Yes, you are very correct in saying that taking coral worm out , does damage the black muscle bed.
However i must also state that if taken out correctly by just lifting the large greenbrown sea grass found close to the black muscle on the rock, plenty of coral worm can be found there without touching the young muscles.
This grass grows back extremely fast and a responsible fisherman must also know not to remove any bait over and above a limit for a days fishing. Another point to remember is that large and only large black muscle can be picked for cooking or bait purposes. By 'picking' off the muscle you actually not destroying the bed, infact it gives the smaller one space to grow. There must also always be responsible bait collecting, wether it be coral worm or wonder worm, the same sustainable resposible enviromental rules always apply.
OH and EVERY TYPE OF BAIT COLLECTED HAS A LIMIT. PLS FAMILIARISE YOURSLVE WITH MARINE AND COASTAL MANAGEMENT REGULATIONS AT ALL TIMES WHEN COLLECTING BAITS.