So that would mean that the guys you saw were poachers then... Thanks for the input.You might try Danger Point as well.
It is on the Kleinbaai side of Gansbaai but seeing that you come from Pearly Beach it won't make much of a difference.
As you drive towards the light tower you will see a small bay with a couple of houses next to the sea. I personally don't dive but you normally see the guys there.