Just plan your trip. Make sure you don't have to paddle against the wind in the afternoon when you have to leave. That said, I've been there that I paddled from the island to Kotze street into a howling SE and therefore I know it is doable. Uncomfortable maybe, but doable. We do it for the exercise, don't we? And that is a good workout.
I nearly forgot, a good anchor system is of the utmost importance. A small foldable anchor did not work for me the last time, I want to try my new "bruce" anchor on Saturday.