If I remember it correctly, Ghamidi Sahib said it is just as any other type of knowledge; just like we predict weather. I think the problem lies in the fact that almost all of the astrologers are lying and making money based on lies. Even if they are really good at Astrology, this is not an exact science. They may be sometimes correct, but if we let them dictate our life based on their predictions, then it creates a serious problem in the society and the society will not be able to function properly. If it was nearly a perfect science with very specific predictions, then perhaps following the science will not be an issue.
Still think about it. If an astrologer predicts that, for example, you will get into an accident today so do not step outside. Following that prediction, you don’t step outside and never get into an accident. Doesn’t it mean that it was already in your fate that you will go to an astrologer and based on his prediction you will not step outside. So, the astrologer never changed your fate and it was never in your fate that you will get into an accident.