This is why our kingdom was lost and our Temple was destroyed and why we were brought to this; for our fathers sinned and are no more because they found many books dealing with these themes of the star gazers, these things being the root of idolatry, as we have made clear in Laws Concerning Idolatry. They erred and were drawn after them, imagining them to be glorious science and to be of great utility. They did not busy themselves with the art of war or with the conquest of lands, but imagined that those studies would help them. Therefore the prophets called them “fools and dolts” (Jer. 4:22).
A number of commentators insisted that this letter must be interpreted in light of traditional doctrine about the Destruction being punishment for the cardinal sins. As such, Rambam is not saying that it was actually the lack of military strength (resulting from pursuing astrology) that caused the Churban, but merely pointing to a problem resulting from astrology. I think that this is an exceedingly forced way of reading the above paragraph. Moreover, there is a fundamental misunderstanding here about Rambam's worldview. Rambam held that there is no such thing as arbitrary reward and punishment, which God inserts into the world. Rather, the mitzvot are the path to intellectual, moral and societal perfection, while aveirot detract from that. To the extent that there is reward and punishment, it is the natural consequence of one's actions. Thus, Rambam's view is that the people were pursuing astrology - which he explains to be the root of idolatry - and as a natural consequence, did not engage in the material, worldly efforts that would have helped them have a defensible kingdom. Rambam is not arguing with the idea that the Destruction was a punishment for idolatry; rather, he is explaining what, in his view, this actually means.
In general, to understand Rambam's views on any topic requires a thorough grasp of his overall worldview, which was radically different than anything we have been taught in yeshivah. A careful study of The Guide for the Perplexed, with the help of those that have unlocked its difficulties, is indispensable for this. In the future, I plan to discuss Rambam's view of the different types of harm that befall man and their causes. For now, I would like to point out that Rambam's view on reward and punishment in general were extremely different from conventional rabbinic doctrine. Prof. Menachem Kellner has a discussion of this topic in the appendix to his book Must a Jew Believe Anything? which he kindly consented to make freely available. You can download it here; please read it before commenting on this post!