If one is the head of the family then one should prioritize the safety of their family first because for that you are definitely answerable to Allah.
It is the duty of the scholars and those in power who can speak, that should speak the truth even if it has consequences. Not the common Muslim. For them, one should be careful:
[Believers! Those among you] who disbelieve after believing, if they are compelled and their hearts are content on belief, then there is no penalty for this……. (Qur’an 16:106)
As for the fortress part. There is context behind it. It is referring to people at time of Messenger (PBUH) on which Allah was putting them through a special test, time when fighting was necessity for survival and the munafiq were complaining even in time of when one should be thinking about self-defence at the very least, not complaints:
Then, after this sorrow, God once again let peace descend upon you in the form of a sleep which was overtaking some of you and there were some who were fearing for their lives. They were engaging in totally false thoughts of the age of jāhiliyyah about God.They were saying: “Do we also have any say in these matters?” Tell them: All matters are in the hands of God. They are concealing in their hearts what they do not reveal to you. They say: “If we had any say in this matter, we would not have been slain here [like this.”] Say: Had you stayed back in your homes, those of you who were destined to be killed would have gone to their places of death nevertheless. [So, once again try to understand the fact that] all this happened because God wanted to make you distinct from one another….. (Qur’an 3:154)
However, when war was made obligatory upon them, their situation now is that one group from among them fears people the way God should be feared or even more. They say: “Lord! Why have you made this war obligatory upon us? Why did you not give us some more respite?” Tell them: Very little is the provision of life, and for those who adhere to piety, the Hereafter is much better for them and there you shall not face injustice the slightest.
As for death, wherever you are it is bound to come to you in all circumstances even if you are in strong and towering fortresses.[These people accept your prophethood by their tongue] and if they are blessed with some success, they say: “This is from God.” And if they are inflicted with loss, they say: “It is because of you.” Say: Everything is from God! [This is because whatever happens, happens with His permission.] So, what has come over these people that they are not prepared to understand anything?(Qur’an 4:77-79)
These verses are not for the common Muslim, these verses are for those who were among the Messenger of God (PBUH) and Allah was separating true believers from the rest so the punishment after Itmam-e-Hujjat may follow as it does with every Messenger of God.