If this is something of a serious concern to you and our epistemological approach does not make sense to you where Quran is given supreme authority and all ahadith are studied in the light of Quran, then you can follow the Hadith in literal sense and keep your trousers above your ankles. However, the hadith talks about Izar (which was tehband at that time) but not about other legwear like trousers, Pajamas etc. So such intricacies will create more trouble for you when following hadith in their literal meaning.
Ghamidi Sahab writes while concluding this topic:
“The issue of isbal-i izar is no different. It was very common among the arrogant in the pre-Islamic Arab society to wear a long shirt, let the loose end of their turban hang below their back and let their legware (izar) dangle so far below the ankles that half of it would drag behind on the ground. In Arabic, this is called isbal. The Prophet (sws) showed his great dislike for this, and once remarked that the Almighty would not like to see the person on the Day of Judgement who walked while arrogantly dragging his leg-cloth (tehband). All narratives regarding izar relate to this appearance.
It can however be said about the tehband that if it is allowed to dangle below the ankles, then it to some extent resembles the appearance of the arrogant just discussed; so care must be exercised even if the purpose to make it dangle is not arrogance. Thus this can be said about the tehband. However, it is a fact that this resemblance is only reflected in the tehband; it has no similarity with a shalwar, a pajama and a trouser.”