I think there are a lot of factors.
My DND group spent a while talking about the mechanics of DND, character knowledge, and player knowledge. Specifically, let's say there's a big knight dude. The dm knows he makes four attacks at +9 for 1d10+7 each. What do the players know?
The dm can say he looks tough. A master of dueling. But what does that mean? Is the dm just being flavorful and this is a "level appropriate challenge"? Or is he trying to tell us he's like a 17th level fighter? Two DMs might use the same exact phrasing and mean different things.
This problem is kind of aggravated by dnd's rules. There's not a lot of unified, coherent systems underpinning it. NPCs don't use the same rules as PCs, and don't really have any rules at all. As a player you can't really reason about what you're facing.
Compare CofD, because I've been playing that lately. Most things are rated 0-5. If the gm says "he's a professional [whatever]" you can be pretty confident the guy has about six dice, and from there you can make an informed decision. Your defense is 3 so he'll probably hit, and with a big gun and your seven health that's risky but if you bump your defense first etc etc.
DND doesn't typically give you that information outright, and there aren't really mappings between common words and power levels. With bounded accuracy you could try to be like newbie is +3, decent is +6, pro is +9, etc, but there's nothing for number of attacks or damage dice.
So it's hard to communicate to your players what they're looking at.
Plus, a lot of people do a sort of oblivion style scaling. The king's guard in the first chapter is like +7 to hit so he's tough and scarier than the PCs, but in the second act (or just a game that starts higher) he might be +10. Not a fan of scaling to the PCs.
Even moving away from human foes, what are the appropriate stats for a cave troll? You can infer it's strong so that's probably a +5 or more, but there's no way to know it's number of attacks or damage dice. How am I supposed to gauge if I can win this fight without that? There's a huge difference between 1 attack and four.
Now, some players are definitely at fault. Not paying attention, not understanding the information they do have, wanting a popcorn game where they just win when the dm said that's not what they're doing, etc. But the game itself also doesn't give the players enough information to make informed decisions , so of course sometimes they're going to be like "yeah the dm is being dramatic let's fight this troll WHAT DO YOU MEAN HE HAS FOUR ATTACKS?"
It's interesting because not knowing exactly how powerful something is feels more realistic. I think the uncertainty has it's place as a narrative device. One way to mitigate the problem you're describing would be more liberal use of passive checks, like if the party is facing down a troll, a nature check might reveal it's a powerful ancient beast. For your 17th level fighter example, a good insight check (or passive insight) would let the DM say 'dont mess with this guy' without it breaking immersion too much.
We talked about doing exactly that in my group. The problem is this clashes with the "don't hide essential information behind a check" guideline.
Like, you can say a nature check of 15 tells them some information about the troll. Let 's leave aside the problem of "well the dm is just being dramatic when he said it cuts known knights like paper" problem and say you tell the players some usable ballpark numbers. Cool, they can make an informed decision that something that swings at +10 and has an average dpr of 25 is too much.
But what if they fail the check? Do you shrug and say they don't know? Do you just let them blend themselves with this troll? That puts us back to square 1, really.
A working solution might be degree-of-success (which 5e tends to underuse). A very high check might give you the actual to-hit, damage dice, ac, but even a very low check might give something like "it can easily exceed your HP per round in damage" or something vague but mechanically grounded.
You would need to be clear when you're giving actual information and not fun flavor.
If you want your players to run from a fight, do not make it their choice. You have designed a fight that they can not win and expect them to know they can not win when it is impossible for them to learn that information unless you tell them thus defeating the point of making the fight in the first place. So unless you're okay with killing members of the party just so they can run away, the only way to ensure they will leave is if you don't make it their decision.
It seems a pretty safe assumption that players never run away until it's too late.
Multiple hints. Multiple. It doesn't matter how many, but the simple fact that it's always more than one is what boggles the damn mind.
I mean, over the years, I've come to terms with never really knowing how, but: why? That at least can be delved into, I think. What is it that inspires such courses of action? Is it blithe & blissful ignorance, or actually something more like cogently willful disregard? Is it some collective allure of pushing one's luck, or some deep-seated need to watch it all burn? 😅
I think it boils down to the fact that scenarios and encounters are designed to be the adventure. Extreme circumstances aside, to a player, running away feels like avoiding the adventure or worse- failing it.
As DMs, we know that hints are left for periphery details. A DM shouldn't be designing an encounter to be certain death, and they also have the power to have the players fail forward into new scenarios even if it becomes one. If an encounter must be deadly, then there should be no hints, you should be saying flat out this is not an encounter where the goal is victory through combat. Combat will mean death. Their characters can feel it in their bones.
DMs ought to avoid making encounters where the best course of action is to not engage with it on the players terms. And at worst, have more creative fail states than a TPK.
All fair points, though more could be said for player contribution to such scenarios; it's not always up to the DM whether death is on the line. Everyone, and I mean everyone , knows: "don't bet against a Sicilian", et al. Beyond that, encounters where the best course of action is not metagaming is often the most satisfactory option when creating scenes for knowledgeable players. I agree, though, that a TPK should always be a surprise to everyone at the table and a mutual decision to not pull the ejection ripcord on the deus ex/mcguffin.
