So I wrote down some of my thoughts below, if you want to chat about it, feel free to leave a comment or DM me. (I don't have Matrix, sorry)
So the core component of any social network is the network effect. In short, the reason we use Lemmy is because others also use Lemmy. We wouldn't use it if there was nobody here (with thanks to Spez).
Indeed, that's why people still use Reddit, Facebook, or any social medium: they don't care about the platform, but about the people.
If Threads federates with Lemmy, they will instantly dominate the entire space: the most popular communities and most of the user activity will all be over there. (There are already 100M Threads users compared to under 2M Lemmy users, so this is essentially inevitable.)
If Meta would then later decide to defederate, the majority of Lemmings will be cut off from their communities. If in the meantime those communities had become important to them, their hand may be forced to sign up for Threads. Lemmy will survive, but it won't be the same for most of its users.
Essentially, we have a choice to make: either we welcome Threads and accept that we will be at the whims of Meta for >90% of our communities and content (given their massively higher user count), OR we choose to be independent at the cost of having smaller communities and less content.
I would argue though, that having smaller communities may even be a pro rather than a con. I've seen many comments that the smaller scale of the Fediverse feels like a breath of fresh air compared to Reddit (and personally I agree with that).