I don't have any experience with the swinging lifestyle, but I can say that you're right that it will be an uphill battle. I've never seen a swingers club that didn't have a 'no single males' rule in place. If they allowed them then unfortunately that's the only people who would ever be there. Finding a like-minded female partner is certainly going to be the easiest way to get into that specific community.
You might have luck on FetLife and connecting with some poly-friendly people there as a start if you're just looking for people to talk to, maybe go to a local munch. You can also keep an eye on classified sites or Reddit for couples looking for a bull. Competition can be fierce though, unless you're very well endowed. All of this kind of depends on what region you live in though.
As a single man, you're one of very many, and part of a group who is usually frowned upon, as some single men just don't know how to behave.
Meeting swingers should not be the major issue, clubs will let you in if you dress and behave accoridngly, though couple nights will be out-of-reach for you. Then, don't be pushy, don't be salty, behave like a human being and remember that swingers are regular people. It might not be "all the fun", but you'll get an idea of the scene. Have fun!
There is an abundance of straight guys in the scene when we were playing but that's who we got with the most. Sometimes bi guys but mostly straight guys. Just be nice, there are couples who want a 3rd for MFM, just not quite as many as there are guys looking to be that 3rd.
A lot of guys seem to have friends (female) to bring along too, even if not girlfriends. We weren't so into that as I am not bi and husband got off more on sharing me than vice versa, but presumably it helps if they only like to do swap.
If you are hung big, people look for that too, a lot of guys (my husband included) like to watch their partners with guys hung big, which always seems to be defined as bigger than they are. If you are hung and fit, there are more guys than I thought interested in cuckolding play too, that wasn't exactly our thing but it's popular.
And if course if you are looking just for one couple to play with on the regular, keep your eyes and ears open when you are with open-minded friends, some people aren't swingers per se but might be open to sharing. Yes the trope is that couples only want bi ladies but that's not really the only thing people like to do, the only swingerly stuff I like is MFM or MMF, love women as friends but not into women sexually at all, and husband liked that best too, particularly straight guys but open to bi guys (receiving oral from guys, basically).
Sad to have to mention it but keep in mind it's a shallow scene, looks matter a lot.
You nailed it from my perspective at least. I like to fantasize with my SO about her getting fucked by a hung guy. We don't do it in person but it's a common fantasy, at least when we fantasize together.
It isn't just that the guy is bigger than me. It also has to do with the shape and general appearance outside of size. But it does play a part in that a cock smaller than mine wouldn't do it for the fantasy very well.