As a single man, you're one of very many, and part of a group who is usually frowned upon, as some single men just don't know how to behave.
Meeting swingers should not be the major issue, clubs will let you in if you dress and behave accoridngly, though couple nights will be out-of-reach for you. Then, don't be pushy, don't be salty, behave like a human being and remember that swingers are regular people. It might not be "all the fun", but you'll get an idea of the scene. Have fun!