Here's what I think.
A passion is something that is stable throughout my life, so I would say I have a passion for marginalized people, helping, and listening to authentic and vulnerable music.
A special interest is something that I am almost obsessed with to the point that I learn about it with disregard to my regular bodily needs (e.g. urinating, eating, or sleeping).
When they match up, it's great because I will create something fantastic. However, I can neglect important areas of my life, so I need to be considerate of them too.
It is definitely tough. I would say they are mostly the same thing, but called something different based on how other people feel about them. At least in my experience. I'm no professional though.
A special interest becomes part of my identity, and I am eager to interact with it a majority of the time. A passion does not necessarily meet those criteria.