Some of that is content categorization in the eyes of the all-seeing algorithm. Let's say you upload a type of content "A" that gets big views but you've been uploading a type of content "B" that gets small views for a while. The youtube algorithm will aggressively try to grow content A and massively deprioritize content B, even among other channels that produce content B.
A guy I know who does youtube/twitch had to create a second channel for his content B because it would get sub-1k views when he would get tens of thousands of views on his content A. Just by uploading somewhere else he started to get higher view counts.
Exactly why that happens isn't known, but a common theory is that youtube wants to push what it knows works. They have no real reason to give your content B a chance because they know content A will sell. And they do this even though this outcome was the result of a feedback loop.
That may be a contributing factor, my most viewed stuff are a couple of anime clips sitting at hundreds of thousands of views, but niche is still the #1 explanation in my eyes, as even the big and successful Master Duel channels stay at only a couple thousand views per upload. There's just not enough people interested in certain types of content.
