Is it possible for the bike to fall off the rollers, sending everything crashing into the nearest wall?
No, if you fall off the rollers, your wheels just stop. Bicycle wheels don't have anywhere near enough mass for their momentum alone to move your body.
I have heard that it is VERY hard to get started and then very quickly becomes thoughtlessly easy
you're absolutely right. I searched it before making that comment and there's one random dude who did exactly that, he even wondered himself if anyone has ever done that.