Well, because straight earth as in mudbrick (or concrete, normal bricks etc) is not insulation. That's thermal mass. It stores energy. Insulation (like strawbales) slows heat movement. So you need insulation on the exterior and thermal mass on the interior for a properly thermally regulated building.
At 2 inches thick of limestone, you can sure bust it up with a sledgehammer or similar. A punch won't do much more than hurt your hand. Still, if you take tools to the majority of homes they break quite fast.
Couldn't say houw tall they can get but from memory I think I've seen 3 stories? Over that you're talking more full on construction. I've seen a 4 story using super bales that was in an commercial carpentry shop
