People lived in peace with other people much of the time, but we never lived in peace with nature itself. Drought, flood, fire, predators, disease, infection, and many many other natural threats to our human rights have always existed. Furthermore, natural scarcity meant that even other people could become a threat - people from a different gens/tribe could decide that it's better if the other gens/tribe dies than if everyone eats a little less to make the food last longer. Nature was humanity's first enemy, before we divided up into class society and started to war with each other.
I will say "natural" is misleading, because humans are natural too. That's why I said in order for "natural" to mean anything at all then it must be distinct from human choices. If we include humans into nature, then natural rights don't mean anything because everything is natural including human oppression.