I keep hearing that it is "prosperity" that made fertility drop, but hasn't the expectation that women have to work also make having children difficult ?

If you make a society where everyone HAS to work in order to survive but also provide no public childcare, schooling and healthcare then obviously people are not going to be able to afford to have babies.