The main issue is that the length of a day is not actually constant. Leap second occur (in either direction) which mean that a day is sometimes one second shorter or longer. Timezones and DST also can make a day a whole hour longer or shorter.
Seconds are a unit for physical measurement. They're always the same length. Minutes, days, weeks, months, years, etc are imprecise shortcuts that are convenient for our society but this convenience sometimes comes at the price of being bonkers units from the physics standpoint.
