Temperatures are fine tho, it's at around 82-83°C.
+80C is just asking for trouble if you ask me... I wouldn't be comfortable with anything above 70C.
But, than again, I don't own a Pi, so I wouldn't know. Maybe working in the middle of the second half of the 100C mark is normal for a Pi 🤷.
Does compiling natively with extra optimizations make a noticable difference?
In some cases, yes. Codecs are known to greatly increase performace if adequate CPU optimizations are used. Don't know what ARM uses, but AVX, AVX2, SSE3, etc. on Intel CPUs does significantly increase performance in codecs. It's mostly noticable when encoding, but it helps decoding as well (not to the extent that it helps in encoding though, but adequate switches when building can reduce CPU usage when decoding by about 10 to 20%... which should help a lot in your case).
But, you have to compile the software with those switches, otherwise, it won't use the optimizations 🤷.
If you can, also have a look at the build scripts of the AV1 binary/library, see if they used CPU specific optimizations or not. Maybe they did, maybe they didn't 🤷.