Yes, the server is authoritative and the physics are simulated on the server, with the resulting transforms pushed to the clients.
The problem is I'm using Unity's wheel colliders, which are known to be CPU intensive, and with 16 of them in one server process that's already 5-7% CPU usage. I tried another physics package which is supposed to be more optimised, but I couldn't get the driving feel that I was looking for.
The other problem is, as an indie developer, paying for dedicated servers on a monthly basis in the can work out to be non-profitable in the long-run.