Might be related to #3.
Recording to 4k first worked, then I tried 8k60 (just to see what happens) which failed, and now 4k continues to fail (after Unity restart).

My understanding of that message would be that the GPU is busy encoding and thus can't keep up with frames. but since 4k first worked and then didn't I think something might be stuck on the unmanaged side.