When you say 720x400 "kills" the Xserver in scaling mode = full ... what do you
mean exactly? Xorg process dies? Kernel hangs?

I don't remember where the scaling is done, but IIRC it's in the kernel, so we
could be driving the scaling engine incorrectly in certain cases.