Just because you can mmap() a file on an NFS server in New Zealand doesn't mean that you should have the same expectations of memory access to that file as you do to of a pointer from alloca(). Even if the primitives look the same, you are crossing significant boundaries, and those do not come without a compromise and a penalty.
Anyway, one of the great things about winsys (there are
some! trust me) is we don't need to be as hopelessly general
as for game engines, nor as hyperoptimised. We place strict
demands on our clients, and we literally kill them every
single time they get something wrong in a way that's visible
to us. Our demands on the GPU are so embarrassingly simple
that you can run every modern desktop environment on GPUs
which don't have unified shaders. And on certain platforms
who don't share tiling formats between
texture/render-target/scanout ... and it all still runs fast
enough that people don't complain.
Ignoring everything below since that is the display pipeline I'm not
really interested in. My concern is how to get the buffer from the
client to the server without allowing the client to get the server
into trouble?
My thinking is still to use timeouts to acquire texture locks. E.g.
when the compositor needs to access texture it grabs a lock and if
that lock isn't available in less than 20ms whoever is holding it is
killed hard and the lock given to the compositor.
It's perfectly fine if a process has a hung queue, but if it tries
to send buffers which should be filled by that queue to the
compositor it just gets a corrupted window content.
Kill the client hard. If the compositor has speculatively queued sampling against rendering which never completed, let it access garbage. You'll have one frame of garbage (outdated content, all black, random pattern; the failure mode is equally imperfect, because there is no perfect answer), then the compositor will notice the client has disappeared and remove all its resources.
It's not possible to completely prevent this situation if the compositor wants to speculatively pipeline work, only ameliorate it. From a system-global point of view, just expose the situation and let it bubble up. Watch the number of fences which failed to retire in time, and destroy the context if there are enough of them (maybe 1, maybe 100). Watch the number of contexts the file description get forcibly destroyed, and destroy the file description if there are enough of them. Watch the number of descriptions which get forcibly destroyed, and destroy the process if there are enough of them. Watch the number of processes in a cgroup/pidns which get forcibly destroyed, and destroy the ... etc. Whether it's the DRM driver or an external monitor such as systemd/Flatpak/podman/Docker doing that is pretty immaterial, as long as the concept of failure bubbling up remains.
(20ms is objectively the wrong answer FWIW, because we're not a hard RTOS. But if our biggest point of disagreement is 20 vs. 200 vs. 2000 vs. 20000 ms, then this thread has been a huge success!)
Cheers,
Daniel