So I think this might have something to do with something Xorg is doing because I've not had it happen while gaming for many hours since just seeing if it happened on wayland on a whim. I now have 21 hours of uptime with no random crashes.