I happen to be using debian packaging. I think I found the root cause of the problem, which is conflicting dependencies of the package xserver-nodm-init (xserver-common) with other x packages.
xserver-nodm-init happens to be in 2 versions in 2 places, version 2 is in meta-openembedded/tree/meta-oe/recipes-graphics/xserver-nodm-init and version 1 is in meta/recipes-graphics/x11-common/
version 1 does not depend on xserver-common and thus will install. debian packages work this out, it looks like the other don't do any checks or don't care.
I have fixed it in my local.conf by forcing the version of xerver-nodm-init:
PREFERRED_VERSION_xserver-nodm-init = "1.0"