After being able to identify xfsettingsd as a potential cause (ignoring whether
or not the monitor was actually Disconnected/Connected), I had a look at its
bug tracker - and https://bugzilla.xfce.org/show_bug.cgi?id=14096 looks
extremely suspicious - after applying the patch and turning on/off the monitor
10 times in a row (waiting 10 seconds inbetween), I haven't had a single
desktop reconfiguration (the patch makes complete sense too WRT this problem),
so I'm going to sit on this for a week to make sure its actually solved.

There seem to be a number of other people complaining of this issue over there
too...