Re: advice sought: practicality of SMP cache coherency implemented in assembler (and a hardware detect line)

[Alan Cox <alan@lxorguk.ukuu.org.uk>]

>  hmmm, the question is, therefore: would the MOSIX DSM solution be
> preferable, which i presume assumes that memory cannot be shared at
> all, to a situation where you *could* at least get cache coherency in
> userspace, if you're happy to tolerate a software interrupt handler
> flushing the cache line manually?

In theory DSM goes further than this. One way to think about DSM is cache
coherency in software with a page size granularity. So you could imagine
a hypothetical example where the physical MMU of each node and a memory
manager layer comnunicating between them implemented a virtualised
machine on top which was cache coherent.

The detail (and devil no doubt) is in the performance.

Basically however providing your MMU can trap both reads and writes you
can implement a MESI cache in software. Mosix just took this to an
extreme as part of a distributed Unix (originally V7 based).

So you've got

Modified: page on one node, MMU set to fault on any other so you can
	  fix it up

Exclusive: page on one node, MMU set to fault on any other or on writes
	   by self (latter taking you to modified so you know to write
	   back)

Shared:    any write set to be caught by the MMU, the fun bit then is
	   handling invalidating across other nodes with the page in
	   cache. (and the fact multiple nodes may fault the page at once)

Invalid:   our copy is invalid (its M or E elsewhere probably), MMU set so
	   we fault on any access. For shared this is also relevant so
	   you can track for faster invalidates

and the rest is a software problem.