Re: advice sought: practicality of SMP cache coherency implemented in assembler (and a hardware detect line)

[Luke Kenneth Casson Leighton <luke.leighton@gmail.com>]

On Mon, Mar 28, 2011 at 11:18 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>>  ok - well, having thought about this a little bit (in a non-detailed
>> high-level way) i was sort-of hoping, as alan hinted at, to still do
>> SMP, even if it's slow, for userspace.   the primary thing to prevent
>> from happening is to have kernelspace data structures from
>> conflicting.
>>
>>  i found kerrigan, btw, spoke to the people on it: louis agreed that
>> the whole idea was mad as hell and was therefore actually very
>> interesting to attempt :)
>>
>>  as a first approximation i'm absolutely happy for existing pthreads
>> applications to be forced to run on the same core.
>
> The underlying problem across a cluster of nodes can be handled
> transparently. MOSIX solved that problem a very long time ago using DSM
> (distributed shared memory). It's not pretty, it requires a lot of tuning
> to make it fly but they did it over comparatively slow interconnects.

 hmmm, the question is, therefore: would the MOSIX DSM solution be
preferable, which i presume assumes that memory cannot be shared at
all, to a situation where you *could* at least get cache coherency in
userspace, if you're happy to tolerate a software interrupt handler
flushing the cache line manually?

 it had occurred to me, btw, that it would be good to have separate
interrupts for userspace and kernelspace.  kernelspace would have a
"serious problem occurred!" interrupt handler and userspace would have
the horribly-slow-but-tolerable cache-flush assembly code.

 is that preferable over - faster than - the MOSIX DSM solution, do you think?

 l.

 p.s. alan am not ignoring what you wrote, it's just that if this goes
ahead, it has to be done _quickly_ and without requiring
re-verification of large VHDL macro blocks.  of the two companies
whose cores are under consideration, neither of them have done SMP
variants (yet) and we haven't got the time to wait around whilst they
get it done.  so these beautiful and hilarious hacks, which can be
tacked onto the outside, are what we have to live with for at least oo
18 months, get a successful saleable core out, that pays for the work
to be done proppa :)