CONFIG_COMPAT for UM?

CONFIG_COMPAT for UM?

[Johannes Berg]

Hi,

It seems to me that it ought to be possible - at least if the host
kernel has 32/64 compatibility, to extend the same to the UML kernel?

Has anyone ever given this a thought?

I just realized that our test suite is based on 32-bit binaries ... :-)

johannes


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: CONFIG_COMPAT for UM?

[Anton Ivanov]

On 17/09/2019 21:47, Johannes Berg wrote:
> Hi,
> 
> It seems to me that it ought to be possible - at least if the host
> kernel has 32/64 compatibility, to extend the same to the UML kernel?
> 
> Has anyone ever given this a thought?
> 
> I just realized that our test suite is based on 32-bit binaries ... :-)
> 
> johannes
> 
> 
> _______________________________________________
> linux-um mailing list
> linux-um@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-um
> 

I have run 32 on 64 with no issues. In fact, I had to build a product 
prototype using that a while back because the target userspace load did 
not have 64 bit support. Even vector IO works despite the differences 
between 32 bit and 64 bit syscalls for that.

I am not sure we need to do anything here. What in particular do you 
have in mind?

-- 
Anton R. Ivanov
https://www.kot-begemot.co.uk/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: CONFIG_COMPAT for UM?

[Anton Ivanov]

On 17/09/2019 22:54, Anton Ivanov wrote:
> 
> 
> On 17/09/2019 21:47, Johannes Berg wrote:
>> Hi,
>>
>> It seems to me that it ought to be possible - at least if the host
>> kernel has 32/64 compatibility, to extend the same to the UML kernel?
>>
>> Has anyone ever given this a thought?
>>
>> I just realized that our test suite is based on 32-bit binaries ... :-)
>>
>> johannes
>>
>>
>> _______________________________________________
>> linux-um mailing list
>> linux-um@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-um
>>
> 
> I have run 32 on 64 with no issues. In fact, I had to build a product 
> prototype using that a while back because the target userspace load did 
> not have 64 bit support. Even vector IO works despite the differences 
> between 32 bit and 64 bit syscalls for that.
> 
> I am not sure we need to do anything here. What in particular do you 
> have in mind?
> 

Sorry, not thinking. Serves me right answering mails at 11pm.

I was thinking about running 32 bit instances on a 64 bit host.

Pure 32 bit userspace on 64 kernel - yeah... why not. Though to be 
honest, nearly all distributions now have multiarch support so the need 
for this is nowhere near as pressing as 10+ years ago when the first 
AMD64 processors came out.

-- 
Anton R. Ivanov
https://www.kot-begemot.co.uk/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: CONFIG_COMPAT for UM?

[Johannes Berg]

On Wed, 2019-09-18 at 07:48 +0100, Anton Ivanov wrote:
> 
> I was thinking about running 32 bit instances on a 64 bit host.

Yeah, I guess I could run as a 32-bit instance, but I'd rather actually
use a 64-bit instance. If I ever manage to get KASAN running (among
other things) this will totally be required :)

> Pure 32 bit userspace on 64 kernel - yeah... why not. Though to be 
> honest, nearly all distributions now have multiarch support so the need 
> for this is nowhere near as pressing as 10+ years ago when the first 
> AMD64 processors came out.

True, it's not really of that much use these days. I guess it could be
helpful to test the compat code (ioctls) itself.

I've mostly worked around my problem by making the test suite bootstrap
64-bit compatible, which was annoying but doable :-)

johannes


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: CONFIG_COMPAT for UM?

[Anton Ivanov]

On 18/09/2019 08:07, Johannes Berg wrote:
> On Wed, 2019-09-18 at 07:48 +0100, Anton Ivanov wrote:
>>
>> I was thinking about running 32 bit instances on a 64 bit host.
> 
> Yeah, I guess I could run as a 32-bit instance, but I'd rather actually
> use a 64-bit instance. 

Not if you want to run many of them.

The 3-level page tables in 64 bit spin up a whole raft threads making it 
unfeasible. Compared to that the 32 bit uses 5 threads in total per 
instance.

I have tried a couple of times to figure out a way to make the 64 bit 
drop down to two-level for small instances like f.e running OpenWRT with 
64 or 128MB RAM, but it does not work.

To be honest, I was not trying very hard :)

> If I ever manage to get KASAN running (among
> other things) this will totally be required :)
> 
>> Pure 32 bit userspace on 64 kernel - yeah... why not. Though to be
>> honest, nearly all distributions now have multiarch support so the need
>> for this is nowhere near as pressing as 10+ years ago when the first
>> AMD64 processors came out.
> 
> True, it's not really of that much use these days. I guess it could be
> helpful to test the compat code (ioctls) itself.
> 
> I've mostly worked around my problem by making the test suite bootstrap
> 64-bit compatible, which was annoying but doable :-)
> 
> johannes
> 
> 

-- 
Anton R. Ivanov
https://www.kot-begemot.co.uk/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: CONFIG_COMPAT for UM?

[Johannes Berg]

> > Yeah, I guess I could run as a 32-bit instance, but I'd rather actually
> > use a 64-bit instance. 
> 
> Not if you want to run many of them.

Probably not more than a dozen I guess?

> The 3-level page tables in 64 bit spin up a whole raft threads making it 
> unfeasible. Compared to that the 32 bit uses 5 threads in total per 
> instance.

I'm probably missing something, how are threads related to page tables?

I can see how that would use more memory, but where do threads come in?

> I have tried a couple of times to figure out a way to make the 64 bit 
> drop down to two-level for small instances like f.e running OpenWRT with 
> 64 or 128MB RAM, but it does not work.
> 
> To be honest, I was not trying very hard :)

:)

johannes


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: CONFIG_COMPAT for UM?

[Anton Ivanov]

On 18/09/2019 09:25, Johannes Berg wrote:
>>> Yeah, I guess I could run as a 32-bit instance, but I'd rather actually
>>> use a 64-bit instance.
>> Not if you want to run many of them.
> Probably not more than a dozen I guess?

256

>
>> The 3-level page tables in 64 bit spin up a whole raft threads making it
>> unfeasible. Compared to that the 32 bit uses 5 threads in total per
>> instance.
> I'm probably missing something, how are threads related to page tables?

Uml memory management spins up a thread every time it creates a level3 
page table. It is somewhere around kernel/skas

>
> I can see how that would use more memory, but where do threads come in?

It is not memory use, it is actually performance and stability on larger 
machines.

This is less acute now after the ubd transaction bulking, but it used to 
be the case that taskset -c 0 ./run-your-uml-testcase was faster to boot 
a linux system by a factor of 2+ times compared to just leaving all the 
threads wonder to different CPUs.

It was even more fun on larger machines. I had a 5 liner script to kill 
a 2 (or 4) socket Xeon in those days - it died on the 6-9th instance 
being spun up if they were not being assigned specific NUMA node 
affinity. It just keeled over - NMIs due to not being able to keep up 
the cache in sync.

>
>> I have tried a couple of times to figure out a way to make the 64 bit
>> drop down to two-level for small instances like f.e running OpenWRT with
>> 64 or 128MB RAM, but it does not work.
>>
>> To be honest, I was not trying very hard :)
> :)
>
> johannes
>
>
> _______________________________________________
> linux-um mailing list
> linux-um@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-um
>
-- 
Anton R. Ivanov
https://www.kot-begemot.co.uk/


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: CONFIG_COMPAT for UM?

[Johannes Berg]

On Wed, 2019-09-18 at 09:32 +0100, Anton Ivanov wrote:
> On 18/09/2019 09:25, Johannes Berg wrote:
> > > > Yeah, I guess I could run as a 32-bit instance, but I'd rather actually
> > > > use a 64-bit instance.
> > > Not if you want to run many of them.
> > Probably not more than a dozen I guess?
> 
> 256

That's a bit more than I was thinking :-)

> > > The 3-level page tables in 64 bit spin up a whole raft threads making it
> > > unfeasible. Compared to that the 32 bit uses 5 threads in total per
> > > instance.
> > I'm probably missing something, how are threads related to page tables?
> 
> Uml memory management spins up a thread every time it creates a level3 
> page table. It is somewhere around kernel/skas

Ah, ok, thanks for the pointer.

> > I can see how that would use more memory, but where do threads come in?
> 
> It is not memory use, it is actually performance and stability on larger 
> machines.
> 
> This is less acute now after the ubd transaction bulking, but it used to 
> be the case that taskset -c 0 ./run-your-uml-testcase was faster to boot 
> a linux system by a factor of 2+ times compared to just leaving all the 
> threads wonder to different CPUs.
> 
> It was even more fun on larger machines. I had a 5 liner script to kill 
> a 2 (or 4) socket Xeon in those days - it died on the 6-9th instance 
> being spun up if they were not being assigned specific NUMA node 
> affinity. It just keeled over - NMIs due to not being able to keep up 
> the cache in sync.

Fun!

Well, I guess if I run into it I'll know what's going on. For now, not
very likely...

johannes


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: CONFIG_COMPAT for UM?

[Anton Ivanov]

On 18/09/2019 09:47, Johannes Berg wrote:
> On Wed, 2019-09-18 at 09:32 +0100, Anton Ivanov wrote:
>> On 18/09/2019 09:25, Johannes Berg wrote:
>>>>> Yeah, I guess I could run as a 32-bit instance, but I'd rather actually
>>>>> use a 64-bit instance.
>>>> Not if you want to run many of them.
>>> Probably not more than a dozen I guess?
>>
>> 256
> 
> That's a bit more than I was thinking :-)
> 
>>>> The 3-level page tables in 64 bit spin up a whole raft threads making it
>>>> unfeasible. Compared to that the 32 bit uses 5 threads in total per
>>>> instance.
>>> I'm probably missing something, how are threads related to page tables?
>>
>> Uml memory management spins up a thread every time it creates a level3
>> page table. It is somewhere around kernel/skas
> 
> Ah, ok, thanks for the pointer.
> 
>>> I can see how that would use more memory, but where do threads come in?
>>
>> It is not memory use, it is actually performance and stability on larger
>> machines.
>>
>> This is less acute now after the ubd transaction bulking, but it used to
>> be the case that taskset -c 0 ./run-your-uml-testcase was faster to boot
>> a linux system by a factor of 2+ times compared to just leaving all the
>> threads wonder to different CPUs.
>>
>> It was even more fun on larger machines. I had a 5 liner script to kill
>> a 2 (or 4) socket Xeon in those days - it died on the 6-9th instance
>> being spun up if they were not being assigned specific NUMA node
>> affinity. It just keeled over - NMIs due to not being able to keep up
>> the cache in sync.
> 
> Fun!
> 
> Well, I guess if I run into it I'll know what's going on. For now, not
> very likely...

I may get around and see if the APIs have improved a bit to add some 
host CPU affinity and NUMA awareness as an option. Anything else aside, 
it should improve the performance quite a bit.

Last time I looked at it, using them was rather painful - f.e. there was 
no way to say "keep these things together on X", you can only do hard 
assignments to a particular core which is not suitable for a generic use 
case.

> 
> johannes
> 
> 
> _______________________________________________
> linux-um mailing list
> linux-um@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-um
> 

-- 
Anton R. Ivanov
https://www.kot-begemot.co.uk/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: CONFIG_COMPAT for UM?

[Johannes Berg]

On Wed, 2019-09-18 at 09:32 +0100, Anton Ivanov wrote:
> On 18/09/2019 09:25, Johannes Berg wrote:
> > > > Yeah, I guess I could run as a 32-bit instance, but I'd rather actually
> > > > use a 64-bit instance.
> > > Not if you want to run many of them.
> > Probably not more than a dozen I guess?
> 
> 256

I just started 255 (yeah, was using a stupid script for IP addresses)
instances co-scheduling simulated time, and it took only 5:42 min for
all of them to run "sleep 600".

Now, yeah, obviously they weren't doing anything. I assume this whole
thread for the page tables really only kicks in when they're working and
faulting.

Also, the simulated time means only one of them really runs at a time,
so perhaps that even helps ...

Anyway, I don't think I'll need virtual labs as big as this, so ...

johannes


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um