Re: PowerPC KVM build directions

Re: PowerPC KVM build directions

[Hollis Blanchard]

(CCing kvm-ppc list.)

Great, so you've managed to get Linux running under KVM on e500?

On Wednesday 01 April 2009 13:58:00 Rahul Kulkarni wrote:
> Liu/Hollis, I have just started digging into the kvm/qemu code..dont have 
specific questions yet..but for a head-start ..I'd appreciate you give me any 
pointers on  places in the code to look which are linux specific which could 
potentially need to reworked for netbsd..

I guess step 1 is to get NetBSD loaded. You have two options:
1. load your bootloader/firmware and have it load your kernel.
2. bypass the bootloader/firmware and have qemu load your kernel directly.

Option 1 requires emulating enough system hardware in qemu to make your 
bootloader/firmware happy. If your firmware expects a lot of IO devices to 
exist that qemu doesn't yet emulate, or doesn't emulate completely, you will 
have IO emulation development to do. (If your firmware offers runtime services 
to your kernel, this is really your only option, but most embedded firmwares 
don't.)

Option 2 requires that qemu set up the initial VM state exactly how firmware 
would have. That means loading the same kernel executable into the same memory 
location, same initial register state, initial TLB mappings, same (emulated) 
system hardware state, etc.

For Linux, we used option 2. After loading the ELF file into guest memory, 
qemu creates the device tree (hierarchical data structure describing the 
physical system) and passes its address to Linux in a GPR, which is a job 
usually performed by u-boot. The decision really depends on the functionality 
and complexity of your firmware.

-- 
Hollis Blanchard
IBM Linux Technology Center

RE: PowerPC KVM build directions

[Rahul Kulkarni]

Thanks Hollis. See inline..

-----Original Message-----
From: Hollis Blanchard [mailto:hollisb@us.ibm.com] 
Sent: Wednesday, April 01, 2009 1:07 PM
To: Rahul Kulkarni
Cc: Liu Yu-B13201; kvm-ppc@vger.kernel.org
Subject: Re: PowerPC KVM build directions

(CCing kvm-ppc list.)

Great, so you've managed to get Linux running under KVM on e500?
Rahul>> Yes.  I can run Linux under kvm on a e500 based platform

Option 2 requires that qemu set up the initial VM state exactly how firmware 
would have. That means loading the same kernel executable into the same memory 
location, same initial register state, initial TLB mappings, same (emulated) 
system hardware state, etc.

For Linux, we used option 2. After loading the ELF file into guest memory, 
qemu creates the device tree (hierarchical data structure describing the 
physical system) and passes its address to Linux in a GPR, which is a job 
usually performed by u-boot. The decision really depends on the functionality 
and complexity of your firmware.

Rahul>>I'll start with option 2.  Initial goal would be to run a NetBSD guest on an emulated e500/8544 platform - which Liu has added support for recently in qemu/kvm to run as a linux guest. Our eventual goal is to emulate an e500/8548 CDS based platform.. Since KVM supports a NetBSD 4.0  guest (I think) and 8544/e500 emulation is already present in qemu -- theoretically the first part should work...but I recall Liu mentioning that there might be some OS specific quirks present in the port...and that was what I my question was hinting at earlier.. 
-- 
Hollis Blanchard
IBM Linux Technology Centera

RE: PowerPC KVM build directions

[Liu Yu-B13201]

> -----Original Message-----
> From: Rahul Kulkarni [mailto:rkulkarn@force10networks.com] 
> Sent: Thursday, April 02, 2009 7:47 AM
> To: Hollis Blanchard
> Cc: Liu Yu-B13201; kvm-ppc@vger.kernel.org
> Subject: RE: PowerPC KVM build directions
> 
> Thanks Hollis. See inline..
> 
> -----Original Message-----
> From: Hollis Blanchard [mailto:hollisb@us.ibm.com] 
> Sent: Wednesday, April 01, 2009 1:07 PM
> To: Rahul Kulkarni
> Cc: Liu Yu-B13201; kvm-ppc@vger.kernel.org
> Subject: Re: PowerPC KVM build directions
> 
> (CCing kvm-ppc list.)
> 
> Great, so you've managed to get Linux running under KVM on e500?
> Rahul>> Yes.  I can run Linux under kvm on a e500 based platform
> 
> Option 2 requires that qemu set up the initial VM state 
> exactly how firmware 
> would have. That means loading the same kernel executable 
> into the same memory 
> location, same initial register state, initial TLB mappings, 
> same (emulated) 
> system hardware state, etc.
> 
> For Linux, we used option 2. After loading the ELF file into 
> guest memory, 
> qemu creates the device tree (hierarchical data structure 
> describing the 
> physical system) and passes its address to Linux in a GPR, 
> which is a job 
> usually performed by u-boot. The decision really depends on 
> the functionality 
> and complexity of your firmware.
> 
> Rahul>>I'll start with option 2.  Initial goal would be to 
> run a NetBSD guest on an emulated e500/8544 platform - which 
> Liu has added support for recently in qemu/kvm to run as a 
> linux guest. Our eventual goal is to emulate an e500/8548 CDS 
> based platform.. 

Guest 8544 is just a name of e500 VM.
Actually it hasn't implement most of 8544's devices and it can have
device which real 8544 doesn't have.
I once had named it to be 85xx...

The reason I chose 8544 as it's name is that the e500 kvm is first
developed on 8544 board 
and 8544 has pci controller which virtual devices in qemu only connect
with...

So if that name dosen't bother you, you can just add the device you want
to support into the guest 8544.

> Since KVM supports a NetBSD 4.0  guest (I 
> think) and 8544/e500 emulation is already present in qemu -- 
> theoretically the first part should work...but I recall Liu 
> mentioning that there might be some OS specific quirks 
> present in the port...and that was what I my question was 
> hinting at earlier.. 

There must be a lot of differences between Linux and NetBSD.
So the design of kvmppc may not be careful considerate.

The main trick is to hijack the system interrupts.
You should check if this part of code cater to NetBSD.
Context switch needs to be taken care as well.

At least there is one thing I can point out.
See comments in the file booke_interrupts.S  line 195. :-)

RE: PowerPC KVM build directions

[Hollis Blanchard]

On Thu, 2009-04-02 at 10:52 +0800, Liu Yu-B13201 wrote:
> 
> > Since KVM supports a NetBSD 4.0  guest (I 
> > think) and 8544/e500 emulation is already present in qemu -- 
> > theoretically the first part should work...but I recall Liu 
> > mentioning that there might be some OS specific quirks 
> > present in the port...and that was what I my question was 
> > hinting at earlier.. 
> 
> There must be a lot of differences between Linux and NetBSD.
> So the design of kvmppc may not be careful considerate.
> 
> The main trick is to hijack the system interrupts.
> You should check if this part of code cater to NetBSD.
> Context switch needs to be taken care as well.
> 
> At least there is one thing I can point out.
> See comments in the file booke_interrupts.S  line 195. :-)

Liu, you're talking about BSD as the *host*. Rahul is asking about BSD
as the guest.

Rahul, one major quirk we exploit is that Linux does not use the MSR[AS]
bit at all. One way that bit could be used is to give 32-bit userspace a
separate 4GB address space from the kernel. Instead, Linux puts both
kernel and userspace into the same 4GB address space (with Linux
mappings above 0xc0000000 and user mappings below). If NetBSD uses
MSR[AS]=1 for userspace (which I think is what the hardware architects
envisioned), you're going to have a lot of MMU fun.

Another potential issue could be the initial environment (described
earlier as option 2) not being what BSD expects. Do you use u-boot? You
can see the initial environment set up in kvm_arch_vcpu_setup() in KVM
and mpc8544ds_init() in Qemu.

Does NetBSD use flattened device trees at all? KVM (Qemu) supplies a
stripped-down device tree to the guest so that the guest won't try to
access IO devices not currently emulated by qemu. If BSD has a hardcoded
device configuration system (e.g. "we built for 8544, therefore we
always have the following SoC devices") that will be an issue.

As a concrete example, qemu doesn't emulate the TSEC ethernet
controller. You need to either convince your guest not to try to use the
TSEC (and use e1000 or some other qemu-supported device instead), or add
just enough TSEC emulation to qemu to make your guest happy. That could
be as simple as always reporting "link down" so the guest doesn't try to
use it.

Please keep us posted about any issues you encounter. Also, documenting
the hurdles and how you overcame them might be an interesting conference
paper, if you're into that sort of thing. :)

-- 
Hollis Blanchard
IBM Linux Technology Center

RE: PowerPC KVM build directions

[Rahul Kulkarni]

Thanks Hollis for the pointer's..pls see inline..

-----Original Message-----
From: Hollis Blanchard [mailto:hollisb@us.ibm.com] 
Sent: Thursday, April 02, 2009 9:00 AM
To: Liu Yu-B13201
Cc: Rahul Kulkarni; kvm-ppc@vger.kernel.org
Subject: RE: PowerPC KVM build directions

On Thu, 2009-04-02 at 10:52 +0800, Liu Yu-B13201 wrote:
> 
> > Since KVM supports a NetBSD 4.0  guest (I 
> > think) and 8544/e500 emulation is already present in qemu -- 
> > theoretically the first part should work...but I recall Liu 
> > mentioning that there might be some OS specific quirks 
> > present in the port...and that was what I my question was 
> > hinting at earlier.. 
> 
> There must be a lot of differences between Linux and NetBSD.
> So the design of kvmppc may not be careful considerate.
> 
> The main trick is to hijack the system interrupts.
> You should check if this part of code cater to NetBSD.
> Context switch needs to be taken care as well.
> 
> At least there is one thing I can point out.
> See comments in the file booke_interrupts.S  line 195. :-)

Liu, you're talking about BSD as the *host*. Rahul is asking about BSD
as the guest.

Rahul, one major quirk we exploit is that Linux does not use the MSR[AS]
bit at all. One way that bit could be used is to give 32-bit userspace a
separate 4GB address space from the kernel. Instead, Linux puts both
kernel and userspace into the same 4GB address space (with Linux
mappings above 0xc0000000 and user mappings below). If NetBSD uses
MSR[AS]=1 for userspace (which I think is what the hardware architects
envisioned), you're going to have a lot of MMU fun.

Rahul>> The NetBSD port for e500/85xx which we have uses the MSR[AS] (IS/DS) for user/kernel address space separation which keep the address spaces split. So that's a major problem to start with. How do we get creative with this to provide guest mappings is something, which has to be explored. Let me know if you have any thoughts..

Another potential issue could be the initial environment (described
earlier as option 2) not being what BSD expects. Do you use u-boot? You
can see the initial environment set up in kvm_arch_vcpu_setup() in KVM
and mpc8544ds_init() in Qemu.

Rahul>> Yes..I will look into those functions..We do use uboot..Are you hinting to go with option 1?

Does NetBSD use flattened device trees at all? KVM (Qemu) supplies a
stripped-down device tree to the guest so that the guest won't try to
access IO devices not currently emulated by qemu. If BSD has a hardcoded
device configuration system (e.g. "we built for 8544, therefore we
always have the following SoC devices") that will be an issue.

Rahul>> The device config is hardcoded our NetBSD code base(more so because of the embedded nature it's a preferred way) but since I see NetBSD supported on Qemu..I would think there is a support available for a flattened device tree to be passed in from qemu..I'll look at x86 implementations.

As a concrete example, qemu doesn't emulate the TSEC ethernet
controller. You need to either convince your guest not to try to use the
TSEC (and use e1000 or some other qemu-supported device instead), or add
just enough TSEC emulation to qemu to make your guest happy. That could
be as simple as always reporting "link down" so the guest doesn't try to
use it.

Rahul>>  Yes ok.. 

Please keep us posted about any issues you encounter. Also, documenting
the hurdles and how you overcame them might be an interesting conference
paper, if you're into that sort of thing. :)

Rahul>> Will do and hopefully we pull this off :>

-- 
Hollis Blanchard
IBM Linux Technology Center

RE: PowerPC KVM build directions

[Liu Yu-B13201]

> -----Original Message-----
> From: Hollis Blanchard [mailto:hollisb@us.ibm.com] 
> Sent: Friday, April 03, 2009 12:00 AM
> To: Liu Yu-B13201
> Cc: Rahul Kulkarni; kvm-ppc@vger.kernel.org
> Subject: RE: PowerPC KVM build directions
> 
> On Thu, 2009-04-02 at 10:52 +0800, Liu Yu-B13201 wrote:
> > 
> > > Since KVM supports a NetBSD 4.0  guest (I 
> > > think) and 8544/e500 emulation is already present in qemu -- 
> > > theoretically the first part should work...but I recall Liu 
> > > mentioning that there might be some OS specific quirks 
> > > present in the port...and that was what I my question was 
> > > hinting at earlier.. 
> > 
> > There must be a lot of differences between Linux and NetBSD.
> > So the design of kvmppc may not be careful considerate.
> > 
> > The main trick is to hijack the system interrupts.
> > You should check if this part of code cater to NetBSD.
> > Context switch needs to be taken care as well.
> > 
> > At least there is one thing I can point out.
> > See comments in the file booke_interrupts.S  line 195. :-)
> 
> Liu, you're talking about BSD as the *host*. Rahul is asking about BSD
> as the guest.
> 

Sorry, I stand corrected. :-/