arch/xen is a bad idea

arch/xen is a bad idea

[Andi Kleen]

"Andi Kleen" <ak@suse.de> writes:

[again this time with subject. sorry for the screwup]
[very late answer]

> Stunned silence I guess - merging an architecture is
> usually much more controversial ;)

In my opinion it's still an extremly bad idea to have arch/xen
an own architecture. It will cause a lot of work long term
to maintain it, especially when it gets x86-64 support too.
It would be much better to just merge it with i386/x86-64.

Currently it's already difficult enough to get people to
add fixes to both i386 and x86-64, adding fixes to three
or rather four (xen32 and xen64) architectures will be quite bad.
In practice we'll likely get much worse code drift and missing
fixes. Also I still suspect Ian is underestimating how much
work it is long term to keep an Linux architecture uptodate.

I cannot imagine the virtualization hooks are intrusive anyways. The
only things it needs to hook idle and the page table updates, right?
Doing that cleanly in the existing architectures shouldn't be that
hard.

I suspect xen64 will be rather different from xen32 anyways
because as far as I can see the tricks Xen32 uses to be
fast (segment limits) just plain don't work on 64bit
because the segments don't extend into 64bit space.
So having both in one architecture may also end up messy.

And i386 and x86-64 are in many pieces very different anyways,
I have my doubts that trying to mesh them together in arch/xen
will be very pretty.

Also the other thing I'm worried about is that there is no clear
specification on how the Xen<->Linux interface works. Assuming
there will be other para Hypervisors in the future too will we
end up with even more virtual architectures? It would be much
better to have at least a somewhat defined "linux virtual interface"
first that is actually understood by multiple people outside
the Xen group.

I think before merging stuff the hypervisor interfaces need to be
discussed on linux-kernel. Splitting the patches and posting them
as individual pieces for i386 with good description will be a good
first step for that.

-Andi

Re: arch/xen is a bad idea

[Antonio Vargas]

On 14 Dec 2004 19:59:50 +0100, Andi Kleen <ak@suse.de> wrote:
> "Andi Kleen" <ak@suse.de> writes:
> 
> [again this time with subject. sorry for the screwup]
> [very late answer]
> 
> > Stunned silence I guess - merging an architecture is
> > usually much more controversial ;)
> 
> In my opinion it's still an extremly bad idea to have arch/xen
> an own architecture. It will cause a lot of work long term
> to maintain it, especially when it gets x86-64 support too.
> It would be much better to just merge it with i386/x86-64.
> 
> Currently it's already difficult enough to get people to
> add fixes to both i386 and x86-64, adding fixes to three
> or rather four (xen32 and xen64) architectures will be quite bad.
> In practice we'll likely get much worse code drift and missing
> fixes. Also I still suspect Ian is underestimating how much
> work it is long term to keep an Linux architecture uptodate.
> 
> I cannot imagine the virtualization hooks are intrusive anyways. The
> only things it needs to hook idle and the page table updates, right?
> Doing that cleanly in the existing architectures shouldn't be that
> hard.
> 
> I suspect xen64 will be rather different from xen32 anyways
> because as far as I can see the tricks Xen32 uses to be
> fast (segment limits) just plain don't work on 64bit
> because the segments don't extend into 64bit space.
> So having both in one architecture may also end up messy.
> 
> And i386 and x86-64 are in many pieces very different anyways,
> I have my doubts that trying to mesh them together in arch/xen
> will be very pretty.
> 
> Also the other thing I'm worried about is that there is no clear
> specification on how the Xen<->Linux interface works. Assuming
> there will be other para Hypervisors in the future too will we
> end up with even more virtual architectures? It would be much
> better to have at least a somewhat defined "linux virtual interface"
> first that is actually understood by multiple people outside
> the Xen group.
> 
> I think before merging stuff the hypervisor interfaces need to be
> discussed on linux-kernel. Splitting the patches and posting them
> as individual pieces for i386 with good description will be a good
> first step for that.
> 
> -Andi
> -

Andi, there is at least one other hypervisor interface, mac-on-linux
features a kernel module that allows booting other kernels inside the
running one, and keeps very good speed anyways.

Their code, at least the user-space part, was also very good coded
for my eyes.

Driver support is done by exporting a customized open-firmware
device tree and then implementing drivers for these devices on
the client OSs.

(just goto http://www.maconlinux.org/ and have a look)

Oh, this is obviusly ppc-only ATM :)

-- 
Greetz, Antonio Vargas aka winden of network

http://wind.codepixel.com/

Las cosas no son lo que parecen, excepto cuando parecen lo que si son.

Re: arch/xen is a bad idea

[Ian Pratt]

> > Stunned silence I guess - merging an architecture is
> > usually much more controversial ;)
> 
> In my opinion it's still an extremly bad idea to have arch/xen
> an own architecture. It will cause a lot of work long term
> to maintain it, especially when it gets x86-64 support too.
> It would be much better to just merge it with i386/x86-64.

Andi, I totally agree that merging into i386 could be a long term
goal. However, its just not feasible right now. The changes
required are way too intrusive. We put considerable effort into
investigating this approach, but came to the conclusion that with
the current structure of arch i386 it was going to be way too
messy. 

I really think the best approach is to get arch xen into
mainstream Linux, and then work toward integrating i386, x86_64
and xen. From our point of view, the first stage of this is to
increase the number of files that are shared unmodified between
i386 and xen/i386 (i.e. linked from xen into i386). There's
already many such files, but with a few relatively simple changes
to i386 we could get quite a few more.

> Currently it's already difficult enough to get people to
> add fixes to both i386 and x86-64, adding fixes to three
> or rather four (xen32 and xen64) architectures will be quite bad.
> In practice we'll likely get much worse code drift and missing
> fixes. Also I still suspect Ian is underestimating how much
> work it is long term to keep an Linux architecture uptodate.

We're actually very well setup to handle this, having been doing
it for some time. Whenever Linus issues a new mainstream patch,
we have a script that rewrites the patch to duplicate the hunks
that apply to i386 such that they also hit files that we've
modified in xen/i386. This way we keep arch xen/i386 in perfect
sync with i386. It takes discipline, but we're pretty good at it
now.

> I cannot imagine the virtualization hooks are intrusive anyways. The
> only things it needs to hook idle and the page table updates,
> right?

It's rather more complicated than that if you want a clean
interface that gives good performance. We've taken a very
benchmark-driven approach to minimise the overhead of
virtualization. The aim is to have such a small overhead that
people are happy running virtualized the whole time. I think this
is a really important aim.

> Doing that cleanly in the existing architectures shouldn't be that
> hard.

I've appended a list of some of the areas we need to modify. I
think you may have underestimated what needs to happen.

> I suspect xen64 will be rather different from xen32 anyways
> because as far as I can see the tricks Xen32 uses to be
> fast (segment limits) just plain don't work on 64bit
> because the segments don't extend into 64bit space.
> So having both in one architecture may also end up messy.

We have subdirectories for the i386 and x86_64 specific files,
along with a common directory for stuff which is shared between
the two e.g. virtual interrupt control etc.
 
> Also the other thing I'm worried about is that there is no clear
> specification on how the Xen<->Linux interface works.

We have an interface manual in the Xen bk repo, but I acknowledge
that we haven't always been totally prompt in keeping it up to
date and fully detailed. Now the Xen2 interface is frozen, that
should be fixable. Even so, it hasn't prevented other independent
groups porting OSes to Xen e.g. NetBSD, FreeBSD, Plan9.

Ian

Differences between i386 and xen/i386 files:
- irq setup
- pci bus scanning
- gdt/ldt must be in dedicated read-only pages
- gdt/ldt install/updates
- debug register updates
- pte quicklist/cache
- pmd/pgd are read-only pages
- highmem pte mapping
- dma memory allocation
- idle loop
- different fixmap
- *_to_* macros differ (like page_to_phys)
- *_val macros (pte_val, pmd_val)
- inb/outb
- switching cr3
- cr4 updates
- a few cpu flags need to be cleared
- msr access
- wbinvd call
- mtrr access
- io permission handling
- ioremap 
- access to hardware memory
- interrupt enabling/disabling
- setup of trap gates
- tlb flush
- fpu stack switches
- user/kernel mode test
- different segment selectors since the kernel runs in ring1
- pagefault handler stack layout is different
- startup is in 32-bit mode
- start of day initialization is different
- start of day memory probing and pagetable setup
- trap/fault handling
- timer is virtualized
- smp addtl. cpu startup
- smp ipi's

Re: arch/xen is a bad idea

[Andi Kleen]

On Tue, Dec 14, 2004 at 10:40:20PM +0000, Ian Pratt wrote:
> I really think the best approach is to get arch xen into
> mainstream Linux, and then work toward integrating i386, x86_64
> and xen. From our point of view, the first stage of this is to

I think that's definitely the wrong way. Also in Linux 
the standard strategy is to first clean up and restructure/refactor 
code and then merge, not the other way round.

> increase the number of files that are shared unmodified between
> i386 and xen/i386 (i.e. linked from xen into i386). There's
> already many such files, but with a few relatively simple changes
> to i386 we could get quite a few more.

That will still have most of the disadvantages I listed.

> 
> > Currently it's already difficult enough to get people to
> > add fixes to both i386 and x86-64, adding fixes to three
> > or rather four (xen32 and xen64) architectures will be quite bad.
> > In practice we'll likely get much worse code drift and missing
> > fixes. Also I still suspect Ian is underestimating how much
> > work it is long term to keep an Linux architecture uptodate.
> 
> We're actually very well setup to handle this, having been doing
> it for some time. Whenever Linus issues a new mainstream patch,
> we have a script that rewrites the patch to duplicate the hunks
> that apply to i386 such that they also hit files that we've
> modified in xen/i386. This way we keep arch xen/i386 in perfect
> sync with i386. It takes discipline, but we're pretty good at it
> now.

That won't anymore at some time. I found this out on x86-64.
It works at the beginning, but eventually the code drift gets
so much that it's near impossible to apply any hunk and you
have to redo everything. One reason for that is that in Linux
code often gets restructured, which makes it very difficult
for such mechanized merging procedures to work long term.

Also you have to review every change anyways.

> > I cannot imagine the virtualization hooks are intrusive anyways. The
> > only things it needs to hook idle and the page table updates,
> > right?
> 
> It's rather more complicated than that if you want a clean
> interface that gives good performance. We've taken a very
> benchmark-driven approach to minimise the overhead of
> virtualization. The aim is to have such a small overhead that
> people are happy running virtualized the whole time. I think this
> is a really important aim.

Can you please be a bit more precise on that? What exactly do you
need? 

> 
> > Doing that cleanly in the existing architectures shouldn't be that
> > hard.
> 
> I've appended a list of some of the areas we need to modify. I
> think you may have underestimated what needs to happen.

Ok that's a start.

> 
> > I suspect xen64 will be rather different from xen32 anyways
> > because as far as I can see the tricks Xen32 uses to be
> > fast (segment limits) just plain don't work on 64bit
> > because the segments don't extend into 64bit space.
> > So having both in one architecture may also end up messy.
> 
> We have subdirectories for the i386 and x86_64 specific files,
> along with a common directory for stuff which is shared between
> the two e.g. virtual interrupt control etc.

Ok, but I think there will be significant differences in the 
64bit part and meshing both together will get extremly messy.

What's your strategy for example to merge changes from arch/x86_64
to xen64? I don't think the way you described above will work 
in any way.

>  
> > Also the other thing I'm worried about is that there is no clear
> > specification on how the Xen<->Linux interface works.
> 
> We have an interface manual in the Xen bk repo, but I acknowledge
> that we haven't always been totally prompt in keeping it up to
> date and fully detailed. Now the Xen2 interface is frozen, that
> should be fixable. Even so, it hasn't prevented other independent
> groups porting OSes to Xen e.g. NetBSD, FreeBSD, Plan9.
> 
> Ian


Comments on some issues:

You are saying you ran benchmarks on each of them and it turns
out to be too slow to emulate them in the standard architectural
way? At least for some of them it's hard to believe.

> 
> Differences between i386 and xen/i386 files:
> - irq setup

Already two different ways (MSI and non MSI). Adding a third
is probably feasible.

> - pci bus scanning

We already have at least three different ways to do that, adding
a fourth wouldn't be very messy I guess.

> - gdt/ldt must be in dedicated read-only pages

That can be probably done in the standard kernel
(without the actual read protection bit) 

> - gdt/ldt install/updates

gdt load could be just trapped, it only happens once
at startup. 

LDT i can see some effort needed, but it's already
only at a single function in a header file.

> - debug register updates

This shouldn't be performance critical. Why is it not
simply emulated by the hypervisor?

> - pte quicklist/cache

i386 had a pte quicklist some time ago. I think there were
plans anyways to readd it. 

> - pmd/pgd are read-only pages

I suspect this just needs another include layer like the existing
PAE/non PAE layer.

> - highmem pte mapping
> - dma memory allocation
> - idle loop

This is already pluggable (e.g. ACPI has its own) 

> - different fixmap
> - *_to_* macros differ (like page_to_phys)
> - *_val macros (pte_val, pmd_val)

Can be also handled like 2/3 levels I bet.

> - inb/outb

Why is it not trapped? 

> - switching cr3
> - cr4 updates
> - a few cpu flags need to be cleared
> - msr access

Why don't you just simply trap it? MSRs are normally
not performance critical (except on x86-64 for the context switch
but you didn't seem to have attacked that at all yet ;-) 

> - wbinvd call
> - mtrr access

Same.

For all of these it would seem much cleaner to me to just
provide instruction emulation in the Hypervisor.

> - io permission handling

What is different? 

> - ioremap 
> - access to hardware memory
> - interrupt enabling/disabling
> - setup of trap gates
> - tlb flush

This is already well separated, shouldn't be a big issue 
to make it different.

> - fpu stack switches
> - user/kernel mode test

That can be just changed everywhere in i386 without ill effects.

> - different segment selectors since the kernel runs in ring1
> - pagefault handler stack layout is different
> - startup is in 32-bit mode

That's a good cleanup anyways. The early boot interface
between 16 and 32bit should be clearly defined and then
a general 32bit booting interface be defined. There are various
other people who would like to have this too. I had some plans
to do the equivalent on x86-64 with a 64bit boot interface.

> - start of day initialization is different
> - start of day memory probing and pagetable setup

Sounds similar to EFI. 

> - trap/fault handling
> - timer is virtualized

Details?

First impression is that there is much cleanup potential and that
there should be probably some discussion on each of these items. 

-Andi

Re: arch/xen is a bad idea

[Pavel Machek]

Hi!

> > > Stunned silence I guess - merging an architecture is
> > > usually much more controversial ;)
> > 
> > In my opinion it's still an extremly bad idea to have arch/xen
> > an own architecture. It will cause a lot of work long term
> > to maintain it, especially when it gets x86-64 support too.
> > It would be much better to just merge it with i386/x86-64.
> 
> Andi, I totally agree that merging into i386 could be a long term
> goal. However, its just not feasible right now. The changes
> required are way too intrusive. We put considerable effort into
> investigating this approach, but came to the conclusion that with
> the current structure of arch i386 it was going to be way too
> messy. 

Okay, what about this one:

You merge xen hooks in mainline, but keep maintaining arch/xen
out-of-tree? You have to maintain it yourself, anyway, and having
hooks merged should make it easy.

When xen is merged into i386 (you said that is your long-term goal
anyway), you can merge that into mainline...
							Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: arch/xen is a bad idea

[Pavel Machek]

Hi!

> > > Stunned silence I guess - merging an architecture is
> > > usually much more controversial ;)
> > 
> > In my opinion it's still an extremly bad idea to have arch/xen
> > an own architecture. It will cause a lot of work long term
> > to maintain it, especially when it gets x86-64 support too.
> > It would be much better to just merge it with i386/x86-64.
> 
> Andi, I totally agree that merging into i386 could be a long term
> goal. However, its just not feasible right now. The changes
> required are way too intrusive. We put considerable effort into
> investigating this approach, but came to the conclusion that with
> the current structure of arch i386 it was going to be way too
> messy. 

BTW if you merge xen as separate architecture, it will be *very* hard
to merge it back to i386. That patch would be huge, and would need to
go in "atomically".

							Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: arch/xen is a bad idea

[Alan Cox]

On Mer, 2004-12-15 at 04:49, Andi Kleen wrote:
> I think that's definitely the wrong way. Also in Linux 
> the standard strategy is to first clean up and restructure/refactor 
> code and then merge, not the other way round.

Must be two different Linux OS's out there then. I thought it was
- get interfaces right
- get working
- get correct
- merge
- evolve

If we did it as you describe we'd still not have a SATA layer merged for
example.

Re: arch/xen is a bad idea

[Ian Pratt]

> You merge xen hooks in mainline, but keep maintaining arch/xen
> out-of-tree? You have to maintain it yourself, anyway, and having
> hooks merged should make it easy.

Having the hooks merged would certainly help. 

However, I think the main benefit of being in the mainline tree
would be that it would make it easy for other developers to see
how arch xen works and thus begin to take us into account when
considering changes. If there's going to be a big effort to merge
i386, x86_64 and xen going forward, it will certainly help if
more people have looked through our code. We'll only get that
kind of exposure to developers if we're in the mainstream tree. 

The other key area is that our top priority is to decrease the
number of files we need to modified from standard i386. For this
to happen, we need to submit patches into i386 that abstract a
few things behind macros/constants. For example, we'd like to
abstract the test to see whether the CPU is in the kernel or not
(we run the kernel in ring 1 rather than 0).  If arch xen is in
the tree, this kind of patch will make rather more sense to
people.

> BTW if you merge xen as separate architecture, it will be *very* hard
> to merge it back to i386. That patch would be huge, and would need to
> go in "atomically".

I don't see it like that. While continuing to track changes in
i386/x86_64, we'd restructure the code under arch xen such that
it could build (or even boot) time switch between running native
and over Xen. At some point the arch directory could then be
renamed.  This would be a big project, and one that would involve
a lot more people than just the Xen team. Because the x86
architecture is so critical to Linux, I just can't see this
happening unless the development takes place in the mainstream
tree. 

Of course, another reason for inclusion in mainstream is that
there's a lot of momentum behind Xen in both the user and
developer communities. My guess is that virtualization is going
to be pretty pervasive in the data centre, and its one of the key
technologies that Linux can win on. The paravirtualized approach
used by Xen is going to offer better performance than full x86
virtualization, even if there's extra support in the processor.

Cheers,
Ian

Re: arch/xen is a bad idea

[Pavel Machek]

Hi!

> > BTW if you merge xen as separate architecture, it will be *very* hard
> > to merge it back to i386. That patch would be huge, and would need to
> > go in "atomically".
> 
> I don't see it like that. While continuing to track changes in
> i386/x86_64, we'd restructure the code under arch xen such that
> it could build (or even boot) time switch between running native
> and over Xen. At some point the arch directory could then be
> renamed.  This would be a big project, and one that would involve
> a lot more people than just the Xen team. Because the x86

So you plan to

merge arch/xen

then modify arch/xen to do all arch/i386 can do

then rm -rf arch/i386, mv arch/xen arch/i386? Well, I'd say that's
rather ambitious plan....
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: arch/xen is a bad idea

[Andi Kleen]

On Thu, Dec 16, 2004 at 12:09:44AM +0000, Alan Cox wrote:
> On Mer, 2004-12-15 at 04:49, Andi Kleen wrote:
> > I think that's definitely the wrong way. Also in Linux 
> > the standard strategy is to first clean up and restructure/refactor 
> > code and then merge, not the other way round.
> 
> Must be two different Linux OS's out there then. I thought it was
> - get interfaces right

That is exactly the part that is wrong currently imho. The arch/xen
interface is a mess and in its current form unlikely to be maintainable.

-Andi

Re: arch/xen is a bad idea

[Alan Cox]

On Iau, 2004-12-16 at 04:01, Andi Kleen wrote:
> That is exactly the part that is wrong currently imho. The arch/xen
> interface is a mess and in its current form unlikely to be maintainable.

It seems maintainable and well documented to me. I just don't see where
your problem is with this. The kernel/hypervisor interface is clear, and
the arch/xen code seems quite sane.

Re: arch/xen is a bad idea

[Alan Cox]

On Iau, 2004-12-16 at 14:09, Andi Kleen wrote:
> Also e.g. for non performance critical 
> things like changing MTRRs or debug registers it would be IMHO much 
> cleaner to just emulate the instructions (the ISA is very well 
> defined) and not change the kernel here.  From a look at Ian's list
> the majority of the changes needed for Xen actually fall into
> this category. 

There are so many problems in snooping and decoding instructions it
isn't funny. Aside from the mmap pci buffer half way through instruction
that will emulate type stuff there are a lot of awkward issues if you
want to emulate multiple mtrr sets (you need PAT).

Xen has tried the decode/patch stuff earlier - see the early NPTL
handling and it was neither pretty nor reliable.

Re: arch/xen is a bad idea

[Andi Kleen]

On Thu, Dec 16, 2004 at 12:54:17PM +0000, Alan Cox wrote:
> On Iau, 2004-12-16 at 04:01, Andi Kleen wrote:
> > That is exactly the part that is wrong currently imho. The arch/xen
> > interface is a mess and in its current form unlikely to be maintainable.
> 
> It seems maintainable and well documented to me. I just don't see where
> your problem is with this. The kernel/hypervisor interface is clear, and
> the arch/xen code seems quite sane.

The main problem I see is that it is source code copy, and especially 
when they both support i386 and x86-64 there will be no sane
way to keep it all synchronized with the i386 and x86-64 
code bases. It's already hard from a single source like I can
attest from x86-64, with two sources it will be likely much more
difficult longer term. I just can't see it working well in
practice. It will be also nasty for people doing changes
because they will need to duplicate i386+x86_64 changes four 
times in the worst case (i386,x86_64,xen32,xen64) 

I guess it may be acceptable if we were maintaining obscure Lance 
drivers this way ;_), but for a important architecture it just doesn't seem
like the right approach to me. 

Also e.g. for non performance critical 
things like changing MTRRs or debug registers it would be IMHO much 
cleaner to just emulate the instructions (the ISA is very well 
defined) and not change the kernel here.  From a look at Ian's list
the majority of the changes needed for Xen actually fall into
this category. 

I suspect when the kernel is only changed for the truly performance 
critical interfaces that cannot be efficiently emulated (like idle/timers/page 
table updates) the required changes for the para virtualization will become 
much more manageable and can be cleanly integrated into the respective ports.

And as Pavel points out first merging arch/xen and then migrating
into i386 and x86_64 like it was proposed sounds extremly hard and is 
probably not really practical. 

-Andi

Re: arch/xen is a bad idea

[Andi Kleen]

On Thu, Dec 16, 2004 at 01:14:09AM +0000, Ian Pratt wrote:
> The other key area is that our top priority is to decrease the
> number of files we need to modified from standard i386. For this
> to happen, we need to submit patches into i386 that abstract a
> few things behind macros/constants. For example, we'd like to
> abstract the test to see whether the CPU is in the kernel or not
> (we run the kernel in ring 1 rather than 0).  If arch xen is in
> the tree, this kind of patch will make rather more sense to
> people.

That would be a good first step, especially if it results in cleanups.
Please go for it.

> I don't see it like that. While continuing to track changes in
> i386/x86_64, we'd restructure the code under arch xen such that
> it could build (or even boot) time switch between running native
> and over Xen. At some point the arch directory could then be
> renamed.  This would be a big project, and one that would involve

This sounds like a massive duplication of effort. You would need
to do all that work on arch/xen and in parallel on the native
port for the slow merge, and in parallel track a changing target
and keep the code usable in mainline.

-Andi

Re: arch/xen is a bad idea

[Andi Kleen]

On Thu, Dec 16, 2004 at 01:19:06PM +0000, Alan Cox wrote:
> On Iau, 2004-12-16 at 14:09, Andi Kleen wrote:
> > Also e.g. for non performance critical 
> > things like changing MTRRs or debug registers it would be IMHO much 
> > cleaner to just emulate the instructions (the ISA is very well 
> > defined) and not change the kernel here.  From a look at Ian's list
> > the majority of the changes needed for Xen actually fall into
> > this category. 
> 
> There are so many problems in snooping and decoding instructions it
> isn't funny. Aside from the mmap pci buffer half way through instruction

Hmm? From what I remember reading the code of the Xen hypervisor
they are already emulating a lot of instructions (e.g. take a look
at xen/arch/x86/x86_32/seg_fixup.c) Emulating some more doesn't 
seem like a big issue to me.

> that will emulate type stuff there are a lot of awkward issues if you
> want to emulate multiple mtrr sets (you need PAT).

No way different from doing it with a magic interface. MTRR is exposed
to kernel subsystems (a lot of things in the service OS will want to
play with it) and to user space (with also a lot of users). The magic
interface will need to provide a full MTRR interface anyways.  The 
only difference would be basically how you tell the hypervisor to
change them.

[Not saying that PAT is actually needed for this, but as a general 
comment:]
Also using PAT for that would be definitely a good idea, MTRRs just
have so many problems that any step away from them is a good thing.


-Andi

Re: arch/xen is a bad idea

[Andrew Morton]

Andi Kleen <ak@suse.de> wrote:
>
> And as Pavel points out first merging arch/xen and then migrating
>  into i386 and x86_64 like it was proposed sounds extremly hard and is 
>  probably not really practical. 

It will also create more patchwork and will make the the change history
more obscure and harder to follow.

It is a bit pervers to be putting something into the tree with the
intention of ripping it up and redoing it in the near future.  Why not do
it up-front and keep the change history cleaner?  From Ian's change list
and Andi's comments it doesn't look to me that this is all as much work as
people seem to be implying?


I guess if we were to go the way which Ian is proposing it would be

a) Add arch/xen

b) Spend N weeks integrating xen into arch/i386, while also separately
   maintaining arch/xen.

c) Remove arch/xen

So...  why not skip a), c) and half of b)?

Re: arch/xen is a bad idea

[Alan Cox]

On Iau, 2004-12-16 at 18:26, Andrew Morton wrote:
> I guess if we were to go the way which Ian is proposing it would be
> 
> a) Add arch/xen
> 
> b) Spend N weeks integrating xen into arch/i386, while also separately
>    maintaining arch/xen.
> 
> c) Remove arch/xen
> 
> So...  why not skip a), c) and half of b)?

Because until xen is in the kernel it won't get good exposure
Because nobody is sure that b) needs to occur or would occur for another
year our two (Xen3)

Xen is not "a slightly x86 arch variant" (and a slight ia64 arch
variant) Xen is quite different in its design and its performance comes
from exactly that - paravirtualisation by being a virtual CPU that isn't
a true x86 is very very fast.

b is a long term research project it is not a "clean this up next month"
project as I understand it. 

Re: arch/xen is a bad idea

[Ian Pratt]

> > There are so many problems in snooping and decoding instructions it
> > isn't funny. Aside from the mmap pci buffer half way through instruction
> 
> Hmm? From what I remember reading the code of the Xen hypervisor
> they are already emulating a lot of instructions (e.g. take a look
> at xen/arch/x86/x86_32/seg_fixup.c) Emulating some more doesn't 
> seem like a big issue to me.

There's actually no emulation in Xen at all. seg_fixup is the
closest we come, which is a tiny decoder that is able to work out
the effective address of an instruction. It's only through grim
necessity we need this in order to be able to make NPTL thread
local storage accesses work (-ve segment offsets). We'd much
prefer that glibc was slightly tweaked such that we didn't have
to do this (it would be a lot faster too).

Anyhow, I really don't think emulation is the issue. The handful
of cases that you pointed out that could relatively easily be
emulated constitute a tiny fraction of the arch xen patch.  Even
so, you have to be a bit careful from a performance point of
view: loading debug registers can occur quite frequently, and its
much quicker to be able to load them as a batch rather than
taking individual faults. 


Ian

Re: arch/xen is a bad idea

[Ian Pratt]

 
> I guess if we were to go the way which Ian is proposing it would be
> 
> a) Add arch/xen
> 
> b) Spend N weeks integrating xen into arch/i386, while also separately
>    maintaining arch/xen.
> 
> c) Remove arch/xen
> 
> So...  why not skip a), c) and half of b)?

That's not quite what I'm proposing. 

Once arch/xen is in the tree, we'd start submitting patches that
try and unify more of arch xen and i386 to reduce the number of
files that we have to modify. 

I think we'd then have to make a decision as to whether merging
i386 and xen/i386 is feasible. Further, we'd have to make a
decision as to whether what is really wanted is a single kernel
that's able to boot-time switch between native i386 and xen,
rather than just having a single source base. The two options
would probably result in rather different implementations.

In short, merging is far from trivial, and its not even clear
quite what is wanted. 

I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many
files unmodified from i386. Keeping i386 and xen/i386 in sync is
fairly mechanical: we can apply most of the patches to i386 to
xen/i386 directly. 


Ian

Re: arch/xen is a bad idea

[Andrew Morton]

Ian Pratt <Ian.Pratt@cl.cam.ac.uk> wrote:
>
> I'm not convinced that maintaining xen/i386 in its current form
>  is going to be as hard as Andi thinks. We already share many
>  files unmodified from i386.

How are they shared?  Inclusion, symlinks, makefile references or
copy-n-paste?

Re: arch/xen is a bad idea

[Ian Pratt]

> Ian Pratt <Ian.Pratt@cl.cam.ac.uk> wrote:
> >
> > I'm not convinced that maintaining xen/i386 in its current form
> >  is going to be as hard as Andi thinks. We already share many
> >  files unmodified from i386.
> 
> How are they shared?  Inclusion, symlinks, makefile references or

The makefile creates symlinks. 

Ian

Re: arch/xen is a bad idea

[Rik van Riel]

On Thu, 16 Dec 2004, Ian Pratt wrote:
>> Ian Pratt <Ian.Pratt@cl.cam.ac.uk> wrote:
>>>
>>> I'm not convinced that maintaining xen/i386 in its current form
>>>  is going to be as hard as Andi thinks. We already share many
>>>  files unmodified from i386.
>>
>> How are they shared?  Inclusion, symlinks, makefile references or
>
> The makefile creates symlinks.

Which is really very nice.  If you update arch/i386/ then
arch/xen/ automatically gets the update too.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: arch/xen is a bad idea

[Philip R Auld]

Rumor has it that on Thu, Dec 16, 2004 at 09:00:52PM +0000 Ian Pratt said:
>  
> I think we'd then have to make a decision as to whether merging
> i386 and xen/i386 is feasible. Further, we'd have to make a
> decision as to whether what is really wanted is a single kernel
> that's able to boot-time switch between native i386 and xen,
> rather than just having a single source base. The two options
> would probably result in rather different implementations.
> 

The boot-time switch seems to be the ideal. This would allow 
enterprise Linux vendors to support using Xen w/o having to
deal with a whole archicture release (including install kernel 
etc.). One of the (really strong IMO) arguments for full 
virtualization methods is beinag able to run stock OSes. 
At least for Linux guests, a Xen merge with the boot-time 
option would remove that argument. 

Of course, the resultant kernel changes would need to have no 
measurable impact on native i386... and be possible...


Cheers,

Phil

Re: arch/xen is a bad idea

[Bill Davidsen]

Pavel Machek wrote:

> Okay, what about this one:
> 
> You merge xen hooks in mainline, but keep maintaining arch/xen
> out-of-tree? You have to maintain it yourself, anyway, and having
> hooks merged should make it easy.
> 
> When xen is merged into i386 (you said that is your long-term goal
> anyway), you can merge that into mainline...
> 							Pavel

Assuming this is practical and would let xen get exposure sooner rather 
than later, it would be nice if the hooks could be used for other 
projects. My impression is that the merge part would come after there is 
some extensive experience with the operation "in the wild" so to speak. 
Who knows what operating systems might run in this way.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: arch/xen is a bad idea

[Rik van Riel]

On Thu, 16 Dec 2004, Philip R Auld wrote:

> The boot-time switch seems to be the ideal. This would allow
> enterprise Linux vendors to support using Xen w/o having to
> deal with a whole archicture release (including install kernel

I have no idea how such a boot-time switch would work
for 3rd party device drivers, though, so don't count
yourself lucky just yet ;)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: arch/xen is a bad idea

[Rik van Riel]

On Thu, 16 Dec 2004, Bill Davidsen wrote:

> Assuming this is practical and would let xen get exposure sooner rather 
> than later, it would be nice if the hooks could be used for other 
> projects.

> Who knows what operating systems might run in this way.

NetBSD and Plan 9 already run on Xen 2.0.  IIRC FreeBSD and
one of the other BSDs are being ported, too...


-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: arch/xen is a bad idea

[Philip R Auld]

Rumor has it that on Thu, Dec 16, 2004 at 06:08:24PM -0500 Rik van Riel said:
> On Thu, 16 Dec 2004, Philip R Auld wrote:
> 
> >The boot-time switch seems to be the ideal. This would allow
> >enterprise Linux vendors to support using Xen w/o having to
> >deal with a whole archicture release (including install kernel
> 
> I have no idea how such a boot-time switch would work
> for 3rd party device drivers, though, so don't count
> yourself lucky just yet ;)

I actually meant _OS_ vendors like you ;)
 
Cheers,

Phil


> 
> -- 
> "Debugging is twice as hard as writing the code in the first place.
> Therefore, if you write the code as cleverly as possible, you are,
> by definition, not smart enough to debug it." - Brian W. Kernighan

Re: arch/xen is a bad idea

[Andi Kleen]

On Thu, Dec 16, 2004 at 09:00:52PM +0000, Ian Pratt wrote:
>  
> > I guess if we were to go the way which Ian is proposing it would be
> > 
> > a) Add arch/xen
> > 
> > b) Spend N weeks integrating xen into arch/i386, while also separately
> >    maintaining arch/xen.
> > 
> > c) Remove arch/xen
> > 
> > So...  why not skip a), c) and half of b)?
> 
> That's not quite what I'm proposing. 
> 
> Once arch/xen is in the tree, we'd start submitting patches that
> try and unify more of arch xen and i386 to reduce the number of
> files that we have to modify. 

IMHO reducing the number of files to modify should be done
first before attempting a mainline merge. That is how a lot
of other more intrusive changes have been merged in the past.

There was a intensive review process and eventually the interfaces
were cleaned up a lot of and the thing was merged piece by piece
after considerable changes. 

> 
> I think we'd then have to make a decision as to whether merging
> i386 and xen/i386 is feasible. Further, we'd have to make a
> decision as to whether what is really wanted is a single kernel
> that's able to boot-time switch between native i386 and xen,
> rather than just having a single source base. The two options
> would probably result in rather different implementations.

Even without boot time switch but only CONFIG switch 
a more shared code base would be probably the only saner
way long time. I'm not sure a run time switch is really
that good an idea, it would essentially require to run
most of pgtable.h etc. through function pointers and 
I don't like this idea very much. But at least doing the
reorganization on the source code level would probably
clean up the interface considerably.

> 
> In short, merging is far from trivial, and its not even clear
> quite what is wanted. 
> 
> I'm not convinced that maintaining xen/i386 in its current form
> is going to be as hard as Andi thinks. We already share many
> files unmodified from i386. Keeping i386 and xen/i386 in sync is
> fairly mechanical: we can apply most of the patches to i386 to
> xen/i386 directly. 

Perhaps for now, but eventually this will not work anymore due
to inevitable code drift (I went exactly through the same phase for a year
or so on x86-64. These days I have to redo most patches unless
i'm lucky enough that the contributor does both) 

And with also supporting x86-64 it will not really work this
way well because you would need to merge from two sources. 
I can't see any mechanical scheme handling that.
The only sane way to maintain the merging scheme you proposed
would be then arch/xen64, multiplying work again.

-Andi

Re: arch/xen is a bad idea

[Andi Kleen]

On Thu, Dec 16, 2004 at 09:36:36PM +0000, Ian Pratt wrote:
> > Ian Pratt <Ian.Pratt@cl.cam.ac.uk> wrote:
> > >
> > > I'm not convinced that maintaining xen/i386 in its current form
> > >  is going to be as hard as Andi thinks. We already share many
> > >  files unmodified from i386.
> > 
> > How are they shared?  Inclusion, symlinks, makefile references or
> 
> The makefile creates symlinks. 

That's deprecated because it breaks with separate objdirs (make O=/other/dir) 
See the x86-64 makefiles on how to do it properly. 

-Andi

Re: arch/xen is a bad idea

[Ian Pratt]

> On Thu, Dec 16, 2004 at 09:36:36PM +0000, Ian Pratt wrote:
> > > Ian Pratt <Ian.Pratt@cl.cam.ac.uk> wrote:
> > > >
> > > > I'm not convinced that maintaining xen/i386 in its current form
> > > >  is going to be as hard as Andi thinks. We already share many
> > > >  files unmodified from i386.
> > > 
> > > How are they shared?  Inclusion, symlinks, makefile references or
> > 
> > The makefile creates symlinks. 
> 
> That's deprecated because it breaks with separate objdirs (make O=/other/dir) 
> See the x86-64 makefiles on how to do it properly. 

Ah, I wasn't previously aware of the O= capability to know that we
were breaking it. 

We can trivially change over to referencing files in arch/i386
directly from the Makefile. The only downside is that it will
make navigating the source slightly harder.


Ian

Re: arch/xen is a bad idea

[William Lee Irwin III]

On Tue, Dec 14, 2004 at 07:59:50PM +0100, Andi Kleen wrote:
> I suspect xen64 will be rather different from xen32 anyways
> because as far as I can see the tricks Xen32 uses to be
> fast (segment limits) just plain don't work on 64bit
> because the segments don't extend into 64bit space.
> So having both in one architecture may also end up messy.
> And i386 and x86-64 are in many pieces very different anyways,
> I have my doubts that trying to mesh them together in arch/xen
> will be very pretty.

I have an inkling that the xen implementors may have plots of
additional architecture ports. If it really is x86/x86-64 -only it
isn't worth bothering with a separate arch/ dir, but it would be if it
were ported to a large number of architectures.

Maybe they're modelling it after UML which has its own arch/ dir but
then grabs assorted things from other architectures; in that event I
wouldn't consider it so misguided.

Probably best if the implementors chimed in about all this.


-- wli

Re: arch/xen is a bad idea

[Ian Pratt]

> On Tue, Dec 14, 2004 at 07:59:50PM +0100, Andi Kleen wrote:
> > I suspect xen64 will be rather different from xen32 anyways
> > because as far as I can see the tricks Xen32 uses to be
> > fast (segment limits) just plain don't work on 64bit
> > because the segments don't extend into 64bit space.
> > So having both in one architecture may also end up messy.
> > And i386 and x86-64 are in many pieces very different anyways,
> > I have my doubts that trying to mesh them together in arch/xen
> > will be very pretty.
> 
> I have an inkling that the xen implementors may have plots of
> additional architecture ports. If it really is x86/x86-64 -only it
> isn't worth bothering with a separate arch/ dir, but it would be if it
> were ported to a large number of architectures.

For non x86 architectures, the changes required are typically
much less, and it makes sense to integrate them into the normal
architecture tree. This is what has been done with ia64, and will
be done with ppc. i386 and x86_64 are a special case where the
extensive changes required justify having an arch xen directory
with i386 and x86_64 subtrees beneath it (although many files are
just linked from the normal architecture).

> Maybe they're modelling it after UML which has its own arch/ dir but
> then grabs assorted things from other architectures; in that event I
> wouldn't consider it so misguided.

Yep, that's the model, for the x86 family.

Ian

arch/xen clue?

[Dorn Hetzel]

For those of us who are clueless as to even what arch/xen
is :)  Where would be a good place to read and become
informed?

-Dorn

Re: arch/xen clue?

[Ian Pratt]

> 
> For those of us who are clueless as to even what arch/xen
> is :)  Where would be a good place to read and become
> informed?

http://xen.sf.net

There's a bunch of papers and documentation etc.

Ian

Re: arch/xen clue?

[Anton Altaparmakov]

On Mon, 2004-12-20 at 10:08 -0500, Dorn Hetzel wrote:
> For those of us who are clueless as to even what arch/xen
> is :)  Where would be a good place to read and become
> informed?

http://www.cl.cam.ac.uk/Research/SRG/netos/xen/index.html

Best regards,

        Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: arch/xen clue?

[Måns Rullgård]

Dorn Hetzel <kernel@dorn.hetzel.org> writes:

> For those of us who are clueless as to even what arch/xen
> is :)  Where would be a good place to read and become
> informed?

http://xen.sf.net/

-- 
Måns Rullgård
mru@inprovide.com

Re: arch/xen is a bad idea

[Andrew Morton]

Andi Kleen <ak@suse.de> wrote:
>
> In my opinion it's still an extremly bad idea to have arch/xen
>  an own architecture.

Guys, I'd like to kick this a bit further down the road.  Things still seem
to be somewhat deadlocked.

To summarise my understanding:

The Xen team still believe that it's best to keep arch/xen, arch/xen/i386,
arch/xen/x86_64, etc.  And I believe that Andi (who is the world expert on
maintaining an i386 derivative) thinks that this is will be a long-term
maintenance problem.

I tend to agree with Andi, and I'm not sure that the Xen team fully
appreciate the downside of haveing an own-architecture in the kernel.org
kernel and the upside of having their code integrated with the
most-maintained architecture.  It could be that the potential problems
haven't been sufficiently well communicated.

Christian has mentioned that Xen would need to hook into the i386 code in
~60 places, which is somewhat more than Ian's 37-bullet-point list.

I get the impression that the Xen team are overly reluctant to make changes
to the arch/i386 code and to arch-neutral kernel code.  Don't do that - new
abstractions, refactoring and generally moving things about is generally a
safe thing to do, and can often make things better anyway.

So.  Has anyone changed position or otherwise converged?  How do we get
this resolved?

kernel 2.6.8-24.11-smp errors

[Marcel Smeets]

Hello,

I am using SuSE 9.2 Pro with kernel uname -a:

Linux 2.6.8-24.11-smp #1 SMP Fri Jan 14 13:01:26 UTC 2005 x86_64
x86_64 x86_64 GNU/Linux

Intel(R) Xeon(TM) CPU 2.80GHz

2 GB RAM
1 GB SWAP

and these strange kernel messages keep appearing::

it is a high load machine with iptables running. The
/proc/net/ip_conntrack number keeps running higher and does not get
less. After a while the maximum is reached. th e only way to solve
this is stop iptables, unload the ip modules from the kernel, load
them again and start iptables again. Then the messages will not come
again if I repeat this frequently. Could it be a kernel bug??? Or
maybe something misconfigured???

dmesg:

swapper: page allocation failure. order:1, mode:0x20

Call Trace:<IRQ> <ffffffff8015ecef>{__alloc_pages+1135}
<ffffffff8015e830>{__get_free_pages+16}
      <ffffffff80162676>{kmem_getpages+38}
<ffffffff8031e3ea>{tcp_v4_route_req+250}
      <ffffffff80162a99>{cache_alloc_refill+665}
<ffffffff80162c46>{kmem_cache_alloc+54}
      <ffffffff802e2f0a>{sk_alloc+58}
<ffffffff80323d99>{tcp_create_openreq_child+41}
      <ffffffff803229e7>{tcp_v4_syn_recv_sock+87}
<ffffffff80323b7e>{tcp_check_req+606}
      <ffffffff8030b4dc>{ip_finish_output+508}
<ffffffff80308b90>{ip_dst_output+0}
      <ffffffff8030b668>{ip_output+216} <ffffffff80308bc8>{ip_dst_output+56}
      <ffffffff80132f9b>{recalc_task_prio+635}
<ffffffff80308b90>{ip_dst_output+0}
      <ffffffff80132f9b>{recalc_task_prio+635}
<ffffffff80133c91>{activate_task+129}
      <ffffffff80137a89>{autoremove_wake_function+9}
<ffffffff80132633>{__wake_up_common+67}
      <ffffffff80132c13>{__wake_up+67} <ffffffff802e2344>{sock_def_readable+68}
      <ffffffff80320b79>{tcp_v4_do_rcv+233}
<ffffffffa017b034>{:ipt_state:match+36}
      <ffffffff803214cb>{tcp_v4_rcv+1835}
<ffffffff80305fb9>{ip_local_deliver_finish+297}
      <ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff802f3a43>{nf_hook_slow+195}
      <ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff803062fe>{ip_local_deliver+622}
      <ffffffff8030592e>{ip_rcv_finish+574} <ffffffff803056f0>{ip_rcv_finish+0}
      <ffffffff803056f0>{ip_rcv_finish+0} <ffffffff802f3a43>{nf_hook_slow+195}
      <ffffffff803056f0>{ip_rcv_finish+0} <ffffffff80305e34>{ip_rcv+1188}
      <ffffffff802e9749>{netif_receive_skb+729}
<ffffffffa00e9c9c>{:e1000:e1000_clean+1820}
      <ffffffff802e85b4>{net_rx_action+132}
<ffffffff8013dca1>{__do_softirq+113}
      <ffffffff8013dd55>{do_softirq+53} <ffffffff80113e3f>{do_IRQ+335}
      <ffffffff8010f540>{mwait_idle+0} <ffffffff80110cf5>{ret_from_intr+0}
       <EOI> <ffffffff8010f596>{mwait_idle+86} <ffffffff8010f9ea>{cpu_idle+26}
      <ffffffff804e971a>{start_kernel+490} <ffffffff804e91e0>{_sinittext+480}

swapper: page allocation failure. order:1, mode:0x20

Call Trace:<IRQ> <ffffffff8015ecef>{__alloc_pages+1135}
<ffffffff8015e830>{__get_free_pages+16}
      <ffffffff80162676>{kmem_getpages+38}
<ffffffff8031e3ea>{tcp_v4_route_req+250}
      <ffffffff80162a99>{cache_alloc_refill+665}
<ffffffff80162c46>{kmem_cache_alloc+54}
      <ffffffff802e2f0a>{sk_alloc+58}
<ffffffff80323d99>{tcp_create_openreq_child+41}
      <ffffffff803229e7>{tcp_v4_syn_recv_sock+87}
<ffffffff80323b7e>{tcp_check_req+606}
      <ffffffff8030b4dc>{ip_finish_output+508}
<ffffffff80308b90>{ip_dst_output+0}
      <ffffffff8030b668>{ip_output+216} <ffffffff80308bc8>{ip_dst_output+56}
      <ffffffff802f3a43>{nf_hook_slow+195} <ffffffff80308b90>{ip_dst_output+0}
      <ffffffff8013340a>{task_rq_lock+74}
<ffffffff80133f9b>{try_to_wake_up+747}
      <ffffffff8030ad6b>{ip_queue_xmit+1211}
<ffffffff8013340a>{task_rq_lock+74}
      <ffffffff8013340a>{task_rq_lock+74}
<ffffffff80132633>{__wake_up_common+67}
      <ffffffff80132c13>{__wake_up+67}
<ffffffff80312996>{tcp_ack_saw_tstamp+22}
      <ffffffff80314d90>{tcp_ack+1040} <ffffffff80320b79>{tcp_v4_do_rcv+233}
      <ffffffffa017b034>{:ipt_state:match+36}
<ffffffff803214cb>{tcp_v4_rcv+1835}
      <ffffffff80305fb9>{ip_local_deliver_finish+297}
<ffffffff80305e90>{ip_local_deliver_finish+0}
      <ffffffff802f3a43>{nf_hook_slow+195}
<ffffffff80305e90>{ip_local_deliver_finish+0}
      <ffffffff803062fe>{ip_local_deliver+622}
<ffffffff8030592e>{ip_rcv_finish+574}
      <ffffffff803056f0>{ip_rcv_finish+0} <ffffffff803056f0>{ip_rcv_finish+0}
      <ffffffff802f3a43>{nf_hook_slow+195} <ffffffff803056f0>{ip_rcv_finish+0}
      <ffffffff80305e34>{ip_rcv+1188} <ffffffff802e9749>{netif_receive_skb+729}
      <ffffffffa00e9c9c>{:e1000:e1000_clean+1820}
<ffffffff80133c91>{activate_task+129}
      <ffffffff80133f9b>{try_to_wake_up+747}
<ffffffff802e85b4>{net_rx_action+132}
      <ffffffff8013dca1>{__do_softirq+113} <ffffffff8013dd55>{do_softirq+53}
      <ffffffff80113e3f>{do_IRQ+335} <ffffffff8010f540>{mwait_idle+0}
      <ffffffff80110cf5>{ret_from_intr+0}  <EOI>
<ffffffff8010f596>{mwait_idle+86}
      <ffffffff8010f9ea>{cpu_idle+26} <ffffffff804e971a>{start_kernel+490}
      <ffffffff804e91e0>{_sinittext+480}
swapper: page allocation failure. order:1, mode:0x20

Call Trace:<IRQ> <ffffffff8015ecef>{__alloc_pages+1135}
<ffffffff8015e830>{__get_free_pages+16}
      <ffffffff80162676>{kmem_getpages+38}
<ffffffff8031e3ea>{tcp_v4_route_req+250}
      <ffffffff80162a99>{cache_alloc_refill+665}
<ffffffff80162c46>{kmem_cache_alloc+54}
      <ffffffff802e2f0a>{sk_alloc+58}
<ffffffff80323d99>{tcp_create_openreq_child+41}
      <ffffffff803229e7>{tcp_v4_syn_recv_sock+87}
<ffffffff80323b7e>{tcp_check_req+606}
      <ffffffff8030b4dc>{ip_finish_output+508}
<ffffffff80308b90>{ip_dst_output+0}
      <ffffffff8030b668>{ip_output+216} <ffffffff80308bc8>{ip_dst_output+56}
      <ffffffff802f3a43>{nf_hook_slow+195} <ffffffff80308b90>{ip_dst_output+0}
      <ffffffff80132f9b>{recalc_task_prio+635}
<ffffffff80133c91>{activate_task+129}
      <ffffffff8013340a>{task_rq_lock+74}
<ffffffff80132633>{__wake_up_common+67}
      <ffffffff80313e1b>{tcp_fastretrans_alert+859}
<ffffffff8031536d>{tcp_ack+2541}
      <ffffffff80320b79>{tcp_v4_do_rcv+233}
<ffffffffa017b034>{:ipt_state:match+36}
      <ffffffff803214cb>{tcp_v4_rcv+1835}
<ffffffff80305fb9>{ip_local_deliver_finish+297}
      <ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff802f3a43>{nf_hook_slow+195}
      <ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff803062fe>{ip_local_deliver+622}
      <ffffffff8030592e>{ip_rcv_finish+574} <ffffffff803056f0>{ip_rcv_finish+0}
      <ffffffff803056f0>{ip_rcv_finish+0} <ffffffff802f3a43>{nf_hook_slow+195}
      <ffffffff803056f0>{ip_rcv_finish+0} <ffffffff80305e34>{ip_rcv+1188}
      <ffffffff802e9749>{netif_receive_skb+729}
<ffffffffa00e9c9c>{:e1000:e1000_clean+1820}
      <ffffffff80132c13>{__wake_up+67} <ffffffff802e85b4>{net_rx_action+132}
      <ffffffff8013dca1>{__do_softirq+113} <ffffffff8013dd55>{do_softirq+53}
      <ffffffff80113e3f>{do_IRQ+335} <ffffffff8010f540>{mwait_idle+0}
      <ffffffff80110cf5>{ret_from_intr+0}  <EOI>
<ffffffff8010f596>{mwait_idle+86}
      <ffffffff8010f9ea>{cpu_idle+26} <ffffffff804e971a>{start_kernel+490}
      <ffffffff804e91e0>{_sinittext+480}

RE: arch/xen is a bad idea

[Ian Pratt]

 > > I think there's an interim compromise position that 
> everyone might go
> > for:
> > 
> > Phase 1 is for us to submit a load of patches that squeeze 
> out the low
> > hanging fruit in unifying xen/i386 and i386. Most of these will be
> > strict cleanups to i386, and the result will be to almost halve the
> > number of files that we need to modify.
> 
> OK.  It would be good to have a phase 0: any refactoring, 
> abstracting, etc
> to the core kernel and to i386 which is a preparatory step, prior to
> introducing any Xen code.  After phase 0 everything should 
> still compile
> and run.  The subsequent Xen patches should merely add stuff 
> and not move
> existing code around.

I think my phase 1 and your phase 0 are actually the same. I'm not
proposing that we put any Xen code in the tree at this point, but that
we do the many easy cleanups to i386 which will mean that we can remove
modifications to files from the arch/xen tree we maintain .

Example cleanups would be: use a macro for testing to see whether you're
in the kernel or not; use isa_bus_to_virt instead of phys_to_virt for
the access the address of the VGA console etc etc.

> What would you propose doing with the i386 header files?  Such as the
> pagetable handling?

We'd certainly still need to have xen-specific versions of some of the
i386 header files. Perhaps the neatest thing is to have an
include/asm-i386/xen ahead of include/asm-i386 on the include search
path?  Not sure.


Ian 

Re: arch/xen is a bad idea

[Andrew Morton]

"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> wrote:
>
>  
> > The Xen team still believe that it's best to keep arch/xen, 
> > arch/xen/i386,
> > arch/xen/x86_64, etc.  And I believe that Andi (who is the 
> > world expert on
> > maintaining an i386 derivative) thinks that this is will be a 
> > long-term
> > maintenance problem.
> 
> I think there's an interim compromise position that everyone might go
> for:
> 
> Phase 1 is for us to submit a load of patches that squeeze out the low
> hanging fruit in unifying xen/i386 and i386. Most of these will be
> strict cleanups to i386, and the result will be to almost halve the
> number of files that we need to modify.

OK.  It would be good to have a phase 0: any refactoring, abstracting, etc
to the core kernel and to i386 which is a preparatory step, prior to
introducing any Xen code.  After phase 0 everything should still compile
and run.  The subsequent Xen patches should merely add stuff and not move
existing code around.

> The next phase is that we re-organise the current arch/xen as follows:
> 
> We move the remaining (reduced) contents of arch/xen/i386 to
> arch/i386/xen (ditto for x86_64). We then move the xen-specific files
> that are shared between all the different xen architectures to
> drivers/xen/core. I know this last step is a bit odd, but it's the best
> location that Rusty Russel and I could come up with.
> 
> At this point, I'd hope that we could get xen into the main-line tree.

What would you propose doing with the i386 header files?  Such as the
pagetable handling?

> The final phase is to see if we can further unify more native and xen
> files. This is going to require some significant i386 code refactoring,
> and I think its going to be much easier to do if all the code is in the
> main-line tree so that people can see the motivation for what's going
> on.
> 
> What do you think?

It sounds decent.  The main objective is to minimise code duplication.  The
question of where in the tree all the resulting code actually lands is
secondary from a maintainability POV.

Re: arch/xen is a bad idea

[Andi Kleen]

> Phase 1 is for us to submit a load of patches that squeeze out the low
> hanging fruit in unifying xen/i386 and i386. Most of these will be
> strict cleanups to i386, and the result will be to almost halve the
> number of files that we need to modify.

Sounds good. I would try to track that for x86-64 too then when
possible to make the later x86-64 merge easier.

> 
> The next phase is that we re-organise the current arch/xen as follows:
> 
> We move the remaining (reduced) contents of arch/xen/i386 to
> arch/i386/xen (ditto for x86_64). We then move the xen-specific files

What would these files be? 

> that are shared between all the different xen architectures to
> drivers/xen/core. I know this last step is a bit odd, but it's the best
> location that Rusty Russel and I could come up with.
> 
> At this point, I'd hope that we could get xen into the main-line tree.
> 
> The final phase is to see if we can further unify more native and xen
> files. This is going to require some significant i386 code refactoring,
> and I think its going to be much easier to do if all the code is in the
> main-line tree so that people can see the motivation for what's going
> on.

Hmm, I would prefer to do that during the merge. I'm not sure there
will be that much push afterwards to unify stuff, and then we might
be stuck with an inferior setup.

I don't think it makes much difference for review if the previous
code is in mainline or not.

-Andi

RE: arch/xen is a bad idea

[Ian Pratt]

 
> The Xen team still believe that it's best to keep arch/xen, 
> arch/xen/i386,
> arch/xen/x86_64, etc.  And I believe that Andi (who is the 
> world expert on
> maintaining an i386 derivative) thinks that this is will be a 
> long-term
> maintenance problem.

I think there's an interim compromise position that everyone might go
for:

Phase 1 is for us to submit a load of patches that squeeze out the low
hanging fruit in unifying xen/i386 and i386. Most of these will be
strict cleanups to i386, and the result will be to almost halve the
number of files that we need to modify.

The next phase is that we re-organise the current arch/xen as follows:

We move the remaining (reduced) contents of arch/xen/i386 to
arch/i386/xen (ditto for x86_64). We then move the xen-specific files
that are shared between all the different xen architectures to
drivers/xen/core. I know this last step is a bit odd, but it's the best
location that Rusty Russel and I could come up with.

At this point, I'd hope that we could get xen into the main-line tree.

The final phase is to see if we can further unify more native and xen
files. This is going to require some significant i386 code refactoring,
and I think its going to be much easier to do if all the code is in the
main-line tree so that people can see the motivation for what's going
on.

What do you think?

Best,
Ian