panic in e100_exec_cb()

panic in e100_exec_cb()

[Charles Coffing]

With today's build, my domain 0 crashes during boot when it tries to
bring eth0 up (it's an E100).

cb->prev (eax) is NULL in e100_exec_cb() (e100.c:827).  Just from code
inspection, I don't see how this can be.  e100_alloc_cbs() was just
called, which looks like it should have correctly linked up all the
cb->prev/cb->next pointers.

It happens regardless of CONFIG_XEN_WRITABLE_PAGETABLES.

Has anyone else seen this?


e100: eth0: e100_watchdog: link up, 100Mbps, full-duplex
Unable to handle kernel NULL pointer dereference at virtual address
00000002
 printing eip:
c022d6d2
*pde = ma 00000000 pa 55555000
 [<c022ea3a>] e100_alloc_cbs+0x74/0x140
 [<c022f5bb>] e100_up+0x39/0x1d6
 [<c0127f8b>] queue_delayed_work+0x50/0x6d
 [<c02307b3>] e100_open+0x23/0x69
 [<c031198b>] dev_open+0x8a/0xe6
 [<c0312e6e>] dev_change_flags+0x51/0x11f
 [<c03479d6>] devinet_ioctl+0x4f1/0x5b2
 [<c03496b9>] inet_ioctl+0x7e/0xa8
 [<c030a026>] sock_ioctl+0x184/0x25d
 [<c015c4e6>] sys_ioctl+0x16f/0x211
 [<c010d1b7>] syscall_call+0x7/0xb
Oops: 0002 [#1]
Modules linked in:
CPU:    0
EIP:    0061:[<c022d6d2>]    Not tainted
EFLAGS: 00013202   (2.6.8.1-xen0)
EIP is at e100_hw_init+0x19c/0x6e3
eax: 00000000   ebx: c66c8000   ecx: 00000005   edx: c7c19220
esi: c7c19220   edi: 0000003f   ebp: 00000000   esp: c73cfe40
ds: 0069   es: 0069   ss: 0069
Process ip (pid: 1708, threadinfo=c73ce000 task=c745a680)
Stack: c7c19220 c66c8000 00000000 00000040 c7c19220 00000000 c022ea3a
c1199844
       00006700 c7c19484 00000000 00000000 00000000 c7c19220 c7c19220
00000000
       c022f5bb c7c19220 c7fee260 0000004f c0127f8b c03f6698 c7c19000
00000000
Call Trace:
 [<c022ea3a>] e100_alloc_cbs+0x74/0x140
 [<c022f5bb>] e100_up+0x39/0x1d6
 [<c0127f8b>] queue_delayed_work+0x50/0x6d
 [<c02307b3>] e100_open+0x23/0x69
 [<c031198b>] dev_open+0x8a/0xe6
 [<c0312e6e>] dev_change_flags+0x51/0x11f
 [<c03479d6>] devinet_ioctl+0x4f1/0x5b2
 [<c03496b9>] inet_ioctl+0x7e/0xa8
 [<c030a026>] sock_ioctl+0x184/0x25d
 [<c015c4e6>] sys_ioctl+0x16f/0x211
 [<c010d1b7>] syscall_call+0x7/0xb
Code: 66 81 60 02 ff bf 8b 86 14 01 00 00 3b 86 10 01 00 00 3e 75



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: panic in e100_exec_cb()

[Paul Dorman]

Yes, I get the same thing. Server reboots right after the line 
'e1000_watchdog: NIC link is up...' (similar hardware to yours no doubt). I 
don't see any messages. It sits for about 10 seconds before restarting.

I do see this in the syslog, which appears strange to me:

Xen1 kernel: Symbols match kernel version 2.6.8.
Xen1 kernel: No module symbols loaded - kernel modules not enabled.

Also:

Xen1 kernel: Kernel command line: /boot/vmlinuz-2.6.8.1-xen0 root=/dev/sda1 ro 
console=tty0 console=ttyS0
Xen1 kernel: Unknown boot option `/boot/vmlinuz-2.6.8.1-xen0' : ignoring

By grub boot instructions are: 

kernel /boot/xen.gz dom0_mem=131072 com21=115200,8n1
module /boot/vmlinuz-2.6.8.1-xen0 root=/dev/sda1 ro console=tty0 console=ttyS0

Is this not correct for the 2.6.x kernels?

Regards,
Paul


On Tuesday 21 September 2004 10:22 am, Charles Coffing wrote:
> With today's build, my domain 0 crashes during boot when it tries to
> bring eth0 up (it's an E100).
>
> cb->prev (eax) is NULL in e100_exec_cb() (e100.c:827).  Just from code
> inspection, I don't see how this can be.  e100_alloc_cbs() was just
> called, which looks like it should have correctly linked up all the
> cb->prev/cb->next pointers.
>
> It happens regardless of CONFIG_XEN_WRITABLE_PAGETABLES.
>
> Has anyone else seen this?
>
>
> e100: eth0: e100_watchdog: link up, 100Mbps, full-duplex
> Unable to handle kernel NULL pointer dereference at virtual address
> 00000002
>  printing eip:
> c022d6d2
> *pde = ma 00000000 pa 55555000
>  [<c022ea3a>] e100_alloc_cbs+0x74/0x140
>  [<c022f5bb>] e100_up+0x39/0x1d6
>  [<c0127f8b>] queue_delayed_work+0x50/0x6d
>  [<c02307b3>] e100_open+0x23/0x69
>  [<c031198b>] dev_open+0x8a/0xe6
>  [<c0312e6e>] dev_change_flags+0x51/0x11f
>  [<c03479d6>] devinet_ioctl+0x4f1/0x5b2
>  [<c03496b9>] inet_ioctl+0x7e/0xa8
>  [<c030a026>] sock_ioctl+0x184/0x25d
>  [<c015c4e6>] sys_ioctl+0x16f/0x211
>  [<c010d1b7>] syscall_call+0x7/0xb
> Oops: 0002 [#1]
> Modules linked in:
> CPU:    0
> EIP:    0061:[<c022d6d2>]    Not tainted
> EFLAGS: 00013202   (2.6.8.1-xen0)
> EIP is at e100_hw_init+0x19c/0x6e3
> eax: 00000000   ebx: c66c8000   ecx: 00000005   edx: c7c19220
> esi: c7c19220   edi: 0000003f   ebp: 00000000   esp: c73cfe40
> ds: 0069   es: 0069   ss: 0069
> Process ip (pid: 1708, threadinfo=c73ce000 task=c745a680)
> Stack: c7c19220 c66c8000 00000000 00000040 c7c19220 00000000 c022ea3a
> c1199844
>        00006700 c7c19484 00000000 00000000 00000000 c7c19220 c7c19220
> 00000000
>        c022f5bb c7c19220 c7fee260 0000004f c0127f8b c03f6698 c7c19000
> 00000000
> Call Trace:
>  [<c022ea3a>] e100_alloc_cbs+0x74/0x140
>  [<c022f5bb>] e100_up+0x39/0x1d6
>  [<c0127f8b>] queue_delayed_work+0x50/0x6d
>  [<c02307b3>] e100_open+0x23/0x69
>  [<c031198b>] dev_open+0x8a/0xe6
>  [<c0312e6e>] dev_change_flags+0x51/0x11f
>  [<c03479d6>] devinet_ioctl+0x4f1/0x5b2
>  [<c03496b9>] inet_ioctl+0x7e/0xa8
>  [<c030a026>] sock_ioctl+0x184/0x25d
>  [<c015c4e6>] sys_ioctl+0x16f/0x211
>  [<c010d1b7>] syscall_call+0x7/0xb
> Code: 66 81 60 02 ff bf 8b 86 14 01 00 00 3b 86 10 01 00 00 3e 75
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: panic in e100_exec_cb()

[Keir Fraser]

> With today's build, my domain 0 crashes during boot when it tries to
> bring eth0 up (it's an E100).
> 
> cb->prev (eax) is NULL in e100_exec_cb() (e100.c:827).  Just from code
> inspection, I don't see how this can be.  e100_alloc_cbs() was just
> called, which looks like it should have correctly linked up all the
> cb->prev/cb->next pointers.
> 
> It happens regardless of CONFIG_XEN_WRITABLE_PAGETABLES.
> 
> Has anyone else seen this?

When was your last successful build? Most likely the changeset in
which I introduced page scrubbing is causing you problems. It looks
like you are failing on the very first use of a 'cb', so tracing
should lead you to the cause of the problem quite easily.

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: panic in e100_exec_cb()

[Keir Fraser]

> I do see this in the syslog, which appears strange to me:
> 
> Xen1 kernel: Symbols match kernel version 2.6.8.
> Xen1 kernel: No module symbols loaded - kernel modules not enabled.

You built a kernel with no module support?

> Also:
> 
> Xen1 kernel: Kernel command line: /boot/vmlinuz-2.6.8.1-xen0 root=/dev/sda1 ro 
> console=tty0 console=ttyS0
> Xen1 kernel: Unknown boot option `/boot/vmlinuz-2.6.8.1-xen0' : ignoring

That looks like a bug in our handling of the command line. We should
strip the kernel-image name.

> By grub boot instructions are: 
> 
> kernel /boot/xen.gz dom0_mem=131072 com21=115200,8n1
> module /boot/vmlinuz-2.6.8.1-xen0 root=/dev/sda1 ro console=tty0 console=ttyS0
> 
> Is this not correct for the 2.6.x kernels?

In general, handling of two 'console=' commands is a bit suspect --
this is true of vanilla Linux as well. IT's not always obvious where
your boot messages will go after init starts. :-)

Apart from that it looks fine.

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: panic in e100_exec_cb()

[Ian Pratt]

> > With today's build, my domain 0 crashes during boot when it tries to
> > bring eth0 up (it's an E100).
> > 
> > cb->prev (eax) is NULL in e100_exec_cb() (e100.c:827).  Just from code
> > inspection, I don't see how this can be.  e100_alloc_cbs() was just
> > called, which looks like it should have correctly linked up all the
> > cb->prev/cb->next pointers.
> > 
> > Has anyone else seen this?

Yep -- I can reproduce. It appears to be something e100 specific,
as I've tested that tg3, e1000 and 3c95x all work fine. 

Should be easy to fix.

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: panic in e100_exec_cb()

[Ian Pratt]

> With today's build, my domain 0 crashes during boot when it tries to
> bring eth0 up (it's an E100).
> 
> cb->prev (eax) is NULL in e100_exec_cb() (e100.c:827).  Just from code
> inspection, I don't see how this can be.  e100_alloc_cbs() was just
> called, which looks like it should have correctly linked up all the
> cb->prev/cb->next pointers.

OK, we've tracked this down to a long-standing bug due to a
missing local_flush_tlb when creating a contiguous mapping of a
PCI region.

The scrub_pages change was having the effect of nicely loading
the TLB with mappings that were just about to to become stale...

This could explain most of the odd behaviour that people have
been seeing since the apparently innocuous scrub pages checkin.

We'll do some more testing and push out a fix later today.

Ian






-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Problems rebooting xen guest

[Thomas E. Conway-Poulsen]

Hi,

I am having problems rebooting the xen guest OS using the build from the 
28 Aug 2004.

I get this following trace, where it then stops.

Rebooting... Restarting system.
Linux version 2.6.8.1-xenU (root@tecp-fw) (gcc version 3.3.4 (Debian 
1:3.3.4-9)) #1 Mon Sep 6 18:56:40 CEST 2004
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 0000000008000000 (usable)
128MB LOWMEM available.
DMI not present.
Built 1 zonelists
Kernel command line:  ip=:1.2.3.4::::eth0:dhcp root=/dev/sda1 rw 4 
sched=fvbt
Initializing CPU#0
PID hash table entries: 1024 (order 10: 8192 bytes)
Xen reported: 1470.058 MHz processor.
Using tsc for high-res timesource
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 127112k/131072k available (1550k kernel code, 3792k reserved, 
468k data, 92k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay loop... 418.61 BogoMIPS
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 256K (64 bytes/line)
CPU: AMD Athlon(tm) XP 1700+ stepping 02
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... disabled
NET: Registered protocol family 16
Initializing Cryptographic API
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Initialising Xen virtual block device
Xen virtual console successfully installed as tty
Event-channel device installed.
Initialising Xen virtual ethernet frontend driver.
[XEN] Failed to connect all virtual interfaces: err=-100
NET: Registered protocol family 2
IP: routing cache hash table of 1024 buckets, 8Kbytes
TCP: Hash tables configured (established 8192 bind 16384)
NET: Registered protocol family 1
NET: Registered protocol family 17
IP-Config: Device `eth0' not found.

I am using dummy0 on the bridge.
Is this something that may have been fixed in newer versions?

Also, the hole box also seems to break stop functioning some times, when 
there is heavy network load. I will probably have to select debug when 
compiling to figure that out.

Even if I destroy the xen instance and create it again, its the same 
stop position.

Best regards,

Thomas Conway.


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: panic in e100_exec_cb()

[Brian Wolfe]

On Tue, 2004-09-21 at 02:36, Keir Fraser wrote:
> > I do see this in the syslog, which appears strange to me:
> > 
> > Xen1 kernel: Symbols match kernel version 2.6.8.
> > Xen1 kernel: No module symbols loaded - kernel modules not enabled.
> 
> You built a kernel with no module support?

in my experience, this is a very common config scenario for dedicated
use machines with competant (and control intensive) admins. 



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: panic in e100_exec_cb()

[Paul Dorman]

On Tuesday 21 September 2004 07:36 pm, Keir Fraser wrote:
> > I do see this in the syslog, which appears strange to me:
> >
> > Xen1 kernel: Symbols match kernel version 2.6.8.
> > Xen1 kernel: No module symbols loaded - kernel modules not enabled.
>
> You built a kernel with no module support?

I followed the instructions in the Xen v2.0 user's manual: 'make world', 'make 
install'. :o) I haven't tried to do anything more sophisticated yet. Once I 
have Xen working, I'll muck around with customizing my kernel builds.

>
> > Also:
> >
> > Xen1 kernel: Kernel command line: /boot/vmlinuz-2.6.8.1-xen0
> > root=/dev/sda1 ro console=tty0 console=ttyS0
> > Xen1 kernel: Unknown boot option `/boot/vmlinuz-2.6.8.1-xen0' : ignoring
>
> That looks like a bug in our handling of the command line. We should
> strip the kernel-image name.

Glad to point it out then!

> > By grub boot instructions are:
> >
> > kernel /boot/xen.gz dom0_mem=131072 com21=115200,8n1
> > module /boot/vmlinuz-2.6.8.1-xen0 root=/dev/sda1 ro console=tty0
> > console=ttyS0
> >
> > Is this not correct for the 2.6.x kernels?
> In general, handling of two 'console=' commands is a bit suspect --
> this is true of vanilla Linux as well. IT's not always obvious where
> your boot messages will go after init starts. :-)
>
> Apart from that it looks fine.

This also is in the v2.0 user's manual. Am I right in thinking it needs a bit 
of editing? I'll remove one of them then.

Thanks Keir!

Paul.


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: panic in e100_exec_cb()

[Paul Dorman]

On Tuesday 21 September 2004 07:37 pm, Ian Pratt wrote:
> > > With today's build, my domain 0 crashes during boot when it tries to
> > > bring eth0 up (it's an E100).
> > >
> > > cb->prev (eax) is NULL in e100_exec_cb() (e100.c:827).  Just from code
> > > inspection, I don't see how this can be.  e100_alloc_cbs() was just
> > > called, which looks like it should have correctly linked up all the
> > > cb->prev/cb->next pointers.
> > >
> > > Has anyone else seen this?
>
> Yep -- I can reproduce. It appears to be something e100 specific,
> as I've tested that tg3, e1000 and 3c95x all work fine.

Ooh! Looks like I have a different problem then! Any ideas? I imagine it could 
be the no modules issue. I'll investigate over the next few hours.

>
> Should be easy to fix.
>
> Ian


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: panic in e100_exec_cb()

[Keir Fraser]

> On Tuesday 21 September 2004 07:37 pm, Ian Pratt wrote:
> > > > With today's build, my domain 0 crashes during boot when it tries to
> > > > bring eth0 up (it's an E100).
> > > >
> > > > cb->prev (eax) is NULL in e100_exec_cb() (e100.c:827).  Just from code
> > > > inspection, I don't see how this can be.  e100_alloc_cbs() was just
> > > > called, which looks like it should have correctly linked up all the
> > > > cb->prev/cb->next pointers.
> > > >
> > > > Has anyone else seen this?
> >
> > Yep -- I can reproduce. It appears to be something e100 specific,
> > as I've tested that tg3, e1000 and 3c95x all work fine.
> 
> Ooh! Looks like I have a different problem then! Any ideas? I imagine it could 
> be the no modules issue. I'll investigate over the next few hours.

Definitely only have one 'console=' option on DOM0's command line. It
may be that some infomration is only getting output to a device that
you are not looking at. It's very unlikely that the machine will
reboot with no visible error message at all.

Are you viewing DOM0 booting over serial line or on the VGA console?
If the former, then specify console=ttyS0 on DOM0's command line. If
the latter then specify console=tty0.

Not that for all other domains you should always specify console=tty0.

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: Problems rebooting xen guest

[Ian Pratt]

> I am having problems rebooting the xen guest OS using the build from the 
> 28 Aug 2004.

I'd certainly upgrade before doing anything further. There's been
some useful bug fixes.

> Initialising Xen virtual ethernet frontend driver.
> [XEN] Failed to connect all virtual interfaces: err=-100
> IP-Config: Device `eth0' not found.
> 
> I am using dummy0 on the bridge.
> Is this something that may have been fixed in newer versions?

Why do you need to use dummy0 on the bridge? Are you doing this
in lieu of a LAN interface?

Can you look in /var/log/xend.* to see if anything is being
logged as to why the netfront driver isn't being connected.
Are you seeing a vif appear in domain 0 if you do an ifconfig to
list them?

> Also, the hole box also seems to break stop functioning some times, when 
> there is heavy network load. I will probably have to select debug when 
> compiling to figure that out.

This should be fixed in newer builds.

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php

Re: Problems rebooting xen guest

[Keir Fraser]

> Hi,
> 
> I am having problems rebooting the xen guest OS using the build from the 
> 28 Aug 2004.
> 
> I get this following trace, where it then stops.

You have stale config files. Do a 'make mrproper ; make world' at the
root of the xen repository.

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php