kernel BUG at drivers/net/xen-netback/netback.c:540!

kernel BUG at drivers/net/xen-netback/netback.c:540!

[Atom2]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

Hi guys,
I just updated my kernel version for dom0 from 3.13.10 to 3.14.5 (both 
gentoo hardened sources) when one of my virtual machines still running 
on 3.13.10 suddenly froze. At that time I was about to update the kernel 
for my domU when it froze. My XEN version is 4.3.2.

I then tried to shutdown the domU from the dom0 (xl shutdown name) which 
seemingly worked (shutdown messages appeard on the console for the 
domU). A restart threw an error message which I missed to record . In xl 
list however, the old domU line was still there with a name of (null) 
and a state of "--ps-d".

Shutdown of other domUs to be able to restart the machine resulted in 
the same xl list output: name as (null) and "--ps-d" state, although 
according to their log files under /var/lo/xen-consoles they seem to 
stop correctly. What I however noted was that the LVMs for all domUs 
were still open after their shutdown and they [blkback.N.xvdayN] were 
still available in the process list on dom0.

Investigation then revealed that the messages as per the attached file 
had been recorded in dmesg on dom0.

For now I have decided to switch back to kernel version 3.13.10 which 
was running absolutely stable for a number of weeks.

I am more than happy to provide further information if needed.

Thanks Atom2

[-- Attachment #2: BUG.txt --]
[-- Type: text/plain, Size: 5720 bytes --]

[ 6653.419840] ------------[ cut here ]------------
[ 6653.420547] kernel BUG at drivers/net/xen-netback/netback.c:540!
[ 6653.421246] invalid opcode: 0000 [#1] SMP
[ 6653.421957] Modules linked in: md5 i915 cfbfillrect cfbimgblt i2c_algo_bit cfbcopyarea drm_kms_helper drm xhci_hcd 8139too mii i2c_i801 i2c_core
[ 6653.423482] CPU: 0 PID: 4939 Comm: vif5.0 Not tainted 3.14.5-hardened-r2 #1
[ 6653.424257] Hardware name: iEi B199/B199, BIOS B199AR23 09/17/2013
[ 6653.425037] task: ffff880119486c80 ti: ffff880119487198 task.ti: ffff880119487198
[ 6653.425827] RIP: e030:[<ffffffff81480e71>]  [<ffffffff81480e71>] xenvif_rx_action+0x811/0x820
[ 6653.426637] RSP: e02b:ffff8800d32cdd98  EFLAGS: 00010297
[ 6653.427445] RAX: 0000000000000001 RBX: ffff8800d3330740 RCX: 3fffffffffffff00
[ 6653.428278] RDX: 0000000000000013 RSI: ffff8800d5153388 RDI: 0000000000000000
[ 6653.429115] RBP: 0000000000000011 R08: ffff8800d32cddd4 R09: 0000000000000001
[ 6653.429957] R10: 0000160000000000 R11: ffff8800d3330740 R12: 0000000000000011
[ 6653.430796] R13: ffff8800d5153388 R14: ffff8800d32cddd4 R15: ffff8800d3330740
[ 6653.431649] FS:  00007fc94120d700(0000) GS:ffff88011ee00000(0000) knlGS:0000000000000000
[ 6653.432520] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6653.433394] CR2: 00007f986cc730a0 CR3: 00000000d0aa5000 CR4: 0000000000042660
[ 6653.434295] Stack:
[ 6653.435186]  ffff8800d32cddd4 ffff88011ee0f180 ffff8800d32cdde8 000000121a2b15aa
[ 6653.436113]  0002b7e200000011 ffff8800d3337258 0000000000000000 0000000081098185
[ 6653.437053]  ffff8800d32cddd8 ffff8800d32cddd8 ffff8800d51538c8 ffff8800d51538c8
[ 6653.437997] Call Trace:
[ 6653.438925]  [<ffffffff8148299c>] ? xenvif_kthread+0x4c/0x1c0
[ 6653.439868]  [<ffffffff810a7fd0>] ? __wake_up_sync+0x10/0x10
[ 6653.440817]  [<ffffffff81482950>] ? xenvif_stop_queue+0x50/0x50
[ 6653.441772]  [<ffffffff8108facc>] ? kthread+0xbc/0xe0
[ 6653.442735]  [<ffffffff810151f5>] ? sched_clock+0x5/0x10
[ 6653.443694]  [<ffffffff81090000>] ? kthread_should_park+0x10/0x20
[ 6653.444656]  [<ffffffff8108fa10>] ? kthread_freezable_should_stop+0x60/0x60
[ 6653.445637]  [<ffffffff815ea044>] ? ret_from_fork+0x74/0xa0
[ 6653.446627]  [<ffffffff8108fa10>] ? kthread_freezable_should_stop+0x60/0x60
[ 6653.447627] Code: 00 c7 42 0c 00 00 00 00 e9 7c fe ff ff c6 44 24 20 00 e9 a3 f9 ff ff 83 c8 03 e9 1c fb ff ff 31 ed b8 04 00 00 00 e9 f5 fa ff ff <0f> 0b 0f 0b 0f 0b 66 0f 1f 84 00 00 00 00 00 44 8b 87 98 00 00
[ 6653.449830] RIP  [<ffffffff81480e71>] xenvif_rx_action+0x811/0x820
[ 6653.450913]  RSP <ffff8800d32cdd98>
[ 6653.624856] ---[ end trace 5d9a58b0d1c65342 ]---
[ 7006.778776] BUG: unable to handle kernel NULL pointer dereference at            (nil)
[ 7006.779951] IP: [<ffffffff81094a22>] exit_creds+0x12/0x60
[ 7006.781106] PGD 0
[ 7006.782239] Oops: 0000 [#2] SMP
[ 7006.783381] Modules linked in: md5 i915 cfbfillrect cfbimgblt i2c_algo_bit cfbcopyarea drm_kms_helper drm xhci_hcd 8139too mii i2c_i801 i2c_core
[ 7006.785786] CPU: 7 PID: 56 Comm: xenwatch Tainted: G      D      3.14.5-hardened-r2 #1
[ 7006.787009] Hardware name: iEi B199/B199, BIOS B199AR23 09/17/2013
[ 7006.788252] task: ffff88011a3e0f80 ti: ffff88011a3e1498 task.ti: ffff88011a3e1498
[ 7006.789507] RIP: e030:[<ffffffff81094a22>]  [<ffffffff81094a22>] exit_creds+0x12/0x60
[ 7006.790783] RSP: e02b:ffff880119c63d68  EFLAGS: 00010287
[ 7006.792049] RAX: 0000000000000000 RBX: ffff880119486c80 RCX: 0000000000000020
[ 7006.793328] RDX: 0000000000000001 RSI: 0000000080000200 RDI: 0000000000000000
[ 7006.794609] RBP: 0000000000000000 R08: 0000000080000200 R09: ffff88011a578000
[ 7006.795898] R10: 000000000000000e R11: 0000000000000000 R12: ffffffff813c0d70
[ 7006.797179] R13: 0000000000000020 R14: ffff8800d4226880 R15: 0000000000000000
[ 7006.798455] FS:  00007f0ce67f2700(0000) GS:ffff88011efc0000(0000) knlGS:0000000000000000
[ 7006.799733] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7006.801013] CR2: 0000000000000000 CR3: 00000000d47ea000 CR4: 0000000000042660
[ 7006.802294] Stack:
[ 7006.803550]  ffff880119486c80 ffffffff8106b3bc ffff880119c63d78 ffff880119486c80
[ 7006.804816]  0000000000000000 ffffffff8108fddc 0000000000000020 ffffffff00000000
[ 7006.806077]  0000000000000020 ffff8800d3330740 0000000000000020 ffffffff814845b4
[ 7006.807329] Call Trace:
[ 7006.808565]  [<ffffffff8106b3bc>] ? __put_task_struct+0x3c/0x110
[ 7006.809802]  [<ffffffff8108fddc>] ? kthread_stop+0x5c/0x70
[ 7006.811025]  [<ffffffff814845b4>] ? xenvif_disconnect+0x24/0x70
[ 7006.812240]  [<ffffffff81482dd3>] ? set_backend_state+0x103/0x920
[ 7006.813453]  [<ffffffff813bf666>] ? xenbus_read_driver_state+0x26/0x40
[ 7006.814683]  [<ffffffff813c0d70>] ? xs_watch+0x60/0x60
[ 7006.815905]  [<ffffffff813c0df2>] ? xenwatch_thread+0x82/0x140
[ 7006.817130]  [<ffffffff810a7fd0>] ? __wake_up_sync+0x10/0x10
[ 7006.818333]  [<ffffffff8108facc>] ? kthread+0xbc/0xe0
[ 7006.819529]  [<ffffffff810151f5>] ? sched_clock+0x5/0x10
[ 7006.820714]  [<ffffffff81090000>] ? kthread_should_park+0x10/0x20
[ 7006.821887]  [<ffffffff8108fa10>] ? kthread_freezable_should_stop+0x60/0x60
[ 7006.823054]  [<ffffffff815ea044>] ? ret_from_fork+0x74/0xa0
[ 7006.824216]  [<ffffffff8108fa10>] ? kthread_freezable_should_stop+0x60/0x60
[ 7006.825377] Code: 0f 85 72 fe ff ff 85 c0 0f 84 8e fe ff ff e9 65 fe ff ff 0f 0b 0f 0b 66 90 53 48 89 fb 48 8b 87 e8 06 00 00 48 8b bf 50 06 00 00 <8b> 00 48 c7 83 50 06 00 00 00 00 00 00 f0 ff 0f 74 1c 48 8b bb
[ 7006.827940] RIP  [<ffffffff81094a22>] exit_creds+0x12/0x60
[ 7006.829167]  RSP <ffff880119c63d68>
[ 7006.830389] CR2: 0000000000000000
[ 7006.831605] ---[ end trace 5d9a58b0d1c65343 ]---
[ 7006.832975] xenbr0: port 6(vif5.0) entered disabled state

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: kernel BUG at drivers/net/xen-netback/netback.c:540!

[Ian Campbell]

On Thu, 2014-06-12 at 01:05 +0200, Atom2 wrote:
> Hi guys,
> I just updated my kernel version for dom0 from 3.13.10 to 3.14.5 (both 
> gentoo hardened sources) when one of my virtual machines still running 
> on 3.13.10 suddenly froze. At that time I was about to update the kernel 
> for my domU when it froze. My XEN version is 4.3.2.

Looks to be the same as
http://lists.xen.org/archives/html/xen-devel/2014-06/msg01307.html
for which Zoltan referred to:
http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/drivers/net/xen-netback?id=59ae9fc67007da8b5aea7b0a31c3607745cfbfee

Cheers,
Ian.

Re: kernel BUG at drivers/net/xen-netback/netback.c:540!

[Atom2]

Am 12.06.14 10:41, schrieb Ian Campbell:
> On Thu, 2014-06-12 at 01:05 +0200, Atom2 wrote:
>> Hi guys,
>> I just updated my kernel version for dom0 from 3.13.10 to 3.14.5 (both
>> gentoo hardened sources) when one of my virtual machines still running
>> on 3.13.10 suddenly froze. At that time I was about to update the kernel
>> for my domU when it froze. My XEN version is 4.3.2.
>
> Looks to be the same as
> http://lists.xen.org/archives/html/xen-devel/2014-06/msg01307.html
> for which Zoltan referred to:
> http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/drivers/net/xen-netback?id=59ae9fc67007da8b5aea7b0a31c3607745cfbfee
>
Thanks Ian for your quick reply. The issue does indeed look similar on 
the face of it (alos given that I also do use an NFS mounted filesystem 
that was accessed at the time the problem occured). Unfortunately the 
patch seems to be for 3.15 only as it does not apply to my 3.14.5 
kernel. The error messages read as follows:

patching file drivers/net/xen-netback/netback.c
Hunk #1 succeeded at 164 (offset 1 line).
Hunk #2 succeeded at 177 (offset 1 line).
Hunk #3 succeeded at 188 (offset 1 line).
Hunk #4 succeeded at 197 (offset 1 line).
Hunk #5 succeeded at 233 (offset 1 line).
Hunk #6 succeeded at 272 (offset -1 lines).
Hunk #7 FAILED at 556.
Hunk #8 succeeded at 526 (offset -91 lines).
1 out of 8 hunks FAILED -- saving rejects to file 
drivers/net/xen-netback/netback.c.rej

Hunk #7 refers to the following piece of code in the patch:
@@ -541,12 +556,6 @@ static void xenvif_add_frag_responses(struct xenvif 
*vif, int status,
         }
  }

-struct xenvif_rx_cb {
-       int meta_slots_used;
-};
-
-#define XENVIF_RX_CB(skb) ((struct xenvif_rx_cb *)(skb)->cb)
-
  void xenvif_kick_thread(struct xenvif *vif)
  {
         wake_up(&vif->wq);


Now in my kernel sources after the function xenvif_add_frag_responses 
referred to by the patch there's neither a definition of a
struct xenvif_rx_cb { ...
nor a line reading
#define XENVIF_RX_CB ...
(both of which the patch tries to remove)

but rather the definition of a
struct skb_cb_overlay {
	int meta_slots_used;
};
(which obviously the patch does not expect to see there)

I guess I'd therefore need a slightly modified patch for the 3.14.5 
kernel sources to make it compile.

Many thanks in advance,
Atom2