[Qemu-devel] Kernel Panic on Yum update

[Qemu-devel] Kernel Panic on Yum update

[Gerhard Wiesinger]

Helllo,

I'm using latest qemu-kvm-2.3.0-3.fc21.x86_64 from libvirt repository 
(updated afterwards). Running yum regularly crashes the VM like below. 
VM is stripped down to minimum memory requirements (256MB) for owncloud. 
See below.

Looks like a problem in virtio-net or a kernel bug with low memory 
situations.

Any ideas?

Thank you.

Ciao,
Gerhard

http://www.wiesinger.com/

kernel: swapper/0: page allocation failure: order:0, mode:0x20
kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.7-200.fc21.x86_64 #1
kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
1.8.1-20150414_141626-myhost.mydomain 04/01/2014
kernel: 0000000000000000 c35c12d2c14e1ce8 ffff88000fc03ac8 ffffffff8176f1f7
kernel: 0000000000000000 0000000000000020 ffff88000fc03b58 ffffffff811a1c9e
kernel: 0000000000000000 0000000000000040 ffff880000000000 ffff88000ffcdb28
kernel: Call Trace:
kernel: <IRQ>  [<ffffffff8176f1f7>] dump_stack+0x45/0x57
kernel: [<ffffffff811a1c9e>] warn_alloc_failed+0xfe/0x170
kernel: [<ffffffff811a607a>] __alloc_pages_nodemask+0x7ca/0xb10
kernel: [<ffffffff81691854>] ? nf_hook_slow+0x84/0x130
kernel: [<ffffffff816482df>] __alloc_page_frag+0x12f/0x150
kernel: [<ffffffff8164832f>] __netdev_alloc_frag+0x2f/0x50
kernel: [<ffffffff8164d446>] __alloc_rx_skb+0xe6/0x110
kernel: [<ffffffff8164d48f>] __netdev_alloc_skb+0x1f/0x40
kernel: [<ffffffffa008b125>] page_to_skb+0x65/0x370 [virtio_net]
kernel: [<ffffffffa008d611>] virtnet_receive+0x2c1/0x980 [virtio_net]
kernel: [<ffffffffa008dcf1>] virtnet_poll+0x21/0x90 [virtio_net]
kernel: [<ffffffff8165b6aa>] net_rx_action+0x21a/0x350
kernel: [<ffffffff810a01ab>] __do_softirq+0x10b/0x2b0
kernel: [<ffffffff810a0565>] irq_exit+0x125/0x130
kernel: [<ffffffff81778788>] do_IRQ+0x58/0xf0
kernel: [<ffffffff8177656d>] common_interrupt+0x6d/0x6d
kernel: <EOI>  [<ffffffff81103cc8>] ? hrtimer_start+0x18/0x20
kernel: [<ffffffff8105ea76>] ? native_safe_halt+0x6/0x10
kernel: [<ffffffff810fabe3>] ? rcu_eqs_enter+0xa3/0xb0
kernel: [<ffffffff8101f97e>] default_idle+0x1e/0xc0
kernel: [<ffffffff810203df>] arch_cpu_idle+0xf/0x20
kernel: [<ffffffff810de1fa>] cpu_startup_entry+0x37a/0x3c0
kernel: [<ffffffff81765757>] rest_init+0x77/0x80
kernel: [<ffffffff81d4f02c>] start_kernel+0x4a4/0x4c5
kernel: [<ffffffff81d4e120>] ? early_idt_handlers+0x120/0x120
kernel: [<ffffffff81d4e4d7>] x86_64_start_reservations+0x2a/0x2c
kernel: [<ffffffff81d4e62b>] x86_64_start_kernel+0x152/0x175
kernel: swapper/0: page allocation failure: order:0, mode:0x20
kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.7-200.fc21.x86_64 #1
kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
1.8.1-20150414_141626-myhost.mydomain 04/01/2014
kernel: 0000000000000000 c35c12d2c14e1ce8 ffff88000fc03ac8 ffffffff8176f1f7
kernel: 0000000000000000 0000000000000020 ffff88000fc03b58 ffffffff811a1c9e
kernel: 0000000000000000 0000000000000040 ffff880000000000 ffff88000ffcdb28
kernel: Call Trace:
kernel: <IRQ>  [<ffffffff8176f1f7>] dump_stack+0x45/0x57
kernel: [<ffffffff811a1c9e>] warn_alloc_failed+0xfe/0x170
kernel: [<ffffffff811a607a>] __alloc_pages_nodemask+0x7ca/0xb10
kernel: [<ffffffff81691854>] ? nf_hook_slow+0x84/0x130
kernel: [<ffffffff816482df>] __alloc_page_frag+0x12f/0x150
kernel: [<ffffffff8164832f>] __netdev_alloc_frag+0x2f/0x50
kernel: [<ffffffff8164d446>] __alloc_rx_skb+0xe6/0x110
kernel: [<ffffffff8164d48f>] __netdev_alloc_skb+0x1f/0x40
kernel: [<ffffffffa008b125>] page_to_skb+0x65/0x370 [virtio_net]
kernel: [<ffffffffa008d611>] virtnet_receive+0x2c1/0x980 [virtio_net]
kernel: [<ffffffffa008dcf1>] virtnet_poll+0x21/0x90 [virtio_net]
kernel: [<ffffffff8165b6aa>] net_rx_action+0x21a/0x350
kernel: [<ffffffff810a01ab>] __do_softirq+0x10b/0x2b0
kernel: [<ffffffff810a0565>] irq_exit+0x125/0x130
kernel: [<ffffffff81778788>] do_IRQ+0x58/0xf0
kernel: [<ffffffff8177656d>] common_interrupt+0x6d/0x6d
kernel: <EOI>  [<ffffffff81103cc8>] ? hrtimer_start+0x18/0x20
kernel: [<ffffffff8105ea76>] ? native_safe_halt+0x6/0x10
kernel: [<ffffffff810fabe3>] ? rcu_eqs_enter+0xa3/0xb0
kernel: [<ffffffff8101f97e>] default_idle+0x1e/0xc0
kernel: [<ffffffff810203df>] arch_cpu_idle+0xf/0x20
kernel: [<ffffffff810de1fa>] cpu_startup_entry+0x37a/0x3c0
kernel: [<ffffffff81765757>] rest_init+0x77/0x80
kernel: [<ffffffff81d4f02c>] start_kernel+0x4a4/0x4c5
kernel: [<ffffffff81d4e120>] ? early_idt_handlers+0x120/0x120
kernel: [<ffffffff81d4e4d7>] x86_64_start_reservations+0x2a/0x2c
kernel: [<ffffffff81d4e62b>] x86_64_start_kernel+0x152/0x175
kernel: swapper/0: page allocation failure: order:0, mode:0x20
kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.7-200.fc21.x86_64 #1
kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
1.8.1-20150414_141626-myhost.mydomain 04/01/2014
kernel: 0000000000000000 c35c12d2c14e1ce8 ffff88000fc03ac8 ffffffff8176f1f7
kernel: 0000000000000000 0000000000000020 ffff88000fc03b58 ffffffff811a1c9e
kernel: 0000000000000000 0000000000000040 ffff880000000000 ffff88000ffcdb28
kernel: Call Trace:
kernel: <IRQ>  [<ffffffff8176f1f7>] dump_stack+0x45/0x57
kernel: [<ffffffff811a1c9e>] warn_alloc_failed+0xfe/0x170
kernel: [<ffffffff811a607a>] __alloc_pages_nodemask+0x7ca/0xb10
kernel: [<ffffffff81691854>] ? nf_hook_slow+0x84/0x130
kernel: [<ffffffff816482df>] __alloc_page_frag+0x12f/0x150
kernel: [<ffffffff8164832f>] __netdev_alloc_frag+0x2f/0x50
kernel: [<ffffffff8164d446>] __alloc_rx_skb+0xe6/0x110
kernel: [<ffffffff8164d48f>] __netdev_alloc_skb+0x1f/0x40
kernel: [<ffffffffa008b125>] page_to_skb+0x65/0x370 [virtio_net]
kernel: [<ffffffffa008d611>] virtnet_receive+0x2c1/0x980 [virtio_net]
kernel: [<ffffffffa008dcf1>] virtnet_poll+0x21/0x90 [virtio_net]
kernel: [<ffffffff8165b6aa>] net_rx_action+0x21a/0x350
kernel: [<ffffffff810a01ab>] __do_softirq+0x10b/0x2b0
kernel: [<ffffffff810a0565>] irq_exit+0x125/0x130
kernel: [<ffffffff81778788>] do_IRQ+0x58/0xf0
kernel: [<ffffffff8177656d>] common_interrupt+0x6d/0x6d
kernel: <EOI>  [<ffffffff81103cc8>] ? hrtimer_start+0x18/0x20
kernel: [<ffffffff8105ea76>] ? native_safe_halt+0x6/0x10
kernel: [<ffffffff810fabe3>] ? rcu_eqs_enter+0xa3/0xb0
kernel: [<ffffffff8101f97e>] default_idle+0x1e/0xc0
kernel: [<ffffffff810203df>] arch_cpu_idle+0xf/0x20
kernel: [<ffffffff810de1fa>] cpu_startup_entry+0x37a/0x3c0
kernel: [<ffffffff81765757>] rest_init+0x77/0x80
kernel: [<ffffffff81d4f02c>] start_kernel+0x4a4/0x4c5
kernel: [<ffffffff81d4e120>] ? early_idt_handlers+0x120/0x120
kernel: [<ffffffff81d4e4d7>] x86_64_start_reservations+0x2a/0x2c
kernel: [<ffffffff81d4e62b>] x86_64_start_kernel+0x152/0x175

Re: [Qemu-devel] Kernel Panic on Yum update

[Paolo Bonzini]

On 15/05/2015 08:34, Gerhard Wiesinger wrote:
> Helllo,
> 
> I'm using latest qemu-kvm-2.3.0-3.fc21.x86_64 from libvirt repository
> (updated afterwards). Running yum regularly crashes the VM like below.
> VM is stripped down to minimum memory requirements (256MB) for owncloud.
> See below.
> 
> Looks like a problem in virtio-net or a kernel bug with low memory
> situations.

You're simply running out of memory.  Yum can be a memory hog.

Paolo

> Any ideas?
> 
> Thank you.
> 
> Ciao,
> Gerhard
> 
> http://www.wiesinger.com/
> 
> kernel: swapper/0: page allocation failure: order:0, mode:0x20
> kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.7-200.fc21.x86_64 #1
> kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> 1.8.1-20150414_141626-myhost.mydomain 04/01/2014
> kernel: 0000000000000000 c35c12d2c14e1ce8 ffff88000fc03ac8 ffffffff8176f1f7
> kernel: 0000000000000000 0000000000000020 ffff88000fc03b58 ffffffff811a1c9e
> kernel: 0000000000000000 0000000000000040 ffff880000000000 ffff88000ffcdb28
> kernel: Call Trace:
> kernel: <IRQ>  [<ffffffff8176f1f7>] dump_stack+0x45/0x57
> kernel: [<ffffffff811a1c9e>] warn_alloc_failed+0xfe/0x170
> kernel: [<ffffffff811a607a>] __alloc_pages_nodemask+0x7ca/0xb10
> kernel: [<ffffffff81691854>] ? nf_hook_slow+0x84/0x130
> kernel: [<ffffffff816482df>] __alloc_page_frag+0x12f/0x150
> kernel: [<ffffffff8164832f>] __netdev_alloc_frag+0x2f/0x50
> kernel: [<ffffffff8164d446>] __alloc_rx_skb+0xe6/0x110
> kernel: [<ffffffff8164d48f>] __netdev_alloc_skb+0x1f/0x40
> kernel: [<ffffffffa008b125>] page_to_skb+0x65/0x370 [virtio_net]
> kernel: [<ffffffffa008d611>] virtnet_receive+0x2c1/0x980 [virtio_net]
> kernel: [<ffffffffa008dcf1>] virtnet_poll+0x21/0x90 [virtio_net]
> kernel: [<ffffffff8165b6aa>] net_rx_action+0x21a/0x350
> kernel: [<ffffffff810a01ab>] __do_softirq+0x10b/0x2b0
> kernel: [<ffffffff810a0565>] irq_exit+0x125/0x130
> kernel: [<ffffffff81778788>] do_IRQ+0x58/0xf0
> kernel: [<ffffffff8177656d>] common_interrupt+0x6d/0x6d

Re: [Qemu-devel] Kernel Panic on Yum update

[Gerhard Wiesinger]

On 15.05.2015 08:51, Paolo Bonzini wrote:
>
> On 15/05/2015 08:34, Gerhard Wiesinger wrote:
>> Helllo,
>>
>> I'm using latest qemu-kvm-2.3.0-3.fc21.x86_64 from libvirt repository
>> (updated afterwards). Running yum regularly crashes the VM like below.
>> VM is stripped down to minimum memory requirements (256MB) for owncloud.
>> See below.
>>
>> Looks like a problem in virtio-net or a kernel bug with low memory
>> situations.
> You're simply running out of memory.  Yum can be a memory hog.
>
>

Yes, yum takes memory. But there is ~2.2 GB virt memory available. That 
should be enough. Therefore I think it is a kernel problem. As in 
previous crashes on the mailing list there is a lot of swap available 
(2GB) which isn't touched in ANY way.

Under normal conditions without yum:
free
               total        used        free      shared buff/cache   
available
Mem:         243036      132540       12716       15520 97780       74964
Swap:       2064380           0     2064380

And running it after reboot works fine (with all the services up).

Ciao,
Gerhard

Re: [Qemu-devel] Kernel Panic on Yum update

[Paolo Bonzini]

On 15/05/2015 09:37, Gerhard Wiesinger wrote:
>>
> 
> Yes, yum takes memory. But there is ~2.2 GB virt memory available. That
> should be enough. Therefore I think it is a kernel problem. As in
> previous crashes on the mailing list there is a lot of swap available
> (2GB) which isn't touched in ANY way.
> 
> Under normal conditions without yum:
> free
>               total        used        free      shared buff/cache  
> available
> Mem:         243036      132540       12716       15520 97780       74964
> Swap:       2064380           0     2064380

Not all memory is the same.  Some memory cannot be swapped, and some
memory can be swapped to disk without a swap file (e.g. executables).

Failing an order 0 allocation is weird indeed, but this is a GFP_ATOMIC
allocation so it's a bit less weird.

Paolo

Re: [Qemu-devel] Kernel Panic on Yum update

[Gerhard Wiesinger]

On 15.05.2015 10:10, Paolo Bonzini wrote:
>
> On 15/05/2015 09:37, Gerhard Wiesinger wrote:
>> Yes, yum takes memory. But there is ~2.2 GB virt memory available. That
>> should be enough. Therefore I think it is a kernel problem. As in
>> previous crashes on the mailing list there is a lot of swap available
>> (2GB) which isn't touched in ANY way.
>>
>> Under normal conditions without yum:
>> free
>>                total        used        free      shared buff/cache
>> available
>> Mem:         243036      132540       12716       15520 97780       74964
>> Swap:       2064380           0     2064380
> Not all memory is the same.  Some memory cannot be swapped, and some
> memory can be swapped to disk without a swap file (e.g. executables).

Yes, I know the different memory types (pageable, non pageable, etc.).

> Failing an order 0 allocation is weird indeed, but this is a GFP_ATOMIC
> allocation so it's a bit less weird.

Nevertheless the kernel should never go out of non pageable memory if a 
USER process requires memory (which is of course pageable).

Ciao,
Gerhard