Re: [BUG] kernel BUG at drivers/block/xen-blkfront.c:1711

[Evgenii Shatokhin <eshatokhin@virtuozzo.com>]

On 10.08.2016 15:49, Bob Liu wrote:
>
> On 08/10/2016 08:33 PM, Evgenii Shatokhin wrote:
>> On 14.07.2016 15:04, Bob Liu wrote:
>>>
>>> On 07/14/2016 07:49 PM, Evgenii Shatokhin wrote:
>>>> On 11.07.2016 15:04, Bob Liu wrote:
>>>>>
>>>>>
>>>>> On 07/11/2016 04:50 PM, Evgenii Shatokhin wrote:
>>>>>> On 06.06.2016 11:42, Dario Faggioli wrote:
>>>>>>> Just Cc-ing some Linux, block, and Xen on CentOS people...
>>>>>>>
>>>>>>
>>>>>> Ping.
>>>>>>
>>>>>> Any suggestions how to debug this or what might cause the problem?
>>>>>>
>>>>>> Obviously, we cannot control Xen on the Amazon's servers. But perhaps there is something we can do at the kernel's side, is it?
>>>>>>
>>>>>>> On Mon, 2016-06-06 at 11:24 +0300, Evgenii Shatokhin wrote:
>>>>>>>> (Resending this bug report because the message I sent last week did
>>>>>>>> not
>>>>>>>> make it to the mailing list somehow.)
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> One of our users gets kernel panics from time to time when he tries
>>>>>>>> to
>>>>>>>> use his Amazon EC2 instance with CentOS7 x64 in it [1]. Kernel panic
>>>>>>>> happens within minutes from the moment the instance starts. The
>>>>>>>> problem
>>>>>>>> does not show up every time, however.
>>>>>>>>
>>>>>>>> The user first observed the problem with a custom kernel, but it was
>>>>>>>> found later that the stock kernel 3.10.0-327.18.2.el7.x86_64 from
>>>>>>>> CentOS7 was affected as well.
>>>>>
>>>>> Please try this patch:
>>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7b0767502b5db11cb1f0daef2d01f6d71b1192dc
>>>>>
>>>>> Regards,
>>>>> Bob
>>>>>
>>>>
>>>> Unfortunately, it did not help. The same BUG_ON() in blkfront_setup_indirect() still triggers in our kernel based on RHEL's 3.10.0-327.18.2, where I added the patch.
>>>>
>>>> As far as I can see, the patch makes sure the indirect pages are added to the list only if (!info->feature_persistent) holds. I suppose it holds in our case and the pages are added to the list because the triggered BUG_ON() is here:
>>>>
>>>>       if (!info->feature_persistent && info->max_indirect_segments) {
>>>>           <...>
>>>>           BUG_ON(!list_empty(&info->indirect_pages));
>>>>           <...>
>>>>       }
>>>>
>>>
>>> That's odd.
>>> Could you please try to reproduce this issue with a recent upstream kernel?
>>>
>>> Thanks,
>>> Bob
>>
>> No luck with the upstream kernel 4.7.0 so far due to unrelated issues (bad initrd, I suppose, so the system does not even boot).
>>
>> However, the problem reproduced with the stable upstream kernel 3.14.74. After the system booted the second time with this kernel, that BUG_ON triggered:
>>       kernel BUG at drivers/block/xen-blkfront.c:1701
>>
>
> Could you please provide more detail on how to reproduce this bug? I'd like to have a test.
>
> Thanks!
> Bob

As the user says, he uses an Amazon EC2 instance. Namely: HVM CentOS7 
AMI on a c3.large instance with EBS magnetic storage.

At least 2 LVM partitions are needed:
* /, 20-30 Gb should be enough, ext4
* /vz, 5-10 Gb should be enough, ext4

Kernel 3.14.74 I was talking about: 
https://www.dropbox.com/s/bhus3mubza87z86/kernel-3.14.74-1.test.x86_64.rpm?dl=1

Not sure if it is relevant, but the user may have installed additional 
packages from 
https://download.openvz.org/virtuozzo/releases/7.0-rtm/x86_64/os/ 
repository. Namely: vzctl, vzmigrate, vzprocps, vztt-lib, vzctcalc, 
ploop, prlctl, centos-7-x86_64-ez.

After the kernel and the other mentioned packages have been installed,
the user rebooted the instance to run that kernel 3.14.74.

Then - start the instance, wait 5 minutes, stop the instance, repeat. 
2-20 such iterations were usually enough to reproduce the problem. Can 
be automated with the help of Amazon's API.

BTW, before the BUG_ON triggered this time, there was the following in 
dmesg. Not sure if it is related but still:

----------------------
[    2.835034] scsi0 : ata_piix
[    2.840317] scsi1 : ata_piix
[    2.842267] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc100 irq 14
[    2.845861] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc108 irq 15
[    2.853840] AVX version of gcm_enc/dec engaged.
[    2.859963] xen_netfront: Initialising Xen virtual ethernet driver
[    2.867156] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[    2.885861] blkfront: xvda: barrier or flush: disabled; persistent 
grants: disabled; indirect descriptors: enabled;
[    2.889046] alg: No test for crc32 (crc32-pclmul)
[    2.899290]  xvda: xvda1
[    2.997751] blkfront: xvdc: flush diskcache: enabled; persistent 
grants: disabled; indirect descriptors: enabled;
[    3.007401]  xvdc: unknown partition table
[    3.010465] Setting capacity to 31992832
[    3.012922] xvdc: detected capacity change from 0 to 16380329984
[    3.017408] blkfront: xvdd: flush diskcache: enabled; persistent 
grants: disabled; indirect descriptors: enabled;
[    3.023861]  xvdd: unknown partition table
[    3.026481] Setting capacity to 31992832
[    3.029051] xvdd: detected capacity change from 0 to 16380329984
[    3.033320] blkfront: xvdf: barrier or flush: disabled; persistent 
grants: disabled; indirect descriptors: enabled;
[    3.040712] random: nonblocking pool is initialized
[    3.057432]  xvdf: unknown partition table
[    3.060807] Setting capacity to 41943040
[    3.063194] xvdf: detected capacity change from 0 to 21474836480
[    3.067684] blkfront: xvdb: barrier or flush: disabled; persistent 
grants: disabled; indirect descriptors: enabled;
[    3.076835]  xvdb: unknown partition table
[    3.079692] Setting capacity to 16777216
[    3.082112] xvdb: detected capacity change from 0 to 8589934592
[    3.086853] vbd vbd-51712: 16 xlvbd_add at 
/local/domain/0/backend/vbd/9543/51712
----------------------

>
>>>
>>>> So the problem is still out there somewhere, it seems.
>>>>
>>>> Regards,
>>>> Evgenii
>>>>
>>>>>>>>
>>>>>>>> The part of the system log he was able to retrieve is attached. Here
>>>>>>>> is
>>>>>>>> the bug info, for convenience:
>>>>>>>>
>>>>>>>> ------------------------------------
>>>>>>>> [    2.246912] kernel BUG at drivers/block/xen-blkfront.c:1711!
>>>>>>>> [    2.246912] invalid opcode: 0000 [#1] SMP
>>>>>>>> [    2.246912] Modules linked in: ata_generic pata_acpi
>>>>>>>> crct10dif_pclmul
>>>>>>>> crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel
>>>>>>>> xen_netfront xen_blkfront(+) aesni_intel lrw ata_piix gf128mul
>>>>>>>> glue_helper ablk_helper cryptd libata serio_raw floppy sunrpc
>>>>>>>> dm_mirror
>>>>>>>> dm_region_hash dm_log dm_mod scsi_transport_iscsi
>>>>>>>> [    2.246912] CPU: 1 PID: 50 Comm: xenwatch Not tainted
>>>>>>>> 3.10.0-327.18.2.el7.x86_64 #1
>>>>>>>> [    2.246912] Hardware name: Xen HVM domU, BIOS 4.2.amazon
>>>>>>>> 12/07/2015
>>>>>>>> [    2.246912] task: ffff8800e9fcb980 ti: ffff8800e98bc000 task.ti:
>>>>>>>> ffff8800e98bc000
>>>>>>>> [    2.246912] RIP: 0010:[<ffffffffa015584f>]  [<ffffffffa015584f>]
>>>>>>>> blkfront_setup_indirect+0x41f/0x430 [xen_blkfront]
>>>>>>>> [    2.246912] RSP: 0018:ffff8800e98bfcd0  EFLAGS: 00010283
>>>>>>>> [    2.246912] RAX: ffff8800353e15c0 RBX: ffff8800e98c52c8 RCX:
>>>>>>>> 0000000000000020
>>>>>>>> [    2.246912] RDX: ffff8800353e15b0 RSI: ffff8800e98c52b8 RDI:
>>>>>>>> ffff8800353e15d0
>>>>>>>> [    2.246912] RBP: ffff8800e98bfd20 R08: ffff8800353e15b0 R09:
>>>>>>>> ffff8800eb403c00
>>>>>>>> [    2.246912] R10: ffffffffa0155532 R11: ffffffffffffffe8 R12:
>>>>>>>> ffff8800e98c4000
>>>>>>>> [    2.246912] R13: ffff8800e98c52b8 R14: 0000000000000020 R15:
>>>>>>>> ffff8800353e15c0
>>>>>>>> [    2.246912] FS:  0000000000000000(0000) GS:ffff8800efc20000(0000)
>>>>>>>> knlGS:0000000000000000
>>>>>>>> [    2.246912] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>>>> [    2.246912] CR2: 00007f1b615ef000 CR3: 00000000e2b44000 CR4:
>>>>>>>> 00000000001406e0
>>>>>>>> [    2.246912] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>>>>>>>> 0000000000000000
>>>>>>>> [    2.246912] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>>>>>>>> 0000000000000400
>>>>>>>> [    2.246912] Stack:
>>>>>>>> [    2.246912]  0000000000000020 0000000000000001 00000020a0157217
>>>>>>>> 00000100e98bfdbc
>>>>>>>> [    2.246912]  0000000027efa3ef ffff8800e98bfdbc ffff8800e98ce000
>>>>>>>> ffff8800e98c4000
>>>>>>>> [    2.246912]  ffff8800e98ce040 0000000000000001 ffff8800e98bfe08
>>>>>>>> ffffffffa0155d4c
>>>>>>>> [    2.246912] Call Trace:
>>>>>>>> [    2.246912]  [<ffffffffa0155d4c>] blkback_changed+0x4ec/0xfc8
>>>>>>>> [xen_blkfront]
>>>>>>>> [    2.246912]  [<ffffffff813a6fd0>] ? xenbus_gather+0x170/0x190
>>>>>>>> [    2.246912]  [<ffffffff816322f5>] ? __slab_free+0x10e/0x277
>>>>>>>> [    2.246912]  [<ffffffff813a805d>]
>>>>>>>> xenbus_otherend_changed+0xad/0x110
>>>>>>>> [    2.246912]  [<ffffffff813a7257>] ? xenwatch_thread+0x77/0x180
>>>>>>>> [    2.246912]  [<ffffffff813a9ba3>] backend_changed+0x13/0x20
>>>>>>>> [    2.246912]  [<ffffffff813a7246>] xenwatch_thread+0x66/0x180
>>>>>>>> [    2.246912]  [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30
>>>>>>>> [    2.246912]  [<ffffffff813a71e0>] ?
>>>>>>>> unregister_xenbus_watch+0x1f0/0x1f0
>>>>>>>> [    2.246912]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
>>>>>>>> [    2.246912]  [<ffffffff810a5a20>] ?
>>>>>>>> kthread_create_on_node+0x140/0x140
>>>>>>>> [    2.246912]  [<ffffffff81646118>] ret_from_fork+0x58/0x90
>>>>>>>> [    2.246912]  [<ffffffff810a5a20>] ?
>>>>>>>> kthread_create_on_node+0x140/0x140
>>>>>>>> [    2.246912] Code: e1 48 85 c0 75 ce 49 8d 84 24 40 01 00 00 48 89
>>>>>>>> 45
>>>>>>>> b8 e9 91 fd ff ff 4c 89 ff e8 8d ae 06 e1 e9 f2 fc ff ff 31 c0 e9 2e
>>>>>>>> fe
>>>>>>>> ff ff <0f> 0b e8 9a 57 f2 e0 0f 0b 0f 1f 84 00 00 00 00 00 0f 1f 44
>>>>>>>> 00
>>>>>>>> [    2.246912] RIP  [<ffffffffa015584f>]
>>>>>>>> blkfront_setup_indirect+0x41f/0x430 [xen_blkfront]
>>>>>>>> [    2.246912]  RSP <ffff8800e98bfcd0>
>>>>>>>> [    2.491574] ---[ end trace 8a9b992812627c71 ]---
>>>>>>>> [    2.495618] Kernel panic - not syncing: Fatal exception
>>>>>>>> ------------------------------------
>>>>>>>>
>>>>>>>> Xen version 4.2.
>>>>>>>>
>>>>>>>> EC2 instance type: c3.large with EBS magnetic storage, if that
>>>>>>>> matters.
>>>>>>>>
>>>>>>>> Here is the code where the BUG_ON triggers (drivers/block/xen-
>>>>>>>> blkfront.c):
>>>>>>>> ------------------------------------
>>>>>>>> if (!info->feature_persistent && info->max_indirect_segments) {
>>>>>>>>          /*
>>>>>>>>              * We are using indirect descriptors but not persistent
>>>>>>>>              * grants, we need to allocate a set of pages that can be
>>>>>>>>              * used for mapping indirect grefs
>>>>>>>>              */
>>>>>>>>          int num = INDIRECT_GREFS(segs) * BLK_RING_SIZE;
>>>>>>>>
>>>>>>>>          BUG_ON(!list_empty(&info->indirect_pages)); // << This one hits.
>>>>>>>>          for (i = 0; i < num; i++) {
>>>>>>>>              struct page *indirect_page = alloc_page(GFP_NOIO);
>>>>>>>>              if (!indirect_page)
>>>>>>>>                  goto out_of_memory;
>>>>>>>>              list_add(&indirect_page->lru, &info->indirect_pages);
>>>>>>>>          }
>>>>>>>> }
>>>>>>>> ------------------------------------
>>>>>>>>
>>>>>>>> As we checked, 'info->indirect_pages' list indeed contained around
>>>>>>>> 30
>>>>>>>> elements at that point.
>>>>>>>>
>>>>>>>> Any ideas what may cause this and how to fix it?
>>>>>>>>
>>>>>>>> If any other data are needed, please let me know.
>>>>>>>>
>>>>>>>> References:
>>>>>>>> [1] https://bugs.openvz.org/browse/OVZ-6718
> .
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel