Re: [PATCH v6 0/5] Fix potential kernel panic when increase hardware queue

[Weiping Zhang <zwp10758@gmail.com>]

On Tue, May 12, 2020 at 8:09 PM Weiping Zhang <zwp10758@gmail.com> wrote:
>
> On Tue, May 12, 2020 at 9:31 AM Bart Van Assche <bvanassche@acm.org> wrote:
> >
> > On 2020-05-07 06:03, Weiping Zhang wrote:
> > > This series mainly fix the kernel panic when increase hardware queue,
> > > and also fix some other misc issue.
> >
> > Does this patch series survive blktests? I'm asking this because
> > blktests triggers the crash shown below for Jens' block-for-next branch.
> > I think this report is the result of a recent change.
> >
> > run blktests block/030
> >
> > null_blk: module loaded
> > Increasing nr_hw_queues to 8 fails, fallback to 1
> > ==================================================================
> > BUG: KASAN: null-ptr-deref in blk_mq_map_swqueue+0x2f2/0x830
> > Read of size 8 at addr 0000000000000128 by task nproc/8541
> >
> > CPU: 5 PID: 8541 Comm: nproc Not tainted 5.7.0-rc4-dbg+ #3
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> > rel-1.13.0-0-gf21b5a4-rebuilt.opensuse.org 04/01/2014
> > Call Trace:
> >  dump_stack+0xa5/0xe6
> >  __kasan_report.cold+0x65/0xbb
> >  kasan_report+0x45/0x60
> >  check_memory_region+0x15e/0x1c0
> >  __kasan_check_read+0x15/0x20
> >  blk_mq_map_swqueue+0x2f2/0x830
> >  __blk_mq_update_nr_hw_queues+0x3df/0x690
> >  blk_mq_update_nr_hw_queues+0x32/0x50
> >  nullb_device_submit_queues_store+0xde/0x160 [null_blk]
> >  configfs_write_file+0x1c4/0x250 [configfs]
> >  __vfs_write+0x4c/0x90
> >  vfs_write+0x14b/0x2d0
> >  ksys_write+0xdd/0x180
> >  __x64_sys_write+0x47/0x50
> >  do_syscall_64+0x6f/0x310
> >  entry_SYSCALL_64_after_hwframe+0x49/0xb3
> >
> > Thanks,
> >
>
> Hi Bart,
>
> I don't test block/030, since I don't pull blktest very often.
>
> It's a different problem,
> because the mapping cann't be reset when do fallback, so the
> cpu[>=1] will point to a hctx(!=0).
>
>  it should be fixed by:
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index bc34d6b572b6..d82cefb0474f 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -3365,8 +3365,8 @@ static void __blk_mq_update_nr_hw_queues(struct
> blk_mq_tag_set *set,
>                 goto reregister;
>
>         set->nr_hw_queues = nr_hw_queues;
> -       blk_mq_update_queue_map(set);
>  fallback:
> +       blk_mq_update_queue_map(set);
>         list_for_each_entry(q, &set->tag_list, tag_set_list) {
>                 blk_mq_realloc_hw_ctxs(set, q);
>                 if (q->nr_hw_queues != set->nr_hw_queues) {

And block/030 should also be improved ?

 35         # Since older null_blk versions do not allow "submit_queues" to be
 36         # modified, check first whether that configs attribute is writeable.
 37         # Each iteration of the loop below triggers $(nproc) + 1
 38         # null_init_hctx() calls. Since <interval>=$(nproc), all possible
 39         # blk_mq_realloc_hw_ctxs() error paths will be triggered. Whether or
 40         # not this test succeeds depends on whether or not _check_dmesg()
 41         # detects a kernel warning.
 42         if { echo "$(<"$sq")" >$sq; } 2>/dev/null; then
 43                 for ((i = 0; i < 100; i++)); do
 44                         echo 1 > $sq
 45                         nproc > $sq  # this line output lots
"nproc: write error: Cannot allocate memory"
 46                 done
 47         else
 48                 SKIP_REASON="Skipping test because $sq cannot be modified"
 49         fi


The test result show this test case [failed], actually it [pass],
there is no warning detect
in kernel log, if apply above patch.

block/030 (trigger the blk_mq_realloc_hw_ctxs() error path)  [failed]
    runtime  1.999s  ...  2.115s
    --- tests/block/030.out 2020-05-12 10:42:26.345782849 +0800
    +++ /data1/zwp/src/blktests/results/nodev/block/030.out.bad
2020-05-12 20:14:59.878915218 +0800
    @@ -1 +1,51 @@
    +nproc: write error: Cannot allocate memory
    +nproc: write error: Cannot allocate memory
    +nproc: write error: Cannot allocate memory
    +nproc: write error: Cannot allocate memory
    +nproc: write error: Cannot allocate memory
    +nproc: write error: Cannot allocate memory
    +nproc: write error: Cannot allocate memory
    ...
    (Run 'diff -u tests/block/030.out
/data1/zwp/src/blktests/results/nodev/block/030.out.bad' to see the
entire diff)

Thanks
Weiping