[bug report] NBD: rbd-nbd + ext4 stuck after nbd resized

* [bug report] NBD: rbd-nbd + ext4 stuck after nbd resized
@ 2020-10-19  3:29 lining
  0 siblings, 0 replies; 4+ messages in thread
From: lining @ 2020-10-19  3:29 UTC (permalink / raw)
  To: josef, axboe, linux-block, yunchuan.wen, ceph-users; +Cc: donglifekernel

Hi kernel、ceph comunity:

We run into an issue that mainly related to the (kernel) nbd driver and (ceph) rbd-nbd. 
After some investigations, I found that the root cause of the problem seems to be related to the change in the block size of nbd.

I am not sure whether it is the nbd driver or rbd-nbd bug, however there is such a problem.

What happened：
It will always hang when accessing the mount point of nbd device with ext4 after nbd resized. 

Environment information:
- kernel:               v4.19.25 or master
- rbd-nbd(ceph):  v12.2.0 Luminous or master
- the fs of nbd:    ext4

Steps to reproduce:
1. rbd create --size 2G rbdpool/foo  # create a 2G size rbd image
2. rbd-nbd map rbdpool/foo            # map the rbd image as a local block device /dev/nbd0, block size is 512(the default block size is set in rbd-nbd code when nbd mapped).
3. mkfs.ext4 /dev/nbd0                     # mkfs.ext4 on nbd0, only nbd + ext4 can reproduce the problem
4. mount /dev/nbd0 /mnt                # mount nbd0 on /mnt
5. rbd resize --size 4G rbdpool/foo   # expand the nbd backend image from 2G to 4G size
6. ls /mnt                                         # `ls` stuck here forever

ln@ubuntu:linux>$ ps -ef |grep mnt
root        8670    7519 98 10:16 pts/5    00:28:46 ls --color=auto /mnt/
ln          9508    9293  0 10:45 pts/6    00:00:00 grep --color=auto mnt

ln@ubuntu:linux>$ sudo cat /proc/8670/stack
[<0>] io_schedule+0x1a/0x40
[<0>] __lock_page+0x105/0x150
[<0>] pagecache_get_page+0x199/0x2c0
[<0>] __getblk_gfp+0xef/0x290
[<0>] ext4_getblk+0x83/0x1a0
[<0>] ext4_bread+0x26/0xb0
[<0>] __ext4_read_dirblock+0x34/0x2c0
[<0>] htree_dirblock_to_tree+0x56/0x1c0
[<0>] ext4_htree_fill_tree+0xad/0x330
[<0>] ext4_readdir+0x6a3/0x980
[<0>] iterate_dir+0x9e/0x1a0
[<0>] ksys_getdents64+0xa0/0x130
[<0>] __x64_sys_getdents64+0x1e/0x30
[<0>] do_syscall_64+0x5e/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] 0xffffffffffffffff

Some investigations on the kernel side:
By git bisect, I found the problem is related to this commit: https://github.com/torvalds/linux/commit/9a9c3c02eacecf4bfde74b08ed32749a4929a2cf .
The kernel with this commit (9a9c3c02) can reproduce the problem, revert the commit and the problem disappears.

Some Logical analysis about the nbd block size changing:
1. rbd-nbd map rbdpool/foo      
    => ioctl NBD_BLKSZSET 512 
      => nbd_size_set() 
        => nbd_size_update(nbd) 
          =>{
                  bdev = bdget_disk(nbd->disk, 0);
                  bd_set_size(bdev, 512)  
                  set_blocksize(bdev, 512)
               }

2. mkfs.ext4 /dev/nbd0

3. mount /dev/nbd0 /mnt    
    => vfs mount
      =>  ext4_mount() 
        => … 
          => sb_set_blocksize() 
            => set_blocksize(bdev, 4096)   <= mount ext4 will set the nbd blocksize to 4096

4. rbd resize –size 4G rbdpool/foo   
    => ioctl NBD_SET_SIZE 4G   <= rbd-nbd will update the latest total size of nbd device
      =>  nbd_size_set() 
        => nbd_size_update(nbd) 
          =>{
                  bdev = bdget_disk(nbd->disk, 0);
                  bd_set_size(bdev, 512)  
                  set_blocksize(bdev, 512)   <= the blocksize is set back to 512 [code line: set_blocksize(bdev, config->blksize);  ]. It seems to be the root cause.
               } 

^ permalink raw reply	[flat|nested] 4+ messages in thread