ext4/xfs: about switching underlying 512B sector devices to 4K ones

ext4/xfs: about switching underlying 512B sector devices to 4K ones

[Takuya Yoshikawa]

I have a question: is it possible to make existing ext4/xfs filesystems
formatted on 512B sector devices run as is on 4k sector devices?


Problem:

We are maintaining some legacy servers whose data is stored on
ext4/xfs filesystems formatted on lvm2 raid1 devices.

These raid1 devices consist of a few iSCSI devices, so the
remote storage servers running as iSCSI targets are the actual
data storage.

  /dev/md127 --  /dev/sda  --(iSCSI)-- remote storage server
                 /dev/sdb  --(iSCSI)-- remote storage server

A problem happened when we tried to add a new storage server with
4k sector disks as an iSCSI target. After lvm2 added that iSCSI
device and started syncing the blocks from existing 512B sector
storage servers to the new 4k sector ones, we got
"Bad block number requested" messages, and soon after that,
the new device was removed from the lvm2 raid1 device.

  /dev/md127 --  /dev/sda  --(iSCSI)-- remote storage server(512)
                 /dev/sdb  --(iSCSI)-- remote storage server(512)
              *  /dev/sdc  --(iSCSI)-- remote storage server(4k)

  The combined raid1 device had been recognized as a 4k device
  as described in this article:
    https://access.redhat.com/articles/3911611

It seemed like 512B unaligned requests from the xfs filesystem
were sent to the raid1 device, and mirrored requests caused
the problem on the newly added 4k sector storage.

The xfs was formatted with its sector_size_options set to the
default (512).
See https://www.man7.org/linux/man-pages/man8/mkfs.xfs.8.html

In the case of ext4, the device continued to run, but I was not
sure if there could be any problems.


Question:

Is it possible to change the underlying storage to 4k sector ones
as written above without copying the data on the ext4/xfs
filesystems to outside of the raid1 device?

ext4: I am not seeing any apparent errors after adding the 4k
  device. Is this an expected behavior?

xfs: is it possible to change the filesystem sector size?

  I read this explanation and thought if I could change the
  journal related metadata, it might be possible.
  https://www.spinics.net/lists/linux-xfs/msg14495.html


Thanks,
  Takuya

Re: ext4/xfs: about switching underlying 512B sector devices to 4K ones

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 3108 bytes --]

On Jul 29, 2020, at 4:38 AM, Takuya Yoshikawa <takuya.yoshikawa@gmail.com> wrote:
> 
> I have a question: is it possible to make existing ext4/xfs filesystems
> formatted on 512B sector devices run as is on 4k sector devices?

For ext4 filesystems that are formatted with 4KB block size, there
is no problem to change the underlying sector size to 4KB.  It will
never access blocks that are not sized/aligned at 4KB.

Most ext4 filesystem are formatted with 4KB blocksize by default,
unless it is a tiny partition like /boot that *may* be formatted with
1KB block size to save space.  In such cases, it would be easy to
format a new /boot with 4KB blocksize and copy the file contents over,
since it would be a very small partition.

You can check with "dumpe2fs -h /dev/md127 | grep | grep 'Block size' "
to see what the filesystem block size is.

Cheers, Andreas

> Problem:
> 
> We are maintaining some legacy servers whose data is stored on
> ext4/xfs filesystems formatted on lvm2 raid1 devices.
> 
> These raid1 devices consist of a few iSCSI devices, so the
> remote storage servers running as iSCSI targets are the actual
> data storage.
> 
>  /dev/md127 --  /dev/sda  --(iSCSI)-- remote storage server
>                 /dev/sdb  --(iSCSI)-- remote storage server
> 
> A problem happened when we tried to add a new storage server with
> 4k sector disks as an iSCSI target. After lvm2 added that iSCSI
> device and started syncing the blocks from existing 512B sector
> storage servers to the new 4k sector ones, we got
> "Bad block number requested" messages, and soon after that,
> the new device was removed from the lvm2 raid1 device.
> 
>  /dev/md127 --  /dev/sda  --(iSCSI)-- remote storage server(512)
>                 /dev/sdb  --(iSCSI)-- remote storage server(512)
>              *  /dev/sdc  --(iSCSI)-- remote storage server(4k)
> 
>  The combined raid1 device had been recognized as a 4k device
>  as described in this article:
>    https://access.redhat.com/articles/3911611
> 
> It seemed like 512B unaligned requests from the xfs filesystem
> were sent to the raid1 device, and mirrored requests caused
> the problem on the newly added 4k sector storage.
> 
> The xfs was formatted with its sector_size_options set to the
> default (512).
> See https://www.man7.org/linux/man-pages/man8/mkfs.xfs.8.html
> 
> In the case of ext4, the device continued to run, but I was not
> sure if there could be any problems.
> 
> 
> Question:
> 
> Is it possible to change the underlying storage to 4k sector ones
> as written above without copying the data on the ext4/xfs
> filesystems to outside of the raid1 device?
> 
> ext4: I am not seeing any apparent errors after adding the 4k
>  device. Is this an expected behavior?
> 
> xfs: is it possible to change the filesystem sector size?
> 
>  I read this explanation and thought if I could change the
>  journal related metadata, it might be possible.
>  https://www.spinics.net/lists/linux-xfs/msg14495.html
> 
> 
> Thanks,
>  Takuya


Cheers, Andreas






[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 873 bytes --]

Re: ext4/xfs: about switching underlying 512B sector devices to 4K ones

[Dave Chinner]

On Wed, Jul 29, 2020 at 07:38:33PM +0900, Takuya Yoshikawa wrote:
> I have a question: is it possible to make existing ext4/xfs filesystems
> formatted on 512B sector devices run as is on 4k sector devices?
> 
> 
> Problem:
> 
> We are maintaining some legacy servers whose data is stored on
> ext4/xfs filesystems formatted on lvm2 raid1 devices.
> 
> These raid1 devices consist of a few iSCSI devices, so the
> remote storage servers running as iSCSI targets are the actual
> data storage.
> 
>   /dev/md127 --  /dev/sda  --(iSCSI)-- remote storage server
>                  /dev/sdb  --(iSCSI)-- remote storage server
> 
> A problem happened when we tried to add a new storage server with
> 4k sector disks as an iSCSI target. After lvm2 added that iSCSI
> device and started syncing the blocks from existing 512B sector
> storage servers to the new 4k sector ones, we got
> "Bad block number requested" messages, and soon after that,
> the new device was removed from the lvm2 raid1 device.
> 
>   /dev/md127 --  /dev/sda  --(iSCSI)-- remote storage server(512)
>                  /dev/sdb  --(iSCSI)-- remote storage server(512)
>               *  /dev/sdc  --(iSCSI)-- remote storage server(4k)
> 
>   The combined raid1 device had been recognized as a 4k device
>   as described in this article:
>     https://access.redhat.com/articles/3911611

Rule of thumb: growing must always be done with devices that have
the same or smaller logical sector sizes. IOWs, the above will break
any filesystem that is formatted with alignment to the logical
sector size of 512 bytes...

> It seemed like 512B unaligned requests from the xfs filesystem
> were sent to the raid1 device, and mirrored requests caused
> the problem on the newly added 4k sector storage.

Yes, because XFS has permanent metadata that is logical sector sized
and aligned. Hence if the device has a logical sector size of 512
at mkfs time, you will get this:

> The xfs was formatted with its sector_size_options set to the
> default (512).
> See https://www.man7.org/linux/man-pages/man8/mkfs.xfs.8.html

and that filesystem will not work on 4k physical/logical
storage devices.

if you start with 4k devices, mkfs.xfs will detect 4k
physical/logical devices and set it's sector size to 4k
automatically and hence will work on those devices, but you can't
change this retrospectively....

> xfs: is it possible to change the filesystem sector size?

Only at mkfs time.

Cheers,

Dave.

-- 
Dave Chinner
david@fromorbit.com

Re: ext4/xfs: about switching underlying 512B sector devices to 4K ones

[Christoph Hellwig]

On Thu, Jul 30, 2020 at 09:16:05AM +1000, Dave Chinner wrote:
> if you start with 4k devices, mkfs.xfs will detect 4k
> physical/logical devices and set it's sector size to 4k
> automatically and hence will work on those devices, but you can't
> change this retrospectively....

Alternatively you can force a larger sectors size manually at mkfs
time.