[linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

Hi,

we just encountered an error when using LVM's pvmove command to move the data from an un-encrypted LVM physical volume onto an encrypted volume. 
After the pvmove has completed, the file system on the logical volume that resides on the moved physical volumes is corrupted and all data on this LV is lost.

As a special condition, we are using the sector-size option with cryptsetup (LUKS2) to use 4096 byte sectors to increase the crypt performance. 

According to Ondrej Kozina in https://www.saout.de/pipermail/dm-crypt/2019-February/006078.html this is due to the fact that a sector size of 4096 also causes the physical block size to be (at least) 4096. Using the default sector size of 512 does not show the problem. 

The source PV of the pvmove has a physical block size of 512 (SCSI disk in our case). So when moving this to the encrypted volume the physical block size becomes 4096. The file system does not seem to like this....

Please note that this problem can also happen in other cases, such as mixing disks with different block sizes (e.g. SCSI disks with 512 bytes and s390x-DASDs with 4096 block size).

In my opinion this is a general problem that needs to be addressed by LVM. 
Whenever you pvmove data to a PV with a larger physical block size you will corrupt the file system and thus cause data loss.
The same probably happens when using lvextend to enlarge an LV onto a PV with a larger physical block size. 

Bottom line: LVM should reject extending a volume group with a PV that has a larger physical block size than the PVs that belong to the volume group already. There might be a force option to override this check in case the user really knows what he does. But the default behavior should be to protect the user from file system corruptions and thus from data loss.

Yes, having a backup before doing pvmove or similar is of course wise, but I still think that LVM should prevent a user for damaging the file system by using LVM's command line tools. 


Here is how to reproduce this (note the error messages on the very last command): 

# sudo dd if=/dev/zero of=loopbackfile1.img bs=500M count=1
1+0 records in
1+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 2.32777 s, 225 MB/s

# sudo dd if=/dev/zero of=loopbackfile2.img bs=500M count=1
1+0 records in
1+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 1.89992 s, 276 MB/s

# losetup -fP loopbackfile1.img

# losetup -fP loopbackfile2.img

# pvcreate /dev/loop0
  Physical volume "/dev/loop0" successfully created.

# vgcreate LOOP_VG /dev/loop0
  Volume group "LOOP_VG" successfully created

# lvcreate -L 300MB LOOP_VG -n LV /dev/loop0
  Logical volume "LV" created.

# mkfs.ext4 /dev/mapper/LOOP_VG-LV
mke2fs 1.44.1 (24-Mar-2018)
Discarding device blocks: done
Creating filesystem with 307200 1k blocks and 76912 inodes
Filesystem UUID: 344289a3-e251-4d88-b03d-a71a4be2a8ec
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729, 204801, 221185

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

# mount /dev/mapper/LOOP_VG-LV /mnt

# cryptsetup luksFormat --type luks2 --sector-size 4096 /dev/loop1

WARNING!
========
This will overwrite data on /dev/loop1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/loop1: loop
Verify passphrase: loop

# cryptsetup luksOpen /dev/loop1 enc-loop
Enter passphrase for /dev/loop1: loop

# pvcreate /dev/mapper/enc-loop
  Physical volume "/dev/mapper/enc-loop" successfully created.

# vgextend LOOP_VG /dev/mapper/enc-loop
  Volume group "LOOP_VG" successfully extended

# pvs
  PV                   VG      Fmt  Attr PSize   PFree
  /dev/loop0           LOOP_VG lvm2 a--  496.00m 196.00m
  /dev/mapper/enc-loop LOOP_VG lvm2 a--  492.00m 492.00m

# pvmove /dev/loop0 /dev/mapper/enc-loop
  /dev/loop0: Moved: 30.67%
  /dev/loop0: Moved: 100.00%

# pvs
  /dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument
  /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument
  /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument
  /dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument
  PV                   VG      Fmt  Attr PSize   PFree
  /dev/loop0           LOOP_VG lvm2 a--  496.00m 496.00m
  /dev/mapper/enc-loop LOOP_VG lvm2 a--  492.00m 192.00m

In case the filesystem of the logical volume is not mounted at the time of pvmove, it gets corrupted anyway, but you only see errors when trying to mount it.

-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Cesare Leonardi]

On 25/02/19 16:33, Ingo Franzki wrote:
> we just encountered an error when using LVM's pvmove command to move the data from an un-encrypted LVM physical volume onto an encrypted volume.
> After the pvmove has completed, the file system on the logical volume that resides on the moved physical volumes is corrupted and all data on this LV is lost.

Hello, your message is interesting. And also this thread:
https://www.redhat.com/archives/linux-lvm/2019-February/msg00002.html

But I'd like to know if I understood correctly.
Should I care about the physical disk size when I use LVM? Mixing disk 
with different sector size (512b and 4k) is dangerous?

Your message and others from the other thread, seems to say that LVM 
doesn't handle correctly that situation and that if I pvmove data 
between a 512b disk and a 4k disk (or viceversa), it will lead to a 
massive filesystem corruption. If I understood correctly, the problem 
that you described looks unrelated to encrypted volume and was only 
exacerbated by that. Right?

Cesare.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 27.02.2019 01:00, Cesare Leonardi wrote:
> On 25/02/19 16:33, Ingo Franzki wrote:
>> we just encountered an error when using LVM's pvmove command to move the data from an un-encrypted LVM physical volume onto an encrypted volume.
>> After the pvmove has completed, the file system on the logical volume that resides on the moved physical volumes is corrupted and all data on this LV is lost.
> 
> Hello, your message is interesting. And also this thread:
> https://www.redhat.com/archives/linux-lvm/2019-February/msg00002.html
> 
> But I'd like to know if I understood correctly.
> Should I care about the physical disk size when I use LVM? Mixing disk with different sector size (512b and 4k) is dangerous?

As far as I can tell: Yes if you pvmove data around or lvextend an LV onto another PV with a larger physical block size that is dangerous.
Creating new LVs and thus new file systems on mixed configurations seem to be OK. 

> 
> Your message and others from the other thread, seems to say that LVM doesn't handle correctly that situation and that if I pvmove data between a 512b disk and a 4k disk (or viceversa), it will lead to a massive filesystem corruption. If I understood correctly, the problem that you described looks unrelated to encrypted volume and was only exacerbated by that. Right?

Moving from 512 to 4096 seems to cause FS corruption, moving from 4096 to 512 does not. So I guess its only a problem when moving to larger physical block sizes.

And yes, its unrelated to encrypted volumes, it can happen with any block device of different physical block sizes that you use as PV.
E.g. SCSI disks exist in 512 bytes block size and 4096 bytes block size. Or s390-DASDs which always have 4096 bytes blocks. 

We just encountered it using encrypted disks with the sector-size option to increase encryption performance. 
So actually we want people to use larger sector sizes, but this seems to cause problems with LVM. 

The good thing about the example with encrypted volumes on loopback devices is that you can reproduce the problem on any platform, without having certain hardware requirements.

> 
> Cesare.
> 
> 


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Stuart D. Gathman]

On Wed, 27 Feb 2019, Ingo Franzki wrote:

> The good thing about the example with encrypted volumes on loopback
> devices is that you can reproduce the problem on any platform, without
> having certain hardware requirements.

The losetup command has a --sector-size option that sets the logical
sector size.  I wonder if that is sufficient to reproduce the problem.

-- 
 	      Stuart D. Gathman <stuart@gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 27.02.2019 15:59, Stuart D. Gathman wrote:
> On Wed, 27 Feb 2019, Ingo Franzki wrote:
> 
>> The good thing about the example with encrypted volumes on loopback
>> devices is that you can reproduce the problem on any platform, without
>> having certain hardware requirements.
> 
> The losetup command has a --sector-size option that sets the logical
> sector size.  I wonder if that is sufficient to reproduce the problem.
> 
Yes that should work: 

# losetup -fP loopbackfile.img --sector-size 4096
# blockdev --getpbsz /dev/loop0
4096


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Cesare Leonardi]

On 27/02/19 09:49, Ingo Franzki wrote:
> As far as I can tell: Yes if you pvmove data around or lvextend an LV onto another PV with a larger physical block size that is dangerous.
> Creating new LVs and thus new file systems on mixed configurations seem to be OK.

[...]

> And yes, its unrelated to encrypted volumes, it can happen with any block device of different physical block sizes that you use as PV.

Thank you Ingo for the precious informations you are giving here.

Not to be pedantic, but what do you mean with physical block? Because 
with modern disks the term is not always clear. Let's take a mechanical 
disk with 512e sectors, that is with 4k sectors but exposed as 512 byte 
sectors. Fdisk will refer to it with these terms:
Sector size (logical/physical): 512 bytes / 4096 bytes

What you are referring as physical size is actually the logical size 
reported by fdisk, right? And if it's correct, I guess that should be 
safe to add the above disk with 512e sectors to an LVM storage composed 
only by disks with real 512 byte sectors. I expect that from the LVM 
point of view this should not be even considered a mixed sector size 
setup, even if the real physical sector size of the added disk is 4096 byte.

Do you agree or do you think it would be better to test this specific setup?

Cesare.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Stuart D. Gathman]

On Thu, 28 Feb 2019, Cesare Leonardi wrote:

> Not to be pedantic, but what do you mean with physical block? Because with 
> modern disks the term is not always clear. Let's take a mechanical disk with 
> 512e sectors, that is with 4k sectors but exposed as 512 byte sectors. Fdisk 
> will refer to it with these terms:
> Sector size (logical/physical): 512 bytes / 4096 bytes
>
> What you are referring as physical size is actually the logical size reported 
> by fdisk, right? And if it's correct, I guess that should be safe to add the 
> above disk with 512e sectors to an LVM storage composed only by disks with 
> real 512 byte sectors. I expect that from the LVM point of view this should 
> not be even considered a mixed sector size setup, even if the real physical 
> sector size of the added disk is 4096 byte.
>
> Do you agree or do you think it would be better to test this specific setup?

I would definitely test it, using the same test script that reproduces the 
problem with loopback devices.

That said, I believe you are right - it should definitely work.  Most of
my drives are 512/4096 logical/phys.  If you actually write a single 512
byte sector, however, the disk firmware will have to do a
read/modify/write cycle - which can tank performance.

hdparm will report logical and physical sector size - but there doesn't
seem to be an option to set logical sectory size.  There really is no
need once you already support a smaller logical sector size, as the
performance hit can be avoided by aligned filesystems with 4k+ block
size (most modern filesystems).

Once I encountered a bug in drive firmware where the R/M/W did not
work correctly with certain read/write patterns (involving unaligned
multi sector writes).  I do not wish that on anyone.  (don't worry,
that drive model is long gone...).

-- 
 	      Stuart D. Gathman <stuart@gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 28.02.2019 02:31, Cesare Leonardi wrote:
> On 27/02/19 09:49, Ingo Franzki wrote:
>> As far as I can tell: Yes if you pvmove data around or lvextend an LV onto another PV with a larger physical block size that is dangerous.
>> Creating new LVs and thus new file systems on mixed configurations seem to be OK.
> 
> [...]
> 
>> And yes, its unrelated to encrypted volumes, it can happen with any block device of different physical block sizes that you use as PV.
> 
> Thank you Ingo for the precious informations you are giving here.
> 
> Not to be pedantic, but what do you mean with physical block? Because with modern disks the term is not always clear. Let's take a mechanical disk with 512e sectors, that is with 4k sectors but exposed as 512 byte sectors. Fdisk will refer to it with these terms:
> Sector size (logical/physical): 512 bytes / 4096 bytes
> 
> What you are referring as physical size is actually the logical size reported by fdisk, right? And if it's correct, I guess that should be safe to add the above disk with 512e sectors to an LVM storage composed only by disks with real 512 byte sectors. I expect that from the LVM point of view this should not be even considered a mixed sector size setup, even if the real physical sector size of the added disk is 4096 byte.
> 
> Do you agree or do you think it would be better to test this specific setup?

Well, there are the following 2 commands:

Get physical block size: 
 blockdev --getpbsz <device>
Get logical block size:
 blockdev --getbsz <device>

Filesystems seem to care about the physical block size only, not the logical block size.

So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...
> 
> Cesare.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ilia Zykov]

[-- Attachment #1: Type: text/plain, Size: 1246 bytes --]

> 
> Well, there are the following 2 commands:
> 
> Get physical block size: 
>  blockdev --getpbsz <device>
> Get logical block size:
>  blockdev --getbsz <device>
> 
> Filesystems seem to care about the physical block size only, not the logical block size.
> 
> So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...

Hello everybody.
Maybe, I don’t understand what do you mean. What the logical block size
mean? But on my machines(CentOS7), this utility get me the strange
results (output reduced):

 smartctl -i /dev/sda; blockdev --getbsz --getpbsz /dev/sda
Device Model:     INTEL SSDSC2KB480G8
User Capacity:    480,103,981,056 bytes [480 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
4096
4096

 smartctl -i /dev/sdb; blockdev --getbsz --getpbsz /dev/sdb
Device Model:     HGST HUS722T2TALA604
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
4096
512

As you see “–getbsz” forever 4096.
But I think it must be forever 512.
What does it mean?

Thank you.
Ilia.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3591 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 28.02.2019 10:48, Ilia Zykov wrote:
>>
>> Well, there are the following 2 commands:
>>
>> Get physical block size: 
>>  blockdev --getpbsz <device>
>> Get logical block size:
>>  blockdev --getbsz <device>
>>
>> Filesystems seem to care about the physical block size only, not the logical block size.
>>
>> So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...
> 
> Hello everybody.
> Maybe, I don’t understand what do you mean. What the logical block size
> mean? But on my machines(CentOS7), this utility get me the strange
> results (output reduced):
> 
>  smartctl -i /dev/sda; blockdev --getbsz --getpbsz /dev/sda
> Device Model:     INTEL SSDSC2KB480G8
> User Capacity:    480,103,981,056 bytes [480 GB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    Solid State Device
> 4096
> 4096
> 
>  smartctl -i /dev/sdb; blockdev --getbsz --getpbsz /dev/sdb
> Device Model:     HGST HUS722T2TALA604
> User Capacity:    2,000,398,934,016 bytes [2.00 TB]
> Sector Size:      512 bytes logical/physical
> Rotation Rate:    7200 rpm
> Form Factor:      3.5 inches
> 4096
> 512
> 
> As you see “–getbsz” forever 4096.
I also see logical block size to be 4096 for all devices on my system.
> But I think it must be forever 512.
> What does it mean?
I have seen the following description about logical and physical block sizes somewhere in the internet:
"Logical block sizes are the units used by the 'kernel' for read/write operations.
Physical block sizes are the units which 'disk controllers' use for read/write operations."

For the problem mentioned in this thread, the physical block size is what you are looking for.
> 
> Thank you.
> Ilia.
> 


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ilia Zykov]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

>>
>>  smartctl -i /dev/sdb; blockdev --getbsz --getpbsz /dev/sdb
>> Device Model:     HGST HUS722T2TALA604
>> User Capacity:    2,000,398,934,016 bytes [2.00 TB]
>> Sector Size:      512 bytes logical/physical
>> Rotation Rate:    7200 rpm
>> Form Factor:      3.5 inches
>> 4096
>> 512
>>
>> As you see “–getbsz” forever 4096.
> I also see logical block size to be 4096 for all devices on my system.
>> But I think it must be forever 512.
>> What does it mean?
> I have seen the following description about logical and physical block sizes somewhere in the internet:
> "Logical block sizes are the units used by the 'kernel' for read/write operations.

Kernel can but usually does not want, because reduce performance.

> Physical block sizes are the units which 'disk controllers' use for read/write operations."

Not disk controller on the motherboard, but controller inside disk. We
don't have access to it.

> 
> For the problem mentioned in this thread, the physical block size is what you are looking for.
>>

I think it is BUG in the "blockdev".
My question was:

Can this error(or similar) be related to a problem in pvmove?


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3599 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ilia Zykov]

[-- Attachment #1: Type: text/plain, Size: 1189 bytes --]

>>
>>  smartctl -i /dev/sdb; blockdev --getbsz --getpbsz /dev/sdb
>> Device Model:     HGST HUS722T2TALA604
>> User Capacity:    2,000,398,934,016 bytes [2.00 TB]
>> Sector Size:      512 bytes logical/physical
>> Rotation Rate:    7200 rpm
>> Form Factor:      3.5 inches
>> 4096
>> 512
>>
>> As you see “–getbsz” forever 4096.
> I also see logical block size to be 4096 for all devices on my system.
>> But I think it must be forever 512.
>> What does it mean?
> I have seen the following description about logical and physical block sizes somewhere in the internet:
> "Logical block sizes are the units used by the 'kernel' for read/write operations.

Kernel can but usually does not want, because reduce performance.

> Physical block sizes are the units which 'disk controllers' use for read/write operations."

Not disk controller on the motherboard, but controller inside disk. We
don't have access to it.

> 
> For the problem mentioned in this thread, the physical block size is what you are looking for.
>>

I think it is BUG in the "blockdev(util-linux)".
My question was:

Can this error(or similar) be related to a problem in pvmove?



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3591 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ilia Zykov]

[-- Attachment #1: Type: text/plain, Size: 457 bytes --]

> 
>>
>> For the problem mentioned in this thread, the physical block size is what you are looking for.
>>>
> 
> I think it is BUG in the "blockdev(util-linux)".

Tt's not a bug it's a feature :O
https://bugzilla.redhat.com/show_bug.cgi?id=1684078

> My question was:
> 
> Can this error(or similar) be related to a problem in pvmove?
> 
> 

My question now:
Can this feature(or similar) be related to a problem in pvmove?

Thank you.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3591 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ilia Zykov]

[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]

> Discarding device blocks: done
> Creating filesystem with 307200 1k blocks and 76912 inodes
> ......
> # pvs
>   /dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument
>   /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument
>   /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument
>   /dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument
>   PV                   VG      Fmt  Attr PSize   PFree
>   /dev/loop0           LOOP_VG lvm2 a--  496.00m 496.00m
>   /dev/mapper/enc-loop LOOP_VG lvm2 a--  492.00m 192.00m
> 
> In case the filesystem of the logical volume is not mounted at the time of pvmove, it gets corrupted anyway, but you only see errors when trying to mount it.
> 

It's because you FS had 1k blocks.
New device can't read with block 1k.
If you plan pvmove on device with 4k. Maybe you need make fs with:
"mkfs.ext4 -b 4096"

See comments:
https://bugzilla.redhat.com/show_bug.cgi?id=1684078


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3591 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 28.02.2019 15:36, Ilia Zykov wrote:
>> Discarding device blocks: done
>> Creating filesystem with 307200 1k blocks and 76912 inodes
>> ......
>> # pvs
>>   /dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument
>>   /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument
>>   /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument
>>   /dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument
>>   PV                   VG      Fmt  Attr PSize   PFree
>>   /dev/loop0           LOOP_VG lvm2 a--  496.00m 496.00m
>>   /dev/mapper/enc-loop LOOP_VG lvm2 a--  492.00m 192.00m
>>
>> In case the filesystem of the logical volume is not mounted at the time of pvmove, it gets corrupted anyway, but you only see errors when trying to mount it.
>>
> 
> It's because you FS had 1k blocks.
> New device can't read with block 1k.
> If you plan pvmove on device with 4k. Maybe you need make fs with:
> "mkfs.ext4 -b 40
The beauty of pvmove is that it works on existing LVs with filesystems on it.
So I might have filesystems on LVs that used the default block size that I want to move via pvmove.
At the time the file system was created (possibly may years ago), I did not know that I would ever move it to a device with a larger block size.
If I have to create a new filesystem after the move, then pvmove is useless.... I then could just create a new LV on the desired PV(s). 
> 
> See comments:
> https://bugzilla.redhat.com/show_bug.cgi?id=1684078
> 


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ilia Zykov]

[-- Attachment #1: Type: text/plain, Size: 300 bytes --]

> At the time the file system was created (possibly may years ago), I did not know that I would ever move it to a device with a larger block size.
>

For this purpose all 4k disks have logical sector size 512.
Don't look at "blockdev --getbsz" it's not property of physical(real)
device.




[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3591 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Cesare Leonardi]

On 28/02/19 09:41, Ingo Franzki wrote:
> Well, there are the following 2 commands:
> 
> Get physical block size:
>   blockdev --getpbsz <device>
> Get logical block size:
>   blockdev --getbsz <device>

I didn't know the blockdev command and, to recap, we have:
--getpbsz: physical sector size
--getss: logical sector size
--getbsz: blocksize used internally by the kernel

getpbsz/getss correspond to physical/logical sector size reported by 
fdisk, smartctl, etc.

> Filesystems seem to care about the physical block size only, not the logical block size.
> 
> So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...

I've done the test suggested by Stuart and it seems to contradict this.
I have pvmoved data from a 512/512 (logical/physical) disk to a newly 
added 512/4096 disk but I had no data corruption. Unfortunately I 
haven't any native 4k disk to repeat the same test.

Here is what I've done.
/dev/sdb: SSD with 512/512 sector size
/dev/sdc: mechanical disk with 512/4096 sector size

# blockdev -v --getss --getpbsz --getbsz /dev/sdb
get logical block (sector) size: 512
get physical block (sector) size: 512
get blocksize: 4096

# blockdev -v --getss --getpbsz --getbsz /dev/sdc
get logical block (sector) size: 512
get physical block (sector) size: 4096
get blocksize: 4096

# pvcreate /dev/sdb4
# vgcreate vgtest /dev/sdb4
# lvcreate -L 1G vgtest /dev/sdb4
# mkfs.ext4 /dev/mapper/vgtest-lvol0
# mkdir /media/test
# mount /dev/mapper/vgtest-lvol0 /media/test
# cp -a SOMEDATA /media/test/
# umount /media/test
# fsck.ext4 -f /dev/mapper/vgtest-lvol0

Filesystem created and no error on it. Now the disk with different 
physical size:

# pvcreate /dev/sdc1
# vgextend vgtest /dev/sdc1
# pvmove /dev/sdb4 /dev/sdc1
# fsck.ext4 -f /dev/mapper/vgtest-lvol0
# mount /dev/mapper/vgtest-lvol0 /media/test
# ls /media/test/

The fsck command reports no errors, the filesystem is mountable and the 
data is there.

Looks like physical sector size didn't matter here. Or I'm missing 
something?

Cesare.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PVwith a larger physical block size

[Bernd Eckenfels]

[-- Attachment #1: Type: text/plain, Size: 3267 bytes --]

Hello,

I think the filesystems address in Logical blocks, so this is the size which should match.

However the physical size might be relevant for alignment/sizing decisions of mkfs  (but you would expect the to be encoded in the metadata of the filesystem so you can transport them (losing proper alignment which might affect Performance or robustness).

For ext3/4 I think the mkfs will use -b 4k by Default if your FS is at least 0,5GB.

BTW: some applications (like SQL Server) also care about the physical size to make sure they always write complete sectors in transactions and avoid read-modify-write scenarios.
 
Gruss
Bernd
-- 
http://bernd.eckenfels.net

Von: Cesare Leonardi
Gesendet: Freitag, 1. März 2019 02:24
An: Ingo Franzki; LVM general discussion and development
Betreff: Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PVwith a larger physical block size

On 28/02/19 09:41, Ingo Franzki wrote:
> Well, there are the following 2 commands:
> 
> Get physical block size:
>   blockdev --getpbsz <device>
> Get logical block size:
>   blockdev --getbsz <device>

I didn't know the blockdev command and, to recap, we have:
--getpbsz: physical sector size
--getss: logical sector size
--getbsz: blocksize used internally by the kernel

getpbsz/getss correspond to physical/logical sector size reported by 
fdisk, smartctl, etc.

> Filesystems seem to care about the physical block size only, not the logical block size.
> 
> So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...

I've done the test suggested by Stuart and it seems to contradict this.
I have pvmoved data from a 512/512 (logical/physical) disk to a newly 
added 512/4096 disk but I had no data corruption. Unfortunately I 
haven't any native 4k disk to repeat the same test.

Here is what I've done.
/dev/sdb: SSD with 512/512 sector size
/dev/sdc: mechanical disk with 512/4096 sector size

# blockdev -v --getss --getpbsz --getbsz /dev/sdb
get logical block (sector) size: 512
get physical block (sector) size: 512
get blocksize: 4096

# blockdev -v --getss --getpbsz --getbsz /dev/sdc
get logical block (sector) size: 512
get physical block (sector) size: 4096
get blocksize: 4096

# pvcreate /dev/sdb4
# vgcreate vgtest /dev/sdb4
# lvcreate -L 1G vgtest /dev/sdb4
# mkfs.ext4 /dev/mapper/vgtest-lvol0
# mkdir /media/test
# mount /dev/mapper/vgtest-lvol0 /media/test
# cp -a SOMEDATA /media/test/
# umount /media/test
# fsck.ext4 -f /dev/mapper/vgtest-lvol0

Filesystem created and no error on it. Now the disk with different 
physical size:

# pvcreate /dev/sdc1
# vgextend vgtest /dev/sdc1
# pvmove /dev/sdb4 /dev/sdc1
# fsck.ext4 -f /dev/mapper/vgtest-lvol0
# mount /dev/mapper/vgtest-lvol0 /media/test
# ls /media/test/

The fsck command reports no errors, the filesystem is mountable and the 
data is there.

Looks like physical sector size didn't matter here. Or I'm missing 
something?

Cesare.

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


[-- Attachment #2: Type: text/html, Size: 7020 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Stuart D. Gathman]

On Fri, 1 Mar 2019, Cesare Leonardi wrote:

> I've done the test suggested by Stuart and it seems to contradict this.
> I have pvmoved data from a 512/512 (logical/physical) disk to a newly added 
> 512/4096 disk but I had no data corruption. Unfortunately I haven't any 
> native 4k disk to repeat the same test.

Use a loopback device with logical block size set to 4096 to confirm
that your test does detect corruption (using the same LV, filesystem,
data).

I believe by "physical sector", the original reporter means logical,
as he was using an encrypted block device that was virtual - there
was no "physical" sector size.  It was "physical" as far as the
file system was concerned - where "physical" means "the next layer
down".

Indeed, even the rotating disk drives make the physical sector size
invisible except to performance tests.  SSD drives have a "sector" size
of 128k or 256k - the erase block, and performance improves when aligned
to that.

-- 
 	      Stuart D. Gathman <stuart@gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 01.03.2019 04:41, Stuart D. Gathman wrote:
> On Fri, 1 Mar 2019, Cesare Leonardi wrote:
> 
>> I've done the test suggested by Stuart and it seems to contradict this.
>> I have pvmoved data from a 512/512 (logical/physical) disk to a newly added 512/4096 disk but I had no data corruption. Unfortunately I haven't any native 4k disk to repeat the same test.
> 
> Use a loopback device with logical block size set to 4096 to confirm
> that your test does detect corruption (using the same LV, filesystem,
> data).
> 
> I believe by "physical sector", the original reporter means logical,
> as he was using an encrypted block device that was virtual - there
> was no "physical" sector size.  It was "physical" as far as the
> file system was concerned - where "physical" means "the next layer
> down".
Well, let me cite from https://www.saout.de/pipermail/dm-crypt/2019-February/006078.html from Ondrej Kozina which is also referenced in my original post:

"dm-crypt advertise itself as a block device with physical sector size 
*at least* equal to encryption sector size. Traditionally it's been only 
512B. So classical dm-crypt mapped over device with phys. sector size = 
512B has no impact. If you mapped dm-crypt over block device with native 
physical sector size = 4096 you got dm-crypt exposing same limits as 
underlying block device. Again no problem. Just internally dm-crypt 
performed encryption on 512B blocks, but it had no impact on exposed limits.

But things get a bit different with encryption sector size > 512B.

If you map dm-crypt with encryption sector size set to 4096B over block 
device with physical sector size = 512B, dm-crypt must increase device 
limits to 4096. Because when it does encryption it must be aligned to 
4096 bytes (and same wrt minimal i/o size)."

> 
> Indeed, even the rotating disk drives make the physical sector size
> invisible except to performance tests.  SSD drives have a "sector" size
> of 128k or 256k - the erase block, and performance improves when aligned
> to that.
> 


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PVwith a larger physical block size

[Ingo Franzki]

On 01.03.2019 03:56, Bernd Eckenfels wrote:
> Hello,
> 
> I think the filesystems address in Logical blocks, so this is the size which should match.
> 
> However the physical size might be relevant for alignment/sizing decisions of mkfs  (but you would expect the to be encoded in the metadata of the filesystem so you can transport them (losing proper alignment which might affect Performance or robustness).
> 
> For ext3/4 I think the mkfs will use -b 4k by Default if your FS is at least 0,5GB.
That may make a difference. My tests were with relatively small volumes, so it might not be using -b 4k due to the size of the volume?
> 
> BTW: some applications (like SQL Server) also care about the physical size to make sure they always write complete sectors in transactions and avoid read-modify-write scenarios.
>  
> Gruss
> Bernd
> 


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 01.03.2019 02:24, Cesare Leonardi wrote:
> On 28/02/19 09:41, Ingo Franzki wrote:
>> Well, there are the following 2 commands:
>>
>> Get physical block size:
>>   blockdev --getpbsz <device>
>> Get logical block size:
>>   blockdev --getbsz <device>
> 
> I didn't know the blockdev command and, to recap, we have:
> --getpbsz: physical sector size
> --getss: logical sector size
> --getbsz: blocksize used internally by the kernel
> 
> getpbsz/getss correspond to physical/logical sector size reported by fdisk, smartctl, etc.
> 
>> Filesystems seem to care about the physical block size only, not the logical block size.
>>
>> So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...
> 
> I've done the test suggested by Stuart and it seems to contradict this.
> I have pvmoved data from a 512/512 (logical/physical) disk to a newly added 512/4096 disk but I had no data corruption. Unfortunately I haven't any native 4k disk to repeat the same test.

Hmm, maybe the size of the volume plays a role as Bernd has pointed out. ext4 may use -b 4K by default on larger devices. 
Once the FS uses 4K block anyway you wont see the problem.

Use  tune2fs -l <device> after you created the file system and check if it is using 4K blocks on your 512/512 device. If so, then you won't see the problem when moved to a 4K block size device.

> 
> Here is what I've done.
> /dev/sdb: SSD with 512/512 sector size
> /dev/sdc: mechanical disk with 512/4096 sector size
> 
> # blockdev -v --getss --getpbsz --getbsz /dev/sdb
> get logical block (sector) size: 512
> get physical block (sector) size: 512
> get blocksize: 4096
> 
> # blockdev -v --getss --getpbsz --getbsz /dev/sdc
> get logical block (sector) size: 512
> get physical block (sector) size: 4096
> get blocksize: 4096
> 
> # pvcreate /dev/sdb4
> # vgcreate vgtest /dev/sdb4
> # lvcreate -L 1G vgtest /dev/sdb4
> # mkfs.ext4 /dev/mapper/vgtest-lvol0
> # mkdir /media/test
> # mount /dev/mapper/vgtest-lvol0 /media/test
> # cp -a SOMEDATA /media/test/
> # umount /media/test
> # fsck.ext4 -f /dev/mapper/vgtest-lvol0
> 
> Filesystem created and no error on it. Now the disk with different physical size:
> 
> # pvcreate /dev/sdc1
> # vgextend vgtest /dev/sdc1
> # pvmove /dev/sdb4 /dev/sdc1
> # fsck.ext4 -f /dev/mapper/vgtest-lvol0
> # mount /dev/mapper/vgtest-lvol0 /media/test
> # ls /media/test/
> 
> The fsck command reports no errors, the filesystem is mountable and the data is there.
> 
> Looks like physical sector size didn't matter here. Or I'm missing something?
> 
> Cesare.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Cesare Leonardi]

Hello Ingo, I've made several tests but I was unable to trigger any 
filesystem corruption. Maybe the trouble you encountered are specific to 
encrypted device?

Yesterday and today I've used:
Debian unstable
kernel 4.19.20
lvm2 2.03.02
e2fsprogs 1.44.5

On 01/03/19 09:05, Ingo Franzki wrote:
> Hmm, maybe the size of the volume plays a role as Bernd has pointed out. ext4 may use -b 4K by default on larger devices.
> Once the FS uses 4K block anyway you wont see the problem.
> 
> Use  tune2fs -l <device> after you created the file system and check if it is using 4K blocks on your 512/512 device. If so, then you won't see the problem when moved to a 4K block size device.

I confirm that tune2fs reports 4096 block size for the 1 GB ext4 
filesystem I've used.
I've also verified what Bernd said: mkfs.ext4 still use 4096 block size 
for a +512M partition, but use 1024 for +500M.

As suggested by Stuart, I also made a test using a 4k loop device and 
pvmoving the LV into it. As you expected, no data corruption.
To do it I've recreated the same setup ad yesterday: 
/dev/mapper/vgtest-lvol0 on /dev/sdb4, a 512/512 disk, with some data on 
it. Then:
# fallocate -l 10G testdisk.img
# losetup -f -L -P -b 4096 testdisk.img
# pvcreate /dev/loop0
# vgextend vgtest /dev/loop0
# pvmove /dev/sdb4 /dev/loop0
# fsck.ext4 -f /dev/mapper/vgtest-lvol0

While I was there, out of curiosity, I've created an ext4 filesystem on 
a <500MB LV (block size = 1024) and I've tried pvmoving data from the 
512/512 disk to 512/4096, then to the 4096/4096 loop device.
New partitions and a new VG was used for that.

The setup:
/dev/sdb5: 512/512
/dev/sdc2: 512/4096
/dev/loop0 4096/4096

# blockdev -v --getss --getpbsz --getbsz /dev/sdb
get logical block (sector) size: 512
get physical block (sector) size: 512
get blocksize: 4096

# blockdev -v --getss --getpbsz --getbsz /dev/sdc
get logical block (sector) size: 512
get physical block (sector) size: 4096
get blocksize: 4096

# blockdev -v --getss --getpbsz --getbsz /dev/loop0
get logical block (sector) size: 4096
get physical block (sector) size: 4096
get blocksize: 4096

# pvcreate /dev/sdb5
# vgcreate vgtest2 /dev/sdb5
# lvcreate -L 400M vgtest2 /dev/sdb5
# mkfs.ext4 /dev/mapper/vgtest2-lvol0

# tune2fs -l /dev/mapper/vgtest2-lvol0
[...]
Block size:               1024
[...]

# mount /dev/mapper/vgtest2-lvol0 /media/test
# cp -a SOMEDATA /media/test/
# umount /media/test
# fsck.ext4 -f /dev/mapper/vgtest2-lvol0

Now I've moved data from the 512/512 to the 512/4096 disk:
# pvcreate /dev/sdc2
# vgextend vgtest2 /dev/sdc2
# pvmove /dev/sdb5 /dev/sdc2
# fsck.ext4 -f /dev/mapper/vgtest2-lvol0

No error reported.
Now I've moved data to the 4096/4096 loop device:
# pvcreate /dev/loop0
# vgextend vgtest2 /dev/loop0
# pvmove /dev/sdc2 /dev/loop0
# fsck.ext4 -f /dev/mapper/vgtest2-lvol0

Still no data corruption.

Cesare.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[L A Walsh]

On 2/27/2019 9:05 AM, Ingo Franzki wrote:
> Yes that should work:
> # losetup -fP loopbackfile.img --sector-size 4096
> # blockdev --getpbsz /dev/loop0
> 4096
>   
-----
Something I noticed that is troublesome.  When I first got my 4K sectory
size disks, one of the numbers in the kernel listed it as 4K, now ~4-5
years later, that difference no longer appears in the kernel!  To me, this
seems a bit unwise -- as even though these disks are "512_e_"  (a 512 byte
sector size can be emulated but writes will usually or often take a minimum
of 2 revolutions so that the rest of the 4k sector can be read, have
512-bytes
updated, then write the whole 4k sector size out.  In otherwords -- hideous
performance on writes. 

Now the RAID controller that manages this disk also has info on the
disks -- and
there it correctly shows the 4k sector size.  So am wondering what in
the kernel
"broke"(?) to disable detection & display of its native 4k sector size.


Anyway, I don't think it should cause problems unless I reformat or move the
partitions,  but it would be nice to know the various disk software knows
not to try anything smaller than a 4k sector size.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Nir Soffer]

[-- Attachment #1: Type: text/plain, Size: 7552 bytes --]

On Sat, Mar 2, 2019 at 3:38 AM Cesare Leonardi <celeonar@gmail.com> wrote:

> Hello Ingo, I've made several tests but I was unable to trigger any
> filesystem corruption. Maybe the trouble you encountered are specific to
> encrypted device?
>
> Yesterday and today I've used:
> Debian unstable
> kernel 4.19.20
> lvm2 2.03.02
> e2fsprogs 1.44.5
>
> On 01/03/19 09:05, Ingo Franzki wrote:
> > Hmm, maybe the size of the volume plays a role as Bernd has pointed out.
> ext4 may use -b 4K by default on larger devices.
> > Once the FS uses 4K block anyway you wont see the problem.
> >
> > Use  tune2fs -l <device> after you created the file system and check if
> it is using 4K blocks on your 512/512 device. If so, then you won't see the
> problem when moved to a 4K block size device.
>
> I confirm that tune2fs reports 4096 block size for the 1 GB ext4
> filesystem I've used.
> I've also verified what Bernd said: mkfs.ext4 still use 4096 block size
> for a +512M partition, but use 1024 for +500M.
>
> As suggested by Stuart, I also made a test using a 4k loop device and
> pvmoving the LV into it. As you expected, no data corruption.
> To do it I've recreated the same setup ad yesterday:
> /dev/mapper/vgtest-lvol0 on /dev/sdb4, a 512/512 disk, with some data on
> it. Then:
> # fallocate -l 10G testdisk.img
> # losetup -f -L -P -b 4096 testdisk.img
> # pvcreate /dev/loop0
> # vgextend vgtest /dev/loop0
> # pvmove /dev/sdb4 /dev/loop0
> # fsck.ext4 -f /dev/mapper/vgtest-lvol0
>
> While I was there, out of curiosity, I've created an ext4 filesystem on
> a <500MB LV (block size = 1024) and I've tried pvmoving data from the
> 512/512 disk to 512/4096, then to the 4096/4096 loop device.
> New partitions and a new VG was used for that.
>
> The setup:
> /dev/sdb5: 512/512
> /dev/sdc2: 512/4096
> /dev/loop0 4096/4096
>
> # blockdev -v --getss --getpbsz --getbsz /dev/sdb
> get logical block (sector) size: 512
> get physical block (sector) size: 512
> get blocksize: 4096
>
> # blockdev -v --getss --getpbsz --getbsz /dev/sdc
> get logical block (sector) size: 512
> get physical block (sector) size: 4096
> get blocksize: 4096
>
> # blockdev -v --getss --getpbsz --getbsz /dev/loop0
> get logical block (sector) size: 4096
> get physical block (sector) size: 4096
> get blocksize: 4096
>
> # pvcreate /dev/sdb5
> # vgcreate vgtest2 /dev/sdb5
> # lvcreate -L 400M vgtest2 /dev/sdb5
> # mkfs.ext4 /dev/mapper/vgtest2-lvol0
>
> # tune2fs -l /dev/mapper/vgtest2-lvol0
> [...]
> Block size:               1024
> [...]
>
> # mount /dev/mapper/vgtest2-lvol0 /media/test
> # cp -a SOMEDATA /media/test/
> # umount /media/test
> # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
>
> Now I've moved data from the 512/512 to the 512/4096 disk:
> # pvcreate /dev/sdc2
> # vgextend vgtest2 /dev/sdc2
> # pvmove /dev/sdb5 /dev/sdc2
> # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
>
> No error reported.
>

Did you try to mount the lv after the pvmove?


> Now I've moved data to the 4096/4096 loop device:
> # pvcreate /dev/loop0
> # vgextend vgtest2 /dev/loop0
> # pvmove /dev/sdc2 /dev/loop0
> # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
>
> Still no data corruption.
>

I can reproduce this without moving data, just by extending vg with 4k
device,
and then extending lv to use both devices.

Here is what I tested:

# truncate -s 500m disk1
# truncate -s 500m disk2
# losetup -f disk1 --sector-size 512 --show
/dev/loop2
# losetup -f disk2 --sector-size 4096 --show
/dev/loop3

# pvcreate /dev/loop2
  Physical volume "/dev/loop2" successfully created.
# pvcreate /dev/loop3
  Physical volume "/dev/loop3" successfully created.
# vgcreate test /dev/loop2
  Volume group "test" successfully created
# lvcreate -L400m -n lv1 test
  Logical volume "lv1" created.

# mkfs.xfs /dev/test/lv1
meta-data=/dev/test/lv1          isize=512    agcount=4, agsize=25600 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0,
reflink=0
data     =                       bsize=4096   blocks=102400, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=855, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

# mkdir /tmp/mnt
# mount /dev/test/lv1 /tmp/mnt
# vgextend test /dev/loop3
  Volume group "test" successfully extended
# lvextend -L+400m test/lv1
  Size of logical volume test/lv1 changed from 400.00 MiB (100 extents) to
800.00 MiB (200 extents).
  Logical volume test/lv1 successfully resized.
# umount /tmp/mnt

# mount /dev/test/lv1 /tmp/mnt
mount: /tmp/mnt: mount(2) system call failed: Function not implemented.

From journalctl:
Mar 02 21:52:53 lean.local kernel: XFS (dm-7): Unmounting Filesystem
Mar 02 21:53:01 lean.local kernel: XFS (dm-7): device supports 4096 byte
sectors (not 512)


I also tried the same with ext4:

(same disks/vg/lv setup as above)

# mkfs.ext4 /dev/test/lv1
mke2fs 1.44.2 (14-May-2018)
Discarding device blocks: done
Creating filesystem with 409600 1k blocks and 102400 inodes
Filesystem UUID: 9283880e-ee89-4d79-9c29-41f4af98f894
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

# vgextend test /dev/loop3
  Volume group "test" successfully extended
# lvextend -L+400 test/lv1
  Size of logical volume test/lv1 changed from 400.00 MiB (100 extents) to
800.00 MiB (200 extents).
  Logical volume test/lv1 successfully resized.

# mount /dev/test/lv1 /tmp/mnt
mount: /tmp/mnt: wrong fs type, bad option, bad superblock on
/dev/mapper/test-lv1, missing codepage or helper program, or other error.

From journalctl:
Mar 02 22:06:09 lean.local kernel: EXT4-fs (dm-7): bad block size 1024


Now same with pvmove:

(same setup as above, using xfs)

# mount /dev/test/lv1 /tmp/mnt
# dd if=/dev/urandom bs=8M count=1 of=/tmp/mnt/data
# vgextend test /dev/loop3
  Physical volume "/dev/loop3" successfully created.
  Volume group "test" successfully extended

# pvmove -v /dev/loop2 /dev/loop3
    Cluster mirror log daemon is not running.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Archiving volume group "test" metadata (seqno 3).
    Creating logical volume pvmove0
    Moving 100 extents of logical volume test/lv1.
    activation/volume_list configuration setting not defined: Checking only
host tags for test/lv1.
    Creating test-pvmove0
    Loading table for test-pvmove0 (253:8).
    Loading table for test-lv1 (253:7).
    Suspending test-lv1 (253:7) with device flush
    Resuming test-pvmove0 (253:8).
    Resuming test-lv1 (253:7).
    Creating volume group backup "/etc/lvm/backup/test" (seqno 4).
    activation/volume_list configuration setting not defined: Checking only
host tags for test/pvmove0.
    Checking progress before waiting every 15 seconds.
  /dev/loop2: Moved: 15.00%
  /dev/loop2: Moved: 100.00%
    Polling finished successfully.
# umount /tmp/mnt
# mount /dev/test/lv1 /tmp/mnt
mount: /tmp/mnt: mount(2) system call failed: Function not implemented.

From journalctl:
Mar 02 22:20:36 lean.local kernel: XFS (dm-7): device supports 4096 byte
sectors (not 512)


Tested on Fedora 28 with:
kernel-4.20.5-100.fc28.x86_64
lvm2-2.02.177-5.fc28.x86_64

Nir

[-- Attachment #2: Type: text/html, Size: 16506 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 02.03.2019 02:36, Cesare Leonardi wrote:
> Hello Ingo, I've made several tests but I was unable to trigger any filesystem corruption. Maybe the trouble you encountered are specific to encrypted device?
> 
> Yesterday and today I've used:
> Debian unstable
> kernel 4.19.20
> lvm2 2.03.02
> e2fsprogs 1.44.5
> 
> On 01/03/19 09:05, Ingo Franzki wrote:
>> Hmm, maybe the size of the volume plays a role as Bernd has pointed out. ext4 may use -b 4K by default on larger devices.
>> Once the FS uses 4K block anyway you wont see the problem.
>>
>> Use  tune2fs -l <device> after you created the file system and check if it is using 4K blocks on your 512/512 device. If so, then you won't see the problem when moved to a 4K block size device.
> 
> I confirm that tune2fs reports 4096 block size for the 1 GB ext4 filesystem I've used.
> I've also verified what Bernd said: mkfs.ext4 still use 4096 block size for a +512M partition, but use 1024 for +500M.
> 
> As suggested by Stuart, I also made a test using a 4k loop device and pvmoving the LV into it. As you expected, no data corruption.
> To do it I've recreated the same setup ad yesterday: /dev/mapper/vgtest-lvol0 on /dev/sdb4, a 512/512 disk, with some data on it. Then:
> # fallocate -l 10G testdisk.img
> # losetup -f -L -P -b 4096 testdisk.img
> # pvcreate /dev/loop0
> # vgextend vgtest /dev/loop0
> # pvmove /dev/sdb4 /dev/loop0
> # fsck.ext4 -f /dev/mapper/vgtest-lvol0
> 
> While I was there, out of curiosity, I've created an ext4 filesystem on a <500MB LV (block size = 1024) and I've tried pvmoving data from the 512/512 disk to 512/4096, then to the 4096/4096 loop device.
> New partitions and a new VG was used for that.
> 
> The setup:
> /dev/sdb5: 512/512
> /dev/sdc2: 512/4096
> /dev/loop0 4096/4096
> 
> # blockdev -v --getss --getpbsz --getbsz /dev/sdb
> get logical block (sector) size: 512
> get physical block (sector) size: 512
> get blocksize: 4096
You display the physical block size of /dev/sdb here, but you use /dev/sdb5 later on.
Not sure if this makes a difference ....
> 
> # blockdev -v --getss --getpbsz --getbsz /dev/sdc
> get logical block (sector) size: 512
> get physical block (sector) size: 4096
> get blocksize: 4096
Again, you display the physical block size of /dev/sdc here, but you use /dev/sdc2 later on.
> 
> # blockdev -v --getss --getpbsz --getbsz /dev/loop0
> get logical block (sector) size: 4096
> get physical block (sector) size: 4096
> get blocksize: 4096
> 
> # pvcreate /dev/sdb5
> # vgcreate vgtest2 /dev/sdb5
> # lvcreate -L 400M vgtest2 /dev/sdb5
> # mkfs.ext4 /dev/mapper/vgtest2-lvol0
> 
> # tune2fs -l /dev/mapper/vgtest2-lvol0
> [...]
> Block size:               1024
> [...]
> 
> # mount /dev/mapper/vgtest2-lvol0 /media/test
> # cp -a SOMEDATA /media/test/
> # umount /media/test
> # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
> 
> Now I've moved data from the 512/512 to the 512/4096 disk:
> # pvcreate /dev/sdc2
> # vgextend vgtest2 /dev/sdc2
> # pvmove /dev/sdb5 /dev/sdc2
> # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
Please note that fsck.ext4 does not seem to detect this kind of corruption. 
In my case fsck.ext4 reported that the FS would be clean (!), but a mount count not mount it anymore...

Do a 'pvs' command here, this should show some error messages.
> 
> No error reported.
> Now I've moved data to the 4096/4096 loop device:
> # pvcreate /dev/loop0
> # vgextend vgtest2 /dev/loop0
> # pvmove /dev/sdc2 /dev/loop0
> # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
Again, fsck did not show the corruption for me, but pvs did show error messages.
> 
> Still no data corruption.
> 
> Cesare.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Cesare Leonardi]

On 04/03/19 10:12, Ingo Franzki wrote:
>> # blockdev -v --getss --getpbsz --getbsz /dev/sdb
>> get logical block (sector) size: 512
>> get physical block (sector) size: 512
>> get blocksize: 4096
> You display the physical block size of /dev/sdb here, but you use /dev/sdb5 later on.
> Not sure if this makes a difference ....

I thought that was the right thing to do, as they are disk parameters. 
At least the first two, for the last I'm not sure.
However the output looks the same:
# blockdev -v --getss --getpbsz --getbsz /dev/sdb5
get logical block (sector) size: 512
get physical block (sector) size: 512
get blocksize: 4096

# blockdev -v --getss --getpbsz --getbsz /dev/sdc2
get logical block (sector) size: 512
get physical block (sector) size: 4096
get blocksize: 4096

> Please note that fsck.ext4 does not seem to detect this kind of corruption.
> In my case fsck.ext4 reported that the FS would be clean (!), but a mount count not mount it anymore...
> 
> Do a 'pvs' command here, this should show some error messages.

Uh, I didn't really expect that such corruption could pass unoticed an 
fsck.ext4 check. During my tests, initially I surely tried to mount the 
filesystem and I did ls on it but it's possible that after some steps I 
only trusted fsck.

Today I repeated all the tests and indeed in one case the mount failed: 
after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV 
ext4 using 1024 block size.

Here is what I've tested:
/dev/sdb: SSD with 512/512 sector size
/dev/sdc: mechanical disk with 512/4096 sector size
/dev/loop0: emulated disk with 4096/4096 sector size

TEST 1
VG vgtest1: /dev/sdb4 /dev/sdc1 /dev/loop0p1
LV vgtest1-lvol0: filesystem ext4 with 4096 block size
pvmove ext4-4096:
- from /dev/sdb4 (512/512) to /dev/sdc1 (512/4096): ok
- from /dev/sdc1 (512/4096) to /dev/loop0p1 (4096/4096): ok

TEST 2
VG vgtest2: /dev/sdb5 /dev/sdc2 /dev/loop0p2
LV vgtest2-lvol0: filesystem ext4 with 1024 block size
pvmove ext4-1024:
- from /dev/sdb5 (512/512) to /dev/sdc2 (512/4096): ok
- from /dev/sdc2 (512/4096) to /dev/loop0p2 (4096/4096): fail

Here the outputs of the failed test:
=======================
# pvmove /dev/sdc2 /dev/loop0p2
   /dev/sdc2: Moved: 9,00%
   /dev/sdc2: Moved: 100,00%

# mount /dev/mapper/vgtest2-lvol0 /media/test/
mount: /media/test: wrong fs type, bad option, bad superblock on
/dev/mapper/vgtest2-lvol0, missing codepage or helper program, or other
error.

# fsck.ext4 -f  /dev/mapper/vgtest2-lvol0
e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/vgtest2-lvol0: 35/102400 files (17.1% non-contiguous), 
304877/409600 blocks
=======================

The error happened where you guys expected. And also for me fsck showed 
no errors.

But doesn't look like a filesystem corruption: if you pvmove back the 
data, it will become readable again:
# pvmove /dev/loop0p2 /dev/sdc2
   /dev/loop0p2: Moved: 1,00%
   /dev/loop0p2: Moved: 100,00%
# mount /dev/mapper/vgtest2-lvol0 /media/test/
# ls /media/test/
epson  hp  kerio  lost+found

And also notice that the pvmove that generated the unreadable filesystem 
starts with an unusual high percentage (9%). In all other test it start 
from 0% or a small number near 1%. This happened in more that one case.

Cesare.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Cesare Leonardi]

On 02/03/19 21:25, Nir Soffer wrote:
> # mkfs.xfs /dev/test/lv1
> meta-data=/dev/test/lv1          isize=512    agcount=4, agsize=25600 blks
>           =                       sectsz=512   attr=2, projid32bit=1
>           =                       crc=1        finobt=1, sparse=0, 
> rmapbt=0, reflink=0
> data     =                       bsize=4096   blocks=102400, imaxpct=25
>           =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
> log      =internal log           bsize=4096   blocks=855, version=2
>           =                       sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0

Has the problem here the same root as for ext4? I guess sectsz should be 
 >=4096 to avoid troubles, isn't it?

Just to draw some conlusion, could we say that currently, if we are 
going to move data around with LVM, it's better to check that the 
filesystem is using a block size >= than "blockdev --getbsz 
DESTINATIONDEVICE"? At least with ext4 and xfs.

Something that couldn't be true with really small devices (< 500 MB).

Is there already an open bug regarding the problem discussed in this thread?

Cesare.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Nir Soffer]

[-- Attachment #1: Type: text/plain, Size: 1587 bytes --]

On Tue, Mar 5, 2019 at 12:45 AM Cesare Leonardi <celeonar@gmail.com> wrote:

> On 02/03/19 21:25, Nir Soffer wrote:
> > # mkfs.xfs /dev/test/lv1
> > meta-data=/dev/test/lv1          isize=512    agcount=4, agsize=25600
> blks
> >           =                       sectsz=512   attr=2, projid32bit=1
> >           =                       crc=1        finobt=1, sparse=0,
> > rmapbt=0, reflink=0
> > data     =                       bsize=4096   blocks=102400, imaxpct=25
> >           =                       sunit=0      swidth=0 blks
> > naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
> > log      =internal log           bsize=4096   blocks=855, version=2
> >           =                       sectsz=512   sunit=0 blks, lazy-count=1
> > realtime =none                   extsz=4096   blocks=0, rtextents=0
>
> Has the problem here the same root as for ext4? I guess sectsz should be
>  >=4096 to avoid troubles, isn't it?
>
> Just to draw some conlusion, could we say that currently, if we are
> going to move data around with LVM, it's better to check that the
> filesystem is using a block size >= than "blockdev --getbsz
> DESTINATIONDEVICE"? At least with ext4 and xfs.
>
> Something that couldn't be true with really small devices (< 500 MB).
>
> Is there already an open bug regarding the problem discussed in this
> thread?
>

There is this bug about lvextend:
https://bugzilla.redhat.com/1669751

And this old bug from 2011, discussing mixing PVs with different block size.
Comment 2 is very clear about this issue:
https://bugzilla.redhat.com/732980#c2

Nir

[-- Attachment #2: Type: text/html, Size: 2995 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Stuart D. Gathman]

On Mon, 4 Mar 2019, Cesare Leonardi wrote:

> Today I repeated all the tests and indeed in one case the mount failed: after 
> pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV ext4 using 
> 1024 block size.
  ...
> The error happened where you guys expected. And also for me fsck showed no 
> errors.
>
> But doesn't look like a filesystem corruption: if you pvmove back the data, 
> it will become readable again:
  ...

THAT is a crucial observation.  It's not an LVM bug, but the filesystem
trying to read 1024 bytes on a 4096 device.  I suspect it could also
happen with an unaligned filesystem on a 4096 device.

-- 
 	      Stuart D. Gathman <stuart@gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 05.03.2019 01:12, Stuart D. Gathman wrote:
> On Mon, 4 Mar 2019, Cesare Leonardi wrote:
> 
>> Today I repeated all the tests and indeed in one case the mount failed: after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV ext4 using 1024 block size.
>  ...
>> The error happened where you guys expected. And also for me fsck showed no errors.
>>
>> But doesn't look like a filesystem corruption: if you pvmove back the data, it will become readable again:
>  ...
> 
> THAT is a crucial observation.  It's not an LVM bug, but the filesystem
> trying to read 1024 bytes on a 4096 device.  
Yes that's probably the reason. Nevertheless, its not really the FS's fault, since it was moved by LVM to a 4069 device.
The FS does not know anything about the move, so it reads in the block size it was created with (1024 in this case).

I still think LVM should prevent one from mixing devices with different physical block sizes, or at least warn when pvmoving or lvextending onto a PV with a larger block size, since this can cause trouble.

> I suspect it could also
> happen with an unaligned filesystem on a 4096 device.
> 

-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 05.03.2019 00:22, Nir Soffer wrote:
> On Tue, Mar 5, 2019 at 12:45 AM Cesare Leonardi <celeonar@gmail.com> wrote:
> 
>> On 02/03/19 21:25, Nir Soffer wrote:
>>> # mkfs.xfs /dev/test/lv1
>>> meta-data=/dev/test/lv1          isize=512    agcount=4, agsize=25600
>> blks
>>>           =                       sectsz=512   attr=2, projid32bit=1
>>>           =                       crc=1        finobt=1, sparse=0,
>>> rmapbt=0, reflink=0
>>> data     =                       bsize=4096   blocks=102400, imaxpct=25
>>>           =                       sunit=0      swidth=0 blks
>>> naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
>>> log      =internal log           bsize=4096   blocks=855, version=2
>>>           =                       sectsz=512   sunit=0 blks, lazy-count=1
>>> realtime =none                   extsz=4096   blocks=0, rtextents=0
>>
>> Has the problem here the same root as for ext4? I guess sectsz should be
>>  >=4096 to avoid troubles, isn't it?
>>
>> Just to draw some conlusion, could we say that currently, if we are
>> going to move data around with LVM, it's better to check that the
>> filesystem is using a block size >= than "blockdev --getbsz
>> DESTINATIONDEVICE"? At least with ext4 and xfs.
>>
>> Something that couldn't be true with really small devices (< 500 MB).
>>
>> Is there already an open bug regarding the problem discussed in this
>> thread?
>>
> 
> There is this bug about lvextend:
> https://bugzilla.redhat.com/1669751
> 
> And this old bug from 2011, discussing mixing PVs with different block size.
> Comment 2 is very clear about this issue:
> https://bugzilla.redhat.com/732980#c2
I don't have access to that one, can you cite comment 2 ?
> 
> Nir
> 


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ilia Zykov]

[-- Attachment #1: Type: text/plain, Size: 1110 bytes --]

Hello.

>> THAT is a crucial observation.  It's not an LVM bug, but the filesystem
>> trying to read 1024 bytes on a 4096 device.  
> Yes that's probably the reason. Nevertheless, its not really the FS's fault, since it was moved by LVM to a 4069 device.
> The FS does not know anything about the move, so it reads in the block size it was created with (1024 in this case).
> 
> I still think LVM should prevent one from mixing devices with different physical block sizes, or at least warn when pvmoving or lvextending onto a PV with a larger block size, since this can cause trouble.
> 

In this case, "dd" tool and others should prevent too.

Because after:

dd if=/dev/DiskWith512block bs=4096 of=/dev/DiskWith4Kblock

You couldn't mount the "/dev/DiskWith4Kblock" with the same error ;)
/dev/DiskWith512block has ext4 fs with 1k block.

P.S.
LVM,dd .. are low level tools and doesn't know about hi level anything.
And in the your case and others cases can't know. You should test(if you
need) the block size with other tools before moving or copying.
Not a lvm bug.
Thank you.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3591 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Ingo Franzki]

On 05.03.2019 10:29, Ilia Zykov wrote:
> Hello.
> 
>>> THAT is a crucial observation.  It's not an LVM bug, but the filesystem
>>> trying to read 1024 bytes on a 4096 device.  
>> Yes that's probably the reason. Nevertheless, its not really the FS's fault, since it was moved by LVM to a 4069 device.
>> The FS does not know anything about the move, so it reads in the block size it was created with (1024 in this case).
>>
>> I still think LVM should prevent one from mixing devices with different physical block sizes, or at least warn when pvmoving or lvextending onto a PV with a larger block size, since this can cause trouble.
>>
> 
> In this case, "dd" tool and others should prevent too.

Well, no, its LVM's pvmove who moves the data from a 512 block size to 4096 block size device.
So its not dd that's causing the problem, but pvmove. 

> 
> Because after:
> 
> dd if=/dev/DiskWith512block bs=4096 of=/dev/DiskWith4Kblock
> 
> You couldn't mount the "/dev/DiskWith4Kblock" with the same error ;)
> /dev/DiskWith512block has ext4 fs with 1k block.
> 
> P.S.
> LVM,dd .. are low level tools and doesn't know about hi level anything.
> And in the your case and others cases can't know. You should test(if you
> need) the block size with other tools before moving or copying.
> Not a lvm bug.
Well, maybe not a bug, but I would still like to see that LVM's pvmove/lvextend and/or vgextend issues at least a warning when you have mixed block sizes. 

LVM knows the block sizes of the PVs it manages and it also knows when it changes the block size of an LV due to a pvmove or lvextend. So it can issue a warning (better a confirmation prompt) when this happens.
> Thank you.
> 


-- 
Ingo Franzki
eMail: ifranzki@linux.ibm.com  
Tel: ++49 (0)7031-16-4648
Fax: ++49 (0)7031-16-3456
Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany

IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Nir Soffer]

[-- Attachment #1: Type: text/plain, Size: 1704 bytes --]

On Tue, Mar 5, 2019 at 11:30 AM Ilia Zykov <mail@izyk.ru> wrote:

> Hello.
>
> >> THAT is a crucial observation.  It's not an LVM bug, but the filesystem
> >> trying to read 1024 bytes on a 4096 device.
> > Yes that's probably the reason. Nevertheless, its not really the FS's
> fault, since it was moved by LVM to a 4069 device.
> > The FS does not know anything about the move, so it reads in the block
> size it was created with (1024 in this case).
> >
> > I still think LVM should prevent one from mixing devices with different
> physical block sizes, or at least warn when pvmoving or lvextending onto a
> PV with a larger block size, since this can cause trouble.
> >
>
> In this case, "dd" tool and others should prevent too.
>
> Because after:
>
> dd if=/dev/DiskWith512block bs=4096 of=/dev/DiskWith4Kblock
>
> You couldn't mount the "/dev/DiskWith4Kblock" with the same error ;)
> /dev/DiskWith512block has ext4 fs with 1k block.
>
> P.S.
> LVM,dd .. are low level tools and doesn't know about hi level anything.
> And in the your case and others cases can't know. You should test(if you
> need) the block size with other tools before moving or copying.
> Not a lvm bug.
>

I don't this way of thinking is useful. If we go in this way, then write()
should not
let you write data, and later maybe the disk controller should avoid this?

LVM is not a low level tool like dd. It is high level tool for managing
device mapper,
and providing high level tools to create user level abstractions. We can
expect it
to prevent system administrator from doing the wrong thing.

Maybe LVM should let you mix PVs with different logical block size, but it
should
require --force.

David, what do you think?

[-- Attachment #2: Type: text/html, Size: 3054 bytes --]

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[David Teigland]

On Tue, Mar 05, 2019 at 06:29:31PM +0200, Nir Soffer wrote:
> I don't this way of thinking is useful. If we go in this way, then write()
> should not
> let you write data, and later maybe the disk controller should avoid this?
> 
> LVM is not a low level tool like dd. It is high level tool for managing
> device mapper,
> and providing high level tools to create user level abstractions. We can
> expect it
> to prevent system administrator from doing the wrong thing.
> 
> Maybe LVM should let you mix PVs with different logical block size, but it
> should
> require --force.
> 
> David, what do you think?

LVM needs to fix this, your solution sounds like the right one.

Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size

[Stuart D. Gathman]

On Tue, 5 Mar 2019, David Teigland wrote:

> On Tue, Mar 05, 2019 at 06:29:31PM +0200, Nir Soffer wrote:
>> Maybe LVM should let you mix PVs with different logical block size, but it
>> should
>> require --force.
>
> LVM needs to fix this, your solution sounds like the right one.

Also, since nearly every modern device device has a physical block size of
4k or more, and even when the logical block size is (emulated) 512,
performance degradation occurs with smaller filesystem blocks, 
then the savvy admin should ensure that all filesystem have a min of 
4k block size - except in special circustances.

-- 
 	      Stuart D. Gathman <stuart@gathman.org>
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.