Any way in LVM to deal with 512e vs 4Kn physical devices?

Any way in LVM to deal with 512e vs 4Kn physical devices?

[Andy Smith]

Hi,

On machine 'A' I have a pair of:

Device Model:     Samsung SSD 870 EVO 4TB
Sector Size:      512 bytes logical/physical

on top of this is an mdadm RAID-1 and that is an LVM PV.

One of the LVs has been partitioned with an MBR and a single
partition spanning the whole of the 400GiB LV.

I took a dd of this LV and transferred it to an identically-sized
LV on machine 'B' which has a pair of:

Device Model:     HGST HUS726T6TALN6L4
Sector Size:      4096 bytes logical/physical

The LV there when examined in a partitioning tool such as "fdisk"
now thinks it has a 3.2TiB partition and it is not usable.
Correcting the partition sector numbers allows for use of, for
example, "kpartx", to expose the partition as a loop device but the
ext4 driver and fsck.ext4 remain unable to detect a superblock.

I have confirmed with sha256sum that the content of the
image/partition remains the same on source and destination.

So, clearly the issue is the 512e sector size on source vs 4Kn on
destination. Is there any way to work around this in LVM? My issue
is that I would like to be able to move images of disks/filesystems
around at the block level without mounting/creating filesystem and
transferring with an fs-level application.

If not, then possibly I can use hdparm to set the 4Kn drives to 512,
which will obviously involve destroying their contents, but that is
fine at this stage.

I don't think the presence of a partition (as opposed to an ext4
filesystem directly upon the LV) is relevant; I think the same
issues would occur with a direct filesystem. I mention it only for
completeness. Also, I realise that the problems would also happen
without LVM. I just wonder if there is any workaround at the LVM
layer, since that is already used here.

Thanks,
Andy

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Roger Heflin]

On Mon, Jan 15, 2024 at 1:49 AM Andy Smith <andy@strugglers.net> wrote:
>
> Hi,
>
> On machine 'A' I have a pair of:
>
> Device Model:     Samsung SSD 870 EVO 4TB
> Sector Size:      512 bytes logical/physical
>
> on top of this is an mdadm RAID-1 and that is an LVM PV.
>
> One of the LVs has been partitioned with an MBR and a single
> partition spanning the whole of the 400GiB LV.
>
> I took a dd of this LV and transferred it to an identically-sized
> LV on machine 'B' which has a pair of:
>
> Device Model:     HGST HUS726T6TALN6L4
> Sector Size:      4096 bytes logical/physical
>
> The LV there when examined in a partitioning tool such as "fdisk"
> now thinks it has a 3.2TiB partition and it is not usable.
> Correcting the partition sector numbers allows for use of, for
> example, "kpartx", to expose the partition as a loop device but the
> ext4 driver and fsck.ext4 remain unable to detect a superblock.
>
> I have confirmed with sha256sum that the content of the
> image/partition remains the same on source and destination.
>
> So, clearly the issue is the 512e sector size on source vs 4Kn on
> destination. Is there any way to work around this in LVM? My issue
> is that I would like to be able to move images of disks/filesystems
> around at the block level without mounting/creating filesystem and
> transferring with an fs-level application.
>
> If not, then possibly I can use hdparm to set the 4Kn drives to 512,
> which will obviously involve destroying their contents, but that is
> fine at this stage.
>
> I don't think the presence of a partition (as opposed to an ext4
> filesystem directly upon the LV) is relevant; I think the same
> issues would occur with a direct filesystem. I mention it only for
> completeness. Also, I realise that the problems would also happen
> without LVM. I just wonder if there is any workaround at the LVM
> layer, since that is already used here.

512e vs 4096 should not matter at all for LVM.  LVM and Linux at their
core work at page-size and that is 4k.

You should probably include an fdisk -l <lv> from the working system
and a show what device it is mounted from.  And a fdisk -l
<device-it-was-dd-ed-to>

Note PV == physical device (a disk sometimes partitioned).
VG = Volume Group (a group of PVs)
LV = logical volume (section and/or entire PV's mapped together,
rarely partitioned).

If you dd'ed the entire LV over to another disk, did you put it on the
entire disk or inside a partition on a disk?

And if you put the partition on the LV after the ext4 fs then the
partition may have damaged the ext4 fs.

If inside a partition on a disk, then partitions on partitions get
tricky as I have found you often need to force a read/create of the
partition entries inside that device.  If you find the right command
and run it against the partitioned device (that has its own partition)
then you may be able to mount it (kpartx -a /dev/sdx1, assuming sdx1
is where you copied the LV).  Note this will not survive reboot, and
you will need to run it again).  The device after partition of a
partition is created would be something like /dev/sdx1p1.

If that is the case you may also have better luck copying just the
specific partition inside the LV over that has the filesystem and not
messing with the partition.

And if you changed the start point of the partition were the ext4 fs
is then there will be no ext4 fs were it is expected and nothing will
find it.

Note the partition on the LV means that the ext4 fs does not start
where any tools expect it to start and so nothing can find it, and as
such you need to get the partition mapping defined.

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Phillip Susi]

Andy Smith <andy@strugglers.net> writes:

> The LV there when examined in a partitioning tool such as "fdisk"
> now thinks it has a 3.2TiB partition and it is not usable.
> Correcting the partition sector numbers allows for use of, for
> example, "kpartx", to expose the partition as a loop device but the
> ext4 driver and fsck.ext4 remain unable to detect a superblock.

You mean you put a partition table inside of the logical volume?  Why?
IIRC, the dos partition table uses units of logical sectors, so if the
logical sector size changes, the start sector and length will be off.
You would need to create a new partition table and use the correctly
adjusted numbers when creating the partition, if that is even possible.
In your case, you will need to divide them all by 8, but if any of them
is not an even multiple of 8, you won't be able to do that.

> I have confirmed with sha256sum that the content of the
> image/partition remains the same on source and destination.

If you have already modified the partition table, then how could it
still have the same sha256sum?

> So, clearly the issue is the 512e sector size on source vs 4Kn on
> destination. Is there any way to work around this in LVM? My issue
> is that I would like to be able to move images of disks/filesystems
> around at the block level without mounting/creating filesystem and
> transferring with an fs-level application.

LVM doesn't really know or care about it.

> I don't think the presence of a partition (as opposed to an ext4
> filesystem directly upon the LV) is relevant; I think the same
> issues would occur with a direct filesystem. I mention it only for

No, it wouldn't be a problem without the partition table.  ext4 uses its
own block size, which is pretty much always 4k.  It doesn't know or care
about the underlying disk logical sector size.

If this is the only partition in the LV, then I think the best thing for
you to do is to dd only the partition and don't bother with a partition
table on the new machine.  Otherwise, you might need to manually create
the new partition table, then dd each partition individually.

For that matter, using dd wastes a lot of time and bandwidth copying all
of the unused space in the filesystem.  I'd suggest using e2image
instead.  It's smart enough to skip the unused space.

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Ilia Zykov]

Hello.

Sorry, I could be wrong, but I was encountered this problem a long time ago.
You cannot transfer ext4 from a device with a phys sector 512 bite to a 
device with phys 4k sector device.
As far as I remember, ext4 uses this size to perform atomic operations.
Because on the new disk it is not possible to perform an atomic 
operation with data of 512 bytes,
then it is impossible to transfer such a FS.
See ENVIROMENTS for mkfs.ext4: "MKE2FS_DEVICE_SECTSIZE", 
"MKE2FS_DEVICE_PHYS_SECTSIZE".

In datasheet for you model tells:
* 512e models can be converted to 4Kn format and
vice versa.
I don't understand what is it mean, but maybe you can try convert to 
512e if possible.


----

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Andy Smith]

Hi,

On Tue, Jan 16, 2024 at 01:24:29PM -0500, Phillip Susi wrote:
> Andy Smith <andy@strugglers.net> writes:
> 
> > The LV there when examined in a partitioning tool such as "fdisk"
> > now thinks it has a 3.2TiB partition and it is not usable.
> > Correcting the partition sector numbers allows for use of, for
> > example, "kpartx", to expose the partition as a loop device but the
> > ext4 driver and fsck.ext4 remain unable to detect a superblock.
> 
> You mean you put a partition table inside of the logical volume?  Why?

It's a disk image for a virtual machine that belongs to someone else
and I don't have a say in how they choose to lay out their data
inside of it. If they want to take what appears to them to be a
plain block device and put a partition table on it, or write their
own invented fs directly on it, in this case I have no latitude to
prohibit that.

It hasn't been an issue before.

> > I have confirmed with sha256sum that the content of the
> > image/partition remains the same on source and destination.
> 
> If you have already modified the partition table, then how could it
> still have the same sha256sum?

I did a sha256sum of the image before I tampered with it and it
matched.

I then recreated the partition table, which allowed me to use
"kpartx" to expose the first partition as a loop device. A sha256sum
of this first partition matches a sha256sum of the first partition
on the source disk image.

> > So, clearly the issue is the 512e sector size on source vs 4Kn on
> > destination. Is there any way to work around this in LVM? My issue
> > is that I would like to be able to move images of disks/filesystems
> > around at the block level without mounting/creating filesystem and
> > transferring with an fs-level application.
> 
> LVM doesn't really know or care about it.

I agree, but I was wondering if it would allow me some way around
the issue, and it's what I have as "my" top layer anyway.

> No, it wouldn't be a problem without the partition table.  ext4 uses its
> own block size, which is pretty much always 4k.  It doesn't know or care
> about the underlying disk logical sector size.

I've found quite a few people having similar problems to me so I'm
not sure about this, but I haven't had chance to test it yet. I
will try it out before I explore hdparm.

> For that matter, using dd wastes a lot of time and bandwidth copying all
> of the unused space in the filesystem.  I'd suggest using e2image
> instead.  It's smart enough to skip the unused space.

Since I don't generally know what is on the disk image as mentioned,
I can't really do this. Empty space isn't too much of an issue since
I generally actually use something that does a block-based sync
without copying matching chunks, so an empty source chunk will match
with an empty destination chunk and be skipped. But I re-did it with
plain dd just to be sure it wasn't a tooling issue.

This is getting a bit off track though, as the issue appears to be
the 512e vs 4Kn nature of the underlying drives, which I can't
sidestep by using fs-level tools.

Thanks,
Andy

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Andy Smith]

Hi,

On Tue, Jan 16, 2024 at 10:30:36PM +0300, Ilia Zykov wrote:
> Sorry, I could be wrong, but I was encountered this problem a long time ago.
> You cannot transfer ext4 from a device with a phys sector 512 bite to a
> device with phys 4k sector device.

I have read several other accounts which agree with this. I will
test just directly putting an ext4 fs on an LV that's backed by 512e
drives and then copying an image of that to an LV that is backed by
4Kn drives, just to see what happens, though.

> In datasheet for you model tells:
> * 512e models can be converted to 4Kn format and
> vice versa.
> I don't understand what is it mean, but maybe you can try convert to 512e if
> possible.

I think it means that using hdparm to set its sector size to 512 is
possible, though that would invalidate all the data currently on
the drive (which is fine in this case).

There would of course also be vendor tools to change the sector
size, but I think hdparm should work if the data sheet says the
procedure is possible.

I'll let you know!

Thanks,
Andy

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Andy Smith]

Hi Phillip,

On Tue, Jan 16, 2024 at 08:13:15PM +0000, Andy Smith wrote:
> On Tue, Jan 16, 2024 at 01:24:29PM -0500, Phillip Susi wrote:
> > No, it wouldn't be a problem without the partition table.  ext4 uses its
> > own block size, which is pretty much always 4k.  It doesn't know or care
> > about the underlying disk logical sector size.
> 
> I've found quite a few people having similar problems to me so I'm
> not sure about this, but I haven't had chance to test it yet. I
> will try it out before I explore hdparm.

I've tested ext4 directly on the LV with no partition table now and
you're correct - no issues there! Apologies for doubting you - so
many different vague accounts of issues out there.

So, the actual problem here is something about the MBR partition
table. Off-topic now for this list but I wonder if there is a safe
and reliable way to modify such a partition table after sync to
allow this to work…

Thanks,
Andy

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Zdenek Kabelac]

Dne 16. 01. 24 v 21:17 Andy Smith napsal(a):
> Hi,
> 
> On Tue, Jan 16, 2024 at 10:30:36PM +0300, Ilia Zykov wrote:
>> Sorry, I could be wrong, but I was encountered this problem a long time ago.
>> You cannot transfer ext4 from a device with a phys sector 512 bite to a
>> device with phys 4k sector device.
> 
> I have read several other accounts which agree with this. I will
> test just directly putting an ext4 fs on an LV that's backed by 512e
> drives and then copying an image of that to an LV that is backed by
> 4Kn drives, just to see what happens, though.
> 
>> In datasheet for you model tells:
>> * 512e models can be converted to 4Kn format and
>> vice versa.
>> I don't understand what is it mean, but maybe you can try convert to 512e if
>> possible.
> 
> I think it means that using hdparm to set its sector size to 512 is
> possible, though that would invalidate all the data currently on
> the drive (which is fine in this case).
> 
> There would of course also be vendor tools to change the sector
> size, but I think hdparm should work if the data sheet says the
> procedure is possible.
> 
> I'll let you know!
> 

Some words from lvm2 perspective.

Lvm tool as such doesn't really have any problems with mixing 4K and 512b 
disks within a single VG - however there are number of issues that will rise 
when an LV starts to be 'mixed' and using such 2 different devices together. 
This is confusing number of filesystem (XFS is one of those), where it 
suddenly starts to fall apart - as devices have different level of write 
atomicity - so it may severely corrupt your filesystem.

So in lvm2 we rather decided to disable the mixed usasge - if the user is 
skilled enough to deal with troubles, such protection can be disabled (see 
allow_mixed_block_size in lvm.conf).

Regards

Zdenek

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Andy Smith]

Hi Zdenek,

On Wed, Jan 17, 2024 at 11:36:24AM +0100, Zdenek Kabelac wrote:
> Lvm tool as such doesn't really have any problems with mixing 4K and 512b
> disks within a single VG

That's interesting, though in my case I have done a block-based
copy of an LV from a system where all PVs are 512b to one where all
PVs are 4K.

So out of interest, what does LVM do when one PV within same VG says
512b and another says 4K - just pass on 512b for every LV I assume,
unless allow_mixed_block_size is set as you mention?

Thanks,
Andy

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Zdenek Kabelac]

Dne 17. 01. 24 v 12:21 Andy Smith napsal(a):
> Hi Zdenek,
> 
> On Wed, Jan 17, 2024 at 11:36:24AM +0100, Zdenek Kabelac wrote:
>> Lvm tool as such doesn't really have any problems with mixing 4K and 512b
>> disks within a single VG
> 
> That's interesting, though in my case I have done a block-based
> copy of an LV from a system where all PVs are 512b to one where all
> PVs are 4K.
> 
> So out of interest, what does LVM do when one PV within same VG says
> 512b and another says 4K - just pass on 512b for every LV I assume,
> unless allow_mixed_block_size is set as you mention?

lvm2 does nothing with disks.
How they are configured, that way they work.

There is nothing that would try to make 4K disks to be like 512b.

Although I believe there were some DM targets for such emulation built, but 
they are not upstream.

So if user had an old VG with mixed blocks created before protection was added 
to lvm2 -- they will continue to work with all the 'risks' - and lvm2 will 
show up some warnings here and there.

With new disks users are not allowed (by default) to mix them together with 
one VG.

Regards

Zdenek

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Roger Heflin]

On Wed, Jan 17, 2024 at 1:22 AM Andy Smith <andy@strugglers.net> wrote:
>
> Hi Phillip,
>
> On Tue, Jan 16, 2024 at 08:13:15PM +0000, Andy Smith wrote:
> > On Tue, Jan 16, 2024 at 01:24:29PM -0500, Phillip Susi wrote:
> > > No, it wouldn't be a problem without the partition table.  ext4 uses its
> > > own block size, which is pretty much always 4k.  It doesn't know or care
> > > about the underlying disk logical sector size.
> >
> > I've found quite a few people having similar problems to me so I'm
> > not sure about this, but I haven't had chance to test it yet. I
> > will try it out before I explore hdparm.
>
> I've tested ext4 directly on the LV with no partition table now and
> you're correct - no issues there! Apologies for doubting you - so
> many different vague accounts of issues out there.
>
> So, the actual problem here is something about the MBR partition
> table. Off-topic now for this list but I wonder if there is a safe
> and reliable way to modify such a partition table after sync to
> allow this to work…
>
> Thanks,
> Andy
>


You might have the 4k alignment off.   IE the first 4k block on the
partition actually is across 2 partial 4k blocks on the disk.
What does fdisk -l look like on the 2 partition tables?

You would want to make sure that the first partition is divisible by 8
and that the partition in the partition is also divisible by 8.

Typically fdisk and other partitioning programs try to do that, but if
someone manually adjusted it and/or an old enough version was used
then it might be wrong.

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Phillip Susi]

Andy Smith <andy@strugglers.net> writes:

> It's a disk image for a virtual machine that belongs to someone else
> and I don't have a say in how they choose to lay out their data
> inside of it. If they want to take what appears to them to be a
> plain block device and put a partition table on it, or write their
> own invented fs directly on it, in this case I have no latitude to
> prohibit that.

If it's a VM then IIRC, it should still be presented to the VM as if it
were using 512 byte sectors, no matter what size the storage in the host
system uses.  Did you try just booting the VM or did you decide to use
kpartx to verify the image from the host first?

> Since I don't generally know what is on the disk image as mentioned,
> I can't really do this. Empty space isn't too much of an issue since
> I generally actually use something that does a block-based sync
> without copying matching chunks, so an empty source chunk will match
> with an empty destination chunk and be skipped. But I re-did it with
> plain dd just to be sure it wasn't a tooling issue.

It sounds like you are assuming that the empty space contains all zeros
and so will be de-duplicated with other space that contains all zeros.
Instead it often tends to contain old data that isn't going to match
something else, and so will be sent, even though it isn't needed.

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Phillip Susi]

Andy Smith <andy@strugglers.net> writes:

> So, the actual problem here is something about the MBR partition
> table. Off-topic now for this list but I wonder if there is a safe
> and reliable way to modify such a partition table after sync to
> allow this to work…

Like I said before, as long as the start and length of each partition is
an even multiple of 8, just divide them by 8 and recreate the partition
table using those values.

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Phillip Susi]

Ilia Zykov <mail@service4.ru> writes:

> Hello.
>
> Sorry, I could be wrong, but I was encountered this problem a long time ago.
> You cannot transfer ext4 from a device with a phys sector 512 bite to a 
> device with phys 4k sector device.
> As far as I remember, ext4 uses this size to perform atomic operations.
> Because on the new disk it is not possible to perform an atomic 
> operation with data of 512 bytes,
> then it is impossible to transfer such a FS.
> See ENVIROMENTS for mkfs.ext4: "MKE2FS_DEVICE_SECTSIZE", 
> "MKE2FS_DEVICE_PHYS_SECTSIZE".

Ext[234] does IO in units of blocks.  In the days of 80 MB hard disks,
it could use a block size of 1 or 2 KB.  These days it pretty much
always uses 4 KB.  I'm almost certain that those environment variables
are to override what the kernel detects for testing purposes, and the
only thing mkfs does with this information is to force a 4K block size (
when that is the sector size ), even if the fs is small enough that it
otherwise would use 1 or 2.

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Ilia Zykov]

On 17.01.2024 17:24, Phillip Susi wrote:
> Ilia Zykov <mail@service4.ru> writes:
> 
>> Hello.
>>
>> Sorry, I could be wrong, but I was encountered this problem a long time ago.
>> You cannot transfer ext4 from a device with a phys sector 512 bite to a
>> device with phys 4k sector device.
>> As far as I remember, ext4 uses this size to perform atomic operations.
>> Because on the new disk it is not possible to perform an atomic
>> operation with data of 512 bytes,
>> then it is impossible to transfer such a FS.
>> See ENVIROMENTS for mkfs.ext4: "MKE2FS_DEVICE_SECTSIZE",
>> "MKE2FS_DEVICE_PHYS_SECTSIZE".
> 
> Ext[234] does IO in units of blocks.  In the days of 80 MB hard disks,
> it could use a block size of 1 or 2 KB.  These days it pretty much
> always uses 4 KB.  I'm almost certain that those environment variables
> are to override what the kernel detects for testing purposes, and the
> only thing mkfs does with this information is to force a 4K block size (
> when that is the sector size ), even if the fs is small enough that it
> otherwise would use 1 or 2.
> 
> 

Yes, Andy confirmed that there are no problems with 4k blocks, without 
partition tables. And now I think so too )
Thank you.

----

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Andy Smith]

(Happy to take this off-topic thread off-list if it's bothering
anyone…)

Hi Phillip,

On Wed, Jan 17, 2024 at 09:10:15AM -0500, Phillip Susi wrote:
> Like I said before, as long as the start and length of each partition is
> an even multiple of 8, just divide them by 8 and recreate the partition
> table using those values.

So here's an attempt.

Commands prefixed with "srchost$" done on the source host; those with
"dsthost$" done on the destination host, and I've left bigger gaps
between when transitioning hosts to hopefully make that clearer.

srchost$ sudo lvcreate -L128m -n /dev/slowvg/4ksectortest
  Logical volume "4ksectortest" created.


$ sudo lvcreate -L128m -n /dev/slowvg/4ksectortest
  Logical volume "4ksectortest" created.


srchost$ sudo parted /dev/slowvg/4ksectortest
GNU Parted 3.2
Using /dev/dm-126
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit s
(parted) mklabel msdos
(parted) mkpart
Partition type?  primary/extended? p
File system type?  [ext2]? ext4
Start? 2048
End? 100%
(parted) p
Model: Linux device-mapper (linear) (dm)
Disk /dev/dm-126: 262144s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start  End      Size     Type     File system  Flags
 1      2048s  262143s  260096s  primary  ext4         lba
(parted) q
Information: You may need to update /etc/fstab.

srchost$ sudo kpartx -va /dev/slowvg/4ksectortest
add map slowvg-4ksectortest1 (253:127): 0 260096 linear 253:126 2048
srchost$ sudo mkfs.ext4 -L 4ksectortest /dev/mapper/slowvg-4ksectortest1
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 130048 1k blocks and 32512 inodes
Filesystem UUID: c39bf28a-c432-4a2a-9545-5da13b8574f2
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

srchost$ sudo sha256sum /dev/mapper/slowvg-4ksectortest1
7e61c1383b3b97db98f1b07022a2a01a926c2f6d98b324b96a83ba9db5eb40b9  /dev/mapper/slowvg-4ksectortest1
srchost$ sudo kpartx -vd /dev/slowvg/4ksectortest
del devmap : slowvg-4ksectortest1
srchost$ sudo fdisk -u -l /dev/slowvg/4ksectortest
Disk /dev/slowvg/4ksectortest: 128 MiB, 134217728 bytes, 262144 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa0cd277d

Device                    Boot Start    End Sectors  Size Id Type
/dev/slowvg/4ksectortest1       2048 262143  260096  127M 83 Linux
srchost$ sudo sh -c 'pv < /dev/slowvg/4ksectortest' | sudo ssh root@desthost 'cat > /dev/slowvg/4ksectortest'
 128MiB 0:00:01 [96.0MiB/s] [========================================>] 100%


dsthost$ sudo fdisk -u -l /dev/slowvg/4ksectortest
Disk /dev/slowvg/4ksectortest: 128 MiB, 134217728 bytes, 32768 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xa0cd277d

Device                    Boot Start    End Sectors  Size Id Type
/dev/slowvg/4ksectortest1       2048 262143  260096 1016M 83 Linux
dsthost$ sudo fdisk -u /dev/slowvg/4ksectortest

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): d
Selected partition 1
Partition 1 has been deleted.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (1-4, default 1):
First sector (256-32767, default 256): 256
Last sector, +/-sectors or +/-size{K,M,G,T,P} (256-32767, default 32767):

Created a new partition 1 of type 'Linux' and of size 127 MiB.
Partition #1 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: n

Command (m for help): p

Disk /dev/slowvg/4ksectortest: 128 MiB, 134217728 bytes, 32768 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xa0cd277d

Device                    Boot Start   End Sectors  Size Id Type
/dev/slowvg/4ksectortest1        256 32767   32512  127M 83 Linux
Command (m for help): w
The partition table has been altered.
Failed to remove partition 1 from system: No such device or address
Failed to add partition 1 to system: Invalid argument

The kernel still uses the old partitions. The new table will be used at the next reboot.

Syncing disks.

dsthost$ sudo fdisk -u -l /dev/slowvg/4ksectortest
Disk /dev/slowvg/4ksectortest: 128 MiB, 134217728 bytes, 32768 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xa0cd277d

Device                    Boot Start   End Sectors  Size Id Type
/dev/slowvg/4ksectortest1        256 32767   32512  127M 83 Linux
dsthost$ sudo kpartx -v -a /dev/slowvg/4ksectortest
add map slowvg-4ksectortest1 (253:52): 0 260096 linear 253:51 2048
dsthost$ sudo sha256sum /dev/mapper/slowvg-4ksectortest1
7e61c1383b3b97db98f1b07022a2a01a926c2f6d98b324b96a83ba9db5eb40b9  /dev/mapper/slowvg-4ksectortest1
dsthost$ sudo mount /dev/mapper/slowvg-4ksectortest1 /mnt
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mapper/slowvg-4ksectortest1, missing codepage or helper program, or other error.

So at the end here, despite msdos partition table looking correct,
sha256sum matching and kpartx appearing to work, dsthost can't mount
this filesystem. Did I miss a step or misunderstand?

I wondered if the part about "The kernel still uses the old
partitions" meant that the modified partition table was not being
used, so I tried with a "partprobe /dev/slowvg/4ksectortest" in
there. It did not help. Also, before modifying the partition table
kpartx would not work because it saw a first partition 8x larger
than the drive. After modifying the partition table, kpartx does
work, so from that I conclude that the system is seeing an updated
partition table on that disk image.

Another weird thing:

dsthost$ sudo lvcreate -L128m -n /dev/slowvg/anothertest
  Logical volume "anothertest" created.
dsthost$ sudo sh -c 'pv < /dev/mapper/slowvg-4ksectortest1 > /dev/slowvg/anothertest'
 127MiB 0:00:00 [ 233MiB/s] [========================================>] 100%
dsthost$ sudo mount -v /dev/slowvg/anothertest /mnt
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mapper/slowvg-anothertest, missing codepage or helper program, or other error.
dsthost$ sudo tune2fs -l /dev/slowvg/anothertest
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name:   4ksectortest
Last mounted on:          <not available>
Filesystem UUID:          c39bf28a-c432-4a2a-9545-5da13b8574f2
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              32512
Block count:              130048
Reserved block count:     6502
Free blocks:              120293
Free inodes:              32501
First block:              1
Block size:               1024
Fragment size:            1024
Group descriptor size:    64
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         2032
Inode blocks per group:   254
Flex block group size:    16
Filesystem created:       Sat Jan 20 04:03:42 2024
Last mount time:          n/a
Last write time:          Sat Jan 20 04:03:42 2024
Mount count:              0
Maximum mount count:      -1
Last checked:             Sat Jan 20 04:03:42 2024
Check interval:           0 (<none>)
Lifetime writes:          4441 kB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      2a074d90-0ade-4dfb-b8f2-3588ffa56cf6
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0xc7322510

Yet, putting an ext4 fs directly on an LV and shipping that image
across worked when I last tested it, at your suggestion that it
would work.

Thanks,
Andy

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Phillip Susi]

Andy Smith <andy@strugglers.net> writes:

> Created a new partition 1 of type 'Linux' and of size 127 MiB.
> Partition #1 contains a ext4 signature.
>
> Do you want to remove the signature? [Y]es/[N]o: n

fdisk sees the filesystem.  So far, everything looks perfect.

> Failed to remove partition 1 from system: No such device or address
> Failed to add partition 1 to system: Invalid argument

This is just because device-mapper doesn't support partitions, so fdisk
couldn't get the kernel to re-read the partition table.

> dsthost$ sudo mount /dev/mapper/slowvg-4ksectortest1 /mnt
> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mapper/slowvg-4ksectortest1, missing codepage or helper program, or other error.
>
> So at the end here, despite msdos partition table looking correct,
> sha256sum matching and kpartx appearing to work, dsthost can't mount
> this filesystem. Did I miss a step or misunderstand?

Try mount -t ext4.  If that doesn't work, see what e2fsck/tune2fs say.

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Andy Smith]

Hi Phillip,

On Sat, Jan 20, 2024 at 01:00:54PM -0500, Phillip Susi wrote:
> Andy Smith <andy@strugglers.net> writes:
> > dsthost$ sudo mount /dev/mapper/slowvg-4ksectortest1 /mnt
> > mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mapper/slowvg-4ksectortest1, missing codepage or helper program, or other error.
> >
> > So at the end here, despite msdos partition table looking correct,
> > sha256sum matching and kpartx appearing to work, dsthost can't mount
> > this filesystem. Did I miss a step or misunderstand?
> 
> Try mount -t ext4.  If that doesn't work, see what e2fsck/tune2fs say.

I've not needed that before and my instinct is that if it is needed,
something has gone wrong. But in any case it did not help:

$ sudo mount -t ext4 /dev/mapper/slowvg-4ksectortest1 /mnt
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mapper/slowvg-4ksectortest1, missing codepage or helper program, or other error.

I had posted tune2fs -l at the end of the email you're replying to;
it was from dsthost, after the move. It looks pretty normal. And as
I say, the sha256sum of the partition matches on both sides.

Here's the tuen2fs -l again:

$ sudo tune2fs -l /dev/mapper/slowvg-4ksectortest1
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name:   4ksectortest
Last mounted on:          <not available>
Filesystem UUID:          c39bf28a-c432-4a2a-9545-5da13b8574f2
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              32512
Block count:              130048
Reserved block count:     6502
Free blocks:              120293
Free inodes:              32501
First block:              1
Block size:               1024
Fragment size:            1024
Group descriptor size:    64
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         2032
Inode blocks per group:   254
Flex block group size:    16
Filesystem created:       Sat Jan 20 04:03:42 2024
Last mount time:          n/a
Last write time:          Sat Jan 20 04:03:42 2024
Mount count:              0
Maximum mount count:      -1
Last checked:             Sat Jan 20 04:03:42 2024
Check interval:           0 (<none>)
Lifetime writes:          4441 kB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      2a074d90-0ade-4dfb-b8f2-3588ffa56cf6
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0xc7322510

In a way this is all becoming academic, as it seems very
filesystem-specific. Given I generally can't poke about inside these
disk images, it looks like I will never come up with a robust
procedure for block-wise copying of these LVs.

Using hdparm to set a 512b sector may be my only way forward.

I am interested in personally knowing more though, so am happy to
try out any ideas anyone has for a while.

Thanks,
Andy

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Phillip Susi]

Andy Smith <andy@strugglers.net> writes:

>> Try mount -t ext4.  If that doesn't work, see what e2fsck/tune2fs say.
>
> I've not needed that before and my instinct is that if it is needed,
> something has gone wrong. But in any case it did not help:

If the fs is listed in /etc/fstab, then mount finds the fs type from
there.  Otherwise, you have to specify it.  At least, that used to be
the case.  This may have changed at some point and I still do it from
mussle memory.

> Here's the tuen2fs -l again:

If tune2fs recognizes it then it should be fine.  e2fsck -f should be
the final arbiter of whether there is anything wrong with the filesystem
though.  You might check dmesg as mount suggests for any errors from the
kernel explainging why it can't mount the filesystem.

> In a way this is all becoming academic, as it seems very
> filesystem-specific. Given I generally can't poke about inside these
> disk images, it looks like I will never come up with a robust
> procedure for block-wise copying of these LVs.

It is filesystem specific.  I believe FAT won't work right from this
transition because it does embed the sector size in its boot sector.
Due to its backward compatability, NTFS does this as well, though fixing
this to reflect the new size isn't hard.  Possibly HFS and iso9660 as
well, but most common linux filesystems, such as ext4 and btrfs, already
use 4k block sizes and so don't care about the sector size.

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Roger Heflin]

On Wed, Jan 24, 2024 at 10:19 AM Phillip Susi <phill@thesusis.net> wrote:
>
> Andy Smith <andy@strugglers.net> writes:
>
> >> Try mount -t ext4.  If that doesn't work, see what e2fsck/tune2fs say.
> >
> > I've not needed that before and my instinct is that if it is needed,
> > something has gone wrong. But in any case it did not help:
>
> If the fs is listed in /etc/fstab, then mount finds the fs type from
> there.  Otherwise, you have to specify it.  At least, that used to be
> the case.  This may have changed at some point and I still do it from
> mussle memory.
>
>

I have never had adding the -t <fstype> be necessary and/or work
better than mount without a -t even when the fs is not in /etc/fstab.
 That is going back at least 15+ years (note rhel4 did not need -t,
and it is old).

The mount program seems to find the type of fs (from the first few
bytes on the fs) and handle it, and if it does not mount, then adding
-t <correctfstype> had never made the mount work (usually because the
fs is seriously damaged in some manner, or completely gone).

So if "mount <device> some-location" does not work, then either the fs
is corrupted (for some reason), or the partition table changed making
it appear to be corrupted(data at wrong locations), or the fs module
is not loaded and/or supported in the kernel.

And I have debugged a lot of fs missing, and in the environment I am
in it is usually someone "fixed" the partition table start to be
different for some reason (or removed the partition table), or put a
partition table on top of the fs.

You might do a "dmsetup table" and specify what the lv names is and
maybe a ls -l /dev/mapper so we can see what the partitioned lv dm
mapping looks like.  and a fdisk -l against the partitioned lv, and a
ls -l /dev/<vgname>/

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Phillip Susi]

Roger Heflin <rogerheflin@gmail.com> writes:

> The mount program seems to find the type of fs (from the first few
> bytes on the fs) and handle it, and if it does not mount, then adding

I'm pretty sure that has to do with libblkid and if if that's been
around for 15 years already I feel old indeed.

> You might do a "dmsetup table" and specify what the lv names is and
> maybe a ls -l /dev/mapper so we can see what the partitioned lv dm
> mapping looks like.  and a fdisk -l against the partitioned lv, and a
> ls -l /dev/<vgname>/

He posted the dm table and partition table and they looked fine.  He
also showed tune2fs showing the superblock info like nothing was wrong,
so at least the superblock appears to be in the right place.

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Glenn Washburn]

On Mon, 15 Jan 2024 07:30:51 +0000
Andy Smith <andy@strugglers.net> wrote:

> Hi,
> 
> On machine 'A' I have a pair of:
> 
> Device Model:     Samsung SSD 870 EVO 4TB
> Sector Size:      512 bytes logical/physical
> 
> on top of this is an mdadm RAID-1 and that is an LVM PV.
> 
> One of the LVs has been partitioned with an MBR and a single
> partition spanning the whole of the 400GiB LV.
> 
> I took a dd of this LV and transferred it to an identically-sized
> LV on machine 'B' which has a pair of:
> 
> Device Model:     HGST HUS726T6TALN6L4
> Sector Size:      4096 bytes logical/physical
> 
> The LV there when examined in a partitioning tool such as "fdisk"
> now thinks it has a 3.2TiB partition and it is not usable.
> Correcting the partition sector numbers allows for use of, for
> example, "kpartx", to expose the partition as a loop device but the
> ext4 driver and fsck.ext4 remain unable to detect a superblock.
> 
> I have confirmed with sha256sum that the content of the
> image/partition remains the same on source and destination.
> 
> So, clearly the issue is the 512e sector size on source vs 4Kn on
> destination. Is there any way to work around this in LVM? My issue
> is that I would like to be able to move images of disks/filesystems
> around at the block level without mounting/creating filesystem and
> transferring with an fs-level application.
> 
> If not, then possibly I can use hdparm to set the 4Kn drives to 512,
> which will obviously involve destroying their contents, but that is
> fine at this stage.
> 
> I don't think the presence of a partition (as opposed to an ext4
> filesystem directly upon the LV) is relevant; I think the same
> issues would occur with a direct filesystem. I mention it only for
> completeness. Also, I realise that the problems would also happen
> without LVM. I just wonder if there is any workaround at the LVM
> layer, since that is already used here.

I've had this issue before and there is a very simple solution. It does
not work at the LVM layer though, but I suspect what you really care
about is having it work at the software, as opposed to hardware or
firmware layer.

Since the software that created the image did so assuming a 512b sector
size, create a block device that has that sector size. The trick is to
use loopdev to create a layer that does the translation from 512b to 4k
sector size. See the "--sector-size" argument to losetup.

Glenn

Re: Any way in LVM to deal with 512e vs 4Kn physical devices?

[Demi Marie Obenour]

[-- Attachment #1: Type: text/plain, Size: 3137 bytes --]

On Thu, Jan 25, 2024 at 07:21:55PM -0600, Glenn Washburn wrote:
> On Mon, 15 Jan 2024 07:30:51 +0000
> Andy Smith <andy@strugglers.net> wrote:
> 
> > Hi,
> > 
> > On machine 'A' I have a pair of:
> > 
> > Device Model:     Samsung SSD 870 EVO 4TB
> > Sector Size:      512 bytes logical/physical
> > 
> > on top of this is an mdadm RAID-1 and that is an LVM PV.
> > 
> > One of the LVs has been partitioned with an MBR and a single
> > partition spanning the whole of the 400GiB LV.
> > 
> > I took a dd of this LV and transferred it to an identically-sized
> > LV on machine 'B' which has a pair of:
> > 
> > Device Model:     HGST HUS726T6TALN6L4
> > Sector Size:      4096 bytes logical/physical
> > 
> > The LV there when examined in a partitioning tool such as "fdisk"
> > now thinks it has a 3.2TiB partition and it is not usable.
> > Correcting the partition sector numbers allows for use of, for
> > example, "kpartx", to expose the partition as a loop device but the
> > ext4 driver and fsck.ext4 remain unable to detect a superblock.
> > 
> > I have confirmed with sha256sum that the content of the
> > image/partition remains the same on source and destination.
> > 
> > So, clearly the issue is the 512e sector size on source vs 4Kn on
> > destination. Is there any way to work around this in LVM? My issue
> > is that I would like to be able to move images of disks/filesystems
> > around at the block level without mounting/creating filesystem and
> > transferring with an fs-level application.
> > 
> > If not, then possibly I can use hdparm to set the 4Kn drives to 512,
> > which will obviously involve destroying their contents, but that is
> > fine at this stage.
> > 
> > I don't think the presence of a partition (as opposed to an ext4
> > filesystem directly upon the LV) is relevant; I think the same
> > issues would occur with a direct filesystem. I mention it only for
> > completeness. Also, I realise that the problems would also happen
> > without LVM. I just wonder if there is any workaround at the LVM
> > layer, since that is already used here.
> 
> I've had this issue before and there is a very simple solution. It does
> not work at the LVM layer though, but I suspect what you really care
> about is having it work at the software, as opposed to hardware or
> firmware layer.
> 
> Since the software that created the image did so assuming a 512b sector
> size, create a block device that has that sector size. The trick is to
> use loopdev to create a layer that does the translation from 512b to 4k
> sector size. See the "--sector-size" argument to losetup.

The atomicity guarantees of devices with different sector sizes are
different, so this is lying to the guest and could cause data corruption
in the event of a power failure.  The only “clean” way to do this is
with something that supports atomic writes with a granularity that is
different than what the hardware does.  ZFS zvols might be able to do
that, since they are copy-on-write internally.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]