From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx1.redhat.com (ext-mx09.extmail.prod.ext.phx2.redhat.com [10.5.110.38]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 166AC5C21F for ; Wed, 13 Feb 2019 21:41:52 +0000 (UTC) Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5BB9DE6A9E for ; Wed, 13 Feb 2019 21:41:52 +0000 (UTC) Received: by mail-vs1-f71.google.com with SMTP id c1so997274vsq.23 for ; Wed, 13 Feb 2019 13:41:52 -0800 (PST) MIME-Version: 1.0 References: <20190204162527.GA2896@redhat.com> <2837066.rp6GCmz5LT@localhost.localdomain> <20190213203958.GA9718@redhat.com> In-Reply-To: <20190213203958.GA9718@redhat.com> From: Nir Soffer Date: Wed, 13 Feb 2019 23:41:39 +0200 Message-ID: Content-Type: multipart/alternative; boundary="000000000000d602630581cd685f" Subject: Re: [linux-lvm] Mixing devices with different logical or physical block size in oVirt LVM based storage Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: To: Mike Snitzer Cc: Denis Chaplygin , Vojtech Juranek , David Teigland , linux-lvm@redhat.com --000000000000d602630581cd685f Content-Type: text/plain; charset="UTF-8" On Wed, Feb 13, 2019 at 10:40 PM Mike Snitzer wrote: > On Wed, Feb 13 2019 at 4:14am -0500, > Vojtech Juranek wrote: > > > Hi Mike, > > > > > > > > Nir Soffer wrote: > > > > We working on enabling 4k block size in oVirt block storage > domain, > > > > built > > > > using VG > > > > on multipath devices on shared storage. > > > > > > > > We have incomplete support for 4k, added in 2011, for this bug: > > > > [1]https://bugzilla.redhat.com/732980 > > > > > > > > When creating or extending a VG, we check that all PVs are using > same > > > > logical and > > > > phyisical block size, and we store both logical and physical > block size > > > > in > > > > the VG tags. > > > > We get the block sizes from > > > > /sys/block/dm-X/queue/{logical,physical}_block_size. > > > > We also enforce that device physical block size is not smaller > than > > > > logical block size, > > > > This check was added in this patch, trying to enable block size > != 512. > > > > There is no > > > > explanation in the patch or in the review comments why we need to > > > > validate > > > > this. > > > > > > > > [2] > https://github.com/oVirt/vdsm/commit/7e79153705891a91a06eb31cd642fb2 > > > > 09d10ff86 When we start to use a VG, we validate that all the > devices > > > > are using the stored logical > > > > and physical block size. > > > > In vdsm itself, we use the logical block size to manage vdsm > metadata, > > > > assuming that writing > > > > and reading one block of logical block size bytes is atomic, and > we can > > > > read and write > > > > different blocks from different hosts at the same time. > > > > The relevant code validating PV block sizes is here: > > > > > > > > [3] > https://github.com/oVirt/vdsm/blob/8b043e402f41d8a82b9f832be5f582b85 > > > > 20b38bc/lib/vdsm/storage/lvm.py#L1110 Reading the comments in bug > > > > 732980, I don't see anything about physical block size. It looks > > > > like this is unnecessary check, and we should check only the > logical > > > > block > > > > size. > > > > Regarding mixing devices with different logical block size, > according > > > > to > > > > > > > > [4]https://bugzilla.redhat.com/show_bug.cgi?id=732980#c8 > > > > > > > > We should not extend an LV over devices with different block > size, as > > > > this > > > > will change the device > > > > logical block size (e.g change from 512 to 4k), and the change may > > > > break > > > > the upper layer that > > > > already use the device and assume the previous logical block size. > > > > > > This idea that 4K writes to a 512b physical drive aren't going to be > > > atomic, and that that is going to be the basis for some upper level > > > failure is handwaving and overly paranoid TBH. > > > > > > > Based on this, I think we are ok with limiting VG to devices with > same > > > > logical block size, so any > > > > LV can be extended to any device. > > > > I think this code should change to: > > > > 1. When creating a VG, check that all PVs use the same logical > block > > > > size > > > > 2. Store the logical block size in the VG tag > > > > 3. When extending the VG, check that the new PVs use the same > logical > > > > block size > > > > 4. When starting to use a VG, check that stored logical block size > > > > matches > > > > PVs logical block size > > > > What do you think? > > > > > > I think you shouldn't care. Or please show me a case where all this > > > concern matters. > > > > I'm sorry, but I'm still quite confused what needs to be checked and > what not. > > > > In [1] you wrote > > > > "So the appropriate VDSM constraint is to not allow a larger > > logical_block_size device (4K) to be added to a VG that has only ever > > contained small logical_block_size (512b) devices." > > > > and > > > > "If an LV is already in use then the admin needs to avoid extending the > LV in > > a way that upper layers may get upset with." > > > > and here that we shouldn't care. Could you be please more specific what > one > > needs to check (regarding block sizes) when creating or extending VG and > start > > using it? > > > > Thanks > > Vojta > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=732980 > > Ha, only going back 8 years in the archive for that BZ! > Thanks for looking at this. I'd need to revisit all the details of what VDSM/oVirt are so concerned > about relative to just _always_ using 4K for the sanlock volumes. > For sanlock volumes we don't care, we trust David to get this right :-) The issue is vdsm metadata. My contention is the constraint likely wasn't ever _really_ needed. But > maybe it was.. again, I'll look back at the BZ in more detail to see > what I'm missing. > > Concerns about 4K issued to 512b physical devices _not_ being atomic > (could have 5 of the 8 512b written, so old 3 bytes could cause > issues). IIRC I shared those concerns with Martin Petersen before > (Martin is an upstream Linux SCSI maintainer) and he felt the atomicity > concerns were overstated. Thinking now, it was possibly for devices > that advertise 4K physical and 512b logical. Whereas issuing 4K to a > 512b/512b device could easily not be atomic for that 4K IO. > > I can revisit this with Martin. Also, I'm happy to adjust my > understanding based on further anecdotal real-world evidence that > issuing 4K IOs to a 512b device and expecting any 4K IO operation to be > atomic is _wrong_. > I want more info why we care about atomic write to 512 bytes blocks. One use case is managing vdsm volumes metadata. In current version we keep one 512 bytes block for every vdsm volume. We keep that on a special "metadata" lv. The number of the block is kept in the lv tags. Here is an example: # lvs -o lv_name,tags fb5cab8c-08ba-4781-9532-ccc78ddb21ec LV LV Tags 3ad2d445-6505-4442-915b-ab3a6a2fd55b IU_c4622768-4173-403a-811c-096376d28c26,MD_7,PU_00000000-0000-0000-0000-000000000000 416573b6-caf0-49b8-ba36-8b64336d742f IU_1f05ff49-e97b-4a13-a973-59260dd13b87,MD_8,PU_00000000-0000-0000-0000-000000000000 ... metadata The metadata of the lv 3ad2d445-6505-4442-915b-ab3a6a2fd55b is stored at offset 7 * 512 (MD_7) in the metadata lv. # dd if=/dev/fb5cab8c-08ba-4781-9532-ccc78ddb21ec/metadata bs=512 count=1 skip=7 DOMAIN=fb5cab8c-08ba-4781-9532-ccc78ddb21ec CTIME=1542309274 FORMAT=RAW DISKTYPE=ISOF LEGALITY=LEGAL SIZE=6291456 VOLTYPE=LEAF DESCRIPTION={"DiskAlias":"Fedora-Server-dvd-x86_64-29-1.2.iso","DiskDescription":"Uploaded disk"} IMAGE=c4622768-4173-403a-811c-096376d28c26 PUUID=00000000-0000-0000-0000-000000000000 MTIME=0 POOL_UUID= TYPE=PREALLOCATED GEN=0 EOF 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00085428 s, 599 kB/s We use sanlock to synchronize access to the metadata lv, but this lv is active on many hosts at the same time, and different hosts are reading and writing volume metadata at the same time. We may have 2 storage jobs reading and writing the blocks at offset 7 and 8. If the writes are not atomic, one host can overwrite other host write. To support 4k drives, we are modifying this format to keep 8k per volume so we can have the same format regardless of the underlying block size, reading and writing 512 bytes blocks or 4k blocks. However we still have to support the old format using 512 bytes blocks per volume. We can simplify the code to always read and write 4k blocks, but I believe that we may have short read/write, and handling that may be more complicated then writing always one block. The underlying storage that we try to support is anything that can be shared using FC/SAS/iSCSI. We want to be compatible with the most stupid storage. Nir --000000000000d602630581cd685f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Wed, Feb 13, 2019 at 10:40 PM Mike Snitzer <= ;snitzer@redhat.com> wrote:
On Wed, Feb 13 2019 at=C2=A0 4:14am -0500,
Vojtech Juranek <vjuranek@redhat.com> wrote:

> Hi Mike,
>
> >
> > Nir Soffer <nsoffer@redhat.com> wrote:
> > >=C2=A0 =C2=A0 We working on enabling 4k block size in oVirt b= lock storage domain,
> > >=C2=A0 =C2=A0 built
> > >=C2=A0 =C2=A0 using VG
> > >=C2=A0 =C2=A0 on multipath devices on shared storage.
> > >=C2=A0 =C2=A0
> > >=C2=A0 =C2=A0 We have incomplete support for 4k, added in 201= 1, for this bug:
> > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 [1]https://bugzilla.redh= at.com/732980
> > >=C2=A0 =C2=A0
> > >=C2=A0 =C2=A0 When creating or extending a VG, we check that = all PVs are using same
> > >=C2=A0 =C2=A0 logical and
> > >=C2=A0 =C2=A0 phyisical block size, and we store both logical= and physical block size
> > >=C2=A0 =C2=A0 in
> > >=C2=A0 =C2=A0 the VG tags.
> > >=C2=A0 =C2=A0 We get the block sizes from
> > >=C2=A0 =C2=A0 /sys/block/dm-X/queue/{logical,physical}_block_= size.
> > >=C2=A0 =C2=A0 We also enforce that device physical block size= is not smaller than
> > >=C2=A0 =C2=A0 logical block size,
> > >=C2=A0 =C2=A0 This check was added in this patch, trying to e= nable block size !=3D 512.
> > >=C2=A0 =C2=A0 There is no
> > >=C2=A0 =C2=A0 explanation in the patch or in the review comme= nts why we need to
> > >=C2=A0 =C2=A0 validate
> > >=C2=A0 =C2=A0 this.
> > >=C2=A0 =C2=A0
> > >=C2=A0 =C2=A0 [2]h= ttps://github.com/oVirt/vdsm/commit/7e79153705891a91a06eb31cd642fb2
> > >=C2=A0 =C2=A0 09d10ff86 When we start to use a VG, we validat= e that all the devices
> > >=C2=A0 =C2=A0 are using the stored logical
> > >=C2=A0 =C2=A0 and physical block size.
> > >=C2=A0 =C2=A0 In vdsm itself, we use the logical block size t= o manage vdsm metadata,
> > >=C2=A0 =C2=A0 assuming that writing
> > >=C2=A0 =C2=A0 and reading one block of logical block size byt= es is atomic, and we can
> > >=C2=A0 =C2=A0 read and write
> > >=C2=A0 =C2=A0 different blocks from different hosts at the sa= me time.
> > >=C2=A0 =C2=A0 The relevant code validating PV block sizes is = here:
> > >=C2=A0 =C2=A0
> > >=C2=A0 =C2=A0 [3]h= ttps://github.com/oVirt/vdsm/blob/8b043e402f41d8a82b9f832be5f582b85
> > >=C2=A0 =C2=A0 20b38bc/lib/vdsm/storage/lvm.py#L1110 Reading t= he comments in bug
> > >=C2=A0 =C2=A0 732980, I don't see anything about physical= block size. It looks
> > >=C2=A0 =C2=A0 like this is unnecessary check, and we should c= heck only the logical
> > >=C2=A0 =C2=A0 block
> > >=C2=A0 =C2=A0 size.
> > >=C2=A0 =C2=A0 Regarding mixing devices with different logical= block size, according
> > >=C2=A0 =C2=A0 to
> > >=C2=A0 =C2=A0
> > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 [4]= https://bugzilla.redhat.com/show_bug.cgi?id=3D732980#c8
> > >=C2=A0 =C2=A0
> > >=C2=A0 =C2=A0 We should not extend an LV over devices with di= fferent block size, as
> > >=C2=A0 =C2=A0 this
> > >=C2=A0 =C2=A0 will change the device
> > >=C2=A0 =C2=A0 logical block size (e.g change from 512 to 4k),= and the change may
> > >=C2=A0 =C2=A0 break
> > >=C2=A0 =C2=A0 the upper layer that
> > >=C2=A0 =C2=A0 already use the device and assume the previous = logical block size.
> >
> > This idea that 4K writes to a 512b physical drive aren't goin= g to be
> > atomic, and that that is going to be the basis for some upper lev= el
> > failure is handwaving and overly paranoid TBH.
> >
> > >=C2=A0 =C2=A0 Based on this, I think we are ok with limiting = VG to devices with same
> > >=C2=A0 =C2=A0 logical block size, so any
> > >=C2=A0 =C2=A0 LV can be extended to any device.
> > >=C2=A0 =C2=A0 I think this code should change to:
> > >=C2=A0 =C2=A0 1. When creating a VG, check that all PVs use t= he same logical block
> > >=C2=A0 =C2=A0 size
> > >=C2=A0 =C2=A0 2. Store the logical block size in the VG tag > > >=C2=A0 =C2=A0 3. When extending the VG, check that the new PV= s use the same logical
> > >=C2=A0 =C2=A0 block size
> > >=C2=A0 =C2=A0 4. When starting to use a VG, check that stored= logical block size
> > >=C2=A0 =C2=A0 matches
> > >=C2=A0 =C2=A0 PVs logical block size
> > >=C2=A0 =C2=A0 What do you think?
> >
> > I think you shouldn't care.=C2=A0 Or please show me a case wh= ere all this
> > concern matters.
>
> I'm sorry, but I'm still quite confused what needs to be check= ed and what not.
>
> In [1] you wrote
>
> "So the appropriate VDSM constraint is to not allow a larger
> logical_block_size device (4K) to be added to a VG that has only ever =
> contained small logical_block_size (512b) devices."
>
> and
>
> "If an LV is already in use then the admin needs to avoid extendi= ng the LV in
> a way that upper layers may get upset with."
>
> and here that we shouldn't care. Could you be please more specific= what one
> needs to check (regarding block sizes) when creating or extending VG a= nd start
> using it?
>
> Thanks
> Vojta
>
> [1] https://bugzilla.redhat.com/show_bug.cg= i?id=3D732980

Ha, only going back 8 years in the archive for that BZ!

Thanks for looking at this.

I'd need to revisit all the details of what VDSM/oVirt are so concerned=
about relative to just _always_ using 4K for the sanlock volumes.

For sanlock volumes we don't care, we trust Dav= id to get this right :-)

The issue is vdsm metadata.

<= /div>
My contention is the constraint likely wasn't ever _really_ needed.=C2= =A0 But
maybe it was.. again, I'll look back at the BZ in more detail to see what I'm missing.

Concerns about 4K issued to 512b physical devices _not_ being atomic
(could have 5 of the 8 512b written, so old 3 bytes could cause
issues).=C2=A0 IIRC I shared those concerns with Martin Petersen before
(Martin is an upstream Linux SCSI maintainer) and he felt the atomicity
concerns were overstated.=C2=A0 Thinking now, it was possibly for devices that advertise 4K physical and 512b logical.=C2=A0 Whereas issuing 4K to a<= br> 512b/512b device could easily not be atomic for that 4K IO.

I can revisit this with Martin.=C2=A0 Also, I'm happy to adjust my
understanding based on further anecdotal real-world evidence that
issuing 4K IOs to a 512b device and expecting any 4K IO operation to be
atomic is _wrong_.

I want more info why we= care about atomic write to 512 bytes blocks.

One use case is mana= ging vdsm volumes metadata. In current version we keep
one 512 bytes bloc= k for every vdsm volume. We keep that on a special "metadata"
l= v. The number of the block is kept in the lv tags.

Here is an ex= ample:

# lvs -o lv_name,tags fb5cab8c= -08ba-4781-9532-ccc78ddb21ec
=C2=A0 LV=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0LV Tags=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0=C2=A0
=C2=A0 3ad2d445-6505-4442-915b-ab3a6a2fd55b IU_c4622768-4173-40= 3a-811c-096376d28c26,MD_7,PU_00000000-0000-0000-0000-000000000000=C2=A0
= =C2=A0 416573b6-caf0-49b8-ba36-8b64336d742f IU_1f05ff49-e97b-4a13-a973-5926= 0dd13b87,MD_8,PU_00000000-0000-0000-0000-000000000000=C2=A0
=C2=A0 ...
= =C2=A0 metadata=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0

The metadata of the lv 3ad2d445-65= 05-4442-915b-ab3a6a2fd55b is stored
at offset 7 * 512 (MD_7) in the metadata lv.

# dd if=3D/dev/fb5cab8c-08ba-4781-9532-ccc78ddb21ec/metadata bs=3D51= 2 count=3D1 skip=3D7
DOMAIN=3Dfb5c= ab8c-08ba-4781-9532-ccc78ddb21ec
C= TIME=3D1542309274
FORMAT=3DRAW
DISKTYPE=3DISOF
LEGALITY=3DLEGAL
SIZE=3D6291456
VOLTYPE=3DLEAF
DESCRIPTION=3D{"DiskAlias"= ;:"Fedora-Server-dvd-x86_64-29-1.2.iso","DiskDescription&quo= t;:"Uploaded disk"}
IMAG= E=3Dc4622768-4173-403a-811c-096376d28c26
PUUID=3D00000000-0000-0000-0000-000000000000
MTIME=3D0
POOL_U= UID=3D
TYPE=3DPREALLOCATED<= /div>
GEN=3D0
EOF
1+0 records in<= /div>
1+0 records out
512 bytes (512 B) copied, 0.00085428 s, 599 kB/s

We use sanlock to synchronize access= to the metadata lv, but this lv is active
on many hosts at the same time, and different hosts ar= e reading and writing
= volume metadata at the same time.

We ma= y have 2 storage jobs reading and writing the blocks at offset 7 and 8.
If the writes are not ato= mic, one host can overwrite other host write.

To support 4k drives, we are modifying this format to keep 8k per vo= lume so
we can have th= e same format regardless of the underlying block size, reading
and writing 512 bytes blocks or 4k= blocks. However we still have to support
the old format using 512 bytes blocks per volume.
=

We can simplify the code to always read and write 4k blocks, but = I believe that
we may have short read/write, and handling that may be mor= e complicated then
writing always one block.=C2=A0

The underlyin= g storage that we try to support is anything that can be shared using
=
FC/= SAS/iSCSI. We want to be compatible with the most stupid storage.

= Nir
--000000000000d602630581cd685f--