Re: [linux-lvm] "write failed.. No space left", "Failed to write VG", and "Failed to write a MDA"

[David Teigland <teigland@redhat.com>]

On Sun, Jun 03, 2018 at 07:47:35PM -0700, Jim Haddad wrote:
> Writing past the end of the disk seems to be fixed in git master.

> Hoping I understood the situation well enough that it wouldn't cause
> harm, using 2.02.177, I ran:

You'll notice some ongoing changes with releases and branches.  I'd
suggest using 2.02.176 and 2.02.178 (skip 2.02.177).  If you want to use a
git branch directly, you may want to look at 2018-06-01-stable since the
master branch may be unstable for a while.

As you suggest, the changes in 2.02.177 related to rounding up write sizes
seems to be your problem with writing beyond the end of the device:

> #format_text/format-text.c:799
  Writing lvm metadata to /dev/sdh3 at 106496 len 20934
  (rounded to 24576) of 20934 aligned to 4096

> #device/dev-io.c:96
  Write /dev/sdh3:   24576 bytes (sync) at 106496

> #format_text/format-text.c:799
  Writing lvm metadata to /dev/sdh3 at 4993488961536 len 20934
  (rounded to 24576) of 20934 aligned to 4096

> #device/dev-io.c:96
  Write /dev/sdh3:   24576 bytes (sync) at 4993488961536

> #device/dev-io.c:129
  /dev/sdh3: write failed after 24064 of 24576 at 4993488961536:
  No space left on device


> With git master, I ran the same command.  It no longer says exactly
> how much and where it's writing, just the header address. 

You should see more writing debug information than you included, like
this...

The first writes will occur in the metadata areas which will be the
closest to the end of the disk, and where you had the errors above:

format-text.c:331 Reading mda header sector from /dev/sdb at 4096
format-text.c:678 Writing metadata for VG foo to /dev/sdb at 7168 len 1525 (wrap 0)
format-text.c:331 Reading mda header sector from /dev/sdb at 999665172480
format-text.c:678 Writing metadata for VG foo to /dev/sdb at 999665175552 len 1525 (wrap 0)
format-text.c:331 Reading mda header sector from /dev/sdg at 4096
format-text.c:678 Writing metadata for VG foo to /dev/sdg at 7168 len 1525 (wrap 0)
format-text.c:331 Reading mda header sector from /dev/sdg at 999665172480
format-text.c:678 Writing metadata for VG foo to /dev/sdg at 999665175552 len 1525 (wrap 0)

Then the subsequent writes (to precommit/commit the changes) will occur to
the metadata headers which are just a 512 byte sector, not as close to the
end:

format-text.c:331 Reading mda header sector from /dev/sdb at 4096
format-text.c:790 Pre-Committing foo metadata (3) to /dev/sdb header at 4096
format-text.c:331 Reading mda header sector from /dev/sdb at 999665172480
format-text.c:790 Pre-Committing foo metadata (3) to /dev/sdb header at 999665172480
format-text.c:331 Reading mda header sector from /dev/sdg at 4096
format-text.c:790 Pre-Committing foo metadata (3) to /dev/sdg header at 4096
format-text.c:331 Reading mda header sector from /dev/sdg at 999665172480
format-text.c:790 Pre-Committing foo metadata (3) to /dev/sdg header at 999665172480

format-text.c:331 Reading mda header sector from /dev/sdb at 4096
format-text.c:790 Committing foo metadata (3) to /dev/sdb header at 4096
format-text.c:331 Reading mda header sector from /dev/sdb at 999665172480
format-text.c:790 Committing foo metadata (3) to /dev/sdb header at 999665172480
format-text.c:331 Reading mda header sector from /dev/sdg at 4096
format-text.c:790 Committing foo metadata (3) to /dev/sdg header at 4096
format-text.c:331 Reading mda header sector from /dev/sdg at 999665172480
format-text.c:790 Committing foo metadata (3) to /dev/sdg header at 999665172480


> But, it doesn't give an error, so I'm hoping it's properly handling the
> situation again:

Most of the metadata writing changes from 2.02.177 should not be in 178.