linux-lvm.redhat.com archive mirror
 help / color / mirror / Atom feed
From: Jim Haddad <jimhaddad46@gmail.com>
To: linux-lvm@redhat.com, teigland@redhat.com, agk@redhat.com,
	ejt@redhat.com
Subject: Re: [linux-lvm] "write failed.. No space left", "Failed to write VG", and "Failed to write a MDA"
Date: Sun, 3 Jun 2018 19:47:35 -0700	[thread overview]
Message-ID: <CAANaKqeKDaBcAHAh91ycm9SwB0gDD+cNHqsAVkW6y-RzNzgifg@mail.gmail.com> (raw)
In-Reply-To: <CAANaKqd98zjbv2Mh60aeny=UL8oMGadGcLUXgOURbQmnFxojEA@mail.gmail.com>

Writing past the end of the disk seems to be fixed in git master.

Looks like the culprit is lib/format_text/format-text.c::_vg_write_raw().

I think the past the end of the disk bug got added in the days before
2.02.177 was released by some of the Alasdair G Kergon commits to the
function regarding wrapping and rounding.  I am not 100% positive of
this.

I think this commit reverting those changes FIXES the bug:

commit 00f1b208a1bf44665ec97a791355b1fcf525a3a7
Author: Joe Thornber <ejt@redhat.com>
Date:   Fri Apr 20 10:43:50 2018 -0500

    [io paths] Unpick agk's aio stuff

It also might be fixed/helped by David Tiegland's commits regarding bcache.



Hoping I understood the situation well enough that it wouldn't cause
harm, using 2.02.177, I ran:

# lvcreate -ddddddvvvv -V200G lvm/disk3thin -n test3
...
#device/dev-io.c:654           Closed /dev/sdh3
#device/dev-io.c:599           Opened /dev/sdh3 RW O_DIRECT
#device/dev-io.c:168           /dev/sdh3: Block size is 512 bytes
#device/dev-io.c:179           /dev/sdh3: Physical block size is 4096 bytes
#device/dev-io.c:96            Read  /dev/sdh3:     512 bytes (sync)
at 4096 (for VG metadata header)
#device/dev-io.c:255           Widening request for 130 bytes at 81920
to 512 bytes at 81920 on /dev/sdh3 (for VG metadata content)
#device/dev-io.c:96            Read  /dev/sdh3:     512 bytes (sync)
at 81920 (for VG metadata content)
#format_text/format-text.c:799           Writing lvm metadata to
/dev/sdh3 at 106496 len 20934 (rounded to 24576) of 20934 aligned to
4096
#device/dev-io.c:96            Write /dev/sdh3:   24576 bytes (sync)
at 106496 (for VG metadata content)
#device/dev-io.c:96            Read  /dev/sdh3:     512 bytes (sync)
at 4993488781312 (for extra VG metadata header)
#device/dev-io.c:255           Widening request for 130 bytes at
4993488936960 to 512 bytes at 4993488936960 on /dev/sdh3 (for extra VG
metadata content)
#device/dev-io.c:96            Read  /dev/sdh3:     512 bytes (sync)
at 4993488936960 (for extra VG metadata content)
#format_text/format-text.c:799           Writing lvm metadata to
/dev/sdh3 at 4993488961536 len 20934 (rounded to 24576) of 20934
aligned to 4096
#device/dev-io.c:96            Write /dev/sdh3:   24576 bytes (sync)
at 4993488961536 (for extra VG metadata content)
#device/dev-io.c:129     /dev/sdh3: write failed after 24064 of 24576
at 4993488961536: No space left on device
#device/dev-io.c:288           <backtrace>
#format_text/format-text.c:806           <backtrace>
#metadata/metadata.c:3055          <backtrace>
#metadata/metadata.c:3064    Failed to write VG lvm.
#device/dev-io.c:96            Read  /dev/sdh3:     512 bytes (sync)
at 4096 (for VG metadata header)
#device/dev-io.c:255           Widening request for 130 bytes at 81920
to 512 bytes at 81920 on /dev/sdh3 (for VG metadata content)
#device/dev-io.c:96            Read  /dev/sdh3:     512 bytes (sync)
at 81920 (for VG metadata content)
#format_text/format-text.c:920           Wiping pre-committed lvm
metadata from /dev/sdh3 header at 4096
#device/dev-io.c:96            Write /dev/sdh3:     512 bytes (sync)
at 4096 (for VG metadata header)
#metadata/lv_manip.c:7802          <backtrace>
#metadata/lv_manip.c:8078          <backtrace>
#lvcreate.c:1652          <backtrace>
#toollib.c:1987          <backtrace>
...



With git master, I ran the same command.  It no longer says exactly
how much and where it's writing, just the header address.  But, it
doesn't give an error, so I'm hoping it's properly handling the
situation again:
...
#format_text/format-text.c:331           Reading mda header sector
from /dev/sdh3 at 4096
#format_text/format-text.c:790           Committing lvm metadata (550)
to /dev/sdh3 header at 4096
#format_text/format-text.c:331           Reading mda header sector
from /dev/sdh3 at 4993488781312
#format_text/format-text.c:790           Committing lvm metadata (550)
to /dev/sdh3 header at 4993488781312
#format_text/format-text.c:331           Reading mda header sector
from /dev/sdg3 at 4096
#format_text/format-text.c:790           Committing lvm metadata (550)
to /dev/sdg3 header at 4096
#format_text/format-text.c:331           Reading mda header sector
from /dev/sdg3 at 4993488781312
#format_text/format-text.c:790           Committing lvm metadata (550)
to /dev/sdg3 header at 4993488781312
#format_text/format-text.c:331           Reading mda header sector
from /dev/sdf3 at 4096
#format_text/format-text.c:790           Committing lvm metadata (550)
to /dev/sdf3 header at 4096
#format_text/format-text.c:331           Reading mda header sector
from /dev/sdf3 at 4993488781312
#format_text/format-text.c:790           Committing lvm metadata (550)
to /dev/sdf3 header at 4993488781312
#format_text/format-text.c:331           Reading mda header sector
from /dev/sde3 at 4096
#format_text/format-text.c:790           Committing lvm metadata (550)
to /dev/sde3 header at 4096
...



So, I'm thinking I can upgrade to git master, or at least 178-rc1, and
leave mda1 incredibly small.  While knowing that if my VG XML a bit
more than doubles beyond the 48,640 available in mda1 for it, I'd need
to run "vgconvert --pvmetadatacopies 1" or move the data off and back
on.

  reply	other threads:[~2018-06-04  2:47 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-06-03 20:13 [linux-lvm] "write failed.. No space left", "Failed to write VG", and "Failed to write a MDA" Inbox
2018-06-03 20:18 ` Inbox
2018-06-03 22:21 ` Inbox
2018-06-03 23:57   ` Inbox
2018-06-04  0:19     ` Inbox
2018-06-04  0:46       ` Jim Haddad
2018-06-04  2:47         ` Jim Haddad [this message]
2018-06-04 15:34           ` David Teigland
2018-06-04 16:35             ` David Teigland
2018-06-04 18:26             ` Jim Haddad
2018-06-04 18:52               ` David Teigland
2018-06-04  6:54     ` Patrick Mitchell

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAANaKqeKDaBcAHAh91ycm9SwB0gDD+cNHqsAVkW6y-RzNzgifg@mail.gmail.com \
    --to=jimhaddad46@gmail.com \
    --cc=agk@redhat.com \
    --cc=ejt@redhat.com \
    --cc=linux-lvm@redhat.com \
    --cc=teigland@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).