From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx1.redhat.com (ext-mx14.extmail.prod.ext.phx2.redhat.com [10.5.110.43]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 34704100191B for ; Sun, 3 Jun 2018 22:22:07 +0000 (UTC) Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0E6A630041D4 for ; Sun, 3 Jun 2018 22:21:56 +0000 (UTC) Received: by mail-wm0-f42.google.com with SMTP id p126-v6so3032781wmb.2 for ; Sun, 03 Jun 2018 15:21:55 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: From: Inbox Date: Sun, 3 Jun 2018 15:21:54 -0700 Message-ID: Content-Type: multipart/alternative; boundary="0000000000009780e4056dc43e1d" Subject: Re: [linux-lvm] "write failed.. No space left", "Failed to write VG", and "Failed to write a MDA" Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: To: linux-lvm@redhat.com --0000000000009780e4056dc43e1d Content-Type: text/plain; charset="UTF-8" Sorry to be mailing again, think this info helps though... On Sun, Jun 3, 2018 at 1:13 PM, Inbox wrote: ... > # lvcreate -V200G lvm/disk3thin -n test3 > ... > /dev/sdh3: write failed after 24064 of 24576 at 4993488961536: No space > left on device > ... > # lvremove lvm/test3 > /dev/sdh3: write failed after 24064 of 24576 at 4993488961536: No space > left on device > WARNING: Failed to write an MDA of VG lvm. > /dev/sdg3: write failed after 24064 of 24576 at 4993488961536: No space > left on device > WARNING: Failed to write an MDA of VG lvm. > /dev/sdf3: write failed after 24064 of 24576 at 4993488961536: No space > left on device > WARNING: Failed to write an MDA of VG lvm. > fdisk -l shows sdf3, sdg3, and sdh3, sdg3, are 4993488985600 bytes. After some reading, I'm guessing mda means metadata area. If LVM is trying to write mda2 at 4993488961536, there's only 24064 bytes left in the partition, which is exactly where it's saying the write is failing. I did use "--pvmetadatacopies 2" when running pvcreate. So, am I right that it's trying to write more than 24k of metadata at the end of the disk, but there's only 24k left at the end of the disk for a metadata copy? If so, where the other locations for metadata (both the main one, and the first copy) only 24k? Did I lose metadata in those areas? Did it overwrite what was after the metadata area? Where do I go from here? # pvdisplay --maps /dev/sdh3 --- Physical volume --- PV Name /dev/sdh3 VG Name lvm PV Size 4.54 TiB / not usable 2.19 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 1190540 Free PE 0 Allocated PE 1190540 PV UUID BK8suJ-dqiy-mdK4-IyUH-aeRR-TPUo-3Dj6JG --- Physical Segments --- Physical extent 0 to 127: Logical volume /dev/lvm/disk1thin_tmeta Logical extents 32 to 159 Physical extent 128 to 4095: Logical volume /dev/lvm/swap1 Logical extents 0 to 3967 Physical extent 4096 to 4127: Logical volume /dev/lvm/lvol0_pmspare Logical extents 0 to 31 Physical extent 4128 to 132127: Logical volume /dev/lvm/disk1thin_tdata Logical extents 0 to 127999 Physical extent 132128 to 132159: Logical volume /dev/lvm/disk1thin_tmeta Logical extents 0 to 31 Physical extent 132160 to 1190539: Logical volume /dev/lvm/disk1thin_tdata Logical extents 128000 to 1186379 Those physical extents translate to beginning/ending bytes (4MiB PE size) of: 0 532676608 536870912 17175674880 17179869184 17309892608 17314086912 554180804608 554184998912 554315022336 554319216640 4993482489856 Disk size (4993488961536) - last physical extent ending byte (4993482489856 + 4*1024*1024 assuming worst case scenario here, the x856 is the starting byte of the last extent) still leaves 2277376 bytes, way more than 24k. --0000000000009780e4056dc43e1d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Sorr= y to be mailing again, think this info helps though...



On Sun, Jun 3, 2018 at 1:13= PM, Inbox <jimhaddad46@gmail.com> wrote:
...
=
# lvcreate -V200G lvm/disk3thin -n test3
...
=C2=A0 /dev/sdh3: write failed after 24064 of 24= 576 at 4993488961536: No space left on device
= ...=C2=A0
# lvremove lvm/test3
=C2=A0 /dev/sdh3: write fa= iled after 24064 of 24576 at 4993488961536: No space left on device
=C2=A0 WARNING: Failed to write an MDA of VG lvm.
=C2=A0 /dev/= sdg3: write failed after 24064 of 24576 at 4993488961536: No space left on = device
=C2=A0 WARNING: Failed to write an MDA of VG lvm.
=C2=A0 /dev/sdf3: write failed after 24064 of 24576 at 4993488961536: No = space left on device
=C2=A0 WARNING: Failed to write an MDA of VG= lvm.

fdisk -l=C2=A0 shows sdf3= , sdg3, and sdh3, sdg3, are=C2=A04993488985600 bytes.

<= div>After some reading, I'm guessing mda means metadata area.

If LVM is trying to write mda2 at=C2=A04993488961536, there's only 24064 bytes left in the partition, wh= ich is exactly where it's saying the write is failing.

I did use "--pvmetada= tacopies 2" when running pvcreate.

So, am I r= ight that it's trying to write more than 24k of metadata at the end of = the disk, but there's only 24k left at the end of the disk for a metada= ta copy?

If so, where the other locations for meta= data (both the main one, and the first copy) only 24k?=C2=A0 Did I lose met= adata in those areas?=C2=A0 Did it overwrite what was after the metadata ar= ea?

Where do I go from here?

<= div>
# pvdisplay --maps /dev/sdh3
=C2=A0 --- Physical volume = ---
=C2=A0 PV Name=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0/dev/sdh3
=C2=A0 VG Name=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0lvm
=C2=A0 PV Size=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04.54 TiB / not usable 2.19 MiB
= =C2=A0 Allocatable=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0yes (but full)
=C2=A0 PE Size=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A04.00 MiB
=C2=A0 Total PE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 1190540
=C2=A0 Free PE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00
=C2=A0 Allocated PE=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 1190540
=C2=A0 PV UUID=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0BK8suJ-dqiy-mdK4-IyUH-aeRR-TPUo-3Dj6JG

=C2=A0 --- Physical Segments ---
=C2=A0 Physical = extent 0 to 127:
=C2=A0 =C2=A0 Logical volume=C2=A0 =C2=A0 =C2=A0= /dev/lvm/disk1thin_tmeta
=C2=A0 =C2=A0 Logical extents=C2=A0 =C2= =A0 =C2=A032 to 159
=C2=A0 Physical extent 128 to 4095:
=C2=A0 =C2=A0 Logical volume=C2=A0 =C2=A0 =C2=A0 /dev/lvm/swap1
= =C2=A0 =C2=A0 Logical extents=C2=A0 =C2=A0 =C2=A00 to 3967
=C2=A0= Physical extent 4096 to 4127:
=C2=A0 =C2=A0 Logical volume=C2=A0= =C2=A0 =C2=A0 /dev/lvm/lvol0_pmspare
=C2=A0 =C2=A0 Logical exten= ts=C2=A0 =C2=A0 =C2=A00 to 31
=C2=A0 Physical extent 4128 to 1321= 27:
=C2=A0 =C2=A0 Logical volume=C2=A0 =C2=A0 =C2=A0 /dev/lvm/dis= k1thin_tdata
=C2=A0 =C2=A0 Logical extents=C2=A0 =C2=A0 =C2=A00 t= o 127999
=C2=A0 Physical extent 132128 to 132159:
=C2= =A0 =C2=A0 Logical volume=C2=A0 =C2=A0 =C2=A0 /dev/lvm/disk1thin_tmeta
=C2=A0 =C2=A0 Logical extents=C2=A0 =C2=A0 =C2=A00 to 31
= =C2=A0 Physical extent 132160 to 1190539:
=C2=A0 =C2=A0 Logical v= olume=C2=A0 =C2=A0 =C2=A0 /dev/lvm/disk1thin_tdata
=C2=A0 =C2=A0 = Logical extents=C2=A0 =C2=A0 =C2=A0128000 to 1186379

Those physical extents translate to beginning/ending bytes (4MiB P= E size) of:

0= 532676608
536870912 17175674880
17179869184 = 17309892608
17314086912 55= 4180804608
554184998912 55= 4315022336
554319216640 49= 93482489856


Disk size (499348= 8961536) - last physical extent ending byte (4993482489856 + 4*1024*1024 as= suming worst case scenario here, the x856 is the starting byte of the last = extent) still leaves=C2=A02277376 bytes, way more than 24k.
--0000000000009780e4056dc43e1d--