[linux-lvm] Scrub errors after extending LVM RAID1 mirror

[linux-lvm] Scrub errors after extending LVM RAID1 mirror

[Steve Dodd]

[-- Attachment #1: Type: text/plain, Size: 167 bytes --]

Hi everyone,

I am experiencing a mystery scrub failure after extending a particular LV
which is a raid1 type mirror. I am using Ubuntu 18.04,
LVM 2.02.176-4.1ubuntu3

[-- Attachment #2: Type: text/html, Size: 235 bytes --]

Re: [linux-lvm] Scrub errors after extending LVM RAID1 mirror [full email]

[Steve Dodd]

[-- Attachment #1: Type: text/plain, Size: 1718 bytes --]

Sorry, user error sent the last email before I'd finished typing, trying
again..

Hi everyone,

I am experiencing a mystery scrub failure after extending a particular LV
which is a raid1 type mirror. I am using Ubuntu 18.04, LVM
2.02.176-4.1ubuntu3, Ubuntu kernel 4.15.0-29-generic. I mentioned this on
IRC, thought an email might reach more people and allow me to provide more
detail.

As far as I can tell, the LV was *not* created with --nosync:

# lvs rvg/backups
>   LV      VG  Attr       LSize  Pool Origin Data%  Meta%  Move Log
> Cpy%Sync Convert
>   backups rvg rwi-aor--- 96.64G                                    100.00
>


The only odd thing I tend to do is specify extents for the extension
manually, being a bit OCD about on disk segment layouts. Having mined
.bash_history, it seems that last time I ran:

lvextend -l+2561 rvg/backups /dev/sdc3:20480-23041 /dev/sdb3:80097-82658


After that, a *lvchange --syncaction check rvg/backups* showed a huge
number for raid_mismatch_count (seemed roughly consistent with the newly
extended portion not being synced), but dumping the actual filesystem with
partclone from both legs of the mirror through md5sum showed no
inconsistencies; the contents are mostly borg repositories and for good
measure I verified the data in those using borg as well - no problems.

After a full resync all is well again. This is the second time this
happened to me on the same LV (I think - certainly the same VG.)

Any clues? Any known bugs fixed recently that might not have made it into
Ubuntu 1804? I am trying to reproduce with a test LV but can't. Only other
thing I can think might be relevant was that the volume was mounted (but
quiescent) at the time.

Thanks,
Steve

[-- Attachment #2: Type: text/html, Size: 2587 bytes --]

Re: [linux-lvm] Scrub errors after extending LVM RAID1 mirror [full email]

[Steve Dodd]

[-- Attachment #1: Type: text/plain, Size: 2229 bytes --]

Weirdly, I thought I had failed to reproduce this bug, but my auto-scrub
job ran this morning (first Sat of month), and I got:

03:15:15: Starting scrub of rvg/test ...
03:15:15: ... scrub started ...
03:18:36: FAILED:      7926656 mismatches

So I really have no idea what's going on there. I will wade through my bash
history and see if I can see what I did last week and what triggered this..

S.

On Wed, 23 Jan 2019 at 10:46, Steve Dodd <steved424@gmail.com> wrote:

> Sorry, user error sent the last email before I'd finished typing, trying
> again..
>
> Hi everyone,
>
> I am experiencing a mystery scrub failure after extending a particular LV
> which is a raid1 type mirror. I am using Ubuntu 18.04, LVM
> 2.02.176-4.1ubuntu3, Ubuntu kernel 4.15.0-29-generic. I mentioned this on
> IRC, thought an email might reach more people and allow me to provide more
> detail.
>
> As far as I can tell, the LV was *not* created with --nosync:
>
> # lvs rvg/backups
>>   LV      VG  Attr       LSize  Pool Origin Data%  Meta%  Move Log
>> Cpy%Sync Convert
>>   backups rvg rwi-aor--- 96.64G
>> 100.00
>
>
> The only odd thing I tend to do is specify extents for the extension
> manually, being a bit OCD about on disk segment layouts. Having mined
> .bash_history, it seems that last time I ran:
>
> lvextend -l+2561 rvg/backups /dev/sdc3:20480-23041 /dev/sdb3:80097-82658
>
>
> After that, a *lvchange --syncaction check rvg/backups* showed a huge
> number for raid_mismatch_count (seemed roughly consistent with the newly
> extended portion not being synced), but dumping the actual filesystem with
> partclone from both legs of the mirror through md5sum showed no
> inconsistencies; the contents are mostly borg repositories and for good
> measure I verified the data in those using borg as well - no problems.
>
> After a full resync all is well again. This is the second time this
> happened to me on the same LV (I think - certainly the same VG.)
>
> Any clues? Any known bugs fixed recently that might not have made it into
> Ubuntu 1804? I am trying to reproduce with a test LV but can't. Only other
> thing I can think might be relevant was that the volume was mounted (but
> quiescent) at the time.
>
> Thanks,
> Steve
>

[-- Attachment #2: Type: text/html, Size: 3482 bytes --]

Re: [linux-lvm] Scrub errors after extending LVM RAID1 mirror [full email]

[Steve Dodd]

[-- Attachment #1: Type: text/plain, Size: 2583 bytes --]

I think I've finally nailed down a test case, if anyone is bored and wants
to play:

https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/1814389

I can't file upstream as RH bugzilla is down at the moment..

S.

On Sat, 2 Feb 2019 at 13:34, Steve Dodd <steved424@gmail.com> wrote:

> Weirdly, I thought I had failed to reproduce this bug, but my auto-scrub
> job ran this morning (first Sat of month), and I got:
>
> 03:15:15: Starting scrub of rvg/test ...
> 03:15:15: ... scrub started ...
> 03:18:36: FAILED:      7926656 mismatches
>
> So I really have no idea what's going on there. I will wade through my
> bash history and see if I can see what I did last week and what triggered
> this..
>
> S.
>
> On Wed, 23 Jan 2019 at 10:46, Steve Dodd <steved424@gmail.com> wrote:
>
>> Sorry, user error sent the last email before I'd finished typing, trying
>> again..
>>
>> Hi everyone,
>>
>> I am experiencing a mystery scrub failure after extending a particular LV
>> which is a raid1 type mirror. I am using Ubuntu 18.04, LVM
>> 2.02.176-4.1ubuntu3, Ubuntu kernel 4.15.0-29-generic. I mentioned this on
>> IRC, thought an email might reach more people and allow me to provide more
>> detail.
>>
>> As far as I can tell, the LV was *not* created with --nosync:
>>
>> # lvs rvg/backups
>>>   LV      VG  Attr       LSize  Pool Origin Data%  Meta%  Move Log
>>> Cpy%Sync Convert
>>>   backups rvg rwi-aor--- 96.64G
>>> 100.00
>>
>>
>> The only odd thing I tend to do is specify extents for the extension
>> manually, being a bit OCD about on disk segment layouts. Having mined
>> .bash_history, it seems that last time I ran:
>>
>> lvextend -l+2561 rvg/backups /dev/sdc3:20480-23041 /dev/sdb3:80097-82658
>>
>>
>> After that, a *lvchange --syncaction check rvg/backups* showed a huge
>> number for raid_mismatch_count (seemed roughly consistent with the newly
>> extended portion not being synced), but dumping the actual filesystem with
>> partclone from both legs of the mirror through md5sum showed no
>> inconsistencies; the contents are mostly borg repositories and for good
>> measure I verified the data in those using borg as well - no problems.
>>
>> After a full resync all is well again. This is the second time this
>> happened to me on the same LV (I think - certainly the same VG.)
>>
>> Any clues? Any known bugs fixed recently that might not have made it into
>> Ubuntu 1804? I am trying to reproduce with a test LV but can't. Only other
>> thing I can think might be relevant was that the volume was mounted (but
>> quiescent) at the time.
>>
>> Thanks,
>> Steve
>>
>

[-- Attachment #2: Type: text/html, Size: 4238 bytes --]