All of lore.kernel.org
 help / color / mirror / Atom feed
* Array size dropped from 40TB to 7TB when upgrading to 5.10
@ 2020-12-15 18:29 Sébastien Luttringer
  2020-12-15 23:53 ` Song Liu
  0 siblings, 1 reply; 4+ messages in thread
From: Sébastien Luttringer @ 2020-12-15 18:29 UTC (permalink / raw)
  To: linux-raid

[-- Attachment #1: Type: text/plain, Size: 3769 bytes --]

Hello,

After a clean reboot to the new kernel 5.10.0 my 40TB md raid5 array size
droped to 7TB.
The previous kernel was 5.9.5. Rebooting back to the 5.9.5 didn't fix the
issue.

# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdf[9] sdd[10] sda[7] sdb[6] sdc[11] sde[8]
      6857871360 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
[UUUUUU]
      
unused devices: <none>


journalctl -oshort-iso --no-hostname -b -6|grep md0
2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sdf operational as raid
disk 0
2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sda operational as raid
disk 5
2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sdd operational as raid
disk 4
2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sde operational as raid
disk 2
2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sdc operational as raid
disk 1
2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sdb operational as raid
disk 3
2020-12-04T02:30:47+0100 kernel: md/raid:md0: raid level 5 active with 6 out of
6 devices, algorithm 2
2020-12-04T02:30:47+0100 kernel: md0: detected capacity change from 0 to
40007809105920
2020-12-04T02:31:47+0100 kernel: EXT4-fs (md0): mounted filesystem with ordered
data mode. Opts: (null)

# journalctl -oshort-iso --no-hostname -b -5|grep md0
2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sdf operational as raid
disk 0
2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sda operational as raid
disk 5
2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sde operational as raid
disk 2
2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sdd operational as raid
disk 4
2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sdc operational as raid
disk 1
2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sdb operational as raid
disk 3
2020-12-15T03:53:00+0100 kernel: md/raid:md0: raid level 5 active with 6 out of
6 devices, algorithm 2
2020-12-15T03:53:00+0100 kernel: md0: detected capacity change from 0 to
7022460272640
2020-12-15T03:54:20+0100 systemd-fsck[1009]: fsck.ext4: Invalid argument while
trying to open /dev/md0

There is no log of hardware errors or unclean unmounting.

# mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Mon Jan 24 02:53:21 2011
        Raid Level : raid5
        Array Size : 6857871360 (6540.18 GiB 7022.46 GB)
     Used Dev Size : 1371574272 (1308.04 GiB 1404.49 GB)
      Raid Devices : 6
     Total Devices : 6
       Persistence : Superblock is persistent

       Update Time : Tue Dec 15 17:53:13 2020
             State : clean 
    Active Devices : 6
   Working Devices : 6
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : resync

              Name : white:0  (local to host white)
              UUID : affd87df:da503e3b:52a8b97f:77b80c0c
            Events : 1791763

    Number   Major   Minor   RaidDevice State
       9       8       80        0      active sync   /dev/sdf
      11       8       32        1      active sync   /dev/sdc
       8       8       64        2      active sync   /dev/sde
       6       8       16        3      active sync   /dev/sdb
      10       8       48        4      active sync   /dev/sdd
       7       8        0        5      active sync   /dev/sda

The mdadm userspace as not been updated.
# mdadm -V
mdadm - v4.1 - 2018-10-01

An `mdadm --action check /dev/md0` was run without errors.

1) What's the best option to restore the size without loosing the data?
2) Is this issue can be related to the kernel upgrade or it's fortuitous?

Regards,

Sébastien "Seblu" Luttringer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Array size dropped from 40TB to 7TB when upgrading to 5.10
  2020-12-15 18:29 Array size dropped from 40TB to 7TB when upgrading to 5.10 Sébastien Luttringer
@ 2020-12-15 23:53 ` Song Liu
  2020-12-16  2:20   ` Sébastien Luttringer
  0 siblings, 1 reply; 4+ messages in thread
From: Song Liu @ 2020-12-15 23:53 UTC (permalink / raw)
  To: Sébastien Luttringer; +Cc: linux-raid

On Tue, Dec 15, 2020 at 10:40 AM Sébastien Luttringer <seblu@seblu.net> wrote:
>
> Hello,
>
> After a clean reboot to the new kernel 5.10.0 my 40TB md raid5 array size
> droped to 7TB.
> The previous kernel was 5.9.5. Rebooting back to the 5.9.5 didn't fix the
> issue.
>
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 sdf[9] sdd[10] sda[7] sdb[6] sdc[11] sde[8]
>       6857871360 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
> [UUUUUU]
>
> unused devices: <none>
>
>
> journalctl -oshort-iso --no-hostname -b -6|grep md0
> 2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sdf operational as raid
> disk 0
> 2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sda operational as raid
> disk 5
> 2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sdd operational as raid
> disk 4
> 2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sde operational as raid
> disk 2
> 2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sdc operational as raid
> disk 1
> 2020-12-04T02:30:47+0100 kernel: md/raid:md0: device sdb operational as raid
> disk 3
> 2020-12-04T02:30:47+0100 kernel: md/raid:md0: raid level 5 active with 6 out of
> 6 devices, algorithm 2
> 2020-12-04T02:30:47+0100 kernel: md0: detected capacity change from 0 to
> 40007809105920
> 2020-12-04T02:31:47+0100 kernel: EXT4-fs (md0): mounted filesystem with ordered
> data mode. Opts: (null)
>
> # journalctl -oshort-iso --no-hostname -b -5|grep md0
> 2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sdf operational as raid
> disk 0
> 2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sda operational as raid
> disk 5
> 2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sde operational as raid
> disk 2
> 2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sdd operational as raid
> disk 4
> 2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sdc operational as raid
> disk 1
> 2020-12-15T03:53:00+0100 kernel: md/raid:md0: device sdb operational as raid
> disk 3
> 2020-12-15T03:53:00+0100 kernel: md/raid:md0: raid level 5 active with 6 out of
> 6 devices, algorithm 2
> 2020-12-15T03:53:00+0100 kernel: md0: detected capacity change from 0 to
> 7022460272640
> 2020-12-15T03:54:20+0100 systemd-fsck[1009]: fsck.ext4: Invalid argument while
> trying to open /dev/md0
>
> There is no log of hardware errors or unclean unmounting.
>
> # mdadm -D /dev/md0
> /dev/md0:
>            Version : 1.2
>      Creation Time : Mon Jan 24 02:53:21 2011
>         Raid Level : raid5
>         Array Size : 6857871360 (6540.18 GiB 7022.46 GB)
>      Used Dev Size : 1371574272 (1308.04 GiB 1404.49 GB)
>       Raid Devices : 6
>      Total Devices : 6
>        Persistence : Superblock is persistent
>
>        Update Time : Tue Dec 15 17:53:13 2020
>              State : clean
>     Active Devices : 6
>    Working Devices : 6
>     Failed Devices : 0
>      Spare Devices : 0
>
>             Layout : left-symmetric
>         Chunk Size : 512K
>
> Consistency Policy : resync
>
>               Name : white:0  (local to host white)
>               UUID : affd87df:da503e3b:52a8b97f:77b80c0c
>             Events : 1791763
>
>     Number   Major   Minor   RaidDevice State
>        9       8       80        0      active sync   /dev/sdf
>       11       8       32        1      active sync   /dev/sdc
>        8       8       64        2      active sync   /dev/sde
>        6       8       16        3      active sync   /dev/sdb
>       10       8       48        4      active sync   /dev/sdd
>        7       8        0        5      active sync   /dev/sda
>
> The mdadm userspace as not been updated.
> # mdadm -V
> mdadm - v4.1 - 2018-10-01
>
> An `mdadm --action check /dev/md0` was run without errors.
>
> 1) What's the best option to restore the size without loosing the data?
> 2) Is this issue can be related to the kernel upgrade or it's fortuitous?

Hi,

I am very sorry for this problem. This is a bug in 5.10 which is fixed
in 5.10.1.
To fix it, please upgrade your kernel to 5.10.1 (or downgrade to previous
version). In many cases, the array should be back normal. If not, please try

       mdadm --grow --size <size> /dev/mdXXX.

If the original array uses the full disk/partition, you can use "max" for <size>
to safe some calculation.

Please let me know if you have future problem with it.

Thanks,
Song

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Array size dropped from 40TB to 7TB when upgrading to 5.10
  2020-12-15 23:53 ` Song Liu
@ 2020-12-16  2:20   ` Sébastien Luttringer
  2020-12-16  4:28     ` Song Liu
  0 siblings, 1 reply; 4+ messages in thread
From: Sébastien Luttringer @ 2020-12-16  2:20 UTC (permalink / raw)
  To: Song Liu; +Cc: linux-raid

[-- Attachment #1: Type: text/plain, Size: 934 bytes --]


On Tue, 2020-12-15 at 15:53 -0800, Song Liu wrote:
> On Tue, Dec 15, 2020 at 10:40 AM Sébastien Luttringer <seblu@seblu.net>
> wrote:
> 
> 
> Hi,
> 
> I am very sorry for this problem. This is a bug in 5.10 which is fixed
> in 5.10.1.
> To fix it, please upgrade your kernel to 5.10.1 (or downgrade to previous
> version). In many cases, the array should be back normal. If not, please try
> 
>        mdadm --grow --size <size> /dev/mdXXX.
> 
> If the original array uses the full disk/partition, you can use "max" for
> <size>
> to safe some calculation.
> 
> Please let me know if you have future problem with it.

Hello,

The array didn't returned to normal after a reboot on 5.10.1. The `mdadm --grow
--size max /dev/md0` command did the trick. The e2fsck detect no error, so we
are back online.

Thanks you for your help and especially your quick answer.


Sébastien "Seblu" Luttringer


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Array size dropped from 40TB to 7TB when upgrading to 5.10
  2020-12-16  2:20   ` Sébastien Luttringer
@ 2020-12-16  4:28     ` Song Liu
  0 siblings, 0 replies; 4+ messages in thread
From: Song Liu @ 2020-12-16  4:28 UTC (permalink / raw)
  To: Sébastien Luttringer; +Cc: linux-raid

On Tue, Dec 15, 2020 at 6:28 PM Sébastien Luttringer <seblu@seblu.net> wrote:
>
>
> On Tue, 2020-12-15 at 15:53 -0800, Song Liu wrote:
> > On Tue, Dec 15, 2020 at 10:40 AM Sébastien Luttringer <seblu@seblu.net>
> > wrote:
> >
> >
> > Hi,
> >
> > I am very sorry for this problem. This is a bug in 5.10 which is fixed
> > in 5.10.1.
> > To fix it, please upgrade your kernel to 5.10.1 (or downgrade to previous
> > version). In many cases, the array should be back normal. If not, please try
> >
> >        mdadm --grow --size <size> /dev/mdXXX.
> >
> > If the original array uses the full disk/partition, you can use "max" for
> > <size>
> > to safe some calculation.
> >
> > Please let me know if you have future problem with it.
>
> Hello,
>
> The array didn't returned to normal after a reboot on 5.10.1. The `mdadm --grow
> --size max /dev/md0` command did the trick. The e2fsck detect no error, so we
> are back online.
>
> Thanks you for your help and especially your quick answer.

I am glad that helped.

Song

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2020-12-16  4:29 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-15 18:29 Array size dropped from 40TB to 7TB when upgrading to 5.10 Sébastien Luttringer
2020-12-15 23:53 ` Song Liu
2020-12-16  2:20   ` Sébastien Luttringer
2020-12-16  4:28     ` Song Liu

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.