Linux-Raid Archives on lore.kernel.org
 help / color / Atom feed
* RAID5 issue with UBUNTU 20.04.1 on my desktop
@ 2020-09-16  9:53 Sung, KoWei
  2020-09-17  7:03 ` Song Liu
  0 siblings, 1 reply; 6+ messages in thread
From: Sung, KoWei @ 2020-09-16  9:53 UTC (permalink / raw)
  To: song, linux-raid
  Cc: Bshara, Saeed, Duan, HanShen, Tokoyo, Hiroshi, Fortin, Mike


[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

Hi,
 
I found RAID5 stability issue while doing disk expansion.
I attached 4 disks (/dev/sda, /dev/sdb, /dev/sdc and /dev/sdd) and create partition by “create_partition.sh” scripts on my PC and run my test scripts “raid_reshape_12.sh” (as attached).
Basically, the test will add partitions to RAID5 (/dev/md3) and write files to /dev/md3 (ext4) at the same time.
Within 1 or 2 hours, kernel will get crashed (Oops) and reshape/resync cannot be finished forever (log as attached).
 
The issue happens randomly, but it most likely happens at beginning of reshape process. When kernel crash happens, the reshape stops at about 3-10% complete only.
Moreover, it is not related to any partition size, because I’ve tried different size, but issue still exists.
I've also tried different kernel (4.1/4.2/4.9/4.19/5.4/5.8), and all kernel version can see this issue.

Best Regards,
Winder


[-- Attachment #2: 4disk_x86.tar.bz2 --]
[-- Type: application/x-bzip2, Size: 4125 bytes --]

[-- Attachment #3: dmesg_raid5_crash.log --]
[-- Type: application/octet-stream, Size: 64856 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: RAID5 issue with UBUNTU 20.04.1 on my desktop
  2020-09-16  9:53 RAID5 issue with UBUNTU 20.04.1 on my desktop Sung, KoWei
@ 2020-09-17  7:03 ` Song Liu
  2020-09-22  0:34   ` Sung, KoWei
  0 siblings, 1 reply; 6+ messages in thread
From: Song Liu @ 2020-09-17  7:03 UTC (permalink / raw)
  To: Sung, KoWei
  Cc: linux-raid, Bshara, Saeed, Duan, HanShen, Tokoyo, Hiroshi, Fortin, Mike

Hi Winder,

On Wed, Sep 16, 2020 at 2:53 AM Sung, KoWei <winders@amazon.com> wrote:
>
> Hi,
>
> I found RAID5 stability issue while doing disk expansion.
> I attached 4 disks (/dev/sda, /dev/sdb, /dev/sdc and /dev/sdd) and create partition by “create_partition.sh” scripts on my PC and run my test scripts “raid_reshape_12.sh” (as attached).
> Basically, the test will add partitions to RAID5 (/dev/md3) and write files to /dev/md3 (ext4) at the same time.
> Within 1 or 2 hours, kernel will get crashed (Oops) and reshape/resync cannot be finished forever (log as attached).
>
> The issue happens randomly, but it most likely happens at beginning of reshape process. When kernel crash happens, the reshape stops at about 3-10% complete only.
> Moreover, it is not related to any partition size, because I’ve tried different size, but issue still exists.
> I've also tried different kernel (4.1/4.2/4.9/4.19/5.4/5.8), and all kernel version can see this issue.

Thanks for the report. I just started some tests with the script. I
will update whether it repros the issue.

Song

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: RAID5 issue with UBUNTU 20.04.1 on my desktop
  2020-09-17  7:03 ` Song Liu
@ 2020-09-22  0:34   ` Sung, KoWei
  2020-09-28 17:15     ` Song Liu
  0 siblings, 1 reply; 6+ messages in thread
From: Sung, KoWei @ 2020-09-22  0:34 UTC (permalink / raw)
  To: Song Liu
  Cc: linux-raid, Bshara, Saeed, Duan, HanShen, Tokoyo, Hiroshi, Fortin, Mike

Hi, Song Liu:

May I know if you're able to reproduce this issue? Thanks a lot for your help.

Best Regards,
Winder
________________________________________
From: Song Liu <song@kernel.org>
Sent: Thursday, September 17, 2020 3:03 PM
To: Sung, KoWei
Cc: linux-raid@vger.kernel.org; Bshara, Saeed; Duan, HanShen; Tokoyo, Hiroshi; Fortin, Mike
Subject: RE: [EXTERNAL] RAID5 issue with UBUNTU 20.04.1 on my desktop

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Hi Winder,

On Wed, Sep 16, 2020 at 2:53 AM Sung, KoWei <winders@amazon.com> wrote:
>
> Hi,
>
> I found RAID5 stability issue while doing disk expansion.
> I attached 4 disks (/dev/sda, /dev/sdb, /dev/sdc and /dev/sdd) and create partition by “create_partition.sh” scripts on my PC and run my test scripts “raid_reshape_12.sh” (as attached).
> Basically, the test will add partitions to RAID5 (/dev/md3) and write files to /dev/md3 (ext4) at the same time.
> Within 1 or 2 hours, kernel will get crashed (Oops) and reshape/resync cannot be finished forever (log as attached).
>
> The issue happens randomly, but it most likely happens at beginning of reshape process. When kernel crash happens, the reshape stops at about 3-10% complete only.
> Moreover, it is not related to any partition size, because I’ve tried different size, but issue still exists.
> I've also tried different kernel (4.1/4.2/4.9/4.19/5.4/5.8), and all kernel version can see this issue.

Thanks for the report. I just started some tests with the script. I
will update whether it repros the issue.

Song

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: RAID5 issue with UBUNTU 20.04.1 on my desktop
  2020-09-22  0:34   ` Sung, KoWei
@ 2020-09-28 17:15     ` Song Liu
  2020-10-05  7:40       ` Song Liu
  0 siblings, 1 reply; 6+ messages in thread
From: Song Liu @ 2020-09-28 17:15 UTC (permalink / raw)
  To: Sung, KoWei
  Cc: linux-raid, Bshara, Saeed, Duan, HanShen, Tokoyo, Hiroshi, Fortin, Mike

On Mon, Sep 21, 2020 at 5:34 PM Sung, KoWei <winders@amazon.com> wrote:
>
> Hi, Song Liu:
>
> May I know if you're able to reproduce this issue? Thanks a lot for your help.

Sorry for the delay.

Yes, I was able to reproduce the issue. But I haven't got a proper fix
yet. Hopefully
I will be able to fix it soon.

Thanks,
Song

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: RAID5 issue with UBUNTU 20.04.1 on my desktop
  2020-09-28 17:15     ` Song Liu
@ 2020-10-05  7:40       ` Song Liu
  2020-10-07  7:02         ` Sung, KoWei
  0 siblings, 1 reply; 6+ messages in thread
From: Song Liu @ 2020-10-05  7:40 UTC (permalink / raw)
  To: Sung, KoWei
  Cc: linux-raid, Bshara, Saeed, Duan, HanShen, Tokoyo, Hiroshi, Fortin, Mike

Hi KoWei,

On Mon, Sep 28, 2020 at 10:15 AM Song Liu <song@kernel.org> wrote:
>
> On Mon, Sep 21, 2020 at 5:34 PM Sung, KoWei <winders@amazon.com> wrote:
> >
> > Hi, Song Liu:
> >
> > May I know if you're able to reproduce this issue? Thanks a lot for your help.
>

Could you please verify whether the following patch fixes it? If it
works well, please
reply with your Test-by tag.

Thanks,
Song

diff --git i/drivers/md/raid5.c w/drivers/md/raid5.c
index 66690b40818e7..39343479ac2a9 100644
--- i/drivers/md/raid5.c
+++ w/drivers/md/raid5.c
@@ -2585,8 +2585,6 @@ static int resize_stripes(struct r5conf *conf,
int newsize)
        } else
                err = -ENOMEM;

-       mutex_unlock(&conf->cache_size_mutex);
-
        conf->slab_cache = sc;
        conf->active_name = 1-conf->active_name;

@@ -2628,6 +2626,8 @@ static int resize_stripes(struct r5conf *conf,
int newsize)

        if (!err)
                conf->pool_size = newsize;
+       mutex_unlock(&conf->cache_size_mutex);
+
        return err;
 }

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: RAID5 issue with UBUNTU 20.04.1 on my desktop
  2020-10-05  7:40       ` Song Liu
@ 2020-10-07  7:02         ` Sung, KoWei
  0 siblings, 0 replies; 6+ messages in thread
From: Sung, KoWei @ 2020-10-07  7:02 UTC (permalink / raw)
  To: Song Liu
  Cc: linux-raid, Bshara, Saeed, Duan, HanShen, Tokoyo, Hiroshi, Fortin, Mike

Hi, Song:

Thanks a lot for your fix.
I've tested the patch by "v4.19.149" tag from Linux kernel stable tree (kernel/git/stable/linux.git).
The test runs over 1000 rounds in 24 hours without kernel crash, so the patch should fix this issue.

Best Regards,
Winder
________________________________________
From: Song Liu <song@kernel.org>
Sent: Monday, October 5, 2020 3:40 PM
To: Sung, KoWei
Cc: linux-raid@vger.kernel.org; Bshara, Saeed; Duan, HanShen; Tokoyo, Hiroshi; Fortin, Mike
Subject: RE: [EXTERNAL] RAID5 issue with UBUNTU 20.04.1 on my desktop

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Hi KoWei,

On Mon, Sep 28, 2020 at 10:15 AM Song Liu <song@kernel.org> wrote:
>
> On Mon, Sep 21, 2020 at 5:34 PM Sung, KoWei <winders@amazon.com> wrote:
> >
> > Hi, Song Liu:
> >
> > May I know if you're able to reproduce this issue? Thanks a lot for your help.
>

Could you please verify whether the following patch fixes it? If it
works well, please
reply with your Test-by tag.

Thanks,
Song

diff --git i/drivers/md/raid5.c w/drivers/md/raid5.c
index 66690b40818e7..39343479ac2a9 100644
--- i/drivers/md/raid5.c
+++ w/drivers/md/raid5.c
@@ -2585,8 +2585,6 @@ static int resize_stripes(struct r5conf *conf,
int newsize)
        } else
                err = -ENOMEM;

-       mutex_unlock(&conf->cache_size_mutex);
-
        conf->slab_cache = sc;
        conf->active_name = 1-conf->active_name;

@@ -2628,6 +2626,8 @@ static int resize_stripes(struct r5conf *conf,
int newsize)

        if (!err)
                conf->pool_size = newsize;
+       mutex_unlock(&conf->cache_size_mutex);
+
        return err;
 }

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, back to index

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-16  9:53 RAID5 issue with UBUNTU 20.04.1 on my desktop Sung, KoWei
2020-09-17  7:03 ` Song Liu
2020-09-22  0:34   ` Sung, KoWei
2020-09-28 17:15     ` Song Liu
2020-10-05  7:40       ` Song Liu
2020-10-07  7:02         ` Sung, KoWei

Linux-Raid Archives on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-raid/0 linux-raid/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-raid linux-raid/ https://lore.kernel.org/linux-raid \
		linux-raid@vger.kernel.org
	public-inbox-index linux-raid

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-raid


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git