resize2fs hang at read function

resize2fs hang at read function

[rong zhao]

Hi,

I met a problem on resize2fs ext4 filesystem.

# Environment
OS: Redhat 7.4 64bit, Kernel: 3.10.0-693.11.6.el7.x86_64

Storage: connected 3PAR storage with multipath, create LVM on LUNs
Filesystem: ext4

# Problem Description
1. create new filesystem, format ext4
2. I tried extend LVM, and resize2fs online, it worked indeed
3. I copied 27T data to the new filesystem
4. Then, I try to extend LVM and resize2fs online again, resize2fs
hang at read function, the strace records are below:
----there are more lines above, do not paste here -------
lseek(3, 38465727102976, SEEK_SET)      = 38465727102976
read(3, "\1\0\300/\21\0\300/!\0\300/_\177\200\0\0\0\5\0\0\0\0\0\0\0\0\0\200\0\330\326"...,
4096) = 4096
lseek(3, 38474317037568, SEEK_SET)      = 38474317037568
read(3, "\1\0\340/\21\0\340/!\0\340/_\177\200\0\0\0\5\0\0\0\0\0\0\0\0\0\200\0000\214"...,
4096) = 4096

--- hang here

5. the filesystem state is clean:
dumpe2fs /dev/billdb_new/billdb_new | grep state
dumpe2fs 1.42.9 (28-Dec-2013)
Filesystem state:         clean

6. e2fsck show the filesystem is healthy.

# Other Information
1. I had old filesystem with same issue, to solve this problem, I
requested new LUNs from storage and create new filesystem
2. Copy data from old filesystem to new filesystem
3. The old fiesystem and new filesyste can be read/written without problem


Please give me some suggestions, have googled long time..

Thanks.

Re: resize2fs hang at read function

[rong zhao]

Hi All,

I downloaded the latest e2fsprogs source code from kernel.org, it works.

Thanks.

2018-03-07 9:59 GMT+08:00 rong zhao <zhaorbox@gmail.com>:
> Hi,
>
> I met a problem on resize2fs ext4 filesystem.
>
> # Environment
> OS: Redhat 7.4 64bit, Kernel: 3.10.0-693.11.6.el7.x86_64
>
> Storage: connected 3PAR storage with multipath, create LVM on LUNs
> Filesystem: ext4
>
> # Problem Description
> 1. create new filesystem, format ext4
> 2. I tried extend LVM, and resize2fs online, it worked indeed
> 3. I copied 27T data to the new filesystem
> 4. Then, I try to extend LVM and resize2fs online again, resize2fs
> hang at read function, the strace records are below:
> ----there are more lines above, do not paste here -------
> lseek(3, 38465727102976, SEEK_SET)      = 38465727102976
> read(3, "\1\0\300/\21\0\300/!\0\300/_\177\200\0\0\0\5\0\0\0\0\0\0\0\0\0\200\0\330\326"...,
> 4096) = 4096
> lseek(3, 38474317037568, SEEK_SET)      = 38474317037568
> read(3, "\1\0\340/\21\0\340/!\0\340/_\177\200\0\0\0\5\0\0\0\0\0\0\0\0\0\200\0000\214"...,
> 4096) = 4096
>
> --- hang here
>
> 5. the filesystem state is clean:
> dumpe2fs /dev/billdb_new/billdb_new | grep state
> dumpe2fs 1.42.9 (28-Dec-2013)
> Filesystem state:         clean
>
> 6. e2fsck show the filesystem is healthy.
>
> # Other Information
> 1. I had old filesystem with same issue, to solve this problem, I
> requested new LUNs from storage and create new filesystem
> 2. Copy data from old filesystem to new filesystem
> 3. The old fiesystem and new filesyste can be read/written without problem
>
>
> Please give me some suggestions, have googled long time..
>
> Thanks.

Re: resize2fs hang at read function

[Theodore Y. Ts'o]

On Wed, Mar 07, 2018 at 11:01:40AM +0800, rong zhao wrote:
> 
> I downloaded the latest e2fsprogs source code from kernel.org, it works.

I'm not sure what the difference is, but I'm *shocked* that RHEL 7.4
is still using e2fsprogs 1.42.x.  There are a very large number of
resize2fs bugs, especially with very large file systems, with
e2fsprogs 1.42.x.  Especially in the case of off-line resizes with a
lage ext4 file system, with a resize2fs from e2fsprogs 1.42.x, data
corruption is almost a certainty.

If you have used resize2fs 1.42.x on that file system, I would suggest
running e2fsck on your large ext4 file system unmounted when you have
a chance, to make sure it's in a good and healthy state.

Regards,

					- Ted

Re: resize2fs hang at read function

[Lukas Czerner]

On Wed, Mar 07, 2018 at 09:59:35AM +0800, rong zhao wrote:
> Hi,
> 
> I met a problem on resize2fs ext4 filesystem.
> 
> # Environment
> OS: Redhat 7.4 64bit, Kernel: 3.10.0-693.11.6.el7.x86_64
> 
> Storage: connected 3PAR storage with multipath, create LVM on LUNs
> Filesystem: ext4
> 
> # Problem Description
> 1. create new filesystem, format ext4
> 2. I tried extend LVM, and resize2fs online, it worked indeed
> 3. I copied 27T data to the new filesystem
> 4. Then, I try to extend LVM and resize2fs online again, resize2fs
> hang at read function, the strace records are below:
> ----there are more lines above, do not paste here -------
> lseek(3, 38465727102976, SEEK_SET)      = 38465727102976
> read(3, "\1\0\300/\21\0\300/!\0\300/_\177\200\0\0\0\5\0\0\0\0\0\0\0\0\0\200\0\330\326"...,
> 4096) = 4096
> lseek(3, 38474317037568, SEEK_SET)      = 38474317037568
> read(3, "\1\0\340/\21\0\340/!\0\340/_\177\200\0\0\0\5\0\0\0\0\0\0\0\0\0\200\0000\214"...,
> 4096) = 4096
> 
> --- hang here

Hi,

thanks for report. Could you please fill a bug at bugzilla.redhat.com
and cc me ?

Thanks!
-Lukas

> 
> 5. the filesystem state is clean:
> dumpe2fs /dev/billdb_new/billdb_new | grep state
> dumpe2fs 1.42.9 (28-Dec-2013)
> Filesystem state:         clean
> 
> 6. e2fsck show the filesystem is healthy.
> 
> # Other Information
> 1. I had old filesystem with same issue, to solve this problem, I
> requested new LUNs from storage and create new filesystem
> 2. Copy data from old filesystem to new filesystem
> 3. The old fiesystem and new filesyste can be read/written without problem
> 
> 
> Please give me some suggestions, have googled long time..
> 
> Thanks.

Re: resize2fs hang at read function

[Lukas Czerner]

On Wed, Mar 07, 2018 at 12:08:46AM -0500, Theodore Y. Ts'o wrote:
> On Wed, Mar 07, 2018 at 11:01:40AM +0800, rong zhao wrote:
> > 
> > I downloaded the latest e2fsprogs source code from kernel.org, it works.
> 
> I'm not sure what the difference is, but I'm *shocked* that RHEL 7.4
> is still using e2fsprogs 1.42.x.  There are a very large number of
> resize2fs bugs, especially with very large file systems, with
> e2fsprogs 1.42.x.  Especially in the case of off-line resizes with a
> lage ext4 file system, with a resize2fs from e2fsprogs 1.42.x, data
> corruption is almost a certainty.

The version number (as with kernel) does not really say much. We
backport a lot of stuff.

Regards,
-Lukas

> 
> If you have used resize2fs 1.42.x on that file system, I would suggest
> running e2fsck on your large ext4 file system unmounted when you have
> a chance, to make sure it's in a good and healthy state.
> 
> Regards,
> 
> 					- Ted

Re: resize2fs hang at read function

[Jan Kara]

On Wed 07-03-18 00:08:46, Theodore Y. Ts'o wrote:
> On Wed, Mar 07, 2018 at 11:01:40AM +0800, rong zhao wrote:
> > 
> > I downloaded the latest e2fsprogs source code from kernel.org, it works.
> 
> I'm not sure what the difference is, but I'm *shocked* that RHEL 7.4
> is still using e2fsprogs 1.42.x.  There are a very large number of
> resize2fs bugs, especially with very large file systems, with
> e2fsprogs 1.42.x.  Especially in the case of off-line resizes with a
> lage ext4 file system, with a resize2fs from e2fsprogs 1.42.x, data
> corruption is almost a certainty.

Yeah, welcome to the world of enterprise distributions :-|. Even in SLES 12
SP3 (our current latest "enterprise offering") we have e2fsprogs 1.42.11 as
well. And the reason is that at the time we were looking what e2fsprogs to
put there (which was about two years back), 1.43 was still "WIP" and so I
had some doubts whether we can ship it in a distro supported for another 10
years. So we have 1.42.11 and only backport fixes based on customers' bug
reports.

Luckily we are somewhat flexible at least on service pack releases so now
that e2fsprogs is having more regular releases (big thanks for that!), I
actually take your comment as a good reminder to talk to our PMs about
pushing 1.43.x to SLE12 SP4 :).

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: resize2fs hang at read function

[rong zhao]

OK, will do it.

Maybe enterprise world looks forward to more steady, so very conservative.

Rong

2018-03-07 16:49 GMT+08:00 Jan Kara <jack@suse.cz>:
> On Wed 07-03-18 00:08:46, Theodore Y. Ts'o wrote:
>> On Wed, Mar 07, 2018 at 11:01:40AM +0800, rong zhao wrote:
>> >
>> > I downloaded the latest e2fsprogs source code from kernel.org, it works.
>>
>> I'm not sure what the difference is, but I'm *shocked* that RHEL 7.4
>> is still using e2fsprogs 1.42.x.  There are a very large number of
>> resize2fs bugs, especially with very large file systems, with
>> e2fsprogs 1.42.x.  Especially in the case of off-line resizes with a
>> lage ext4 file system, with a resize2fs from e2fsprogs 1.42.x, data
>> corruption is almost a certainty.
>
> Yeah, welcome to the world of enterprise distributions :-|. Even in SLES 12
> SP3 (our current latest "enterprise offering") we have e2fsprogs 1.42.11 as
> well. And the reason is that at the time we were looking what e2fsprogs to
> put there (which was about two years back), 1.43 was still "WIP" and so I
> had some doubts whether we can ship it in a distro supported for another 10
> years. So we have 1.42.11 and only backport fixes based on customers' bug
> reports.
>
> Luckily we are somewhat flexible at least on service pack releases so now
> that e2fsprogs is having more regular releases (big thanks for that!), I
> actually take your comment as a good reminder to talk to our PMs about
> pushing 1.43.x to SLE12 SP4 :).
>
>                                                                 Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR

Re: resize2fs hang at read function

[rong zhao]

Hi Lukas,

   Bug 1553004 filed for this, if anything needed, please let me know,
hope can learn from you how to find root cause.

Thanks.

2018-03-08 7:32 GMT+08:00 rong zhao <zhaorbox@gmail.com>:
> OK, will do it.
>
> Maybe enterprise world looks forward to more steady, so very conservative.
>
> Rong
>
> 2018-03-07 16:49 GMT+08:00 Jan Kara <jack@suse.cz>:
>> On Wed 07-03-18 00:08:46, Theodore Y. Ts'o wrote:
>>> On Wed, Mar 07, 2018 at 11:01:40AM +0800, rong zhao wrote:
>>> >
>>> > I downloaded the latest e2fsprogs source code from kernel.org, it works.
>>>
>>> I'm not sure what the difference is, but I'm *shocked* that RHEL 7.4
>>> is still using e2fsprogs 1.42.x.  There are a very large number of
>>> resize2fs bugs, especially with very large file systems, with
>>> e2fsprogs 1.42.x.  Especially in the case of off-line resizes with a
>>> lage ext4 file system, with a resize2fs from e2fsprogs 1.42.x, data
>>> corruption is almost a certainty.
>>
>> Yeah, welcome to the world of enterprise distributions :-|. Even in SLES 12
>> SP3 (our current latest "enterprise offering") we have e2fsprogs 1.42.11 as
>> well. And the reason is that at the time we were looking what e2fsprogs to
>> put there (which was about two years back), 1.43 was still "WIP" and so I
>> had some doubts whether we can ship it in a distro supported for another 10
>> years. So we have 1.42.11 and only backport fixes based on customers' bug
>> reports.
>>
>> Luckily we are somewhat flexible at least on service pack releases so now
>> that e2fsprogs is having more regular releases (big thanks for that!), I
>> actually take your comment as a good reminder to talk to our PMs about
>> pushing 1.43.x to SLE12 SP4 :).
>>
>>                                                                 Honza
>> --
>> Jan Kara <jack@suse.com>
>> SUSE Labs, CR

Re: resize2fs hang at read function

[Lukas Czerner]

On Thu, Mar 08, 2018 at 10:20:02AM +0800, rong zhao wrote:
> Hi Lukas,
> 
>    Bug 1553004 filed for this, if anything needed, please let me know,
> hope can learn from you how to find root cause.
> 
> Thanks.

Thank you.

-Lukas

> 
> 2018-03-08 7:32 GMT+08:00 rong zhao <zhaorbox@gmail.com>:
> > OK, will do it.
> >
> > Maybe enterprise world looks forward to more steady, so very conservative.
> >
> > Rong
> >
> > 2018-03-07 16:49 GMT+08:00 Jan Kara <jack@suse.cz>:
> >> On Wed 07-03-18 00:08:46, Theodore Y. Ts'o wrote:
> >>> On Wed, Mar 07, 2018 at 11:01:40AM +0800, rong zhao wrote:
> >>> >
> >>> > I downloaded the latest e2fsprogs source code from kernel.org, it works.
> >>>
> >>> I'm not sure what the difference is, but I'm *shocked* that RHEL 7.4
> >>> is still using e2fsprogs 1.42.x.  There are a very large number of
> >>> resize2fs bugs, especially with very large file systems, with
> >>> e2fsprogs 1.42.x.  Especially in the case of off-line resizes with a
> >>> lage ext4 file system, with a resize2fs from e2fsprogs 1.42.x, data
> >>> corruption is almost a certainty.
> >>
> >> Yeah, welcome to the world of enterprise distributions :-|. Even in SLES 12
> >> SP3 (our current latest "enterprise offering") we have e2fsprogs 1.42.11 as
> >> well. And the reason is that at the time we were looking what e2fsprogs to
> >> put there (which was about two years back), 1.43 was still "WIP" and so I
> >> had some doubts whether we can ship it in a distro supported for another 10
> >> years. So we have 1.42.11 and only backport fixes based on customers' bug
> >> reports.
> >>
> >> Luckily we are somewhat flexible at least on service pack releases so now
> >> that e2fsprogs is having more regular releases (big thanks for that!), I
> >> actually take your comment as a good reminder to talk to our PMs about
> >> pushing 1.43.x to SLE12 SP4 :).
> >>
> >>                                                                 Honza
> >> --
> >> Jan Kara <jack@suse.com>
> >> SUSE Labs, CR