[PATCH] nvme/012 & 013: avoid extremely slow xfs IO

[PATCH] nvme/012 & 013: avoid extremely slow xfs IO

[Ming Lei]

It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
even worse it may trigger IO timeout on nvme-loop.

Eric and Dave replied that it is because of too small log size on small
disk.

So pass '-l size=32m' to avoid the issue.

With this patch, nvme/012 can be completed in one minute.

Cc: Eric Sandeen <esandeen@redhat.com>
Cc: Dave Chinner <dchinner@redhat.com>
Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
Cc: linux-xfs@vger.kernel.org
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
 tests/nvme/012 | 2 +-
 tests/nvme/013 | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/nvme/012 b/tests/nvme/012
index 9a6801511df7..d7a8751ec752 100755
--- a/tests/nvme/012
+++ b/tests/nvme/012
@@ -46,7 +46,7 @@ test() {
 
 	umount ${mount_dir} > /dev/null 2>&1
 
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 
 	mount /dev/"${nvmedev}n1" "${mount_dir}"
 
diff --git a/tests/nvme/013 b/tests/nvme/013
index e346cd4baa35..bc435f752dca 100755
--- a/tests/nvme/013
+++ b/tests/nvme/013
@@ -44,7 +44,7 @@ test() {
 
 	umount ${mount_dir} > /dev/null 2>&1
 
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m  -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 
 	mount /dev/"${nvmedev}n1" "${mount_dir}"
 
-- 
2.9.5

Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO

[Chaitanya Kulkarni]

Thanks for the patch Ming. Couple of comments below.
On 4/14/19 6:22 PM, Ming Lei wrote:
> It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
> even worse it may trigger IO timeout on nvme-loop.
> 
> Eric and Dave replied that it is because of too small log size on small
> disk.
> 
> So pass '-l size=32m' to avoid the issue.
> 
> With this patch, nvme/012 can be completed in one minute.
> 
Then we should set the QUICK=1 if its taking shorter time.

> Cc: Eric Sandeen <esandeen@redhat.com>
> Cc: Dave Chinner <dchinner@redhat.com>
> Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
> Cc: linux-xfs@vger.kernel.org
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
>   tests/nvme/012 | 2 +-
>   tests/nvme/013 | 2 +-
>   2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/nvme/012 b/tests/nvme/012
> index 9a6801511df7..d7a8751ec752 100755
> --- a/tests/nvme/012
> +++ b/tests/nvme/012
> @@ -46,7 +46,7 @@ test() {
>   
>   	umount ${mount_dir} > /dev/null 2>&1
>   
> -	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
> +	mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>   
As a part of this series lets move this to the helper mkfs in the 
nvme/rc and use that call in all the file-backed ns related testcases.
Let me know if you want me to do that or you would like to do that as a 
part of this series. I'm okay with anything.
>   	mount /dev/"${nvmedev}n1" "${mount_dir}"
>   
> diff --git a/tests/nvme/013 b/tests/nvme/013
> index e346cd4baa35..bc435f752dca 100755
> --- a/tests/nvme/013
> +++ b/tests/nvme/013
> @@ -44,7 +44,7 @@ test() {
>   
>   	umount ${mount_dir} > /dev/null 2>&1
>   
> -	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
> +	mkfs.xfs -l size=32m  -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>   
Extra space here but can be ignored once we move mkfs to the helper call.
>   	mount /dev/"${nvmedev}n1" "${mount_dir}"
>   
> 

Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO

[Chaitanya Kulkarni]

On 4/14/19 7:01 PM, Chaitanya Kulkarni wrote:
> Thanks for the patch Ming. Couple of comments below.
> On 4/14/19 6:22 PM, Ming Lei wrote:
>> It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
>> even worse it may trigger IO timeout on nvme-loop.
>>
>> Eric and Dave replied that it is because of too small log size on small
>> disk.
>>
>> So pass '-l size=32m' to avoid the issue.
>>
>> With this patch, nvme/012 can be completed in one minute.
>>
> Then we should set the QUICK=1 if its taking shorter time.
> 
>> Cc: Eric Sandeen <esandeen@redhat.com>
>> Cc: Dave Chinner <dchinner@redhat.com>
>> Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
>> Cc: linux-xfs@vger.kernel.org
>> Signed-off-by: Ming Lei <ming.lei@redhat.com>
>> ---
>>    tests/nvme/012 | 2 +-
>>    tests/nvme/013 | 2 +-
>>    2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/nvme/012 b/tests/nvme/012
>> index 9a6801511df7..d7a8751ec752 100755
>> --- a/tests/nvme/012
>> +++ b/tests/nvme/012
>> @@ -46,7 +46,7 @@ test() {
>>    
>>    	umount ${mount_dir} > /dev/null 2>&1
>>    
>> -	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>> +	mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>    
> As a part of this series lets move this to the helper mkfs in the
> nvme/rc and use that call in all the file-backed ns related testcases.
> Let me know if you want me to do that or you want me to do that as a
> part of this series. I'm okay with anything.
>>    	mount /dev/"${nvmedev}n1" "${mount_dir}"
>>    
>> diff --git a/tests/nvme/013 b/tests/nvme/013
>> index e346cd4baa35..bc435f752dca 100755
>> --- a/tests/nvme/013
>> +++ b/tests/nvme/013
>> @@ -44,7 +44,7 @@ test() {
>>    
>>    	umount ${mount_dir} > /dev/null 2>&1
>>    
>> -	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>> +	mkfs.xfs -l size=32m  -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>    
> Extra space here but can be ignored once we move mkfs to the helper call.
>>    	mount /dev/"${nvmedev}n1" "${mount_dir}"
>>    
>>
> 
> 

Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO

[Ming Lei]

On Mon, Apr 15, 2019 at 10:02 AM Chaitanya Kulkarni
<Chaitanya.Kulkarni@wdc.com> wrote:
>
> Thanks for the patch Ming. Couple of comments below.
> On 4/14/19 6:22 PM, Ming Lei wrote:
> > It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
> > even worse it may trigger IO timeout on nvme-loop.
> >
> > Eric and Dave replied that it is because of too small log size on small
> > disk.
> >
> > So pass '-l size=32m' to avoid the issue.
> >
> > With this patch, nvme/012 can be completed in one minute.
> >
> Then we should set the QUICK=1 if its taking shorter time.

That should have been a QUICK test, it is fine to set it, but it shouldn't
belong to this patch.

>
> > Cc: Eric Sandeen <esandeen@redhat.com>
> > Cc: Dave Chinner <dchinner@redhat.com>
> > Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
> > Cc: linux-xfs@vger.kernel.org
> > Signed-off-by: Ming Lei <ming.lei@redhat.com>
> > ---
> >   tests/nvme/012 | 2 +-
> >   tests/nvme/013 | 2 +-
> >   2 files changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/tests/nvme/012 b/tests/nvme/012
> > index 9a6801511df7..d7a8751ec752 100755
> > --- a/tests/nvme/012
> > +++ b/tests/nvme/012
> > @@ -46,7 +46,7 @@ test() {
> >
> >       umount ${mount_dir} > /dev/null 2>&1
> >
> > -     mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
> > +     mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
> >
> As a part of this series lets move this to the helper mkfs in the
> nvme/rc and use that call in all the file-backed ns related testcases.
> Let me know if you want me to do that or you would like to do that as a
> part of this series. I'm okay with anything.

I am fine with either way.

Thanks,
Ming Lei

Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO

[Chaitanya Kulkarni]

On 4/14/19 7:11 PM, Ming Lei wrote:
> On Mon, Apr 15, 2019 at 10:02 AM Chaitanya Kulkarni
> <Chaitanya.Kulkarni@wdc.com> wrote:
>>
>> Thanks for the patch Ming. Couple of comments below.
>> On 4/14/19 6:22 PM, Ming Lei wrote:
>>> It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
>>> even worse it may trigger IO timeout on nvme-loop.
>>>
>>> Eric and Dave replied that it is because of too small log size on small
>>> disk.
>>>
>>> So pass '-l size=32m' to avoid the issue.
>>>
>>> With this patch, nvme/012 can be completed in one minute.
>>>
>> Then we should set the QUICK=1 if its taking shorter time.
> 
> That should have been a QUICK test, it is fine to set it, but it shouldn't
> belong to this patch.
> 
>>
>>> Cc: Eric Sandeen <esandeen@redhat.com>
>>> Cc: Dave Chinner <dchinner@redhat.com>
>>> Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
>>> Cc: linux-xfs@vger.kernel.org
>>> Signed-off-by: Ming Lei <ming.lei@redhat.com>
>>> ---
>>>    tests/nvme/012 | 2 +-
>>>    tests/nvme/013 | 2 +-
>>>    2 files changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/tests/nvme/012 b/tests/nvme/012
>>> index 9a6801511df7..d7a8751ec752 100755
>>> --- a/tests/nvme/012
>>> +++ b/tests/nvme/012
>>> @@ -46,7 +46,7 @@ test() {
>>>
>>>        umount ${mount_dir} > /dev/null 2>&1
>>>
>>> -     mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>> +     mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>>
>> As a part of this series lets move this to the helper mkfs in the
>> nvme/rc and use that call in all the file-backed ns related testcases.
>> Let me know if you want me to do that or you would like to do that as a
>> part of this series. I'm okay with anything.
> 
> I am fine with either way.
> 
Okay, let's get this in, will send out the refactoring.
> Thanks,
> Ming Lei
> 

Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO

[Eric Sandeen]

On 4/14/19 9:01 PM, Chaitanya Kulkarni wrote:
> Thanks for the patch Ming. Couple of comments below.
> On 4/14/19 6:22 PM, Ming Lei wrote:
>> It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
>> even worse it may trigger IO timeout on nvme-loop.
>>
>> Eric and Dave replied that it is because of too small log size on small
>> disk.
>>
>> So pass '-l size=32m' to avoid the issue.
>>
>> With this patch, nvme/012 can be completed in one minute.
>>
> Then we should set the QUICK=1 if its taking shorter time.
> 
>> Cc: Eric Sandeen <esandeen@redhat.com>
>> Cc: Dave Chinner <dchinner@redhat.com>
>> Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
>> Cc: linux-xfs@vger.kernel.org
>> Signed-off-by: Ming Lei <ming.lei@redhat.com>
>> ---
>>   tests/nvme/012 | 2 +-
>>   tests/nvme/013 | 2 +-
>>   2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/nvme/012 b/tests/nvme/012
>> index 9a6801511df7..d7a8751ec752 100755
>> --- a/tests/nvme/012
>> +++ b/tests/nvme/012
>> @@ -46,7 +46,7 @@ test() {
>>   
>>   	umount ${mount_dir} > /dev/null 2>&1
>>   
>> -	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>> +	mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>   
> As a part of this series lets move this to the helper mkfs in the 
> nvme/rc and use that call in all the file-backed ns related testcases.
> Let me know if you want me to do that or you would like to do that as a 
> part of this series. I'm okay with anything.

We also discussed making the log size larger by default, on small fielsystems,
but that won't help you yet.

My only caution is that manually setting the log to 32m may actually create
a smaller than default log if your backing file happens to be very large.
Just something to consider.  If the backing files are typically around the
4G size of this test (?) then 32m seems reasonable, 128m certainly would
not hurt.

-Eric

>>   	mount /dev/"${nvmedev}n1" "${mount_dir}"
>>   
>> diff --git a/tests/nvme/013 b/tests/nvme/013
>> index e346cd4baa35..bc435f752dca 100755
>> --- a/tests/nvme/013
>> +++ b/tests/nvme/013
>> @@ -44,7 +44,7 @@ test() {
>>   
>>   	umount ${mount_dir} > /dev/null 2>&1
>>   
>> -	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>> +	mkfs.xfs -l size=32m  -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>   
> Extra space here but can be ignored once we move mkfs to the helper call.
>>   	mount /dev/"${nvmedev}n1" "${mount_dir}"
>>   
>>
> 

Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO

[Ming Lei]

On Mon, Apr 15, 2019 at 09:46:02AM -0400, Eric Sandeen wrote:
> On 4/14/19 9:01 PM, Chaitanya Kulkarni wrote:
> > Thanks for the patch Ming. Couple of comments below.
> > On 4/14/19 6:22 PM, Ming Lei wrote:
> >> It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
> >> even worse it may trigger IO timeout on nvme-loop.
> >>
> >> Eric and Dave replied that it is because of too small log size on small
> >> disk.
> >>
> >> So pass '-l size=32m' to avoid the issue.
> >>
> >> With this patch, nvme/012 can be completed in one minute.
> >>
> > Then we should set the QUICK=1 if its taking shorter time.
> > 
> >> Cc: Eric Sandeen <esandeen@redhat.com>
> >> Cc: Dave Chinner <dchinner@redhat.com>
> >> Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
> >> Cc: linux-xfs@vger.kernel.org
> >> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> >> ---
> >>   tests/nvme/012 | 2 +-
> >>   tests/nvme/013 | 2 +-
> >>   2 files changed, 2 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/tests/nvme/012 b/tests/nvme/012
> >> index 9a6801511df7..d7a8751ec752 100755
> >> --- a/tests/nvme/012
> >> +++ b/tests/nvme/012
> >> @@ -46,7 +46,7 @@ test() {
> >>   
> >>   	umount ${mount_dir} > /dev/null 2>&1
> >>   
> >> -	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
> >> +	mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
> >>   
> > As a part of this series lets move this to the helper mkfs in the 
> > nvme/rc and use that call in all the file-backed ns related testcases.
> > Let me know if you want me to do that or you would like to do that as a 
> > part of this series. I'm okay with anything.
> 
> We also discussed making the log size larger by default, on small fielsystems,
> but that won't help you yet.
> 
> My only caution is that manually setting the log to 32m may actually create
> a smaller than default log if your backing file happens to be very large.
> Just something to consider.  If the backing files are typically around the
> 4G size of this test (?) then 32m seems reasonable, 128m certainly would
> not hurt.

In the two tests, the device size is 1G, and looks 32m log size works
just fine. 

thanks,
Ming

Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO

[Omar Sandoval]

On Mon, Apr 15, 2019 at 09:22:29AM +0800, Ming Lei wrote:
> It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
> even worse it may trigger IO timeout on nvme-loop.
> 
> Eric and Dave replied that it is because of too small log size on small
> disk.
> 
> So pass '-l size=32m' to avoid the issue.
> 
> With this patch, nvme/012 can be completed in one minute.

Nice:

nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]
    runtime  402.605s  ...  27.431s

Thanks, Ming, applied.