* Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO
2019-04-15 2:01 ` Chaitanya Kulkarni
@ 2019-04-15 2:06 ` Chaitanya Kulkarni
2019-04-15 2:10 ` Ming Lei
2019-04-15 13:46 ` Eric Sandeen
2 siblings, 0 replies; 8+ messages in thread
From: Chaitanya Kulkarni @ 2019-04-15 2:06 UTC (permalink / raw)
To: Ming Lei, Omar Sandoval
Cc: linux-block, Eric Sandeen, Dave Chinner, Darrick J. Wong, linux-xfs
On 4/14/19 7:01 PM, Chaitanya Kulkarni wrote:
> Thanks for the patch Ming. Couple of comments below.
> On 4/14/19 6:22 PM, Ming Lei wrote:
>> It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
>> even worse it may trigger IO timeout on nvme-loop.
>>
>> Eric and Dave replied that it is because of too small log size on small
>> disk.
>>
>> So pass '-l size=32m' to avoid the issue.
>>
>> With this patch, nvme/012 can be completed in one minute.
>>
> Then we should set the QUICK=1 if its taking shorter time.
>
>> Cc: Eric Sandeen <esandeen@redhat.com>
>> Cc: Dave Chinner <dchinner@redhat.com>
>> Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
>> Cc: linux-xfs@vger.kernel.org
>> Signed-off-by: Ming Lei <ming.lei@redhat.com>
>> ---
>> tests/nvme/012 | 2 +-
>> tests/nvme/013 | 2 +-
>> 2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/nvme/012 b/tests/nvme/012
>> index 9a6801511df7..d7a8751ec752 100755
>> --- a/tests/nvme/012
>> +++ b/tests/nvme/012
>> @@ -46,7 +46,7 @@ test() {
>>
>> umount ${mount_dir} > /dev/null 2>&1
>>
>> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>
> As a part of this series lets move this to the helper mkfs in the
> nvme/rc and use that call in all the file-backed ns related testcases.
> Let me know if you want me to do that or you want me to do that as a
> part of this series. I'm okay with anything.
>> mount /dev/"${nvmedev}n1" "${mount_dir}"
>>
>> diff --git a/tests/nvme/013 b/tests/nvme/013
>> index e346cd4baa35..bc435f752dca 100755
>> --- a/tests/nvme/013
>> +++ b/tests/nvme/013
>> @@ -44,7 +44,7 @@ test() {
>>
>> umount ${mount_dir} > /dev/null 2>&1
>>
>> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>
> Extra space here but can be ignored once we move mkfs to the helper call.
>> mount /dev/"${nvmedev}n1" "${mount_dir}"
>>
>>
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO
2019-04-15 2:01 ` Chaitanya Kulkarni
2019-04-15 2:06 ` Chaitanya Kulkarni
@ 2019-04-15 2:10 ` Ming Lei
2019-04-15 2:21 ` Chaitanya Kulkarni
2019-04-15 13:46 ` Eric Sandeen
2 siblings, 1 reply; 8+ messages in thread
From: Ming Lei @ 2019-04-15 2:10 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: Ming Lei, Omar Sandoval, linux-block, Eric Sandeen, Dave Chinner,
Darrick J. Wong, linux-xfs
On Mon, Apr 15, 2019 at 10:02 AM Chaitanya Kulkarni
<Chaitanya.Kulkarni@wdc.com> wrote:
>
> Thanks for the patch Ming. Couple of comments below.
> On 4/14/19 6:22 PM, Ming Lei wrote:
> > It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
> > even worse it may trigger IO timeout on nvme-loop.
> >
> > Eric and Dave replied that it is because of too small log size on small
> > disk.
> >
> > So pass '-l size=32m' to avoid the issue.
> >
> > With this patch, nvme/012 can be completed in one minute.
> >
> Then we should set the QUICK=1 if its taking shorter time.
That should have been a QUICK test, it is fine to set it, but it shouldn't
belong to this patch.
>
> > Cc: Eric Sandeen <esandeen@redhat.com>
> > Cc: Dave Chinner <dchinner@redhat.com>
> > Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
> > Cc: linux-xfs@vger.kernel.org
> > Signed-off-by: Ming Lei <ming.lei@redhat.com>
> > ---
> > tests/nvme/012 | 2 +-
> > tests/nvme/013 | 2 +-
> > 2 files changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/tests/nvme/012 b/tests/nvme/012
> > index 9a6801511df7..d7a8751ec752 100755
> > --- a/tests/nvme/012
> > +++ b/tests/nvme/012
> > @@ -46,7 +46,7 @@ test() {
> >
> > umount ${mount_dir} > /dev/null 2>&1
> >
> > - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
> > + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
> >
> As a part of this series lets move this to the helper mkfs in the
> nvme/rc and use that call in all the file-backed ns related testcases.
> Let me know if you want me to do that or you would like to do that as a
> part of this series. I'm okay with anything.
I am fine with either way.
Thanks,
Ming Lei
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO
2019-04-15 2:10 ` Ming Lei
@ 2019-04-15 2:21 ` Chaitanya Kulkarni
0 siblings, 0 replies; 8+ messages in thread
From: Chaitanya Kulkarni @ 2019-04-15 2:21 UTC (permalink / raw)
To: Ming Lei
Cc: Ming Lei, Omar Sandoval, linux-block, Eric Sandeen, Dave Chinner,
Darrick J. Wong, linux-xfs
On 4/14/19 7:11 PM, Ming Lei wrote:
> On Mon, Apr 15, 2019 at 10:02 AM Chaitanya Kulkarni
> <Chaitanya.Kulkarni@wdc.com> wrote:
>>
>> Thanks for the patch Ming. Couple of comments below.
>> On 4/14/19 6:22 PM, Ming Lei wrote:
>>> It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
>>> even worse it may trigger IO timeout on nvme-loop.
>>>
>>> Eric and Dave replied that it is because of too small log size on small
>>> disk.
>>>
>>> So pass '-l size=32m' to avoid the issue.
>>>
>>> With this patch, nvme/012 can be completed in one minute.
>>>
>> Then we should set the QUICK=1 if its taking shorter time.
>
> That should have been a QUICK test, it is fine to set it, but it shouldn't
> belong to this patch.
>
>>
>>> Cc: Eric Sandeen <esandeen@redhat.com>
>>> Cc: Dave Chinner <dchinner@redhat.com>
>>> Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
>>> Cc: linux-xfs@vger.kernel.org
>>> Signed-off-by: Ming Lei <ming.lei@redhat.com>
>>> ---
>>> tests/nvme/012 | 2 +-
>>> tests/nvme/013 | 2 +-
>>> 2 files changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/tests/nvme/012 b/tests/nvme/012
>>> index 9a6801511df7..d7a8751ec752 100755
>>> --- a/tests/nvme/012
>>> +++ b/tests/nvme/012
>>> @@ -46,7 +46,7 @@ test() {
>>>
>>> umount ${mount_dir} > /dev/null 2>&1
>>>
>>> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>>
>> As a part of this series lets move this to the helper mkfs in the
>> nvme/rc and use that call in all the file-backed ns related testcases.
>> Let me know if you want me to do that or you would like to do that as a
>> part of this series. I'm okay with anything.
>
> I am fine with either way.
>
Okay, let's get this in, will send out the refactoring.
> Thanks,
> Ming Lei
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO
2019-04-15 2:01 ` Chaitanya Kulkarni
2019-04-15 2:06 ` Chaitanya Kulkarni
2019-04-15 2:10 ` Ming Lei
@ 2019-04-15 13:46 ` Eric Sandeen
2019-04-16 0:06 ` Ming Lei
2 siblings, 1 reply; 8+ messages in thread
From: Eric Sandeen @ 2019-04-15 13:46 UTC (permalink / raw)
To: Chaitanya Kulkarni, Ming Lei, Omar Sandoval
Cc: linux-block, Dave Chinner, Darrick J. Wong, linux-xfs
On 4/14/19 9:01 PM, Chaitanya Kulkarni wrote:
> Thanks for the patch Ming. Couple of comments below.
> On 4/14/19 6:22 PM, Ming Lei wrote:
>> It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
>> even worse it may trigger IO timeout on nvme-loop.
>>
>> Eric and Dave replied that it is because of too small log size on small
>> disk.
>>
>> So pass '-l size=32m' to avoid the issue.
>>
>> With this patch, nvme/012 can be completed in one minute.
>>
> Then we should set the QUICK=1 if its taking shorter time.
>
>> Cc: Eric Sandeen <esandeen@redhat.com>
>> Cc: Dave Chinner <dchinner@redhat.com>
>> Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
>> Cc: linux-xfs@vger.kernel.org
>> Signed-off-by: Ming Lei <ming.lei@redhat.com>
>> ---
>> tests/nvme/012 | 2 +-
>> tests/nvme/013 | 2 +-
>> 2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/nvme/012 b/tests/nvme/012
>> index 9a6801511df7..d7a8751ec752 100755
>> --- a/tests/nvme/012
>> +++ b/tests/nvme/012
>> @@ -46,7 +46,7 @@ test() {
>>
>> umount ${mount_dir} > /dev/null 2>&1
>>
>> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>
> As a part of this series lets move this to the helper mkfs in the
> nvme/rc and use that call in all the file-backed ns related testcases.
> Let me know if you want me to do that or you would like to do that as a
> part of this series. I'm okay with anything.
We also discussed making the log size larger by default, on small fielsystems,
but that won't help you yet.
My only caution is that manually setting the log to 32m may actually create
a smaller than default log if your backing file happens to be very large.
Just something to consider. If the backing files are typically around the
4G size of this test (?) then 32m seems reasonable, 128m certainly would
not hurt.
-Eric
>> mount /dev/"${nvmedev}n1" "${mount_dir}"
>>
>> diff --git a/tests/nvme/013 b/tests/nvme/013
>> index e346cd4baa35..bc435f752dca 100755
>> --- a/tests/nvme/013
>> +++ b/tests/nvme/013
>> @@ -44,7 +44,7 @@ test() {
>>
>> umount ${mount_dir} > /dev/null 2>&1
>>
>> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
>>
> Extra space here but can be ignored once we move mkfs to the helper call.
>> mount /dev/"${nvmedev}n1" "${mount_dir}"
>>
>>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] nvme/012 & 013: avoid extremely slow xfs IO
2019-04-15 13:46 ` Eric Sandeen
@ 2019-04-16 0:06 ` Ming Lei
0 siblings, 0 replies; 8+ messages in thread
From: Ming Lei @ 2019-04-16 0:06 UTC (permalink / raw)
To: sandeen
Cc: Chaitanya Kulkarni, Omar Sandoval, linux-block, Dave Chinner,
Darrick J. Wong, linux-xfs
On Mon, Apr 15, 2019 at 09:46:02AM -0400, Eric Sandeen wrote:
> On 4/14/19 9:01 PM, Chaitanya Kulkarni wrote:
> > Thanks for the patch Ming. Couple of comments below.
> > On 4/14/19 6:22 PM, Ming Lei wrote:
> >> It is observed that nvme/012 may take ~17 minutes to complete on aarch64,
> >> even worse it may trigger IO timeout on nvme-loop.
> >>
> >> Eric and Dave replied that it is because of too small log size on small
> >> disk.
> >>
> >> So pass '-l size=32m' to avoid the issue.
> >>
> >> With this patch, nvme/012 can be completed in one minute.
> >>
> > Then we should set the QUICK=1 if its taking shorter time.
> >
> >> Cc: Eric Sandeen <esandeen@redhat.com>
> >> Cc: Dave Chinner <dchinner@redhat.com>
> >> Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
> >> Cc: linux-xfs@vger.kernel.org
> >> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> >> ---
> >> tests/nvme/012 | 2 +-
> >> tests/nvme/013 | 2 +-
> >> 2 files changed, 2 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/tests/nvme/012 b/tests/nvme/012
> >> index 9a6801511df7..d7a8751ec752 100755
> >> --- a/tests/nvme/012
> >> +++ b/tests/nvme/012
> >> @@ -46,7 +46,7 @@ test() {
> >>
> >> umount ${mount_dir} > /dev/null 2>&1
> >>
> >> - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
> >> + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
> >>
> > As a part of this series lets move this to the helper mkfs in the
> > nvme/rc and use that call in all the file-backed ns related testcases.
> > Let me know if you want me to do that or you would like to do that as a
> > part of this series. I'm okay with anything.
>
> We also discussed making the log size larger by default, on small fielsystems,
> but that won't help you yet.
>
> My only caution is that manually setting the log to 32m may actually create
> a smaller than default log if your backing file happens to be very large.
> Just something to consider. If the backing files are typically around the
> 4G size of this test (?) then 32m seems reasonable, 128m certainly would
> not hurt.
In the two tests, the device size is 1G, and looks 32m log size works
just fine.
thanks,
Ming
^ permalink raw reply [flat|nested] 8+ messages in thread