* XFS mounted with 'discard' option - deleting fio test files slow
@ 2017-09-07 18:22 Lu, Qian
2017-09-07 23:34 ` Dave Chinner
2017-09-08 17:17 ` Lu, Qian
0 siblings, 2 replies; 6+ messages in thread
From: Lu, Qian @ 2017-09-07 18:22 UTC (permalink / raw)
To: linux-xfs; +Cc: linux-kernel
Hi XFS mailing list,
Recently we received a bug report in the XFS filesystem with 'discard' option. I have been able to reproduce this issue. I used XFS filesystem to format NVMe SSD and mounted with 'discard' option. When I tried to delete the test fio files, the session took long time. This issue is based on Linux 4.9 stable tree. I have also repeated this test with Linux 4.13, 4.12, and we are facing the same issue. Tests were repeated several times and it was consistent.
Please see details below.
1. Kernel version: Linux ip-172-31-6-243 4.9.32-15.41.amzn1.x86_64 #1 SMP Thu Jun 22 06:20:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
--> Interrupt with Ctrl+C
# time rm -rf fio_test_file.*
--> The session hangs and in 'blocked' state
$ dmesg
...
[ 492.329896] INFO: task rm:9231 blocked for more than 120 seconds.
...
Then I tried to backport some patches and repeated the test. The issue has been improved. Eventually 'rm' command completed but took long time(2min).
* Backported patch: 4560e78 xfs: don't block the log commit handler for discards
# fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
--> Interrupt with Ctrl+C
# time rm -rf fio_test_file.*
real 2m2.242s
user 0m0.000s
sys 0m25.524s
2. With Linux 4.12 and 4.13.0-rc1, the issue has been improved and the command is not stuck. But 'rm' command still takes long time (more than 1min). Please see details below.
Kernel version: Linux version 4.13.0-rc1+ (ec2-user@ip-172-31-21-25) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC)) #1 SMP Fri Jul 21 17:31:06 UTC 2017
# fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
--> Interrupt at about 37%
# time rm -rf fio_test_file.*
real 1m57.912s
user 0m0.000s
sys 0m28.810s
Compare this result with:
a) XFS mounted with 'nodiscard' option: It took less than 1min to run 'rm' command.
# fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
--> Interrupt at about 39%
# time rm -rf fio_test_file.*
real 0m31.176s
user 0m0.000s
sys 0m30.005s
b) EXT4 file system mounted with 'discard' option: It only took about several seconds to run 'rm' command.
# fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
--> Interrupt at about 36.2%
# time rm -rf fio_test_file.*
real 0m4.661s
user 0m0.000s
sys 0m4.657s
Please note if I wait for 'fio' command 100% done, 'rm' command took less than 1s (0m0.001s).
3. Shell script which triggers the problem
sudo su -
yum install xfsprogs fio -y
mkfs.xfs -K -f -s size=4096 /dev/nvme0n1
mkdir -p /media/disk1
mount -o discard /dev/nvme0n1 /media/disk1
cd /media/disk1/
fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
# Interrupt with Ctrl+C
time rm -rf fio_test_file.*
Best Regards,
Qian Lu
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: XFS mounted with 'discard' option - deleting fio test files slow
2017-09-07 18:22 XFS mounted with 'discard' option - deleting fio test files slow Lu, Qian
@ 2017-09-07 23:34 ` Dave Chinner
2017-09-08 7:16 ` Christoph Hellwig
2017-09-08 17:17 ` Lu, Qian
1 sibling, 1 reply; 6+ messages in thread
From: Dave Chinner @ 2017-09-07 23:34 UTC (permalink / raw)
To: Lu, Qian; +Cc: linux-xfs, linux-kernel
[Your email is full of wacky symbols and needs line wrapping at 72
columns]
On Thu, Sep 07, 2017 at 06:22:22PM +0000, Lu, Qian wrote:
> Recently we received a bug report in the XFS filesystem with
> 'discard' option. I have been able to reproduce this issue. I used
> XFS filesystem to format NVMe SSD and mounted with 'discard'
> option. When I tried to delete the test fio files, the session
> took long time. This issue is based on Linux 4.9 stable tree. I
> have also repeated this test with Linux 4.13, 4.12, and we are
> facing the same issue. Tests were repeated several times and it
> was consistent.
.....
>
> # fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
> --> Interrupt with Ctrl+C
> # time rm -rf fio_test_file.*
> real 2m2.242s
> user 0m0.000s
> sys 0m25.524s
4k random write with direct IO. 5GB file. Probably got a million 4k
extents in it. Which means XFS has sent a million tiny 4k discards
to the device. Run 'xfs_bmap -vvp fio_test_file.*' to confirm.
Don't use "-o discard" if you care about performance.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: XFS mounted with 'discard' option - deleting fio test files slow
2017-09-07 23:34 ` Dave Chinner
@ 2017-09-08 7:16 ` Christoph Hellwig
0 siblings, 0 replies; 6+ messages in thread
From: Christoph Hellwig @ 2017-09-08 7:16 UTC (permalink / raw)
To: Dave Chinner; +Cc: Lu, Qian, linux-xfs, linux-kernel
On Fri, Sep 08, 2017 at 09:34:02AM +1000, Dave Chinner wrote:
> 4k random write with direct IO. 5GB file. Probably got a million 4k
> extents in it. Which means XFS has sent a million tiny 4k discards
> to the device. Run 'xfs_bmap -vvp fio_test_file.*' to confirm.
>
> Don't use "-o discard" if you care about performance.
And your device sucks :)
Some numbers here:
https://events.linuxfoundation.org/sites/events/files/slides/discard_0.pdf
Discard will cause extra work, and for worst case sceanarios lots of
it. But some device handle that extra load very well.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: XFS mounted with 'discard' option - deleting fio test files slow
2017-09-07 18:22 XFS mounted with 'discard' option - deleting fio test files slow Lu, Qian
2017-09-07 23:34 ` Dave Chinner
@ 2017-09-08 17:17 ` Lu, Qian
2017-09-08 23:21 ` Hao Chen
1 sibling, 1 reply; 6+ messages in thread
From: Lu, Qian @ 2017-09-08 17:17 UTC (permalink / raw)
To: linux-xfs, darrick.wong; +Cc: linux-kernel, amazon-linux-kernel
Adding amazon-linux-kernel@amazon.com
On 9/7/17, 11:22 AM, "Lu, Qian" <luqia@amazon.com> wrote:
Hi XFS mailing list,
Recently we received a bug report in the XFS filesystem with 'discard' option. I have been able to reproduce this issue. I used XFS filesystem to format NVMe SSD and mounted with 'discard' option. When I tried to delete the test fio files, the session took long time. This issue is based on Linux 4.9 stable tree. I have also repeated this test with Linux 4.13, 4.12, and we are facing the same issue. Tests were repeated several times and it was consistent.
Please see details below.
1. Kernel version: Linux ip-172-31-6-243 4.9.32-15.41.amzn1.x86_64 #1 SMP Thu Jun 22 06:20:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
--> Interrupt with Ctrl+C
# time rm -rf fio_test_file.*
--> The session hangs and in 'blocked' state
$ dmesg
...
[ 492.329896] INFO: task rm:9231 blocked for more than 120 seconds.
...
Then I tried to backport some patches and repeated the test. The issue has been improved. Eventually 'rm' command completed but took long time(2min).
* Backported patch: 4560e78 xfs: don't block the log commit handler for discards
# fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
--> Interrupt with Ctrl+C
# time rm -rf fio_test_file.*
real 2m2.242s
user 0m0.000s
sys 0m25.524s
2. With Linux 4.12 and 4.13.0-rc1, the issue has been improved and the command is not stuck. But 'rm' command still takes long time (more than 1min). Please see details below.
Kernel version: Linux version 4.13.0-rc1+ (ec2-user@ip-172-31-21-25) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC)) #1 SMP Fri Jul 21 17:31:06 UTC 2017
# fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
--> Interrupt at about 37%
# time rm -rf fio_test_file.*
real 1m57.912s
user 0m0.000s
sys 0m28.810s
Compare this result with:
a) XFS mounted with 'nodiscard' option: It took less than 1min to run 'rm' command.
# fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
--> Interrupt at about 39%
# time rm -rf fio_test_file.*
real 0m31.176s
user 0m0.000s
sys 0m30.005s
b) EXT4 file system mounted with 'discard' option: It only took about several seconds to run 'rm' command.
# fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
--> Interrupt at about 36.2%
# time rm -rf fio_test_file.*
real 0m4.661s
user 0m0.000s
sys 0m4.657s
Please note if I wait for 'fio' command 100% done, 'rm' command took less than 1s (0m0.001s).
3. Shell script which triggers the problem
sudo su -
yum install xfsprogs fio -y
mkfs.xfs -K -f -s size=4096 /dev/nvme0n1
mkdir -p /media/disk1
mount -o discard /dev/nvme0n1 /media/disk1
cd /media/disk1/
fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
# Interrupt with Ctrl+C
time rm -rf fio_test_file.*
Best Regards,
Qian Lu
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: XFS mounted with 'discard' option - deleting fio test files slow
2017-09-08 17:17 ` Lu, Qian
@ 2017-09-08 23:21 ` Hao Chen
2017-09-11 6:42 ` Christoph Hellwig
0 siblings, 1 reply; 6+ messages in thread
From: Hao Chen @ 2017-09-08 23:21 UTC (permalink / raw)
To: Lu, Qian; +Cc: linux-xfs, darrick.wong, linux-kernel, amazon-linux-kernel
This is expected. To get a better performance of XFS on NVMe drive, it
is advised not use use the realtime discard option.
https://www.intel.com/content/dam/support/us/en/documents/ssdc/data-center-ssds/Intel_Linux_NVMe_Guide_330602-002.pdf
Hao
==== Lu, Qian <luqia@amazon.com> [2017-09-08 17:17:07 +0000] ====
> Adding amazon-linux-kernel@amazon.com
>
> On 9/7/17, 11:22 AM, "Lu, Qian" <luqia@amazon.com> wrote:
>
> Hi XFS mailing list,
>
> Recently we received a bug report in the XFS filesystem with 'discard' option. I have been able to reproduce this issue. I used XFS filesystem to format NVMe SSD and mounted with 'discard' option. When I tried to delete the test fio files, the session took long time. This issue is based on Linux 4.9 stable tree. I have also repeated this test with Linux 4.13, 4.12, and we are facing the same issue. Tests were repeated several times and it was consistent.
>
> Please see details below.
>
> 1. Kernel version: Linux ip-172-31-6-243 4.9.32-15.41.amzn1.x86_64 #1 SMP Thu Jun 22 06:20:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
>
> # fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
> --> Interrupt with Ctrl+C
> # time rm -rf fio_test_file.*
> --> The session hangs and in 'blocked' state
> $ dmesg
> ...
> [ 492.329896] INFO: task rm:9231 blocked for more than 120 seconds.
> ...
>
> Then I tried to backport some patches and repeated the test. The issue has been improved. Eventually 'rm' command completed but took long time(2min).
>
> * Backported patch: 4560e78 xfs: don't block the log commit handler for discards
>
> # fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
> --> Interrupt with Ctrl+C
> # time rm -rf fio_test_file.*
> real 2m2.242s
> user 0m0.000s
> sys 0m25.524s
>
>
> 2. With Linux 4.12 and 4.13.0-rc1, the issue has been improved and the command is not stuck. But 'rm' command still takes long time (more than 1min). Please see details below.
>
> Kernel version: Linux version 4.13.0-rc1+ (ec2-user@ip-172-31-21-25) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC)) #1 SMP Fri Jul 21 17:31:06 UTC 2017
>
> # fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
> --> Interrupt at about 37%
> # time rm -rf fio_test_file.*
> real 1m57.912s
> user 0m0.000s
> sys 0m28.810s
>
> Compare this result with:
> a) XFS mounted with 'nodiscard' option: It took less than 1min to run 'rm' command.
>
> # fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
> --> Interrupt at about 39%
> # time rm -rf fio_test_file.*
> real 0m31.176s
> user 0m0.000s
> sys 0m30.005s
>
> b) EXT4 file system mounted with 'discard' option: It only took about several seconds to run 'rm' command.
>
> # fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
> --> Interrupt at about 36.2%
> # time rm -rf fio_test_file.*
> real 0m4.661s
> user 0m0.000s
> sys 0m4.657s
>
> Please note if I wait for 'fio' command 100% done, 'rm' command took less than 1s (0m0.001s).
>
>
> 3. Shell script which triggers the problem
>
> sudo su -
> yum install xfsprogs fio -y
> mkfs.xfs -K -f -s size=4096 /dev/nvme0n1
> mkdir -p /media/disk1
> mount -o discard /dev/nvme0n1 /media/disk1
> cd /media/disk1/
> fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=5G --numjobs=8 --group_reporting
> # Interrupt with Ctrl+C
> time rm -rf fio_test_file.*
>
>
> Best Regards,
> Qian Lu
>
>
>
>
--
Hao Chen
Cordite SDM@Cupertino
408-480-6632
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: XFS mounted with 'discard' option - deleting fio test files slow
2017-09-08 23:21 ` Hao Chen
@ 2017-09-11 6:42 ` Christoph Hellwig
0 siblings, 0 replies; 6+ messages in thread
From: Christoph Hellwig @ 2017-09-11 6:42 UTC (permalink / raw)
To: Hao Chen
Cc: Lu, Qian, linux-xfs, darrick.wong, linux-kernel, amazon-linux-kernel
On Fri, Sep 08, 2017 at 04:21:52PM -0700, Hao Chen wrote:
> This is expected. To get a better performance of XFS on NVMe drive, it
> is advised not use use the realtime discard option.
>
> https://www.intel.com/content/dam/support/us/en/documents/ssdc/data-center-ssds/Intel_Linux_NVMe_Guide_330602-002.pdf
Again, this will depend on your device. Intel in particular was the
example in my presentation for horrible performance of the NVMe
discard support - other vendors implement much better discard support,
so you'll see much better results.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-09-11 6:42 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-07 18:22 XFS mounted with 'discard' option - deleting fio test files slow Lu, Qian
2017-09-07 23:34 ` Dave Chinner
2017-09-08 7:16 ` Christoph Hellwig
2017-09-08 17:17 ` Lu, Qian
2017-09-08 23:21 ` Hao Chen
2017-09-11 6:42 ` Christoph Hellwig
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).