Fwd: Strange I/O slowdown with 4.x kernels

Fwd: Strange I/O slowdown with 4.x kernels

[Rus]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

Hi,

After switching from 3.x to 4.x I'm observing strange filesystem (ext4
used) slowdown just after fresh boot - it is 100% reproducible. The
simple test just after boot at root fs first shows the good
performance :

dd if=/dev/zero of=/output bs=8k count=10k; rm -f /output
10240+0 records in
10240+0 records out
83886080 bytes (84 MB) copied, 0.5676 s, 150.3 MB/s

But after some small i/o stress (completed updatedb command for ex)
the i/o speed drops dramatically :

dd if=/dev/zero of=/output bs=8k count=10k; rm -f /output
10240+0 records in
10240+0 records out
83886080 bytes (84 MB) copied, 19.5703 s, 4.3 MB/s

What is interesting is that hdparm test still shows normal performance :

hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 448 MB in  3.01 seconds = 148.79 MB/sec

How to debug such thing (bissect can take too much time because of big
version range) ? The config attached, this is ordinary 32-bit system
with 32GB ECC RAM.
 The slowdown clearly triggered with last stable (4.5.2) and latest rc
(4.6-rc5) kernels. The 3.x kernels do not have such bug.
Can supply any additional info.

TIA, Rus

-- 
SfinxSoft
http://sfinxsoft.com



-- 
SfinxSoft
http://sfinxsoft.com

[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 37199 bytes --]

Re: Strange I/O slowdown with 4.x kernels

[Ming Lei]

On Fri, Apr 29, 2016 at 9:36 PM, Rus <rus@sfinxsoft.com> wrote:
> Hi,
>
> After switching from 3.x to 4.x I'm observing strange filesystem (ext4
> used) slowdown just after fresh boot - it is 100% reproducible. The
> simple test just after boot at root fs first shows the good
> performance :
>
> dd if=/dev/zero of=/output bs=8k count=10k; rm -f /output
> 10240+0 records in
> 10240+0 records out
> 83886080 bytes (84 MB) copied, 0.5676 s, 150.3 MB/s
>
> But after some small i/o stress (completed updatedb command for ex)
> the i/o speed drops dramatically :
>
> dd if=/dev/zero of=/output bs=8k count=10k; rm -f /output
> 10240+0 records in
> 10240+0 records out
> 83886080 bytes (84 MB) copied, 19.5703 s, 4.3 MB/s
>
> What is interesting is that hdparm test still shows normal performance :
>
> hdparm -t /dev/sda1
>
> /dev/sda1:
>  Timing buffered disk reads: 448 MB in  3.01 seconds = 148.79 MB/sec
>
> How to debug such thing (bissect can take too much time because of big
> version range) ? The config attached, this is ordinary 32-bit system

git bisect uses bineary search which is often quick and takes
not more than 20 iterations, for example:

$ git bisect start
$ git bisect good v3.12
$ git bisect bad HEAD
Bisecting: 93973 revisions left to test after this (roughly 17 steps)
[b0d4c5ec9730f7544772f50f84dc0dc730b1d551] mwifiex: update IEs after
AP has started

> with 32GB ECC RAM.
>  The slowdown clearly triggered with last stable (4.5.2) and latest rc
> (4.6-rc5) kernels. The 3.x kernels do not have such bug.
> Can supply any additional info.
>
> TIA, Rus
>
> --
> SfinxSoft
> http://sfinxsoft.com
>
>
>
> --
> SfinxSoft
> http://sfinxsoft.com



-- 
Ming Lei