Re: Bug#779336: fio: calc_clat_percentiles() create unaligned access on armhf

Re: Bug#779336: fio: calc_clat_percentiles() create unaligned access on armhf

[Martin Steigerwald]

Hello Andreas, hello Jens,

thank you for your bug report.

This sounds like an upstream bug, thus I cc fio mailing list.

Jens, any idea?


For reference, Debian bug report is at:
https://bugs.debian.org/779336

Thanks,
Martin

Am Freitag, 27. Februar 2015, 12:15:44 schrieben Sie:
> Package: fio
> Version: 2.1.11-2
> Severity: important
> 
> Dear Maintainer,
> 
> when performing an extended workload with fio on a armhf box fio ends witch a
> bus error. It happens while calculating the clat values of a specific test case.
> The same error does not show up when running a simple read test.
> 
> The program output is a follows
> ---8<---
> andreas@cubieboard2 % fio four-threads-randio.fio
> bgwriter: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
> queryA: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=mmap, iodepth=1
> queryB: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=mmap, iodepth=1
> bgupdater: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
> fio-2.1.11
> Starting 4 processes
> Jobs: 1 (f=1): [_(1),r(1),_(2)] [99.6% done] [3017KB/0KB/0KB /s] [754/0/0 iops] [eta 00m:02s]    
> bgwriter: (groupid=0, jobs=1): err= 0: pid=3459: Fri Feb 27 10:40:38 2015
>   write: io=128000KB, bw=333561B/s, iops=81, runt=392947msec
>     slat (usec): min=33, max=81636, avg=141.64, stdev=1609.52
>     clat (usec): min=644, max=4307.2K, avg=392735.17, stdev=619005.07
>      lat (usec): min=748, max=4307.3K, avg=392881.68, stdev=619036.45
>     clat percentiles (usec):
>      |  1.00th=[ 1352],  5.00th=[ 2128], 10.00th=[ 3280], 20.00th=[ 5344],
>      | 30.00th=[ 7520], 40.00th=[10816], 50.00th=[19840], 60.00th=[37120],
>      | 70.00th=[428032], 80.00th=[1220608], 90.00th=[1302528], 95.00th=[1351680],
>      | 99.00th=[2539520], 99.50th=[2637824], 99.90th=[3751936], 99.95th=[3883008],
>      | 99.99th=[4292608]
>     bw (KB  /s): min=   57, max= 3233, per=96.43%, avg=353.90, stdev=310.19
>     lat (usec) : 750=0.08%, 1000=0.24%
>     lat (msec) : 2=4.10%, 4=8.83%, 10=25.07%, 20=11.80%, 50=13.58%
>     lat (msec) : 100=4.88%, 250=0.43%, 500=2.27%, 750=1.83%, 1000=2.26%
>     lat (msec) : 2000=22.83%, >=2000=1.78%
>   cpu          : usr=0.25%, sys=1.12%, ctx=15458, majf=0, minf=14
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
>      issued    : total=r=0/w=32000/d=0, short=r=0/w=0/d=0
>      latency   : target=0, window=0, percentile=100.00%, depth=32
> queryA: (groupid=0, jobs=1): err= 0: pid=3460: Fri Feb 27 10:40:38 2015
>   read : io=128000KB, bw=264627B/s, iops=64, runt=495307msec
>     clat (usec): min=211, max=670833, avg=15437.00, stdev=28506.89
>      lat (usec): min=213, max=670837, avg=15440.17, stdev=28506.95
> [1]    3457 bus error (core dumped)  fio four-threads-randio.fio
> fio four-threads-randio.fio  6,64s user 24,31s system 6% cpu 8:16,44 total
> ---->8---
> 
> The bus error turns out to be a alignment exception on my armhf box.
> 
> Kernel issues the following
> ---8<---
> [57912.593192] Alignment trap: not handling instruction ed937b00 at [<0001e2b0>]
> [57912.600408] Unhandled fault: alignment exception (0x221) at 0x00f625be
> --->8---
> 
> Investigating the core gives me
> ---8<---
> andreas@cubieboard2 % gdb /usr/bin/fio core
> GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "arm-linux-gnueabihf".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/bin/fio...(no debugging symbols found)...done.
> [New LWP 3457]
> [New LWP 3458]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
> Core was generated by `fio four-threads-randio.fio'.
> Program terminated with signal SIGBUS, Bus error.
> #0  0x0001e2b4 in calc_clat_percentiles ()
> (gdb) bt
> #0  0x0001e2b4 in calc_clat_percentiles ()
> #1  0x00000000 in ?? ()
> (gdb) 
> --->8---
> 
> The test runs with the following configuration
> ---8<---
> ; Four threads, two query, two writers.
> 
> [global]
> rw=randread
> size=125m
> directory=/mnt/fio-testing/data
> ioengine=libaio
> iodepth=4
> invalidate=1
> direct=1
> 
> [bgwriter]
> rw=randwrite
> iodepth=32
> 
> [queryA]
> iodepth=1
> ioengine=mmap
> direct=0
> thinktime=3
> 
> [queryB]
> iodepth=1
> ioengine=mmap
> direct=0
> thinktime=5
> 
> [bgupdater]
> rw=randrw
> iodepth=16
> thinktime=40
> size=32m
> --->8---
> 
> 
> -- System Information:
> Debian Release: 8.0
>   APT prefers testing-updates
>   APT policy: (500, 'testing-updates'), (500, 'testing')
> Architecture: armhf (armv7l)
> 
> Kernel: Linux 3.16.0-4-armmp-lpae (SMP w/2 CPU cores)
> Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to de_DE.UTF-8)
> Shell: /bin/sh linked to /bin/dash
> Init: systemd (via /run/systemd/system)
> 
> Versions of packages fio depends on:
> ii  libaio1      0.3.110-1
> ii  libc6        2.19-13
> ii  libibverbs1  1.1.8-1
> ii  librados2    0.80.7-2
> ii  librbd1      0.80.7-2
> ii  librdmacm1   1.0.19.1-1
> ii  python       2.7.8-3
> ii  zlib1g       1:1.2.8.dfsg-2+b1
> 
> fio recommends no packages.
> 
> Versions of packages fio suggests:
> pn  gnuplot  <none>
> 
> -- no debconf information


-- 
Martin Steigerwald  | Consultant / Trainer

teamix GmbH
Südwestpark 43
90449 Nürnberg

Tel.:  +49 911 30999 55 | Fax: +49 911 30999 99
mail: martin.steigerwald@teamix.de | web:  http://www.teamix.de | blog: http://blog.teamix.de

Amtsgericht Nürnberg, HRB 18320 | Geschäftsführer: Oliver Kügow, Richard Müller


***Nicht verpassen: Wissenstag - Software defined Datacenter (SDDC)***
10.03. Nürnberg | 11.03. Mainz | 24.03. Bayreuth | 25.03. München
Jetzt anmelden unter: www.teamix.de/wissenstag

Re: Bug#779336: fio: calc_clat_percentiles() create unaligned access on armhf

[Jens Axboe]

On 03/04/2015 02:12 AM, Martin Steigerwald wrote:
> Hello Andreas, hello Jens,
>
> thank you for your bug report.
>
> This sounds like an upstream bug, thus I cc fio mailing list.
>
> Jens, any idea?

This is fixed in later releases, the debian package should be updated to 
2.2.6.

-- 
Jens Axboe