RE: fio signal 11

[Jeff Furlong <jeff.furlong@hgst.com>]

I'd like to use the latest git version so that I can pickup patch 356014ff351c6eb69339652650af5f6af72e5c22 to fix the ramp time issue.  

I tried the latest git version fio-2.11-21-g2b762, but got a new issue:

[New LWP 120298]
[New LWP 120281]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `fio --name=test_job --ioengine=libaio --direct=1 --rw=write --iodepth=32'.
Program terminated with signal 11, Segmentation fault.
#0  __add_log_sample (iolog=iolog@entry=0x7f8376222c30, val=2230, ddir=ddir@entry=DDIR_WRITE, bs=bs@entry=0, t=t@entry=1056000, 
    offset=offset@entry=0) at stat.c:2015
2015			s->val = val;
(gdb) bt
#0  __add_log_sample (iolog=iolog@entry=0x7f8376222c30, val=2230, ddir=ddir@entry=DDIR_WRITE, bs=bs@entry=0, t=t@entry=1056000, 
    offset=offset@entry=0) at stat.c:2015
#1  0x0000000000422411 in __add_log_sample (offset=0, t=1056000, bs=0, ddir=DDIR_WRITE, val=<optimized out>, iolog=0x7f8376222c30)
    at stat.c:2004
#2  __add_stat_to_log (log_max=<optimized out>, elapsed=1056000, ddir=DDIR_WRITE, iolog=0x7f8376222c30) at stat.c:2091
#3  _add_stat_to_log (log_max=<optimized out>, elapsed=<optimized out>, iolog=0x7f8376222c30) at stat.c:2103
#4  add_log_sample (td=td@entry=0x7f8361f3d000, iolog=0x7f8376222c30, val=2230, ddir=ddir@entry=DDIR_WRITE, bs=1048576, 
    offset=offset@entry=0) at stat.c:2139
#5  0x000000000042f45f in add_log_sample (offset=0, bs=<optimized out>, ddir=DDIR_WRITE, val=<optimized out>, iolog=<optimized out>, 
    td=0x7f8361f3d000) at stat.c:2357
#6  add_iops_samples (t=0x7f835a4bae50, td=0x7f8361f3d000) at stat.c:2362
#7  calc_log_samples () at stat.c:2402
#8  0x0000000000467508 in helper_thread_main (data=0x7f8376222d90) at helper_thread.c:125
#9  0x00007f8377e7cdf5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f83779a61ad in clone () from /lib64/libc.so.6

Maybe I should try the ramp time patch only on 2.11 release?

Regards,
Jeff


-----Original Message-----
From: Jens Axboe [mailto:axboe@kernel.dk] 
Sent: Saturday, June 11, 2016 8:30 PM
To: Jeff Furlong <jeff.furlong@hgst.com>; Sitsofe Wheeler <sitsofe@gmail.com>
Cc: fio@vger.kernel.org; Jan Kara <jack@suse.cz>
Subject: Re: fio signal 11

On 06/11/2016 08:56 PM, Jens Axboe wrote:
> On 06/10/2016 12:42 PM, Jeff Furlong wrote:
>> Good point.  Here is the trace:
>>
>> [New LWP 59231]
>> [Thread debugging using libthread_db enabled] Using host libthread_db 
>> library "/lib64/libthread_db.so.1".
>> Core was generated by `fio --name=test_job --ioengine=libaio
>> --direct=1 --rw=write --iodepth=32'.
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x0000000000421e39 in regrow_log (iolog=0x7f828c0c5ad0) at
>> stat.c:1909
>> 1909        if (!cur_log) {
>>
>> (gdb) bt
>> #0  0x0000000000421e39 in regrow_log (iolog=0x7f828c0c5ad0) at
>> stat.c:1909
>> #1  0x000000000042d4df in regrow_logs (td=td@entry=0x7f8277de0000) at
>> stat.c:1965
>> #2  0x000000000040ca90 in wait_for_completions 
>> (td=td@entry=0x7f8277de0000, time=time@entry=0x7fffcfb6b300) at
>> backend.c:446
>> #3  0x000000000045ade7 in do_io (bytes_done=<synthetic pointer>,
>> td=0x7f8277de0000) at backend.c:991
>> #4  thread_main (data=data@entry=0x264d450) at backend.c:1667
>> #5  0x000000000045cfec in run_threads (sk_out=sk_out@entry=0x0) at
>> backend.c:2217
>> #6  0x000000000045d2cd in fio_backend (sk_out=sk_out@entry=0x0) at
>> backend.c:2349
>> #7  0x000000000040d09c in main (argc=22, argv=0x7fffcfb6f638, 
>> envp=<optimized out>) at fio.c:63
>
> That looks odd, thanks for reporting this. I'll see if I can get to 
> this on Monday, if not, it'll have to wait until after my vacation... 
> So while I appreciate people running -git and finding issues like 
> these before they show up in a release, might be best to revert back 
> to 2.2.11 until I can get this debugged.

I take that back - continue using -git! Just pull a fresh copy, should be fixed now.

Jan, the reporter is right, 2.11 works and -git does not. So I just ran a quick bisect, changing the logging from every second to every 100ms to make it reproduce faster. I don't have time to look into why yet, so I just reverted the commit.

commit d7982dd0ab2a1a315b5f9859c67a02414ce6274f
Author: Jan Kara <jack@suse.cz>
Date:   Tue May 24 17:03:21 2016 +0200

     fio: Simplify forking of processes

--
Jens Axboe

Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.