Re: Proper way to shut down FIO in Linux

[Sitsofe Wheeler <sitsofe@gmail.com>]

Matt,

You didn't include the fio line you are running which cause the
problem. That sort of thing is useful extra information - see
https://github.com/axboe/fio/blob/master/REPORTING-BUGS .

1. Which file are we talking about? For example if a job is abandoned
due to it hanging you start skipping past code like this:

backend.c
2497         if (!fio_abort) {
2498                 __show_run_stats();
2499                 if (write_bw_log) {
2500                         for (i = 0; i < DDIR_RWDIR_CNT; i++) {
2501                                 struct io_log *log = agg_io_log[i];
2502
2503                                 flush_log(log, false);
2504                                 free_log(log);
2505                         }
2506                 }
2507         }

So that's an example of a log file won't necessarily be flushed if a
job is believed to be stuck. There are also logs that may not be
written if the job itself is stuck in the running state:

1525 static void *thread_main(void *data)
1526 {
[...]
1755
1756         while (keep_running(td)) {
[...]
1853         }
1854
[...]
1882         td_writeout_logs(td, true);

So I wouldn't depend on all the logs being correct if you have stuck
jobs that end up being abandoned. With regard to a) the general stats
might be OK but you're going to potentially have data at the end of
them that's indeterminate depending on why the job became stuck and
since we don't know the thread is dead the "final" stats might be
pulled while the job is in the middle of changing them...

2. What you're doing will send a kill to all fio processes which may
mean that when in process mode fio's child jobs get signalled before
the main job. You might things get better if you just the main fio
backend thread and let that then send the kill message to the other
processes.

Nonetheless, it would be useful to know the minimal fio command line
that generates the hangs you are referring to. If we had that then we
might be able to make things more robust by debugging the problem.

On 14 March 2018 at 15:07, Matt Freel <matt.freel@broadcom.com> wrote:
> I'm using it to generate IO -- not necessarily as a benchmark.  I'm running
> IO, taking some other measurements, then killing it to kick off a different
> workload.  The time it needs to run is not constant -- it depends on a bunch
> of different things.
>
> -----Original Message-----
> From: Erwan Velu <evelu@redhat.com>
> Sent: Wednesday, March 14, 2018 12:47 AM
> To: Matt Freel <matt.freel@broadcom.com>
> Cc: fio@vger.kernel.org
> Subject: Re: Proper way to shut down FIO in Linux
>
> Hey,
>
> Why do you want to kill fio ? That sounds weird to me.
>
> If you need to run your benchmark on constant time then use time_based &
> runtime instructions.
>
> ----- Mail original -----
> De: "Matt Freel" <matt.freel@broadcom.com>
> À: fio@vger.kernel.org
> Envoyé: Mardi 13 Mars 2018 19:56:10
> Objet: Proper way to shut down FIO in Linux
>
> I'm using FIO to run IOs to a number of block devices.  I'm looking for the
> proper way to shut down all the threads that are spawned.
>
> I'm doing the following:
>
> /usr/bin/pkill --signal INT fio
>
> Most of the time this works fine, but I do have cases where some of the FIO
> processes remain open.  Eventually I get a 300s timeout and then they're
> killed.
>
> A couple questions:
>
> 1.      When these threads have to be ungracefully killed, do the results
> still get counted in the output file?
> a.      I'm using JSON output file
> 2.      Is there a better way I should be killing all the threads?

-- 
Sitsofe | http://sucs.org/~sits/