Proper way to shut down FIO in Linux

Proper way to shut down FIO in Linux

[Matt Freel]

I'm using FIO to run IOs to a number of block devices.  I'm looking for
the proper way to shut down all the threads that are spawned.

I'm doing the following:

/usr/bin/pkill --signal INT fio

Most of the time this works fine, but I do have cases where some of the
FIO processes remain open.  Eventually I get a 300s timeout and then
they're killed.

A couple questions:

1.	When these threads have to be ungracefully killed, do the results
still get counted in the output file?
a.	I'm using JSON output file
2.	Is there a better way I should be killing all the threads?

Re: Proper way to shut down FIO in Linux

[Erwan Velu]

Hey,

Why do you want to kill fio ? That sounds weird to me.

If you need to run your benchmark on constant time then use time_based & runtime instructions.

----- Mail original -----
De: "Matt Freel" <matt.freel@broadcom.com>
À: fio@vger.kernel.org
Envoyé: Mardi 13 Mars 2018 19:56:10
Objet: Proper way to shut down FIO in Linux

I'm using FIO to run IOs to a number of block devices.  I'm looking for
the proper way to shut down all the threads that are spawned.

I'm doing the following:

/usr/bin/pkill --signal INT fio

Most of the time this works fine, but I do have cases where some of the
FIO processes remain open.  Eventually I get a 300s timeout and then
they're killed.

A couple questions:

1.	When these threads have to be ungracefully killed, do the results
still get counted in the output file?
a.	I'm using JSON output file
2.	Is there a better way I should be killing all the threads?
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Proper way to shut down FIO in Linux

[Matt Freel]

I'm using it to generate IO -- not necessarily as a benchmark.  I'm running
IO, taking some other measurements, then killing it to kick off a different
workload.  The time it needs to run is not constant -- it depends on a bunch
of different things.

-----Original Message-----
From: Erwan Velu <evelu@redhat.com>
Sent: Wednesday, March 14, 2018 12:47 AM
To: Matt Freel <matt.freel@broadcom.com>
Cc: fio@vger.kernel.org
Subject: Re: Proper way to shut down FIO in Linux

Hey,

Why do you want to kill fio ? That sounds weird to me.

If you need to run your benchmark on constant time then use time_based &
runtime instructions.

----- Mail original -----
De: "Matt Freel" <matt.freel@broadcom.com>
À: fio@vger.kernel.org
Envoyé: Mardi 13 Mars 2018 19:56:10
Objet: Proper way to shut down FIO in Linux

I'm using FIO to run IOs to a number of block devices.  I'm looking for the
proper way to shut down all the threads that are spawned.

I'm doing the following:

/usr/bin/pkill --signal INT fio

Most of the time this works fine, but I do have cases where some of the FIO
processes remain open.  Eventually I get a 300s timeout and then they're
killed.

A couple questions:

1.	When these threads have to be ungracefully killed, do the results
still get counted in the output file?
a.	I'm using JSON output file
2.	Is there a better way I should be killing all the threads?
--
To unsubscribe from this list: send the line "unsubscribe fio" in the body
of a message to majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html

Re: Proper way to shut down FIO in Linux

[Sitsofe Wheeler]

Matt,

You didn't include the fio line you are running which cause the
problem. That sort of thing is useful extra information - see
https://github.com/axboe/fio/blob/master/REPORTING-BUGS .

1. Which file are we talking about? For example if a job is abandoned
due to it hanging you start skipping past code like this:

backend.c
2497         if (!fio_abort) {
2498                 __show_run_stats();
2499                 if (write_bw_log) {
2500                         for (i = 0; i < DDIR_RWDIR_CNT; i++) {
2501                                 struct io_log *log = agg_io_log[i];
2502
2503                                 flush_log(log, false);
2504                                 free_log(log);
2505                         }
2506                 }
2507         }

So that's an example of a log file won't necessarily be flushed if a
job is believed to be stuck. There are also logs that may not be
written if the job itself is stuck in the running state:

1525 static void *thread_main(void *data)
1526 {
[...]
1755
1756         while (keep_running(td)) {
[...]
1853         }
1854
[...]
1882         td_writeout_logs(td, true);

So I wouldn't depend on all the logs being correct if you have stuck
jobs that end up being abandoned. With regard to a) the general stats
might be OK but you're going to potentially have data at the end of
them that's indeterminate depending on why the job became stuck and
since we don't know the thread is dead the "final" stats might be
pulled while the job is in the middle of changing them...

2. What you're doing will send a kill to all fio processes which may
mean that when in process mode fio's child jobs get signalled before
the main job. You might things get better if you just the main fio
backend thread and let that then send the kill message to the other
processes.

Nonetheless, it would be useful to know the minimal fio command line
that generates the hangs you are referring to. If we had that then we
might be able to make things more robust by debugging the problem.

On 14 March 2018 at 15:07, Matt Freel <matt.freel@broadcom.com> wrote:
> I'm using it to generate IO -- not necessarily as a benchmark.  I'm running
> IO, taking some other measurements, then killing it to kick off a different
> workload.  The time it needs to run is not constant -- it depends on a bunch
> of different things.
>
> -----Original Message-----
> From: Erwan Velu <evelu@redhat.com>
> Sent: Wednesday, March 14, 2018 12:47 AM
> To: Matt Freel <matt.freel@broadcom.com>
> Cc: fio@vger.kernel.org
> Subject: Re: Proper way to shut down FIO in Linux
>
> Hey,
>
> Why do you want to kill fio ? That sounds weird to me.
>
> If you need to run your benchmark on constant time then use time_based &
> runtime instructions.
>
> ----- Mail original -----
> De: "Matt Freel" <matt.freel@broadcom.com>
> À: fio@vger.kernel.org
> Envoyé: Mardi 13 Mars 2018 19:56:10
> Objet: Proper way to shut down FIO in Linux
>
> I'm using FIO to run IOs to a number of block devices.  I'm looking for the
> proper way to shut down all the threads that are spawned.
>
> I'm doing the following:
>
> /usr/bin/pkill --signal INT fio
>
> Most of the time this works fine, but I do have cases where some of the FIO
> processes remain open.  Eventually I get a 300s timeout and then they're
> killed.
>
> A couple questions:
>
> 1.      When these threads have to be ungracefully killed, do the results
> still get counted in the output file?
> a.      I'm using JSON output file
> 2.      Is there a better way I should be killing all the threads?

-- 
Sitsofe | http://sucs.org/~sits/

Re: Proper way to shut down FIO in Linux

[Jens Axboe]

On 3/13/18 12:56 PM, Matt Freel wrote:
> I'm using FIO to run IOs to a number of block devices.  I'm looking for
> the proper way to shut down all the threads that are spawned.
> 
> I'm doing the following:
> 
> /usr/bin/pkill --signal INT fio
> 
> Most of the time this works fine, but I do have cases where some of the
> FIO processes remain open.  Eventually I get a 300s timeout and then
> they're killed.
> 
> A couple questions:
> 
> 1.	When these threads have to be ungracefully killed, do the results
> still get counted in the output file?
> a.	I'm using JSON output file
> 2.	Is there a better way I should be killing all the threads?

You want to SIGINT the main process, not all the jobs that are running.
If you do that, you would get the same behavior as when you ctrl-c
a running job. The way you are doing it, you are randomly killing
job processes. You want to let the main thread shut things down
instead.

-- 
Jens Axboe

RE: Proper way to shut down FIO in Linux

[Matt Freel]

Thanks.  Making that change makes things work correctly.  There was some
confusion on my part with Python because I wasn't getting the PID of the FIO
process.  Once I figured out why it's working sending sigint to the main
process.

-----Original Message-----
From: Jens Axboe <axboe@kernel.dk>
Sent: Wednesday, March 21, 2018 8:37 AM
To: Matt Freel <matt.freel@broadcom.com>; fio@vger.kernel.org
Subject: Re: Proper way to shut down FIO in Linux

On 3/13/18 12:56 PM, Matt Freel wrote:
> I'm using FIO to run IOs to a number of block devices.  I'm looking
> for the proper way to shut down all the threads that are spawned.
>
> I'm doing the following:
>
> /usr/bin/pkill --signal INT fio
>
> Most of the time this works fine, but I do have cases where some of
> the FIO processes remain open.  Eventually I get a 300s timeout and
> then they're killed.
>
> A couple questions:
>
> 1.	When these threads have to be ungracefully killed, do the results
> still get counted in the output file?
> a.	I'm using JSON output file
> 2.	Is there a better way I should be killing all the threads?

You want to SIGINT the main process, not all the jobs that are running.
If you do that, you would get the same behavior as when you ctrl-c a running
job. The way you are doing it, you are randomly killing job processes. You
want to let the main thread shut things down instead.

--
Jens Axboe