--minimal does not work in client/server mode

--minimal does not work in client/server mode

[Nisha Miller]

Hi,

When running fio locally, --minimal --status-interval=1 works fine.
Terse output is dumped once a second.

When I run fio remotely like this:

fio <jobfile> --client=10.10.1.165,55555 --minimal --status-interval=1

fio dumps normal output once a second and at the end of the test it
dumps terse output just once. I've tried this using fio version 2.12
and 2.20. Both have the same issue.

I see others have faced the same problems, but I don't see any solutions.

Can anyone help me?

thanks
Nisha Miller

Re: --minimal does not work in client/server mode

[Sitsofe Wheeler]

Hi,

On 30 June 2017 at 00:51, Nisha Miller <nisha43222@gmail.com> wrote:
>
> When running fio locally, --minimal --status-interval=1 works fine.
> Terse output is dumped once a second.
>
> When I run fio remotely like this:
>
> fio <jobfile> --client=10.10.1.165,55555 --minimal --status-interval=1
>
> fio dumps normal output once a second and at the end of the test it
> dumps terse output just once. I've tried this using fio version 2.12
> and 2.20. Both have the same issue.
>
> I see others have faced the same problems, but I don't see any solutions.
>
> Can anyone help me?

Does applying https://github.com/sitsofe/fio/commit/bd5dc65b5b89b5c179f9c2ac1e0b72640d166ed9
(see https://github.com/sitsofe/fio/commit/bd5dc65b5b89b5c179f9c2ac1e0b72640d166ed9.patch
for the same thing in patch form) help? If so I'll ask Jens to pull it
via https://github.com/sitsofe/fio/tree/client_status_interval .

-- 
Sitsofe | http://sucs.org/~sits/

Re: --minimal does not work in client/server mode

[Nisha Miller]

Hi Sitsofe,

I made the change to client.c but it did not help?

thanks
Nisha

On Thu, Jun 29, 2017 at 8:17 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> On 30 June 2017 at 00:51, Nisha Miller <nisha43222@gmail.com> wrote:
>>
>> When running fio locally, --minimal --status-interval=1 works fine.
>> Terse output is dumped once a second.
>>
>> When I run fio remotely like this:
>>
>> fio <jobfile> --client=10.10.1.165,55555 --minimal --status-interval=1
>>
>> fio dumps normal output once a second and at the end of the test it
>> dumps terse output just once. I've tried this using fio version 2.12
>> and 2.20. Both have the same issue.
>>
>> I see others have faced the same problems, but I don't see any solutions.
>>
>> Can anyone help me?
>
> Does applying https://github.com/sitsofe/fio/commit/bd5dc65b5b89b5c179f9c2ac1e0b72640d166ed9
> (see https://github.com/sitsofe/fio/commit/bd5dc65b5b89b5c179f9c2ac1e0b72640d166ed9.patch
> for the same thing in patch form) help? If so I'll ask Jens to pull it
> via https://github.com/sitsofe/fio/tree/client_status_interval .
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: --minimal does not work in client/server mode

[Nisha Miller]

Sorry, my previous reply was a bit confusing.

I made the changes to client.c as per the patch. But that did not
solve the problem. I still get just one terse output at the end of the
test.

thanks
Nisha

On Thu, Jun 29, 2017 at 8:17 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> On 30 June 2017 at 00:51, Nisha Miller <nisha43222@gmail.com> wrote:
>>
>> When running fio locally, --minimal --status-interval=1 works fine.
>> Terse output is dumped once a second.
>>
>> When I run fio remotely like this:
>>
>> fio <jobfile> --client=10.10.1.165,55555 --minimal --status-interval=1
>>
>> fio dumps normal output once a second and at the end of the test it
>> dumps terse output just once. I've tried this using fio version 2.12
>> and 2.20. Both have the same issue.
>>
>> I see others have faced the same problems, but I don't see any solutions.
>>
>> Can anyone help me?
>
> Does applying https://github.com/sitsofe/fio/commit/bd5dc65b5b89b5c179f9c2ac1e0b72640d166ed9
> (see https://github.com/sitsofe/fio/commit/bd5dc65b5b89b5c179f9c2ac1e0b72640d166ed9.patch
> for the same thing in patch form) help? If so I'll ask Jens to pull it
> via https://github.com/sitsofe/fio/tree/client_status_interval .
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: --minimal does not work in client/server mode

[Sitsofe Wheeler]

Hi,

On 1 July 2017 at 00:06, Nisha Miller <nisha43222@gmail.com> wrote:
> Sorry, my previous reply was a bit confusing.
>
> I made the changes to client.c as per the patch. But that did not
> solve the problem. I still get just one terse output at the end of the
> test.

You're right - that patch doesn't solve the problem. Looking closer,
the way that client/server mode operates suggests you'd need to add a
brand new client/server command to arrange for each client to send the
cumulative information you're after to the server so it can print it
out. If so that work looks more involved than I have spare time for...

-- 
Sitsofe | http://sucs.org/~sits/

Re: --minimal does not work in client/server mode

[Nisha Miller]

If you can summarize what needs to be done to add this new command, I
can have a go at it.

thanks


On Fri, Jun 30, 2017 at 4:46 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> Hi,
>
> On 1 July 2017 at 00:06, Nisha Miller <nisha43222@gmail.com> wrote:
>> Sorry, my previous reply was a bit confusing.
>>
>> I made the changes to client.c as per the patch. But that did not
>> solve the problem. I still get just one terse output at the end of the
>> test.
>
> You're right - that patch doesn't solve the problem. Looking closer,
> the way that client/server mode operates suggests you'd need to add a
> brand new client/server command to arrange for each client to send the
> cumulative information you're after to the server so it can print it
> out. If so that work looks more involved than I have spare time for...
>
> --
> Sitsofe | http://sucs.org/~sits/

Re: --minimal does not work in client/server mode

[Sitsofe Wheeler]

Hi,

I'm guessing because I'm not familiar with the code in question:
Over in server.c (e.g.
https://github.com/axboe/fio/blob/a2c95580b468a1ddd72ecb5532aca7d94f6efa5b/server.c#L986
) you will need to add a new command (e.g. FIO_NET_CMD_SEND_STATUS).
You will need to have the client send this command to the server
periodically when the status interval times out (e.g.
https://github.com/axboe/fio/blob/a2c95580b468a1ddd72ecb5532aca7d94f6efa5b/client.c#L1929
) presumably by adding something like request_client_statuses that
does has timing logic similar to that in check_for_running_stats
https://github.com/axboe/fio/blob/b2f4b559a3bc69c384f6c83ee7d8efca7712adc8/stat.c#L1970
. That's the easy part...

When the server receives that command you need to actually summarise
the appropriate stats and send them back to the client. Hopefully the
server fio has been keeping a running cumulative totals somewhere so
you can just look at those and send them back without having to do
further calculation (regular fio seems to do the heavy lifting in a
helper thread that runs __show_running_run_stats -
https://github.com/axboe/fio/blob/b2f4b559a3bc69c384f6c83ee7d8efca7712adc8/stat.c#L1878
).

A drawback with the approach I'm outlining is that each server will
send stats back to the client so the client may print stats info for
different servers at different offsets to others but I think that
simplifies this task.

Good luck!

On 1 July 2017 at 01:05, Nisha Miller <nisha43222@gmail.com> wrote:
> If you can summarize what needs to be done to add this new command, I
> can have a go at it.
>
> On Fri, Jun 30, 2017 at 4:46 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>
>> You're right - that patch doesn't solve the problem. Looking closer,
>> the way that client/server mode operates suggests you'd need to add a
>> brand new client/server command to arrange for each client to send the
>> cumulative information you're after to the server so it can print it
>> out. If so that work looks more involved than I have spare time for...
>>
>> On 1 July 2017 at 00:06, Nisha Miller <nisha43222@gmail.com> wrote:
>>> Sorry, my previous reply was a bit confusing.
>>>
>>> I made the changes to client.c as per the patch. But that did not
>>> solve the problem. I still get just one terse output at the end of the
>>> test.

-- 
Sitsofe | http://sucs.org/~sits/

Re: --minimal does not work in client/server mode

[Jens Axboe]

On Thu, Aug 29 2013, asomers@gmail.com wrote:
> On Mon, Jul 29, 2013 at 10:23 AM, Jens Axboe <axboe@kernel.dk> wrote:
> >
> > On 07/29/2013 10:00 AM, asomers@gmail.com wrote:
> > > I found a few more things that need to be suppressed in terse mode.
> > > The updated patch is here.  In addition to working with JSON output,
> > > we also need to check that the patch doesn't break disk statistics.  I
> > > haven't tried using those at all yet.
> > >
> > > https://gist.github.com/asomers/6105396
> >
> > I would suggest making a show_thread_status_all() that does something ala:
> >
> > void show_thread_status_all()
> > {
> >    if (is_backend)
> >        fio_server_send_ts(ts, rs);
> >    else if (output_format == FIO_OUTPUT_TERSE)
> >        show_thread_status_terse(ts, rs);
> >    else if (output_format == FIO_OUTPUT_JSON) {
> >        struct json_object *tmp = show_thread_status_json(ts, rs);
> >
> >        json_array_add_value_object(array, tmp);
> >    } else
> >        show_thread_status(ts, rs);
> > }
> >
> > which should work for all. If you could check and fix the json bit too,
> > bonus points, and I'll test and commit it. Thanks!
> >
> >
> > --
> > Jens Axboe
> >
> 
> Try this new set of patches.  They get json output working in client/server
> mode. This is my third attempt to send them; your server keeps rejecting
> gmail's HTML.  Sorry for the overposts.
> 
> https://gist.github.com/asomers/6105396

Thanks! I'll take a look at this soon and hopefully get it added.

-- 
Jens Axboe

Re: --minimal does not work in client/server mode

[asomers]

On Mon, Jul 29, 2013 at 10:23 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 07/29/2013 10:00 AM, asomers@gmail.com wrote:
>> I found a few more things that need to be suppressed in terse mode.
>> The updated patch is here.  In addition to working with JSON output,
>> we also need to check that the patch doesn't break disk statistics.  I
>> haven't tried using those at all yet.
>>
>> https://gist.github.com/asomers/6105396
>
> I would suggest making a show_thread_status_all() that does something ala:
>
> void show_thread_status_all()
> {
>    if (is_backend)
>        fio_server_send_ts(ts, rs);
>    else if (output_format == FIO_OUTPUT_TERSE)
>        show_thread_status_terse(ts, rs);
>    else if (output_format == FIO_OUTPUT_JSON) {
>        struct json_object *tmp = show_thread_status_json(ts, rs);
>
>        json_array_add_value_object(array, tmp);
>    } else
>        show_thread_status(ts, rs);
> }
>
> which should work for all. If you could check and fix the json bit too,
> bonus points, and I'll test and commit it. Thanks!
>
>
> --
> Jens Axboe
>

Try this new set of patches.  They get json output working in
client/server mode. This is my fourth attempt to send them; your
server keeps rejecting gmail's HTML.  Sorry for the overposts.

https://gist.github.com/asomers/6105396

Re: --minimal does not work in client/server mode

[asomers]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

On Mon, Jul 29, 2013 at 10:23 AM, Jens Axboe <axboe@kernel.dk> wrote:
>
> On 07/29/2013 10:00 AM, asomers@gmail.com wrote:
> > I found a few more things that need to be suppressed in terse mode.
> > The updated patch is here.  In addition to working with JSON output,
> > we also need to check that the patch doesn't break disk statistics.  I
> > haven't tried using those at all yet.
> >
> > https://gist.github.com/asomers/6105396
>
> I would suggest making a show_thread_status_all() that does something ala:
>
> void show_thread_status_all()
> {
>    if (is_backend)
>        fio_server_send_ts(ts, rs);
>    else if (output_format == FIO_OUTPUT_TERSE)
>        show_thread_status_terse(ts, rs);
>    else if (output_format == FIO_OUTPUT_JSON) {
>        struct json_object *tmp = show_thread_status_json(ts, rs);
>
>        json_array_add_value_object(array, tmp);
>    } else
>        show_thread_status(ts, rs);
> }
>
> which should work for all. If you could check and fix the json bit too,
> bonus points, and I'll test and commit it. Thanks!
>
>
> --
> Jens Axboe
>

Try this new set of patches.  They get json output working in client/server
mode. This is my third attempt to send them; your server keeps rejecting
gmail's HTML.  Sorry for the overposts.

https://gist.github.com/asomers/6105396

[-- Attachment #2: Type: text/html, Size: 1781 bytes --]

Re: --minimal does not work in client/server mode

[asomers]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

On Mon, Jul 29, 2013 at 10:23 AM, Jens Axboe <axboe@kernel.dk> wrote:

> On 07/29/2013 10:00 AM, asomers@gmail.com wrote:
> > I found a few more things that need to be suppressed in terse mode.
> > The updated patch is here.  In addition to working with JSON output,
> > we also need to check that the patch doesn't break disk statistics.  I
> > haven't tried using those at all yet.
> >
> > https://gist.github.com/asomers/6105396
>
> I would suggest making a show_thread_status_all() that does something ala:
>
> void show_thread_status_all()
> {
>    if (is_backend)
>        fio_server_send_ts(ts, rs);
>    else if (output_format == FIO_OUTPUT_TERSE)
>        show_thread_status_terse(ts, rs);
>    else if (output_format == FIO_OUTPUT_JSON) {
>        struct json_object *tmp = show_thread_status_json(ts, rs);
>
>        json_array_add_value_object(array, tmp);
>    } else
>        show_thread_status(ts, rs);
> }
>
> which should work for all. If you could check and fix the json bit too,
> bonus points, and I'll test and commit it. Thanks!
>
>
> --
> Jens Axboe
>
> Try this new set of patches.  These get json output working in
client/server mode.

https://gist.github.com/asomers/6105396

[-- Attachment #2: Type: text/html, Size: 1925 bytes --]

Re: --minimal does not work in client/server mode

[asomers]

[-- Attachment #1: Type: text/plain, Size: 1203 bytes --]

Try this new set of patches.  These get json output working in
client/server mode.

https://gist.github.com/asomers/6105396


On Mon, Jul 29, 2013 at 10:23 AM, Jens Axboe <axboe@kernel.dk> wrote:

> On 07/29/2013 10:00 AM, asomers@gmail.com wrote:
> > I found a few more things that need to be suppressed in terse mode.
> > The updated patch is here.  In addition to working with JSON output,
> > we also need to check that the patch doesn't break disk statistics.  I
> > haven't tried using those at all yet.
> >
> > https://gist.github.com/asomers/6105396
>
> I would suggest making a show_thread_status_all() that does something ala:
>
> void show_thread_status_all()
> {
>    if (is_backend)
>        fio_server_send_ts(ts, rs);
>    else if (output_format == FIO_OUTPUT_TERSE)
>        show_thread_status_terse(ts, rs);
>    else if (output_format == FIO_OUTPUT_JSON) {
>        struct json_object *tmp = show_thread_status_json(ts, rs);
>
>        json_array_add_value_object(array, tmp);
>    } else
>        show_thread_status(ts, rs);
> }
>
> which should work for all. If you could check and fix the json bit too,
> bonus points, and I'll test and commit it. Thanks!
>
>
> --
> Jens Axboe
>
>

[-- Attachment #2: Type: text/html, Size: 1880 bytes --]

Re: --minimal does not work in client/server mode

[Jens Axboe]

On 07/29/2013 10:00 AM, asomers@gmail.com wrote:
> I found a few more things that need to be suppressed in terse mode.
> The updated patch is here.  In addition to working with JSON output,
> we also need to check that the patch doesn't break disk statistics.  I
> haven't tried using those at all yet.
> 
> https://gist.github.com/asomers/6105396

I would suggest making a show_thread_status_all() that does something ala:

void show_thread_status_all()
{
   if (is_backend)
       fio_server_send_ts(ts, rs);
   else if (output_format == FIO_OUTPUT_TERSE)
       show_thread_status_terse(ts, rs);
   else if (output_format == FIO_OUTPUT_JSON) {
       struct json_object *tmp = show_thread_status_json(ts, rs);

       json_array_add_value_object(array, tmp);
   } else
       show_thread_status(ts, rs);
}

which should work for all. If you could check and fix the json bit too,
bonus points, and I'll test and commit it. Thanks!


-- 
Jens Axboe

Re: --minimal does not work in client/server mode

[asomers]

I found a few more things that need to be suppressed in terse mode.
The updated patch is here.  In addition to working with JSON output,
we also need to check that the patch doesn't break disk statistics.  I
haven't tried using those at all yet.

https://gist.github.com/asomers/6105396

On Mon, Jul 29, 2013 at 9:02 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 07/26/2013 05:16 PM, asomers@gmail.com wrote:
>> When I use fio in client/server mode, no matter how I try to specify
>> --minimal, I always get the normal output.  I've tried all of these
>> methods, and their combinations:
>>
>> server# fio --server
>> client#  fio --minimal --client=localhost /tmp/cpuio.fio
>>
>> server# fio --server
>> client#  fio --client=localhost --minimal /tmp/cpuio.fio
>>
>> server# fio --minimal --server
>> client# fio --client=localhost  /tmp/cpuio.fio
>>
>> The following patch fixes the problem, though I can't promise it won't
>> break anything else:
>
> It's a good starting point. It needs to handle the json based output too
> though. I'll take a look at fixing that part up and pass you back
> something to test.
>
> --
> Jens Axboe
>

Re: --minimal does not work in client/server mode

[Jens Axboe]

On 07/26/2013 05:16 PM, asomers@gmail.com wrote:
> When I use fio in client/server mode, no matter how I try to specify
> --minimal, I always get the normal output.  I've tried all of these
> methods, and their combinations:
> 
> server# fio --server
> client#  fio --minimal --client=localhost /tmp/cpuio.fio
> 
> server# fio --server
> client#  fio --client=localhost --minimal /tmp/cpuio.fio
> 
> server# fio --minimal --server
> client# fio --client=localhost  /tmp/cpuio.fio
> 
> The following patch fixes the problem, though I can't promise it won't
> break anything else:

It's a good starting point. It needs to handle the json based output too
though. I'll take a look at fixing that part up and pass you back
something to test.

-- 
Jens Axboe

--minimal does not work in client/server mode

[asomers]

When I use fio in client/server mode, no matter how I try to specify
--minimal, I always get the normal output.  I've tried all of these
methods, and their combinations:

server# fio --server
client#  fio --minimal --client=localhost /tmp/cpuio.fio

server# fio --server
client#  fio --client=localhost --minimal /tmp/cpuio.fio

server# fio --minimal --server
client# fio --client=localhost  /tmp/cpuio.fio

The following patch fixes the problem, though I can't promise it won't
break anything else:

--- fio-2.1.1/client.c	2013-06-05 08:55:53.000000000 -0600
+++ fio-2.1.1.new/client.c	2013-07-26 17:13:18.504740214 -0600
@@ -788,11 +788,15 @@
 	dst->unified_rw_rep	= le32_to_cpu(src->unified_rw_rep);
 }

+
 static void handle_ts(struct fio_client *client, struct fio_net_cmd *cmd)
 {
 	struct cmd_ts_pdu *p = (struct cmd_ts_pdu *) cmd->payload;

-	show_thread_status(&p->ts, &p->rs);
+	if (output_format == FIO_OUTPUT_TERSE)
+		show_thread_status_terse(&p->ts, &p->rs);
+	else
+		show_thread_status(&p->ts, &p->rs);
 	client->did_stat = 1;

 	if (!do_output_all_clients)
@@ -808,7 +812,10 @@

 	if (++sum_stat_nr == sum_stat_clients) {
 		strcpy(client_ts.name, "All clients");
-		show_thread_status(&client_ts, &client_gs);
+		if (output_format == FIO_OUTPUT_TERSE)
+			show_thread_status_terse(&client_ts, &client_gs);
+		else
+			show_thread_status(&client_ts, &client_gs);
 	}
 }

diff -dur fio-2.1.1/stat.c fio-2.1.1.new/stat.c
--- fio-2.1.1/stat.c	2013-06-05 08:55:53.000000000 -0600
+++ fio-2.1.1.new/stat.c	2013-07-26 17:03:05.332755580 -0600
@@ -885,7 +885,8 @@
 		log_info(";%3.2f%%", io_u_lat_m[i]);

 	/* disk util stats, if any */
-	show_disk_util(1, NULL);
+	if (is_backend)
+		show_disk_util(1, NULL);

 	/* Additional output if continue_on_error set - default off*/
 	if (ts->continue_on_error)
@@ -982,7 +983,7 @@
 	return root;
 }

-static void show_thread_status_terse(struct thread_stat *ts,
+void show_thread_status_terse(struct thread_stat *ts,
 				     struct group_run_stats *rs)
 {
 	if (terse_version == 2)
diff -dur fio-2.1.1/stat.h fio-2.1.1.new/stat.h
--- fio-2.1.1/stat.h	2013-06-05 08:55:53.000000000 -0600
+++ fio-2.1.1.new/stat.h	2013-07-26 16:46:12.972780950 -0600
@@ -203,6 +203,7 @@
 extern void stat_exit(void);

 extern void show_thread_status(struct thread_stat *ts, struct
group_run_stats *rs);
+extern void show_thread_status_terse(struct thread_stat *ts, struct
group_run_stats *rs);
 extern void show_group_stats(struct group_run_stats *rs);
 extern int calc_thread_status(struct jobs_eta *je, int force);
 extern void display_thread_status(struct jobs_eta *je);