client/server mode fails with Assertion `client->eta_in_flight = eta' failed

client/server mode fails with Assertion `client->eta_in_flight = eta' failed

[Srinivasa Chamarthy]

I started FIO in client/server mode with two clients running 24 jobs
each on individual LUNs. After a while, FIO fails with the following
assertion, even though the jobs are still running on both the clients.

# fio --client=client1 --remote-config=/root/all_powerpath.fio
--client=client2 --remote-config=/root/all_multipaths.fio
hostname=client1, be=0, 64-bit, os=Linux, arch=x86-64,
fio=fio-2.6-13-g8a76, flags=1
hostname=client2, be=0, 64-bit, os=Linux, arch=x86-64,
fio=fio-2.6-13-g8a76, flags=1

<client2> readwrite-mpatha-8k-para: (g=0): rw=randrw, <client1>
readwrite-emcpowera-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
ioengine=libaio, iodepth=8
<client2> readwrite-mpathb-8k-para: (g=0): rw=randrw, <client1>
readwrite-emcpowerb-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
ioengine=libaio, iodepth=8
<client2> readwrite-mpathc-8k-para: (g=0): rw=randrw, <client1>
readwrite-emcpowerc-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
ioengine=libaio, iodepth=8
<client2> readwrite-mpathd-8k-para: (g=0): rw=randrw, <client1>
readwrite-emcpowerd-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
ioengine=libaio, iodepth=8

<<< Omit other info >>>

<client2> Starting 24 processes
<client1> Starting 24 processes

client <client1>: timeout on SEND_ETA/405.7M/0K /s] [51.2K/50.7K/0
iops] [eta 02d:23h:59m:49s]
fio: client: unable to find matching tag (dbebe0)2M/0K /s]
[60.8K/58.6K/0 iops] [eta 02d:23h:59m:54s]
fio: client.c:1241: handle_eta: Assertion `client->eta_in_flight == eta' failed.
Aborted

# fio --version
fio-2.6-13-g8a76

Also sometimes the job information gets mixed up from both the clients
as above.

Thanks for your support.
--
Srinivasa R Chamarthy

Re: client/server mode fails with Assertion `client->eta_in_flight = eta' failed

[Jens Axboe]

On 02/16/2016 01:19 AM, Srinivasa Chamarthy wrote:
> I started FIO in client/server mode with two clients running 24 jobs
> each on individual LUNs. After a while, FIO fails with the following
> assertion, even though the jobs are still running on both the clients.
>
> # fio --client=client1 --remote-config=/root/all_powerpath.fio
> --client=client2 --remote-config=/root/all_multipaths.fio
> hostname=client1, be=0, 64-bit, os=Linux, arch=x86-64,
> fio=fio-2.6-13-g8a76, flags=1
> hostname=client2, be=0, 64-bit, os=Linux, arch=x86-64,
> fio=fio-2.6-13-g8a76, flags=1
>
> <client2> readwrite-mpatha-8k-para: (g=0): rw=randrw, <client1>
> readwrite-emcpowera-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
> ioengine=libaio, iodepth=8
> <client2> readwrite-mpathb-8k-para: (g=0): rw=randrw, <client1>
> readwrite-emcpowerb-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
> ioengine=libaio, iodepth=8
> <client2> readwrite-mpathc-8k-para: (g=0): rw=randrw, <client1>
> readwrite-emcpowerc-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
> ioengine=libaio, iodepth=8
> <client2> readwrite-mpathd-8k-para: (g=0): rw=randrw, <client1>
> readwrite-emcpowerd-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
> ioengine=libaio, iodepth=8
>
> <<< Omit other info >>>
>
> <client2> Starting 24 processes
> <client1> Starting 24 processes
>
> client <client1>: timeout on SEND_ETA/405.7M/0K /s] [51.2K/50.7K/0
> iops] [eta 02d:23h:59m:49s]
> fio: client: unable to find matching tag (dbebe0)2M/0K /s]
> [60.8K/58.6K/0 iops] [eta 02d:23h:59m:54s]
> fio: client.c:1241: handle_eta: Assertion `client->eta_in_flight == eta' failed.
> Aborted
>
> # fio --version
> fio-2.6-13-g8a76
>
> Also sometimes the job information gets mixed up from both the clients
> as above.

Can you include the job file(s) that you used?

-- 
Jens Axboe

Re: client/server mode fails with Assertion `client->eta_in_flight = eta' failed

[Srinivasa Chamarthy]

Hi Jens,

Job Files:

Client1:
# cat all_powerpath.fio
[global]
ioengine=libaio
invalidate=1
ramp_time=10
direct=1
refill_buffers=1
time_based
runtime=259200

[readwrite-emcpowera-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowera
size=10g
rw=randrw

[readwrite-emcpowerb-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowerb
size=10g
rw=randrw

[readwrite-emcpowerc-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowerc
size=10g
rw=randrw

[readwrite-emcpowerd-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowerd
size=10g
rw=randrw

[readwrite-emcpowere-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowere
size=10g
rw=randrw

[readwrite-emcpowerf-8k-para]
bs=4k
iodepth=8
filename=/dev/emcpowerf
size=10g
rw=randrw

...
... [Extends till /dev/emcpowerx -- 24 devices

client2:
# cat all_multipaths.fio q

                                                  [112/2161]
[global]
ioengine=libaio
invalidate=1
ramp_time=10
direct=1
refill_buffers=1
time_based
runtime=259200
norandommap

[readwrite-mpatha-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpatha
size=10g
rw=randrw

[readwrite-mpathb-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathb
size=10g
rw=randrw

[readwrite-mpathc-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathc
size=10g
rw=randrw

[readwrite-mpathd-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathd
size=10g
rw=randrw

[readwrite-mpathe-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathe
size=10g
rw=randrw

[readwrite-mpathf-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathf
size=10g
rw=randrw

[readwrite-mpathg-8k-para]
bs=8k
iodepth=8
filename=/dev/mapper/mpathg
size=10g
rw=randrw
...
... [ Extends till /dev/mapper/mpathx --- 24 devices ]

--
Srinivasa R Chamarthy


On Thu, Feb 18, 2016 at 4:28 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 02/16/2016 01:19 AM, Srinivasa Chamarthy wrote:
>>
>> I started FIO in client/server mode with two clients running 24 jobs
>> each on individual LUNs. After a while, FIO fails with the following
>> assertion, even though the jobs are still running on both the clients.
>>
>> # fio --client=client1 --remote-config=/root/all_powerpath.fio
>> --client=client2 --remote-config=/root/all_multipaths.fio
>> hostname=client1, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.6-13-g8a76, flags=1
>> hostname=client2, be=0, 64-bit, os=Linux, arch=x86-64,
>> fio=fio-2.6-13-g8a76, flags=1
>>
>> <client2> readwrite-mpatha-8k-para: (g=0): rw=randrw, <client1>
>> readwrite-emcpowera-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>> ioengine=libaio, iodepth=8
>> <client2> readwrite-mpathb-8k-para: (g=0): rw=randrw, <client1>
>> readwrite-emcpowerb-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>> ioengine=libaio, iodepth=8
>> <client2> readwrite-mpathc-8k-para: (g=0): rw=randrw, <client1>
>> readwrite-emcpowerc-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>> ioengine=libaio, iodepth=8
>> <client2> readwrite-mpathd-8k-para: (g=0): rw=randrw, <client1>
>> readwrite-emcpowerd-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>> ioengine=libaio, iodepth=8
>>
>> <<< Omit other info >>>
>>
>> <client2> Starting 24 processes
>> <client1> Starting 24 processes
>>
>> client <client1>: timeout on SEND_ETA/405.7M/0K /s] [51.2K/50.7K/0
>> iops] [eta 02d:23h:59m:49s]
>> fio: client: unable to find matching tag (dbebe0)2M/0K /s]
>> [60.8K/58.6K/0 iops] [eta 02d:23h:59m:54s]
>> fio: client.c:1241: handle_eta: Assertion `client->eta_in_flight == eta'
>> failed.
>> Aborted
>>
>> # fio --version
>> fio-2.6-13-g8a76
>>
>> Also sometimes the job information gets mixed up from both the clients
>> as above.
>
>
> Can you include the job file(s) that you used?
>
> --
> Jens Axboe
>

Re: client/server mode fails with Assertion `client->eta_in_flight = eta' failed

[Chris Worley]

Was there ever a solution?  I'm seeing this a lot on v2.9.

On Wed, Feb 17, 2016 at 6:30 PM, Srinivasa Chamarthy
<chamarthy.raju@gmail.com> wrote:
> Hi Jens,
>
> Job Files:
>
> Client1:
> # cat all_powerpath.fio
> [global]
> ioengine=libaio
> invalidate=1
> ramp_time=10
> direct=1
> refill_buffers=1
> time_based
> runtime=259200
>
> [readwrite-emcpowera-8k-para]
> bs=4k
> iodepth=8
> filename=/dev/emcpowera
> size=10g
> rw=randrw
>
> [readwrite-emcpowerb-8k-para]
> bs=4k
> iodepth=8
> filename=/dev/emcpowerb
> size=10g
> rw=randrw
>
> [readwrite-emcpowerc-8k-para]
> bs=4k
> iodepth=8
> filename=/dev/emcpowerc
> size=10g
> rw=randrw
>
> [readwrite-emcpowerd-8k-para]
> bs=4k
> iodepth=8
> filename=/dev/emcpowerd
> size=10g
> rw=randrw
>
> [readwrite-emcpowere-8k-para]
> bs=4k
> iodepth=8
> filename=/dev/emcpowere
> size=10g
> rw=randrw
>
> [readwrite-emcpowerf-8k-para]
> bs=4k
> iodepth=8
> filename=/dev/emcpowerf
> size=10g
> rw=randrw
>
> ...
> ... [Extends till /dev/emcpowerx -- 24 devices
>
> client2:
> # cat all_multipaths.fio q
>
>                                                   [112/2161]
> [global]
> ioengine=libaio
> invalidate=1
> ramp_time=10
> direct=1
> refill_buffers=1
> time_based
> runtime=259200
> norandommap
>
> [readwrite-mpatha-8k-para]
> bs=8k
> iodepth=8
> filename=/dev/mapper/mpatha
> size=10g
> rw=randrw
>
> [readwrite-mpathb-8k-para]
> bs=8k
> iodepth=8
> filename=/dev/mapper/mpathb
> size=10g
> rw=randrw
>
> [readwrite-mpathc-8k-para]
> bs=8k
> iodepth=8
> filename=/dev/mapper/mpathc
> size=10g
> rw=randrw
>
> [readwrite-mpathd-8k-para]
> bs=8k
> iodepth=8
> filename=/dev/mapper/mpathd
> size=10g
> rw=randrw
>
> [readwrite-mpathe-8k-para]
> bs=8k
> iodepth=8
> filename=/dev/mapper/mpathe
> size=10g
> rw=randrw
>
> [readwrite-mpathf-8k-para]
> bs=8k
> iodepth=8
> filename=/dev/mapper/mpathf
> size=10g
> rw=randrw
>
> [readwrite-mpathg-8k-para]
> bs=8k
> iodepth=8
> filename=/dev/mapper/mpathg
> size=10g
> rw=randrw
> ...
> ... [ Extends till /dev/mapper/mpathx --- 24 devices ]
>
> --
> Srinivasa R Chamarthy
>
>
> On Thu, Feb 18, 2016 at 4:28 AM, Jens Axboe <axboe@kernel.dk> wrote:
>> On 02/16/2016 01:19 AM, Srinivasa Chamarthy wrote:
>>>
>>> I started FIO in client/server mode with two clients running 24 jobs
>>> each on individual LUNs. After a while, FIO fails with the following
>>> assertion, even though the jobs are still running on both the clients.
>>>
>>> # fio --client=client1 --remote-config=/root/all_powerpath.fio
>>> --client=client2 --remote-config=/root/all_multipaths.fio
>>> hostname=client1, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.6-13-g8a76, flags=1
>>> hostname=client2, be=0, 64-bit, os=Linux, arch=x86-64,
>>> fio=fio-2.6-13-g8a76, flags=1
>>>
>>> <client2> readwrite-mpatha-8k-para: (g=0): rw=randrw, <client1>
>>> readwrite-emcpowera-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>>> ioengine=libaio, iodepth=8
>>> <client2> readwrite-mpathb-8k-para: (g=0): rw=randrw, <client1>
>>> readwrite-emcpowerb-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>>> ioengine=libaio, iodepth=8
>>> <client2> readwrite-mpathc-8k-para: (g=0): rw=randrw, <client1>
>>> readwrite-emcpowerc-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>>> ioengine=libaio, iodepth=8
>>> <client2> readwrite-mpathd-8k-para: (g=0): rw=randrw, <client1>
>>> readwrite-emcpowerd-8k-para: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K,
>>> bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=8
>>> ioengine=libaio, iodepth=8
>>>
>>> <<< Omit other info >>>
>>>
>>> <client2> Starting 24 processes
>>> <client1> Starting 24 processes
>>>
>>> client <client1>: timeout on SEND_ETA/405.7M/0K /s] [51.2K/50.7K/0
>>> iops] [eta 02d:23h:59m:49s]
>>> fio: client: unable to find matching tag (dbebe0)2M/0K /s]
>>> [60.8K/58.6K/0 iops] [eta 02d:23h:59m:54s]
>>> fio: client.c:1241: handle_eta: Assertion `client->eta_in_flight == eta'
>>> failed.
>>> Aborted
>>>
>>> # fio --version
>>> fio-2.6-13-g8a76
>>>
>>> Also sometimes the job information gets mixed up from both the clients
>>> as above.
>>
>>
>> Can you include the job file(s) that you used?
>>
>> --
>> Jens Axboe
>>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: client/server mode fails with Assertion `client->eta_in_flight = eta' failed

[Jens Axboe]

On 08/30/2016 09:28 AM, Chris Worley wrote:
> Was there ever a solution?  I'm seeing this a lot on v2.9.

Chris/Srinivasa, can you try the below patch? Looks like we're racing
sometimes on the ETA command. And if we don't find a reply tag, we
already decremented the ETA count, so we should just ignore it.

diff --git a/client.c b/client.c
index 3456665009b8..c8069a0159ac 100644
--- a/client.c
+++ b/client.c
@@ -1183,7 +1183,7 @@ void fio_client_sum_jobs_eta(struct jobs_eta *dst, 
struct jobs_eta *je)
  	strcpy((char *) dst->run_str, (char *) je->run_str);
  }

-static void remove_reply_cmd(struct fio_client *client, struct 
fio_net_cmd *cmd)
+static bool remove_reply_cmd(struct fio_client *client, struct 
fio_net_cmd *cmd)
  {
  	struct fio_net_cmd_reply *reply = NULL;
  	struct flist_head *entry;
@@ -1199,12 +1199,13 @@ static void remove_reply_cmd(struct fio_client 
*client, struct fio_net_cmd *cmd)

  	if (!reply) {
  		log_err("fio: client: unable to find matching tag (%llx)\n", 
(unsigned long long) cmd->tag);
-		return;
+		return false;
  	}

  	flist_del(&reply->list);
  	cmd->tag = reply->saved_tag;
  	free(reply);
+	return true;
  }

  int fio_client_wait_for_reply(struct fio_client *client, uint64_t tag)
@@ -1653,7 +1654,8 @@ int fio_handle_client(struct fio_client *client)
  	case FIO_NET_CMD_ETA: {
  		struct jobs_eta *je = (struct jobs_eta *) cmd->payload;

-		remove_reply_cmd(client, cmd);
+		if (!remove_reply_cmd(client, cmd))
+			break;
  		convert_jobs_eta(je);
  		handle_eta(client, cmd);
  		break;

-- 
Jens Axboe

Re: client/server mode fails with Assertion `client->eta_in_flight = eta' failed

[Jens Axboe]

On 09/13/2016 09:04 AM, Jens Axboe wrote:
> On 08/30/2016 09:28 AM, Chris Worley wrote:
>> Was there ever a solution?  I'm seeing this a lot on v2.9.
>
> Chris/Srinivasa, can you try the below patch? Looks like we're racing
> sometimes on the ETA command. And if we don't find a reply tag, we
> already decremented the ETA count, so we should just ignore it.

I committed this fix, so you can just update to current -git to get it.

-- 
Jens Axboe

Re: client/server mode fails with Assertion `client->eta_in_flight = eta' failed

[Srinivasa Chamarthy]

I have stated the workload and seems to be working fine for the last
one hour. If i hit the issue i will let you know.

# fio --version
fio-2.13-95-g308d

# fio --client=client1 --remote-config=/root/fio_randrw_8k.fio
--client=client2 --remote-config=/root/fio_randrw_8k.fio

<client1> Starting 24 processes
<client2> Starting 24 processes
Jobs: 48 (f=48): [m(24)] [2.0% done] [687.5M/687.8M/0K /s]
[85.1K/85.1K/0 iops] [eta 02d:23h:15m:22s]

One question though, related to the number of jobs. I stated 24 jobs
on each client which should total to 48. But the job info shows only
m(24). I have asked the same query using another thread
http://www.spinics.net/lists/fio/msg04604.html.

Thanks for the great support.

--
Srinivasa R Chamarthy


On Tue, Sep 13, 2016 at 11:09 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 09/13/2016 09:04 AM, Jens Axboe wrote:
>>
>> On 08/30/2016 09:28 AM, Chris Worley wrote:
>>>
>>> Was there ever a solution?  I'm seeing this a lot on v2.9.
>>
>>
>> Chris/Srinivasa, can you try the below patch? Looks like we're racing
>> sometimes on the ETA command. And if we don't find a reply tag, we
>> already decremented the ETA count, so we should just ignore it.
>
>
> I committed this fix, so you can just update to current -git to get it.
>
>
> --
> Jens Axboe
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html