engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f

engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f

[Jens Rosenboom]

When running fio-2.3 with the rbd engine, the tests themselves run
fine, but instead of the results being printed, I get:

fio: mutex.c:163: fio_mutex_down: Assertion `mutex->magic ==
0x4d555445U' failed.

The command I am using is

fio --ioengine=rbd --clientname=admin --pool=test --rbdname=test1
--rw=randwrite --runtime=10 --ramp_time=None --numjobs=1 --direct=1
--bs=1048576B --iodepth=64 --name=test1

If I instead use another engine, everything is fine, e.g.:

fio --ioengine=libaio --filename=/dev/sdb1 --rw=randwrite --runtime=10
--ramp_time=None --numjobs=1 --direct=1 --bs=1048576B --iodepth=64
--name=test1

Going back to fio-2.2.13 makes the error go away and running a git
bisect leads to the commit mentioned in the subject as the culprit,
the error turns into a Segmentation Fault at that point, though.

Re: engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f

[Jens Rosenboom]

2016-01-08 12:15 GMT+01:00 Jens Rosenboom <j.rosenboom@x-ion.de>:
> When running fio-2.3 with the rbd engine, the tests themselves run
> fine, but instead of the results being printed, I get:
>
> fio: mutex.c:163: fio_mutex_down: Assertion `mutex->magic ==
> 0x4d555445U' failed.
>
> The command I am using is
>
> fio --ioengine=rbd --clientname=admin --pool=test --rbdname=test1
> --rw=randwrite --runtime=10 --ramp_time=None --numjobs=1 --direct=1
> --bs=1048576B --iodepth=64 --name=test1
>
> If I instead use another engine, everything is fine, e.g.:
>
> fio --ioengine=libaio --filename=/dev/sdb1 --rw=randwrite --runtime=10
> --ramp_time=None --numjobs=1 --direct=1 --bs=1048576B --iodepth=64
> --name=test1
>
> Going back to fio-2.2.13 makes the error go away and running a git
> bisect leads to the commit mentioned in the subject as the culprit,
> the error turns into a Segmentation Fault at that point, though.

Accidentally I found out that I can also reproduce the bug with libaio if I set

LD_PRELOAD=../jemalloc/lib/libjemalloc.so

compiled from https://github.com/jemalloc/jemalloc.git (tested with
3.6.0 and 4.0.4)

So it seems there is some use-after-free issue in this commit. I'll
try to dig into this a bit more when I get the time, but I won't be
sad if someone else is faster. ;)

Re: engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f

[Jens Rosenboom]

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]

2016-01-11 12:09 GMT+01:00 Jens Rosenboom <j.rosenboom@x-ion.de>:
> 2016-01-08 12:15 GMT+01:00 Jens Rosenboom <j.rosenboom@x-ion.de>:
>> When running fio-2.3 with the rbd engine, the tests themselves run
>> fine, but instead of the results being printed, I get:
>>
>> fio: mutex.c:163: fio_mutex_down: Assertion `mutex->magic ==
>> 0x4d555445U' failed.
>>
>> The command I am using is
>>
>> fio --ioengine=rbd --clientname=admin --pool=test --rbdname=test1
>> --rw=randwrite --runtime=10 --ramp_time=None --numjobs=1 --direct=1
>> --bs=1048576B --iodepth=64 --name=test1
>>
>> If I instead use another engine, everything is fine, e.g.:
>>
>> fio --ioengine=libaio --filename=/dev/sdb1 --rw=randwrite --runtime=10
>> --ramp_time=None --numjobs=1 --direct=1 --bs=1048576B --iodepth=64
>> --name=test1
>>
>> Going back to fio-2.2.13 makes the error go away and running a git
>> bisect leads to the commit mentioned in the subject as the culprit,
>> the error turns into a Segmentation Fault at that point, though.
>
> Accidentally I found out that I can also reproduce the bug with libaio if I set
>
> LD_PRELOAD=../jemalloc/lib/libjemalloc.so
>
> compiled from https://github.com/jemalloc/jemalloc.git (tested with
> 3.6.0 and 4.0.4)
>
> So it seems there is some use-after-free issue in this commit. I'll
> try to dig into this a bit more when I get the time, but I won't be
> sad if someone else is faster. ;)

In the standalone case, sk_out_key is used uninitialized, which may
happen to work fine until some other library also uses a pthread_key,
like it seems both librbd and libjemalloc do.

Attached is a patch that fixes this.

[-- Attachment #2: 0001-Don-t-use-uninitialized-pthread_key.patch --]
[-- Type: text/x-diff, Size: 1836 bytes --]

From fd4f2df3243b215622fc35ed7c175c9e6c0d1f62 Mon Sep 17 00:00:00 2001
From: Jens Rosenboom <j.rosenboom@x-ion.de>
Date: Mon, 11 Jan 2016 14:41:41 +0100
Subject: [PATCH] Don't use uninitialized pthread_key

When running standalone instead of as a server, sk_out_key doesn't get
initialized, but still will be referenced lateron, which according to
POSIX is unspecified.

Signed-off-by: Jens Rosenboom <j.rosenboom@x-ion.de>
---
 fio.c    |  6 ++++--
 server.c | 14 ++++++++++----
 server.h |  2 +-
 3 files changed, 15 insertions(+), 7 deletions(-)

diff --git a/fio.c b/fio.c
index bd3e260..d59c567 100644
--- a/fio.c
+++ b/fio.c
@@ -56,8 +56,10 @@ int main(int argc, char *argv[], char *envp[])
 		if (fio_start_all_clients())
 			goto done;
 		ret = fio_handle_clients(&fio_client_ops);
-	} else
-		ret = fio_backend(NULL);
+	} else {
+		if (!server_create_sk_out_key())
+			ret = fio_backend(NULL);
+	}
 
 done:
 	deinitialize_fio();
diff --git a/server.c b/server.c
index eccd9d3..85174c2 100644
--- a/server.c
+++ b/server.c
@@ -2186,17 +2186,23 @@ static void set_sig_handlers(void)
 	sigaction(SIGINT, &act, NULL);
 }
 
-static int fio_server(void)
-{
-	int sk, ret;
-
+int server_create_sk_out_key(void) {
 	if (pthread_key_create(&sk_out_key, NULL)) {
 		log_err("fio: can't create sk_out backend key\n");
 		return -1;
 	}
 
 	pthread_setspecific(sk_out_key, NULL);
+	return 0;
+}
 
+static int fio_server(void)
+{
+	int sk, ret;
+
+
+	if (server_create_sk_out_key())
+		return -1;
 	dprint(FD_NET, "starting server\n");
 
 	if (fio_handle_server_arg())
diff --git a/server.h b/server.h
index 9205ae6..3ae55d4 100644
--- a/server.h
+++ b/server.h
@@ -222,5 +222,5 @@ extern int fio_net_send_quit(int sk);
 
 extern int exit_backend;
 extern int fio_net_port;
-
+extern int server_create_sk_out_key(void);
 #endif
-- 
2.4.10

Re: engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f

[Jens Axboe]

On 01/11/2016 07:03 AM, Jens Rosenboom wrote:
> 2016-01-11 12:09 GMT+01:00 Jens Rosenboom <j.rosenboom@x-ion.de>:
>> 2016-01-08 12:15 GMT+01:00 Jens Rosenboom <j.rosenboom@x-ion.de>:
>>> When running fio-2.3 with the rbd engine, the tests themselves run
>>> fine, but instead of the results being printed, I get:
>>>
>>> fio: mutex.c:163: fio_mutex_down: Assertion `mutex->magic ==
>>> 0x4d555445U' failed.
>>>
>>> The command I am using is
>>>
>>> fio --ioengine=rbd --clientname=admin --pool=test --rbdname=test1
>>> --rw=randwrite --runtime=10 --ramp_time=None --numjobs=1 --direct=1
>>> --bs=1048576B --iodepth=64 --name=test1
>>>
>>> If I instead use another engine, everything is fine, e.g.:
>>>
>>> fio --ioengine=libaio --filename=/dev/sdb1 --rw=randwrite --runtime=10
>>> --ramp_time=None --numjobs=1 --direct=1 --bs=1048576B --iodepth=64
>>> --name=test1
>>>
>>> Going back to fio-2.2.13 makes the error go away and running a git
>>> bisect leads to the commit mentioned in the subject as the culprit,
>>> the error turns into a Segmentation Fault at that point, though.
>>
>> Accidentally I found out that I can also reproduce the bug with libaio if I set
>>
>> LD_PRELOAD=../jemalloc/lib/libjemalloc.so
>>
>> compiled from https://github.com/jemalloc/jemalloc.git (tested with
>> 3.6.0 and 4.0.4)
>>
>> So it seems there is some use-after-free issue in this commit. I'll
>> try to dig into this a bit more when I get the time, but I won't be
>> sad if someone else is faster. ;)
>
> In the standalone case, sk_out_key is used uninitialized, which may
> happen to work fine until some other library also uses a pthread_key,
> like it seems both librbd and libjemalloc do.
>
> Attached is a patch that fixes this.

Thanks for reporting this! I merged a cleaner fix, we can just set up 
the key earlier and not have to worry about server vs non-server. Can 
you check if current -git works for you?


-- 
Jens Axboe

Re: engine rbd broken after commit 2466096336bd0fbc1a94811aa338926af6baf42f

[Jens Rosenboom]

2016-01-13 17:36 GMT+01:00 Jens Axboe <axboe@kernel.dk>:
...
> Thanks for reporting this! I merged a cleaner fix, we can just set up the
> key earlier and not have to worry about server vs non-server. Can you check
> if current -git works for you?

Yep, working fine, thx.