[LTP] [PATCH 1/2] fs_fill: Add max_runtime = 60

[LTP] [PATCH 1/2] fs_fill: Add max_runtime = 60

[Richard Palethorpe via ltp]

Usually it takes a few seconds (3-6) to fill up the device with
default settings. Which is too close to the default runtime.

This sets the max_runtime to 10x that to avoid random failures caused
by rare events.

Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
Cc: Petr Cervinka <pcervinka@suse.com>
---
 testcases/kernel/fs/fs_fill/fs_fill.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/testcases/kernel/fs/fs_fill/fs_fill.c b/testcases/kernel/fs/fs_fill/fs_fill.c
index 66b3974fc..95dfc2cb6 100644
--- a/testcases/kernel/fs/fs_fill/fs_fill.c
+++ b/testcases/kernel/fs/fs_fill/fs_fill.c
@@ -109,6 +109,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.max_runtime = 60,
 	.needs_root = 1,
 	.mount_device = 1,
 	.mntpoint = MNTPOINT,
-- 
2.38.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 2/2] fill_fs: Ensure written data is not easily compressed

[Richard Palethorpe via ltp]

If the stack is auto initialized to zero, then we will write all
zeros. Some FS may treat this as a special case and just record the
number of zero bytes or sectors.

This could alter the test behaviour in unpredictable ways. For example
a large number of (slow) syscalls may be required to fill up the
drive, extending the required test time. Or we could overflow the file
size/offset causing EFBIG.

So this writes random data to the buffer if it is zero.

Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
Cc: Cyril Hrubis <chrubis@suse.cz>
---
 lib/tst_fill_fs.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/tst_fill_fs.c b/lib/tst_fill_fs.c
index 121dd2f20..ade0a52ba 100644
--- a/lib/tst_fill_fs.c
+++ b/lib/tst_fill_fs.c
@@ -16,13 +16,20 @@ void tst_fill_fs(const char *path, int verbose)
 {
 	int i = 0;
 	char file[PATH_MAX];
-	char buf[4096];
+	static char buf[4096];
 	size_t len;
 	ssize_t ret;
 	int fd;
 	struct statvfs fi;
+
 	statvfs(path, &fi);
 
+	if (!(buf[0] || buf[4095])) {
+		fd = SAFE_OPEN("/dev/urandom", O_RDONLY);
+		SAFE_READ(0, fd, buf, 4096);
+		SAFE_CLOSE(fd);
+	}
+
 	for (;;) {
 		len = random() % (1024 * 102400);
 
-- 
2.38.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/2] fill_fs: Ensure written data is not easily compressed

[Cyril Hrubis]

Hi!
> --- a/lib/tst_fill_fs.c
> +++ b/lib/tst_fill_fs.c
> @@ -16,13 +16,20 @@ void tst_fill_fs(const char *path, int verbose)
>  {
>  	int i = 0;
>  	char file[PATH_MAX];
> -	char buf[4096];
> +	static char buf[4096];

I'm not sure if caching the random data is worth here, I bet that
reading the random data would be neglectible to the rest of the write
operations we do.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/2] fill_fs: Ensure written data is not easily compressed

[Richard Palethorpe]

Hello,

Cyril Hrubis <chrubis@suse.cz> writes:

> Hi!
>> --- a/lib/tst_fill_fs.c
>> +++ b/lib/tst_fill_fs.c
>> @@ -16,13 +16,20 @@ void tst_fill_fs(const char *path, int verbose)
>>  {
>>  	int i = 0;
>>  	char file[PATH_MAX];
>> -	char buf[4096];
>> +	static char buf[4096];
>
> I'm not sure if caching the random data is worth here, I bet that
> reading the random data would be neglectible to the rest of the write
> operations we do.

I suppose that instead of writing random lengths we could just copy
/dev/urandom to <path> in static chunks of a reasonable size.

Furthermore we can use copy_file_range on newer kernels.

-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 1/2] fs_fill: Add max_runtime = 60

[Petr Vorel]

Hi Richie,

Reviewed-by: Petr Vorel <pvorel@suse.cz>

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/2] fill_fs: Ensure written data is not easily compressed

[Cyril Hrubis]

Hi!
> I suppose that instead of writing random lengths we could just copy
> /dev/urandom to <path> in static chunks of a reasonable size.

Actually it would make sense to do random length writes as well, at
least for a subset of files. I guess that in real life scenario we would
encounter both, block writes and randomly sized writes for files.

I would do something as:

#define BLOCK_SIZE 4096

..
	char buf[2*BLOCK_SIZE];

	fd = SAFE_OPEN("/dev/urandom", O_RDONLY);
	SAFE_READ(1, fd, buf, sizeof(buf));
	SAFE_CLOSE(fd);

	...

	random_size = random() % 2;

	while (len) {
		if (random_size)
			len = random() % BOCK_SIZE;
		else
			len = BLOCK_SIZE;

		off = random() % BLOCK_SIZE;

		ret = write(fd, buf + off, len);

	...


But feel free to implement anything that you find sensible.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/2] fill_fs: Ensure written data is not easily compressed

[Richard Palethorpe]

Hello,

Cyril Hrubis <chrubis@suse.cz> writes:

> Hi!
>> I suppose that instead of writing random lengths we could just copy
>> /dev/urandom to <path> in static chunks of a reasonable size.
>
> Actually it would make sense to do random length writes as well, at
> least for a subset of files. I guess that in real life scenario we would
> encounter both, block writes and randomly sized writes for files.
>
> I would do something as:
>
> #define BLOCK_SIZE 4096
>
> ..
> 	char buf[2*BLOCK_SIZE];
>
> 	fd = SAFE_OPEN("/dev/urandom", O_RDONLY);
> 	SAFE_READ(1, fd, buf, sizeof(buf));
> 	SAFE_CLOSE(fd);
>
> 	...
>
> 	random_size = random() % 2;
>
> 	while (len) {
> 		if (random_size)
> 			len = random() % BOCK_SIZE;
> 		else
> 			len = BLOCK_SIZE;
>
> 		off = random() % BLOCK_SIZE;
>
> 		ret = write(fd, buf + off, len);
>
> 	...
>
>
> But feel free to implement anything that you find sensible.

What are we trying to do though, simply fill the device to test the
ENOSPC condition or some kind of poor man's fuzzing?

-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/2] fill_fs: Ensure written data is not easily compressed

[Cyril Hrubis]

Hi!
> What are we trying to do though, simply fill the device to test the
> ENOSPC condition or some kind of poor man's fuzzing?

The test is supposed to test what happens when filesystem is altmost
full and being written to, which may trigger all kinds of corner cases.
In that sense it makes sense to randomize the access patterns a bit so
that we have higher chances of utilizing different code paths. But of
course the question where should we stop in randomizing things and what
makes sense and what does not.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 1/2] fs_fill: Add max_runtime = 60

[Li Wang]

Merged this 1/2, thanks!


-- 
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/2] fill_fs: Ensure written data is not easily compressed

[Richard Palethorpe]

Hello,

Cyril Hrubis <chrubis@suse.cz> writes:

> Hi!
>> What are we trying to do though, simply fill the device to test the
>> ENOSPC condition or some kind of poor man's fuzzing?
>
> The test is supposed to test what happens when filesystem is altmost
> full and being written to, which may trigger all kinds of corner cases.
> In that sense it makes sense to randomize the access patterns a bit so
> that we have higher chances of utilizing different code paths. But of
> course the question where should we stop in randomizing things and what
> makes sense and what does not.

I think there are multiple scenarious which are totally different.

For example, Redis uses an append only file (AOF) and IIRC you can
choose to batch writes for performance at the expense of data
integrity. It's common for the AOF to have it's own volume.

OTOH if we have a classic server with 1000s of daemons running, then we
can expect writes to happen in parallel and be random.

I'd be in favor of trying to test these things separately and to keep
each scenario as simple (and reproducible) as possible. I think mixing
together different access patterns is better handled by fuzzing.

-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/2] fill_fs: Ensure written data is not easily compressed

[Cyril Hrubis]

Hi!
> > The test is supposed to test what happens when filesystem is altmost
> > full and being written to, which may trigger all kinds of corner cases.
> > In that sense it makes sense to randomize the access patterns a bit so
> > that we have higher chances of utilizing different code paths. But of
> > course the question where should we stop in randomizing things and what
> > makes sense and what does not.
> 
> I think there are multiple scenarious which are totally different.
> 
> For example, Redis uses an append only file (AOF) and IIRC you can
> choose to batch writes for performance at the expense of data
> integrity. It's common for the AOF to have it's own volume.
> 
> OTOH if we have a classic server with 1000s of daemons running, then we
> can expect writes to happen in parallel and be random.
> 
> I'd be in favor of trying to test these things separately and to keep
> each scenario as simple (and reproducible) as possible. I think mixing
> together different access patterns is better handled by fuzzing.

I suppose.

Or we can run the test a few times with different access patters and do
not mix them in a single run.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp