[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[Li Wang]

Do more tries with size in half when write() getting ENOSPC, until the size
is less than the filesystem block size. Though we can't really fill a filesystem
full, this could make the routine more robust.

Signed-off-by: Li Wang <liwang@redhat.com>
Cc: Jan Stancek <jstancek@redhat.com>
---
 lib/tst_fill_fs.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/lib/tst_fill_fs.c b/lib/tst_fill_fs.c
index 4003dce97..00adf7287 100644
--- a/lib/tst_fill_fs.c
+++ b/lib/tst_fill_fs.c
@@ -6,6 +6,7 @@
 #include <errno.h>
 #include <stdio.h>
 #include <stdlib.h>
+#include <sys/statvfs.h>
 
 #define TST_NO_DEFAULT_MAIN
 #include "tst_test.h"
@@ -19,6 +20,8 @@ void tst_fill_fs(const char *path, int verbose)
 	size_t len;
 	ssize_t ret;
 	int fd;
+	struct statvfs fi;
+	statvfs(path, &fi);
 
 	for (;;) {
 		len = random() % (1024 * 102400);
@@ -37,10 +40,15 @@ void tst_fill_fs(const char *path, int verbose)
 			return;
 		}
 
-		while (len) {
+		while (len >= fi.f_bsize) {
 			ret = write(fd, buf, MIN(len, sizeof(buf)));
 
 			if (ret < 0) {
+				if (errno == ENOSPC) {
+					len /= 2;
+					continue;
+				}
+
 				SAFE_CLOSE(fd);
 
 				if (errno != ENOSPC)
-- 
2.20.1

[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[Jan Stancek]

----- Original Message -----
> -		while (len) {
> +		while (len >= fi.f_bsize) {

Let's say f_bsize is 512, and len is 1023. We hit ENOSPC for 1023,
len for next round is 511. Loop ends, but we never tried write with
f_bsize or smaller value.

I think we should try len < fi.f_bsize@least once.

>  			ret = write(fd, buf, MIN(len, sizeof(buf)));
>  
>  			if (ret < 0) {
> +				if (errno == ENOSPC) {
> +					len /= 2;
> +					continue;
> +				}

The return below (in original code) doesn't appear to be reachable now.

[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[Li Wang]

On Tue, Nov 19, 2019 at 7:07 PM Jan Stancek <jstancek@redhat.com> wrote:

>
>
> ----- Original Message -----
> > -             while (len) {
> > +             while (len >= fi.f_bsize) {
>
> Let's say f_bsize is 512, and len is 1023. We hit ENOSPC for 1023,
> len for next round is 511. Loop ends, but we never tried write with
> f_bsize or smaller value.
>
> I think we should try len < fi.f_bsize@least once.
>

Good point. At least for len = fi.f_bsize.


>
> >                       ret = write(fd, buf, MIN(len, sizeof(buf)));
> >
> >                       if (ret < 0) {
> > +                             if (errno == ENOSPC) {
> > +                                     len /= 2;
> > +                                     continue;
> > +                             }
>
> The return below (in original code) doesn't appear to be reachable now.
>

Yes, but it doesn't make much sense there, maybe we could remove that part
directly?

-- 
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191119/e94973b4/attachment.htm>

[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[Martin Doucha]

On 11/19/19 3:03 PM, Li Wang wrote:
> On Tue, Nov 19, 2019 at 7:07 PM Jan Stancek <jstancek@redhat.com
> <mailto:jstancek@redhat.com>> wrote:
> 
>     ----- Original Message -----
>     > -? ? ? ? ? ? ?while (len) {
>     > +? ? ? ? ? ? ?while (len >= fi.f_bsize) {
> 
>     Let's say f_bsize is 512, and len is 1023. We hit ENOSPC for 1023,
>     len for next round is 511. Loop ends, but we never tried write with
>     f_bsize or smaller value.
> 
>     I think we should try len < fi.f_bsize@least once.
> 
> 
> Good point. At least for len = fi.f_bsize.

You don't need to reduce the write size in the first place. Kernel will
happily do short writes for you if the file system somehow finds some
free blocks. The more important question is how to trigger garbage
collection of internal file system book-keeping data.

-- 
Martin Doucha   mdoucha@suse.cz
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[Li Wang]

On Tue, Nov 19, 2019 at 10:24 PM Martin Doucha <mdoucha@suse.cz> wrote:

> On 11/19/19 3:03 PM, Li Wang wrote:
> > On Tue, Nov 19, 2019 at 7:07 PM Jan Stancek <jstancek@redhat.com
> > <mailto:jstancek@redhat.com>> wrote:
> >
> >     ----- Original Message -----
> >     > -             while (len) {
> >     > +             while (len >= fi.f_bsize) {
> >
> >     Let's say f_bsize is 512, and len is 1023. We hit ENOSPC for 1023,
> >     len for next round is 511. Loop ends, but we never tried write with
> >     f_bsize or smaller value.
> >
> >     I think we should try len < fi.f_bsize@least once.
> >
> >
> > Good point. At least for len = fi.f_bsize.
>
> You don't need to reduce the write size in the first place. Kernel will
> happily do short writes for you if the file system somehow finds some
>

You're probably right, but here I just want to enhance the tst_fill_fs
routine, it obviously couldn't  be guaranteed way to perfectly fill a
filesystem, this action from userspace is what I can think of.


> free blocks. The more important question is how to trigger garbage
> collection of internal file system book-keeping data.
>

Okay. I'm not good at FS issues. Or, do you have a better idea for that?

-- 
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191120/12246ea0/attachment.htm>

[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[Jan Stancek]

----- Original Message -----
> > You don't need to reduce the write size in the first place. Kernel will
> > happily do short writes for you if the file system somehow finds some
> >
> 
> You're probably right, but here I just want to enhance the tst_fill_fs
> routine, it obviously couldn't  be guaranteed way to perfectly fill a
> filesystem, this action from userspace is what I can think of.
> 
> 
> > free blocks. The more important question is how to trigger garbage
> > collection of internal file system book-keeping data.
> >
> 
> Okay. I'm not good at FS issues. Or, do you have a better idea for that?

FS devs mentioned couple ideas. Repeated writes (this thread), fsync, mount+umount.
But all suggestions were from "better" category, don't think we have 100% reliable one.

[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[Martin Doucha]

On 11/20/19 3:19 AM, Li Wang wrote:
>     You don't need to reduce the write size in the first place. Kernel will
>     happily do short writes for you if the file system somehow finds some
> 
> 
> You're probably right, but here I just want to enhance the tst_fill_fs
> routine, it obviously couldn't? be guaranteed way to perfectly fill a
> filesystem, this action from userspace is what I can think of.

I was also talking about enhancing tst_fill_fs(). You can simply repeat
write()s with the maximum size until you're sure the file system is
really full. Reducing the write size and then also using the write size
as iteration counter creates some really nasty edge cases that can be
easily avoided.

-- 
Martin Doucha   mdoucha@suse.cz
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[Jan Stancek]

----- Original Message -----
> On 11/20/19 3:19 AM, Li Wang wrote:
> >     You don't need to reduce the write size in the first place. Kernel will
> >     happily do short writes for you if the file system somehow finds some
> > 
> > 
> > You're probably right, but here I just want to enhance the tst_fill_fs
> > routine, it obviously couldn't? be guaranteed way to perfectly fill a
> > filesystem, this action from userspace is what I can think of.
> 
> I was also talking about enhancing tst_fill_fs(). You can simply repeat
> write()s with the maximum size until you're sure the file system is
> really full.

I'd rather we mirror xfstests, which halves the length on ENOSPC.

> Reducing the write size and then also using the write size
> as iteration counter

We could do fixed number of retries. In the end it still feels
like rain dance.

[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[Martin Doucha]

Hello,
here's something fun which I've discovered while playing with
fallocate05 test on XFS:
- fallocate(x blocks)
- tst_fill_fs()
- write(x blocks) - OK
- write(1 block) - ENOSPC
- fallocate(1 block) - OK
- write(1 block) - OK
- fallocate(1 block) - ENOSPC

Somehow, fallocate() can find another free block on XFS which wasn't
available to write() at first. But then that block becomes write()able.

-- 
Martin Doucha   mdoucha@suse.cz
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

[LTP] [PATCH] tst_fill_fs: enhance the filesystem filling routine

[Li Wang]

On Wed, Nov 27, 2019 at 10:29 PM Martin Doucha <mdoucha@suse.cz> wrote:

> Hello,
> here's something fun which I've discovered while playing with
> fallocate05 test on XFS:
> - fallocate(x blocks)
> - tst_fill_fs()
> - write(x blocks) - OK
> - write(1 block) - ENOSPC
> - fallocate(1 block) - OK
> - write(1 block) - OK
> - fallocate(1 block) - ENOSPC
>
> Somehow, fallocate() can find another free block on XFS which wasn't
> available to write() at first. But then that block becomes write()able.
>

That's interesting.

It does prove that an xfs filesystem can't be really fill full. As FS dev
says though a filesystem return -ENOSPC, after some internal bookkeeping,
flushing, then have space available again.

And that also means to increases FALLOCATE_SIZE is a wise way to avoid this
problem.


>
> --
> Martin Doucha   mdoucha@suse.cz
> QA Engineer for Software Maintenance
> SUSE LINUX, s.r.o.
> CORSO IIa
> Krizikova 148/34
> 186 00 Prague 8
> Czech Republic
>
>

-- 
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191128/5a04406e/attachment-0001.htm>