[ANNOUNCE] unionmount-testsuite: master branch updated to 9c60a9c

[ANNOUNCE] unionmount-testsuite: master branch updated to 9c60a9c

[Amir Goldstein]

Hi All,

The master branch on the unionmount-testsuite tree [1] has been updated.

Changes in this update:
- Support user configurable underlying filesystem

So far, unionmount-testsuite used hardcoded paths for layers and
mount point.  Using underlying filesystem other than tmpfs was possible,
but not very easy to setup.

This update brings the ability for user to configure custom paths
with a custom filesystem for the underlying layers.
This is intended to be used for integration with xfstests [2].

Here is an excerpt from the README:
---
  The following environment variables are supported:

  UNIONMOUNT_BASEDIR  - parent dir of all samefs layers (default: /base)
  UNIONMOUNT_LOWERDIR - lower layer path for non samefs (default: /lower)
  UNIONMOUNT_MNTPOINT - mount point for executing tests (default: /mnt)

  When user provides UNIONMOUNT_LOWERDIR:

  1) Path should be an existing directory whose content will be deleted.
  2) Path is assumed to be on a different filesystem than base dir, so
     --samefs setup is not supported.

  When user provides UNIONMOUNT_BASEDIR:

  1) Path should be an existing directory whose content will be deleted.
  2) Upper layer and middle layers will be created under base dir.
  3) If UNIONMOUNT_MNTPOINT is not provided, the overlay mount point will
     be created under base dir.
  4) If UNIONMOUNT_LOWERDIR is not provided, the lower layer dir will be
     created under base dir.
  5) If UNIONMOUNT_LOWERDIR is not provided, the test setup defaults to
     --samefs (i.e. lower and upper layers are on the same base fs).
     However, if --maxfs=<M> is specified, a tmpfs instance will be
     mounted on the lower layer dir that was created under base dir.
---

Many thanks to Vivek for review and testing.

Thanks,
Amir.

[1] https://github.com/amir73il/unionmount-testsuite
[2] https://github.com/amir73il/xfstests/commits/unionmount

The head of the master branch is commit:

9c60a9c Configure custom layers via environment variables

New commits:

Amir Goldstein (3):
  Add command run --clean-up to cleanup old test mounts
  Stop using bind mounts for --samefs
  Configure custom layers via environment variables

 README           |  26 ++++++++++
 mount_union.py   |  13 +----
 run              |  32 +++++-------
 set_up.py        | 127 +++++++++++++++++++++++++----------------------
 settings.py      |  85 +++++++++++++++++++++++++------
 unmount_union.py |  15 +++---
 6 files changed, 187 insertions(+), 111 deletions(-)

-- 
2.17.1

Re: [ANNOUNCE] unionmount-testsuite: master branch updated to 9c60a9c

[Vivek Goyal]

On Fri, May 29, 2020 at 07:40:58PM +0300, Amir Goldstein wrote:
> Hi All,
> 
> The master branch on the unionmount-testsuite tree [1] has been updated.
> 
> Changes in this update:
> - Support user configurable underlying filesystem
> 
> So far, unionmount-testsuite used hardcoded paths for layers and
> mount point.  Using underlying filesystem other than tmpfs was possible,
> but not very easy to setup.
> 
> This update brings the ability for user to configure custom paths
> with a custom filesystem for the underlying layers.
> This is intended to be used for integration with xfstests [2].
> 
> Here is an excerpt from the README:
> ---
>   The following environment variables are supported:
> 
>   UNIONMOUNT_BASEDIR  - parent dir of all samefs layers (default: /base)

Hi Amir,

I am running these tests with.

UNIONMOUNT_BASEDIR="/mnt/foo/"

- ./run --ov runs fine. But when I try to run it again it complains
  that.

  rm: cannot remove '/mnt/overlayfs//m': Device or resource busy

So I have to first unmount /mnt/overlayfs/m/ and then run tests
again.

I think it will be nice if it can clear the environment by itself.

- I am running one the recent kernel (5.7.0+) and following errors
  out.

# ./run --ov --verify
Environment variables:
UNIONMOUNT_BASEDIR=/mnt/overlayfs/

***
*** ./run --ov --samefs --ts=0 open-plain
***
TEST open-plain.py:10: Open O_RDONLY
/mnt/overlayfs/m/a/foo100: not on union mount

Will spend more time to figure out what happened.

- I am planning to use these environment variables and run overlay over
  virtiofs tests. Can I do the same thing with xfstests overlay tests.
  In README.overlay I see that I need to specify two separate devices.
  Can I specify to directories (and not devices) to be used as TEST
  and SCRATCH and run overlay test.

Thanks
Vivek


> 
>   1) Path should be an existing directory whose content will be deleted.
>   2) Path is assumed to be on a different filesystem than base dir, so
>      --samefs setup is not supported.
> 
>   When user provides UNIONMOUNT_BASEDIR:
> 
>   1) Path should be an existing directory whose content will be deleted.
>   2) Upper layer and middle layers will be created under base dir.
>   3) If UNIONMOUNT_MNTPOINT is not provided, the overlay mount point will
>      be created under base dir.
>   4) If UNIONMOUNT_LOWERDIR is not provided, the lower layer dir will be
>      created under base dir.
>   5) If UNIONMOUNT_LOWERDIR is not provided, the test setup defaults to
>      --samefs (i.e. lower and upper layers are on the same base fs).
>      However, if --maxfs=<M> is specified, a tmpfs instance will be
>      mounted on the lower layer dir that was created under base dir.
> ---
> 
> Many thanks to Vivek for review and testing.
> 
> Thanks,
> Amir.
> 
> [1] https://github.com/amir73il/unionmount-testsuite
> [2] https://github.com/amir73il/xfstests/commits/unionmount
> 
> The head of the master branch is commit:
> 
> 9c60a9c Configure custom layers via environment variables
> 
> New commits:
> 
> Amir Goldstein (3):
>   Add command run --clean-up to cleanup old test mounts
>   Stop using bind mounts for --samefs
>   Configure custom layers via environment variables
> 
>  README           |  26 ++++++++++
>  mount_union.py   |  13 +----
>  run              |  32 +++++-------
>  set_up.py        | 127 +++++++++++++++++++++++++----------------------
>  settings.py      |  85 +++++++++++++++++++++++++------
>  unmount_union.py |  15 +++---
>  6 files changed, 187 insertions(+), 111 deletions(-)
> 
> -- 
> 2.17.1
> 

Re: [ANNOUNCE] unionmount-testsuite: master branch updated to 9c60a9c

[Vivek Goyal]

On Thu, Jun 18, 2020 at 05:38:31PM -0400, Vivek Goyal wrote:

[..]
> 
> # ./run --ov --verify
> Environment variables:
> UNIONMOUNT_BASEDIR=/mnt/overlayfs/
> 
> ***
> *** ./run --ov --samefs --ts=0 open-plain
> ***
> TEST open-plain.py:10: Open O_RDONLY
> /mnt/overlayfs/m/a/foo100: not on union mount
> 
> Will spend more time to figure out what happened.

Ok, this is an issue only if I specify UNIONMOUNT_BASEDIR. If I don't,
then it passes.

Thanks
Vivek

Re: [ANNOUNCE] unionmount-testsuite: master branch updated to 9c60a9c

[Amir Goldstein]

On Fri, Jun 19, 2020 at 12:38 AM Vivek Goyal <vgoyal@redhat.com> wrote:
>
> On Fri, May 29, 2020 at 07:40:58PM +0300, Amir Goldstein wrote:
> > Hi All,
> >
> > The master branch on the unionmount-testsuite tree [1] has been updated.
> >
> > Changes in this update:
> > - Support user configurable underlying filesystem
> >
> > So far, unionmount-testsuite used hardcoded paths for layers and
> > mount point.  Using underlying filesystem other than tmpfs was possible,
> > but not very easy to setup.
> >
> > This update brings the ability for user to configure custom paths
> > with a custom filesystem for the underlying layers.
> > This is intended to be used for integration with xfstests [2].
> >
> > Here is an excerpt from the README:
> > ---
> >   The following environment variables are supported:
> >
> >   UNIONMOUNT_BASEDIR  - parent dir of all samefs layers (default: /base)
>
> Hi Amir,
>
> I am running these tests with.
>
> UNIONMOUNT_BASEDIR="/mnt/foo/"

I suspect there is a bug with trailing / in UNIONMOUNT_BASEDIR
I did not test with trailing /
Can you try without it?

>
> - ./run --ov runs fine. But when I try to run it again it complains
>   that.
>
>   rm: cannot remove '/mnt/overlayfs//m': Device or resource busy
>
> So I have to first unmount /mnt/overlayfs/m/ and then run tests
> again.
>
> I think it will be nice if it can clear the environment by itself.
>

It should.
All the tests I tried cleaned up previous test env automatically.
The only case where explicit cleanup is needed is before changing
context and modifying envvars.

> - I am running one the recent kernel (5.7.0+) and following errors
>   out.
>
> # ./run --ov --verify
> Environment variables:
> UNIONMOUNT_BASEDIR=/mnt/overlayfs/
>
> ***
> *** ./run --ov --samefs --ts=0 open-plain
> ***
> TEST open-plain.py:10: Open O_RDONLY
> /mnt/overlayfs/m/a/foo100: not on union mount
>
> Will spend more time to figure out what happened.
>
> - I am planning to use these environment variables and run overlay over
>   virtiofs tests. Can I do the same thing with xfstests overlay tests.
>   In README.overlay I see that I need to specify two separate devices.
>   Can I specify to directories (and not devices) to be used as TEST
>   and SCRATCH and run overlay test.

As far as I can tell, SCRATCH_DEV and TEST_DEV are directories
in all those non-block filesystems including virtiofs:
_scratch_mkfs()
...
      case $FSTYP in
        nfs*|cifs|ceph|overlay|glusterfs|pvfs2|9p|virtiofs)
                # unable to re-create this fstyp, just remove all files in
                # $SCRATCH_MNT to avoid EEXIST caused by the leftover files
                # created in previous runs
                _scratch_cleanup_files

So I think you just need to set SCRATCH_DEV and TEST_DEV to
two different directories and you are good to go for running
check or check -overlay.

Thanks,
Amir.

Re: [ANNOUNCE] unionmount-testsuite: master branch updated to 9c60a9c

[Vivek Goyal]

On Fri, Jun 19, 2020 at 06:29:08AM +0300, Amir Goldstein wrote:

[..]
> > Hi Amir,
> >
> > I am running these tests with.
> >
> > UNIONMOUNT_BASEDIR="/mnt/foo/"
> 
> I suspect there is a bug with trailing / in UNIONMOUNT_BASEDIR
> I did not test with trailing /
> Can you try without it?

Hi Amir,

This indeed is trailing / issue. I removed it and things are working fine.

> 
> >
> > - ./run --ov runs fine. But when I try to run it again it complains
> >   that.
> >
> >   rm: cannot remove '/mnt/overlayfs//m': Device or resource busy
> >
> > So I have to first unmount /mnt/overlayfs/m/ and then run tests
> > again.
> >
> > I think it will be nice if it can clear the environment by itself.
> >
> 
> It should.
> All the tests I tried cleaned up previous test env automatically.
> The only case where explicit cleanup is needed is before changing
> context and modifying envvars.

After removing trailing /, cleanup was fine too.

> 
> > - I am running one the recent kernel (5.7.0+) and following errors
> >   out.
> >
> > # ./run --ov --verify
> > Environment variables:
> > UNIONMOUNT_BASEDIR=/mnt/overlayfs/
> >
> > ***
> > *** ./run --ov --samefs --ts=0 open-plain
> > ***
> > TEST open-plain.py:10: Open O_RDONLY
> > /mnt/overlayfs/m/a/foo100: not on union mount
> >
> > Will spend more time to figure out what happened.
> >
> > - I am planning to use these environment variables and run overlay over
> >   virtiofs tests. Can I do the same thing with xfstests overlay tests.
> >   In README.overlay I see that I need to specify two separate devices.
> >   Can I specify to directories (and not devices) to be used as TEST
> >   and SCRATCH and run overlay test.
> 
> As far as I can tell, SCRATCH_DEV and TEST_DEV are directories
> in all those non-block filesystems including virtiofs:
> _scratch_mkfs()
> ...
>       case $FSTYP in
>         nfs*|cifs|ceph|overlay|glusterfs|pvfs2|9p|virtiofs)
>                 # unable to re-create this fstyp, just remove all files in
>                 # $SCRATCH_MNT to avoid EEXIST caused by the leftover files
>                 # created in previous runs
>                 _scratch_cleanup_files
> 
> So I think you just need to set SCRATCH_DEV and TEST_DEV to
> two different directories and you are good to go for running
> check or check -overlay.

It seems to work. I am seeing bunch of failures but that probably 
are issues with virtiofs and overlayfs interaction. Will look into
these one by one.

Thanks
Vivek