Overlayfs @Plumbers

Overlayfs @Plumbers

[Amir Goldstein]

Hi Guys,

It's been nice to virtually meet with you yesterday.
Some of you wanted to follow up on overlayfs related issues.

If you want to discuss, try to find me in one of the
https://meet.2020.linuxplumbersconf.org/hackrooms
today between 16:00-17:00 UTC
(No need to enter the room to see who's inside)

If those times do not work for you, contact me and we can try
to schedule another time.

Thanks,
Amir.

Re: Overlayfs @Plumbers

[Eric W. Biederman]

Amir Goldstein <amir73il@gmail.com> writes:

> Hi Guys,
>
> It's been nice to virtually meet with you yesterday.
> Some of you wanted to follow up on overlayfs related issues.
>
> If you want to discuss, try to find me in one of the
> https://meet.2020.linuxplumbersconf.org/hackrooms
> today between 16:00-17:00 UTC
> (No need to enter the room to see who's inside)
>
> If those times do not work for you, contact me and we can try
> to schedule another time.

Did this conversation wind up happening?  Do we need to reschedule?

Eric

Re: Overlayfs @Plumbers

[Aleksa Sarai]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote:
> Amir Goldstein <amir73il@gmail.com> writes:
> 
> > Hi Guys,
> >
> > It's been nice to virtually meet with you yesterday.
> > Some of you wanted to follow up on overlayfs related issues.
> >
> > If you want to discuss, try to find me in one of the
> > https://meet.2020.linuxplumbersconf.org/hackrooms
> > today between 16:00-17:00 UTC
> > (No need to enter the room to see who's inside)
> >
> > If those times do not work for you, contact me and we can try
> > to schedule another time.
> 
> Did this conversation wind up happening?  Do we need to reschedule?

This conversation already happened in a Hackroom on Tuesday. I'm not
sure if the Hackrooms will have their recordings published, so maybe
Amir can post any of the takeaways we had?

-- 
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
<https://www.cyphar.com/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Overlayfs @Plumbers

[Sargun Dhillon]

On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote:
>
> On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote:
> > Amir Goldstein <amir73il@gmail.com> writes:
> >
> > > Hi Guys,
> > >
> > > It's been nice to virtually meet with you yesterday.
> > > Some of you wanted to follow up on overlayfs related issues.
> > >
> > > If you want to discuss, try to find me in one of the
> > > https://meet.2020.linuxplumbersconf.org/hackrooms
> > > today between 16:00-17:00 UTC
> > > (No need to enter the room to see who's inside)
> > >
> > > If those times do not work for you, contact me and we can try
> > > to schedule another time.
> >
> > Did this conversation wind up happening?  Do we need to reschedule?
>
> This conversation already happened in a Hackroom on Tuesday. I'm not
> sure if the Hackrooms will have their recordings published, so maybe
> Amir can post any of the takeaways we had?
>
> --
> Aleksa Sarai
> Senior Software Engineer (Containers)
> SUSE Linux GmbH
> <https://www.cyphar.com/>

I unfortunately missed this conversation. I wanted to bring up OverlayFS, and
ephemeral upper dirs. We use overlayfs with Docker containers, and we waste
a lot of time on writing things back to disk.

We're not so peeved about the fact that OVL does any sync operations, as that's
what our users have been used to. The big problem is on unmount, ovelfs decides
syncing the upperdirs is a good idea. IIRC, this regression was
introduced somewhere
in the 4.X series.

We've been carrying a patch to short-circuit this behaviour for a while now:
https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253

I know that this behaviour (and any behaviour that short-circuits
O_SYNC / FUA is
technically "wrong", but in this case, can we make an exception? I originally
thought about using device mapper to remove the FUA bit from all BIOs, but it
turns out that my underlying storage *always* persists data to disk,
so every write
takes...a long time.

Amir, what's your take?

Re: Overlayfs @Plumbers

[Giuseppe Scrivano]

Hi Sargun,

Sargun Dhillon <sargun@sargun.me> writes:

Q> On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote:
>>
>> On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote:
>> > Amir Goldstein <amir73il@gmail.com> writes:
>> >
>> > > Hi Guys,
>> > >
>> > > It's been nice to virtually meet with you yesterday.
>> > > Some of you wanted to follow up on overlayfs related issues.
>> > >
>> > > If you want to discuss, try to find me in one of the
>> > > https://meet.2020.linuxplumbersconf.org/hackrooms
>> > > today between 16:00-17:00 UTC
>> > > (No need to enter the room to see who's inside)
>> > >
>> > > If those times do not work for you, contact me and we can try
>> > > to schedule another time.
>> >
>> > Did this conversation wind up happening?  Do we need to reschedule?
>>
>> This conversation already happened in a Hackroom on Tuesday. I'm not
>> sure if the Hackrooms will have their recordings published, so maybe
>> Amir can post any of the takeaways we had?
>>
>> --
>> Aleksa Sarai
>> Senior Software Engineer (Containers)
>> SUSE Linux GmbH
>> <https://www.cyphar.com/>
>
> I unfortunately missed this conversation. I wanted to bring up OverlayFS, and
> ephemeral upper dirs. We use overlayfs with Docker containers, and we waste
> a lot of time on writing things back to disk.
>
> We're not so peeved about the fact that OVL does any sync operations, as that's
> what our users have been used to. The big problem is on unmount, ovelfs decides
> syncing the upperdirs is a good idea. IIRC, this regression was
> introduced somewhere
> in the 4.X series.
>
> We've been carrying a patch to short-circuit this behaviour for a while now:
> https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253
>
> I know that this behaviour (and any behaviour that short-circuits
> O_SYNC / FUA is
> technically "wrong", but in this case, can we make an exception? I originally
> thought about using device mapper to remove the FUA bit from all BIOs, but it
> turns out that my underlying storage *always* persists data to disk,
> so every write
> takes...a long time.
>
> Amir, what's your take?

we are having the same issue when building images.  Would the following
patch be helpful?

https://www.spinics.net/lists/linux-unionfs/msg08267.html

Regards,
Giuseppe

Re: Overlayfs @Plumbers

[Amir Goldstein]

On Sat, Aug 29, 2020 at 6:41 PM Sargun Dhillon <sargun@sargun.me> wrote:
>
> On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote:
> >
> > On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote:
> > > Amir Goldstein <amir73il@gmail.com> writes:
> > >
> > > > Hi Guys,
> > > >
> > > > It's been nice to virtually meet with you yesterday.
> > > > Some of you wanted to follow up on overlayfs related issues.
> > > >
> > > > If you want to discuss, try to find me in one of the
> > > > https://meet.2020.linuxplumbersconf.org/hackrooms
> > > > today between 16:00-17:00 UTC
> > > > (No need to enter the room to see who's inside)
> > > >
> > > > If those times do not work for you, contact me and we can try
> > > > to schedule another time.
> > >
> > > Did this conversation wind up happening?  Do we need to reschedule?
> >
> > This conversation already happened in a Hackroom on Tuesday. I'm not
> > sure if the Hackrooms will have their recordings published, so maybe
> > Amir can post any of the takeaways we had?
> >
> > --
> > Aleksa Sarai
> > Senior Software Engineer (Containers)
> > SUSE Linux GmbH
> > <https://www.cyphar.com/>
>
> I unfortunately missed this conversation. I wanted to bring up OverlayFS, and
> ephemeral upper dirs. We use overlayfs with Docker containers, and we waste
> a lot of time on writing things back to disk.
>
> We're not so peeved about the fact that OVL does any sync operations, as that's
> what our users have been used to. The big problem is on unmount, ovelfs decides
> syncing the upperdirs is a good idea. IIRC, this regression was
> introduced somewhere
> in the 4.X series.
>
> We've been carrying a patch to short-circuit this behaviour for a while now:
> https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253
>
> I know that this behaviour (and any behaviour that short-circuits
> O_SYNC / FUA is
> technically "wrong", but in this case, can we make an exception? I originally
> thought about using device mapper to remove the FUA bit from all BIOs, but it
> turns out that my underlying storage *always* persists data to disk,
> so every write
> takes...a long time.
>
> Amir, what's your take?

It's not only FUA that is causing slow down.
syncfs() takes internal filesystem locks (e.g. to commit a journal transaction),
so it causes interference with other writers on the same underlying filesystem.

As Giuseppe pointed out, a patch has already been submitted to address
this issue.

Thanks,
Amir.

Re: Overlayfs @Plumbers

[Amir Goldstein]

On Fri, Aug 28, 2020 at 6:58 PM Aleksa Sarai <asarai@suse.de> wrote:
>
> On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote:
> > Amir Goldstein <amir73il@gmail.com> writes:
> >
> > > Hi Guys,
> > >
> > > It's been nice to virtually meet with you yesterday.
> > > Some of you wanted to follow up on overlayfs related issues.
> > >
> > > If you want to discuss, try to find me in one of the
> > > https://meet.2020.linuxplumbersconf.org/hackrooms
> > > today between 16:00-17:00 UTC
> > > (No need to enter the room to see who's inside)
> > >
> > > If those times do not work for you, contact me and we can try
> > > to schedule another time.
> >
> > Did this conversation wind up happening?  Do we need to reschedule?
>
> This conversation already happened in a Hackroom on Tuesday. I'm not
> sure if the Hackrooms will have their recordings published, so maybe
> Amir can post any of the takeaways we had?
>

AFAIK we had discussed two issues.
1. Watching changes on upper layer
2. OCI format v2

#1 was brought up by Josh Triplett.
He asked whether watching changes using inotify/fanotify on upper dir
and reading changes from upper dir is a reasonable practice.
I suggested that he use fanotify filesystem mark to watch changes
on overlay sb and read the changes from overlay mount.

#2 TBH I lost track and you guys continued the discussion after I left.
From what I gathered we need a way to describe clones and renames
more efficiently, but I did not hear any solid plan of how to do that.
So perhaps you would be able to fill in some details here.

Thanks,
Amir.