* Re: Overlayfs @Plumbers
@ 2020-08-29 15:40 ` Sargun Dhillon
0 siblings, 0 replies; 13+ messages in thread
From: Sargun Dhillon @ 2020-08-29 15:40 UTC (permalink / raw)
To: Aleksa Sarai
Cc: Eric W. Biederman, Amir Goldstein, Linux Containers, overlayfs
On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote:
>
> On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote:
> > Amir Goldstein <amir73il@gmail.com> writes:
> >
> > > Hi Guys,
> > >
> > > It's been nice to virtually meet with you yesterday.
> > > Some of you wanted to follow up on overlayfs related issues.
> > >
> > > If you want to discuss, try to find me in one of the
> > > https://meet.2020.linuxplumbersconf.org/hackrooms
> > > today between 16:00-17:00 UTC
> > > (No need to enter the room to see who's inside)
> > >
> > > If those times do not work for you, contact me and we can try
> > > to schedule another time.
> >
> > Did this conversation wind up happening? Do we need to reschedule?
>
> This conversation already happened in a Hackroom on Tuesday. I'm not
> sure if the Hackrooms will have their recordings published, so maybe
> Amir can post any of the takeaways we had?
>
> --
> Aleksa Sarai
> Senior Software Engineer (Containers)
> SUSE Linux GmbH
> <https://www.cyphar.com/>
I unfortunately missed this conversation. I wanted to bring up OverlayFS, and
ephemeral upper dirs. We use overlayfs with Docker containers, and we waste
a lot of time on writing things back to disk.
We're not so peeved about the fact that OVL does any sync operations, as that's
what our users have been used to. The big problem is on unmount, ovelfs decides
syncing the upperdirs is a good idea. IIRC, this regression was
introduced somewhere
in the 4.X series.
We've been carrying a patch to short-circuit this behaviour for a while now:
https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253
I know that this behaviour (and any behaviour that short-circuits
O_SYNC / FUA is
technically "wrong", but in this case, can we make an exception? I originally
thought about using device mapper to remove the FUA bit from all BIOs, but it
turns out that my underlying storage *always* persists data to disk,
so every write
takes...a long time.
Amir, what's your take?
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Overlayfs @Plumbers
2020-08-29 15:40 ` Sargun Dhillon
@ 2020-08-30 11:29 ` Giuseppe Scrivano
-1 siblings, 0 replies; 13+ messages in thread
From: Giuseppe Scrivano @ 2020-08-30 11:29 UTC (permalink / raw)
To: Sargun Dhillon; +Cc: Linux Containers, Eric W. Biederman, overlayfs
Hi Sargun,
Sargun Dhillon <sargun@sargun.me> writes:
Q> On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote:
>>
>> On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote:
>> > Amir Goldstein <amir73il@gmail.com> writes:
>> >
>> > > Hi Guys,
>> > >
>> > > It's been nice to virtually meet with you yesterday.
>> > > Some of you wanted to follow up on overlayfs related issues.
>> > >
>> > > If you want to discuss, try to find me in one of the
>> > > https://meet.2020.linuxplumbersconf.org/hackrooms
>> > > today between 16:00-17:00 UTC
>> > > (No need to enter the room to see who's inside)
>> > >
>> > > If those times do not work for you, contact me and we can try
>> > > to schedule another time.
>> >
>> > Did this conversation wind up happening? Do we need to reschedule?
>>
>> This conversation already happened in a Hackroom on Tuesday. I'm not
>> sure if the Hackrooms will have their recordings published, so maybe
>> Amir can post any of the takeaways we had?
>>
>> --
>> Aleksa Sarai
>> Senior Software Engineer (Containers)
>> SUSE Linux GmbH
>> <https://www.cyphar.com/>
>
> I unfortunately missed this conversation. I wanted to bring up OverlayFS, and
> ephemeral upper dirs. We use overlayfs with Docker containers, and we waste
> a lot of time on writing things back to disk.
>
> We're not so peeved about the fact that OVL does any sync operations, as that's
> what our users have been used to. The big problem is on unmount, ovelfs decides
> syncing the upperdirs is a good idea. IIRC, this regression was
> introduced somewhere
> in the 4.X series.
>
> We've been carrying a patch to short-circuit this behaviour for a while now:
> https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253
>
> I know that this behaviour (and any behaviour that short-circuits
> O_SYNC / FUA is
> technically "wrong", but in this case, can we make an exception? I originally
> thought about using device mapper to remove the FUA bit from all BIOs, but it
> turns out that my underlying storage *always* persists data to disk,
> so every write
> takes...a long time.
>
> Amir, what's your take?
we are having the same issue when building images. Would the following
patch be helpful?
https://www.spinics.net/lists/linux-unionfs/msg08267.html
Regards,
Giuseppe
_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Overlayfs @Plumbers
@ 2020-08-30 11:29 ` Giuseppe Scrivano
0 siblings, 0 replies; 13+ messages in thread
From: Giuseppe Scrivano @ 2020-08-30 11:29 UTC (permalink / raw)
To: Sargun Dhillon
Cc: Aleksa Sarai, Linux Containers, Eric W. Biederman, overlayfs
Hi Sargun,
Sargun Dhillon <sargun@sargun.me> writes:
Q> On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote:
>>
>> On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote:
>> > Amir Goldstein <amir73il@gmail.com> writes:
>> >
>> > > Hi Guys,
>> > >
>> > > It's been nice to virtually meet with you yesterday.
>> > > Some of you wanted to follow up on overlayfs related issues.
>> > >
>> > > If you want to discuss, try to find me in one of the
>> > > https://meet.2020.linuxplumbersconf.org/hackrooms
>> > > today between 16:00-17:00 UTC
>> > > (No need to enter the room to see who's inside)
>> > >
>> > > If those times do not work for you, contact me and we can try
>> > > to schedule another time.
>> >
>> > Did this conversation wind up happening? Do we need to reschedule?
>>
>> This conversation already happened in a Hackroom on Tuesday. I'm not
>> sure if the Hackrooms will have their recordings published, so maybe
>> Amir can post any of the takeaways we had?
>>
>> --
>> Aleksa Sarai
>> Senior Software Engineer (Containers)
>> SUSE Linux GmbH
>> <https://www.cyphar.com/>
>
> I unfortunately missed this conversation. I wanted to bring up OverlayFS, and
> ephemeral upper dirs. We use overlayfs with Docker containers, and we waste
> a lot of time on writing things back to disk.
>
> We're not so peeved about the fact that OVL does any sync operations, as that's
> what our users have been used to. The big problem is on unmount, ovelfs decides
> syncing the upperdirs is a good idea. IIRC, this regression was
> introduced somewhere
> in the 4.X series.
>
> We've been carrying a patch to short-circuit this behaviour for a while now:
> https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253
>
> I know that this behaviour (and any behaviour that short-circuits
> O_SYNC / FUA is
> technically "wrong", but in this case, can we make an exception? I originally
> thought about using device mapper to remove the FUA bit from all BIOs, but it
> turns out that my underlying storage *always* persists data to disk,
> so every write
> takes...a long time.
>
> Amir, what's your take?
we are having the same issue when building images. Would the following
patch be helpful?
https://www.spinics.net/lists/linux-unionfs/msg08267.html
Regards,
Giuseppe
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Overlayfs @Plumbers
2020-08-29 15:40 ` Sargun Dhillon
@ 2020-08-30 11:37 ` Amir Goldstein
-1 siblings, 0 replies; 13+ messages in thread
From: Amir Goldstein @ 2020-08-30 11:37 UTC (permalink / raw)
To: Sargun Dhillon; +Cc: Linux Containers, Eric W. Biederman, overlayfs
On Sat, Aug 29, 2020 at 6:41 PM Sargun Dhillon <sargun@sargun.me> wrote:
>
> On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote:
> >
> > On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote:
> > > Amir Goldstein <amir73il@gmail.com> writes:
> > >
> > > > Hi Guys,
> > > >
> > > > It's been nice to virtually meet with you yesterday.
> > > > Some of you wanted to follow up on overlayfs related issues.
> > > >
> > > > If you want to discuss, try to find me in one of the
> > > > https://meet.2020.linuxplumbersconf.org/hackrooms
> > > > today between 16:00-17:00 UTC
> > > > (No need to enter the room to see who's inside)
> > > >
> > > > If those times do not work for you, contact me and we can try
> > > > to schedule another time.
> > >
> > > Did this conversation wind up happening? Do we need to reschedule?
> >
> > This conversation already happened in a Hackroom on Tuesday. I'm not
> > sure if the Hackrooms will have their recordings published, so maybe
> > Amir can post any of the takeaways we had?
> >
> > --
> > Aleksa Sarai
> > Senior Software Engineer (Containers)
> > SUSE Linux GmbH
> > <https://www.cyphar.com/>
>
> I unfortunately missed this conversation. I wanted to bring up OverlayFS, and
> ephemeral upper dirs. We use overlayfs with Docker containers, and we waste
> a lot of time on writing things back to disk.
>
> We're not so peeved about the fact that OVL does any sync operations, as that's
> what our users have been used to. The big problem is on unmount, ovelfs decides
> syncing the upperdirs is a good idea. IIRC, this regression was
> introduced somewhere
> in the 4.X series.
>
> We've been carrying a patch to short-circuit this behaviour for a while now:
> https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253
>
> I know that this behaviour (and any behaviour that short-circuits
> O_SYNC / FUA is
> technically "wrong", but in this case, can we make an exception? I originally
> thought about using device mapper to remove the FUA bit from all BIOs, but it
> turns out that my underlying storage *always* persists data to disk,
> so every write
> takes...a long time.
>
> Amir, what's your take?
It's not only FUA that is causing slow down.
syncfs() takes internal filesystem locks (e.g. to commit a journal transaction),
so it causes interference with other writers on the same underlying filesystem.
As Giuseppe pointed out, a patch has already been submitted to address
this issue.
Thanks,
Amir.
_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Overlayfs @Plumbers
@ 2020-08-30 11:37 ` Amir Goldstein
0 siblings, 0 replies; 13+ messages in thread
From: Amir Goldstein @ 2020-08-30 11:37 UTC (permalink / raw)
To: Sargun Dhillon
Cc: Aleksa Sarai, Eric W. Biederman, Linux Containers, overlayfs
On Sat, Aug 29, 2020 at 6:41 PM Sargun Dhillon <sargun@sargun.me> wrote:
>
> On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote:
> >
> > On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote:
> > > Amir Goldstein <amir73il@gmail.com> writes:
> > >
> > > > Hi Guys,
> > > >
> > > > It's been nice to virtually meet with you yesterday.
> > > > Some of you wanted to follow up on overlayfs related issues.
> > > >
> > > > If you want to discuss, try to find me in one of the
> > > > https://meet.2020.linuxplumbersconf.org/hackrooms
> > > > today between 16:00-17:00 UTC
> > > > (No need to enter the room to see who's inside)
> > > >
> > > > If those times do not work for you, contact me and we can try
> > > > to schedule another time.
> > >
> > > Did this conversation wind up happening? Do we need to reschedule?
> >
> > This conversation already happened in a Hackroom on Tuesday. I'm not
> > sure if the Hackrooms will have their recordings published, so maybe
> > Amir can post any of the takeaways we had?
> >
> > --
> > Aleksa Sarai
> > Senior Software Engineer (Containers)
> > SUSE Linux GmbH
> > <https://www.cyphar.com/>
>
> I unfortunately missed this conversation. I wanted to bring up OverlayFS, and
> ephemeral upper dirs. We use overlayfs with Docker containers, and we waste
> a lot of time on writing things back to disk.
>
> We're not so peeved about the fact that OVL does any sync operations, as that's
> what our users have been used to. The big problem is on unmount, ovelfs decides
> syncing the upperdirs is a good idea. IIRC, this regression was
> introduced somewhere
> in the 4.X series.
>
> We've been carrying a patch to short-circuit this behaviour for a while now:
> https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253
>
> I know that this behaviour (and any behaviour that short-circuits
> O_SYNC / FUA is
> technically "wrong", but in this case, can we make an exception? I originally
> thought about using device mapper to remove the FUA bit from all BIOs, but it
> turns out that my underlying storage *always* persists data to disk,
> so every write
> takes...a long time.
>
> Amir, what's your take?
It's not only FUA that is causing slow down.
syncfs() takes internal filesystem locks (e.g. to commit a journal transaction),
so it causes interference with other writers on the same underlying filesystem.
As Giuseppe pointed out, a patch has already been submitted to address
this issue.
Thanks,
Amir.
^ permalink raw reply [flat|nested] 13+ messages in thread