* Overlayfs @Plumbers @ 2020-08-25 6:07 Amir Goldstein 2020-08-28 15:33 ` Eric W. Biederman 0 siblings, 1 reply; 7+ messages in thread From: Amir Goldstein @ 2020-08-25 6:07 UTC (permalink / raw) To: Linux Containers; +Cc: overlayfs, Vivek Goyal Hi Guys, It's been nice to virtually meet with you yesterday. Some of you wanted to follow up on overlayfs related issues. If you want to discuss, try to find me in one of the https://meet.2020.linuxplumbersconf.org/hackrooms today between 16:00-17:00 UTC (No need to enter the room to see who's inside) If those times do not work for you, contact me and we can try to schedule another time. Thanks, Amir. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Overlayfs @Plumbers 2020-08-25 6:07 Overlayfs @Plumbers Amir Goldstein @ 2020-08-28 15:33 ` Eric W. Biederman 2020-08-28 15:58 ` Aleksa Sarai 0 siblings, 1 reply; 7+ messages in thread From: Eric W. Biederman @ 2020-08-28 15:33 UTC (permalink / raw) To: Amir Goldstein; +Cc: Linux Containers, overlayfs Amir Goldstein <amir73il@gmail.com> writes: > Hi Guys, > > It's been nice to virtually meet with you yesterday. > Some of you wanted to follow up on overlayfs related issues. > > If you want to discuss, try to find me in one of the > https://meet.2020.linuxplumbersconf.org/hackrooms > today between 16:00-17:00 UTC > (No need to enter the room to see who's inside) > > If those times do not work for you, contact me and we can try > to schedule another time. Did this conversation wind up happening? Do we need to reschedule? Eric ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Overlayfs @Plumbers 2020-08-28 15:33 ` Eric W. Biederman @ 2020-08-28 15:58 ` Aleksa Sarai 2020-08-29 15:40 ` Sargun Dhillon 2020-08-30 11:44 ` Amir Goldstein 0 siblings, 2 replies; 7+ messages in thread From: Aleksa Sarai @ 2020-08-28 15:58 UTC (permalink / raw) To: Eric W. Biederman; +Cc: Amir Goldstein, Linux Containers, overlayfs [-- Attachment #1: Type: text/plain, Size: 933 bytes --] On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote: > Amir Goldstein <amir73il@gmail.com> writes: > > > Hi Guys, > > > > It's been nice to virtually meet with you yesterday. > > Some of you wanted to follow up on overlayfs related issues. > > > > If you want to discuss, try to find me in one of the > > https://meet.2020.linuxplumbersconf.org/hackrooms > > today between 16:00-17:00 UTC > > (No need to enter the room to see who's inside) > > > > If those times do not work for you, contact me and we can try > > to schedule another time. > > Did this conversation wind up happening? Do we need to reschedule? This conversation already happened in a Hackroom on Tuesday. I'm not sure if the Hackrooms will have their recordings published, so maybe Amir can post any of the takeaways we had? -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH <https://www.cyphar.com/> [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Overlayfs @Plumbers 2020-08-28 15:58 ` Aleksa Sarai @ 2020-08-29 15:40 ` Sargun Dhillon 2020-08-30 11:29 ` Giuseppe Scrivano 2020-08-30 11:37 ` Amir Goldstein 2020-08-30 11:44 ` Amir Goldstein 1 sibling, 2 replies; 7+ messages in thread From: Sargun Dhillon @ 2020-08-29 15:40 UTC (permalink / raw) To: Aleksa Sarai Cc: Eric W. Biederman, Amir Goldstein, Linux Containers, overlayfs On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote: > > On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote: > > Amir Goldstein <amir73il@gmail.com> writes: > > > > > Hi Guys, > > > > > > It's been nice to virtually meet with you yesterday. > > > Some of you wanted to follow up on overlayfs related issues. > > > > > > If you want to discuss, try to find me in one of the > > > https://meet.2020.linuxplumbersconf.org/hackrooms > > > today between 16:00-17:00 UTC > > > (No need to enter the room to see who's inside) > > > > > > If those times do not work for you, contact me and we can try > > > to schedule another time. > > > > Did this conversation wind up happening? Do we need to reschedule? > > This conversation already happened in a Hackroom on Tuesday. I'm not > sure if the Hackrooms will have their recordings published, so maybe > Amir can post any of the takeaways we had? > > -- > Aleksa Sarai > Senior Software Engineer (Containers) > SUSE Linux GmbH > <https://www.cyphar.com/> I unfortunately missed this conversation. I wanted to bring up OverlayFS, and ephemeral upper dirs. We use overlayfs with Docker containers, and we waste a lot of time on writing things back to disk. We're not so peeved about the fact that OVL does any sync operations, as that's what our users have been used to. The big problem is on unmount, ovelfs decides syncing the upperdirs is a good idea. IIRC, this regression was introduced somewhere in the 4.X series. We've been carrying a patch to short-circuit this behaviour for a while now: https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253 I know that this behaviour (and any behaviour that short-circuits O_SYNC / FUA is technically "wrong", but in this case, can we make an exception? I originally thought about using device mapper to remove the FUA bit from all BIOs, but it turns out that my underlying storage *always* persists data to disk, so every write takes...a long time. Amir, what's your take? ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Overlayfs @Plumbers 2020-08-29 15:40 ` Sargun Dhillon @ 2020-08-30 11:29 ` Giuseppe Scrivano 2020-08-30 11:37 ` Amir Goldstein 1 sibling, 0 replies; 7+ messages in thread From: Giuseppe Scrivano @ 2020-08-30 11:29 UTC (permalink / raw) To: Sargun Dhillon Cc: Aleksa Sarai, Linux Containers, Eric W. Biederman, overlayfs Hi Sargun, Sargun Dhillon <sargun@sargun.me> writes: Q> On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote: >> >> On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote: >> > Amir Goldstein <amir73il@gmail.com> writes: >> > >> > > Hi Guys, >> > > >> > > It's been nice to virtually meet with you yesterday. >> > > Some of you wanted to follow up on overlayfs related issues. >> > > >> > > If you want to discuss, try to find me in one of the >> > > https://meet.2020.linuxplumbersconf.org/hackrooms >> > > today between 16:00-17:00 UTC >> > > (No need to enter the room to see who's inside) >> > > >> > > If those times do not work for you, contact me and we can try >> > > to schedule another time. >> > >> > Did this conversation wind up happening? Do we need to reschedule? >> >> This conversation already happened in a Hackroom on Tuesday. I'm not >> sure if the Hackrooms will have their recordings published, so maybe >> Amir can post any of the takeaways we had? >> >> -- >> Aleksa Sarai >> Senior Software Engineer (Containers) >> SUSE Linux GmbH >> <https://www.cyphar.com/> > > I unfortunately missed this conversation. I wanted to bring up OverlayFS, and > ephemeral upper dirs. We use overlayfs with Docker containers, and we waste > a lot of time on writing things back to disk. > > We're not so peeved about the fact that OVL does any sync operations, as that's > what our users have been used to. The big problem is on unmount, ovelfs decides > syncing the upperdirs is a good idea. IIRC, this regression was > introduced somewhere > in the 4.X series. > > We've been carrying a patch to short-circuit this behaviour for a while now: > https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253 > > I know that this behaviour (and any behaviour that short-circuits > O_SYNC / FUA is > technically "wrong", but in this case, can we make an exception? I originally > thought about using device mapper to remove the FUA bit from all BIOs, but it > turns out that my underlying storage *always* persists data to disk, > so every write > takes...a long time. > > Amir, what's your take? we are having the same issue when building images. Would the following patch be helpful? https://www.spinics.net/lists/linux-unionfs/msg08267.html Regards, Giuseppe ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Overlayfs @Plumbers 2020-08-29 15:40 ` Sargun Dhillon 2020-08-30 11:29 ` Giuseppe Scrivano @ 2020-08-30 11:37 ` Amir Goldstein 1 sibling, 0 replies; 7+ messages in thread From: Amir Goldstein @ 2020-08-30 11:37 UTC (permalink / raw) To: Sargun Dhillon Cc: Aleksa Sarai, Eric W. Biederman, Linux Containers, overlayfs On Sat, Aug 29, 2020 at 6:41 PM Sargun Dhillon <sargun@sargun.me> wrote: > > On Fri, Aug 28, 2020 at 8:59 AM Aleksa Sarai <asarai@suse.de> wrote: > > > > On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote: > > > Amir Goldstein <amir73il@gmail.com> writes: > > > > > > > Hi Guys, > > > > > > > > It's been nice to virtually meet with you yesterday. > > > > Some of you wanted to follow up on overlayfs related issues. > > > > > > > > If you want to discuss, try to find me in one of the > > > > https://meet.2020.linuxplumbersconf.org/hackrooms > > > > today between 16:00-17:00 UTC > > > > (No need to enter the room to see who's inside) > > > > > > > > If those times do not work for you, contact me and we can try > > > > to schedule another time. > > > > > > Did this conversation wind up happening? Do we need to reschedule? > > > > This conversation already happened in a Hackroom on Tuesday. I'm not > > sure if the Hackrooms will have their recordings published, so maybe > > Amir can post any of the takeaways we had? > > > > -- > > Aleksa Sarai > > Senior Software Engineer (Containers) > > SUSE Linux GmbH > > <https://www.cyphar.com/> > > I unfortunately missed this conversation. I wanted to bring up OverlayFS, and > ephemeral upper dirs. We use overlayfs with Docker containers, and we waste > a lot of time on writing things back to disk. > > We're not so peeved about the fact that OVL does any sync operations, as that's > what our users have been used to. The big problem is on unmount, ovelfs decides > syncing the upperdirs is a good idea. IIRC, this regression was > introduced somewhere > in the 4.X series. > > We've been carrying a patch to short-circuit this behaviour for a while now: > https://github.com/Netflix-Skunkworks/linux/commit/edb195d9b73cc22d095078010a14a690f41ee253 > > I know that this behaviour (and any behaviour that short-circuits > O_SYNC / FUA is > technically "wrong", but in this case, can we make an exception? I originally > thought about using device mapper to remove the FUA bit from all BIOs, but it > turns out that my underlying storage *always* persists data to disk, > so every write > takes...a long time. > > Amir, what's your take? It's not only FUA that is causing slow down. syncfs() takes internal filesystem locks (e.g. to commit a journal transaction), so it causes interference with other writers on the same underlying filesystem. As Giuseppe pointed out, a patch has already been submitted to address this issue. Thanks, Amir. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Overlayfs @Plumbers 2020-08-28 15:58 ` Aleksa Sarai 2020-08-29 15:40 ` Sargun Dhillon @ 2020-08-30 11:44 ` Amir Goldstein 1 sibling, 0 replies; 7+ messages in thread From: Amir Goldstein @ 2020-08-30 11:44 UTC (permalink / raw) To: Aleksa Sarai Cc: Eric W. Biederman, Linux Containers, overlayfs, Josh Triplett On Fri, Aug 28, 2020 at 6:58 PM Aleksa Sarai <asarai@suse.de> wrote: > > On 2020-08-28, Eric W. Biederman <ebiederm@xmission.com> wrote: > > Amir Goldstein <amir73il@gmail.com> writes: > > > > > Hi Guys, > > > > > > It's been nice to virtually meet with you yesterday. > > > Some of you wanted to follow up on overlayfs related issues. > > > > > > If you want to discuss, try to find me in one of the > > > https://meet.2020.linuxplumbersconf.org/hackrooms > > > today between 16:00-17:00 UTC > > > (No need to enter the room to see who's inside) > > > > > > If those times do not work for you, contact me and we can try > > > to schedule another time. > > > > Did this conversation wind up happening? Do we need to reschedule? > > This conversation already happened in a Hackroom on Tuesday. I'm not > sure if the Hackrooms will have their recordings published, so maybe > Amir can post any of the takeaways we had? > AFAIK we had discussed two issues. 1. Watching changes on upper layer 2. OCI format v2 #1 was brought up by Josh Triplett. He asked whether watching changes using inotify/fanotify on upper dir and reading changes from upper dir is a reasonable practice. I suggested that he use fanotify filesystem mark to watch changes on overlay sb and read the changes from overlay mount. #2 TBH I lost track and you guys continued the discussion after I left. From what I gathered we need a way to describe clones and renames more efficiently, but I did not hear any solid plan of how to do that. So perhaps you would be able to fill in some details here. Thanks, Amir. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-08-30 11:44 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-08-25 6:07 Overlayfs @Plumbers Amir Goldstein 2020-08-28 15:33 ` Eric W. Biederman 2020-08-28 15:58 ` Aleksa Sarai 2020-08-29 15:40 ` Sargun Dhillon 2020-08-30 11:29 ` Giuseppe Scrivano 2020-08-30 11:37 ` Amir Goldstein 2020-08-30 11:44 ` Amir Goldstein
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).