* seccomp: Delay filter activation @ 2021-02-20 9:31 Sargun Dhillon 2021-03-01 11:09 ` Christian Brauner 0 siblings, 1 reply; 8+ messages in thread From: Sargun Dhillon @ 2021-02-20 9:31 UTC (permalink / raw) To: Kees Cook Cc: LKML, Giuseppe Scrivano, Christian Brauner, Tycho Andersen, Hariharan Ananthakrishnan, Keerti Lakshminarayan, Kyle Anderson We've run into a problem where attaching a filter can be quite messy business because the filter itself intercepts sendmsg, and other syscalls related to exfiltrating the listener FD. I believe that this problem set has been brought up before, and although there are "simpler" methods of exfiltrating the listener, like clone3 or pidfd_getfd, but these are still less than ideal. One of the ideas that's been talked about (I want to say back at LSS NA) is the idea of "delayed activation". I was thinking that it might be nice to have a mechanism to do delayed attach, either activated on execve / fork, or an ioctl on the listenerfd to activate the filter and have a flag like SECCOMP_FILTER_FLAG_NEW_LISTENER_INACTIVE, which indicates that the listener should be setup, but not enforcing, and another ioctl to activate it. The later approach is preferred due to simplicity, but I can see a situation where you could accidentally get into a state where the filter is not being enforced. Additionally, this may have unforeseen implications with CRIU. I'm curious whether this is a problem others share, and whether any of the aforementioned approaches seem reasonable. -Thanks, Sargun ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: seccomp: Delay filter activation 2021-02-20 9:31 seccomp: Delay filter activation Sargun Dhillon @ 2021-03-01 11:09 ` Christian Brauner 2021-03-01 13:21 ` Christian Brauner 0 siblings, 1 reply; 8+ messages in thread From: Christian Brauner @ 2021-03-01 11:09 UTC (permalink / raw) To: Sargun Dhillon Cc: Kees Cook, LKML, Giuseppe Scrivano, Tycho Andersen, Hariharan Ananthakrishnan, Keerti Lakshminarayan, Kyle Anderson, Linux Containers List, stgraber, Andy Lutomirski On Sat, Feb 20, 2021 at 01:31:57AM -0800, Sargun Dhillon wrote: > We've run into a problem where attaching a filter can be quite messy > business because the filter itself intercepts sendmsg, and other > syscalls related to exfiltrating the listener FD. I believe that this > problem set has been brought up before, and although there are > "simpler" methods of exfiltrating the listener, like clone3 or > pidfd_getfd, but these are still less than ideal. (You really like sending patches and discussion points in the middle of the merge window. :D I think everyone's panicked about getting their PRs in shape so it's not unlikely that this sometimes gets lost on the list. :)) It hasn't been a huge problem for us, especially since we added pidfd_getfd() this seemed like a straightforward problem to solve by selecting a fix fd number that is to be used for the listener. But I can see why it is annoying. > > One of the ideas that's been talked about (I want to say back at LSS > NA) is the idea of "delayed activation". I was thinking that it might > be nice to have a mechanism to do delayed attach, either activated on > execve / fork, or an ioctl on the listenerfd to activate the filter > and have a flag like SECCOMP_FILTER_FLAG_NEW_LISTENER_INACTIVE, which > indicates that the listener should be setup, but not enforcing, and > another ioctl to activate it. > > The later approach is preferred due to simplicity, but I can see a > situation where you could accidentally get into a state where the > filter is not being enforced. Additionally, this may have unforeseen > implications with CRIU. (If you were to expose an ioctl() that allows userspace to query the notifer state then CRIU shouldn't have a problem restoring the notifier in the correct state. Right now it doesn't do anyting fancy about the notifier, it just restores the task with the filter. It just has to learn about the new feature and that's fine imho.) > > I'm curious whether this is a problem others share, and whether any of > the aforementioned approaches seem reasonable. So when I originally suggested the delayed activation I I had another related idea that I think I might have mentioned too: if we're already considering delaying filter activation I like to discuss the possibility of attaching a seccomp filter to a task. Right now, if one task wants to attach to another task they need to recreate the whole seccomp filter and load it. That's not just pretty expensive but also only works if you have access to the rules that the filter was generated with. For container that's usually some sort of pseudo seccomp filter configuration language dumped into a config file from which it can be read. So right now the status quo is: struct sock_filter filter[] = { BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, nr)), BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, nr, 0, 1), BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_USER_NOTIF), /* Get me a listener fd */ BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ALLOW), }; struct sock_fprog prog = { .len = (unsigned short)ARRAY_SIZE(filter), .filter = filter, }; int fd = seccomp(SECCOMP_SET_MODE_FILTER, flags, &prog); and then the caller must send the fd to the manager or the manager uses pidfd_getfd(). But, why not get a bit crazy^wcreative; especially since seccomp() is already a multiplexer. We introduce a new seccomp flag: #define SECCOMP_FILTER_DETACHED and a new seccomp command: #define SECCOMP_ATTACH_FILTER And now we could do something like: pid_t pid = fork(); if (pid < 0) return; if (pid == 0) { // do stuff BARRIER_WAKE_SETUP_DONE; // do more unrelated stuff BARRIER_WAIT_SECCOMP_FILTER; execve(exec-something); } else { int fd_filter; struct sock_filter filter[] = { BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, nr)), BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, nr, 0, 1), BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ALLOW), }; struct sock_fprog prog = { .len = (unsigned short)ARRAY_SIZE(filter), .filter = filter, }; int fd_filter = seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_DETACHED, &prog); BARRIER_WAIT_SETUP_DONE; int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, INT_TO_PTR(fd_listener)); BARRIER_WAKE_SECCOMP_FILTER; } And now you have attached a filter to another task. This would be super elegant for a container manager. The container manager could also stash the filter fd and when attaching to a container the manager can send the attaching task the fd and the attaching task can do: int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, INT_TO_PTR(fd_filter)); too and would be attached to the same filter as the target task. And for the listener fd case a container manager could simply set SECCOMP_RET_USER_NOTIF as before struct sock_filter filter[] = { BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, nr)), BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, nr, 0, 1), BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_USER_NOTIF), BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ALLOW), }; and now fd_filter simply functions as the notifier fd after seccomp(SECCOMP_ATTACH_FILTER) that's basically the fancy version of my delayed notifier activiation idea. I'm sure there's nastiness to figure out but I would love to see something like this. Christian ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: seccomp: Delay filter activation 2021-03-01 11:09 ` Christian Brauner @ 2021-03-01 13:21 ` Christian Brauner 2021-03-01 23:44 ` Kees Cook 0 siblings, 1 reply; 8+ messages in thread From: Christian Brauner @ 2021-03-01 13:21 UTC (permalink / raw) To: Sargun Dhillon Cc: Kees Cook, LKML, Giuseppe Scrivano, Tycho Andersen, Hariharan Ananthakrishnan, Keerti Lakshminarayan, Kyle Anderson, Linux Containers List, stgraber, Andy Lutomirski On Mon, Mar 01, 2021 at 12:09:09PM +0100, Christian Brauner wrote: > On Sat, Feb 20, 2021 at 01:31:57AM -0800, Sargun Dhillon wrote: > > We've run into a problem where attaching a filter can be quite messy > > business because the filter itself intercepts sendmsg, and other > > syscalls related to exfiltrating the listener FD. I believe that this > > problem set has been brought up before, and although there are > > "simpler" methods of exfiltrating the listener, like clone3 or > > pidfd_getfd, but these are still less than ideal. > > (You really like sending patches and discussion points in the middle of > the merge window. :D I think everyone's panicked about getting their PRs > in shape so it's not unlikely that this sometimes gets lost on the list. :)) > > It hasn't been a huge problem for us, especially since we added > pidfd_getfd() this seemed like a straightforward problem to solve by > selecting a fix fd number that is to be used for the listener. But I can > see why it is annoying. > > > > > One of the ideas that's been talked about (I want to say back at LSS > > NA) is the idea of "delayed activation". I was thinking that it might > > be nice to have a mechanism to do delayed attach, either activated on > > execve / fork, or an ioctl on the listenerfd to activate the filter > > and have a flag like SECCOMP_FILTER_FLAG_NEW_LISTENER_INACTIVE, which > > indicates that the listener should be setup, but not enforcing, and > > another ioctl to activate it. > > > > The later approach is preferred due to simplicity, but I can see a > > situation where you could accidentally get into a state where the > > filter is not being enforced. Additionally, this may have unforeseen > > implications with CRIU. > > (If you were to expose an ioctl() that allows userspace to query the > notifer state then CRIU shouldn't have a problem restoring the notifier > in the correct state. Right now it doesn't do anyting fancy about the > notifier, it just restores the task with the filter. It just has to > learn about the new feature and that's fine imho.) > > > > > I'm curious whether this is a problem others share, and whether any of > > the aforementioned approaches seem reasonable. > > So when I originally suggested the delayed activation I I had another > related idea that I think I might have mentioned too: if we're already > considering delaying filter activation I like to discuss the possibility > of attaching a seccomp filter to a task. > > Right now, if one task wants to attach to another task they need to > recreate the whole seccomp filter and load it. That's not just pretty > expensive but also only works if you have access to the rules that the > filter was generated with. For container that's usually some sort of > pseudo seccomp filter configuration language dumped into a config file > from which it can be read. > > So right now the status quo is: > > struct sock_filter filter[] = { > BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, nr)), > BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, nr, 0, 1), > BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_USER_NOTIF), /* Get me a listener fd */ > BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ALLOW), > }; > struct sock_fprog prog = { > .len = (unsigned short)ARRAY_SIZE(filter), > .filter = filter, > }; > int fd = seccomp(SECCOMP_SET_MODE_FILTER, flags, &prog); > > and then the caller must send the fd to the manager or the manager uses > pidfd_getfd(). > > But, why not get a bit crazy^wcreative; especially since seccomp() is > already a multiplexer. We introduce a new seccomp flag: > > #define SECCOMP_FILTER_DETACHED > > and a new seccomp command: > > #define SECCOMP_ATTACH_FILTER > > And now we could do something like: > > pid_t pid = fork(); > if (pid < 0) > return; > > if (pid == 0) { > // do stuff > BARRIER_WAKE_SETUP_DONE; > > // do more unrelated stuff > > BARRIER_WAIT_SECCOMP_FILTER; > execve(exec-something); > } else { > > int fd_filter; > > struct sock_filter filter[] = { > BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, nr)), > BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, nr, 0, 1), > BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ALLOW), > }; > > struct sock_fprog prog = { > .len = (unsigned short)ARRAY_SIZE(filter), > .filter = filter, > }; > > int fd_filter = seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_DETACHED, &prog); > > BARRIER_WAIT_SETUP_DONE; > > int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, INT_TO_PTR(fd_listener)); This obviously should've been sm like: struct seccomp_filter_attach { union { __s32 pidfd; __s32 pid; }; __u32 fd_filter; }; and then int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, seccomp_filter_attach); > > BARRIER_WAKE_SECCOMP_FILTER; > } > > And now you have attached a filter to another task. This would be super > elegant for a container manager. The container manager could also stash > the filter fd and when attaching to a container the manager can send the > attaching task the fd and the attaching task can do: > > int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, INT_TO_PTR(fd_filter)); > > too and would be attached to the same filter as the target task. > > And for the listener fd case a container manager could simply set > SECCOMP_RET_USER_NOTIF as before > > struct sock_filter filter[] = { > BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, nr)), > BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, nr, 0, 1), > BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_USER_NOTIF), > BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ALLOW), > }; > > and now fd_filter simply functions as the notifier fd after > seccomp(SECCOMP_ATTACH_FILTER) that's basically the fancy version of my > delayed notifier activiation idea. > > I'm sure there's nastiness to figure out but I would love to see > something like this. > > Christian ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: seccomp: Delay filter activation 2021-03-01 13:21 ` Christian Brauner @ 2021-03-01 23:44 ` Kees Cook 2021-03-18 14:54 ` Christian Brauner 0 siblings, 1 reply; 8+ messages in thread From: Kees Cook @ 2021-03-01 23:44 UTC (permalink / raw) To: Christian Brauner Cc: Sargun Dhillon, LKML, Giuseppe Scrivano, Tycho Andersen, Hariharan Ananthakrishnan, Keerti Lakshminarayan, Kyle Anderson, Linux Containers List, stgraber, Andy Lutomirski On Mon, Mar 01, 2021 at 02:21:56PM +0100, Christian Brauner wrote: > On Mon, Mar 01, 2021 at 12:09:09PM +0100, Christian Brauner wrote: > > On Sat, Feb 20, 2021 at 01:31:57AM -0800, Sargun Dhillon wrote: > > > We've run into a problem where attaching a filter can be quite messy > > > business because the filter itself intercepts sendmsg, and other > > > syscalls related to exfiltrating the listener FD. I believe that this > > > problem set has been brought up before, and although there are > > > "simpler" methods of exfiltrating the listener, like clone3 or > > > pidfd_getfd, but these are still less than ideal. I'm trying to make sure I understand: the target process would like to have a filter attached that blocks sendmsg, but that would mean it has no way to send the listener FD to its manager? And you'd want to have listening working for sendmsg (otherwise you could do it with two filters, I imagine)? > > int fd_filter = seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_DETACHED, &prog); > > > > BARRIER_WAIT_SETUP_DONE; > > > > int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, INT_TO_PTR(fd_listener)); > > This obviously should've been sm like: > > struct seccomp_filter_attach { > union { > __s32 pidfd; > __s32 pid; > }; > __u32 fd_filter; > }; > > and then > > int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, seccomp_filter_attach); Given the difficulty with TSYNC, I'm not excited about adding an "apply this filter to another process" API. :) The prior thread was here: https://lore.kernel.org/lkml/20201029075841.GB29881@ircssh-2.c.rugged-nimbus-611.internal/ But I haven't had time to follow up. Both Andy and Sargun discuss filter "replacement", but I'm not a fan of that, since I'd really like to keep the "additive-only" property of seccomp. So, I'm still back to wanting an answer to my questions at the end of https://lore.kernel.org/lkml/202010281503.3D1FCFE0@keescook/ Namely, how to best indicate the point of execution where "delayed" filters become applied? If we require supporting the "2b" (launched oblivious target) case (which I think we must), we need to signal it externally, or via an automatic trip point. Since synchronizing with an oblivious target is rather nasty (e.g. involving ptrace or at least ptrace access checking), I'd rather create a predefined trip point. Having it be "execve" limits the utility of this feature for cooperating targets, though, so I think "apply on exec" isn't great. struct seccomp_filter_attach_trigger { u64 nr; unsigned char *filter; }; seccomp(SECCOMP_ATTACH_FILTER_TRIGGER, 0, seccomp_filter_attach_trigger); after "nr" is evaluated (but before it runs), seccomp installs the filter. And by "installs", I'm not sure if it needs to keep it in a queue, with separate ref coutning, or if it should be in the main filter stack, but have an "alive" toggle, or what. -- Kees Cook ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: seccomp: Delay filter activation 2021-03-01 23:44 ` Kees Cook @ 2021-03-18 14:54 ` Christian Brauner 2021-03-18 20:39 ` Sargun Dhillon 0 siblings, 1 reply; 8+ messages in thread From: Christian Brauner @ 2021-03-18 14:54 UTC (permalink / raw) To: Kees Cook Cc: Sargun Dhillon, LKML, Giuseppe Scrivano, Tycho Andersen, Hariharan Ananthakrishnan, Keerti Lakshminarayan, Kyle Anderson, Linux Containers List, stgraber, Andy Lutomirski Sorry, I just found that mail. On Mon, Mar 01, 2021 at 03:44:06PM -0800, Kees Cook wrote: > On Mon, Mar 01, 2021 at 02:21:56PM +0100, Christian Brauner wrote: > > On Mon, Mar 01, 2021 at 12:09:09PM +0100, Christian Brauner wrote: > > > On Sat, Feb 20, 2021 at 01:31:57AM -0800, Sargun Dhillon wrote: > > > > We've run into a problem where attaching a filter can be quite messy > > > > business because the filter itself intercepts sendmsg, and other > > > > syscalls related to exfiltrating the listener FD. I believe that this > > > > problem set has been brought up before, and although there are > > > > "simpler" methods of exfiltrating the listener, like clone3 or > > > > pidfd_getfd, but these are still less than ideal. > > I'm trying to make sure I understand: the target process would like to > have a filter attached that blocks sendmsg, but that would mean it has > no way to send the listener FD to its manager? With pidfd_getfd() that wouldn't be a problem, I think which is what I was trying to say. Unless the supervising task doen't have enough privilege over the supervised task which seems like an odd scenario but is technically possible, I guess. > > And you'd want to have listening working for sendmsg (otherwise you > could do it with two filters, I imagine)? > > > > int fd_filter = seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_DETACHED, &prog); > > > > > > BARRIER_WAIT_SETUP_DONE; > > > > > > int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, INT_TO_PTR(fd_listener)); > > > > This obviously should've been sm like: > > > > struct seccomp_filter_attach { > > union { > > __s32 pidfd; > > __s32 pid; > > }; > > __u32 fd_filter; > > }; > > > > and then > > > > int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, seccomp_filter_attach); > > Given the difficulty with TSYNC, I'm not excited about adding an > "apply this filter to another process" API. :) Just to give a more complete reason for suggesting something like this without trying to argue that we must have this: seccomp() has so far been an API that is caller-centric and by that I mean that the caller loaded it's seccomp profile and sandboxed itself. As such seccomp is an example of "caller-managed" security. This security model has obvious advantages and fits into the general fork()-like world of unix. But imho that self-management model breaks down as soon as a file descriptor that can be used to refer to the object in question enters into the picture. For seccomp this "breaking point" was the seccomp notifier fd. Because with the introduction of that fd we have introduced the concept of supervisor and supervisee for seccomp which imho didn't really exist in the same way before. It's pretty obvious from the type of language that we now use both in userspace and in kernelspace when we talk about the seccomp notifier. At the current point we're somewhere in the middle between caller-managed and supervised seccomp which brings up funny probelms and edge-cases. One of them most obvious examples is in fact the question how to get the seccomp notify fd out of the supervised task. This clearly points to the fact that we're missing one of the fundamentals of an fd-based supervision model: open(). This is why I was suggesting the SECCOMP_ATTACH_FILTER command. It's in a sense an open-call for the seccomp notify fd. That all being said I know that it can be weird to implement this and if you prefer we go with another simpler model to work around such things than I fully understand. Christian ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: seccomp: Delay filter activation 2021-03-18 14:54 ` Christian Brauner @ 2021-03-18 20:39 ` Sargun Dhillon 2021-03-19 10:06 ` Rodrigo Campos 2021-03-19 11:03 ` Christian Brauner 0 siblings, 2 replies; 8+ messages in thread From: Sargun Dhillon @ 2021-03-18 20:39 UTC (permalink / raw) To: Christian Brauner Cc: Kees Cook, LKML, Giuseppe Scrivano, Tycho Andersen, Hariharan Ananthakrishnan, Keerti Lakshminarayan, Kyle Anderson, Linux Containers List, stgraber, Andy Lutomirski On Thu, Mar 18, 2021 at 03:54:54PM +0100, Christian Brauner wrote: > Sorry, I just found that mail. > > On Mon, Mar 01, 2021 at 03:44:06PM -0800, Kees Cook wrote: > > On Mon, Mar 01, 2021 at 02:21:56PM +0100, Christian Brauner wrote: > > > On Mon, Mar 01, 2021 at 12:09:09PM +0100, Christian Brauner wrote: > > > > On Sat, Feb 20, 2021 at 01:31:57AM -0800, Sargun Dhillon wrote: > > > > > We've run into a problem where attaching a filter can be quite messy > > > > > business because the filter itself intercepts sendmsg, and other > > > > > syscalls related to exfiltrating the listener FD. I believe that this > > > > > problem set has been brought up before, and although there are > > > > > "simpler" methods of exfiltrating the listener, like clone3 or > > > > > pidfd_getfd, but these are still less than ideal. > > > > I'm trying to make sure I understand: the target process would like to > > have a filter attached that blocks sendmsg, but that would mean it has > > no way to send the listener FD to its manager? > > With pidfd_getfd() that wouldn't be a problem, I think which is what I > was trying to say. Unless the supervising task doen't have enough > privilege over the supervised task which seems like an odd scenario but > is technically possible, I guess. > > > > > And you'd want to have listening working for sendmsg (otherwise you > > could do it with two filters, I imagine)? > > > > > > int fd_filter = seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_DETACHED, &prog); > > > > > > > > BARRIER_WAIT_SETUP_DONE; > > > > > > > > int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, INT_TO_PTR(fd_listener)); > > > > > > This obviously should've been sm like: > > > > > > struct seccomp_filter_attach { > > > union { > > > __s32 pidfd; > > > __s32 pid; > > > }; > > > __u32 fd_filter; > > > }; > > > > > > and then > > > > > > int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, seccomp_filter_attach); > > > > Given the difficulty with TSYNC, I'm not excited about adding an > > "apply this filter to another process" API. :) > > Just to give a more complete reason for suggesting something like this > without trying to argue that we must have this: > > seccomp() has so far been an API that is caller-centric and by that I > mean that the caller loaded it's seccomp profile and sandboxed itself. As > such seccomp is an example of "caller-managed" security. This security > model has obvious advantages and fits into the general fork()-like world > of unix. But imho that self-management model breaks down as soon as a > file descriptor that can be used to refer to the object in question > enters into the picture. For seccomp this "breaking point" was the > seccomp notifier fd. > > Because with the introduction of that fd we have introduced the concept > of supervisor and supervisee for seccomp which imho didn't really exist > in the same way before. It's pretty obvious from the type of language > that we now use both in userspace and in kernelspace when we talk about > the seccomp notifier. > > At the current point we're somewhere in the middle between caller-managed > and supervised seccomp which brings up funny probelms and edge-cases. > One of them most obvious examples is in fact the question how to get the > seccomp notify fd out of the supervised task. This clearly points to the > fact that we're missing one of the fundamentals of an fd-based > supervision model: open(). This is why I was suggesting the > SECCOMP_ATTACH_FILTER command. It's in a sense an open-call for the > seccomp notify fd. > > That all being said I know that it can be weird to implement this and if > you prefer we go with another simpler model to work around such things > than I fully understand. > > Christian So, beyond clone3 to get pidfds being kind of awkward, how do you see this pattern actually working? How does the filter installer let the supervisor know that it's ready for extraction? pause() + signaling the parent? In the case that you're not fork-execing, how do you communicate to the notifier? My coworkers have been working on this code, where they need to connect to a daemon that does the supervision, and it's gnarly[1]. They're looking at adding sendmsg to the filter list, and that complicates things. I think that pidfd_getfd works well if the child has some way to signal to the parent that it's ready and that the filter has been installed, but when your filter intercepts connect, and sendmsg, and the supervisor is not your parent (and your parent is some compiled, COTS software), it becomes complicated to handle this. I believe that the OCI spec[2] is going to run into this class of problem unless we introduce an out of band signaling mechanism. I think a valid way to handle this is do a send() of the fd number (literal), and wait for the other side to pidfd_getfd the seccomp filter, and wait for the socket to be closed to continue, but I think we should maybe create an example (I volunteer) showing how to do this. [1]: https://github.com/Netflix/titus-executor/blob/393d71fa3b4ab836c6036c6a242a7f4fd2d0307e/tini/src/seccomp_fd_notify.c#L139-L201 [2]: https://github.com/opencontainers/runtime-spec/commit/a8c4a9ee0f6b5a0b994c5c23c68725394e2b0d9d ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: seccomp: Delay filter activation 2021-03-18 20:39 ` Sargun Dhillon @ 2021-03-19 10:06 ` Rodrigo Campos 2021-03-19 11:03 ` Christian Brauner 1 sibling, 0 replies; 8+ messages in thread From: Rodrigo Campos @ 2021-03-19 10:06 UTC (permalink / raw) To: Sargun Dhillon Cc: Christian Brauner, Giuseppe Scrivano, Kees Cook, Keerti Lakshminarayan, Linux Containers List, LKML, Hariharan Ananthakrishnan, Kyle Anderson, Andy Lutomirski On Thu, Mar 18, 2021 at 9:39 PM Sargun Dhillon <sargun@sargun.me> wrote: > I believe that the OCI spec[2] is going to run into this class of problem unless > we introduce an out of band signaling mechanism. I think a valid way to handle > this is do a send() of the fd number (literal), and wait for the other side to > pidfd_getfd the seccomp filter, and wait for the socket to be closed to continue, > but I think we should maybe create an example (I volunteer) showing how to do this. Well, we created a runc implementation for that OCI spec change and we hit exactly that[1]. runc has a pipe mechanism to communicate already, so we use that. What we do is: do the seccomp syscall, send the plain fd number over the pipe and the parent gets the fd with pidfd_getfd()[2]. We use the pipe to sync, so no issues with that part. But, of course, if the seccomp filter blocks the syscall to send over the pipe, this fails. Christian, can you please elaborate on how you solve this on lxd? I'm curious to understand if we can use the same in runc or not. [1]: https://github.com/opencontainers/runc/pull/2682 [2]: https://github.com/opencontainers/runc/pull/2682/files#diff-f0214a0f16408fc7f168c6fc9837d189590025cc1813ebf7c1d751136936dfbfR172 -- Rodrigo Campos --- Kinvolk GmbH | Adalbertstr.6a, 10999 Berlin | tel: +491755589364 Geschäftsführer/Directors: Alban Crequy, Chris Kühl, Iago López Galeiras Registergericht/Court of registration: Amtsgericht Charlottenburg Registernummer/Registration number: HRB 171414 B Ust-ID-Nummer/VAT ID number: DE302207000 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: seccomp: Delay filter activation 2021-03-18 20:39 ` Sargun Dhillon 2021-03-19 10:06 ` Rodrigo Campos @ 2021-03-19 11:03 ` Christian Brauner 1 sibling, 0 replies; 8+ messages in thread From: Christian Brauner @ 2021-03-19 11:03 UTC (permalink / raw) To: Sargun Dhillon Cc: Kees Cook, LKML, Giuseppe Scrivano, Tycho Andersen, Hariharan Ananthakrishnan, Keerti Lakshminarayan, Kyle Anderson, Linux Containers List, stgraber, Andy Lutomirski On Thu, Mar 18, 2021 at 08:39:13PM +0000, Sargun Dhillon wrote: > On Thu, Mar 18, 2021 at 03:54:54PM +0100, Christian Brauner wrote: > > Sorry, I just found that mail. > > > > On Mon, Mar 01, 2021 at 03:44:06PM -0800, Kees Cook wrote: > > > On Mon, Mar 01, 2021 at 02:21:56PM +0100, Christian Brauner wrote: > > > > On Mon, Mar 01, 2021 at 12:09:09PM +0100, Christian Brauner wrote: > > > > > On Sat, Feb 20, 2021 at 01:31:57AM -0800, Sargun Dhillon wrote: > > > > > > We've run into a problem where attaching a filter can be quite messy > > > > > > business because the filter itself intercepts sendmsg, and other > > > > > > syscalls related to exfiltrating the listener FD. I believe that this > > > > > > problem set has been brought up before, and although there are > > > > > > "simpler" methods of exfiltrating the listener, like clone3 or > > > > > > pidfd_getfd, but these are still less than ideal. > > > > > > I'm trying to make sure I understand: the target process would like to > > > have a filter attached that blocks sendmsg, but that would mean it has > > > no way to send the listener FD to its manager? > > > > With pidfd_getfd() that wouldn't be a problem, I think which is what I > > was trying to say. Unless the supervising task doen't have enough > > privilege over the supervised task which seems like an odd scenario but > > is technically possible, I guess. > > > > > > > > And you'd want to have listening working for sendmsg (otherwise you > > > could do it with two filters, I imagine)? > > > > > > > > int fd_filter = seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_DETACHED, &prog); > > > > > > > > > > BARRIER_WAIT_SETUP_DONE; > > > > > > > > > > int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, INT_TO_PTR(fd_listener)); > > > > > > > > This obviously should've been sm like: > > > > > > > > struct seccomp_filter_attach { > > > > union { > > > > __s32 pidfd; > > > > __s32 pid; > > > > }; > > > > __u32 fd_filter; > > > > }; > > > > > > > > and then > > > > > > > > int ret = seccomp(SECCOMP_ATTACH_FILTER, 0, seccomp_filter_attach); > > > > > > Given the difficulty with TSYNC, I'm not excited about adding an > > > "apply this filter to another process" API. :) > > > > Just to give a more complete reason for suggesting something like this > > without trying to argue that we must have this: > > > > seccomp() has so far been an API that is caller-centric and by that I > > mean that the caller loaded it's seccomp profile and sandboxed itself. As > > such seccomp is an example of "caller-managed" security. This security > > model has obvious advantages and fits into the general fork()-like world > > of unix. But imho that self-management model breaks down as soon as a > > file descriptor that can be used to refer to the object in question > > enters into the picture. For seccomp this "breaking point" was the > > seccomp notifier fd. > > > > Because with the introduction of that fd we have introduced the concept > > of supervisor and supervisee for seccomp which imho didn't really exist > > in the same way before. It's pretty obvious from the type of language > > that we now use both in userspace and in kernelspace when we talk about > > the seccomp notifier. > > > > At the current point we're somewhere in the middle between caller-managed > > and supervised seccomp which brings up funny probelms and edge-cases. > > One of them most obvious examples is in fact the question how to get the > > seccomp notify fd out of the supervised task. This clearly points to the > > fact that we're missing one of the fundamentals of an fd-based > > supervision model: open(). This is why I was suggesting the > > SECCOMP_ATTACH_FILTER command. It's in a sense an open-call for the > > seccomp notify fd. > > > > That all being said I know that it can be weird to implement this and if > > you prefer we go with another simpler model to work around such things > > than I fully understand. > > > > Christian > > So, beyond clone3 to get pidfds being kind of awkward, how do you see this Afaict we're talking about getting a file descriptor out of another task. So just for my understanding what does clone3()'s way of returning pidfds have to do with this and why is returning a pidfd in clone3() awkward? > pattern actually working? How does the filter installer let the supervisor Sorry, what pattern are you referring to, i.e. is this a critique/questions of my attach api proposal or of my statement about using pidfd_getfd() to retrieve the fd? I think I'm just a little confused about where you chose to place your reply in the thread. :) Just to reiterate briefly: my preferred variant is to have an api similar to what I proposed above where we can essentially prepare a filter for another task and attach it to that task. In that model the fd is retrieved directly by the supervisor without any need for cooperation from the supervisee. After that you just need simply barriers (pipe, socket etc.) to synchronize with the supervisee. For example, a supervise does all the setup it wants and then sends a message over a socket or pipe to the supervisor signaling it's ready and then the parent attaches the child to its seccomp filter via some form of the command I proposed above. > know that it's ready for extraction? pause() + signaling the parent? > > In the case that you're not fork-execing, how do you communicate to the > notifier? My coworkers have been working on this code, where they need to > connect to a daemon that does the supervision, and it's gnarly[1]. They're > looking at adding sendmsg to the filter list, and that complicates things. > > I think that pidfd_getfd works well if the child has some way to signal to the > parent that it's ready and that the filter has been installed, but when your > filter intercepts connect, and sendmsg, and the supervisor is not your parent > (and your parent is some compiled, COTS software), it becomes complicated to > handle this. > > I believe that the OCI spec[2] is going to run into this class of problem unless > we introduce an out of band signaling mechanism. I think a valid way to handle > this is do a send() of the fd number (literal), and wait for the other side to > pidfd_getfd the seccomp filter, and wait for the socket to be closed to continue, > but I think we should maybe create an example (I volunteer) showing how to do this. So at first it seems were looking at very different cases here: - Supervisor and supervisee do not have a child-parent relationship - Supervisor and supervisee do have a child-parent relationship But in both scenarios you must have a way for the supervisor and supervisee to communicate with each other. If you don't then I'd be curious how you manage to supervise the setup process of the supervisee even independent of the seccomp notifier. (Your first link seems to show a non-child-parent supervisor-supervisee relationship and you communicate between supervisor and supervisee via a socket.) I think you have two immediate ways of solving this problem (Possibly more but I would need to think a bit more and I think that the two solutions would solve most cases.): 1. Send the fd number and use pidfd_getfd() (Which is also what Rodrigo mentioned.). This can technically be a problem when somehow the supervisee is blocked from using any kind of write method to a pipe which I find to be a very artifical example. And in fact looking at [1] runC is writing to a fifo after the seccomp profile has been loaded so a similar method of telling the parent which fd number to retrieve should always work. 2. Use a shared memory location: Have the supervisor create a file created via memfd_create(). Send that fd to the supervisee. (Make the fd close-on-exec in case you for some reason want to block close() which again looking at [1] would mean you'd violate assumptions about what syscalls are available after loading the seccomp profile.) Have the supervisee mmap() that location (In case you're blocking write which again, doesn't work with current assumption.). Have it load it's seccomp profile. Write the fd number to that memory location. The mmap() memory will be cleared on exec so no need to have munmap() available. But again, I prefer a model I suggested above. [1]: https://github.com/opencontainers/runc/blob/4d4d19ce528ac40cc357ef92cd3a6931dba19316/libcontainer/standard_init_linux.go#L192 Christian ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-03-19 11:06 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-02-20 9:31 seccomp: Delay filter activation Sargun Dhillon 2021-03-01 11:09 ` Christian Brauner 2021-03-01 13:21 ` Christian Brauner 2021-03-01 23:44 ` Kees Cook 2021-03-18 14:54 ` Christian Brauner 2021-03-18 20:39 ` Sargun Dhillon 2021-03-19 10:06 ` Rodrigo Campos 2021-03-19 11:03 ` Christian Brauner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).