* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures [not found] <8584c14f-5c28-9d70-c054-7c78127d84ea@arm.com> @ 2020-10-22 7:18 ` Lennart Poettering [not found] ` <20201022075447.GO3819@arm.com> 2020-10-26 16:24 ` Dave Martin 2 siblings, 0 replies; 88+ messages in thread From: Lennart Poettering @ 2020-10-22 7:18 UTC (permalink / raw) To: Jeremy Linton Cc: linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Kees Cook, Szabolcs Nagy, Catalin Marinas, Will Deacon, Mark Brown, Dave Martin On Mi, 21.10.20 22:44, Jeremy Linton (jeremy.linton@arm.com) wrote: > Hi, > > There is a problem with glibc+systemd on BTI enabled systems. Systemd > has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny > PROT_EXEC changes. Glibc enables BTI only on segments which are marked as > being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is > caught by the seccomp filter, resulting in service failures. > > So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. > This is obviously not desirable. > > Various changes have been suggested, replacing the mprotect with mmap calls > having PROT_BTI set on the original mapping, re-mmapping the segments, > implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, > and various modification to seccomp to allow particular mprotect cases to > bypass the filters. In each case there seems to be an undesirable attribute > to the solution. > > So, whats the best solution? Did you see Topi's comments on the systemd issue? https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 I think I agree with this: it's a bit weird to alter the bits after the fact. Can't glibc set up everything right from the begining? That would keep both concepts working. Lennart -- Lennart Poettering, Berlin ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 7:18 ` Lennart Poettering 0 siblings, 0 replies; 88+ messages in thread From: Lennart Poettering @ 2020-10-22 7:18 UTC (permalink / raw) To: Jeremy Linton Cc: Mark Rutland, systemd-devel, Kees Cook, Szabolcs Nagy, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On Mi, 21.10.20 22:44, Jeremy Linton (jeremy.linton@arm.com) wrote: > Hi, > > There is a problem with glibc+systemd on BTI enabled systems. Systemd > has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny > PROT_EXEC changes. Glibc enables BTI only on segments which are marked as > being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is > caught by the seccomp filter, resulting in service failures. > > So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. > This is obviously not desirable. > > Various changes have been suggested, replacing the mprotect with mmap calls > having PROT_BTI set on the original mapping, re-mmapping the segments, > implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, > and various modification to seccomp to allow particular mprotect cases to > bypass the filters. In each case there seems to be an undesirable attribute > to the solution. > > So, whats the best solution? Did you see Topi's comments on the systemd issue? https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 I think I agree with this: it's a bit weird to alter the bits after the fact. Can't glibc set up everything right from the begining? That would keep both concepts working. Lennart -- Lennart Poettering, Berlin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 7:18 ` Lennart Poettering @ 2020-10-22 7:54 ` Florian Weimer -1 siblings, 0 replies; 88+ messages in thread From: Florian Weimer @ 2020-10-22 7:54 UTC (permalink / raw) To: Lennart Poettering Cc: Jeremy Linton, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel * Lennart Poettering: > On Mi, 21.10.20 22:44, Jeremy Linton (jeremy.linton@arm.com) wrote: > >> Hi, >> >> There is a problem with glibc+systemd on BTI enabled systems. Systemd >> has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny >> PROT_EXEC changes. Glibc enables BTI only on segments which are marked as >> being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is >> caught by the seccomp filter, resulting in service failures. >> >> So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. >> This is obviously not desirable. >> >> Various changes have been suggested, replacing the mprotect with mmap calls >> having PROT_BTI set on the original mapping, re-mmapping the segments, >> implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, >> and various modification to seccomp to allow particular mprotect cases to >> bypass the filters. In each case there seems to be an undesirable attribute >> to the solution. >> >> So, whats the best solution? > > Did you see Topi's comments on the systemd issue? > > https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 > > I think I agree with this: it's a bit weird to alter the bits after > the fact. Can't glibc set up everything right from the begining? That > would keep both concepts working. The dynamic loader has to process the LOAD segments to get to the ELF note that says to enable BTI. Maybe we could do a first pass and load only the segments that cover notes. But that requires lots of changes to generic code in the loader. Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 7:54 ` Florian Weimer 0 siblings, 0 replies; 88+ messages in thread From: Florian Weimer @ 2020-10-22 7:54 UTC (permalink / raw) To: Lennart Poettering Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel * Lennart Poettering: > On Mi, 21.10.20 22:44, Jeremy Linton (jeremy.linton@arm.com) wrote: > >> Hi, >> >> There is a problem with glibc+systemd on BTI enabled systems. Systemd >> has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny >> PROT_EXEC changes. Glibc enables BTI only on segments which are marked as >> being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is >> caught by the seccomp filter, resulting in service failures. >> >> So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. >> This is obviously not desirable. >> >> Various changes have been suggested, replacing the mprotect with mmap calls >> having PROT_BTI set on the original mapping, re-mmapping the segments, >> implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, >> and various modification to seccomp to allow particular mprotect cases to >> bypass the filters. In each case there seems to be an undesirable attribute >> to the solution. >> >> So, whats the best solution? > > Did you see Topi's comments on the systemd issue? > > https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 > > I think I agree with this: it's a bit weird to alter the bits after > the fact. Can't glibc set up everything right from the begining? That > would keep both concepts working. The dynamic loader has to process the LOAD segments to get to the ELF note that says to enable BTI. Maybe we could do a first pass and load only the segments that cover notes. But that requires lots of changes to generic code in the loader. Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 7:54 ` Florian Weimer @ 2020-10-22 8:17 ` Topi Miettinen -1 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 8:17 UTC (permalink / raw) To: Florian Weimer, Lennart Poettering Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On 22.10.2020 10.54, Florian Weimer wrote: > * Lennart Poettering: > >> On Mi, 21.10.20 22:44, Jeremy Linton (jeremy.linton@arm.com) wrote: >> >>> Hi, >>> >>> There is a problem with glibc+systemd on BTI enabled systems. Systemd >>> has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny >>> PROT_EXEC changes. Glibc enables BTI only on segments which are marked as >>> being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is >>> caught by the seccomp filter, resulting in service failures. >>> >>> So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. >>> This is obviously not desirable. >>> >>> Various changes have been suggested, replacing the mprotect with mmap calls >>> having PROT_BTI set on the original mapping, re-mmapping the segments, >>> implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, >>> and various modification to seccomp to allow particular mprotect cases to >>> bypass the filters. In each case there seems to be an undesirable attribute >>> to the solution. >>> >>> So, whats the best solution? >> >> Did you see Topi's comments on the systemd issue? >> >> https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 >> >> I think I agree with this: it's a bit weird to alter the bits after >> the fact. Can't glibc set up everything right from the begining? That >> would keep both concepts working. > > The dynamic loader has to process the LOAD segments to get to the ELF > note that says to enable BTI. Maybe we could do a first pass and load > only the segments that cover notes. But that requires lots of changes > to generic code in the loader. What if the loader always enabled BTI for PROT_EXEC pages, but then when discovering that this was a mistake, mprotect() the pages without BTI? Then both BTI and MDWX would work and the penalty of not getting MDWX would fall to non-BTI programs. What's the expected proportion of BTI enabled code vs. disabled in the future, is it perhaps expected that a distro would enable the flag globally so eventually only a few legacy programs might be unprotected? -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 8:17 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 8:17 UTC (permalink / raw) To: Florian Weimer, Lennart Poettering Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On 22.10.2020 10.54, Florian Weimer wrote: > * Lennart Poettering: > >> On Mi, 21.10.20 22:44, Jeremy Linton (jeremy.linton@arm.com) wrote: >> >>> Hi, >>> >>> There is a problem with glibc+systemd on BTI enabled systems. Systemd >>> has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny >>> PROT_EXEC changes. Glibc enables BTI only on segments which are marked as >>> being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is >>> caught by the seccomp filter, resulting in service failures. >>> >>> So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. >>> This is obviously not desirable. >>> >>> Various changes have been suggested, replacing the mprotect with mmap calls >>> having PROT_BTI set on the original mapping, re-mmapping the segments, >>> implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, >>> and various modification to seccomp to allow particular mprotect cases to >>> bypass the filters. In each case there seems to be an undesirable attribute >>> to the solution. >>> >>> So, whats the best solution? >> >> Did you see Topi's comments on the systemd issue? >> >> https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 >> >> I think I agree with this: it's a bit weird to alter the bits after >> the fact. Can't glibc set up everything right from the begining? That >> would keep both concepts working. > > The dynamic loader has to process the LOAD segments to get to the ELF > note that says to enable BTI. Maybe we could do a first pass and load > only the segments that cover notes. But that requires lots of changes > to generic code in the loader. What if the loader always enabled BTI for PROT_EXEC pages, but then when discovering that this was a mistake, mprotect() the pages without BTI? Then both BTI and MDWX would work and the penalty of not getting MDWX would fall to non-BTI programs. What's the expected proportion of BTI enabled code vs. disabled in the future, is it perhaps expected that a distro would enable the flag globally so eventually only a few legacy programs might be unprotected? -Topi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 8:17 ` Topi Miettinen @ 2020-10-22 8:25 ` Florian Weimer -1 siblings, 0 replies; 88+ messages in thread From: Florian Weimer @ 2020-10-22 8:25 UTC (permalink / raw) To: Topi Miettinen Cc: Lennart Poettering, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel * Topi Miettinen: >> The dynamic loader has to process the LOAD segments to get to the ELF >> note that says to enable BTI. Maybe we could do a first pass and >> load only the segments that cover notes. But that requires lots of >> changes to generic code in the loader. > > What if the loader always enabled BTI for PROT_EXEC pages, but then > when discovering that this was a mistake, mprotect() the pages without > BTI? Is that architecturally supported? How costly is the mprotect change if the pages have not been faulted in yet? > Then both BTI and MDWX would work and the penalty of not getting > MDWX would fall to non-BTI programs. What's the expected proportion of > BTI enabled code vs. disabled in the future, is it perhaps expected > that a distro would enable the flag globally so eventually only a few > legacy programs might be unprotected? Eventually, I expect that mainstream distributions build everything for BTI, so yes, the PROT_BTI removal would only be needed for legacy programs. Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 8:25 ` Florian Weimer 0 siblings, 0 replies; 88+ messages in thread From: Florian Weimer @ 2020-10-22 8:25 UTC (permalink / raw) To: Topi Miettinen Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, Lennart Poettering, libc-alpha, Dave Martin, linux-arm-kernel * Topi Miettinen: >> The dynamic loader has to process the LOAD segments to get to the ELF >> note that says to enable BTI. Maybe we could do a first pass and >> load only the segments that cover notes. But that requires lots of >> changes to generic code in the loader. > > What if the loader always enabled BTI for PROT_EXEC pages, but then > when discovering that this was a mistake, mprotect() the pages without > BTI? Is that architecturally supported? How costly is the mprotect change if the pages have not been faulted in yet? > Then both BTI and MDWX would work and the penalty of not getting > MDWX would fall to non-BTI programs. What's the expected proportion of > BTI enabled code vs. disabled in the future, is it perhaps expected > that a distro would enable the flag globally so eventually only a few > legacy programs might be unprotected? Eventually, I expect that mainstream distributions build everything for BTI, so yes, the PROT_BTI removal would only be needed for legacy programs. Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 8:17 ` Topi Miettinen @ 2020-10-22 8:29 ` Szabolcs Nagy -1 siblings, 0 replies; 88+ messages in thread From: Szabolcs Nagy @ 2020-10-22 8:29 UTC (permalink / raw) To: Topi Miettinen Cc: Florian Weimer, Lennart Poettering, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel The 10/22/2020 11:17, Topi Miettinen via Libc-alpha wrote: > On 22.10.2020 10.54, Florian Weimer wrote: > > * Lennart Poettering: > > > Did you see Topi's comments on the systemd issue? > > > > > > https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 > > > > > > I think I agree with this: it's a bit weird to alter the bits after > > > the fact. Can't glibc set up everything right from the begining? That > > > would keep both concepts working. > > > > The dynamic loader has to process the LOAD segments to get to the ELF > > note that says to enable BTI. Maybe we could do a first pass and load > > only the segments that cover notes. But that requires lots of changes > > to generic code in the loader. > > What if the loader always enabled BTI for PROT_EXEC pages, but then when > discovering that this was a mistake, mprotect() the pages without BTI? Then > both BTI and MDWX would work and the penalty of not getting MDWX would fall > to non-BTI programs. What's the expected proportion of BTI enabled code vs. > disabled in the future, is it perhaps expected that a distro would enable > the flag globally so eventually only a few legacy programs might be > unprotected? i thought mprotect(PROT_EXEC) would get filtered with or without bti, is that not the case? then i guess we can do the protection that way around, but then i don't see why the filter cannot treat PROT_EXEC|PROT_BTI the same as PROT_EXEC. ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 8:29 ` Szabolcs Nagy 0 siblings, 0 replies; 88+ messages in thread From: Szabolcs Nagy @ 2020-10-22 8:29 UTC (permalink / raw) To: Topi Miettinen Cc: Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, Lennart Poettering, libc-alpha, Dave Martin, linux-arm-kernel The 10/22/2020 11:17, Topi Miettinen via Libc-alpha wrote: > On 22.10.2020 10.54, Florian Weimer wrote: > > * Lennart Poettering: > > > Did you see Topi's comments on the systemd issue? > > > > > > https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 > > > > > > I think I agree with this: it's a bit weird to alter the bits after > > > the fact. Can't glibc set up everything right from the begining? That > > > would keep both concepts working. > > > > The dynamic loader has to process the LOAD segments to get to the ELF > > note that says to enable BTI. Maybe we could do a first pass and load > > only the segments that cover notes. But that requires lots of changes > > to generic code in the loader. > > What if the loader always enabled BTI for PROT_EXEC pages, but then when > discovering that this was a mistake, mprotect() the pages without BTI? Then > both BTI and MDWX would work and the penalty of not getting MDWX would fall > to non-BTI programs. What's the expected proportion of BTI enabled code vs. > disabled in the future, is it perhaps expected that a distro would enable > the flag globally so eventually only a few legacy programs might be > unprotected? i thought mprotect(PROT_EXEC) would get filtered with or without bti, is that not the case? then i guess we can do the protection that way around, but then i don't see why the filter cannot treat PROT_EXEC|PROT_BTI the same as PROT_EXEC. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 8:29 ` Szabolcs Nagy @ 2020-10-22 8:38 ` Lennart Poettering -1 siblings, 0 replies; 88+ messages in thread From: Lennart Poettering @ 2020-10-22 8:38 UTC (permalink / raw) To: Szabolcs Nagy Cc: Topi Miettinen, Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On Do, 22.10.20 09:29, Szabolcs Nagy (szabolcs.nagy@arm.com) wrote: > > > The dynamic loader has to process the LOAD segments to get to the ELF > > > note that says to enable BTI. Maybe we could do a first pass and load > > > only the segments that cover notes. But that requires lots of changes > > > to generic code in the loader. > > > > What if the loader always enabled BTI for PROT_EXEC pages, but then when > > discovering that this was a mistake, mprotect() the pages without BTI? Then > > both BTI and MDWX would work and the penalty of not getting MDWX would fall > > to non-BTI programs. What's the expected proportion of BTI enabled code vs. > > disabled in the future, is it perhaps expected that a distro would enable > > the flag globally so eventually only a few legacy programs might be > > unprotected? > > i thought mprotect(PROT_EXEC) would get filtered > with or without bti, is that not the case? We can adjust the filter in systemd to match any combination of flags to allow and to deny. Lennart -- Lennart Poettering, Berlin ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 8:38 ` Lennart Poettering 0 siblings, 0 replies; 88+ messages in thread From: Lennart Poettering @ 2020-10-22 8:38 UTC (permalink / raw) To: Szabolcs Nagy Cc: Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, Topi Miettinen, libc-alpha, Dave Martin, linux-arm-kernel On Do, 22.10.20 09:29, Szabolcs Nagy (szabolcs.nagy@arm.com) wrote: > > > The dynamic loader has to process the LOAD segments to get to the ELF > > > note that says to enable BTI. Maybe we could do a first pass and load > > > only the segments that cover notes. But that requires lots of changes > > > to generic code in the loader. > > > > What if the loader always enabled BTI for PROT_EXEC pages, but then when > > discovering that this was a mistake, mprotect() the pages without BTI? Then > > both BTI and MDWX would work and the penalty of not getting MDWX would fall > > to non-BTI programs. What's the expected proportion of BTI enabled code vs. > > disabled in the future, is it perhaps expected that a distro would enable > > the flag globally so eventually only a few legacy programs might be > > unprotected? > > i thought mprotect(PROT_EXEC) would get filtered > with or without bti, is that not the case? We can adjust the filter in systemd to match any combination of flags to allow and to deny. Lennart -- Lennart Poettering, Berlin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 8:38 ` Lennart Poettering @ 2020-10-22 9:31 ` Catalin Marinas -1 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-22 9:31 UTC (permalink / raw) To: Lennart Poettering Cc: Szabolcs Nagy, Topi Miettinen, Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On Thu, Oct 22, 2020 at 10:38:23AM +0200, Lennart Poettering wrote: > On Do, 22.10.20 09:29, Szabolcs Nagy (szabolcs.nagy@arm.com) wrote: > > > > The dynamic loader has to process the LOAD segments to get to the ELF > > > > note that says to enable BTI. Maybe we could do a first pass and load > > > > only the segments that cover notes. But that requires lots of changes > > > > to generic code in the loader. > > > > > > What if the loader always enabled BTI for PROT_EXEC pages, but then when > > > discovering that this was a mistake, mprotect() the pages without BTI? Then > > > both BTI and MDWX would work and the penalty of not getting MDWX would fall > > > to non-BTI programs. What's the expected proportion of BTI enabled code vs. > > > disabled in the future, is it perhaps expected that a distro would enable > > > the flag globally so eventually only a few legacy programs might be > > > unprotected? > > > > i thought mprotect(PROT_EXEC) would get filtered > > with or without bti, is that not the case? > > We can adjust the filter in systemd to match any combination of > flags to allow and to deny. Yes but Szabolcs' point to Topi was that if we can adjust the filters to allow mprotect(PROT_EXEC), why not allow mprotect(PROT_EXEC|PROT_BTI) instead? Anyway, I see the MDWX and BTI as complementary policies so ideally we shouldn't have to choose between one or the other. If we allow mprotect(PROT_EXEC), that would override MDWX and also disable BTI. IIUC, the problem is with the main executable which is mapped by the kernel without PROT_BTI. The dynamic loader wants to set PROT_BTI but does not have the original file descriptor to be able to remap. Its only choice is mprotect() and this fails because of the MDWX policy. Not sure whether the kernel has the right information but could it map the main executable with PROT_BTI if the corresponding PT_GNU_PROPERTY is found? The current ABI states it only sets PROT_BTI for the interpreter who'd be responsible for setting the PROT_BTI on the main executable. I can't tell whether it would break anything but it's worth a try: diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 4784011cecac..0a08fb9133e8 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -730,14 +730,6 @@ asmlinkage void __sched arm64_preempt_schedule_irq(void) int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state, bool has_interp, bool is_interp) { - /* - * For dynamically linked executables the interpreter is - * responsible for setting PROT_BTI on everything except - * itself. - */ - if (is_interp != has_interp) - return prot; - if (!(state->flags & ARM64_ELF_BTI)) return prot; -- Catalin ^ permalink raw reply related [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 9:31 ` Catalin Marinas 0 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-22 9:31 UTC (permalink / raw) To: Lennart Poettering Cc: Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, Szabolcs Nagy, Will Deacon, linux-kernel, Mark Brown, Topi Miettinen, libc-alpha, Dave Martin, linux-arm-kernel On Thu, Oct 22, 2020 at 10:38:23AM +0200, Lennart Poettering wrote: > On Do, 22.10.20 09:29, Szabolcs Nagy (szabolcs.nagy@arm.com) wrote: > > > > The dynamic loader has to process the LOAD segments to get to the ELF > > > > note that says to enable BTI. Maybe we could do a first pass and load > > > > only the segments that cover notes. But that requires lots of changes > > > > to generic code in the loader. > > > > > > What if the loader always enabled BTI for PROT_EXEC pages, but then when > > > discovering that this was a mistake, mprotect() the pages without BTI? Then > > > both BTI and MDWX would work and the penalty of not getting MDWX would fall > > > to non-BTI programs. What's the expected proportion of BTI enabled code vs. > > > disabled in the future, is it perhaps expected that a distro would enable > > > the flag globally so eventually only a few legacy programs might be > > > unprotected? > > > > i thought mprotect(PROT_EXEC) would get filtered > > with or without bti, is that not the case? > > We can adjust the filter in systemd to match any combination of > flags to allow and to deny. Yes but Szabolcs' point to Topi was that if we can adjust the filters to allow mprotect(PROT_EXEC), why not allow mprotect(PROT_EXEC|PROT_BTI) instead? Anyway, I see the MDWX and BTI as complementary policies so ideally we shouldn't have to choose between one or the other. If we allow mprotect(PROT_EXEC), that would override MDWX and also disable BTI. IIUC, the problem is with the main executable which is mapped by the kernel without PROT_BTI. The dynamic loader wants to set PROT_BTI but does not have the original file descriptor to be able to remap. Its only choice is mprotect() and this fails because of the MDWX policy. Not sure whether the kernel has the right information but could it map the main executable with PROT_BTI if the corresponding PT_GNU_PROPERTY is found? The current ABI states it only sets PROT_BTI for the interpreter who'd be responsible for setting the PROT_BTI on the main executable. I can't tell whether it would break anything but it's worth a try: diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 4784011cecac..0a08fb9133e8 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -730,14 +730,6 @@ asmlinkage void __sched arm64_preempt_schedule_irq(void) int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state, bool has_interp, bool is_interp) { - /* - * For dynamically linked executables the interpreter is - * responsible for setting PROT_BTI on everything except - * itself. - */ - if (is_interp != has_interp) - return prot; - if (!(state->flags & ARM64_ELF_BTI)) return prot; -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 9:31 ` Catalin Marinas @ 2020-10-22 10:12 ` Topi Miettinen -1 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 10:12 UTC (permalink / raw) To: Catalin Marinas, Lennart Poettering Cc: Szabolcs Nagy, Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On 22.10.2020 12.31, Catalin Marinas wrote: > On Thu, Oct 22, 2020 at 10:38:23AM +0200, Lennart Poettering wrote: >> On Do, 22.10.20 09:29, Szabolcs Nagy (szabolcs.nagy@arm.com) wrote: >>>>> The dynamic loader has to process the LOAD segments to get to the ELF >>>>> note that says to enable BTI. Maybe we could do a first pass and load >>>>> only the segments that cover notes. But that requires lots of changes >>>>> to generic code in the loader. >>>> >>>> What if the loader always enabled BTI for PROT_EXEC pages, but then when >>>> discovering that this was a mistake, mprotect() the pages without BTI? Then >>>> both BTI and MDWX would work and the penalty of not getting MDWX would fall >>>> to non-BTI programs. What's the expected proportion of BTI enabled code vs. >>>> disabled in the future, is it perhaps expected that a distro would enable >>>> the flag globally so eventually only a few legacy programs might be >>>> unprotected? >>> >>> i thought mprotect(PROT_EXEC) would get filtered >>> with or without bti, is that not the case? >> >> We can adjust the filter in systemd to match any combination of >> flags to allow and to deny. > > Yes but Szabolcs' point to Topi was that if we can adjust the filters to > allow mprotect(PROT_EXEC), why not allow mprotect(PROT_EXEC|PROT_BTI) > instead? Anyway, I see the MDWX and BTI as complementary policies so > ideally we shouldn't have to choose between one or the other. If we > allow mprotect(PROT_EXEC), that would override MDWX and also disable > BTI. Allowing mprotect(PROT_EXEC|PROT_BTI) would mean that all you need to circumvent MDWX is to add PROT_BTI flag. I'd suggest getting the flags right at mmap() time or failing that, reverting the PROT_BTI for legacy programs later. Could the kernel tell the loader of the BTI situation with auxiliary vectors? Then it would be easy for the loader to always use the best mmap() flags without ever needing to mprotect(). -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 10:12 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 10:12 UTC (permalink / raw) To: Catalin Marinas, Lennart Poettering Cc: Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, Szabolcs Nagy, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On 22.10.2020 12.31, Catalin Marinas wrote: > On Thu, Oct 22, 2020 at 10:38:23AM +0200, Lennart Poettering wrote: >> On Do, 22.10.20 09:29, Szabolcs Nagy (szabolcs.nagy@arm.com) wrote: >>>>> The dynamic loader has to process the LOAD segments to get to the ELF >>>>> note that says to enable BTI. Maybe we could do a first pass and load >>>>> only the segments that cover notes. But that requires lots of changes >>>>> to generic code in the loader. >>>> >>>> What if the loader always enabled BTI for PROT_EXEC pages, but then when >>>> discovering that this was a mistake, mprotect() the pages without BTI? Then >>>> both BTI and MDWX would work and the penalty of not getting MDWX would fall >>>> to non-BTI programs. What's the expected proportion of BTI enabled code vs. >>>> disabled in the future, is it perhaps expected that a distro would enable >>>> the flag globally so eventually only a few legacy programs might be >>>> unprotected? >>> >>> i thought mprotect(PROT_EXEC) would get filtered >>> with or without bti, is that not the case? >> >> We can adjust the filter in systemd to match any combination of >> flags to allow and to deny. > > Yes but Szabolcs' point to Topi was that if we can adjust the filters to > allow mprotect(PROT_EXEC), why not allow mprotect(PROT_EXEC|PROT_BTI) > instead? Anyway, I see the MDWX and BTI as complementary policies so > ideally we shouldn't have to choose between one or the other. If we > allow mprotect(PROT_EXEC), that would override MDWX and also disable > BTI. Allowing mprotect(PROT_EXEC|PROT_BTI) would mean that all you need to circumvent MDWX is to add PROT_BTI flag. I'd suggest getting the flags right at mmap() time or failing that, reverting the PROT_BTI for legacy programs later. Could the kernel tell the loader of the BTI situation with auxiliary vectors? Then it would be easy for the loader to always use the best mmap() flags without ever needing to mprotect(). -Topi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 10:12 ` Topi Miettinen @ 2020-10-22 10:27 ` Florian Weimer -1 siblings, 0 replies; 88+ messages in thread From: Florian Weimer @ 2020-10-22 10:27 UTC (permalink / raw) To: Topi Miettinen Cc: Catalin Marinas, Lennart Poettering, Szabolcs Nagy, Mark Rutland, systemd-devel, Kees Cook, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel * Topi Miettinen: > Allowing mprotect(PROT_EXEC|PROT_BTI) would mean that all you need to > circumvent MDWX is to add PROT_BTI flag. I'd suggest getting the flags > right at mmap() time or failing that, reverting the PROT_BTI for > legacy programs later. > > Could the kernel tell the loader of the BTI situation with auxiliary > vectors? Then it would be easy for the loader to always use the best > mmap() flags without ever needing to mprotect(). I think what we want is a mprotect2 call with a flags argument (separate from protection flags) that tells the kernel that the request *removes* protection flags and should fail otherwise. seccomp could easily filter that then. But like the other proposals, the migration story isn't great. You would need kernel and seccomp/systemd etc. updates before glibc starts working, even if glibc has a fallback from mprotect2 to mprotect (because the latter would be blocked). Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 10:27 ` Florian Weimer 0 siblings, 0 replies; 88+ messages in thread From: Florian Weimer @ 2020-10-22 10:27 UTC (permalink / raw) To: Topi Miettinen Cc: Mark Rutland, systemd-devel, Kees Cook, Szabolcs Nagy, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, Lennart Poettering, libc-alpha, Dave Martin, linux-arm-kernel * Topi Miettinen: > Allowing mprotect(PROT_EXEC|PROT_BTI) would mean that all you need to > circumvent MDWX is to add PROT_BTI flag. I'd suggest getting the flags > right at mmap() time or failing that, reverting the PROT_BTI for > legacy programs later. > > Could the kernel tell the loader of the BTI situation with auxiliary > vectors? Then it would be easy for the loader to always use the best > mmap() flags without ever needing to mprotect(). I think what we want is a mprotect2 call with a flags argument (separate from protection flags) that tells the kernel that the request *removes* protection flags and should fail otherwise. seccomp could easily filter that then. But like the other proposals, the migration story isn't great. You would need kernel and seccomp/systemd etc. updates before glibc starts working, even if glibc has a fallback from mprotect2 to mprotect (because the latter would be blocked). Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 9:31 ` Catalin Marinas @ 2020-10-23 6:13 ` Szabolcs Nagy -1 siblings, 0 replies; 88+ messages in thread From: Szabolcs Nagy @ 2020-10-23 6:13 UTC (permalink / raw) To: Catalin Marinas Cc: Lennart Poettering, Topi Miettinen, Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel The 10/22/2020 10:31, Catalin Marinas wrote: > IIUC, the problem is with the main executable which is mapped by the > kernel without PROT_BTI. The dynamic loader wants to set PROT_BTI but > does not have the original file descriptor to be able to remap. Its only > choice is mprotect() and this fails because of the MDWX policy. > > Not sure whether the kernel has the right information but could it map > the main executable with PROT_BTI if the corresponding PT_GNU_PROPERTY > is found? The current ABI states it only sets PROT_BTI for the > interpreter who'd be responsible for setting the PROT_BTI on the main > executable. I can't tell whether it would break anything but it's worth > a try: i think it would work, but now i can't easily tell from the libc if i have to do the mprotect on the main exe or not. i guess i can just always mprotect and ignore the failure? > > diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c > index 4784011cecac..0a08fb9133e8 100644 > --- a/arch/arm64/kernel/process.c > +++ b/arch/arm64/kernel/process.c > @@ -730,14 +730,6 @@ asmlinkage void __sched arm64_preempt_schedule_irq(void) > int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state, > bool has_interp, bool is_interp) > { > - /* > - * For dynamically linked executables the interpreter is > - * responsible for setting PROT_BTI on everything except > - * itself. > - */ > - if (is_interp != has_interp) > - return prot; > - > if (!(state->flags & ARM64_ELF_BTI)) > return prot; > > > -- > Catalin -- ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-23 6:13 ` Szabolcs Nagy 0 siblings, 0 replies; 88+ messages in thread From: Szabolcs Nagy @ 2020-10-23 6:13 UTC (permalink / raw) To: Catalin Marinas Cc: Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, libc-alpha, Will Deacon, linux-kernel, Mark Brown, Topi Miettinen, Lennart Poettering, Dave Martin, linux-arm-kernel The 10/22/2020 10:31, Catalin Marinas wrote: > IIUC, the problem is with the main executable which is mapped by the > kernel without PROT_BTI. The dynamic loader wants to set PROT_BTI but > does not have the original file descriptor to be able to remap. Its only > choice is mprotect() and this fails because of the MDWX policy. > > Not sure whether the kernel has the right information but could it map > the main executable with PROT_BTI if the corresponding PT_GNU_PROPERTY > is found? The current ABI states it only sets PROT_BTI for the > interpreter who'd be responsible for setting the PROT_BTI on the main > executable. I can't tell whether it would break anything but it's worth > a try: i think it would work, but now i can't easily tell from the libc if i have to do the mprotect on the main exe or not. i guess i can just always mprotect and ignore the failure? > > diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c > index 4784011cecac..0a08fb9133e8 100644 > --- a/arch/arm64/kernel/process.c > +++ b/arch/arm64/kernel/process.c > @@ -730,14 +730,6 @@ asmlinkage void __sched arm64_preempt_schedule_irq(void) > int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state, > bool has_interp, bool is_interp) > { > - /* > - * For dynamically linked executables the interpreter is > - * responsible for setting PROT_BTI on everything except > - * itself. > - */ > - if (is_interp != has_interp) > - return prot; > - > if (!(state->flags & ARM64_ELF_BTI)) > return prot; > > > -- > Catalin -- _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-23 6:13 ` Szabolcs Nagy @ 2020-10-23 9:04 ` Catalin Marinas -1 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-23 9:04 UTC (permalink / raw) To: Szabolcs Nagy Cc: Lennart Poettering, Topi Miettinen, Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On Fri, Oct 23, 2020 at 07:13:17AM +0100, Szabolcs Nagy wrote: > The 10/22/2020 10:31, Catalin Marinas wrote: > > IIUC, the problem is with the main executable which is mapped by the > > kernel without PROT_BTI. The dynamic loader wants to set PROT_BTI but > > does not have the original file descriptor to be able to remap. Its only > > choice is mprotect() and this fails because of the MDWX policy. > > > > Not sure whether the kernel has the right information but could it map > > the main executable with PROT_BTI if the corresponding PT_GNU_PROPERTY > > is found? The current ABI states it only sets PROT_BTI for the > > interpreter who'd be responsible for setting the PROT_BTI on the main > > executable. I can't tell whether it would break anything but it's worth > > a try: > > i think it would work, but now i can't easily > tell from the libc if i have to do the mprotect > on the main exe or not. > > i guess i can just always mprotect and ignore > the failure? I replied to Keys before reading your email. So yeah, still issue mprotect() but ignore the failure. -- Catalin ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-23 9:04 ` Catalin Marinas 0 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-23 9:04 UTC (permalink / raw) To: Szabolcs Nagy Cc: Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, libc-alpha, Will Deacon, linux-kernel, Mark Brown, Topi Miettinen, Lennart Poettering, Dave Martin, linux-arm-kernel On Fri, Oct 23, 2020 at 07:13:17AM +0100, Szabolcs Nagy wrote: > The 10/22/2020 10:31, Catalin Marinas wrote: > > IIUC, the problem is with the main executable which is mapped by the > > kernel without PROT_BTI. The dynamic loader wants to set PROT_BTI but > > does not have the original file descriptor to be able to remap. Its only > > choice is mprotect() and this fails because of the MDWX policy. > > > > Not sure whether the kernel has the right information but could it map > > the main executable with PROT_BTI if the corresponding PT_GNU_PROPERTY > > is found? The current ABI states it only sets PROT_BTI for the > > interpreter who'd be responsible for setting the PROT_BTI on the main > > executable. I can't tell whether it would break anything but it's worth > > a try: > > i think it would work, but now i can't easily > tell from the libc if i have to do the mprotect > on the main exe or not. > > i guess i can just always mprotect and ignore > the failure? I replied to Keys before reading your email. So yeah, still issue mprotect() but ignore the failure. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 8:29 ` Szabolcs Nagy @ 2020-10-22 10:03 ` Topi Miettinen -1 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 10:03 UTC (permalink / raw) To: Szabolcs Nagy Cc: Florian Weimer, Lennart Poettering, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On 22.10.2020 11.29, Szabolcs Nagy wrote: > The 10/22/2020 11:17, Topi Miettinen via Libc-alpha wrote: >> On 22.10.2020 10.54, Florian Weimer wrote: >>> * Lennart Poettering: >>>> Did you see Topi's comments on the systemd issue? >>>> >>>> https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 >>>> >>>> I think I agree with this: it's a bit weird to alter the bits after >>>> the fact. Can't glibc set up everything right from the begining? That >>>> would keep both concepts working. >>> >>> The dynamic loader has to process the LOAD segments to get to the ELF >>> note that says to enable BTI. Maybe we could do a first pass and load >>> only the segments that cover notes. But that requires lots of changes >>> to generic code in the loader. >> >> What if the loader always enabled BTI for PROT_EXEC pages, but then when >> discovering that this was a mistake, mprotect() the pages without BTI? Then >> both BTI and MDWX would work and the penalty of not getting MDWX would fall >> to non-BTI programs. What's the expected proportion of BTI enabled code vs. >> disabled in the future, is it perhaps expected that a distro would enable >> the flag globally so eventually only a few legacy programs might be >> unprotected? > > i thought mprotect(PROT_EXEC) would get filtered > with or without bti, is that not the case? It would be filtered, but the idea is that with modern binaries this would not happen since the pages would be mapped with mmap(,, PROT_EXEC | PROT_BTI,,) which is OK for purposes MDWX. The loader would have to use mprotect(PROT_EXEC) to get rid of PROT_BTI only for the legacy binaries. -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 10:03 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 10:03 UTC (permalink / raw) To: Szabolcs Nagy Cc: Florian Weimer, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, Lennart Poettering, libc-alpha, Dave Martin, linux-arm-kernel On 22.10.2020 11.29, Szabolcs Nagy wrote: > The 10/22/2020 11:17, Topi Miettinen via Libc-alpha wrote: >> On 22.10.2020 10.54, Florian Weimer wrote: >>> * Lennart Poettering: >>>> Did you see Topi's comments on the systemd issue? >>>> >>>> https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 >>>> >>>> I think I agree with this: it's a bit weird to alter the bits after >>>> the fact. Can't glibc set up everything right from the begining? That >>>> would keep both concepts working. >>> >>> The dynamic loader has to process the LOAD segments to get to the ELF >>> note that says to enable BTI. Maybe we could do a first pass and load >>> only the segments that cover notes. But that requires lots of changes >>> to generic code in the loader. >> >> What if the loader always enabled BTI for PROT_EXEC pages, but then when >> discovering that this was a mistake, mprotect() the pages without BTI? Then >> both BTI and MDWX would work and the penalty of not getting MDWX would fall >> to non-BTI programs. What's the expected proportion of BTI enabled code vs. >> disabled in the future, is it perhaps expected that a distro would enable >> the flag globally so eventually only a few legacy programs might be >> unprotected? > > i thought mprotect(PROT_EXEC) would get filtered > with or without bti, is that not the case? It would be filtered, but the idea is that with modern binaries this would not happen since the pages would be mapped with mmap(,, PROT_EXEC | PROT_BTI,,) which is OK for purposes MDWX. The loader would have to use mprotect(PROT_EXEC) to get rid of PROT_BTI only for the legacy binaries. -Topi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 7:18 ` Lennart Poettering @ 2020-10-22 8:05 ` Szabolcs Nagy -1 siblings, 0 replies; 88+ messages in thread From: Szabolcs Nagy @ 2020-10-22 8:05 UTC (permalink / raw) To: Lennart Poettering Cc: Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Kees Cook, Catalin Marinas, Will Deacon, Mark Brown, Dave Martin The 10/22/2020 09:18, Lennart Poettering wrote: > On Mi, 21.10.20 22:44, Jeremy Linton (jeremy.linton@arm.com) wrote: > > > Hi, > > > > There is a problem with glibc+systemd on BTI enabled systems. Systemd > > has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny > > PROT_EXEC changes. Glibc enables BTI only on segments which are marked as > > being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is > > caught by the seccomp filter, resulting in service failures. > > > > So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. > > This is obviously not desirable. > > > > Various changes have been suggested, replacing the mprotect with mmap calls > > having PROT_BTI set on the original mapping, re-mmapping the segments, > > implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, > > and various modification to seccomp to allow particular mprotect cases to > > bypass the filters. In each case there seems to be an undesirable attribute > > to the solution. > > > > So, whats the best solution? > > Did you see Topi's comments on the systemd issue? > > https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 > > I think I agree with this: it's a bit weird to alter the bits after > the fact. Can't glibc set up everything right from the begining? That > would keep both concepts working. that's hard to do and does not work for the main exe currently (which is mmaped by the kernel). (it's hard to do because to know that the elf module requires bti the PT_GNU_PROPERTY notes have to be accessed that are often in the executable load segment, so either you mmap that or have to read that, but the latter has a lot more failure modes, so if i have to get the mmap flags right i'd do a mmap and then re-mmap if the flags were not right) ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 8:05 ` Szabolcs Nagy 0 siblings, 0 replies; 88+ messages in thread From: Szabolcs Nagy @ 2020-10-22 8:05 UTC (permalink / raw) To: Lennart Poettering Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel The 10/22/2020 09:18, Lennart Poettering wrote: > On Mi, 21.10.20 22:44, Jeremy Linton (jeremy.linton@arm.com) wrote: > > > Hi, > > > > There is a problem with glibc+systemd on BTI enabled systems. Systemd > > has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny > > PROT_EXEC changes. Glibc enables BTI only on segments which are marked as > > being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is > > caught by the seccomp filter, resulting in service failures. > > > > So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. > > This is obviously not desirable. > > > > Various changes have been suggested, replacing the mprotect with mmap calls > > having PROT_BTI set on the original mapping, re-mmapping the segments, > > implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, > > and various modification to seccomp to allow particular mprotect cases to > > bypass the filters. In each case there seems to be an undesirable attribute > > to the solution. > > > > So, whats the best solution? > > Did you see Topi's comments on the systemd issue? > > https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 > > I think I agree with this: it's a bit weird to alter the bits after > the fact. Can't glibc set up everything right from the begining? That > would keep both concepts working. that's hard to do and does not work for the main exe currently (which is mmaped by the kernel). (it's hard to do because to know that the elf module requires bti the PT_GNU_PROPERTY notes have to be accessed that are often in the executable load segment, so either you mmap that or have to read that, but the latter has a lot more failure modes, so if i have to get the mmap flags right i'd do a mmap and then re-mmap if the flags were not right) _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 8:05 ` Szabolcs Nagy @ 2020-10-22 8:31 ` Lennart Poettering -1 siblings, 0 replies; 88+ messages in thread From: Lennart Poettering @ 2020-10-22 8:31 UTC (permalink / raw) To: Szabolcs Nagy Cc: Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Kees Cook, Catalin Marinas, Will Deacon, Mark Brown, Dave Martin On Do, 22.10.20 09:05, Szabolcs Nagy (szabolcs.nagy@arm.com) wrote: > > > Various changes have been suggested, replacing the mprotect with mmap calls > > > having PROT_BTI set on the original mapping, re-mmapping the segments, > > > implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, > > > and various modification to seccomp to allow particular mprotect cases to > > > bypass the filters. In each case there seems to be an undesirable attribute > > > to the solution. > > > > > > So, whats the best solution? > > > > Did you see Topi's comments on the systemd issue? > > > > https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 > > > > I think I agree with this: it's a bit weird to alter the bits after > > the fact. Can't glibc set up everything right from the begining? That > > would keep both concepts working. > > that's hard to do and does not work for the main exe currently > (which is mmaped by the kernel). > > (it's hard to do because to know that the elf module requires > bti the PT_GNU_PROPERTY notes have to be accessed that are > often in the executable load segment, so either you mmap that > or have to read that, but the latter has a lot more failure > modes, so if i have to get the mmap flags right i'd do a mmap > and then re-mmap if the flags were not right) Only other option I then see is to neuter one of the two mechanisms. We could certainly turn off MDWE on arm in systemd, if people want that. Or make it a build-time choice, so that distros make the choice: build everything with BTI xor suppport MDWE. (Might make sense for glibc to gracefully fallback to non-BTI mode if the mprotect() fails though, to make sure BTI-built binaries work everywhere.) I figure your interest in ARM system security is bigger than mine. I am totally fine to turn off MDWE on ARM if that's what the Linux ARM folks want. I ave no horse in the race. Just let me know. [An acceptable compromise might be to allow mprotect(PROT_EXEC|PROT_BTI) if MDWE is on, but prohibit mprotect(PROT_EXEC) without PROT_BTI. Then at least you get one of the two protections, but not both. I mean, MDWE is not perfect anyway on non-x86-64 already: on 32bit i386 MDWE protection is not complete, due to ipc() syscall multiplexing being unmatchable with seccomp. I personally am happy as long as it works fully on x86-64] Lennart -- Lennart Poettering, Berlin ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 8:31 ` Lennart Poettering 0 siblings, 0 replies; 88+ messages in thread From: Lennart Poettering @ 2020-10-22 8:31 UTC (permalink / raw) To: Szabolcs Nagy Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On Do, 22.10.20 09:05, Szabolcs Nagy (szabolcs.nagy@arm.com) wrote: > > > Various changes have been suggested, replacing the mprotect with mmap calls > > > having PROT_BTI set on the original mapping, re-mmapping the segments, > > > implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, > > > and various modification to seccomp to allow particular mprotect cases to > > > bypass the filters. In each case there seems to be an undesirable attribute > > > to the solution. > > > > > > So, whats the best solution? > > > > Did you see Topi's comments on the systemd issue? > > > > https://github.com/systemd/systemd/issues/17368#issuecomment-710485532 > > > > I think I agree with this: it's a bit weird to alter the bits after > > the fact. Can't glibc set up everything right from the begining? That > > would keep both concepts working. > > that's hard to do and does not work for the main exe currently > (which is mmaped by the kernel). > > (it's hard to do because to know that the elf module requires > bti the PT_GNU_PROPERTY notes have to be accessed that are > often in the executable load segment, so either you mmap that > or have to read that, but the latter has a lot more failure > modes, so if i have to get the mmap flags right i'd do a mmap > and then re-mmap if the flags were not right) Only other option I then see is to neuter one of the two mechanisms. We could certainly turn off MDWE on arm in systemd, if people want that. Or make it a build-time choice, so that distros make the choice: build everything with BTI xor suppport MDWE. (Might make sense for glibc to gracefully fallback to non-BTI mode if the mprotect() fails though, to make sure BTI-built binaries work everywhere.) I figure your interest in ARM system security is bigger than mine. I am totally fine to turn off MDWE on ARM if that's what the Linux ARM folks want. I ave no horse in the race. Just let me know. [An acceptable compromise might be to allow mprotect(PROT_EXEC|PROT_BTI) if MDWE is on, but prohibit mprotect(PROT_EXEC) without PROT_BTI. Then at least you get one of the two protections, but not both. I mean, MDWE is not perfect anyway on non-x86-64 already: on 32bit i386 MDWE protection is not complete, due to ipc() syscall multiplexing being unmatchable with seccomp. I personally am happy as long as it works fully on x86-64] Lennart -- Lennart Poettering, Berlin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
[parent not found: <20201022075447.GO3819@arm.com>]
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures [not found] ` <20201022075447.GO3819@arm.com> @ 2020-10-22 10:39 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 10:39 UTC (permalink / raw) To: Szabolcs Nagy, Jeremy Linton Cc: linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Kees Cook, Catalin Marinas, Will Deacon On 22.10.2020 10.54, Szabolcs Nagy wrote: > The 10/21/2020 22:44, Jeremy Linton wrote: >> There is a problem with glibc+systemd on BTI enabled systems. Systemd >> has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny >> PROT_EXEC changes. Glibc enables BTI only on segments which are marked as >> being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is >> caught by the seccomp filter, resulting in service failures. >> >> So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. >> This is obviously not desirable. >> >> Various changes have been suggested, replacing the mprotect with mmap calls >> having PROT_BTI set on the original mapping, re-mmapping the segments, >> implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, >> and various modification to seccomp to allow particular mprotect cases to >> bypass the filters. In each case there seems to be an undesirable attribute >> to the solution. >> >> So, whats the best solution? > > the easy fix in glibc is to ignore mprotect(PROT_BTI|PROT_EXEC) > failures, so programs work with seccomp filters, but bti gets > disabled (it's unreasonable to expect bti protection if mprotect > is filtered). it will be a nasty silent failure though. Some may also want to use seccomp filters so that they will immediately kill the process and in this case they couldn't do it. > and i'm also considering a fix that re-mmaps the executable > segment with PROT_BTI instead of mprotect since that is not > filtered. unfortunately the main exe is mmaped by the kernel > without PROT_BTI and the libc does not have the fd to re-mmap. > (bti can be left off for the main exe if mprotect fails and > later we can teach the kernel to add bti there.) currently > this is not a complete fix so i'm a bit hesitant about it. > > as for a kernel side fix: if there is a way to only filter > PROT_EXEC mprotect on mappings that are not yet PROT_EXEC > that would solve this problem (but likely needs new syscall > or seccomp capability). Problem with seccomp MDWX is that it's still possible for malicious programs to circumvent the filter by using memfd_create(), fill the memory with desired content and then use mmap(,,PROT_EXEC) to make it executable without triggering seccomp. This can be mitigated by filtering also memfd_create(), but then some programs want to use it. Also the protection can be bypassed if the program can write to a file system which isn't mounted with "noexec". This can be mitigated with private mount namespaces and global mount options, but again some programs are written to expect W & X. But I think SELinux has a more complete solution (execmem) which can track the pages better than is possible with seccomp solution which has a very narrow field of view. Maybe this facility could be made available to non-SELinux systems, for example with prctl()? Then the in-kernel MDWX could allow mprotect(PROT_EXEC | PROT_BTI) in case the backing file hasn't been modified, the source filesystem isn't writable for the calling process and the file descriptor isn't created with memfd_create(). -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 10:39 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 10:39 UTC (permalink / raw) To: Szabolcs Nagy, Jeremy Linton Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, libc-alpha, Dave Martin, linux-arm-kernel On 22.10.2020 10.54, Szabolcs Nagy wrote: > The 10/21/2020 22:44, Jeremy Linton wrote: >> There is a problem with glibc+systemd on BTI enabled systems. Systemd >> has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny >> PROT_EXEC changes. Glibc enables BTI only on segments which are marked as >> being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is >> caught by the seccomp filter, resulting in service failures. >> >> So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. >> This is obviously not desirable. >> >> Various changes have been suggested, replacing the mprotect with mmap calls >> having PROT_BTI set on the original mapping, re-mmapping the segments, >> implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, >> and various modification to seccomp to allow particular mprotect cases to >> bypass the filters. In each case there seems to be an undesirable attribute >> to the solution. >> >> So, whats the best solution? > > the easy fix in glibc is to ignore mprotect(PROT_BTI|PROT_EXEC) > failures, so programs work with seccomp filters, but bti gets > disabled (it's unreasonable to expect bti protection if mprotect > is filtered). it will be a nasty silent failure though. Some may also want to use seccomp filters so that they will immediately kill the process and in this case they couldn't do it. > and i'm also considering a fix that re-mmaps the executable > segment with PROT_BTI instead of mprotect since that is not > filtered. unfortunately the main exe is mmaped by the kernel > without PROT_BTI and the libc does not have the fd to re-mmap. > (bti can be left off for the main exe if mprotect fails and > later we can teach the kernel to add bti there.) currently > this is not a complete fix so i'm a bit hesitant about it. > > as for a kernel side fix: if there is a way to only filter > PROT_EXEC mprotect on mappings that are not yet PROT_EXEC > that would solve this problem (but likely needs new syscall > or seccomp capability). Problem with seccomp MDWX is that it's still possible for malicious programs to circumvent the filter by using memfd_create(), fill the memory with desired content and then use mmap(,,PROT_EXEC) to make it executable without triggering seccomp. This can be mitigated by filtering also memfd_create(), but then some programs want to use it. Also the protection can be bypassed if the program can write to a file system which isn't mounted with "noexec". This can be mitigated with private mount namespaces and global mount options, but again some programs are written to expect W & X. But I think SELinux has a more complete solution (execmem) which can track the pages better than is possible with seccomp solution which has a very narrow field of view. Maybe this facility could be made available to non-SELinux systems, for example with prctl()? Then the in-kernel MDWX could allow mprotect(PROT_EXEC | PROT_BTI) in case the backing file hasn't been modified, the source filesystem isn't writable for the calling process and the file descriptor isn't created with memfd_create(). -Topi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 10:39 ` Topi Miettinen (?) @ 2020-10-22 20:02 ` Kees Cook -1 siblings, 0 replies; 88+ messages in thread From: Kees Cook @ 2020-10-22 20:02 UTC (permalink / raw) To: Topi Miettinen Cc: Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On Thu, Oct 22, 2020 at 01:39:07PM +0300, Topi Miettinen wrote: > But I think SELinux has a more complete solution (execmem) which can track > the pages better than is possible with seccomp solution which has a very > narrow field of view. Maybe this facility could be made available to > non-SELinux systems, for example with prctl()? Then the in-kernel MDWX could > allow mprotect(PROT_EXEC | PROT_BTI) in case the backing file hasn't been > modified, the source filesystem isn't writable for the calling process and > the file descriptor isn't created with memfd_create(). Right. The problem here is that systemd is attempting to mediate a state change using only syscall details (i.e. with seccomp) instead of a stateful analysis. Using a MAC is likely the only sane way to do that. SELinux is a bit difficult to adjust "on the fly" the way systemd would like to do things, and the more dynamic approach seen with SARA[1] isn't yet in the kernel. Trying to enforce memory W^X protection correctly via seccomp isn't really going to work well, as far as I can see. Regardless, it makes sense to me to have the kernel load the executable itself with BTI enabled by default. I prefer gaining Catalin's suggested patch[2]. :) [1] https://lore.kernel.org/kernel-hardening/1562410493-8661-1-git-send-email-s.mesoraca16@gmail.com/ [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ -- Kees Cook ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 20:02 ` Kees Cook 0 siblings, 0 replies; 88+ messages in thread From: Kees Cook @ 2020-10-22 20:02 UTC (permalink / raw) To: Topi Miettinen Cc: Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On Thu, Oct 22, 2020 at 01:39:07PM +0300, Topi Miettinen wrote: > But I think SELinux has a more complete solution (execmem) which can track > the pages better than is possible with seccomp solution which has a very > narrow field of view. Maybe this facility could be made available to > non-SELinux systems, for example with prctl()? Then the in-kernel MDWX could > allow mprotect(PROT_EXEC | PROT_BTI) in case the backing file hasn't been > modified, the source filesystem isn't writable for the calling process and > the file descriptor isn't created with memfd_create(). Right. The problem here is that systemd is attempting to mediate a state change using only syscall details (i.e. with seccomp) instead of a stateful analysis. Using a MAC is likely the only sane way to do that. SELinux is a bit difficult to adjust "on the fly" the way systemd would like to do things, and the more dynamic approach seen with SARA[1] isn't yet in the kernel. Trying to enforce memory W^X protection correctly via seccomp isn't really going to work well, as far as I can see. Regardless, it makes sense to me to have the kernel load the executable itself with BTI enabled by default. I prefer gaining Catalin's suggested patch[2]. :) [1] https://lore.kernel.org/kernel-hardening/1562410493-8661-1-git-send-email-s.mesoraca16@gmail.com/ [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ -- Kees Cook ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 20:02 ` Kees Cook 0 siblings, 0 replies; 88+ messages in thread From: Kees Cook @ 2020-10-22 20:02 UTC (permalink / raw) To: Topi Miettinen Cc: Mark Rutland, Salvatore Mesoraca, systemd-devel, kernel-hardening, Szabolcs Nagy, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, linux-hardening, libc-alpha, Dave Martin, linux-arm-kernel On Thu, Oct 22, 2020 at 01:39:07PM +0300, Topi Miettinen wrote: > But I think SELinux has a more complete solution (execmem) which can track > the pages better than is possible with seccomp solution which has a very > narrow field of view. Maybe this facility could be made available to > non-SELinux systems, for example with prctl()? Then the in-kernel MDWX could > allow mprotect(PROT_EXEC | PROT_BTI) in case the backing file hasn't been > modified, the source filesystem isn't writable for the calling process and > the file descriptor isn't created with memfd_create(). Right. The problem here is that systemd is attempting to mediate a state change using only syscall details (i.e. with seccomp) instead of a stateful analysis. Using a MAC is likely the only sane way to do that. SELinux is a bit difficult to adjust "on the fly" the way systemd would like to do things, and the more dynamic approach seen with SARA[1] isn't yet in the kernel. Trying to enforce memory W^X protection correctly via seccomp isn't really going to work well, as far as I can see. Regardless, it makes sense to me to have the kernel load the executable itself with BTI enabled by default. I prefer gaining Catalin's suggested patch[2]. :) [1] https://lore.kernel.org/kernel-hardening/1562410493-8661-1-git-send-email-s.mesoraca16@gmail.com/ [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ -- Kees Cook _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 20:02 ` Kees Cook (?) @ 2020-10-22 22:24 ` Topi Miettinen -1 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 22:24 UTC (permalink / raw) To: Kees Cook Cc: Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On 22.10.2020 23.02, Kees Cook wrote: > On Thu, Oct 22, 2020 at 01:39:07PM +0300, Topi Miettinen wrote: >> But I think SELinux has a more complete solution (execmem) which can track >> the pages better than is possible with seccomp solution which has a very >> narrow field of view. Maybe this facility could be made available to >> non-SELinux systems, for example with prctl()? Then the in-kernel MDWX could >> allow mprotect(PROT_EXEC | PROT_BTI) in case the backing file hasn't been >> modified, the source filesystem isn't writable for the calling process and >> the file descriptor isn't created with memfd_create(). > > Right. The problem here is that systemd is attempting to mediate a > state change using only syscall details (i.e. with seccomp) instead of > a stateful analysis. Using a MAC is likely the only sane way to do that. > SELinux is a bit difficult to adjust "on the fly" the way systemd would > like to do things, and the more dynamic approach seen with SARA[1] isn't > yet in the kernel. SARA looks interesting. What is missing is a prctl() to enable all W^X protections irrevocably for the current process, then systemd could enable it for services with MemoryDenyWriteExecute=yes. I didn't also see specific measures against memfd_create() or file system W&X, but perhaps those can be added later. Maybe pkey_mprotect() is not handled either unless it uses the same LSM hook as mprotect(). > Trying to enforce memory W^X protection correctly > via seccomp isn't really going to work well, as far as I can see. Not in general, but I think it can work well in context of system services. Then you can ensure that for a specific service, memfd_create() is blocked by seccomp and the file systems are W^X because of mount namespaces etc., so there should not be any means to construct arbitrary executable pages. -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 22:24 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 22:24 UTC (permalink / raw) To: Kees Cook Cc: Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On 22.10.2020 23.02, Kees Cook wrote: > On Thu, Oct 22, 2020 at 01:39:07PM +0300, Topi Miettinen wrote: >> But I think SELinux has a more complete solution (execmem) which can track >> the pages better than is possible with seccomp solution which has a very >> narrow field of view. Maybe this facility could be made available to >> non-SELinux systems, for example with prctl()? Then the in-kernel MDWX could >> allow mprotect(PROT_EXEC | PROT_BTI) in case the backing file hasn't been >> modified, the source filesystem isn't writable for the calling process and >> the file descriptor isn't created with memfd_create(). > > Right. The problem here is that systemd is attempting to mediate a > state change using only syscall details (i.e. with seccomp) instead of > a stateful analysis. Using a MAC is likely the only sane way to do that. > SELinux is a bit difficult to adjust "on the fly" the way systemd would > like to do things, and the more dynamic approach seen with SARA[1] isn't > yet in the kernel. SARA looks interesting. What is missing is a prctl() to enable all W^X protections irrevocably for the current process, then systemd could enable it for services with MemoryDenyWriteExecute=yes. I didn't also see specific measures against memfd_create() or file system W&X, but perhaps those can be added later. Maybe pkey_mprotect() is not handled either unless it uses the same LSM hook as mprotect(). > Trying to enforce memory W^X protection correctly > via seccomp isn't really going to work well, as far as I can see. Not in general, but I think it can work well in context of system services. Then you can ensure that for a specific service, memfd_create() is blocked by seccomp and the file systems are W^X because of mount namespaces etc., so there should not be any means to construct arbitrary executable pages. -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-22 22:24 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-22 22:24 UTC (permalink / raw) To: Kees Cook Cc: Mark Rutland, Salvatore Mesoraca, systemd-devel, kernel-hardening, Szabolcs Nagy, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, linux-hardening, libc-alpha, Dave Martin, linux-arm-kernel On 22.10.2020 23.02, Kees Cook wrote: > On Thu, Oct 22, 2020 at 01:39:07PM +0300, Topi Miettinen wrote: >> But I think SELinux has a more complete solution (execmem) which can track >> the pages better than is possible with seccomp solution which has a very >> narrow field of view. Maybe this facility could be made available to >> non-SELinux systems, for example with prctl()? Then the in-kernel MDWX could >> allow mprotect(PROT_EXEC | PROT_BTI) in case the backing file hasn't been >> modified, the source filesystem isn't writable for the calling process and >> the file descriptor isn't created with memfd_create(). > > Right. The problem here is that systemd is attempting to mediate a > state change using only syscall details (i.e. with seccomp) instead of > a stateful analysis. Using a MAC is likely the only sane way to do that. > SELinux is a bit difficult to adjust "on the fly" the way systemd would > like to do things, and the more dynamic approach seen with SARA[1] isn't > yet in the kernel. SARA looks interesting. What is missing is a prctl() to enable all W^X protections irrevocably for the current process, then systemd could enable it for services with MemoryDenyWriteExecute=yes. I didn't also see specific measures against memfd_create() or file system W&X, but perhaps those can be added later. Maybe pkey_mprotect() is not handled either unless it uses the same LSM hook as mprotect(). > Trying to enforce memory W^X protection correctly > via seccomp isn't really going to work well, as far as I can see. Not in general, but I think it can work well in context of system services. Then you can ensure that for a specific service, memfd_create() is blocked by seccomp and the file systems are W^X because of mount namespaces etc., so there should not be any means to construct arbitrary executable pages. -Topi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 22:24 ` Topi Miettinen (?) @ 2020-10-23 17:52 ` Salvatore Mesoraca -1 siblings, 0 replies; 88+ messages in thread From: Salvatore Mesoraca @ 2020-10-23 17:52 UTC (permalink / raw) To: Topi Miettinen Cc: Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Kernel Hardening, linux-hardening Hi, On Thu, 22 Oct 2020 at 23:24, Topi Miettinen <toiwoton@gmail.com> wrote: > SARA looks interesting. What is missing is a prctl() to enable all W^X > protections irrevocably for the current process, then systemd could > enable it for services with MemoryDenyWriteExecute=yes. SARA actually has a procattr[0] interface to do just that. There is also a library[1] to help using it. > I didn't also see specific measures against memfd_create() or file > system W&X, but perhaps those can be added later. You are right, there are no measures against those vectors. It would be interesting to add them, though. > Maybe pkey_mprotect() > is not handled either unless it uses the same LSM hook as mprotect(). IIRC mprotect is implemented more or less as a pkey_mprotect with -1 as pkey. The same LSM hook should cover both. Salvatore [0] https://lore.kernel.org/lkml/1562410493-8661-10-git-send-email-s.mesoraca16@gmail.com/ [1] https://github.com/smeso/libsara ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-23 17:52 ` Salvatore Mesoraca 0 siblings, 0 replies; 88+ messages in thread From: Salvatore Mesoraca @ 2020-10-23 17:52 UTC (permalink / raw) To: Topi Miettinen Cc: Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Kernel Hardening, linux-hardening Hi, On Thu, 22 Oct 2020 at 23:24, Topi Miettinen <toiwoton@gmail.com> wrote: > SARA looks interesting. What is missing is a prctl() to enable all W^X > protections irrevocably for the current process, then systemd could > enable it for services with MemoryDenyWriteExecute=yes. SARA actually has a procattr[0] interface to do just that. There is also a library[1] to help using it. > I didn't also see specific measures against memfd_create() or file > system W&X, but perhaps those can be added later. You are right, there are no measures against those vectors. It would be interesting to add them, though. > Maybe pkey_mprotect() > is not handled either unless it uses the same LSM hook as mprotect(). IIRC mprotect is implemented more or less as a pkey_mprotect with -1 as pkey. The same LSM hook should cover both. Salvatore [0] https://lore.kernel.org/lkml/1562410493-8661-10-git-send-email-s.mesoraca16@gmail.com/ [1] https://github.com/smeso/libsara ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-23 17:52 ` Salvatore Mesoraca 0 siblings, 0 replies; 88+ messages in thread From: Salvatore Mesoraca @ 2020-10-23 17:52 UTC (permalink / raw) To: Topi Miettinen Cc: Mark Rutland, systemd-devel, Kees Cook, Kernel Hardening, Szabolcs Nagy, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, linux-hardening, libc-alpha, Dave Martin, linux-arm-kernel Hi, On Thu, 22 Oct 2020 at 23:24, Topi Miettinen <toiwoton@gmail.com> wrote: > SARA looks interesting. What is missing is a prctl() to enable all W^X > protections irrevocably for the current process, then systemd could > enable it for services with MemoryDenyWriteExecute=yes. SARA actually has a procattr[0] interface to do just that. There is also a library[1] to help using it. > I didn't also see specific measures against memfd_create() or file > system W&X, but perhaps those can be added later. You are right, there are no measures against those vectors. It would be interesting to add them, though. > Maybe pkey_mprotect() > is not handled either unless it uses the same LSM hook as mprotect(). IIRC mprotect is implemented more or less as a pkey_mprotect with -1 as pkey. The same LSM hook should cover both. Salvatore [0] https://lore.kernel.org/lkml/1562410493-8661-10-git-send-email-s.mesoraca16@gmail.com/ [1] https://github.com/smeso/libsara _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-23 17:52 ` Salvatore Mesoraca (?) @ 2020-10-24 11:34 ` Topi Miettinen -1 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-24 11:34 UTC (permalink / raw) To: Salvatore Mesoraca Cc: Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Kernel Hardening, linux-hardening On 23.10.2020 20.52, Salvatore Mesoraca wrote: > Hi, > > On Thu, 22 Oct 2020 at 23:24, Topi Miettinen <toiwoton@gmail.com> wrote: >> SARA looks interesting. What is missing is a prctl() to enable all W^X >> protections irrevocably for the current process, then systemd could >> enable it for services with MemoryDenyWriteExecute=yes. > > SARA actually has a procattr[0] interface to do just that. > There is also a library[1] to help using it. That means that /proc has to be available and writable at that point, so setting up procattrs has to be done before mount namespaces are set up. In general, it would be nice for sandboxing facilities in kernel if there would be a way to start enforcing restrictions only at next execve(), like setexeccon() for SELinux and aa_change_onexec() for AppArmor. Otherwise the exact order of setting up various sandboxing options can be very tricky to arrange correctly, since each option may have a subtle effect to the sandboxing features enabled later. In case of SARA, the operations done between shuffling the mount namespace and before execve() shouldn't be affected so it isn't important. Even if it did (a new sandboxing feature in the future would need trampolines or JIT code generation), maybe the procattr file could be opened early but it could be written closer to execve(). -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-24 11:34 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-24 11:34 UTC (permalink / raw) To: Salvatore Mesoraca Cc: Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Kernel Hardening, linux-hardening On 23.10.2020 20.52, Salvatore Mesoraca wrote: > Hi, > > On Thu, 22 Oct 2020 at 23:24, Topi Miettinen <toiwoton@gmail.com> wrote: >> SARA looks interesting. What is missing is a prctl() to enable all W^X >> protections irrevocably for the current process, then systemd could >> enable it for services with MemoryDenyWriteExecute=yes. > > SARA actually has a procattr[0] interface to do just that. > There is also a library[1] to help using it. That means that /proc has to be available and writable at that point, so setting up procattrs has to be done before mount namespaces are set up. In general, it would be nice for sandboxing facilities in kernel if there would be a way to start enforcing restrictions only at next execve(), like setexeccon() for SELinux and aa_change_onexec() for AppArmor. Otherwise the exact order of setting up various sandboxing options can be very tricky to arrange correctly, since each option may have a subtle effect to the sandboxing features enabled later. In case of SARA, the operations done between shuffling the mount namespace and before execve() shouldn't be affected so it isn't important. Even if it did (a new sandboxing feature in the future would need trampolines or JIT code generation), maybe the procattr file could be opened early but it could be written closer to execve(). -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-24 11:34 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-24 11:34 UTC (permalink / raw) To: Salvatore Mesoraca Cc: Mark Rutland, systemd-devel, Kees Cook, Kernel Hardening, Szabolcs Nagy, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, linux-hardening, libc-alpha, Dave Martin, linux-arm-kernel On 23.10.2020 20.52, Salvatore Mesoraca wrote: > Hi, > > On Thu, 22 Oct 2020 at 23:24, Topi Miettinen <toiwoton@gmail.com> wrote: >> SARA looks interesting. What is missing is a prctl() to enable all W^X >> protections irrevocably for the current process, then systemd could >> enable it for services with MemoryDenyWriteExecute=yes. > > SARA actually has a procattr[0] interface to do just that. > There is also a library[1] to help using it. That means that /proc has to be available and writable at that point, so setting up procattrs has to be done before mount namespaces are set up. In general, it would be nice for sandboxing facilities in kernel if there would be a way to start enforcing restrictions only at next execve(), like setexeccon() for SELinux and aa_change_onexec() for AppArmor. Otherwise the exact order of setting up various sandboxing options can be very tricky to arrange correctly, since each option may have a subtle effect to the sandboxing features enabled later. In case of SARA, the operations done between shuffling the mount namespace and before execve() shouldn't be affected so it isn't important. Even if it did (a new sandboxing feature in the future would need trampolines or JIT code generation), maybe the procattr file could be opened early but it could be written closer to execve(). -Topi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-24 11:34 ` Topi Miettinen (?) @ 2020-10-24 14:12 ` Salvatore Mesoraca -1 siblings, 0 replies; 88+ messages in thread From: Salvatore Mesoraca @ 2020-10-24 14:12 UTC (permalink / raw) To: Topi Miettinen Cc: Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Kernel Hardening, linux-hardening On Sat, 24 Oct 2020 at 12:34, Topi Miettinen <toiwoton@gmail.com> wrote: > > On 23.10.2020 20.52, Salvatore Mesoraca wrote: > > Hi, > > > > On Thu, 22 Oct 2020 at 23:24, Topi Miettinen <toiwoton@gmail.com> wrote: > >> SARA looks interesting. What is missing is a prctl() to enable all W^X > >> protections irrevocably for the current process, then systemd could > >> enable it for services with MemoryDenyWriteExecute=yes. > > > > SARA actually has a procattr[0] interface to do just that. > > There is also a library[1] to help using it. > > That means that /proc has to be available and writable at that point, so > setting up procattrs has to be done before mount namespaces are set up. > In general, it would be nice for sandboxing facilities in kernel if > there would be a way to start enforcing restrictions only at next > execve(), like setexeccon() for SELinux and aa_change_onexec() for > AppArmor. Otherwise the exact order of setting up various sandboxing > options can be very tricky to arrange correctly, since each option may > have a subtle effect to the sandboxing features enabled later. In case > of SARA, the operations done between shuffling the mount namespace and > before execve() shouldn't be affected so it isn't important. Even if it > did (a new sandboxing feature in the future would need trampolines or > JIT code generation), maybe the procattr file could be opened early but > it could be written closer to execve(). A new "apply on exec" procattr file seems reasonable and relatively easy to add. As Kees pointed out, the main obstacle here is the fact that SARA is not upstream :( Salvatore ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-24 14:12 ` Salvatore Mesoraca 0 siblings, 0 replies; 88+ messages in thread From: Salvatore Mesoraca @ 2020-10-24 14:12 UTC (permalink / raw) To: Topi Miettinen Cc: Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Kernel Hardening, linux-hardening On Sat, 24 Oct 2020 at 12:34, Topi Miettinen <toiwoton@gmail.com> wrote: > > On 23.10.2020 20.52, Salvatore Mesoraca wrote: > > Hi, > > > > On Thu, 22 Oct 2020 at 23:24, Topi Miettinen <toiwoton@gmail.com> wrote: > >> SARA looks interesting. What is missing is a prctl() to enable all W^X > >> protections irrevocably for the current process, then systemd could > >> enable it for services with MemoryDenyWriteExecute=yes. > > > > SARA actually has a procattr[0] interface to do just that. > > There is also a library[1] to help using it. > > That means that /proc has to be available and writable at that point, so > setting up procattrs has to be done before mount namespaces are set up. > In general, it would be nice for sandboxing facilities in kernel if > there would be a way to start enforcing restrictions only at next > execve(), like setexeccon() for SELinux and aa_change_onexec() for > AppArmor. Otherwise the exact order of setting up various sandboxing > options can be very tricky to arrange correctly, since each option may > have a subtle effect to the sandboxing features enabled later. In case > of SARA, the operations done between shuffling the mount namespace and > before execve() shouldn't be affected so it isn't important. Even if it > did (a new sandboxing feature in the future would need trampolines or > JIT code generation), maybe the procattr file could be opened early but > it could be written closer to execve(). A new "apply on exec" procattr file seems reasonable and relatively easy to add. As Kees pointed out, the main obstacle here is the fact that SARA is not upstream :( Salvatore ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-24 14:12 ` Salvatore Mesoraca 0 siblings, 0 replies; 88+ messages in thread From: Salvatore Mesoraca @ 2020-10-24 14:12 UTC (permalink / raw) To: Topi Miettinen Cc: Mark Rutland, systemd-devel, Kees Cook, Kernel Hardening, Szabolcs Nagy, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, linux-hardening, libc-alpha, Dave Martin, linux-arm-kernel On Sat, 24 Oct 2020 at 12:34, Topi Miettinen <toiwoton@gmail.com> wrote: > > On 23.10.2020 20.52, Salvatore Mesoraca wrote: > > Hi, > > > > On Thu, 22 Oct 2020 at 23:24, Topi Miettinen <toiwoton@gmail.com> wrote: > >> SARA looks interesting. What is missing is a prctl() to enable all W^X > >> protections irrevocably for the current process, then systemd could > >> enable it for services with MemoryDenyWriteExecute=yes. > > > > SARA actually has a procattr[0] interface to do just that. > > There is also a library[1] to help using it. > > That means that /proc has to be available and writable at that point, so > setting up procattrs has to be done before mount namespaces are set up. > In general, it would be nice for sandboxing facilities in kernel if > there would be a way to start enforcing restrictions only at next > execve(), like setexeccon() for SELinux and aa_change_onexec() for > AppArmor. Otherwise the exact order of setting up various sandboxing > options can be very tricky to arrange correctly, since each option may > have a subtle effect to the sandboxing features enabled later. In case > of SARA, the operations done between shuffling the mount namespace and > before execve() shouldn't be affected so it isn't important. Even if it > did (a new sandboxing feature in the future would need trampolines or > JIT code generation), maybe the procattr file could be opened early but > it could be written closer to execve(). A new "apply on exec" procattr file seems reasonable and relatively easy to add. As Kees pointed out, the main obstacle here is the fact that SARA is not upstream :( Salvatore _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-24 14:12 ` Salvatore Mesoraca (?) @ 2020-10-25 13:42 ` Jordan Glover -1 siblings, 0 replies; 88+ messages in thread From: Jordan Glover @ 2020-10-25 13:42 UTC (permalink / raw) To: Salvatore Mesoraca Cc: Topi Miettinen, Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Kernel Hardening, linux-hardening On Saturday, October 24, 2020 2:12 PM, Salvatore Mesoraca <s.mesoraca16@gmail.com> wrote: > On Sat, 24 Oct 2020 at 12:34, Topi Miettinen toiwoton@gmail.com wrote: > > > On 23.10.2020 20.52, Salvatore Mesoraca wrote: > > > > > Hi, > > > On Thu, 22 Oct 2020 at 23:24, Topi Miettinen toiwoton@gmail.com wrote: > > > > > > > SARA looks interesting. What is missing is a prctl() to enable all W^X > > > > protections irrevocably for the current process, then systemd could > > > > enable it for services with MemoryDenyWriteExecute=yes. > > > > > > SARA actually has a procattr[0] interface to do just that. > > > There is also a library[1] to help using it. > > > > That means that /proc has to be available and writable at that point, so > > setting up procattrs has to be done before mount namespaces are set up. > > In general, it would be nice for sandboxing facilities in kernel if > > there would be a way to start enforcing restrictions only at next > > execve(), like setexeccon() for SELinux and aa_change_onexec() for > > AppArmor. Otherwise the exact order of setting up various sandboxing > > options can be very tricky to arrange correctly, since each option may > > have a subtle effect to the sandboxing features enabled later. In case > > of SARA, the operations done between shuffling the mount namespace and > > before execve() shouldn't be affected so it isn't important. Even if it > > did (a new sandboxing feature in the future would need trampolines or > > JIT code generation), maybe the procattr file could be opened early but > > it could be written closer to execve(). > > A new "apply on exec" procattr file seems reasonable and relatively easy to add. > As Kees pointed out, the main obstacle here is the fact that SARA is > not upstream :( > > Salvatore Is there a chance we will see new SARA iteration soon on lkml? :) Jordan ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-25 13:42 ` Jordan Glover 0 siblings, 0 replies; 88+ messages in thread From: Jordan Glover @ 2020-10-25 13:42 UTC (permalink / raw) To: Salvatore Mesoraca Cc: Topi Miettinen, Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Catalin Marinas, Will Deacon, Kernel Hardening, linux-hardening On Saturday, October 24, 2020 2:12 PM, Salvatore Mesoraca <s.mesoraca16@gmail.com> wrote: > On Sat, 24 Oct 2020 at 12:34, Topi Miettinen toiwoton@gmail.com wrote: > > > On 23.10.2020 20.52, Salvatore Mesoraca wrote: > > > > > Hi, > > > On Thu, 22 Oct 2020 at 23:24, Topi Miettinen toiwoton@gmail.com wrote: > > > > > > > SARA looks interesting. What is missing is a prctl() to enable all W^X > > > > protections irrevocably for the current process, then systemd could > > > > enable it for services with MemoryDenyWriteExecute=yes. > > > > > > SARA actually has a procattr[0] interface to do just that. > > > There is also a library[1] to help using it. > > > > That means that /proc has to be available and writable at that point, so > > setting up procattrs has to be done before mount namespaces are set up. > > In general, it would be nice for sandboxing facilities in kernel if > > there would be a way to start enforcing restrictions only at next > > execve(), like setexeccon() for SELinux and aa_change_onexec() for > > AppArmor. Otherwise the exact order of setting up various sandboxing > > options can be very tricky to arrange correctly, since each option may > > have a subtle effect to the sandboxing features enabled later. In case > > of SARA, the operations done between shuffling the mount namespace and > > before execve() shouldn't be affected so it isn't important. Even if it > > did (a new sandboxing feature in the future would need trampolines or > > JIT code generation), maybe the procattr file could be opened early but > > it could be written closer to execve(). > > A new "apply on exec" procattr file seems reasonable and relatively easy to add. > As Kees pointed out, the main obstacle here is the fact that SARA is > not upstream :( > > Salvatore Is there a chance we will see new SARA iteration soon on lkml? :) Jordan ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-25 13:42 ` Jordan Glover 0 siblings, 0 replies; 88+ messages in thread From: Jordan Glover @ 2020-10-25 13:42 UTC (permalink / raw) To: Salvatore Mesoraca Cc: Mark Rutland, libc-alpha, Kees Cook, Kernel Hardening, Szabolcs Nagy, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, Topi Miettinen, linux-hardening, systemd-devel, Dave Martin, linux-arm-kernel On Saturday, October 24, 2020 2:12 PM, Salvatore Mesoraca <s.mesoraca16@gmail.com> wrote: > On Sat, 24 Oct 2020 at 12:34, Topi Miettinen toiwoton@gmail.com wrote: > > > On 23.10.2020 20.52, Salvatore Mesoraca wrote: > > > > > Hi, > > > On Thu, 22 Oct 2020 at 23:24, Topi Miettinen toiwoton@gmail.com wrote: > > > > > > > SARA looks interesting. What is missing is a prctl() to enable all W^X > > > > protections irrevocably for the current process, then systemd could > > > > enable it for services with MemoryDenyWriteExecute=yes. > > > > > > SARA actually has a procattr[0] interface to do just that. > > > There is also a library[1] to help using it. > > > > That means that /proc has to be available and writable at that point, so > > setting up procattrs has to be done before mount namespaces are set up. > > In general, it would be nice for sandboxing facilities in kernel if > > there would be a way to start enforcing restrictions only at next > > execve(), like setexeccon() for SELinux and aa_change_onexec() for > > AppArmor. Otherwise the exact order of setting up various sandboxing > > options can be very tricky to arrange correctly, since each option may > > have a subtle effect to the sandboxing features enabled later. In case > > of SARA, the operations done between shuffling the mount namespace and > > before execve() shouldn't be affected so it isn't important. Even if it > > did (a new sandboxing feature in the future would need trampolines or > > JIT code generation), maybe the procattr file could be opened early but > > it could be written closer to execve(). > > A new "apply on exec" procattr file seems reasonable and relatively easy to add. > As Kees pointed out, the main obstacle here is the fact that SARA is > not upstream :( > > Salvatore Is there a chance we will see new SARA iteration soon on lkml? :) Jordan _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-22 20:02 ` Kees Cook (?) @ 2020-10-23 9:02 ` Catalin Marinas -1 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-23 9:02 UTC (permalink / raw) To: Kees Cook Cc: Topi Miettinen, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: > Regardless, it makes sense to me to have the kernel load the executable > itself with BTI enabled by default. I prefer gaining Catalin's suggested > patch[2]. :) [...] > [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ I think I first heard the idea at Mark R ;). It still needs glibc changes to avoid the mprotect(), or at least ignore the error. Since this is an ABI change and we don't know which kernels would have it backported, maybe better to still issue the mprotect() but ignore the failure. -- Catalin ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-23 9:02 ` Catalin Marinas 0 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-23 9:02 UTC (permalink / raw) To: Kees Cook Cc: Topi Miettinen, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: > Regardless, it makes sense to me to have the kernel load the executable > itself with BTI enabled by default. I prefer gaining Catalin's suggested > patch[2]. :) [...] > [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ I think I first heard the idea at Mark R ;). It still needs glibc changes to avoid the mprotect(), or at least ignore the error. Since this is an ABI change and we don't know which kernels would have it backported, maybe better to still issue the mprotect() but ignore the failure. -- Catalin ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-23 9:02 ` Catalin Marinas 0 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-23 9:02 UTC (permalink / raw) To: Kees Cook Cc: Mark Rutland, Salvatore Mesoraca, libc-alpha, kernel-hardening, Szabolcs Nagy, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, Topi Miettinen, linux-hardening, systemd-devel, Dave Martin, linux-arm-kernel On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: > Regardless, it makes sense to me to have the kernel load the executable > itself with BTI enabled by default. I prefer gaining Catalin's suggested > patch[2]. :) [...] > [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ I think I first heard the idea at Mark R ;). It still needs glibc changes to avoid the mprotect(), or at least ignore the error. Since this is an ABI change and we don't know which kernels would have it backported, maybe better to still issue the mprotect() but ignore the failure. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-23 9:02 ` Catalin Marinas (?) @ 2020-10-24 11:01 ` Topi Miettinen -1 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-24 11:01 UTC (permalink / raw) To: Catalin Marinas, Kees Cook Cc: Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On 23.10.2020 12.02, Catalin Marinas wrote: > On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: >> Regardless, it makes sense to me to have the kernel load the executable >> itself with BTI enabled by default. I prefer gaining Catalin's suggested >> patch[2]. :) > [...] >> [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ > > I think I first heard the idea at Mark R ;). > > It still needs glibc changes to avoid the mprotect(), or at least ignore > the error. Since this is an ABI change and we don't know which kernels > would have it backported, maybe better to still issue the mprotect() but > ignore the failure. What about kernel adding an auxiliary vector as a flag to indicate that BTI is supported and recommended by the kernel? Then dynamic loader could use that to detect that a) the main executable is BTI protected and there's no need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC pages. In absence of the vector, the dynamic loader might choose to skip doing PROT_BTI at all (since the main executable isn't protected anyway either, or maybe even the kernel is up-to-date but it knows that it's not recommended for some reason, or maybe the kernel is so ancient that it doesn't know about BTI). Optionally it could still read the flag from ELF later (for compatibility with old kernels) and then do the mprotect() dance, which may trip seccomp filters, possibly fatally. -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-24 11:01 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-24 11:01 UTC (permalink / raw) To: Catalin Marinas, Kees Cook Cc: Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On 23.10.2020 12.02, Catalin Marinas wrote: > On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: >> Regardless, it makes sense to me to have the kernel load the executable >> itself with BTI enabled by default. I prefer gaining Catalin's suggested >> patch[2]. :) > [...] >> [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ > > I think I first heard the idea at Mark R ;). > > It still needs glibc changes to avoid the mprotect(), or at least ignore > the error. Since this is an ABI change and we don't know which kernels > would have it backported, maybe better to still issue the mprotect() but > ignore the failure. What about kernel adding an auxiliary vector as a flag to indicate that BTI is supported and recommended by the kernel? Then dynamic loader could use that to detect that a) the main executable is BTI protected and there's no need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC pages. In absence of the vector, the dynamic loader might choose to skip doing PROT_BTI at all (since the main executable isn't protected anyway either, or maybe even the kernel is up-to-date but it knows that it's not recommended for some reason, or maybe the kernel is so ancient that it doesn't know about BTI). Optionally it could still read the flag from ELF later (for compatibility with old kernels) and then do the mprotect() dance, which may trip seccomp filters, possibly fatally. -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-24 11:01 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-24 11:01 UTC (permalink / raw) To: Catalin Marinas, Kees Cook Cc: Mark Rutland, Salvatore Mesoraca, systemd-devel, kernel-hardening, Szabolcs Nagy, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, linux-hardening, libc-alpha, Dave Martin, linux-arm-kernel On 23.10.2020 12.02, Catalin Marinas wrote: > On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: >> Regardless, it makes sense to me to have the kernel load the executable >> itself with BTI enabled by default. I prefer gaining Catalin's suggested >> patch[2]. :) > [...] >> [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ > > I think I first heard the idea at Mark R ;). > > It still needs glibc changes to avoid the mprotect(), or at least ignore > the error. Since this is an ABI change and we don't know which kernels > would have it backported, maybe better to still issue the mprotect() but > ignore the failure. What about kernel adding an auxiliary vector as a flag to indicate that BTI is supported and recommended by the kernel? Then dynamic loader could use that to detect that a) the main executable is BTI protected and there's no need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC pages. In absence of the vector, the dynamic loader might choose to skip doing PROT_BTI at all (since the main executable isn't protected anyway either, or maybe even the kernel is up-to-date but it knows that it's not recommended for some reason, or maybe the kernel is so ancient that it doesn't know about BTI). Optionally it could still read the flag from ELF later (for compatibility with old kernels) and then do the mprotect() dance, which may trip seccomp filters, possibly fatally. -Topi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-24 11:01 ` Topi Miettinen (?) @ 2020-10-26 14:52 ` Catalin Marinas -1 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-26 14:52 UTC (permalink / raw) To: Topi Miettinen Cc: Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On Sat, Oct 24, 2020 at 02:01:30PM +0300, Topi Miettinen wrote: > On 23.10.2020 12.02, Catalin Marinas wrote: > > On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: > > > Regardless, it makes sense to me to have the kernel load the executable > > > itself with BTI enabled by default. I prefer gaining Catalin's suggested > > > patch[2]. :) > > [...] > > > [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ > > > > I think I first heard the idea at Mark R ;). > > > > It still needs glibc changes to avoid the mprotect(), or at least ignore > > the error. Since this is an ABI change and we don't know which kernels > > would have it backported, maybe better to still issue the mprotect() but > > ignore the failure. > > What about kernel adding an auxiliary vector as a flag to indicate that BTI > is supported and recommended by the kernel? Then dynamic loader could use > that to detect that a) the main executable is BTI protected and there's no > need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC > pages. We could add a bit to AT_FLAGS, it's always been 0 for Linux. > In absence of the vector, the dynamic loader might choose to skip doing > PROT_BTI at all (since the main executable isn't protected anyway either, or > maybe even the kernel is up-to-date but it knows that it's not recommended > for some reason, or maybe the kernel is so ancient that it doesn't know > about BTI). Optionally it could still read the flag from ELF later (for > compatibility with old kernels) and then do the mprotect() dance, which may > trip seccomp filters, possibly fatally. I think the safest is for the dynamic loader to issue an mprotect() and ignore the EPERM error. Not all user deployments have this seccomp filter, so they can still benefit, and user can't tell whether the kernel change has been backported. Now, if the dynamic loader silently ignores the mprotect() failure on the main executable, is there much value in exposing a flag in the aux vectors? It saves a few (one?) mprotect() calls but I don't think it matters much. Anyway, I don't mind the flag. The only potential risk is if the dynamic loader decides not to turn PROT_BTI one because of some mix and match of objects but AFAIK BTI allows interworking. -- Catalin ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 14:52 ` Catalin Marinas 0 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-26 14:52 UTC (permalink / raw) To: Topi Miettinen Cc: Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On Sat, Oct 24, 2020 at 02:01:30PM +0300, Topi Miettinen wrote: > On 23.10.2020 12.02, Catalin Marinas wrote: > > On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: > > > Regardless, it makes sense to me to have the kernel load the executable > > > itself with BTI enabled by default. I prefer gaining Catalin's suggested > > > patch[2]. :) > > [...] > > > [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ > > > > I think I first heard the idea at Mark R ;). > > > > It still needs glibc changes to avoid the mprotect(), or at least ignore > > the error. Since this is an ABI change and we don't know which kernels > > would have it backported, maybe better to still issue the mprotect() but > > ignore the failure. > > What about kernel adding an auxiliary vector as a flag to indicate that BTI > is supported and recommended by the kernel? Then dynamic loader could use > that to detect that a) the main executable is BTI protected and there's no > need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC > pages. We could add a bit to AT_FLAGS, it's always been 0 for Linux. > In absence of the vector, the dynamic loader might choose to skip doing > PROT_BTI at all (since the main executable isn't protected anyway either, or > maybe even the kernel is up-to-date but it knows that it's not recommended > for some reason, or maybe the kernel is so ancient that it doesn't know > about BTI). Optionally it could still read the flag from ELF later (for > compatibility with old kernels) and then do the mprotect() dance, which may > trip seccomp filters, possibly fatally. I think the safest is for the dynamic loader to issue an mprotect() and ignore the EPERM error. Not all user deployments have this seccomp filter, so they can still benefit, and user can't tell whether the kernel change has been backported. Now, if the dynamic loader silently ignores the mprotect() failure on the main executable, is there much value in exposing a flag in the aux vectors? It saves a few (one?) mprotect() calls but I don't think it matters much. Anyway, I don't mind the flag. The only potential risk is if the dynamic loader decides not to turn PROT_BTI one because of some mix and match of objects but AFAIK BTI allows interworking. -- Catalin ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 14:52 ` Catalin Marinas 0 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-26 14:52 UTC (permalink / raw) To: Topi Miettinen Cc: Mark Rutland, Salvatore Mesoraca, systemd-devel, Kees Cook, kernel-hardening, Szabolcs Nagy, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, linux-hardening, libc-alpha, Dave Martin, linux-arm-kernel On Sat, Oct 24, 2020 at 02:01:30PM +0300, Topi Miettinen wrote: > On 23.10.2020 12.02, Catalin Marinas wrote: > > On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: > > > Regardless, it makes sense to me to have the kernel load the executable > > > itself with BTI enabled by default. I prefer gaining Catalin's suggested > > > patch[2]. :) > > [...] > > > [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ > > > > I think I first heard the idea at Mark R ;). > > > > It still needs glibc changes to avoid the mprotect(), or at least ignore > > the error. Since this is an ABI change and we don't know which kernels > > would have it backported, maybe better to still issue the mprotect() but > > ignore the failure. > > What about kernel adding an auxiliary vector as a flag to indicate that BTI > is supported and recommended by the kernel? Then dynamic loader could use > that to detect that a) the main executable is BTI protected and there's no > need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC > pages. We could add a bit to AT_FLAGS, it's always been 0 for Linux. > In absence of the vector, the dynamic loader might choose to skip doing > PROT_BTI at all (since the main executable isn't protected anyway either, or > maybe even the kernel is up-to-date but it knows that it's not recommended > for some reason, or maybe the kernel is so ancient that it doesn't know > about BTI). Optionally it could still read the flag from ELF later (for > compatibility with old kernels) and then do the mprotect() dance, which may > trip seccomp filters, possibly fatally. I think the safest is for the dynamic loader to issue an mprotect() and ignore the EPERM error. Not all user deployments have this seccomp filter, so they can still benefit, and user can't tell whether the kernel change has been backported. Now, if the dynamic loader silently ignores the mprotect() failure on the main executable, is there much value in exposing a flag in the aux vectors? It saves a few (one?) mprotect() calls but I don't think it matters much. Anyway, I don't mind the flag. The only potential risk is if the dynamic loader decides not to turn PROT_BTI one because of some mix and match of objects but AFAIK BTI allows interworking. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-26 14:52 ` Catalin Marinas (?) @ 2020-10-26 15:56 ` Dave Martin -1 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-26 15:56 UTC (permalink / raw) To: Catalin Marinas Cc: Topi Miettinen, Mark Rutland, Salvatore Mesoraca, systemd-devel, Kees Cook, kernel-hardening, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, linux-hardening, libc-alpha, linux-arm-kernel On Mon, Oct 26, 2020 at 02:52:46PM +0000, Catalin Marinas via Libc-alpha wrote: > On Sat, Oct 24, 2020 at 02:01:30PM +0300, Topi Miettinen wrote: > > On 23.10.2020 12.02, Catalin Marinas wrote: > > > On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: > > > > Regardless, it makes sense to me to have the kernel load the executable > > > > itself with BTI enabled by default. I prefer gaining Catalin's suggested > > > > patch[2]. :) > > > [...] > > > > [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ > > > > > > I think I first heard the idea at Mark R ;). > > > > > > It still needs glibc changes to avoid the mprotect(), or at least ignore > > > the error. Since this is an ABI change and we don't know which kernels > > > would have it backported, maybe better to still issue the mprotect() but > > > ignore the failure. > > > > What about kernel adding an auxiliary vector as a flag to indicate that BTI > > is supported and recommended by the kernel? Then dynamic loader could use > > that to detect that a) the main executable is BTI protected and there's no > > need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC > > pages. > > We could add a bit to AT_FLAGS, it's always been 0 for Linux. > > > In absence of the vector, the dynamic loader might choose to skip doing > > PROT_BTI at all (since the main executable isn't protected anyway either, or > > maybe even the kernel is up-to-date but it knows that it's not recommended > > for some reason, or maybe the kernel is so ancient that it doesn't know > > about BTI). Optionally it could still read the flag from ELF later (for > > compatibility with old kernels) and then do the mprotect() dance, which may > > trip seccomp filters, possibly fatally. > > I think the safest is for the dynamic loader to issue an mprotect() and > ignore the EPERM error. Not all user deployments have this seccomp > filter, so they can still benefit, and user can't tell whether the > kernel change has been backported. > > Now, if the dynamic loader silently ignores the mprotect() failure on > the main executable, is there much value in exposing a flag in the aux > vectors? It saves a few (one?) mprotect() calls but I don't think it > matters much. Anyway, I don't mind the flag. I don't see a problem with the aforementioned patch [2] to pre-set BTI on the pages of the main binary. The original rationale here was that ld.so doesn't _need_ this, since it is going to examine the binary's ELF headers anyway. But equally, if the binary is marked as supporting BTI then it's safe to enable BTI for the binary's own pages. I'd tend to agree that an AT_FLAGS flag doesn't add much. I think real EPERMs would only be seen in assert-fail type situations. Failure of mmap() is likely to result in a segfault later on, or correct operation with weakened permissions on some pages. Given the likely failure modes, that situation doesn't feel too bad. > The only potential risk is if the dynamic loader decides not to turn > PROT_BTI one because of some mix and match of objects but AFAIK BTI > allows interworking. Yes, the design means that a page's PROT_BTI can be set safely if the code in that page was compiled for BTI, irrespective of how other pages were compiled. The reasons why we don't do this at finer granularity are (a) is't not very useful, and (b) ELF images only contain a BTI property note for the whole image, not per segment. I think that ld.so already makes this decision at ELF image granularity (unless someone contradicts me). Cheers ---Dave ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 15:56 ` Dave Martin 0 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-26 15:56 UTC (permalink / raw) To: Catalin Marinas Cc: Topi Miettinen, Mark Rutland, Salvatore Mesoraca, systemd-devel, Kees Cook, kernel-hardening, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, linux-hardening, libc-alpha, linux-arm-kernel On Mon, Oct 26, 2020 at 02:52:46PM +0000, Catalin Marinas via Libc-alpha wrote: > On Sat, Oct 24, 2020 at 02:01:30PM +0300, Topi Miettinen wrote: > > On 23.10.2020 12.02, Catalin Marinas wrote: > > > On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: > > > > Regardless, it makes sense to me to have the kernel load the executable > > > > itself with BTI enabled by default. I prefer gaining Catalin's suggested > > > > patch[2]. :) > > > [...] > > > > [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ > > > > > > I think I first heard the idea at Mark R ;). > > > > > > It still needs glibc changes to avoid the mprotect(), or at least ignore > > > the error. Since this is an ABI change and we don't know which kernels > > > would have it backported, maybe better to still issue the mprotect() but > > > ignore the failure. > > > > What about kernel adding an auxiliary vector as a flag to indicate that BTI > > is supported and recommended by the kernel? Then dynamic loader could use > > that to detect that a) the main executable is BTI protected and there's no > > need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC > > pages. > > We could add a bit to AT_FLAGS, it's always been 0 for Linux. > > > In absence of the vector, the dynamic loader might choose to skip doing > > PROT_BTI at all (since the main executable isn't protected anyway either, or > > maybe even the kernel is up-to-date but it knows that it's not recommended > > for some reason, or maybe the kernel is so ancient that it doesn't know > > about BTI). Optionally it could still read the flag from ELF later (for > > compatibility with old kernels) and then do the mprotect() dance, which may > > trip seccomp filters, possibly fatally. > > I think the safest is for the dynamic loader to issue an mprotect() and > ignore the EPERM error. Not all user deployments have this seccomp > filter, so they can still benefit, and user can't tell whether the > kernel change has been backported. > > Now, if the dynamic loader silently ignores the mprotect() failure on > the main executable, is there much value in exposing a flag in the aux > vectors? It saves a few (one?) mprotect() calls but I don't think it > matters much. Anyway, I don't mind the flag. I don't see a problem with the aforementioned patch [2] to pre-set BTI on the pages of the main binary. The original rationale here was that ld.so doesn't _need_ this, since it is going to examine the binary's ELF headers anyway. But equally, if the binary is marked as supporting BTI then it's safe to enable BTI for the binary's own pages. I'd tend to agree that an AT_FLAGS flag doesn't add much. I think real EPERMs would only be seen in assert-fail type situations. Failure of mmap() is likely to result in a segfault later on, or correct operation with weakened permissions on some pages. Given the likely failure modes, that situation doesn't feel too bad. > The only potential risk is if the dynamic loader decides not to turn > PROT_BTI one because of some mix and match of objects but AFAIK BTI > allows interworking. Yes, the design means that a page's PROT_BTI can be set safely if the code in that page was compiled for BTI, irrespective of how other pages were compiled. The reasons why we don't do this at finer granularity are (a) is't not very useful, and (b) ELF images only contain a BTI property note for the whole image, not per segment. I think that ld.so already makes this decision at ELF image granularity (unless someone contradicts me). Cheers ---Dave ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 15:56 ` Dave Martin 0 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-26 15:56 UTC (permalink / raw) To: Catalin Marinas Cc: Mark Rutland, Salvatore Mesoraca, systemd-devel, Kees Cook, kernel-hardening, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, Topi Miettinen, linux-hardening, libc-alpha, linux-arm-kernel On Mon, Oct 26, 2020 at 02:52:46PM +0000, Catalin Marinas via Libc-alpha wrote: > On Sat, Oct 24, 2020 at 02:01:30PM +0300, Topi Miettinen wrote: > > On 23.10.2020 12.02, Catalin Marinas wrote: > > > On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: > > > > Regardless, it makes sense to me to have the kernel load the executable > > > > itself with BTI enabled by default. I prefer gaining Catalin's suggested > > > > patch[2]. :) > > > [...] > > > > [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ > > > > > > I think I first heard the idea at Mark R ;). > > > > > > It still needs glibc changes to avoid the mprotect(), or at least ignore > > > the error. Since this is an ABI change and we don't know which kernels > > > would have it backported, maybe better to still issue the mprotect() but > > > ignore the failure. > > > > What about kernel adding an auxiliary vector as a flag to indicate that BTI > > is supported and recommended by the kernel? Then dynamic loader could use > > that to detect that a) the main executable is BTI protected and there's no > > need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC > > pages. > > We could add a bit to AT_FLAGS, it's always been 0 for Linux. > > > In absence of the vector, the dynamic loader might choose to skip doing > > PROT_BTI at all (since the main executable isn't protected anyway either, or > > maybe even the kernel is up-to-date but it knows that it's not recommended > > for some reason, or maybe the kernel is so ancient that it doesn't know > > about BTI). Optionally it could still read the flag from ELF later (for > > compatibility with old kernels) and then do the mprotect() dance, which may > > trip seccomp filters, possibly fatally. > > I think the safest is for the dynamic loader to issue an mprotect() and > ignore the EPERM error. Not all user deployments have this seccomp > filter, so they can still benefit, and user can't tell whether the > kernel change has been backported. > > Now, if the dynamic loader silently ignores the mprotect() failure on > the main executable, is there much value in exposing a flag in the aux > vectors? It saves a few (one?) mprotect() calls but I don't think it > matters much. Anyway, I don't mind the flag. I don't see a problem with the aforementioned patch [2] to pre-set BTI on the pages of the main binary. The original rationale here was that ld.so doesn't _need_ this, since it is going to examine the binary's ELF headers anyway. But equally, if the binary is marked as supporting BTI then it's safe to enable BTI for the binary's own pages. I'd tend to agree that an AT_FLAGS flag doesn't add much. I think real EPERMs would only be seen in assert-fail type situations. Failure of mmap() is likely to result in a segfault later on, or correct operation with weakened permissions on some pages. Given the likely failure modes, that situation doesn't feel too bad. > The only potential risk is if the dynamic loader decides not to turn > PROT_BTI one because of some mix and match of objects but AFAIK BTI > allows interworking. Yes, the design means that a page's PROT_BTI can be set safely if the code in that page was compiled for BTI, irrespective of how other pages were compiled. The reasons why we don't do this at finer granularity are (a) is't not very useful, and (b) ELF images only contain a BTI property note for the whole image, not per segment. I think that ld.so already makes this decision at ELF image granularity (unless someone contradicts me). Cheers ---Dave _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-26 15:56 ` Dave Martin (?) @ 2020-10-26 16:51 ` Mark Brown -1 siblings, 0 replies; 88+ messages in thread From: Mark Brown @ 2020-10-26 16:51 UTC (permalink / raw) To: Dave Martin Cc: Catalin Marinas, Topi Miettinen, Mark Rutland, Salvatore Mesoraca, systemd-devel, Kees Cook, kernel-hardening, Will Deacon, linux-kernel, Jeremy Linton, linux-hardening, libc-alpha, linux-arm-kernel [-- Attachment #1: Type: text/plain, Size: 541 bytes --] On Mon, Oct 26, 2020 at 03:56:35PM +0000, Dave Martin wrote: > On Mon, Oct 26, 2020 at 02:52:46PM +0000, Catalin Marinas via Libc-alpha wrote: > > Now, if the dynamic loader silently ignores the mprotect() failure on > > the main executable, is there much value in exposing a flag in the aux > > vectors? It saves a few (one?) mprotect() calls but I don't think it > > matters much. Anyway, I don't mind the flag. > I don't see a problem with the aforementioned patch [2] to pre-set BTI > on the pages of the main binary. Me either FWIW. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 16:51 ` Mark Brown 0 siblings, 0 replies; 88+ messages in thread From: Mark Brown @ 2020-10-26 16:51 UTC (permalink / raw) To: Dave Martin Cc: Catalin Marinas, Topi Miettinen, Mark Rutland, Salvatore Mesoraca, systemd-devel, Kees Cook, kernel-hardening, Will Deacon, linux-kernel, Jeremy Linton, linux-hardening, libc-alpha, linux-arm-kernel [-- Attachment #1: Type: text/plain, Size: 541 bytes --] On Mon, Oct 26, 2020 at 03:56:35PM +0000, Dave Martin wrote: > On Mon, Oct 26, 2020 at 02:52:46PM +0000, Catalin Marinas via Libc-alpha wrote: > > Now, if the dynamic loader silently ignores the mprotect() failure on > > the main executable, is there much value in exposing a flag in the aux > > vectors? It saves a few (one?) mprotect() calls but I don't think it > > matters much. Anyway, I don't mind the flag. > I don't see a problem with the aforementioned patch [2] to pre-set BTI > on the pages of the main binary. Me either FWIW. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 16:51 ` Mark Brown 0 siblings, 0 replies; 88+ messages in thread From: Mark Brown @ 2020-10-26 16:51 UTC (permalink / raw) To: Dave Martin Cc: Mark Rutland, Salvatore Mesoraca, systemd-devel, Kees Cook, kernel-hardening, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Topi Miettinen, linux-hardening, libc-alpha, linux-arm-kernel [-- Attachment #1.1: Type: text/plain, Size: 541 bytes --] On Mon, Oct 26, 2020 at 03:56:35PM +0000, Dave Martin wrote: > On Mon, Oct 26, 2020 at 02:52:46PM +0000, Catalin Marinas via Libc-alpha wrote: > > Now, if the dynamic loader silently ignores the mprotect() failure on > > the main executable, is there much value in exposing a flag in the aux > > vectors? It saves a few (one?) mprotect() calls but I don't think it > > matters much. Anyway, I don't mind the flag. > I don't see a problem with the aforementioned patch [2] to pre-set BTI > on the pages of the main binary. Me either FWIW. [-- Attachment #1.2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] [-- Attachment #2: Type: text/plain, Size: 176 bytes --] _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-26 14:52 ` Catalin Marinas (?) @ 2020-10-26 16:31 ` Topi Miettinen -1 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-26 16:31 UTC (permalink / raw) To: Catalin Marinas Cc: Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On 26.10.2020 16.52, Catalin Marinas wrote: > On Sat, Oct 24, 2020 at 02:01:30PM +0300, Topi Miettinen wrote: >> On 23.10.2020 12.02, Catalin Marinas wrote: >>> On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: >>>> Regardless, it makes sense to me to have the kernel load the executable >>>> itself with BTI enabled by default. I prefer gaining Catalin's suggested >>>> patch[2]. :) >>> [...] >>>> [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ >>> >>> I think I first heard the idea at Mark R ;). >>> >>> It still needs glibc changes to avoid the mprotect(), or at least ignore >>> the error. Since this is an ABI change and we don't know which kernels >>> would have it backported, maybe better to still issue the mprotect() but >>> ignore the failure. >> >> What about kernel adding an auxiliary vector as a flag to indicate that BTI >> is supported and recommended by the kernel? Then dynamic loader could use >> that to detect that a) the main executable is BTI protected and there's no >> need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC >> pages. > > We could add a bit to AT_FLAGS, it's always been 0 for Linux. Great! >> In absence of the vector, the dynamic loader might choose to skip doing >> PROT_BTI at all (since the main executable isn't protected anyway either, or >> maybe even the kernel is up-to-date but it knows that it's not recommended >> for some reason, or maybe the kernel is so ancient that it doesn't know >> about BTI). Optionally it could still read the flag from ELF later (for >> compatibility with old kernels) and then do the mprotect() dance, which may >> trip seccomp filters, possibly fatally. > > I think the safest is for the dynamic loader to issue an mprotect() and > ignore the EPERM error. Not all user deployments have this seccomp > filter, so they can still benefit, and user can't tell whether the > kernel change has been backported. But the seccomp filter can be set to kill the process, so that's definitely not the safest way. I think safest is that when the AT_FLAGS bit is seen, ld.so doesn't do any mprotect() calls but instead when mapping the segments, mmap() flags are adjusted to include PROT_BTI, so mprotect() calls are not necessary. If there's no seccomp filter, there's no disadvantage for avoiding the useless mprotect() calls. I'd expect the backported kernel change to include both aux vector and also using PROT_BTI for the main executable. Then the logic would work with backported kernels as well. If there's no aux vector, all bets are off. The kernel could be old and unpatched, even so old that PROT_BTI is not known. Perhaps also in the future there may be new technologies which have replaced BTI and the kernel could want a previous generation ld.so not to try to use BTI, so this could be also indicated with the lack of aux vector. The dynamic loader could still attempt to mprotect() the pages, but that could be fatal. Getting to the point where the error can be ignored means that there's no seccomp filter, at least none set to kill. Perhaps the pain is only temporary, new or patched kernels should eventually replace the old versions. > Now, if the dynamic loader silently ignores the mprotect() failure on > the main executable, is there much value in exposing a flag in the aux > vectors? It saves a few (one?) mprotect() calls but I don't think it > matters much. Anyway, I don't mind the flag. Saving a few system calls is indeed not an issue, but not being able to use MDWX and PROT_BTI simultaneously was the original problem (service failures). -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 16:31 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-26 16:31 UTC (permalink / raw) To: Catalin Marinas Cc: Kees Cook, Szabolcs Nagy, Jeremy Linton, linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Mark Brown, Dave Martin, Will Deacon, Salvatore Mesoraca, kernel-hardening, linux-hardening On 26.10.2020 16.52, Catalin Marinas wrote: > On Sat, Oct 24, 2020 at 02:01:30PM +0300, Topi Miettinen wrote: >> On 23.10.2020 12.02, Catalin Marinas wrote: >>> On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: >>>> Regardless, it makes sense to me to have the kernel load the executable >>>> itself with BTI enabled by default. I prefer gaining Catalin's suggested >>>> patch[2]. :) >>> [...] >>>> [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ >>> >>> I think I first heard the idea at Mark R ;). >>> >>> It still needs glibc changes to avoid the mprotect(), or at least ignore >>> the error. Since this is an ABI change and we don't know which kernels >>> would have it backported, maybe better to still issue the mprotect() but >>> ignore the failure. >> >> What about kernel adding an auxiliary vector as a flag to indicate that BTI >> is supported and recommended by the kernel? Then dynamic loader could use >> that to detect that a) the main executable is BTI protected and there's no >> need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC >> pages. > > We could add a bit to AT_FLAGS, it's always been 0 for Linux. Great! >> In absence of the vector, the dynamic loader might choose to skip doing >> PROT_BTI at all (since the main executable isn't protected anyway either, or >> maybe even the kernel is up-to-date but it knows that it's not recommended >> for some reason, or maybe the kernel is so ancient that it doesn't know >> about BTI). Optionally it could still read the flag from ELF later (for >> compatibility with old kernels) and then do the mprotect() dance, which may >> trip seccomp filters, possibly fatally. > > I think the safest is for the dynamic loader to issue an mprotect() and > ignore the EPERM error. Not all user deployments have this seccomp > filter, so they can still benefit, and user can't tell whether the > kernel change has been backported. But the seccomp filter can be set to kill the process, so that's definitely not the safest way. I think safest is that when the AT_FLAGS bit is seen, ld.so doesn't do any mprotect() calls but instead when mapping the segments, mmap() flags are adjusted to include PROT_BTI, so mprotect() calls are not necessary. If there's no seccomp filter, there's no disadvantage for avoiding the useless mprotect() calls. I'd expect the backported kernel change to include both aux vector and also using PROT_BTI for the main executable. Then the logic would work with backported kernels as well. If there's no aux vector, all bets are off. The kernel could be old and unpatched, even so old that PROT_BTI is not known. Perhaps also in the future there may be new technologies which have replaced BTI and the kernel could want a previous generation ld.so not to try to use BTI, so this could be also indicated with the lack of aux vector. The dynamic loader could still attempt to mprotect() the pages, but that could be fatal. Getting to the point where the error can be ignored means that there's no seccomp filter, at least none set to kill. Perhaps the pain is only temporary, new or patched kernels should eventually replace the old versions. > Now, if the dynamic loader silently ignores the mprotect() failure on > the main executable, is there much value in exposing a flag in the aux > vectors? It saves a few (one?) mprotect() calls but I don't think it > matters much. Anyway, I don't mind the flag. Saving a few system calls is indeed not an issue, but not being able to use MDWX and PROT_BTI simultaneously was the original problem (service failures). -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 16:31 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-26 16:31 UTC (permalink / raw) To: Catalin Marinas Cc: Mark Rutland, Salvatore Mesoraca, systemd-devel, Kees Cook, kernel-hardening, Szabolcs Nagy, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, linux-hardening, libc-alpha, Dave Martin, linux-arm-kernel On 26.10.2020 16.52, Catalin Marinas wrote: > On Sat, Oct 24, 2020 at 02:01:30PM +0300, Topi Miettinen wrote: >> On 23.10.2020 12.02, Catalin Marinas wrote: >>> On Thu, Oct 22, 2020 at 01:02:18PM -0700, Kees Cook wrote: >>>> Regardless, it makes sense to me to have the kernel load the executable >>>> itself with BTI enabled by default. I prefer gaining Catalin's suggested >>>> patch[2]. :) >>> [...] >>>> [2] https://lore.kernel.org/linux-arm-kernel/20201022093104.GB1229@gaia/ >>> >>> I think I first heard the idea at Mark R ;). >>> >>> It still needs glibc changes to avoid the mprotect(), or at least ignore >>> the error. Since this is an ABI change and we don't know which kernels >>> would have it backported, maybe better to still issue the mprotect() but >>> ignore the failure. >> >> What about kernel adding an auxiliary vector as a flag to indicate that BTI >> is supported and recommended by the kernel? Then dynamic loader could use >> that to detect that a) the main executable is BTI protected and there's no >> need to mprotect() it and b) PROT_BTI flag should be added to all PROT_EXEC >> pages. > > We could add a bit to AT_FLAGS, it's always been 0 for Linux. Great! >> In absence of the vector, the dynamic loader might choose to skip doing >> PROT_BTI at all (since the main executable isn't protected anyway either, or >> maybe even the kernel is up-to-date but it knows that it's not recommended >> for some reason, or maybe the kernel is so ancient that it doesn't know >> about BTI). Optionally it could still read the flag from ELF later (for >> compatibility with old kernels) and then do the mprotect() dance, which may >> trip seccomp filters, possibly fatally. > > I think the safest is for the dynamic loader to issue an mprotect() and > ignore the EPERM error. Not all user deployments have this seccomp > filter, so they can still benefit, and user can't tell whether the > kernel change has been backported. But the seccomp filter can be set to kill the process, so that's definitely not the safest way. I think safest is that when the AT_FLAGS bit is seen, ld.so doesn't do any mprotect() calls but instead when mapping the segments, mmap() flags are adjusted to include PROT_BTI, so mprotect() calls are not necessary. If there's no seccomp filter, there's no disadvantage for avoiding the useless mprotect() calls. I'd expect the backported kernel change to include both aux vector and also using PROT_BTI for the main executable. Then the logic would work with backported kernels as well. If there's no aux vector, all bets are off. The kernel could be old and unpatched, even so old that PROT_BTI is not known. Perhaps also in the future there may be new technologies which have replaced BTI and the kernel could want a previous generation ld.so not to try to use BTI, so this could be also indicated with the lack of aux vector. The dynamic loader could still attempt to mprotect() the pages, but that could be fatal. Getting to the point where the error can be ignored means that there's no seccomp filter, at least none set to kill. Perhaps the pain is only temporary, new or patched kernels should eventually replace the old versions. > Now, if the dynamic loader silently ignores the mprotect() failure on > the main executable, is there much value in exposing a flag in the aux > vectors? It saves a few (one?) mprotect() calls but I don't think it > matters much. Anyway, I don't mind the flag. Saving a few system calls is indeed not an issue, but not being able to use MDWX and PROT_BTI simultaneously was the original problem (service failures). -Topi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures [not found] <8584c14f-5c28-9d70-c054-7c78127d84ea@arm.com> @ 2020-10-26 16:24 ` Dave Martin [not found] ` <20201022075447.GO3819@arm.com> 2020-10-26 16:24 ` Dave Martin 2 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-26 16:24 UTC (permalink / raw) To: Jeremy Linton Cc: linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Kees Cook, Catalin Marinas, Will Deacon, Mark Brown, toiwoton On Wed, Oct 21, 2020 at 10:44:46PM -0500, Jeremy Linton via Libc-alpha wrote: > Hi, > > There is a problem with glibc+systemd on BTI enabled systems. Systemd > has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny > PROT_EXEC changes. Glibc enables BTI only on segments which are marked as > being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is > caught by the seccomp filter, resulting in service failures. > > So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. > This is obviously not desirable. > > Various changes have been suggested, replacing the mprotect with mmap calls > having PROT_BTI set on the original mapping, re-mmapping the segments, > implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, > and various modification to seccomp to allow particular mprotect cases to > bypass the filters. In each case there seems to be an undesirable attribute > to the solution. > > So, whats the best solution? Unrolling this discussion a bit, this problem comes from a few sources: 1) systemd is trying to implement a policy that doesn't fit SECCOMP syscall filtering very well. 2) The program is trying to do something not expressible through the syscall interface: really the intent is to set PROT_BTI on the page, with no intent to set PROT_EXEC on any page that didn't already have it set. This limitation of mprotect() was known when I originally added PROT_BTI, but at that time we weren't aware of a clear use case that would fail. Would it now help to add something like: int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) { int ret = -EINVAL; mmap_write_lock(current->mm); if (all vmas in [addr .. addr + len) have their mprotect flags set to old_flags) { ret = mprotect(addr, len, new_flags); } mmap_write_unlock(current->mm); return ret; } libc would now be able to do mchangeprot(addr, len, PROT_EXEC | PROT_READ, PROT_EXEC | PROT_READ | PROT_BTI); while systemd's MDWX filter would reject the call if (new_flags & PROT_EXEC) && (!(old_flags & PROT_EXEC) || (new_flags & PROT_WRITE) This won't magically fix current code, but something along these lines might be better going forward. Thoughts? ---Dave ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 16:24 ` Dave Martin 0 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-26 16:24 UTC (permalink / raw) To: Jeremy Linton Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel On Wed, Oct 21, 2020 at 10:44:46PM -0500, Jeremy Linton via Libc-alpha wrote: > Hi, > > There is a problem with glibc+systemd on BTI enabled systems. Systemd > has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny > PROT_EXEC changes. Glibc enables BTI only on segments which are marked as > being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is > caught by the seccomp filter, resulting in service failures. > > So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. > This is obviously not desirable. > > Various changes have been suggested, replacing the mprotect with mmap calls > having PROT_BTI set on the original mapping, re-mmapping the segments, > implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, > and various modification to seccomp to allow particular mprotect cases to > bypass the filters. In each case there seems to be an undesirable attribute > to the solution. > > So, whats the best solution? Unrolling this discussion a bit, this problem comes from a few sources: 1) systemd is trying to implement a policy that doesn't fit SECCOMP syscall filtering very well. 2) The program is trying to do something not expressible through the syscall interface: really the intent is to set PROT_BTI on the page, with no intent to set PROT_EXEC on any page that didn't already have it set. This limitation of mprotect() was known when I originally added PROT_BTI, but at that time we weren't aware of a clear use case that would fail. Would it now help to add something like: int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) { int ret = -EINVAL; mmap_write_lock(current->mm); if (all vmas in [addr .. addr + len) have their mprotect flags set to old_flags) { ret = mprotect(addr, len, new_flags); } mmap_write_unlock(current->mm); return ret; } libc would now be able to do mchangeprot(addr, len, PROT_EXEC | PROT_READ, PROT_EXEC | PROT_READ | PROT_BTI); while systemd's MDWX filter would reject the call if (new_flags & PROT_EXEC) && (!(old_flags & PROT_EXEC) || (new_flags & PROT_WRITE) This won't magically fix current code, but something along these lines might be better going forward. Thoughts? ---Dave _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-26 16:24 ` Dave Martin @ 2020-10-26 16:39 ` Topi Miettinen -1 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-26 16:39 UTC (permalink / raw) To: Dave Martin, Jeremy Linton Cc: linux-arm-kernel, libc-alpha, systemd-devel, linux-kernel, Mark Rutland, Kees Cook, Catalin Marinas, Will Deacon, Mark Brown On 26.10.2020 18.24, Dave Martin wrote: > On Wed, Oct 21, 2020 at 10:44:46PM -0500, Jeremy Linton via Libc-alpha wrote: >> Hi, >> >> There is a problem with glibc+systemd on BTI enabled systems. Systemd >> has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny >> PROT_EXEC changes. Glibc enables BTI only on segments which are marked as >> being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is >> caught by the seccomp filter, resulting in service failures. >> >> So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. >> This is obviously not desirable. >> >> Various changes have been suggested, replacing the mprotect with mmap calls >> having PROT_BTI set on the original mapping, re-mmapping the segments, >> implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, >> and various modification to seccomp to allow particular mprotect cases to >> bypass the filters. In each case there seems to be an undesirable attribute >> to the solution. >> >> So, whats the best solution? > > Unrolling this discussion a bit, this problem comes from a few sources: > > 1) systemd is trying to implement a policy that doesn't fit SECCOMP > syscall filtering very well. > > 2) The program is trying to do something not expressible through the > syscall interface: really the intent is to set PROT_BTI on the page, > with no intent to set PROT_EXEC on any page that didn't already have it > set. > > > This limitation of mprotect() was known when I originally added PROT_BTI, > but at that time we weren't aware of a clear use case that would fail. > > > Would it now help to add something like: > > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > { > int ret = -EINVAL; > mmap_write_lock(current->mm); > if (all vmas in [addr .. addr + len) have > their mprotect flags set to old_flags) { > > ret = mprotect(addr, len, new_flags); > } > > mmap_write_unlock(current->mm); > return ret; > } > > > libc would now be able to do > > mchangeprot(addr, len, PROT_EXEC | PROT_READ, > PROT_EXEC | PROT_READ | PROT_BTI); > > while systemd's MDWX filter would reject the call if > > (new_flags & PROT_EXEC) && > (!(old_flags & PROT_EXEC) || (new_flags & PROT_WRITE) > > > > This won't magically fix current code, but something along these lines > might be better going forward. > > > Thoughts? Looks good to me. -Topi ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 16:39 ` Topi Miettinen 0 siblings, 0 replies; 88+ messages in thread From: Topi Miettinen @ 2020-10-26 16:39 UTC (permalink / raw) To: Dave Martin, Jeremy Linton Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, libc-alpha, linux-arm-kernel On 26.10.2020 18.24, Dave Martin wrote: > On Wed, Oct 21, 2020 at 10:44:46PM -0500, Jeremy Linton via Libc-alpha wrote: >> Hi, >> >> There is a problem with glibc+systemd on BTI enabled systems. Systemd >> has a service flag "MemoryDenyWriteExecute" which uses seccomp to deny >> PROT_EXEC changes. Glibc enables BTI only on segments which are marked as >> being BTI compatible by calling mprotect PROT_EXEC|PROT_BTI. That call is >> caught by the seccomp filter, resulting in service failures. >> >> So, at the moment one has to pick either denying PROT_EXEC changes, or BTI. >> This is obviously not desirable. >> >> Various changes have been suggested, replacing the mprotect with mmap calls >> having PROT_BTI set on the original mapping, re-mmapping the segments, >> implying PROT_EXEC on mprotect PROT_BTI calls when VM_EXEC is already set, >> and various modification to seccomp to allow particular mprotect cases to >> bypass the filters. In each case there seems to be an undesirable attribute >> to the solution. >> >> So, whats the best solution? > > Unrolling this discussion a bit, this problem comes from a few sources: > > 1) systemd is trying to implement a policy that doesn't fit SECCOMP > syscall filtering very well. > > 2) The program is trying to do something not expressible through the > syscall interface: really the intent is to set PROT_BTI on the page, > with no intent to set PROT_EXEC on any page that didn't already have it > set. > > > This limitation of mprotect() was known when I originally added PROT_BTI, > but at that time we weren't aware of a clear use case that would fail. > > > Would it now help to add something like: > > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > { > int ret = -EINVAL; > mmap_write_lock(current->mm); > if (all vmas in [addr .. addr + len) have > their mprotect flags set to old_flags) { > > ret = mprotect(addr, len, new_flags); > } > > mmap_write_unlock(current->mm); > return ret; > } > > > libc would now be able to do > > mchangeprot(addr, len, PROT_EXEC | PROT_READ, > PROT_EXEC | PROT_READ | PROT_BTI); > > while systemd's MDWX filter would reject the call if > > (new_flags & PROT_EXEC) && > (!(old_flags & PROT_EXEC) || (new_flags & PROT_WRITE) > > > > This won't magically fix current code, but something along these lines > might be better going forward. > > > Thoughts? Looks good to me. -Topi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-26 16:24 ` Dave Martin @ 2020-10-26 16:45 ` Florian Weimer -1 siblings, 0 replies; 88+ messages in thread From: Florian Weimer @ 2020-10-26 16:45 UTC (permalink / raw) To: Dave Martin via Libc-alpha Cc: Jeremy Linton, Dave Martin, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, toiwoton, linux-arm-kernel * Dave Martin via Libc-alpha: > Would it now help to add something like: > > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > { > int ret = -EINVAL; > mmap_write_lock(current->mm); > if (all vmas in [addr .. addr + len) have > their mprotect flags set to old_flags) { > > ret = mprotect(addr, len, new_flags); > } > > mmap_write_unlock(current->mm); > return ret; > } I suggested something similar as well. Ideally, the interface would subsume pkey_mprotect, though, and have a separate flags argument from the protection flags. But then we run into argument list length limits. Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 16:45 ` Florian Weimer 0 siblings, 0 replies; 88+ messages in thread From: Florian Weimer @ 2020-10-26 16:45 UTC (permalink / raw) To: Dave Martin via Libc-alpha Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, Dave Martin, linux-arm-kernel * Dave Martin via Libc-alpha: > Would it now help to add something like: > > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > { > int ret = -EINVAL; > mmap_write_lock(current->mm); > if (all vmas in [addr .. addr + len) have > their mprotect flags set to old_flags) { > > ret = mprotect(addr, len, new_flags); > } > > mmap_write_unlock(current->mm); > return ret; > } I suggested something similar as well. Ideally, the interface would subsume pkey_mprotect, though, and have a separate flags argument from the protection flags. But then we run into argument list length limits. Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-26 16:45 ` Florian Weimer @ 2020-10-27 14:22 ` Dave Martin -1 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-27 14:22 UTC (permalink / raw) To: Florian Weimer Cc: Dave Martin via Libc-alpha, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, linux-arm-kernel On Mon, Oct 26, 2020 at 05:45:42PM +0100, Florian Weimer via Libc-alpha wrote: > * Dave Martin via Libc-alpha: > > > Would it now help to add something like: > > > > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > > { > > int ret = -EINVAL; > > mmap_write_lock(current->mm); > > if (all vmas in [addr .. addr + len) have > > their mprotect flags set to old_flags) { > > > > ret = mprotect(addr, len, new_flags); > > } > > > > mmap_write_unlock(current->mm); > > return ret; > > } > > I suggested something similar as well. Ideally, the interface would > subsume pkey_mprotect, though, and have a separate flags argument from > the protection flags. But then we run into argument list length limits. > > Thanks, > Florian I suppose. Assuming that a syscall filter can inspect memory, we might be able to bundle arguments into a struct if necessary. [...] Cheers ---Dave ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-27 14:22 ` Dave Martin 0 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-27 14:22 UTC (permalink / raw) To: Florian Weimer Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, Dave Martin via Libc-alpha, linux-arm-kernel On Mon, Oct 26, 2020 at 05:45:42PM +0100, Florian Weimer via Libc-alpha wrote: > * Dave Martin via Libc-alpha: > > > Would it now help to add something like: > > > > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > > { > > int ret = -EINVAL; > > mmap_write_lock(current->mm); > > if (all vmas in [addr .. addr + len) have > > their mprotect flags set to old_flags) { > > > > ret = mprotect(addr, len, new_flags); > > } > > > > mmap_write_unlock(current->mm); > > return ret; > > } > > I suggested something similar as well. Ideally, the interface would > subsume pkey_mprotect, though, and have a separate flags argument from > the protection flags. But then we run into argument list length limits. > > Thanks, > Florian I suppose. Assuming that a syscall filter can inspect memory, we might be able to bundle arguments into a struct if necessary. [...] Cheers ---Dave _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-27 14:22 ` Dave Martin @ 2020-10-27 14:41 ` Florian Weimer -1 siblings, 0 replies; 88+ messages in thread From: Florian Weimer @ 2020-10-27 14:41 UTC (permalink / raw) To: Dave Martin via Libc-alpha Cc: Dave Martin, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, linux-arm-kernel * Dave Martin via Libc-alpha: > On Mon, Oct 26, 2020 at 05:45:42PM +0100, Florian Weimer via Libc-alpha wrote: >> * Dave Martin via Libc-alpha: >> >> > Would it now help to add something like: >> > >> > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) >> > { >> > int ret = -EINVAL; >> > mmap_write_lock(current->mm); >> > if (all vmas in [addr .. addr + len) have >> > their mprotect flags set to old_flags) { >> > >> > ret = mprotect(addr, len, new_flags); >> > } >> > >> > mmap_write_unlock(current->mm); >> > return ret; >> > } >> >> I suggested something similar as well. Ideally, the interface would >> subsume pkey_mprotect, though, and have a separate flags argument from >> the protection flags. But then we run into argument list length limits. >> >> Thanks, >> Florian > > I suppose. Assuming that a syscall filter can inspect memory, we might > be able to bundle arguments into a struct if necessary. But that leads to a discussion about batch mmap/mprotect/munmap, and that's again incompatible with seccomp (it would need a checking loop). Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-27 14:41 ` Florian Weimer 0 siblings, 0 replies; 88+ messages in thread From: Florian Weimer @ 2020-10-27 14:41 UTC (permalink / raw) To: Dave Martin via Libc-alpha Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, Dave Martin, linux-arm-kernel * Dave Martin via Libc-alpha: > On Mon, Oct 26, 2020 at 05:45:42PM +0100, Florian Weimer via Libc-alpha wrote: >> * Dave Martin via Libc-alpha: >> >> > Would it now help to add something like: >> > >> > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) >> > { >> > int ret = -EINVAL; >> > mmap_write_lock(current->mm); >> > if (all vmas in [addr .. addr + len) have >> > their mprotect flags set to old_flags) { >> > >> > ret = mprotect(addr, len, new_flags); >> > } >> > >> > mmap_write_unlock(current->mm); >> > return ret; >> > } >> >> I suggested something similar as well. Ideally, the interface would >> subsume pkey_mprotect, though, and have a separate flags argument from >> the protection flags. But then we run into argument list length limits. >> >> Thanks, >> Florian > > I suppose. Assuming that a syscall filter can inspect memory, we might > be able to bundle arguments into a struct if necessary. But that leads to a discussion about batch mmap/mprotect/munmap, and that's again incompatible with seccomp (it would need a checking loop). Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-26 16:24 ` Dave Martin @ 2020-10-26 16:57 ` Szabolcs Nagy -1 siblings, 0 replies; 88+ messages in thread From: Szabolcs Nagy @ 2020-10-26 16:57 UTC (permalink / raw) To: Dave Martin Cc: Jeremy Linton, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel The 10/26/2020 16:24, Dave Martin via Libc-alpha wrote: > Unrolling this discussion a bit, this problem comes from a few sources: > > 1) systemd is trying to implement a policy that doesn't fit SECCOMP > syscall filtering very well. > > 2) The program is trying to do something not expressible through the > syscall interface: really the intent is to set PROT_BTI on the page, > with no intent to set PROT_EXEC on any page that didn't already have it > set. > > > This limitation of mprotect() was known when I originally added PROT_BTI, > but at that time we weren't aware of a clear use case that would fail. > > > Would it now help to add something like: > > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > { > int ret = -EINVAL; > mmap_write_lock(current->mm); > if (all vmas in [addr .. addr + len) have > their mprotect flags set to old_flags) { > > ret = mprotect(addr, len, new_flags); > } > > mmap_write_unlock(current->mm); > return ret; > } if more prot flags are introduced then the exact match for old_flags may be restrictive and currently there is no way to query these flags to figure out how to toggle one prot flag in a future proof way, so i don't think this solves the issue completely. i think we might need a new api, given that aarch64 now has PROT_BTI and PROT_MTE while existing code expects RWX only, but i don't know what api is best. > libc would now be able to do > > mchangeprot(addr, len, PROT_EXEC | PROT_READ, > PROT_EXEC | PROT_READ | PROT_BTI); > > while systemd's MDWX filter would reject the call if > > (new_flags & PROT_EXEC) && > (!(old_flags & PROT_EXEC) || (new_flags & PROT_WRITE) > > > > This won't magically fix current code, but something along these lines > might be better going forward. > > > Thoughts? > > ---Dave ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 16:57 ` Szabolcs Nagy 0 siblings, 0 replies; 88+ messages in thread From: Szabolcs Nagy @ 2020-10-26 16:57 UTC (permalink / raw) To: Dave Martin Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel The 10/26/2020 16:24, Dave Martin via Libc-alpha wrote: > Unrolling this discussion a bit, this problem comes from a few sources: > > 1) systemd is trying to implement a policy that doesn't fit SECCOMP > syscall filtering very well. > > 2) The program is trying to do something not expressible through the > syscall interface: really the intent is to set PROT_BTI on the page, > with no intent to set PROT_EXEC on any page that didn't already have it > set. > > > This limitation of mprotect() was known when I originally added PROT_BTI, > but at that time we weren't aware of a clear use case that would fail. > > > Would it now help to add something like: > > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > { > int ret = -EINVAL; > mmap_write_lock(current->mm); > if (all vmas in [addr .. addr + len) have > their mprotect flags set to old_flags) { > > ret = mprotect(addr, len, new_flags); > } > > mmap_write_unlock(current->mm); > return ret; > } if more prot flags are introduced then the exact match for old_flags may be restrictive and currently there is no way to query these flags to figure out how to toggle one prot flag in a future proof way, so i don't think this solves the issue completely. i think we might need a new api, given that aarch64 now has PROT_BTI and PROT_MTE while existing code expects RWX only, but i don't know what api is best. > libc would now be able to do > > mchangeprot(addr, len, PROT_EXEC | PROT_READ, > PROT_EXEC | PROT_READ | PROT_BTI); > > while systemd's MDWX filter would reject the call if > > (new_flags & PROT_EXEC) && > (!(old_flags & PROT_EXEC) || (new_flags & PROT_WRITE) > > > > This won't magically fix current code, but something along these lines > might be better going forward. > > > Thoughts? > > ---Dave _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-26 16:57 ` Szabolcs Nagy @ 2020-10-26 17:52 ` Dave Martin -1 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-26 17:52 UTC (permalink / raw) To: Szabolcs Nagy Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel On Mon, Oct 26, 2020 at 04:57:55PM +0000, Szabolcs Nagy via Libc-alpha wrote: > The 10/26/2020 16:24, Dave Martin via Libc-alpha wrote: > > Unrolling this discussion a bit, this problem comes from a few sources: > > > > 1) systemd is trying to implement a policy that doesn't fit SECCOMP > > syscall filtering very well. > > > > 2) The program is trying to do something not expressible through the > > syscall interface: really the intent is to set PROT_BTI on the page, > > with no intent to set PROT_EXEC on any page that didn't already have it > > set. > > > > > > This limitation of mprotect() was known when I originally added PROT_BTI, > > but at that time we weren't aware of a clear use case that would fail. > > > > > > Would it now help to add something like: > > > > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > > { > > int ret = -EINVAL; > > mmap_write_lock(current->mm); > > if (all vmas in [addr .. addr + len) have > > their mprotect flags set to old_flags) { > > > > ret = mprotect(addr, len, new_flags); > > } > > > > mmap_write_unlock(current->mm); > > return ret; > > } > > if more prot flags are introduced then the exact > match for old_flags may be restrictive and currently > there is no way to query these flags to figure out > how to toggle one prot flag in a future proof way, > so i don't think this solves the issue completely. Ack -- I illustrated this model because it makes the seccomp filter's job easy, but it does have limitations. > i think we might need a new api, given that aarch64 > now has PROT_BTI and PROT_MTE while existing code > expects RWX only, but i don't know what api is best. An alternative option would be a call that sets / clears chosen flags and leaves others unchanged. The trouble with that is that the MDWX policy then becomes hard to implement again. But policies might be best set via another route, such as a prctl, rather than being implemented completely in a seccomp filter. Cheers ---Dave ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 17:52 ` Dave Martin 0 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-26 17:52 UTC (permalink / raw) To: Szabolcs Nagy Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel On Mon, Oct 26, 2020 at 04:57:55PM +0000, Szabolcs Nagy via Libc-alpha wrote: > The 10/26/2020 16:24, Dave Martin via Libc-alpha wrote: > > Unrolling this discussion a bit, this problem comes from a few sources: > > > > 1) systemd is trying to implement a policy that doesn't fit SECCOMP > > syscall filtering very well. > > > > 2) The program is trying to do something not expressible through the > > syscall interface: really the intent is to set PROT_BTI on the page, > > with no intent to set PROT_EXEC on any page that didn't already have it > > set. > > > > > > This limitation of mprotect() was known when I originally added PROT_BTI, > > but at that time we weren't aware of a clear use case that would fail. > > > > > > Would it now help to add something like: > > > > int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > > { > > int ret = -EINVAL; > > mmap_write_lock(current->mm); > > if (all vmas in [addr .. addr + len) have > > their mprotect flags set to old_flags) { > > > > ret = mprotect(addr, len, new_flags); > > } > > > > mmap_write_unlock(current->mm); > > return ret; > > } > > if more prot flags are introduced then the exact > match for old_flags may be restrictive and currently > there is no way to query these flags to figure out > how to toggle one prot flag in a future proof way, > so i don't think this solves the issue completely. Ack -- I illustrated this model because it makes the seccomp filter's job easy, but it does have limitations. > i think we might need a new api, given that aarch64 > now has PROT_BTI and PROT_MTE while existing code > expects RWX only, but i don't know what api is best. An alternative option would be a call that sets / clears chosen flags and leaves others unchanged. The trouble with that is that the MDWX policy then becomes hard to implement again. But policies might be best set via another route, such as a prctl, rather than being implemented completely in a seccomp filter. Cheers ---Dave _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-26 17:52 ` Dave Martin @ 2020-10-26 22:39 ` Jeremy Linton -1 siblings, 0 replies; 88+ messages in thread From: Jeremy Linton @ 2020-10-26 22:39 UTC (permalink / raw) To: Dave Martin, Szabolcs Nagy Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel Hi, On 10/26/20 12:52 PM, Dave Martin wrote: > On Mon, Oct 26, 2020 at 04:57:55PM +0000, Szabolcs Nagy via Libc-alpha wrote: >> The 10/26/2020 16:24, Dave Martin via Libc-alpha wrote: >>> Unrolling this discussion a bit, this problem comes from a few sources: >>> >>> 1) systemd is trying to implement a policy that doesn't fit SECCOMP >>> syscall filtering very well. >>> >>> 2) The program is trying to do something not expressible through the >>> syscall interface: really the intent is to set PROT_BTI on the page, >>> with no intent to set PROT_EXEC on any page that didn't already have it >>> set. >>> >>> >>> This limitation of mprotect() was known when I originally added PROT_BTI, >>> but at that time we weren't aware of a clear use case that would fail. >>> >>> >>> Would it now help to add something like: >>> >>> int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) >>> { >>> int ret = -EINVAL; >>> mmap_write_lock(current->mm); >>> if (all vmas in [addr .. addr + len) have >>> their mprotect flags set to old_flags) { >>> >>> ret = mprotect(addr, len, new_flags); >>> } >>> >>> mmap_write_unlock(current->mm); >>> return ret; >>> } >> >> if more prot flags are introduced then the exact >> match for old_flags may be restrictive and currently >> there is no way to query these flags to figure out >> how to toggle one prot flag in a future proof way, >> so i don't think this solves the issue completely. > > Ack -- I illustrated this model because it makes the seccomp filter's > job easy, but it does have limitations. > >> i think we might need a new api, given that aarch64 >> now has PROT_BTI and PROT_MTE while existing code >> expects RWX only, but i don't know what api is best. > > An alternative option would be a call that sets / clears chosen > flags and leaves others unchanged. I tend to favor a set/clear API, but that could also just be done by creating a new PROT_BTI_IF_X which enables BTI for areas already set to _EXEC. That goes right by the seccomp filters too, and actually is closer to what glibc wants to do anyway. > > The trouble with that is that the MDWX policy then becomes hard to > implement again. > > > But policies might be best set via another route, such as a prctl, > rather than being implemented completely in a seccomp filter. > > Cheers > ---Dave > ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-26 22:39 ` Jeremy Linton 0 siblings, 0 replies; 88+ messages in thread From: Jeremy Linton @ 2020-10-26 22:39 UTC (permalink / raw) To: Dave Martin, Szabolcs Nagy Cc: Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel Hi, On 10/26/20 12:52 PM, Dave Martin wrote: > On Mon, Oct 26, 2020 at 04:57:55PM +0000, Szabolcs Nagy via Libc-alpha wrote: >> The 10/26/2020 16:24, Dave Martin via Libc-alpha wrote: >>> Unrolling this discussion a bit, this problem comes from a few sources: >>> >>> 1) systemd is trying to implement a policy that doesn't fit SECCOMP >>> syscall filtering very well. >>> >>> 2) The program is trying to do something not expressible through the >>> syscall interface: really the intent is to set PROT_BTI on the page, >>> with no intent to set PROT_EXEC on any page that didn't already have it >>> set. >>> >>> >>> This limitation of mprotect() was known when I originally added PROT_BTI, >>> but at that time we weren't aware of a clear use case that would fail. >>> >>> >>> Would it now help to add something like: >>> >>> int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) >>> { >>> int ret = -EINVAL; >>> mmap_write_lock(current->mm); >>> if (all vmas in [addr .. addr + len) have >>> their mprotect flags set to old_flags) { >>> >>> ret = mprotect(addr, len, new_flags); >>> } >>> >>> mmap_write_unlock(current->mm); >>> return ret; >>> } >> >> if more prot flags are introduced then the exact >> match for old_flags may be restrictive and currently >> there is no way to query these flags to figure out >> how to toggle one prot flag in a future proof way, >> so i don't think this solves the issue completely. > > Ack -- I illustrated this model because it makes the seccomp filter's > job easy, but it does have limitations. > >> i think we might need a new api, given that aarch64 >> now has PROT_BTI and PROT_MTE while existing code >> expects RWX only, but i don't know what api is best. > > An alternative option would be a call that sets / clears chosen > flags and leaves others unchanged. I tend to favor a set/clear API, but that could also just be done by creating a new PROT_BTI_IF_X which enables BTI for areas already set to _EXEC. That goes right by the seccomp filters too, and actually is closer to what glibc wants to do anyway. > > The trouble with that is that the MDWX policy then becomes hard to > implement again. > > > But policies might be best set via another route, such as a prctl, > rather than being implemented completely in a seccomp filter. > > Cheers > ---Dave > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-26 22:39 ` Jeremy Linton @ 2020-10-27 14:15 ` Dave Martin -1 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-27 14:15 UTC (permalink / raw) To: Jeremy Linton Cc: Szabolcs Nagy, Mark Rutland, systemd-devel, Kees Cook, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel On Mon, Oct 26, 2020 at 05:39:42PM -0500, Jeremy Linton via Libc-alpha wrote: > Hi, > > On 10/26/20 12:52 PM, Dave Martin wrote: > >On Mon, Oct 26, 2020 at 04:57:55PM +0000, Szabolcs Nagy via Libc-alpha wrote: > >>The 10/26/2020 16:24, Dave Martin via Libc-alpha wrote: > >>>Unrolling this discussion a bit, this problem comes from a few sources: > >>> > >>>1) systemd is trying to implement a policy that doesn't fit SECCOMP > >>>syscall filtering very well. > >>> > >>>2) The program is trying to do something not expressible through the > >>>syscall interface: really the intent is to set PROT_BTI on the page, > >>>with no intent to set PROT_EXEC on any page that didn't already have it > >>>set. > >>> > >>> > >>>This limitation of mprotect() was known when I originally added PROT_BTI, > >>>but at that time we weren't aware of a clear use case that would fail. > >>> > >>> > >>>Would it now help to add something like: > >>> > >>>int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > >>>{ > >>> int ret = -EINVAL; > >>> mmap_write_lock(current->mm); > >>> if (all vmas in [addr .. addr + len) have > >>> their mprotect flags set to old_flags) { > >>> > >>> ret = mprotect(addr, len, new_flags); > >>> } > >>> > >>> mmap_write_unlock(current->mm); > >>> return ret; > >>>} > >> > >>if more prot flags are introduced then the exact > >>match for old_flags may be restrictive and currently > >>there is no way to query these flags to figure out > >>how to toggle one prot flag in a future proof way, > >>so i don't think this solves the issue completely. > > > >Ack -- I illustrated this model because it makes the seccomp filter's > >job easy, but it does have limitations. > > > >>i think we might need a new api, given that aarch64 > >>now has PROT_BTI and PROT_MTE while existing code > >>expects RWX only, but i don't know what api is best. > > > >An alternative option would be a call that sets / clears chosen > >flags and leaves others unchanged. > > I tend to favor a set/clear API, but that could also just be done by > creating a new PROT_BTI_IF_X which enables BTI for areas already set to > _EXEC. That goes right by the seccomp filters too, and actually is closer to > what glibc wants to do anyway. That works, though I'm not so keen on teating PROT_BTI as a special case, since the problem is likely to recur when other weird per-arch flags get added... I also wonder whether we actually care whether the pages are marked executable or not here; probably the flags can just be independent. This rather depends on whether the how the architecture treats the BTI (a.k.a GP) pagetable bit for non-executable pages. I have a feeling we already allow PROT_BTI && !PROT_EXEC through anyway. What about a generic-ish set/clear interface that still works by just adding a couple of PROT_ flags: switch (flags & (PROT_SET | PROT_CLEAR)) { case PROT_SET: prot |= flags; break; case PROT_CLEAR: prot &= ~flags; break; case 0: prot = flags; break; default: return -EINVAL; } This can't atomically set some flags while clearing some others, but for simple stuff it seems sufficient and shouldn't be too invasive on the kernel side. We will still have to take the mm lock when doing a SET or CLEAR, but not for the non-set/clear case. Anyway, libc could now do: mprotect(addr, len, PROT_SET | PROT_BTI); with much the same effect as your PROT_BTI_IF_X. JITting or breakpoint setting code that wants to change the permissions temporarily, without needing to know whether PROT_BTI is set, say: mprotect(addr, len, PROT_SET | PROT_WRITE); *addr = BKPT_INSN; mprotect(addr, len, PROT_CLEAR | PROT_WRITE); Thoughts? I won't claim this doesn't still have some limitations... Cheers ---Dave ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-27 14:15 ` Dave Martin 0 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-10-27 14:15 UTC (permalink / raw) To: Jeremy Linton Cc: Mark Rutland, systemd-devel, Kees Cook, Szabolcs Nagy, Catalin Marinas, Will Deacon, linux-kernel, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel On Mon, Oct 26, 2020 at 05:39:42PM -0500, Jeremy Linton via Libc-alpha wrote: > Hi, > > On 10/26/20 12:52 PM, Dave Martin wrote: > >On Mon, Oct 26, 2020 at 04:57:55PM +0000, Szabolcs Nagy via Libc-alpha wrote: > >>The 10/26/2020 16:24, Dave Martin via Libc-alpha wrote: > >>>Unrolling this discussion a bit, this problem comes from a few sources: > >>> > >>>1) systemd is trying to implement a policy that doesn't fit SECCOMP > >>>syscall filtering very well. > >>> > >>>2) The program is trying to do something not expressible through the > >>>syscall interface: really the intent is to set PROT_BTI on the page, > >>>with no intent to set PROT_EXEC on any page that didn't already have it > >>>set. > >>> > >>> > >>>This limitation of mprotect() was known when I originally added PROT_BTI, > >>>but at that time we weren't aware of a clear use case that would fail. > >>> > >>> > >>>Would it now help to add something like: > >>> > >>>int mchangeprot(void *addr, size_t len, int old_flags, int new_flags) > >>>{ > >>> int ret = -EINVAL; > >>> mmap_write_lock(current->mm); > >>> if (all vmas in [addr .. addr + len) have > >>> their mprotect flags set to old_flags) { > >>> > >>> ret = mprotect(addr, len, new_flags); > >>> } > >>> > >>> mmap_write_unlock(current->mm); > >>> return ret; > >>>} > >> > >>if more prot flags are introduced then the exact > >>match for old_flags may be restrictive and currently > >>there is no way to query these flags to figure out > >>how to toggle one prot flag in a future proof way, > >>so i don't think this solves the issue completely. > > > >Ack -- I illustrated this model because it makes the seccomp filter's > >job easy, but it does have limitations. > > > >>i think we might need a new api, given that aarch64 > >>now has PROT_BTI and PROT_MTE while existing code > >>expects RWX only, but i don't know what api is best. > > > >An alternative option would be a call that sets / clears chosen > >flags and leaves others unchanged. > > I tend to favor a set/clear API, but that could also just be done by > creating a new PROT_BTI_IF_X which enables BTI for areas already set to > _EXEC. That goes right by the seccomp filters too, and actually is closer to > what glibc wants to do anyway. That works, though I'm not so keen on teating PROT_BTI as a special case, since the problem is likely to recur when other weird per-arch flags get added... I also wonder whether we actually care whether the pages are marked executable or not here; probably the flags can just be independent. This rather depends on whether the how the architecture treats the BTI (a.k.a GP) pagetable bit for non-executable pages. I have a feeling we already allow PROT_BTI && !PROT_EXEC through anyway. What about a generic-ish set/clear interface that still works by just adding a couple of PROT_ flags: switch (flags & (PROT_SET | PROT_CLEAR)) { case PROT_SET: prot |= flags; break; case PROT_CLEAR: prot &= ~flags; break; case 0: prot = flags; break; default: return -EINVAL; } This can't atomically set some flags while clearing some others, but for simple stuff it seems sufficient and shouldn't be too invasive on the kernel side. We will still have to take the mm lock when doing a SET or CLEAR, but not for the non-set/clear case. Anyway, libc could now do: mprotect(addr, len, PROT_SET | PROT_BTI); with much the same effect as your PROT_BTI_IF_X. JITting or breakpoint setting code that wants to change the permissions temporarily, without needing to know whether PROT_BTI is set, say: mprotect(addr, len, PROT_SET | PROT_WRITE); *addr = BKPT_INSN; mprotect(addr, len, PROT_CLEAR | PROT_WRITE); Thoughts? I won't claim this doesn't still have some limitations... Cheers ---Dave _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-27 14:15 ` Dave Martin @ 2020-10-29 11:02 ` Catalin Marinas -1 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-29 11:02 UTC (permalink / raw) To: Dave Martin Cc: Jeremy Linton, Szabolcs Nagy, Mark Rutland, systemd-devel, Kees Cook, Will Deacon, linux-kernel, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel On Tue, Oct 27, 2020 at 02:15:22PM +0000, Dave P Martin wrote: > I also wonder whether we actually care whether the pages are marked > executable or not here; probably the flags can just be independent. This > rather depends on whether the how the architecture treats the BTI (a.k.a > GP) pagetable bit for non-executable pages. I have a feeling we already > allow PROT_BTI && !PROT_EXEC through anyway. > > > What about a generic-ish set/clear interface that still works by just > adding a couple of PROT_ flags: > > switch (flags & (PROT_SET | PROT_CLEAR)) { > case PROT_SET: prot |= flags; break; > case PROT_CLEAR: prot &= ~flags; break; > case 0: prot = flags; break; > > default: > return -EINVAL; > } > > This can't atomically set some flags while clearing some others, but for > simple stuff it seems sufficient and shouldn't be too invasive on the > kernel side. > > We will still have to take the mm lock when doing a SET or CLEAR, but > not for the non-set/clear case. > > > Anyway, libc could now do: > > mprotect(addr, len, PROT_SET | PROT_BTI); > > with much the same effect as your PROT_BTI_IF_X. > > > JITting or breakpoint setting code that wants to change the permissions > temporarily, without needing to know whether PROT_BTI is set, say: > > mprotect(addr, len, PROT_SET | PROT_WRITE); > *addr = BKPT_INSN; > mprotect(addr, len, PROT_CLEAR | PROT_WRITE); The problem with this approach is that you can't catch PROT_EXEC|PROT_WRITE mappings via seccomp. So you'd have to limit it to some harmless PROT_ flags only. I don't like this limitation, nor the PROT_BTI_IF_X approach. The only generic solutions I see are to either use a stateful filter in systemd or pass the old state to the kernel in a cmpxchg style so that seccomp can check it (I think you suggest this at some point). The latter requires a new syscall which is not something we can address as a quick, back-portable fix here. If systemd cannot be changed to use a stateful filter for w^x detection, my suggestion is to go for the kernel setting PROT_BTI on the main executable with glibc changed to tolerate EPERM on mprotect(). I don't mind adding an AT_FLAGS bit if needed but I don't think it buys us much. Once the current problem is fixed, we can look at a better solution longer term as a new syscall. -- Catalin ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-10-29 11:02 ` Catalin Marinas 0 siblings, 0 replies; 88+ messages in thread From: Catalin Marinas @ 2020-10-29 11:02 UTC (permalink / raw) To: Dave Martin Cc: Mark Rutland, systemd-devel, Kees Cook, Szabolcs Nagy, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel On Tue, Oct 27, 2020 at 02:15:22PM +0000, Dave P Martin wrote: > I also wonder whether we actually care whether the pages are marked > executable or not here; probably the flags can just be independent. This > rather depends on whether the how the architecture treats the BTI (a.k.a > GP) pagetable bit for non-executable pages. I have a feeling we already > allow PROT_BTI && !PROT_EXEC through anyway. > > > What about a generic-ish set/clear interface that still works by just > adding a couple of PROT_ flags: > > switch (flags & (PROT_SET | PROT_CLEAR)) { > case PROT_SET: prot |= flags; break; > case PROT_CLEAR: prot &= ~flags; break; > case 0: prot = flags; break; > > default: > return -EINVAL; > } > > This can't atomically set some flags while clearing some others, but for > simple stuff it seems sufficient and shouldn't be too invasive on the > kernel side. > > We will still have to take the mm lock when doing a SET or CLEAR, but > not for the non-set/clear case. > > > Anyway, libc could now do: > > mprotect(addr, len, PROT_SET | PROT_BTI); > > with much the same effect as your PROT_BTI_IF_X. > > > JITting or breakpoint setting code that wants to change the permissions > temporarily, without needing to know whether PROT_BTI is set, say: > > mprotect(addr, len, PROT_SET | PROT_WRITE); > *addr = BKPT_INSN; > mprotect(addr, len, PROT_CLEAR | PROT_WRITE); The problem with this approach is that you can't catch PROT_EXEC|PROT_WRITE mappings via seccomp. So you'd have to limit it to some harmless PROT_ flags only. I don't like this limitation, nor the PROT_BTI_IF_X approach. The only generic solutions I see are to either use a stateful filter in systemd or pass the old state to the kernel in a cmpxchg style so that seccomp can check it (I think you suggest this at some point). The latter requires a new syscall which is not something we can address as a quick, back-portable fix here. If systemd cannot be changed to use a stateful filter for w^x detection, my suggestion is to go for the kernel setting PROT_BTI on the main executable with glibc changed to tolerate EPERM on mprotect(). I don't mind adding an AT_FLAGS bit if needed but I don't think it buys us much. Once the current problem is fixed, we can look at a better solution longer term as a new syscall. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures 2020-10-29 11:02 ` Catalin Marinas @ 2020-11-04 12:18 ` Dave Martin -1 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-11-04 12:18 UTC (permalink / raw) To: Catalin Marinas Cc: Mark Rutland, systemd-devel, Kees Cook, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel On Thu, Oct 29, 2020 at 11:02:22AM +0000, Catalin Marinas via Libc-alpha wrote: > On Tue, Oct 27, 2020 at 02:15:22PM +0000, Dave P Martin wrote: > > I also wonder whether we actually care whether the pages are marked > > executable or not here; probably the flags can just be independent. This > > rather depends on whether the how the architecture treats the BTI (a.k.a > > GP) pagetable bit for non-executable pages. I have a feeling we already > > allow PROT_BTI && !PROT_EXEC through anyway. > > > > > > What about a generic-ish set/clear interface that still works by just > > adding a couple of PROT_ flags: > > > > switch (flags & (PROT_SET | PROT_CLEAR)) { > > case PROT_SET: prot |= flags; break; > > case PROT_CLEAR: prot &= ~flags; break; > > case 0: prot = flags; break; > > > > default: > > return -EINVAL; > > } > > > > This can't atomically set some flags while clearing some others, but for > > simple stuff it seems sufficient and shouldn't be too invasive on the > > kernel side. > > > > We will still have to take the mm lock when doing a SET or CLEAR, but > > not for the non-set/clear case. > > > > > > Anyway, libc could now do: > > > > mprotect(addr, len, PROT_SET | PROT_BTI); > > > > with much the same effect as your PROT_BTI_IF_X. > > > > > > JITting or breakpoint setting code that wants to change the permissions > > temporarily, without needing to know whether PROT_BTI is set, say: > > > > mprotect(addr, len, PROT_SET | PROT_WRITE); > > *addr = BKPT_INSN; > > mprotect(addr, len, PROT_CLEAR | PROT_WRITE); > > The problem with this approach is that you can't catch > PROT_EXEC|PROT_WRITE mappings via seccomp. So you'd have to limit it to > some harmless PROT_ flags only. I don't like this limitation, nor the > PROT_BTI_IF_X approach. Ack; this is just one flavour of interface, and every approach seems to have some shortcomings. > The only generic solutions I see are to either use a stateful filter in > systemd or pass the old state to the kernel in a cmpxchg style so that > seccomp can check it (I think you suggest this at some point). The "cmpxchg" option has the disadvantage that the caller needs to know the original permissions. It seems that glibc is prepared to work around this, but it won't always be feasible in ancillary / instrumentation code or libraries. IMHO it would be preferable to apply a policy to mmap/mprotect in the kernel proper rather then BPF being the only way to do it -- in any case, the required checks seem to be out of the scope of what can be done efficiently (or perhaps at all) in a syscall filter. > The latter requires a new syscall which is not something we can address > as a quick, back-portable fix here. If systemd cannot be changed to use > a stateful filter for w^x detection, my suggestion is to go for the > kernel setting PROT_BTI on the main executable with glibc changed to > tolerate EPERM on mprotect(). I don't mind adding an AT_FLAGS bit if > needed but I don't think it buys us much. I agree, this seems the best short-term approach. > Once the current problem is fixed, we can look at a better solution > longer term as a new syscall. Agreed, I think if we try to rush the addition of new syscalls, the chance of coming up with a bad design is high... Cheers ---Dave ^ permalink raw reply [flat|nested] 88+ messages in thread
* Re: BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures @ 2020-11-04 12:18 ` Dave Martin 0 siblings, 0 replies; 88+ messages in thread From: Dave Martin @ 2020-11-04 12:18 UTC (permalink / raw) To: Catalin Marinas Cc: Mark Rutland, systemd-devel, Kees Cook, Will Deacon, linux-kernel, Jeremy Linton, Mark Brown, toiwoton, libc-alpha, linux-arm-kernel On Thu, Oct 29, 2020 at 11:02:22AM +0000, Catalin Marinas via Libc-alpha wrote: > On Tue, Oct 27, 2020 at 02:15:22PM +0000, Dave P Martin wrote: > > I also wonder whether we actually care whether the pages are marked > > executable or not here; probably the flags can just be independent. This > > rather depends on whether the how the architecture treats the BTI (a.k.a > > GP) pagetable bit for non-executable pages. I have a feeling we already > > allow PROT_BTI && !PROT_EXEC through anyway. > > > > > > What about a generic-ish set/clear interface that still works by just > > adding a couple of PROT_ flags: > > > > switch (flags & (PROT_SET | PROT_CLEAR)) { > > case PROT_SET: prot |= flags; break; > > case PROT_CLEAR: prot &= ~flags; break; > > case 0: prot = flags; break; > > > > default: > > return -EINVAL; > > } > > > > This can't atomically set some flags while clearing some others, but for > > simple stuff it seems sufficient and shouldn't be too invasive on the > > kernel side. > > > > We will still have to take the mm lock when doing a SET or CLEAR, but > > not for the non-set/clear case. > > > > > > Anyway, libc could now do: > > > > mprotect(addr, len, PROT_SET | PROT_BTI); > > > > with much the same effect as your PROT_BTI_IF_X. > > > > > > JITting or breakpoint setting code that wants to change the permissions > > temporarily, without needing to know whether PROT_BTI is set, say: > > > > mprotect(addr, len, PROT_SET | PROT_WRITE); > > *addr = BKPT_INSN; > > mprotect(addr, len, PROT_CLEAR | PROT_WRITE); > > The problem with this approach is that you can't catch > PROT_EXEC|PROT_WRITE mappings via seccomp. So you'd have to limit it to > some harmless PROT_ flags only. I don't like this limitation, nor the > PROT_BTI_IF_X approach. Ack; this is just one flavour of interface, and every approach seems to have some shortcomings. > The only generic solutions I see are to either use a stateful filter in > systemd or pass the old state to the kernel in a cmpxchg style so that > seccomp can check it (I think you suggest this at some point). The "cmpxchg" option has the disadvantage that the caller needs to know the original permissions. It seems that glibc is prepared to work around this, but it won't always be feasible in ancillary / instrumentation code or libraries. IMHO it would be preferable to apply a policy to mmap/mprotect in the kernel proper rather then BPF being the only way to do it -- in any case, the required checks seem to be out of the scope of what can be done efficiently (or perhaps at all) in a syscall filter. > The latter requires a new syscall which is not something we can address > as a quick, back-portable fix here. If systemd cannot be changed to use > a stateful filter for w^x detection, my suggestion is to go for the > kernel setting PROT_BTI on the main executable with glibc changed to > tolerate EPERM on mprotect(). I don't mind adding an AT_FLAGS bit if > needed but I don't think it buys us much. I agree, this seems the best short-term approach. > Once the current problem is fixed, we can look at a better solution > longer term as a new syscall. Agreed, I think if we try to rush the addition of new syscalls, the chance of coming up with a bad design is high... Cheers ---Dave _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 88+ messages in thread
end of thread, other threads:[~2020-11-04 12:19 UTC | newest]
Thread overview: 88+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <8584c14f-5c28-9d70-c054-7c78127d84ea@arm.com>
2020-10-22 7:18 ` [systemd-devel] BTI interaction between seccomp filters in systemd and glibc mprotect calls, causing service failures Lennart Poettering
2020-10-22 7:18 ` Lennart Poettering
2020-10-22 7:54 ` Florian Weimer
2020-10-22 7:54 ` Florian Weimer
2020-10-22 8:17 ` Topi Miettinen
2020-10-22 8:17 ` Topi Miettinen
2020-10-22 8:25 ` Florian Weimer
2020-10-22 8:25 ` Florian Weimer
2020-10-22 8:29 ` Szabolcs Nagy
2020-10-22 8:29 ` Szabolcs Nagy
2020-10-22 8:38 ` Lennart Poettering
2020-10-22 8:38 ` Lennart Poettering
2020-10-22 9:31 ` Catalin Marinas
2020-10-22 9:31 ` Catalin Marinas
2020-10-22 10:12 ` Topi Miettinen
2020-10-22 10:12 ` Topi Miettinen
2020-10-22 10:27 ` Florian Weimer
2020-10-22 10:27 ` Florian Weimer
2020-10-23 6:13 ` Szabolcs Nagy
2020-10-23 6:13 ` Szabolcs Nagy
2020-10-23 9:04 ` Catalin Marinas
2020-10-23 9:04 ` Catalin Marinas
2020-10-22 10:03 ` Topi Miettinen
2020-10-22 10:03 ` Topi Miettinen
2020-10-22 8:05 ` Szabolcs Nagy
2020-10-22 8:05 ` Szabolcs Nagy
2020-10-22 8:31 ` Lennart Poettering
2020-10-22 8:31 ` Lennart Poettering
[not found] ` <20201022075447.GO3819@arm.com>
2020-10-22 10:39 ` Topi Miettinen
2020-10-22 10:39 ` Topi Miettinen
2020-10-22 20:02 ` Kees Cook
2020-10-22 20:02 ` Kees Cook
2020-10-22 20:02 ` Kees Cook
2020-10-22 22:24 ` Topi Miettinen
2020-10-22 22:24 ` Topi Miettinen
2020-10-22 22:24 ` Topi Miettinen
2020-10-23 17:52 ` Salvatore Mesoraca
2020-10-23 17:52 ` Salvatore Mesoraca
2020-10-23 17:52 ` Salvatore Mesoraca
2020-10-24 11:34 ` Topi Miettinen
2020-10-24 11:34 ` Topi Miettinen
2020-10-24 11:34 ` Topi Miettinen
2020-10-24 14:12 ` Salvatore Mesoraca
2020-10-24 14:12 ` Salvatore Mesoraca
2020-10-24 14:12 ` Salvatore Mesoraca
2020-10-25 13:42 ` Jordan Glover
2020-10-25 13:42 ` Jordan Glover
2020-10-25 13:42 ` Jordan Glover
2020-10-23 9:02 ` Catalin Marinas
2020-10-23 9:02 ` Catalin Marinas
2020-10-23 9:02 ` Catalin Marinas
2020-10-24 11:01 ` Topi Miettinen
2020-10-24 11:01 ` Topi Miettinen
2020-10-24 11:01 ` Topi Miettinen
2020-10-26 14:52 ` Catalin Marinas
2020-10-26 14:52 ` Catalin Marinas
2020-10-26 14:52 ` Catalin Marinas
2020-10-26 15:56 ` Dave Martin
2020-10-26 15:56 ` Dave Martin
2020-10-26 15:56 ` Dave Martin
2020-10-26 16:51 ` Mark Brown
2020-10-26 16:51 ` Mark Brown
2020-10-26 16:51 ` Mark Brown
2020-10-26 16:31 ` Topi Miettinen
2020-10-26 16:31 ` Topi Miettinen
2020-10-26 16:31 ` Topi Miettinen
2020-10-26 16:24 ` Dave Martin
2020-10-26 16:24 ` Dave Martin
2020-10-26 16:39 ` Topi Miettinen
2020-10-26 16:39 ` Topi Miettinen
2020-10-26 16:45 ` Florian Weimer
2020-10-26 16:45 ` Florian Weimer
2020-10-27 14:22 ` Dave Martin
2020-10-27 14:22 ` Dave Martin
2020-10-27 14:41 ` Florian Weimer
2020-10-27 14:41 ` Florian Weimer
2020-10-26 16:57 ` Szabolcs Nagy
2020-10-26 16:57 ` Szabolcs Nagy
2020-10-26 17:52 ` Dave Martin
2020-10-26 17:52 ` Dave Martin
2020-10-26 22:39 ` Jeremy Linton
2020-10-26 22:39 ` Jeremy Linton
2020-10-27 14:15 ` Dave Martin
2020-10-27 14:15 ` Dave Martin
2020-10-29 11:02 ` Catalin Marinas
2020-10-29 11:02 ` Catalin Marinas
2020-11-04 12:18 ` Dave Martin
2020-11-04 12:18 ` Dave Martin
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.