* Re: [PATCHv2 1/4] perf/core: Add support to exclude kernel mode PMU tracing [not found] ` <def1a6b37cbb54cb15329765266ed90c2f7aa24e.1614624041.git.saiprakash.ranjan@codeaurora.org> @ 2021-03-01 22:42 ` Doug Anderson 0 siblings, 0 replies; 8+ messages in thread From: Doug Anderson @ 2021-03-01 22:42 UTC (permalink / raw) To: Sai Prakash Ranjan Cc: Mark Rutland, Al Grant, Denis Nikitin, Mathieu Poirier, Suzuki K Poulose, Peter Zijlstra, linux-arm-msm, coresight, Mattias Nissler, LKML, Arnaldo Carvalho de Melo, Stephen Boyd, Alexander Shishkin, Ingo Molnar, Leo Yan, Namhyung Kim, Jiri Olsa, Linux ARM, Mike Leach Hi, On Mon, Mar 1, 2021 at 11:05 AM Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> wrote: > > Hardware assisted tracing families such as ARM Coresight, Intel PT > provides rich tracing capabilities including instruction level > tracing and accurate timestamps which are very useful for profiling > and also pose a significant security risk. One such example of > security risk is when kernel mode tracing is not excluded and these > hardware assisted tracing can be used to analyze cryptographic code > execution. In this case, even the root user must not be able to infer > anything. > > To explain it more clearly in the words of a security team member > (credits: Mattias Nissler), > > "Consider a system where disk contents are encrypted and the encryption > key is set up by the user when mounting the file system. From that point > on the encryption key resides in the kernel. It seems reasonable to > expect that the disk encryption key be protected from exfiltration even > if the system later suffers a root compromise (or even against insiders > that have root access), at least as long as the attacker doesn't > manage to compromise the kernel." > > Here the idea is to protect such important information from all users > including root users since root privileges does not have to mean full > control over the kernel [1] and root compromise does not have to be > the end of the world. > > But "Peter said even the regular counters can be used for full branch > trace, the information isn't as accurate as PT and friends and not easier > but is good enough to infer plenty". This would mean that a global tunable > config for all kernel mode pmu tracing is more appropriate than the one > targeting the hardware assisted instruction tracing. > > Currently we can exclude kernel mode tracing via perf_event_paranoid > sysctl but it has following limitations, > > * No option to restrict kernel mode instruction tracing by the > root user. > * Not possible to restrict kernel mode instruction tracing when the > hardware assisted tracing IPs like ARM Coresight ETMs use an > additional interface via sysfs for tracing in addition to perf > interface. > > So introduce a new config CONFIG_EXCLUDE_KERNEL_PMU_TRACE to exclude > kernel mode pmu tracing which will be generic and applicable to all > hardware tracing families and which can also be used with other > interfaces like sysfs in case of ETMs. > > [1] https://lwn.net/Articles/796866/ > > Suggested-by: Suzuki K Poulose <suzuki.poulose@arm.com> > Suggested-by: Al Grant <al.grant@arm.com> > Tested-by: Denis Nikitin <denik@chromium.org> > Link: https://lore.kernel.org/lkml/20201015124522.1876-1-saiprakash.ranjan@codeaurora.org/ > Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> > --- > init/Kconfig | 11 +++++++++++ > kernel/events/core.c | 3 +++ > 2 files changed, 14 insertions(+) I'm not really knowledgeable at all about the perf subsystem so my review doesn't hold a lot of weight. However, Sai's patch seems sane to me. Reviewed-by: Douglas Anderson <dianders@chromium.org> _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <523a5af43615b804aa1211a3f27e06226d7159bc.1614624041.git.saiprakash.ranjan@codeaurora.org>]
* Re: [PATCHv2 2/4] perf evsel: Print warning for excluding kernel mode instruction tracing [not found] ` <523a5af43615b804aa1211a3f27e06226d7159bc.1614624041.git.saiprakash.ranjan@codeaurora.org> @ 2021-03-01 22:43 ` Doug Anderson 0 siblings, 0 replies; 8+ messages in thread From: Doug Anderson @ 2021-03-01 22:43 UTC (permalink / raw) To: Sai Prakash Ranjan Cc: Mark Rutland, Al Grant, Denis Nikitin, Mathieu Poirier, Suzuki K Poulose, Peter Zijlstra, linux-arm-msm, coresight, Mattias Nissler, LKML, Arnaldo Carvalho de Melo, Stephen Boyd, Alexander Shishkin, Ingo Molnar, Leo Yan, Namhyung Kim, Jiri Olsa, Linux ARM, Mike Leach Hi, On Mon, Mar 1, 2021 at 11:05 AM Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> wrote: > > Add a warning message to check CONFIG_EXCLUDE_KERNEL_HW_ITRACE kernel > config which excludes kernel mode instruction tracing to help perf tool > users identify the perf event open failure when they attempt kernel mode > tracing with this config enabled. > > Tested-by: Denis Nikitin <denik@chromium.org> > Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> > --- > tools/perf/util/evsel.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) I'm not really knowledgeable at all about the perf subsystem so my review doesn't hold a lot of weight. However, Sai's patch seems sane to me. Reviewed-by: Douglas Anderson <dianders@chromium.org> _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <defd34e2a8744fc93404abdaad2d429fa3f69850.1614624041.git.saiprakash.ranjan@codeaurora.org>]
* Re: [PATCHv2 3/4] coresight: etm4x: Add support to exclude kernel mode tracing [not found] ` <defd34e2a8744fc93404abdaad2d429fa3f69850.1614624041.git.saiprakash.ranjan@codeaurora.org> @ 2021-03-01 22:43 ` Doug Anderson 0 siblings, 0 replies; 8+ messages in thread From: Doug Anderson @ 2021-03-01 22:43 UTC (permalink / raw) To: Sai Prakash Ranjan Cc: Mark Rutland, Al Grant, Denis Nikitin, Mathieu Poirier, Suzuki K Poulose, Peter Zijlstra, linux-arm-msm, coresight, Mattias Nissler, LKML, Arnaldo Carvalho de Melo, Stephen Boyd, Alexander Shishkin, Ingo Molnar, Leo Yan, Namhyung Kim, Jiri Olsa, Linux ARM, Mike Leach Hi, On Mon, Mar 1, 2021 at 11:05 AM Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> wrote: > > On production systems with ETMs enabled, it is preferred to exclude > kernel mode(NS EL1) tracing for security concerns and support only > userspace(NS EL0) tracing. Perf subsystem interface uses the newly > introduced kernel config CONFIG_EXCLUDE_KERNEL_PMU_TRACE to exclude > kernel mode tracing, but there is an additional interface via sysfs > for ETMs which also needs to be handled to exclude kernel > mode tracing. So we use this same generic kernel config to handle > the sysfs mode of tracing. This config is disabled by default and > would not affect the current configuration which has both kernel and > userspace tracing enabled by default. > > Tested-by: Denis Nikitin <denik@chromium.org> > Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> > --- > drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++- > drivers/hwtracing/coresight/coresight-etm4x-sysfs.c | 6 ++++++ > 2 files changed, 11 insertions(+), 1 deletion(-) Not that I'm an expert in the perf subsystem, but the concern I had with v1 is now addressed. FWIW this seems fine to me now. Reviewed-by: Douglas Anderson <dianders@chromium.org> > --- a/drivers/hwtracing/coresight/coresight-etm4x-sysfs.c > +++ b/drivers/hwtracing/coresight/coresight-etm4x-sysfs.c > @@ -296,6 +296,12 @@ static ssize_t mode_store(struct device *dev, > if (kstrtoul(buf, 16, &val)) > return -EINVAL; > > + if (IS_ENABLED(CONFIG_EXCLUDE_KERNEL_PMU_TRACE) && (!(val & ETM_MODE_EXCL_KERN))) { > + dev_warn(dev, > + "Kernel mode tracing is not allowed, check your kernel config\n"); slight nit that I think your string needs to be indented by 1 space. ;-) _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <0b165b2608eed1a61240539f4b812bd80b6ad1bd.1614624041.git.saiprakash.ranjan@codeaurora.org>]
* Re: [PATCHv2 4/4] coresight: etm3x: Add support to exclude kernel mode tracing [not found] ` <0b165b2608eed1a61240539f4b812bd80b6ad1bd.1614624041.git.saiprakash.ranjan@codeaurora.org> @ 2021-03-01 22:43 ` Doug Anderson 0 siblings, 0 replies; 8+ messages in thread From: Doug Anderson @ 2021-03-01 22:43 UTC (permalink / raw) To: Sai Prakash Ranjan Cc: Mark Rutland, Al Grant, Denis Nikitin, Mathieu Poirier, Suzuki K Poulose, Peter Zijlstra, linux-arm-msm, coresight, Mattias Nissler, LKML, Arnaldo Carvalho de Melo, Stephen Boyd, Alexander Shishkin, Ingo Molnar, Leo Yan, Namhyung Kim, Jiri Olsa, Linux ARM, Mike Leach Hi, On Mon, Mar 1, 2021 at 11:05 AM Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> wrote: > > On production systems with ETMs enabled, it is preferred to exclude > kernel mode(NS EL1) tracing for security concerns and support only > userspace(NS EL0) tracing. Perf subsystem interface uses the newly > introduced kernel config CONFIG_EXCLUDE_KERNEL_PMU_TRACE to exclude > kernel mode tracing, but there is an additional interface > via sysfs for ETMs which also needs to be handled to exclude kernel > mode tracing. So we use this same generic kernel config to handle > the sysfs mode of tracing. This config is disabled by default and > would not affect the current configuration which has both kernel and > userspace tracing enabled by default. > > Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> > --- > drivers/hwtracing/coresight/coresight-etm3x-core.c | 3 +++ > drivers/hwtracing/coresight/coresight-etm3x-sysfs.c | 6 ++++++ > 2 files changed, 9 insertions(+) Reviewed-by: Douglas Anderson <dianders@chromium.org> > diff --git a/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c b/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c > index e8c7649f123e..f522fc2e01b3 100644 > --- a/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c > +++ b/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c > @@ -116,6 +116,12 @@ static ssize_t mode_store(struct device *dev, > if (ret) > return ret; > > + if (IS_ENABLED(CONFIG_EXCLUDE_KERNEL_PMU_TRACE) && (!(val & ETM_MODE_EXCL_KERN))) { > + dev_warn(dev, > + "Kernel mode tracing is not allowed, check your kernel config\n"); Same nit as in patch #3 that the above string should be indented by 1 more space. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCHv2 0/4] perf/core: Add support to exclude kernel mode PMU tracing [not found] <cover.1614624041.git.saiprakash.ranjan@codeaurora.org> ` (3 preceding siblings ...) [not found] ` <0b165b2608eed1a61240539f4b812bd80b6ad1bd.1614624041.git.saiprakash.ranjan@codeaurora.org> @ 2021-03-04 19:59 ` Andi Kleen 2021-03-04 20:17 ` Andi Kleen 4 siblings, 1 reply; 8+ messages in thread From: Andi Kleen @ 2021-03-04 19:59 UTC (permalink / raw) To: Sai Prakash Ranjan Cc: Mathieu Poirier, Suzuki K Poulose, Mike Leach, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Leo Yan, Jiri Olsa, Namhyung Kim, coresight, Stephen Boyd, Denis Nikitin, Mattias Nissler, Al Grant, linux-arm-msm, linux-kernel, linux-arm-kernel, Douglas Anderson Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> writes: > > "Consider a system where disk contents are encrypted and the encryption > key is set up by the user when mounting the file system. From that point > on the encryption key resides in the kernel. It seems reasonable to > expect that the disk encryption key be protected from exfiltration even > if the system later suffers a root compromise (or even against insiders > that have root access), at least as long as the attacker doesn't > manage to compromise the kernel." Normally disk encryption is in specialized work queues. It's total overkill to restrict all of the kernel if you just want to restrict those work queues. I would suggest some more analysis where secrets are actually stored and handled first. -Andi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCHv2 0/4] perf/core: Add support to exclude kernel mode PMU tracing 2021-03-04 19:59 ` [PATCHv2 0/4] perf/core: Add support to exclude kernel mode PMU tracing Andi Kleen @ 2021-03-04 20:17 ` Andi Kleen [not found] ` <20210309063828.26392-1-saiprakash.ranjan@codeaurora.org> 0 siblings, 1 reply; 8+ messages in thread From: Andi Kleen @ 2021-03-04 20:17 UTC (permalink / raw) To: Sai Prakash Ranjan Cc: Mathieu Poirier, Suzuki K Poulose, Mike Leach, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Leo Yan, Jiri Olsa, Namhyung Kim, coresight, Stephen Boyd, Denis Nikitin, Mattias Nissler, Al Grant, linux-arm-msm, linux-kernel, linux-arm-kernel, Douglas Anderson Andi Kleen <ak@linux.intel.com> writes: > > Normally disk encryption is in specialized work queues. It's total > overkill to restrict all of the kernel if you just want to restrict > those work queues. > > I would suggest some more analysis where secrets are actually stored > and handled first. Also thinking about this more: You really only want to limit data tracing here. If tracing branches could reveal secrets the crypto code would be already insecure due to timing side channels. If that's the case it would already require fixing the crypto code. -Andi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <20210309063828.26392-1-saiprakash.ranjan@codeaurora.org>]
* Re: [PATCHv2 0/4] perf/core: Add support to exclude kernel mode PMU tracing [not found] ` <20210309063828.26392-1-saiprakash.ranjan@codeaurora.org> @ 2021-03-09 14:44 ` Andi Kleen [not found] ` <24e0d604750babd3461768897bb2ae82@codeaurora.org> 0 siblings, 1 reply; 8+ messages in thread From: Andi Kleen @ 2021-03-09 14:44 UTC (permalink / raw) To: Sai Prakash Ranjan Cc: acme, al.grant, alexander.shishkin, coresight, denik, dianders, jolsa, leo.yan, linux-arm-kernel, linux-arm-msm, linux-kernel, mark.rutland, mathieu.poirier, mike.leach, mingo, mnissler, namhyung, peterz, suzuki.poulose, swboyd > The disk encryption is just one example and there might be others which > we might not be aware of yet and we are not suspecting there is something > wrong with the crypto code that needs to be fixed. Then you don't have any leaks relating to branch tracing. > restrict an external(in the sense that its not related to crypto or any > other security related component) entity such as hardware assisted tracing > like ARM coresight and so on. I don't see why or how the crypto code needs > to be fixed for something that is not related to it although it is affected. It's just a general property that if some code that is handling secrets is data dependent it already leaks. > The analogy would be like of the victims and a perpetrator. Lets take coresight > as an example for perpetrator and crypto as the victim here. Now we can try There's no victim with branch tracing, unless it is already leaky. > If we just know one victim (lets say crypto code here), what happens to the > others which we haven't identified yet? Do we just wait for someone to write > an exploit based on this and then scramble to fix it? For a useful security mitigation you need a threat model first I would say. So you need to have at least some idea how an attack with branch tracing would work. > Initial change was to restrict this only to HW assisted instruction tracing [1] I don't think it's needed for instruction tracing. -Andi _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <24e0d604750babd3461768897bb2ae82@codeaurora.org>]
* Re: [PATCHv2 0/4] perf/core: Add support to exclude kernel mode PMU tracing [not found] ` <24e0d604750babd3461768897bb2ae82@codeaurora.org> @ 2021-06-10 13:28 ` Mattias Nissler 0 siblings, 0 replies; 8+ messages in thread From: Mattias Nissler @ 2021-06-10 13:28 UTC (permalink / raw) To: Sai Prakash Ranjan Cc: Andi Kleen, acme, Al Grant, alexander.shishkin, coresight, Denis Nikitin, Douglas Anderson, jolsa, Leo Yan, linux-arm-kernel, linux-arm-msm, linux-kernel, mark.rutland, Mathieu Poirier, Mike Leach, mingo, namhyung, Peter Zijlstra, Suzuki K Poulose, Stephen Boyd I know this reply is hopelessly late, but I still want to clarify a few things (see inline for that) and provide some background on the thinking that led to this proposal and where we ultimately landed for the benefit of folks that come across this thread in the future. The team working on tracing had asked me to provide input on the security angle. We generally follow principle of least privilege / attack surface minimization principles, and so the conclusion "kernel tracing not needed" -> let's turn it off so we have one less thing to potentially worry about was an easy one. What I hadn't been aware of is that (a) no such config option exists in the kernel and (b) the approach that the kernel is taking is to limit access to tracing data to privileged users (via perf_event_paranoid sysctl, CAP_PERFMON). Same for leaking kernel code pointers (note that kernel tracing trivially defeats KASLR), privileged userspace can already read /proc/kallsysms. While I would still like the kernel to give us a build time option to disable kernel tracing, we can work with the current state of affairs. On Wed, Mar 10, 2021 at 4:17 PM Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> wrote: > > Hi Andi, > > On 2021-03-09 20:14, Andi Kleen wrote: > >> The disk encryption is just one example The intention behind that example was to illustrate the point that there are some data items that the kernel should hide to all of userspace, including privileged processes. I know this has been controversial in the past, but I hope we all agree that the kernel (if configured appropriately) is aiming to do so. > > and there might be others > >> which > >> we might not be aware of yet and we are not suspecting there is > >> something > >> wrong with the crypto code that needs to be fixed. > > > > Then you don't have any leaks relating to branch tracing. > > > > >> restrict an external(in the sense that its not related to crypto or > >> any > >> other security related component) entity such as hardware assisted > >> tracing > >> like ARM coresight and so on. I don't see why or how the crypto code > >> needs > >> to be fixed for something that is not related to it although it is > >> affected. > > > > It's just a general property that if some code that is handling secrets > > is data dependent it already leaks. Timing side channels have been a constant source of grief in crypto implementations for decades now. Things have become better in the most popular implementations, but bugs continue to be found. I happened to chat about this topic with David Benjamin (boringssl maintainer) recently, and his take was that timing side channels are a well understood problem meanwhile, but in practice few implementations get the details right. And the thing with high-resolution timestamps in traces is that it gives you a tool to observe timing differences closer to the source (so less noise) than if you just measure syscall latency from userspace. So, in theory you are right - data-dependent branches should not exist in crypto code, and if they do we should just fix them, since they're potentially exploitable already. In practice, timestamp information in tracing data can act as a magnifying glass that may well make a difference between whether a timing side channel is problematic in practice or not. > > > > > >> The analogy would be like of the victims and a perpetrator. Lets take > >> coresight > >> as an example for perpetrator and crypto as the victim here. Now we > >> can try > > > > There's no victim with branch tracing, unless it is already leaky. > > > >> If we just know one victim (lets say crypto code here), what happens > >> to the > >> others which we haven't identified yet? Do we just wait for someone to > >> write > >> an exploit based on this and then scramble to fix it? > > > > For a useful security mitigation you need a threat model first I would > > say. > > > > So you need to have at least some idea how an attack with branch > > tracing would work. > > > > > >> Initial change was to restrict this only to HW assisted instruction > >> tracing [1] > > > > I don't think it's needed for instruction tracing. > > > > From what I know, newer ARM A-profile cores doesn't allow data tracing. > And you > are saying that just the instruction tracing cannot be used to infer any > important data. > > There are few security folks in CC who probably can give us more details > on how > branch tracing can be used for an exploit. @mnissler? > > Thanks, > Sai > > -- > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a > member > of Code Aurora Forum, hosted by The Linux Foundation _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-06-10 13:32 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <cover.1614624041.git.saiprakash.ranjan@codeaurora.org> [not found] ` <def1a6b37cbb54cb15329765266ed90c2f7aa24e.1614624041.git.saiprakash.ranjan@codeaurora.org> 2021-03-01 22:42 ` [PATCHv2 1/4] perf/core: Add support to exclude kernel mode PMU tracing Doug Anderson [not found] ` <523a5af43615b804aa1211a3f27e06226d7159bc.1614624041.git.saiprakash.ranjan@codeaurora.org> 2021-03-01 22:43 ` [PATCHv2 2/4] perf evsel: Print warning for excluding kernel mode instruction tracing Doug Anderson [not found] ` <defd34e2a8744fc93404abdaad2d429fa3f69850.1614624041.git.saiprakash.ranjan@codeaurora.org> 2021-03-01 22:43 ` [PATCHv2 3/4] coresight: etm4x: Add support to exclude kernel mode tracing Doug Anderson [not found] ` <0b165b2608eed1a61240539f4b812bd80b6ad1bd.1614624041.git.saiprakash.ranjan@codeaurora.org> 2021-03-01 22:43 ` [PATCHv2 4/4] coresight: etm3x: " Doug Anderson 2021-03-04 19:59 ` [PATCHv2 0/4] perf/core: Add support to exclude kernel mode PMU tracing Andi Kleen 2021-03-04 20:17 ` Andi Kleen [not found] ` <20210309063828.26392-1-saiprakash.ranjan@codeaurora.org> 2021-03-09 14:44 ` Andi Kleen [not found] ` <24e0d604750babd3461768897bb2ae82@codeaurora.org> 2021-06-10 13:28 ` Mattias Nissler
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).