[PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

On production systems with ETMs enabled, it is preferred to
exclude kernel mode(NS EL1) tracing for security concerns and
support only userspace(NS EL0) tracing. So provide an option
via kconfig to exclude kernel mode tracing if it is required.
This config is disabled by default and would not affect the
current configuration which has both kernel and userspace
tracing enabled by default.

Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org>
---
 drivers/hwtracing/coresight/Kconfig                | 9 +++++++++
 drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++-
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig
index c1198245461d..52435de8824c 100644
--- a/drivers/hwtracing/coresight/Kconfig
+++ b/drivers/hwtracing/coresight/Kconfig
@@ -110,6 +110,15 @@ config CORESIGHT_SOURCE_ETM4X
 	  To compile this driver as a module, choose M here: the
 	  module will be called coresight-etm4x.
 
+config CORESIGHT_ETM4X_EXCL_KERN
+	bool "Coresight ETM 4.x exclude kernel mode tracing"
+	depends on CORESIGHT_SOURCE_ETM4X
+	help
+	  This will exclude kernel mode(NS EL1) tracing if enabled. This option
+	  will be useful to provide more flexible options on production systems
+	  where only userspace(NS EL0) tracing might be preferred for security
+	  reasons.
+
 config CORESIGHT_STM
 	tristate "CoreSight System Trace Macrocell driver"
 	depends on (ARM && !(CPU_32v3 || CPU_32v4 || CPU_32v4T)) || ARM64
diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c
index abd706b216ac..7e5669e5cd1f 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x-core.c
+++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c
@@ -832,6 +832,9 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config)
 {
 	u64 access_type = 0;
 
+	if (IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN))
+		config->mode |= ETM_MODE_EXCL_KERN;
+
 	/*
 	 * EXLEVEL_NS, bits[15:12]
 	 * The Exception levels are:
@@ -849,7 +852,8 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config)
 		access_type = ETM_EXLEVEL_NS_HYP;
 	}
 
-	if (config->mode & ETM_MODE_EXCL_USER)
+	if (config->mode & ETM_MODE_EXCL_USER &&
+	    !IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN))
 		access_type |= ETM_EXLEVEL_NS_APP;
 
 	return access_type;

base-commit: 3477326277451000bc667dfcc4fd0774c039184c
-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Suzuki K Poulose]

Hi Sai,

On 10/15/2020 01:45 PM, Sai Prakash Ranjan wrote:
> On production systems with ETMs enabled, it is preferred to
> exclude kernel mode(NS EL1) tracing for security concerns and
> support only userspace(NS EL0) tracing. So provide an option
> via kconfig to exclude kernel mode tracing if it is required.
> This config is disabled by default and would not affect the
> current configuration which has both kernel and userspace
> tracing enabled by default.

While this solution works for ETM4x, I would prefer if we did
this in a more generic way. There are other hardware tracing
PMUs that provide instruction tracing (e.g, Intel PT, even ETM3x)
and it would be good to have a single option that works everywhere.

Something like EXCLUDE_KERNEL_HW_ITRACE, which can be honored by
all tracing drivers ?
> 
> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org>
> ---
>   drivers/hwtracing/coresight/Kconfig                | 9 +++++++++
>   drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++-
>   2 files changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig
> index c1198245461d..52435de8824c 100644
> --- a/drivers/hwtracing/coresight/Kconfig
> +++ b/drivers/hwtracing/coresight/Kconfig
> @@ -110,6 +110,15 @@ config CORESIGHT_SOURCE_ETM4X
>   	  To compile this driver as a module, choose M here: the
>   	  module will be called coresight-etm4x.
>   
> +config CORESIGHT_ETM4X_EXCL_KERN
> +	bool "Coresight ETM 4.x exclude kernel mode tracing"
> +	depends on CORESIGHT_SOURCE_ETM4X
> +	help
> +	  This will exclude kernel mode(NS EL1) tracing if enabled. This option
> +	  will be useful to provide more flexible options on production systems
> +	  where only userspace(NS EL0) tracing might be preferred for security
> +	  reasons.
> +
>   config CORESIGHT_STM
>   	tristate "CoreSight System Trace Macrocell driver"
>   	depends on (ARM && !(CPU_32v3 || CPU_32v4 || CPU_32v4T)) || ARM64
> diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c
> index abd706b216ac..7e5669e5cd1f 100644
> --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c
> +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c
> @@ -832,6 +832,9 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config)
>   {
>   	u64 access_type = 0;
>   
> +	if (IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN))
> +		config->mode |= ETM_MODE_EXCL_KERN;
> +

Rather than hacking the mode behind the back, could we always make sure that
mode is not set in the first place and return this back to the user with
proper errors (see below) ?

>   	/*
>   	 * EXLEVEL_NS, bits[15:12]
>   	 * The Exception levels are:
> @@ -849,7 +852,8 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config)
>   		access_type = ETM_EXLEVEL_NS_HYP;
>   	}
>   
> -	if (config->mode & ETM_MODE_EXCL_USER)
> +	if (config->mode & ETM_MODE_EXCL_USER &&
> +	    !IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN))
>   		access_type |= ETM_EXLEVEL_NS_APP;

Why is this needed ?

Also we should return an error if the sysfs mode ever tries to clear the mode bit
for kernel in config->mode.

Suzuki

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

Hi Suzuki,

On 2020-10-15 19:57, Suzuki K Poulose wrote:
> Hi Sai,
> 
> On 10/15/2020 01:45 PM, Sai Prakash Ranjan wrote:
>> On production systems with ETMs enabled, it is preferred to
>> exclude kernel mode(NS EL1) tracing for security concerns and
>> support only userspace(NS EL0) tracing. So provide an option
>> via kconfig to exclude kernel mode tracing if it is required.
>> This config is disabled by default and would not affect the
>> current configuration which has both kernel and userspace
>> tracing enabled by default.
> 
> While this solution works for ETM4x, I would prefer if we did
> this in a more generic way. There are other hardware tracing
> PMUs that provide instruction tracing (e.g, Intel PT, even ETM3x)
> and it would be good to have a single option that works everywhere.
> 
> Something like EXCLUDE_KERNEL_HW_ITRACE, which can be honored by
> all tracing drivers ?

I can add this for ETM3x as well but I have zero idea regarding
Intel PTs, is there any code in those hwtracing PMUs actually
excluding kernel mode tracing currently?

>> 
>> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org>
>> ---
>>   drivers/hwtracing/coresight/Kconfig                | 9 +++++++++
>>   drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++-
>>   2 files changed, 14 insertions(+), 1 deletion(-)
>> 
>> diff --git a/drivers/hwtracing/coresight/Kconfig 
>> b/drivers/hwtracing/coresight/Kconfig
>> index c1198245461d..52435de8824c 100644
>> --- a/drivers/hwtracing/coresight/Kconfig
>> +++ b/drivers/hwtracing/coresight/Kconfig
>> @@ -110,6 +110,15 @@ config CORESIGHT_SOURCE_ETM4X
>>   	  To compile this driver as a module, choose M here: the
>>   	  module will be called coresight-etm4x.
>>   +config CORESIGHT_ETM4X_EXCL_KERN
>> +	bool "Coresight ETM 4.x exclude kernel mode tracing"
>> +	depends on CORESIGHT_SOURCE_ETM4X
>> +	help
>> +	  This will exclude kernel mode(NS EL1) tracing if enabled. This 
>> option
>> +	  will be useful to provide more flexible options on production 
>> systems
>> +	  where only userspace(NS EL0) tracing might be preferred for 
>> security
>> +	  reasons.
>> +
>>   config CORESIGHT_STM
>>   	tristate "CoreSight System Trace Macrocell driver"
>>   	depends on (ARM && !(CPU_32v3 || CPU_32v4 || CPU_32v4T)) || ARM64
>> diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c 
>> b/drivers/hwtracing/coresight/coresight-etm4x-core.c
>> index abd706b216ac..7e5669e5cd1f 100644
>> --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c
>> +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c
>> @@ -832,6 +832,9 @@ static u64 etm4_get_ns_access_type(struct 
>> etmv4_config *config)
>>   {
>>   	u64 access_type = 0;
>>   +	if (IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN))
>> +		config->mode |= ETM_MODE_EXCL_KERN;
>> +
> 
> Rather than hacking the mode behind the back, could we always make sure 
> that
> mode is not set in the first place and return this back to the user 
> with
> proper errors (see below) ?
> 

Sure, this was the minimal change with which I could keep the
check in one place which would work for both sysfs and perf,
but I'll change as you suggested and move the check to
etm4_parse_event_config() and etm4_config_trace_mode() and
return errors properly.

>>   	/*
>>   	 * EXLEVEL_NS, bits[15:12]
>>   	 * The Exception levels are:
>> @@ -849,7 +852,8 @@ static u64 etm4_get_ns_access_type(struct 
>> etmv4_config *config)
>>   		access_type = ETM_EXLEVEL_NS_HYP;
>>   	}
>>   -	if (config->mode & ETM_MODE_EXCL_USER)
>> +	if (config->mode & ETM_MODE_EXCL_USER &&
>> +	    !IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN))
>>   		access_type |= ETM_EXLEVEL_NS_APP;
> 
> Why is this needed ?
> 

Yes this will not be required as excluding both doesn't make
sense and we print warning in that case already, will drop
this.

> Also we should return an error if the sysfs mode ever tries to clear
> the mode bit
> for kernel in config->mode.
> 

Yes will change as explained in above comment.

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Mathieu Poirier]

On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
> On production systems with ETMs enabled, it is preferred to
> exclude kernel mode(NS EL1) tracing for security concerns and
> support only userspace(NS EL0) tracing. So provide an option
> via kconfig to exclude kernel mode tracing if it is required.
> This config is disabled by default and would not affect the
> current configuration which has both kernel and userspace
> tracing enabled by default.
>

One requires root access (or be part of a special trace group) to be able to use
the cs_etm PMU.  With this kind of elevated access restricting tracing at EL1
provides little in terms of security.

Thanks,
Mathieu
 
> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org>
> ---
>  drivers/hwtracing/coresight/Kconfig                | 9 +++++++++
>  drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++-
>  2 files changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig
> index c1198245461d..52435de8824c 100644
> --- a/drivers/hwtracing/coresight/Kconfig
> +++ b/drivers/hwtracing/coresight/Kconfig
> @@ -110,6 +110,15 @@ config CORESIGHT_SOURCE_ETM4X
>  	  To compile this driver as a module, choose M here: the
>  	  module will be called coresight-etm4x.
>  
> +config CORESIGHT_ETM4X_EXCL_KERN
> +	bool "Coresight ETM 4.x exclude kernel mode tracing"
> +	depends on CORESIGHT_SOURCE_ETM4X
> +	help
> +	  This will exclude kernel mode(NS EL1) tracing if enabled. This option
> +	  will be useful to provide more flexible options on production systems
> +	  where only userspace(NS EL0) tracing might be preferred for security
> +	  reasons.
> +
>  config CORESIGHT_STM
>  	tristate "CoreSight System Trace Macrocell driver"
>  	depends on (ARM && !(CPU_32v3 || CPU_32v4 || CPU_32v4T)) || ARM64
> diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c
> index abd706b216ac..7e5669e5cd1f 100644
> --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c
> +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c
> @@ -832,6 +832,9 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config)
>  {
>  	u64 access_type = 0;
>  
> +	if (IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN))
> +		config->mode |= ETM_MODE_EXCL_KERN;
> +
>  	/*
>  	 * EXLEVEL_NS, bits[15:12]
>  	 * The Exception levels are:
> @@ -849,7 +852,8 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config)
>  		access_type = ETM_EXLEVEL_NS_HYP;
>  	}
>  
> -	if (config->mode & ETM_MODE_EXCL_USER)
> +	if (config->mode & ETM_MODE_EXCL_USER &&
> +	    !IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN))
>  		access_type |= ETM_EXLEVEL_NS_APP;
>  
>  	return access_type;
> 
> base-commit: 3477326277451000bc667dfcc4fd0774c039184c
> -- 
> QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
> of Code Aurora Forum, hosted by The Linux Foundation
> 

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Leo Yan]

On Thu, Oct 15, 2020 at 11:40:05PM -0700, Denis Nikitin wrote:
> Hi Mathieu,
> 
> I think one of the use cases could be VMs.
> Is there isolation between EL1 guest kernels which we can control from perf
> in a system wide mode?

Sorry for suddenly jumping in.

For KVM, I think we need to implement mechanism for saving/restoring
CoreSight context for every guest OS, the CPU PMUs has implemented
related features [1].

Thanks,
Leo

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/pmu.c

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

Hi Leo,

On 2020-10-16 12:54, Leo Yan wrote:
> On Thu, Oct 15, 2020 at 11:40:05PM -0700, Denis Nikitin wrote:
>> Hi Mathieu,
>> 
>> I think one of the use cases could be VMs.
>> Is there isolation between EL1 guest kernels which we can control from 
>> perf
>> in a system wide mode?
> 
> Sorry for suddenly jumping in.
> 
> For KVM, I think we need to implement mechanism for saving/restoring
> CoreSight context for every guest OS, the CPU PMUs has implemented
> related features [1].
> 
> Thanks,
> Leo
> 
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/pmu.c
> 

What happens to the sysfs mode of tracing? For that we would still
need a config right to exclude kernel mode tracing completely.

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Leo Yan]

Hi Sai,

On Fri, Oct 16, 2020 at 02:10:47PM +0530, Sai Prakash Ranjan wrote:
> Hi Leo,
> 
> On 2020-10-16 12:54, Leo Yan wrote:
> > On Thu, Oct 15, 2020 at 11:40:05PM -0700, Denis Nikitin wrote:
> > > Hi Mathieu,
> > > 
> > > I think one of the use cases could be VMs.
> > > Is there isolation between EL1 guest kernels which we can control
> > > from perf
> > > in a system wide mode?
> > 
> > Sorry for suddenly jumping in.
> > 
> > For KVM, I think we need to implement mechanism for saving/restoring
> > CoreSight context for every guest OS, the CPU PMUs has implemented
> > related features [1].
> > 
> > Thanks,
> > Leo
> > 
> > [1]
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/pmu.c
> > 
> 
> What happens to the sysfs mode of tracing? For that we would still
> need a config right to exclude kernel mode tracing completely.

IIUC, sysfs mode and perf mode both can apply the same approach, the
guest OS runs a thread context for the host, so when a guest OS is
switched in or out, the hypervisor can save/restore the context for
the guest OS; thus every guest OS will have its dedicated context and
trace data ideally.

Thanks,
Leo

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

Hi Leo,

On 2020-10-16 14:54, Leo Yan wrote:
> Hi Sai,
> 
> On Fri, Oct 16, 2020 at 02:10:47PM +0530, Sai Prakash Ranjan wrote:
>> Hi Leo,
>> 
>> On 2020-10-16 12:54, Leo Yan wrote:
>> > On Thu, Oct 15, 2020 at 11:40:05PM -0700, Denis Nikitin wrote:
>> > > Hi Mathieu,
>> > >
>> > > I think one of the use cases could be VMs.
>> > > Is there isolation between EL1 guest kernels which we can control
>> > > from perf
>> > > in a system wide mode?
>> >
>> > Sorry for suddenly jumping in.
>> >
>> > For KVM, I think we need to implement mechanism for saving/restoring
>> > CoreSight context for every guest OS, the CPU PMUs has implemented
>> > related features [1].
>> >
>> > Thanks,
>> > Leo
>> >
>> > [1]
>> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/pmu.c
>> >
>> 
>> What happens to the sysfs mode of tracing? For that we would still
>> need a config right to exclude kernel mode tracing completely.
> 
> IIUC, sysfs mode and perf mode both can apply the same approach, the
> guest OS runs a thread context for the host, so when a guest OS is
> switched in or out, the hypervisor can save/restore the context for
> the guest OS; thus every guest OS will have its dedicated context and
> trace data ideally.
> 

Thanks for the explanation, so for this usecase then we would have to
implement something as you suggested, not sure how hard would that be
looking at my KVM knowledge(which at the moment is almost nil) when
compared to a kconfig ;)

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Suzuki Poulose]

On 10/16/20 10:24 AM, Leo Yan wrote:
> Hi Sai,
> 
> On Fri, Oct 16, 2020 at 02:10:47PM +0530, Sai Prakash Ranjan wrote:
>> Hi Leo,
>>
>> On 2020-10-16 12:54, Leo Yan wrote:
>>> On Thu, Oct 15, 2020 at 11:40:05PM -0700, Denis Nikitin wrote:
>>>> Hi Mathieu,
>>>>
>>>> I think one of the use cases could be VMs.
>>>> Is there isolation between EL1 guest kernels which we can control
>>>> from perf
>>>> in a system wide mode?
>>>
>>> Sorry for suddenly jumping in.
>>>
>>> For KVM, I think we need to implement mechanism for saving/restoring
>>> CoreSight context for every guest OS, the CPU PMUs has implemented
>>> related features [1].
>>>
>>> Thanks,
>>> Leo
>>>
>>> [1]
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/pmu.c
>>>

Its not as easy as the CPU PMU for virtualizing the ETMs (with memory
mapped access only), i.e supporting ETMs from VMs.
We could definitely stop/resume on guest entry/exit, to support 
attr.exclude_guest.

>>
>> What happens to the sysfs mode of tracing? For that we would still
>> need a config right to exclude kernel mode tracing completely.
> 
> IIUC, sysfs mode and perf mode both can apply the same approach, the
> guest OS runs a thread context for the host, so when a guest OS is
> switched in or out, the hypervisor can save/restore the context for
> the guest OS; thus every guest OS will have its dedicated context and
> trace data ideally.

I don't think Guest Context is something we can support as mentioned
above, at least for systems without sysreg access for ETMs (and 
virtualizing ETRs is a different story !)

Cheers
Suzuki

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Leo Yan]

On Fri, Oct 16, 2020 at 12:38:47PM +0100, Suzuki Kuruppassery Poulose wrote:

[...]

> > > What happens to the sysfs mode of tracing? For that we would still
> > > need a config right to exclude kernel mode tracing completely.
> > 
> > IIUC, sysfs mode and perf mode both can apply the same approach, the
> > guest OS runs a thread context for the host, so when a guest OS is
> > switched in or out, the hypervisor can save/restore the context for
> > the guest OS; thus every guest OS will have its dedicated context and
> > trace data ideally.
> 
> I don't think Guest Context is something we can support as mentioned
> above, at least for systems without sysreg access for ETMs (and virtualizing
> ETRs is a different story !)

Thanks for sharing thoughts, Suzuki.

I missed the device virtulisation.  Here should virtualize all devices
(includes CoreSight ETM/funnel/ETR/ETF)?  Or only need to virtualize
ETRs?

Obviously, this is a difficult task :)

Thanks,
Leo

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Suzuki Poulose]

On 10/16/20 2:14 PM, Leo Yan wrote:
> On Fri, Oct 16, 2020 at 12:38:47PM +0100, Suzuki Kuruppassery Poulose wrote:
> 
> [...]
> 
>>>> What happens to the sysfs mode of tracing? For that we would still
>>>> need a config right to exclude kernel mode tracing completely.
>>>
>>> IIUC, sysfs mode and perf mode both can apply the same approach, the
>>> guest OS runs a thread context for the host, so when a guest OS is
>>> switched in or out, the hypervisor can save/restore the context for
>>> the guest OS; thus every guest OS will have its dedicated context and
>>> trace data ideally.
>>
>> I don't think Guest Context is something we can support as mentioned
>> above, at least for systems without sysreg access for ETMs (and virtualizing
>> ETRs is a different story !)
> 
> Thanks for sharing thoughts, Suzuki.
> 
> I missed the device virtulisation.  Here should virtualize all devices
> (includes CoreSight ETM/funnel/ETR/ETF)?  Or only need to virtualize
> ETRs?

I wouldn't even think of virtualizing the components without sysreg
access. So let us not worry about it :-)

Cheers
Suzuki

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Suzuki Poulose]

On 10/16/20 7:40 AM, Denis Nikitin wrote:
> Hi Mathieu,
> 
> I think one of the use cases could be VMs.
> Is there isolation between EL1 guest kernels which we can control from 
> perf in a system wide mode?

The proposed solution doesn't solve this for VMs anyway. It only
excludes EL1 *OR* EL2, depending on the host kernel's running  EL.
We cannot support Virtual ETM access for VMs with memory mapped
accesses.

Unforutnately, trace filtering is the solution for preventing tracing
for EL1 guest/kernel (available from v8.4 Self Hosted extensions). Other
option is to add support for "exclude_guest" support for CoreSight for perf.
But again this can't be controlled by sysfs. And it can't be enforced 
for perf, if not specified. Again it all goes back to the root
permission hammer lock which Mathieu pointed out.


With the v8.4 Self hosted trace extensions, Guest and Host both could
control individually if they can be traced (both EL0 and EL1/2).

Suzuki

> 
> Thanks,
> Denis
> 
> On Thu, Oct 15, 2020 at 9:03 AM Mathieu Poirier 
> <mathieu.poirier@linaro.org <mailto:mathieu.poirier@linaro.org>> wrote:
> 
>     On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
>      > On production systems with ETMs enabled, it is preferred to
>      > exclude kernel mode(NS EL1) tracing for security concerns and
>      > support only userspace(NS EL0) tracing. So provide an option
>      > via kconfig to exclude kernel mode tracing if it is required.
>      > This config is disabled by default and would not affect the
>      > current configuration which has both kernel and userspace
>      > tracing enabled by default.
>      >
> 
>     One requires root access (or be part of a special trace group) to be
>     able to use
>     the cs_etm PMU.  With this kind of elevated access restricting
>     tracing at EL1
>     provides little in terms of security.
> 
>     Thanks,
>     Mathieu
> 
>      > Signed-off-by: Sai Prakash Ranjan
>     <saiprakash.ranjan@codeaurora.org
>     <mailto:saiprakash.ranjan@codeaurora.org>>
>      > ---
>      >  drivers/hwtracing/coresight/Kconfig                | 9 +++++++++
>      >  drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++-
>      >  2 files changed, 14 insertions(+), 1 deletion(-)
>      >
>      > diff --git a/drivers/hwtracing/coresight/Kconfig
>     b/drivers/hwtracing/coresight/Kconfig
>      > index c1198245461d..52435de8824c 100644
>      > --- a/drivers/hwtracing/coresight/Kconfig
>      > +++ b/drivers/hwtracing/coresight/Kconfig
>      > @@ -110,6 +110,15 @@ config CORESIGHT_SOURCE_ETM4X
>      >         To compile this driver as a module, choose M here: the
>      >         module will be called coresight-etm4x.
>      >
>      > +config CORESIGHT_ETM4X_EXCL_KERN
>      > +     bool "Coresight ETM 4.x exclude kernel mode tracing"
>      > +     depends on CORESIGHT_SOURCE_ETM4X
>      > +     help
>      > +       This will exclude kernel mode(NS EL1) tracing if enabled.
>     This option
>      > +       will be useful to provide more flexible options on
>     production systems
>      > +       where only userspace(NS EL0) tracing might be preferred
>     for security
>      > +       reasons.
>      > +
>      >  config CORESIGHT_STM
>      >       tristate "CoreSight System Trace Macrocell driver"
>      >       depends on (ARM && !(CPU_32v3 || CPU_32v4 || CPU_32v4T)) ||
>     ARM64
>      > diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c
>     b/drivers/hwtracing/coresight/coresight-etm4x-core.c
>      > index abd706b216ac..7e5669e5cd1f 100644
>      > --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c
>      > +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c
>      > @@ -832,6 +832,9 @@ static u64 etm4_get_ns_access_type(struct
>     etmv4_config *config)
>      >  {
>      >       u64 access_type = 0;
>      >
>      > +     if (IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN))
>      > +             config->mode |= ETM_MODE_EXCL_KERN;
>      > +
>      >       /*
>      >        * EXLEVEL_NS, bits[15:12]
>      >        * The Exception levels are:
>      > @@ -849,7 +852,8 @@ static u64 etm4_get_ns_access_type(struct
>     etmv4_config *config)
>      >               access_type = ETM_EXLEVEL_NS_HYP;
>      >       }
>      >
>      > -     if (config->mode & ETM_MODE_EXCL_USER)
>      > +     if (config->mode & ETM_MODE_EXCL_USER &&
>      > +         !IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN))
>      >               access_type |= ETM_EXLEVEL_NS_APP;
>      >
>      >       return access_type;
>      >
>      > base-commit: 3477326277451000bc667dfcc4fd0774c039184c
>      > --
>      > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is
>     a member
>      > of Code Aurora Forum, hosted by The Linux Foundation
>      >
> 

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

Hello Mathieu, Suzuki

On 2020-10-15 21:32, Mathieu Poirier wrote:
> On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
>> On production systems with ETMs enabled, it is preferred to
>> exclude kernel mode(NS EL1) tracing for security concerns and
>> support only userspace(NS EL0) tracing. So provide an option
>> via kconfig to exclude kernel mode tracing if it is required.
>> This config is disabled by default and would not affect the
>> current configuration which has both kernel and userspace
>> tracing enabled by default.
>> 
> 
> One requires root access (or be part of a special trace group) to be 
> able to use
> the cs_etm PMU.  With this kind of elevated access restricting tracing 
> at EL1
> provides little in terms of security.
> 

Apart from the VM usecase discussed, I am told there are other
security concerns here regarding need to exclude kernel mode tracing
even for the privileged users/root. One such case being the ability
to analyze cryptographic code execution since ETMs can record all
branch instructions including timestamps in the kernel and there may
be other cases as well which I may not be aware of and hence have
added Denis and Mattias. Please let us know if you have any questions
further regarding this not being a security concern.

After this discussion, I would like to post a v2 based on Suzuki's
feedback earlier. @Suzuki, I have a common config for ETM3 and ETM4
but couldn't get much idea on how to implement it for Intel PTs, if
you have any suggestions there, please do share or we can have this
only for Coresight ETMs.

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Mattias Nissler]

On Fri, Jan 15, 2021 at 6:46 AM Sai Prakash Ranjan
<saiprakash.ranjan@codeaurora.org> wrote:
>
> Hello Mathieu, Suzuki
>
> On 2020-10-15 21:32, Mathieu Poirier wrote:
> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
> >> On production systems with ETMs enabled, it is preferred to
> >> exclude kernel mode(NS EL1) tracing for security concerns and
> >> support only userspace(NS EL0) tracing. So provide an option
> >> via kconfig to exclude kernel mode tracing if it is required.
> >> This config is disabled by default and would not affect the
> >> current configuration which has both kernel and userspace
> >> tracing enabled by default.
> >>
> >
> > One requires root access (or be part of a special trace group) to be
> > able to use
> > the cs_etm PMU.  With this kind of elevated access restricting tracing
> > at EL1
> > provides little in terms of security.
> >
>
> Apart from the VM usecase discussed, I am told there are other
> security concerns here regarding need to exclude kernel mode tracing
> even for the privileged users/root. One such case being the ability
> to analyze cryptographic code execution since ETMs can record all
> branch instructions including timestamps in the kernel and there may
> be other cases as well which I may not be aware of and hence have
> added Denis and Mattias. Please let us know if you have any questions
> further regarding this not being a security concern.

Well, the idea that root privileges != full control over the kernel
isn't new and at the very least since lockdown became part of mainline
[1] no longer an esoteric edge case. Regarding the use case Sai hints
at (namely protection of secrets in the kernel), Matthew Garret
actually has some more thoughts about confidentiality mode for
lockdown for secret protection [2]. And thus, unless someone can make
a compelling case that instruction-level tracing will not leak secrets
held by the kernel, I think an option for the kernel to prevent itself
from being traced (even by root) is valuable.

Finally, to sketch a practical use case scenario: Consider a system
where disk contents are encrypted and the encryption key is set up by
the user when mounting the file system. From that point on the
encryption key resides in the kernel. It seems reasonable to expect
that the disk encryption key be protected from exfiltration even if
the system later suffers a root compromise (or even against insiders
that have root access), at least as long as the attacker doesn't
manage to compromise the kernel.

[1] https://lwn.net/Articles/796866/
[2] https://mjg59.dreamwidth.org/55105.html

>
> After this discussion, I would like to post a v2 based on Suzuki's
> feedback earlier. @Suzuki, I have a common config for ETM3 and ETM4
> but couldn't get much idea on how to implement it for Intel PTs, if
> you have any suggestions there, please do share or we can have this
> only for Coresight ETMs.
>
> Thanks,
> Sai
>
> --
> QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a
> member
> of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

On 2021-01-18 20:17, Mattias Nissler wrote:
> On Fri, Jan 15, 2021 at 6:46 AM Sai Prakash Ranjan
> <saiprakash.ranjan@codeaurora.org> wrote:
>> 
>> Hello Mathieu, Suzuki
>> 
>> On 2020-10-15 21:32, Mathieu Poirier wrote:
>> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
>> >> On production systems with ETMs enabled, it is preferred to
>> >> exclude kernel mode(NS EL1) tracing for security concerns and
>> >> support only userspace(NS EL0) tracing. So provide an option
>> >> via kconfig to exclude kernel mode tracing if it is required.
>> >> This config is disabled by default and would not affect the
>> >> current configuration which has both kernel and userspace
>> >> tracing enabled by default.
>> >>
>> >
>> > One requires root access (or be part of a special trace group) to be
>> > able to use
>> > the cs_etm PMU.  With this kind of elevated access restricting tracing
>> > at EL1
>> > provides little in terms of security.
>> >
>> 
>> Apart from the VM usecase discussed, I am told there are other
>> security concerns here regarding need to exclude kernel mode tracing
>> even for the privileged users/root. One such case being the ability
>> to analyze cryptographic code execution since ETMs can record all
>> branch instructions including timestamps in the kernel and there may
>> be other cases as well which I may not be aware of and hence have
>> added Denis and Mattias. Please let us know if you have any questions
>> further regarding this not being a security concern.
> 
> Well, the idea that root privileges != full control over the kernel
> isn't new and at the very least since lockdown became part of mainline
> [1] no longer an esoteric edge case. Regarding the use case Sai hints
> at (namely protection of secrets in the kernel), Matthew Garret
> actually has some more thoughts about confidentiality mode for
> lockdown for secret protection [2]. And thus, unless someone can make
> a compelling case that instruction-level tracing will not leak secrets
> held by the kernel, I think an option for the kernel to prevent itself
> from being traced (even by root) is valuable.
> 
> Finally, to sketch a practical use case scenario: Consider a system
> where disk contents are encrypted and the encryption key is set up by
> the user when mounting the file system. From that point on the
> encryption key resides in the kernel. It seems reasonable to expect
> that the disk encryption key be protected from exfiltration even if
> the system later suffers a root compromise (or even against insiders
> that have root access), at least as long as the attacker doesn't
> manage to compromise the kernel.
> 
> [1] https://lwn.net/Articles/796866/
> [2] https://mjg59.dreamwidth.org/55105.html
> 

Thanks for the detailed description, it is way better put than my crude
explanation.

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Mathieu Poirier]

On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote:
> Hello Mathieu, Suzuki
> 
> On 2020-10-15 21:32, Mathieu Poirier wrote:
> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
> > > On production systems with ETMs enabled, it is preferred to
> > > exclude kernel mode(NS EL1) tracing for security concerns and
> > > support only userspace(NS EL0) tracing. So provide an option
> > > via kconfig to exclude kernel mode tracing if it is required.
> > > This config is disabled by default and would not affect the
> > > current configuration which has both kernel and userspace
> > > tracing enabled by default.
> > > 
> > 
> > One requires root access (or be part of a special trace group) to be
> > able to use
> > the cs_etm PMU.  With this kind of elevated access restricting tracing
> > at EL1
> > provides little in terms of security.
> > 
> 
> Apart from the VM usecase discussed, I am told there are other
> security concerns here regarding need to exclude kernel mode tracing
> even for the privileged users/root. One such case being the ability
> to analyze cryptographic code execution since ETMs can record all
> branch instructions including timestamps in the kernel and there may
> be other cases as well which I may not be aware of and hence have
> added Denis and Mattias. Please let us know if you have any questions
> further regarding this not being a security concern.

Even if we were to apply this patch there are many ways to compromise a system
or get the kernel to reveal important information using the perf subsystem.  I
would perfer to tackle the problem at that level rather than concentrating on
coresight.

> 
> After this discussion, I would like to post a v2 based on Suzuki's
> feedback earlier. @Suzuki, I have a common config for ETM3 and ETM4
> but couldn't get much idea on how to implement it for Intel PTs, if
> you have any suggestions there, please do share or we can have this
> only for Coresight ETMs.
> 
> Thanks,
> Sai
> 
> -- 
> QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
> of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

Hi Mathieu,

On 2021-01-19 01:53, Mathieu Poirier wrote:
> On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote:
>> Hello Mathieu, Suzuki
>> 
>> On 2020-10-15 21:32, Mathieu Poirier wrote:
>> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
>> > > On production systems with ETMs enabled, it is preferred to
>> > > exclude kernel mode(NS EL1) tracing for security concerns and
>> > > support only userspace(NS EL0) tracing. So provide an option
>> > > via kconfig to exclude kernel mode tracing if it is required.
>> > > This config is disabled by default and would not affect the
>> > > current configuration which has both kernel and userspace
>> > > tracing enabled by default.
>> > >
>> >
>> > One requires root access (or be part of a special trace group) to be
>> > able to use
>> > the cs_etm PMU.  With this kind of elevated access restricting tracing
>> > at EL1
>> > provides little in terms of security.
>> >
>> 
>> Apart from the VM usecase discussed, I am told there are other
>> security concerns here regarding need to exclude kernel mode tracing
>> even for the privileged users/root. One such case being the ability
>> to analyze cryptographic code execution since ETMs can record all
>> branch instructions including timestamps in the kernel and there may
>> be other cases as well which I may not be aware of and hence have
>> added Denis and Mattias. Please let us know if you have any questions
>> further regarding this not being a security concern.
> 
> Even if we were to apply this patch there are many ways to compromise a 
> system
> or get the kernel to reveal important information using the perf 
> subsystem.  I
> would perfer to tackle the problem at that level rather than 
> concentrating on
> coresight.
> 

Sorry but I did not understand your point. We are talking about the
capabilities of coresight etm tracing which has the instruction level
tracing and a lot more. Perf subsystem is just the framework used for 
it.
In other words, its not the perf subsystem which does instruction level
tracing, its the coresight etm. Why the perf subsystem should be
modified to lockdown kernel mode? If we were to let perf handle all the
trace filtering for different exception levels, then why do we need
the register settings in coresight etm driver to filter out NS EL*
tracing? And more importantly, how do you suppose we handle sysfs mode
of coresight tracing with perf subsystem?

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation

RE: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Al Grant]

Hi Sai,

> From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org
> Hi Mathieu,
> 
> On 2021-01-19 01:53, Mathieu Poirier wrote:
> > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote:
> >> Hello Mathieu, Suzuki
> >>
> >> On 2020-10-15 21:32, Mathieu Poirier wrote:
> >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
> >> > > On production systems with ETMs enabled, it is preferred to
> >> > > exclude kernel mode(NS EL1) tracing for security concerns and
> >> > > support only userspace(NS EL0) tracing. So provide an option via
> >> > > kconfig to exclude kernel mode tracing if it is required.
> >> > > This config is disabled by default and would not affect the
> >> > > current configuration which has both kernel and userspace tracing
> >> > > enabled by default.
> >> > >
> >> >
> >> > One requires root access (or be part of a special trace group) to
> >> > be able to use the cs_etm PMU.  With this kind of elevated access
> >> > restricting tracing at EL1 provides little in terms of security.
> >> >
> >>
> >> Apart from the VM usecase discussed, I am told there are other
> >> security concerns here regarding need to exclude kernel mode tracing
> >> even for the privileged users/root. One such case being the ability
> >> to analyze cryptographic code execution since ETMs can record all
> >> branch instructions including timestamps in the kernel and there may
> >> be other cases as well which I may not be aware of and hence have
> >> added Denis and Mattias. Please let us know if you have any questions
> >> further regarding this not being a security concern.
> >
> > Even if we were to apply this patch there are many ways to compromise
> > a system or get the kernel to reveal important information using the
> > perf subsystem.  I would perfer to tackle the problem at that level
> > rather than concentrating on coresight.
> >
> 
> Sorry but I did not understand your point. We are talking about the capabilities
> of coresight etm tracing which has the instruction level tracing and a lot more.
> Perf subsystem is just the framework used for it.
> In other words, its not the perf subsystem which does instruction level tracing,
> its the coresight etm. Why the perf subsystem should be modified to lockdown
> kernel mode? If we were to let perf handle all the trace filtering for different
> exception levels, then why do we need the register settings in coresight etm
> driver to filter out NS EL* tracing? And more importantly, how do you suppose
> we handle sysfs mode of coresight tracing with perf subsystem?

You both have good points. Mathieu is right that this is not a CoreSight
issue specifically, it is a matter of kernel security policy, and other hardware
tracing mechanisms ought to be within its scope. There should be a general
"anti kernel exfiltration" config that applies to all mechanisms within
its scope, and we'd definitely expect that to include Intel PT as well as ETM.

A kernel config that forced exclude_kernel on all perf events would deal with
ETM and PT in one place, but miss the sysfs interface to ETM.

On the other hand, doing it in the ETM drivers would cover the perf and sysfs
interfaces to ETM, but would miss Intel PT.

So I think what is needed is a general config option that is both implemented
in perf (excluding all kernel tracing events) and by any drivers that provide
an alternative interface to hardware tracing events.

Al


> 
> Thanks,
> Sai
> 
> --
> QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
> of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

Hi Al,

On 2021-01-19 14:06, Al Grant wrote:
> Hi Sai,
> 
>> From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org
>> Hi Mathieu,
>> 
>> On 2021-01-19 01:53, Mathieu Poirier wrote:
>> > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote:
>> >> Hello Mathieu, Suzuki
>> >>
>> >> On 2020-10-15 21:32, Mathieu Poirier wrote:
>> >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
>> >> > > On production systems with ETMs enabled, it is preferred to
>> >> > > exclude kernel mode(NS EL1) tracing for security concerns and
>> >> > > support only userspace(NS EL0) tracing. So provide an option via
>> >> > > kconfig to exclude kernel mode tracing if it is required.
>> >> > > This config is disabled by default and would not affect the
>> >> > > current configuration which has both kernel and userspace tracing
>> >> > > enabled by default.
>> >> > >
>> >> >
>> >> > One requires root access (or be part of a special trace group) to
>> >> > be able to use the cs_etm PMU.  With this kind of elevated access
>> >> > restricting tracing at EL1 provides little in terms of security.
>> >> >
>> >>
>> >> Apart from the VM usecase discussed, I am told there are other
>> >> security concerns here regarding need to exclude kernel mode tracing
>> >> even for the privileged users/root. One such case being the ability
>> >> to analyze cryptographic code execution since ETMs can record all
>> >> branch instructions including timestamps in the kernel and there may
>> >> be other cases as well which I may not be aware of and hence have
>> >> added Denis and Mattias. Please let us know if you have any questions
>> >> further regarding this not being a security concern.
>> >
>> > Even if we were to apply this patch there are many ways to compromise
>> > a system or get the kernel to reveal important information using the
>> > perf subsystem.  I would perfer to tackle the problem at that level
>> > rather than concentrating on coresight.
>> >
>> 
>> Sorry but I did not understand your point. We are talking about the 
>> capabilities
>> of coresight etm tracing which has the instruction level tracing and a 
>> lot more.
>> Perf subsystem is just the framework used for it.
>> In other words, its not the perf subsystem which does instruction 
>> level tracing,
>> its the coresight etm. Why the perf subsystem should be modified to 
>> lockdown
>> kernel mode? If we were to let perf handle all the trace filtering for 
>> different
>> exception levels, then why do we need the register settings in 
>> coresight etm
>> driver to filter out NS EL* tracing? And more importantly, how do you 
>> suppose
>> we handle sysfs mode of coresight tracing with perf subsystem?
> 
> You both have good points. Mathieu is right that this is not a 
> CoreSight
> issue specifically, it is a matter of kernel security policy, and other 
> hardware
> tracing mechanisms ought to be within its scope. There should be a 
> general
> "anti kernel exfiltration" config that applies to all mechanisms within
> its scope, and we'd definitely expect that to include Intel PT as well 
> as ETM.
> 

I agree with this part where there should be a generic config for all
hardware tracing families(atleast for Intel PT and ARM Coresight),
Suzuki suggested that as well. I am under the impression that Mathieu
didn't like adding such a config and wanted perf subsystem to handle
it since initial discussion was around whether root compromise meant
everything is lost already and such a kconfig would not help, but
Mattias already gave some good examples where that is not true.

> A kernel config that forced exclude_kernel on all perf events would 
> deal with
> ETM and PT in one place, but miss the sysfs interface to ETM.
> 
> On the other hand, doing it in the ETM drivers would cover the perf and 
> sysfs
> interfaces to ETM, but would miss Intel PT.
> 
> So I think what is needed is a general config option that is both 
> implemented
> in perf (excluding all kernel tracing events) and by any drivers that 
> provide
> an alternative interface to hardware tracing events.
> 

I am good with this approach, once Mathieu confirms, I can add a kernel
wide kconfig as Suzuki suggested earlier and make ETM{3,4}x as the
initial users. Someone more familiar with Intel PTs can then make use
of this kconfig.

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Suzuki K Poulose]

On 1/19/21 9:51 AM, Sai Prakash Ranjan wrote:
> Hi Al,
> 
> On 2021-01-19 14:06, Al Grant wrote:
>> Hi Sai,
>>
>>> From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org
>>> Hi Mathieu,
>>>
>>> On 2021-01-19 01:53, Mathieu Poirier wrote:
>>> > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote:
>>> >> Hello Mathieu, Suzuki
>>> >>
>>> >> On 2020-10-15 21:32, Mathieu Poirier wrote:
>>> >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
>>> >> > > On production systems with ETMs enabled, it is preferred to
>>> >> > > exclude kernel mode(NS EL1) tracing for security concerns and
>>> >> > > support only userspace(NS EL0) tracing. So provide an option via
>>> >> > > kconfig to exclude kernel mode tracing if it is required.
>>> >> > > This config is disabled by default and would not affect the
>>> >> > > current configuration which has both kernel and userspace tracing
>>> >> > > enabled by default.
>>> >> > >
>>> >> >
>>> >> > One requires root access (or be part of a special trace group) to
>>> >> > be able to use the cs_etm PMU.  With this kind of elevated access
>>> >> > restricting tracing at EL1 provides little in terms of security.
>>> >> >
>>> >>
>>> >> Apart from the VM usecase discussed, I am told there are other
>>> >> security concerns here regarding need to exclude kernel mode tracing
>>> >> even for the privileged users/root. One such case being the ability
>>> >> to analyze cryptographic code execution since ETMs can record all
>>> >> branch instructions including timestamps in the kernel and there may
>>> >> be other cases as well which I may not be aware of and hence have
>>> >> added Denis and Mattias. Please let us know if you have any questions
>>> >> further regarding this not being a security concern.
>>> >
>>> > Even if we were to apply this patch there are many ways to compromise
>>> > a system or get the kernel to reveal important information using the
>>> > perf subsystem.  I would perfer to tackle the problem at that level
>>> > rather than concentrating on coresight.
>>> >
>>>
>>> Sorry but I did not understand your point. We are talking about the capabilities
>>> of coresight etm tracing which has the instruction level tracing and a lot more.
>>> Perf subsystem is just the framework used for it.
>>> In other words, its not the perf subsystem which does instruction level tracing,
>>> its the coresight etm. Why the perf subsystem should be modified to lockdown
>>> kernel mode? If we were to let perf handle all the trace filtering for different
>>> exception levels, then why do we need the register settings in coresight etm
>>> driver to filter out NS EL* tracing? And more importantly, how do you suppose
>>> we handle sysfs mode of coresight tracing with perf subsystem?
>>
>> You both have good points. Mathieu is right that this is not a CoreSight
>> issue specifically, it is a matter of kernel security policy, and other hardware
>> tracing mechanisms ought to be within its scope. There should be a general
>> "anti kernel exfiltration" config that applies to all mechanisms within
>> its scope, and we'd definitely expect that to include Intel PT as well as ETM.
>>
> 
> I agree with this part where there should be a generic config for all
> hardware tracing families(atleast for Intel PT and ARM Coresight),
> Suzuki suggested that as well. I am under the impression that Mathieu
> didn't like adding such a config and wanted perf subsystem to handle
> it since initial discussion was around whether root compromise meant
> everything is lost already and such a kconfig would not help, but
> Mattias already gave some good examples where that is not true.
> 
>> A kernel config that forced exclude_kernel on all perf events would deal with
>> ETM and PT in one place, but miss the sysfs interface to ETM.
>>
>> On the other hand, doing it in the ETM drivers would cover the perf and sysfs
>> interfaces to ETM, but would miss Intel PT.
>>
>> So I think what is needed is a general config option that is both implemented
>> in perf (excluding all kernel tracing events) and by any drivers that provide
>> an alternative interface to hardware tracing events.
>>
> 
> I am good with this approach, once Mathieu confirms, I can add a kernel
> wide kconfig as Suzuki suggested earlier and make ETM{3,4}x as the
> initial users. Someone more familiar with Intel PTs can then make use
> of this kconfig.

Instead of adding the support for individual drivers, you could handle this
in the generic perf layer. e.g, Fail perf_event create with an attribute
which allows kernel tracing ?

if (!attr.exclude_kernel)
	return -EINVAL;

Or even exclude the kernel silently always.

This could also be limited to PMUs with PERF_PMU_CAP_ITRACE, if you
want to limit this to PMUs that instruction level tracing.

Suzuki

RE: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Al Grant]

> From: Suzuki K Poulose <suzuki.poulose@arm.com>
> On 1/19/21 9:51 AM, Sai Prakash Ranjan wrote:
> > Hi Al,
> >
> > On 2021-01-19 14:06, Al Grant wrote:
> >> Hi Sai,
> >>
> >>> From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org
> >>> Hi Mathieu,
> >>>
> >>> On 2021-01-19 01:53, Mathieu Poirier wrote:
> >>> > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote:
> >>> >> Hello Mathieu, Suzuki
> >>> >>
> >>> >> On 2020-10-15 21:32, Mathieu Poirier wrote:
> >>> >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
> >>> >> > > On production systems with ETMs enabled, it is preferred to
> >>> >> > > exclude kernel mode(NS EL1) tracing for security concerns and
> >>> >> > > support only userspace(NS EL0) tracing. So provide an option
> >>> >> > > via kconfig to exclude kernel mode tracing if it is required.
> >>> >> > > This config is disabled by default and would not affect the
> >>> >> > > current configuration which has both kernel and userspace
> >>> >> > > tracing enabled by default.
> >>> >> > >
> >>> >> >
> >>> >> > One requires root access (or be part of a special trace group)
> >>> >> > to be able to use the cs_etm PMU.  With this kind of elevated
> >>> >> > access restricting tracing at EL1 provides little in terms of security.
> >>> >> >
> >>> >>
> >>> >> Apart from the VM usecase discussed, I am told there are other
> >>> >> security concerns here regarding need to exclude kernel mode
> >>> >> tracing even for the privileged users/root. One such case being
> >>> >> the ability to analyze cryptographic code execution since ETMs
> >>> >> can record all branch instructions including timestamps in the
> >>> >> kernel and there may be other cases as well which I may not be
> >>> >> aware of and hence have added Denis and Mattias. Please let us
> >>> >> know if you have any questions further regarding this not being a security
> concern.
> >>> >
> >>> > Even if we were to apply this patch there are many ways to
> >>> > compromise a system or get the kernel to reveal important
> >>> > information using the perf subsystem.  I would perfer to tackle
> >>> > the problem at that level rather than concentrating on coresight.
> >>> >
> >>>
> >>> Sorry but I did not understand your point. We are talking about the
> >>> capabilities of coresight etm tracing which has the instruction level tracing
> and a lot more.
> >>> Perf subsystem is just the framework used for it.
> >>> In other words, its not the perf subsystem which does instruction
> >>> level tracing, its the coresight etm. Why the perf subsystem should
> >>> be modified to lockdown kernel mode? If we were to let perf handle
> >>> all the trace filtering for different exception levels, then why do
> >>> we need the register settings in coresight etm driver to filter out
> >>> NS EL* tracing? And more importantly, how do you suppose we handle sysfs
> mode of coresight tracing with perf subsystem?
> >>
> >> You both have good points. Mathieu is right that this is not a
> >> CoreSight issue specifically, it is a matter of kernel security
> >> policy, and other hardware tracing mechanisms ought to be within its
> >> scope. There should be a general "anti kernel exfiltration" config
> >> that applies to all mechanisms within its scope, and we'd definitely expect
> that to include Intel PT as well as ETM.
> >>
> >
> > I agree with this part where there should be a generic config for all
> > hardware tracing families(atleast for Intel PT and ARM Coresight),
> > Suzuki suggested that as well. I am under the impression that Mathieu
> > didn't like adding such a config and wanted perf subsystem to handle
> > it since initial discussion was around whether root compromise meant
> > everything is lost already and such a kconfig would not help, but
> > Mattias already gave some good examples where that is not true.
> >
> >> A kernel config that forced exclude_kernel on all perf events would
> >> deal with ETM and PT in one place, but miss the sysfs interface to ETM.
> >>
> >> On the other hand, doing it in the ETM drivers would cover the perf
> >> and sysfs interfaces to ETM, but would miss Intel PT.
> >>
> >> So I think what is needed is a general config option that is both
> >> implemented in perf (excluding all kernel tracing events) and by any
> >> drivers that provide an alternative interface to hardware tracing events.
> >>
> >
> > I am good with this approach, once Mathieu confirms, I can add a
> > kernel wide kconfig as Suzuki suggested earlier and make ETM{3,4}x as
> > the initial users. Someone more familiar with Intel PTs can then make
> > use of this kconfig.
> 
> Instead of adding the support for individual drivers, you could handle this in the
> generic perf layer. e.g, Fail perf_event create with an attribute which allows
> kernel tracing ?
> 
> if (!attr.exclude_kernel)
> 	return -EINVAL;
> 
> Or even exclude the kernel silently always.
> 
> This could also be limited to PMUs with PERF_PMU_CAP_ITRACE, if you want to
> limit this to PMUs that instruction level tracing.

The sysfs interface to ETM also needs to deny access to kernel trace, so it's
safest to enforce it in the drivers in addition to any enforcement done in perf.

Also, forcing exclude_kernel on all perf events may be too strong. Including
the kernel in counted events e.g. cache misses can help understand the effect
of system calls on performance, and isn't a big side channel compared to
userspace event counts. It doesn't reveal detailed timings in the way trace does.

So there's an argument for locking out kernel trace specifically (ETM or PT
on the kernel); or even, for locking out timed trace with timestamps and
cycle counts, and allowing untimed trace. So, that could be done in perf, with
a more specific test on the type of event, before it forced exclude_kernel.

Al


> 
> Suzuki

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

Hi Al,

On 2021-01-19 17:26, Al Grant wrote:
>> From: Suzuki K Poulose <suzuki.poulose@arm.com>
>> On 1/19/21 9:51 AM, Sai Prakash Ranjan wrote:
>> > Hi Al,
>> >
>> > On 2021-01-19 14:06, Al Grant wrote:
>> >> Hi Sai,
>> >>
>> >>> From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org
>> >>> Hi Mathieu,
>> >>>
>> >>> On 2021-01-19 01:53, Mathieu Poirier wrote:
>> >>> > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote:
>> >>> >> Hello Mathieu, Suzuki
>> >>> >>
>> >>> >> On 2020-10-15 21:32, Mathieu Poirier wrote:
>> >>> >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
>> >>> >> > > On production systems with ETMs enabled, it is preferred to
>> >>> >> > > exclude kernel mode(NS EL1) tracing for security concerns and
>> >>> >> > > support only userspace(NS EL0) tracing. So provide an option
>> >>> >> > > via kconfig to exclude kernel mode tracing if it is required.
>> >>> >> > > This config is disabled by default and would not affect the
>> >>> >> > > current configuration which has both kernel and userspace
>> >>> >> > > tracing enabled by default.
>> >>> >> > >
>> >>> >> >
>> >>> >> > One requires root access (or be part of a special trace group)
>> >>> >> > to be able to use the cs_etm PMU.  With this kind of elevated
>> >>> >> > access restricting tracing at EL1 provides little in terms of security.
>> >>> >> >
>> >>> >>
>> >>> >> Apart from the VM usecase discussed, I am told there are other
>> >>> >> security concerns here regarding need to exclude kernel mode
>> >>> >> tracing even for the privileged users/root. One such case being
>> >>> >> the ability to analyze cryptographic code execution since ETMs
>> >>> >> can record all branch instructions including timestamps in the
>> >>> >> kernel and there may be other cases as well which I may not be
>> >>> >> aware of and hence have added Denis and Mattias. Please let us
>> >>> >> know if you have any questions further regarding this not being a security
>> concern.
>> >>> >
>> >>> > Even if we were to apply this patch there are many ways to
>> >>> > compromise a system or get the kernel to reveal important
>> >>> > information using the perf subsystem.  I would perfer to tackle
>> >>> > the problem at that level rather than concentrating on coresight.
>> >>> >
>> >>>
>> >>> Sorry but I did not understand your point. We are talking about the
>> >>> capabilities of coresight etm tracing which has the instruction level tracing
>> and a lot more.
>> >>> Perf subsystem is just the framework used for it.
>> >>> In other words, its not the perf subsystem which does instruction
>> >>> level tracing, its the coresight etm. Why the perf subsystem should
>> >>> be modified to lockdown kernel mode? If we were to let perf handle
>> >>> all the trace filtering for different exception levels, then why do
>> >>> we need the register settings in coresight etm driver to filter out
>> >>> NS EL* tracing? And more importantly, how do you suppose we handle sysfs
>> mode of coresight tracing with perf subsystem?
>> >>
>> >> You both have good points. Mathieu is right that this is not a
>> >> CoreSight issue specifically, it is a matter of kernel security
>> >> policy, and other hardware tracing mechanisms ought to be within its
>> >> scope. There should be a general "anti kernel exfiltration" config
>> >> that applies to all mechanisms within its scope, and we'd definitely expect
>> that to include Intel PT as well as ETM.
>> >>
>> >
>> > I agree with this part where there should be a generic config for all
>> > hardware tracing families(atleast for Intel PT and ARM Coresight),
>> > Suzuki suggested that as well. I am under the impression that Mathieu
>> > didn't like adding such a config and wanted perf subsystem to handle
>> > it since initial discussion was around whether root compromise meant
>> > everything is lost already and such a kconfig would not help, but
>> > Mattias already gave some good examples where that is not true.
>> >
>> >> A kernel config that forced exclude_kernel on all perf events would
>> >> deal with ETM and PT in one place, but miss the sysfs interface to ETM.
>> >>
>> >> On the other hand, doing it in the ETM drivers would cover the perf
>> >> and sysfs interfaces to ETM, but would miss Intel PT.
>> >>
>> >> So I think what is needed is a general config option that is both
>> >> implemented in perf (excluding all kernel tracing events) and by any
>> >> drivers that provide an alternative interface to hardware tracing events.
>> >>
>> >
>> > I am good with this approach, once Mathieu confirms, I can add a
>> > kernel wide kconfig as Suzuki suggested earlier and make ETM{3,4}x as
>> > the initial users. Someone more familiar with Intel PTs can then make
>> > use of this kconfig.
>> 
>> Instead of adding the support for individual drivers, you could handle 
>> this in the
>> generic perf layer. e.g, Fail perf_event create with an attribute 
>> which allows
>> kernel tracing ?
>> 
>> if (!attr.exclude_kernel)
>> 	return -EINVAL;
>> 
>> Or even exclude the kernel silently always.
>> 
>> This could also be limited to PMUs with PERF_PMU_CAP_ITRACE, if you 
>> want to
>> limit this to PMUs that instruction level tracing.
> 
> The sysfs interface to ETM also needs to deny access to kernel trace, 
> so it's
> safest to enforce it in the drivers in addition to any enforcement done 
> in perf.
> 

Yes, it will be done in drivers for sysfs interface as well based on the
same kconfig.

> Also, forcing exclude_kernel on all perf events may be too strong. 
> Including
> the kernel in counted events e.g. cache misses can help understand the 
> effect
> of system calls on performance, and isn't a big side channel compared 
> to
> userspace event counts. It doesn't reveal detailed timings in the way
> trace does.
> 
> So there's an argument for locking out kernel trace specifically (ETM 
> or PT
> on the kernel); or even, for locking out timed trace with timestamps 
> and
> cycle counts, and allowing untimed trace. So, that could be done in 
> perf, with
> a more specific test on the type of event, before it forced 
> exclude_kernel.
> 

Yes exclude_kernel for all events might not be possible, so it would be
better if it is initially applied for PMUs with PERF_PMU_CAP_ITRACE as
Suzuki suggested.

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

Hi Suzuki,

On 2021-01-19 16:03, Suzuki K Poulose wrote:
> On 1/19/21 9:51 AM, Sai Prakash Ranjan wrote:
>> Hi Al,
>> 
>> On 2021-01-19 14:06, Al Grant wrote:
>>> Hi Sai,
>>> 
>>>> From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org
>>>> Hi Mathieu,
>>>> 
>>>> On 2021-01-19 01:53, Mathieu Poirier wrote:
>>>> > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote:
>>>> >> Hello Mathieu, Suzuki
>>>> >>
>>>> >> On 2020-10-15 21:32, Mathieu Poirier wrote:
>>>> >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
>>>> >> > > On production systems with ETMs enabled, it is preferred to
>>>> >> > > exclude kernel mode(NS EL1) tracing for security concerns and
>>>> >> > > support only userspace(NS EL0) tracing. So provide an option via
>>>> >> > > kconfig to exclude kernel mode tracing if it is required.
>>>> >> > > This config is disabled by default and would not affect the
>>>> >> > > current configuration which has both kernel and userspace tracing
>>>> >> > > enabled by default.
>>>> >> > >
>>>> >> >
>>>> >> > One requires root access (or be part of a special trace group) to
>>>> >> > be able to use the cs_etm PMU.  With this kind of elevated access
>>>> >> > restricting tracing at EL1 provides little in terms of security.
>>>> >> >
>>>> >>
>>>> >> Apart from the VM usecase discussed, I am told there are other
>>>> >> security concerns here regarding need to exclude kernel mode tracing
>>>> >> even for the privileged users/root. One such case being the ability
>>>> >> to analyze cryptographic code execution since ETMs can record all
>>>> >> branch instructions including timestamps in the kernel and there may
>>>> >> be other cases as well which I may not be aware of and hence have
>>>> >> added Denis and Mattias. Please let us know if you have any questions
>>>> >> further regarding this not being a security concern.
>>>> >
>>>> > Even if we were to apply this patch there are many ways to compromise
>>>> > a system or get the kernel to reveal important information using the
>>>> > perf subsystem.  I would perfer to tackle the problem at that level
>>>> > rather than concentrating on coresight.
>>>> >
>>>> 
>>>> Sorry but I did not understand your point. We are talking about the 
>>>> capabilities
>>>> of coresight etm tracing which has the instruction level tracing and 
>>>> a lot more.
>>>> Perf subsystem is just the framework used for it.
>>>> In other words, its not the perf subsystem which does instruction 
>>>> level tracing,
>>>> its the coresight etm. Why the perf subsystem should be modified to 
>>>> lockdown
>>>> kernel mode? If we were to let perf handle all the trace filtering 
>>>> for different
>>>> exception levels, then why do we need the register settings in 
>>>> coresight etm
>>>> driver to filter out NS EL* tracing? And more importantly, how do 
>>>> you suppose
>>>> we handle sysfs mode of coresight tracing with perf subsystem?
>>> 
>>> You both have good points. Mathieu is right that this is not a 
>>> CoreSight
>>> issue specifically, it is a matter of kernel security policy, and 
>>> other hardware
>>> tracing mechanisms ought to be within its scope. There should be a 
>>> general
>>> "anti kernel exfiltration" config that applies to all mechanisms 
>>> within
>>> its scope, and we'd definitely expect that to include Intel PT as 
>>> well as ETM.
>>> 
>> 
>> I agree with this part where there should be a generic config for all
>> hardware tracing families(atleast for Intel PT and ARM Coresight),
>> Suzuki suggested that as well. I am under the impression that Mathieu
>> didn't like adding such a config and wanted perf subsystem to handle
>> it since initial discussion was around whether root compromise meant
>> everything is lost already and such a kconfig would not help, but
>> Mattias already gave some good examples where that is not true.
>> 
>>> A kernel config that forced exclude_kernel on all perf events would 
>>> deal with
>>> ETM and PT in one place, but miss the sysfs interface to ETM.
>>> 
>>> On the other hand, doing it in the ETM drivers would cover the perf 
>>> and sysfs
>>> interfaces to ETM, but would miss Intel PT.
>>> 
>>> So I think what is needed is a general config option that is both 
>>> implemented
>>> in perf (excluding all kernel tracing events) and by any drivers that 
>>> provide
>>> an alternative interface to hardware tracing events.
>>> 
>> 
>> I am good with this approach, once Mathieu confirms, I can add a 
>> kernel
>> wide kconfig as Suzuki suggested earlier and make ETM{3,4}x as the
>> initial users. Someone more familiar with Intel PTs can then make use
>> of this kconfig.
> 
> Instead of adding the support for individual drivers, you could handle 
> this
> in the generic perf layer. e.g, Fail perf_event create with an 
> attribute
> which allows kernel tracing ?
> 
> if (!attr.exclude_kernel)
> 	return -EINVAL;
> 
> Or even exclude the kernel silently always.
> 
> This could also be limited to PMUs with PERF_PMU_CAP_ITRACE, if you
> want to limit this to PMUs that instruction level tracing.
> 

Ah nice, wasn't aware of such a flag for instruction level tracing.
This sounds really good to me, I will use this in generic perf layer
and will have a config EXCLUDE_KERNEL_HW_ITRACE as you suggested
earlier.

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Mathieu Poirier]

On Tue, Jan 19, 2021 at 08:36:22AM +0000, Al Grant wrote:
> Hi Sai,
> 
> > From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org
> > Hi Mathieu,
> > 
> > On 2021-01-19 01:53, Mathieu Poirier wrote:
> > > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote:
> > >> Hello Mathieu, Suzuki
> > >>
> > >> On 2020-10-15 21:32, Mathieu Poirier wrote:
> > >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
> > >> > > On production systems with ETMs enabled, it is preferred to
> > >> > > exclude kernel mode(NS EL1) tracing for security concerns and
> > >> > > support only userspace(NS EL0) tracing. So provide an option via
> > >> > > kconfig to exclude kernel mode tracing if it is required.
> > >> > > This config is disabled by default and would not affect the
> > >> > > current configuration which has both kernel and userspace tracing
> > >> > > enabled by default.
> > >> > >
> > >> >
> > >> > One requires root access (or be part of a special trace group) to
> > >> > be able to use the cs_etm PMU.  With this kind of elevated access
> > >> > restricting tracing at EL1 provides little in terms of security.
> > >> >
> > >>
> > >> Apart from the VM usecase discussed, I am told there are other
> > >> security concerns here regarding need to exclude kernel mode tracing
> > >> even for the privileged users/root. One such case being the ability
> > >> to analyze cryptographic code execution since ETMs can record all
> > >> branch instructions including timestamps in the kernel and there may
> > >> be other cases as well which I may not be aware of and hence have
> > >> added Denis and Mattias. Please let us know if you have any questions
> > >> further regarding this not being a security concern.
> > >
> > > Even if we were to apply this patch there are many ways to compromise
> > > a system or get the kernel to reveal important information using the
> > > perf subsystem.  I would perfer to tackle the problem at that level
> > > rather than concentrating on coresight.
> > >
> > 
> > Sorry but I did not understand your point. We are talking about the capabilities
> > of coresight etm tracing which has the instruction level tracing and a lot more.
> > Perf subsystem is just the framework used for it.
> > In other words, its not the perf subsystem which does instruction level tracing,
> > its the coresight etm. Why the perf subsystem should be modified to lockdown
> > kernel mode? If we were to let perf handle all the trace filtering for different
> > exception levels, then why do we need the register settings in coresight etm
> > driver to filter out NS EL* tracing? And more importantly, how do you suppose
> > we handle sysfs mode of coresight tracing with perf subsystem?
> 
> You both have good points. Mathieu is right that this is not a CoreSight
> issue specifically, it is a matter of kernel security policy, and other hardware
> tracing mechanisms ought to be within its scope. There should be a general
> "anti kernel exfiltration" config that applies to all mechanisms within
> its scope, and we'd definitely expect that to include Intel PT as well as ETM.
> 
> A kernel config that forced exclude_kernel on all perf events would deal with
> ETM and PT in one place, but miss the sysfs interface to ETM.
> 
> On the other hand, doing it in the ETM drivers would cover the perf and sysfs
> interfaces to ETM, but would miss Intel PT.
> 
> So I think what is needed is a general config option that is both implemented
> in perf (excluding all kernel tracing events) and by any drivers that provide
> an alternative interface to hardware tracing events.
>

I also think this is the right solution.

Thanks,
Mathieu
 
> Al
> 
> 
> > 
> > Thanks,
> > Sai
> > 
> > --
> > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
> > of Code Aurora Forum, hosted by The Linux Foundation

Re: [PATCH] coresight: etm4x: Add config to exclude kernel mode tracing

[Sai Prakash Ranjan]

On 2021-01-21 00:18, Mathieu Poirier wrote:
> On Tue, Jan 19, 2021 at 08:36:22AM +0000, Al Grant wrote:
>> Hi Sai,
>> 
>> > From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org
>> > Hi Mathieu,
>> >
>> > On 2021-01-19 01:53, Mathieu Poirier wrote:
>> > > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote:
>> > >> Hello Mathieu, Suzuki
>> > >>
>> > >> On 2020-10-15 21:32, Mathieu Poirier wrote:
>> > >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote:
>> > >> > > On production systems with ETMs enabled, it is preferred to
>> > >> > > exclude kernel mode(NS EL1) tracing for security concerns and
>> > >> > > support only userspace(NS EL0) tracing. So provide an option via
>> > >> > > kconfig to exclude kernel mode tracing if it is required.
>> > >> > > This config is disabled by default and would not affect the
>> > >> > > current configuration which has both kernel and userspace tracing
>> > >> > > enabled by default.
>> > >> > >
>> > >> >
>> > >> > One requires root access (or be part of a special trace group) to
>> > >> > be able to use the cs_etm PMU.  With this kind of elevated access
>> > >> > restricting tracing at EL1 provides little in terms of security.
>> > >> >
>> > >>
>> > >> Apart from the VM usecase discussed, I am told there are other
>> > >> security concerns here regarding need to exclude kernel mode tracing
>> > >> even for the privileged users/root. One such case being the ability
>> > >> to analyze cryptographic code execution since ETMs can record all
>> > >> branch instructions including timestamps in the kernel and there may
>> > >> be other cases as well which I may not be aware of and hence have
>> > >> added Denis and Mattias. Please let us know if you have any questions
>> > >> further regarding this not being a security concern.
>> > >
>> > > Even if we were to apply this patch there are many ways to compromise
>> > > a system or get the kernel to reveal important information using the
>> > > perf subsystem.  I would perfer to tackle the problem at that level
>> > > rather than concentrating on coresight.
>> > >
>> >
>> > Sorry but I did not understand your point. We are talking about the capabilities
>> > of coresight etm tracing which has the instruction level tracing and a lot more.
>> > Perf subsystem is just the framework used for it.
>> > In other words, its not the perf subsystem which does instruction level tracing,
>> > its the coresight etm. Why the perf subsystem should be modified to lockdown
>> > kernel mode? If we were to let perf handle all the trace filtering for different
>> > exception levels, then why do we need the register settings in coresight etm
>> > driver to filter out NS EL* tracing? And more importantly, how do you suppose
>> > we handle sysfs mode of coresight tracing with perf subsystem?
>> 
>> You both have good points. Mathieu is right that this is not a 
>> CoreSight
>> issue specifically, it is a matter of kernel security policy, and 
>> other hardware
>> tracing mechanisms ought to be within its scope. There should be a 
>> general
>> "anti kernel exfiltration" config that applies to all mechanisms 
>> within
>> its scope, and we'd definitely expect that to include Intel PT as well 
>> as ETM.
>> 
>> A kernel config that forced exclude_kernel on all perf events would 
>> deal with
>> ETM and PT in one place, but miss the sysfs interface to ETM.
>> 
>> On the other hand, doing it in the ETM drivers would cover the perf 
>> and sysfs
>> interfaces to ETM, but would miss Intel PT.
>> 
>> So I think what is needed is a general config option that is both 
>> implemented
>> in perf (excluding all kernel tracing events) and by any drivers that 
>> provide
>> an alternative interface to hardware tracing events.
>> 
> 
> I also think this is the right solution.
> 

Thanks for confirming, I will be working on this suggestion.

Thanks,
Sai

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member
of Code Aurora Forum, hosted by The Linux Foundation