[RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

Hi,

Currently, the PCI subsystem marks the PCI devices as "untrusted", if
the firmware asks it to:

617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
9cb30a71acd4 ("PCI: OF: Support "external-facing" property")

An "untrusted" device indicates a (likely external facing) device that
may be malicious, and can trigger DMA attacks on the system. It may
also try to exploit any vulnerabilities exposed by the driver, that
may allow it to read/write unintended addresses in the host (e.g. if
DMA buffers for the device, share memory pages with other driver data
structures or code etc).

High Level proposal
===============
Currently, the "untrusted" device property is used as a hint to enable
IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
go a step further, and allow the administrator to build a list of
whitelisted drivers for these "untrusted" devices. This whitelist of
drivers are the ones that he trusts enough to have little or no
vulnerabilities. (He may have built this list of whitelisted drivers
by a combination of code analysis of drivers, or by extensive testing
using PCIe fuzzing etc). We propose that the administrator be allowed
to specify this list of whitelisted drivers to the kernel, and the PCI
subsystem to impose this behavior:

1) The "untrusted" devices can bind to only "whitelisted drivers".
2) The other devices (i.e. dev->untrusted=0) can bind to any driver.

Of course this behavior is to be imposed only if such a whitelist is
provided by the administrator.

Details
======

1) A kernel argument ("pci.impose_driver_whitelisting") to enable
imposing of whitelisting by PCI subsystem.

2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
the driver is whitelisted.

3) Use the driver's "whitelisted" flag and the device's "untrusted"
flag, to make a decision about whether to bind or not in
pci_bus_match() or similar.

4) A mechanism to allow the administrator to specify the whitelist of
drivers. I think this needs more thought as there are multiple
options.

a) Expose individual driver's "whitelisted" flag to userspace so a
boot script can whitelist that driver. There are questions that still
need answered though e.g. what to do about the devices that may have
already been enumerated and rejected by then? What to do with the
already bound devices, if the user changes a driver to remove it from
the whitelist. etc.

      b) Provide a way to specify the whitelist via the kernel command
line. Accept a ("pci.whitelist") kernel parameter which is a comma
separated list of driver names (just like "module_blacklist"), and
then use it to initialize each driver's "whitelisted" flag as the
drivers are registered. Essentially this would mean that the whitelist
of devices cannot be changed after boot.

To me (b) looks a better option but I think a future requirement would
be the ability to remove the drivers from the whitelist after boot
(adding drivers to whitelist at runtime may not be that critical IMO)

 WDYT?

Thanks,

Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Jean-Philippe Brucker]

Hi,

On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> Hi,
> 
> Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> the firmware asks it to:
> 
> 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> 
> An "untrusted" device indicates a (likely external facing) device that
> may be malicious, and can trigger DMA attacks on the system. It may
> also try to exploit any vulnerabilities exposed by the driver, that
> may allow it to read/write unintended addresses in the host (e.g. if
> DMA buffers for the device, share memory pages with other driver data
> structures or code etc).
> 
> High Level proposal
> ===============
> Currently, the "untrusted" device property is used as a hint to enable
> IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> go a step further, and allow the administrator to build a list of
> whitelisted drivers for these "untrusted" devices.

How about letting the administrator whitelist devices that are trusted,
rather than whitelisting drivers?

The "thunderclap" attack [1] emulates an existing device using an FPGA in
order to get probed by the device driver, and then bypasses a weakened
IOMMU. By design the driver cannot differentiate a well-behaved device
from a malicious one, so changing the trust level of the driver doesn't
feel like the right way. What the admin wants to say is "I trust this
port, no one is plugging any malicious device in here."

Then you could also make the option 3-way: either keep the default trust
fixed by FW, or manually set "trusted" or "untrusted".

For reference there have been several discussions, recently, about letting
admins change IOMMU configuration for a device. A PCI command-line option
[2] was suggested, but I think the current proposal is a sysfs knob on
IOMMU groups [3], that can be changed while devices are unbound from
drivers. It's not completely relevant since the "untrusted" property isn't
tied to the IOMMU subsystem, but seemed worth mentioning.

[1] https://thunderclap.io/thunderclap-paper-ndss2019.pdf
[2] https://lore.kernel.org/linux-iommu/20200101052648.14295-3-baolu.lu@linux.intel.com/
[3] https://lore.kernel.org/linux-iommu/5aa5ef20ff81f706aafa9a6af68cef98fe60ad0f.1581619464.git.sai.praneeth.prakhya@intel.com/

Thanks,
Jean

> This whitelist of
> drivers are the ones that he trusts enough to have little or no
> vulnerabilities. (He may have built this list of whitelisted drivers
> by a combination of code analysis of drivers, or by extensive testing
> using PCIe fuzzing etc). We propose that the administrator be allowed
> to specify this list of whitelisted drivers to the kernel, and the PCI
> subsystem to impose this behavior:
> 
> 1) The "untrusted" devices can bind to only "whitelisted drivers".
> 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> 
> Of course this behavior is to be imposed only if such a whitelist is
> provided by the administrator.
> 
> Details
> ======
> 
> 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> imposing of whitelisting by PCI subsystem.
> 
> 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> the driver is whitelisted.
> 
> 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> flag, to make a decision about whether to bind or not in
> pci_bus_match() or similar.
> 
> 4) A mechanism to allow the administrator to specify the whitelist of
> drivers. I think this needs more thought as there are multiple
> options.
> 
> a) Expose individual driver's "whitelisted" flag to userspace so a
> boot script can whitelist that driver. There are questions that still
> need answered though e.g. what to do about the devices that may have
> already been enumerated and rejected by then? What to do with the
> already bound devices, if the user changes a driver to remove it from
> the whitelist. etc.
> 
>       b) Provide a way to specify the whitelist via the kernel command
> line. Accept a ("pci.whitelist") kernel parameter which is a comma
> separated list of driver names (just like "module_blacklist"), and
> then use it to initialize each driver's "whitelisted" flag as the
> drivers are registered. Essentially this would mean that the whitelist
> of devices cannot be changed after boot.
> 
> To me (b) looks a better option but I think a future requirement would
> be the ability to remove the drivers from the whitelist after boot
> (adding drivers to whitelist at runtime may not be that critical IMO)
> 
>  WDYT?
> 
> Thanks,
> 
> Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Jean-Philippe Brucker]

On Mon, May 04, 2020 at 01:47:27PM +0200, Jean-Philippe Brucker wrote:
> Hi,
> 
> On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > Hi,
> > 
> > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > the firmware asks it to:
> > 
> > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> > 
> > An "untrusted" device indicates a (likely external facing) device that
> > may be malicious, and can trigger DMA attacks on the system. It may
> > also try to exploit any vulnerabilities exposed by the driver, that
> > may allow it to read/write unintended addresses in the host (e.g. if
> > DMA buffers for the device, share memory pages with other driver data
> > structures or code etc).
> > 
> > High Level proposal
> > ===============
> > Currently, the "untrusted" device property is used as a hint to enable
> > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > go a step further, and allow the administrator to build a list of
> > whitelisted drivers for these "untrusted" devices.
> 
> How about letting the administrator whitelist devices that are trusted,
> rather than whitelisting drivers?

Uh, I completely missed the point. Your proposal is about preventing from
binding any untrusted devices to non-whitelisted drivers. Please disregard
my reply :)

Thanks,
Jean

> 
> The "thunderclap" attack [1] emulates an existing device using an FPGA in
> order to get probed by the device driver, and then bypasses a weakened
> IOMMU. By design the driver cannot differentiate a well-behaved device
> from a malicious one, so changing the trust level of the driver doesn't
> feel like the right way. What the admin wants to say is "I trust this
> port, no one is plugging any malicious device in here."
> 
> Then you could also make the option 3-way: either keep the default trust
> fixed by FW, or manually set "trusted" or "untrusted".
> 
> For reference there have been several discussions, recently, about letting
> admins change IOMMU configuration for a device. A PCI command-line option
> [2] was suggested, but I think the current proposal is a sysfs knob on
> IOMMU groups [3], that can be changed while devices are unbound from
> drivers. It's not completely relevant since the "untrusted" property isn't
> tied to the IOMMU subsystem, but seemed worth mentioning.
> 
> [1] https://thunderclap.io/thunderclap-paper-ndss2019.pdf
> [2] https://lore.kernel.org/linux-iommu/20200101052648.14295-3-baolu.lu@linux.intel.com/
> [3] https://lore.kernel.org/linux-iommu/5aa5ef20ff81f706aafa9a6af68cef98fe60ad0f.1581619464.git.sai.praneeth.prakhya@intel.com/
> 
> Thanks,
> Jean
> 
> > This whitelist of
> > drivers are the ones that he trusts enough to have little or no
> > vulnerabilities. (He may have built this list of whitelisted drivers
> > by a combination of code analysis of drivers, or by extensive testing
> > using PCIe fuzzing etc). We propose that the administrator be allowed
> > to specify this list of whitelisted drivers to the kernel, and the PCI
> > subsystem to impose this behavior:
> > 
> > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> > 
> > Of course this behavior is to be imposed only if such a whitelist is
> > provided by the administrator.
> > 
> > Details
> > ======
> > 
> > 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> > imposing of whitelisting by PCI subsystem.
> > 
> > 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> > the driver is whitelisted.
> > 
> > 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> > flag, to make a decision about whether to bind or not in
> > pci_bus_match() or similar.
> > 
> > 4) A mechanism to allow the administrator to specify the whitelist of
> > drivers. I think this needs more thought as there are multiple
> > options.
> > 
> > a) Expose individual driver's "whitelisted" flag to userspace so a
> > boot script can whitelist that driver. There are questions that still
> > need answered though e.g. what to do about the devices that may have
> > already been enumerated and rejected by then? What to do with the
> > already bound devices, if the user changes a driver to remove it from
> > the whitelist. etc.
> > 
> >       b) Provide a way to specify the whitelist via the kernel command
> > line. Accept a ("pci.whitelist") kernel parameter which is a comma
> > separated list of driver names (just like "module_blacklist"), and
> > then use it to initialize each driver's "whitelisted" flag as the
> > drivers are registered. Essentially this would mean that the whitelist
> > of devices cannot be changed after boot.
> > 
> > To me (b) looks a better option but I think a future requirement would
> > be the ability to remove the drivers from the whitelist after boot
> > (adding drivers to whitelist at runtime may not be that critical IMO)
> > 
> >  WDYT?
> > 
> > Thanks,
> > 
> > Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

On Mon, May 4, 2020 at 4:59 AM Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> On Mon, May 04, 2020 at 01:47:27PM +0200, Jean-Philippe Brucker wrote:
> > Hi,
> >
> > On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > > Hi,
> > >
> > > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > > the firmware asks it to:
> > >
> > > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> > >
> > > An "untrusted" device indicates a (likely external facing) device that
> > > may be malicious, and can trigger DMA attacks on the system. It may
> > > also try to exploit any vulnerabilities exposed by the driver, that
> > > may allow it to read/write unintended addresses in the host (e.g. if
> > > DMA buffers for the device, share memory pages with other driver data
> > > structures or code etc).
> > >
> > > High Level proposal
> > > ===============
> > > Currently, the "untrusted" device property is used as a hint to enable
> > > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > > go a step further, and allow the administrator to build a list of
> > > whitelisted drivers for these "untrusted" devices.
> >
> > How about letting the administrator whitelist devices that are trusted,
> > rather than whitelisting drivers?
>
> Uh, I completely missed the point. Your proposal is about preventing from
> binding any untrusted devices to non-whitelisted drivers. Please disregard
> my reply :)

Yes, my proposal is about ensuring untrusted devices can be bound to
only trusted drivers (if the administrator so desires).

Thanks for the links though - I think they may be additionally useful
for what we're trying to do.

Thanks,

Rajat


>
> Thanks,
> Jean
>
> >
> > The "thunderclap" attack [1] emulates an existing device using an FPGA in
> > order to get probed by the device driver, and then bypasses a weakened
> > IOMMU. By design the driver cannot differentiate a well-behaved device
> > from a malicious one, so changing the trust level of the driver doesn't
> > feel like the right way. What the admin wants to say is "I trust this
> > port, no one is plugging any malicious device in here."
> >
> > Then you could also make the option 3-way: either keep the default trust
> > fixed by FW, or manually set "trusted" or "untrusted".
> >
> > For reference there have been several discussions, recently, about letting
> > admins change IOMMU configuration for a device. A PCI command-line option
> > [2] was suggested, but I think the current proposal is a sysfs knob on
> > IOMMU groups [3], that can be changed while devices are unbound from
> > drivers. It's not completely relevant since the "untrusted" property isn't
> > tied to the IOMMU subsystem, but seemed worth mentioning.
> >
> > [1] https://thunderclap.io/thunderclap-paper-ndss2019.pdf
> > [2] https://lore.kernel.org/linux-iommu/20200101052648.14295-3-baolu.lu@linux.intel.com/
> > [3] https://lore.kernel.org/linux-iommu/5aa5ef20ff81f706aafa9a6af68cef98fe60ad0f.1581619464.git.sai.praneeth.prakhya@intel.com/
> >
> > Thanks,
> > Jean
> >
> > > This whitelist of
> > > drivers are the ones that he trusts enough to have little or no
> > > vulnerabilities. (He may have built this list of whitelisted drivers
> > > by a combination of code analysis of drivers, or by extensive testing
> > > using PCIe fuzzing etc). We propose that the administrator be allowed
> > > to specify this list of whitelisted drivers to the kernel, and the PCI
> > > subsystem to impose this behavior:
> > >
> > > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> > >
> > > Of course this behavior is to be imposed only if such a whitelist is
> > > provided by the administrator.
> > >
> > > Details
> > > ======
> > >
> > > 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> > > imposing of whitelisting by PCI subsystem.
> > >
> > > 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> > > the driver is whitelisted.
> > >
> > > 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> > > flag, to make a decision about whether to bind or not in
> > > pci_bus_match() or similar.
> > >
> > > 4) A mechanism to allow the administrator to specify the whitelist of
> > > drivers. I think this needs more thought as there are multiple
> > > options.
> > >
> > > a) Expose individual driver's "whitelisted" flag to userspace so a
> > > boot script can whitelist that driver. There are questions that still
> > > need answered though e.g. what to do about the devices that may have
> > > already been enumerated and rejected by then? What to do with the
> > > already bound devices, if the user changes a driver to remove it from
> > > the whitelist. etc.
> > >
> > >       b) Provide a way to specify the whitelist via the kernel command
> > > line. Accept a ("pci.whitelist") kernel parameter which is a comma
> > > separated list of driver names (just like "module_blacklist"), and
> > > then use it to initialize each driver's "whitelisted" flag as the
> > > drivers are registered. Essentially this would mean that the whitelist
> > > of devices cannot be changed after boot.
> > >
> > > To me (b) looks a better option but I think a future requirement would
> > > be the ability to remove the drivers from the whitelist after boot
> > > (adding drivers to whitelist at runtime may not be that critical IMO)
> > >
> > >  WDYT?
> > >
> > > Thanks,
> > >
> > > Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Mika Westerberg]

On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> Hi,

Hi,

> Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> the firmware asks it to:
> 
> 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> 
> An "untrusted" device indicates a (likely external facing) device that
> may be malicious, and can trigger DMA attacks on the system. It may
> also try to exploit any vulnerabilities exposed by the driver, that
> may allow it to read/write unintended addresses in the host (e.g. if
> DMA buffers for the device, share memory pages with other driver data
> structures or code etc).
> 
> High Level proposal
> ===============
> Currently, the "untrusted" device property is used as a hint to enable
> IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> go a step further, and allow the administrator to build a list of
> whitelisted drivers for these "untrusted" devices. This whitelist of
> drivers are the ones that he trusts enough to have little or no
> vulnerabilities. (He may have built this list of whitelisted drivers
> by a combination of code analysis of drivers, or by extensive testing
> using PCIe fuzzing etc). We propose that the administrator be allowed
> to specify this list of whitelisted drivers to the kernel, and the PCI
> subsystem to impose this behavior:
> 
> 1) The "untrusted" devices can bind to only "whitelisted drivers".
> 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> 
> Of course this behavior is to be imposed only if such a whitelist is
> provided by the administrator.
> 
> Details
> ======
> 
> 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> imposing of whitelisting by PCI subsystem.

In addition this could be a Kconfig option, I think.

> 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> the driver is whitelisted.
> 
> 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> flag, to make a decision about whether to bind or not in
> pci_bus_match() or similar.
> 
> 4) A mechanism to allow the administrator to specify the whitelist of
> drivers. I think this needs more thought as there are multiple
> options.
> 
> a) Expose individual driver's "whitelisted" flag to userspace so a
> boot script can whitelist that driver. There are questions that still
> need answered though e.g. what to do about the devices that may have
> already been enumerated and rejected by then? What to do with the
> already bound devices, if the user changes a driver to remove it from
> the whitelist. etc.
> 
>       b) Provide a way to specify the whitelist via the kernel command
> line. Accept a ("pci.whitelist") kernel parameter which is a comma
> separated list of driver names (just like "module_blacklist"), and
> then use it to initialize each driver's "whitelisted" flag as the
> drivers are registered. Essentially this would mean that the whitelist
> of devices cannot be changed after boot.
> 
> To me (b) looks a better option but I think a future requirement would
> be the ability to remove the drivers from the whitelist after boot
> (adding drivers to whitelist at runtime may not be that critical IMO)

What about adding "module.whitelisted" parameter in the same way than we
have for example "module.dyndbg"? Then you can pass these in the command
line, during module load time and also trough /sys/module/.

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

On Tue, May 5, 2020 at 5:33 AM Mika Westerberg
<mika.westerberg@linux.intel.com> wrote:
>
> On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > Hi,
>
> Hi,
>
> > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > the firmware asks it to:
> >
> > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> >
> > An "untrusted" device indicates a (likely external facing) device that
> > may be malicious, and can trigger DMA attacks on the system. It may
> > also try to exploit any vulnerabilities exposed by the driver, that
> > may allow it to read/write unintended addresses in the host (e.g. if
> > DMA buffers for the device, share memory pages with other driver data
> > structures or code etc).
> >
> > High Level proposal
> > ===============
> > Currently, the "untrusted" device property is used as a hint to enable
> > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > go a step further, and allow the administrator to build a list of
> > whitelisted drivers for these "untrusted" devices. This whitelist of
> > drivers are the ones that he trusts enough to have little or no
> > vulnerabilities. (He may have built this list of whitelisted drivers
> > by a combination of code analysis of drivers, or by extensive testing
> > using PCIe fuzzing etc). We propose that the administrator be allowed
> > to specify this list of whitelisted drivers to the kernel, and the PCI
> > subsystem to impose this behavior:
> >
> > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> >
> > Of course this behavior is to be imposed only if such a whitelist is
> > provided by the administrator.
> >
> > Details
> > ======
> >
> > 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> > imposing of whitelisting by PCI subsystem.
>
> In addition this could be a Kconfig option, I think.
>
> > 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> > the driver is whitelisted.
> >
> > 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> > flag, to make a decision about whether to bind or not in
> > pci_bus_match() or similar.
> >
> > 4) A mechanism to allow the administrator to specify the whitelist of
> > drivers. I think this needs more thought as there are multiple
> > options.
> >
> > a) Expose individual driver's "whitelisted" flag to userspace so a
> > boot script can whitelist that driver. There are questions that still
> > need answered though e.g. what to do about the devices that may have
> > already been enumerated and rejected by then? What to do with the
> > already bound devices, if the user changes a driver to remove it from
> > the whitelist. etc.
> >
> >       b) Provide a way to specify the whitelist via the kernel command
> > line. Accept a ("pci.whitelist") kernel parameter which is a comma
> > separated list of driver names (just like "module_blacklist"), and
> > then use it to initialize each driver's "whitelisted" flag as the
> > drivers are registered. Essentially this would mean that the whitelist
> > of devices cannot be changed after boot.
> >
> > To me (b) looks a better option but I think a future requirement would
> > be the ability to remove the drivers from the whitelist after boot
> > (adding drivers to whitelist at runtime may not be that critical IMO)
>
> What about adding "module.whitelisted" parameter in the same way than we
> have for example "module.dyndbg"? Then you can pass these in the command
> line, during module load time and also trough /sys/module/.

Sure we can do that.

However, I suspect ability to add to the whitelist after booting up
might be frowned upon. May be if we do that, we can somehow ensure
that the drivers can only be removed from the whitelist, and not
added.

Thanks,

Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

Hi Bjorn,

On Fri, May 1, 2020 at 4:07 PM Rajat Jain <rajatja@google.com> wrote:
>
> Hi,
>
> Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> the firmware asks it to:
>
> 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
>
> An "untrusted" device indicates a (likely external facing) device that
> may be malicious, and can trigger DMA attacks on the system. It may
> also try to exploit any vulnerabilities exposed by the driver, that
> may allow it to read/write unintended addresses in the host (e.g. if
> DMA buffers for the device, share memory pages with other driver data
> structures or code etc).
>
> High Level proposal
> ===============
> Currently, the "untrusted" device property is used as a hint to enable
> IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> go a step further, and allow the administrator to build a list of
> whitelisted drivers for these "untrusted" devices. This whitelist of
> drivers are the ones that he trusts enough to have little or no
> vulnerabilities. (He may have built this list of whitelisted drivers
> by a combination of code analysis of drivers, or by extensive testing
> using PCIe fuzzing etc). We propose that the administrator be allowed
> to specify this list of whitelisted drivers to the kernel, and the PCI
> subsystem to impose this behavior:
>
> 1) The "untrusted" devices can bind to only "whitelisted drivers".
> 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
>
> Of course this behavior is to be imposed only if such a whitelist is
> provided by the administrator.

I was wondering if you got a chance to look at this proposal? WDYT?

Thanks & Best Regards,

Rajat


>
> Details
> ======
>
> 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> imposing of whitelisting by PCI subsystem.
>
> 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> the driver is whitelisted.
>
> 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> flag, to make a decision about whether to bind or not in
> pci_bus_match() or similar.
>
> 4) A mechanism to allow the administrator to specify the whitelist of
> drivers. I think this needs more thought as there are multiple
> options.
>
> a) Expose individual driver's "whitelisted" flag to userspace so a
> boot script can whitelist that driver. There are questions that still
> need answered though e.g. what to do about the devices that may have
> already been enumerated and rejected by then? What to do with the
> already bound devices, if the user changes a driver to remove it from
> the whitelist. etc.
>
>       b) Provide a way to specify the whitelist via the kernel command
> line. Accept a ("pci.whitelist") kernel parameter which is a comma
> separated list of driver names (just like "module_blacklist"), and
> then use it to initialize each driver's "whitelisted" flag as the
> drivers are registered. Essentially this would mean that the whitelist
> of devices cannot be changed after boot.
>
> To me (b) looks a better option but I think a future requirement would
> be the ability to remove the drivers from the whitelist after boot
> (adding drivers to whitelist at runtime may not be that critical IMO)
>
>  WDYT?
>
> Thanks,
>
> Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Bjorn Helgaas]

[+cc Christian (bolt maintainer), Alex, Joerg (IOMMU folks)]

On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> Hi,
> 
> Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> the firmware asks it to:
> 
> 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> 
> An "untrusted" device indicates a (likely external facing) device that
> may be malicious, and can trigger DMA attacks on the system. It may
> also try to exploit any vulnerabilities exposed by the driver, that
> may allow it to read/write unintended addresses in the host (e.g. if
> DMA buffers for the device, share memory pages with other driver data
> structures or code etc).
> 
> High Level proposal
> ===============
> Currently, the "untrusted" device property is used as a hint to enable
> IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> go a step further, and allow the administrator to build a list of
> whitelisted drivers for these "untrusted" devices. This whitelist of
> drivers are the ones that he trusts enough to have little or no
> vulnerabilities. (He may have built this list of whitelisted drivers
> by a combination of code analysis of drivers, or by extensive testing
> using PCIe fuzzing etc). We propose that the administrator be allowed
> to specify this list of whitelisted drivers to the kernel, and the PCI
> subsystem to impose this behavior:
> 
> 1) The "untrusted" devices can bind to only "whitelisted drivers".
> 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> 
> Of course this behavior is to be imposed only if such a whitelist is
> provided by the administrator.
> 
> Details
> ======
> 
> 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> imposing of whitelisting by PCI subsystem.
> 
> 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> the driver is whitelisted.
> 
> 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> flag, to make a decision about whether to bind or not in
> pci_bus_match() or similar.
> 
> 4) A mechanism to allow the administrator to specify the whitelist of
> drivers. I think this needs more thought as there are multiple
> options.
> 
> a) Expose individual driver's "whitelisted" flag to userspace so a
> boot script can whitelist that driver. There are questions that still
> need answered though e.g. what to do about the devices that may have
> already been enumerated and rejected by then? What to do with the
> already bound devices, if the user changes a driver to remove it from
> the whitelist. etc.
> 
>       b) Provide a way to specify the whitelist via the kernel command
> line. Accept a ("pci.whitelist") kernel parameter which is a comma
> separated list of driver names (just like "module_blacklist"), and
> then use it to initialize each driver's "whitelisted" flag as the
> drivers are registered. Essentially this would mean that the whitelist
> of devices cannot be changed after boot.
> 
> To me (b) looks a better option but I think a future requirement would
> be the ability to remove the drivers from the whitelist after boot
> (adding drivers to whitelist at runtime may not be that critical IMO)

We definitely have some problems in this area.

- Thunderbolt has similar security issues, and "bolt"
  (https://gitlab.freedesktop.org/bolt/bolt) provides a user interface
  for authorizing devices.  Bolt is device-oriented (and specifically
  for Thunderbolt), not driver-oriented, and I have no idea what
  kernel interfaces it uses, but I wonder if there's some overlap with
  this proposal.  It seems like both bolt and this proposal could
  ultimately be part of the same user interface.

- ATS allows PCIe endpoints to cache address translations so they
  can generate DMAs with translated addresses (TLP Address Type 10b,
  see PCIe r5.0, sec 10.2.1).  These DMAs can potentially bypass
  the IOMMU.

  AFAICT, amd_iommu always turns on ATS when possible.  It looks
  like intel-iommu and arm-smmu-v3 turn it on except for "untrusted"
  (external) devices.

  There's nothing to prevent a malicious external device from
  generating DMA with translated addresses even if we haven't
  enabled ATS.

  I think all three IOMMUs have mechanisms to block TLPs with
  translated addresses, but I can't tell whether they all *use*
  them.

- ACS is an optional capability, but if implemented at all, downstream
  ports (including root ports) are required to implement Translation
  Blocking.  When enabled, this blocks upstream memory requests with
  non-default AT fields.

  Linux currently never enables Translation Blocking.  Maybe the IOMMU
  protection is sufficient, but I think we should consider enabling TB
  by default and disabling it only when required to enable ATS.  That
  may catch malicious TLPs closer to the source and help when there is
  no IOMMU at all.

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

+ashok.raj@intel.com
+lalithambika.krishnakumar@intel.com

On Wed, May 13, 2020 at 8:19 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Christian (bolt maintainer), Alex, Joerg (IOMMU folks)]
>
> On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > Hi,
> >
> > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > the firmware asks it to:
> >
> > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> >
> > An "untrusted" device indicates a (likely external facing) device that
> > may be malicious, and can trigger DMA attacks on the system. It may
> > also try to exploit any vulnerabilities exposed by the driver, that
> > may allow it to read/write unintended addresses in the host (e.g. if
> > DMA buffers for the device, share memory pages with other driver data
> > structures or code etc).
> >
> > High Level proposal
> > ===============
> > Currently, the "untrusted" device property is used as a hint to enable
> > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > go a step further, and allow the administrator to build a list of
> > whitelisted drivers for these "untrusted" devices. This whitelist of
> > drivers are the ones that he trusts enough to have little or no
> > vulnerabilities. (He may have built this list of whitelisted drivers
> > by a combination of code analysis of drivers, or by extensive testing
> > using PCIe fuzzing etc). We propose that the administrator be allowed
> > to specify this list of whitelisted drivers to the kernel, and the PCI
> > subsystem to impose this behavior:
> >
> > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> >
> > Of course this behavior is to be imposed only if such a whitelist is
> > provided by the administrator.
> >
> > Details
> > ======
> >
> > 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> > imposing of whitelisting by PCI subsystem.
> >
> > 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> > the driver is whitelisted.
> >
> > 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> > flag, to make a decision about whether to bind or not in
> > pci_bus_match() or similar.
> >
> > 4) A mechanism to allow the administrator to specify the whitelist of
> > drivers. I think this needs more thought as there are multiple
> > options.
> >
> > a) Expose individual driver's "whitelisted" flag to userspace so a
> > boot script can whitelist that driver. There are questions that still
> > need answered though e.g. what to do about the devices that may have
> > already been enumerated and rejected by then? What to do with the
> > already bound devices, if the user changes a driver to remove it from
> > the whitelist. etc.
> >
> >       b) Provide a way to specify the whitelist via the kernel command
> > line. Accept a ("pci.whitelist") kernel parameter which is a comma
> > separated list of driver names (just like "module_blacklist"), and
> > then use it to initialize each driver's "whitelisted" flag as the
> > drivers are registered. Essentially this would mean that the whitelist
> > of devices cannot be changed after boot.
> >
> > To me (b) looks a better option but I think a future requirement would
> > be the ability to remove the drivers from the whitelist after boot
> > (adding drivers to whitelist at runtime may not be that critical IMO)
>
> We definitely have some problems in this area.
>
> - Thunderbolt has similar security issues, and "bolt"
>   (https://gitlab.freedesktop.org/bolt/bolt) provides a user interface
>   for authorizing devices.  Bolt is device-oriented (and specifically
>   for Thunderbolt), not driver-oriented, and I have no idea what
>   kernel interfaces it uses, but I wonder if there's some overlap with
>   this proposal.  It seems like both bolt and this proposal could
>   ultimately be part of the same user interface.

Thanks for pointing to it! My proposal does indeed stem from enabling
of thunderbolt in our devices, and the PCIe tunneling (and thus the
additional threat from external devices) that it brings along. I took
a brief look at its documentation and it seems (Christian can correct
me) that it identifies devices with "UUID" and then uses that to drive
all its decisions. So essentially the problem it is trying to solve is
determining whether or not to enable PCIe tunnels based on the UUID of
the device. It seems to me that it assumes that the devices are
trustworthy (i.e. for eg. they will not spoof any other whitelisted
UUID). Christian, can you please help explain if bolt is capable of
dealing with malicious devices that can spoof other devices in order
to try and do bad things to the system?

>
> - ATS allows PCIe endpoints to cache address translations so they
>   can generate DMAs with translated addresses (TLP Address Type 10b,
>   see PCIe r5.0, sec 10.2.1).  These DMAs can potentially bypass
>   the IOMMU.
>
>   AFAICT, amd_iommu always turns on ATS when possible.  It looks
>   like intel-iommu and arm-smmu-v3 turn it on except for "untrusted"
>   (external) devices.

Correct. The point here is to turn on more restrictions on "untrusted" devices.

>
>   There's nothing to prevent a malicious external device from
>   generating DMA with translated addresses even if we haven't
>   enabled ATS.
>
>   I think all three IOMMUs have mechanisms to block TLPs with
>   translated addresses, but I can't tell whether they all *use*
>   them.

Understood.

>
> - ACS is an optional capability, but if implemented at all, downstream
>   ports (including root ports) are required to implement Translation
>   Blocking.  When enabled, this blocks upstream memory requests with
>   non-default AT fields.
>
>   Linux currently never enables Translation Blocking.  Maybe the IOMMU
>   protection is sufficient, but I think we should consider enabling TB
>   by default and disabling it only when required to enable ATS.  That
>   may catch malicious TLPs closer to the source and help when there is
>   no IOMMU at all.

Understood and point taken. Note that enabling IOMMU protection (and
even disabling ATS and enabling TB) is not enough though. This isn't
to say that they shouldn't be done. Yes, they definitely need to be
done. As these can help ensure that a device can generate transactions
*only* to the memory areas (DMA buffers) that the driver has allotted
to it, [and thus all the security mitigations (IOMMU/ ACS/AT/TB) have
been configured so as to provide the device access for those areas].

What these settings can't help with, though, is a malicious device
trying to exploit certain driver vulnerabilities, that allow the
device to do bad things even while *restricting transactions within
the IOMMU allowed memory*. An attacker can do this by carefully
looking at drivers to identify and exploit driver vulnerabilities
(driver negligence). There is a lot of research work, but we for e.g.
are looking at this:
https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04A-1_Song_paper.pdf.
Here are the examples of driver vulnerabilities that it found, that
could be exploited even with the IOMMU/ACS and other restrictions in
place (please check case studies in sections F/G/H in the above paper)
since I may not be able to explain that well:

* A driver could be double fetching the memory, causing it to do
different things than intended. E.g.
* A driver could be (negligently) passing some kernel addresses to the device.
* A driver could be using (for memory dereferencing, for e.g.) the
address/indices, given by the device, without enough validation.
* A driver may negligently be sharing the DMA memory with some other
driver data in the same PAGE. Since the IOMMU restrictions are PAGE
granular, this might give device access to that driver data.

I think the points I am trying to make here are that

1) Since malicious devices can spoof other (potentially whitelisted)
devices, classifying devices into trusted and non-trusted is a good
step, but it is not enough. We do need to go one step further and
classify drivers into trusted/untrusted also, so as to (allow to)
impose more restrictions.

2) Drivers can be vulnerable / exploitable; and finding, fixing, and
introducing new exploits is a never ending cat and mouse game. But
everyone's appetite for risk is different depending on use case, and
thus administrators need a way to say, "I trust these drivers enough
that I consider them safe for my use, even on untrusted ports".

There is going to be a class of threat vectors that cannot be
addressed by IOMMU and ACS alone. And my proposal aims at those cases
specifically. It makes the case for an admin to actually look at the
various drivers and use various techniques available (PCIe fuzzing,
code analysis etc) to bless drivers. I once again suspect that I may
have failed to elaborate on the threat vectors clearly. Please let me
know if that is the case, and in that case, I'll probably ask our
security folks to chime in.

Thanks,

Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Mika Westerberg]

On Wed, May 13, 2020 at 02:26:18PM -0700, Rajat Jain wrote:
> +ashok.raj@intel.com
> +lalithambika.krishnakumar@intel.com
> 
> On Wed, May 13, 2020 at 8:19 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > [+cc Christian (bolt maintainer), Alex, Joerg (IOMMU folks)]
> >
> > On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > > Hi,
> > >
> > > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > > the firmware asks it to:
> > >
> > > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> > >
> > > An "untrusted" device indicates a (likely external facing) device that
> > > may be malicious, and can trigger DMA attacks on the system. It may
> > > also try to exploit any vulnerabilities exposed by the driver, that
> > > may allow it to read/write unintended addresses in the host (e.g. if
> > > DMA buffers for the device, share memory pages with other driver data
> > > structures or code etc).
> > >
> > > High Level proposal
> > > ===============
> > > Currently, the "untrusted" device property is used as a hint to enable
> > > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > > go a step further, and allow the administrator to build a list of
> > > whitelisted drivers for these "untrusted" devices. This whitelist of
> > > drivers are the ones that he trusts enough to have little or no
> > > vulnerabilities. (He may have built this list of whitelisted drivers
> > > by a combination of code analysis of drivers, or by extensive testing
> > > using PCIe fuzzing etc). We propose that the administrator be allowed
> > > to specify this list of whitelisted drivers to the kernel, and the PCI
> > > subsystem to impose this behavior:
> > >
> > > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> > >
> > > Of course this behavior is to be imposed only if such a whitelist is
> > > provided by the administrator.
> > >
> > > Details
> > > ======
> > >
> > > 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> > > imposing of whitelisting by PCI subsystem.
> > >
> > > 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> > > the driver is whitelisted.
> > >
> > > 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> > > flag, to make a decision about whether to bind or not in
> > > pci_bus_match() or similar.
> > >
> > > 4) A mechanism to allow the administrator to specify the whitelist of
> > > drivers. I think this needs more thought as there are multiple
> > > options.
> > >
> > > a) Expose individual driver's "whitelisted" flag to userspace so a
> > > boot script can whitelist that driver. There are questions that still
> > > need answered though e.g. what to do about the devices that may have
> > > already been enumerated and rejected by then? What to do with the
> > > already bound devices, if the user changes a driver to remove it from
> > > the whitelist. etc.
> > >
> > >       b) Provide a way to specify the whitelist via the kernel command
> > > line. Accept a ("pci.whitelist") kernel parameter which is a comma
> > > separated list of driver names (just like "module_blacklist"), and
> > > then use it to initialize each driver's "whitelisted" flag as the
> > > drivers are registered. Essentially this would mean that the whitelist
> > > of devices cannot be changed after boot.
> > >
> > > To me (b) looks a better option but I think a future requirement would
> > > be the ability to remove the drivers from the whitelist after boot
> > > (adding drivers to whitelist at runtime may not be that critical IMO)
> >
> > We definitely have some problems in this area.
> >
> > - Thunderbolt has similar security issues, and "bolt"
> >   (https://gitlab.freedesktop.org/bolt/bolt) provides a user interface
> >   for authorizing devices.  Bolt is device-oriented (and specifically
> >   for Thunderbolt), not driver-oriented, and I have no idea what
> >   kernel interfaces it uses, but I wonder if there's some overlap with
> >   this proposal.  It seems like both bolt and this proposal could
> >   ultimately be part of the same user interface.
> 
> Thanks for pointing to it! My proposal does indeed stem from enabling
> of thunderbolt in our devices, and the PCIe tunneling (and thus the
> additional threat from external devices) that it brings along. I took
> a brief look at its documentation and it seems (Christian can correct
> me) that it identifies devices with "UUID" and then uses that to drive
> all its decisions. So essentially the problem it is trying to solve is
> determining whether or not to enable PCIe tunnels based on the UUID of
> the device. It seems to me that it assumes that the devices are
> trustworthy (i.e. for eg. they will not spoof any other whitelisted
> UUID). Christian, can you please help explain if bolt is capable of
> dealing with malicious devices that can spoof other devices in order
> to try and do bad things to the system?

It is not. It basically implements the TBT security levels that were
used in PCs before IOMMU was deployed. These security levels are based
on user "approving" a device so if anyone does something malicious to
the "approved" device replaces it with another with the same identity
then all bets are off.

For current systems and USB4 ones we still use bolt to perform two
important things but this is not about security that much:

  1. Disable PCIe tunneling completely
  2. Implement whitelist of TBT/USB4 devices

The whitelist here means that for example administrator can limit users
to use only certain TBT/USB4 devices that their organization wants to
support.

DMA security is handled by IOMMU.

> > - ATS allows PCIe endpoints to cache address translations so they
> >   can generate DMAs with translated addresses (TLP Address Type 10b,
> >   see PCIe r5.0, sec 10.2.1).  These DMAs can potentially bypass
> >   the IOMMU.
> >
> >   AFAICT, amd_iommu always turns on ATS when possible.  It looks
> >   like intel-iommu and arm-smmu-v3 turn it on except for "untrusted"
> >   (external) devices.
> 
> Correct. The point here is to turn on more restrictions on "untrusted" devices.
> 
> >
> >   There's nothing to prevent a malicious external device from
> >   generating DMA with translated addresses even if we haven't
> >   enabled ATS.
> >
> >   I think all three IOMMUs have mechanisms to block TLPs with
> >   translated addresses, but I can't tell whether they all *use*
> >   them.
> 
> Understood.
> 
> >
> > - ACS is an optional capability, but if implemented at all, downstream
> >   ports (including root ports) are required to implement Translation
> >   Blocking.  When enabled, this blocks upstream memory requests with
> >   non-default AT fields.
> >
> >   Linux currently never enables Translation Blocking.  Maybe the IOMMU
> >   protection is sufficient, but I think we should consider enabling TB
> >   by default and disabling it only when required to enable ATS.  That
> >   may catch malicious TLPs closer to the source and help when there is
> >   no IOMMU at all.
> 
> Understood and point taken. Note that enabling IOMMU protection (and
> even disabling ATS and enabling TB) is not enough though. This isn't
> to say that they shouldn't be done. Yes, they definitely need to be
> done. As these can help ensure that a device can generate transactions
> *only* to the memory areas (DMA buffers) that the driver has allotted
> to it, [and thus all the security mitigations (IOMMU/ ACS/AT/TB) have
> been configured so as to provide the device access for those areas].
> 
> What these settings can't help with, though, is a malicious device
> trying to exploit certain driver vulnerabilities, that allow the
> device to do bad things even while *restricting transactions within
> the IOMMU allowed memory*. An attacker can do this by carefully
> looking at drivers to identify and exploit driver vulnerabilities
> (driver negligence). There is a lot of research work, but we for e.g.
> are looking at this:
> https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04A-1_Song_paper.pdf.
> Here are the examples of driver vulnerabilities that it found, that
> could be exploited even with the IOMMU/ACS and other restrictions in
> place (please check case studies in sections F/G/H in the above paper)
> since I may not be able to explain that well:
> 
> * A driver could be double fetching the memory, causing it to do
> different things than intended. E.g.
> * A driver could be (negligently) passing some kernel addresses to the device.
> * A driver could be using (for memory dereferencing, for e.g.) the
> address/indices, given by the device, without enough validation.
> * A driver may negligently be sharing the DMA memory with some other
> driver data in the same PAGE. Since the IOMMU restrictions are PAGE
> granular, this might give device access to that driver data.
> 
> I think the points I am trying to make here are that
> 
> 1) Since malicious devices can spoof other (potentially whitelisted)
> devices, classifying devices into trusted and non-trusted is a good
> step, but it is not enough. We do need to go one step further and
> classify drivers into trusted/untrusted also, so as to (allow to)
> impose more restrictions.
> 
> 2) Drivers can be vulnerable / exploitable; and finding, fixing, and
> introducing new exploits is a never ending cat and mouse game. But
> everyone's appetite for risk is different depending on use case, and
> thus administrators need a way to say, "I trust these drivers enough
> that I consider them safe for my use, even on untrusted ports".
> 
> There is going to be a class of threat vectors that cannot be
> addressed by IOMMU and ACS alone. And my proposal aims at those cases
> specifically. It makes the case for an admin to actually look at the
> various drivers and use various techniques available (PCIe fuzzing,
> code analysis etc) to bless drivers. I once again suspect that I may
> have failed to elaborate on the threat vectors clearly. Please let me
> know if that is the case, and in that case, I'll probably ask our
> security folks to chime in.

I think this pretty well explains the issue. At least for me :)

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Raj, Ashok]

Hi Rajat,


On Wed, May 13, 2020 at 02:26:18PM -0700, Rajat Jain wrote:
> +ashok.raj@intel.com
> +lalithambika.krishnakumar@intel.com
> 
> On Wed, May 13, 2020 at 8:19 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > [+cc Christian (bolt maintainer), Alex, Joerg (IOMMU folks)]
> >
> > On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > > Hi,
> > >
> > > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > > the firmware asks it to:
> > >
> > > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> > >
> > > An "untrusted" device indicates a (likely external facing) device that
> > > may be malicious, and can trigger DMA attacks on the system. It may
> > > also try to exploit any vulnerabilities exposed by the driver, that
> > > may allow it to read/write unintended addresses in the host (e.g. if
> > > DMA buffers for the device, share memory pages with other driver data
> > > structures or code etc).
> > >
> > > High Level proposal
> > > ===============
> > > Currently, the "untrusted" device property is used as a hint to enable
> > > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > > go a step further, and allow the administrator to build a list of
> > > whitelisted drivers for these "untrusted" devices. This whitelist of
> > > drivers are the ones that he trusts enough to have little or no
> > > vulnerabilities. (He may have built this list of whitelisted drivers
> > > by a combination of code analysis of drivers, or by extensive testing
> > > using PCIe fuzzing etc). We propose that the administrator be allowed
> > > to specify this list of whitelisted drivers to the kernel, and the PCI
> > > subsystem to impose this behavior:
> > >
> > > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> > >
> > > Of course this behavior is to be imposed only if such a whitelist is
> > > provided by the administrator.
> > >
> > > Details
> > > ======
> > >
> > > 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> > > imposing of whitelisting by PCI subsystem.
> > >
> > > 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> > > the driver is whitelisted.

I'm not sure if a driver certifying itself as secure is acceptable.

Probably the pcie-component-authentication type mechanisms can establish
proper root of trust. Othewise we are just hand waving and any method
has its own gaps. You can probably say use the fuzzer etc, but that more
falls in every adminstrator needs to run and qualify every device. Once you 
have a firmware update that component needs to be re-certified as well.


> > >
> > > 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> > > flag, to make a decision about whether to bind or not in
> > > pci_bus_match() or similar.
> > >
> > > 4) A mechanism to allow the administrator to specify the whitelist of
> > > drivers. I think this needs more thought as there are multiple
> > > options.

A default could be we:

* Trust nothing - need to have a challenge to establish ROT.
* Trust RCiEP devices. These are integrated components and you can probably
  think its not some FPGA plugged in trying to fake itself to defeat
  security.
* A sysadm supplied list of devices to trust. 
   - This could be maybe a RP and all devices below. Since they might be
     all internal facing, the sysadm put those things together. Not plugged
     in external facing ports. 

> > >
> > > a) Expose individual driver's "whitelisted" flag to userspace so a
> > > boot script can whitelist that driver. There are questions that still
> > > need answered though e.g. what to do about the devices that may have
> > > already been enumerated and rejected by then? What to do with the
> > > already bound devices, if the user changes a driver to remove it from
> > > the whitelist. etc.
> > >
> > >       b) Provide a way to specify the whitelist via the kernel command
> > > line. Accept a ("pci.whitelist") kernel parameter which is a comma
> > > separated list of driver names (just like "module_blacklist"), and
> > > then use it to initialize each driver's "whitelisted" flag as the
> > > drivers are registered. Essentially this would mean that the whitelist
> > > of devices cannot be changed after boot.

As @Jean suggested in other thread, maybe sysfs attribute to flip after
reboot is a good idea. One needs to be root, probably a good start. And 
you don't need to reboot to fix.

> > >
> > > To me (b) looks a better option but I think a future requirement would
> > > be the ability to remove the drivers from the whitelist after boot
> > > (adding drivers to whitelist at runtime may not be that critical IMO)
> >
> > We definitely have some problems in this area.
> >
> > - Thunderbolt has similar security issues, and "bolt"
> >   (https://gitlab.freedesktop.org/bolt/bolt) provides a user interface
> >   for authorizing devices.  Bolt is device-oriented (and specifically
> >   for Thunderbolt), not driver-oriented, and I have no idea what
> >   kernel interfaces it uses, but I wonder if there's some overlap with
> >   this proposal.  It seems like both bolt and this proposal could
> >   ultimately be part of the same user interface.
> 
> Thanks for pointing to it! My proposal does indeed stem from enabling
> of thunderbolt in our devices, and the PCIe tunneling (and thus the
> additional threat from external devices) that it brings along. I took
> a brief look at its documentation and it seems (Christian can correct
> me) that it identifies devices with "UUID" and then uses that to drive
> all its decisions. So essentially the problem it is trying to solve is
> determining whether or not to enable PCIe tunnels based on the UUID of
> the device. It seems to me that it assumes that the devices are
> trustworthy (i.e. for eg. they will not spoof any other whitelisted
> UUID). Christian, can you please help explain if bolt is capable of
> dealing with malicious devices that can spoof other devices in order
> to try and do bad things to the system?
> 
> >
> > - ATS allows PCIe endpoints to cache address translations so they
> >   can generate DMAs with translated addresses (TLP Address Type 10b,
> >   see PCIe r5.0, sec 10.2.1).  These DMAs can potentially bypass
> >   the IOMMU.
> >
> >   AFAICT, amd_iommu always turns on ATS when possible.  It looks
> >   like intel-iommu and arm-smmu-v3 turn it on except for "untrusted"
> >   (external) devices.
> 
> Correct. The point here is to turn on more restrictions on "untrusted" devices.
> 
> >
> >   There's nothing to prevent a malicious external device from
> >   generating DMA with translated addresses even if we haven't
> >   enabled ATS.

@Bjorn: Intel Root ports behave as follows: at least for servers:

Translation Requests: Are always non-posted. So RP will always respond with
UR if IOMMU.TRANSLATION_ENABLE=0

Translated Requests can be non-posted (reads), or Posted (Writes).
If IOMMU.TE=0, RP will return UR for reads, and drop writes.


* 
> >
> >   I think all three IOMMUs have mechanisms to block TLPs with
> >   translated addresses, but I can't tell whether they all *use*
> >   them.
> 
> Understood.
> 
> >
> > - ACS is an optional capability, but if implemented at all, downstream
> >   ports (including root ports) are required to implement Translation
> >   Blocking.  When enabled, this blocks upstream memory requests with
> >   non-default AT fields.
> >
> >   Linux currently never enables Translation Blocking.  Maybe the IOMMU
> >   protection is sufficient, but I think we should consider enabling TB
> >   by default and disabling it only when required to enable ATS.  That
> >   may catch malicious TLPs closer to the source and help when there is
> >   no IOMMU at all.
> 
> Understood and point taken. Note that enabling IOMMU protection (and
> even disabling ATS and enabling TB) is not enough though. This isn't
> to say that they shouldn't be done. Yes, they definitely need to be
> done. As these can help ensure that a device can generate transactions
> *only* to the memory areas (DMA buffers) that the driver has allotted
> to it, [and thus all the security mitigations (IOMMU/ ACS/AT/TB) have
> been configured so as to provide the device access for those areas].

I'm not sure how much difference it makes if IOMMU's behave for translation
request and requests with AT=1 accordingly to ensure safety. What
additional protection does Translation blocking provides if we do not turn
on ATS for those untrusted devices.

> 
> What these settings can't help with, though, is a malicious device
> trying to exploit certain driver vulnerabilities, that allow the
> device to do bad things even while *restricting transactions within
> the IOMMU allowed memory*. An attacker can do this by carefully
> looking at drivers to identify and exploit driver vulnerabilities
> (driver negligence). There is a lot of research work, but we for e.g.
> are looking at this:
> https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04A-1_Song_paper.pdf.
> Here are the examples of driver vulnerabilities that it found, that
> could be exploited even with the IOMMU/ACS and other restrictions in
> place (please check case studies in sections F/G/H in the above paper)
> since I may not be able to explain that well:
> 
> * A driver could be double fetching the memory, causing it to do
> different things than intended. E.g.
> * A driver could be (negligently) passing some kernel addresses to the device.
> * A driver could be using (for memory dereferencing, for e.g.) the
> address/indices, given by the device, without enough validation.
> * A driver may negligently be sharing the DMA memory with some other
> driver data in the same PAGE. Since the IOMMU restrictions are PAGE
> granular, this might give device access to that driver data.
> 
> I think the points I am trying to make here are that
> 
> 1) Since malicious devices can spoof other (potentially whitelisted)
> devices, classifying devices into trusted and non-trusted is a good
> step, but it is not enough. We do need to go one step further and
> classify drivers into trusted/untrusted also, so as to (allow to)
> impose more restrictions.
> 
> 2) Drivers can be vulnerable / exploitable; and finding, fixing, and
> introducing new exploits is a never ending cat and mouse game. But
> everyone's appetite for risk is different depending on use case, and
> thus administrators need a way to say, "I trust these drivers enough
> that I consider them safe for my use, even on untrusted ports".

with efforts like lockdown kernel, you ensure the entire kernel and drivers
move to-gether. My fear is if we don't keep this security properties small
enough, the pure permutation and combinations would become a validation
nightmare that by itself can't ensure what works and what doesn't.

> 
> There is going to be a class of threat vectors that cannot be
> addressed by IOMMU and ACS alone. And my proposal aims at those cases
> specifically. It makes the case for an admin to actually look at the
> various drivers and use various techniques available (PCIe fuzzing,
> code analysis etc) to bless drivers. I once again suspect that I may
> have failed to elaborate on the threat vectors clearly. Please let me
> know if that is the case, and in that case, I'll probably ask our
> security folks to chime in.

When you say "Admin should actually look at the various driver" what does
that mean?  I think we should give a simple security policy enforcement
that is simple enough to keep up with. Until we get those device security
enhancements are placed in practice.

https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html

Cheers,
Ashok

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

Hi,

On Thu, May 14, 2020 at 12:13 PM Raj, Ashok <ashok.raj@intel.com> wrote:
>
> Hi Rajat,
>
>
> On Wed, May 13, 2020 at 02:26:18PM -0700, Rajat Jain wrote:
> > +ashok.raj@intel.com
> > +lalithambika.krishnakumar@intel.com
> >
> > On Wed, May 13, 2020 at 8:19 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > >
> > > [+cc Christian (bolt maintainer), Alex, Joerg (IOMMU folks)]
> > >
> > > On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > > > Hi,
> > > >
> > > > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > > > the firmware asks it to:
> > > >
> > > > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > > > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> > > >
> > > > An "untrusted" device indicates a (likely external facing) device that
> > > > may be malicious, and can trigger DMA attacks on the system. It may
> > > > also try to exploit any vulnerabilities exposed by the driver, that
> > > > may allow it to read/write unintended addresses in the host (e.g. if
> > > > DMA buffers for the device, share memory pages with other driver data
> > > > structures or code etc).
> > > >
> > > > High Level proposal
> > > > ===============
> > > > Currently, the "untrusted" device property is used as a hint to enable
> > > > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > > > go a step further, and allow the administrator to build a list of
> > > > whitelisted drivers for these "untrusted" devices. This whitelist of
> > > > drivers are the ones that he trusts enough to have little or no
> > > > vulnerabilities. (He may have built this list of whitelisted drivers
> > > > by a combination of code analysis of drivers, or by extensive testing
> > > > using PCIe fuzzing etc). We propose that the administrator be allowed
> > > > to specify this list of whitelisted drivers to the kernel, and the PCI
> > > > subsystem to impose this behavior:
> > > >
> > > > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > > > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> > > >
> > > > Of course this behavior is to be imposed only if such a whitelist is
> > > > provided by the administrator.
> > > >
> > > > Details
> > > > ======
> > > >
> > > > 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> > > > imposing of whitelisting by PCI subsystem.
> > > >
> > > > 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> > > > the driver is whitelisted.
>
> I'm not sure if a driver certifying itself as secure is acceptable.
>
> Probably the pcie-component-authentication type mechanisms can establish
> proper root of trust. Othewise we are just hand waving and any method
> has its own gaps. You can probably say use the fuzzer etc, but that more
> falls in every adminstrator needs to run and qualify every device. Once you
> have a firmware update that component needs to be re-certified as well.

Yes and No. Yes, the whitelist may have to be re-evaluated for any
changes to kernel/drivers. But No, this will not be needed for any
device firmware updates.

>
>
> > > >
> > > > 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> > > > flag, to make a decision about whether to bind or not in
> > > > pci_bus_match() or similar.
> > > >
> > > > 4) A mechanism to allow the administrator to specify the whitelist of
> > > > drivers. I think this needs more thought as there are multiple
> > > > options.
>
> A default could be we:
>
> * Trust nothing - need to have a challenge to establish ROT.
> * Trust RCiEP devices. These are integrated components and you can probably
>   think its not some FPGA plugged in trying to fake itself to defeat
>   security.
> * A sysadm supplied list of devices to trust.
>    - This could be maybe a RP and all devices below. Since they might be
>      all internal facing, the sysadm put those things together. Not plugged
>      in external facing ports.

Right, but the main problem that we want to solve (about the untrusted
devices) still remains unaddressed.

I believe that the approach taken by the paper you sent below, where
we're building a root of trust using device certificates, key pairs,
and challenges is definitely the right long term path. (Although I
feel there is still scope of some attacks there, but let's not go
there). But it requires the entire device ecosystem to come to an
agreement and then move to that, and is a big change (requiring change
in HW, FW and SW). That is still far away, and we need to think about
what we can do to deal with the current set of (external) devices that
we still want to support, and they can only plug in at untrusted
ports.

>
> > > >
> > > > a) Expose individual driver's "whitelisted" flag to userspace so a
> > > > boot script can whitelist that driver. There are questions that still
> > > > need answered though e.g. what to do about the devices that may have
> > > > already been enumerated and rejected by then? What to do with the
> > > > already bound devices, if the user changes a driver to remove it from
> > > > the whitelist. etc.
> > > >
> > > >       b) Provide a way to specify the whitelist via the kernel command
> > > > line. Accept a ("pci.whitelist") kernel parameter which is a comma
> > > > separated list of driver names (just like "module_blacklist"), and
> > > > then use it to initialize each driver's "whitelisted" flag as the
> > > > drivers are registered. Essentially this would mean that the whitelist
> > > > of devices cannot be changed after boot.
>
> As @Jean suggested in other thread, maybe sysfs attribute to flip after
> reboot is a good idea. One needs to be root, probably a good start. And
> you don't need to reboot to fix.
>
> > > >
> > > > To me (b) looks a better option but I think a future requirement would
> > > > be the ability to remove the drivers from the whitelist after boot
> > > > (adding drivers to whitelist at runtime may not be that critical IMO)
> > >
> > > We definitely have some problems in this area.
> > >
> > > - Thunderbolt has similar security issues, and "bolt"
> > >   (https://gitlab.freedesktop.org/bolt/bolt) provides a user interface
> > >   for authorizing devices.  Bolt is device-oriented (and specifically
> > >   for Thunderbolt), not driver-oriented, and I have no idea what
> > >   kernel interfaces it uses, but I wonder if there's some overlap with
> > >   this proposal.  It seems like both bolt and this proposal could
> > >   ultimately be part of the same user interface.
> >
> > Thanks for pointing to it! My proposal does indeed stem from enabling
> > of thunderbolt in our devices, and the PCIe tunneling (and thus the
> > additional threat from external devices) that it brings along. I took
> > a brief look at its documentation and it seems (Christian can correct
> > me) that it identifies devices with "UUID" and then uses that to drive
> > all its decisions. So essentially the problem it is trying to solve is
> > determining whether or not to enable PCIe tunnels based on the UUID of
> > the device. It seems to me that it assumes that the devices are
> > trustworthy (i.e. for eg. they will not spoof any other whitelisted
> > UUID). Christian, can you please help explain if bolt is capable of
> > dealing with malicious devices that can spoof other devices in order
> > to try and do bad things to the system?
> >
> > >
> > > - ATS allows PCIe endpoints to cache address translations so they
> > >   can generate DMAs with translated addresses (TLP Address Type 10b,
> > >   see PCIe r5.0, sec 10.2.1).  These DMAs can potentially bypass
> > >   the IOMMU.
> > >
> > >   AFAICT, amd_iommu always turns on ATS when possible.  It looks
> > >   like intel-iommu and arm-smmu-v3 turn it on except for "untrusted"
> > >   (external) devices.
> >
> > Correct. The point here is to turn on more restrictions on "untrusted" devices.
> >
> > >
> > >   There's nothing to prevent a malicious external device from
> > >   generating DMA with translated addresses even if we haven't
> > >   enabled ATS.

Reading this mail again, I think I now finally understand what Bjorn
was trying to say above, and I agree.

>
> @Bjorn: Intel Root ports behave as follows: at least for servers:
>
> Translation Requests: Are always non-posted. So RP will always respond with
> UR if IOMMU.TRANSLATION_ENABLE=0
>
> Translated Requests can be non-posted (reads), or Posted (Writes).
> If IOMMU.TE=0, RP will return UR for reads, and drop writes.
>

This is good info, thanks for confirming.

>
> *
> > >
> > >   I think all three IOMMUs have mechanisms to block TLPs with
> > >   translated addresses, but I can't tell whether they all *use*
> > >   them.
> >
> > Understood.
> >
> > >
> > > - ACS is an optional capability, but if implemented at all, downstream
> > >   ports (including root ports) are required to implement Translation
> > >   Blocking.  When enabled, this blocks upstream memory requests with
> > >   non-default AT fields.
> > >
> > >   Linux currently never enables Translation Blocking.  Maybe the IOMMU
> > >   protection is sufficient, but I think we should consider enabling TB
> > >   by default and disabling it only when required to enable ATS.  That
> > >   may catch malicious TLPs closer to the source and help when there is
> > >   no IOMMU at all.

Agree. Additionally for untrusted ports, we should probably never
disable TB and never allow to enable ATS.

> >
> > Understood and point taken. Note that enabling IOMMU protection (and
> > even disabling ATS and enabling TB) is not enough though. This isn't
> > to say that they shouldn't be done. Yes, they definitely need to be
> > done. As these can help ensure that a device can generate transactions
> > *only* to the memory areas (DMA buffers) that the driver has allotted
> > to it, [and thus all the security mitigations (IOMMU/ ACS/AT/TB) have
> > been configured so as to provide the device access for those areas].
>
> I'm not sure how much difference it makes if IOMMU's behave for translation
> request and requests with AT=1 accordingly to ensure safety. What
> additional protection does Translation blocking provides if we do not turn
> on ATS for those untrusted devices.

AFAICT nothing for Intel systems like you explained above. But maybe for others?

>
> >
> > What these settings can't help with, though, is a malicious device
> > trying to exploit certain driver vulnerabilities, that allow the
> > device to do bad things even while *restricting transactions within
> > the IOMMU allowed memory*. An attacker can do this by carefully
> > looking at drivers to identify and exploit driver vulnerabilities
> > (driver negligence). There is a lot of research work, but we for e.g.
> > are looking at this:
> > https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04A-1_Song_paper.pdf.
> > Here are the examples of driver vulnerabilities that it found, that
> > could be exploited even with the IOMMU/ACS and other restrictions in
> > place (please check case studies in sections F/G/H in the above paper)
> > since I may not be able to explain that well:
> >
> > * A driver could be double fetching the memory, causing it to do
> > different things than intended. E.g.
> > * A driver could be (negligently) passing some kernel addresses to the device.
> > * A driver could be using (for memory dereferencing, for e.g.) the
> > address/indices, given by the device, without enough validation.
> > * A driver may negligently be sharing the DMA memory with some other
> > driver data in the same PAGE. Since the IOMMU restrictions are PAGE
> > granular, this might give device access to that driver data.
> >
> > I think the points I am trying to make here are that
> >
> > 1) Since malicious devices can spoof other (potentially whitelisted)
> > devices, classifying devices into trusted and non-trusted is a good
> > step, but it is not enough. We do need to go one step further and
> > classify drivers into trusted/untrusted also, so as to (allow to)
> > impose more restrictions.
> >
> > 2) Drivers can be vulnerable / exploitable; and finding, fixing, and
> > introducing new exploits is a never ending cat and mouse game. But
> > everyone's appetite for risk is different depending on use case, and
> > thus administrators need a way to say, "I trust these drivers enough
> > that I consider them safe for my use, even on untrusted ports".
>
> with efforts like lockdown kernel, you ensure the entire kernel and drivers
> move to-gether.

Thanks for pointing. That would be helpful, but I'm not sure if it
will address the problems I identified above.

> My fear is if we don't keep this security properties small
> enough, the pure permutation and combinations would become a validation
> nightmare that by itself can't ensure what works and what doesn't.
>
> >
> > There is going to be a class of threat vectors that cannot be
> > addressed by IOMMU and ACS alone. And my proposal aims at those cases
> > specifically. It makes the case for an admin to actually look at the
> > various drivers and use various techniques available (PCIe fuzzing,
> > code analysis etc) to bless drivers. I once again suspect that I may
> > have failed to elaborate on the threat vectors clearly. Please let me
> > know if that is the case, and in that case, I'll probably ask our
> > security folks to chime in.
>
> When you say "Admin should actually look at the various driver" what does
> that mean?  I think we should give a simple security policy enforcement
> that is simple enough to keep up with. Until we get those device security
> enhancements are placed in practice.
>
> https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html

This I agree with, but until we get those enhanced secured devices in
place, we need to build a solution for existing devices that can be
plugged on untrusted ports.  Since currently there isn't a way for us
to verify device identity, any scheme that builds upon device provided
identification, falls apart as soon as we introduce "device spoofing"
in the threat model.

The proposal allows a Linux distribution/system designer to choose
which drivers he wants to allow on the untrusted ports. I think this
is a fair ask - given that there isn't any other solution at this time
to address the issues I pointed out.


Thanks!

Rajat

PS; A dimension that I think I'd like to mention again are the issues
arising out of "driver negligences" (like the vulnerabilities I
pointed above). These may not necessarily require a malicious device.
A driver whitelist also helps for that.

>
> Cheers,
> Ashok

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

Hi Bjorn,

On Thu, May 14, 2020 at 7:18 PM Rajat Jain <rajatja@google.com> wrote:
>
> Hi,
>
> On Thu, May 14, 2020 at 12:13 PM Raj, Ashok <ashok.raj@intel.com> wrote:
> >
> > Hi Rajat,
> >
> >
> > On Wed, May 13, 2020 at 02:26:18PM -0700, Rajat Jain wrote:
> > > +ashok.raj@intel.com
> > > +lalithambika.krishnakumar@intel.com
> > >
> > > On Wed, May 13, 2020 at 8:19 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > >
> > > > [+cc Christian (bolt maintainer), Alex, Joerg (IOMMU folks)]
> > > >
> > > > On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > > > > Hi,
> > > > >
> > > > > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > > > > the firmware asks it to:
> > > > >
> > > > > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > > > > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> > > > >
> > > > > An "untrusted" device indicates a (likely external facing) device that
> > > > > may be malicious, and can trigger DMA attacks on the system. It may
> > > > > also try to exploit any vulnerabilities exposed by the driver, that
> > > > > may allow it to read/write unintended addresses in the host (e.g. if
> > > > > DMA buffers for the device, share memory pages with other driver data
> > > > > structures or code etc).
> > > > >
> > > > > High Level proposal
> > > > > ===============
> > > > > Currently, the "untrusted" device property is used as a hint to enable
> > > > > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > > > > go a step further, and allow the administrator to build a list of
> > > > > whitelisted drivers for these "untrusted" devices. This whitelist of
> > > > > drivers are the ones that he trusts enough to have little or no
> > > > > vulnerabilities. (He may have built this list of whitelisted drivers
> > > > > by a combination of code analysis of drivers, or by extensive testing
> > > > > using PCIe fuzzing etc). We propose that the administrator be allowed
> > > > > to specify this list of whitelisted drivers to the kernel, and the PCI
> > > > > subsystem to impose this behavior:
> > > > >
> > > > > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > > > > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> > > > >
> > > > > Of course this behavior is to be imposed only if such a whitelist is
> > > > > provided by the administrator.

I haven't heard much on this proposal after the initial inputs (to
which I responded). Essentially, I agree that IO-MMU and ACS
restrictions need to be put in plcase. But I think we need this
additionally. Does this look acceptable to you? I wanted to start
spinning out the patches, but wanted to see if there are any pending
comments or concerns.

Thanks & Best Regards,

Rajat


> > > > >
> > > > > Details
> > > > > ======
> > > > >
> > > > > 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> > > > > imposing of whitelisting by PCI subsystem.
> > > > >
> > > > > 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> > > > > the driver is whitelisted.
> >
> > I'm not sure if a driver certifying itself as secure is acceptable.
> >
> > Probably the pcie-component-authentication type mechanisms can establish
> > proper root of trust. Othewise we are just hand waving and any method
> > has its own gaps. You can probably say use the fuzzer etc, but that more
> > falls in every adminstrator needs to run and qualify every device. Once you
> > have a firmware update that component needs to be re-certified as well.
>
> Yes and No. Yes, the whitelist may have to be re-evaluated for any
> changes to kernel/drivers. But No, this will not be needed for any
> device firmware updates.
>
> >
> >
> > > > >
> > > > > 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> > > > > flag, to make a decision about whether to bind or not in
> > > > > pci_bus_match() or similar.
> > > > >
> > > > > 4) A mechanism to allow the administrator to specify the whitelist of
> > > > > drivers. I think this needs more thought as there are multiple
> > > > > options.
> >
> > A default could be we:
> >
> > * Trust nothing - need to have a challenge to establish ROT.
> > * Trust RCiEP devices. These are integrated components and you can probably
> >   think its not some FPGA plugged in trying to fake itself to defeat
> >   security.
> > * A sysadm supplied list of devices to trust.
> >    - This could be maybe a RP and all devices below. Since they might be
> >      all internal facing, the sysadm put those things together. Not plugged
> >      in external facing ports.
>
> Right, but the main problem that we want to solve (about the untrusted
> devices) still remains unaddressed.
>
> I believe that the approach taken by the paper you sent below, where
> we're building a root of trust using device certificates, key pairs,
> and challenges is definitely the right long term path. (Although I
> feel there is still scope of some attacks there, but let's not go
> there). But it requires the entire device ecosystem to come to an
> agreement and then move to that, and is a big change (requiring change
> in HW, FW and SW). That is still far away, and we need to think about
> what we can do to deal with the current set of (external) devices that
> we still want to support, and they can only plug in at untrusted
> ports.
>
> >
> > > > >
> > > > > a) Expose individual driver's "whitelisted" flag to userspace so a
> > > > > boot script can whitelist that driver. There are questions that still
> > > > > need answered though e.g. what to do about the devices that may have
> > > > > already been enumerated and rejected by then? What to do with the
> > > > > already bound devices, if the user changes a driver to remove it from
> > > > > the whitelist. etc.
> > > > >
> > > > >       b) Provide a way to specify the whitelist via the kernel command
> > > > > line. Accept a ("pci.whitelist") kernel parameter which is a comma
> > > > > separated list of driver names (just like "module_blacklist"), and
> > > > > then use it to initialize each driver's "whitelisted" flag as the
> > > > > drivers are registered. Essentially this would mean that the whitelist
> > > > > of devices cannot be changed after boot.
> >
> > As @Jean suggested in other thread, maybe sysfs attribute to flip after
> > reboot is a good idea. One needs to be root, probably a good start. And
> > you don't need to reboot to fix.
> >
> > > > >
> > > > > To me (b) looks a better option but I think a future requirement would
> > > > > be the ability to remove the drivers from the whitelist after boot
> > > > > (adding drivers to whitelist at runtime may not be that critical IMO)
> > > >
> > > > We definitely have some problems in this area.
> > > >
> > > > - Thunderbolt has similar security issues, and "bolt"
> > > >   (https://gitlab.freedesktop.org/bolt/bolt) provides a user interface
> > > >   for authorizing devices.  Bolt is device-oriented (and specifically
> > > >   for Thunderbolt), not driver-oriented, and I have no idea what
> > > >   kernel interfaces it uses, but I wonder if there's some overlap with
> > > >   this proposal.  It seems like both bolt and this proposal could
> > > >   ultimately be part of the same user interface.
> > >
> > > Thanks for pointing to it! My proposal does indeed stem from enabling
> > > of thunderbolt in our devices, and the PCIe tunneling (and thus the
> > > additional threat from external devices) that it brings along. I took
> > > a brief look at its documentation and it seems (Christian can correct
> > > me) that it identifies devices with "UUID" and then uses that to drive
> > > all its decisions. So essentially the problem it is trying to solve is
> > > determining whether or not to enable PCIe tunnels based on the UUID of
> > > the device. It seems to me that it assumes that the devices are
> > > trustworthy (i.e. for eg. they will not spoof any other whitelisted
> > > UUID). Christian, can you please help explain if bolt is capable of
> > > dealing with malicious devices that can spoof other devices in order
> > > to try and do bad things to the system?
> > >
> > > >
> > > > - ATS allows PCIe endpoints to cache address translations so they
> > > >   can generate DMAs with translated addresses (TLP Address Type 10b,
> > > >   see PCIe r5.0, sec 10.2.1).  These DMAs can potentially bypass
> > > >   the IOMMU.
> > > >
> > > >   AFAICT, amd_iommu always turns on ATS when possible.  It looks
> > > >   like intel-iommu and arm-smmu-v3 turn it on except for "untrusted"
> > > >   (external) devices.
> > >
> > > Correct. The point here is to turn on more restrictions on "untrusted" devices.
> > >
> > > >
> > > >   There's nothing to prevent a malicious external device from
> > > >   generating DMA with translated addresses even if we haven't
> > > >   enabled ATS.
>
> Reading this mail again, I think I now finally understand what Bjorn
> was trying to say above, and I agree.
>
> >
> > @Bjorn: Intel Root ports behave as follows: at least for servers:
> >
> > Translation Requests: Are always non-posted. So RP will always respond with
> > UR if IOMMU.TRANSLATION_ENABLE=0
> >
> > Translated Requests can be non-posted (reads), or Posted (Writes).
> > If IOMMU.TE=0, RP will return UR for reads, and drop writes.
> >
>
> This is good info, thanks for confirming.
>
> >
> > *
> > > >
> > > >   I think all three IOMMUs have mechanisms to block TLPs with
> > > >   translated addresses, but I can't tell whether they all *use*
> > > >   them.
> > >
> > > Understood.
> > >
> > > >
> > > > - ACS is an optional capability, but if implemented at all, downstream
> > > >   ports (including root ports) are required to implement Translation
> > > >   Blocking.  When enabled, this blocks upstream memory requests with
> > > >   non-default AT fields.
> > > >
> > > >   Linux currently never enables Translation Blocking.  Maybe the IOMMU
> > > >   protection is sufficient, but I think we should consider enabling TB
> > > >   by default and disabling it only when required to enable ATS.  That
> > > >   may catch malicious TLPs closer to the source and help when there is
> > > >   no IOMMU at all.
>
> Agree. Additionally for untrusted ports, we should probably never
> disable TB and never allow to enable ATS.
>
> > >
> > > Understood and point taken. Note that enabling IOMMU protection (and
> > > even disabling ATS and enabling TB) is not enough though. This isn't
> > > to say that they shouldn't be done. Yes, they definitely need to be
> > > done. As these can help ensure that a device can generate transactions
> > > *only* to the memory areas (DMA buffers) that the driver has allotted
> > > to it, [and thus all the security mitigations (IOMMU/ ACS/AT/TB) have
> > > been configured so as to provide the device access for those areas].
> >
> > I'm not sure how much difference it makes if IOMMU's behave for translation
> > request and requests with AT=1 accordingly to ensure safety. What
> > additional protection does Translation blocking provides if we do not turn
> > on ATS for those untrusted devices.
>
> AFAICT nothing for Intel systems like you explained above. But maybe for others?
>
> >
> > >
> > > What these settings can't help with, though, is a malicious device
> > > trying to exploit certain driver vulnerabilities, that allow the
> > > device to do bad things even while *restricting transactions within
> > > the IOMMU allowed memory*. An attacker can do this by carefully
> > > looking at drivers to identify and exploit driver vulnerabilities
> > > (driver negligence). There is a lot of research work, but we for e.g.
> > > are looking at this:
> > > https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04A-1_Song_paper.pdf.
> > > Here are the examples of driver vulnerabilities that it found, that
> > > could be exploited even with the IOMMU/ACS and other restrictions in
> > > place (please check case studies in sections F/G/H in the above paper)
> > > since I may not be able to explain that well:
> > >
> > > * A driver could be double fetching the memory, causing it to do
> > > different things than intended. E.g.
> > > * A driver could be (negligently) passing some kernel addresses to the device.
> > > * A driver could be using (for memory dereferencing, for e.g.) the
> > > address/indices, given by the device, without enough validation.
> > > * A driver may negligently be sharing the DMA memory with some other
> > > driver data in the same PAGE. Since the IOMMU restrictions are PAGE
> > > granular, this might give device access to that driver data.
> > >
> > > I think the points I am trying to make here are that
> > >
> > > 1) Since malicious devices can spoof other (potentially whitelisted)
> > > devices, classifying devices into trusted and non-trusted is a good
> > > step, but it is not enough. We do need to go one step further and
> > > classify drivers into trusted/untrusted also, so as to (allow to)
> > > impose more restrictions.
> > >
> > > 2) Drivers can be vulnerable / exploitable; and finding, fixing, and
> > > introducing new exploits is a never ending cat and mouse game. But
> > > everyone's appetite for risk is different depending on use case, and
> > > thus administrators need a way to say, "I trust these drivers enough
> > > that I consider them safe for my use, even on untrusted ports".
> >
> > with efforts like lockdown kernel, you ensure the entire kernel and drivers
> > move to-gether.
>
> Thanks for pointing. That would be helpful, but I'm not sure if it
> will address the problems I identified above.
>
> > My fear is if we don't keep this security properties small
> > enough, the pure permutation and combinations would become a validation
> > nightmare that by itself can't ensure what works and what doesn't.
> >
> > >
> > > There is going to be a class of threat vectors that cannot be
> > > addressed by IOMMU and ACS alone. And my proposal aims at those cases
> > > specifically. It makes the case for an admin to actually look at the
> > > various drivers and use various techniques available (PCIe fuzzing,
> > > code analysis etc) to bless drivers. I once again suspect that I may
> > > have failed to elaborate on the threat vectors clearly. Please let me
> > > know if that is the case, and in that case, I'll probably ask our
> > > security folks to chime in.
> >
> > When you say "Admin should actually look at the various driver" what does
> > that mean?  I think we should give a simple security policy enforcement
> > that is simple enough to keep up with. Until we get those device security
> > enhancements are placed in practice.
> >
> > https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html
>
> This I agree with, but until we get those enhanced secured devices in
> place, we need to build a solution for existing devices that can be
> plugged on untrusted ports.  Since currently there isn't a way for us
> to verify device identity, any scheme that builds upon device provided
> identification, falls apart as soon as we introduce "device spoofing"
> in the threat model.
>
> The proposal allows a Linux distribution/system designer to choose
> which drivers he wants to allow on the untrusted ports. I think this
> is a fair ask - given that there isn't any other solution at this time
> to address the issues I pointed out.
>
>
> Thanks!
>
> Rajat
>
> PS; A dimension that I think I'd like to mention again are the issues
> arising out of "driver negligences" (like the vulnerabilities I
> pointed above). These may not necessarily require a malicious device.
> A driver whitelist also helps for that.
>
> >
> > Cheers,
> > Ashok

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Bjorn Helgaas]

[+cc Greg, linux-kernel for wider exposure]

On Tue, May 26, 2020 at 09:30:08AM -0700, Rajat Jain wrote:
> On Thu, May 14, 2020 at 7:18 PM Rajat Jain <rajatja@google.com> wrote:
> > On Thu, May 14, 2020 at 12:13 PM Raj, Ashok <ashok.raj@intel.com> wrote:
> > > On Wed, May 13, 2020 at 02:26:18PM -0700, Rajat Jain wrote:
> > > > On Wed, May 13, 2020 at 8:19 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > > > > > Hi,
> > > > > >
> > > > > > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > > > > > the firmware asks it to:
> > > > > >
> > > > > > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > > > > > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> > > > > >
> > > > > > An "untrusted" device indicates a (likely external facing) device that
> > > > > > may be malicious, and can trigger DMA attacks on the system. It may
> > > > > > also try to exploit any vulnerabilities exposed by the driver, that
> > > > > > may allow it to read/write unintended addresses in the host (e.g. if
> > > > > > DMA buffers for the device, share memory pages with other driver data
> > > > > > structures or code etc).
> > > > > >
> > > > > > High Level proposal
> > > > > > ===============
> > > > > > Currently, the "untrusted" device property is used as a hint to enable
> > > > > > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > > > > > go a step further, and allow the administrator to build a list of
> > > > > > whitelisted drivers for these "untrusted" devices. This whitelist of
> > > > > > drivers are the ones that he trusts enough to have little or no
> > > > > > vulnerabilities. (He may have built this list of whitelisted drivers
> > > > > > by a combination of code analysis of drivers, or by extensive testing
> > > > > > using PCIe fuzzing etc). We propose that the administrator be allowed
> > > > > > to specify this list of whitelisted drivers to the kernel, and the PCI
> > > > > > subsystem to impose this behavior:
> > > > > >
> > > > > > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > > > > > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> > > > > >
> > > > > > Of course this behavior is to be imposed only if such a whitelist is
> > > > > > provided by the administrator.
> 
> I haven't heard much on this proposal after the initial inputs (to
> which I responded). Essentially, I agree that IO-MMU and ACS
> restrictions need to be put in plcase. But I think we need this
> additionally. Does this look acceptable to you? I wanted to start
> spinning out the patches, but wanted to see if there are any pending
> comments or concerns.

I think it makes sense to code this up and see what it would look
like.  The bare minimum seems like a driver "bind-to-external-devices"
bit that's visible in sysfs plus something in the driver probe path
that checks it.  Neither is inherently PCI-specific, but maybe the
right place will become obvious when implementing it.

I'm still not 100% sure the device "external/untrusted" bit is the
right thing to check.  If you don't trust a driver enough to expose it
to an external device, is it reasonable to trust it for internal
devices?  It seems like one could attack the driver of even an
internal device like a NIC by controlling the data fed to it.  

The existing use of "external/untrusted" for IOMMU protection is
different.  There we're acknowledging that the *device* itself is
unknown and we need to protect ourselves from malicious DMA.

Here we're concerned about a *driver* that's completely under our
control.  If we don't trust the driver, we could (a) fix the problems
in the driver, (b) change the driver so it handles external/untrusted
devices differently, or (c) not ship the driver at all.

I'm also not sure about the administrative details of this.  Certain
versions of the driver may be trusted while others are untrusted.
That would all have to be managed in userspace, so it's not really our
problem, but it sounds like a hassle.  Putting the information in the
driver itself would reduce that.

> > > > > > Details
> > > > > > ======
> > > > > >
> > > > > > 1) A kernel argument ("pci.impose_driver_whitelisting") to enable
> > > > > > imposing of whitelisting by PCI subsystem.
> > > > > >
> > > > > > 2) Add a flag ("whitelisted") in struct pci_driver to indicate whether
> > > > > > the driver is whitelisted.
> > >
> > > I'm not sure if a driver certifying itself as secure is acceptable.
> > >
> > > Probably the pcie-component-authentication type mechanisms can establish
> > > proper root of trust. Othewise we are just hand waving and any method
> > > has its own gaps. You can probably say use the fuzzer etc, but that more
> > > falls in every adminstrator needs to run and qualify every device. Once you
> > > have a firmware update that component needs to be re-certified as well.
> >
> > Yes and No. Yes, the whitelist may have to be re-evaluated for any
> > changes to kernel/drivers. But No, this will not be needed for any
> > device firmware updates.
> >
> > > > > > 3) Use the driver's "whitelisted" flag and the device's "untrusted"
> > > > > > flag, to make a decision about whether to bind or not in
> > > > > > pci_bus_match() or similar.
> > > > > >
> > > > > > 4) A mechanism to allow the administrator to specify the whitelist of
> > > > > > drivers. I think this needs more thought as there are multiple
> > > > > > options.
> > >
> > > A default could be we:
> > >
> > > * Trust nothing - need to have a challenge to establish ROT.
> > > * Trust RCiEP devices. These are integrated components and you can probably
> > >   think its not some FPGA plugged in trying to fake itself to defeat
> > >   security.
> > > * A sysadm supplied list of devices to trust.
> > >    - This could be maybe a RP and all devices below. Since they might be
> > >      all internal facing, the sysadm put those things together. Not plugged
> > >      in external facing ports.
> >
> > Right, but the main problem that we want to solve (about the untrusted
> > devices) still remains unaddressed.
> >
> > I believe that the approach taken by the paper you sent below, where
> > we're building a root of trust using device certificates, key pairs,
> > and challenges is definitely the right long term path. (Although I
> > feel there is still scope of some attacks there, but let's not go
> > there). But it requires the entire device ecosystem to come to an
> > agreement and then move to that, and is a big change (requiring change
> > in HW, FW and SW). That is still far away, and we need to think about
> > what we can do to deal with the current set of (external) devices that
> > we still want to support, and they can only plug in at untrusted
> > ports.
> >
> > > > > > a) Expose individual driver's "whitelisted" flag to userspace so a
> > > > > > boot script can whitelist that driver. There are questions that still
> > > > > > need answered though e.g. what to do about the devices that may have
> > > > > > already been enumerated and rejected by then? What to do with the
> > > > > > already bound devices, if the user changes a driver to remove it from
> > > > > > the whitelist. etc.
> > > > > >
> > > > > >       b) Provide a way to specify the whitelist via the kernel command
> > > > > > line. Accept a ("pci.whitelist") kernel parameter which is a comma
> > > > > > separated list of driver names (just like "module_blacklist"), and
> > > > > > then use it to initialize each driver's "whitelisted" flag as the
> > > > > > drivers are registered. Essentially this would mean that the whitelist
> > > > > > of devices cannot be changed after boot.
> > >
> > > As @Jean suggested in other thread, maybe sysfs attribute to flip after
> > > reboot is a good idea. One needs to be root, probably a good start. And
> > > you don't need to reboot to fix.
> > >
> > > > > > To me (b) looks a better option but I think a future requirement would
> > > > > > be the ability to remove the drivers from the whitelist after boot
> > > > > > (adding drivers to whitelist at runtime may not be that critical IMO)
> > > > >
> > > > > We definitely have some problems in this area.
> > > > >
> > > > > - Thunderbolt has similar security issues, and "bolt"
> > > > >   (https://gitlab.freedesktop.org/bolt/bolt) provides a user interface
> > > > >   for authorizing devices.  Bolt is device-oriented (and specifically
> > > > >   for Thunderbolt), not driver-oriented, and I have no idea what
> > > > >   kernel interfaces it uses, but I wonder if there's some overlap with
> > > > >   this proposal.  It seems like both bolt and this proposal could
> > > > >   ultimately be part of the same user interface.
> > > >
> > > > Thanks for pointing to it! My proposal does indeed stem from enabling
> > > > of thunderbolt in our devices, and the PCIe tunneling (and thus the
> > > > additional threat from external devices) that it brings along. I took
> > > > a brief look at its documentation and it seems (Christian can correct
> > > > me) that it identifies devices with "UUID" and then uses that to drive
> > > > all its decisions. So essentially the problem it is trying to solve is
> > > > determining whether or not to enable PCIe tunnels based on the UUID of
> > > > the device. It seems to me that it assumes that the devices are
> > > > trustworthy (i.e. for eg. they will not spoof any other whitelisted
> > > > UUID). Christian, can you please help explain if bolt is capable of
> > > > dealing with malicious devices that can spoof other devices in order
> > > > to try and do bad things to the system?
> > > >
> > > > > - ATS allows PCIe endpoints to cache address translations so they
> > > > >   can generate DMAs with translated addresses (TLP Address Type 10b,
> > > > >   see PCIe r5.0, sec 10.2.1).  These DMAs can potentially bypass
> > > > >   the IOMMU.
> > > > >
> > > > >   AFAICT, amd_iommu always turns on ATS when possible.  It looks
> > > > >   like intel-iommu and arm-smmu-v3 turn it on except for "untrusted"
> > > > >   (external) devices.
> > > >
> > > > Correct. The point here is to turn on more restrictions on "untrusted" devices.
> > > >
> > > > >   There's nothing to prevent a malicious external device from
> > > > >   generating DMA with translated addresses even if we haven't
> > > > >   enabled ATS.
> >
> > Reading this mail again, I think I now finally understand what Bjorn
> > was trying to say above, and I agree.
> >
> > > @Bjorn: Intel Root ports behave as follows: at least for servers:
> > >
> > > Translation Requests: Are always non-posted. So RP will always respond with
> > > UR if IOMMU.TRANSLATION_ENABLE=0
> > >
> > > Translated Requests can be non-posted (reads), or Posted (Writes).
> > > If IOMMU.TE=0, RP will return UR for reads, and drop writes.
> >
> > This is good info, thanks for confirming.
> >
> > > > >   I think all three IOMMUs have mechanisms to block TLPs with
> > > > >   translated addresses, but I can't tell whether they all *use*
> > > > >   them.
> > > >
> > > > Understood.
> > > >
> > > > > - ACS is an optional capability, but if implemented at all, downstream
> > > > >   ports (including root ports) are required to implement Translation
> > > > >   Blocking.  When enabled, this blocks upstream memory requests with
> > > > >   non-default AT fields.
> > > > >
> > > > >   Linux currently never enables Translation Blocking.  Maybe the IOMMU
> > > > >   protection is sufficient, but I think we should consider enabling TB
> > > > >   by default and disabling it only when required to enable ATS.  That
> > > > >   may catch malicious TLPs closer to the source and help when there is
> > > > >   no IOMMU at all.
> >
> > Agree. Additionally for untrusted ports, we should probably never
> > disable TB and never allow to enable ATS.
> >
> > > > Understood and point taken. Note that enabling IOMMU protection (and
> > > > even disabling ATS and enabling TB) is not enough though. This isn't
> > > > to say that they shouldn't be done. Yes, they definitely need to be
> > > > done. As these can help ensure that a device can generate transactions
> > > > *only* to the memory areas (DMA buffers) that the driver has allotted
> > > > to it, [and thus all the security mitigations (IOMMU/ ACS/AT/TB) have
> > > > been configured so as to provide the device access for those areas].
> > >
> > > I'm not sure how much difference it makes if IOMMU's behave for translation
> > > request and requests with AT=1 accordingly to ensure safety. What
> > > additional protection does Translation blocking provides if we do not turn
> > > on ATS for those untrusted devices.
> >
> > AFAICT nothing for Intel systems like you explained above. But maybe for others?
> >
> > > > What these settings can't help with, though, is a malicious device
> > > > trying to exploit certain driver vulnerabilities, that allow the
> > > > device to do bad things even while *restricting transactions within
> > > > the IOMMU allowed memory*. An attacker can do this by carefully
> > > > looking at drivers to identify and exploit driver vulnerabilities
> > > > (driver negligence). There is a lot of research work, but we for e.g.
> > > > are looking at this:
> > > > https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04A-1_Song_paper.pdf.
> > > > Here are the examples of driver vulnerabilities that it found, that
> > > > could be exploited even with the IOMMU/ACS and other restrictions in
> > > > place (please check case studies in sections F/G/H in the above paper)
> > > > since I may not be able to explain that well:
> > > >
> > > > * A driver could be double fetching the memory, causing it to do
> > > > different things than intended. E.g.
> > > > * A driver could be (negligently) passing some kernel addresses to the device.
> > > > * A driver could be using (for memory dereferencing, for e.g.) the
> > > > address/indices, given by the device, without enough validation.
> > > > * A driver may negligently be sharing the DMA memory with some other
> > > > driver data in the same PAGE. Since the IOMMU restrictions are PAGE
> > > > granular, this might give device access to that driver data.
> > > >
> > > > I think the points I am trying to make here are that
> > > >
> > > > 1) Since malicious devices can spoof other (potentially whitelisted)
> > > > devices, classifying devices into trusted and non-trusted is a good
> > > > step, but it is not enough. We do need to go one step further and
> > > > classify drivers into trusted/untrusted also, so as to (allow to)
> > > > impose more restrictions.
> > > >
> > > > 2) Drivers can be vulnerable / exploitable; and finding, fixing, and
> > > > introducing new exploits is a never ending cat and mouse game. But
> > > > everyone's appetite for risk is different depending on use case, and
> > > > thus administrators need a way to say, "I trust these drivers enough
> > > > that I consider them safe for my use, even on untrusted ports".
> > >
> > > with efforts like lockdown kernel, you ensure the entire kernel and drivers
> > > move to-gether.
> >
> > Thanks for pointing. That would be helpful, but I'm not sure if it
> > will address the problems I identified above.
> >
> > > My fear is if we don't keep this security properties small
> > > enough, the pure permutation and combinations would become a validation
> > > nightmare that by itself can't ensure what works and what doesn't.
> > >
> > > > There is going to be a class of threat vectors that cannot be
> > > > addressed by IOMMU and ACS alone. And my proposal aims at those cases
> > > > specifically. It makes the case for an admin to actually look at the
> > > > various drivers and use various techniques available (PCIe fuzzing,
> > > > code analysis etc) to bless drivers. I once again suspect that I may
> > > > have failed to elaborate on the threat vectors clearly. Please let me
> > > > know if that is the case, and in that case, I'll probably ask our
> > > > security folks to chime in.
> > >
> > > When you say "Admin should actually look at the various driver" what does
> > > that mean?  I think we should give a simple security policy enforcement
> > > that is simple enough to keep up with. Until we get those device security
> > > enhancements are placed in practice.
> > >
> > > https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html
> >
> > This I agree with, but until we get those enhanced secured devices in
> > place, we need to build a solution for existing devices that can be
> > plugged on untrusted ports.  Since currently there isn't a way for us
> > to verify device identity, any scheme that builds upon device provided
> > identification, falls apart as soon as we introduce "device spoofing"
> > in the threat model.
> >
> > The proposal allows a Linux distribution/system designer to choose
> > which drivers he wants to allow on the untrusted ports. I think this
> > is a fair ask - given that there isn't any other solution at this time
> > to address the issues I pointed out.
> >
> > Thanks!
> >
> > Rajat
> >
> > PS; A dimension that I think I'd like to mention again are the issues
> > arising out of "driver negligences" (like the vulnerabilities I
> > pointed above). These may not necessarily require a malicious device.
> > A driver whitelist also helps for that.

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Mon, Jun 01, 2020 at 06:25:42PM -0500, Bjorn Helgaas wrote:
> [+cc Greg, linux-kernel for wider exposure]

Thanks for the cc:, missed this...

> 
> On Tue, May 26, 2020 at 09:30:08AM -0700, Rajat Jain wrote:
> > On Thu, May 14, 2020 at 7:18 PM Rajat Jain <rajatja@google.com> wrote:
> > > On Thu, May 14, 2020 at 12:13 PM Raj, Ashok <ashok.raj@intel.com> wrote:
> > > > On Wed, May 13, 2020 at 02:26:18PM -0700, Rajat Jain wrote:
> > > > > On Wed, May 13, 2020 at 8:19 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > > On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > > > > > > the firmware asks it to:
> > > > > > >
> > > > > > > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > > > > > > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> > > > > > >
> > > > > > > An "untrusted" device indicates a (likely external facing) device that
> > > > > > > may be malicious, and can trigger DMA attacks on the system. It may
> > > > > > > also try to exploit any vulnerabilities exposed by the driver, that
> > > > > > > may allow it to read/write unintended addresses in the host (e.g. if
> > > > > > > DMA buffers for the device, share memory pages with other driver data
> > > > > > > structures or code etc).
> > > > > > >
> > > > > > > High Level proposal
> > > > > > > ===============
> > > > > > > Currently, the "untrusted" device property is used as a hint to enable
> > > > > > > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > > > > > > go a step further, and allow the administrator to build a list of
> > > > > > > whitelisted drivers for these "untrusted" devices. This whitelist of
> > > > > > > drivers are the ones that he trusts enough to have little or no
> > > > > > > vulnerabilities. (He may have built this list of whitelisted drivers
> > > > > > > by a combination of code analysis of drivers, or by extensive testing
> > > > > > > using PCIe fuzzing etc). We propose that the administrator be allowed
> > > > > > > to specify this list of whitelisted drivers to the kernel, and the PCI
> > > > > > > subsystem to impose this behavior:
> > > > > > >
> > > > > > > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > > > > > > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> > > > > > >
> > > > > > > Of course this behavior is to be imposed only if such a whitelist is
> > > > > > > provided by the administrator.
> > 
> > I haven't heard much on this proposal after the initial inputs (to
> > which I responded). Essentially, I agree that IO-MMU and ACS
> > restrictions need to be put in plcase. But I think we need this
> > additionally. Does this look acceptable to you? I wanted to start
> > spinning out the patches, but wanted to see if there are any pending
> > comments or concerns.
> 
> I think it makes sense to code this up and see what it would look
> like.  The bare minimum seems like a driver "bind-to-external-devices"
> bit that's visible in sysfs plus something in the driver probe path
> that checks it.  Neither is inherently PCI-specific, but maybe the
> right place will become obvious when implementing it.
> 
> I'm still not 100% sure the device "external/untrusted" bit is the
> right thing to check.  If you don't trust a driver enough to expose it
> to an external device, is it reasonable to trust it for internal
> devices?  It seems like one could attack the driver of even an
> internal device like a NIC by controlling the data fed to it.  
> 
> The existing use of "external/untrusted" for IOMMU protection is
> different.  There we're acknowledging that the *device* itself is
> unknown and we need to protect ourselves from malicious DMA.
> 
> Here we're concerned about a *driver* that's completely under our
> control.  If we don't trust the driver, we could (a) fix the problems
> in the driver, (b) change the driver so it handles external/untrusted
> devices differently, or (c) not ship the driver at all.
> 
> I'm also not sure about the administrative details of this.  Certain
> versions of the driver may be trusted while others are untrusted.
> That would all have to be managed in userspace, so it's not really our
> problem, but it sounds like a hassle.  Putting the information in the
> driver itself would reduce that.

What about taking what we have today for USB devices/drivers where we
have different levels of "authorized"?  There's no reason that could not
just move to the driver core and be available for all devices/drivers as
that model has proven to work really well over time.

See the "authorized" sysfs file documentation in
Documentation/ABI/testing/sysfs-bus-usb for some details.  Lots of
userspace tools have been built on top of that api to control how and
when specific USB devices are "allowed" to be used by the kernel and
userspace.

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

On Mon, Jun 1, 2020 at 10:06 PM Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Mon, Jun 01, 2020 at 06:25:42PM -0500, Bjorn Helgaas wrote:
> > [+cc Greg, linux-kernel for wider exposure]
>
> Thanks for the cc:, missed this...
>
> >
> > On Tue, May 26, 2020 at 09:30:08AM -0700, Rajat Jain wrote:
> > > On Thu, May 14, 2020 at 7:18 PM Rajat Jain <rajatja@google.com> wrote:
> > > > On Thu, May 14, 2020 at 12:13 PM Raj, Ashok <ashok.raj@intel.com> wrote:
> > > > > On Wed, May 13, 2020 at 02:26:18PM -0700, Rajat Jain wrote:
> > > > > > On Wed, May 13, 2020 at 8:19 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > > > On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > > > > > > > the firmware asks it to:
> > > > > > > >
> > > > > > > > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > > > > > > > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> > > > > > > >
> > > > > > > > An "untrusted" device indicates a (likely external facing) device that
> > > > > > > > may be malicious, and can trigger DMA attacks on the system. It may
> > > > > > > > also try to exploit any vulnerabilities exposed by the driver, that
> > > > > > > > may allow it to read/write unintended addresses in the host (e.g. if
> > > > > > > > DMA buffers for the device, share memory pages with other driver data
> > > > > > > > structures or code etc).
> > > > > > > >
> > > > > > > > High Level proposal
> > > > > > > > ===============
> > > > > > > > Currently, the "untrusted" device property is used as a hint to enable
> > > > > > > > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > > > > > > > go a step further, and allow the administrator to build a list of
> > > > > > > > whitelisted drivers for these "untrusted" devices. This whitelist of
> > > > > > > > drivers are the ones that he trusts enough to have little or no
> > > > > > > > vulnerabilities. (He may have built this list of whitelisted drivers
> > > > > > > > by a combination of code analysis of drivers, or by extensive testing
> > > > > > > > using PCIe fuzzing etc). We propose that the administrator be allowed
> > > > > > > > to specify this list of whitelisted drivers to the kernel, and the PCI
> > > > > > > > subsystem to impose this behavior:
> > > > > > > >
> > > > > > > > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > > > > > > > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> > > > > > > >
> > > > > > > > Of course this behavior is to be imposed only if such a whitelist is
> > > > > > > > provided by the administrator.
> > >
> > > I haven't heard much on this proposal after the initial inputs (to
> > > which I responded). Essentially, I agree that IO-MMU and ACS
> > > restrictions need to be put in plcase. But I think we need this
> > > additionally. Does this look acceptable to you? I wanted to start
> > > spinning out the patches, but wanted to see if there are any pending
> > > comments or concerns.
> >
> > I think it makes sense to code this up and see what it would look
> > like.  The bare minimum seems like a driver "bind-to-external-devices"
> > bit that's visible in sysfs plus something in the driver probe path
> > that checks it.  Neither is inherently PCI-specific, but maybe the
> > right place will become obvious when implementing it.


Agree. I'll try to code it up.

My proposal became PCI specific because

* The need for my proposal arrived out of the potentially malicious
*external* devices that can (NOW, with the advent of thunderbolt)
directly DMA into the CPU memory space. PCI (enabled by Thunderbolt 3
and USB4) is the only interface that fits the bill for laptops at
least (There are few more interfaces that allow DMA directly into host
memory such as LPC etc, but they are all internal so far).

* It hinges on the "untrusted" attribute (I see your concerns on this,
and more on this later) which is part of the "struct pci_dev". If we
can move that flag higher up to "struct device", then we can make this
proposal not PCI specific I think.

> >
> > I'm still not 100% sure the device "external/untrusted" bit is the
> > right thing to check.  If you don't trust a driver enough to expose it
> > to an external device, is it reasonable to trust it for internal
> > devices?  It seems like one could attack the driver of even an
> > internal device like a NIC by controlling the data fed to it.
> >
> > The existing use of "external/untrusted" for IOMMU protection is
> > different.  There we're acknowledging that the *device* itself is
> > unknown and we need to protect ourselves from malicious DMA.
> >
> > Here we're concerned about a *driver* that's completely under our
> > control.  If we don't trust the driver, we could (a) fix the problems
> > in the driver, (b) change the driver so it handles external/untrusted
> > devices differently, or (c) not ship the driver at all.

So here is our dilemma. In the laptop world:

1) Today (Pre-Thunderbolt 3 / Pre-USB4), there is a mix of trusted /
untrusted drivers that we (or any OEMs) are shipping on their laptops.
Yes, there is some (calculated) risk that everyone is taking - because
currently PCI bus does not extend outside the laptop *easily*. Yes I
understand systems may have external PCI slots, but that is rather
rare in the laptop world I think. The risks of the existing drivers
are limited to the devices that were built into the system, and since
the drivers, firmware updates, (and supply chain in some cases) are
controlled by us, such internal devices are conceivably more secure
than something random that the user may plug in. If the user opens the
chassis to replace a piece of hardware with something else, all bets
are off. Yes, we're still susceptible to the NIC driver attacks that
you talk about it along with other potential vulnerabilities, but this
is just convey our current baseline level of risk/security.

2) Now, we want to enable technology of tomorrow :-) (Thunderbolt 3 /
USB4) on laptops, which allows to very *easily* extend the internal
PCI bus to the outside devices. Note it doesn't require to open a
laptop, and anyone can plug a device onto a port. This throws the
system open to a lot of DMA attacks now, which it did not have to deal
with earlier. Essentially with the advent of technology to expand PCIe
outside of system chassis, the attacks have become much more easier,
we can no longer control or monitor device hardware or firmware, and
thus the level of risk has clearly increased. So what we are trying to
find here, is a good path to enable these new technologies, that keeps
keeps our baseline level of risk/security unchanged, and to also not
regress in functionality in supporting devices as much as possible.

3) Now you are certainly right that one path could be a binary
decision to ship or not ship a driver, or fix any issues with the
driver, or change the driver to differentiate between external and
internal ports. However, there are multiple factors that pose
practical problems (why regress internal devices that we tightly
control? Why regress systems that don't have such external ports? Need
to front load all effort in vetting the drivers before hand before the
first release. Work with each and individual driver etc).

4) The other path that this proposal aims to take is that by applying
a whitelist of drivers to external ports only, we're going to be able
to *slowly* build this whitelist. We can start with a NULL whitelist.
Which means that existing internal devices continue to work, and
external devices on PCI don't pose a risk. With ACS and IOMMU
restrictions in place, the security/risk baseline remains unchanged.
The existing devices are not regressed. As we vet and whitelist the
drivers, we start supporting more and more USB4 and Thunderbolt3
devices. Until then, those devices when plugged, can continue to work
in the "USB / legacy mode" (I forgot what it is called).

5) To give an example, assume we don't trust the PCI nvme driver and
don't want to whitelist it for external devices given there are so
many off the shelf devices with questionable firmware. But we
certainly need to enable it for internal NVME devices (that we may
have audited the firmware for, and control our supply chain) in order
to boot. With my proposal, until we whitelist it, the internal devices
continue to work, the external NVMEs switch to "USB storage device"
mode and thus go via a USB bridge so they cannot directly DMA into
host memory directly. Keep in mind that whitelisting a driver may be
handled by a separate security team, and may take long time depending
on the driver. The proposal allows us to release laptops with
Thunderbolt3/USB4 support and add peripheral support as we go.

6) Also small nit: consider the other scenario (I think this may not
be as important but still worth a thought). Assume the security team
finds a new vulnerability in a whitelisted driver, and want to take it
out of whitelist. Now, this really isn't possible if there was no
distinction between internal / external devices, and an internal
device uses that driver to boot.

> >
> > I'm also not sure about the administrative details of this.  Certain
> > versions of the driver may be trusted while others are untrusted.
> > That would all have to be managed in userspace, so it's not really our
> > problem, but it sounds like a hassle.  Putting the information in the
> > driver itself would reduce that.

I agree to the points you are making. *

- Who do you think should certify the driver? The driver maintainer?
Do we really think driver authors / maintainers will responsibly
update this field? Also, how often?

- Also, this being a policy decision, and thus best left outside the kernel?

- You are correct that version 1 may be trusted and version 2 may not
be. But I think that provides more reason for this to be maintained by
administrators. They (are supposed to) consciously uprev kernels (or
not), or cherry-pick security patches and can decide whether or not to
uprev a driver, or whether to pick a patch for the entire kernel on
their systems. In other words, they have more control over versions of
what they run on their system. OTOH, if this is maintained within the
upstream kernel by a driver maintainer, he may have no control over
other parts of the kernel (that may have an impact on his driver) and
also his driver needs to move on. I feel this has a lot of room for
mistakes.

>
> What about taking what we have today for USB devices/drivers where we
> have different levels of "authorized"?  There's no reason that could not
> just move to the driver core and be available for all devices/drivers as
> that model has proven to work really well over time.
>
> See the "authorized" sysfs file documentation in
> Documentation/ABI/testing/sysfs-bus-usb for some details.  Lots of
> userspace tools have been built on top of that api to control how and
> when specific USB devices are "allowed" to be used by the kernel and
> userspace.

Thanks for the pointer! I'm still looking at the details yet, but a
quick look (usb_dev_authorized()) seems to suggest that this API is
"device based". The multiple levels of "authorized" seem to take shape
from either how it is wired or from userspace choice. Once authorized,
USB device or interface is authorized to be used by *anyone* (can be
attached to any drivers). Do I understand it right that it does not
differentiate between drivers?

Thanks & Best Regards,

Rajat


>
> thanks,
>
> greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Wed, Jun 03, 2020 at 02:27:33AM +0000, Rajat Jain wrote:
> On Mon, Jun 1, 2020 at 10:06 PM Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Mon, Jun 01, 2020 at 06:25:42PM -0500, Bjorn Helgaas wrote:
> > > [+cc Greg, linux-kernel for wider exposure]
> >
> > Thanks for the cc:, missed this...
> >
> > >
> > > On Tue, May 26, 2020 at 09:30:08AM -0700, Rajat Jain wrote:
> > > > On Thu, May 14, 2020 at 7:18 PM Rajat Jain <rajatja@google.com> wrote:
> > > > > On Thu, May 14, 2020 at 12:13 PM Raj, Ashok <ashok.raj@intel.com> wrote:
> > > > > > On Wed, May 13, 2020 at 02:26:18PM -0700, Rajat Jain wrote:
> > > > > > > On Wed, May 13, 2020 at 8:19 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > > > > > On Fri, May 01, 2020 at 04:07:10PM -0700, Rajat Jain wrote:
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > Currently, the PCI subsystem marks the PCI devices as "untrusted", if
> > > > > > > > > the firmware asks it to:
> > > > > > > > >
> > > > > > > > > 617654aae50e ("PCI / ACPI: Identify untrusted PCI devices")
> > > > > > > > > 9cb30a71acd4 ("PCI: OF: Support "external-facing" property")
> > > > > > > > >
> > > > > > > > > An "untrusted" device indicates a (likely external facing) device that
> > > > > > > > > may be malicious, and can trigger DMA attacks on the system. It may
> > > > > > > > > also try to exploit any vulnerabilities exposed by the driver, that
> > > > > > > > > may allow it to read/write unintended addresses in the host (e.g. if
> > > > > > > > > DMA buffers for the device, share memory pages with other driver data
> > > > > > > > > structures or code etc).
> > > > > > > > >
> > > > > > > > > High Level proposal
> > > > > > > > > ===============
> > > > > > > > > Currently, the "untrusted" device property is used as a hint to enable
> > > > > > > > > IOMMU restrictions (on Intel), disable ATS (on ARM) etc. We'd like to
> > > > > > > > > go a step further, and allow the administrator to build a list of
> > > > > > > > > whitelisted drivers for these "untrusted" devices. This whitelist of
> > > > > > > > > drivers are the ones that he trusts enough to have little or no
> > > > > > > > > vulnerabilities. (He may have built this list of whitelisted drivers
> > > > > > > > > by a combination of code analysis of drivers, or by extensive testing
> > > > > > > > > using PCIe fuzzing etc). We propose that the administrator be allowed
> > > > > > > > > to specify this list of whitelisted drivers to the kernel, and the PCI
> > > > > > > > > subsystem to impose this behavior:
> > > > > > > > >
> > > > > > > > > 1) The "untrusted" devices can bind to only "whitelisted drivers".
> > > > > > > > > 2) The other devices (i.e. dev->untrusted=0) can bind to any driver.
> > > > > > > > >
> > > > > > > > > Of course this behavior is to be imposed only if such a whitelist is
> > > > > > > > > provided by the administrator.
> > > >
> > > > I haven't heard much on this proposal after the initial inputs (to
> > > > which I responded). Essentially, I agree that IO-MMU and ACS
> > > > restrictions need to be put in plcase. But I think we need this
> > > > additionally. Does this look acceptable to you? I wanted to start
> > > > spinning out the patches, but wanted to see if there are any pending
> > > > comments or concerns.
> > >
> > > I think it makes sense to code this up and see what it would look
> > > like.  The bare minimum seems like a driver "bind-to-external-devices"
> > > bit that's visible in sysfs plus something in the driver probe path
> > > that checks it.  Neither is inherently PCI-specific, but maybe the
> > > right place will become obvious when implementing it.
> 
> 
> Agree. I'll try to code it up.
> 
> My proposal became PCI specific because
> 
> * The need for my proposal arrived out of the potentially malicious
> *external* devices that can (NOW, with the advent of thunderbolt)
> directly DMA into the CPU memory space. PCI (enabled by Thunderbolt 3
> and USB4) is the only interface that fits the bill for laptops at
> least (There are few more interfaces that allow DMA directly into host
> memory such as LPC etc, but they are all internal so far).

Again, that fits in exactly with what USB does, so you should just steal
that code and move it into the driver core for all busses to use.

> > > I'm also not sure about the administrative details of this.  Certain
> > > versions of the driver may be trusted while others are untrusted.
> > > That would all have to be managed in userspace, so it's not really our
> > > problem, but it sounds like a hassle.  Putting the information in the
> > > driver itself would reduce that.
> 
> I agree to the points you are making. *
> 
> - Who do you think should certify the driver? The driver maintainer?
> Do we really think driver authors / maintainers will responsibly
> update this field? Also, how often?

No, that way lies madness, don't have "certified" drivers or anything
like that.  Just put policy in place for if you can trust the _device_
or not.

> - Also, this being a policy decision, and thus best left outside the kernel?

Yes.  There are tools already that do this today for USB, take a look at
how they work for examples of the process.

> Thanks for the pointer! I'm still looking at the details yet, but a
> quick look (usb_dev_authorized()) seems to suggest that this API is
> "device based". The multiple levels of "authorized" seem to take shape
> from either how it is wired or from userspace choice. Once authorized,
> USB device or interface is authorized to be used by *anyone* (can be
> attached to any drivers). Do I understand it right that it does not
> differentiate between drivers?

Yes, and that is what you should do, don't fixate on drivers.  Users
know how to control and manage devices.  Us kernel developers are
responsible for writing solid drivers and getting them merged into the
kernel tree and maintaining them over time.  Drivers in the kernel
should always be trusted, if not, then we have bigger architectural
issues we need to deal with.

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

Hello,

>
> > Thanks for the pointer! I'm still looking at the details yet, but a
> > quick look (usb_dev_authorized()) seems to suggest that this API is
> > "device based". The multiple levels of "authorized" seem to take shape
> > from either how it is wired or from userspace choice. Once authorized,
> > USB device or interface is authorized to be used by *anyone* (can be
> > attached to any drivers). Do I understand it right that it does not
> > differentiate between drivers?
>
> Yes, and that is what you should do, don't fixate on drivers.  Users
> know how to control and manage devices.  Us kernel developers are
> responsible for writing solid drivers and getting them merged into the
> kernel tree and maintaining them over time.  Drivers in the kernel
> should always be trusted, ...

1) Yes, I agree that this would be ideal, and this should be our
mission. I should clarify that I may have used the wrong term
"Trusted/Certified drivers". I didn't really mean that the drivers may
be malicious by intent. What I really meant is that a driver may have
an attack surface, which is a vulnerability that may be exploited.
Realistically speaking, finding vulnerabilities in drivers, creating
attacks to exploit them, and fixing them is a never ending cat and
mouse game. At Least "identifying the vulnerabilities" part is better
performed by security folks rather than driver writers. Earlier in the
thread I had mentioned certain studies/projects that identified and
exploited such vulnerabilities in the drivers. I should have used the
term "Vetted Drivers" maybe to convey the intent better - drivers that
have been vetted by a security focussed team (admin). What I'm
advocating here is an administrator's right to control the drivers
that he wants to allow for external ports on his systems.

2) In addition to the problem of driver negligences / vulnerabilities
to be exploited, we ran into another problem with the "whitelist
devices only" approach. We did start with the "device based" approach
only initially - but quickly realized that anything we use to
whitelist an external device can only be based on the info provided by
*that device* itself. So until we have devices that exchange
certificates with kernel [1], it is easy for a malicious device to
spoof a whitelisted device (by presenting the same VID:DID or any
other data that is used by us to whitelist it).

[1] https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html

I hope that helps somewhat clarify how / why we reached here?

Thanks & Best Regards,

Rajat

> thanks,
>
> greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Wed, Jun 03, 2020 at 04:51:18AM -0700, Rajat Jain wrote:
> Hello,
> 
> >
> > > Thanks for the pointer! I'm still looking at the details yet, but a
> > > quick look (usb_dev_authorized()) seems to suggest that this API is
> > > "device based". The multiple levels of "authorized" seem to take shape
> > > from either how it is wired or from userspace choice. Once authorized,
> > > USB device or interface is authorized to be used by *anyone* (can be
> > > attached to any drivers). Do I understand it right that it does not
> > > differentiate between drivers?
> >
> > Yes, and that is what you should do, don't fixate on drivers.  Users
> > know how to control and manage devices.  Us kernel developers are
> > responsible for writing solid drivers and getting them merged into the
> > kernel tree and maintaining them over time.  Drivers in the kernel
> > should always be trusted, ...
> 
> 1) Yes, I agree that this would be ideal, and this should be our
> mission. I should clarify that I may have used the wrong term
> "Trusted/Certified drivers". I didn't really mean that the drivers may
> be malicious by intent. What I really meant is that a driver may have
> an attack surface, which is a vulnerability that may be exploited.

Any code has such a thing, proving otherwise is a tough problem :)

> Realistically speaking, finding vulnerabilities in drivers, creating
> attacks to exploit them, and fixing them is a never ending cat and
> mouse game. At Least "identifying the vulnerabilities" part is better
> performed by security folks rather than driver writers.

Are you sure about that?  It's hard to prove a negative :)

> Earlier in the
> thread I had mentioned certain studies/projects that identified and
> exploited such vulnerabilities in the drivers. I should have used the
> term "Vetted Drivers" maybe to convey the intent better - drivers that
> have been vetted by a security focussed team (admin). What I'm
> advocating here is an administrator's right to control the drivers
> that he wants to allow for external ports on his systems.

That's an odd thing, but sure, if you want to write up such a policy for
your systems, great.  But that policy does not belong in the kernel, it
belongs in userspace.

> 2) In addition to the problem of driver negligences / vulnerabilities
> to be exploited, we ran into another problem with the "whitelist
> devices only" approach. We did start with the "device based" approach
> only initially - but quickly realized that anything we use to
> whitelist an external device can only be based on the info provided by
> *that device* itself. So until we have devices that exchange
> certificates with kernel [1], it is easy for a malicious device to
> spoof a whitelisted device (by presenting the same VID:DID or any
> other data that is used by us to whitelist it).
> 
> [1] https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html
> 
> I hope that helps somewhat clarify how / why we reached here?

Kind of, I still think all you need to do is worry about controling the
devices and if a driver should bind to it or not.  Again, much like USB
has been doing for a very long time now.  The idea of "spoofing" ids
also is not new, and has been around for a very long time as well, and
again, the controls that the USB core gives you allows you to make any
type of policy decision you want to, in userspace.

So please, in summary:
	- don't think this is some new type of thing, it's an old issue
	  transferred to yet-another-hardware-bus.  Not to say this is
	  not important, just please look at the work that others have
	  done in the past to help mitigate and solve this (reading the
	  Wireless USB spec should help you out here too, as they
	  detailed all of this.)
	- do copy what USB did, by moving that logic into the driver
	  core so that all busses who want to take advantage of this
	  type of functionality, easily can do so.

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

Hi Greg,

Thanks for looking into this thread.

On Wed, Jun 3, 2020 at 5:16 AM Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Wed, Jun 03, 2020 at 04:51:18AM -0700, Rajat Jain wrote:
> > Hello,
> >
> > >
> > > > Thanks for the pointer! I'm still looking at the details yet, but a
> > > > quick look (usb_dev_authorized()) seems to suggest that this API is
> > > > "device based". The multiple levels of "authorized" seem to take shape
> > > > from either how it is wired or from userspace choice. Once authorized,
> > > > USB device or interface is authorized to be used by *anyone* (can be
> > > > attached to any drivers). Do I understand it right that it does not
> > > > differentiate between drivers?
> > >
> > > Yes, and that is what you should do, don't fixate on drivers.  Users
> > > know how to control and manage devices.  Us kernel developers are
> > > responsible for writing solid drivers and getting them merged into the
> > > kernel tree and maintaining them over time.  Drivers in the kernel
> > > should always be trusted, ...
> >
> > 1) Yes, I agree that this would be ideal, and this should be our
> > mission. I should clarify that I may have used the wrong term
> > "Trusted/Certified drivers". I didn't really mean that the drivers may
> > be malicious by intent. What I really meant is that a driver may have
> > an attack surface, which is a vulnerability that may be exploited.
>
> Any code has such a thing, proving otherwise is a tough problem :)
>
> > Realistically speaking, finding vulnerabilities in drivers, creating
> > attacks to exploit them, and fixing them is a never ending cat and
> > mouse game. At Least "identifying the vulnerabilities" part is better
> > performed by security folks rather than driver writers.
>
> Are you sure about that?  It's hard to prove a negative :)
>
> > Earlier in the
> > thread I had mentioned certain studies/projects that identified and
> > exploited such vulnerabilities in the drivers. I should have used the
> > term "Vetted Drivers" maybe to convey the intent better - drivers that
> > have been vetted by a security focussed team (admin). What I'm
> > advocating here is an administrator's right to control the drivers
> > that he wants to allow for external ports on his systems.
>
> That's an odd thing, but sure, if you want to write up such a policy for
> your systems, great.  But that policy does not belong in the kernel, it
> belongs in userspace.

Agree.

>
> > 2) In addition to the problem of driver negligences / vulnerabilities
> > to be exploited, we ran into another problem with the "whitelist
> > devices only" approach. We did start with the "device based" approach
> > only initially - but quickly realized that anything we use to
> > whitelist an external device can only be based on the info provided by
> > *that device* itself. So until we have devices that exchange
> > certificates with kernel [1], it is easy for a malicious device to
> > spoof a whitelisted device (by presenting the same VID:DID or any
> > other data that is used by us to whitelist it).
> >
> > [1] https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html
> >
> > I hope that helps somewhat clarify how / why we reached here?
>
> Kind of, I still think all you need to do is worry about controlling the
> devices and if a driver should bind to it or not.

Agree. That is precisely what this RFC had in mind: (1) controlling
whether a device is authorized and if so (2) What drivers can bind to
it.

>  Again, much like USB
> has been doing for a very long time now.  The idea of "spoofing" ids
> also is not new, and has been around for a very long time as well, and
> again, the controls that the USB core gives you allows you to make any
> type of policy decision you want to, in userspace.
>
> So please, in summary:
>         - don't think this is some new type of thing, it's an old issue
>           transferred to yet-another-hardware-bus.  Not to say this is
>           not important, just please look at the work that others have
>           done in the past to help mitigate and solve this (reading the
>           Wireless USB spec should help you out here too, as they
>           detailed all of this.)
>         - do copy what USB did, by moving that logic into the driver
>           core so that all busses who want to take advantage of this
>           type of functionality, easily can do so.

Understood, will keep that in mind. I may first present a "PCI
subsystem only" draft just to get a feel for it, since that is more
familiar to me and also already has some bits and pieces we need for
this.

Thanks & Best Regards,

Rajat

>
> thanks,
>
> greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Wed, Jun 03, 2020 at 05:57:02AM -0700, Rajat Jain wrote:
> > So please, in summary:
> >         - don't think this is some new type of thing, it's an old issue
> >           transferred to yet-another-hardware-bus.  Not to say this is
> >           not important, just please look at the work that others have
> >           done in the past to help mitigate and solve this (reading the
> >           Wireless USB spec should help you out here too, as they
> >           detailed all of this.)
> >         - do copy what USB did, by moving that logic into the driver
> >           core so that all busses who want to take advantage of this
> >           type of functionality, easily can do so.
> 
> Understood, will keep that in mind. I may first present a "PCI
> subsystem only" draft just to get a feel for it, since that is more
> familiar to me and also already has some bits and pieces we need for
> this.

Why?  Do it right the first time.  To waste reviewer's time on something
that you know you have to throw away and do it "correctly" again, is not
very nice.  I know I would put you on the bottom of my "patches to
review" list if you did that to me :)

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

Hello,

I spent some more thoughts into this...

On Wed, Jun 3, 2020 at 5:16 AM Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Wed, Jun 03, 2020 at 04:51:18AM -0700, Rajat Jain wrote:
> > Hello,
> >
> > >
> > > > Thanks for the pointer! I'm still looking at the details yet, but a
> > > > quick look (usb_dev_authorized()) seems to suggest that this API is
> > > > "device based". The multiple levels of "authorized" seem to take shape
> > > > from either how it is wired or from userspace choice. Once authorized,
> > > > USB device or interface is authorized to be used by *anyone* (can be
> > > > attached to any drivers). Do I understand it right that it does not
> > > > differentiate between drivers?
> > >
> > > Yes, and that is what you should do, don't fixate on drivers.  Users
> > > know how to control and manage devices.  Us kernel developers are
> > > responsible for writing solid drivers and getting them merged into the
> > > kernel tree and maintaining them over time.  Drivers in the kernel
> > > should always be trusted, ...
> >
> > 1) Yes, I agree that this would be ideal, and this should be our
> > mission. I should clarify that I may have used the wrong term
> > "Trusted/Certified drivers". I didn't really mean that the drivers may
> > be malicious by intent. What I really meant is that a driver may have
> > an attack surface, which is a vulnerability that may be exploited.
>
> Any code has such a thing, proving otherwise is a tough problem :)
>
> > Realistically speaking, finding vulnerabilities in drivers, creating
> > attacks to exploit them, and fixing them is a never ending cat and
> > mouse game. At Least "identifying the vulnerabilities" part is better
> > performed by security folks rather than driver writers.
>
> Are you sure about that?  It's hard to prove a negative :)
>
> > Earlier in the
> > thread I had mentioned certain studies/projects that identified and
> > exploited such vulnerabilities in the drivers. I should have used the
> > term "Vetted Drivers" maybe to convey the intent better - drivers that
> > have been vetted by a security focussed team (admin). What I'm
> > advocating here is an administrator's right to control the drivers
> > that he wants to allow for external ports on his systems.
>
> That's an odd thing, but sure, if you want to write up such a policy for
> your systems, great.  But that policy does not belong in the kernel, it
> belongs in userspace.
>
> > 2) In addition to the problem of driver negligences / vulnerabilities
> > to be exploited, we ran into another problem with the "whitelist
> > devices only" approach. We did start with the "device based" approach
> > only initially - but quickly realized that anything we use to
> > whitelist an external device can only be based on the info provided by
> > *that device* itself. So until we have devices that exchange
> > certificates with kernel [1], it is easy for a malicious device to
> > spoof a whitelisted device (by presenting the same VID:DID or any
> > other data that is used by us to whitelist it).
> >
> > [1] https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html
> >
> > I hope that helps somewhat clarify how / why we reached here?
>
> Kind of, I still think all you need to do is worry about controling the
> devices and if a driver should bind to it or not.  Again, much like USB
> has been doing for a very long time now.  The idea of "spoofing" ids
> also is not new, and has been around for a very long time as well, and
> again, the controls that the USB core gives you allows you to make any
> type of policy decision you want to, in userspace.

Er, *currently* it doesn't allow the userspace to make the particular
policy I want to, right? Specifically, today an administrator can not
control which USB *drivers* he wants to allow on an *external* USB
port. He can only control which USB devices he wants to authorize, but
once authorized, they are free to bind to any of the USB drivers. So
if I want to allow the administrator to implement a policy that allows
him to control the drivers for external ports, we'll need to enhance
the current code (whether we want to do it specific to a bus, or more
generically in the driver core). Are we on the same page?

To implement the policy that I want to in the driver core, what is
missing today in driver core is a distinction between "internal" and
"external" devices. Some buses have this knowledge locally today (PCI
has "untrusted" flag which can be used, USB uses hcd->wireless and
hub->port->connect_type) but it is not shared with the core.

So just to make sure if I'm thinking in the right direction, this is
what I'm thinking:

1) The device core needs a notion of internal vs external devices (a
flag) - a knowledge that needs to be filled in by the bus as it
discovers the device.

2) The driver core needs to allow an admin to provide a whitelist of
drivers for external devices. (Via Command line or a driver flag.
Default = everything is whitelisted).

3) While matching a driver to a device, the driver core needs to
impose the whitelist if the device is external, and if the
administrator has provided a whitelist.

Any bus that wants to use this can use it if it wants to, for external devices.

Thoughts?

Thanks & Best Regards,

Rajat

>
> So please, in summary:
>         - don't think this is some new type of thing, it's an old issue
>           transferred to yet-another-hardware-bus.  Not to say this is
>           not important, just please look at the work that others have
>           done in the past to help mitigate and solve this (reading the
>           Wireless USB spec should help you out here too, as they
>           detailed all of this.)
>         - do copy what USB did, by moving that logic into the driver
>           core so that all busses who want to take advantage of this
>           type of functionality, easily can do so.
>
> thanks,
>
> greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Thu, Jun 04, 2020 at 12:38:18PM -0700, Rajat Jain wrote:
> Hello,
> 
> I spent some more thoughts into this...
> 
> On Wed, Jun 3, 2020 at 5:16 AM Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Wed, Jun 03, 2020 at 04:51:18AM -0700, Rajat Jain wrote:
> > > Hello,
> > >
> > > >
> > > > > Thanks for the pointer! I'm still looking at the details yet, but a
> > > > > quick look (usb_dev_authorized()) seems to suggest that this API is
> > > > > "device based". The multiple levels of "authorized" seem to take shape
> > > > > from either how it is wired or from userspace choice. Once authorized,
> > > > > USB device or interface is authorized to be used by *anyone* (can be
> > > > > attached to any drivers). Do I understand it right that it does not
> > > > > differentiate between drivers?
> > > >
> > > > Yes, and that is what you should do, don't fixate on drivers.  Users
> > > > know how to control and manage devices.  Us kernel developers are
> > > > responsible for writing solid drivers and getting them merged into the
> > > > kernel tree and maintaining them over time.  Drivers in the kernel
> > > > should always be trusted, ...
> > >
> > > 1) Yes, I agree that this would be ideal, and this should be our
> > > mission. I should clarify that I may have used the wrong term
> > > "Trusted/Certified drivers". I didn't really mean that the drivers may
> > > be malicious by intent. What I really meant is that a driver may have
> > > an attack surface, which is a vulnerability that may be exploited.
> >
> > Any code has such a thing, proving otherwise is a tough problem :)
> >
> > > Realistically speaking, finding vulnerabilities in drivers, creating
> > > attacks to exploit them, and fixing them is a never ending cat and
> > > mouse game. At Least "identifying the vulnerabilities" part is better
> > > performed by security folks rather than driver writers.
> >
> > Are you sure about that?  It's hard to prove a negative :)
> >
> > > Earlier in the
> > > thread I had mentioned certain studies/projects that identified and
> > > exploited such vulnerabilities in the drivers. I should have used the
> > > term "Vetted Drivers" maybe to convey the intent better - drivers that
> > > have been vetted by a security focussed team (admin). What I'm
> > > advocating here is an administrator's right to control the drivers
> > > that he wants to allow for external ports on his systems.
> >
> > That's an odd thing, but sure, if you want to write up such a policy for
> > your systems, great.  But that policy does not belong in the kernel, it
> > belongs in userspace.
> >
> > > 2) In addition to the problem of driver negligences / vulnerabilities
> > > to be exploited, we ran into another problem with the "whitelist
> > > devices only" approach. We did start with the "device based" approach
> > > only initially - but quickly realized that anything we use to
> > > whitelist an external device can only be based on the info provided by
> > > *that device* itself. So until we have devices that exchange
> > > certificates with kernel [1], it is easy for a malicious device to
> > > spoof a whitelisted device (by presenting the same VID:DID or any
> > > other data that is used by us to whitelist it).
> > >
> > > [1] https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html
> > >
> > > I hope that helps somewhat clarify how / why we reached here?
> >
> > Kind of, I still think all you need to do is worry about controling the
> > devices and if a driver should bind to it or not.  Again, much like USB
> > has been doing for a very long time now.  The idea of "spoofing" ids
> > also is not new, and has been around for a very long time as well, and
> > again, the controls that the USB core gives you allows you to make any
> > type of policy decision you want to, in userspace.
> 
> Er, *currently* it doesn't allow the userspace to make the particular
> policy I want to, right? Specifically, today an administrator can not
> control which USB *drivers* he wants to allow on an *external* USB
> port.

Not true, you can do that today with the explicit binding/unbinding of
devices to drivers in userspace.  Been there for many decades :)

But, think this through, since when do you have _multiple_ drivers that
have support to control the same type of device?  We almost never allow
that in the kernel today as that way lies madness (no heiarchy of
drivers to bind to what devices and so on.)

We always strive to keep a one-to-one mapping of "this device is only
allowed to be controlled by this one driver" today, why would you want
to change that basic premise now?

> He can only control which USB devices he wants to authorize, but
> once authorized, they are free to bind to any of the USB drivers.

Since when do different drivers control the same type of USB device?  :)

> So if I want to allow the administrator to implement a policy that
> allows him to control the drivers for external ports, we'll need to
> enhance the current code (whether we want to do it specific to a bus,
> or more generically in the driver core). Are we on the same page?
> 
> To implement the policy that I want to in the driver core, what is
> missing today in driver core is a distinction between "internal" and
> "external" devices. Some buses have this knowledge locally today (PCI
> has "untrusted" flag which can be used, USB uses hcd->wireless and
> hub->port->connect_type) but it is not shared with the core.

Note the wireless USB code should now be gone from the tree.  If you see
any remants of it floating around, let me know and I will remove them, I
think there might be a few bits left that I missed.

> So just to make sure if I'm thinking in the right direction, this is
> what I'm thinking:
> 
> 1) The device core needs a notion of internal vs external devices (a
> flag) - a knowledge that needs to be filled in by the bus as it
> discovers the device.

Nope, don't go down this path.  We tried to do this for USB where the
BIOS tells us that a device is "internal" vs. "external" but in reality,
BIOSes get this wrong and it's not always all that useful.

And why would you somehow "trust" a device that is in your system more
than one is not?  The same driver binds to it no matter what (as I state
above), so you should be able to trust it the same.

> 2) The driver core needs to allow an admin to provide a whitelist of
> drivers for external devices. (Via Command line or a driver flag.
> Default = everything is whitelisted).

Again, nope, no difference, see above.

> 3) While matching a driver to a device, the driver core needs to
> impose the whitelist if the device is external, and if the
> administrator has provided a whitelist.

Ick, no, again, work on a per-device authorized setting.  That way it
works the same all across the system.  Don't get stuck in a "external
vs. internal" discussion as this will get messy really quickly (think
about "internal" devices with "external" links to them like PCI
"drawers" of devices that we currently support on large systems.  Or
things like thunderbolt hubs with "internal" devices like I have on my
desktop right now.

In summary, if a driver is "trusted enough" to control an internal
device, it should be "trusted enough" to control an external device.  If
not, then fix that driver so that you do "trust" it.

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

Hello Greg,

Thank you for continuing to work with me through this.

On Fri, Jun 5, 2020 at 1:02 AM Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Thu, Jun 04, 2020 at 12:38:18PM -0700, Rajat Jain wrote:
> > Hello,
> >
> > I spent some more thoughts into this...
> >
> > On Wed, Jun 3, 2020 at 5:16 AM Greg Kroah-Hartman
> > <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Wed, Jun 03, 2020 at 04:51:18AM -0700, Rajat Jain wrote:
> > > > Hello,
> > > >
> > > > >
> > > > > > Thanks for the pointer! I'm still looking at the details yet, but a
> > > > > > quick look (usb_dev_authorized()) seems to suggest that this API is
> > > > > > "device based". The multiple levels of "authorized" seem to take shape
> > > > > > from either how it is wired or from userspace choice. Once authorized,
> > > > > > USB device or interface is authorized to be used by *anyone* (can be
> > > > > > attached to any drivers). Do I understand it right that it does not
> > > > > > differentiate between drivers?
> > > > >
> > > > > Yes, and that is what you should do, don't fixate on drivers.  Users
> > > > > know how to control and manage devices.  Us kernel developers are
> > > > > responsible for writing solid drivers and getting them merged into the
> > > > > kernel tree and maintaining them over time.  Drivers in the kernel
> > > > > should always be trusted, ...
> > > >
> > > > 1) Yes, I agree that this would be ideal, and this should be our
> > > > mission. I should clarify that I may have used the wrong term
> > > > "Trusted/Certified drivers". I didn't really mean that the drivers may
> > > > be malicious by intent. What I really meant is that a driver may have
> > > > an attack surface, which is a vulnerability that may be exploited.
> > >
> > > Any code has such a thing, proving otherwise is a tough problem :)
> > >
> > > > Realistically speaking, finding vulnerabilities in drivers, creating
> > > > attacks to exploit them, and fixing them is a never ending cat and
> > > > mouse game. At Least "identifying the vulnerabilities" part is better
> > > > performed by security folks rather than driver writers.
> > >
> > > Are you sure about that?  It's hard to prove a negative :)
> > >
> > > > Earlier in the
> > > > thread I had mentioned certain studies/projects that identified and
> > > > exploited such vulnerabilities in the drivers. I should have used the
> > > > term "Vetted Drivers" maybe to convey the intent better - drivers that
> > > > have been vetted by a security focussed team (admin). What I'm
> > > > advocating here is an administrator's right to control the drivers
> > > > that he wants to allow for external ports on his systems.
> > >
> > > That's an odd thing, but sure, if you want to write up such a policy for
> > > your systems, great.  But that policy does not belong in the kernel, it
> > > belongs in userspace.
> > >
> > > > 2) In addition to the problem of driver negligences / vulnerabilities
> > > > to be exploited, we ran into another problem with the "whitelist
> > > > devices only" approach. We did start with the "device based" approach
> > > > only initially - but quickly realized that anything we use to
> > > > whitelist an external device can only be based on the info provided by
> > > > *that device* itself. So until we have devices that exchange
> > > > certificates with kernel [1], it is easy for a malicious device to
> > > > spoof a whitelisted device (by presenting the same VID:DID or any
> > > > other data that is used by us to whitelist it).
> > > >
> > > > [1] https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html
> > > >
> > > > I hope that helps somewhat clarify how / why we reached here?
> > >
> > > Kind of, I still think all you need to do is worry about controling the
> > > devices and if a driver should bind to it or not.  Again, much like USB
> > > has been doing for a very long time now.  The idea of "spoofing" ids
> > > also is not new, and has been around for a very long time as well, and
> > > again, the controls that the USB core gives you allows you to make any
> > > type of policy decision you want to, in userspace.
> >
> > Er, *currently* it doesn't allow the userspace to make the particular
> > policy I want to, right? Specifically, today an administrator can not
> > control which USB *drivers* he wants to allow on an *external* USB
> > port.
>
> Not true, you can do that today with the explicit binding/unbinding of
> devices to drivers in userspace.  Been there for many decades :)

Not sure if I understood. Can you please elaborate how that helps
implement the policy I want?

>
> But, think this through, since when do you have _multiple_ drivers that
> have support to control the same type of device?  We almost never allow
> that in the kernel today as that way lies madness (no heiarchy of
> drivers to bind to what devices and so on.)
>
> We always strive to keep a one-to-one mapping of "this device is only
> allowed to be controlled by this one driver" today, why would you want
> to change that basic premise now?

No, I don't want to change that premise. Multiple drivers for a single
device is not the goal at all.

>
> > He can only control which USB devices he wants to authorize, but
> > once authorized, they are free to bind to any of the USB drivers.
>
> Since when do different drivers control the same type of USB device?  :)

Sorry, I should have used better wording:
"..., a malicious device can choose which (authorized/whitelisted)
device to spoof, and *thus* choose a driver to attach to"
Since the only data admin can use to decide to authorize the device,
is provided by device itself, authorization is really just a farce.

From Documentation/usb/authorization.rst:
"Just checking if the class, type and protocol match something is the worse
security verification you can make (or the best, for someone willing
to break it). If you need something secure, use crypto and Certificate
Authentication or stuff like that."

Truth be told, there is nothing else really available today. While
certificate exchanges may be the future, the challenge is to deal with
devices at hand.

> > So if I want to allow the administrator to implement a policy that
> > allows him to control the drivers for external ports, we'll need to
> > enhance the current code (whether we want to do it specific to a bus,
> > or more generically in the driver core). Are we on the same page?
> >
> > To implement the policy that I want to in the driver core, what is
> > missing today in driver core is a distinction between "internal" and
> > "external" devices. Some buses have this knowledge locally today (PCI
> > has "untrusted" flag which can be used, USB uses hcd->wireless and
> > hub->port->connect_type) but it is not shared with the core.
>
> Note the wireless USB code should now be gone from the tree.  If you see
> any remants of it floating around, let me know and I will remove them, I
> think there might be a few bits left that I missed.
>
> > So just to make sure if I'm thinking in the right direction, this is
> > what I'm thinking:
> >
> > 1) The device core needs a notion of internal vs external devices (a
> > flag) - a knowledge that needs to be filled in by the bus as it
> > discovers the device.
>
> Nope, don't go down this path.  We tried to do this for USB where the
> BIOS tells us that a device is "internal" vs. "external" but in reality,
> BIOSes get this wrong and it's not always all that useful.
>
> And why would you somehow "trust" a device that is in your system more
> than one is not?  The same driver binds to it no matter what (as I state
> above), so you should be able to trust it the same.

There are multiple reasons for trust level to be different for
"internal" vs "external" devices. Speaking for the laptop world at
least (and I suspect same is true for most of OEM products):

1) The hardware, firmware, and in some cases even the supply chain is
quite tightly controlled and audited for "internal devices". OTOH, we
don't even know what an "external device" may look like.

2) Most of the internal devices are soldered on board, and can't be
accessed by a malicious person, atleast without the owner knowing.
OTOH, external device attack is very easy (Imagine malicious user
plugging in a USB stick on an unattended laptop / internet cafes /
airports etc) - the owner wouldn't even know.

3) Internal devices do not physically travel to another system. OTOH,
external devices, even if not malicious, may get infected since they
travel between multiple systems.

4) New devices' support (new drivers) keeps on getting added in the
kernel (which is a good thing!). But that means while the "internal
device space" is fixed at product release, the "external device space"
is unbounded and keeps on increasing. It is good from a functionality
point of view, but not from a security point of view.

>
> > 2) The driver core needs to allow an admin to provide a whitelist of
> > drivers for external devices. (Via Command line or a driver flag.
> > Default = everything is whitelisted).
>
> Again, nope, no difference, see above.
>
> > 3) While matching a driver to a device, the driver core needs to
> > impose the whitelist if the device is external, and if the
> > administrator has provided a whitelist.
>
> Ick, no, again, work on a per-device authorized setting.  That way it
> works the same all across the system.  Don't get stuck in a "external
> vs. internal" discussion as this will get messy really quickly (think
> about "internal" devices with "external" links to them like PCI
> "drawers" of devices that we currently support on large systems.  Or
> things like thunderbolt hubs with "internal" devices like I have on my
> desktop right now.

Good point. I'm not good at terminologies - by "external", I meant
anything that is not in the physical boundary of the host system as
shipped. It should be defined by the individual buses who learn it
from the platform (BIOS / Device tree / Discovery process etc).

>
> In summary, if a driver is "trusted enough" to control an internal
> device, it should be "trusted enough" to control an external device.  If
> not, then fix that driver so that you do "trust" it.

That is indeed our goal, and we do intend to inspect and send patch
fixes upstream. However the problem is inspecting drivers for security
and finding and fixing issues is a long drawn process - and has a lot
of dependencies on different maintainers. It is not possible to front
load all the effort and release the product only when *all* drivers
are fixed (We want to begin with a *NULL* whitelist of drivers and
then build it *slowly*). OTOH, it would be unfair to block a product
because not all drivers could be inspected or fixed in time for
security issues.

I feel that I have failed to explain the problems clearly. I'm copying
a blurb earlier from this thread, which explains the context and the
problem we're trying to solve.

================ BEGIN ============================
So here is our dilemma. In the laptop world:

1) Today (Pre-Thunderbolt 3 / Pre-USB4), there is a mix of trusted /
untrusted drivers that we (or any OEMs) are shipping on their laptops.
Yes, there is some (calculated) risk that everyone is taking - because
currently PCI bus does not extend outside the laptop *easily*. Yes I
understand systems may have external PCI slots, but that is rather
rare in the laptop world I think. The risks of the existing drivers
are limited to the devices that were built into the system, and since
the drivers, firmware updates, (and supply chain in some cases) are
controlled by us, such internal devices are conceivably more secure
than something random that the user may plug in. If the user opens the
chassis to replace a piece of hardware with something else, all bets
are off. Yes, we're still susceptible to the NIC driver attacks that
you talk about it along with other potential vulnerabilities, but this
is just convey our current baseline level of risk/security.

2) Now, we want to enable technology of tomorrow :-) (Thunderbolt 3 /
USB4) on laptops, which allows to very *easily* extend the internal
PCI bus to the outside devices. Note it doesn't require to open a
laptop, and anyone can plug a device onto a port. This throws the
system open to a lot of DMA attacks now, which it did not have to deal
with earlier. Essentially with the advent of technology to expand PCIe
outside of system chassis, the attacks have become much more easier,
we can no longer control or monitor device hardware or firmware, and
thus the level of risk has clearly increased. So what we are trying to
find here, is a good path to enable these new technologies, that keeps
keeps our baseline level of risk/security unchanged, and to also not
regress in functionality in supporting devices as much as possible.

3) Now you are certainly right that one path could be a binary
decision to ship or not ship a driver, or fix any issues with the
driver, or change the driver to differentiate between external and
internal ports. However, there are multiple factors that pose
practical problems (why regress internal devices that we tightly
control? Why regress systems that don't have such external ports? Need
to front load all effort in vetting the drivers before hand before the
first release. Work with each and individual driver etc).

4) The other path that this proposal aims to take is that by applying
a whitelist of drivers to external ports only, we're going to be able
to *slowly* build this whitelist. We can start with a NULL whitelist.
Which means that existing internal devices continue to work, and
external devices on PCI don't pose a risk. With ACS and IOMMU
restrictions in place, the security/risk baseline remains unchanged.
The existing devices are not regressed. As we vet and whitelist the
drivers, we start supporting more and more USB4 and Thunderbolt3
devices. Until then, those devices when plugged, can continue to work
in the "USB / legacy mode" (I forgot what it is called).

5) To give an example, assume we don't trust the PCI nvme driver and
don't want to whitelist it for external devices given there are so
many off the shelf devices with questionable firmware. But we
certainly need to enable it for internal NVME devices (that we may
have audited the firmware for, and control our supply chain) in order
to boot. With my proposal, until we whitelist it, the internal devices
continue to work, the external NVMEs switch to "USB storage device"
mode and thus go via a USB bridge so they cannot directly DMA into
host memory directly. Keep in mind that whitelisting a driver may be
handled by a separate security team, and may take long time depending
on the driver. The proposal allows us to release laptops with
Thunderbolt3/USB4 support and add peripheral support as we go.

6) Also small nit: consider the other scenario (I think this may not
be as important but still worth a thought). Assume the security team
finds a new vulnerability in a whitelisted driver, and want to take it
out of whitelist. Now, this really isn't possible if there was no
distinction between internal / external devices, and an internal
device uses that driver to boot.

================ END ============================

I feel I've described a problem, looked around to see what is
available, and tried to explain why it doesn't work for us given the
constraints. When it comes to security, the world is far from ideal.
Though in principle a "driver vulnerability" applies equally to any
device, in reality the threat vector is different for "internal" vs
"external" devices.

I feel a lot of resistance to the proposal, however, I'm not hearing
any realistic solutions that may help us to move forward. We want to
go with a solution that is acceptable upstream as that is our mission,
and also helps the community, however the behemoth task of "inspect
all drivers and fix them" before launching a product is really an
unfair ask I feel :-(. Can you help us by suggesting a proposal that
does not require us to trust a driver equally for internal / external
devices?

Looking for some guidance here.

Thanks & Best Regards,

Rajat

>
> thanks,
>
> greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Fri, Jun 05, 2020 at 06:08:28PM -0700, Rajat Jain wrote:
> Hello Greg,
> 
> Thank you for continuing to work with me through this.
> 
> On Fri, Jun 5, 2020 at 1:02 AM Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Jun 04, 2020 at 12:38:18PM -0700, Rajat Jain wrote:
> > > Hello,
> > >
> > > I spent some more thoughts into this...
> > >
> > > On Wed, Jun 3, 2020 at 5:16 AM Greg Kroah-Hartman
> > > <gregkh@linuxfoundation.org> wrote:
> > > >
> > > > On Wed, Jun 03, 2020 at 04:51:18AM -0700, Rajat Jain wrote:
> > > > > Hello,
> > > > >
> > > > > >
> > > > > > > Thanks for the pointer! I'm still looking at the details yet, but a
> > > > > > > quick look (usb_dev_authorized()) seems to suggest that this API is
> > > > > > > "device based". The multiple levels of "authorized" seem to take shape
> > > > > > > from either how it is wired or from userspace choice. Once authorized,
> > > > > > > USB device or interface is authorized to be used by *anyone* (can be
> > > > > > > attached to any drivers). Do I understand it right that it does not
> > > > > > > differentiate between drivers?
> > > > > >
> > > > > > Yes, and that is what you should do, don't fixate on drivers.  Users
> > > > > > know how to control and manage devices.  Us kernel developers are
> > > > > > responsible for writing solid drivers and getting them merged into the
> > > > > > kernel tree and maintaining them over time.  Drivers in the kernel
> > > > > > should always be trusted, ...
> > > > >
> > > > > 1) Yes, I agree that this would be ideal, and this should be our
> > > > > mission. I should clarify that I may have used the wrong term
> > > > > "Trusted/Certified drivers". I didn't really mean that the drivers may
> > > > > be malicious by intent. What I really meant is that a driver may have
> > > > > an attack surface, which is a vulnerability that may be exploited.
> > > >
> > > > Any code has such a thing, proving otherwise is a tough problem :)
> > > >
> > > > > Realistically speaking, finding vulnerabilities in drivers, creating
> > > > > attacks to exploit them, and fixing them is a never ending cat and
> > > > > mouse game. At Least "identifying the vulnerabilities" part is better
> > > > > performed by security folks rather than driver writers.
> > > >
> > > > Are you sure about that?  It's hard to prove a negative :)
> > > >
> > > > > Earlier in the
> > > > > thread I had mentioned certain studies/projects that identified and
> > > > > exploited such vulnerabilities in the drivers. I should have used the
> > > > > term "Vetted Drivers" maybe to convey the intent better - drivers that
> > > > > have been vetted by a security focussed team (admin). What I'm
> > > > > advocating here is an administrator's right to control the drivers
> > > > > that he wants to allow for external ports on his systems.
> > > >
> > > > That's an odd thing, but sure, if you want to write up such a policy for
> > > > your systems, great.  But that policy does not belong in the kernel, it
> > > > belongs in userspace.
> > > >
> > > > > 2) In addition to the problem of driver negligences / vulnerabilities
> > > > > to be exploited, we ran into another problem with the "whitelist
> > > > > devices only" approach. We did start with the "device based" approach
> > > > > only initially - but quickly realized that anything we use to
> > > > > whitelist an external device can only be based on the info provided by
> > > > > *that device* itself. So until we have devices that exchange
> > > > > certificates with kernel [1], it is easy for a malicious device to
> > > > > spoof a whitelisted device (by presenting the same VID:DID or any
> > > > > other data that is used by us to whitelist it).
> > > > >
> > > > > [1] https://www.intel.com/content/www/us/en/io/pci-express/pcie-device-security-enhancements-spec.html
> > > > >
> > > > > I hope that helps somewhat clarify how / why we reached here?
> > > >
> > > > Kind of, I still think all you need to do is worry about controling the
> > > > devices and if a driver should bind to it or not.  Again, much like USB
> > > > has been doing for a very long time now.  The idea of "spoofing" ids
> > > > also is not new, and has been around for a very long time as well, and
> > > > again, the controls that the USB core gives you allows you to make any
> > > > type of policy decision you want to, in userspace.
> > >
> > > Er, *currently* it doesn't allow the userspace to make the particular
> > > policy I want to, right? Specifically, today an administrator can not
> > > control which USB *drivers* he wants to allow on an *external* USB
> > > port.
> >
> > Not true, you can do that today with the explicit binding/unbinding of
> > devices to drivers in userspace.  Been there for many decades :)
> 
> Not sure if I understood. Can you please elaborate how that helps
> implement the policy I want?
> 
> >
> > But, think this through, since when do you have _multiple_ drivers that
> > have support to control the same type of device?  We almost never allow
> > that in the kernel today as that way lies madness (no heiarchy of
> > drivers to bind to what devices and so on.)
> >
> > We always strive to keep a one-to-one mapping of "this device is only
> > allowed to be controlled by this one driver" today, why would you want
> > to change that basic premise now?
> 
> No, I don't want to change that premise. Multiple drivers for a single
> device is not the goal at all.
> 
> >
> > > He can only control which USB devices he wants to authorize, but
> > > once authorized, they are free to bind to any of the USB drivers.
> >
> > Since when do different drivers control the same type of USB device?  :)
> 
> Sorry, I should have used better wording:
> "..., a malicious device can choose which (authorized/whitelisted)
> device to spoof, and *thus* choose a driver to attach to"
> Since the only data admin can use to decide to authorize the device,
> is provided by device itself, authorization is really just a farce.
> 
> >From Documentation/usb/authorization.rst:
> "Just checking if the class, type and protocol match something is the worse
> security verification you can make (or the best, for someone willing
> to break it). If you need something secure, use crypto and Certificate
> Authentication or stuff like that."
> 
> Truth be told, there is nothing else really available today. While
> certificate exchanges may be the future, the challenge is to deal with
> devices at hand.
> 
> > > So if I want to allow the administrator to implement a policy that
> > > allows him to control the drivers for external ports, we'll need to
> > > enhance the current code (whether we want to do it specific to a bus,
> > > or more generically in the driver core). Are we on the same page?
> > >
> > > To implement the policy that I want to in the driver core, what is
> > > missing today in driver core is a distinction between "internal" and
> > > "external" devices. Some buses have this knowledge locally today (PCI
> > > has "untrusted" flag which can be used, USB uses hcd->wireless and
> > > hub->port->connect_type) but it is not shared with the core.
> >
> > Note the wireless USB code should now be gone from the tree.  If you see
> > any remants of it floating around, let me know and I will remove them, I
> > think there might be a few bits left that I missed.
> >
> > > So just to make sure if I'm thinking in the right direction, this is
> > > what I'm thinking:
> > >
> > > 1) The device core needs a notion of internal vs external devices (a
> > > flag) - a knowledge that needs to be filled in by the bus as it
> > > discovers the device.
> >
> > Nope, don't go down this path.  We tried to do this for USB where the
> > BIOS tells us that a device is "internal" vs. "external" but in reality,
> > BIOSes get this wrong and it's not always all that useful.
> >
> > And why would you somehow "trust" a device that is in your system more
> > than one is not?  The same driver binds to it no matter what (as I state
> > above), so you should be able to trust it the same.
> 
> There are multiple reasons for trust level to be different for
> "internal" vs "external" devices. Speaking for the laptop world at
> least (and I suspect same is true for most of OEM products):
> 
> 1) The hardware, firmware, and in some cases even the supply chain is
> quite tightly controlled and audited for "internal devices". OTOH, we
> don't even know what an "external device" may look like.
> 
> 2) Most of the internal devices are soldered on board, and can't be
> accessed by a malicious person, atleast without the owner knowing.
> OTOH, external device attack is very easy (Imagine malicious user
> plugging in a USB stick on an unattended laptop / internet cafes /
> airports etc) - the owner wouldn't even know.
> 
> 3) Internal devices do not physically travel to another system. OTOH,
> external devices, even if not malicious, may get infected since they
> travel between multiple systems.
> 
> 4) New devices' support (new drivers) keeps on getting added in the
> kernel (which is a good thing!). But that means while the "internal
> device space" is fixed at product release, the "external device space"
> is unbounded and keeps on increasing. It is good from a functionality
> point of view, but not from a security point of view.
> 
> >
> > > 2) The driver core needs to allow an admin to provide a whitelist of
> > > drivers for external devices. (Via Command line or a driver flag.
> > > Default = everything is whitelisted).
> >
> > Again, nope, no difference, see above.
> >
> > > 3) While matching a driver to a device, the driver core needs to
> > > impose the whitelist if the device is external, and if the
> > > administrator has provided a whitelist.
> >
> > Ick, no, again, work on a per-device authorized setting.  That way it
> > works the same all across the system.  Don't get stuck in a "external
> > vs. internal" discussion as this will get messy really quickly (think
> > about "internal" devices with "external" links to them like PCI
> > "drawers" of devices that we currently support on large systems.  Or
> > things like thunderbolt hubs with "internal" devices like I have on my
> > desktop right now.
> 
> Good point. I'm not good at terminologies - by "external", I meant
> anything that is not in the physical boundary of the host system as
> shipped. It should be defined by the individual buses who learn it
> from the platform (BIOS / Device tree / Discovery process etc).
> 
> >
> > In summary, if a driver is "trusted enough" to control an internal
> > device, it should be "trusted enough" to control an external device.  If
> > not, then fix that driver so that you do "trust" it.
> 
> That is indeed our goal, and we do intend to inspect and send patch
> fixes upstream. However the problem is inspecting drivers for security
> and finding and fixing issues is a long drawn process - and has a lot
> of dependencies on different maintainers. It is not possible to front
> load all the effort and release the product only when *all* drivers
> are fixed (We want to begin with a *NULL* whitelist of drivers and
> then build it *slowly*). OTOH, it would be unfair to block a product
> because not all drivers could be inspected or fixed in time for
> security issues.
> 
> I feel that I have failed to explain the problems clearly. I'm copying
> a blurb earlier from this thread, which explains the context and the
> problem we're trying to solve.
> 
> ================ BEGIN ============================
> So here is our dilemma. In the laptop world:
> 
> 1) Today (Pre-Thunderbolt 3 / Pre-USB4), there is a mix of trusted /
> untrusted drivers that we (or any OEMs) are shipping on their laptops.
> Yes, there is some (calculated) risk that everyone is taking - because
> currently PCI bus does not extend outside the laptop *easily*. Yes I
> understand systems may have external PCI slots, but that is rather
> rare in the laptop world I think. The risks of the existing drivers
> are limited to the devices that were built into the system, and since
> the drivers, firmware updates, (and supply chain in some cases) are
> controlled by us, such internal devices are conceivably more secure
> than something random that the user may plug in. If the user opens the
> chassis to replace a piece of hardware with something else, all bets
> are off. Yes, we're still susceptible to the NIC driver attacks that
> you talk about it along with other potential vulnerabilities, but this
> is just convey our current baseline level of risk/security.
> 
> 2) Now, we want to enable technology of tomorrow :-) (Thunderbolt 3 /
> USB4) on laptops, which allows to very *easily* extend the internal
> PCI bus to the outside devices. Note it doesn't require to open a
> laptop, and anyone can plug a device onto a port. This throws the
> system open to a lot of DMA attacks now, which it did not have to deal
> with earlier. Essentially with the advent of technology to expand PCIe
> outside of system chassis, the attacks have become much more easier,
> we can no longer control or monitor device hardware or firmware, and
> thus the level of risk has clearly increased. So what we are trying to
> find here, is a good path to enable these new technologies, that keeps
> keeps our baseline level of risk/security unchanged, and to also not
> regress in functionality in supporting devices as much as possible.
> 
> 3) Now you are certainly right that one path could be a binary
> decision to ship or not ship a driver, or fix any issues with the
> driver, or change the driver to differentiate between external and
> internal ports. However, there are multiple factors that pose
> practical problems (why regress internal devices that we tightly
> control? Why regress systems that don't have such external ports? Need
> to front load all effort in vetting the drivers before hand before the
> first release. Work with each and individual driver etc).
> 
> 4) The other path that this proposal aims to take is that by applying
> a whitelist of drivers to external ports only, we're going to be able
> to *slowly* build this whitelist. We can start with a NULL whitelist.
> Which means that existing internal devices continue to work, and
> external devices on PCI don't pose a risk. With ACS and IOMMU
> restrictions in place, the security/risk baseline remains unchanged.
> The existing devices are not regressed. As we vet and whitelist the
> drivers, we start supporting more and more USB4 and Thunderbolt3
> devices. Until then, those devices when plugged, can continue to work
> in the "USB / legacy mode" (I forgot what it is called).
> 
> 5) To give an example, assume we don't trust the PCI nvme driver and
> don't want to whitelist it for external devices given there are so
> many off the shelf devices with questionable firmware. But we
> certainly need to enable it for internal NVME devices (that we may
> have audited the firmware for, and control our supply chain) in order
> to boot. With my proposal, until we whitelist it, the internal devices
> continue to work, the external NVMEs switch to "USB storage device"
> mode and thus go via a USB bridge so they cannot directly DMA into
> host memory directly. Keep in mind that whitelisting a driver may be
> handled by a separate security team, and may take long time depending
> on the driver. The proposal allows us to release laptops with
> Thunderbolt3/USB4 support and add peripheral support as we go.
> 
> 6) Also small nit: consider the other scenario (I think this may not
> be as important but still worth a thought). Assume the security team
> finds a new vulnerability in a whitelisted driver, and want to take it
> out of whitelist. Now, this really isn't possible if there was no
> distinction between internal / external devices, and an internal
> device uses that driver to boot.
> 
> ================ END ============================
> 
> I feel I've described a problem, looked around to see what is
> available, and tried to explain why it doesn't work for us given the
> constraints. When it comes to security, the world is far from ideal.
> Though in principle a "driver vulnerability" applies equally to any
> device, in reality the threat vector is different for "internal" vs
> "external" devices.

Your "problem" I think can be summed up a bit more concise:
	- you don't trust kernel drivers to be "secure" for untrusted
	  devices
	- you only want to bind kernel drivers to "internal" devices
	  automatically as you "trust" drivers in that situation.
	- you want to only bind specific kernel drivers that you somehow
	  feel are "secure" to untrusted devices "outside" of a system
	  when those devices are added to the system.

Is that correct?

If so, fine, you can do that today with the bind/unbind ability of
drivers, right?  After boot with your "trusted" drivers bound to
"internal" devices, turn off autobind of drivers to devices and then
manually bind them when you see new devices show up, as those "must" be
from external devices (see the bind/unbind files that all drivers export
for how to do this, and old lwn.net articles, this feature has been
around for a very long time.)

I know for USB you can do this, odds are PCI you can turn off
autobinding as well, as I think this is a per-bus flag somewhere.  If
that's not exported to userspace, should be trivial to do so, should be
somewere in the driver model already...

Ah, yes, look at the "drivers_autoprobe" and "drivers_probe" files in
sysfs for all busses.  Do those not work for you?

My other points are the fact that you don't want to put policy in the
kernel, and I think that you can do everything you want in userspace
today, except maybe the fact that trying to determine what is "inside"
and "outside" is not always easy given that most hardware does not
export this information properly, if at all.  Go work with the firmware
people on that issue please, that would be most helpful for everyone
involved to get that finally straightened out.

> I feel a lot of resistance to the proposal, however, I'm not hearing
> any realistic solutions that may help us to move forward. We want to
> go with a solution that is acceptable upstream as that is our mission,
> and also helps the community, however the behemoth task of "inspect
> all drivers and fix them" before launching a product is really an
> unfair ask I feel :-(. Can you help us by suggesting a proposal that
> does not require us to trust a driver equally for internal / external
> devices?

I have no idea why you feel you have to "inspect all drivers" other than
the fact that for some reason _you_ do not feel they are secure today.

What type of "assurance" are you, or anyone else going to be able to
provide for any kernel driver that would meet such a "I feel good now"
level?  Have you done that work for any specific driver already so that
you can show us what you mean by this effort?  Perhaps it's as simple as
"oh look, this tool over here runs 'clean' on the source code, all is
good!", or not, I really have no idea.

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Jesse Barnes]

> > I feel a lot of resistance to the proposal, however, I'm not hearing
> > any realistic solutions that may help us to move forward. We want to
> > go with a solution that is acceptable upstream as that is our mission,
> > and also helps the community, however the behemoth task of "inspect
> > all drivers and fix them" before launching a product is really an
> > unfair ask I feel :-(. Can you help us by suggesting a proposal that
> > does not require us to trust a driver equally for internal / external
> > devices?
>
> I have no idea why you feel you have to "inspect all drivers" other than
> the fact that for some reason _you_ do not feel they are secure today.
>
> What type of "assurance" are you, or anyone else going to be able to
> provide for any kernel driver that would meet such a "I feel good now"
> level?  Have you done that work for any specific driver already so that
> you can show us what you mean by this effort?  Perhaps it's as simple as
> "oh look, this tool over here runs 'clean' on the source code, all is
> good!", or not, I really have no idea.

I think there's a disconnect somewhere in this discussion... maybe
we're just approaching this with different assumptions?

I think you recognize the potential for driver vulnerabilities when
binding to new or potentially hostile devices that may be spoofing
DID/VID/class/etc than then go on to abuse driver trust or the driver
using unvalidated inputs from the device to crash or run arbitrary
code on the target system.

Yes such drivers should be fixed, no doubt.  But without lots of
fuzzing (we're working on this) and testing we'd like to avoid
exposing that attack surface at all.

I think your suggestion to disable driver binding once the initial
bus/slot devices have been bound will probably work for this
situation.  I just wanted to be clear that without some auditing,
fuzzing, and additional testing, we simply have to assume that drivers
are *not* secure and avoid using them on untrusted devices until we're
fairly confident they can handle them (whether just misbehaving or
malicious), in combination with other approaches like IOMMUs of
course.  And this isn't because we don't trust driver authors or
kernel developers to dtrt, it's just that for many devices (maybe USB
is an exception) I think driver authors haven't had to consider this
case much, and so I think it's prudent to expect bugs in this area
that we need to find & fix.

Thanks,
Jesse

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Mon, Jun 08, 2020 at 10:03:38AM -0700, Jesse Barnes wrote:
> > > I feel a lot of resistance to the proposal, however, I'm not hearing
> > > any realistic solutions that may help us to move forward. We want to
> > > go with a solution that is acceptable upstream as that is our mission,
> > > and also helps the community, however the behemoth task of "inspect
> > > all drivers and fix them" before launching a product is really an
> > > unfair ask I feel :-(. Can you help us by suggesting a proposal that
> > > does not require us to trust a driver equally for internal / external
> > > devices?
> >
> > I have no idea why you feel you have to "inspect all drivers" other than
> > the fact that for some reason _you_ do not feel they are secure today.
> >
> > What type of "assurance" are you, or anyone else going to be able to
> > provide for any kernel driver that would meet such a "I feel good now"
> > level?  Have you done that work for any specific driver already so that
> > you can show us what you mean by this effort?  Perhaps it's as simple as
> > "oh look, this tool over here runs 'clean' on the source code, all is
> > good!", or not, I really have no idea.
> 
> I think there's a disconnect somewhere in this discussion... maybe
> we're just approaching this with different assumptions?
> 
> I think you recognize the potential for driver vulnerabilities when
> binding to new or potentially hostile devices that may be spoofing
> DID/VID/class/etc than then go on to abuse driver trust or the driver
> using unvalidated inputs from the device to crash or run arbitrary
> code on the target system.
> 
> Yes such drivers should be fixed, no doubt.  But without lots of
> fuzzing (we're working on this) and testing we'd like to avoid
> exposing that attack surface at all.
> 
> I think your suggestion to disable driver binding once the initial
> bus/slot devices have been bound will probably work for this
> situation.  I just wanted to be clear that without some auditing,
> fuzzing, and additional testing, we simply have to assume that drivers
> are *not* secure and avoid using them on untrusted devices until we're
> fairly confident they can handle them (whether just misbehaving or
> malicious), in combination with other approaches like IOMMUs of
> course.  And this isn't because we don't trust driver authors or
> kernel developers to dtrt, it's just that for many devices (maybe USB
> is an exception) I think driver authors haven't had to consider this
> case much, and so I think it's prudent to expect bugs in this area
> that we need to find & fix.

For USB, yes, we have now had to deal with "untrusted devices" lieing
about their ids and sending us horrible data.  That's all due to the
fuzzing tools that have been written over the past few years, and now we
have some of those in the kernel tree itself to help with that testing.

For PCI, heh, good luck, those assumptions about "devices sending valid
data" are everywhere, if our experience with USB is any indication.

But, to take USB as an example, this is exactly what the USB
"authorized" flag is there for, it's a "trust" setting that userspace
has control over.  This came from the wireless USB spec, where it was
determined that you could not trust devices.  So just use that same
model here, move it to the driver core for all busses to use and you
should be fine.

If that doesn't meet your needs, please let me know the specifics of
why, with patches :)

Now, as to you all getting some sort of "Hardware flag" to determine
"inside" vs. "outside" devices, hah, good luck!  It took us a long time
to get that for USB, and even then, BIOSes lie and get it wrong all the
time.  So you will have to also deal with that in some way, for your
userspace policy.

good luck!

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Jesse Barnes]

> > I think your suggestion to disable driver binding once the initial
> > bus/slot devices have been bound will probably work for this
> > situation.  I just wanted to be clear that without some auditing,
> > fuzzing, and additional testing, we simply have to assume that drivers
> > are *not* secure and avoid using them on untrusted devices until we're
> > fairly confident they can handle them (whether just misbehaving or
> > malicious), in combination with other approaches like IOMMUs of
> > course.  And this isn't because we don't trust driver authors or
> > kernel developers to dtrt, it's just that for many devices (maybe USB
> > is an exception) I think driver authors haven't had to consider this
> > case much, and so I think it's prudent to expect bugs in this area
> > that we need to find & fix.
>
> For USB, yes, we have now had to deal with "untrusted devices" lieing
> about their ids and sending us horrible data.  That's all due to the
> fuzzing tools that have been written over the past few years, and now we
> have some of those in the kernel tree itself to help with that testing.
>
> For PCI, heh, good luck, those assumptions about "devices sending valid
> data" are everywhere, if our experience with USB is any indication.
>
> But, to take USB as an example, this is exactly what the USB
> "authorized" flag is there for, it's a "trust" setting that userspace
> has control over.  This came from the wireless USB spec, where it was
> determined that you could not trust devices.  So just use that same
> model here, move it to the driver core for all busses to use and you
> should be fine.
>
> If that doesn't meet your needs, please let me know the specifics of
> why, with patches :)

Yeah will do for sure.  I don't want to carry a big infra for this on our own!

> Now, as to you all getting some sort of "Hardware flag" to determine
> "inside" vs. "outside" devices, hah, good luck!  It took us a long time
> to get that for USB, and even then, BIOSes lie and get it wrong all the
> time.  So you will have to also deal with that in some way, for your
> userspace policy.

I think that's inherently platform specific to some extent.  We can do
it with our coreboot based firmware, but there's no guarantee other
vendors will adopt the same approach.  But I think at least for the
ChromeOS ecosystem we can come up with something that'll work, and
allow us to dtrt in userspace wrt driver binding.

Thanks,
Jesse

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

Hi Jesse and Greg,

On Mon, Jun 8, 2020 at 11:30 AM Jesse Barnes <jsbarnes@google.com> wrote:
>
> > > I think your suggestion to disable driver binding once the initial
> > > bus/slot devices have been bound will probably work for this
> > > situation.  I just wanted to be clear that without some auditing,
> > > fuzzing, and additional testing, we simply have to assume that drivers
> > > are *not* secure and avoid using them on untrusted devices until we're
> > > fairly confident they can handle them (whether just misbehaving or
> > > malicious), in combination with other approaches like IOMMUs of
> > > course.  And this isn't because we don't trust driver authors or
> > > kernel developers to dtrt, it's just that for many devices (maybe USB
> > > is an exception) I think driver authors haven't had to consider this
> > > case much, and so I think it's prudent to expect bugs in this area
> > > that we need to find & fix.
> >
> > For USB, yes, we have now had to deal with "untrusted devices" lieing
> > about their ids and sending us horrible data.  That's all due to the
> > fuzzing tools that have been written over the past few years, and now we
> > have some of those in the kernel tree itself to help with that testing.

This is great to hear! I tried to look up but didn't find anything
else in-kernel, except the kcov support to export coverage info for
userspace fuzzers. Can you please give us some pointers for in-kernel
fuzzing tools?

> >
> > For PCI, heh, good luck, those assumptions about "devices sending valid
> > data" are everywhere, if our experience with USB is any indication.
> >
> > But, to take USB as an example, this is exactly what the USB
> > "authorized" flag is there for, it's a "trust" setting that userspace
> > has control over.  This came from the wireless USB spec, where it was
> > determined that you could not trust devices.  So just use that same
> > model here, move it to the driver core for all busses to use and you
> > should be fine.
> >
> > If that doesn't meet your needs, please let me know the specifics of
> > why, with patches :)
>
> Yeah will do for sure.  I don't want to carry a big infra for this on our own!
>
> > Now, as to you all getting some sort of "Hardware flag" to determine
> > "inside" vs. "outside" devices, hah, good luck!  It took us a long time
> > to get that for USB, and even then, BIOSes lie and get it wrong all the
> > time.  So you will have to also deal with that in some way, for your
> > userspace policy.
>
> I think that's inherently platform specific to some extent.  We can do
> it with our coreboot based firmware, but there's no guarantee other
> vendors will adopt the same approach.  But I think at least for the
> ChromeOS ecosystem we can come up with something that'll work, and
> allow us to dtrt in userspace wrt driver binding.

Agree, we can work with our firmware teams to get that right, and then
expose it from kernel to userspace to help it implement the policy we
want.

Thanks & Best Regards,

Rajat

>
> Thanks,
> Jesse

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Mon, Jun 08, 2020 at 11:41:19AM -0700, Rajat Jain wrote:
> Hi Jesse and Greg,
> 
> On Mon, Jun 8, 2020 at 11:30 AM Jesse Barnes <jsbarnes@google.com> wrote:
> >
> > > > I think your suggestion to disable driver binding once the initial
> > > > bus/slot devices have been bound will probably work for this
> > > > situation.  I just wanted to be clear that without some auditing,
> > > > fuzzing, and additional testing, we simply have to assume that drivers
> > > > are *not* secure and avoid using them on untrusted devices until we're
> > > > fairly confident they can handle them (whether just misbehaving or
> > > > malicious), in combination with other approaches like IOMMUs of
> > > > course.  And this isn't because we don't trust driver authors or
> > > > kernel developers to dtrt, it's just that for many devices (maybe USB
> > > > is an exception) I think driver authors haven't had to consider this
> > > > case much, and so I think it's prudent to expect bugs in this area
> > > > that we need to find & fix.
> > >
> > > For USB, yes, we have now had to deal with "untrusted devices" lieing
> > > about their ids and sending us horrible data.  That's all due to the
> > > fuzzing tools that have been written over the past few years, and now we
> > > have some of those in the kernel tree itself to help with that testing.
> 
> This is great to hear! I tried to look up but didn't find anything
> else in-kernel, except the kcov support to export coverage info for
> userspace fuzzers. Can you please give us some pointers for in-kernel
> fuzzing tools?

For USB, it's a combination of using syzbot with the "raw gadget" driver
and the loopback gadget/host controller.  See many posts from Andrey
Konovalov <andreyknvl@google.com> on the linux-usb@vger.kernel.org list
for details as to how he does this.

> > > For PCI, heh, good luck, those assumptions about "devices sending valid
> > > data" are everywhere, if our experience with USB is any indication.
> > >
> > > But, to take USB as an example, this is exactly what the USB
> > > "authorized" flag is there for, it's a "trust" setting that userspace
> > > has control over.  This came from the wireless USB spec, where it was
> > > determined that you could not trust devices.  So just use that same
> > > model here, move it to the driver core for all busses to use and you
> > > should be fine.
> > >
> > > If that doesn't meet your needs, please let me know the specifics of
> > > why, with patches :)
> >
> > Yeah will do for sure.  I don't want to carry a big infra for this on our own!
> >
> > > Now, as to you all getting some sort of "Hardware flag" to determine
> > > "inside" vs. "outside" devices, hah, good luck!  It took us a long time
> > > to get that for USB, and even then, BIOSes lie and get it wrong all the
> > > time.  So you will have to also deal with that in some way, for your
> > > userspace policy.
> >
> > I think that's inherently platform specific to some extent.  We can do
> > it with our coreboot based firmware, but there's no guarantee other
> > vendors will adopt the same approach.  But I think at least for the
> > ChromeOS ecosystem we can come up with something that'll work, and
> > allow us to dtrt in userspace wrt driver binding.
> 
> Agree, we can work with our firmware teams to get that right, and then
> expose it from kernel to userspace to help it implement the policy we
> want.

This is already in the spec for USB, I suggest working to get this added
to the other bus type specs that you care about as well (UEFI, PCI,
etc.)

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

Hi!

> > > I think that's inherently platform specific to some extent.  We can do
> > > it with our coreboot based firmware, but there's no guarantee other
> > > vendors will adopt the same approach.  But I think at least for the
> > > ChromeOS ecosystem we can come up with something that'll work, and
> > > allow us to dtrt in userspace wrt driver binding.
> > 
> > Agree, we can work with our firmware teams to get that right, and then
> > expose it from kernel to userspace to help it implement the policy we
> > want.
> 
> This is already in the spec for USB, I suggest working to get this added
> to the other bus type specs that you care about as well (UEFI, PCI,
> etc.)

Note that "you can only use authorized SSD and authorized WIFI card in
this notebook" was done before, but is considered antisocial.

Best regards,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Mon, Jun 08, 2020 at 11:29:58AM -0700, Jesse Barnes wrote:
> > Now, as to you all getting some sort of "Hardware flag" to determine
> > "inside" vs. "outside" devices, hah, good luck!  It took us a long time
> > to get that for USB, and even then, BIOSes lie and get it wrong all the
> > time.  So you will have to also deal with that in some way, for your
> > userspace policy.
> 
> I think that's inherently platform specific to some extent.  We can do
> it with our coreboot based firmware, but there's no guarantee other
> vendors will adopt the same approach.  But I think at least for the
> ChromeOS ecosystem we can come up with something that'll work, and
> allow us to dtrt in userspace wrt driver binding.

Why not work with the UEFI group to add this to their spec so that it
will work for all future firmware releases, not just your
vendor-specific one?  :)

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1704 bytes --]

Hi!

> Yes such drivers should be fixed, no doubt.  But without lots of
> fuzzing (we're working on this) and testing we'd like to avoid
> exposing that attack surface at all.
> 
> I think your suggestion to disable driver binding once the initial
> bus/slot devices have been bound will probably work for this
> situation.  I just wanted to be clear that without some auditing,
> fuzzing, and additional testing, we simply have to assume that drivers
> are *not* secure and avoid using them on untrusted devices until we're
> fairly confident they can handle them (whether just misbehaving or
> malicious), in combination with other approaches like IOMMUs of
> course.  And this isn't because we don't trust driver authors or
> kernel developers to dtrt, it's just that for many devices (maybe USB
> is an exception) I think driver authors haven't had to consider this
> case much, and so I think it's prudent to expect bugs in this area
> that we need to find & fix.

We normally trust the hardware NOT to be malicious. (Because if hacker
has physical access to hardware and lot of resources, you lost).

This is still true today, but maybe trusting USB devices is bad idea,
so drivers are being cleaned up. PCI drivers will be WORSE in this
regard. And you can't really protect against malicious CPU, and it is
very very hard to protect against malicous RAM (probably not practical
without explicit CPU support).

Linux was designed with "don't let hackers near your hardware" threat
model in mind.

Best regards,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Tue, Jun 30, 2020 at 11:45:59PM +0200, Pavel Machek wrote:
> Hi!
> 
> > Yes such drivers should be fixed, no doubt.  But without lots of
> > fuzzing (we're working on this) and testing we'd like to avoid
> > exposing that attack surface at all.
> > 
> > I think your suggestion to disable driver binding once the initial
> > bus/slot devices have been bound will probably work for this
> > situation.  I just wanted to be clear that without some auditing,
> > fuzzing, and additional testing, we simply have to assume that drivers
> > are *not* secure and avoid using them on untrusted devices until we're
> > fairly confident they can handle them (whether just misbehaving or
> > malicious), in combination with other approaches like IOMMUs of
> > course.  And this isn't because we don't trust driver authors or
> > kernel developers to dtrt, it's just that for many devices (maybe USB
> > is an exception) I think driver authors haven't had to consider this
> > case much, and so I think it's prudent to expect bugs in this area
> > that we need to find & fix.
> 
> We normally trust the hardware NOT to be malicious. (Because if hacker
> has physical access to hardware and lot of resources, you lost).

That is what we originally thought, however the world has changed and we
need to be better about this, now that it is trivial to create a "bad"
device.

> This is still true today, but maybe trusting USB devices is bad idea,
> so drivers are being cleaned up. PCI drivers will be WORSE in this
> regard. And you can't really protect against malicious CPU, and it is
> very very hard to protect against malicous RAM (probably not practical
> without explicit CPU support).
> 
> Linux was designed with "don't let hackers near your hardware" threat
> model in mind.

Yes, it originally was designed that way, but again, the world has
changed so we have to change with it.  That is why USB has for a long
time now, allowed you to not bind drivers to devices that you do not
"trust", and that trust can be determined by userspace.  That all came
about thanks to the work done by the wireless USB spec people and kernel
authors, which showed that maybe you just don't want to trust any device
that comes within range of your system :)

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2092 bytes --]

Hi!

> > We normally trust the hardware NOT to be malicious. (Because if hacker
> > has physical access to hardware and lot of resources, you lost).
> 
> That is what we originally thought, however the world has changed and we
> need to be better about this, now that it is trivial to create a "bad"
> device.

I'm not disagreeing.

> > This is still true today, but maybe trusting USB devices is bad idea,
> > so drivers are being cleaned up. PCI drivers will be WORSE in this
> > regard. And you can't really protect against malicious CPU, and it is
> > very very hard to protect against malicous RAM (probably not practical
> > without explicit CPU support).
> > 
> > Linux was designed with "don't let hackers near your hardware" threat
> > model in mind.
> 
> Yes, it originally was designed that way, but again, the world has
> changed so we have to change with it.  That is why USB has for a long
> time now, allowed you to not bind drivers to devices that you do not
> "trust", and that trust can be determined by userspace.  That all came
> about thanks to the work done by the wireless USB spec people and kernel
> authors, which showed that maybe you just don't want to trust any device
> that comes within range of your system :)

Again, not disagreeing; but note the scale here.

It is mandatory to defend against malicious wireless USB devices.

We probably should work on robustness against malicious USB devices.

Malicious PCI-express devices are lot less of concern.

Defending against malicious CPU/RAM does not make much sense.

Notice that it is quite easy to generate -100V on the USB and kill
your motherboard. Also notice that malicious parts of the hardware
don't need to be electrically connected to the rest of system, and
that they don't even have to contain any electronics. You just have to
be careful. https://en.wikipedia.org/wiki/The_Thing_(listening_device)

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Wed, Jul 01, 2020 at 10:47:50AM +0200, Pavel Machek wrote:
> Hi!
> 
> > > We normally trust the hardware NOT to be malicious. (Because if hacker
> > > has physical access to hardware and lot of resources, you lost).
> > 
> > That is what we originally thought, however the world has changed and we
> > need to be better about this, now that it is trivial to create a "bad"
> > device.
> 
> I'm not disagreeing.
> 
> > > This is still true today, but maybe trusting USB devices is bad idea,
> > > so drivers are being cleaned up. PCI drivers will be WORSE in this
> > > regard. And you can't really protect against malicious CPU, and it is
> > > very very hard to protect against malicous RAM (probably not practical
> > > without explicit CPU support).
> > > 
> > > Linux was designed with "don't let hackers near your hardware" threat
> > > model in mind.
> > 
> > Yes, it originally was designed that way, but again, the world has
> > changed so we have to change with it.  That is why USB has for a long
> > time now, allowed you to not bind drivers to devices that you do not
> > "trust", and that trust can be determined by userspace.  That all came
> > about thanks to the work done by the wireless USB spec people and kernel
> > authors, which showed that maybe you just don't want to trust any device
> > that comes within range of your system :)
> 
> Again, not disagreeing; but note the scale here.
> 
> It is mandatory to defend against malicious wireless USB devices.

Turns out there are no more wireless USB devices in the world, and the
code for that is gone from Linux :)

> We probably should work on robustness against malicious USB devices.

We are, and do have, that support today.

> Malicious PCI-express devices are lot less of concern.

Not really, they are a lot of concern to some people.  Valid attacks are
out there today, see the thunderbolt attacks that numerous people have
done and published recently and for many years.

> Defending against malicious CPU/RAM does not make much sense.

That's what the spectre and rowhammer fixes have been for :)

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]

Hi!

> > > Yes, it originally was designed that way, but again, the world has
> > > changed so we have to change with it.  That is why USB has for a long
> > > time now, allowed you to not bind drivers to devices that you do not
> > > "trust", and that trust can be determined by userspace.  That all came
> > > about thanks to the work done by the wireless USB spec people and kernel
> > > authors, which showed that maybe you just don't want to trust any device
> > > that comes within range of your system :)
> > 
> > Again, not disagreeing; but note the scale here.
> > 
> > It is mandatory to defend against malicious wireless USB devices.
> 
> Turns out there are no more wireless USB devices in the world, and the
> code for that is gone from Linux :)
> 
> > We probably should work on robustness against malicious USB devices.
> 
> We are, and do have, that support today.
> 
> > Malicious PCI-express devices are lot less of concern.
> 
> Not really, they are a lot of concern to some people.  Valid attacks are
> out there today, see the thunderbolt attacks that numerous people have
> done and published recently and for many years.

In this case PCI-express meant internal cards in PCs. Yes, thunderbolt
would be higher concern than internal card.

> > Defending against malicious CPU/RAM does not make much sense.
> 
> That's what the spectre and rowhammer fixes have been for :)

Yeah, and that's why we have whitelist of working CPUs and only work
on those, riiight? :-). [There's difference between "malicious" and
"buggy".]

								Pavel
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Bjorn Helgaas]

On Sun, Jun 07, 2020 at 01:36:32PM +0200, Greg Kroah-Hartman wrote:

> Your "problem" I think can be summed up a bit more concise:
> 	- you don't trust kernel drivers to be "secure" for untrusted
> 	  devices
> 	- you only want to bind kernel drivers to "internal" devices
> 	  automatically as you "trust" drivers in that situation.
> 	- you want to only bind specific kernel drivers that you somehow
> 	  feel are "secure" to untrusted devices "outside" of a system
> 	  when those devices are added to the system.
> 
> Is that correct?
> 
> If so, fine, you can do that today with the bind/unbind ability of
> drivers, right?  After boot with your "trusted" drivers bound to
> "internal" devices, turn off autobind of drivers to devices and then
> manually bind them when you see new devices show up, as those "must" be
> from external devices (see the bind/unbind files that all drivers export
> for how to do this, and old lwn.net articles, this feature has been
> around for a very long time.)
> 
> I know for USB you can do this, odds are PCI you can turn off
> autobinding as well, as I think this is a per-bus flag somewhere.  If
> that's not exported to userspace, should be trivial to do so, should be
> somewere in the driver model already...
> 
> Ah, yes, look at the "drivers_autoprobe" and "drivers_probe" files in
> sysfs for all busses.  Do those not work for you?
> 
> My other points are the fact that you don't want to put policy in the
> kernel, and I think that you can do everything you want in userspace
> today, except maybe the fact that trying to determine what is "inside"
> and "outside" is not always easy given that most hardware does not
> export this information properly, if at all.  Go work with the firmware
> people on that issue please, that would be most helpful for everyone
> involved to get that finally straightened out.

To sketch this out, my understanding of how this would work is:

  - Expose the PCI pdev->untrusted bit in sysfs.  We don't expose this
    today, but doing so would be trivial.  I think I would prefer a
    sysfs name like "external" so it's more descriptive and less of a
    judgment.

    This comes from either the DT "external-facing" property or the
    ACPI "ExternalFacingPort" property.  

  - All devices present at boot are enumerated.  Any statically built
    drivers will bind to them before any userspace code runs.

    If you want to keep statically built drivers from binding, you'd
    need to invent some mechanism so pci_driver_init() could clear
    drivers_autoprobe after registering pci_bus_type.

  - Early userspace code prevents modular drivers from automatically
    binding to PCI devices:

      echo 0 > /sys/bus/pci/drivers_autoprobe

    This prevents modular drivers from binding to all devices, whether
    present at boot or hot-added.

  - Userspace code uses the sysfs "bind" file to control which drivers
    are loaded and can bind to each device, e.g.,

      echo 0000:02:00.0 > /sys/bus/pci/drivers/nvme/bind

Is that what you're thinking?  Is that enough for the control you
need, Rajat?

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

Hi Bjorn,

Thanks for sending out the summary, I was about to send it out but got lazy.

On Tue, Jun 9, 2020 at 2:04 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Sun, Jun 07, 2020 at 01:36:32PM +0200, Greg Kroah-Hartman wrote:
>
> > Your "problem" I think can be summed up a bit more concise:
> >       - you don't trust kernel drivers to be "secure" for untrusted
> >         devices
> >       - you only want to bind kernel drivers to "internal" devices
> >         automatically as you "trust" drivers in that situation.
> >       - you want to only bind specific kernel drivers that you somehow
> >         feel are "secure" to untrusted devices "outside" of a system
> >         when those devices are added to the system.
> >
> > Is that correct?
> >
> > If so, fine, you can do that today with the bind/unbind ability of
> > drivers, right?  After boot with your "trusted" drivers bound to
> > "internal" devices, turn off autobind of drivers to devices and then
> > manually bind them when you see new devices show up, as those "must" be
> > from external devices (see the bind/unbind files that all drivers export
> > for how to do this, and old lwn.net articles, this feature has been
> > around for a very long time.)
> >
> > I know for USB you can do this, odds are PCI you can turn off
> > autobinding as well, as I think this is a per-bus flag somewhere.  If
> > that's not exported to userspace, should be trivial to do so, should be
> > somewere in the driver model already...
> >
> > Ah, yes, look at the "drivers_autoprobe" and "drivers_probe" files in
> > sysfs for all busses.  Do those not work for you?
> >
> > My other points are the fact that you don't want to put policy in the
> > kernel, and I think that you can do everything you want in userspace
> > today, except maybe the fact that trying to determine what is "inside"
> > and "outside" is not always easy given that most hardware does not
> > export this information properly, if at all.  Go work with the firmware
> > people on that issue please, that would be most helpful for everyone
> > involved to get that finally straightened out.
>
> To sketch this out, my understanding of how this would work is:
>
>   - Expose the PCI pdev->untrusted bit in sysfs.  We don't expose this
>     today, but doing so would be trivial.  I think I would prefer a
>     sysfs name like "external" so it's more descriptive and less of a
>     judgment.

Yes. I think we should probably semantically differentiate between
"external" and "external facing" devices. Root ports and downstream
ports can be "external facing" but are actually internal devices.
Anything below an "external facing" device is "external". So the sysfs
attribute "external" should be set only for devices that are truly
external.

So I think we can possibly synthesize "external" sysfs attribute from
"untrusted" bit like this (Sorry code looks more complex than it is):

parent = pdev->bus->self;

if (pdev->untrusted) {
        if (parent && parent->untrusted)
                pdev->external = true;   /* Device downstream of
un-trusted device = external */
        else {
                pdev->external = false   /* Root complex or an
internal Downstream port */
} else {
        pdev->external = false; /* Trusted device = Internal device */
}

For platforms that don't expose and untrusted" device, everything is
assumed to be an "internal device".

Just a suggestion: Do you think an enum attribute may be better
instead, whose values could be "internal" / "external" /
"external-facing" in case need arises later to distinguish between
them?

>
>     This comes from either the DT "external-facing" property or the
>     ACPI "ExternalFacingPort" property.
>
>   - All devices present at boot are enumerated.  Any statically built
>     drivers will bind to them before any userspace code runs.

Yes. For our (thunderbolt / USB4) use case, we'd still be protected
because we can control the PCIe tunnels to thunderbolt / USB4 devices
and will not enable them until we are ready. So while this approach
may not work for a system that always enables PCIe connections to
external devices at boot, it works for our use case as we are looking
for only thunderbolt / USB4 devices. (Not a problem or concern, just
wanted to be clear).

>
>     If you want to keep statically built drivers from binding, you'd
>     need to invent some mechanism so pci_driver_init() could clear
>     drivers_autoprobe after registering pci_bus_type.

At present I am not planning this.

>
>   - Early userspace code prevents modular drivers from automatically
>     binding to PCI devices:
>
>       echo 0 > /sys/bus/pci/drivers_autoprobe

Yes.
I believe this setting will apply it equally to both modular and
statically linked drivers?

>
>     This prevents modular drivers from binding to all devices, whether
>     present at boot or hot-added.

Yes, at this time, the userspace will need to monitor udev events for
any new PCI devices hot added, and lookup the VID/DID in pci driver
database (Isn't it somewhere like modules.pcimap modules.dap or
something in /lib/modules?) to get the driver name. and then after
consulting a maintained whitelist, do the following:

>
>   - Userspace code uses the sysfs "bind" file to control which drivers
>     are loaded and can bind to each device, e.g.,
>
>       echo 0000:02:00.0 > /sys/bus/pci/drivers/nvme/bind
>
> Is that what you're thinking?  Is that enough for the control you
> need, Rajat?

Yes, It sounds like it. That is my current plan.

The one thing that still needs more thought is how about the
"pcieport" driver that enumerates the PCI bridges. I'm unsure if it
needs to be whitelisted for further enumeration downstream. What do
you think?

Thanks & Best Regards,

Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Bjorn Helgaas]

On Tue, Jun 09, 2020 at 04:23:54PM -0700, Rajat Jain wrote:
> Hi Bjorn,
> 
> Thanks for sending out the summary, I was about to send it out but got lazy.
> 
> On Tue, Jun 9, 2020 at 2:04 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > On Sun, Jun 07, 2020 at 01:36:32PM +0200, Greg Kroah-Hartman wrote:
> >
> > > Your "problem" I think can be summed up a bit more concise:
> > >       - you don't trust kernel drivers to be "secure" for untrusted
> > >         devices
> > >       - you only want to bind kernel drivers to "internal" devices
> > >         automatically as you "trust" drivers in that situation.
> > >       - you want to only bind specific kernel drivers that you somehow
> > >         feel are "secure" to untrusted devices "outside" of a system
> > >         when those devices are added to the system.
> > >
> > > Is that correct?
> > >
> > > If so, fine, you can do that today with the bind/unbind ability of
> > > drivers, right?  After boot with your "trusted" drivers bound to
> > > "internal" devices, turn off autobind of drivers to devices and then
> > > manually bind them when you see new devices show up, as those "must" be
> > > from external devices (see the bind/unbind files that all drivers export
> > > for how to do this, and old lwn.net articles, this feature has been
> > > around for a very long time.)
> > >
> > > I know for USB you can do this, odds are PCI you can turn off
> > > autobinding as well, as I think this is a per-bus flag somewhere.  If
> > > that's not exported to userspace, should be trivial to do so, should be
> > > somewere in the driver model already...
> > >
> > > Ah, yes, look at the "drivers_autoprobe" and "drivers_probe" files in
> > > sysfs for all busses.  Do those not work for you?
> > >
> > > My other points are the fact that you don't want to put policy in the
> > > kernel, and I think that you can do everything you want in userspace
> > > today, except maybe the fact that trying to determine what is "inside"
> > > and "outside" is not always easy given that most hardware does not
> > > export this information properly, if at all.  Go work with the firmware
> > > people on that issue please, that would be most helpful for everyone
> > > involved to get that finally straightened out.
> >
> > To sketch this out, my understanding of how this would work is:
> >
> >   - Expose the PCI pdev->untrusted bit in sysfs.  We don't expose this
> >     today, but doing so would be trivial.  I think I would prefer a
> >     sysfs name like "external" so it's more descriptive and less of a
> >     judgment.
> 
> Yes. I think we should probably semantically differentiate between
> "external" and "external facing" devices. Root ports and downstream
> ports can be "external facing" but are actually internal devices.
> Anything below an "external facing" device is "external". So the sysfs
> attribute "external" should be set only for devices that are truly
> external.

Good point; we (maybe you? :)) should fix that edge case.

> Just a suggestion: Do you think an enum attribute may be better
> instead, whose values could be "internal" / "external" /
> "external-facing" in case need arises later to distinguish between
> them?

I don't see the need for an enum yet.  Maybe we should add that
if/when we do need it?

> >   - Early userspace code prevents modular drivers from automatically
> >     binding to PCI devices:
> >
> >       echo 0 > /sys/bus/pci/drivers_autoprobe
> 
> Yes.
> I believe this setting will apply it equally to both modular and
> statically linked drivers?

Yes.  The test is in bus_probe_device(), and it does the same for both
modular and statically linked drivers.

But for statically linked drivers, it only prevents them from binding
to *hot-added* devices.  They will claim devices present at boot even
before userspace code can run.

> The one thing that still needs more thought is how about the
> "pcieport" driver that enumerates the PCI bridges. I'm unsure if it
> needs to be whitelisted for further enumeration downstream. What do
> you think?

The pcieport driver is required for AER, PCIe native hotplug, PME,
etc., and it cannot be a module, so the whitelist wouldn't apply to
it.  I assume you need hotplug support, so you would have pcieport
enabled and built in statically.

If you're using ACPI hotplug, that doesn't require pcieport.

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

On Tue, Jun 9, 2020 at 5:04 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Tue, Jun 09, 2020 at 04:23:54PM -0700, Rajat Jain wrote:
> > Hi Bjorn,
> >
> > Thanks for sending out the summary, I was about to send it out but got lazy.
> >
> > On Tue, Jun 9, 2020 at 2:04 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > >
> > > On Sun, Jun 07, 2020 at 01:36:32PM +0200, Greg Kroah-Hartman wrote:
> > >
> > > > Your "problem" I think can be summed up a bit more concise:
> > > >       - you don't trust kernel drivers to be "secure" for untrusted
> > > >         devices
> > > >       - you only want to bind kernel drivers to "internal" devices
> > > >         automatically as you "trust" drivers in that situation.
> > > >       - you want to only bind specific kernel drivers that you somehow
> > > >         feel are "secure" to untrusted devices "outside" of a system
> > > >         when those devices are added to the system.
> > > >
> > > > Is that correct?
> > > >
> > > > If so, fine, you can do that today with the bind/unbind ability of
> > > > drivers, right?  After boot with your "trusted" drivers bound to
> > > > "internal" devices, turn off autobind of drivers to devices and then
> > > > manually bind them when you see new devices show up, as those "must" be
> > > > from external devices (see the bind/unbind files that all drivers export
> > > > for how to do this, and old lwn.net articles, this feature has been
> > > > around for a very long time.)
> > > >
> > > > I know for USB you can do this, odds are PCI you can turn off
> > > > autobinding as well, as I think this is a per-bus flag somewhere.  If
> > > > that's not exported to userspace, should be trivial to do so, should be
> > > > somewere in the driver model already...
> > > >
> > > > Ah, yes, look at the "drivers_autoprobe" and "drivers_probe" files in
> > > > sysfs for all busses.  Do those not work for you?
> > > >
> > > > My other points are the fact that you don't want to put policy in the
> > > > kernel, and I think that you can do everything you want in userspace
> > > > today, except maybe the fact that trying to determine what is "inside"
> > > > and "outside" is not always easy given that most hardware does not
> > > > export this information properly, if at all.  Go work with the firmware
> > > > people on that issue please, that would be most helpful for everyone
> > > > involved to get that finally straightened out.
> > >
> > > To sketch this out, my understanding of how this would work is:
> > >
> > >   - Expose the PCI pdev->untrusted bit in sysfs.  We don't expose this
> > >     today, but doing so would be trivial.  I think I would prefer a
> > >     sysfs name like "external" so it's more descriptive and less of a
> > >     judgment.
> >
> > Yes. I think we should probably semantically differentiate between
> > "external" and "external facing" devices. Root ports and downstream
> > ports can be "external facing" but are actually internal devices.
> > Anything below an "external facing" device is "external". So the sysfs
> > attribute "external" should be set only for devices that are truly
> > external.
>
> Good point; we (maybe you? :)) should fix that edge case.

Sure, happy to. I will start a fresh conversation about that (in a
separate thread).

>
> > Just a suggestion: Do you think an enum attribute may be better
> > instead, whose values could be "internal" / "external" /
> > "external-facing" in case need arises later to distinguish between
> > them?
>
> I don't see the need for an enum yet.  Maybe we should add that
> if/when we do need it?

Sure, no problems. (I just wanted to slip the thought into the
conversation as UAPI is hard to change later).

>
> > >   - Early userspace code prevents modular drivers from automatically
> > >     binding to PCI devices:
> > >
> > >       echo 0 > /sys/bus/pci/drivers_autoprobe
> >
> > Yes.
> > I believe this setting will apply it equally to both modular and
> > statically linked drivers?
>
> Yes.  The test is in bus_probe_device(), and it does the same for both
> modular and statically linked drivers.
>
> But for statically linked drivers, it only prevents them from binding
> to *hot-added* devices.  They will claim devices present at boot even
> before userspace code can run.

Yes, understood.

>
> > The one thing that still needs more thought is how about the
> > "pcieport" driver that enumerates the PCI bridges. I'm unsure if it
> > needs to be whitelisted for further enumeration downstream. What do
> > you think?
>
> The pcieport driver is required for AER, PCIe native hotplug, PME,
> etc., and it cannot be a module, so the whitelist wouldn't apply to
> it.

Not that I see the need, but slight clarification needed just to make
sure I understand it clearly:

Since pcieport driver is statically compiled in, AER, pciehp, PME, DPC
etc will always be enabled for devices plugged in during boot. But I
can still choose to choose to allow or deny for devices added *after
boot* using the whitelist, right?

Also, denying pcieport driver for hot-added PCIe bridges only disables
these pcieport services on those bridges, but device enumeration
further downstream those bridges is not an issue?

> I assume you need hotplug support, so you would have pcieport
> enabled and built in statically.
>
> If you're using ACPI hotplug, that doesn't require pcieport.

Thank you, this was indeed a long and useful thread :-)

Best Regards,

Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

On Tue, Jun 9, 2020 at 5:30 PM Rajat Jain <rajatja@google.com> wrote:
>
> On Tue, Jun 9, 2020 at 5:04 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > On Tue, Jun 09, 2020 at 04:23:54PM -0700, Rajat Jain wrote:
> > > Hi Bjorn,
> > >
> > > Thanks for sending out the summary, I was about to send it out but got lazy.
> > >
> > > On Tue, Jun 9, 2020 at 2:04 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > >
> > > > On Sun, Jun 07, 2020 at 01:36:32PM +0200, Greg Kroah-Hartman wrote:
> > > >
> > > > > Your "problem" I think can be summed up a bit more concise:
> > > > >       - you don't trust kernel drivers to be "secure" for untrusted
> > > > >         devices
> > > > >       - you only want to bind kernel drivers to "internal" devices
> > > > >         automatically as you "trust" drivers in that situation.
> > > > >       - you want to only bind specific kernel drivers that you somehow
> > > > >         feel are "secure" to untrusted devices "outside" of a system
> > > > >         when those devices are added to the system.
> > > > >
> > > > > Is that correct?
> > > > >
> > > > > If so, fine, you can do that today with the bind/unbind ability of
> > > > > drivers, right?  After boot with your "trusted" drivers bound to
> > > > > "internal" devices, turn off autobind of drivers to devices and then
> > > > > manually bind them when you see new devices show up, as those "must" be
> > > > > from external devices (see the bind/unbind files that all drivers export
> > > > > for how to do this, and old lwn.net articles, this feature has been
> > > > > around for a very long time.)
> > > > >
> > > > > I know for USB you can do this, odds are PCI you can turn off
> > > > > autobinding as well, as I think this is a per-bus flag somewhere.  If
> > > > > that's not exported to userspace, should be trivial to do so, should be
> > > > > somewere in the driver model already...
> > > > >
> > > > > Ah, yes, look at the "drivers_autoprobe" and "drivers_probe" files in
> > > > > sysfs for all busses.  Do those not work for you?
> > > > >
> > > > > My other points are the fact that you don't want to put policy in the
> > > > > kernel, and I think that you can do everything you want in userspace
> > > > > today, except maybe the fact that trying to determine what is "inside"
> > > > > and "outside" is not always easy given that most hardware does not
> > > > > export this information properly, if at all.  Go work with the firmware
> > > > > people on that issue please, that would be most helpful for everyone
> > > > > involved to get that finally straightened out.
> > > >
> > > > To sketch this out, my understanding of how this would work is:
> > > >
> > > >   - Expose the PCI pdev->untrusted bit in sysfs.  We don't expose this
> > > >     today, but doing so would be trivial.  I think I would prefer a
> > > >     sysfs name like "external" so it's more descriptive and less of a
> > > >     judgment.
> > >
> > > Yes. I think we should probably semantically differentiate between
> > > "external" and "external facing" devices. Root ports and downstream
> > > ports can be "external facing" but are actually internal devices.
> > > Anything below an "external facing" device is "external". So the sysfs
> > > attribute "external" should be set only for devices that are truly
> > > external.
> >
> > Good point; we (maybe you? :)) should fix that edge case.
>

I realized that we may not need to distinguish between internal and
external devices if we can assume that no internal PCI devices will
show up after boot. That assumption is 99% true for our use case
(leaving 1% out because we have some corner cases i.e. PCIe rescans,
module insertions etc that may probably make some internal devices
disappear and reappear).  If I find that I can do without the need for
a UAPI to distinguish internal vs external devices, do you still want
me to fix this edge case (i.e. "break" the pdev->untrusted flag into
"external_facing" and "external" devices)?

Thanks,

Rajat

> Sure, happy to. I will start a fresh conversation about that (in a
> separate thread).
>
> >
> > > Just a suggestion: Do you think an enum attribute may be better
> > > instead, whose values could be "internal" / "external" /
> > > "external-facing" in case need arises later to distinguish between
> > > them?
> >
> > I don't see the need for an enum yet.  Maybe we should add that
> > if/when we do need it?
>
> Sure, no problems. (I just wanted to slip the thought into the
> conversation as UAPI is hard to change later).
>
> >
> > > >   - Early userspace code prevents modular drivers from automatically
> > > >     binding to PCI devices:
> > > >
> > > >       echo 0 > /sys/bus/pci/drivers_autoprobe
> > >
> > > Yes.
> > > I believe this setting will apply it equally to both modular and
> > > statically linked drivers?
> >
> > Yes.  The test is in bus_probe_device(), and it does the same for both
> > modular and statically linked drivers.
> >
> > But for statically linked drivers, it only prevents them from binding
> > to *hot-added* devices.  They will claim devices present at boot even
> > before userspace code can run.
>
> Yes, understood.
>
> >
> > > The one thing that still needs more thought is how about the
> > > "pcieport" driver that enumerates the PCI bridges. I'm unsure if it
> > > needs to be whitelisted for further enumeration downstream. What do
> > > you think?
> >
> > The pcieport driver is required for AER, PCIe native hotplug, PME,
> > etc., and it cannot be a module, so the whitelist wouldn't apply to
> > it.
>
> Not that I see the need, but slight clarification needed just to make
> sure I understand it clearly:
>
> Since pcieport driver is statically compiled in, AER, pciehp, PME, DPC
> etc will always be enabled for devices plugged in during boot. But I
> can still choose to choose to allow or deny for devices added *after
> boot* using the whitelist, right?
>
> Also, denying pcieport driver for hot-added PCIe bridges only disables
> these pcieport services on those bridges, but device enumeration
> further downstream those bridges is not an issue?
>
> > I assume you need hotplug support, so you would have pcieport
> > enabled and built in statically.
> >
> > If you're using ACPI hotplug, that doesn't require pcieport.
>
> Thank you, this was indeed a long and useful thread :-)
>
> Best Regards,
>
> Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Bjorn Helgaas]

On Wed, Jun 10, 2020 at 01:17:54PM -0700, Rajat Jain wrote:
> On Tue, Jun 9, 2020 at 5:30 PM Rajat Jain <rajatja@google.com> wrote:
> >
> > On Tue, Jun 9, 2020 at 5:04 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > >
> > > On Tue, Jun 09, 2020 at 04:23:54PM -0700, Rajat Jain wrote:
> > > > Hi Bjorn,
> > > >
> > > > Thanks for sending out the summary, I was about to send it out but got lazy.
> > > >
> > > > On Tue, Jun 9, 2020 at 2:04 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > > >
> > > > > On Sun, Jun 07, 2020 at 01:36:32PM +0200, Greg Kroah-Hartman wrote:
> > > > >
> > > > > > Your "problem" I think can be summed up a bit more concise:
> > > > > >       - you don't trust kernel drivers to be "secure" for untrusted
> > > > > >         devices
> > > > > >       - you only want to bind kernel drivers to "internal" devices
> > > > > >         automatically as you "trust" drivers in that situation.
> > > > > >       - you want to only bind specific kernel drivers that you somehow
> > > > > >         feel are "secure" to untrusted devices "outside" of a system
> > > > > >         when those devices are added to the system.
> > > > > >
> > > > > > Is that correct?
> > > > > >
> > > > > > If so, fine, you can do that today with the bind/unbind ability of
> > > > > > drivers, right?  After boot with your "trusted" drivers bound to
> > > > > > "internal" devices, turn off autobind of drivers to devices and then
> > > > > > manually bind them when you see new devices show up, as those "must" be
> > > > > > from external devices (see the bind/unbind files that all drivers export
> > > > > > for how to do this, and old lwn.net articles, this feature has been
> > > > > > around for a very long time.)
> > > > > >
> > > > > > I know for USB you can do this, odds are PCI you can turn off
> > > > > > autobinding as well, as I think this is a per-bus flag somewhere.  If
> > > > > > that's not exported to userspace, should be trivial to do so, should be
> > > > > > somewere in the driver model already...
> > > > > >
> > > > > > Ah, yes, look at the "drivers_autoprobe" and "drivers_probe" files in
> > > > > > sysfs for all busses.  Do those not work for you?
> > > > > >
> > > > > > My other points are the fact that you don't want to put policy in the
> > > > > > kernel, and I think that you can do everything you want in userspace
> > > > > > today, except maybe the fact that trying to determine what is "inside"
> > > > > > and "outside" is not always easy given that most hardware does not
> > > > > > export this information properly, if at all.  Go work with the firmware
> > > > > > people on that issue please, that would be most helpful for everyone
> > > > > > involved to get that finally straightened out.
> > > > >
> > > > > To sketch this out, my understanding of how this would work is:
> > > > >
> > > > >   - Expose the PCI pdev->untrusted bit in sysfs.  We don't expose this
> > > > >     today, but doing so would be trivial.  I think I would prefer a
> > > > >     sysfs name like "external" so it's more descriptive and less of a
> > > > >     judgment.
> > > >
> > > > Yes. I think we should probably semantically differentiate between
> > > > "external" and "external facing" devices. Root ports and downstream
> > > > ports can be "external facing" but are actually internal devices.
> > > > Anything below an "external facing" device is "external". So the sysfs
> > > > attribute "external" should be set only for devices that are truly
> > > > external.
> > >
> > > Good point; we (maybe you? :)) should fix that edge case.
> >
> 
> I realized that we may not need to distinguish between internal and
> external devices if we can assume that no internal PCI devices will
> show up after boot. That assumption is 99% true for our use case
> (leaving 1% out because we have some corner cases i.e. PCIe rescans,
> module insertions etc that may probably make some internal devices
> disappear and reappear).  If I find that I can do without the need for
> a UAPI to distinguish internal vs external devices, do you still want
> me to fix this edge case (i.e. "break" the pdev->untrusted flag into
> "external_facing" and "external" devices)?

I'm not sure there's a need to explicitly identify the
"external-facing" devices at all.  AFAIK there's nothing today that
would do anything different with a root port that happens to have an
external connector.

The "untrusted" uses today are things like forcing use of an IOMMU.
Most root ports don't initiate DMA themselves, but even if they did,
there's no reason to treat DMA from the root port differently than DMA
from other internal devices.  What we want is to treat DMA from
devices *connected* to that external connector differently.

So yes, I think we should change the current behavior so we only set
the dev->untrusted flag for devices *downstream* from an
external-facing device, but not the external-facing device itself.

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Bjorn Helgaas]

On Tue, Jun 09, 2020 at 05:30:13PM -0700, Rajat Jain wrote:
> On Tue, Jun 9, 2020 at 5:04 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Tue, Jun 09, 2020 at 04:23:54PM -0700, Rajat Jain wrote:
> > > Thanks for sending out the summary, I was about to send it out but got lazy.
> > > ...

> > > The one thing that still needs more thought is how about the
> > > "pcieport" driver that enumerates the PCI bridges. I'm unsure if it
> > > needs to be whitelisted for further enumeration downstream. What do
> > > you think?
> >
> > The pcieport driver is required for AER, PCIe native hotplug, PME,
> > etc., and it cannot be a module, so the whitelist wouldn't apply to
> > it.
> 
> Not that I see the need, but slight clarification needed just to make
> sure I understand it clearly:
> 
> Since pcieport driver is statically compiled in, AER, pciehp, PME, DPC
> etc will always be enabled for devices plugged in during boot. But I
> can still choose to choose to allow or deny for devices added *after
> boot* using the whitelist, right?

Yes, I think so.  However, if pcieport doesn't claim hot-added devices
like a dock, I don't think hotplug of PCI things downstream from the
dock will work, e.g., if there are Thunderbolt switches in a monitor
or something.

> Also, denying pcieport driver for hot-added PCIe bridges only disables
> these pcieport services on those bridges, but device enumeration
> further downstream those bridges is not an issue?

Right.  Devices without pcieport would not be able to report hotplug
events, so we wouldn't notice hot-adds or -removes involving those
devices.

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

On Wed, Jun 10, 2020 at 4:01 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Tue, Jun 09, 2020 at 05:30:13PM -0700, Rajat Jain wrote:
> > On Tue, Jun 9, 2020 at 5:04 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > > On Tue, Jun 09, 2020 at 04:23:54PM -0700, Rajat Jain wrote:
> > > > Thanks for sending out the summary, I was about to send it out but got lazy.
> > > > ...
>
> > > > The one thing that still needs more thought is how about the
> > > > "pcieport" driver that enumerates the PCI bridges. I'm unsure if it
> > > > needs to be whitelisted for further enumeration downstream. What do
> > > > you think?
> > >
> > > The pcieport driver is required for AER, PCIe native hotplug, PME,
> > > etc., and it cannot be a module, so the whitelist wouldn't apply to
> > > it.
> >
> > Not that I see the need, but slight clarification needed just to make
> > sure I understand it clearly:
> >
> > Since pcieport driver is statically compiled in, AER, pciehp, PME, DPC
> > etc will always be enabled for devices plugged in during boot. But I
> > can still choose to choose to allow or deny for devices added *after
> > boot* using the whitelist, right?
>
> Yes, I think so.  However, if pcieport doesn't claim hot-added devices
> like a dock, I don't think hotplug of PCI things downstream from the
> dock will work, e.g., if there are Thunderbolt switches in a monitor
> or something.

Yes, understood, thanks.

>
> > Also, denying pcieport driver for hot-added PCIe bridges only disables
> > these pcieport services on those bridges, but device enumeration
> > further downstream those bridges is not an issue?
>
> Right.  Devices without pcieport would not be able to report hotplug
> events, so we wouldn't notice hot-adds or -removes involving those
> devices.

Understood.

Thanks,

Rajat

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Tue, Jun 09, 2020 at 04:23:54PM -0700, Rajat Jain wrote:
> The one thing that still needs more thought is how about the
> "pcieport" driver that enumerates the PCI bridges. I'm unsure if it
> needs to be whitelisted for further enumeration downstream. What do
> you think?

Why not just do whatever type of "code review" you need to do for that
one core driver to get that off of your "drivers to worry about" list?
:)

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Oliver O'Halloran]

On Wed, Jun 10, 2020 at 7:04 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> To sketch this out, my understanding of how this would work is:
>
>   - Expose the PCI pdev->untrusted bit in sysfs.  We don't expose this
>     today, but doing so would be trivial.  I think I would prefer a
>     sysfs name like "external" so it's more descriptive and less of a
>     judgment.
>
>     This comes from either the DT "external-facing" property or the
>     ACPI "ExternalFacingPort" property.

I don't think internal / external is the right distinction to be
making. We have a similar trust issue with the BMC in servers even
though they're internal devices. They're typically network accessible
and infrequently updated so treating them as trustworthy isn't a great
idea. We have been slowly de-privileging the BMC over the last few
years, but the PCIe interface isn't locked down enough for my liking
since the SoCs we use do allow software to set the VDID and perform
arbitrary DMAs (thankfully limited to 32bit). If we're going to add in
infrastructure for handling possibly untrustworthy PCI devices then
I'd like to use that for BMCs too.

>   - All devices present at boot are enumerated.  Any statically built
>     drivers will bind to them before any userspace code runs.
>
>     If you want to keep statically built drivers from binding, you'd
>     need to invent some mechanism so pci_driver_init() could clear
>     drivers_autoprobe after registering pci_bus_type.
>
>   - Early userspace code prevents modular drivers from automatically
>     binding to PCI devices:
>
>       echo 0 > /sys/bus/pci/drivers_autoprobe
>
>     This prevents modular drivers from binding to all devices, whether
>     present at boot or hot-added.

I don't see why this is preferable to just disabling autoprobe for
untrusted devices. That would dovetail nicely with Rajat's whitelist
idea if we want to go down that route and I think we might want to.
The BMC usually provides some form of VGA console and we'd like that
to continue working out-of-the-box without too much user (or distro)
intervention.

Oliver

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

On Tue, Jun 9, 2020 at 6:34 PM Oliver O'Halloran <oohall@gmail.com> wrote:
>
> On Wed, Jun 10, 2020 at 7:04 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > To sketch this out, my understanding of how this would work is:
> >
> >   - Expose the PCI pdev->untrusted bit in sysfs.  We don't expose this
> >     today, but doing so would be trivial.  I think I would prefer a
> >     sysfs name like "external" so it's more descriptive and less of a
> >     judgment.
> >
> >     This comes from either the DT "external-facing" property or the
> >     ACPI "ExternalFacingPort" property.
>
> I don't think internal / external is the right distinction to be
> making. We have a similar trust issue with the BMC in servers even
> though they're internal devices. They're typically network accessible
> and infrequently updated so treating them as trustworthy isn't a great
> idea. We have been slowly de-privileging the BMC over the last few
> years, but the PCIe interface isn't locked down enough for my liking
> since the SoCs we use do allow software to set the VDID and perform
> arbitrary DMAs (thankfully limited to 32bit). If we're going to add in
> infrastructure for handling possibly untrustworthy PCI devices then
> I'd like to use that for BMCs too.
>
> >   - All devices present at boot are enumerated.  Any statically built
> >     drivers will bind to them before any userspace code runs.
> >
> >     If you want to keep statically built drivers from binding, you'd
> >     need to invent some mechanism so pci_driver_init() could clear
> >     drivers_autoprobe after registering pci_bus_type.
> >
> >   - Early userspace code prevents modular drivers from automatically
> >     binding to PCI devices:
> >
> >       echo 0 > /sys/bus/pci/drivers_autoprobe
> >
> >     This prevents modular drivers from binding to all devices, whether
> >     present at boot or hot-added.
>
> I don't see why this is preferable to just disabling autoprobe for
> untrusted devices. That would dovetail nicely with Rajat's whitelist
> idea if we want to go down that route and I think we might want to.
> The BMC usually provides some form of VGA console and we'd like that
> to continue working out-of-the-box without too much user (or distro)
> intervention.

I wouldn't mind introducing a kernel parameter to disable auto-probing
of untrusted devices if there is a wider agreement here.
The only notch is that in my opinion, if present, that parameter
should disable auto-probing for "external" devices only (i.e.
"external-facing" devices should still be auto-probed).

Thanks,

Rajat

>
> Oliver

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Rajat Jain]

On Wed, Jun 10, 2020 at 12:57 PM Rajat Jain <rajatxjain@gmail.com> wrote:
>
> On Tue, Jun 9, 2020 at 6:34 PM Oliver O'Halloran <oohall@gmail.com> wrote:
> >
> > On Wed, Jun 10, 2020 at 7:04 AM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > >
> > > To sketch this out, my understanding of how this would work is:
> > >
> > >   - Expose the PCI pdev->untrusted bit in sysfs.  We don't expose this
> > >     today, but doing so would be trivial.  I think I would prefer a
> > >     sysfs name like "external" so it's more descriptive and less of a
> > >     judgment.
> > >
> > >     This comes from either the DT "external-facing" property or the
> > >     ACPI "ExternalFacingPort" property.
> >
> > I don't think internal / external is the right distinction to be
> > making. We have a similar trust issue with the BMC in servers even
> > though they're internal devices. They're typically network accessible
> > and infrequently updated so treating them as trustworthy isn't a great
> > idea. We have been slowly de-privileging the BMC over the last few
> > years, but the PCIe interface isn't locked down enough for my liking
> > since the SoCs we use do allow software to set the VDID and perform
> > arbitrary DMAs (thankfully limited to 32bit). If we're going to add in
> > infrastructure for handling possibly untrustworthy PCI devices then
> > I'd like to use that for BMCs too.
> >
> > >   - All devices present at boot are enumerated.  Any statically built
> > >     drivers will bind to them before any userspace code runs.
> > >
> > >     If you want to keep statically built drivers from binding, you'd
> > >     need to invent some mechanism so pci_driver_init() could clear
> > >     drivers_autoprobe after registering pci_bus_type.
> > >
> > >   - Early userspace code prevents modular drivers from automatically
> > >     binding to PCI devices:
> > >
> > >       echo 0 > /sys/bus/pci/drivers_autoprobe
> > >
> > >     This prevents modular drivers from binding to all devices, whether
> > >     present at boot or hot-added.
> >
> > I don't see why this is preferable to just disabling autoprobe for
> > untrusted devices. That would dovetail nicely with Rajat's whitelist
> > idea if we want to go down that route and I think we might want to.
> > The BMC usually provides some form of VGA console and we'd like that
> > to continue working out-of-the-box without too much user (or distro)
> > intervention.
>
> I wouldn't mind introducing a kernel parameter to disable auto-probing
> of untrusted devices if there is a wider agreement here.
> The only notch is that in my opinion, if present, that parameter
> should disable auto-probing for "external" devices only (i.e.
> "external-facing" devices should still be auto-probed).

So I looked around at my systems, and I realized that I will have to
go this way (introduce a parameter to disable auto-probing of
untrusted devices), because we do have systems on which we might not
have control over what devices will show up when (external devices'
PCI link may come up before the userspace gets a chance to run). I
shall be sending out a patch soon. I've sent out a couple of other
loosely related patches here:

https://lkml.org/lkml/2020/6/15/1447

Thanks,

Rajat

>
> Thanks,
>
> Rajat
>
> >
> > Oliver

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Greg Kroah-Hartman]

On Tue, Jun 09, 2020 at 04:04:00PM -0500, Bjorn Helgaas wrote:
> On Sun, Jun 07, 2020 at 01:36:32PM +0200, Greg Kroah-Hartman wrote:
> 
> > Your "problem" I think can be summed up a bit more concise:
> > 	- you don't trust kernel drivers to be "secure" for untrusted
> > 	  devices
> > 	- you only want to bind kernel drivers to "internal" devices
> > 	  automatically as you "trust" drivers in that situation.
> > 	- you want to only bind specific kernel drivers that you somehow
> > 	  feel are "secure" to untrusted devices "outside" of a system
> > 	  when those devices are added to the system.
> > 
> > Is that correct?
> > 
> > If so, fine, you can do that today with the bind/unbind ability of
> > drivers, right?  After boot with your "trusted" drivers bound to
> > "internal" devices, turn off autobind of drivers to devices and then
> > manually bind them when you see new devices show up, as those "must" be
> > from external devices (see the bind/unbind files that all drivers export
> > for how to do this, and old lwn.net articles, this feature has been
> > around for a very long time.)
> > 
> > I know for USB you can do this, odds are PCI you can turn off
> > autobinding as well, as I think this is a per-bus flag somewhere.  If
> > that's not exported to userspace, should be trivial to do so, should be
> > somewere in the driver model already...
> > 
> > Ah, yes, look at the "drivers_autoprobe" and "drivers_probe" files in
> > sysfs for all busses.  Do those not work for you?
> > 
> > My other points are the fact that you don't want to put policy in the
> > kernel, and I think that you can do everything you want in userspace
> > today, except maybe the fact that trying to determine what is "inside"
> > and "outside" is not always easy given that most hardware does not
> > export this information properly, if at all.  Go work with the firmware
> > people on that issue please, that would be most helpful for everyone
> > involved to get that finally straightened out.
> 
> To sketch this out, my understanding of how this would work is:
> 
>   - Expose the PCI pdev->untrusted bit in sysfs.  We don't expose this
>     today, but doing so would be trivial.  I think I would prefer a
>     sysfs name like "external" so it's more descriptive and less of a
>     judgment.
> 
>     This comes from either the DT "external-facing" property or the
>     ACPI "ExternalFacingPort" property.  

Good idea, but as people have pointed out, even these don't always work
so userspace will need to be able to override that somehow :(

>   - All devices present at boot are enumerated.  Any statically built
>     drivers will bind to them before any userspace code runs.
> 
>     If you want to keep statically built drivers from binding, you'd
>     need to invent some mechanism so pci_driver_init() could clear
>     drivers_autoprobe after registering pci_bus_type.
> 
>   - Early userspace code prevents modular drivers from automatically
>     binding to PCI devices:
> 
>       echo 0 > /sys/bus/pci/drivers_autoprobe
> 
>     This prevents modular drivers from binding to all devices, whether
>     present at boot or hot-added.
> 
>   - Userspace code uses the sysfs "bind" file to control which drivers
>     are loaded and can bind to each device, e.g.,
> 
>       echo 0000:02:00.0 > /sys/bus/pci/drivers/nvme/bind

Seems good to me, and also matches what the current USB tools do for
this type of thing.

thanks,

greg k-h

Re: [RFC] Restrict the untrusted devices, to bind to only a set of "whitelisted" drivers

[Joerg Roedel]

On Wed, May 13, 2020 at 02:26:18PM -0700, Rajat Jain wrote:
> * A driver could be double fetching the memory, causing it to do
> different things than intended. E.g.
> * A driver could be (negligently) passing some kernel addresses to the device.
> * A driver could be using (for memory dereferencing, for e.g.) the
> address/indices, given by the device, without enough validation.
> * A driver may negligently be sharing the DMA memory with some other
> driver data in the same PAGE. Since the IOMMU restrictions are PAGE
> granular, this might give device access to that driver data.

The Intel IOMMU driver has a solution for that problem as it has iommu
based bounce-buffer dma ops. This means that a driver can't
accidentially share sensitive information on the same page with a
device.

This idea should be generalized and made available for all iommu-drivers
in the form of integrating it into the dma-iommu code, or have a
separate generic dma-ops implementation, which does:

	1) Give the device direct access to DMA buffers if they are
	   IOMMU-page aligned (both start and size).

	2) Use bounce buffering for DMA buffers that don't align with
	   iommu page-size.

This would at least eliminate this type of attack made possible by
uncautious drivers.

Regards,

	Joerg