[PATCH v2] iommu: add support for drivers that manage iommu explicitly

[PATCH v2] iommu: add support for drivers that manage iommu explicitly

[Rob Clark]

From: Rob Clark <robdclark@chromium.org>

Avoid attaching any non-driver managed domain if the driver indicates
that it manages the iommu directly.

This avoids a problem on devices where the bootloader takes the SMMU out
of bypass and enables scanout, such as is the case on snapdragon aarch64
laptops and newer snapdragon android devices.  Attaching an IDENTITY or
DMA domain before the driver has a chance to intervene will break efifb
scanout and start triggering iommu faults.

If the driver manages the iommu directly (as does drm/msm), it can
shut down scanout when it is ready to take over the display, before
attaching an UNMANAGED domain.

Signed-off-by: Rob Clark <robdclark@chromium.org>
---
v2. Move the check into arm_smmu_attach_dev() (as I *think* this is
    what Robin preferred; update commit msg to focus on the display
    related issue that this solves.

We also need Bjorn's patch set to inherit SMR and CB config during
init:

https://www.spinics.net/lists/arm-kernel/msg732246.html

 drivers/iommu/arm-smmu.c | 11 +++++++++++
 include/linux/device.h   |  3 ++-
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 1a5efa7c8767..4a80710124db 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c
@@ -1411,6 +1411,17 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
 		return -ENXIO;
 	}
 
+	/*
+	 * If driver is going to manage iommu directly, then avoid
+	 * attaching any non driver managed domain.  There could
+	 * be already active dma underway (ie. scanout in case of
+	 * bootloader enabled display), and interfering with that
+	 * will make things go *boom*
+	 */
+	if ((domain->type != IOMMU_DOMAIN_UNMANAGED) &&
+	    dev->driver && dev->driver->driver_manages_iommu)
+		return 0;
+
 	/*
 	 * FIXME: The arch/arm DMA API code tries to attach devices to its own
 	 * domains between of_xlate() and add_device() - we have no way to cope
diff --git a/include/linux/device.h b/include/linux/device.h
index e138baabe01e..d98aa4d3c8c3 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -282,7 +282,8 @@ struct device_driver {
 	struct module		*owner;
 	const char		*mod_name;	/* used for built-in modules */
 
-	bool suppress_bind_attrs;	/* disables bind/unbind via sysfs */
+	bool suppress_bind_attrs:1;	/* disables bind/unbind via sysfs */
+	bool driver_manages_iommu:1;	/* driver manages IOMMU explicitly */
 	enum probe_type probe_type;
 
 	const struct of_device_id	*of_match_table;
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly

[Joerg Roedel]

On Wed, Jul 10, 2019 at 11:28:30AM -0700, Rob Clark wrote:
> --- a/include/linux/device.h
> +++ b/include/linux/device.h
> @@ -282,7 +282,8 @@ struct device_driver {
>  	struct module		*owner;
>  	const char		*mod_name;	/* used for built-in modules */
>  
> -	bool suppress_bind_attrs;	/* disables bind/unbind via sysfs */
> +	bool suppress_bind_attrs:1;	/* disables bind/unbind via sysfs */
> +	bool driver_manages_iommu:1;	/* driver manages IOMMU explicitly */

Who will set this bit?


Regards,

	Joerg

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly

[Rob Clark]

On Mon, Jul 22, 2019 at 7:28 AM Joerg Roedel <joro@8bytes.org> wrote:
>
> On Wed, Jul 10, 2019 at 11:28:30AM -0700, Rob Clark wrote:
> > --- a/include/linux/device.h
> > +++ b/include/linux/device.h
> > @@ -282,7 +282,8 @@ struct device_driver {
> >       struct module           *owner;
> >       const char              *mod_name;      /* used for built-in modules */
> >
> > -     bool suppress_bind_attrs;       /* disables bind/unbind via sysfs */
> > +     bool suppress_bind_attrs:1;     /* disables bind/unbind via sysfs */
> > +     bool driver_manages_iommu:1;    /* driver manages IOMMU explicitly */
>
> Who will set this bit?
>

It is set by the driver:

https://patchwork.freedesktop.org/patch/315291/

(This doesn't really belong in devicetree, since it isn't a
description of the hardware, so the driver is really the only place to
set this.. which is fine because it is about a detail of how the
driver works.)

BR,
-R

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly

[Joerg Roedel]

On Mon, Jul 22, 2019 at 08:41:34AM -0700, Rob Clark wrote:
> It is set by the driver:
> 
> https://patchwork.freedesktop.org/patch/315291/
> 
> (This doesn't really belong in devicetree, since it isn't a
> description of the hardware, so the driver is really the only place to
> set this.. which is fine because it is about a detail of how the
> driver works.)

It is more a detail about how the firmware works. IIUC the problem is
that the firmware initializes the context mappings for the GPU and the
OS doesn't know anything about that and just overwrites them, causing
the firmware GPU driver to fail badly.

So I think it is the task of the firmware to tell the OS not to touch
the devices mappings until the OS device driver takes over. On x86 there
is something similar with the RMRR/unity-map tables from the firmware.

Regards,

	Joerg

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly

[Rob Clark]

On Mon, Jul 22, 2019 at 8:48 AM Joerg Roedel <joro@8bytes.org> wrote:
>
> On Mon, Jul 22, 2019 at 08:41:34AM -0700, Rob Clark wrote:
> > It is set by the driver:
> >
> > https://patchwork.freedesktop.org/patch/315291/
> >
> > (This doesn't really belong in devicetree, since it isn't a
> > description of the hardware, so the driver is really the only place to
> > set this.. which is fine because it is about a detail of how the
> > driver works.)
>
> It is more a detail about how the firmware works. IIUC the problem is
> that the firmware initializes the context mappings for the GPU and the
> OS doesn't know anything about that and just overwrites them, causing
> the firmware GPU driver to fail badly.
>
> So I think it is the task of the firmware to tell the OS not to touch
> the devices mappings until the OS device driver takes over. On x86 there
> is something similar with the RMRR/unity-map tables from the firmware.
>

Bjorn had a patchset[1] to inherit the config from firmware/bootloader
when arm-smmu is probed which handles that part of the problem.  My
patch is intended to be used on top of his patchset.  This seems to me
like the best solution, if we don't have control over the firmware.

BR,
-R

[1] https://www.spinics.net/lists/arm-kernel/msg732246.html

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly

[Will Deacon]

On Mon, Jul 22, 2019 at 09:23:48AM -0700, Rob Clark wrote:
> On Mon, Jul 22, 2019 at 8:48 AM Joerg Roedel <joro@8bytes.org> wrote:
> >
> > On Mon, Jul 22, 2019 at 08:41:34AM -0700, Rob Clark wrote:
> > > It is set by the driver:
> > >
> > > https://patchwork.freedesktop.org/patch/315291/
> > >
> > > (This doesn't really belong in devicetree, since it isn't a
> > > description of the hardware, so the driver is really the only place to
> > > set this.. which is fine because it is about a detail of how the
> > > driver works.)
> >
> > It is more a detail about how the firmware works. IIUC the problem is
> > that the firmware initializes the context mappings for the GPU and the
> > OS doesn't know anything about that and just overwrites them, causing
> > the firmware GPU driver to fail badly.
> >
> > So I think it is the task of the firmware to tell the OS not to touch
> > the devices mappings until the OS device driver takes over. On x86 there
> > is something similar with the RMRR/unity-map tables from the firmware.
> >
> 
> Bjorn had a patchset[1] to inherit the config from firmware/bootloader
> when arm-smmu is probed which handles that part of the problem.  My
> patch is intended to be used on top of his patchset.  This seems to me
> like the best solution, if we don't have control over the firmware.

Hmm, but the feedback from Robin on the thread you cite was that this should
be generalised to look more like RMRR, so there seems to be a clear message
here.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly

[Rob Clark]

On Tue, Jul 23, 2019 at 8:38 AM Will Deacon <will@kernel.org> wrote:
>
> On Mon, Jul 22, 2019 at 09:23:48AM -0700, Rob Clark wrote:
> > On Mon, Jul 22, 2019 at 8:48 AM Joerg Roedel <joro@8bytes.org> wrote:
> > >
> > > On Mon, Jul 22, 2019 at 08:41:34AM -0700, Rob Clark wrote:
> > > > It is set by the driver:
> > > >
> > > > https://patchwork.freedesktop.org/patch/315291/
> > > >
> > > > (This doesn't really belong in devicetree, since it isn't a
> > > > description of the hardware, so the driver is really the only place to
> > > > set this.. which is fine because it is about a detail of how the
> > > > driver works.)
> > >
> > > It is more a detail about how the firmware works. IIUC the problem is
> > > that the firmware initializes the context mappings for the GPU and the
> > > OS doesn't know anything about that and just overwrites them, causing
> > > the firmware GPU driver to fail badly.
> > >
> > > So I think it is the task of the firmware to tell the OS not to touch
> > > the devices mappings until the OS device driver takes over. On x86 there
> > > is something similar with the RMRR/unity-map tables from the firmware.
> > >
> >
> > Bjorn had a patchset[1] to inherit the config from firmware/bootloader
> > when arm-smmu is probed which handles that part of the problem.  My
> > patch is intended to be used on top of his patchset.  This seems to me
> > like the best solution, if we don't have control over the firmware.
>
> Hmm, but the feedback from Robin on the thread you cite was that this should
> be generalised to look more like RMRR, so there seems to be a clear message
> here.
>

Perhaps it is a lack of creativity, or lack of familiarity w/ iommu vs
virtualization, but I'm not quite seeing how RMRR would help.. in
particular when dealing with both DT and ACPI cases.  So I kinda
prefer, when possible, if arm-smmu can figure out what is going on by
looking at the hw state at boot (since that approach would work
equally well for DT and ACPI).

I *think* (but need to confirm if Bjorn hasn't already) that the
memory for the pagetables that firmware/bootloader sets up is already
removed from the memory map efi passes to kernel, so we don't need to
worry about kernel stomping in-use pagetables.

BR,
-R

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly

[Will Deacon]

On Tue, Jul 23, 2019 at 10:40:55AM -0700, Rob Clark wrote:
> On Tue, Jul 23, 2019 at 8:38 AM Will Deacon <will@kernel.org> wrote:
> >
> > On Mon, Jul 22, 2019 at 09:23:48AM -0700, Rob Clark wrote:
> > > On Mon, Jul 22, 2019 at 8:48 AM Joerg Roedel <joro@8bytes.org> wrote:
> > > >
> > > > On Mon, Jul 22, 2019 at 08:41:34AM -0700, Rob Clark wrote:
> > > > > It is set by the driver:
> > > > >
> > > > > https://patchwork.freedesktop.org/patch/315291/
> > > > >
> > > > > (This doesn't really belong in devicetree, since it isn't a
> > > > > description of the hardware, so the driver is really the only place to
> > > > > set this.. which is fine because it is about a detail of how the
> > > > > driver works.)
> > > >
> > > > It is more a detail about how the firmware works. IIUC the problem is
> > > > that the firmware initializes the context mappings for the GPU and the
> > > > OS doesn't know anything about that and just overwrites them, causing
> > > > the firmware GPU driver to fail badly.
> > > >
> > > > So I think it is the task of the firmware to tell the OS not to touch
> > > > the devices mappings until the OS device driver takes over. On x86 there
> > > > is something similar with the RMRR/unity-map tables from the firmware.
> > > >
> > >
> > > Bjorn had a patchset[1] to inherit the config from firmware/bootloader
> > > when arm-smmu is probed which handles that part of the problem.  My
> > > patch is intended to be used on top of his patchset.  This seems to me
> > > like the best solution, if we don't have control over the firmware.
> >
> > Hmm, but the feedback from Robin on the thread you cite was that this should
> > be generalised to look more like RMRR, so there seems to be a clear message
> > here.
> >
> 
> Perhaps it is a lack of creativity, or lack of familiarity w/ iommu vs
> virtualization, but I'm not quite seeing how RMRR would help.. in
> particular when dealing with both DT and ACPI cases.

Well, I suppose we'd have something for DT and something for ACPI and we'd
try to make them look similar enough that we don't need lots of divergent
code in the kernel. The RMRR-like description would describe that, for a
particular device, a specific virt:phys mapping needs to exist in the
small window between initialising the SMMU and re-initialising the device
(GPU).

I would prefer this to be framebuffer-specific, since we're already in
flagrant violation of the arm64 boot requirements wrt ongoing DMA and making
this too general could lead to all sorts of brain damage. That would
probably also allow us to limit the flexibility, by mandating things like
alignment and memory attributes.

Having said that, I just realised I'm still a bit confused about the
problem: why does the bootloader require SMMU translation for the GPU at
all? If we could limit this whole thing to be identity-mapped/bypassed,
then all we need is a per-device flag in the firmware to indicate that the
SMMU should be initialised to allow passthrough for transactions originating
from that device.

> So I kinda prefer, when possible, if arm-smmu can figure out what is going
> on by looking at the hw state at boot (since that approach would work
> equally well for DT and ACPI).

That's not going to fly.

Forcing Linux to infer the state of the system by effectively parsing the
hardware configuration directly is fragile, error-prone and may not even be
possible in the general case. Worse, if this goes wrong, the symptoms are
very likely to be undiagnosable memory corruption, which is pretty awful in
my opinion.

Not only would you need separate parsing code for every IOMMU out there,
but you'd also need to make Linux aware of device aspects that it otherwise
doesn't care about, just in case the firmware decided to use them.
Furthermore, running an older kernel on newer hardware (which may have some
extensions), could cause the parsing to silently ignore parts of the device
that indicate memory regions which are in use. On top of that, there made be
device-global state that we are unable to reconfigure and that affect
devices other than the ones in question.

So no, I'm very much against this approach and the solution absolutely needs
to come in the form of a more abstract description from firmware.

> I *think* (but need to confirm if Bjorn hasn't already) that the
> memory for the pagetables that firmware/bootloader sets up is already
> removed from the memory map efi passes to kernel, so we don't need to
> worry about kernel stomping in-use pagetables.

It's precisely this sort of fragility that makes me nervous about this whole
approach.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v2] iommu: add support for drivers that manage iommu explicitly

[Rob Clark]

On Wed, Jul 24, 2019 at 3:51 AM Will Deacon <will@kernel.org> wrote:
>
> On Tue, Jul 23, 2019 at 10:40:55AM -0700, Rob Clark wrote:
> > On Tue, Jul 23, 2019 at 8:38 AM Will Deacon <will@kernel.org> wrote:
> > >
> > > On Mon, Jul 22, 2019 at 09:23:48AM -0700, Rob Clark wrote:
> > > > On Mon, Jul 22, 2019 at 8:48 AM Joerg Roedel <joro@8bytes.org> wrote:
> > > > >
> > > > > On Mon, Jul 22, 2019 at 08:41:34AM -0700, Rob Clark wrote:
> > > > > > It is set by the driver:
> > > > > >
> > > > > > https://patchwork.freedesktop.org/patch/315291/
> > > > > >
> > > > > > (This doesn't really belong in devicetree, since it isn't a
> > > > > > description of the hardware, so the driver is really the only place to
> > > > > > set this.. which is fine because it is about a detail of how the
> > > > > > driver works.)
> > > > >
> > > > > It is more a detail about how the firmware works. IIUC the problem is
> > > > > that the firmware initializes the context mappings for the GPU and the
> > > > > OS doesn't know anything about that and just overwrites them, causing
> > > > > the firmware GPU driver to fail badly.
> > > > >
> > > > > So I think it is the task of the firmware to tell the OS not to touch
> > > > > the devices mappings until the OS device driver takes over. On x86 there
> > > > > is something similar with the RMRR/unity-map tables from the firmware.
> > > > >
> > > >
> > > > Bjorn had a patchset[1] to inherit the config from firmware/bootloader
> > > > when arm-smmu is probed which handles that part of the problem.  My
> > > > patch is intended to be used on top of his patchset.  This seems to me
> > > > like the best solution, if we don't have control over the firmware.
> > >
> > > Hmm, but the feedback from Robin on the thread you cite was that this should
> > > be generalised to look more like RMRR, so there seems to be a clear message
> > > here.
> > >
> >
> > Perhaps it is a lack of creativity, or lack of familiarity w/ iommu vs
> > virtualization, but I'm not quite seeing how RMRR would help.. in
> > particular when dealing with both DT and ACPI cases.
>
> Well, I suppose we'd have something for DT and something for ACPI and we'd
> try to make them look similar enough that we don't need lots of divergent
> code in the kernel. The RMRR-like description would describe that, for a
> particular device, a specific virt:phys mapping needs to exist in the
> small window between initialising the SMMU and re-initialising the device
> (GPU).

For both DT and ACPI (or perhaps more accurately UEFI and non-UEFI) we
often don't have much/any control of the firmware.  In the UEFI case,
we can at least get the physical address of the scanout buffer from
EFI GOP (since VA=PA at that point).  In either case we could get the
iova by reading back controller state.  We kinda just need the iommu
driver to not change anything about the context bank the display is
using until the display driver is ready to install it's own
pagetables.

Initially I just want to shut down display, and then bring it back up
w/ my own pagetables.. but eventually, once iommu/clk/genpd issues are
sorted upstream, I'm planning to read out more completely the display
state, and remap the existing scanout buffer at same iova in my own
pagetables/iommu_domain, for a flicker-free display handover... that
is a lot more work but at least it is self contained in the display
(and bridge/panel) drivers.

> I would prefer this to be framebuffer-specific, since we're already in
> flagrant violation of the arm64 boot requirements wrt ongoing DMA and making
> this too general could lead to all sorts of brain damage. That would
> probably also allow us to limit the flexibility, by mandating things like
> alignment and memory attributes.

I'd be pretty happy if we could convince qcom to use EFI GOP on
android devices too..

Although there is a lot more activity these days with people bringing
upstream support to various existing android phones/tablets.. and I'd
like to see that continue without downstream hacks due to lack of
control over firmware.

> Having said that, I just realised I'm still a bit confused about the
> problem: why does the bootloader require SMMU translation for the GPU at
> all? If we could limit this whole thing to be identity-mapped/bypassed,
> then all we need is a per-device flag in the firmware to indicate that the
> SMMU should be initialised to allow passthrough for transactions originating
> from that device.

I was chatting last night w/ Bjorn on IRC.. and he mentioned that it
looked like TTBRn was 0x0.  Which I wasn't expecting, and I didn't
realize was a legit thing.  Maybe the purpose is to allow display to
access memory w/ iova==pa but disallow memory access from other
devices using different context banks of the same iommu?  Maybe this
makes more sense to you?

> > So I kinda prefer, when possible, if arm-smmu can figure out what is going
> > on by looking at the hw state at boot (since that approach would work
> > equally well for DT and ACPI).
>
> That's not going to fly.
>
> Forcing Linux to infer the state of the system by effectively parsing the
> hardware configuration directly is fragile, error-prone and may not even be
> possible in the general case. Worse, if this goes wrong, the symptoms are
> very likely to be undiagnosable memory corruption, which is pretty awful in
> my opinion.

So, I guess in a DT world, we could have some boot-on; flag in the
device node.. that tells the iommu driver the stream-id(s) which might
be in use.  I guess the iommu driver would still need to map that back
to context bank, and then leave that CB and stream<->CB mapping alone.
But I think that would at least prevent "false positives", which I
think is what you are worrying about.

I'm not sure how this would work for ACPI, but tbh I haven't looked
yet at how iommu and device are connected w/ ACPI.

> Not only would you need separate parsing code for every IOMMU out there,
> but you'd also need to make Linux aware of device aspects that it otherwise
> doesn't care about, just in case the firmware decided to use them.
> Furthermore, running an older kernel on newer hardware (which may have some
> extensions), could cause the parsing to silently ignore parts of the device
> that indicate memory regions which are in use. On top of that, there made be
> device-global state that we are unable to reconfigure and that affect
> devices other than the ones in question.

I'm not sure you can avoid having *some* amount of code in each iommu
driver, since the exact meaning of iommu-cells can be different.. but
I guess we could come up with a generic way of determining which
device(s) attached to an iommu might be active when the iommu driver
probes.

> So no, I'm very much against this approach and the solution absolutely needs
> to come in the form of a more abstract description from firmware.
>
> > I *think* (but need to confirm if Bjorn hasn't already) that the
> > memory for the pagetables that firmware/bootloader sets up is already
> > removed from the memory map efi passes to kernel, so we don't need to
> > worry about kernel stomping in-use pagetables.
>
> It's precisely this sort of fragility that makes me nervous about this whole
> approach.

See above about TTBRn==null.. although if there were actually
pagetables, they would need to be reserved memory, otherwise I guess
windows would also have a problem w/ stomping pagetables.

BR,
-R

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel