Master-aware devices and sideband ID data

Master-aware devices and sideband ID data

[Mark Rutland]

Hi all,

For some devices, identification of particular masters is critical to
their operation (e.g. IOMMUs, MSI controllers). The identity of masters
is determined from sideband signals on the bus, and it may or may not be
possible to probe and/or configure this information. Worse still, these
information may be rewritten by intermediate buses, so the
information for a given master may differ at different points in the bus
hierarchy.

We currently describe this master information for devices attached to
IOMMUs, with a master ID being encoded in the iommu-cells. However this
only covers the ID between the master and its IOMMU(s), and we don't
currently have a mechanism to describe the master IDs as they are seen
by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).

The following are example of master IDs:
* PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
* SMMU Stream IDs (SIDs)
* GICv3 ITS Device IDs (DIDs)

In the case of a system with PCIe, SMMU and GICv3, the master IDs are
rewritten in a chain of RID=>SID=>DID, as in the following diagram:

+-------------+
| PCIe master |
+-------------+
    ||
    || Requester ID (RID)
    || Probeable from RC.
    \/
+-------------------+
| PCIe Root Complex |
+-------------------+
    ||
    || SMMU Stream ID (SID)
    || Derived from RID, static non-probeable mapping.
    \/
+--------------+
| SMMU (IOMMU) |
+--------------+
    ||
    || ITS Device ID (DID)
    || Derived from SID, static non-probeable mapping.
    \/
+----------------------------+
| GICv3 ITS (MSI controller) |
+----------------------------+

In simpler cases, you might have a simpler set of master ID
translations, e.g. just a DID:

+-----------------+
| Platform device |
+-----------------+
    ||
    || ITS Device ID (DID)
    || Hard-wired on the bus.
    \/
+----------------------------+
| GICv3 ITS (MSI controller) |
+----------------------------+

Ignoring current bindings for the moment, I can see how we can describe
this with a generic master id-binding, with master-id-map along the
lines of interrupt-map, with a tuple format like:
<child-id-base child-id-length parent parent-id-base>

For the PCIe example, this would look something like the following (with
properties omitted for brevity):

PCI: pci@af000000 {
	...
	
	/* Requester ID of PCIe endpoints, implicit at runtime */
	master-id-cells = <1>;

	/* RIDS idmapped to SIDS @ SMMU */
	master-id-map = <0 0x10000 &SMMU 0>;
}

SMMU: iommu@bf000000 {
	...

	/* SID, derived from RID */
	master-id-cells = <1>;

	/* 
	 * For some reason the high bit of the ID was negated.
	 */
	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
	                <0x8000 0x8000 &ITS 0x0 0x0000>;

};

ITS: its@cf000000 {
	...

	/* DID, derived from SID */
	master-id-cells = <2>;

	/* 
	 * Master traffic not propagated beyond this point, so no
	 * master-id-ranges
	 */
};

For the simpler case, this would look something like:

DEV: device@af000000 {
	master-id-cells = <1>;
	master-ids = <0xf>, <0xb>;
	master-id-map = <0 0xf &ITS 0 0>;
};

ITS: its@cf000000 {
	...

	/* DID */
	master-id-cells = <2>;
};


However, this conflicts/overlaps with existing bindings (at least iommus
and msi-parent), and I'm not sure how to reconcile them. Am I missing a
neat way of describing this that works with the existing bindings?

It's also not clear to me if it's sufficient to have a generic
"master-ids" property (with the relevant type being implicit from
context), or if each type of ID needs each type of ID (RID, SID, DID,
etc) needs its own.

Which other devices out there which require side-band master
information, and what do they require?

Thanks,
Mark.

Master-aware devices and sideband ID data

[Mark Rutland]

Hi all,

For some devices, identification of particular masters is critical to
their operation (e.g. IOMMUs, MSI controllers). The identity of masters
is determined from sideband signals on the bus, and it may or may not be
possible to probe and/or configure this information. Worse still, these
information may be rewritten by intermediate buses, so the
information for a given master may differ at different points in the bus
hierarchy.

We currently describe this master information for devices attached to
IOMMUs, with a master ID being encoded in the iommu-cells. However this
only covers the ID between the master and its IOMMU(s), and we don't
currently have a mechanism to describe the master IDs as they are seen
by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).

The following are example of master IDs:
* PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
* SMMU Stream IDs (SIDs)
* GICv3 ITS Device IDs (DIDs)

In the case of a system with PCIe, SMMU and GICv3, the master IDs are
rewritten in a chain of RID=>SID=>DID, as in the following diagram:

+-------------+
| PCIe master |
+-------------+
    ||
    || Requester ID (RID)
    || Probeable from RC.
    \/
+-------------------+
| PCIe Root Complex |
+-------------------+
    ||
    || SMMU Stream ID (SID)
    || Derived from RID, static non-probeable mapping.
    \/
+--------------+
| SMMU (IOMMU) |
+--------------+
    ||
    || ITS Device ID (DID)
    || Derived from SID, static non-probeable mapping.
    \/
+----------------------------+
| GICv3 ITS (MSI controller) |
+----------------------------+

In simpler cases, you might have a simpler set of master ID
translations, e.g. just a DID:

+-----------------+
| Platform device |
+-----------------+
    ||
    || ITS Device ID (DID)
    || Hard-wired on the bus.
    \/
+----------------------------+
| GICv3 ITS (MSI controller) |
+----------------------------+

Ignoring current bindings for the moment, I can see how we can describe
this with a generic master id-binding, with master-id-map along the
lines of interrupt-map, with a tuple format like:
<child-id-base child-id-length parent parent-id-base>

For the PCIe example, this would look something like the following (with
properties omitted for brevity):

PCI: pci at af000000 {
	...
	
	/* Requester ID of PCIe endpoints, implicit at runtime */
	master-id-cells = <1>;

	/* RIDS idmapped to SIDS @ SMMU */
	master-id-map = <0 0x10000 &SMMU 0>;
}

SMMU: iommu at bf000000 {
	...

	/* SID, derived from RID */
	master-id-cells = <1>;

	/* 
	 * For some reason the high bit of the ID was negated.
	 */
	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
	                <0x8000 0x8000 &ITS 0x0 0x0000>;

};

ITS: its at cf000000 {
	...

	/* DID, derived from SID */
	master-id-cells = <2>;

	/* 
	 * Master traffic not propagated beyond this point, so no
	 * master-id-ranges
	 */
};

For the simpler case, this would look something like:

DEV: device at af000000 {
	master-id-cells = <1>;
	master-ids = <0xf>, <0xb>;
	master-id-map = <0 0xf &ITS 0 0>;
};

ITS: its at cf000000 {
	...

	/* DID */
	master-id-cells = <2>;
};


However, this conflicts/overlaps with existing bindings (at least iommus
and msi-parent), and I'm not sure how to reconcile them. Am I missing a
neat way of describing this that works with the existing bindings?

It's also not clear to me if it's sufficient to have a generic
"master-ids" property (with the relevant type being implicit from
context), or if each type of ID needs each type of ID (RID, SID, DID,
etc) needs its own.

Which other devices out there which require side-band master
information, and what do they require?

Thanks,
Mark.

Re: Master-aware devices and sideband ID data

[Stuart Yoder]

On Tue, Mar 24, 2015 at 10:50 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> Hi all,
>
> For some devices, identification of particular masters is critical to
> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> is determined from sideband signals on the bus, and it may or may not be
> possible to probe and/or configure this information. Worse still, these
> information may be rewritten by intermediate buses, so the
> information for a given master may differ at different points in the bus
> hierarchy.
>
> We currently describe this master information for devices attached to
> IOMMUs, with a master ID being encoded in the iommu-cells. However this
> only covers the ID between the master and its IOMMU(s), and we don't
> currently have a mechanism to describe the master IDs as they are seen
> by devices beyond the IOMMU(s),

Is there a specific case you've run across where conveying this additional
master info would be needed,  or are you just thinking through how to fully
describe hardware?

Are there really cases out there were there is a hardwired hardware
relationship between RID and SID?

> or in the absence of any IOMMU(s).
>
> The following are example of master IDs:
> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> * SMMU Stream IDs (SIDs)
> * GICv3 ITS Device IDs (DIDs)
>
> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
>
> +-------------+
> | PCIe master |
> +-------------+
>     ||
>     || Requester ID (RID)
>     || Probeable from RC.
>     \/
> +-------------------+
> | PCIe Root Complex |
> +-------------------+
>     ||
>     || SMMU Stream ID (SID)
>     || Derived from RID, static non-probeable mapping.

The FSL LS2085A SoC has an actual RID->SID mapping table in the PCI
controller, but it is not static in the sense of fixed in hardware or
firmware.  It's
a programmable mapping table, and we envision Linux programming this table.

>     \/
> +--------------+
> | SMMU (IOMMU) |
> +--------------+
>     ||
>     || ITS Device ID (DID)
>     || Derived from SID, static non-probeable mapping.
>     \/

Is this even architecturally possible on ARM?  Can the SMMU transform stream
IDs into some different number?

> +----------------------------+
> | GICv3 ITS (MSI controller) |
> +----------------------------+
>
> In simpler cases, you might have a simpler set of master ID
> translations, e.g. just a DID:
>
> +-----------------+
> | Platform device |
> +-----------------+
>     ||
>     || ITS Device ID (DID)
>     || Hard-wired on the bus.
>     \/
> +----------------------------+
> | GICv3 ITS (MSI controller) |
> +----------------------------+
>
> Ignoring current bindings for the moment, I can see how we can describe
> this with a generic master id-binding, with master-id-map along the
> lines of interrupt-map, with a tuple format like:
> <child-id-base child-id-length parent parent-id-base>
>
> For the PCIe example, this would look something like the following (with
> properties omitted for brevity):
>
> PCI: pci@af000000 {
>         ...
>
>         /* Requester ID of PCIe endpoints, implicit at runtime */
>         master-id-cells = <1>;
>
>         /* RIDS idmapped to SIDS @ SMMU */
>         master-id-map = <0 0x10000 &SMMU 0>;
> }
>
> SMMU: iommu@bf000000 {
>         ...
>
>         /* SID, derived from RID */
>         master-id-cells = <1>;
>
>         /*
>          * For some reason the high bit of the ID was negated.
>          */
>         master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
>                         <0x8000 0x8000 &ITS 0x0 0x0000>;
>
> };
>
> ITS: its@cf000000 {
>         ...
>
>         /* DID, derived from SID */
>         master-id-cells = <2>;
>
>         /*
>          * Master traffic not propagated beyond this point, so no
>          * master-id-ranges
>          */
> };
>
> For the simpler case, this would look something like:
>
> DEV: device@af000000 {
>         master-id-cells = <1>;
>         master-ids = <0xf>, <0xb>;
>         master-id-map = <0 0xf &ITS 0 0>;
> };
>
> ITS: its@cf000000 {
>         ...
>
>         /* DID */
>         master-id-cells = <2>;
> };
>
>
> However, this conflicts/overlaps with existing bindings (at least iommus
> and msi-parent), and I'm not sure how to reconcile them. Am I missing a
> neat way of describing this that works with the existing bindings?
>
> It's also not clear to me if it's sufficient to have a generic
> "master-ids" property (with the relevant type being implicit from
> context), or if each type of ID needs each type of ID (RID, SID, DID,
> etc) needs its own.
>
> Which other devices out there which require side-band master
> information, and what do they require?

For us, at least, the  master-id-map cannot be a static property.
Our PCI controller's DID->SID mapping table is programmable
and we won't know how to program it until Linux is running.  A PCI
SRIOV card may have virtual functions that appear as hot-plugged
devices and there is no way that boot firmware would have the knowledge
to have set that table up ahead of time.

What we do need is some way to have boot firmware convey to
the OS what SIDs are available to be used in the DID->SID
table.   We have a very limited number of SIDs available and boot
firmware may need to convey to Linux that for example-- PCI controller #2
has only 8 SIDs available for DID->SID mappings.

We had invented a property called "available-stream-ids" for some
proof of concept work, but perhaps the master-id-map property could
be used to convey available stream IDs by using a child-id-length
of 0.  So to convey that stream IDs 5,6,7,8 are available:

  PCI: pci@af000000 {
         ...
         /* Requester ID of PCIe endpoints, implicit at runtime */
         master-id-cells = <1>;

         /* RIDS idmapped to SIDS @ SMMU */
         master-id-map = <0 0 &SMMU 5
                                    0 0 &SMMU 6
                                    0 0 &SMMU 7
                                    0 0 &SMMU 8>;
  };

I'm not sure the above would conflict with the existing IOMMU
binding, but would seem to be an extension of it.

If you have a simple master with 2 stream IDS do this:          ...
          iommus = <&{/smmu} 23>, <&{/smmu} 24>;

If you have a static DID->SID mapping to convey, use the map as
you proposed:
         master-id-cells = <1>;
         master-id-map = <0 0x10000 &SMMU 0>;;

If your DID->SID is not static, but you need to convey the SIDs that are
available to this bus:
         master-id-cells = <1>;
         master-id-map = <0 0 &SMMU 23
                                    0 0 &SMMU 24>;

I am less sure about how to deal with expressing the relationship of masters to
the GIC ITS, but is there really a problem there?

Thanks,
Stuart

Master-aware devices and sideband ID data

[Stuart Yoder]

On Tue, Mar 24, 2015 at 10:50 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> Hi all,
>
> For some devices, identification of particular masters is critical to
> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> is determined from sideband signals on the bus, and it may or may not be
> possible to probe and/or configure this information. Worse still, these
> information may be rewritten by intermediate buses, so the
> information for a given master may differ at different points in the bus
> hierarchy.
>
> We currently describe this master information for devices attached to
> IOMMUs, with a master ID being encoded in the iommu-cells. However this
> only covers the ID between the master and its IOMMU(s), and we don't
> currently have a mechanism to describe the master IDs as they are seen
> by devices beyond the IOMMU(s),

Is there a specific case you've run across where conveying this additional
master info would be needed,  or are you just thinking through how to fully
describe hardware?

Are there really cases out there were there is a hardwired hardware
relationship between RID and SID?

> or in the absence of any IOMMU(s).
>
> The following are example of master IDs:
> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> * SMMU Stream IDs (SIDs)
> * GICv3 ITS Device IDs (DIDs)
>
> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
>
> +-------------+
> | PCIe master |
> +-------------+
>     ||
>     || Requester ID (RID)
>     || Probeable from RC.
>     \/
> +-------------------+
> | PCIe Root Complex |
> +-------------------+
>     ||
>     || SMMU Stream ID (SID)
>     || Derived from RID, static non-probeable mapping.

The FSL LS2085A SoC has an actual RID->SID mapping table in the PCI
controller, but it is not static in the sense of fixed in hardware or
firmware.  It's
a programmable mapping table, and we envision Linux programming this table.

>     \/
> +--------------+
> | SMMU (IOMMU) |
> +--------------+
>     ||
>     || ITS Device ID (DID)
>     || Derived from SID, static non-probeable mapping.
>     \/

Is this even architecturally possible on ARM?  Can the SMMU transform stream
IDs into some different number?

> +----------------------------+
> | GICv3 ITS (MSI controller) |
> +----------------------------+
>
> In simpler cases, you might have a simpler set of master ID
> translations, e.g. just a DID:
>
> +-----------------+
> | Platform device |
> +-----------------+
>     ||
>     || ITS Device ID (DID)
>     || Hard-wired on the bus.
>     \/
> +----------------------------+
> | GICv3 ITS (MSI controller) |
> +----------------------------+
>
> Ignoring current bindings for the moment, I can see how we can describe
> this with a generic master id-binding, with master-id-map along the
> lines of interrupt-map, with a tuple format like:
> <child-id-base child-id-length parent parent-id-base>
>
> For the PCIe example, this would look something like the following (with
> properties omitted for brevity):
>
> PCI: pci at af000000 {
>         ...
>
>         /* Requester ID of PCIe endpoints, implicit at runtime */
>         master-id-cells = <1>;
>
>         /* RIDS idmapped to SIDS @ SMMU */
>         master-id-map = <0 0x10000 &SMMU 0>;
> }
>
> SMMU: iommu at bf000000 {
>         ...
>
>         /* SID, derived from RID */
>         master-id-cells = <1>;
>
>         /*
>          * For some reason the high bit of the ID was negated.
>          */
>         master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
>                         <0x8000 0x8000 &ITS 0x0 0x0000>;
>
> };
>
> ITS: its at cf000000 {
>         ...
>
>         /* DID, derived from SID */
>         master-id-cells = <2>;
>
>         /*
>          * Master traffic not propagated beyond this point, so no
>          * master-id-ranges
>          */
> };
>
> For the simpler case, this would look something like:
>
> DEV: device at af000000 {
>         master-id-cells = <1>;
>         master-ids = <0xf>, <0xb>;
>         master-id-map = <0 0xf &ITS 0 0>;
> };
>
> ITS: its at cf000000 {
>         ...
>
>         /* DID */
>         master-id-cells = <2>;
> };
>
>
> However, this conflicts/overlaps with existing bindings (at least iommus
> and msi-parent), and I'm not sure how to reconcile them. Am I missing a
> neat way of describing this that works with the existing bindings?
>
> It's also not clear to me if it's sufficient to have a generic
> "master-ids" property (with the relevant type being implicit from
> context), or if each type of ID needs each type of ID (RID, SID, DID,
> etc) needs its own.
>
> Which other devices out there which require side-band master
> information, and what do they require?

For us, at least, the  master-id-map cannot be a static property.
Our PCI controller's DID->SID mapping table is programmable
and we won't know how to program it until Linux is running.  A PCI
SRIOV card may have virtual functions that appear as hot-plugged
devices and there is no way that boot firmware would have the knowledge
to have set that table up ahead of time.

What we do need is some way to have boot firmware convey to
the OS what SIDs are available to be used in the DID->SID
table.   We have a very limited number of SIDs available and boot
firmware may need to convey to Linux that for example-- PCI controller #2
has only 8 SIDs available for DID->SID mappings.

We had invented a property called "available-stream-ids" for some
proof of concept work, but perhaps the master-id-map property could
be used to convey available stream IDs by using a child-id-length
of 0.  So to convey that stream IDs 5,6,7,8 are available:

  PCI: pci at af000000 {
         ...
         /* Requester ID of PCIe endpoints, implicit at runtime */
         master-id-cells = <1>;

         /* RIDS idmapped to SIDS @ SMMU */
         master-id-map = <0 0 &SMMU 5
                                    0 0 &SMMU 6
                                    0 0 &SMMU 7
                                    0 0 &SMMU 8>;
  };

I'm not sure the above would conflict with the existing IOMMU
binding, but would seem to be an extension of it.

If you have a simple master with 2 stream IDS do this:          ...
          iommus = <&{/smmu} 23>, <&{/smmu} 24>;

If you have a static DID->SID mapping to convey, use the map as
you proposed:
         master-id-cells = <1>;
         master-id-map = <0 0x10000 &SMMU 0>;;

If your DID->SID is not static, but you need to convey the SIDs that are
available to this bus:
         master-id-cells = <1>;
         master-id-map = <0 0 &SMMU 23
                                    0 0 &SMMU 24>;

I am less sure about how to deal with expressing the relationship of masters to
the GIC ITS, but is there really a problem there?

Thanks,
Stuart

Re: Master-aware devices and sideband ID data

[Will Deacon]

Hi Stuart,

On Thu, May 07, 2015 at 06:49:32PM +0100, Stuart Yoder wrote:
> On Tue, Mar 24, 2015 at 10:50 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> > Hi all,
> >
> > For some devices, identification of particular masters is critical to
> > their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> > is determined from sideband signals on the bus, and it may or may not be
> > possible to probe and/or configure this information. Worse still, these
> > information may be rewritten by intermediate buses, so the
> > information for a given master may differ at different points in the bus
> > hierarchy.
> >
> > We currently describe this master information for devices attached to
> > IOMMUs, with a master ID being encoded in the iommu-cells. However this
> > only covers the ID between the master and its IOMMU(s), and we don't
> > currently have a mechanism to describe the master IDs as they are seen
> > by devices beyond the IOMMU(s),
> 
> Is there a specific case you've run across where conveying this additional
> master info would be needed,  or are you just thinking through how to fully
> describe hardware?

It's a combination of a few things:

  1. We are aware of hardware that has ID transformations
  2. ACPI allows this to be described in IORT
  3. We need to be general enough to encompass current and future designs
     in the device-tree binding.

> Are there really cases out there were there is a hardwired hardware
> relationship between RID and SID?

Yes, I believe it's actually the common case (at least, from Linux's
perspective).

> > or in the absence of any IOMMU(s).
> >
> > The following are example of master IDs:
> > * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> > * SMMU Stream IDs (SIDs)
> > * GICv3 ITS Device IDs (DIDs)
> >
> > In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> > rewritten in a chain of RID=>SID=>DID, as in the following diagram:
> >
> > +-------------+
> > | PCIe master |
> > +-------------+
> >     ||
> >     || Requester ID (RID)
> >     || Probeable from RC.
> >     \/
> > +-------------------+
> > | PCIe Root Complex |
> > +-------------------+
> >     ||
> >     || SMMU Stream ID (SID)
> >     || Derived from RID, static non-probeable mapping.
> 
> The FSL LS2085A SoC has an actual RID->SID mapping table in the PCI
> controller, but it is not static in the sense of fixed in hardware or
> firmware.  It's
> a programmable mapping table, and we envision Linux programming this table.

Ok, so I assume you're not planning to use ACPI with this system?

Also, we can't just program this table willy-nilly, as I'm sure you're
aware. For example, updating a the SMRs in a live SMMU is a big no-no,
so although Linux may initialise the table, it can only be safely changed
at device init/teardown time, surely?

> >     \/
> > +--------------+
> > | SMMU (IOMMU) |
> > +--------------+
> >     ||
> >     || ITS Device ID (DID)
> >     || Derived from SID, static non-probeable mapping.
> >     \/
> 
> Is this even architecturally possible on ARM?  Can the SMMU transform stream
> IDs into some different number?

>From the OS perspective, this is what it looks like. What we'd actually
have in hardware is multiple SMMUs sharing an ITS, so there would need
to be something prepending an SMMU ID onto the StreamID to construct a
device ID, otherwise you'd have devices inexplicably sharing MSIs.

> > However, this conflicts/overlaps with existing bindings (at least iommus
> > and msi-parent), and I'm not sure how to reconcile them. Am I missing a
> > neat way of describing this that works with the existing bindings?
> >
> > It's also not clear to me if it's sufficient to have a generic
> > "master-ids" property (with the relevant type being implicit from
> > context), or if each type of ID needs each type of ID (RID, SID, DID,
> > etc) needs its own.
> >
> > Which other devices out there which require side-band master
> > information, and what do they require?
> 
> For us, at least, the  master-id-map cannot be a static property.
> Our PCI controller's DID->SID mapping table is programmable
> and we won't know how to program it until Linux is running.  A PCI
> SRIOV card may have virtual functions that appear as hot-plugged
> devices and there is no way that boot firmware would have the knowledge
> to have set that table up ahead of time.
> 
> What we do need is some way to have boot firmware convey to
> the OS what SIDs are available to be used in the DID->SID
> table.   We have a very limited number of SIDs available and boot
> firmware may need to convey to Linux that for example-- PCI controller #2
> has only 8 SIDs available for DID->SID mappings.
> 
> We had invented a property called "available-stream-ids" for some
> proof of concept work, but perhaps the master-id-map property could
> be used to convey available stream IDs by using a child-id-length
> of 0.  So to convey that stream IDs 5,6,7,8 are available:
> 
>   PCI: pci@af000000 {
>          ...
>          /* Requester ID of PCIe endpoints, implicit at runtime */
>          master-id-cells = <1>;
> 
>          /* RIDS idmapped to SIDS @ SMMU */
>          master-id-map = <0 0 &SMMU 5
>                                     0 0 &SMMU 6
>                                     0 0 &SMMU 7
>                                     0 0 &SMMU 8>;
>   };
> 
> I'm not sure the above would conflict with the existing IOMMU
> binding, but would seem to be an extension of it.
> 
> If you have a simple master with 2 stream IDS do this:          ...
>           iommus = <&{/smmu} 23>, <&{/smmu} 24>;
> 
> If you have a static DID->SID mapping to convey, use the map as
> you proposed:
>          master-id-cells = <1>;
>          master-id-map = <0 0x10000 &SMMU 0>;;
> 
> If your DID->SID is not static, but you need to convey the SIDs that are
> available to this bus:
>          master-id-cells = <1>;
>          master-id-map = <0 0 &SMMU 23
>                                     0 0 &SMMU 24>;

We need to be careful with terminology here. Can we use RID to mean the PCI
requester ID and DID to mean the GIC ITS device ID?

> I am less sure about how to deal with expressing the relationship of masters to
> the GIC ITS, but is there really a problem there?

Well we need to know the DID at the ITS for an arbitrary device, which is
constructed by performing the RID->SID and SID->DID transformations.

Mark probably has more feedback on the binding side of things.

Will
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Will Deacon]

Hi Stuart,

On Thu, May 07, 2015 at 06:49:32PM +0100, Stuart Yoder wrote:
> On Tue, Mar 24, 2015 at 10:50 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> > Hi all,
> >
> > For some devices, identification of particular masters is critical to
> > their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> > is determined from sideband signals on the bus, and it may or may not be
> > possible to probe and/or configure this information. Worse still, these
> > information may be rewritten by intermediate buses, so the
> > information for a given master may differ at different points in the bus
> > hierarchy.
> >
> > We currently describe this master information for devices attached to
> > IOMMUs, with a master ID being encoded in the iommu-cells. However this
> > only covers the ID between the master and its IOMMU(s), and we don't
> > currently have a mechanism to describe the master IDs as they are seen
> > by devices beyond the IOMMU(s),
> 
> Is there a specific case you've run across where conveying this additional
> master info would be needed,  or are you just thinking through how to fully
> describe hardware?

It's a combination of a few things:

  1. We are aware of hardware that has ID transformations
  2. ACPI allows this to be described in IORT
  3. We need to be general enough to encompass current and future designs
     in the device-tree binding.

> Are there really cases out there were there is a hardwired hardware
> relationship between RID and SID?

Yes, I believe it's actually the common case (at least, from Linux's
perspective).

> > or in the absence of any IOMMU(s).
> >
> > The following are example of master IDs:
> > * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> > * SMMU Stream IDs (SIDs)
> > * GICv3 ITS Device IDs (DIDs)
> >
> > In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> > rewritten in a chain of RID=>SID=>DID, as in the following diagram:
> >
> > +-------------+
> > | PCIe master |
> > +-------------+
> >     ||
> >     || Requester ID (RID)
> >     || Probeable from RC.
> >     \/
> > +-------------------+
> > | PCIe Root Complex |
> > +-------------------+
> >     ||
> >     || SMMU Stream ID (SID)
> >     || Derived from RID, static non-probeable mapping.
> 
> The FSL LS2085A SoC has an actual RID->SID mapping table in the PCI
> controller, but it is not static in the sense of fixed in hardware or
> firmware.  It's
> a programmable mapping table, and we envision Linux programming this table.

Ok, so I assume you're not planning to use ACPI with this system?

Also, we can't just program this table willy-nilly, as I'm sure you're
aware. For example, updating a the SMRs in a live SMMU is a big no-no,
so although Linux may initialise the table, it can only be safely changed
at device init/teardown time, surely?

> >     \/
> > +--------------+
> > | SMMU (IOMMU) |
> > +--------------+
> >     ||
> >     || ITS Device ID (DID)
> >     || Derived from SID, static non-probeable mapping.
> >     \/
> 
> Is this even architecturally possible on ARM?  Can the SMMU transform stream
> IDs into some different number?

>From the OS perspective, this is what it looks like. What we'd actually
have in hardware is multiple SMMUs sharing an ITS, so there would need
to be something prepending an SMMU ID onto the StreamID to construct a
device ID, otherwise you'd have devices inexplicably sharing MSIs.

> > However, this conflicts/overlaps with existing bindings (at least iommus
> > and msi-parent), and I'm not sure how to reconcile them. Am I missing a
> > neat way of describing this that works with the existing bindings?
> >
> > It's also not clear to me if it's sufficient to have a generic
> > "master-ids" property (with the relevant type being implicit from
> > context), or if each type of ID needs each type of ID (RID, SID, DID,
> > etc) needs its own.
> >
> > Which other devices out there which require side-band master
> > information, and what do they require?
> 
> For us, at least, the  master-id-map cannot be a static property.
> Our PCI controller's DID->SID mapping table is programmable
> and we won't know how to program it until Linux is running.  A PCI
> SRIOV card may have virtual functions that appear as hot-plugged
> devices and there is no way that boot firmware would have the knowledge
> to have set that table up ahead of time.
> 
> What we do need is some way to have boot firmware convey to
> the OS what SIDs are available to be used in the DID->SID
> table.   We have a very limited number of SIDs available and boot
> firmware may need to convey to Linux that for example-- PCI controller #2
> has only 8 SIDs available for DID->SID mappings.
> 
> We had invented a property called "available-stream-ids" for some
> proof of concept work, but perhaps the master-id-map property could
> be used to convey available stream IDs by using a child-id-length
> of 0.  So to convey that stream IDs 5,6,7,8 are available:
> 
>   PCI: pci at af000000 {
>          ...
>          /* Requester ID of PCIe endpoints, implicit at runtime */
>          master-id-cells = <1>;
> 
>          /* RIDS idmapped to SIDS @ SMMU */
>          master-id-map = <0 0 &SMMU 5
>                                     0 0 &SMMU 6
>                                     0 0 &SMMU 7
>                                     0 0 &SMMU 8>;
>   };
> 
> I'm not sure the above would conflict with the existing IOMMU
> binding, but would seem to be an extension of it.
> 
> If you have a simple master with 2 stream IDS do this:          ...
>           iommus = <&{/smmu} 23>, <&{/smmu} 24>;
> 
> If you have a static DID->SID mapping to convey, use the map as
> you proposed:
>          master-id-cells = <1>;
>          master-id-map = <0 0x10000 &SMMU 0>;;
> 
> If your DID->SID is not static, but you need to convey the SIDs that are
> available to this bus:
>          master-id-cells = <1>;
>          master-id-map = <0 0 &SMMU 23
>                                     0 0 &SMMU 24>;

We need to be careful with terminology here. Can we use RID to mean the PCI
requester ID and DID to mean the GIC ITS device ID?

> I am less sure about how to deal with expressing the relationship of masters to
> the GIC ITS, but is there really a problem there?

Well we need to know the DID at the ITS for an arbitrary device, which is
constructed by performing the RID->SID and SID->DID transformations.

Mark probably has more feedback on the binding side of things.

Will

Re: Master-aware devices and sideband ID data

[Stuart Yoder]

On Fri, May 8, 2015 at 10:49 AM, Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org> wrote:
> Hi Stuart,
>
> On Thu, May 07, 2015 at 06:49:32PM +0100, Stuart Yoder wrote:
>> On Tue, Mar 24, 2015 at 10:50 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
>> > Hi all,
>> >
>> > For some devices, identification of particular masters is critical to
>> > their operation (e.g. IOMMUs, MSI controllers). The identity of masters
>> > is determined from sideband signals on the bus, and it may or may not be
>> > possible to probe and/or configure this information. Worse still, these
>> > information may be rewritten by intermediate buses, so the
>> > information for a given master may differ at different points in the bus
>> > hierarchy.
>> >
>> > We currently describe this master information for devices attached to
>> > IOMMUs, with a master ID being encoded in the iommu-cells. However this
>> > only covers the ID between the master and its IOMMU(s), and we don't
>> > currently have a mechanism to describe the master IDs as they are seen
>> > by devices beyond the IOMMU(s),
>>
>> Is there a specific case you've run across where conveying this additional
>> master info would be needed,  or are you just thinking through how to fully
>> describe hardware?
>
> It's a combination of a few things:
>
>   1. We are aware of hardware that has ID transformations
>   2. ACPI allows this to be described in IORT
>   3. We need to be general enough to encompass current and future designs
>      in the device-tree binding.
>
>> Are there really cases out there were there is a hardwired hardware
>> relationship between RID and SID?
>
> Yes, I believe it's actually the common case (at least, from Linux's
> perspective).
>
>> > or in the absence of any IOMMU(s).
>> >
>> > The following are example of master IDs:
>> > * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
>> > * SMMU Stream IDs (SIDs)
>> > * GICv3 ITS Device IDs (DIDs)
>> >
>> > In the case of a system with PCIe, SMMU and GICv3, the master IDs are
>> > rewritten in a chain of RID=>SID=>DID, as in the following diagram:
>> >
>> > +-------------+
>> > | PCIe master |
>> > +-------------+
>> >     ||
>> >     || Requester ID (RID)
>> >     || Probeable from RC.
>> >     \/
>> > +-------------------+
>> > | PCIe Root Complex |
>> > +-------------------+
>> >     ||
>> >     || SMMU Stream ID (SID)
>> >     || Derived from RID, static non-probeable mapping.
>>
>> The FSL LS2085A SoC has an actual RID->SID mapping table in the PCI
>> controller, but it is not static in the sense of fixed in hardware or
>> firmware.  It's
>> a programmable mapping table, and we envision Linux programming this table.
>
> Ok, so I assume you're not planning to use ACPI with this system?

Not initially, but perhaps in the future.  I guess I don't know the implications
to ACPI yet.

> Also, we can't just program this table willy-nilly, as I'm sure you're
> aware. For example, updating a the SMRs in a live SMMU is a big no-no,
> so although Linux may initialise the table, it can only be safely changed
> at device init/teardown time, surely?

Correct.  That would not be updated when a device is live.

>> >     \/
>> > +--------------+
>> > | SMMU (IOMMU) |
>> > +--------------+
>> >     ||
>> >     || ITS Device ID (DID)
>> >     || Derived from SID, static non-probeable mapping.
>> >     \/
>>
>> Is this even architecturally possible on ARM?  Can the SMMU transform stream
>> IDs into some different number?
>
> From the OS perspective, this is what it looks like. What we'd actually
> have in hardware is multiple SMMUs sharing an ITS, so there would need
> to be something prepending an SMMU ID onto the StreamID to construct a
> device ID, otherwise you'd have devices inexplicably sharing MSIs.
>
>> > However, this conflicts/overlaps with existing bindings (at least iommus
>> > and msi-parent), and I'm not sure how to reconcile them. Am I missing a
>> > neat way of describing this that works with the existing bindings?
>> >
>> > It's also not clear to me if it's sufficient to have a generic
>> > "master-ids" property (with the relevant type being implicit from
>> > context), or if each type of ID needs each type of ID (RID, SID, DID,
>> > etc) needs its own.
>> >
>> > Which other devices out there which require side-band master
>> > information, and what do they require?
>>
>> For us, at least, the  master-id-map cannot be a static property.
>> Our PCI controller's DID->SID mapping table is programmable
>> and we won't know how to program it until Linux is running.  A PCI
>> SRIOV card may have virtual functions that appear as hot-plugged
>> devices and there is no way that boot firmware would have the knowledge
>> to have set that table up ahead of time.
>>
>> What we do need is some way to have boot firmware convey to
>> the OS what SIDs are available to be used in the DID->SID
>> table.   We have a very limited number of SIDs available and boot
>> firmware may need to convey to Linux that for example-- PCI controller #2
>> has only 8 SIDs available for DID->SID mappings.
>>
>> We had invented a property called "available-stream-ids" for some
>> proof of concept work, but perhaps the master-id-map property could
>> be used to convey available stream IDs by using a child-id-length
>> of 0.  So to convey that stream IDs 5,6,7,8 are available:
>>
>>   PCI: pci@af000000 {
>>          ...
>>          /* Requester ID of PCIe endpoints, implicit at runtime */
>>          master-id-cells = <1>;
>>
>>          /* RIDS idmapped to SIDS @ SMMU */
>>          master-id-map = <0 0 &SMMU 5
>>                                     0 0 &SMMU 6
>>                                     0 0 &SMMU 7
>>                                     0 0 &SMMU 8>;
>>   };
>>
>> I'm not sure the above would conflict with the existing IOMMU
>> binding, but would seem to be an extension of it.
>>
>> If you have a simple master with 2 stream IDS do this:          ...
>>           iommus = <&{/smmu} 23>, <&{/smmu} 24>;
>>
>> If you have a static DID->SID mapping to convey, use the map as
>> you proposed:
>>          master-id-cells = <1>;
>>          master-id-map = <0 0x10000 &SMMU 0>;;
>>
>> If your DID->SID is not static, but you need to convey the SIDs that are
>> available to this bus:
>>          master-id-cells = <1>;
>>          master-id-map = <0 0 &SMMU 23
>>                                     0 0 &SMMU 24>;
>
> We need to be careful with terminology here. Can we use RID to mean the PCI
> requester ID and DID to mean the GIC ITS device ID?

Sorry, that was a typo on my part...where I said DID I meant RID.

Thanks,
Stuart
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Stuart Yoder]

On Fri, May 8, 2015 at 10:49 AM, Will Deacon <will.deacon@arm.com> wrote:
> Hi Stuart,
>
> On Thu, May 07, 2015 at 06:49:32PM +0100, Stuart Yoder wrote:
>> On Tue, Mar 24, 2015 at 10:50 AM, Mark Rutland <mark.rutland@arm.com> wrote:
>> > Hi all,
>> >
>> > For some devices, identification of particular masters is critical to
>> > their operation (e.g. IOMMUs, MSI controllers). The identity of masters
>> > is determined from sideband signals on the bus, and it may or may not be
>> > possible to probe and/or configure this information. Worse still, these
>> > information may be rewritten by intermediate buses, so the
>> > information for a given master may differ at different points in the bus
>> > hierarchy.
>> >
>> > We currently describe this master information for devices attached to
>> > IOMMUs, with a master ID being encoded in the iommu-cells. However this
>> > only covers the ID between the master and its IOMMU(s), and we don't
>> > currently have a mechanism to describe the master IDs as they are seen
>> > by devices beyond the IOMMU(s),
>>
>> Is there a specific case you've run across where conveying this additional
>> master info would be needed,  or are you just thinking through how to fully
>> describe hardware?
>
> It's a combination of a few things:
>
>   1. We are aware of hardware that has ID transformations
>   2. ACPI allows this to be described in IORT
>   3. We need to be general enough to encompass current and future designs
>      in the device-tree binding.
>
>> Are there really cases out there were there is a hardwired hardware
>> relationship between RID and SID?
>
> Yes, I believe it's actually the common case (at least, from Linux's
> perspective).
>
>> > or in the absence of any IOMMU(s).
>> >
>> > The following are example of master IDs:
>> > * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
>> > * SMMU Stream IDs (SIDs)
>> > * GICv3 ITS Device IDs (DIDs)
>> >
>> > In the case of a system with PCIe, SMMU and GICv3, the master IDs are
>> > rewritten in a chain of RID=>SID=>DID, as in the following diagram:
>> >
>> > +-------------+
>> > | PCIe master |
>> > +-------------+
>> >     ||
>> >     || Requester ID (RID)
>> >     || Probeable from RC.
>> >     \/
>> > +-------------------+
>> > | PCIe Root Complex |
>> > +-------------------+
>> >     ||
>> >     || SMMU Stream ID (SID)
>> >     || Derived from RID, static non-probeable mapping.
>>
>> The FSL LS2085A SoC has an actual RID->SID mapping table in the PCI
>> controller, but it is not static in the sense of fixed in hardware or
>> firmware.  It's
>> a programmable mapping table, and we envision Linux programming this table.
>
> Ok, so I assume you're not planning to use ACPI with this system?

Not initially, but perhaps in the future.  I guess I don't know the implications
to ACPI yet.

> Also, we can't just program this table willy-nilly, as I'm sure you're
> aware. For example, updating a the SMRs in a live SMMU is a big no-no,
> so although Linux may initialise the table, it can only be safely changed
> at device init/teardown time, surely?

Correct.  That would not be updated when a device is live.

>> >     \/
>> > +--------------+
>> > | SMMU (IOMMU) |
>> > +--------------+
>> >     ||
>> >     || ITS Device ID (DID)
>> >     || Derived from SID, static non-probeable mapping.
>> >     \/
>>
>> Is this even architecturally possible on ARM?  Can the SMMU transform stream
>> IDs into some different number?
>
> From the OS perspective, this is what it looks like. What we'd actually
> have in hardware is multiple SMMUs sharing an ITS, so there would need
> to be something prepending an SMMU ID onto the StreamID to construct a
> device ID, otherwise you'd have devices inexplicably sharing MSIs.
>
>> > However, this conflicts/overlaps with existing bindings (at least iommus
>> > and msi-parent), and I'm not sure how to reconcile them. Am I missing a
>> > neat way of describing this that works with the existing bindings?
>> >
>> > It's also not clear to me if it's sufficient to have a generic
>> > "master-ids" property (with the relevant type being implicit from
>> > context), or if each type of ID needs each type of ID (RID, SID, DID,
>> > etc) needs its own.
>> >
>> > Which other devices out there which require side-band master
>> > information, and what do they require?
>>
>> For us, at least, the  master-id-map cannot be a static property.
>> Our PCI controller's DID->SID mapping table is programmable
>> and we won't know how to program it until Linux is running.  A PCI
>> SRIOV card may have virtual functions that appear as hot-plugged
>> devices and there is no way that boot firmware would have the knowledge
>> to have set that table up ahead of time.
>>
>> What we do need is some way to have boot firmware convey to
>> the OS what SIDs are available to be used in the DID->SID
>> table.   We have a very limited number of SIDs available and boot
>> firmware may need to convey to Linux that for example-- PCI controller #2
>> has only 8 SIDs available for DID->SID mappings.
>>
>> We had invented a property called "available-stream-ids" for some
>> proof of concept work, but perhaps the master-id-map property could
>> be used to convey available stream IDs by using a child-id-length
>> of 0.  So to convey that stream IDs 5,6,7,8 are available:
>>
>>   PCI: pci at af000000 {
>>          ...
>>          /* Requester ID of PCIe endpoints, implicit at runtime */
>>          master-id-cells = <1>;
>>
>>          /* RIDS idmapped to SIDS @ SMMU */
>>          master-id-map = <0 0 &SMMU 5
>>                                     0 0 &SMMU 6
>>                                     0 0 &SMMU 7
>>                                     0 0 &SMMU 8>;
>>   };
>>
>> I'm not sure the above would conflict with the existing IOMMU
>> binding, but would seem to be an extension of it.
>>
>> If you have a simple master with 2 stream IDS do this:          ...
>>           iommus = <&{/smmu} 23>, <&{/smmu} 24>;
>>
>> If you have a static DID->SID mapping to convey, use the map as
>> you proposed:
>>          master-id-cells = <1>;
>>          master-id-map = <0 0x10000 &SMMU 0>;;
>>
>> If your DID->SID is not static, but you need to convey the SIDs that are
>> available to this bus:
>>          master-id-cells = <1>;
>>          master-id-map = <0 0 &SMMU 23
>>                                     0 0 &SMMU 24>;
>
> We need to be careful with terminology here. Can we use RID to mean the PCI
> requester ID and DID to mean the GIC ITS device ID?

Sorry, that was a typo on my part...where I said DID I meant RID.

Thanks,
Stuart

Re: Master-aware devices and sideband ID data

[Will Deacon]

On Fri, May 08, 2015 at 08:30:00PM +0100, Stuart Yoder wrote:
> On Fri, May 8, 2015 at 10:49 AM, Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org> wrote:
> > On Thu, May 07, 2015 at 06:49:32PM +0100, Stuart Yoder wrote:
> >> The FSL LS2085A SoC has an actual RID->SID mapping table in the PCI
> >> controller, but it is not static in the sense of fixed in hardware or
> >> firmware.  It's
> >> a programmable mapping table, and we envision Linux programming this table.
> >
> > Ok, so I assume you're not planning to use ACPI with this system?
> 
> Not initially, but perhaps in the future.  I guess I don't know the implications
> to ACPI yet.

Well, I suggest you take a look at the IORT spec[1] asap, because I don't think
it can describe your system if you want to go in the direction of dynamic
mappings.

> > Also, we can't just program this table willy-nilly, as I'm sure you're
> > aware. For example, updating a the SMRs in a live SMMU is a big no-no,
> > so although Linux may initialise the table, it can only be safely changed
> > at device init/teardown time, surely?
> 
> Correct.  That would not be updated when a device is live.

Ok, so the best way for Linux is probably to abstract this in the bus code
and have that allocate the IDs when a device is `hotplugged' on. This would
also fit well with consolidating the group creation for platform and PCI
devices.

Will

[1] http://infocenter.arm.com/help/topic/com.arm.doc.den0049a/DEN0049A_IO_Remapping_Table.pdf
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Will Deacon]

On Fri, May 08, 2015 at 08:30:00PM +0100, Stuart Yoder wrote:
> On Fri, May 8, 2015 at 10:49 AM, Will Deacon <will.deacon@arm.com> wrote:
> > On Thu, May 07, 2015 at 06:49:32PM +0100, Stuart Yoder wrote:
> >> The FSL LS2085A SoC has an actual RID->SID mapping table in the PCI
> >> controller, but it is not static in the sense of fixed in hardware or
> >> firmware.  It's
> >> a programmable mapping table, and we envision Linux programming this table.
> >
> > Ok, so I assume you're not planning to use ACPI with this system?
> 
> Not initially, but perhaps in the future.  I guess I don't know the implications
> to ACPI yet.

Well, I suggest you take a look at the IORT spec[1] asap, because I don't think
it can describe your system if you want to go in the direction of dynamic
mappings.

> > Also, we can't just program this table willy-nilly, as I'm sure you're
> > aware. For example, updating a the SMRs in a live SMMU is a big no-no,
> > so although Linux may initialise the table, it can only be safely changed
> > at device init/teardown time, surely?
> 
> Correct.  That would not be updated when a device is live.

Ok, so the best way for Linux is probably to abstract this in the bus code
and have that allocate the IDs when a device is `hotplugged' on. This would
also fit well with consolidating the group creation for platform and PCI
devices.

Will

[1] http://infocenter.arm.com/help/topic/com.arm.doc.den0049a/DEN0049A_IO_Remapping_Table.pdf

Re: Master-aware devices and sideband ID data

[Chalamarla, Tirumalesh]

This is some thing we also like to see in ITS and SMMU drivers. 
> On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> 
> Hi all,
> 
> For some devices, identification of particular masters is critical to
> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> is determined from sideband signals on the bus, and it may or may not be
> possible to probe and/or configure this information. Worse still, these
> information may be rewritten by intermediate buses, so the
> information for a given master may differ at different points in the bus
> hierarchy.
> 
> We currently describe this master information for devices attached to
> IOMMUs, with a master ID being encoded in the iommu-cells. However this
> only covers the ID between the master and its IOMMU(s), and we don't
> currently have a mechanism to describe the master IDs as they are seen
> by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
> 
> The following are example of master IDs:
> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> * SMMU Stream IDs (SIDs)
> * GICv3 ITS Device IDs (DIDs)
> 
> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
> 
> +-------------+
> | PCIe master |
> +-------------+
>    ||
>    || Requester ID (RID)
>    || Probeable from RC.
>    \/
> +-------------------+
> | PCIe Root Complex |
> +-------------------+
>    ||
>    || SMMU Stream ID (SID)
>    || Derived from RID, static non-probeable mapping.
>    \/
> +--------------+
> | SMMU (IOMMU) |
> +--------------+
>    ||
>    || ITS Device ID (DID)
>    || Derived from SID, static non-probeable mapping.
>    \/
> +----------------------------+
> | GICv3 ITS (MSI controller) |
> +----------------------------+
> 
> In simpler cases, you might have a simpler set of master ID
> translations, e.g. just a DID:
> 
> +-----------------+
> | Platform device |
> +-----------------+
>    ||
>    || ITS Device ID (DID)
>    || Hard-wired on the bus.
>    \/
> +----------------------------+
> | GICv3 ITS (MSI controller) |
> +----------------------------+
> 
> Ignoring current bindings for the moment, I can see how we can describe
> this with a generic master id-binding, with master-id-map along the
> lines of interrupt-map, with a tuple format like:
> <child-id-base child-id-length parent parent-id-base>
> 
> For the PCIe example, this would look something like the following (with
> properties omitted for brevity):
> 
> PCI: pci@af000000 {
> 	...
> 	
> 	/* Requester ID of PCIe endpoints, implicit at runtime */
> 	master-id-cells = <1>;
> 
> 	/* RIDS idmapped to SIDS @ SMMU */
> 	master-id-map = <0 0x10000 &SMMU 0>;
> }
> 
> SMMU: iommu@bf000000 {
> 	...
> 
> 	/* SID, derived from RID */
> 	master-id-cells = <1>;
> 
> 	/* 
> 	 * For some reason the high bit of the ID was negated.
> 	 */
> 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
> 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
> 
> };
> 
> ITS: its@cf000000 {
> 	...
> 
> 	/* DID, derived from SID */
> 	master-id-cells = <2>;
> 
> 	/* 
> 	 * Master traffic not propagated beyond this point, so no
> 	 * master-id-ranges
> 	 */
> };

I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  

> 
> For the simpler case, this would look something like:
> 
> DEV: device@af000000 {
> 	master-id-cells = <1>;
> 	master-ids = <0xf>, <0xb>;
> 	master-id-map = <0 0xf &ITS 0 0>;
> };
> 
> ITS: its@cf000000 {
> 	...
> 
> 	/* DID */
> 	master-id-cells = <2>;
> };
> 
Is this is not depending heavily on discover order, how do drivers know which device to get which ID. is it implicitly assumed in discovery order? 
what happens to hot pluggable devices.  

Thanks,
Tirumalesh. 

> 
> However, this conflicts/overlaps with existing bindings (at least iommus
> and msi-parent), and I'm not sure how to reconcile them. Am I missing a
> neat way of describing this that works with the existing bindings?
> 
> It's also not clear to me if it's sufficient to have a generic
> "master-ids" property (with the relevant type being implicit from
> context), or if each type of ID needs each type of ID (RID, SID, DID,
> etc) needs its own.
> 
> Which other devices out there which require side-band master
> information, and what do they require?
> 
> Thanks,
> Mark.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Chalamarla, Tirumalesh]

This is some thing we also like to see in ITS and SMMU drivers. 
> On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> 
> Hi all,
> 
> For some devices, identification of particular masters is critical to
> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> is determined from sideband signals on the bus, and it may or may not be
> possible to probe and/or configure this information. Worse still, these
> information may be rewritten by intermediate buses, so the
> information for a given master may differ at different points in the bus
> hierarchy.
> 
> We currently describe this master information for devices attached to
> IOMMUs, with a master ID being encoded in the iommu-cells. However this
> only covers the ID between the master and its IOMMU(s), and we don't
> currently have a mechanism to describe the master IDs as they are seen
> by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
> 
> The following are example of master IDs:
> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> * SMMU Stream IDs (SIDs)
> * GICv3 ITS Device IDs (DIDs)
> 
> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
> 
> +-------------+
> | PCIe master |
> +-------------+
>    ||
>    || Requester ID (RID)
>    || Probeable from RC.
>    \/
> +-------------------+
> | PCIe Root Complex |
> +-------------------+
>    ||
>    || SMMU Stream ID (SID)
>    || Derived from RID, static non-probeable mapping.
>    \/
> +--------------+
> | SMMU (IOMMU) |
> +--------------+
>    ||
>    || ITS Device ID (DID)
>    || Derived from SID, static non-probeable mapping.
>    \/
> +----------------------------+
> | GICv3 ITS (MSI controller) |
> +----------------------------+
> 
> In simpler cases, you might have a simpler set of master ID
> translations, e.g. just a DID:
> 
> +-----------------+
> | Platform device |
> +-----------------+
>    ||
>    || ITS Device ID (DID)
>    || Hard-wired on the bus.
>    \/
> +----------------------------+
> | GICv3 ITS (MSI controller) |
> +----------------------------+
> 
> Ignoring current bindings for the moment, I can see how we can describe
> this with a generic master id-binding, with master-id-map along the
> lines of interrupt-map, with a tuple format like:
> <child-id-base child-id-length parent parent-id-base>
> 
> For the PCIe example, this would look something like the following (with
> properties omitted for brevity):
> 
> PCI: pci at af000000 {
> 	...
> 	
> 	/* Requester ID of PCIe endpoints, implicit at runtime */
> 	master-id-cells = <1>;
> 
> 	/* RIDS idmapped to SIDS @ SMMU */
> 	master-id-map = <0 0x10000 &SMMU 0>;
> }
> 
> SMMU: iommu at bf000000 {
> 	...
> 
> 	/* SID, derived from RID */
> 	master-id-cells = <1>;
> 
> 	/* 
> 	 * For some reason the high bit of the ID was negated.
> 	 */
> 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
> 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
> 
> };
> 
> ITS: its at cf000000 {
> 	...
> 
> 	/* DID, derived from SID */
> 	master-id-cells = <2>;
> 
> 	/* 
> 	 * Master traffic not propagated beyond this point, so no
> 	 * master-id-ranges
> 	 */
> };

I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  

> 
> For the simpler case, this would look something like:
> 
> DEV: device at af000000 {
> 	master-id-cells = <1>;
> 	master-ids = <0xf>, <0xb>;
> 	master-id-map = <0 0xf &ITS 0 0>;
> };
> 
> ITS: its at cf000000 {
> 	...
> 
> 	/* DID */
> 	master-id-cells = <2>;
> };
> 
Is this is not depending heavily on discover order, how do drivers know which device to get which ID. is it implicitly assumed in discovery order? 
what happens to hot pluggable devices.  

Thanks,
Tirumalesh. 

> 
> However, this conflicts/overlaps with existing bindings (at least iommus
> and msi-parent), and I'm not sure how to reconcile them. Am I missing a
> neat way of describing this that works with the existing bindings?
> 
> It's also not clear to me if it's sufficient to have a generic
> "master-ids" property (with the relevant type being implicit from
> context), or if each type of ID needs each type of ID (RID, SID, DID,
> etc) needs its own.
> 
> Which other devices out there which require side-band master
> information, and what do they require?
> 
> Thanks,
> Mark.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Master-aware devices and sideband ID data

[Mark Rutland]

On Tue, May 26, 2015 at 11:20:59PM +0100, Chalamarla, Tirumalesh wrote:
> This is some thing we also like to see in ITS and SMMU drivers. 
> > On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> > 
> > Hi all,
> > 
> > For some devices, identification of particular masters is critical to
> > their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> > is determined from sideband signals on the bus, and it may or may not be
> > possible to probe and/or configure this information. Worse still, these
> > information may be rewritten by intermediate buses, so the
> > information for a given master may differ at different points in the bus
> > hierarchy.
> > 
> > We currently describe this master information for devices attached to
> > IOMMUs, with a master ID being encoded in the iommu-cells. However this
> > only covers the ID between the master and its IOMMU(s), and we don't
> > currently have a mechanism to describe the master IDs as they are seen
> > by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
> > 
> > The following are example of master IDs:
> > * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> > * SMMU Stream IDs (SIDs)
> > * GICv3 ITS Device IDs (DIDs)
> > 
> > In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> > rewritten in a chain of RID=>SID=>DID, as in the following diagram:
> > 
> > +-------------+
> > | PCIe master |
> > +-------------+
> >    ||
> >    || Requester ID (RID)
> >    || Probeable from RC.
> >    \/
> > +-------------------+
> > | PCIe Root Complex |
> > +-------------------+
> >    ||
> >    || SMMU Stream ID (SID)
> >    || Derived from RID, static non-probeable mapping.
> >    \/
> > +--------------+
> > | SMMU (IOMMU) |
> > +--------------+
> >    ||
> >    || ITS Device ID (DID)
> >    || Derived from SID, static non-probeable mapping.
> >    \/
> > +----------------------------+
> > | GICv3 ITS (MSI controller) |
> > +----------------------------+
> > 
> > In simpler cases, you might have a simpler set of master ID
> > translations, e.g. just a DID:
> > 
> > +-----------------+
> > | Platform device |
> > +-----------------+
> >    ||
> >    || ITS Device ID (DID)
> >    || Hard-wired on the bus.
> >    \/
> > +----------------------------+
> > | GICv3 ITS (MSI controller) |
> > +----------------------------+
> > 
> > Ignoring current bindings for the moment, I can see how we can describe
> > this with a generic master id-binding, with master-id-map along the
> > lines of interrupt-map, with a tuple format like:
> > <child-id-base child-id-length parent parent-id-base>
> > 
> > For the PCIe example, this would look something like the following (with
> > properties omitted for brevity):
> > 
> > PCI: pci@af000000 {
> > 	...
> > 	
> > 	/* Requester ID of PCIe endpoints, implicit at runtime */
> > 	master-id-cells = <1>;
> > 
> > 	/* RIDS idmapped to SIDS @ SMMU */
> > 	master-id-map = <0 0x10000 &SMMU 0>;
> > }
> > 
> > SMMU: iommu@bf000000 {
> > 	...
> > 
> > 	/* SID, derived from RID */
> > 	master-id-cells = <1>;
> > 
> > 	/* 
> > 	 * For some reason the high bit of the ID was negated.
> > 	 */
> > 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
> > 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
> > 
> > };
> > 
> > ITS: its@cf000000 {
> > 	...
> > 
> > 	/* DID, derived from SID */
> > 	master-id-cells = <2>;
> > 
> > 	/* 
> > 	 * Master traffic not propagated beyond this point, so no
> > 	 * master-id-ranges
> > 	 */
> > };
> 
> I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  

The set of IDs that we expect to transform (i.e. those masters use)
would be implicit in the first master-id-map from a master. In the PCI
example above, no master is expected to generate an ID outside of the
range 0-0x10000 inclusive (and that's all the SMMU would see).

For devices which are not hotpluggable, the nodes for those devices
would describe the specific set of IDs they use.

Generally, between a master and a slave there might not be one
contiguous set of valid IDs as these may be rewritten along the way (as
happens at the SMMU between the PCI root complex and the ITS in the
example above).

Which drivers do you think need this information? What exactly are they
trying to check and enforce?

> > For the simpler case, this would look something like:
> > 
> > DEV: device@af000000 {
> > 	master-id-cells = <1>;
> > 	master-ids = <0xf>, <0xb>;
> > 	master-id-map = <0 0xf &ITS 0 0>;
> > };
> > 
> > ITS: its@cf000000 {
> > 	...
> > 
> > 	/* DID */
> > 	master-id-cells = <2>;
> > };
> > 
> Is this is not depending heavily on discover order, how do drivers
> know which device to get which ID. is it implicitly assumed in
> discovery order? 

I'm not sure I follow the question. Could you elaborate?

> what happens to hot pluggable devices.  

It would be necessary to be able to discover the ID assigned to the
device by the hotpluggable bus. For example, this could depend on which
slot the device is plugged into.

If you can't discover the ID associated with a hotpluggable device from
the bus it is plugged into I can't see how hotplug could work.

>From that point on I would expect the ID transformations to be static as
in the example.

Thanks,
Mark.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Mark Rutland]

On Tue, May 26, 2015 at 11:20:59PM +0100, Chalamarla, Tirumalesh wrote:
> This is some thing we also like to see in ITS and SMMU drivers. 
> > On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> > 
> > Hi all,
> > 
> > For some devices, identification of particular masters is critical to
> > their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> > is determined from sideband signals on the bus, and it may or may not be
> > possible to probe and/or configure this information. Worse still, these
> > information may be rewritten by intermediate buses, so the
> > information for a given master may differ at different points in the bus
> > hierarchy.
> > 
> > We currently describe this master information for devices attached to
> > IOMMUs, with a master ID being encoded in the iommu-cells. However this
> > only covers the ID between the master and its IOMMU(s), and we don't
> > currently have a mechanism to describe the master IDs as they are seen
> > by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
> > 
> > The following are example of master IDs:
> > * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> > * SMMU Stream IDs (SIDs)
> > * GICv3 ITS Device IDs (DIDs)
> > 
> > In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> > rewritten in a chain of RID=>SID=>DID, as in the following diagram:
> > 
> > +-------------+
> > | PCIe master |
> > +-------------+
> >    ||
> >    || Requester ID (RID)
> >    || Probeable from RC.
> >    \/
> > +-------------------+
> > | PCIe Root Complex |
> > +-------------------+
> >    ||
> >    || SMMU Stream ID (SID)
> >    || Derived from RID, static non-probeable mapping.
> >    \/
> > +--------------+
> > | SMMU (IOMMU) |
> > +--------------+
> >    ||
> >    || ITS Device ID (DID)
> >    || Derived from SID, static non-probeable mapping.
> >    \/
> > +----------------------------+
> > | GICv3 ITS (MSI controller) |
> > +----------------------------+
> > 
> > In simpler cases, you might have a simpler set of master ID
> > translations, e.g. just a DID:
> > 
> > +-----------------+
> > | Platform device |
> > +-----------------+
> >    ||
> >    || ITS Device ID (DID)
> >    || Hard-wired on the bus.
> >    \/
> > +----------------------------+
> > | GICv3 ITS (MSI controller) |
> > +----------------------------+
> > 
> > Ignoring current bindings for the moment, I can see how we can describe
> > this with a generic master id-binding, with master-id-map along the
> > lines of interrupt-map, with a tuple format like:
> > <child-id-base child-id-length parent parent-id-base>
> > 
> > For the PCIe example, this would look something like the following (with
> > properties omitted for brevity):
> > 
> > PCI: pci at af000000 {
> > 	...
> > 	
> > 	/* Requester ID of PCIe endpoints, implicit at runtime */
> > 	master-id-cells = <1>;
> > 
> > 	/* RIDS idmapped to SIDS @ SMMU */
> > 	master-id-map = <0 0x10000 &SMMU 0>;
> > }
> > 
> > SMMU: iommu at bf000000 {
> > 	...
> > 
> > 	/* SID, derived from RID */
> > 	master-id-cells = <1>;
> > 
> > 	/* 
> > 	 * For some reason the high bit of the ID was negated.
> > 	 */
> > 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
> > 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
> > 
> > };
> > 
> > ITS: its at cf000000 {
> > 	...
> > 
> > 	/* DID, derived from SID */
> > 	master-id-cells = <2>;
> > 
> > 	/* 
> > 	 * Master traffic not propagated beyond this point, so no
> > 	 * master-id-ranges
> > 	 */
> > };
> 
> I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  

The set of IDs that we expect to transform (i.e. those masters use)
would be implicit in the first master-id-map from a master. In the PCI
example above, no master is expected to generate an ID outside of the
range 0-0x10000 inclusive (and that's all the SMMU would see).

For devices which are not hotpluggable, the nodes for those devices
would describe the specific set of IDs they use.

Generally, between a master and a slave there might not be one
contiguous set of valid IDs as these may be rewritten along the way (as
happens at the SMMU between the PCI root complex and the ITS in the
example above).

Which drivers do you think need this information? What exactly are they
trying to check and enforce?

> > For the simpler case, this would look something like:
> > 
> > DEV: device at af000000 {
> > 	master-id-cells = <1>;
> > 	master-ids = <0xf>, <0xb>;
> > 	master-id-map = <0 0xf &ITS 0 0>;
> > };
> > 
> > ITS: its at cf000000 {
> > 	...
> > 
> > 	/* DID */
> > 	master-id-cells = <2>;
> > };
> > 
> Is this is not depending heavily on discover order, how do drivers
> know which device to get which ID. is it implicitly assumed in
> discovery order? 

I'm not sure I follow the question. Could you elaborate?

> what happens to hot pluggable devices.  

It would be necessary to be able to discover the ID assigned to the
device by the hotpluggable bus. For example, this could depend on which
slot the device is plugged into.

If you can't discover the ID associated with a hotpluggable device from
the bus it is plugged into I can't see how hotplug could work.

>From that point on I would expect the ID transformations to be static as
in the example.

Thanks,
Mark.

Re: Master-aware devices and sideband ID data

[Chalamarla, Tirumalesh]

> On May 27, 2015, at 10:39 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> 
> On Tue, May 26, 2015 at 11:20:59PM +0100, Chalamarla, Tirumalesh wrote:
>> This is some thing we also like to see in ITS and SMMU drivers. 
>>> On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland@arm.com> wrote:
>>> 
>>> Hi all,
>>> 
>>> For some devices, identification of particular masters is critical to
>>> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
>>> is determined from sideband signals on the bus, and it may or may not be
>>> possible to probe and/or configure this information. Worse still, these
>>> information may be rewritten by intermediate buses, so the
>>> information for a given master may differ at different points in the bus
>>> hierarchy.
>>> 
>>> We currently describe this master information for devices attached to
>>> IOMMUs, with a master ID being encoded in the iommu-cells. However this
>>> only covers the ID between the master and its IOMMU(s), and we don't
>>> currently have a mechanism to describe the master IDs as they are seen
>>> by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
>>> 
>>> The following are example of master IDs:
>>> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
>>> * SMMU Stream IDs (SIDs)
>>> * GICv3 ITS Device IDs (DIDs)
>>> 
>>> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
>>> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
>>> 
>>> +-------------+
>>> | PCIe master |
>>> +-------------+
>>>   ||
>>>   || Requester ID (RID)
>>>   || Probeable from RC.
>>>   \/
>>> +-------------------+
>>> | PCIe Root Complex |
>>> +-------------------+
>>>   ||
>>>   || SMMU Stream ID (SID)
>>>   || Derived from RID, static non-probeable mapping.
>>>   \/
>>> +--------------+
>>> | SMMU (IOMMU) |
>>> +--------------+
>>>   ||
>>>   || ITS Device ID (DID)
>>>   || Derived from SID, static non-probeable mapping.
>>>   \/
>>> +----------------------------+
>>> | GICv3 ITS (MSI controller) |
>>> +----------------------------+
>>> 
>>> In simpler cases, you might have a simpler set of master ID
>>> translations, e.g. just a DID:
>>> 
>>> +-----------------+
>>> | Platform device |
>>> +-----------------+
>>>   ||
>>>   || ITS Device ID (DID)
>>>   || Hard-wired on the bus.
>>>   \/
>>> +----------------------------+
>>> | GICv3 ITS (MSI controller) |
>>> +----------------------------+
>>> 
>>> Ignoring current bindings for the moment, I can see how we can describe
>>> this with a generic master id-binding, with master-id-map along the
>>> lines of interrupt-map, with a tuple format like:
>>> <child-id-base child-id-length parent parent-id-base>
>>> 
>>> For the PCIe example, this would look something like the following (with
>>> properties omitted for brevity):
>>> 
>>> PCI: pci@af000000 {
>>> 	...
>>> 	
>>> 	/* Requester ID of PCIe endpoints, implicit at runtime */
>>> 	master-id-cells = <1>;
>>> 
>>> 	/* RIDS idmapped to SIDS @ SMMU */
>>> 	master-id-map = <0 0x10000 &SMMU 0>;
>>> }
>>> 
>>> SMMU: iommu@bf000000 {
>>> 	...
>>> 
>>> 	/* SID, derived from RID */
>>> 	master-id-cells = <1>;
>>> 
>>> 	/* 
>>> 	 * For some reason the high bit of the ID was negated.
>>> 	 */
>>> 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
>>> 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
>>> 
>>> };
>>> 
>>> ITS: its@cf000000 {
>>> 	...
>>> 
>>> 	/* DID, derived from SID */
>>> 	master-id-cells = <2>;
>>> 
>>> 	/* 
>>> 	 * Master traffic not propagated beyond this point, so no
>>> 	 * master-id-ranges
>>> 	 */
>>> };
>> 
>> I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  
> 
> The set of IDs that we expect to transform (i.e. those masters use)
> would be implicit in the first master-id-map from a master. In the PCI
> example above, no master is expected to generate an ID outside of the
> range 0-0x10000 inclusive (and that's all the SMMU would see).
> 
> For devices which are not hotpluggable, the nodes for those devices
> would describe the specific set of IDs they use.
> 
> Generally, between a master and a slave there might not be one
> contiguous set of valid IDs as these may be rewritten along the way (as
> happens at the SMMU between the PCI root complex and the ITS in the
> example above).
> 
> Which drivers do you think need this information? What exactly are they
> trying to check and enforce?
> 

I think, i miss understood by the names, so it is not base it is some thing like

x => f(x) => y           (and f(x) is bitwise | )

ok now i see it. 


>>> For the simpler case, this would look something like:
>>> 
>>> DEV: device@af000000 {
>>> 	master-id-cells = <1>;
>>> 	master-ids = <0xf>, <0xb>;
>>> 	master-id-map = <0 0xf &ITS 0 0>;
>>> };
>>> 
>>> ITS: its@cf000000 {
>>> 	...
>>> 
>>> 	/* DID */
>>> 	master-id-cells = <2>;
>>> };
>>> 
>> Is this is not depending heavily on discover order, how do drivers
>> know which device to get which ID. is it implicitly assumed in
>> discovery order? 
> 
> I'm not sure I follow the question. Could you elaborate?
> 
>> what happens to hot pluggable devices.  
> 
> It would be necessary to be able to discover the ID assigned to the
> device by the hotpluggable bus. For example, this could depend on which
> slot the device is plugged into.
> 
> If you can't discover the ID associated with a hotpluggable device from
> the bus it is plugged into I can't see how hotplug could work.
> 
> From that point on I would expect the ID transformations to be static as
> in the example.
> 

what if i have an SMMU with two ITS masters and i want to decide which one to use based on runtime like which CPU i am serving.

i am thinking in terms go NUMA based systems, where my pci device is on node 0, if the interrupt is targeted to CPU on node 1, i would like to use ITS1, otherwise 
ITS0. 

is it possible to specify some thing like that or do you think its is best to leave to ITS driver. 

Thanks,
Tirumalesh. 

> Thanks,
> Mark.

Master-aware devices and sideband ID data

[Chalamarla, Tirumalesh]

> On May 27, 2015, at 10:39 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> 
> On Tue, May 26, 2015 at 11:20:59PM +0100, Chalamarla, Tirumalesh wrote:
>> This is some thing we also like to see in ITS and SMMU drivers. 
>>> On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland@arm.com> wrote:
>>> 
>>> Hi all,
>>> 
>>> For some devices, identification of particular masters is critical to
>>> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
>>> is determined from sideband signals on the bus, and it may or may not be
>>> possible to probe and/or configure this information. Worse still, these
>>> information may be rewritten by intermediate buses, so the
>>> information for a given master may differ at different points in the bus
>>> hierarchy.
>>> 
>>> We currently describe this master information for devices attached to
>>> IOMMUs, with a master ID being encoded in the iommu-cells. However this
>>> only covers the ID between the master and its IOMMU(s), and we don't
>>> currently have a mechanism to describe the master IDs as they are seen
>>> by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
>>> 
>>> The following are example of master IDs:
>>> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
>>> * SMMU Stream IDs (SIDs)
>>> * GICv3 ITS Device IDs (DIDs)
>>> 
>>> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
>>> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
>>> 
>>> +-------------+
>>> | PCIe master |
>>> +-------------+
>>>   ||
>>>   || Requester ID (RID)
>>>   || Probeable from RC.
>>>   \/
>>> +-------------------+
>>> | PCIe Root Complex |
>>> +-------------------+
>>>   ||
>>>   || SMMU Stream ID (SID)
>>>   || Derived from RID, static non-probeable mapping.
>>>   \/
>>> +--------------+
>>> | SMMU (IOMMU) |
>>> +--------------+
>>>   ||
>>>   || ITS Device ID (DID)
>>>   || Derived from SID, static non-probeable mapping.
>>>   \/
>>> +----------------------------+
>>> | GICv3 ITS (MSI controller) |
>>> +----------------------------+
>>> 
>>> In simpler cases, you might have a simpler set of master ID
>>> translations, e.g. just a DID:
>>> 
>>> +-----------------+
>>> | Platform device |
>>> +-----------------+
>>>   ||
>>>   || ITS Device ID (DID)
>>>   || Hard-wired on the bus.
>>>   \/
>>> +----------------------------+
>>> | GICv3 ITS (MSI controller) |
>>> +----------------------------+
>>> 
>>> Ignoring current bindings for the moment, I can see how we can describe
>>> this with a generic master id-binding, with master-id-map along the
>>> lines of interrupt-map, with a tuple format like:
>>> <child-id-base child-id-length parent parent-id-base>
>>> 
>>> For the PCIe example, this would look something like the following (with
>>> properties omitted for brevity):
>>> 
>>> PCI: pci at af000000 {
>>> 	...
>>> 	
>>> 	/* Requester ID of PCIe endpoints, implicit at runtime */
>>> 	master-id-cells = <1>;
>>> 
>>> 	/* RIDS idmapped to SIDS @ SMMU */
>>> 	master-id-map = <0 0x10000 &SMMU 0>;
>>> }
>>> 
>>> SMMU: iommu at bf000000 {
>>> 	...
>>> 
>>> 	/* SID, derived from RID */
>>> 	master-id-cells = <1>;
>>> 
>>> 	/* 
>>> 	 * For some reason the high bit of the ID was negated.
>>> 	 */
>>> 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
>>> 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
>>> 
>>> };
>>> 
>>> ITS: its at cf000000 {
>>> 	...
>>> 
>>> 	/* DID, derived from SID */
>>> 	master-id-cells = <2>;
>>> 
>>> 	/* 
>>> 	 * Master traffic not propagated beyond this point, so no
>>> 	 * master-id-ranges
>>> 	 */
>>> };
>> 
>> I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  
> 
> The set of IDs that we expect to transform (i.e. those masters use)
> would be implicit in the first master-id-map from a master. In the PCI
> example above, no master is expected to generate an ID outside of the
> range 0-0x10000 inclusive (and that's all the SMMU would see).
> 
> For devices which are not hotpluggable, the nodes for those devices
> would describe the specific set of IDs they use.
> 
> Generally, between a master and a slave there might not be one
> contiguous set of valid IDs as these may be rewritten along the way (as
> happens at the SMMU between the PCI root complex and the ITS in the
> example above).
> 
> Which drivers do you think need this information? What exactly are they
> trying to check and enforce?
> 

I think, i miss understood by the names, so it is not base it is some thing like

x => f(x) => y           (and f(x) is bitwise | )

ok now i see it. 


>>> For the simpler case, this would look something like:
>>> 
>>> DEV: device at af000000 {
>>> 	master-id-cells = <1>;
>>> 	master-ids = <0xf>, <0xb>;
>>> 	master-id-map = <0 0xf &ITS 0 0>;
>>> };
>>> 
>>> ITS: its at cf000000 {
>>> 	...
>>> 
>>> 	/* DID */
>>> 	master-id-cells = <2>;
>>> };
>>> 
>> Is this is not depending heavily on discover order, how do drivers
>> know which device to get which ID. is it implicitly assumed in
>> discovery order? 
> 
> I'm not sure I follow the question. Could you elaborate?
> 
>> what happens to hot pluggable devices.  
> 
> It would be necessary to be able to discover the ID assigned to the
> device by the hotpluggable bus. For example, this could depend on which
> slot the device is plugged into.
> 
> If you can't discover the ID associated with a hotpluggable device from
> the bus it is plugged into I can't see how hotplug could work.
> 
> From that point on I would expect the ID transformations to be static as
> in the example.
> 

what if i have an SMMU with two ITS masters and i want to decide which one to use based on runtime like which CPU i am serving.

i am thinking in terms go NUMA based systems, where my pci device is on node 0, if the interrupt is targeted to CPU on node 1, i would like to use ITS1, otherwise 
ITS0. 

is it possible to specify some thing like that or do you think its is best to leave to ITS driver. 

Thanks,
Tirumalesh. 

> Thanks,
> Mark.

Re: Master-aware devices and sideband ID data

[Mark Rutland]

On Fri, May 29, 2015 at 06:46:07PM +0100, Chalamarla, Tirumalesh wrote:
> 
> > On May 27, 2015, at 10:39 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> > 
> > On Tue, May 26, 2015 at 11:20:59PM +0100, Chalamarla, Tirumalesh wrote:
> >> This is some thing we also like to see in ITS and SMMU drivers. 
> >>> On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> >>> 
> >>> Hi all,
> >>> 
> >>> For some devices, identification of particular masters is critical to
> >>> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> >>> is determined from sideband signals on the bus, and it may or may not be
> >>> possible to probe and/or configure this information. Worse still, these
> >>> information may be rewritten by intermediate buses, so the
> >>> information for a given master may differ at different points in the bus
> >>> hierarchy.
> >>> 
> >>> We currently describe this master information for devices attached to
> >>> IOMMUs, with a master ID being encoded in the iommu-cells. However this
> >>> only covers the ID between the master and its IOMMU(s), and we don't
> >>> currently have a mechanism to describe the master IDs as they are seen
> >>> by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
> >>> 
> >>> The following are example of master IDs:
> >>> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> >>> * SMMU Stream IDs (SIDs)
> >>> * GICv3 ITS Device IDs (DIDs)
> >>> 
> >>> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> >>> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
> >>> 
> >>> +-------------+
> >>> | PCIe master |
> >>> +-------------+
> >>>   ||
> >>>   || Requester ID (RID)
> >>>   || Probeable from RC.
> >>>   \/
> >>> +-------------------+
> >>> | PCIe Root Complex |
> >>> +-------------------+
> >>>   ||
> >>>   || SMMU Stream ID (SID)
> >>>   || Derived from RID, static non-probeable mapping.
> >>>   \/
> >>> +--------------+
> >>> | SMMU (IOMMU) |
> >>> +--------------+
> >>>   ||
> >>>   || ITS Device ID (DID)
> >>>   || Derived from SID, static non-probeable mapping.
> >>>   \/
> >>> +----------------------------+
> >>> | GICv3 ITS (MSI controller) |
> >>> +----------------------------+
> >>> 
> >>> In simpler cases, you might have a simpler set of master ID
> >>> translations, e.g. just a DID:
> >>> 
> >>> +-----------------+
> >>> | Platform device |
> >>> +-----------------+
> >>>   ||
> >>>   || ITS Device ID (DID)
> >>>   || Hard-wired on the bus.
> >>>   \/
> >>> +----------------------------+
> >>> | GICv3 ITS (MSI controller) |
> >>> +----------------------------+
> >>> 
> >>> Ignoring current bindings for the moment, I can see how we can describe
> >>> this with a generic master id-binding, with master-id-map along the
> >>> lines of interrupt-map, with a tuple format like:
> >>> <child-id-base child-id-length parent parent-id-base>
> >>> 
> >>> For the PCIe example, this would look something like the following (with
> >>> properties omitted for brevity):
> >>> 
> >>> PCI: pci@af000000 {
> >>> 	...
> >>> 	
> >>> 	/* Requester ID of PCIe endpoints, implicit at runtime */
> >>> 	master-id-cells = <1>;
> >>> 
> >>> 	/* RIDS idmapped to SIDS @ SMMU */
> >>> 	master-id-map = <0 0x10000 &SMMU 0>;
> >>> }
> >>> 
> >>> SMMU: iommu@bf000000 {
> >>> 	...
> >>> 
> >>> 	/* SID, derived from RID */
> >>> 	master-id-cells = <1>;
> >>> 
> >>> 	/* 
> >>> 	 * For some reason the high bit of the ID was negated.
> >>> 	 */
> >>> 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
> >>> 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
> >>> 
> >>> };
> >>> 
> >>> ITS: its@cf000000 {
> >>> 	...
> >>> 
> >>> 	/* DID, derived from SID */
> >>> 	master-id-cells = <2>;
> >>> 
> >>> 	/* 
> >>> 	 * Master traffic not propagated beyond this point, so no
> >>> 	 * master-id-ranges
> >>> 	 */
> >>> };
> >> 
> >> I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  
> > 
> > The set of IDs that we expect to transform (i.e. those masters use)
> > would be implicit in the first master-id-map from a master. In the PCI
> > example above, no master is expected to generate an ID outside of the
> > range 0-0x10000 inclusive (and that's all the SMMU would see).
> > 
> > For devices which are not hotpluggable, the nodes for those devices
> > would describe the specific set of IDs they use.
> > 
> > Generally, between a master and a slave there might not be one
> > contiguous set of valid IDs as these may be rewritten along the way (as
> > happens at the SMMU between the PCI root complex and the ITS in the
> > example above).
> > 
> > Which drivers do you think need this information? What exactly are they
> > trying to check and enforce?
> > 
> 
> I think, i miss understood by the names, so it is not base it is some thing like
> 
> x => f(x) => y           (and f(x) is bitwise | )
> 
> ok now i see it. 

It's subtraction and addition rather than bitwise ORing.

Consider a SID 0x8040 entering the SMMU. This falls in the range
described by the SMMU's second master-id-map entry (I've added brackets
to separate each entry:
 
<(0x8000) (0x8000) (&ITS) (0x0 0x0000)>

As 0x8040 falls in the range described by the first two elements, we kow
we can translate the ID to calculate the ID at the ITS. To do so, we
subtract the child-id-base (0x8000), and add the parent-id-base (0x0
0x0000). That give us 0x0 0x0040 as our ITS device ID.

> >>> For the simpler case, this would look something like:
> >>> 
> >>> DEV: device@af000000 {
> >>> 	master-id-cells = <1>;
> >>> 	master-ids = <0xf>, <0xb>;
> >>> 	master-id-map = <0 0xf &ITS 0 0>;
> >>> };
> >>> 
> >>> ITS: its@cf000000 {
> >>> 	...
> >>> 
> >>> 	/* DID */
> >>> 	master-id-cells = <2>;
> >>> };
> >>> 
> >> Is this is not depending heavily on discover order, how do drivers
> >> know which device to get which ID. is it implicitly assumed in
> >> discovery order? 
> > 
> > I'm not sure I follow the question. Could you elaborate?
> > 
> >> what happens to hot pluggable devices.  
> > 
> > It would be necessary to be able to discover the ID assigned to the
> > device by the hotpluggable bus. For example, this could depend on which
> > slot the device is plugged into.
> > 
> > If you can't discover the ID associated with a hotpluggable device from
> > the bus it is plugged into I can't see how hotplug could work.
> > 
> > From that point on I would expect the ID transformations to be static as
> > in the example.
> > 
> 
> what if i have an SMMU with two ITS masters and i want to decide which
> one to use based on runtime like which CPU i am serving.
> 
> i am thinking in terms go NUMA based systems, where my pci device is
> on node 0, if the interrupt is targeted to CPU on node 1, i would like
> to use ITS1, otherwise 
> ITS0. 

The binding can express that both these translation paths exist. For
example, on the SMMU node you could have:

master-id-map = <0x0000 0x8000 &ITS_1 0x0 0x0000>,
                <0x0000 0x8000 &ITS_2 0x0 0x0000>;

Which would mean that master IDs are idmapped to both ITS instances.

> is it possible to specify some thing like that or do you think its is
> best to leave to ITS driver. 

It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

Thanks,
Mark.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Mark Rutland]

On Fri, May 29, 2015 at 06:46:07PM +0100, Chalamarla, Tirumalesh wrote:
> 
> > On May 27, 2015, at 10:39 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> > 
> > On Tue, May 26, 2015 at 11:20:59PM +0100, Chalamarla, Tirumalesh wrote:
> >> This is some thing we also like to see in ITS and SMMU drivers. 
> >>> On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> >>> 
> >>> Hi all,
> >>> 
> >>> For some devices, identification of particular masters is critical to
> >>> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
> >>> is determined from sideband signals on the bus, and it may or may not be
> >>> possible to probe and/or configure this information. Worse still, these
> >>> information may be rewritten by intermediate buses, so the
> >>> information for a given master may differ at different points in the bus
> >>> hierarchy.
> >>> 
> >>> We currently describe this master information for devices attached to
> >>> IOMMUs, with a master ID being encoded in the iommu-cells. However this
> >>> only covers the ID between the master and its IOMMU(s), and we don't
> >>> currently have a mechanism to describe the master IDs as they are seen
> >>> by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
> >>> 
> >>> The following are example of master IDs:
> >>> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
> >>> * SMMU Stream IDs (SIDs)
> >>> * GICv3 ITS Device IDs (DIDs)
> >>> 
> >>> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
> >>> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
> >>> 
> >>> +-------------+
> >>> | PCIe master |
> >>> +-------------+
> >>>   ||
> >>>   || Requester ID (RID)
> >>>   || Probeable from RC.
> >>>   \/
> >>> +-------------------+
> >>> | PCIe Root Complex |
> >>> +-------------------+
> >>>   ||
> >>>   || SMMU Stream ID (SID)
> >>>   || Derived from RID, static non-probeable mapping.
> >>>   \/
> >>> +--------------+
> >>> | SMMU (IOMMU) |
> >>> +--------------+
> >>>   ||
> >>>   || ITS Device ID (DID)
> >>>   || Derived from SID, static non-probeable mapping.
> >>>   \/
> >>> +----------------------------+
> >>> | GICv3 ITS (MSI controller) |
> >>> +----------------------------+
> >>> 
> >>> In simpler cases, you might have a simpler set of master ID
> >>> translations, e.g. just a DID:
> >>> 
> >>> +-----------------+
> >>> | Platform device |
> >>> +-----------------+
> >>>   ||
> >>>   || ITS Device ID (DID)
> >>>   || Hard-wired on the bus.
> >>>   \/
> >>> +----------------------------+
> >>> | GICv3 ITS (MSI controller) |
> >>> +----------------------------+
> >>> 
> >>> Ignoring current bindings for the moment, I can see how we can describe
> >>> this with a generic master id-binding, with master-id-map along the
> >>> lines of interrupt-map, with a tuple format like:
> >>> <child-id-base child-id-length parent parent-id-base>
> >>> 
> >>> For the PCIe example, this would look something like the following (with
> >>> properties omitted for brevity):
> >>> 
> >>> PCI: pci at af000000 {
> >>> 	...
> >>> 	
> >>> 	/* Requester ID of PCIe endpoints, implicit at runtime */
> >>> 	master-id-cells = <1>;
> >>> 
> >>> 	/* RIDS idmapped to SIDS @ SMMU */
> >>> 	master-id-map = <0 0x10000 &SMMU 0>;
> >>> }
> >>> 
> >>> SMMU: iommu at bf000000 {
> >>> 	...
> >>> 
> >>> 	/* SID, derived from RID */
> >>> 	master-id-cells = <1>;
> >>> 
> >>> 	/* 
> >>> 	 * For some reason the high bit of the ID was negated.
> >>> 	 */
> >>> 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
> >>> 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
> >>> 
> >>> };
> >>> 
> >>> ITS: its at cf000000 {
> >>> 	...
> >>> 
> >>> 	/* DID, derived from SID */
> >>> 	master-id-cells = <2>;
> >>> 
> >>> 	/* 
> >>> 	 * Master traffic not propagated beyond this point, so no
> >>> 	 * master-id-ranges
> >>> 	 */
> >>> };
> >> 
> >> I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  
> > 
> > The set of IDs that we expect to transform (i.e. those masters use)
> > would be implicit in the first master-id-map from a master. In the PCI
> > example above, no master is expected to generate an ID outside of the
> > range 0-0x10000 inclusive (and that's all the SMMU would see).
> > 
> > For devices which are not hotpluggable, the nodes for those devices
> > would describe the specific set of IDs they use.
> > 
> > Generally, between a master and a slave there might not be one
> > contiguous set of valid IDs as these may be rewritten along the way (as
> > happens at the SMMU between the PCI root complex and the ITS in the
> > example above).
> > 
> > Which drivers do you think need this information? What exactly are they
> > trying to check and enforce?
> > 
> 
> I think, i miss understood by the names, so it is not base it is some thing like
> 
> x => f(x) => y           (and f(x) is bitwise | )
> 
> ok now i see it. 

It's subtraction and addition rather than bitwise ORing.

Consider a SID 0x8040 entering the SMMU. This falls in the range
described by the SMMU's second master-id-map entry (I've added brackets
to separate each entry:
 
<(0x8000) (0x8000) (&ITS) (0x0 0x0000)>

As 0x8040 falls in the range described by the first two elements, we kow
we can translate the ID to calculate the ID at the ITS. To do so, we
subtract the child-id-base (0x8000), and add the parent-id-base (0x0
0x0000). That give us 0x0 0x0040 as our ITS device ID.

> >>> For the simpler case, this would look something like:
> >>> 
> >>> DEV: device at af000000 {
> >>> 	master-id-cells = <1>;
> >>> 	master-ids = <0xf>, <0xb>;
> >>> 	master-id-map = <0 0xf &ITS 0 0>;
> >>> };
> >>> 
> >>> ITS: its at cf000000 {
> >>> 	...
> >>> 
> >>> 	/* DID */
> >>> 	master-id-cells = <2>;
> >>> };
> >>> 
> >> Is this is not depending heavily on discover order, how do drivers
> >> know which device to get which ID. is it implicitly assumed in
> >> discovery order? 
> > 
> > I'm not sure I follow the question. Could you elaborate?
> > 
> >> what happens to hot pluggable devices.  
> > 
> > It would be necessary to be able to discover the ID assigned to the
> > device by the hotpluggable bus. For example, this could depend on which
> > slot the device is plugged into.
> > 
> > If you can't discover the ID associated with a hotpluggable device from
> > the bus it is plugged into I can't see how hotplug could work.
> > 
> > From that point on I would expect the ID transformations to be static as
> > in the example.
> > 
> 
> what if i have an SMMU with two ITS masters and i want to decide which
> one to use based on runtime like which CPU i am serving.
> 
> i am thinking in terms go NUMA based systems, where my pci device is
> on node 0, if the interrupt is targeted to CPU on node 1, i would like
> to use ITS1, otherwise 
> ITS0. 

The binding can express that both these translation paths exist. For
example, on the SMMU node you could have:

master-id-map = <0x0000 0x8000 &ITS_1 0x0 0x0000>,
                <0x0000 0x8000 &ITS_2 0x0 0x0000>;

Which would mean that master IDs are idmapped to both ITS instances.

> is it possible to specify some thing like that or do you think its is
> best to leave to ITS driver. 

It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

Thanks,
Mark.

Re: Master-aware devices and sideband ID data

[Chalamarla, Tirumalesh]

> On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> 
> On Fri, May 29, 2015 at 06:46:07PM +0100, Chalamarla, Tirumalesh wrote:
>> 
>>> On May 27, 2015, at 10:39 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
>>> 
>>> On Tue, May 26, 2015 at 11:20:59PM +0100, Chalamarla, Tirumalesh wrote:
>>>> This is some thing we also like to see in ITS and SMMU drivers. 
>>>>> On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
>>>>> 
>>>>> Hi all,
>>>>> 
>>>>> For some devices, identification of particular masters is critical to
>>>>> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
>>>>> is determined from sideband signals on the bus, and it may or may not be
>>>>> possible to probe and/or configure this information. Worse still, these
>>>>> information may be rewritten by intermediate buses, so the
>>>>> information for a given master may differ at different points in the bus
>>>>> hierarchy.
>>>>> 
>>>>> We currently describe this master information for devices attached to
>>>>> IOMMUs, with a master ID being encoded in the iommu-cells. However this
>>>>> only covers the ID between the master and its IOMMU(s), and we don't
>>>>> currently have a mechanism to describe the master IDs as they are seen
>>>>> by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
>>>>> 
>>>>> The following are example of master IDs:
>>>>> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
>>>>> * SMMU Stream IDs (SIDs)
>>>>> * GICv3 ITS Device IDs (DIDs)
>>>>> 
>>>>> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
>>>>> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
>>>>> 
>>>>> +-------------+
>>>>> | PCIe master |
>>>>> +-------------+
>>>>>  ||
>>>>>  || Requester ID (RID)
>>>>>  || Probeable from RC.
>>>>>  \/
>>>>> +-------------------+
>>>>> | PCIe Root Complex |
>>>>> +-------------------+
>>>>>  ||
>>>>>  || SMMU Stream ID (SID)
>>>>>  || Derived from RID, static non-probeable mapping.
>>>>>  \/
>>>>> +--------------+
>>>>> | SMMU (IOMMU) |
>>>>> +--------------+
>>>>>  ||
>>>>>  || ITS Device ID (DID)
>>>>>  || Derived from SID, static non-probeable mapping.
>>>>>  \/
>>>>> +----------------------------+
>>>>> | GICv3 ITS (MSI controller) |
>>>>> +----------------------------+
>>>>> 
>>>>> In simpler cases, you might have a simpler set of master ID
>>>>> translations, e.g. just a DID:
>>>>> 
>>>>> +-----------------+
>>>>> | Platform device |
>>>>> +-----------------+
>>>>>  ||
>>>>>  || ITS Device ID (DID)
>>>>>  || Hard-wired on the bus.
>>>>>  \/
>>>>> +----------------------------+
>>>>> | GICv3 ITS (MSI controller) |
>>>>> +----------------------------+
>>>>> 
>>>>> Ignoring current bindings for the moment, I can see how we can describe
>>>>> this with a generic master id-binding, with master-id-map along the
>>>>> lines of interrupt-map, with a tuple format like:
>>>>> <child-id-base child-id-length parent parent-id-base>
>>>>> 
>>>>> For the PCIe example, this would look something like the following (with
>>>>> properties omitted for brevity):
>>>>> 
>>>>> PCI: pci@af000000 {
>>>>> 	...
>>>>> 	
>>>>> 	/* Requester ID of PCIe endpoints, implicit at runtime */
>>>>> 	master-id-cells = <1>;
>>>>> 
>>>>> 	/* RIDS idmapped to SIDS @ SMMU */
>>>>> 	master-id-map = <0 0x10000 &SMMU 0>;
>>>>> }
>>>>> 
>>>>> SMMU: iommu@bf000000 {
>>>>> 	...
>>>>> 
>>>>> 	/* SID, derived from RID */
>>>>> 	master-id-cells = <1>;
>>>>> 
>>>>> 	/* 
>>>>> 	 * For some reason the high bit of the ID was negated.
>>>>> 	 */
>>>>> 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
>>>>> 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
>>>>> 
>>>>> };
>>>>> 
>>>>> ITS: its@cf000000 {
>>>>> 	...
>>>>> 
>>>>> 	/* DID, derived from SID */
>>>>> 	master-id-cells = <2>;
>>>>> 
>>>>> 	/* 
>>>>> 	 * Master traffic not propagated beyond this point, so no
>>>>> 	 * master-id-ranges
>>>>> 	 */
>>>>> };
>>>> 
>>>> I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  
>>> 
>>> The set of IDs that we expect to transform (i.e. those masters use)
>>> would be implicit in the first master-id-map from a master. In the PCI
>>> example above, no master is expected to generate an ID outside of the
>>> range 0-0x10000 inclusive (and that's all the SMMU would see).
>>> 
>>> For devices which are not hotpluggable, the nodes for those devices
>>> would describe the specific set of IDs they use.
>>> 
>>> Generally, between a master and a slave there might not be one
>>> contiguous set of valid IDs as these may be rewritten along the way (as
>>> happens at the SMMU between the PCI root complex and the ITS in the
>>> example above).
>>> 
>>> Which drivers do you think need this information? What exactly are they
>>> trying to check and enforce?
>>> 
>> 
>> I think, i miss understood by the names, so it is not base it is some thing like
>> 
>> x => f(x) => y           (and f(x) is bitwise | )
>> 
>> ok now i see it. 
> 
> It's subtraction and addition rather than bitwise ORing.
> 
> Consider a SID 0x8040 entering the SMMU. This falls in the range
> described by the SMMU's second master-id-map entry (I've added brackets
> to separate each entry:
> 
> <(0x8000) (0x8000) (&ITS) (0x0 0x0000)>
> 
> As 0x8040 falls in the range described by the first two elements, we kow
> we can translate the ID to calculate the ID at the ITS. To do so, we
> subtract the child-id-base (0x8000), and add the parent-id-base (0x0
> 0x0000). That give us 0x0 0x0040 as our ITS device ID.

fine.
> 
>>>>> For the simpler case, this would look something like:
>>>>> 
>>>>> DEV: device@af000000 {
>>>>> 	master-id-cells = <1>;
>>>>> 	master-ids = <0xf>, <0xb>;
>>>>> 	master-id-map = <0 0xf &ITS 0 0>;
>>>>> };
>>>>> 
>>>>> ITS: its@cf000000 {
>>>>> 	...
>>>>> 
>>>>> 	/* DID */
>>>>> 	master-id-cells = <2>;
>>>>> };
>>>>> 
>>>> Is this is not depending heavily on discover order, how do drivers
>>>> know which device to get which ID. is it implicitly assumed in
>>>> discovery order? 
>>> 
>>> I'm not sure I follow the question. Could you elaborate?
>>> 
>>>> what happens to hot pluggable devices.  
>>> 
>>> It would be necessary to be able to discover the ID assigned to the
>>> device by the hotpluggable bus. For example, this could depend on which
>>> slot the device is plugged into.
>>> 
>>> If you can't discover the ID associated with a hotpluggable device from
>>> the bus it is plugged into I can't see how hotplug could work.
>>> 
>>> From that point on I would expect the ID transformations to be static as
>>> in the example.
>>> 
>> 
>> what if i have an SMMU with two ITS masters and i want to decide which
>> one to use based on runtime like which CPU i am serving.
>> 
>> i am thinking in terms go NUMA based systems, where my pci device is
>> on node 0, if the interrupt is targeted to CPU on node 1, i would like
>> to use ITS1, otherwise 
>> ITS0. 
> 
> The binding can express that both these translation paths exist. For
> example, on the SMMU node you could have:
> 
> master-id-map = <0x0000 0x8000 &ITS_1 0x0 0x0000>,
>                <0x0000 0x8000 &ITS_2 0x0 0x0000>;
> 
> Which would mean that master IDs are idmapped to both ITS instances.
> 
>> is it possible to specify some thing like that or do you think its is
>> best to leave to ITS driver. 
> 
> It's possible to specify that the paths exist. I expect that software
> would select which to use at runtime.
> 

My worry is how to define any priorities/preferences between masters. 
in general the proposal looks reasonable.

Thanks,
Tirumalesh.   
> Thanks,
> Mark.

Master-aware devices and sideband ID data

[Chalamarla, Tirumalesh]

> On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> 
> On Fri, May 29, 2015 at 06:46:07PM +0100, Chalamarla, Tirumalesh wrote:
>> 
>>> On May 27, 2015, at 10:39 AM, Mark Rutland <mark.rutland@arm.com> wrote:
>>> 
>>> On Tue, May 26, 2015 at 11:20:59PM +0100, Chalamarla, Tirumalesh wrote:
>>>> This is some thing we also like to see in ITS and SMMU drivers. 
>>>>> On Mar 24, 2015, at 8:50 AM, Mark Rutland <mark.rutland@arm.com> wrote:
>>>>> 
>>>>> Hi all,
>>>>> 
>>>>> For some devices, identification of particular masters is critical to
>>>>> their operation (e.g. IOMMUs, MSI controllers). The identity of masters
>>>>> is determined from sideband signals on the bus, and it may or may not be
>>>>> possible to probe and/or configure this information. Worse still, these
>>>>> information may be rewritten by intermediate buses, so the
>>>>> information for a given master may differ at different points in the bus
>>>>> hierarchy.
>>>>> 
>>>>> We currently describe this master information for devices attached to
>>>>> IOMMUs, with a master ID being encoded in the iommu-cells. However this
>>>>> only covers the ID between the master and its IOMMU(s), and we don't
>>>>> currently have a mechanism to describe the master IDs as they are seen
>>>>> by devices beyond the IOMMU(s), or in the absence of any IOMMU(s).
>>>>> 
>>>>> The following are example of master IDs:
>>>>> * PCIe Requester IDs (RIDs) (bus:device.function AKA BDF)
>>>>> * SMMU Stream IDs (SIDs)
>>>>> * GICv3 ITS Device IDs (DIDs)
>>>>> 
>>>>> In the case of a system with PCIe, SMMU and GICv3, the master IDs are
>>>>> rewritten in a chain of RID=>SID=>DID, as in the following diagram:
>>>>> 
>>>>> +-------------+
>>>>> | PCIe master |
>>>>> +-------------+
>>>>>  ||
>>>>>  || Requester ID (RID)
>>>>>  || Probeable from RC.
>>>>>  \/
>>>>> +-------------------+
>>>>> | PCIe Root Complex |
>>>>> +-------------------+
>>>>>  ||
>>>>>  || SMMU Stream ID (SID)
>>>>>  || Derived from RID, static non-probeable mapping.
>>>>>  \/
>>>>> +--------------+
>>>>> | SMMU (IOMMU) |
>>>>> +--------------+
>>>>>  ||
>>>>>  || ITS Device ID (DID)
>>>>>  || Derived from SID, static non-probeable mapping.
>>>>>  \/
>>>>> +----------------------------+
>>>>> | GICv3 ITS (MSI controller) |
>>>>> +----------------------------+
>>>>> 
>>>>> In simpler cases, you might have a simpler set of master ID
>>>>> translations, e.g. just a DID:
>>>>> 
>>>>> +-----------------+
>>>>> | Platform device |
>>>>> +-----------------+
>>>>>  ||
>>>>>  || ITS Device ID (DID)
>>>>>  || Hard-wired on the bus.
>>>>>  \/
>>>>> +----------------------------+
>>>>> | GICv3 ITS (MSI controller) |
>>>>> +----------------------------+
>>>>> 
>>>>> Ignoring current bindings for the moment, I can see how we can describe
>>>>> this with a generic master id-binding, with master-id-map along the
>>>>> lines of interrupt-map, with a tuple format like:
>>>>> <child-id-base child-id-length parent parent-id-base>
>>>>> 
>>>>> For the PCIe example, this would look something like the following (with
>>>>> properties omitted for brevity):
>>>>> 
>>>>> PCI: pci at af000000 {
>>>>> 	...
>>>>> 	
>>>>> 	/* Requester ID of PCIe endpoints, implicit at runtime */
>>>>> 	master-id-cells = <1>;
>>>>> 
>>>>> 	/* RIDS idmapped to SIDS @ SMMU */
>>>>> 	master-id-map = <0 0x10000 &SMMU 0>;
>>>>> }
>>>>> 
>>>>> SMMU: iommu at bf000000 {
>>>>> 	...
>>>>> 
>>>>> 	/* SID, derived from RID */
>>>>> 	master-id-cells = <1>;
>>>>> 
>>>>> 	/* 
>>>>> 	 * For some reason the high bit of the ID was negated.
>>>>> 	 */
>>>>> 	master-id-map = <0x0000 0x8000 &ITS 0x0 0x8000>,
>>>>> 	                <0x8000 0x8000 &ITS 0x0 0x0000>;
>>>>> 
>>>>> };
>>>>> 
>>>>> ITS: its at cf000000 {
>>>>> 	...
>>>>> 
>>>>> 	/* DID, derived from SID */
>>>>> 	master-id-cells = <2>;
>>>>> 
>>>>> 	/* 
>>>>> 	 * Master traffic not propagated beyond this point, so no
>>>>> 	 * master-id-ranges
>>>>> 	 */
>>>>> };
>>>> 
>>>> I think it is nice to have max IDs supported by masters. so that drivers can check and enforce.  
>>> 
>>> The set of IDs that we expect to transform (i.e. those masters use)
>>> would be implicit in the first master-id-map from a master. In the PCI
>>> example above, no master is expected to generate an ID outside of the
>>> range 0-0x10000 inclusive (and that's all the SMMU would see).
>>> 
>>> For devices which are not hotpluggable, the nodes for those devices
>>> would describe the specific set of IDs they use.
>>> 
>>> Generally, between a master and a slave there might not be one
>>> contiguous set of valid IDs as these may be rewritten along the way (as
>>> happens at the SMMU between the PCI root complex and the ITS in the
>>> example above).
>>> 
>>> Which drivers do you think need this information? What exactly are they
>>> trying to check and enforce?
>>> 
>> 
>> I think, i miss understood by the names, so it is not base it is some thing like
>> 
>> x => f(x) => y           (and f(x) is bitwise | )
>> 
>> ok now i see it. 
> 
> It's subtraction and addition rather than bitwise ORing.
> 
> Consider a SID 0x8040 entering the SMMU. This falls in the range
> described by the SMMU's second master-id-map entry (I've added brackets
> to separate each entry:
> 
> <(0x8000) (0x8000) (&ITS) (0x0 0x0000)>
> 
> As 0x8040 falls in the range described by the first two elements, we kow
> we can translate the ID to calculate the ID at the ITS. To do so, we
> subtract the child-id-base (0x8000), and add the parent-id-base (0x0
> 0x0000). That give us 0x0 0x0040 as our ITS device ID.

fine.
> 
>>>>> For the simpler case, this would look something like:
>>>>> 
>>>>> DEV: device at af000000 {
>>>>> 	master-id-cells = <1>;
>>>>> 	master-ids = <0xf>, <0xb>;
>>>>> 	master-id-map = <0 0xf &ITS 0 0>;
>>>>> };
>>>>> 
>>>>> ITS: its at cf000000 {
>>>>> 	...
>>>>> 
>>>>> 	/* DID */
>>>>> 	master-id-cells = <2>;
>>>>> };
>>>>> 
>>>> Is this is not depending heavily on discover order, how do drivers
>>>> know which device to get which ID. is it implicitly assumed in
>>>> discovery order? 
>>> 
>>> I'm not sure I follow the question. Could you elaborate?
>>> 
>>>> what happens to hot pluggable devices.  
>>> 
>>> It would be necessary to be able to discover the ID assigned to the
>>> device by the hotpluggable bus. For example, this could depend on which
>>> slot the device is plugged into.
>>> 
>>> If you can't discover the ID associated with a hotpluggable device from
>>> the bus it is plugged into I can't see how hotplug could work.
>>> 
>>> From that point on I would expect the ID transformations to be static as
>>> in the example.
>>> 
>> 
>> what if i have an SMMU with two ITS masters and i want to decide which
>> one to use based on runtime like which CPU i am serving.
>> 
>> i am thinking in terms go NUMA based systems, where my pci device is
>> on node 0, if the interrupt is targeted to CPU on node 1, i would like
>> to use ITS1, otherwise 
>> ITS0. 
> 
> The binding can express that both these translation paths exist. For
> example, on the SMMU node you could have:
> 
> master-id-map = <0x0000 0x8000 &ITS_1 0x0 0x0000>,
>                <0x0000 0x8000 &ITS_2 0x0 0x0000>;
> 
> Which would mean that master IDs are idmapped to both ITS instances.
> 
>> is it possible to specify some thing like that or do you think its is
>> best to leave to ITS driver. 
> 
> It's possible to specify that the paths exist. I expect that software
> would select which to use at runtime.
> 

My worry is how to define any priorities/preferences between masters. 
in general the proposal looks reasonable.

Thanks,
Tirumalesh.   
> Thanks,
> Mark.

Re: Master-aware devices and sideband ID data

[Will Deacon]

On Thu, Jun 04, 2015 at 11:19:30PM +0100, Chalamarla, Tirumalesh wrote:
> > On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> > It's possible to specify that the paths exist. I expect that software
> > would select which to use at runtime.
> > 
> My worry is how to define any priorities/preferences between masters. 
> in general the proposal looks reasonable.

I agree that the proposal looks reasonable (in terms of the ability to
describe the sort of topologies that we will face) but I still don't
understand what I need to do in e.g. my IOMMU driver to support this
binding whilst continuing to support the existing iommus binding, which
is relied upon to configure dma-mapping.

Mark: how do you see this co-existing/merging with the current bindings?
I don't think it's practical to throw away what we have and move over to
something totally different all in one go, but there clearly *is* benefit
in your proposal over the existing scheme.

Will
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Will Deacon]

On Thu, Jun 04, 2015 at 11:19:30PM +0100, Chalamarla, Tirumalesh wrote:
> > On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> > It's possible to specify that the paths exist. I expect that software
> > would select which to use at runtime.
> > 
> My worry is how to define any priorities/preferences between masters. 
> in general the proposal looks reasonable.

I agree that the proposal looks reasonable (in terms of the ability to
describe the sort of topologies that we will face) but I still don't
understand what I need to do in e.g. my IOMMU driver to support this
binding whilst continuing to support the existing iommus binding, which
is relied upon to configure dma-mapping.

Mark: how do you see this co-existing/merging with the current bindings?
I don't think it's practical to throw away what we have and move over to
something totally different all in one go, but there clearly *is* benefit
in your proposal over the existing scheme.

Will

Re: Master-aware devices and sideband ID data

[Mark Rutland]

On Fri, Jun 05, 2015 at 10:05:34AM +0100, Will Deacon wrote:
> On Thu, Jun 04, 2015 at 11:19:30PM +0100, Chalamarla, Tirumalesh wrote:
> > > On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> > > It's possible to specify that the paths exist. I expect that software
> > > would select which to use at runtime.
> > > 
> > My worry is how to define any priorities/preferences between masters. 
> > in general the proposal looks reasonable.
> 
> I agree that the proposal looks reasonable (in terms of the ability to
> describe the sort of topologies that we will face) but I still don't
> understand what I need to do in e.g. my IOMMU driver to support this
> binding whilst continuing to support the existing iommus binding, which
> is relied upon to configure dma-mapping.
> 
> Mark: how do you see this co-existing/merging with the current bindings?

As I mentioned in my initial mail, it's not clear to me how this can be
reconciled with the current bindings. Everything I've been able to come
up with so far at best ends up describing the same thing repeatedly.

I'll see what I can come up with. Any sugestions are welcome!

> I don't think it's practical to throw away what we have and move over to
> something totally different all in one go, but there clearly *is* benefit
> in your proposal over the existing scheme.

I can see that's probably not practical. :(

Do we know what we're going to do w.r.t. IORT? That's going to require
the kernel to be able to handle a similar description to this proposal.

Mark.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Mark Rutland]

On Fri, Jun 05, 2015 at 10:05:34AM +0100, Will Deacon wrote:
> On Thu, Jun 04, 2015 at 11:19:30PM +0100, Chalamarla, Tirumalesh wrote:
> > > On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> > > It's possible to specify that the paths exist. I expect that software
> > > would select which to use at runtime.
> > > 
> > My worry is how to define any priorities/preferences between masters. 
> > in general the proposal looks reasonable.
> 
> I agree that the proposal looks reasonable (in terms of the ability to
> describe the sort of topologies that we will face) but I still don't
> understand what I need to do in e.g. my IOMMU driver to support this
> binding whilst continuing to support the existing iommus binding, which
> is relied upon to configure dma-mapping.
> 
> Mark: how do you see this co-existing/merging with the current bindings?

As I mentioned in my initial mail, it's not clear to me how this can be
reconciled with the current bindings. Everything I've been able to come
up with so far at best ends up describing the same thing repeatedly.

I'll see what I can come up with. Any sugestions are welcome!

> I don't think it's practical to throw away what we have and move over to
> something totally different all in one go, but there clearly *is* benefit
> in your proposal over the existing scheme.

I can see that's probably not practical. :(

Do we know what we're going to do w.r.t. IORT? That's going to require
the kernel to be able to handle a similar description to this proposal.

Mark.

Re: Master-aware devices and sideband ID data

[Will Deacon]

On Tue, Jun 09, 2015 at 11:17:54AM +0100, Mark Rutland wrote:
> On Fri, Jun 05, 2015 at 10:05:34AM +0100, Will Deacon wrote:
> > On Thu, Jun 04, 2015 at 11:19:30PM +0100, Chalamarla, Tirumalesh wrote:
> > > > On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> > > > It's possible to specify that the paths exist. I expect that software
> > > > would select which to use at runtime.
> > > > 
> > > My worry is how to define any priorities/preferences between masters. 
> > > in general the proposal looks reasonable.
> > 
> > I agree that the proposal looks reasonable (in terms of the ability to
> > describe the sort of topologies that we will face) but I still don't
> > understand what I need to do in e.g. my IOMMU driver to support this
> > binding whilst continuing to support the existing iommus binding, which
> > is relied upon to configure dma-mapping.
> > 
> > Mark: how do you see this co-existing/merging with the current bindings?
> 
> As I mentioned in my initial mail, it's not clear to me how this can be
> reconciled with the current bindings. Everything I've been able to come
> up with so far at best ends up describing the same thing repeatedly.
> 
> I'll see what I can come up with. Any sugestions are welcome!

Well, I'd start with the current iommu binding and see if you can figure
out which properties need adding to it in order to describe the sort of
topologies you can describe with your original proposal.

> > I don't think it's practical to throw away what we have and move over to
> > something totally different all in one go, but there clearly *is* benefit
> > in your proposal over the existing scheme.
> 
> I can see that's probably not practical. :(
> 
> Do we know what we're going to do w.r.t. IORT? That's going to require
> the kernel to be able to handle a similar description to this proposal.

IORT is part of ACPI, so it doesn't care about the existing DT binding
and doesn't need to co-exist at runtime.

Will

Master-aware devices and sideband ID data

[Will Deacon]

On Tue, Jun 09, 2015 at 11:17:54AM +0100, Mark Rutland wrote:
> On Fri, Jun 05, 2015 at 10:05:34AM +0100, Will Deacon wrote:
> > On Thu, Jun 04, 2015 at 11:19:30PM +0100, Chalamarla, Tirumalesh wrote:
> > > > On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> > > > It's possible to specify that the paths exist. I expect that software
> > > > would select which to use at runtime.
> > > > 
> > > My worry is how to define any priorities/preferences between masters. 
> > > in general the proposal looks reasonable.
> > 
> > I agree that the proposal looks reasonable (in terms of the ability to
> > describe the sort of topologies that we will face) but I still don't
> > understand what I need to do in e.g. my IOMMU driver to support this
> > binding whilst continuing to support the existing iommus binding, which
> > is relied upon to configure dma-mapping.
> > 
> > Mark: how do you see this co-existing/merging with the current bindings?
> 
> As I mentioned in my initial mail, it's not clear to me how this can be
> reconciled with the current bindings. Everything I've been able to come
> up with so far at best ends up describing the same thing repeatedly.
> 
> I'll see what I can come up with. Any sugestions are welcome!

Well, I'd start with the current iommu binding and see if you can figure
out which properties need adding to it in order to describe the sort of
topologies you can describe with your original proposal.

> > I don't think it's practical to throw away what we have and move over to
> > something totally different all in one go, but there clearly *is* benefit
> > in your proposal over the existing scheme.
> 
> I can see that's probably not practical. :(
> 
> Do we know what we're going to do w.r.t. IORT? That's going to require
> the kernel to be able to handle a similar description to this proposal.

IORT is part of ACPI, so it doesn't care about the existing DT binding
and doesn't need to co-exist at runtime.

Will

Re: Master-aware devices and sideband ID data

[Chalamarla, Tirumalesh]

[-- Attachment #1.1: Type: text/plain, Size: 2141 bytes --]

any update on this?
On Jun 10, 2015, at 1:11 AM, Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org<mailto:will.deacon-5wv7dgnIgG8@public.gmane.org>> wrote:

On Tue, Jun 09, 2015 at 11:17:54AM +0100, Mark Rutland wrote:
On Fri, Jun 05, 2015 at 10:05:34AM +0100, Will Deacon wrote:
On Thu, Jun 04, 2015 at 11:19:30PM +0100, Chalamarla, Tirumalesh wrote:
On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org<mailto:mark.rutland-5wv7dgnIgG8@public.gmane.org>> wrote:
It's possible to specify that the paths exist. I expect that software
would select which to use at runtime.

My worry is how to define any priorities/preferences between masters.
in general the proposal looks reasonable.

I agree that the proposal looks reasonable (in terms of the ability to
describe the sort of topologies that we will face) but I still don't
understand what I need to do in e.g. my IOMMU driver to support this
binding whilst continuing to support the existing iommus binding, which
is relied upon to configure dma-mapping.

Mark: how do you see this co-existing/merging with the current bindings?

As I mentioned in my initial mail, it's not clear to me how this can be
reconciled with the current bindings. Everything I've been able to come
up with so far at best ends up describing the same thing repeatedly.

I'll see what I can come up with. Any sugestions are welcome!

Well, I'd start with the current iommu binding and see if you can figure
out which properties need adding to it in order to describe the sort of
topologies you can describe with your original proposal.

I don't think it's practical to throw away what we have and move over to
something totally different all in one go, but there clearly *is* benefit
in your proposal over the existing scheme.

I can see that's probably not practical. :(

Do we know what we're going to do w.r.t. IORT? That's going to require
the kernel to be able to handle a similar description to this proposal.

IORT is part of ACPI, so it doesn't care about the existing DT binding
and doesn't need to co-exist at runtime.

Will


[-- Attachment #1.2: Type: text/html, Size: 9939 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Master-aware devices and sideband ID data

[Mark Rutland]

On Tue, Jun 09, 2015 at 11:17:54AM +0100, Mark Rutland wrote:
> On Fri, Jun 05, 2015 at 10:05:34AM +0100, Will Deacon wrote:
> > On Thu, Jun 04, 2015 at 11:19:30PM +0100, Chalamarla, Tirumalesh wrote:
> > > > On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org> wrote:
> > > > It's possible to specify that the paths exist. I expect that software
> > > > would select which to use at runtime.
> > > > 
> > > My worry is how to define any priorities/preferences between masters. 
> > > in general the proposal looks reasonable.
> > 
> > I agree that the proposal looks reasonable (in terms of the ability to
> > describe the sort of topologies that we will face) but I still don't
> > understand what I need to do in e.g. my IOMMU driver to support this
> > binding whilst continuing to support the existing iommus binding, which
> > is relied upon to configure dma-mapping.
> > 
> > Mark: how do you see this co-existing/merging with the current bindings?
> 
> As I mentioned in my initial mail, it's not clear to me how this can be
> reconciled with the current bindings. Everything I've been able to come
> up with so far at best ends up describing the same thing repeatedly.
> 
> I'll see what I can come up with. Any sugestions are welcome!

I can't see a way of keeping the ID transformations explicit with the
existing bindings, but I think we can simply fold these down into
properties in the master nodes, given we expect each ID to be derived
from some initial master ID anyway.

So, to cater for the ITS we would need to pass master IDs along with the
MSI parent information, which we could do by extending msi-parent or by
introducing a new msis property which behaves similarly to the iommus
property, describing the MSI controllers the device can address (via any
IOMMUs), along with any controller-specific identification data.

Which means we'd have DT fragments like the following for an arbitrary
platform device:

its0: its {
	...
	msi-controller;
	#msi-cells = <1>; // DeviceId
};

its1: its {
	...
	msi-controller;
	#msi-cells = <1>; // DeviceId
};

smmu: smmu {
	...
	iommu-cells = <1>; // StreamId
};

device {
	...
	iommus = <&its 0>;
	/* Can use either ITS, but has a different ID at each */
	msis = <&its0 0x0>, <&its1 0x400>;
};

That doesn't allow you to describe a device with multiple mater ports
where each master port might want to generate MSIs, but I'm not sure if
that's a real case.

For PCIe root complexes, we'd need to describe the BDF -> iommu-cells
and BDF -> msi-cells translations separately with new properties on the
node for the root complex itself.

Is there anything obviously broken with the above approach?

Thanks,
Mark.

Master-aware devices and sideband ID data

[Mark Rutland]

On Tue, Jun 09, 2015 at 11:17:54AM +0100, Mark Rutland wrote:
> On Fri, Jun 05, 2015 at 10:05:34AM +0100, Will Deacon wrote:
> > On Thu, Jun 04, 2015 at 11:19:30PM +0100, Chalamarla, Tirumalesh wrote:
> > > > On Jun 1, 2015, at 3:22 AM, Mark Rutland <mark.rutland@arm.com> wrote:
> > > > It's possible to specify that the paths exist. I expect that software
> > > > would select which to use at runtime.
> > > > 
> > > My worry is how to define any priorities/preferences between masters. 
> > > in general the proposal looks reasonable.
> > 
> > I agree that the proposal looks reasonable (in terms of the ability to
> > describe the sort of topologies that we will face) but I still don't
> > understand what I need to do in e.g. my IOMMU driver to support this
> > binding whilst continuing to support the existing iommus binding, which
> > is relied upon to configure dma-mapping.
> > 
> > Mark: how do you see this co-existing/merging with the current bindings?
> 
> As I mentioned in my initial mail, it's not clear to me how this can be
> reconciled with the current bindings. Everything I've been able to come
> up with so far at best ends up describing the same thing repeatedly.
> 
> I'll see what I can come up with. Any sugestions are welcome!

I can't see a way of keeping the ID transformations explicit with the
existing bindings, but I think we can simply fold these down into
properties in the master nodes, given we expect each ID to be derived
from some initial master ID anyway.

So, to cater for the ITS we would need to pass master IDs along with the
MSI parent information, which we could do by extending msi-parent or by
introducing a new msis property which behaves similarly to the iommus
property, describing the MSI controllers the device can address (via any
IOMMUs), along with any controller-specific identification data.

Which means we'd have DT fragments like the following for an arbitrary
platform device:

its0: its {
	...
	msi-controller;
	#msi-cells = <1>; // DeviceId
};

its1: its {
	...
	msi-controller;
	#msi-cells = <1>; // DeviceId
};

smmu: smmu {
	...
	iommu-cells = <1>; // StreamId
};

device {
	...
	iommus = <&its 0>;
	/* Can use either ITS, but has a different ID at each */
	msis = <&its0 0x0>, <&its1 0x400>;
};

That doesn't allow you to describe a device with multiple mater ports
where each master port might want to generate MSIs, but I'm not sure if
that's a real case.

For PCIe root complexes, we'd need to describe the BDF -> iommu-cells
and BDF -> msi-cells translations separately with new properties on the
node for the root complex itself.

Is there anything obviously broken with the above approach?

Thanks,
Mark.

Re: Master-aware devices and sideband ID data

[Will Deacon]

On Wed, Jul 08, 2015 at 02:30:50PM +0100, Mark Rutland wrote:
> On Tue, Jun 09, 2015 at 11:17:54AM +0100, Mark Rutland wrote:
> > On Fri, Jun 05, 2015 at 10:05:34AM +0100, Will Deacon wrote:
> > > Mark: how do you see this co-existing/merging with the current bindings?
> > 
> > As I mentioned in my initial mail, it's not clear to me how this can be
> > reconciled with the current bindings. Everything I've been able to come
> > up with so far at best ends up describing the same thing repeatedly.
> > 
> > I'll see what I can come up with. Any sugestions are welcome!
> 
> I can't see a way of keeping the ID transformations explicit with the
> existing bindings, but I think we can simply fold these down into
> properties in the master nodes, given we expect each ID to be derived
> from some initial master ID anyway.
> 
> So, to cater for the ITS we would need to pass master IDs along with the
> MSI parent information, which we could do by extending msi-parent or by
> introducing a new msis property which behaves similarly to the iommus
> property, describing the MSI controllers the device can address (via any
> IOMMUs), along with any controller-specific identification data.
> 
> Which means we'd have DT fragments like the following for an arbitrary
> platform device:
> 
> its0: its {
> 	...
> 	msi-controller;
> 	#msi-cells = <1>; // DeviceId
> };
> 
> its1: its {
> 	...
> 	msi-controller;
> 	#msi-cells = <1>; // DeviceId
> };
> 
> smmu: smmu {
> 	...
> 	iommu-cells = <1>; // StreamId
> };
> 
> device {
> 	...
> 	iommus = <&its 0>;
> 	/* Can use either ITS, but has a different ID at each */
> 	msis = <&its0 0x0>, <&its1 0x400>;
> };
> 
> That doesn't allow you to describe a device with multiple mater ports
> where each master port might want to generate MSIs, but I'm not sure if
> that's a real case.

In this case, I think we'd need something extra to define precisely how
those master ports relate to the rest of the system anyway. That would
likely be a device-specific property, I reckon.

> For PCIe root complexes, we'd need to describe the BDF -> iommu-cells
> and BDF -> msi-cells translations separately with new properties on the
> node for the root complex itself.
> 
> Is there anything obviously broken with the above approach?

Works for me. Can you write this up as a binding extension to msi-parent,
please?

Will
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Will Deacon]

On Wed, Jul 08, 2015 at 02:30:50PM +0100, Mark Rutland wrote:
> On Tue, Jun 09, 2015 at 11:17:54AM +0100, Mark Rutland wrote:
> > On Fri, Jun 05, 2015 at 10:05:34AM +0100, Will Deacon wrote:
> > > Mark: how do you see this co-existing/merging with the current bindings?
> > 
> > As I mentioned in my initial mail, it's not clear to me how this can be
> > reconciled with the current bindings. Everything I've been able to come
> > up with so far at best ends up describing the same thing repeatedly.
> > 
> > I'll see what I can come up with. Any sugestions are welcome!
> 
> I can't see a way of keeping the ID transformations explicit with the
> existing bindings, but I think we can simply fold these down into
> properties in the master nodes, given we expect each ID to be derived
> from some initial master ID anyway.
> 
> So, to cater for the ITS we would need to pass master IDs along with the
> MSI parent information, which we could do by extending msi-parent or by
> introducing a new msis property which behaves similarly to the iommus
> property, describing the MSI controllers the device can address (via any
> IOMMUs), along with any controller-specific identification data.
> 
> Which means we'd have DT fragments like the following for an arbitrary
> platform device:
> 
> its0: its {
> 	...
> 	msi-controller;
> 	#msi-cells = <1>; // DeviceId
> };
> 
> its1: its {
> 	...
> 	msi-controller;
> 	#msi-cells = <1>; // DeviceId
> };
> 
> smmu: smmu {
> 	...
> 	iommu-cells = <1>; // StreamId
> };
> 
> device {
> 	...
> 	iommus = <&its 0>;
> 	/* Can use either ITS, but has a different ID at each */
> 	msis = <&its0 0x0>, <&its1 0x400>;
> };
> 
> That doesn't allow you to describe a device with multiple mater ports
> where each master port might want to generate MSIs, but I'm not sure if
> that's a real case.

In this case, I think we'd need something extra to define precisely how
those master ports relate to the rest of the system anyway. That would
likely be a device-specific property, I reckon.

> For PCIe root complexes, we'd need to describe the BDF -> iommu-cells
> and BDF -> msi-cells translations separately with new properties on the
> node for the root complex itself.
> 
> Is there anything obviously broken with the above approach?

Works for me. Can you write this up as a binding extension to msi-parent,
please?

Will

Re: Master-aware devices and sideband ID data

[Mark Rutland]

Hi Will,

The below is an attempt at an MSI binding, derived from my original
example. It extends msi-parent inoto a phandle+(optional args) style
property.

I haven't yet managed to come up with a sane way of describing
the Bus-ID/BDF -> {iommu,msi}-cells translation, but this should
hopefully cover the platform device case.

For the Bus-ID translation case I'm not sure it's sane to attempt to use
msi-parent, given that the transformation description will necessarily
have to describe the parents anyway.

Thanks,
Mark.

---->8----
>From 429dca4bba98732c492e95bdf395aa2ccc634e69 Mon Sep 17 00:00:00 2001
From: Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org>
Date: Thu, 9 Jul 2015 17:53:00 +0100
Subject: [PATCH] Documentation: dt: add generic MSI bindings

Currently msi-parent is in use in a couple of drviers despite being
fairly underspecified. This patch adds a generic binding for MSIs
(including the existing msi-parent property) enabling the description of
platform devices capable of using MSIs.

This binding does not yet cover the general case. Currently the binding
does not cover the relationship between bus IDs (e.g. PCIe BDF) and
sideband data.

Signed-off-by: Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org>
---
 .../bindings/interrupt-controller/msi.txt          | 135 +++++++++++++++++++++
 1 file changed, 135 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt

diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
new file mode 100644
index 0000000..c60c034
--- /dev/null
+++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
@@ -0,0 +1,135 @@
+This document describes the generic device tree binding for MSI controllers and
+their master(s).
+
+Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
+write to an MMIO address.
+
+MSIs were originally specified by PCI (and are used with PCIe), but may also be
+used with other busses, and hence a mechanism is required to relate devices on
+those busses to the MSI controllers which they are capable of using,
+potentially including additional information.
+
+MSIs are distinguished by some combination of:
+
+- The doorbell (the MMIO address written to).
+  
+  Devices may be configured by software to write to arbitrary doorbells which
+  they can address. An MSI controller may feature a number of doorbells.
+
+- The payload (the value written to the doorbell).
+  
+  Devices may be configured to write an arbitrary payload chosen by software.
+  MSI controllers may have restrictions on permitted payloads.
+
+- Sideband information accompanying the write.
+  
+  Typically this is neither configurable nor probeable, and depends on the path
+  taken through the memory system (i.e. it is a property of the combination of
+  MSI controller and device rather than a property of either in isolation).
+
+
+MSI controllers:
+================
+
+An MSI controller signals interrupts to a CPU when a write is made to an MMIO
+address by some master. An MSI controller may feature a number of doorbells.
+
+Required properties:
+--------------------
+
+- msi-controller: Identifies the node as an MSI controller.
+
+Optional properties:
+--------------------
+
+- #msi-cells: The number of cells in an msi-specifier, required if not zero.
+
+  Typically this will encode information related to sideband data, and will
+  not encode doorbells or payloads as these can be configured dynamically.
+
+  The meaning of the msi-specifier is defined by the device tree binding of
+  the specific MSI controller. 
+
+
+MSI clients
+===========
+
+MSI clients are devices which generate MSIs. For each MSI they wish to
+generate, the doorbell and payload may be configured, though sideband
+information may not be configurable.
+
+Required properties:
+--------------------
+
+- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
+  controller which the device is capable of using.
+
+  This property is unordered, and MSIs may be allocated from any combination of
+  MSI controllers listed in the msi-parent property.
+
+  If a device has restrictions on the allocation of MSIs, these restrictions
+  must be described with additional properties.
+
+  When #msi-cells is non-zero, busses with an msi-parent will require
+  additional properties to describe the relationship between devices on the bus
+  and the set of MSIs they can potentially generate.
+
+
+Example
+=======
+
+/ {
+	#address-cells = <1>;
+	#size-cells = <1>;
+
+	msi_a: msi-controller@a {
+		reg = <0xa 0xf00>;
+		compatible = "vendor-a,some-controller";
+		msi-controller;
+		/* No sideband data, so #msi-cells omitted */
+	};
+
+	msi_b: msi-controller@b {
+		reg = <0xb 0xf00>;
+		compatible = "vendor-b,another-controller";
+		msi-controller;
+		/* Each device has some unique ID */
+		#msi-cells = <1>;
+	};
+
+	msi_c: msi-controller@c {
+		reg = <0xb 0xf00>;
+		compatible = "vendor-b,another-controller";
+		msi-controller;
+		/* Each device has some unique ID */
+		#msi-cells = <1>;
+	};
+
+	dev@0 {
+		reg = <0x0 0xf00>;
+		compatible = "vendor-c,some-device";
+
+		/* Can only generate MSIs to msi_a */
+		msi-parent = <&msi_a>;
+	};
+
+	dev@1 {
+		reg = <0x1 0xf00>;
+		compatible = "vendor-c,some-device";
+
+		/* 
+		 * Can generate MSIs to either A or B.
+		 */
+		msi-parent = <&msi_a>, <&msi_b 0x17>;
+	};
+
+	dev@2 {
+		reg = <0x2 0xf00>;
+		compatible = "vendor-c,some-device";
+		/*
+		 * Has different IDs at each MSI controller.
+		 * Can generate MSIs to all of the MSI controllers.
+		 */
+		msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
+	};
+};
-- 
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Mark Rutland]

Hi Will,

The below is an attempt at an MSI binding, derived from my original
example. It extends msi-parent inoto a phandle+(optional args) style
property.

I haven't yet managed to come up with a sane way of describing
the Bus-ID/BDF -> {iommu,msi}-cells translation, but this should
hopefully cover the platform device case.

For the Bus-ID translation case I'm not sure it's sane to attempt to use
msi-parent, given that the transformation description will necessarily
have to describe the parents anyway.

Thanks,
Mark.

---->8----
>From 429dca4bba98732c492e95bdf395aa2ccc634e69 Mon Sep 17 00:00:00 2001
From: Mark Rutland <mark.rutland@arm.com>
Date: Thu, 9 Jul 2015 17:53:00 +0100
Subject: [PATCH] Documentation: dt: add generic MSI bindings

Currently msi-parent is in use in a couple of drviers despite being
fairly underspecified. This patch adds a generic binding for MSIs
(including the existing msi-parent property) enabling the description of
platform devices capable of using MSIs.

This binding does not yet cover the general case. Currently the binding
does not cover the relationship between bus IDs (e.g. PCIe BDF) and
sideband data.

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
---
 .../bindings/interrupt-controller/msi.txt          | 135 +++++++++++++++++++++
 1 file changed, 135 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt

diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
new file mode 100644
index 0000000..c60c034
--- /dev/null
+++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
@@ -0,0 +1,135 @@
+This document describes the generic device tree binding for MSI controllers and
+their master(s).
+
+Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
+write to an MMIO address.
+
+MSIs were originally specified by PCI (and are used with PCIe), but may also be
+used with other busses, and hence a mechanism is required to relate devices on
+those busses to the MSI controllers which they are capable of using,
+potentially including additional information.
+
+MSIs are distinguished by some combination of:
+
+- The doorbell (the MMIO address written to).
+  
+  Devices may be configured by software to write to arbitrary doorbells which
+  they can address. An MSI controller may feature a number of doorbells.
+
+- The payload (the value written to the doorbell).
+  
+  Devices may be configured to write an arbitrary payload chosen by software.
+  MSI controllers may have restrictions on permitted payloads.
+
+- Sideband information accompanying the write.
+  
+  Typically this is neither configurable nor probeable, and depends on the path
+  taken through the memory system (i.e. it is a property of the combination of
+  MSI controller and device rather than a property of either in isolation).
+
+
+MSI controllers:
+================
+
+An MSI controller signals interrupts to a CPU when a write is made to an MMIO
+address by some master. An MSI controller may feature a number of doorbells.
+
+Required properties:
+--------------------
+
+- msi-controller: Identifies the node as an MSI controller.
+
+Optional properties:
+--------------------
+
+- #msi-cells: The number of cells in an msi-specifier, required if not zero.
+
+  Typically this will encode information related to sideband data, and will
+  not encode doorbells or payloads as these can be configured dynamically.
+
+  The meaning of the msi-specifier is defined by the device tree binding of
+  the specific MSI controller. 
+
+
+MSI clients
+===========
+
+MSI clients are devices which generate MSIs. For each MSI they wish to
+generate, the doorbell and payload may be configured, though sideband
+information may not be configurable.
+
+Required properties:
+--------------------
+
+- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
+  controller which the device is capable of using.
+
+  This property is unordered, and MSIs may be allocated from any combination of
+  MSI controllers listed in the msi-parent property.
+
+  If a device has restrictions on the allocation of MSIs, these restrictions
+  must be described with additional properties.
+
+  When #msi-cells is non-zero, busses with an msi-parent will require
+  additional properties to describe the relationship between devices on the bus
+  and the set of MSIs they can potentially generate.
+
+
+Example
+=======
+
+/ {
+	#address-cells = <1>;
+	#size-cells = <1>;
+
+	msi_a: msi-controller at a {
+		reg = <0xa 0xf00>;
+		compatible = "vendor-a,some-controller";
+		msi-controller;
+		/* No sideband data, so #msi-cells omitted */
+	};
+
+	msi_b: msi-controller at b {
+		reg = <0xb 0xf00>;
+		compatible = "vendor-b,another-controller";
+		msi-controller;
+		/* Each device has some unique ID */
+		#msi-cells = <1>;
+	};
+
+	msi_c: msi-controller at c {
+		reg = <0xb 0xf00>;
+		compatible = "vendor-b,another-controller";
+		msi-controller;
+		/* Each device has some unique ID */
+		#msi-cells = <1>;
+	};
+
+	dev at 0 {
+		reg = <0x0 0xf00>;
+		compatible = "vendor-c,some-device";
+
+		/* Can only generate MSIs to msi_a */
+		msi-parent = <&msi_a>;
+	};
+
+	dev at 1 {
+		reg = <0x1 0xf00>;
+		compatible = "vendor-c,some-device";
+
+		/* 
+		 * Can generate MSIs to either A or B.
+		 */
+		msi-parent = <&msi_a>, <&msi_b 0x17>;
+	};
+
+	dev at 2 {
+		reg = <0x2 0xf00>;
+		compatible = "vendor-c,some-device";
+		/*
+		 * Has different IDs at each MSI controller.
+		 * Can generate MSIs to all of the MSI controllers.
+		 */
+		msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
+	};
+};
-- 
1.9.1

Re: Master-aware devices and sideband ID data

[Will Deacon]

On Thu, Jul 16, 2015 at 02:34:41PM +0100, Mark Rutland wrote:
> Hi Will,

Hi Mark,

[adding David, since he's working on PCI/ITS stuff atm]

> The below is an attempt at an MSI binding, derived from my original
> example. It extends msi-parent inoto a phandle+(optional args) style
> property.
> 
> I haven't yet managed to come up with a sane way of describing
> the Bus-ID/BDF -> {iommu,msi}-cells translation, but this should
> hopefully cover the platform device case.
> 
> For the Bus-ID translation case I'm not sure it's sane to attempt to use
> msi-parent, given that the transformation description will necessarily
> have to describe the parents anyway.

We probably want a separate property on the RC node describing the
transformation (i.e. offsetting) in a similar way to ranges or
interrupt-map (potentially replacing msi-parent altogether). Maybe you
could propose something so that David could have a crack at describing
his RequesterID -> DeviceID mapping, which seems to map nicely onto a
per-RC offset IIUC.

David -- does that sound ok to you?

Will

> ---->8----
> From 429dca4bba98732c492e95bdf395aa2ccc634e69 Mon Sep 17 00:00:00 2001
> From: Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org>
> Date: Thu, 9 Jul 2015 17:53:00 +0100
> Subject: [PATCH] Documentation: dt: add generic MSI bindings
> 
> Currently msi-parent is in use in a couple of drviers despite being
> fairly underspecified. This patch adds a generic binding for MSIs
> (including the existing msi-parent property) enabling the description of
> platform devices capable of using MSIs.
> 
> This binding does not yet cover the general case. Currently the binding
> does not cover the relationship between bus IDs (e.g. PCIe BDF) and
> sideband data.
> 
> Signed-off-by: Mark Rutland <mark.rutland-5wv7dgnIgG8@public.gmane.org>
> ---
>  .../bindings/interrupt-controller/msi.txt          | 135 +++++++++++++++++++++
>  1 file changed, 135 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt
> 
> diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> new file mode 100644
> index 0000000..c60c034
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> @@ -0,0 +1,135 @@
> +This document describes the generic device tree binding for MSI controllers and
> +their master(s).
> +
> +Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
> +write to an MMIO address.
> +
> +MSIs were originally specified by PCI (and are used with PCIe), but may also be
> +used with other busses, and hence a mechanism is required to relate devices on
> +those busses to the MSI controllers which they are capable of using,
> +potentially including additional information.
> +
> +MSIs are distinguished by some combination of:
> +
> +- The doorbell (the MMIO address written to).
> +  
> +  Devices may be configured by software to write to arbitrary doorbells which
> +  they can address. An MSI controller may feature a number of doorbells.
> +
> +- The payload (the value written to the doorbell).
> +  
> +  Devices may be configured to write an arbitrary payload chosen by software.
> +  MSI controllers may have restrictions on permitted payloads.
> +
> +- Sideband information accompanying the write.
> +  
> +  Typically this is neither configurable nor probeable, and depends on the path
> +  taken through the memory system (i.e. it is a property of the combination of
> +  MSI controller and device rather than a property of either in isolation).
> +
> +
> +MSI controllers:
> +================
> +
> +An MSI controller signals interrupts to a CPU when a write is made to an MMIO
> +address by some master. An MSI controller may feature a number of doorbells.
> +
> +Required properties:
> +--------------------
> +
> +- msi-controller: Identifies the node as an MSI controller.
> +
> +Optional properties:
> +--------------------
> +
> +- #msi-cells: The number of cells in an msi-specifier, required if not zero.
> +
> +  Typically this will encode information related to sideband data, and will
> +  not encode doorbells or payloads as these can be configured dynamically.
> +
> +  The meaning of the msi-specifier is defined by the device tree binding of
> +  the specific MSI controller. 
> +
> +
> +MSI clients
> +===========
> +
> +MSI clients are devices which generate MSIs. For each MSI they wish to
> +generate, the doorbell and payload may be configured, though sideband
> +information may not be configurable.
> +
> +Required properties:
> +--------------------
> +
> +- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
> +  controller which the device is capable of using.
> +
> +  This property is unordered, and MSIs may be allocated from any combination of
> +  MSI controllers listed in the msi-parent property.
> +
> +  If a device has restrictions on the allocation of MSIs, these restrictions
> +  must be described with additional properties.
> +
> +  When #msi-cells is non-zero, busses with an msi-parent will require
> +  additional properties to describe the relationship between devices on the bus
> +  and the set of MSIs they can potentially generate.
> +
> +
> +Example
> +=======
> +
> +/ {
> +	#address-cells = <1>;
> +	#size-cells = <1>;
> +
> +	msi_a: msi-controller@a {
> +		reg = <0xa 0xf00>;
> +		compatible = "vendor-a,some-controller";
> +		msi-controller;
> +		/* No sideband data, so #msi-cells omitted */
> +	};
> +
> +	msi_b: msi-controller@b {
> +		reg = <0xb 0xf00>;
> +		compatible = "vendor-b,another-controller";
> +		msi-controller;
> +		/* Each device has some unique ID */
> +		#msi-cells = <1>;
> +	};
> +
> +	msi_c: msi-controller@c {
> +		reg = <0xb 0xf00>;
> +		compatible = "vendor-b,another-controller";
> +		msi-controller;
> +		/* Each device has some unique ID */
> +		#msi-cells = <1>;
> +	};
> +
> +	dev@0 {
> +		reg = <0x0 0xf00>;
> +		compatible = "vendor-c,some-device";
> +
> +		/* Can only generate MSIs to msi_a */
> +		msi-parent = <&msi_a>;
> +	};
> +
> +	dev@1 {
> +		reg = <0x1 0xf00>;
> +		compatible = "vendor-c,some-device";
> +
> +		/* 
> +		 * Can generate MSIs to either A or B.
> +		 */
> +		msi-parent = <&msi_a>, <&msi_b 0x17>;
> +	};
> +
> +	dev@2 {
> +		reg = <0x2 0xf00>;
> +		compatible = "vendor-c,some-device";
> +		/*
> +		 * Has different IDs at each MSI controller.
> +		 * Can generate MSIs to all of the MSI controllers.
> +		 */
> +		msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
> +	};
> +};
> -- 
> 1.9.1
> 
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Master-aware devices and sideband ID data

[Will Deacon]

On Thu, Jul 16, 2015 at 02:34:41PM +0100, Mark Rutland wrote:
> Hi Will,

Hi Mark,

[adding David, since he's working on PCI/ITS stuff atm]

> The below is an attempt at an MSI binding, derived from my original
> example. It extends msi-parent inoto a phandle+(optional args) style
> property.
> 
> I haven't yet managed to come up with a sane way of describing
> the Bus-ID/BDF -> {iommu,msi}-cells translation, but this should
> hopefully cover the platform device case.
> 
> For the Bus-ID translation case I'm not sure it's sane to attempt to use
> msi-parent, given that the transformation description will necessarily
> have to describe the parents anyway.

We probably want a separate property on the RC node describing the
transformation (i.e. offsetting) in a similar way to ranges or
interrupt-map (potentially replacing msi-parent altogether). Maybe you
could propose something so that David could have a crack at describing
his RequesterID -> DeviceID mapping, which seems to map nicely onto a
per-RC offset IIUC.

David -- does that sound ok to you?

Will

> ---->8----
> From 429dca4bba98732c492e95bdf395aa2ccc634e69 Mon Sep 17 00:00:00 2001
> From: Mark Rutland <mark.rutland@arm.com>
> Date: Thu, 9 Jul 2015 17:53:00 +0100
> Subject: [PATCH] Documentation: dt: add generic MSI bindings
> 
> Currently msi-parent is in use in a couple of drviers despite being
> fairly underspecified. This patch adds a generic binding for MSIs
> (including the existing msi-parent property) enabling the description of
> platform devices capable of using MSIs.
> 
> This binding does not yet cover the general case. Currently the binding
> does not cover the relationship between bus IDs (e.g. PCIe BDF) and
> sideband data.
> 
> Signed-off-by: Mark Rutland <mark.rutland@arm.com>
> ---
>  .../bindings/interrupt-controller/msi.txt          | 135 +++++++++++++++++++++
>  1 file changed, 135 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/msi.txt
> 
> diff --git a/Documentation/devicetree/bindings/interrupt-controller/msi.txt b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> new file mode 100644
> index 0000000..c60c034
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/interrupt-controller/msi.txt
> @@ -0,0 +1,135 @@
> +This document describes the generic device tree binding for MSI controllers and
> +their master(s).
> +
> +Message Signaled Interrupts (MSIs) are a class of interrupts generated by a
> +write to an MMIO address.
> +
> +MSIs were originally specified by PCI (and are used with PCIe), but may also be
> +used with other busses, and hence a mechanism is required to relate devices on
> +those busses to the MSI controllers which they are capable of using,
> +potentially including additional information.
> +
> +MSIs are distinguished by some combination of:
> +
> +- The doorbell (the MMIO address written to).
> +  
> +  Devices may be configured by software to write to arbitrary doorbells which
> +  they can address. An MSI controller may feature a number of doorbells.
> +
> +- The payload (the value written to the doorbell).
> +  
> +  Devices may be configured to write an arbitrary payload chosen by software.
> +  MSI controllers may have restrictions on permitted payloads.
> +
> +- Sideband information accompanying the write.
> +  
> +  Typically this is neither configurable nor probeable, and depends on the path
> +  taken through the memory system (i.e. it is a property of the combination of
> +  MSI controller and device rather than a property of either in isolation).
> +
> +
> +MSI controllers:
> +================
> +
> +An MSI controller signals interrupts to a CPU when a write is made to an MMIO
> +address by some master. An MSI controller may feature a number of doorbells.
> +
> +Required properties:
> +--------------------
> +
> +- msi-controller: Identifies the node as an MSI controller.
> +
> +Optional properties:
> +--------------------
> +
> +- #msi-cells: The number of cells in an msi-specifier, required if not zero.
> +
> +  Typically this will encode information related to sideband data, and will
> +  not encode doorbells or payloads as these can be configured dynamically.
> +
> +  The meaning of the msi-specifier is defined by the device tree binding of
> +  the specific MSI controller. 
> +
> +
> +MSI clients
> +===========
> +
> +MSI clients are devices which generate MSIs. For each MSI they wish to
> +generate, the doorbell and payload may be configured, though sideband
> +information may not be configurable.
> +
> +Required properties:
> +--------------------
> +
> +- msi-parent: A list of phandle + msi-specifier pairs, one for each MSI
> +  controller which the device is capable of using.
> +
> +  This property is unordered, and MSIs may be allocated from any combination of
> +  MSI controllers listed in the msi-parent property.
> +
> +  If a device has restrictions on the allocation of MSIs, these restrictions
> +  must be described with additional properties.
> +
> +  When #msi-cells is non-zero, busses with an msi-parent will require
> +  additional properties to describe the relationship between devices on the bus
> +  and the set of MSIs they can potentially generate.
> +
> +
> +Example
> +=======
> +
> +/ {
> +	#address-cells = <1>;
> +	#size-cells = <1>;
> +
> +	msi_a: msi-controller at a {
> +		reg = <0xa 0xf00>;
> +		compatible = "vendor-a,some-controller";
> +		msi-controller;
> +		/* No sideband data, so #msi-cells omitted */
> +	};
> +
> +	msi_b: msi-controller at b {
> +		reg = <0xb 0xf00>;
> +		compatible = "vendor-b,another-controller";
> +		msi-controller;
> +		/* Each device has some unique ID */
> +		#msi-cells = <1>;
> +	};
> +
> +	msi_c: msi-controller at c {
> +		reg = <0xb 0xf00>;
> +		compatible = "vendor-b,another-controller";
> +		msi-controller;
> +		/* Each device has some unique ID */
> +		#msi-cells = <1>;
> +	};
> +
> +	dev at 0 {
> +		reg = <0x0 0xf00>;
> +		compatible = "vendor-c,some-device";
> +
> +		/* Can only generate MSIs to msi_a */
> +		msi-parent = <&msi_a>;
> +	};
> +
> +	dev at 1 {
> +		reg = <0x1 0xf00>;
> +		compatible = "vendor-c,some-device";
> +
> +		/* 
> +		 * Can generate MSIs to either A or B.
> +		 */
> +		msi-parent = <&msi_a>, <&msi_b 0x17>;
> +	};
> +
> +	dev at 2 {
> +		reg = <0x2 0xf00>;
> +		compatible = "vendor-c,some-device";
> +		/*
> +		 * Has different IDs at each MSI controller.
> +		 * Can generate MSIs to all of the MSI controllers.
> +		 */
> +		msi-parent = <&msi_a>, <&msi_b 0x17>, <&msi_c 0x53>;
> +	};
> +};
> -- 
> 1.9.1
>