[RFC] DT affinity bindings/representing bus masters with DT

[RFC] DT affinity bindings/representing bus masters with DT

[Lorenzo Pieralisi]

Hi all,

in order to come up with a solid solution to the affinity bindings concept
we are facing in the ARM world, I am posting this RFC so that hopefully people
on the list can chime in and help us in this endeavour.

I tried to keep things simple on purpose and some statements are a bit of an
oversimplification, we can elaborate on those if needed.

Hereafter a summary of what we are trying to achieve.

Current device tree bindings allow to describe HW configurations of systems
in a bus like fashion, where each bus contains bus slaves and mapping
heuristics to translate address spaces across bus layers (AMBA -> PCI).

The device tree by definition represents the view of the system from the
perspective of the CPU(s). This means that all devices (but CPUs) present
in the device tree are to be seen as "slave" components, ie devices sitting on
a bus and accessible from the CPU with an addressing mode that can vary and it
is defined by the bus the device is sitting on.

There are specific cases in current SoCs though where resources belonging to
a slave device should be linked to a master in the SoC system hierarchy.

To the best of my knowledge, the mechanism used to implement this linkage is
not defined by any device tree binding; it probably can be a phandle in the
device tree syntax, but this has some implications that will be described as
follows.

A programmable device, let's call it "foo" for the sake of this discussion,
has several resources (ie memory spaces) that should be mapped to bus masters
in a SoC hierarchy (each resource "belongs" to a master). The only way this
can be currently done through a device tree is by linking the resource in
question to a device representing the master node through a phandle (keeping
in mind that the concept of master node does not exist in current DT
bindings).

An example is worth a thousand words (pseudo dts file):

/ {
	#address-cells = 1;
	#size-cells = 1;

	acme1: acme@4000000 {
		reg = <0x4000000 0x1000>;
	};

	acme2: acme@5000000 {
		reg = <0x5000000 0x1000>;
	};

	foo@2000000 {
		reg = <0x2000000 0x1000
		       0x2001000 0x1000>;
		affinity = <&acme1 &acme2>;
	};
};

where the "affinity" property contains a number of phandles equal to the number
of resources (ie reg properties) in the foo@2000000 node. The "affinity"
property maps a reg property to a device tree node, one phandle per "reg"
property.

acme1 and acme2 are two bus masters in the system (eg a DMA and a GPU).

Each foo@2000000 reg property maps to a device that represents a bus master
(to make it clearer, a foo@2000000 reg property defines an address space that
belongs to a bus master, ie the address space represents a programming
interface specific to that master; in the bindings above address 0x2000000 is
the address at which acme1 device can programme its "foo" interface, address
0x2001000 is the address at which acme2 device can programme its "foo"
interface).

Now, the dts above links, through a phandle, a reg property to a "slave"
device not a "master" device, basically to a device tree node representing the
acme1 device programming interface (address 0x4000000 is the register space of
the acme1 device, which is used to describe that device in the tree and
represents its address space, its "slave interface").

The approach above has two drawbacks:

1 - we are using a slave device node to map a resource to a master device.
    Since there is no notion of "master" device in the current device tree
    bindings that's our best bet. The approach above works as long as there
    is a node in the device tree describing the "master" node (through its
    slave interface), even if it does not have a programming interface (ie it
    cannot be described in the device tree as a memory mapped/addressable
    device). This is also the approach taken by the dma bindings described in

    Documentation/devicetree/bindings/dma/dma.txt (currently in linux-next)

2 - By connecting a device's resource to a slave device, to emulate a resource
    to a bus master connection, we are implicitly taking the assumption that
    the address space where both eg foo and acme1 devices above reside (acme1
    programming interface) is the same as the one seen by the acme1 master
    device (which has no representation in the device tree). This is not a
    problem for now, but the device tree representation is questionable and
    we are inquiring on possible issues this might create.

Any bits of advice is welcome on the subject, and the problem can be
elaborated with further examples to kickstart discussions.

Thank you very much for your feedback,
Lorenzo

Re: [RFC] DT affinity bindings/representing bus masters with DT

[Dave Martin]

On Fri, Feb 15, 2013 at 05:21:02PM +0000, Lorenzo Pieralisi wrote:
> Hi all,
> 
> in order to come up with a solid solution to the affinity bindings concept
> we are facing in the ARM world, I am posting this RFC so that hopefully people
> on the list can chime in and help us in this endeavour.
> 
> I tried to keep things simple on purpose and some statements are a bit of an
> oversimplification, we can elaborate on those if needed.
> 
> Hereafter a summary of what we are trying to achieve.
> 
> Current device tree bindings allow to describe HW configurations of systems
> in a bus like fashion, where each bus contains bus slaves and mapping
> heuristics to translate address spaces across bus layers (AMBA -> PCI).
> 
> The device tree by definition represents the view of the system from the
> perspective of the CPU(s). This means that all devices (but CPUs) present
> in the device tree are to be seen as "slave" components, ie devices sitting on
> a bus and accessible from the CPU with an addressing mode that can vary and it
> is defined by the bus the device is sitting on.
> 
> There are specific cases in current SoCs though where resources belonging to
> a slave device should be linked to a master in the SoC system hierarchy.
> 
> To the best of my knowledge, the mechanism used to implement this linkage is
> not defined by any device tree binding; it probably can be a phandle in the
> device tree syntax, but this has some implications that will be described as
> follows.
> 
> A programmable device, let's call it "foo" for the sake of this discussion,
> has several resources (ie memory spaces) that should be mapped to bus masters
> in a SoC hierarchy (each resource "belongs" to a master). The only way this
> can be currently done through a device tree is by linking the resource in
> question to a device representing the master node through a phandle (keeping
> in mind that the concept of master node does not exist in current DT
> bindings).
> 
> An example is worth a thousand words (pseudo dts file):
> 
> / {
> 	#address-cells = 1;
> 	#size-cells = 1;
> 
> 	acme1: acme@4000000 {
> 		reg = <0x4000000 0x1000>;
> 	};
> 
> 	acme2: acme@5000000 {
> 		reg = <0x5000000 0x1000>;
> 	};
> 
> 	foo@2000000 {
> 		reg = <0x2000000 0x1000
> 		       0x2001000 0x1000>;
> 		affinity = <&acme1 &acme2>;
> 	};
> };
> 
> where the "affinity" property contains a number of phandles equal to the number
> of resources (ie reg properties) in the foo@2000000 node. The "affinity"
> property maps a reg property to a device tree node, one phandle per "reg"
> property.

Maybe we should avoid the word "affinity".  We know what this means with
respct to devices symmetric across multiple CPUs (though the illusion of
symmetry is often less complete than we'd like).

In other contexts, we might just get confused about what this word means
(I do, anyway).

> acme1 and acme2 are two bus masters in the system (eg a DMA and a GPU).
> 
> Each foo@2000000 reg property maps to a device that represents a bus master
> (to make it clearer, a foo@2000000 reg property defines an address space that
> belongs to a bus master, ie the address space represents a programming
> interface specific to that master; in the bindings above address 0x2000000 is
> the address at which acme1 device can programme its "foo" interface, address
> 0x2001000 is the address at which acme2 device can programme its "foo"
> interface).
> 
> Now, the dts above links, through a phandle, a reg property to a "slave"
> device not a "master" device, basically to a device tree node representing the
> acme1 device programming interface (address 0x4000000 is the register space of
> the acme1 device, which is used to describe that device in the tree and
> represents its address space, its "slave interface").
> 
> The approach above has two drawbacks:
> 
> 1 - we are using a slave device node to map a resource to a master device.
>     Since there is no notion of "master" device in the current device tree
>     bindings that's our best bet. The approach above works as long as there
>     is a node in the device tree describing the "master" node (through its
>     slave interface), even if it does not have a programming interface (ie it
>     cannot be described in the device tree as a memory mapped/addressable
>     device). This is also the approach taken by the dma bindings described in
> 
>     Documentation/devicetree/bindings/dma/dma.txt (currently in linux-next)
> 
> 2 - By connecting a device's resource to a slave device, to emulate a resource
>     to a bus master connection, we are implicitly taking the assumption that
>     the address space where both eg foo and acme1 devices above reside (acme1
>     programming interface) is the same as the one seen by the acme1 master
>     device (which has no representation in the device tree). This is not a
>     problem for now, but the device tree representation is questionable and
>     we are inquiring on possible issues this might create.
> 
> Any bits of advice is welcome on the subject, and the problem can be
> elaborated with further examples to kickstart discussions.

Every master potentially has its own address space -- a bus
(either hypothetical or real) which maps the accessible slaves.  This
bus might or might not be shared with other masters in the system.

If could help illustrate the potential issues if we can sketch a system
which we can't adequately describe.

Things which come to mind are:

 * CPU-local and cluster-local peripherals

 * Non-coherent CPUs or microcontrollers

 * Bus masters with slave interfaces (e.g., control interfaces) in a location
   in the bus hierarchy unrelated to where the device makes its master
   accesses, to that we can't guess from the DT how to map addresses
   for those devices (interconnects are an example of this: the CCI
   control register interface is likely to be dangling from an AXI or APB
   bus, topologically distant from the CCI's interconnect and master
   ports, for example)

   Most DMA controllers will have this property too.  For now, we just
   have to guess what view of the system the controller sees in its
   master role.

 * Masters with weird private links (for example, a high-throughput DMA
   controller with its own, non-coherent port on the DRAM controller,
   bypassing the bus through which CPUs and other masters see the DRAM)

We don't necessarily have to solve all of these yet, but they all feel
related.  The fundamental problem is how to describe the fact that
there may be multiple, arbitrarily different, views of the system.

If we can come up with non-tortured ways of describing all these things
in DT, then we don't have a problem...

Cheers
---Dave

Re: [RFC] DT affinity bindings/representing bus masters with DT

[Dave Martin]

On Fri, Feb 15, 2013 at 05:52:06PM +0000, Dave Martin wrote:
> On Fri, Feb 15, 2013 at 05:21:02PM +0000, Lorenzo Pieralisi wrote:
> > Hi all,
> > 
> > in order to come up with a solid solution to the affinity bindings concept
> > we are facing in the ARM world, I am posting this RFC so that hopefully people
> > on the list can chime in and help us in this endeavour.
> > 
> > I tried to keep things simple on purpose and some statements are a bit of an
> > oversimplification, we can elaborate on those if needed.
> > 
> > Hereafter a summary of what we are trying to achieve.
> > 
> > Current device tree bindings allow to describe HW configurations of systems
> > in a bus like fashion, where each bus contains bus slaves and mapping
> > heuristics to translate address spaces across bus layers (AMBA -> PCI).
> > 
> > The device tree by definition represents the view of the system from the
> > perspective of the CPU(s). This means that all devices (but CPUs) present
> > in the device tree are to be seen as "slave" components, ie devices sitting on
> > a bus and accessible from the CPU with an addressing mode that can vary and it
> > is defined by the bus the device is sitting on.
> > 
> > There are specific cases in current SoCs though where resources belonging to
> > a slave device should be linked to a master in the SoC system hierarchy.
> > 
> > To the best of my knowledge, the mechanism used to implement this linkage is
> > not defined by any device tree binding; it probably can be a phandle in the
> > device tree syntax, but this has some implications that will be described as
> > follows.
> > 
> > A programmable device, let's call it "foo" for the sake of this discussion,
> > has several resources (ie memory spaces) that should be mapped to bus masters
> > in a SoC hierarchy (each resource "belongs" to a master). The only way this
> > can be currently done through a device tree is by linking the resource in
> > question to a device representing the master node through a phandle (keeping
> > in mind that the concept of master node does not exist in current DT
> > bindings).
> > 
> > An example is worth a thousand words (pseudo dts file):
> > 
> > / {
> > 	#address-cells = 1;
> > 	#size-cells = 1;
> > 
> > 	acme1: acme@4000000 {
> > 		reg = <0x4000000 0x1000>;
> > 	};
> > 
> > 	acme2: acme@5000000 {
> > 		reg = <0x5000000 0x1000>;
> > 	};
> > 
> > 	foo@2000000 {
> > 		reg = <0x2000000 0x1000
> > 		       0x2001000 0x1000>;
> > 		affinity = <&acme1 &acme2>;
> > 	};
> > };
> > 
> > where the "affinity" property contains a number of phandles equal to the number
> > of resources (ie reg properties) in the foo@2000000 node. The "affinity"
> > property maps a reg property to a device tree node, one phandle per "reg"
> > property.
> 
> Maybe we should avoid the word "affinity".  We know what this means with
> respct to devices symmetric across multiple CPUs (though the illusion of
> symmetry is often less complete than we'd like).
> 
> In other contexts, we might just get confused about what this word means
> (I do, anyway).
> 
> > acme1 and acme2 are two bus masters in the system (eg a DMA and a GPU).
> > 
> > Each foo@2000000 reg property maps to a device that represents a bus master
> > (to make it clearer, a foo@2000000 reg property defines an address space that
> > belongs to a bus master, ie the address space represents a programming
> > interface specific to that master; in the bindings above address 0x2000000 is
> > the address at which acme1 device can programme its "foo" interface, address
> > 0x2001000 is the address at which acme2 device can programme its "foo"
> > interface).
> > 
> > Now, the dts above links, through a phandle, a reg property to a "slave"
> > device not a "master" device, basically to a device tree node representing the
> > acme1 device programming interface (address 0x4000000 is the register space of
> > the acme1 device, which is used to describe that device in the tree and
> > represents its address space, its "slave interface").
> > 
> > The approach above has two drawbacks:
> > 
> > 1 - we are using a slave device node to map a resource to a master device.
> >     Since there is no notion of "master" device in the current device tree
> >     bindings that's our best bet. The approach above works as long as there
> >     is a node in the device tree describing the "master" node (through its
> >     slave interface), even if it does not have a programming interface (ie it
> >     cannot be described in the device tree as a memory mapped/addressable
> >     device). This is also the approach taken by the dma bindings described in
> > 
> >     Documentation/devicetree/bindings/dma/dma.txt (currently in linux-next)
> > 
> > 2 - By connecting a device's resource to a slave device, to emulate a resource
> >     to a bus master connection, we are implicitly taking the assumption that
> >     the address space where both eg foo and acme1 devices above reside (acme1
> >     programming interface) is the same as the one seen by the acme1 master
> >     device (which has no representation in the device tree). This is not a
> >     problem for now, but the device tree representation is questionable and
> >     we are inquiring on possible issues this might create.
> > 
> > Any bits of advice is welcome on the subject, and the problem can be
> > elaborated with further examples to kickstart discussions.
> 
> Every master potentially has its own address space -- a bus
> (either hypothetical or real) which maps the accessible slaves.  This
> bus might or might not be shared with other masters in the system.
> 
> If could help illustrate the potential issues if we can sketch a system
> which we can't adequately describe.
> 
> Things which come to mind are:
> 
>  * CPU-local and cluster-local peripherals
> 
>  * Non-coherent CPUs or microcontrollers
> 
>  * Bus masters with slave interfaces (e.g., control interfaces) in a location
>    in the bus hierarchy unrelated to where the device makes its master
>    accesses, to that we can't guess from the DT how to map addresses
>    for those devices (interconnects are an example of this: the CCI
>    control register interface is likely to be dangling from an AXI or APB
>    bus, topologically distant from the CCI's interconnect and master
>    ports, for example)
> 
>    Most DMA controllers will have this property too.  For now, we just
>    have to guess what view of the system the controller sees in its
>    master role.
> 
>  * Masters with weird private links (for example, a high-throughput DMA
>    controller with its own, non-coherent port on the DRAM controller,
>    bypassing the bus through which CPUs and other masters see the DRAM)
> 
> We don't necessarily have to solve all of these yet, but they all feel
> related.  The fundamental problem is how to describe the fact that
> there may be multiple, arbitrarily different, views of the system.
> 
> If we can come up with non-tortured ways of describing all these things
> in DT, then we don't have a problem...


Below, I attempt to define the problem in a bit more detail and sketch
out some solutions and implications.  It's a bit long -- apologies --
but I couldn't figure out a good way to compress it.


Cheers
---Dave


Thinking a bit about this, CPU nodes feel like a good place to start
thinking about what masters look like, since CPUs are the only pure
masters that DT currently describes today.

Could we describe each CPU's address space by putting child nodes into
a CPU node itself?  ePAPR already specifies that a missing ranges
property means that there is no address mapping between the parent
and the child, so each cpu node can already represent an isolated
memory space:

/ {
	#address-cells = <1>;
	#size-cells = <0>;

	cpus {
		// Note: no ranges property means that CPU reg properties
		// are not mapped in to / as addresses

		#address-cells = <1>;
		#size-cells = <0>;

		cpu@0 {
			#address-cells = <1>;
			#size-cells = <0>;

			// Note: no ranges property means that we have a
			// separate CPU-local address space, nothing to
			// do with CPU ids.

			memory {
				compatible = "arm,tcm";
				reg = <0xe0400000 0x4000>;
			};
		};

		cpu@1 {
			#address-cells = <1>;
			#size-cells = <0>;

			memory {
				compatible = "arm,tcm";
				reg = <0xe0400000 0x4000>;
			};
		};

		// ...
	};

	// ...
};

Representing:

    +------+            +------+
    | CPU0 |            | CPU1 |
    +------+            +------+
      |  |                |  |
      |  v                |  v
      | +------+          | +------+
      | |memory|          | |memory|
      | +------+          | +------+
      |                   |
      v                   v
   +-------------------------+
   |   common interconnect   |
   +-------------------------+
          |     |     |
          v     v     v
        common peripherals


Here, the common interconnect is represented by / in the DT.  The DT
consists of a mixture of nodes which have a reg or ranges propertyo
(peripherals and buses) and therefore has no presence in the / address
space, and nodes which have neither (other stuff which may not represent
hardware at all but which "has to go in the DT somewhere".

I've invented some local memory on each CPU for the purpose of
illustration.  That's probably not a realistic or likely system design
-- but I'm just trying to expore ideas here.


Some outcomes:

 * The primary bus seen by each CPU would be the cpu node itself.

 * Addresses not matching any peripheral on the local bus propagate to /.
   The address mapping between the CPU local bus and / is assumed by
   default to be the identifity mapping.  This also means that a master
   node's #address-cells and #size-cells must match /.

 * There is no risk of circular lookup here, because the cpus { } node
   is isolated from / by the absence of reg or ranges properties.


This suggests a way to represent a master device in the DT, if the
address space it sees in its master role is /:

 * A device with master capabilities must be represented by a node

 * Such a node must define #address-cells and #size-cells to match /.


For simplicity, we could consider every device a potential master,
such that a hypothetical dt_map_address(struct device_node *slave,
struct device_node *master) would always produce a result for any
slave reachable from /.

Alternatively, we could require master nodes to be labelled with a
special property to indicate that the node represents a device with
master capabilities.


This can work when / really is the common merge point for all
masters, but what if it isn't?

Suppose we have a non-coherent DMA engine, like this:

    +------+            +------+
    | CPU0 |            | CPU1 |
    +------+            +------+
      |  |                |  |
      |  v                |  v
      | +------+          | +------+
      | |memory|          | |memory|
      | +------+          | +------+
      |                   |      ,-.
      v                   v     |   v (control slave port)
   +-------------------------+  |  +-------+
   |   common interconnect   |  |  | ncDMA |
   +-------------------------+  |  +-------+
      | |          |      | |   |  | (master port)
      v v          |      |  `-'   |
common peripherals |      |        |
                   v      v        v
          +---------+   +-----------+
          |  DRAM1  |   |   DRAM2   |
          +---------+   +-----------+


Unlike the CPU-local-memory thing, this isn't unrealistic at all.  We
have a situation something like this with the motherboard CLCD
controller on vexpress.  I think that most complex SoCs will have at
least one instance of this kind of configuration (just my guess, based
on no evidence)  Real SoCs are not designed from a clean page, but will
inherit legacy subsystems from older SoCs, especially in areas where
performance is not much of an issue (audio codecs, UARTs ... whatever).


Omitting the CPU nodes, this might look something like:

/ {
	cpus {
		// omitted
	};

	// Question: what does the memory node mean wrt bus topology??
	memory {
		device_type = "memory";
		reg = <0x80000000 0x40000000>; // 1GB DRAM1
	};

	common-bus {
		compatible = "simple-bus";
		ranges = <0x00000000 0x80000000 0x80000000>;

		// DRAM1 omitted because it's in the toplevel memory
		// node.  This may or may not be appropriate.

ncDMA_bus:	bus@42000000 {
			compatible = "simple-bus";
			ranges = <0x00000000 0x42000000 0x01000000>;

			memory {
				reg = <0x00000000 0x01000000>;
			};
		};

		dma@4c00000 {
			// control slave interface
			reg = <0x04c00000 0x00001000>; 

			// device also has master capabilities:
			master;

			// links to shared slaves
			// intentionally similar to the ePAPR dma-ranges
			// property
			slave = <&ncDMA_bus 0x00000000 0x00000000 0x01000000>;
		};
	}
};

"slave" is a prop-encoded array of slave-bus-phandle, master-address,
slave-bus-address, length.  master-address obeys #address-cells of
the master node; slave-bus-address obeys #address-cells of the node
to which slave-bus-phandle points; length obeys the #size-cells of
the master node.  The slave node might contain multiple such tuples.

ePAPR has dma-ranges, which is close to what we need, but too limited
for the general case.  The assumption there seems to be that DMA
accesses are a special type of memory access which can travel up the
tree and then down into other peripherals or subtrees.  dma-ranges
can be thought of as a "slave" property where slave-bus-phandle
implicitly points to the immediate parent of the containing node.

However, while this is probably adequate to describe a cards-in-slots
physical bus, on-SoC peripherals can be crosslinked in arbitrary
ways, bypassing shared buses.

"slave" allows us to eliminate the assumption that all master accesses
originating from a subtree behave identically, by adding an explicit
phandle.


To get around this problem in a way which follows the spirit of
dma-ranges, I created a fictional transparent bus (labelled ncDMA_bus),
and a slave property to refer to it.

Some more outcomes:

 * cpus are probably still special.  We assume those are masters
   even with no "master" property.

 * Masters with no "slaves" property master on /, with an identity
   address mapping.

 * Masters with local subnodes master on those subnodes.

 * Masters with a "slave" property master the referenced devices
   or buses.

 * We can retain the dma-ranges property and continue to use it
   in those cases where it is adequate.

 * The "master" property might not be needed.  Either a device
   has subnodes or a slave or dma-ranges node (in which case it
   can be considered a master) ... or has none of these (in which
   the device is not a master).


Cheers
---Dave

Re: [RFC] DT affinity bindings/representing bus masters with DT

[David Gibson]

[-- Attachment #1.1: Type: text/plain, Size: 5002 bytes --]

On Fri, Feb 15, 2013 at 05:21:02PM +0000, Lorenzo Pieralisi wrote:
> Hi all,
> 
> in order to come up with a solid solution to the affinity bindings concept
> we are facing in the ARM world, I am posting this RFC so that hopefully people
> on the list can chime in and help us in this endeavour.
> 
> I tried to keep things simple on purpose and some statements are a bit of an
> oversimplification, we can elaborate on those if needed.
> 
> Hereafter a summary of what we are trying to achieve.
> 
> Current device tree bindings allow to describe HW configurations of systems
> in a bus like fashion, where each bus contains bus slaves and mapping
> heuristics to translate address spaces across bus layers (AMBA -> PCI).
> 
> The device tree by definition represents the view of the system from the
> perspective of the CPU(s). This means that all devices (but CPUs) present
> in the device tree are to be seen as "slave" components, ie devices sitting on
> a bus and accessible from the CPU with an addressing mode that can vary and it
> is defined by the bus the device is sitting on.
> 
> There are specific cases in current SoCs though where resources belonging to
> a slave device should be linked to a master in the SoC system hierarchy.
> 
> To the best of my knowledge, the mechanism used to implement this linkage is
> not defined by any device tree binding; it probably can be a phandle in the
> device tree syntax, but this has some implications that will be described as
> follows.
> 
> A programmable device, let's call it "foo" for the sake of this discussion,
> has several resources (ie memory spaces) that should be mapped to bus masters
> in a SoC hierarchy (each resource "belongs" to a master). The only way this
> can be currently done through a device tree is by linking the resource in
> question to a device representing the master node through a phandle (keeping
> in mind that the concept of master node does not exist in current DT
> bindings).
> 
> An example is worth a thousand words (pseudo dts file):
> 
> / {
> 	#address-cells = 1;
> 	#size-cells = 1;
> 
> 	acme1: acme@4000000 {
> 		reg = <0x4000000 0x1000>;
> 	};
> 
> 	acme2: acme@5000000 {
> 		reg = <0x5000000 0x1000>;
> 	};
> 
> 	foo@2000000 {
> 		reg = <0x2000000 0x1000
> 		       0x2001000 0x1000>;
> 		affinity = <&acme1 &acme2>;
> 	};
> };
> 
> where the "affinity" property contains a number of phandles equal to the number
> of resources (ie reg properties) in the foo@2000000 node. The "affinity"
> property maps a reg property to a device tree node, one phandle per "reg"
> property.
> 
> acme1 and acme2 are two bus masters in the system (eg a DMA and a GPU).
> 
> Each foo@2000000 reg property maps to a device that represents a bus master
> (to make it clearer, a foo@2000000 reg property defines an address space that
> belongs to a bus master, ie the address space represents a programming
> interface specific to that master; in the bindings above address 0x2000000 is
> the address at which acme1 device can programme its "foo" interface, address
> 0x2001000 is the address at which acme2 device can programme its "foo"
> interface).

Ok.  I think annotating the existing reg property like this is a very
bad idea.  I haven't seen all the previous discussion, so I'm not
totally clean on what this affinity concept is about.  But as I
understand it, these "slave" resources cannot be treated like an
ordinary resource in in the reg property.  That means an older client
will potentially misinterpret "reg" because it doesn't know about
"affinity".

Worse, again, if I've understood correctly, resources with different
"masters" are essentially in different logical address spaces.  "reg"
properties should always sit in the logical address space representing
the parent node's bus.  Different address spaces could also have
different address sizes, which would really complicate parsing "reg".

Now, a device tree extension I've considered before for other reasons
might also be relevant to your case.  That is to add a new "bus-reg"
property that has a meaning similar to "reg" but contains (phandle,
address, size) tuples instead of just (address, size) tuples.  The
phandle in each case would represent the node in whose address space
this resource appears.  This would generalize the "dcr-reg" and
"scom-reg" properties we've sometimes used on ppc.

It's not clear if this essentially incompatible extension would be
worth it, though.  The other way to represent these structures is to
just split the node for this logical device into different pieces
under each address space it has resources on.  These pieces are then
linked together with (binding specific) phandle pointers to each
other.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 192 bytes --]

_______________________________________________
devicetree-discuss mailing list
devicetree-discuss-uLR06cmDAlY/bJ5BZ2RsiQ@public.gmane.org
https://lists.ozlabs.org/listinfo/devicetree-discuss

Re: [RFC] DT affinity bindings/representing bus masters with DT

[Lorenzo Pieralisi]

Hi David,

thanks for your feedback.

On Wed, Mar 06, 2013 at 09:57:14PM +0000, David Gibson wrote:
> On Fri, Feb 15, 2013 at 05:21:02PM +0000, Lorenzo Pieralisi wrote:
> > Hi all,
> > 
> > in order to come up with a solid solution to the affinity bindings concept
> > we are facing in the ARM world, I am posting this RFC so that hopefully people
> > on the list can chime in and help us in this endeavour.
> > 
> > I tried to keep things simple on purpose and some statements are a bit of an
> > oversimplification, we can elaborate on those if needed.
> > 
> > Hereafter a summary of what we are trying to achieve.
> > 
> > Current device tree bindings allow to describe HW configurations of systems
> > in a bus like fashion, where each bus contains bus slaves and mapping
> > heuristics to translate address spaces across bus layers (AMBA -> PCI).
> > 
> > The device tree by definition represents the view of the system from the
> > perspective of the CPU(s). This means that all devices (but CPUs) present
> > in the device tree are to be seen as "slave" components, ie devices sitting on
> > a bus and accessible from the CPU with an addressing mode that can vary and it
> > is defined by the bus the device is sitting on.
> > 
> > There are specific cases in current SoCs though where resources belonging to
> > a slave device should be linked to a master in the SoC system hierarchy.
> > 
> > To the best of my knowledge, the mechanism used to implement this linkage is
> > not defined by any device tree binding; it probably can be a phandle in the
> > device tree syntax, but this has some implications that will be described as
> > follows.
> > 
> > A programmable device, let's call it "foo" for the sake of this discussion,
> > has several resources (ie memory spaces) that should be mapped to bus masters
> > in a SoC hierarchy (each resource "belongs" to a master). The only way this
> > can be currently done through a device tree is by linking the resource in
> > question to a device representing the master node through a phandle (keeping
> > in mind that the concept of master node does not exist in current DT
> > bindings).
> > 
> > An example is worth a thousand words (pseudo dts file):
> > 
> > / {
> > 	#address-cells = 1;
> > 	#size-cells = 1;
> > 
> > 	acme1: acme@4000000 {
> > 		reg = <0x4000000 0x1000>;
> > 	};
> > 
> > 	acme2: acme@5000000 {
> > 		reg = <0x5000000 0x1000>;
> > 	};
> > 
> > 	foo@2000000 {
> > 		reg = <0x2000000 0x1000
> > 		       0x2001000 0x1000>;
> > 		affinity = <&acme1 &acme2>;
> > 	};
> > };
> > 
> > where the "affinity" property contains a number of phandles equal to the number
> > of resources (ie reg properties) in the foo@2000000 node. The "affinity"
> > property maps a reg property to a device tree node, one phandle per "reg"
> > property.
> > 
> > acme1 and acme2 are two bus masters in the system (eg a DMA and a GPU).
> > 
> > Each foo@2000000 reg property maps to a device that represents a bus master
> > (to make it clearer, a foo@2000000 reg property defines an address space that
> > belongs to a bus master, ie the address space represents a programming
> > interface specific to that master; in the bindings above address 0x2000000 is
> > the address at which acme1 device can programme its "foo" interface, address
> > 0x2001000 is the address at which acme2 device can programme its "foo"
> > interface).
> 
> Ok.  I think annotating the existing reg property like this is a very
> bad idea.  I haven't seen all the previous discussion, so I'm not
> totally clean on what this affinity concept is about.  But as I
> understand it, these "slave" resources cannot be treated like an
> ordinary resource in in the reg property.  That means an older client
> will potentially misinterpret "reg" because it doesn't know about
> "affinity".

Not really, "reg" still complies with the current DT bindings. Affinity
is there to associate a reg property to a "master" but the reg property
definition does not change. I do not think backward compatibility is a
problem per-se here.

> Worse, again, if I've understood correctly, resources with different
> "masters" are essentially in different logical address spaces.  "reg"
> properties should always sit in the logical address space representing
> the parent node's bus.  Different address spaces could also have
> different address sizes, which would really complicate parsing "reg".

I think we need to post what we have, it is really complex to explain
the issue without a concrete example. To cut a long story short I
would not say that the resources sit in different address spaces, it is
that we need to associate those address ranges with specific bus masters.

We have to have a way to say:

"Address range 0x80001000 - 0x80001fff is used to programme the control
registers associated with the port connected to master X".

When a CPU wants to programme a control port for a specific master, it
needs to know what address range should be programmed.

I mentioned "resources" instead of addresses since the problem we are
having is the same when it comes to map IRQs to set of CPUS. We need
to associate a resource (IRQ or address) to a set of cpus (or more in
general, masters).

We will be posting the code we have soon, this will simplify the discussion.

> Now, a device tree extension I've considered before for other reasons
> might also be relevant to your case.  That is to add a new "bus-reg"
> property that has a meaning similar to "reg" but contains (phandle,
> address, size) tuples instead of just (address, size) tuples.  The
> phandle in each case would represent the node in whose address space
> this resource appears.  This would generalize the "dcr-reg" and
> "scom-reg" properties we've sometimes used on ppc.

It might be used that way but I need an example to understand if it fits
the purpose. We still have to associate that resource to a set of
masters, so for that to work the address space pointed at by the phandle
must be capable of defining a master (or set of masters).

I am not sure the problem is identical to dcr-reg though.

> It's not clear if this essentially incompatible extension would be
> worth it, though.  The other way to represent these structures is to
> just split the node for this logical device into different pieces
> under each address space it has resources on.  These pieces are then
> linked together with (binding specific) phandle pointers to each
> other.

I think that could be feasible too and we thought about that.

Code coming, let's restart the discussion then to see how we can move
forward.

Thank you very much,
Lorenzo

Re: [RFC] DT affinity bindings/representing bus masters with DT

[David Gibson]

[-- Attachment #1.1: Type: text/plain, Size: 3299 bytes --]

On Mon, Mar 11, 2013 at 05:06:57PM +0000, Lorenzo Pieralisi wrote:
> Hi David,
> 
> thanks for your feedback.
> 
> On Wed, Mar 06, 2013 at 09:57:14PM +0000, David Gibson wrote:
> > On Fri, Feb 15, 2013 at 05:21:02PM +0000, Lorenzo Pieralisi wrote:
[snip]
> > > Each foo@2000000 reg property maps to a device that represents a bus master
> > > (to make it clearer, a foo@2000000 reg property defines an address space that
> > > belongs to a bus master, ie the address space represents a programming
> > > interface specific to that master; in the bindings above address 0x2000000 is
> > > the address at which acme1 device can programme its "foo" interface, address
> > > 0x2001000 is the address at which acme2 device can programme its "foo"
> > > interface).
> > 
> > Ok.  I think annotating the existing reg property like this is a very
> > bad idea.  I haven't seen all the previous discussion, so I'm not
> > totally clean on what this affinity concept is about.  But as I
> > understand it, these "slave" resources cannot be treated like an
> > ordinary resource in in the reg property.  That means an older client
> > will potentially misinterpret "reg" because it doesn't know about
> > "affinity".
> 
> Not really, "reg" still complies with the current DT bindings. Affinity
> is there to associate a reg property to a "master" but the reg property
> definition does not change. I do not think backward compatibility is a
> problem per-se here.

Ok, I did not understand the problem properly.

> > Worse, again, if I've understood correctly, resources with different
> > "masters" are essentially in different logical address spaces.  "reg"
> > properties should always sit in the logical address space representing
> > the parent node's bus.  Different address spaces could also have
> > different address sizes, which would really complicate parsing "reg".
> 
> I think we need to post what we have, it is really complex to explain
> the issue without a concrete example. To cut a long story short I
> would not say that the resources sit in different address spaces, it is
> that we need to associate those address ranges with specific bus masters.
> 
> We have to have a way to say:
> 
> "Address range 0x80001000 - 0x80001fff is used to programme the control
> registers associated with the port connected to master X".
> 
> When a CPU wants to programme a control port for a specific master, it
> needs to know what address range should be programmed.
> 
> I mentioned "resources" instead of addresses since the problem we are
> having is the same when it comes to map IRQs to set of CPUS. We need
> to associate a resource (IRQ or address) to a set of cpus (or more in
> general, masters).

Hrm.  See, I think I may be misunderstanding the problem again,
because with this description I can see no problem.  It's already up
to the device binding to describe the purpose of each entry in the reg
property.  So what's the problem with it just being part of the
binding to say which reg entry is associated with which master port?

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 192 bytes --]

_______________________________________________
devicetree-discuss mailing list
devicetree-discuss-uLR06cmDAlY/bJ5BZ2RsiQ@public.gmane.org
https://lists.ozlabs.org/listinfo/devicetree-discuss

Re: [RFC] DT affinity bindings/representing bus masters with DT

[Lorenzo Pieralisi]

On Mon, Mar 18, 2013 at 03:09:28AM +0000, David Gibson wrote:
> On Mon, Mar 11, 2013 at 05:06:57PM +0000, Lorenzo Pieralisi wrote:
> > Hi David,
> > 
> > thanks for your feedback.
> > 
> > On Wed, Mar 06, 2013 at 09:57:14PM +0000, David Gibson wrote:
> > > On Fri, Feb 15, 2013 at 05:21:02PM +0000, Lorenzo Pieralisi wrote:
> [snip]
> > > > Each foo@2000000 reg property maps to a device that represents a bus master
> > > > (to make it clearer, a foo@2000000 reg property defines an address space that
> > > > belongs to a bus master, ie the address space represents a programming
> > > > interface specific to that master; in the bindings above address 0x2000000 is
> > > > the address at which acme1 device can programme its "foo" interface, address
> > > > 0x2001000 is the address at which acme2 device can programme its "foo"
> > > > interface).
> > > 
> > > Ok.  I think annotating the existing reg property like this is a very
> > > bad idea.  I haven't seen all the previous discussion, so I'm not
> > > totally clean on what this affinity concept is about.  But as I
> > > understand it, these "slave" resources cannot be treated like an
> > > ordinary resource in in the reg property.  That means an older client
> > > will potentially misinterpret "reg" because it doesn't know about
> > > "affinity".
> > 
> > Not really, "reg" still complies with the current DT bindings. Affinity
> > is there to associate a reg property to a "master" but the reg property
> > definition does not change. I do not think backward compatibility is a
> > problem per-se here.
> 
> Ok, I did not understand the problem properly.
> 
> > > Worse, again, if I've understood correctly, resources with different
> > > "masters" are essentially in different logical address spaces.  "reg"
> > > properties should always sit in the logical address space representing
> > > the parent node's bus.  Different address spaces could also have
> > > different address sizes, which would really complicate parsing "reg".
> > 
> > I think we need to post what we have, it is really complex to explain
> > the issue without a concrete example. To cut a long story short I
> > would not say that the resources sit in different address spaces, it is
> > that we need to associate those address ranges with specific bus masters.
> > 
> > We have to have a way to say:
> > 
> > "Address range 0x80001000 - 0x80001fff is used to programme the control
> > registers associated with the port connected to master X".
> > 
> > When a CPU wants to programme a control port for a specific master, it
> > needs to know what address range should be programmed.
> > 
> > I mentioned "resources" instead of addresses since the problem we are
> > having is the same when it comes to map IRQs to set of CPUS. We need
> > to associate a resource (IRQ or address) to a set of cpus (or more in
> > general, masters).
> 
> Hrm.  See, I think I may be misunderstanding the problem again,
> because with this description I can see no problem.  It's already up
> to the device binding to describe the purpose of each entry in the reg
> property.  So what's the problem with it just being part of the
> binding to say which reg entry is associated with which master port?

That's exactly what we are trying to do. But to associate the reg
property to a master port we need a phandle, how can we pull that off
otherwise ?

The purpose of each reg entry is to describe the address space
associated with the programming interface for a port. What reg currently
can't describe is to what component that port is connected to. It is not
something we can define in the binding (unless we make it configurable
with a phandle, and that's what we want to do) because it changes with
different SoC implementations.

In one SoC port 0 can be connected to a GPU, in another one to a DMA.

How can we write a binding that encompasses both cases ? The only
solution is a phandle, to say this port is connected to this master.

When a CPU programmes that port is because it wants to change behaviour
of the port for a specific component (a DMA, a GPU, a cluster of CPUs).

Let's say we have to describe a binding for a component having three
slave ports connected to three masters (to have a more accurate example,
two clusters of cpus and a GPU). Each port can be programmed and has an
address space associated to it (reg property). The programming interface
controls the port behaviour.

Now, when a CPU wants to programme the port connected to CPU X (or GPU), how
can it know what address space (ie what reg property) it has to use ?

We cannot hardcode it in the reg property bindings, the component's ports are
connected to different masters in different SoCs.

Let me know if it is unclear, the patchset we are about to post will help
clarify things I hope.

Lorenzo

Re: [RFC] DT affinity bindings/representing bus masters with DT

[David Gibson]

[-- Attachment #1.1: Type: text/plain, Size: 4390 bytes --]

On Mon, Mar 18, 2013 at 09:48:16AM +0000, Lorenzo Pieralisi wrote:
> On Mon, Mar 18, 2013 at 03:09:28AM +0000, David Gibson wrote:
> > On Mon, Mar 11, 2013 at 05:06:57PM +0000, Lorenzo Pieralisi wrote:
> > > Hi David,
> > > 
> > > thanks for your feedback.
> > > 
> > > On Wed, Mar 06, 2013 at 09:57:14PM +0000, David Gibson wrote:
> > > > On Fri, Feb 15, 2013 at 05:21:02PM +0000, Lorenzo Pieralisi wrote:
> > [snip]
> > > > > Each foo@2000000 reg property maps to a device that represents a bus master
> > > > > (to make it clearer, a foo@2000000 reg property defines an address space that
> > > > > belongs to a bus master, ie the address space represents a programming
> > > > > interface specific to that master; in the bindings above address 0x2000000 is
> > > > > the address at which acme1 device can programme its "foo" interface, address
> > > > > 0x2001000 is the address at which acme2 device can programme its "foo"
> > > > > interface).
> > > > 
> > > > Ok.  I think annotating the existing reg property like this is a very
> > > > bad idea.  I haven't seen all the previous discussion, so I'm not
> > > > totally clean on what this affinity concept is about.  But as I
> > > > understand it, these "slave" resources cannot be treated like an
> > > > ordinary resource in in the reg property.  That means an older client
> > > > will potentially misinterpret "reg" because it doesn't know about
> > > > "affinity".
> > > 
> > > Not really, "reg" still complies with the current DT bindings. Affinity
> > > is there to associate a reg property to a "master" but the reg property
> > > definition does not change. I do not think backward compatibility is a
> > > problem per-se here.
> > 
> > Ok, I did not understand the problem properly.
> > 
> > > > Worse, again, if I've understood correctly, resources with different
> > > > "masters" are essentially in different logical address spaces.  "reg"
> > > > properties should always sit in the logical address space representing
> > > > the parent node's bus.  Different address spaces could also have
> > > > different address sizes, which would really complicate parsing "reg".
> > > 
> > > I think we need to post what we have, it is really complex to explain
> > > the issue without a concrete example. To cut a long story short I
> > > would not say that the resources sit in different address spaces, it is
> > > that we need to associate those address ranges with specific bus masters.
> > > 
> > > We have to have a way to say:
> > > 
> > > "Address range 0x80001000 - 0x80001fff is used to programme the control
> > > registers associated with the port connected to master X".
> > > 
> > > When a CPU wants to programme a control port for a specific master, it
> > > needs to know what address range should be programmed.
> > > 
> > > I mentioned "resources" instead of addresses since the problem we are
> > > having is the same when it comes to map IRQs to set of CPUS. We need
> > > to associate a resource (IRQ or address) to a set of cpus (or more in
> > > general, masters).
> > 
> > Hrm.  See, I think I may be misunderstanding the problem again,
> > because with this description I can see no problem.  It's already up
> > to the device binding to describe the purpose of each entry in the reg
> > property.  So what's the problem with it just being part of the
> > binding to say which reg entry is associated with which master port?
> 
> That's exactly what we are trying to do. But to associate the reg
> property to a master port we need a phandle, how can we pull that off
> otherwise ?

Ah, right.  I was meaning that the binding specifies the abstract port
number that each reg entry is associated with.  I'm fine with adding a
property to map the port numbers to master devices on the other end.
I think treating it in two steps like that is better than thinking of
the master phandles being  directly associated with reg entries,
because it also handles cases like having a bank of common/global
registers not associated with any port/master, or cases where the
ports aren't all identical and some need more resources than others.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 192 bytes --]

_______________________________________________
devicetree-discuss mailing list
devicetree-discuss-uLR06cmDAlY/bJ5BZ2RsiQ@public.gmane.org
https://lists.ozlabs.org/listinfo/devicetree-discuss

Re: [RFC] DT affinity bindings/representing bus masters with DT

[Lorenzo Pieralisi]

On Tue, Mar 19, 2013 at 07:07:34AM +0000, David Gibson wrote:
> On Mon, Mar 18, 2013 at 09:48:16AM +0000, Lorenzo Pieralisi wrote:
> > On Mon, Mar 18, 2013 at 03:09:28AM +0000, David Gibson wrote:
> > > On Mon, Mar 11, 2013 at 05:06:57PM +0000, Lorenzo Pieralisi wrote:
> > > > Hi David,
> > > > 
> > > > thanks for your feedback.
> > > > 
> > > > On Wed, Mar 06, 2013 at 09:57:14PM +0000, David Gibson wrote:
> > > > > On Fri, Feb 15, 2013 at 05:21:02PM +0000, Lorenzo Pieralisi wrote:
> > > [snip]
> > > > > > Each foo@2000000 reg property maps to a device that represents a bus master
> > > > > > (to make it clearer, a foo@2000000 reg property defines an address space that
> > > > > > belongs to a bus master, ie the address space represents a programming
> > > > > > interface specific to that master; in the bindings above address 0x2000000 is
> > > > > > the address at which acme1 device can programme its "foo" interface, address
> > > > > > 0x2001000 is the address at which acme2 device can programme its "foo"
> > > > > > interface).
> > > > > 
> > > > > Ok.  I think annotating the existing reg property like this is a very
> > > > > bad idea.  I haven't seen all the previous discussion, so I'm not
> > > > > totally clean on what this affinity concept is about.  But as I
> > > > > understand it, these "slave" resources cannot be treated like an
> > > > > ordinary resource in in the reg property.  That means an older client
> > > > > will potentially misinterpret "reg" because it doesn't know about
> > > > > "affinity".
> > > > 
> > > > Not really, "reg" still complies with the current DT bindings. Affinity
> > > > is there to associate a reg property to a "master" but the reg property
> > > > definition does not change. I do not think backward compatibility is a
> > > > problem per-se here.
> > > 
> > > Ok, I did not understand the problem properly.
> > > 
> > > > > Worse, again, if I've understood correctly, resources with different
> > > > > "masters" are essentially in different logical address spaces.  "reg"
> > > > > properties should always sit in the logical address space representing
> > > > > the parent node's bus.  Different address spaces could also have
> > > > > different address sizes, which would really complicate parsing "reg".
> > > > 
> > > > I think we need to post what we have, it is really complex to explain
> > > > the issue without a concrete example. To cut a long story short I
> > > > would not say that the resources sit in different address spaces, it is
> > > > that we need to associate those address ranges with specific bus masters.
> > > > 
> > > > We have to have a way to say:
> > > > 
> > > > "Address range 0x80001000 - 0x80001fff is used to programme the control
> > > > registers associated with the port connected to master X".
> > > > 
> > > > When a CPU wants to programme a control port for a specific master, it
> > > > needs to know what address range should be programmed.
> > > > 
> > > > I mentioned "resources" instead of addresses since the problem we are
> > > > having is the same when it comes to map IRQs to set of CPUS. We need
> > > > to associate a resource (IRQ or address) to a set of cpus (or more in
> > > > general, masters).
> > > 
> > > Hrm.  See, I think I may be misunderstanding the problem again,
> > > because with this description I can see no problem.  It's already up
> > > to the device binding to describe the purpose of each entry in the reg
> > > property.  So what's the problem with it just being part of the
> > > binding to say which reg entry is associated with which master port?
> > 
> > That's exactly what we are trying to do. But to associate the reg
> > property to a master port we need a phandle, how can we pull that off
> > otherwise ?
> 
> Ah, right.  I was meaning that the binding specifies the abstract port
> number that each reg entry is associated with.  I'm fine with adding a
> property to map the port numbers to master devices on the other end.
> I think treating it in two steps like that is better than thinking of
> the master phandles being  directly associated with reg entries,
> because it also handles cases like having a bank of common/global
> registers not associated with any port/master, or cases where the
> ports aren't all identical and some need more resources than others.

That's fine by me. Can you provide me with an example of how the bindings
should tie a specific reg property to an abstract port number ? reg properties
ordering (ie index) ? Or port number encoded in the reg property itself ?

Thanks,
Lorenzo

Re: [RFC] DT affinity bindings/representing bus masters with DT

[Grant Likely]

On Mon, 25 Mar 2013 15:20:39 +0000, Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org> wrote:
> On Tue, Mar 19, 2013 at 07:07:34AM +0000, David Gibson wrote:
> > On Mon, Mar 18, 2013 at 09:48:16AM +0000, Lorenzo Pieralisi wrote:
> > > > > I think we need to post what we have, it is really complex to explain
> > > > > the issue without a concrete example. To cut a long story short I
> > > > > would not say that the resources sit in different address spaces, it is
> > > > > that we need to associate those address ranges with specific bus masters.
> > > > > 
> > > > > We have to have a way to say:
> > > > > 
> > > > > "Address range 0x80001000 - 0x80001fff is used to programme the control
> > > > > registers associated with the port connected to master X".
> > > > > 
> > > > > When a CPU wants to programme a control port for a specific master, it
> > > > > needs to know what address range should be programmed.
> > > > > 
> > > > > I mentioned "resources" instead of addresses since the problem we are
> > > > > having is the same when it comes to map IRQs to set of CPUS. We need
> > > > > to associate a resource (IRQ or address) to a set of cpus (or more in
> > > > > general, masters).
> > > > 
> > > > Hrm.  See, I think I may be misunderstanding the problem again,
> > > > because with this description I can see no problem.  It's already up
> > > > to the device binding to describe the purpose of each entry in the reg
> > > > property.  So what's the problem with it just being part of the
> > > > binding to say which reg entry is associated with which master port?
> > > 
> > > That's exactly what we are trying to do. But to associate the reg
> > > property to a master port we need a phandle, how can we pull that off
> > > otherwise ?
> > 
> > Ah, right.  I was meaning that the binding specifies the abstract port
> > number that each reg entry is associated with.  I'm fine with adding a
> > property to map the port numbers to master devices on the other end.
> > I think treating it in two steps like that is better than thinking of
> > the master phandles being  directly associated with reg entries,
> > because it also handles cases like having a bank of common/global
> > registers not associated with any port/master, or cases where the
> > ports aren't all identical and some need more resources than others.
> 
> That's fine by me. Can you provide me with an example of how the bindings
> should tie a specific reg property to an abstract port number ? reg properties
> ordering (ie index) ? Or port number encoded in the reg property itself ?

I think I agree with David (assuming I understand his argument
correctly). Rather than putting the information into the single node, it
is more consistent in DT terms to put that information into each of the
'users' of the device; so for each master, put in a property that
associates it with a specific port. Something like this (completely off
the top of my head; not at all fully thought out):

cpu@0 {
	cci-port = <&cci1 0>;
}

cpu@1 {
	cci-port = <&cci1 1>;
}

cpu@2 {
	cci-port = <&cci1 2>;
}

gpu@c8010000 {
	cci-port = <&cci1 3>;
}

cci1: cci@c8000000 {
	reg = < ... >;
}

An arguement against that would be that it is difficult to determine
what all the bus-mastering devices are from looking at the cci node, but
in real terms you would have a lookup table in the operating system
/anyway/ for matching devices to ports and I would expect that as each
device is discovered from the DT data the OS would populate it's lookup
table.

It may be that Dave Martin's suggestion of using basically a dma-ranges
type property with a phandle would be the correct interface. I've not
thought enough about it yet to really get my head around it.

g.

Re: [RFC] DT affinity bindings/representing bus masters with DT

[Lorenzo Pieralisi]

Thank you for having a look Grant.

On Sat, Apr 13, 2013 at 11:24:48PM +0100, Grant Likely wrote:
> On Mon, 25 Mar 2013 15:20:39 +0000, Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org> wrote:
> > On Tue, Mar 19, 2013 at 07:07:34AM +0000, David Gibson wrote:
> > > On Mon, Mar 18, 2013 at 09:48:16AM +0000, Lorenzo Pieralisi wrote:
> > > > > > I think we need to post what we have, it is really complex to explain
> > > > > > the issue without a concrete example. To cut a long story short I
> > > > > > would not say that the resources sit in different address spaces, it is
> > > > > > that we need to associate those address ranges with specific bus masters.
> > > > > > 
> > > > > > We have to have a way to say:
> > > > > > 
> > > > > > "Address range 0x80001000 - 0x80001fff is used to programme the control
> > > > > > registers associated with the port connected to master X".
> > > > > > 
> > > > > > When a CPU wants to programme a control port for a specific master, it
> > > > > > needs to know what address range should be programmed.
> > > > > > 
> > > > > > I mentioned "resources" instead of addresses since the problem we are
> > > > > > having is the same when it comes to map IRQs to set of CPUS. We need
> > > > > > to associate a resource (IRQ or address) to a set of cpus (or more in
> > > > > > general, masters).
> > > > > 
> > > > > Hrm.  See, I think I may be misunderstanding the problem again,
> > > > > because with this description I can see no problem.  It's already up
> > > > > to the device binding to describe the purpose of each entry in the reg
> > > > > property.  So what's the problem with it just being part of the
> > > > > binding to say which reg entry is associated with which master port?
> > > > 
> > > > That's exactly what we are trying to do. But to associate the reg
> > > > property to a master port we need a phandle, how can we pull that off
> > > > otherwise ?
> > > 
> > > Ah, right.  I was meaning that the binding specifies the abstract port
> > > number that each reg entry is associated with.  I'm fine with adding a
> > > property to map the port numbers to master devices on the other end.
> > > I think treating it in two steps like that is better than thinking of
> > > the master phandles being  directly associated with reg entries,
> > > because it also handles cases like having a bank of common/global
> > > registers not associated with any port/master, or cases where the
> > > ports aren't all identical and some need more resources than others.
> > 
> > That's fine by me. Can you provide me with an example of how the bindings
> > should tie a specific reg property to an abstract port number ? reg properties
> > ordering (ie index) ? Or port number encoded in the reg property itself ?
> 
> I think I agree with David (assuming I understand his argument
> correctly). Rather than putting the information into the single node, it
> is more consistent in DT terms to put that information into each of the
> 'users' of the device; so for each master, put in a property that
> associates it with a specific port. Something like this (completely off
> the top of my head; not at all fully thought out):
> 
> cpu@0 {
> 	cci-port = <&cci1 0>;
> }
> 
> cpu@1 {
> 	cci-port = <&cci1 1>;
> }
> 
> cpu@2 {
> 	cci-port = <&cci1 2>;
> }
> 
> gpu@c8010000 {
> 	cci-port = <&cci1 3>;
> }

Nitpicking: the node above is not really a "master", it is a device tree
node representing GPU resources in the CPU address space since, if I got that
right, the device tree provides the address space view from a CPU perspective.

I am not saying this is right or wrong, just trying to make up my mind
myself to understand if this can cause any issues in the future.

> cci1: cci@c8000000 {
> 	reg = < ... >;
> }
> 
> An arguement against that would be that it is difficult to determine
> what all the bus-mastering devices are from looking at the cci node, but
> in real terms you would have a lookup table in the operating system
> /anyway/ for matching devices to ports and I would expect that as each
> device is discovered from the DT data the OS would populate it's lookup
> table.

Ok, I am not a big fan of this approach since it does not scale IMHO. I would
avoid adding a property to a cpu node every given time we need a device
requiring its resources to be linked to a (possibly set of) cpu(s).

If we add the property(ies) in the cci node, at least the changes become
self contained and the mechanism through which we link a resource to a
"master" becomes standard (phandle). All bits of info become available
through the cci node, but again, it is a choice to be made, I do not
know what's the best approach.

> It may be that Dave Martin's suggestion of using basically a dma-ranges
> type property with a phandle would be the correct interface. I've not
> thought enough about it yet to really get my head around it.

What I really really want to avoid is implementing something for CCI and then
realize that it does not work for other components with similar requirements.
With Mark and Dave Martin we tried to come up with a solution that can be
generalized, that was the main goal.

I will review again Dave Martin's suggestion and take into account your
suggestion (which is what David Gibson would like us to do if I got it
right, you formalized his proposal).

Thx,
Lorenzo

Re: [RFC] DT affinity bindings/representing bus masters with DT

[Dave Martin]

On Mon, Apr 15, 2013 at 02:43:02PM +0100, Lorenzo Pieralisi wrote:
> Thank you for having a look Grant.
> 
> On Sat, Apr 13, 2013 at 11:24:48PM +0100, Grant Likely wrote:
> > On Mon, 25 Mar 2013 15:20:39 +0000, Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org> wrote:
> > > On Tue, Mar 19, 2013 at 07:07:34AM +0000, David Gibson wrote:
> > > > On Mon, Mar 18, 2013 at 09:48:16AM +0000, Lorenzo Pieralisi wrote:
> > > > > > > I think we need to post what we have, it is really complex to explain
> > > > > > > the issue without a concrete example. To cut a long story short I
> > > > > > > would not say that the resources sit in different address spaces, it is
> > > > > > > that we need to associate those address ranges with specific bus masters.
> > > > > > > 
> > > > > > > We have to have a way to say:
> > > > > > > 
> > > > > > > "Address range 0x80001000 - 0x80001fff is used to programme the control
> > > > > > > registers associated with the port connected to master X".
> > > > > > > 
> > > > > > > When a CPU wants to programme a control port for a specific master, it
> > > > > > > needs to know what address range should be programmed.
> > > > > > > 
> > > > > > > I mentioned "resources" instead of addresses since the problem we are
> > > > > > > having is the same when it comes to map IRQs to set of CPUS. We need
> > > > > > > to associate a resource (IRQ or address) to a set of cpus (or more in
> > > > > > > general, masters).
> > > > > > 
> > > > > > Hrm.  See, I think I may be misunderstanding the problem again,
> > > > > > because with this description I can see no problem.  It's already up
> > > > > > to the device binding to describe the purpose of each entry in the reg
> > > > > > property.  So what's the problem with it just being part of the
> > > > > > binding to say which reg entry is associated with which master port?
> > > > > 
> > > > > That's exactly what we are trying to do. But to associate the reg
> > > > > property to a master port we need a phandle, how can we pull that off
> > > > > otherwise ?
> > > > 
> > > > Ah, right.  I was meaning that the binding specifies the abstract port
> > > > number that each reg entry is associated with.  I'm fine with adding a
> > > > property to map the port numbers to master devices on the other end.
> > > > I think treating it in two steps like that is better than thinking of
> > > > the master phandles being  directly associated with reg entries,
> > > > because it also handles cases like having a bank of common/global
> > > > registers not associated with any port/master, or cases where the
> > > > ports aren't all identical and some need more resources than others.
> > > 
> > > That's fine by me. Can you provide me with an example of how the bindings
> > > should tie a specific reg property to an abstract port number ? reg properties
> > > ordering (ie index) ? Or port number encoded in the reg property itself ?
> > 
> > I think I agree with David (assuming I understand his argument
> > correctly). Rather than putting the information into the single node, it
> > is more consistent in DT terms to put that information into each of the
> > 'users' of the device; so for each master, put in a property that
> > associates it with a specific port. Something like this (completely off
> > the top of my head; not at all fully thought out):
> > 
> > cpu@0 {
> > 	cci-port = <&cci1 0>;
> > }
> > 
> > cpu@1 {
> > 	cci-port = <&cci1 1>;
> > }
> > 
> > cpu@2 {
> > 	cci-port = <&cci1 2>;
> > }
> > 
> > gpu@c8010000 {
> > 	cci-port = <&cci1 3>;
> > }
> 
> Nitpicking: the node above is not really a "master", it is a device tree
> node representing GPU resources in the CPU address space since, if I got that
> right, the device tree provides the address space view from a CPU perspective.
> 
> I am not saying this is right or wrong, just trying to make up my mind
> myself to understand if this can cause any issues in the future.
> 
> > cci1: cci@c8000000 {
> > 	reg = < ... >;
> > }
> > 
> > An arguement against that would be that it is difficult to determine
> > what all the bus-mastering devices are from looking at the cci node, but
> > in real terms you would have a lookup table in the operating system
> > /anyway/ for matching devices to ports and I would expect that as each
> > device is discovered from the DT data the OS would populate it's lookup
> > table.
> 
> Ok, I am not a big fan of this approach since it does not scale IMHO. I would
> avoid adding a property to a cpu node every given time we need a device
> requiring its resources to be linked to a (possibly set of) cpu(s).

I think that putting the cci-port references in the relevant cluster node
(assuming that we agree to have cluster nodes) would be the logical
analogy.  This also correctly describes the system.  Putting those
properties in the cpu nodes directly makes it harder to describe/figure
out what topoogical level the coherency controls apply to, and that
really matters if we want to abstract this properly.


Most things in DT point in a master->slave direction, so the proposed CCI
binding looks a bit unusual in that its references currently point the
opposite way.  (Some references go the other way: for clocks and power,
we tend to refer backwards -- i.e., from the thing controlled to the
controller, just as in the current CCI proposal.)

I guess we'd need to figure out whether representing things master->slave
would be excessively painful for handling the CCI, and if so, why.

> If we add the property(ies) in the cci node, at least the changes become
> self contained and the mechanism through which we link a resource to a
> "master" becomes standard (phandle). All bits of info become available
> through the cci node, but again, it is a choice to be made, I do not
> know what's the best approach.
> 
> > It may be that Dave Martin's suggestion of using basically a dma-ranges
> > type property with a phandle would be the correct interface. I've not
> > thought enough about it yet to really get my head around it.
> 
> What I really really want to avoid is implementing something for CCI and then
> realize that it does not work for other components with similar requirements.
> With Mark and Dave Martin we tried to come up with a solution that can be
> generalized, that was the main goal.

I still think we don't describe that fact that CCI really is a bus, and
controlling the ports on there is related to attaching and detaching
devices on that bus.

That could allow for a more general abstraction, but I'm still not clear
on how it would be described, and whether it would be a step too far...

Cheers
---Dave

> I will review again Dave Martin's suggestion and take into account your
> suggestion (which is what David Gibson would like us to do if I got it
> right, you formalized his proposal).


> 
> Thx,
> Lorenzo
> 

Re: [RFC] DT affinity bindings/representing bus masters with DT

[Lorenzo Pieralisi]

On Tue, Apr 16, 2013 at 04:55:43PM +0100, Dave Martin wrote:
> On Mon, Apr 15, 2013 at 02:43:02PM +0100, Lorenzo Pieralisi wrote:
> > Thank you for having a look Grant.
> > 
> > On Sat, Apr 13, 2013 at 11:24:48PM +0100, Grant Likely wrote:
> > > On Mon, 25 Mar 2013 15:20:39 +0000, Lorenzo Pieralisi <lorenzo.pieralisi-5wv7dgnIgG8@public.gmane.org> wrote:
> > > > On Tue, Mar 19, 2013 at 07:07:34AM +0000, David Gibson wrote:
> > > > > On Mon, Mar 18, 2013 at 09:48:16AM +0000, Lorenzo Pieralisi wrote:
> > > > > > > > I think we need to post what we have, it is really complex to explain
> > > > > > > > the issue without a concrete example. To cut a long story short I
> > > > > > > > would not say that the resources sit in different address spaces, it is
> > > > > > > > that we need to associate those address ranges with specific bus masters.
> > > > > > > > 
> > > > > > > > We have to have a way to say:
> > > > > > > > 
> > > > > > > > "Address range 0x80001000 - 0x80001fff is used to programme the control
> > > > > > > > registers associated with the port connected to master X".
> > > > > > > > 
> > > > > > > > When a CPU wants to programme a control port for a specific master, it
> > > > > > > > needs to know what address range should be programmed.
> > > > > > > > 
> > > > > > > > I mentioned "resources" instead of addresses since the problem we are
> > > > > > > > having is the same when it comes to map IRQs to set of CPUS. We need
> > > > > > > > to associate a resource (IRQ or address) to a set of cpus (or more in
> > > > > > > > general, masters).
> > > > > > > 
> > > > > > > Hrm.  See, I think I may be misunderstanding the problem again,
> > > > > > > because with this description I can see no problem.  It's already up
> > > > > > > to the device binding to describe the purpose of each entry in the reg
> > > > > > > property.  So what's the problem with it just being part of the
> > > > > > > binding to say which reg entry is associated with which master port?
> > > > > > 
> > > > > > That's exactly what we are trying to do. But to associate the reg
> > > > > > property to a master port we need a phandle, how can we pull that off
> > > > > > otherwise ?
> > > > > 
> > > > > Ah, right.  I was meaning that the binding specifies the abstract port
> > > > > number that each reg entry is associated with.  I'm fine with adding a
> > > > > property to map the port numbers to master devices on the other end.
> > > > > I think treating it in two steps like that is better than thinking of
> > > > > the master phandles being  directly associated with reg entries,
> > > > > because it also handles cases like having a bank of common/global
> > > > > registers not associated with any port/master, or cases where the
> > > > > ports aren't all identical and some need more resources than others.
> > > > 
> > > > That's fine by me. Can you provide me with an example of how the bindings
> > > > should tie a specific reg property to an abstract port number ? reg properties
> > > > ordering (ie index) ? Or port number encoded in the reg property itself ?
> > > 
> > > I think I agree with David (assuming I understand his argument
> > > correctly). Rather than putting the information into the single node, it
> > > is more consistent in DT terms to put that information into each of the
> > > 'users' of the device; so for each master, put in a property that
> > > associates it with a specific port. Something like this (completely off
> > > the top of my head; not at all fully thought out):
> > > 
> > > cpu@0 {
> > > 	cci-port = <&cci1 0>;
> > > }
> > > 
> > > cpu@1 {
> > > 	cci-port = <&cci1 1>;
> > > }
> > > 
> > > cpu@2 {
> > > 	cci-port = <&cci1 2>;
> > > }
> > > 
> > > gpu@c8010000 {
> > > 	cci-port = <&cci1 3>;
> > > }
> > 
> > Nitpicking: the node above is not really a "master", it is a device tree
> > node representing GPU resources in the CPU address space since, if I got that
> > right, the device tree provides the address space view from a CPU perspective.
> > 
> > I am not saying this is right or wrong, just trying to make up my mind
> > myself to understand if this can cause any issues in the future.
> > 
> > > cci1: cci@c8000000 {
> > > 	reg = < ... >;
> > > }
> > > 
> > > An arguement against that would be that it is difficult to determine
> > > what all the bus-mastering devices are from looking at the cci node, but
> > > in real terms you would have a lookup table in the operating system
> > > /anyway/ for matching devices to ports and I would expect that as each
> > > device is discovered from the DT data the OS would populate it's lookup
> > > table.
> > 
> > Ok, I am not a big fan of this approach since it does not scale IMHO. I would
> > avoid adding a property to a cpu node every given time we need a device
> > requiring its resources to be linked to a (possibly set of) cpu(s).
> 
> I think that putting the cci-port references in the relevant cluster node
> (assuming that we agree to have cluster nodes) would be the logical
> analogy.  This also correctly describes the system.  Putting those
> properties in the cpu nodes directly makes it harder to describe/figure
> out what topoogical level the coherency controls apply to, and that
> really matters if we want to abstract this properly.

This would mean that cluster nodes becomes "HW" entities and would be
treated as cpu nodes are at the moment. That was not my intent, the
idea was to use nodes within cpu-map just to describe topology, that was it,
no property attached to those nodes. We need to figure out what we are
doing with cluster nodes, and then make a decision on CCI.

> Most things in DT point in a master->slave direction, so the proposed CCI
> binding looks a bit unusual in that its references currently point the
> opposite way.  (Some references go the other way: for clocks and power,
> we tend to refer backwards -- i.e., from the thing controlled to the
> controller, just as in the current CCI proposal.)
> 
> I guess we'd need to figure out whether representing things master->slave
> would be excessively painful for handling the CCI, and if so, why.

I would say we need to figure out how the device tree should describe
this kind of dependencies, and try to tackle the problem in a generic way,
not on a case-by-case one.

I had a look at dma bindings and they seem to follow your suggestion
(David's initial suggestion), I will try to follow it as well (but first
topology bindings have to be finalized).

> > If we add the property(ies) in the cci node, at least the changes become
> > self contained and the mechanism through which we link a resource to a
> > "master" becomes standard (phandle). All bits of info become available
> > through the cci node, but again, it is a choice to be made, I do not
> > know what's the best approach.
> > 
> > > It may be that Dave Martin's suggestion of using basically a dma-ranges
> > > type property with a phandle would be the correct interface. I've not
> > > thought enough about it yet to really get my head around it.
> > 
> > What I really really want to avoid is implementing something for CCI and then
> > realize that it does not work for other components with similar requirements.
> > With Mark and Dave Martin we tried to come up with a solution that can be
> > generalized, that was the main goal.
> 
> I still think we don't describe that fact that CCI really is a bus, and
> controlling the ports on there is related to attaching and detaching
> devices on that bus.
> 
> That could allow for a more general abstraction, but I'm still not clear
> on how it would be described, and whether it would be a step too far...

I think CCI should be described as a bus, but this does not change much
wrt its ports programming interfaces. It can be seen as a device in that
respect, and the problem we are facing would be indentical with an IP
block whose programming address space is allocated per-bus-master (ie
every bus master using that IP has a specific programming address range).

I might be wrong, but I have a feeling the two problems should be tackled
separately.

Cheers,
Lorenzo