Defining schemas for Device Tree

Defining schemas for Device Tree

[Tomasz Figa]

Hi,

As promised I am starting a discussion about Device Tree schema. Let's 
first shortly introduce the problem.

Device Tree is a text-based data structure used to describe hardware. Its 
main point is separation from kernel code, which has a lot of benefits, 
but, at the moment, also a huge drawback - there is no verification of 
device tree sources against defined bindings. All the dtc compiler does 
currently are syntax checks - no semantic analysis is performed (except
some really basic things). What this means is that anybody can put
anything in their device tree and end up with the dts compiling fine only
to find out that something is wrong at boot time.

Currently, device tree bindings are described in plain text documentation 
files, which can not be considered a formal way of binding description. 
While such documentation provides information for developers/users that 
need to work with particular bindings, it can not be easily used as input 
for validation of device tree sources. This means that we need to define a 
more formal way of binding description, in other words - Device Tree 
schema.

To find a solution for this problem, we must first answer several 
questions to determine a set of requirements we have to meet.

a) What is a device tree binding?
 
For our purposes, I will define a binding as internal format of some 
device tree node, which can be matched using of_find_matching_node(). In 
other words, key for a binding would be node name and/or value of 
compatible property and/or node type. Value for a binding would be a list 
of properties with their formats and/or subnodes with their bindings.
 
b) What information should be specified in schemas? What level of 
   granularity is required?

For each property we need to have at least following data specified:
 - property name (or property name format, e.g. regex),
 - whether the property is mandatory or optional,
 - data type of value.

As for now, I can think of following data types used in device trees:
 - boolean (i.e. without value),
 - array of strings (including single string),
 - array of u32 (including single u32),
 - specifier (aka phandle with args, including cases with 0 args),
 - variable-length cells (e.g. #address-cells of u32s).

Some properties might require a combination of data types to be specified 
or even an array of combinations, like interrupt-map property, which is an 
array of entries consisting of:
 - #address-cells u32s,
 - #interrupt-cells u32s,
 - specifier (phandle of interrupt controller and u32 of count defined by 
   #interrupt-cells of the controller).

We probably want to define allowed range of values for given property, be 
it contiguous or enumerated.

As for subnodes, I think we need to define following constraints:
 - node name (or node name format, e.g. regex),
 - optional or not,
 - how many nodes of this type can be present (one, limited, unlimited),
 - recursively define binding for such node type.

We probably also want human readable descriptions for all properties and 
subnodes, so a textual documentation (like the one currently available) 
could be generated from schemas.

c) What about generic bindings? (e.g. for subsystems like pinctrl or 
regulators)

This is where things get more interesting. Looks like we need some kind of 
inheritance for bindings or binding templates. Templates sound more 
appropriate here, because most of the generic bindings do not fully 
conform to what I defined as binding and need device-specific parameters 
to become so.

Let's consider first example taken from regulator subsystem.

	device {
		compatible = "foo,mydevice";
		/* ... */
		core-supply = <&regulator_a>;
		io-supply = <&regulator_b>;
		/* ... */
	};

Bindings of regulator subsystem define the way of regulator lookup to be 
based on property matching following definition:

	#define REGULATOR(name) name ## _supply = <&phandle>

As you can see, the binding is parametrized, i.e. part of it is defined 
globally, but part is device-specific. Similarly for pinctrl subsystem:

	device {
		compatible = "foo,mydevice";
		/* ... */
		pinctrl-names = "state0", "state1";
		pinctrl-0 = <&phandle>...;
		pinctrl-1 = <&phandle>...;
		/* ... */
	};

This binding is now parametrized in a more complex way:

	#define PINCTRL(name0, name1, ..., nameN) \
		pinctrl-names = name0, name1, ..., nameN; \
		pinctrl-0 = <&phandle>...; \
		pinctrl-1 = <&phandle>...; \
		... \
		pinctrl-N = <&phandle>...;

We need to have a way to describe this kind of inheritance, if we don't 
want to respecify generic attributes in all device bindings using them.

d) When should the validation happen and what should handle it?

In my opinion, similarly to compilation of board files, validation should 
be happening at dts compile time, to show any warnings or errors as early 
as possible.

Whether this should be integrated into dtc or rather handled by external 
tool is another question. Since we are already processing device tree 
sources in dtc, it might be reasonable to reuse its dts parsing 
infrastructure and add validation there, especially that dtc is supposed 
to already contain some infrastructure for doing checks on device tree as 
well. Nothing stops us from running validation on already compiled dtbs, 
though, using an extra tool.

e) What format should be used for Device Tree schema?

This is a non-trivial problem. Key criteria I can think of are as follows:
 - the whole set of information established above must be representable,
 - human-readable, easy to create and edit (extend), preferably similar to 
   something already existing, so could be easily learnt,
 - something that can be integrated with dtc with reasonable amount of 
   work or can reuse a lot (if not all) of already existing parsing code.

Okay, this should be enough to have some discussion. I will post a
follow-up with my proposal of schema format to separate general discussion
from discussion about the proposal, but this will happen tomorrow, as now
it's time to get some sleep.

For now please think about the points above and feel free to correct
anything wrong or suggest what else should be taken into consideration
for DT schemas. Let the discussion start.

Best regards,
Tomasz

Defining schemas for Device Tree

[jonsmirl at gmail.com]

There is another angle to this problem -- how do we make a single
schema for all device trees accepted by the kernel? We certainly don't
want a schema for each vendor where they can do whatever they want.

Obviously all devices in a device class are not identical but many of
their attributes are. The purpose of a unified schema is stop one
vendor from naming an attribute VOLTAGE, another VOLTS, another V, etc
when all of these attributes are doing the same thing - setting a
voltage. It is possible to use different names right now since they
are different device drivers. There's no overall architecture glue
(schema) holding the device class design together.

The main point of this unified schema is to help people creating new
device drivers. When ypu construct your device tree representation you
should first try your best to get it to fit into the existing device
class schema. Only after you determine that it is impossible to make
do with the current schema should you ask for the generic schema to be
extended to include whatever attribute you think you need. At that
point peer review will happen and hopefully a good solution will
ensue.

----------------

Imagine what might be possible with device trees in a few years...

A single kernel image can load on any embedded system. The device tree
sorts out all of the driver loading and tells how everything is wired
together. There is zero board specific code in the kernel. We're
getting close to this one.

And in the far future - send your device tree off to a cloud PCB
printer and get your custom embedded system back in the mail. Sign me
up for this one!

-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[David Woodhouse]

On Sun, 2013-07-28 at 21:30 -0400, jonsmirl at gmail.com wrote:
> Imagine what might be possible with device trees in a few years...
> 
> A single kernel image can load on any embedded system. The device tree
> sorts out all of the driver loading and tells how everything is wired
> together. There is zero board specific code in the kernel. We're
> getting close to this one.

Weren't we doing this years ago on PowerPC? I'm sure I've heard of
Freescale bringing up a new SoC without touching a line of kernel code.

-- 
dwmw2

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5745 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130729/f5c1e814/attachment.bin>

Defining schemas for Device Tree

[Tomasz Figa]

On Sunday 28 of July 2013 21:30:22 jonsmirl at gmail.com wrote:
> There is another angle to this problem -- how do we make a single
> schema for all device trees accepted by the kernel? We certainly don't
> want a schema for each vendor where they can do whatever they want.

Aren't we trying to do it the other way around? We define a single schema 
for device trees acceptable by mainline kernels, which should be used by 
vendors, if they want their hardware support to be mainlined.

This is exactly the same situation as with introducing new code (arch, 
drivers, whatever). Vendors are free to do whatever they want, but only 
code that matches kernel standards (i.e. uses existing generic interfaces, 
not proprietary ones) is going to be accepted.

> Obviously all devices in a device class are not identical but many of
> their attributes are. The purpose of a unified schema is stop one
> vendor from naming an attribute VOLTAGE, another VOLTS, another V, etc
> when all of these attributes are doing the same thing - setting a
> voltage. It is possible to use different names right now since they
> are different device drivers. There's no overall architecture glue
> (schema) holding the device class design together.

Well, as of now, there are only some unwritten agreements for generic 
bindings, so if we have a generic bindings for voltage regulators, they 
should be used, not something new get introduced. Only then such code can 
be mainlined.

> The main point of this unified schema is to help people creating new
> device drivers. When ypu construct your device tree representation you
> should first try your best to get it to fit into the existing device
> class schema. Only after you determine that it is impossible to make
> do with the current schema should you ask for the generic schema to be
> extended to include whatever attribute you think you need. At that
> point peer review will happen and hopefully a good solution will
> ensue.

Yes, I believe this is what we want to achieve.

> ----------------
> 
> Imagine what might be possible with device trees in a few years...
> 
> A single kernel image can load on any embedded system. The device tree
> sorts out all of the driver loading and tells how everything is wired
> together. There is zero board specific code in the kernel. We're
> getting close to this one.

Well, we already have platforms with zero board specific code in the 
kernel, e.g. Exynos.

> And in the far future - send your device tree off to a cloud PCB
> printer and get your custom embedded system back in the mail. Sign me
> up for this one!

Haha, that would be something.

Best regards,
Tomasz

Defining schemas for Device Tree

[Jason Cooper]

On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> Hi,
> 
> As promised I am starting a discussion about Device Tree schema. Let's 
> first shortly introduce the problem.
> 
> Device Tree is a text-based data structure used to describe hardware.

Just a clarifying point here:  I think it would be more accurate to say
"devicetree describes how hardware on a given board is *configured*".
The driver learns from the compatible property which IP block it is
dealing with and handles the necessary quirks.  How that IP block is
attached, and interfaced with is DT territory, that it needs an extra
10ns delay (outside of spec) is something the driver should know and
work around.

> Its 
> main point is separation from kernel code, which has a lot of benefits, 
> but, at the moment, also a huge drawback - there is no verification of 
> device tree sources against defined bindings. All the dtc compiler does 
> currently are syntax checks - no semantic analysis is performed (except
> some really basic things). What this means is that anybody can put
> anything in their device tree and end up with the dts compiling fine only
> to find out that something is wrong at boot time.
> 
> Currently, device tree bindings are described in plain text documentation 
> files, which can not be considered a formal way of binding description. 
> While such documentation provides information for developers/users that 
> need to work with particular bindings, it can not be easily used as input 
> for validation of device tree sources. This means that we need to define a 
> more formal way of binding description, in other words - Device Tree 
> schema.

+1

> To find a solution for this problem, we must first answer several 
> questions to determine a set of requirements we have to meet.
> 
> a) What is a device tree binding?
>  
> For our purposes, I will define a binding as internal format of some 
> device tree node, which can be matched using of_find_matching_node(). In 
> other words, key for a binding would be node name and/or value of 
> compatible property and/or node type. Value for a binding would be a list 
> of properties with their formats and/or subnodes with their bindings.
>  
> b) What information should be specified in schemas? What level of 
>    granularity is required?

One item I don't see in this list is node ordering.  There's been some
discussion lately on deferred probing (re boot times).  If we were to
intentionally declare that DT are parsed in the order written, then a
lot of deferred probes could be avoided by moving eg the pinctrl node to
near the top of the tree.

This doesn't impact buses as much, since the nodes needing the bus are
already children.  However, anything accessed via phandles: pins,
clocks, regulators, etc could benefit from declaring and enforcing this.
Eg having the dtc warn when a phandle is used before it's corresponding
node is declared.

Not critical though, just a thought.

thx,

Jason.

> For each property we need to have at least following data specified:
>  - property name (or property name format, e.g. regex),
>  - whether the property is mandatory or optional,
>  - data type of value.
> 
> As for now, I can think of following data types used in device trees:
>  - boolean (i.e. without value),
>  - array of strings (including single string),
>  - array of u32 (including single u32),
>  - specifier (aka phandle with args, including cases with 0 args),
>  - variable-length cells (e.g. #address-cells of u32s).
> 
> Some properties might require a combination of data types to be specified 
> or even an array of combinations, like interrupt-map property, which is an 
> array of entries consisting of:
>  - #address-cells u32s,
>  - #interrupt-cells u32s,
>  - specifier (phandle of interrupt controller and u32 of count defined by 
>    #interrupt-cells of the controller).
> 
> We probably want to define allowed range of values for given property, be 
> it contiguous or enumerated.
> 
> As for subnodes, I think we need to define following constraints:
>  - node name (or node name format, e.g. regex),
>  - optional or not,
>  - how many nodes of this type can be present (one, limited, unlimited),
>  - recursively define binding for such node type.
> 
> We probably also want human readable descriptions for all properties and 
> subnodes, so a textual documentation (like the one currently available) 
> could be generated from schemas.
> 
> c) What about generic bindings? (e.g. for subsystems like pinctrl or 
> regulators)
> 
> This is where things get more interesting. Looks like we need some kind of 
> inheritance for bindings or binding templates. Templates sound more 
> appropriate here, because most of the generic bindings do not fully 
> conform to what I defined as binding and need device-specific parameters 
> to become so.
> 
> Let's consider first example taken from regulator subsystem.
> 
> 	device {
> 		compatible = "foo,mydevice";
> 		/* ... */
> 		core-supply = <&regulator_a>;
> 		io-supply = <&regulator_b>;
> 		/* ... */
> 	};
> 
> Bindings of regulator subsystem define the way of regulator lookup to be 
> based on property matching following definition:
> 
> 	#define REGULATOR(name) name ## _supply = <&phandle>
> 
> As you can see, the binding is parametrized, i.e. part of it is defined 
> globally, but part is device-specific. Similarly for pinctrl subsystem:
> 
> 	device {
> 		compatible = "foo,mydevice";
> 		/* ... */
> 		pinctrl-names = "state0", "state1";
> 		pinctrl-0 = <&phandle>...;
> 		pinctrl-1 = <&phandle>...;
> 		/* ... */
> 	};
> 
> This binding is now parametrized in a more complex way:
> 
> 	#define PINCTRL(name0, name1, ..., nameN) \
> 		pinctrl-names = name0, name1, ..., nameN; \
> 		pinctrl-0 = <&phandle>...; \
> 		pinctrl-1 = <&phandle>...; \
> 		... \
> 		pinctrl-N = <&phandle>...;
> 
> We need to have a way to describe this kind of inheritance, if we don't 
> want to respecify generic attributes in all device bindings using them.
> 
> d) When should the validation happen and what should handle it?
> 
> In my opinion, similarly to compilation of board files, validation should 
> be happening at dts compile time, to show any warnings or errors as early 
> as possible.
> 
> Whether this should be integrated into dtc or rather handled by external 
> tool is another question. Since we are already processing device tree 
> sources in dtc, it might be reasonable to reuse its dts parsing 
> infrastructure and add validation there, especially that dtc is supposed 
> to already contain some infrastructure for doing checks on device tree as 
> well. Nothing stops us from running validation on already compiled dtbs, 
> though, using an extra tool.
> 
> e) What format should be used for Device Tree schema?
> 
> This is a non-trivial problem. Key criteria I can think of are as follows:
>  - the whole set of information established above must be representable,
>  - human-readable, easy to create and edit (extend), preferably similar to 
>    something already existing, so could be easily learnt,
>  - something that can be integrated with dtc with reasonable amount of 
>    work or can reuse a lot (if not all) of already existing parsing code.
> 
> Okay, this should be enough to have some discussion. I will post a
> follow-up with my proposal of schema format to separate general discussion
> from discussion about the proposal, but this will happen tomorrow, as now
> it's time to get some sleep.
> 
> For now please think about the points above and feel free to correct
> anything wrong or suggest what else should be taken into consideration
> for DT schemas. Let the discussion start.
> 
> Best regards,
> Tomasz
> 

Defining schemas for Device Tree

[Dave Martin]

On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> > Hi,
> > 
> > As promised I am starting a discussion about Device Tree schema. Let's 
> > first shortly introduce the problem.
> > 
> > Device Tree is a text-based data structure used to describe hardware.
> 
> Just a clarifying point here:  I think it would be more accurate to say
> "devicetree describes how hardware on a given board is *configured*".
> The driver learns from the compatible property which IP block it is
> dealing with and handles the necessary quirks.  How that IP block is
> attached, and interfaced with is DT territory, that it needs an extra
> 10ns delay (outside of spec) is something the driver should know and
> work around.
> 
> > Its 
> > main point is separation from kernel code, which has a lot of benefits, 
> > but, at the moment, also a huge drawback - there is no verification of 
> > device tree sources against defined bindings. All the dtc compiler does 
> > currently are syntax checks - no semantic analysis is performed (except
> > some really basic things). What this means is that anybody can put
> > anything in their device tree and end up with the dts compiling fine only
> > to find out that something is wrong at boot time.
> > 
> > Currently, device tree bindings are described in plain text documentation 
> > files, which can not be considered a formal way of binding description. 
> > While such documentation provides information for developers/users that 
> > need to work with particular bindings, it can not be easily used as input 
> > for validation of device tree sources. This means that we need to define a 
> > more formal way of binding description, in other words - Device Tree 
> > schema.
> 
> +1
> 
> > To find a solution for this problem, we must first answer several 
> > questions to determine a set of requirements we have to meet.
> > 
> > a) What is a device tree binding?
> >  
> > For our purposes, I will define a binding as internal format of some 
> > device tree node, which can be matched using of_find_matching_node(). In 
> > other words, key for a binding would be node name and/or value of 
> > compatible property and/or node type. Value for a binding would be a list 
> > of properties with their formats and/or subnodes with their bindings.
> >  
> > b) What information should be specified in schemas? What level of 
> >    granularity is required?
> 
> One item I don't see in this list is node ordering.  There's been some
> discussion lately on deferred probing (re boot times).  If we were to
> intentionally declare that DT are parsed in the order written, then a
> lot of deferred probes could be avoided by moving eg the pinctrl node to
> near the top of the tree.
> 
> This doesn't impact buses as much, since the nodes needing the bus are
> already children.  However, anything accessed via phandles: pins,
> clocks, regulators, etc could benefit from declaring and enforcing this.
> Eg having the dtc warn when a phandle is used before it's corresponding
> node is declared.
> 
> Not critical though, just a thought.

I don't think that siblings have any defined order in DT.  If reading a
device tree, there's no guarantee you get nodes or properties out in the
same order as the original .dts file.

Provided child/parent relationships are maintained and the set of nodes
and values is the same, I think completely rearranging a .dts file does
not change its meaning.

"depends-on" relationships mostly have to come from the semantics of
the bindings themselves: for example, if a device is connected to some
clocks and regulators, the kernel may need to probe those first.

Cheers
---Dave

Defining schemas for Device Tree

[Jason Gunthorpe]

On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:

> Provided child/parent relationships are maintained and the set of nodes
> and values is the same, I think completely rearranging a .dts file does
> not change its meaning.
> 
> "depends-on" relationships mostly have to come from the semantics of
> the bindings themselves: for example, if a device is connected to some
> clocks and regulators, the kernel may need to probe those first.

There is already code in the kernel to sort the interrupt controllers
from DT and start them up in the right order.

I've often thought to myself that the deferred probe hack could be
avoided if some layer in the DT stack could do the same kind of topo
sorting, but more generally.

Just topo-sorting based on phandles would probably get things quite
far along??

Jason

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[Jason Cooper]

On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:

> > > b) What information should be specified in schemas? What level of 
> > >    granularity is required?
> > 
> > One item I don't see in this list is node ordering.  There's been some
> > discussion lately on deferred probing (re boot times).  If we were to
> > intentionally declare that DT are parsed in the order written, then a
> > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > near the top of the tree.
> > 
> > This doesn't impact buses as much, since the nodes needing the bus are
> > already children.  However, anything accessed via phandles: pins,
> > clocks, regulators, etc could benefit from declaring and enforcing this.
> > Eg having the dtc warn when a phandle is used before it's corresponding
> > node is declared.
> > 
> > Not critical though, just a thought.
> 
> I don't think that siblings have any defined order in DT.  If reading a
> device tree, there's no guarantee you get nodes or properties out in the
> same order as the original .dts file.

That's why I raised the point.  If people think encoding initialization
order in the DT is a good idea, then we should change the dtc so it
compiles/decompiles in the same order.

> Provided child/parent relationships are maintained and the set of nodes
> and values is the same, I think completely rearranging a .dts file does
> not change its meaning.
> 
> "depends-on" relationships mostly have to come from the semantics of
> the bindings themselves: for example, if a device is connected to some
> clocks and regulators, the kernel may need to probe those first.

true, the answer to this problem may be to create a depgraph of the
nodes based on phandles and child status, then init.  However, if the
goal is to accelerate boot times, then that should not be calculated
during each boot, especially since it doesn't likely change from boot to
boot.

Which means it would either go in the dtc (dts node ordering is
irrelevant), or in the dts.  I'm inclined to say dtc should do it, but I
like the aesthetics of things being in the proper order in something I
can read.  After all, C requires functions to be declared before use,
even though the compiler could figure it out.

thx,

Jason.

[Ksummit-2013-discuss] Defining schemas for Device Tree

[Jason Cooper]

On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:

> > > b) What information should be specified in schemas? What level of 
> > >    granularity is required?
> > 
> > One item I don't see in this list is node ordering.  There's been some
> > discussion lately on deferred probing (re boot times).  If we were to
> > intentionally declare that DT are parsed in the order written, then a
> > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > near the top of the tree.
> > 
> > This doesn't impact buses as much, since the nodes needing the bus are
> > already children.  However, anything accessed via phandles: pins,
> > clocks, regulators, etc could benefit from declaring and enforcing this.
> > Eg having the dtc warn when a phandle is used before it's corresponding
> > node is declared.
> > 
> > Not critical though, just a thought.
> 
> I don't think that siblings have any defined order in DT.  If reading a
> device tree, there's no guarantee you get nodes or properties out in the
> same order as the original .dts file.

That's why I raised the point.  If people think encoding initialization
order in the DT is a good idea, then we should change the dtc so it
compiles/decompiles in the same order.

> Provided child/parent relationships are maintained and the set of nodes
> and values is the same, I think completely rearranging a .dts file does
> not change its meaning.
> 
> "depends-on" relationships mostly have to come from the semantics of
> the bindings themselves: for example, if a device is connected to some
> clocks and regulators, the kernel may need to probe those first.

true, the answer to this problem may be to create a depgraph of the
nodes based on phandles and child status, then init.  However, if the
goal is to accelerate boot times, then that should not be calculated
during each boot, especially since it doesn't likely change from boot to
boot.

Which means it would either go in the dtc (dts node ordering is
irrelevant), or in the dts.  I'm inclined to say dtc should do it, but I
like the aesthetics of things being in the proper order in something I
can read.  After all, C requires functions to be declared before use,
even though the compiler could figure it out.

thx,

Jason.

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[Jason Gunthorpe]

On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:

> true, the answer to this problem may be to create a depgraph of the
> nodes based on phandles and child status, then init.  However, if the
> goal is to accelerate boot times, then that should not be calculated
> during each boot, especially since it doesn't likely change from boot to
> boot.

That seems like a really smart idea. dtc could store the startup-order
and dependencies in the dtb someplace and the kernel can just run
through that. Using a combination of declaration order and
topo-phandle sorting would probably cover all cases???

Jason

[Ksummit-2013-discuss] Defining schemas for Device Tree

[Jason Gunthorpe]

On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:

> true, the answer to this problem may be to create a depgraph of the
> nodes based on phandles and child status, then init.  However, if the
> goal is to accelerate boot times, then that should not be calculated
> during each boot, especially since it doesn't likely change from boot to
> boot.

That seems like a really smart idea. dtc could store the startup-order
and dependencies in the dtb someplace and the kernel can just run
through that. Using a combination of declaration order and
topo-phandle sorting would probably cover all cases???

Jason

Defining schemas for Device Tree

[Jason Gunthorpe]

On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
 
> e) What format should be used for Device Tree schema?
> 
> This is a non-trivial problem. Key criteria I can think of are as
> follows:

You may wish to think about this as a linter+checker (like
sparse/coverity, etc) and code the 'schema' in C code.

As you've noticed already there are many complex cases. Look at the
PCI bindings, for instance, the complexity of what needs to be valided
is very high.

Jason

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:

> true, the answer to this problem may be to create a depgraph of the
> nodes based on phandles and child status, then init.  However, if the
> goal is to accelerate boot times, then that should not be calculated
> during each boot, especially since it doesn't likely change from boot to
> boot.

FWIW this is broadly speaking what deferred probing is supposed to do -
it seemed too complex and fragile to make the graph for board files so
Grant came up with the idea of deferred probing which aims to sidestep
the problem and work it out dynamically, also providing a fallback in
case something that has actually optimised the ordering misses some case
(cleaning up after the effects of parallel probes for example).

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] Defining schemas for Device Tree

[Mark Brown]

On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:

> true, the answer to this problem may be to create a depgraph of the
> nodes based on phandles and child status, then init.  However, if the
> goal is to accelerate boot times, then that should not be calculated
> during each boot, especially since it doesn't likely change from boot to
> boot.

FWIW this is broadly speaking what deferred probing is supposed to do -
it seemed too complex and fragile to make the graph for board files so
Grant came up with the idea of deferred probing which aims to sidestep
the problem and work it out dynamically, also providing a fallback in
case something that has actually optimised the ordering misses some case
(cleaning up after the effects of parallel probes for example).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130729/2f39878e/attachment.sig>

Defining schemas for Device Tree

[Stephen Warren]

On 07/28/2013 06:21 PM, Tomasz Figa wrote:
...
> b) What information should be specified in schemas? What level of 
>    granularity is required?
> 
> For each property we need to have at least following data specified:
>  - property name (or property name format, e.g. regex),
>  - whether the property is mandatory or optional,
>  - data type of value.

We might want general restrictions on node/property names, e.g. it seems
we generally prefer - rather than _ in them, although there are a few
entrenched exceptions.

> As for now, I can think of following data types used in device trees:
>  - boolean (i.e. without value),
>  - array of strings (including single string),
>  - array of u32 (including single u32),
>  - specifier (aka phandle with args, including cases with 0 args),
>  - variable-length cells (e.g. #address-cells of u32s).
> 
> Some properties might require a combination of data types to be specified 

I believe phandles are just cells in terms of the storage type at least.
Hence, I'd argue we don't need mixing of types (boolean, cell, string),
but I see your point about semantically needing to represent different
cells meaning different things.

> or even an array of combinations, like interrupt-map property, which is an 
> array of entries consisting of:
>  - #address-cells u32s,
>  - #interrupt-cells u32s,
>  - specifier (phandle of interrupt controller and u32 of count defined by 
>    #interrupt-cells of the controller).
> 
> We probably want to define allowed range of values for given property, be 
> it contiguous or enumerated.

Where defining the length of a property (or part of it) in terms of
another #xxx-cells property, we need to specify where that other
property exists. Sometimes it's the nearest parent node containing the
property (typical for #address-cells), sometimes it's the node
containing the property itself (typical for one of the #address-cells
referenced by a ranges property), and sometimes it's in the node
referenced by the phandle associated with a specifier.

Other content restrictions might be:

* List must contain (at least) these entries/values.

* List can't contain any other entries/values not specified here.

* List must be in this order vs. any order.

* List logical length (in entries, not bytes/cells) must match the
logical length of another cell (consider clocks/clock-names).

* Ordering of one property must match ordering of another property
(again consider clocks/clock-names), although it'd be difficult to tell
whether this condition was met, perhaps we can at least document it.

I'm sure there will be many more criteria to validate that we're
forgetting. This is possible the most complex area?

> As for subnodes, I think we need to define following constraints:
>  - node name (or node name format, e.g. regex),

I don't think node names are supposed to convey any semantic meaning, so
they probably shouldn't be restricted (much?) by schema.

> d) When should the validation happen and what should handle it?
...
> well. Nothing stops us from running validation on already compiled dtbs, 
> though, using an extra tool.

In a DTB, I'm not sure whether all the type information is still present.

For example, in DTS, the following are very obviously different:

prop = "abc", "def";
prop = <0x61626300> <0x64656600>;

... whereas in the DTB, I think they are both just two 32-bit values
that just happen to encode string data or not. Presumably we want the
validator to force string properties to be specified using string syntax?

Equally, I'm not sure there's any difference between:

prop = <&node>;
prop = <0x23648689>;

... in the DTB, except that the value just *happens* to match another
node's phandle value, which could end up leading to false positive
matches if only applied on the raw values not the syntax?

Defining schemas for Device Tree

[Tomasz Figa]

On Monday 29 of July 2013 15:47:55 Stephen Warren wrote:
> On 07/28/2013 06:21 PM, Tomasz Figa wrote:
> ...
> 
> > b) What information should be specified in schemas? What level of
> > 
> >    granularity is required?
> > 
> > For each property we need to have at least following data specified:
> >  - property name (or property name format, e.g. regex),
> >  - whether the property is mandatory or optional,
> >  - data type of value.
> 
> We might want general restrictions on node/property names, e.g. it seems
> we generally prefer - rather than _ in them, although there are a few
> entrenched exceptions.

Yes.

> > As for now, I can think of following data types used in device trees:
> >  - boolean (i.e. without value),
> >  - array of strings (including single string),
> >  - array of u32 (including single u32),
> >  - specifier (aka phandle with args, including cases with 0 args),
> >  - variable-length cells (e.g. #address-cells of u32s).
> > 
> > Some properties might require a combination of data types to be specified
> 
> I believe phandles are just cells in terms of the storage type at least.
> Hence, I'd argue we don't need mixing of types (boolean, cell, string),
> but I see your point about semantically needing to represent different
> cells meaning different things.

Let me return to this point a bit later in this message.

> > or even an array of combinations, like interrupt-map property, which is an
> > 
> > array of entries consisting of:
> >  - #address-cells u32s,
> >  - #interrupt-cells u32s,
> >  - specifier (phandle of interrupt controller and u32 of count defined by
> >  
> >    #interrupt-cells of the controller).
> > 
> > We probably want to define allowed range of values for given property, be
> > it contiguous or enumerated.
> 
> Where defining the length of a property (or part of it) in terms of
> another #xxx-cells property, we need to specify where that other
> property exists. Sometimes it's the nearest parent node containing the
> property (typical for #address-cells), sometimes it's the node
> containing the property itself (typical for one of the #address-cells
> referenced by a ranges property), and sometimes it's in the node
> referenced by the phandle associated with a specifier.
> 
> Other content restrictions might be:
> 
> * List must contain (at least) these entries/values.
> 
> * List can't contain any other entries/values not specified here.
> 
> * List must be in this order vs. any order.
> 
> * List logical length (in entries, not bytes/cells) must match the
> logical length of another cell (consider clocks/clock-names).
> 
> * Ordering of one property must match ordering of another property
> (again consider clocks/clock-names), although it'd be difficult to tell
> whether this condition was met, perhaps we can at least document it.
> 
> I'm sure there will be many more criteria to validate that we're
> forgetting. This is possible the most complex area?

Right. Still, this is about the granularity of checks we really need. Do we 
need to check for all those restrictions?

> > As for subnodes, I think we need to define following constraints:
> >  - node name (or node name format, e.g. regex),
> 
> I don't think node names are supposed to convey any semantic meaning, so
> they probably shouldn't be restricted (much?) by schema.

There are bindings currently that require specific node names. Since 
of_match_node() can do matching by names as well, I think we should care about 
node names too,

> > d) When should the validation happen and what should handle it?
> 
> ...
> 
> > well. Nothing stops us from running validation on already compiled dtbs,
> > though, using an extra tool.
> 
> In a DTB, I'm not sure whether all the type information is still present.
> 
> For example, in DTS, the following are very obviously different:
> 
> prop = "abc", "def";
> prop = <0x61626300> <0x64656600>;
> 
> ... whereas in the DTB, I think they are both just two 32-bit values
> that just happen to encode string data or not. Presumably we want the
> validator to force string properties to be specified using string syntax?
> 
> Equally, I'm not sure there's any difference between:
> 
> prop = <&node>;
> prop = <0x23648689>;
> 
> ... in the DTB, except that the value just *happens* to match another
> node's phandle value, which could end up leading to false positive
> matches if only applied on the raw values not the syntax?

This is a good point. From driver's perspective it doesn't matter how you 
specify value of some property, as long as it can be parsed using the same 
code. As a more trivial example, consider following dts code:

	array-prop = <0>, <1>, <2>;
	array-prop-2 = <0 1 2>;

Both array props contain the same data and can be parsed using the same code, 
but from reader's perspective they mean something slightly different - an array 
with 1 cell per element and a single entry of 3 cells.

Now back to the point I deferred my reply to, I think we need different 
handling for at least phandles and properties of which number of cells is 
specified by one or more properties (possibly in other nodes).

At DTS level phandle properties should use the phandle syntax, pointing at 
some node, while at DTB level a raw phandle value could be validated by 
checking if it matches with phandle of some node.

For properties ussing #xxx-cells properties we need to check for correct 
number of cells., which involves getting values of right #xxx-cells 
properties.

Best regards,
Tomasz

Defining schemas for Device Tree

[jonsmirl at gmail.com]

The schema should enforce some kind of regularity into the description
of similar devices.  Take SPI for example. I pulled these SPI
controller definitions from the kernel source. Obviously all of these
controllers are different, but it seems to me that their definitions
could be made more regular. DMA seems like it could use a lot of help.

spi_0: spi at 13920000 {
compatible = "samsung,exynos4210-spi";
reg = <0x13920000 0x100>;
interrupts = <0 66 0>;
tx-dma-channel = <&pdma0 7>; /* preliminary */
rx-dma-channel = <&pdma0 6>; /* preliminary */
#address-cells = <1>;
#size-cells = <0>;
clocks = <&clock 327>, <&clock 159>;
clock-names = "spi", "spi_busclk0";
pinctrl-names = "default";
pinctrl-0 = <&spi0_bus>;
status = "disabled";
};

spi1: spi at 1f0e000 {
#address-cells = <1>;
#size-cells = <0>;
compatible = "ti,da830-spi";
reg = <0x30e000 0x1000>;
num-cs = <4>;
ti,davinci-spi-intr-line = <1>;
interrupts = <56>;
status = "disabled";
};

spi0: spi at b00d0000 {
cell-index = <0>;
compatible = "sirf,prima2-spi";
reg = <0xb00d0000 0x10000>;
interrupts = <15>;
sirf,spi-num-chipselects = <1>;
cs-gpios = <&gpio 0 0>;
sirf,spi-dma-rx-channel = <25>;
sirf,spi-dma-tx-channel = <20>;
#address-cells = <1>;
#size-cells = <0>;
clocks = <&clks 19>;
status = "disabled";
};

spi0: spi at f0000000 {
#address-cells = <1>;
#size-cells = <0>;
compatible = "atmel,at91rm9200-spi";
reg = <0xf0000000 0x100>;
interrupts = <13 IRQ_TYPE_LEVEL_HIGH 3>;
dmas = <&dma0 1 AT91_DMA_CFG_PER_ID(1)>,
      <&dma0 1 AT91_DMA_CFG_PER_ID(2)>;
dma-names = "tx", "rx";
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_spi0>;
status = "disabled";
};

spi0: spi at 48030000 {
compatible = "ti,omap4-mcspi";
#address-cells = <1>;
#size-cells = <0>;
reg = <0x48030000 0x400>;
interrupts = <65>;
ti,spi-num-cs = <2>;
ti,hwmods = "spi0";
status = "disabled";
};

mcspi1: spi at 48098000 {
compatible = "ti,omap4-mcspi";
reg = <0x48098000 0x200>;
interrupts = <GIC_SPI 65 IRQ_TYPE_LEVEL_HIGH>;
#address-cells = <1>;
#size-cells = <0>;
ti,hwmods = "mcspi1";
ti,spi-num-cs = <4>;
dmas = <&sdma 35>,
      <&sdma 36>,
      <&sdma 37>,
      <&sdma 38>,
      <&sdma 39>,
      <&sdma 40>,
      <&sdma 41>,
      <&sdma 42>;
dma-names = "tx0", "rx0", "tx1", "rx1",
   "tx2", "rx2", "tx3", "rx3";
};

spi at 7000d400 {
compatible = "nvidia,tegra114-spi";
reg = <0x7000d400 0x200>;
interrupts = <GIC_SPI 59 IRQ_TYPE_LEVEL_HIGH>;
nvidia,dma-request-selector = <&apbdma 15>;
#address-cells = <1>;
#size-cells = <0>;
clocks = <&tegra_car TEGRA114_CLK_SBC1>;
clock-names = "spi";
status = "disabled";
};

On Mon, Jul 29, 2013 at 5:47 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 07/28/2013 06:21 PM, Tomasz Figa wrote:
> ...
>> b) What information should be specified in schemas? What level of
>>    granularity is required?
>>
>> For each property we need to have at least following data specified:
>>  - property name (or property name format, e.g. regex),
>>  - whether the property is mandatory or optional,
>>  - data type of value.
>
> We might want general restrictions on node/property names, e.g. it seems
> we generally prefer - rather than _ in them, although there are a few
> entrenched exceptions.
>
>> As for now, I can think of following data types used in device trees:
>>  - boolean (i.e. without value),
>>  - array of strings (including single string),
>>  - array of u32 (including single u32),
>>  - specifier (aka phandle with args, including cases with 0 args),
>>  - variable-length cells (e.g. #address-cells of u32s).
>>
>> Some properties might require a combination of data types to be specified
>
> I believe phandles are just cells in terms of the storage type at least.
> Hence, I'd argue we don't need mixing of types (boolean, cell, string),
> but I see your point about semantically needing to represent different
> cells meaning different things.
>
>> or even an array of combinations, like interrupt-map property, which is an
>> array of entries consisting of:
>>  - #address-cells u32s,
>>  - #interrupt-cells u32s,
>>  - specifier (phandle of interrupt controller and u32 of count defined by
>>    #interrupt-cells of the controller).
>>
>> We probably want to define allowed range of values for given property, be
>> it contiguous or enumerated.
>
> Where defining the length of a property (or part of it) in terms of
> another #xxx-cells property, we need to specify where that other
> property exists. Sometimes it's the nearest parent node containing the
> property (typical for #address-cells), sometimes it's the node
> containing the property itself (typical for one of the #address-cells
> referenced by a ranges property), and sometimes it's in the node
> referenced by the phandle associated with a specifier.
>
> Other content restrictions might be:
>
> * List must contain (at least) these entries/values.
>
> * List can't contain any other entries/values not specified here.
>
> * List must be in this order vs. any order.
>
> * List logical length (in entries, not bytes/cells) must match the
> logical length of another cell (consider clocks/clock-names).
>
> * Ordering of one property must match ordering of another property
> (again consider clocks/clock-names), although it'd be difficult to tell
> whether this condition was met, perhaps we can at least document it.
>
> I'm sure there will be many more criteria to validate that we're
> forgetting. This is possible the most complex area?
>
>> As for subnodes, I think we need to define following constraints:
>>  - node name (or node name format, e.g. regex),
>
> I don't think node names are supposed to convey any semantic meaning, so
> they probably shouldn't be restricted (much?) by schema.
>
>> d) When should the validation happen and what should handle it?
> ...
>> well. Nothing stops us from running validation on already compiled dtbs,
>> though, using an extra tool.
>
> In a DTB, I'm not sure whether all the type information is still present.
>
> For example, in DTS, the following are very obviously different:
>
> prop = "abc", "def";
> prop = <0x61626300> <0x64656600>;
>
> ... whereas in the DTB, I think they are both just two 32-bit values
> that just happen to encode string data or not. Presumably we want the
> validator to force string properties to be specified using string syntax?
>
> Equally, I'm not sure there's any difference between:
>
> prop = <&node>;
> prop = <0x23648689>;
>
> ... in the DTB, except that the value just *happens* to match another
> node's phandle value, which could end up leading to false positive
> matches if only applied on the raw values not the syntax?
>



-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[Tomasz Figa]

On Monday 29 of July 2013 12:15:50 Jason Gunthorpe wrote:
> On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> > e) What format should be used for Device Tree schema?
> > 
> > This is a non-trivial problem. Key criteria I can think of are as
> 
> > follows:
> You may wish to think about this as a linter+checker (like
> sparse/coverity, etc) and code the 'schema' in C code.
> 
> As you've noticed already there are many complex cases. Look at the
> PCI bindings, for instance, the complexity of what needs to be valided
> is very high.

This sounds like an interesting idea. Not sure about usability of such schemas 
as human-readable documentation, though.

Best regards,
Tomasz

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3729 bytes --]

On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> 
> > > > b) What information should be specified in schemas? What level of 
> > > >    granularity is required?
> > > 
> > > One item I don't see in this list is node ordering.  There's been some
> > > discussion lately on deferred probing (re boot times).  If we were to
> > > intentionally declare that DT are parsed in the order written, then a
> > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > near the top of the tree.
> > > 
> > > This doesn't impact buses as much, since the nodes needing the bus are
> > > already children.  However, anything accessed via phandles: pins,
> > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > node is declared.
> > > 
> > > Not critical though, just a thought.
> > 
> > I don't think that siblings have any defined order in DT.  If reading a
> > device tree, there's no guarantee you get nodes or properties out in the
> > same order as the original .dts file.
> 
> That's why I raised the point.  If people think encoding initialization
> order in the DT is a good idea, then we should change the dtc so it
> compiles/decompiles in the same order.

I've always considered the DT to be unordered, although the flattened
representation obviously has to have some order.  It is much safer to
explicitly represent any required orderings with properties, rather
than to rely on the flattened tree order.  I really don't think trying
to have dtc magically understand device initialization ordering in
this way is a good idea.

Fwiw, dtc generally preserves order between input and output, with the
exception of the -s option, which sorts the subnodes of each node by
name (useful for dtdiff).

> > Provided child/parent relationships are maintained and the set of nodes
> > and values is the same, I think completely rearranging a .dts file does
> > not change its meaning.
> > 
> > "depends-on" relationships mostly have to come from the semantics of
> > the bindings themselves: for example, if a device is connected to some
> > clocks and regulators, the kernel may need to probe those first.
> 
> true, the answer to this problem may be to create a depgraph of the
> nodes based on phandles and child status, then init.  However, if the
> goal is to accelerate boot times, then that should not be calculated
> during each boot, especially since it doesn't likely change from boot to
> boot.
> 
> Which means it would either go in the dtc (dts node ordering is
> irrelevant), or in the dts.  I'm inclined to say dtc should do it, but I
> like the aesthetics of things being in the proper order in something I
> can read.  After all, C requires functions to be declared before use,
> even though the compiler could figure it out.

It's not necessarily possible to encode device initialization order in
flattened tree order.  Suppose you have bus A with devices A1 and A2,
and bus B with devices B1 and B2.  A1 must be initialized before B1,
but B2 must be initialized before A2.  There are no loops there, it's
a valid set of initialization order constraints, but you can't get
both of them right in the flat tree ordering.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] Defining schemas for Device Tree

[David Gibson]

On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> 
> > > > b) What information should be specified in schemas? What level of 
> > > >    granularity is required?
> > > 
> > > One item I don't see in this list is node ordering.  There's been some
> > > discussion lately on deferred probing (re boot times).  If we were to
> > > intentionally declare that DT are parsed in the order written, then a
> > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > near the top of the tree.
> > > 
> > > This doesn't impact buses as much, since the nodes needing the bus are
> > > already children.  However, anything accessed via phandles: pins,
> > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > node is declared.
> > > 
> > > Not critical though, just a thought.
> > 
> > I don't think that siblings have any defined order in DT.  If reading a
> > device tree, there's no guarantee you get nodes or properties out in the
> > same order as the original .dts file.
> 
> That's why I raised the point.  If people think encoding initialization
> order in the DT is a good idea, then we should change the dtc so it
> compiles/decompiles in the same order.

I've always considered the DT to be unordered, although the flattened
representation obviously has to have some order.  It is much safer to
explicitly represent any required orderings with properties, rather
than to rely on the flattened tree order.  I really don't think trying
to have dtc magically understand device initialization ordering in
this way is a good idea.

Fwiw, dtc generally preserves order between input and output, with the
exception of the -s option, which sorts the subnodes of each node by
name (useful for dtdiff).

> > Provided child/parent relationships are maintained and the set of nodes
> > and values is the same, I think completely rearranging a .dts file does
> > not change its meaning.
> > 
> > "depends-on" relationships mostly have to come from the semantics of
> > the bindings themselves: for example, if a device is connected to some
> > clocks and regulators, the kernel may need to probe those first.
> 
> true, the answer to this problem may be to create a depgraph of the
> nodes based on phandles and child status, then init.  However, if the
> goal is to accelerate boot times, then that should not be calculated
> during each boot, especially since it doesn't likely change from boot to
> boot.
> 
> Which means it would either go in the dtc (dts node ordering is
> irrelevant), or in the dts.  I'm inclined to say dtc should do it, but I
> like the aesthetics of things being in the proper order in something I
> can read.  After all, C requires functions to be declared before use,
> even though the compiler could figure it out.

It's not necessarily possible to encode device initialization order in
flattened tree order.  Suppose you have bus A with devices A1 and A2,
and bus B with devices B1 and B2.  A1 must be initialized before B1,
but B2 must be initialized before A2.  There are no loops there, it's
a valid set of initialization order constraints, but you can't get
both of them right in the flat tree ordering.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130730/c63ed571/attachment.sig>

Defining schemas for Device Tree

[Tomasz Figa]

On Monday 29 of July 2013 18:23:30 jonsmirl at gmail.com wrote:
> The schema should enforce some kind of regularity into the description
> of similar devices.  Take SPI for example. I pulled these SPI
> controller definitions from the kernel source. Obviously all of these
> controllers are different, but it seems to me that their definitions
> could be made more regular. DMA seems like it could use a lot of help.

Well, there is not much in common in bindings for all those SPI controllers, 
other than general device attributes, such as compatible and reg (not even 
interrupts - see spi-gpio).

There is a lot of generic binding templates (as I called this kind of bindings 
in my first mail) involved here, though.

Taking one of the controllers as the example:
> spi0: spi at f0000000 {
> #address-cells = <1>;
> #size-cells = <0>;
> compatible = "atmel,at91rm9200-spi";
> reg = <0xf0000000 0x100>;
> interrupts = <13 IRQ_TYPE_LEVEL_HIGH 3>;

/inherit/ "interrupts.dtss", 1;

> dmas = <&dma0 1 AT91_DMA_CFG_PER_ID(1)>,
>       <&dma0 1 AT91_DMA_CFG_PER_ID(2)>;
> dma-names = "tx", "rx";

/inherit/ "dma.dtss", "tx", "rx";

> pinctrl-names = "default";
> pinctrl-0 = <&pinctrl_spi0>;

/inherit/ "pinctrl.dtss","default";

> status = "disabled";
> };

Best regards,
Tomasz

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[Jason Cooper]

On Tue, Jul 30, 2013 at 08:29:20AM +1000, David Gibson wrote:
> On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> > On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> > 
> > > > > b) What information should be specified in schemas? What level of 
> > > > >    granularity is required?
> > > > 
> > > > One item I don't see in this list is node ordering.  There's been some
> > > > discussion lately on deferred probing (re boot times).  If we were to
> > > > intentionally declare that DT are parsed in the order written, then a
> > > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > > near the top of the tree.
> > > > 
> > > > This doesn't impact buses as much, since the nodes needing the bus are
> > > > already children.  However, anything accessed via phandles: pins,
> > > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > > node is declared.
> > > > 
> > > > Not critical though, just a thought.
> > > 
> > > I don't think that siblings have any defined order in DT.  If reading a
> > > device tree, there's no guarantee you get nodes or properties out in the
> > > same order as the original .dts file.
> > 
> > That's why I raised the point.  If people think encoding initialization
> > order in the DT is a good idea, then we should change the dtc so it
> > compiles/decompiles in the same order.
> 
> I've always considered the DT to be unordered, although the flattened
> representation obviously has to have some order.  It is much safer to
> explicitly represent any required orderings with properties, rather
> than to rely on the flattened tree order.  I really don't think trying
> to have dtc magically understand device initialization ordering in
> this way is a good idea.
> 
> Fwiw, dtc generally preserves order between input and output, with the
> exception of the -s option, which sorts the subnodes of each node by
> name (useful for dtdiff).
> 
> > > Provided child/parent relationships are maintained and the set of nodes
> > > and values is the same, I think completely rearranging a .dts file does
> > > not change its meaning.
> > > 
> > > "depends-on" relationships mostly have to come from the semantics of
> > > the bindings themselves: for example, if a device is connected to some
> > > clocks and regulators, the kernel may need to probe those first.
> > 
> > true, the answer to this problem may be to create a depgraph of the
> > nodes based on phandles and child status, then init.  However, if the
> > goal is to accelerate boot times, then that should not be calculated
> > during each boot, especially since it doesn't likely change from boot to
> > boot.
> > 
> > Which means it would either go in the dtc (dts node ordering is
> > irrelevant), or in the dts.  I'm inclined to say dtc should do it, but I
> > like the aesthetics of things being in the proper order in something I
> > can read.  After all, C requires functions to be declared before use,
> > even though the compiler could figure it out.
> 
> It's not necessarily possible to encode device initialization order in
> flattened tree order.  Suppose you have bus A with devices A1 and A2,
> and bus B with devices B1 and B2.  A1 must be initialized before B1,
> but B2 must be initialized before A2.  There are no loops there, it's
> a valid set of initialization order constraints, but you can't get
> both of them right in the flat tree ordering.

True, but is there a real scenario where this is the case?  In any
event, this could still fall back to deferred probing.

As I think about it more, working with only what dtc can definitely see,
eg busses and phandles, some ordering optimization could be done to
reduce the number of probe deferrals.

thx,

Jason.

[Ksummit-2013-discuss] Defining schemas for Device Tree

[Jason Cooper]

On Tue, Jul 30, 2013 at 08:29:20AM +1000, David Gibson wrote:
> On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> > On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> > 
> > > > > b) What information should be specified in schemas? What level of 
> > > > >    granularity is required?
> > > > 
> > > > One item I don't see in this list is node ordering.  There's been some
> > > > discussion lately on deferred probing (re boot times).  If we were to
> > > > intentionally declare that DT are parsed in the order written, then a
> > > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > > near the top of the tree.
> > > > 
> > > > This doesn't impact buses as much, since the nodes needing the bus are
> > > > already children.  However, anything accessed via phandles: pins,
> > > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > > node is declared.
> > > > 
> > > > Not critical though, just a thought.
> > > 
> > > I don't think that siblings have any defined order in DT.  If reading a
> > > device tree, there's no guarantee you get nodes or properties out in the
> > > same order as the original .dts file.
> > 
> > That's why I raised the point.  If people think encoding initialization
> > order in the DT is a good idea, then we should change the dtc so it
> > compiles/decompiles in the same order.
> 
> I've always considered the DT to be unordered, although the flattened
> representation obviously has to have some order.  It is much safer to
> explicitly represent any required orderings with properties, rather
> than to rely on the flattened tree order.  I really don't think trying
> to have dtc magically understand device initialization ordering in
> this way is a good idea.
> 
> Fwiw, dtc generally preserves order between input and output, with the
> exception of the -s option, which sorts the subnodes of each node by
> name (useful for dtdiff).
> 
> > > Provided child/parent relationships are maintained and the set of nodes
> > > and values is the same, I think completely rearranging a .dts file does
> > > not change its meaning.
> > > 
> > > "depends-on" relationships mostly have to come from the semantics of
> > > the bindings themselves: for example, if a device is connected to some
> > > clocks and regulators, the kernel may need to probe those first.
> > 
> > true, the answer to this problem may be to create a depgraph of the
> > nodes based on phandles and child status, then init.  However, if the
> > goal is to accelerate boot times, then that should not be calculated
> > during each boot, especially since it doesn't likely change from boot to
> > boot.
> > 
> > Which means it would either go in the dtc (dts node ordering is
> > irrelevant), or in the dts.  I'm inclined to say dtc should do it, but I
> > like the aesthetics of things being in the proper order in something I
> > can read.  After all, C requires functions to be declared before use,
> > even though the compiler could figure it out.
> 
> It's not necessarily possible to encode device initialization order in
> flattened tree order.  Suppose you have bus A with devices A1 and A2,
> and bus B with devices B1 and B2.  A1 must be initialized before B1,
> but B2 must be initialized before A2.  There are no loops there, it's
> a valid set of initialization order constraints, but you can't get
> both of them right in the flat tree ordering.

True, but is there a real scenario where this is the case?  In any
event, this could still fall back to deferred probing.

As I think about it more, working with only what dtc can definitely see,
eg busses and phandles, some ordering optimization could be done to
reduce the number of probe deferrals.

thx,

Jason.

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5462 bytes --]

On Mon, Jul 29, 2013 at 06:48:40PM -0400, Jason Cooper wrote:
> On Tue, Jul 30, 2013 at 08:29:20AM +1000, David Gibson wrote:
> > On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> > > On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > > > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> > > 
> > > > > > b) What information should be specified in schemas? What level of 
> > > > > >    granularity is required?
> > > > > 
> > > > > One item I don't see in this list is node ordering.  There's been some
> > > > > discussion lately on deferred probing (re boot times).  If we were to
> > > > > intentionally declare that DT are parsed in the order written, then a
> > > > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > > > near the top of the tree.
> > > > > 
> > > > > This doesn't impact buses as much, since the nodes needing the bus are
> > > > > already children.  However, anything accessed via phandles: pins,
> > > > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > > > node is declared.
> > > > > 
> > > > > Not critical though, just a thought.
> > > > 
> > > > I don't think that siblings have any defined order in DT.  If reading a
> > > > device tree, there's no guarantee you get nodes or properties out in the
> > > > same order as the original .dts file.
> > > 
> > > That's why I raised the point.  If people think encoding initialization
> > > order in the DT is a good idea, then we should change the dtc so it
> > > compiles/decompiles in the same order.
> > 
> > I've always considered the DT to be unordered, although the flattened
> > representation obviously has to have some order.  It is much safer to
> > explicitly represent any required orderings with properties, rather
> > than to rely on the flattened tree order.  I really don't think trying
> > to have dtc magically understand device initialization ordering in
> > this way is a good idea.
> > 
> > Fwiw, dtc generally preserves order between input and output, with the
> > exception of the -s option, which sorts the subnodes of each node by
> > name (useful for dtdiff).
> > 
> > > > Provided child/parent relationships are maintained and the set of nodes
> > > > and values is the same, I think completely rearranging a .dts file does
> > > > not change its meaning.
> > > > 
> > > > "depends-on" relationships mostly have to come from the semantics of
> > > > the bindings themselves: for example, if a device is connected to some
> > > > clocks and regulators, the kernel may need to probe those first.
> > > 
> > > true, the answer to this problem may be to create a depgraph of the
> > > nodes based on phandles and child status, then init.  However, if the
> > > goal is to accelerate boot times, then that should not be calculated
> > > during each boot, especially since it doesn't likely change from boot to
> > > boot.
> > > 
> > > Which means it would either go in the dtc (dts node ordering is
> > > irrelevant), or in the dts.  I'm inclined to say dtc should do it, but I
> > > like the aesthetics of things being in the proper order in something I
> > > can read.  After all, C requires functions to be declared before use,
> > > even though the compiler could figure it out.
> > 
> > It's not necessarily possible to encode device initialization order in
> > flattened tree order.  Suppose you have bus A with devices A1 and A2,
> > and bus B with devices B1 and B2.  A1 must be initialized before B1,
> > but B2 must be initialized before A2.  There are no loops there, it's
> > a valid set of initialization order constraints, but you can't get
> > both of them right in the flat tree ordering.
> 
> True, but is there a real scenario where this is the case?  In any
> event, this could still fall back to deferred probing.

I never count on weird and wonderful arrangements _not_ appearing in
embedded.

But, in regards to falling back do deferred probing.  If you're
thinking of the fdt ordering as purely an optimization, rather than
_required_ to get device init correct, then that's a very different
matter.  I have no problem with optimizing the ordering, as long as
its expected that the kernel will still be correct with arbitrary
ordering.

> As I think about it more, working with only what dtc can definitely see,
> eg busses and phandles, some ordering optimization could be done to
> reduce the number of probe deferrals.

Well.  It depends what you mean here.  To do this fully would require
dtc to interpret properties much more than it currently does - in
nearly all cases it treats them as opaque blobs, whereas many
different kinds of properties can potentially include phandles, and
you have to know how to parse them to discover them.

On the other hand, if what you're referring to is dtc's &-syntax for
phandle references, then I guess we could topsort on that.  As long as
we bear in mind that that can miss cases, if people hand craft their
phandles, instead of using references.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] Defining schemas for Device Tree

[David Gibson]

On Mon, Jul 29, 2013 at 06:48:40PM -0400, Jason Cooper wrote:
> On Tue, Jul 30, 2013 at 08:29:20AM +1000, David Gibson wrote:
> > On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> > > On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > > > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> > > 
> > > > > > b) What information should be specified in schemas? What level of 
> > > > > >    granularity is required?
> > > > > 
> > > > > One item I don't see in this list is node ordering.  There's been some
> > > > > discussion lately on deferred probing (re boot times).  If we were to
> > > > > intentionally declare that DT are parsed in the order written, then a
> > > > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > > > near the top of the tree.
> > > > > 
> > > > > This doesn't impact buses as much, since the nodes needing the bus are
> > > > > already children.  However, anything accessed via phandles: pins,
> > > > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > > > node is declared.
> > > > > 
> > > > > Not critical though, just a thought.
> > > > 
> > > > I don't think that siblings have any defined order in DT.  If reading a
> > > > device tree, there's no guarantee you get nodes or properties out in the
> > > > same order as the original .dts file.
> > > 
> > > That's why I raised the point.  If people think encoding initialization
> > > order in the DT is a good idea, then we should change the dtc so it
> > > compiles/decompiles in the same order.
> > 
> > I've always considered the DT to be unordered, although the flattened
> > representation obviously has to have some order.  It is much safer to
> > explicitly represent any required orderings with properties, rather
> > than to rely on the flattened tree order.  I really don't think trying
> > to have dtc magically understand device initialization ordering in
> > this way is a good idea.
> > 
> > Fwiw, dtc generally preserves order between input and output, with the
> > exception of the -s option, which sorts the subnodes of each node by
> > name (useful for dtdiff).
> > 
> > > > Provided child/parent relationships are maintained and the set of nodes
> > > > and values is the same, I think completely rearranging a .dts file does
> > > > not change its meaning.
> > > > 
> > > > "depends-on" relationships mostly have to come from the semantics of
> > > > the bindings themselves: for example, if a device is connected to some
> > > > clocks and regulators, the kernel may need to probe those first.
> > > 
> > > true, the answer to this problem may be to create a depgraph of the
> > > nodes based on phandles and child status, then init.  However, if the
> > > goal is to accelerate boot times, then that should not be calculated
> > > during each boot, especially since it doesn't likely change from boot to
> > > boot.
> > > 
> > > Which means it would either go in the dtc (dts node ordering is
> > > irrelevant), or in the dts.  I'm inclined to say dtc should do it, but I
> > > like the aesthetics of things being in the proper order in something I
> > > can read.  After all, C requires functions to be declared before use,
> > > even though the compiler could figure it out.
> > 
> > It's not necessarily possible to encode device initialization order in
> > flattened tree order.  Suppose you have bus A with devices A1 and A2,
> > and bus B with devices B1 and B2.  A1 must be initialized before B1,
> > but B2 must be initialized before A2.  There are no loops there, it's
> > a valid set of initialization order constraints, but you can't get
> > both of them right in the flat tree ordering.
> 
> True, but is there a real scenario where this is the case?  In any
> event, this could still fall back to deferred probing.

I never count on weird and wonderful arrangements _not_ appearing in
embedded.

But, in regards to falling back do deferred probing.  If you're
thinking of the fdt ordering as purely an optimization, rather than
_required_ to get device init correct, then that's a very different
matter.  I have no problem with optimizing the ordering, as long as
its expected that the kernel will still be correct with arbitrary
ordering.

> As I think about it more, working with only what dtc can definitely see,
> eg busses and phandles, some ordering optimization could be done to
> reduce the number of probe deferrals.

Well.  It depends what you mean here.  To do this fully would require
dtc to interpret properties much more than it currently does - in
nearly all cases it treats them as opaque blobs, whereas many
different kinds of properties can potentially include phandles, and
you have to know how to parse them to discover them.

On the other hand, if what you're referring to is dtc's &-syntax for
phandle references, then I guess we could topsort on that.  As long as
we bear in mind that that can miss cases, if people hand craft their
phandles, instead of using references.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130730/04bf655e/attachment.sig>

Defining schemas for Device Tree

[Stephen Warren]

On 07/29/2013 04:20 PM, Tomasz Figa wrote:
> On Monday 29 of July 2013 15:47:55 Stephen Warren wrote:
>> On 07/28/2013 06:21 PM, Tomasz Figa wrote:

...
>> Other content restrictions might be:
>>
>> * List must contain (at least) these entries/values.
>>
>> * List can't contain any other entries/values not specified here.
>>
>> * List must be in this order vs. any order.
>>
>> * List logical length (in entries, not bytes/cells) must match the
>> logical length of another cell (consider clocks/clock-names).
>>
>> * Ordering of one property must match ordering of another property
>> (again consider clocks/clock-names), although it'd be difficult to tell
>> whether this condition was met, perhaps we can at least document it.
>>
>> I'm sure there will be many more criteria to validate that we're
>> forgetting. This is possible the most complex area?
> 
> Right. Still, this is about the granularity of checks we really need. Do we 
> need to check for all those restrictions?

I think all/most of those bullets actually exist in various binding
documents, and I think we want any schema to be able to represent any
restrictions we already impose. So, yes, I do think we want to be able
to validate all of that.

Now, that's not saying that a system which didn't validate that, but
perhaps only validated that certain properties were present, and no
other properties were present. However, I'd still hope that such a
system was just a first step towards the complete system.

>>> As for subnodes, I think we need to define following constraints:
>>>  - node name (or node name format, e.g. regex),
>>
>> I don't think node names are supposed to convey any semantic meaning, so
>> they probably shouldn't be restricted (much?) by schema.
> 
> There are bindings currently that require specific node names. Since 
> of_match_node() can do matching by names as well, I think we should care about 
> node names too,

Hopefully we can discourage this through review though, since I've
definitely been told not to semantically interpret node names.

...
>> Equally, I'm not sure there's any difference between:
>>
>> prop = <&node>;
>> prop = <0x23648689>;
>>
>> ... in the DTB, except that the value just *happens* to match another
>> node's phandle value, which could end up leading to false positive
>> matches if only applied on the raw values not the syntax?
> 
> This is a good point. From driver's perspective it doesn't matter how you 
> specify value of some property, as long as it can be parsed using the same 
> code. As a more trivial example, consider following dts code:
> 
> 	array-prop = <0>, <1>, <2>;
> 	array-prop-2 = <0 1 2>;
> 
> Both array props contain the same data and can be parsed using the same code, 
> but from reader's perspective they mean something slightly different - an array 
> with 1 cell per element and a single entry of 3 cells.

I can certainly understand those two pieces of syntax being interpreted
that way. I've always thought of the above being 100% equivalent, even
though there may be a logical distinction. I not sure that such a
distinction is actually present in all extant DT source files. Perhaps
it's fine as part of the schema effort to re-write the DTs along these
lines though.

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Mon, Jul 29, 2013 at 6:45 PM, Tomasz Figa <tomasz.figa@gmail.com> wrote:
> On Monday 29 of July 2013 18:23:30 jonsmirl at gmail.com wrote:
>> The schema should enforce some kind of regularity into the description
>> of similar devices.  Take SPI for example. I pulled these SPI
>> controller definitions from the kernel source. Obviously all of these
>> controllers are different, but it seems to me that their definitions
>> could be made more regular. DMA seems like it could use a lot of help.
>
> Well, there is not much in common in bindings for all those SPI controllers,
> other than general device attributes, such as compatible and reg (not even
> interrupts - see spi-gpio).

I went and read a little about some of those SPI controllers. I
suspect everyone of these controllers could be mapped into a common
SPI controller schema.

Things like:
num-cs
ti,spi-num-cs

should have the same name.
The description of hardware SPI CS vs GPIOs being used for CS could be
unified too.

This...
tx-dma-channel = <&pdma0 7>; /* preliminary */
rx-dma-channel = <&pdma0 6>; /* preliminary */

Probably should be
dmas = <&pdma0 7>,<&pdma0 6>;
dma-names = "tx", "rx";

With a generic schema as a guide all of these controllers can be
described using the same syntax.

>
> There is a lot of generic binding templates (as I called this kind of bindings
> in my first mail) involved here, though.
>
> Taking one of the controllers as the example:
>> spi0: spi at f0000000 {
>> #address-cells = <1>;
>> #size-cells = <0>;
>> compatible = "atmel,at91rm9200-spi";
>> reg = <0xf0000000 0x100>;
>> interrupts = <13 IRQ_TYPE_LEVEL_HIGH 3>;
>
> /inherit/ "interrupts.dtss", 1;
>
>> dmas = <&dma0 1 AT91_DMA_CFG_PER_ID(1)>,
>>       <&dma0 1 AT91_DMA_CFG_PER_ID(2)>;
>> dma-names = "tx", "rx";
>
> /inherit/ "dma.dtss", "tx", "rx";
>
>> pinctrl-names = "default";
>> pinctrl-0 = <&pinctrl_spi0>;
>
> /inherit/ "pinctrl.dtss","default";
>
>> status = "disabled";
>> };
>
> Best regards,
> Tomasz
>



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[David Lang]

On Mon, 29 Jul 2013, Jason Cooper wrote:

> 
>>
>> I don't think that siblings have any defined order in DT.  If reading a
>> device tree, there's no guarantee you get nodes or properties out in the
>> same order as the original .dts file.
>
> That's why I raised the point.  If people think encoding initialization
> order in the DT is a good idea, then we should change the dtc so it
> compiles/decompiles in the same order.

if you make the initializaiton order 'magicly' correct by following the order of 
the flat representation, how do you reflect the case where initialization can be 
overlapped for different devices?

you are just trading one side of the problem for the other.

David Lang

[Ksummit-2013-discuss] Defining schemas for Device Tree

[David Lang]

On Mon, 29 Jul 2013, Jason Cooper wrote:

> 
>>
>> I don't think that siblings have any defined order in DT.  If reading a
>> device tree, there's no guarantee you get nodes or properties out in the
>> same order as the original .dts file.
>
> That's why I raised the point.  If people think encoding initialization
> order in the DT is a good idea, then we should change the dtc so it
> compiles/decompiles in the same order.

if you make the initializaiton order 'magicly' correct by following the order of 
the flat representation, how do you reflect the case where initialization can be 
overlapped for different devices?

you are just trading one side of the problem for the other.

David Lang

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[jonsmirl]

On Mon, Jul 29, 2013 at 8:41 PM, David Lang <david@lang.hm> wrote:
> On Mon, 29 Jul 2013, Jason Cooper wrote:
>
>>
>>>
>>> I don't think that siblings have any defined order in DT.  If reading a
>>> device tree, there's no guarantee you get nodes or properties out in the
>>> same order as the original .dts file.
>>
>>
>> That's why I raised the point.  If people think encoding initialization
>> order in the DT is a good idea, then we should change the dtc so it
>> compiles/decompiles in the same order.
>
>
> if you make the initializaiton order 'magicly' correct by following the
> order of the flat representation, how do you reflect the case where
> initialization can be overlapped for different devices?

I agree with David, using DT to try and eliminate deferred probes
isn't a good solution.  Overlapped probes and doing probes on multiple
CPUs introduces a temporal angle to the problem. Best to just let the
deferred probing code dynamically solve the problem. From what I can
see the deferred probing solution is working out nicely.

Plus there isn't that much code being run in deferred probing. I
suspect potential savings (if there even is any) are under a
millisecond.

>
> you are just trading one side of the problem for the other.
>
> David Lang



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Mon, Jul 29, 2013 at 8:41 PM, David Lang <david@lang.hm> wrote:
> On Mon, 29 Jul 2013, Jason Cooper wrote:
>
>>
>>>
>>> I don't think that siblings have any defined order in DT.  If reading a
>>> device tree, there's no guarantee you get nodes or properties out in the
>>> same order as the original .dts file.
>>
>>
>> That's why I raised the point.  If people think encoding initialization
>> order in the DT is a good idea, then we should change the dtc so it
>> compiles/decompiles in the same order.
>
>
> if you make the initializaiton order 'magicly' correct by following the
> order of the flat representation, how do you reflect the case where
> initialization can be overlapped for different devices?

I agree with David, using DT to try and eliminate deferred probes
isn't a good solution.  Overlapped probes and doing probes on multiple
CPUs introduces a temporal angle to the problem. Best to just let the
deferred probing code dynamically solve the problem. From what I can
see the deferred probing solution is working out nicely.

Plus there isn't that much code being run in deferred probing. I
suspect potential savings (if there even is any) are under a
millisecond.

>
> you are just trading one side of the problem for the other.
>
> David Lang



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1823 bytes --]

On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> 
> > > > b) What information should be specified in schemas? What level of 
> > > >    granularity is required?
> > > 
> > > One item I don't see in this list is node ordering.  There's been some
> > > discussion lately on deferred probing (re boot times).  If we were to
> > > intentionally declare that DT are parsed in the order written, then a
> > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > near the top of the tree.
> > > 
> > > This doesn't impact buses as much, since the nodes needing the bus are
> > > already children.  However, anything accessed via phandles: pins,
> > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > node is declared.
> > > 
> > > Not critical though, just a thought.
> > 
> > I don't think that siblings have any defined order in DT.  If reading a
> > device tree, there's no guarantee you get nodes or properties out in the
> > same order as the original .dts file.
> 
> That's why I raised the point.  If people think encoding initialization
> order in the DT is a good idea, then we should change the dtc so it
> compiles/decompiles in the same order.

I'm not actually sure what you mean by this.  dtc already preserves
order between input and output.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] Defining schemas for Device Tree

[David Gibson]

On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> 
> > > > b) What information should be specified in schemas? What level of 
> > > >    granularity is required?
> > > 
> > > One item I don't see in this list is node ordering.  There's been some
> > > discussion lately on deferred probing (re boot times).  If we were to
> > > intentionally declare that DT are parsed in the order written, then a
> > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > near the top of the tree.
> > > 
> > > This doesn't impact buses as much, since the nodes needing the bus are
> > > already children.  However, anything accessed via phandles: pins,
> > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > node is declared.
> > > 
> > > Not critical though, just a thought.
> > 
> > I don't think that siblings have any defined order in DT.  If reading a
> > device tree, there's no guarantee you get nodes or properties out in the
> > same order as the original .dts file.
> 
> That's why I raised the point.  If people think encoding initialization
> order in the DT is a good idea, then we should change the dtc so it
> compiles/decompiles in the same order.

I'm not actually sure what you mean by this.  dtc already preserves
order between input and output.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130730/871c21da/attachment.sig>

Defining schemas for Device Tree

[Mark Brown]

On Mon, Jul 29, 2013 at 08:30:29PM -0400, jonsmirl at gmail.com wrote:

> This...
> tx-dma-channel = <&pdma0 7>; /* preliminary */
> rx-dma-channel = <&pdma0 6>; /* preliminary */

> Probably should be
> dmas = <&pdma0 7>,<&pdma0 6>;
> dma-names = "tx", "rx";

It should be - the latter is the generic DMA binding.  There's a lot of
bindings in the kernel that predate that but people are currently
working to transfer over, this is one of the examples of instability
that everyone is talking about.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130730/ac91dfe8/attachment.sig>

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[Jason Cooper]

On Tue, Jul 30, 2013 at 09:45:32AM +1000, David Gibson wrote:
> On Mon, Jul 29, 2013 at 06:48:40PM -0400, Jason Cooper wrote:
> > On Tue, Jul 30, 2013 at 08:29:20AM +1000, David Gibson wrote:
> > > On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> > > > On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > > > > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > > > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> > > > 
> > > > > > > b) What information should be specified in schemas? What level of 
> > > > > > >    granularity is required?
> > > > > > 
> > > > > > One item I don't see in this list is node ordering.  There's been some
> > > > > > discussion lately on deferred probing (re boot times).  If we were to
> > > > > > intentionally declare that DT are parsed in the order written, then a
> > > > > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > > > > near the top of the tree.
> > > > > > 
> > > > > > This doesn't impact buses as much, since the nodes needing the bus are
> > > > > > already children.  However, anything accessed via phandles: pins,
> > > > > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > > > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > > > > node is declared.
> > > > > > 
> > > > > > Not critical though, just a thought.
> > > > > 
> > > > > I don't think that siblings have any defined order in DT.  If reading a
> > > > > device tree, there's no guarantee you get nodes or properties out in the
> > > > > same order as the original .dts file.
> > > > 
> > > > That's why I raised the point.  If people think encoding initialization
> > > > order in the DT is a good idea, then we should change the dtc so it
> > > > compiles/decompiles in the same order.
> > > 
> > > I've always considered the DT to be unordered, although the flattened
> > > representation obviously has to have some order.  It is much safer to
> > > explicitly represent any required orderings with properties, rather
> > > than to rely on the flattened tree order.  I really don't think trying
> > > to have dtc magically understand device initialization ordering in
> > > this way is a good idea.
> > > 
> > > Fwiw, dtc generally preserves order between input and output, with the
> > > exception of the -s option, which sorts the subnodes of each node by
> > > name (useful for dtdiff).
> > > 
> > > > > Provided child/parent relationships are maintained and the set of nodes
> > > > > and values is the same, I think completely rearranging a .dts file does
> > > > > not change its meaning.
> > > > > 
> > > > > "depends-on" relationships mostly have to come from the semantics of
> > > > > the bindings themselves: for example, if a device is connected to some
> > > > > clocks and regulators, the kernel may need to probe those first.
> > > > 
> > > > true, the answer to this problem may be to create a depgraph of the
> > > > nodes based on phandles and child status, then init.  However, if the
> > > > goal is to accelerate boot times, then that should not be calculated
> > > > during each boot, especially since it doesn't likely change from boot to
> > > > boot.
> > > > 
> > > > Which means it would either go in the dtc (dts node ordering is
> > > > irrelevant), or in the dts.  I'm inclined to say dtc should do it, but I
> > > > like the aesthetics of things being in the proper order in something I
> > > > can read.  After all, C requires functions to be declared before use,
> > > > even though the compiler could figure it out.
> > > 
> > > It's not necessarily possible to encode device initialization order in
> > > flattened tree order.  Suppose you have bus A with devices A1 and A2,
> > > and bus B with devices B1 and B2.  A1 must be initialized before B1,
> > > but B2 must be initialized before A2.  There are no loops there, it's
> > > a valid set of initialization order constraints, but you can't get
> > > both of them right in the flat tree ordering.
> > 
> > True, but is there a real scenario where this is the case?  In any
> > event, this could still fall back to deferred probing.
> 
> I never count on weird and wonderful arrangements _not_ appearing in
> embedded.

:-)

> But, in regards to falling back do deferred probing.  If you're
> thinking of the fdt ordering as purely an optimization, rather than
> _required_ to get device init correct, then that's a very different
> matter.  I have no problem with optimizing the ordering, as long as
> its expected that the kernel will still be correct with arbitrary
> ordering.

Yes, after sleeping on it a night, I agree.

> > As I think about it more, working with only what dtc can definitely see,
> > eg busses and phandles, some ordering optimization could be done to
> > reduce the number of probe deferrals.
> 
> Well.  It depends what you mean here.  To do this fully would require
> dtc to interpret properties much more than it currently does - in
> nearly all cases it treats them as opaque blobs, whereas many
> different kinds of properties can potentially include phandles, and
> you have to know how to parse them to discover them.
> 
> On the other hand, if what you're referring to is dtc's &-syntax for
> phandle references, then I guess we could topsort on that.  As long as
> we bear in mind that that can miss cases, if people hand craft their
> phandles, instead of using references.

Just this would eliminate a majority of probe deferrals.

For example, on the Seagate Wireless Plus I was recently trying to put a
mainline kernel on, all of the probe deferrals were pinctrl related.
Just topsorting that node would've fixed the problem.

thx,

Jason.

[Ksummit-2013-discuss] Defining schemas for Device Tree

[Jason Cooper]

On Tue, Jul 30, 2013 at 09:45:32AM +1000, David Gibson wrote:
> On Mon, Jul 29, 2013 at 06:48:40PM -0400, Jason Cooper wrote:
> > On Tue, Jul 30, 2013 at 08:29:20AM +1000, David Gibson wrote:
> > > On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> > > > On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > > > > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > > > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> > > > 
> > > > > > > b) What information should be specified in schemas? What level of 
> > > > > > >    granularity is required?
> > > > > > 
> > > > > > One item I don't see in this list is node ordering.  There's been some
> > > > > > discussion lately on deferred probing (re boot times).  If we were to
> > > > > > intentionally declare that DT are parsed in the order written, then a
> > > > > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > > > > near the top of the tree.
> > > > > > 
> > > > > > This doesn't impact buses as much, since the nodes needing the bus are
> > > > > > already children.  However, anything accessed via phandles: pins,
> > > > > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > > > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > > > > node is declared.
> > > > > > 
> > > > > > Not critical though, just a thought.
> > > > > 
> > > > > I don't think that siblings have any defined order in DT.  If reading a
> > > > > device tree, there's no guarantee you get nodes or properties out in the
> > > > > same order as the original .dts file.
> > > > 
> > > > That's why I raised the point.  If people think encoding initialization
> > > > order in the DT is a good idea, then we should change the dtc so it
> > > > compiles/decompiles in the same order.
> > > 
> > > I've always considered the DT to be unordered, although the flattened
> > > representation obviously has to have some order.  It is much safer to
> > > explicitly represent any required orderings with properties, rather
> > > than to rely on the flattened tree order.  I really don't think trying
> > > to have dtc magically understand device initialization ordering in
> > > this way is a good idea.
> > > 
> > > Fwiw, dtc generally preserves order between input and output, with the
> > > exception of the -s option, which sorts the subnodes of each node by
> > > name (useful for dtdiff).
> > > 
> > > > > Provided child/parent relationships are maintained and the set of nodes
> > > > > and values is the same, I think completely rearranging a .dts file does
> > > > > not change its meaning.
> > > > > 
> > > > > "depends-on" relationships mostly have to come from the semantics of
> > > > > the bindings themselves: for example, if a device is connected to some
> > > > > clocks and regulators, the kernel may need to probe those first.
> > > > 
> > > > true, the answer to this problem may be to create a depgraph of the
> > > > nodes based on phandles and child status, then init.  However, if the
> > > > goal is to accelerate boot times, then that should not be calculated
> > > > during each boot, especially since it doesn't likely change from boot to
> > > > boot.
> > > > 
> > > > Which means it would either go in the dtc (dts node ordering is
> > > > irrelevant), or in the dts.  I'm inclined to say dtc should do it, but I
> > > > like the aesthetics of things being in the proper order in something I
> > > > can read.  After all, C requires functions to be declared before use,
> > > > even though the compiler could figure it out.
> > > 
> > > It's not necessarily possible to encode device initialization order in
> > > flattened tree order.  Suppose you have bus A with devices A1 and A2,
> > > and bus B with devices B1 and B2.  A1 must be initialized before B1,
> > > but B2 must be initialized before A2.  There are no loops there, it's
> > > a valid set of initialization order constraints, but you can't get
> > > both of them right in the flat tree ordering.
> > 
> > True, but is there a real scenario where this is the case?  In any
> > event, this could still fall back to deferred probing.
> 
> I never count on weird and wonderful arrangements _not_ appearing in
> embedded.

:-)

> But, in regards to falling back do deferred probing.  If you're
> thinking of the fdt ordering as purely an optimization, rather than
> _required_ to get device init correct, then that's a very different
> matter.  I have no problem with optimizing the ordering, as long as
> its expected that the kernel will still be correct with arbitrary
> ordering.

Yes, after sleeping on it a night, I agree.

> > As I think about it more, working with only what dtc can definitely see,
> > eg busses and phandles, some ordering optimization could be done to
> > reduce the number of probe deferrals.
> 
> Well.  It depends what you mean here.  To do this fully would require
> dtc to interpret properties much more than it currently does - in
> nearly all cases it treats them as opaque blobs, whereas many
> different kinds of properties can potentially include phandles, and
> you have to know how to parse them to discover them.
> 
> On the other hand, if what you're referring to is dtc's &-syntax for
> phandle references, then I guess we could topsort on that.  As long as
> we bear in mind that that can miss cases, if people hand craft their
> phandles, instead of using references.

Just this would eliminate a majority of probe deferrals.

For example, on the Seagate Wireless Plus I was recently trying to put a
mainline kernel on, all of the probe deferrals were pinctrl related.
Just topsorting that node would've fixed the problem.

thx,

Jason.

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[Jason Cooper]

On Tue, Jul 30, 2013 at 11:50:31AM +1000, David Gibson wrote:
> On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> > On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> > 
> > > > > b) What information should be specified in schemas? What level of 
> > > > >    granularity is required?
> > > > 
> > > > One item I don't see in this list is node ordering.  There's been some
> > > > discussion lately on deferred probing (re boot times).  If we were to
> > > > intentionally declare that DT are parsed in the order written, then a
> > > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > > near the top of the tree.
> > > > 
> > > > This doesn't impact buses as much, since the nodes needing the bus are
> > > > already children.  However, anything accessed via phandles: pins,
> > > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > > node is declared.
> > > > 
> > > > Not critical though, just a thought.
> > > 
> > > I don't think that siblings have any defined order in DT.  If reading a
> > > device tree, there's no guarantee you get nodes or properties out in the
> > > same order as the original .dts file.
> > 
> > That's why I raised the point.  If people think encoding initialization
> > order in the DT is a good idea, then we should change the dtc so it
> > compiles/decompiles in the same order.
> 
> I'm not actually sure what you mean by this.  dtc already preserves
> order between input and output.

This is an old comment (~ 1d, wow).  My position has evolved to seeing
if we can allow dtc to topsort nodes it can easily tell are needed first
as an optimization.  *Not* a requirement.  Deferred probing would still
be a fall back.

thx,

Jason.

[Ksummit-2013-discuss] Defining schemas for Device Tree

[Jason Cooper]

On Tue, Jul 30, 2013 at 11:50:31AM +1000, David Gibson wrote:
> On Mon, Jul 29, 2013 at 01:23:39PM -0400, Jason Cooper wrote:
> > On Mon, Jul 29, 2013 at 05:49:05PM +0100, Dave Martin wrote:
> > > On Mon, Jul 29, 2013 at 11:01:24AM -0400, Jason Cooper wrote:
> > > > On Mon, Jul 29, 2013 at 02:21:52AM +0200, Tomasz Figa wrote:
> > 
> > > > > b) What information should be specified in schemas? What level of 
> > > > >    granularity is required?
> > > > 
> > > > One item I don't see in this list is node ordering.  There's been some
> > > > discussion lately on deferred probing (re boot times).  If we were to
> > > > intentionally declare that DT are parsed in the order written, then a
> > > > lot of deferred probes could be avoided by moving eg the pinctrl node to
> > > > near the top of the tree.
> > > > 
> > > > This doesn't impact buses as much, since the nodes needing the bus are
> > > > already children.  However, anything accessed via phandles: pins,
> > > > clocks, regulators, etc could benefit from declaring and enforcing this.
> > > > Eg having the dtc warn when a phandle is used before it's corresponding
> > > > node is declared.
> > > > 
> > > > Not critical though, just a thought.
> > > 
> > > I don't think that siblings have any defined order in DT.  If reading a
> > > device tree, there's no guarantee you get nodes or properties out in the
> > > same order as the original .dts file.
> > 
> > That's why I raised the point.  If people think encoding initialization
> > order in the DT is a good idea, then we should change the dtc so it
> > compiles/decompiles in the same order.
> 
> I'm not actually sure what you mean by this.  dtc already preserves
> order between input and output.

This is an old comment (~ 1d, wow).  My position has evolved to seeing
if we can allow dtc to topsort nodes it can easily tell are needed first
as an optimization.  *Not* a requirement.  Deferred probing would still
be a fall back.

thx,

Jason.

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Tue, Jul 30, 2013 at 6:25 AM, Mark Brown <broonie@kernel.org> wrote:
> On Mon, Jul 29, 2013 at 08:30:29PM -0400, jonsmirl at gmail.com wrote:
>
>> This...
>> tx-dma-channel = <&pdma0 7>; /* preliminary */
>> rx-dma-channel = <&pdma0 6>; /* preliminary */
>
>> Probably should be
>> dmas = <&pdma0 7>,<&pdma0 6>;
>> dma-names = "tx", "rx";
>
> It should be - the latter is the generic DMA binding.  There's a lot of
> bindings in the kernel that predate that but people are currently
> working to transfer over, this is one of the examples of instability
> that everyone is talking about.

Is something similar to this possible in device tree syntax?
dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;

That's an example of something the quirks system could do. We can make
a quirk for these older definitions and then map them to the new and
improved kernel schema.   And you only need to make quirks for devices
that have been deployed with dtb in boot loaders. There is no need to
accept older dtbs that were appended to the kernel.

The schema system provides the class description for the generic SPI
controller. This generic SPI controller is only generic in the sense
that all of the device syntax is as regular as possible. Each of the
actual SPI controller device drivers is completely different. But now
someone implementing a new SPI controller would have a single DT
example to follow - the schema for the generic SPI controller.

I don't think the quirks system will be that bad to work with. We will
quickly develop a list of device trees that have been deployed in
flash and need to be handled by the quirk system They are already
handled right now by code spread throughout the kernel. Quirks would
just collect everything into an init time segment where it can easily
be discarded.

Also consider that someone like Microsoft might ship SurfaceRT tablets
with a MS specific device tree format. That could be handled by a
quirk. But it would be even better if Microsoft used the generic
schema and followed its format. Right now that schema isn't written
down.

-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[Stephen Warren]

On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:
> On Tue, Jul 30, 2013 at 6:25 AM, Mark Brown <broonie@kernel.org> wrote:
>> On Mon, Jul 29, 2013 at 08:30:29PM -0400, jonsmirl at gmail.com wrote:
>>
>>> This...
>>> tx-dma-channel = <&pdma0 7>; /* preliminary */
>>> rx-dma-channel = <&pdma0 6>; /* preliminary */
>>
>>> Probably should be
>>> dmas = <&pdma0 7>,<&pdma0 6>;
>>> dma-names = "tx", "rx";
>>
>> It should be - the latter is the generic DMA binding.  There's a lot of
>> bindings in the kernel that predate that but people are currently
>> working to transfer over, this is one of the examples of instability
>> that everyone is talking about.
> 
> Is something similar to this possible in device tree syntax?
> dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;

I /think/ you can physically write that in *.dts, or something very
similar; with the strings outside the <>:

dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;

However, there's been strong push-back (i.e. doing that has not been
allowed at all) on attempting to mix variable-length strings and
fixed-length/alignment integer cells in the same property. This is
primarily because you then can't ensure that the integer cell data is
aligned in the DTB (dtc and/or the DTB format spec does/requires/allows
no alignment padding), and hence have to be much more careful about data
alignment when parsing the property.

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Tue, Jul 30, 2013 at 1:19 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>
> On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:
> > On Tue, Jul 30, 2013 at 6:25 AM, Mark Brown <broonie@kernel.org> wrote:
> >> On Mon, Jul 29, 2013 at 08:30:29PM -0400, jonsmirl at gmail.com wrote:
> >>
> >>> This...
> >>> tx-dma-channel = <&pdma0 7>; /* preliminary */
> >>> rx-dma-channel = <&pdma0 6>; /* preliminary */
> >>
> >>> Probably should be
> >>> dmas = <&pdma0 7>,<&pdma0 6>;
> >>> dma-names = "tx", "rx";
> >>
> >> It should be - the latter is the generic DMA binding.  There's a lot of
> >> bindings in the kernel that predate that but people are currently
> >> working to transfer over, this is one of the examples of instability
> >> that everyone is talking about.
> >
> > Is something similar to this possible in device tree syntax?
> > dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;
>
> I /think/ you can physically write that in *.dts, or something very
> similar; with the strings outside the <>:
>
> dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>
> However, there's been strong push-back (i.e. doing that has not been
> allowed at all) on attempting to mix variable-length strings and
> fixed-length/alignment integer cells in the same property. This is
> primarily because you then can't ensure that the integer cell data is
> aligned in the DTB (dtc and/or the DTB format spec does/requires/allows

Can we turn the strings into string phandles? Then they'd be fixed sized.

That might provide a way to internationalize the strings too.


>
> no alignment padding), and hence have to be much more careful about data
> alignment when parsing the property.




-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[Stephen Warren]

On 07/30/2013 11:29 AM, jonsmirl at gmail.com wrote:
> On Tue, Jul 30, 2013 at 1:19 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>
>> On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:
>>> On Tue, Jul 30, 2013 at 6:25 AM, Mark Brown <broonie@kernel.org> wrote:
>>>> On Mon, Jul 29, 2013 at 08:30:29PM -0400, jonsmirl at gmail.com wrote:
>>>>
>>>>> This...
>>>>> tx-dma-channel = <&pdma0 7>; /* preliminary */
>>>>> rx-dma-channel = <&pdma0 6>; /* preliminary */
>>>>
>>>>> Probably should be
>>>>> dmas = <&pdma0 7>,<&pdma0 6>;
>>>>> dma-names = "tx", "rx";
>>>>
>>>> It should be - the latter is the generic DMA binding.  There's a lot of
>>>> bindings in the kernel that predate that but people are currently
>>>> working to transfer over, this is one of the examples of instability
>>>> that everyone is talking about.
>>>
>>> Is something similar to this possible in device tree syntax?
>>> dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;
>>
>> I /think/ you can physically write that in *.dts, or something very
>> similar; with the strings outside the <>:
>>
>> dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>>
>> However, there's been strong push-back (i.e. doing that has not been
>> allowed at all) on attempting to mix variable-length strings and
>> fixed-length/alignment integer cells in the same property. This is
>> primarily because you then can't ensure that the integer cell data is
>> aligned in the DTB (dtc and/or the DTB format spec does/requires/allows
> 
> Can we turn the strings into string phandles? Then they'd be fixed sized.

I believe that phandles point at nodes not arbitrary data, so no.

Now, could the DTB format be enhanced to allow that? Perhaps, although
we'd need a different string syntax to enable it, since existing
bindings would require the current inline-string format. That's getting
outside the realm of schemas and making active changes to the DT format
though...

> That might provide a way to internationalize the strings too.

I don't think internationalization is needed; DT is an interface to
drivers/SW, rather than something end-user-visible.

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Tue, Jul 30, 2013 at 1:34 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 07/30/2013 11:29 AM, jonsmirl at gmail.com wrote:
>> On Tue, Jul 30, 2013 at 1:19 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>
>>> On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:
>>>> On Tue, Jul 30, 2013 at 6:25 AM, Mark Brown <broonie@kernel.org> wrote:
>>>>> On Mon, Jul 29, 2013 at 08:30:29PM -0400, jonsmirl at gmail.com wrote:
>>>>>
>>>>>> This...
>>>>>> tx-dma-channel = <&pdma0 7>; /* preliminary */
>>>>>> rx-dma-channel = <&pdma0 6>; /* preliminary */
>>>>>
>>>>>> Probably should be
>>>>>> dmas = <&pdma0 7>,<&pdma0 6>;
>>>>>> dma-names = "tx", "rx";
>>>>>
>>>>> It should be - the latter is the generic DMA binding.  There's a lot of
>>>>> bindings in the kernel that predate that but people are currently
>>>>> working to transfer over, this is one of the examples of instability
>>>>> that everyone is talking about.
>>>>
>>>> Is something similar to this possible in device tree syntax?
>>>> dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;
>>>
>>> I /think/ you can physically write that in *.dts, or something very
>>> similar; with the strings outside the <>:
>>>
>>> dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>>>
>>> However, there's been strong push-back (i.e. doing that has not been
>>> allowed at all) on attempting to mix variable-length strings and
>>> fixed-length/alignment integer cells in the same property. This is
>>> primarily because you then can't ensure that the integer cell data is
>>> aligned in the DTB (dtc and/or the DTB format spec does/requires/allows

The PowerPC FDT doc says that strings are indirected into a string
table. So they aren't variable length in the data structures.

I was wondering how dmas = "tx", "rx" was implemented. It is two
pointers into the indirect block. So this should work, there is no
variable length data to worry about in the structure.

>>> dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;


>>
>> Can we turn the strings into string phandles? Then they'd be fixed sized.
>
> I believe that phandles point at nodes not arbitrary data, so no.
>
> Now, could the DTB format be enhanced to allow that? Perhaps, although
> we'd need a different string syntax to enable it, since existing
> bindings would require the current inline-string format. That's getting
> outside the realm of schemas and making active changes to the DT format
> though...
>
>> That might provide a way to internationalize the strings too.
>
> I don't think internationalization is needed; DT is an interface to
> drivers/SW, rather than something end-user-visible.



-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[Stephen Warren]

On 07/30/2013 11:45 AM, jonsmirl at gmail.com wrote:
> On Tue, Jul 30, 2013 at 1:34 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> On 07/30/2013 11:29 AM, jonsmirl at gmail.com wrote:
>>> On Tue, Jul 30, 2013 at 1:19 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>>
>>>> On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:
>>>>> On Tue, Jul 30, 2013 at 6:25 AM, Mark Brown <broonie@kernel.org> wrote:
>>>>>> On Mon, Jul 29, 2013 at 08:30:29PM -0400, jonsmirl at gmail.com wrote:
>>>>>>
>>>>>>> This...
>>>>>>> tx-dma-channel = <&pdma0 7>; /* preliminary */
>>>>>>> rx-dma-channel = <&pdma0 6>; /* preliminary */
>>>>>>
>>>>>>> Probably should be
>>>>>>> dmas = <&pdma0 7>,<&pdma0 6>;
>>>>>>> dma-names = "tx", "rx";
>>>>>>
>>>>>> It should be - the latter is the generic DMA binding.  There's a lot of
>>>>>> bindings in the kernel that predate that but people are currently
>>>>>> working to transfer over, this is one of the examples of instability
>>>>>> that everyone is talking about.
>>>>>
>>>>> Is something similar to this possible in device tree syntax?
>>>>> dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;
>>>>
>>>> I /think/ you can physically write that in *.dts, or something very
>>>> similar; with the strings outside the <>:
>>>>
>>>> dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>>>>
>>>> However, there's been strong push-back (i.e. doing that has not been
>>>> allowed at all) on attempting to mix variable-length strings and
>>>> fixed-length/alignment integer cells in the same property. This is
>>>> primarily because you then can't ensure that the integer cell data is
>>>> aligned in the DTB (dtc and/or the DTB format spec does/requires/allows
> 
> The PowerPC FDT doc says that strings are indirected into a string
> table. So they aren't variable length in the data structures.

Do you have a link to the document; I'm not sure which one you mean.

> I was wondering how dmas = "tx", "rx" was implemented. It is two
> pointers into the indirect block. So this should work, there is no
> variable length data to worry about in the structure.

If you look at of_property_read_string_index() in drivers/of/base.c,
you'll see that it expects the strings to just be packed together inline.

What you say may be correct for node and property names, just not
property values.

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Tue, Jul 30, 2013 at 1:49 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 07/30/2013 11:45 AM, jonsmirl at gmail.com wrote:
>> On Tue, Jul 30, 2013 at 1:34 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>> On 07/30/2013 11:29 AM, jonsmirl at gmail.com wrote:
>>>> On Tue, Jul 30, 2013 at 1:19 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>>>
>>>>> On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:
>>>>>> On Tue, Jul 30, 2013 at 6:25 AM, Mark Brown <broonie@kernel.org> wrote:
>>>>>>> On Mon, Jul 29, 2013 at 08:30:29PM -0400, jonsmirl at gmail.com wrote:
>>>>>>>
>>>>>>>> This...
>>>>>>>> tx-dma-channel = <&pdma0 7>; /* preliminary */
>>>>>>>> rx-dma-channel = <&pdma0 6>; /* preliminary */
>>>>>>>
>>>>>>>> Probably should be
>>>>>>>> dmas = <&pdma0 7>,<&pdma0 6>;
>>>>>>>> dma-names = "tx", "rx";
>>>>>>>
>>>>>>> It should be - the latter is the generic DMA binding.  There's a lot of
>>>>>>> bindings in the kernel that predate that but people are currently
>>>>>>> working to transfer over, this is one of the examples of instability
>>>>>>> that everyone is talking about.
>>>>>>
>>>>>> Is something similar to this possible in device tree syntax?
>>>>>> dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;
>>>>>
>>>>> I /think/ you can physically write that in *.dts, or something very
>>>>> similar; with the strings outside the <>:
>>>>>
>>>>> dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>>>>>
>>>>> However, there's been strong push-back (i.e. doing that has not been
>>>>> allowed at all) on attempting to mix variable-length strings and
>>>>> fixed-length/alignment integer cells in the same property. This is
>>>>> primarily because you then can't ensure that the integer cell data is
>>>>> aligned in the DTB (dtc and/or the DTB format spec does/requires/allows
>>
>> The PowerPC FDT doc says that strings are indirected into a string
>> table. So they aren't variable length in the data structures.
>
> Do you have a link to the document; I'm not sure which one you mean.

So it looks like there is one indirection to get to the block, then
the strings are concatenated. The indirection must have been hidden
when the node structure was created.  I had thought that there was a
list of indirects each pointing to a string in the string block.

https://www.power.org/documentation/epapr-version-1-1/

8.5 Strings Block
The strings block contains strings representing all the property names
used in the tree. These null-
terminated strings are simply concatenated together in this section,
and referred to from the structure
block by an offset into the strings block.
The strings block has no alignment constraints and may appear at any
offset from the beginning of the
device tree blob.


>
>> I was wondering how dmas = "tx", "rx" was implemented. It is two
>> pointers into the indirect block. So this should work, there is no
>> variable length data to worry about in the structure.
>
> If you look at of_property_read_string_index() in drivers/of/base.c,
> you'll see that it expects the strings to just be packed together inline.
>
> What you say may be correct for node and property names, just not
> property values.



-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Tue, Jul 30, 2013 at 2:03 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
> On Tue, Jul 30, 2013 at 1:49 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> On 07/30/2013 11:45 AM, jonsmirl at gmail.com wrote:
>>> On Tue, Jul 30, 2013 at 1:34 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>> On 07/30/2013 11:29 AM, jonsmirl at gmail.com wrote:
>>>>> On Tue, Jul 30, 2013 at 1:19 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>>>>
>>>>>> On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:
>>>>>>> On Tue, Jul 30, 2013 at 6:25 AM, Mark Brown <broonie@kernel.org> wrote:
>>>>>>>> On Mon, Jul 29, 2013 at 08:30:29PM -0400, jonsmirl at gmail.com wrote:
>>>>>>>>
>>>>>>>>> This...
>>>>>>>>> tx-dma-channel = <&pdma0 7>; /* preliminary */
>>>>>>>>> rx-dma-channel = <&pdma0 6>; /* preliminary */
>>>>>>>>
>>>>>>>>> Probably should be
>>>>>>>>> dmas = <&pdma0 7>,<&pdma0 6>;
>>>>>>>>> dma-names = "tx", "rx";
>>>>>>>>
>>>>>>>> It should be - the latter is the generic DMA binding.  There's a lot of
>>>>>>>> bindings in the kernel that predate that but people are currently
>>>>>>>> working to transfer over, this is one of the examples of instability
>>>>>>>> that everyone is talking about.
>>>>>>>
>>>>>>> Is something similar to this possible in device tree syntax?
>>>>>>> dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;
>>>>>>
>>>>>> I /think/ you can physically write that in *.dts, or something very
>>>>>> similar; with the strings outside the <>:
>>>>>>
>>>>>> dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>>>>>>
>>>>>> However, there's been strong push-back (i.e. doing that has not been
>>>>>> allowed at all) on attempting to mix variable-length strings and
>>>>>> fixed-length/alignment integer cells in the same property. This is
>>>>>> primarily because you then can't ensure that the integer cell data is
>>>>>> aligned in the DTB (dtc and/or the DTB format spec does/requires/allows
>>>
>>> The PowerPC FDT doc says that strings are indirected into a string
>>> table. So they aren't variable length in the data structures.
>>
>> Do you have a link to the document; I'm not sure which one you mean.
>
> So it looks like there is one indirection to get to the block, then
> the strings are concatenated. The indirection must have been hidden
> when the node structure was created.  I had thought that there was a
> list of indirects each pointing to a string in the string block.
>
> https://www.power.org/documentation/epapr-version-1-1/

So what does an alternating use of strings do?

 dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;


>
> 8.5 Strings Block
> The strings block contains strings representing all the property names
> used in the tree. These null-
> terminated strings are simply concatenated together in this section,
> and referred to from the structure
> block by an offset into the strings block.
> The strings block has no alignment constraints and may appear at any
> offset from the beginning of the
> device tree blob.
>
>
>>
>>> I was wondering how dmas = "tx", "rx" was implemented. It is two
>>> pointers into the indirect block. So this should work, there is no
>>> variable length data to worry about in the structure.
>>
>> If you look at of_property_read_string_index() in drivers/of/base.c,
>> you'll see that it expects the strings to just be packed together inline.
>>
>> What you say may be correct for node and property names, just not
>> property values.
>
>
>
> --
> Jon Smirl
> jonsmirl at gmail.com



-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[Stephen Warren]

On 07/30/2013 12:04 PM, jonsmirl at gmail.com wrote:
...
> So what does an alternating use of strings do?
> 
>  dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;

cat > temp.dts << ENDOFHERE
/dts-v1/;

/ {
    dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;

    pdma0: dma { };
};
ENDOFHERE

dtc -O dtb -o temp.dtb -I dts temp.dts

hexdump -C temp.dtb

yields:

> 00000000  d0 0d fe ed 00 00 00 ab  00 00 00 38 00 00 00 98  |...........8....|
> 00000010  00 00 00 28 00 00 00 11  00 00 00 10 00 00 00 00  |...(............|
> 00000020  00 00 00 13 00 00 00 60  00 00 00 00 00 00 00 00  |.......`........|
> 00000030  00 00 00 00 00 00 00 00  00 00 00 01 00 00 00 00  |................|
> 00000040  00 00 00 03 00 00 00 16  00 00 00 00 74 78 00 00  |............tx..|
                                                 ss ss ss pp
> 00000050  00 00 01 00 00 00 07 72  78 00 00 00 00 01 00 00  |.......rx.......|
            pp pp pp ii ii ii ii ss  ss ss pp pp pp pp ii ii
> 00000060  00 06 00 00 00 00 00 01  64 6d 61 00 00 00 00 03  |........dma.....|
            ii ii

... where ss represents a string byte, pp a phandle byte, ii an integer
byte.

> 00000070  00 00 00 04 00 00 00 05  00 00 00 01 00 00 00 03  |................|
> 00000080  00 00 00 04 00 00 00 0b  00 00 00 01 00 00 00 02  |................|
> 00000090  00 00 00 02 00 00 00 09  64 6d 61 73 00 6c 69 6e  |........dmas.lin|
> 000000a0  75 78 2c 70 68 61 6e 64  6c 65 00                 |ux,phandle.|

Starting about half way through that last line is the string table.

(the string table indirection is for property names, not property
values, and it looks like not for node names either)

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Tue, Jul 30, 2013 at 2:04 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
> On Tue, Jul 30, 2013 at 2:03 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
>> On Tue, Jul 30, 2013 at 1:49 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>> On 07/30/2013 11:45 AM, jonsmirl at gmail.com wrote:
>>>> On Tue, Jul 30, 2013 at 1:34 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>>> On 07/30/2013 11:29 AM, jonsmirl at gmail.com wrote:
>>>>>> On Tue, Jul 30, 2013 at 1:19 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>>>>>
>>>>>>> On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:
>>>>>>>> On Tue, Jul 30, 2013 at 6:25 AM, Mark Brown <broonie@kernel.org> wrote:
>>>>>>>>> On Mon, Jul 29, 2013 at 08:30:29PM -0400, jonsmirl at gmail.com wrote:
>>>>>>>>>
>>>>>>>>>> This...
>>>>>>>>>> tx-dma-channel = <&pdma0 7>; /* preliminary */
>>>>>>>>>> rx-dma-channel = <&pdma0 6>; /* preliminary */
>>>>>>>>>
>>>>>>>>>> Probably should be
>>>>>>>>>> dmas = <&pdma0 7>,<&pdma0 6>;
>>>>>>>>>> dma-names = "tx", "rx";
>>>>>>>>>
>>>>>>>>> It should be - the latter is the generic DMA binding.  There's a lot of
>>>>>>>>> bindings in the kernel that predate that but people are currently
>>>>>>>>> working to transfer over, this is one of the examples of instability
>>>>>>>>> that everyone is talking about.
>>>>>>>>
>>>>>>>> Is something similar to this possible in device tree syntax?
>>>>>>>> dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;
>>>>>>>
>>>>>>> I /think/ you can physically write that in *.dts, or something very
>>>>>>> similar; with the strings outside the <>:
>>>>>>>
>>>>>>> dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>>>>>>>
>>>>>>> However, there's been strong push-back (i.e. doing that has not been
>>>>>>> allowed at all) on attempting to mix variable-length strings and
>>>>>>> fixed-length/alignment integer cells in the same property. This is
>>>>>>> primarily because you then can't ensure that the integer cell data is
>>>>>>> aligned in the DTB (dtc and/or the DTB format spec does/requires/allows
>>>>
>>>> The PowerPC FDT doc says that strings are indirected into a string
>>>> table. So they aren't variable length in the data structures.
>>>
>>> Do you have a link to the document; I'm not sure which one you mean.
>>
>> So it looks like there is one indirection to get to the block, then
>> the strings are concatenated. The indirection must have been hidden
>> when the node structure was created.  I had thought that there was a
>> list of indirects each pointing to a string in the string block.
>>
>> https://www.power.org/documentation/epapr-version-1-1/
>
> So what does an alternating use of strings do?
>
>  dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;

dtc compiles this string without complaint....

gpios = "str1", <&gpio0 10 1>, "str2", <&gpio0 10 1>;

It would a new helper function to interpret it but I this it should work.
I've always disliked paired attributes (dmas, dma-names) is easy to
mess them up by accident.

>
>
>>
>> 8.5 Strings Block
>> The strings block contains strings representing all the property names
>> used in the tree. These null-
>> terminated strings are simply concatenated together in this section,
>> and referred to from the structure
>> block by an offset into the strings block.
>> The strings block has no alignment constraints and may appear at any
>> offset from the beginning of the
>> device tree blob.
>>
>>
>>>
>>>> I was wondering how dmas = "tx", "rx" was implemented. It is two
>>>> pointers into the indirect block. So this should work, there is no
>>>> variable length data to worry about in the structure.
>>>
>>> If you look at of_property_read_string_index() in drivers/of/base.c,
>>> you'll see that it expects the strings to just be packed together inline.
>>>
>>> What you say may be correct for node and property names, just not
>>> property values.
>>
>>
>>
>> --
>> Jon Smirl
>> jonsmirl at gmail.com
>
>
>
> --
> Jon Smirl
> jonsmirl at gmail.com



-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Tue, Jul 30, 2013 at 2:25 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 07/30/2013 12:04 PM, jonsmirl at gmail.com wrote:
> ...
>> So what does an alternating use of strings do?
>>
>>  dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>
> cat > temp.dts << ENDOFHERE
> /dts-v1/;
>
> / {
>     dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>
>     pdma0: dma { };
> };
> ENDOFHERE
>
> dtc -O dtb -o temp.dtb -I dts temp.dts
>
> hexdump -C temp.dtb
>
> yields:
>
>> 00000000  d0 0d fe ed 00 00 00 ab  00 00 00 38 00 00 00 98  |...........8....|
>> 00000010  00 00 00 28 00 00 00 11  00 00 00 10 00 00 00 00  |...(............|
>> 00000020  00 00 00 13 00 00 00 60  00 00 00 00 00 00 00 00  |.......`........|
>> 00000030  00 00 00 00 00 00 00 00  00 00 00 01 00 00 00 00  |................|
>> 00000040  00 00 00 03 00 00 00 16  00 00 00 00 74 78 00 00  |............tx..|
>                                                  ss ss ss pp
>> 00000050  00 00 01 00 00 00 07 72  78 00 00 00 00 01 00 00  |.......rx.......|
>             pp pp pp ii ii ii ii ss  ss ss pp pp pp pp ii ii
>> 00000060  00 06 00 00 00 00 00 01  64 6d 61 00 00 00 00 03  |........dma.....|
>             ii ii
>
> ... where ss represents a string byte, pp a phandle byte, ii an integer
> byte.
>
>> 00000070  00 00 00 04 00 00 00 05  00 00 00 01 00 00 00 03  |................|
>> 00000080  00 00 00 04 00 00 00 0b  00 00 00 01 00 00 00 02  |................|
>> 00000090  00 00 00 02 00 00 00 09  64 6d 61 73 00 6c 69 6e  |........dmas.lin|
>> 000000a0  75 78 2c 70 68 61 6e 64  6c 65 00                 |ux,phandle.|
>
> Starting about half way through that last line is the string table.
>
> (the string table indirection is for property names, not property
> values, and it looks like not for node names either)

Ok, so this won't work without introducing variable length data. Too
bad, I would have liked to eliminate the paired attributes.

-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[Mark Brown]

On Tue, Jul 30, 2013 at 01:29:56PM -0400, jonsmirl at gmail.com wrote:
> On Tue, Jul 30, 2013 at 1:19 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> > On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:

> > > Is something similar to this possible in device tree syntax?
> > > dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;

> > I /think/ you can physically write that in *.dts, or something very
> > similar; with the strings outside the <>:

> > dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;

> Can we turn the strings into string phandles? Then they'd be fixed sized.

> That might provide a way to internationalize the strings too.

I don't know how viable or sane this actually is but one suggestion I
just made elsewhere which might help is to fix this at the dtc level by
having syntatic sugar which will do the parallel arrays trick that
people are currently using to write things out.  The user writes one
array with multiple types of data in it and the actual output is two or
more matched arrays each with a single type of data.

It would help legibility and wouldn't impact the parsers.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130730/a8e63708/attachment.sig>

Defining schemas for Device Tree

[Tomasz Figa]

On Monday 29 of July 2013 02:21:52 Tomasz Figa wrote:
> Hi,
> 
> As promised I am starting a discussion about Device Tree schema. Let's
> first shortly introduce the problem.
> 
> Device Tree is a text-based data structure used to describe hardware. Its
> main point is separation from kernel code, which has a lot of benefits,
> but, at the moment, also a huge drawback - there is no verification of
> device tree sources against defined bindings. All the dtc compiler does
> currently are syntax checks - no semantic analysis is performed (except
> some really basic things). What this means is that anybody can put
> anything in their device tree and end up with the dts compiling fine only
> to find out that something is wrong at boot time.
> 
> Currently, device tree bindings are described in plain text documentation
> files, which can not be considered a formal way of binding description.
> While such documentation provides information for developers/users that
> need to work with particular bindings, it can not be easily used as input
> for validation of device tree sources. This means that we need to define a
> more formal way of binding description, in other words - Device Tree
> schema.
> 
> To find a solution for this problem, we must first answer several
> questions to determine a set of requirements we have to meet.
> 
> a) What is a device tree binding?
> 
> For our purposes, I will define a binding as internal format of some
> device tree node, which can be matched using of_find_matching_node(). In
> other words, key for a binding would be node name and/or value of
> compatible property and/or node type. Value for a binding would be a list
> of properties with their formats and/or subnodes with their bindings.

Now as I think of it, there is one more thing that needs to be considered. Do 
we want the schemas to strictly specify available set of properties, warning 
about any unrecognized ones or just enforce checking of recognized properties 
(including presence of required ones)? Or maybe both, depending on some 
keyword or whatever?

> b) What information should be specified in schemas? What level of
>    granularity is required?
> 
> For each property we need to have at least following data specified:
>  - property name (or property name format, e.g. regex),
>  - whether the property is mandatory or optional,
>  - data type of value.
> 
> As for now, I can think of following data types used in device trees:
>  - boolean (i.e. without value),
>  - array of strings (including single string),
>  - array of u32 (including single u32),
>  - specifier (aka phandle with args, including cases with 0 args),
>  - variable-length cells (e.g. #address-cells of u32s).
> 
> Some properties might require a combination of data types to be specified
> or even an array of combinations, like interrupt-map property, which is an
> array of entries consisting of:
>  - #address-cells u32s,
>  - #interrupt-cells u32s,
>  - specifier (phandle of interrupt controller and u32 of count defined by
>    #interrupt-cells of the controller).
> 
> We probably want to define allowed range of values for given property, be
> it contiguous or enumerated.
> 
> As for subnodes, I think we need to define following constraints:
>  - node name (or node name format, e.g. regex),
>  - optional or not,
>  - how many nodes of this type can be present (one, limited, unlimited),
>  - recursively define binding for such node type.
> 
> We probably also want human readable descriptions for all properties and
> subnodes, so a textual documentation (like the one currently available)
> could be generated from schemas.
> 
> c) What about generic bindings? (e.g. for subsystems like pinctrl or
> regulators)
> 
> This is where things get more interesting. Looks like we need some kind of
> inheritance for bindings or binding templates. Templates sound more
> appropriate here, because most of the generic bindings do not fully
> conform to what I defined as binding and need device-specific parameters
> to become so.
> 
> Let's consider first example taken from regulator subsystem.
> 
> 	device {
> 		compatible = "foo,mydevice";
> 		/* ... */
> 		core-supply = <&regulator_a>;
> 		io-supply = <&regulator_b>;
> 		/* ... */
> 	};
> 
> Bindings of regulator subsystem define the way of regulator lookup to be
> based on property matching following definition:
> 
> 	#define REGULATOR(name) name ## _supply = <&phandle>
> 
> As you can see, the binding is parametrized, i.e. part of it is defined
> globally, but part is device-specific. Similarly for pinctrl subsystem:
> 
> 	device {
> 		compatible = "foo,mydevice";
> 		/* ... */
> 		pinctrl-names = "state0", "state1";
> 		pinctrl-0 = <&phandle>...;
> 		pinctrl-1 = <&phandle>...;
> 		/* ... */
> 	};
> 
> This binding is now parametrized in a more complex way:
> 
> 	#define PINCTRL(name0, name1, ..., nameN) \
> 		pinctrl-names = name0, name1, ..., nameN; \
> 		pinctrl-0 = <&phandle>...; \
> 		pinctrl-1 = <&phandle>...; \
> 		... \
> 		pinctrl-N = <&phandle>...;
> 
> We need to have a way to describe this kind of inheritance, if we don't
> want to respecify generic attributes in all device bindings using them.
> 
> d) When should the validation happen and what should handle it?
> 
> In my opinion, similarly to compilation of board files, validation should
> be happening at dts compile time, to show any warnings or errors as early
> as possible.
> 
> Whether this should be integrated into dtc or rather handled by external
> tool is another question. Since we are already processing device tree
> sources in dtc, it might be reasonable to reuse its dts parsing
> infrastructure and add validation there, especially that dtc is supposed
> to already contain some infrastructure for doing checks on device tree as
> well. Nothing stops us from running validation on already compiled dtbs,
> though, using an extra tool.
> 
> e) What format should be used for Device Tree schema?
> 
> This is a non-trivial problem. Key criteria I can think of are as follows:
>  - the whole set of information established above must be representable,
>  - human-readable, easy to create and edit (extend), preferably similar to
>    something already existing, so could be easily learnt,
>  - something that can be integrated with dtc with reasonable amount of
>    work or can reuse a lot (if not all) of already existing parsing code.
> 
> Okay, this should be enough to have some discussion. I will post a
> follow-up with my proposal of schema format to separate general discussion
> from discussion about the proposal, but this will happen tomorrow, as now
> it's time to get some sleep.

Finally I didn't manage to post it yet, but this might be better, as there 
have been a lot of interesting points in the discussion, which I'm now taking 
into consideration, while writing a sample schema.

Best regards,
Tomasz

> For now please think about the points above and feel free to correct
> anything wrong or suggest what else should be taken into consideration
> for DT schemas. Let the discussion start.
> 
> Best regards,
> Tomasz

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Tue, Jul 30, 2013 at 4:57 PM, Mark Brown <broonie@kernel.org> wrote:
> On Tue, Jul 30, 2013 at 01:29:56PM -0400, jonsmirl at gmail.com wrote:
>> On Tue, Jul 30, 2013 at 1:19 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> > On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:
>
>> > > Is something similar to this possible in device tree syntax?
>> > > dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;
>
>> > I /think/ you can physically write that in *.dts, or something very
>> > similar; with the strings outside the <>:
>
>> > dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>
>> Can we turn the strings into string phandles? Then they'd be fixed sized.
>
>> That might provide a way to internationalize the strings too.
>
> I don't know how viable or sane this actually is but one suggestion I
> just made elsewhere which might help is to fix this at the dtc level by
> having syntatic sugar which will do the parallel arrays trick that
> people are currently using to write things out.  The user writes one
> array with multiple types of data in it and the actual output is two or
> more matched arrays each with a single type of data.
>
> It would help legibility and wouldn't impact the parsers.

FDT is versioned and PowerPC docs say that multiple versions have been
released.  If we find enough flaws we could do a new release then use
the quirks scheme to read the older versions.

So far the only issue I know of is that all of the strings aren't
stored in the string table. If all of the strings were in the string
table then we wouldn't have this variable attribute size problem.

8.2 Header
The layout of the header for the device tree is defined by the
following C structure. All the header
fields are 32-bit integers, stored in big-endian format.
struct fdt_header {
uint32_t magic;
uint32_t totalsize;
uint32_t off_dt_struct;
uint32_t off_dt_strings;
uint32_t off_mem_rsvmap;
uint32_t version;
uint32_t last_comp_version;
uint32_t boot_cpuid_phys;
uint32_t size_dt_strings;
uint32_t size_dt_struct;
};
?
magic
This field shall contain the value 0xd00dfeed (big-endian).
?
totalsize
This field shall contain the total size of the device tree data
structure. This size shall
encompass all sections of the structure: the header, the memory
reservation block, structure
block and strings block, as well as any free space gaps between the
blocks or after the final
block.
?
off_dt_struct
This field shall contain the offset in bytes of the structure block
(see 8.4) from the beginning
of the header.
?
off_dt_strings
This field shall contain the offset in bytes of the strings block (see
8.5) from the beginning of
the header.
?
off_mem_rsvmap
This field shall contain the offset in bytes of the memory reservation
block (see 8.3) from the
beginning of the header.
?
version
This field shall contain the version of the device tree data
structure. The version is 17 if using
the structure as defined in this document. An ePAPR boot program may
provide the device
tree of a later version, in which case this field shall contain the
version number defined in
whichever later document gives the details of that version.
?
last_comp_version
This field shall contain the lowest version of the device tree data
structure with which the
version used is backwards compatible. So, for the structure as defined
in this document
(version 17), this field shall contain 16 because version 17 is
backwards compatible with
version 16, but not earlier versions. As per 8.1, an ePAPR boot
program should provide a
device tree in a format which is backwards compatible with version 16,
and thus this field
shall always contain 16.
?
boot_cpuid_phys
This field shall contain the physical ID of the system?s boot CPU. It
shall be identical to the
physical ID given in the reg property of that CPU node within the device tree.
?
size_dt_strings
This field shall contain the length in bytes of the strings block
section of the device tree blob.
?
size_dt_struct
This field shall contain the length in bytes of the structure block
section of the device tree
blob.

-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[Stephen Warren]

On 07/30/2013 04:16 PM, Tomasz Figa wrote:
> On Monday 29 of July 2013 02:21:52 Tomasz Figa wrote:
>> Hi,
>>
>> As promised I am starting a discussion about Device Tree schema. Let's
>> first shortly introduce the problem.
>>
>> Device Tree is a text-based data structure used to describe hardware. Its
>> main point is separation from kernel code, which has a lot of benefits,
>> but, at the moment, also a huge drawback - there is no verification of
>> device tree sources against defined bindings. All the dtc compiler does
>> currently are syntax checks - no semantic analysis is performed (except
>> some really basic things). What this means is that anybody can put
>> anything in their device tree and end up with the dts compiling fine only
>> to find out that something is wrong at boot time.
>>
>> Currently, device tree bindings are described in plain text documentation
>> files, which can not be considered a formal way of binding description.
>> While such documentation provides information for developers/users that
>> need to work with particular bindings, it can not be easily used as input
>> for validation of device tree sources. This means that we need to define a
>> more formal way of binding description, in other words - Device Tree
>> schema.
>>
>> To find a solution for this problem, we must first answer several
>> questions to determine a set of requirements we have to meet.
>>
>> a) What is a device tree binding?
>>
>> For our purposes, I will define a binding as internal format of some
>> device tree node, which can be matched using of_find_matching_node(). In
>> other words, key for a binding would be node name and/or value of
>> compatible property and/or node type. Value for a binding would be a list
>> of properties with their formats and/or subnodes with their bindings.
> 
> Now as I think of it, there is one more thing that needs to be considered. Do 
> we want the schemas to strictly specify available set of properties, warning 
> about any unrecognized ones or just enforce checking of recognized properties 
> (including presence of required ones)? Or maybe both, depending on some 
> keyword or whatever?

I think one of the use-cases that having schema validation was asked to
cover was the case of a typo in a property name leading to that property
having no effect. This could only be caught by the validation tool if it
warned (or worse) about unknown properties.

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Tue, Jul 30, 2013 at 6:16 PM, Tomasz Figa <tomasz.figa@gmail.com> wrote:
> On Monday 29 of July 2013 02:21:52 Tomasz Figa wrote:
>> Hi,
>>
>> As promised I am starting a discussion about Device Tree schema. Let's
>> first shortly introduce the problem.
>>
>> Device Tree is a text-based data structure used to describe hardware. Its
>> main point is separation from kernel code, which has a lot of benefits,
>> but, at the moment, also a huge drawback - there is no verification of
>> device tree sources against defined bindings. All the dtc compiler does
>> currently are syntax checks - no semantic analysis is performed (except
>> some really basic things). What this means is that anybody can put
>> anything in their device tree and end up with the dts compiling fine only
>> to find out that something is wrong at boot time.
>>
>> Currently, device tree bindings are described in plain text documentation
>> files, which can not be considered a formal way of binding description.
>> While such documentation provides information for developers/users that
>> need to work with particular bindings, it can not be easily used as input
>> for validation of device tree sources. This means that we need to define a
>> more formal way of binding description, in other words - Device Tree
>> schema.
>>
>> To find a solution for this problem, we must first answer several
>> questions to determine a set of requirements we have to meet.
>>
>> a) What is a device tree binding?
>>
>> For our purposes, I will define a binding as internal format of some
>> device tree node, which can be matched using of_find_matching_node(). In
>> other words, key for a binding would be node name and/or value of
>> compatible property and/or node type. Value for a binding would be a list
>> of properties with their formats and/or subnodes with their bindings.
>
> Now as I think of it, there is one more thing that needs to be considered. Do
> we want the schemas to strictly specify available set of properties, warning
> about any unrecognized ones or just enforce checking of recognized properties
> (including presence of required ones)? Or maybe both, depending on some
> keyword or whatever?

I think it should warn about anything it doesn't recognize. That's the
hint for new DT developers to get on lkml and talk about their need
for a new attribute to be added to the master schema.

The device tree quirks should be put into a file which can both be
used at init time by the kernel or by the schema checker.  Now you can
feed those flash based DTB into the schema checker, apply the quirks
to them and ensure that your quirk code is working right. The
validator would take DTB input, apply the quirks, and write the DTS
output plus any errors it finds in the transformed tree.

So the dtc source in the kernel tree would build the quirks files into it.

>
>> b) What information should be specified in schemas? What level of
>>    granularity is required?
>>
>> For each property we need to have at least following data specified:
>>  - property name (or property name format, e.g. regex),
>>  - whether the property is mandatory or optional,
>>  - data type of value.
>>
>> As for now, I can think of following data types used in device trees:
>>  - boolean (i.e. without value),
>>  - array of strings (including single string),
>>  - array of u32 (including single u32),
>>  - specifier (aka phandle with args, including cases with 0 args),
>>  - variable-length cells (e.g. #address-cells of u32s).
>>
>> Some properties might require a combination of data types to be specified
>> or even an array of combinations, like interrupt-map property, which is an
>> array of entries consisting of:
>>  - #address-cells u32s,
>>  - #interrupt-cells u32s,
>>  - specifier (phandle of interrupt controller and u32 of count defined by
>>    #interrupt-cells of the controller).
>>
>> We probably want to define allowed range of values for given property, be
>> it contiguous or enumerated.
>>
>> As for subnodes, I think we need to define following constraints:
>>  - node name (or node name format, e.g. regex),
>>  - optional or not,
>>  - how many nodes of this type can be present (one, limited, unlimited),
>>  - recursively define binding for such node type.
>>
>> We probably also want human readable descriptions for all properties and
>> subnodes, so a textual documentation (like the one currently available)
>> could be generated from schemas.
>>
>> c) What about generic bindings? (e.g. for subsystems like pinctrl or
>> regulators)
>>
>> This is where things get more interesting. Looks like we need some kind of
>> inheritance for bindings or binding templates. Templates sound more
>> appropriate here, because most of the generic bindings do not fully
>> conform to what I defined as binding and need device-specific parameters
>> to become so.
>>
>> Let's consider first example taken from regulator subsystem.
>>
>>       device {
>>               compatible = "foo,mydevice";
>>               /* ... */
>>               core-supply = <&regulator_a>;
>>               io-supply = <&regulator_b>;
>>               /* ... */
>>       };
>>
>> Bindings of regulator subsystem define the way of regulator lookup to be
>> based on property matching following definition:
>>
>>       #define REGULATOR(name) name ## _supply = <&phandle>
>>
>> As you can see, the binding is parametrized, i.e. part of it is defined
>> globally, but part is device-specific. Similarly for pinctrl subsystem:
>>
>>       device {
>>               compatible = "foo,mydevice";
>>               /* ... */
>>               pinctrl-names = "state0", "state1";
>>               pinctrl-0 = <&phandle>...;
>>               pinctrl-1 = <&phandle>...;
>>               /* ... */
>>       };
>>
>> This binding is now parametrized in a more complex way:
>>
>>       #define PINCTRL(name0, name1, ..., nameN) \
>>               pinctrl-names = name0, name1, ..., nameN; \
>>               pinctrl-0 = <&phandle>...; \
>>               pinctrl-1 = <&phandle>...; \
>>               ... \
>>               pinctrl-N = <&phandle>...;
>>
>> We need to have a way to describe this kind of inheritance, if we don't
>> want to respecify generic attributes in all device bindings using them.
>>
>> d) When should the validation happen and what should handle it?
>>
>> In my opinion, similarly to compilation of board files, validation should
>> be happening at dts compile time, to show any warnings or errors as early
>> as possible.
>>
>> Whether this should be integrated into dtc or rather handled by external
>> tool is another question. Since we are already processing device tree
>> sources in dtc, it might be reasonable to reuse its dts parsing
>> infrastructure and add validation there, especially that dtc is supposed
>> to already contain some infrastructure for doing checks on device tree as
>> well. Nothing stops us from running validation on already compiled dtbs,
>> though, using an extra tool.
>>
>> e) What format should be used for Device Tree schema?
>>
>> This is a non-trivial problem. Key criteria I can think of are as follows:
>>  - the whole set of information established above must be representable,
>>  - human-readable, easy to create and edit (extend), preferably similar to
>>    something already existing, so could be easily learnt,
>>  - something that can be integrated with dtc with reasonable amount of
>>    work or can reuse a lot (if not all) of already existing parsing code.
>>
>> Okay, this should be enough to have some discussion. I will post a
>> follow-up with my proposal of schema format to separate general discussion
>> from discussion about the proposal, but this will happen tomorrow, as now
>> it's time to get some sleep.
>
> Finally I didn't manage to post it yet, but this might be better, as there
> have been a lot of interesting points in the discussion, which I'm now taking
> into consideration, while writing a sample schema.
>
> Best regards,
> Tomasz
>
>> For now please think about the points above and feel free to correct
>> anything wrong or suggest what else should be taken into consideration
>> for DT schemas. Let the discussion start.
>>
>> Best regards,
>> Tomasz



-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[Mark Brown]

On Tue, Jul 30, 2013 at 06:19:08PM -0400, jonsmirl at gmail.com wrote:

> FDT is versioned and PowerPC docs say that multiple versions have been
> released.  If we find enough flaws we could do a new release then use
> the quirks scheme to read the older versions.

That does involve updating everything that wants to understand FDTs in
the new FDT format version though.  I'm not sure that's going to be
greeted with universal enthusiasm.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130731/af092b57/attachment-0001.sig>

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Tue, Jul 30, 2013 at 7:03 PM, Mark Brown <broonie@kernel.org> wrote:
> On Tue, Jul 30, 2013 at 06:19:08PM -0400, jonsmirl at gmail.com wrote:
>
>> FDT is versioned and PowerPC docs say that multiple versions have been
>> released.  If we find enough flaws we could do a new release then use
>> the quirks scheme to read the older versions.
>
> That does involve updating everything that wants to understand FDTs in
> the new FDT format version though.  I'm not sure that's going to be
> greeted with universal enthusiasm.

Shipped hardware containing FDTs would keep the version they shipped
with. That's the point of making this  a stable ABI. The Linux quirk
code would convert the older format to the new format at boot time.

So to change this you need to:
1) change dtc to produce the new version
2) adjust the kernel code to understand the new version
3) adjust the quirk code to convert the known set of older firmware
based FDTs into the new format. For the simple string case the strings
would be turned into indirect pointers instead of being in-line.

The only way you could get in trouble would be by upgrading an old
firmware FDT to the new format and then booting an old Linux or
another OS. But this is a general problem not specific to the FDT
versioning - if you replace the firmware FDT with one of the later
Linux ones that has been rewritten, the older stuff isn't going to
boot either since it won't understand the changes.  Simple answer - if
you are going to boot other OSes or old Linux kernels, don't mess with
the firmware FDT that the device shipped with, leave it alone.

Uboot could be an issue but I don't think they are actually doing
anything with FDTs yet, they are just talking about doing it.


-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[Tony Lindgren]

* jonsmirl at gmail.com <jonsmirl@gmail.com> [130730 11:36]:
> On Tue, Jul 30, 2013 at 2:25 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> > On 07/30/2013 12:04 PM, jonsmirl at gmail.com wrote:
> > ...
> >> So what does an alternating use of strings do?
> >>
> >>  dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
> >
> > cat > temp.dts << ENDOFHERE
> > /dts-v1/;
> >
> > / {
> >     dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
> >
> >     pdma0: dma { };
> > };
> > ENDOFHERE
> >
> > dtc -O dtb -o temp.dtb -I dts temp.dts
> >
> > hexdump -C temp.dtb
> >
> > yields:
> >
> >> 00000000  d0 0d fe ed 00 00 00 ab  00 00 00 38 00 00 00 98  |...........8....|
> >> 00000010  00 00 00 28 00 00 00 11  00 00 00 10 00 00 00 00  |...(............|
> >> 00000020  00 00 00 13 00 00 00 60  00 00 00 00 00 00 00 00  |.......`........|
> >> 00000030  00 00 00 00 00 00 00 00  00 00 00 01 00 00 00 00  |................|
> >> 00000040  00 00 00 03 00 00 00 16  00 00 00 00 74 78 00 00  |............tx..|
> >                                                  ss ss ss pp
> >> 00000050  00 00 01 00 00 00 07 72  78 00 00 00 00 01 00 00  |.......rx.......|
> >             pp pp pp ii ii ii ii ss  ss ss pp pp pp pp ii ii
> >> 00000060  00 06 00 00 00 00 00 01  64 6d 61 00 00 00 00 03  |........dma.....|
> >             ii ii
> >
> > ... where ss represents a string byte, pp a phandle byte, ii an integer
> > byte.
> >
> >> 00000070  00 00 00 04 00 00 00 05  00 00 00 01 00 00 00 03  |................|
> >> 00000080  00 00 00 04 00 00 00 0b  00 00 00 01 00 00 00 02  |................|
> >> 00000090  00 00 00 02 00 00 00 09  64 6d 61 73 00 6c 69 6e  |........dmas.lin|
> >> 000000a0  75 78 2c 70 68 61 6e 64  6c 65 00                 |ux,phandle.|
> >
> > Starting about half way through that last line is the string table.
> >
> > (the string table indirection is for property names, not property
> > values, and it looks like not for node names either)
> 
> Ok, so this won't work without introducing variable length data. Too
> bad, I would have liked to eliminate the paired attributes.

Yup that's my experience too. But then again, using string values
in .dts should not be done in general as it bloats things and makes
the parsing slow.

These kind of names for something is usually something that should
be implemented as user space debug tools, and the kernel really does
not need to know names for something coming from DT.

Of course without using string names there's the issue how to know
what is rx dma and what is tx dma in the binding :) Probably what
we've done with interrupt numbers is the way to go with DT to avoid
string names.

Regards,

Tony

Defining schemas for Device Tree

[Mark Brown]

On Tue, Jul 30, 2013 at 07:23:36PM -0400, jonsmirl at gmail.com wrote:

> Uboot could be an issue but I don't think they are actually doing
> anything with FDTs yet, they are just talking about doing it.

Right, plus any other bootloaders that handle the DT in some way.  Those
that just copy the file through without looking at it aren't an issue
but I thought people were doing things like adding in properties for
serial numbers but perhaps that's less widespread than I'd thought and
it's not an issue.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130731/85b18fa2/attachment.sig>

Defining schemas for Device Tree

[jonsmirl at gmail.com]

On Wed, Jul 31, 2013 at 7:34 AM, Mark Brown <broonie@kernel.org> wrote:
> On Tue, Jul 30, 2013 at 07:23:36PM -0400, jonsmirl at gmail.com wrote:
>
>> Uboot could be an issue but I don't think they are actually doing
>> anything with FDTs yet, they are just talking about doing it.
>
> Right, plus any other bootloaders that handle the DT in some way.  Those
> that just copy the file through without looking at it aren't an issue
> but I thought people were doing things like adding in properties for
> serial numbers but perhaps that's less widespread than I'd thought and
> it's not an issue.

uboot patches up the amount of RAM it finds into the FDT. It might
patch up clock speed too, I don't remember. I haven't seen it add
serial numbers but it might, not sure which serial numbers.

There has been talk for several years about turning uboot into a bunch
of device drivers and using FDT to control it. Allwinner has done that
with the fex implementation. The uboot work is still in the talking
about it phase.

-- 
Jon Smirl
jonsmirl at gmail.com

Defining schemas for Device Tree

[Tomasz Figa]

On Wednesday 31 of July 2013 08:01:54 jonsmirl at gmail.com wrote:
> On Wed, Jul 31, 2013 at 7:34 AM, Mark Brown <broonie@kernel.org> wrote:
> > On Tue, Jul 30, 2013 at 07:23:36PM -0400, jonsmirl at gmail.com wrote:
> >> Uboot could be an issue but I don't think they are actually doing
> >> anything with FDTs yet, they are just talking about doing it.
> > 
> > Right, plus any other bootloaders that handle the DT in some way.  Those
> > that just copy the file through without looking at it aren't an issue
> > but I thought people were doing things like adding in properties for
> > serial numbers but perhaps that's less widespread than I'd thought and
> > it's not an issue.
> 
> uboot patches up the amount of RAM it finds into the FDT. It might
> patch up clock speed too, I don't remember. I haven't seen it add
> serial numbers but it might, not sure which serial numbers.
> 
> There has been talk for several years about turning uboot into a bunch
> of device drivers and using FDT to control it. Allwinner has done that
> with the fex implementation. The uboot work is still in the talking
> about it phase.

u-boot still does not have generic driver model or at least did not have when 
I last checked it not so long ago, so it seems like there is still long way to 
go for using FDT as source of information about hardware in u-boot.

barebox [1], aka u-boot-v2, might be closer to this approach (it might be 
already using it, but this needs to be checked) as it already has generic 
driver model and support for DT.

Best regards,
Tomasz

[1] http://barebox.org/

Re: [Ksummit-2013-discuss] Defining schemas for Device Tree

[Thomas Petazzoni]

Dear Tomasz Figa,

On Wed, 31 Jul 2013 14:21:49 +0200, Tomasz Figa wrote:

> u-boot still does not have generic driver model or at least did not have when 
> I last checked it not so long ago, so it seems like there is still long way to 
> go for using FDT as source of information about hardware in u-boot.
> 
> barebox [1], aka u-boot-v2, might be closer to this approach (it might be 
> already using it, but this needs to be checked) as it already has generic 
> driver model and support for DT.

There are already some platforms in Barebox that are using the Device
Tree. For example, the basic Marvell platforms support in Barebox now
uses the Device Tree.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Ksummit-2013-discuss] Defining schemas for Device Tree

[Thomas Petazzoni]

Dear Tomasz Figa,

On Wed, 31 Jul 2013 14:21:49 +0200, Tomasz Figa wrote:

> u-boot still does not have generic driver model or at least did not have when 
> I last checked it not so long ago, so it seems like there is still long way to 
> go for using FDT as source of information about hardware in u-boot.
> 
> barebox [1], aka u-boot-v2, might be closer to this approach (it might be 
> already using it, but this needs to be checked) as it already has generic 
> driver model and support for DT.

There are already some platforms in Barebox that are using the Device
Tree. For example, the basic Marvell platforms support in Barebox now
uses the Device Tree.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Defining schemas for Device Tree

[Sascha Hauer]

On Wed, Jul 31, 2013 at 02:21:49PM +0200, Tomasz Figa wrote:
> On Wednesday 31 of July 2013 08:01:54 jonsmirl at gmail.com wrote:
> > On Wed, Jul 31, 2013 at 7:34 AM, Mark Brown <broonie@kernel.org> wrote:
> > > On Tue, Jul 30, 2013 at 07:23:36PM -0400, jonsmirl at gmail.com wrote:
> > >> Uboot could be an issue but I don't think they are actually doing
> > >> anything with FDTs yet, they are just talking about doing it.
> > > 
> > > Right, plus any other bootloaders that handle the DT in some way.  Those
> > > that just copy the file through without looking at it aren't an issue
> > > but I thought people were doing things like adding in properties for
> > > serial numbers but perhaps that's less widespread than I'd thought and
> > > it's not an issue.
> > 
> > uboot patches up the amount of RAM it finds into the FDT. It might
> > patch up clock speed too, I don't remember. I haven't seen it add
> > serial numbers but it might, not sure which serial numbers.
> > 
> > There has been talk for several years about turning uboot into a bunch
> > of device drivers and using FDT to control it. Allwinner has done that
> > with the fex implementation. The uboot work is still in the talking
> > about it phase.
> 
> u-boot still does not have generic driver model or at least did not have when 
> I last checked it not so long ago, so it seems like there is still long way to 
> go for using FDT as source of information about hardware in u-boot.
> 
> barebox [1], aka u-boot-v2, might be closer to this approach (it might be 
> already using it, but this needs to be checked) as it already has generic 
> driver model and support for DT.

Besides mvebu several newer i.MX platforms are probed from the
devicetree in barebox, so in order to reuse the devicetrees it would be
very good to have the devicetree bindings stable.

Sascha

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Defining schemas for Device Tree

[Dave Martin]

On Tue, Jul 30, 2013 at 09:57:16PM +0100, Mark Brown wrote:
> On Tue, Jul 30, 2013 at 01:29:56PM -0400, jonsmirl at gmail.com wrote:
> > On Tue, Jul 30, 2013 at 1:19 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> > > On 07/30/2013 07:14 AM, jonsmirl at gmail.com wrote:
> 
> > > > Is something similar to this possible in device tree syntax?
> > > > dmas =  <"tx" &pdma0 7>,  <"rx"  &pdma0 6>;
> 
> > > I /think/ you can physically write that in *.dts, or something very
> > > similar; with the strings outside the <>:
> 
> > > dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
> 
> > Can we turn the strings into string phandles? Then they'd be fixed sized.
> 
> > That might provide a way to internationalize the strings too.
> 
> I don't know how viable or sane this actually is but one suggestion I
> just made elsewhere which might help is to fix this at the dtc level by
> having syntatic sugar which will do the parallel arrays trick that
> people are currently using to write things out.  The user writes one
> array with multiple types of data in it and the actual output is two or
> more matched arrays each with a single type of data.
> 
> It would help legibility and wouldn't impact the parsers.

There may often be a conflict between making _good_ bindings, and making
bindings which are convenient for maintainers to edit in .dts syntax.

I think good bindings (in the sense of good ABI) must take precedence:
good bindings need to interoperate well over a long period of time, and
should be robust against future evolution of hardware platforms OSes.  If
they look nice in the .dts file, that's a bonus -- but I don't think it
should be viewed as a requirement.

A pair of properties like dma-names/dmas may be a bit cumbersome, but
it can be precise, unambiguous, and simple and easy to parse and check.
The semantics can be extended in the future if necessary, by adding
additional companion properties.  From my perspective, this can work as
a template for good bindings.


We could add syntactic sugar to make those property pairs easier to
maintain... but I worry that haphazard extensions to dtc may leave us
with an unmanageable syntax which we will regret later.   This seems to
be a common failure mode of domain-specific languages.

Someone trying to maintain a complex .dts can always make their own life
easier with a bit of scripting.  After all, in a future of stable
bindings, a DT really shouldn't be changing much once it is complete.
I don't see a clear reason why these issues must be solved by dtc
itself.

Cheers
---Dave

Defining schemas for Device Tree

[Mark Brown]

On Wed, Jul 31, 2013 at 05:59:10PM +0100, Dave Martin wrote:

> There may often be a conflict between making _good_ bindings, and making
> bindings which are convenient for maintainers to edit in .dts syntax.

> I think good bindings (in the sense of good ABI) must take precedence:
> good bindings need to interoperate well over a long period of time, and
> should be robust against future evolution of hardware platforms OSes.  If
> they look nice in the .dts file, that's a bonus -- but I don't think it
> should be viewed as a requirement.

While I take your point that we need to have a sane ABI I think you're
understating the importance of the human factors here - the harder it is
to reliably understand and edit the device trees the more likely it is
that people will end up shipping mistakes which undermines the work on
defining stable ABIs.

> A pair of properties like dma-names/dmas may be a bit cumbersome, but
> it can be precise, unambiguous, and simple and easy to parse and check.
> The semantics can be extended in the future if necessary, by adding
> additional companion properties.  From my perspective, this can work as
> a template for good bindings.

I think this is fine for bindings but it's just not really legible for
humans once the lists start growing.  We need a human interaction format
which is suitable for humans to work with, it's not good when people
start finding that it's a set backwards in terms of their ability to
work with their systems.

> Someone trying to maintain a complex .dts can always make their own life
> easier with a bit of scripting.  After all, in a future of stable
> bindings, a DT really shouldn't be changing much once it is complete.
> I don't see a clear reason why these issues must be solved by dtc
> itself.

I don't think it's terribly important if they're solved in dtc itself so
long as it's part of the standard tooling that everyone has access to.
There's a reasonable number of new SoCs get made which tend to be the
things that suffer most.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130731/5edf5c3d/attachment.sig>

Defining schemas for Device Tree

[Stephen Warren]

On 07/30/2013 05:23 PM, jonsmirl at gmail.com wrote:
...
> Uboot could be an issue but I don't think they are actually doing
> anything with FDTs yet, they are just talking about doing it.

A couple U-Boot platforms (Tegra, Exynos) actually use DT to
parameterize U-Boot itself, just like the kernel uses it, although:

* often the bindings used there are not the "real" bindings defined in
the kernel:-(

* They use a DTB embedded into the U-Boot binary, which is a different
DTB to that passed to the kernel, which at least in the Tegra case is
always loaded from disk e.g. from /boot, just like zImage is.

U-Boot certainly passes DTs to the kernel, and performs some limited
modification on it such as setting the kernel cmdline, memory size, etc.

Defining schemas for Device Tree

[Stephen Warren]

On 07/30/2013 05:23 PM, jonsmirl at gmail.com wrote:
> On Tue, Jul 30, 2013 at 7:03 PM, Mark Brown <broonie@kernel.org> wrote:
>> On Tue, Jul 30, 2013 at 06:19:08PM -0400, jonsmirl at gmail.com wrote:
>>
>>> FDT is versioned and PowerPC docs say that multiple versions have been
>>> released.  If we find enough flaws we could do a new release then use
>>> the quirks scheme to read the older versions.
>>
>> That does involve updating everything that wants to understand FDTs in
>> the new FDT format version though.  I'm not sure that's going to be
>> greeted with universal enthusiasm.
> 
> Shipped hardware containing FDTs would keep the version they shipped
> with. That's the point of making this  a stable ABI. The Linux quirk
> code would convert the older format to the new format at boot time.
> 
> So to change this you need to:
> 1) change dtc to produce the new version
> 2) adjust the kernel code to understand the new version
> 3) adjust the quirk code to convert the known set of older firmware
> based FDTs into the new format. For the simple string case the strings
> would be turned into indirect pointers instead of being in-line.

There's one issue issue with the quirk/conversion layer in the kernel.

Various combinations are supposed to work:

1) The "current" kernel with the "current" DTB. I think everyone agrees
on that much:-)

2) Using an old DTB with a new kernel. This could be handled by having
stable DT bindings, and/or by having a quirk/conversion layer in the
kernel to convert old DTBs into whatever format the new kernel expects.

3) Using a new DTB with a old kernel. Rob Herring raised the point that
a firmware update (which might include updating the DTB the firmware
passes to the kernel) should not make the kernel you have installed in
the filesystem fail to boot.

I'm assuming that if the quirk/conversion layer you're championing were
the solution chosen for (2), then at least some of the time, people
would start using the new DT binding when writing new DTs, or modifying
existing DTs. That would break point (3) above, since the new DT would
no longer work with the old kernel.

I think the only way to satisfy all three points above is to have truly
stable backwards-compatible bindings, without relying on any in-kernel
conversion layer.

Defining schemas for Device Tree

[Tomasz Figa]

On Wednesday 31 of July 2013 14:47:12 Stephen Warren wrote:
> On 07/30/2013 05:23 PM, jonsmirl at gmail.com wrote:
> > On Tue, Jul 30, 2013 at 7:03 PM, Mark Brown <broonie@kernel.org> 
wrote:
> >> On Tue, Jul 30, 2013 at 06:19:08PM -0400, jonsmirl at gmail.com wrote:
> >>> FDT is versioned and PowerPC docs say that multiple versions have
> >>> been
> >>> released.  If we find enough flaws we could do a new release then
> >>> use
> >>> the quirks scheme to read the older versions.
> >> 
> >> That does involve updating everything that wants to understand FDTs
> >> in
> >> the new FDT format version though.  I'm not sure that's going to be
> >> greeted with universal enthusiasm.
> > 
> > Shipped hardware containing FDTs would keep the version they shipped
> > with. That's the point of making this  a stable ABI. The Linux quirk
> > code would convert the older format to the new format at boot time.
> > 
> > So to change this you need to:
> > 1) change dtc to produce the new version
> > 2) adjust the kernel code to understand the new version
> > 3) adjust the quirk code to convert the known set of older firmware
> > based FDTs into the new format. For the simple string case the strings
> > would be turned into indirect pointers instead of being in-line.
> 
> There's one issue issue with the quirk/conversion layer in the kernel.
> 
> Various combinations are supposed to work:
> 
> 1) The "current" kernel with the "current" DTB. I think everyone agrees
> on that much:-)
> 
> 2) Using an old DTB with a new kernel. This could be handled by having
> stable DT bindings, and/or by having a quirk/conversion layer in the
> kernel to convert old DTBs into whatever format the new kernel expects.
> 
> 3) Using a new DTB with a old kernel. Rob Herring raised the point that
> a firmware update (which might include updating the DTB the firmware
> passes to the kernel) should not make the kernel you have installed in
> the filesystem fail to boot.
>
> I'm assuming that if the quirk/conversion layer you're championing were
> the solution chosen for (2), then at least some of the time, people
> would start using the new DT binding when writing new DTs, or modifying
> existing DTs. That would break point (3) above, since the new DT would
> no longer work with the old kernel.
> 
> I think the only way to satisfy all three points above is to have truly
> stable backwards-compatible bindings, without relying on any in-kernel
> conversion layer.

IMHO stable bindings is the way to go. We already managed to keep lot of 
existing bindings stable, see most of the generic bindings. Those got much 
more thought at design stage, much more review and ended up being 
something that could be stabilized.

If we define the process of bindings introduction correctly, we should be 
able to filter out those that won't work as stable thing and stabilize 
those that will. This is why we may need a staging tree for bindings and a 
way to develop such bindings to make them stable at some point of time.

I'm not really sure how Greg's staging tree works, but we might be able to 
get some hints for creating our staging process from him (now on Cc).

Best regards,
Tomasz

Defining schemas for Device Tree

[Dave Martin]

On Wed, Jul 31, 2013 at 07:59:47PM +0100, Mark Brown wrote:
> On Wed, Jul 31, 2013 at 05:59:10PM +0100, Dave Martin wrote:
> 
> > There may often be a conflict between making _good_ bindings, and making
> > bindings which are convenient for maintainers to edit in .dts syntax.
> 
> > I think good bindings (in the sense of good ABI) must take precedence:
> > good bindings need to interoperate well over a long period of time, and
> > should be robust against future evolution of hardware platforms OSes.  If
> > they look nice in the .dts file, that's a bonus -- but I don't think it
> > should be viewed as a requirement.
> 
> While I take your point that we need to have a sane ABI I think you're
> understating the importance of the human factors here - the harder it is
> to reliably understand and edit the device trees the more likely it is
> that people will end up shipping mistakes which undermines the work on
> defining stable ABIs.
> 
> > A pair of properties like dma-names/dmas may be a bit cumbersome, but
> > it can be precise, unambiguous, and simple and easy to parse and check.
> > The semantics can be extended in the future if necessary, by adding
> > additional companion properties.  From my perspective, this can work as
> > a template for good bindings.
> 
> I think this is fine for bindings but it's just not really legible for
> humans once the lists start growing.  We need a human interaction format
> which is suitable for humans to work with, it's not good when people
> start finding that it's a set backwards in terms of their ability to
> work with their systems.
> 
> > Someone trying to maintain a complex .dts can always make their own life
> > easier with a bit of scripting.  After all, in a future of stable
> > bindings, a DT really shouldn't be changing much once it is complete.
> > I don't see a clear reason why these issues must be solved by dtc
> > itself.
> 
> I don't think it's terribly important if they're solved in dtc itself so
> long as it's part of the standard tooling that everyone has access to.
> There's a reasonable number of new SoCs get made which tend to be the
> things that suffer most.

I agree with your points -- I just wanted to make the point that extensions
to .dts must be considered carefully, focusing on a few key features
that really are useful to everyone -- and with a commitment to support
and use those features.
 
If we start making ad-hoc extensions, I worry that there is a risk of
dtc forking -- has this issue been discussed?  How do we avoid that
situation?


A fork in dtc and .dts might not be so bad a problem as a fork in the
FDT format or DT bindings themselves, but it still sounds best avoided.
A Linux- or ARM-specific dtc would be just as bad as vanilla dtc plus an
obscure preprocessing framework that only the Linux (or ARM) folks use.


If we agree that the only acceptable form for "object-like" properties is
the paired-property model and enforce/strongly encourage this policy
when reviewing new bindings, then it may be worth adding dedicated
syntax to dtc.  We just shouldn't allow this tool to morph into dt-c++
unless we have a relly good reason.

Cheers
---Dave

Defining schemas for Device Tree

[Matt Sealey]

On Tue, Jul 30, 2013 at 1:28 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
> On Tue, Jul 30, 2013 at 2:25 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> On 07/30/2013 12:04 PM, jonsmirl at gmail.com wrote:
>> ...
>>> So what does an alternating use of strings do?
>>>
>>>  dmas = "tx", <&pdma0 7>, "rx", <&pdma0 6>;
>
> Ok, so this won't work without introducing variable length data. Too
> bad, I would have liked to eliminate the paired attributes.

It would be nice to eliminate things from device trees and do them
differently but right now, pairs like clocks and clock-names and dmas
and dma-names are basically so common, you're talking about cleaning
it up and affecting everyone and every board with a change.

There's really not a lot of code reduction you'd get in parsing and
fetching data just by using a single property vs. paired properties.

You're right, in the sense that a (in devicetree.org semantics)
"mixed-property" is the logical way to go here (bear with me, though),
this would be eminently supported by every device-tree-using system in
the world, ever - because properties are just encoded values. A string
should (and usually is..) not encoded directly in a property but in
some place elsewhere and a pointer to it is created. That string in
FDT should be in this indirect table of strings. Inside OF it would be
a Forth string (1 byte with the length, then the string, no NULL
terminator) and the 'pointer' (if you pushed it to the Forth stack)
would be to the size byte.

Context is king here. It doesn't matter what you put in a property but
*how you parse it* and how you parse it is entirely defined by the
binding.

1275 and it's related bindings did a very good job of defining
precisely the format and layout of every property on the basis of how
it is represented if you use the standard fetch method to pull it out,
you would get some reliable result - the "getprop" OFW service fills a
buffer you tell it to. You have to know and expect what you get back
in that buffer. You don't ask for strings or cells or a mix, just to
fill a buffer with whatever that property was. That is the defining
attribute for how FDT properties got done in the first instance.

What's not possible there, which makes it impractical, is to encode a
value that says "the string is here" in the middle of a mixed property
when using "getprop," which means nobody did it. You only gave it a
single buffer pointer and a length, not a place to put strings and a
place to put other values that refer to those strings..

It eases parsing anyway to split them into pairs, so people just
stopped mixing property types. Dealing with the same property type in
an array means no weird variable stepping over data.. to do it with a
phandle, string, int triple means copying the phandle out, moving a
buffer pointer, strlen on the string, and then knowing where to move
the buffer pointer to get the int at the end. Yuck. The whole
mixed-property thing worked great in Forth where you knew what you
pushed and when you popped it, it would still be a cell or a pointer
to a string. It's not great on the "OS" side.

I liked Mark's idea of using magic (preprocessor macros or "syntactic
sugar", either way) to generate pairs, but I relate to his later
comment; I really fear this starts to turn device tree authorship into
more of a chore than it already is. The more functionality you try and
shoe in to it, the more likely it will be to break (or more likely to
annoy people developing DTs). Preprocessing device trees is already
meaning the "source" for an FDT blob is no longer going to be much
like the output. I think that the point it turned a text file into a
compact, concise binary representation is where it should stop. That
text file needn't be full of macros and inheritance. It's starting to
become another programming language, and we're about 3 months away
from just saying "let's define our hardware platforms in Python and
build the interpreter right into Linux"

I'd rather hope that time was spent on maintaining the trees and
coming up with reasonable bindings and hardware descriptions based on
good knowledge of the history and reasoning behind a bunch of stuff
(education! I hope that the kernel summit list isn't on here for
nothing..) than constantly tweaking the infrastructure to compile
trees to output formats.. and changing those output formats.

~

What would help with defining FDTs is if someone could collect up all
the old and existing specifications and bindings and put them
somewhere and start referring to them in any bindings that reproduce
or change behaviors. The IEEE1275 spec is hard to get hold of, the
bindings are not always in "readable" (PostScript) formats. The
devicetree.org website has a bunch of broken links (and on the front
page, basically two different ePAPR specifications are linked) and
says it doesn't hope to redefine what is in the original (broken link)
or ePAPR (link to 2011 version) specs (footer: ePAPR link to the spec
from ~2008).

I got a couple bounces from devicetree-discuss at ozlabs with a note that
it had moved to devicetree at vger.kernel.org. That's fun, because
devicetree.org hasn't been updated to reflect it ;)

I may spend some time on the devicetree.org wiki but someone maybe
should do a review of the changes at some point. I am sure I saw a
tumbleweed go past when I typed it in to Firefox.

BTW if we're still looking for maintainers/'spirited individuals with
an opinion and more importantly, experience' on DTs, I'm up for it
(time willing).

-- Matt