DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Olof Johansson]

[I'm adding LKML and ksummit-discuss to this thread, since the ACPI/DT
discussions have been covered there and this overlaps some with that]

On Thu, Jul 25, 2013 at 7:38 AM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Thu, Jul 25, 2013 at 03:27:02PM +0100, Russell King - ARM Linux wrote:
>> Remember the stated assertion when DT was first added to the kernel: the
>> DT descriptions _will_ become a separately maintained project which is
>> independent of the kernel.  They will _not_ be kernel version specific.
>
> I'm looking forward to this.
>
> Question for the DT guys: what are the plans here? Are we going to get
> rid of the .dts files inside the kernel tree?

In the discussions we had in Dublin, a couple of options on how to
lock in bindings were discussed. I'm a little surprised that Grant
didn't cover them in his initial emails on the new maintainership
model, but maybe he wanted the new group to handle it. And they didn't
bring it up yet either. So I am. :-)

Until now, we have been working under the assumption that the bindings
are _NOT LOCKED_. I.e. they can change as needed, and we _ARE_
assuming that the device tree has to match the kernel. That has been a
good choice as people get up to speed on what is a good binding and
not, and has given us much-needed room to adjust things as needed.
That obviously has to change, but doing so needs to be done carefully.

It's likely that we still want to have a period in which a binding is
tentative and can be changed. Sometimes we don't know what we really
want until after we've used it a while, and sometimes we, like
everybody else, make mistakes on what is a good idea and not. The
alternative is to grind most new binding proposals to a halt while we
spend mind-numbing hours and hours on polishing every single aspect of
the binding to a perfect shine, since we can't go back and fix it.

So, there really seems to be a need for a layered approach, one in
which a binding "graduates" from being tentative to being locked in.
I'm refraining from using the terms "staging" and "stable" here, since
they have overloaded meaning in the kernel world that doesn't apply
here.

One problem that needs to be solved is obviously how a binding
graduates from tentative to locked. This work isn't going to be very
interesting to most people, I suspect. Think standards committee type
work.

A possible way to handle this is to have exactly that: A group of
people that essentially constitute the "standards committee" that meet
on a regular basis to review and approve new bindings. They should be
people not just doing ARM Linux work, but other stakeholders in these
bindings too. One of the things they should probably do is sift
through our current in-kernel bindings and select those who seem ready
to be locked in, review/discuss/decide upon that and once the decision
is made, that specific binding does become part of the static,
never-ever-change ABI of firmware-to-kernel interfaces.   That might
also be the time that the binding is moved from the kernel to a
separate repo, but that's a technicality that we'll let the DT
maintainers decide among themselves, IMHO.

I think that captures most of what we discussed in person. Others
might have changed their opinions since then, so I'm definitely not
claiming that any of this was an official decision made by anybody.
It's just one proposal on how to handle these things moving forward.


-Olof

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Olof Johansson]

[I'm adding LKML and ksummit-discuss to this thread, since the ACPI/DT
discussions have been covered there and this overlaps some with that]

On Thu, Jul 25, 2013 at 7:38 AM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Thu, Jul 25, 2013 at 03:27:02PM +0100, Russell King - ARM Linux wrote:
>> Remember the stated assertion when DT was first added to the kernel: the
>> DT descriptions _will_ become a separately maintained project which is
>> independent of the kernel.  They will _not_ be kernel version specific.
>
> I'm looking forward to this.
>
> Question for the DT guys: what are the plans here? Are we going to get
> rid of the .dts files inside the kernel tree?

In the discussions we had in Dublin, a couple of options on how to
lock in bindings were discussed. I'm a little surprised that Grant
didn't cover them in his initial emails on the new maintainership
model, but maybe he wanted the new group to handle it. And they didn't
bring it up yet either. So I am. :-)

Until now, we have been working under the assumption that the bindings
are _NOT LOCKED_. I.e. they can change as needed, and we _ARE_
assuming that the device tree has to match the kernel. That has been a
good choice as people get up to speed on what is a good binding and
not, and has given us much-needed room to adjust things as needed.
That obviously has to change, but doing so needs to be done carefully.

It's likely that we still want to have a period in which a binding is
tentative and can be changed. Sometimes we don't know what we really
want until after we've used it a while, and sometimes we, like
everybody else, make mistakes on what is a good idea and not. The
alternative is to grind most new binding proposals to a halt while we
spend mind-numbing hours and hours on polishing every single aspect of
the binding to a perfect shine, since we can't go back and fix it.

So, there really seems to be a need for a layered approach, one in
which a binding "graduates" from being tentative to being locked in.
I'm refraining from using the terms "staging" and "stable" here, since
they have overloaded meaning in the kernel world that doesn't apply
here.

One problem that needs to be solved is obviously how a binding
graduates from tentative to locked. This work isn't going to be very
interesting to most people, I suspect. Think standards committee type
work.

A possible way to handle this is to have exactly that: A group of
people that essentially constitute the "standards committee" that meet
on a regular basis to review and approve new bindings. They should be
people not just doing ARM Linux work, but other stakeholders in these
bindings too. One of the things they should probably do is sift
through our current in-kernel bindings and select those who seem ready
to be locked in, review/discuss/decide upon that and once the decision
is made, that specific binding does become part of the static,
never-ever-change ABI of firmware-to-kernel interfaces.   That might
also be the time that the binding is moved from the kernel to a
separate repo, but that's a technicality that we'll let the DT
maintainers decide among themselves, IMHO.

I think that captures most of what we discussed in person. Others
might have changed their opinions since then, so I'm definitely not
claiming that any of this was an official decision made by anybody.
It's just one proposal on how to handle these things moving forward.


-Olof

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Rutland]

Hi Olof,

[Adding the other new dt bindings maintainers to Cc]

On Thu, Jul 25, 2013 at 05:09:19PM +0100, Olof Johansson wrote:
> [I'm adding LKML and ksummit-discuss to this thread, since the ACPI/DT
> discussions have been covered there and this overlaps some with that]
> 
> On Thu, Jul 25, 2013 at 7:38 AM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > On Thu, Jul 25, 2013 at 03:27:02PM +0100, Russell King - ARM Linux wrote:
> >> Remember the stated assertion when DT was first added to the kernel: the
> >> DT descriptions _will_ become a separately maintained project which is
> >> independent of the kernel.  They will _not_ be kernel version specific.
> >
> > I'm looking forward to this.
> >
> > Question for the DT guys: what are the plans here? Are we going to get
> > rid of the .dts files inside the kernel tree?
> 
> In the discussions we had in Dublin, a couple of options on how to
> lock in bindings were discussed. I'm a little surprised that Grant
> didn't cover them in his initial emails on the new maintainership
> model, but maybe he wanted the new group to handle it. And they didn't
> bring it up yet either. So I am. :-)

Apologies for the delay on this, thanks for bringing it up. We certainly
need a kick regarding this.

> 
> Until now, we have been working under the assumption that the bindings
> are _NOT LOCKED_. I.e. they can change as needed, and we _ARE_
> assuming that the device tree has to match the kernel. That has been a
> good choice as people get up to speed on what is a good binding and
> not, and has given us much-needed room to adjust things as needed.
> That obviously has to change, but doing so needs to be done carefully.

While it's been the most natural way for us kernel folk to build the
necessary infrastructure and adapt to the new way of doing things, as
Dave and Russell pointed out in the original thread [1], we've blindly
walked into building a horrible ABI for ourselves. With KVM we've also
exposed this ABI to userspace...

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2013-July/186114.html

> 
> It's likely that we still want to have a period in which a binding is
> tentative and can be changed. Sometimes we don't know what we really
> want until after we've used it a while, and sometimes we, like
> everybody else, make mistakes on what is a good idea and not. The
> alternative is to grind most new binding proposals to a halt while we
> spend mind-numbing hours and hours on polishing every single aspect of
> the binding to a perfect shine, since we can't go back and fix it.
> 
> So, there really seems to be a need for a layered approach, one in
> which a binding "graduates" from being tentative to being locked in.
> I'm refraining from using the terms "staging" and "stable" here, since
> they have overloaded meaning in the kernel world that doesn't apply
> here.

I'm not sure how realistic it is to have drivers in the kernel using
unstable bindings, and expect people to not rely on them, but I don't
have a better option to give. We need a big fat warning that an unstable
binding is in use.

> 
> One problem that needs to be solved is obviously how a binding
> graduates from tentative to locked. This work isn't going to be very
> interesting to most people, I suspect. Think standards committee type
> work.
> 
> A possible way to handle this is to have exactly that: A group of
> people that essentially constitute the "standards committee" that meet
> on a regular basis to review and approve new bindings. They should be
> people not just doing ARM Linux work, but other stakeholders in these
> bindings too. One of the things they should probably do is sift
> through our current in-kernel bindings and select those who seem ready
> to be locked in, review/discuss/decide upon that and once the decision
> is made, that specific binding does become part of the static,
> never-ever-change ABI of firmware-to-kernel interfaces.   That might
> also be the time that the binding is moved from the kernel to a
> separate repo, but that's a technicality that we'll let the DT
> maintainers decide among themselves, IMHO.

We're going to need input from other OSs too, or the bindings will
remain Linux-specific regardless of how far away the bindings and dts
repo(s) is/are.

I'd be happy with a Linux-specific unstable binding and an accepted,
stable binding documented externally, that gives people the most freedom
to prototype while keeping stable and unstable bindings clearly
separated.

> 
> I think that captures most of what we discussed in person. Others
> might have changed their opinions since then, so I'm definitely not
> claiming that any of this was an official decision made by anybody.
> It's just one proposal on how to handle these things moving forward.

I think you've got the gist of it. 

How about the following:

* We reach out to other users of devicetree, and attempt to form the
  standards/review commitee.

* At v3.11, we pull (a copy of?) all binding documents out of the
  kernel, into an 'unstable/linux bindings' repo. 

* The minimal set of 'stable' bindings (e.g. those having been used for
  years without modification by powerpc) will be moved to a stable repo
  after consensus from the committee, and developers outside of the
  committee using said bindings. Hopefully there will be no showstoppers
  here. These should contain no Linux-specific information.

  Those bindings that have been modified recently or are in areas where
  there is currently heavy churn will not be marked stable.

* We attempt to graduate bindings to stable, once the appropriate
  infrastructure is in place to prove the concept, preferably with
  bindings for similar devices from different vendors.

Thoughts?

Thanks,
Mark.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Rutland]

Hi Olof,

[Adding the other new dt bindings maintainers to Cc]

On Thu, Jul 25, 2013 at 05:09:19PM +0100, Olof Johansson wrote:
> [I'm adding LKML and ksummit-discuss to this thread, since the ACPI/DT
> discussions have been covered there and this overlaps some with that]
> 
> On Thu, Jul 25, 2013 at 7:38 AM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > On Thu, Jul 25, 2013 at 03:27:02PM +0100, Russell King - ARM Linux wrote:
> >> Remember the stated assertion when DT was first added to the kernel: the
> >> DT descriptions _will_ become a separately maintained project which is
> >> independent of the kernel.  They will _not_ be kernel version specific.
> >
> > I'm looking forward to this.
> >
> > Question for the DT guys: what are the plans here? Are we going to get
> > rid of the .dts files inside the kernel tree?
> 
> In the discussions we had in Dublin, a couple of options on how to
> lock in bindings were discussed. I'm a little surprised that Grant
> didn't cover them in his initial emails on the new maintainership
> model, but maybe he wanted the new group to handle it. And they didn't
> bring it up yet either. So I am. :-)

Apologies for the delay on this, thanks for bringing it up. We certainly
need a kick regarding this.

> 
> Until now, we have been working under the assumption that the bindings
> are _NOT LOCKED_. I.e. they can change as needed, and we _ARE_
> assuming that the device tree has to match the kernel. That has been a
> good choice as people get up to speed on what is a good binding and
> not, and has given us much-needed room to adjust things as needed.
> That obviously has to change, but doing so needs to be done carefully.

While it's been the most natural way for us kernel folk to build the
necessary infrastructure and adapt to the new way of doing things, as
Dave and Russell pointed out in the original thread [1], we've blindly
walked into building a horrible ABI for ourselves. With KVM we've also
exposed this ABI to userspace...

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2013-July/186114.html

> 
> It's likely that we still want to have a period in which a binding is
> tentative and can be changed. Sometimes we don't know what we really
> want until after we've used it a while, and sometimes we, like
> everybody else, make mistakes on what is a good idea and not. The
> alternative is to grind most new binding proposals to a halt while we
> spend mind-numbing hours and hours on polishing every single aspect of
> the binding to a perfect shine, since we can't go back and fix it.
> 
> So, there really seems to be a need for a layered approach, one in
> which a binding "graduates" from being tentative to being locked in.
> I'm refraining from using the terms "staging" and "stable" here, since
> they have overloaded meaning in the kernel world that doesn't apply
> here.

I'm not sure how realistic it is to have drivers in the kernel using
unstable bindings, and expect people to not rely on them, but I don't
have a better option to give. We need a big fat warning that an unstable
binding is in use.

> 
> One problem that needs to be solved is obviously how a binding
> graduates from tentative to locked. This work isn't going to be very
> interesting to most people, I suspect. Think standards committee type
> work.
> 
> A possible way to handle this is to have exactly that: A group of
> people that essentially constitute the "standards committee" that meet
> on a regular basis to review and approve new bindings. They should be
> people not just doing ARM Linux work, but other stakeholders in these
> bindings too. One of the things they should probably do is sift
> through our current in-kernel bindings and select those who seem ready
> to be locked in, review/discuss/decide upon that and once the decision
> is made, that specific binding does become part of the static,
> never-ever-change ABI of firmware-to-kernel interfaces.   That might
> also be the time that the binding is moved from the kernel to a
> separate repo, but that's a technicality that we'll let the DT
> maintainers decide among themselves, IMHO.

We're going to need input from other OSs too, or the bindings will
remain Linux-specific regardless of how far away the bindings and dts
repo(s) is/are.

I'd be happy with a Linux-specific unstable binding and an accepted,
stable binding documented externally, that gives people the most freedom
to prototype while keeping stable and unstable bindings clearly
separated.

> 
> I think that captures most of what we discussed in person. Others
> might have changed their opinions since then, so I'm definitely not
> claiming that any of this was an official decision made by anybody.
> It's just one proposal on how to handle these things moving forward.

I think you've got the gist of it. 

How about the following:

* We reach out to other users of devicetree, and attempt to form the
  standards/review commitee.

* At v3.11, we pull (a copy of?) all binding documents out of the
  kernel, into an 'unstable/linux bindings' repo. 

* The minimal set of 'stable' bindings (e.g. those having been used for
  years without modification by powerpc) will be moved to a stable repo
  after consensus from the committee, and developers outside of the
  committee using said bindings. Hopefully there will be no showstoppers
  here. These should contain no Linux-specific information.

  Those bindings that have been modified recently or are in areas where
  there is currently heavy churn will not be marked stable.

* We attempt to graduate bindings to stable, once the appropriate
  infrastructure is in place to prove the concept, preferably with
  bindings for similar devices from different vendors.

Thoughts?

Thanks,
Mark.

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 09:09 AM, Olof Johansson wrote:
...
> One problem that needs to be solved is obviously how a binding
> graduates from tentative to locked. This work isn't going to be very
> interesting to most people, I suspect. Think standards committee type
> work.

I assume that's part of what being a maintainer of the DT bindings tree
means?

+1 on everything else you said. I've been traveling this week so haven't
had much time for email, but if I hadn't, I hope I would have said
something similar:-)

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 09:09 AM, Olof Johansson wrote:
...
> One problem that needs to be solved is obviously how a binding
> graduates from tentative to locked. This work isn't going to be very
> interesting to most people, I suspect. Think standards committee type
> work.

I assume that's part of what being a maintainer of the DT bindings tree
means?

+1 on everything else you said. I've been traveling this week so haven't
had much time for email, but if I hadn't, I hope I would have said
something similar:-)

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 10:57 AM, Mark Rutland wrote:
> On Thu, Jul 25, 2013 at 05:09:19PM +0100, Olof Johansson wrote:
...
>> So, there really seems to be a need for a layered approach, one in
>> which a binding "graduates" from being tentative to being locked in.
>> I'm refraining from using the terms "staging" and "stable" here, since
>> they have overloaded meaning in the kernel world that doesn't apply
>> here.
> 
> I'm not sure how realistic it is to have drivers in the kernel using
> unstable bindings, and expect people to not rely on them, but I don't
> have a better option to give. We need a big fat warning that an unstable
> binding is in use.

I don't think having people "rely" on the bindings is the issue so much
as the awareness that if they do, there will be compatibility issues for
unstable bindings.

>> One problem that needs to be solved is obviously how a binding
>> graduates from tentative to locked. This work isn't going to be very
>> interesting to most people, I suspect. Think standards committee type
>> work.
>>
>> A possible way to handle this is to have exactly that: A group of
>> people that essentially constitute the "standards committee" that meet
>> on a regular basis to review and approve new bindings. They should be
>> people not just doing ARM Linux work, but other stakeholders in these
>> bindings too. One of the things they should probably do is sift
>> through our current in-kernel bindings and select those who seem ready
>> to be locked in, review/discuss/decide upon that and once the decision
>> is made, that specific binding does become part of the static,
>> never-ever-change ABI of firmware-to-kernel interfaces.   That might
>> also be the time that the binding is moved from the kernel to a
>> separate repo, but that's a technicality that we'll let the DT
>> maintainers decide among themselves, IMHO.
> 
> We're going to need input from other OSs too, or the bindings will
> remain Linux-specific regardless of how far away the bindings and dts
> repo(s) is/are.

And bootloaders too. Some U-Boot platforms are starting to use DT for
exactly the same reason that the kernel does; to allow a single
bootloader binary to run on a variety of different boards, with all
configuration coming from DT.

On another related topic, something that may be useful for the DT
bindings reviewer team is a basic checklist for new DT bindings.
Something similar to Fedora's package review checklist. Perhaps also
(yet another?) document on a bit of DT philosophy. If this sounds
useful, I could try and take a stab at some basic initial version.

We also need to decide (or just document) exactly what "describes the
HW" means; see the thread on thermal limits, and consider the extension
of describing hard/absolute thermal limits to describing use-cased base
thermal profiles using the same schema, or not allowing that.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 10:57 AM, Mark Rutland wrote:
> On Thu, Jul 25, 2013 at 05:09:19PM +0100, Olof Johansson wrote:
...
>> So, there really seems to be a need for a layered approach, one in
>> which a binding "graduates" from being tentative to being locked in.
>> I'm refraining from using the terms "staging" and "stable" here, since
>> they have overloaded meaning in the kernel world that doesn't apply
>> here.
> 
> I'm not sure how realistic it is to have drivers in the kernel using
> unstable bindings, and expect people to not rely on them, but I don't
> have a better option to give. We need a big fat warning that an unstable
> binding is in use.

I don't think having people "rely" on the bindings is the issue so much
as the awareness that if they do, there will be compatibility issues for
unstable bindings.

>> One problem that needs to be solved is obviously how a binding
>> graduates from tentative to locked. This work isn't going to be very
>> interesting to most people, I suspect. Think standards committee type
>> work.
>>
>> A possible way to handle this is to have exactly that: A group of
>> people that essentially constitute the "standards committee" that meet
>> on a regular basis to review and approve new bindings. They should be
>> people not just doing ARM Linux work, but other stakeholders in these
>> bindings too. One of the things they should probably do is sift
>> through our current in-kernel bindings and select those who seem ready
>> to be locked in, review/discuss/decide upon that and once the decision
>> is made, that specific binding does become part of the static,
>> never-ever-change ABI of firmware-to-kernel interfaces.   That might
>> also be the time that the binding is moved from the kernel to a
>> separate repo, but that's a technicality that we'll let the DT
>> maintainers decide among themselves, IMHO.
> 
> We're going to need input from other OSs too, or the bindings will
> remain Linux-specific regardless of how far away the bindings and dts
> repo(s) is/are.

And bootloaders too. Some U-Boot platforms are starting to use DT for
exactly the same reason that the kernel does; to allow a single
bootloader binary to run on a variety of different boards, with all
configuration coming from DT.

On another related topic, something that may be useful for the DT
bindings reviewer team is a basic checklist for new DT bindings.
Something similar to Fedora's package review checklist. Perhaps also
(yet another?) document on a bit of DT philosophy. If this sounds
useful, I could try and take a stab at some basic initial version.

We also need to decide (or just document) exactly what "describes the
HW" means; see the thread on thermal limits, and consider the extension
of describing hard/absolute thermal limits to describing use-cased base
thermal profiles using the same schema, or not allowing that.

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Olof Johansson]

On Thu, Jul 25, 2013 at 11:05 AM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 07/25/2013 10:57 AM, Mark Rutland wrote:
>> On Thu, Jul 25, 2013 at 05:09:19PM +0100, Olof Johansson wrote:
> ...
>>> So, there really seems to be a need for a layered approach, one in
>>> which a binding "graduates" from being tentative to being locked in.
>>> I'm refraining from using the terms "staging" and "stable" here, since
>>> they have overloaded meaning in the kernel world that doesn't apply
>>> here.
>>
>> I'm not sure how realistic it is to have drivers in the kernel using
>> unstable bindings, and expect people to not rely on them, but I don't
>> have a better option to give. We need a big fat warning that an unstable
>> binding is in use.
>
> I don't think having people "rely" on the bindings is the issue so much
> as the awareness that if they do, there will be compatibility issues for
> unstable bindings.

If we continue doing what we do today, where we have (at least part
of) the device trees hosted with the kernel sources, then we can
introduce tentative changes to the device trees at the same time as
the code, so having the drivers rely on that seems appropriate.

Of course, things should not be static in that state, and have to move
out of it reasonably quickly.

>>> One problem that needs to be solved is obviously how a binding
>>> graduates from tentative to locked. This work isn't going to be very
>>> interesting to most people, I suspect. Think standards committee type
>>> work.
>>>
>>> A possible way to handle this is to have exactly that: A group of
>>> people that essentially constitute the "standards committee" that meet
>>> on a regular basis to review and approve new bindings. They should be
>>> people not just doing ARM Linux work, but other stakeholders in these
>>> bindings too. One of the things they should probably do is sift
>>> through our current in-kernel bindings and select those who seem ready
>>> to be locked in, review/discuss/decide upon that and once the decision
>>> is made, that specific binding does become part of the static,
>>> never-ever-change ABI of firmware-to-kernel interfaces.   That might
>>> also be the time that the binding is moved from the kernel to a
>>> separate repo, but that's a technicality that we'll let the DT
>>> maintainers decide among themselves, IMHO.
>>
>> We're going to need input from other OSs too, or the bindings will
>> remain Linux-specific regardless of how far away the bindings and dts
>> repo(s) is/are.
>
> And bootloaders too. Some U-Boot platforms are starting to use DT for
> exactly the same reason that the kernel does; to allow a single
> bootloader binary to run on a variety of different boards, with all
> configuration coming from DT.

Not only that, but in some cases u-boot might want to reach into the
device-tree of the launching kernel and tweak settings based on board
information. It already does this for things like memory and command
line, but there could be other cases where it wants to do so as well.
If things move around, or bindings change, this will obviously cause
problems. (I've already seen this with the Chrome OS u-boot vs
upstream kernels, partly due to a bug in the local fork of u-boot, but
still).

> On another related topic, something that may be useful for the DT
> bindings reviewer team is a basic checklist for new DT bindings.
> Something similar to Fedora's package review checklist. Perhaps also
> (yet another?) document on a bit of DT philosophy. If this sounds
> useful, I could try and take a stab at some basic initial version.

Sounds reasonable. Starting with one of the existing ones instead of
from scratch is a reasonable approach. A checklist and a best
practices doc would come a long way.

> We also need to decide (or just document) exactly what "describes the
> HW" means; see the thread on thermal limits, and consider the extension
> of describing hard/absolute thermal limits to describing use-cased base
> thermal profiles using the same schema, or not allowing that.

Yes indeed. A basic binding need just specify what the specific
hardware IP is, if the rest of the configuration of the IP can be
determined at runtime through other means (i.e. by autoprobing). It's
stuff beyond that that gets very complicated.

To talk semi-specifics: What about USB PHY tunings for a specific
board, where each of the three USB ports and their registers have
slightly different layout? Sure, we can take 10 properties to describe
each tunable field in each register, but at the end it will just be
used to craft the contents blindly, so we might just stuff the 32-bit
register value as a property instead. And in other cases we might
indeed want to describe each property independently. Determining what
is appropriate when is one of the most difficult parts of the review
workflow.


-Olof

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Olof Johansson]

On Thu, Jul 25, 2013 at 11:05 AM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 07/25/2013 10:57 AM, Mark Rutland wrote:
>> On Thu, Jul 25, 2013 at 05:09:19PM +0100, Olof Johansson wrote:
> ...
>>> So, there really seems to be a need for a layered approach, one in
>>> which a binding "graduates" from being tentative to being locked in.
>>> I'm refraining from using the terms "staging" and "stable" here, since
>>> they have overloaded meaning in the kernel world that doesn't apply
>>> here.
>>
>> I'm not sure how realistic it is to have drivers in the kernel using
>> unstable bindings, and expect people to not rely on them, but I don't
>> have a better option to give. We need a big fat warning that an unstable
>> binding is in use.
>
> I don't think having people "rely" on the bindings is the issue so much
> as the awareness that if they do, there will be compatibility issues for
> unstable bindings.

If we continue doing what we do today, where we have (at least part
of) the device trees hosted with the kernel sources, then we can
introduce tentative changes to the device trees at the same time as
the code, so having the drivers rely on that seems appropriate.

Of course, things should not be static in that state, and have to move
out of it reasonably quickly.

>>> One problem that needs to be solved is obviously how a binding
>>> graduates from tentative to locked. This work isn't going to be very
>>> interesting to most people, I suspect. Think standards committee type
>>> work.
>>>
>>> A possible way to handle this is to have exactly that: A group of
>>> people that essentially constitute the "standards committee" that meet
>>> on a regular basis to review and approve new bindings. They should be
>>> people not just doing ARM Linux work, but other stakeholders in these
>>> bindings too. One of the things they should probably do is sift
>>> through our current in-kernel bindings and select those who seem ready
>>> to be locked in, review/discuss/decide upon that and once the decision
>>> is made, that specific binding does become part of the static,
>>> never-ever-change ABI of firmware-to-kernel interfaces.   That might
>>> also be the time that the binding is moved from the kernel to a
>>> separate repo, but that's a technicality that we'll let the DT
>>> maintainers decide among themselves, IMHO.
>>
>> We're going to need input from other OSs too, or the bindings will
>> remain Linux-specific regardless of how far away the bindings and dts
>> repo(s) is/are.
>
> And bootloaders too. Some U-Boot platforms are starting to use DT for
> exactly the same reason that the kernel does; to allow a single
> bootloader binary to run on a variety of different boards, with all
> configuration coming from DT.

Not only that, but in some cases u-boot might want to reach into the
device-tree of the launching kernel and tweak settings based on board
information. It already does this for things like memory and command
line, but there could be other cases where it wants to do so as well.
If things move around, or bindings change, this will obviously cause
problems. (I've already seen this with the Chrome OS u-boot vs
upstream kernels, partly due to a bug in the local fork of u-boot, but
still).

> On another related topic, something that may be useful for the DT
> bindings reviewer team is a basic checklist for new DT bindings.
> Something similar to Fedora's package review checklist. Perhaps also
> (yet another?) document on a bit of DT philosophy. If this sounds
> useful, I could try and take a stab at some basic initial version.

Sounds reasonable. Starting with one of the existing ones instead of
from scratch is a reasonable approach. A checklist and a best
practices doc would come a long way.

> We also need to decide (or just document) exactly what "describes the
> HW" means; see the thread on thermal limits, and consider the extension
> of describing hard/absolute thermal limits to describing use-cased base
> thermal profiles using the same schema, or not allowing that.

Yes indeed. A basic binding need just specify what the specific
hardware IP is, if the rest of the configuration of the IP can be
determined at runtime through other means (i.e. by autoprobing). It's
stuff beyond that that gets very complicated.

To talk semi-specifics: What about USB PHY tunings for a specific
board, where each of the three USB ports and their registers have
slightly different layout? Sure, we can take 10 properties to describe
each tunable field in each register, but at the end it will just be
used to craft the contents blindly, so we might just stuff the 32-bit
register value as a property instead. And in other cases we might
indeed want to describe each property independently. Determining what
is appropriate when is one of the most difficult parts of the review
workflow.


-Olof

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Rutland]

On Thu, Jul 25, 2013 at 07:05:48PM +0100, Stephen Warren wrote:
> On 07/25/2013 10:57 AM, Mark Rutland wrote:
> > On Thu, Jul 25, 2013 at 05:09:19PM +0100, Olof Johansson wrote:
> ...
> >> So, there really seems to be a need for a layered approach, one in
> >> which a binding "graduates" from being tentative to being locked in.
> >> I'm refraining from using the terms "staging" and "stable" here, since
> >> they have overloaded meaning in the kernel world that doesn't apply
> >> here.
> > 
> > I'm not sure how realistic it is to have drivers in the kernel using
> > unstable bindings, and expect people to not rely on them, but I don't
> > have a better option to give. We need a big fat warning that an unstable
> > binding is in use.
> 
> I don't think having people "rely" on the bindings is the issue so much
> as the awareness that if they do, there will be compatibility issues for
> unstable bindings.

As long as we can make sufficiently clear that trying to use an unstable
binding is going to be *very* painful, and not necessarily supported.

> 
> >> One problem that needs to be solved is obviously how a binding
> >> graduates from tentative to locked. This work isn't going to be very
> >> interesting to most people, I suspect. Think standards committee type
> >> work.
> >>
> >> A possible way to handle this is to have exactly that: A group of
> >> people that essentially constitute the "standards committee" that meet
> >> on a regular basis to review and approve new bindings. They should be
> >> people not just doing ARM Linux work, but other stakeholders in these
> >> bindings too. One of the things they should probably do is sift
> >> through our current in-kernel bindings and select those who seem ready
> >> to be locked in, review/discuss/decide upon that and once the decision
> >> is made, that specific binding does become part of the static,
> >> never-ever-change ABI of firmware-to-kernel interfaces.   That might
> >> also be the time that the binding is moved from the kernel to a
> >> separate repo, but that's a technicality that we'll let the DT
> >> maintainers decide among themselves, IMHO.
> > 
> > We're going to need input from other OSs too, or the bindings will
> > remain Linux-specific regardless of how far away the bindings and dts
> > repo(s) is/are.
> 
> And bootloaders too. Some U-Boot platforms are starting to use DT for
> exactly the same reason that the kernel does; to allow a single
> bootloader binary to run on a variety of different boards, with all
> configuration coming from DT.

Ah, I'd not considered that. We certainly need to include them in the
discussion.

> 
> On another related topic, something that may be useful for the DT
> bindings reviewer team is a basic checklist for new DT bindings.
> Something similar to Fedora's package review checklist. Perhaps also
> (yet another?) document on a bit of DT philosophy. If this sounds
> useful, I could try and take a stab at some basic initial version.

That does indeed sound useful. Please do :)

> 
> We also need to decide (or just document) exactly what "describes the
> HW" means; see the thread on thermal limits, and consider the extension
> of describing hard/absolute thermal limits to describing use-cased base
> thermal profiles using the same schema, or not allowing that.
> 

Certainly. This is an unfortunate gray area.

Thanks,
Mark.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Rutland]

On Thu, Jul 25, 2013 at 07:05:48PM +0100, Stephen Warren wrote:
> On 07/25/2013 10:57 AM, Mark Rutland wrote:
> > On Thu, Jul 25, 2013 at 05:09:19PM +0100, Olof Johansson wrote:
> ...
> >> So, there really seems to be a need for a layered approach, one in
> >> which a binding "graduates" from being tentative to being locked in.
> >> I'm refraining from using the terms "staging" and "stable" here, since
> >> they have overloaded meaning in the kernel world that doesn't apply
> >> here.
> > 
> > I'm not sure how realistic it is to have drivers in the kernel using
> > unstable bindings, and expect people to not rely on them, but I don't
> > have a better option to give. We need a big fat warning that an unstable
> > binding is in use.
> 
> I don't think having people "rely" on the bindings is the issue so much
> as the awareness that if they do, there will be compatibility issues for
> unstable bindings.

As long as we can make sufficiently clear that trying to use an unstable
binding is going to be *very* painful, and not necessarily supported.

> 
> >> One problem that needs to be solved is obviously how a binding
> >> graduates from tentative to locked. This work isn't going to be very
> >> interesting to most people, I suspect. Think standards committee type
> >> work.
> >>
> >> A possible way to handle this is to have exactly that: A group of
> >> people that essentially constitute the "standards committee" that meet
> >> on a regular basis to review and approve new bindings. They should be
> >> people not just doing ARM Linux work, but other stakeholders in these
> >> bindings too. One of the things they should probably do is sift
> >> through our current in-kernel bindings and select those who seem ready
> >> to be locked in, review/discuss/decide upon that and once the decision
> >> is made, that specific binding does become part of the static,
> >> never-ever-change ABI of firmware-to-kernel interfaces.   That might
> >> also be the time that the binding is moved from the kernel to a
> >> separate repo, but that's a technicality that we'll let the DT
> >> maintainers decide among themselves, IMHO.
> > 
> > We're going to need input from other OSs too, or the bindings will
> > remain Linux-specific regardless of how far away the bindings and dts
> > repo(s) is/are.
> 
> And bootloaders too. Some U-Boot platforms are starting to use DT for
> exactly the same reason that the kernel does; to allow a single
> bootloader binary to run on a variety of different boards, with all
> configuration coming from DT.

Ah, I'd not considered that. We certainly need to include them in the
discussion.

> 
> On another related topic, something that may be useful for the DT
> bindings reviewer team is a basic checklist for new DT bindings.
> Something similar to Fedora's package review checklist. Perhaps also
> (yet another?) document on a bit of DT philosophy. If this sounds
> useful, I could try and take a stab at some basic initial version.

That does indeed sound useful. Please do :)

> 
> We also need to decide (or just document) exactly what "describes the
> HW" means; see the thread on thermal limits, and consider the extension
> of describing hard/absolute thermal limits to describing use-cased base
> thermal profiles using the same schema, or not allowing that.
> 

Certainly. This is an unfortunate gray area.

Thanks,
Mark.

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Thu, Jul 25, 2013 at 07:29:20PM +0100, Mark Rutland wrote:
> On Thu, Jul 25, 2013 at 07:05:48PM +0100, Stephen Warren wrote:
> > 
> > I don't think having people "rely" on the bindings is the issue so much
> > as the awareness that if they do, there will be compatibility issues for
> > unstable bindings.
> 
> As long as we can make sufficiently clear that trying to use an unstable
> binding is going to be *very* painful, and not necessarily supported.

Oh, man.

The introduction of DT into ARM Linux was supposed to make everyone's
life sooo much easier. Of course, based on experience with powerpc, I
never believed it*, but still I would expect to hear that the DT
bindings are, well, a *binding* contract between the board developer,
boot loader, and the kernel.

Once it is working with a particular kernel, a DT board description
file should continue to work indefinitely with newer kernels. Anything
less is a regression, pure and simple.

If you go around changing the bindings willy nilly, then what is point
of having DT at all?

Thanks,
Richard

* http://lists.infradead.org/pipermail/linux-arm-kernel/2011-April/046963.html
  http://lists.infradead.org/pipermail/linux-arm-kernel/2011-May/050255.html
  http://lists.infradead.org/pipermail/linux-arm-kernel/2011-May/050256.html
  http://lists.infradead.org/pipermail/linux-arm-kernel/2011-May/050264.html

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Thu, Jul 25, 2013 at 07:29:20PM +0100, Mark Rutland wrote:
> On Thu, Jul 25, 2013 at 07:05:48PM +0100, Stephen Warren wrote:
> > 
> > I don't think having people "rely" on the bindings is the issue so much
> > as the awareness that if they do, there will be compatibility issues for
> > unstable bindings.
> 
> As long as we can make sufficiently clear that trying to use an unstable
> binding is going to be *very* painful, and not necessarily supported.

Oh, man.

The introduction of DT into ARM Linux was supposed to make everyone's
life sooo much easier. Of course, based on experience with powerpc, I
never believed it*, but still I would expect to hear that the DT
bindings are, well, a *binding* contract between the board developer,
boot loader, and the kernel.

Once it is working with a particular kernel, a DT board description
file should continue to work indefinitely with newer kernels. Anything
less is a regression, pure and simple.

If you go around changing the bindings willy nilly, then what is point
of having DT at all?

Thanks,
Richard

* http://lists.infradead.org/pipermail/linux-arm-kernel/2011-April/046963.html
  http://lists.infradead.org/pipermail/linux-arm-kernel/2011-May/050255.html
  http://lists.infradead.org/pipermail/linux-arm-kernel/2011-May/050256.html
  http://lists.infradead.org/pipermail/linux-arm-kernel/2011-May/050264.html

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 11:25 AM, Olof Johansson wrote:
> On Thu, Jul 25, 2013 at 11:05 AM, Stephen Warren <swarren@wwwdotorg.org> wrote:
...
>> On another related topic, something that may be useful for the DT
>> bindings reviewer team is a basic checklist for new DT bindings.
>> Something similar to Fedora's package review checklist. Perhaps also
>> (yet another?) document on a bit of DT philosophy. If this sounds
>> useful, I could try and take a stab at some basic initial version.
> 
> Sounds reasonable. Starting with one of the existing ones instead of
> from scratch is a reasonable approach. A checklist and a best
> practices doc would come a long way.

Do you have a link to an existing check-list? I know there's plenty of
best practices information out there to build on.

>> We also need to decide (or just document) exactly what "describes the
>> HW" means; see the thread on thermal limits, and consider the extension
>> of describing hard/absolute thermal limits to describing use-cased base
>> thermal profiles using the same schema, or not allowing that.
> 
> Yes indeed. A basic binding need just specify what the specific
> hardware IP is, if the rest of the configuration of the IP can be
> determined at runtime through other means (i.e. by autoprobing). It's
> stuff beyond that that gets very complicated.
> 
> To talk semi-specifics: What about USB PHY tunings for a specific
> board,

I was thinking more about the slightly blurry line between representing
HW and representing policy, but the example you gave certainly needs
consideration too.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 11:25 AM, Olof Johansson wrote:
> On Thu, Jul 25, 2013 at 11:05 AM, Stephen Warren <swarren@wwwdotorg.org> wrote:
...
>> On another related topic, something that may be useful for the DT
>> bindings reviewer team is a basic checklist for new DT bindings.
>> Something similar to Fedora's package review checklist. Perhaps also
>> (yet another?) document on a bit of DT philosophy. If this sounds
>> useful, I could try and take a stab at some basic initial version.
> 
> Sounds reasonable. Starting with one of the existing ones instead of
> from scratch is a reasonable approach. A checklist and a best
> practices doc would come a long way.

Do you have a link to an existing check-list? I know there's plenty of
best practices information out there to build on.

>> We also need to decide (or just document) exactly what "describes the
>> HW" means; see the thread on thermal limits, and consider the extension
>> of describing hard/absolute thermal limits to describing use-cased base
>> thermal profiles using the same schema, or not allowing that.
> 
> Yes indeed. A basic binding need just specify what the specific
> hardware IP is, if the rest of the configuration of the IP can be
> determined at runtime through other means (i.e. by autoprobing). It's
> stuff beyond that that gets very complicated.
> 
> To talk semi-specifics: What about USB PHY tunings for a specific
> board,

I was thinking more about the slightly blurry line between representing
HW and representing policy, but the example you gave certainly needs
consideration too.

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 11:48 AM, Richard Cochran wrote:
> On Thu, Jul 25, 2013 at 07:29:20PM +0100, Mark Rutland wrote:
>> On Thu, Jul 25, 2013 at 07:05:48PM +0100, Stephen Warren wrote:
>>>
>>> I don't think having people "rely" on the bindings is the issue so much
>>> as the awareness that if they do, there will be compatibility issues for
>>> unstable bindings.
>>
>> As long as we can make sufficiently clear that trying to use an unstable
>> binding is going to be *very* painful, and not necessarily supported.
> 
> Oh, man.
> 
> The introduction of DT into ARM Linux was supposed to make everyone's
> life sooo much easier. Of course, based on experience with powerpc, I
> never believed it*, but still I would expect to hear that the DT
> bindings are, well, a *binding* contract between the board developer,
> boot loader, and the kernel.
> 
> Once it is working with a particular kernel, a DT board description
> file should continue to work indefinitely with newer kernels. Anything
> less is a regression, pure and simple.
> 
> If you go around changing the bindings willy nilly, then what is point
> of having DT at all?

That's exactly why we're starting to think about which bindings should
be considered stable and immutable, and when that should happen. As Olof
pointed out, we haven't fully enforced that yet. Preferably bindings
will be marked stable very fast, but mistakes are always going to happen
in early development. ABIs are very hard.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 11:48 AM, Richard Cochran wrote:
> On Thu, Jul 25, 2013 at 07:29:20PM +0100, Mark Rutland wrote:
>> On Thu, Jul 25, 2013 at 07:05:48PM +0100, Stephen Warren wrote:
>>>
>>> I don't think having people "rely" on the bindings is the issue so much
>>> as the awareness that if they do, there will be compatibility issues for
>>> unstable bindings.
>>
>> As long as we can make sufficiently clear that trying to use an unstable
>> binding is going to be *very* painful, and not necessarily supported.
> 
> Oh, man.
> 
> The introduction of DT into ARM Linux was supposed to make everyone's
> life sooo much easier. Of course, based on experience with powerpc, I
> never believed it*, but still I would expect to hear that the DT
> bindings are, well, a *binding* contract between the board developer,
> boot loader, and the kernel.
> 
> Once it is working with a particular kernel, a DT board description
> file should continue to work indefinitely with newer kernels. Anything
> less is a regression, pure and simple.
> 
> If you go around changing the bindings willy nilly, then what is point
> of having DT at all?

That's exactly why we're starting to think about which bindings should
be considered stable and immutable, and when that should happen. As Olof
pointed out, we haven't fully enforced that yet. Preferably bindings
will be marked stable very fast, but mistakes are always going to happen
in early development. ABIs are very hard.

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
> [I'm adding LKML and ksummit-discuss to this thread, since the ACPI/DT
> discussions have been covered there and this overlaps some with that]
>
> On Thu, Jul 25, 2013 at 7:38 AM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
>> On Thu, Jul 25, 2013 at 03:27:02PM +0100, Russell King - ARM Linux wrote:
>>> Remember the stated assertion when DT was first added to the kernel: the
>>> DT descriptions _will_ become a separately maintained project which is
>>> independent of the kernel.  They will _not_ be kernel version specific.
>>
>> I'm looking forward to this.
>>
>> Question for the DT guys: what are the plans here? Are we going to get
>> rid of the .dts files inside the kernel tree?
>
> In the discussions we had in Dublin, a couple of options on how to
> lock in bindings were discussed. I'm a little surprised that Grant
> didn't cover them in his initial emails on the new maintainership
> model, but maybe he wanted the new group to handle it. And they didn't
> bring it up yet either. So I am. :-)
>
> Until now, we have been working under the assumption that the bindings
> are _NOT LOCKED_. I.e. they can change as needed, and we _ARE_
> assuming that the device tree has to match the kernel. That has been a
> good choice as people get up to speed on what is a good binding and
> not, and has given us much-needed room to adjust things as needed.
> That obviously has to change, but doing so needs to be done carefully.

I would argue that little has gone in (if reviewed) with the
expectation that the binding will change. Many of the common bindings
have had many rounds of review and they have not changed. My position
any time I see an incompatible change has been that it require anyone
affected to be okay with the change. This has pretty much been limited
to specific platforms from what I've seen.

One area we've gotten into trouble is using ePAPR based bindings and
then deciding those aren't sufficient. For cpu bindings, we've gone
from the cores are describable we don't need them in DT to following
ePAPR to some level to more fully describing cpu info including
topology.

> It's likely that we still want to have a period in which a binding is
> tentative and can be changed. Sometimes we don't know what we really
> want until after we've used it a while, and sometimes we, like
> everybody else, make mistakes on what is a good idea and not. The
> alternative is to grind most new binding proposals to a halt while we
> spend mind-numbing hours and hours on polishing every single aspect of
> the binding to a perfect shine, since we can't go back and fix it.
>
> So, there really seems to be a need for a layered approach, one in
> which a binding "graduates" from being tentative to being locked in.
> I'm refraining from using the terms "staging" and "stable" here, since
> they have overloaded meaning in the kernel world that doesn't apply
> here.

+1

Tentative must still be reviewed to the same level bindings which have
been reviewed up to now were.

> One problem that needs to be solved is obviously how a binding
> graduates from tentative to locked. This work isn't going to be very
> interesting to most people, I suspect. Think standards committee type
> work.

I think a time based stabilization period would be better than a
separate directory to apply bindings too. Or time plus periodic review
perhaps.

> A possible way to handle this is to have exactly that: A group of
> people that essentially constitute the "standards committee" that meet
> on a regular basis to review and approve new bindings. They should be
> people not just doing ARM Linux work, but other stakeholders in these
> bindings too. One of the things they should probably do is sift
> through our current in-kernel bindings and select those who seem ready
> to be locked in, review/discuss/decide upon that and once the decision
> is made, that specific binding does become part of the static,
> never-ever-change ABI of firmware-to-kernel interfaces.   That might
> also be the time that the binding is moved from the kernel to a
> separate repo, but that's a technicality that we'll let the DT
> maintainers decide among themselves, IMHO.

Seems like a good plan.

Rob

>
> I think that captures most of what we discussed in person. Others
> might have changed their opinions since then, so I'm definitely not
> claiming that any of this was an official decision made by anybody.
> It's just one proposal on how to handle these things moving forward.
>
>
> -Olof
> --
> To unsubscribe from this list: send the line "unsubscribe devicetree" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
> [I'm adding LKML and ksummit-discuss to this thread, since the ACPI/DT
> discussions have been covered there and this overlaps some with that]
>
> On Thu, Jul 25, 2013 at 7:38 AM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
>> On Thu, Jul 25, 2013 at 03:27:02PM +0100, Russell King - ARM Linux wrote:
>>> Remember the stated assertion when DT was first added to the kernel: the
>>> DT descriptions _will_ become a separately maintained project which is
>>> independent of the kernel.  They will _not_ be kernel version specific.
>>
>> I'm looking forward to this.
>>
>> Question for the DT guys: what are the plans here? Are we going to get
>> rid of the .dts files inside the kernel tree?
>
> In the discussions we had in Dublin, a couple of options on how to
> lock in bindings were discussed. I'm a little surprised that Grant
> didn't cover them in his initial emails on the new maintainership
> model, but maybe he wanted the new group to handle it. And they didn't
> bring it up yet either. So I am. :-)
>
> Until now, we have been working under the assumption that the bindings
> are _NOT LOCKED_. I.e. they can change as needed, and we _ARE_
> assuming that the device tree has to match the kernel. That has been a
> good choice as people get up to speed on what is a good binding and
> not, and has given us much-needed room to adjust things as needed.
> That obviously has to change, but doing so needs to be done carefully.

I would argue that little has gone in (if reviewed) with the
expectation that the binding will change. Many of the common bindings
have had many rounds of review and they have not changed. My position
any time I see an incompatible change has been that it require anyone
affected to be okay with the change. This has pretty much been limited
to specific platforms from what I've seen.

One area we've gotten into trouble is using ePAPR based bindings and
then deciding those aren't sufficient. For cpu bindings, we've gone
from the cores are describable we don't need them in DT to following
ePAPR to some level to more fully describing cpu info including
topology.

> It's likely that we still want to have a period in which a binding is
> tentative and can be changed. Sometimes we don't know what we really
> want until after we've used it a while, and sometimes we, like
> everybody else, make mistakes on what is a good idea and not. The
> alternative is to grind most new binding proposals to a halt while we
> spend mind-numbing hours and hours on polishing every single aspect of
> the binding to a perfect shine, since we can't go back and fix it.
>
> So, there really seems to be a need for a layered approach, one in
> which a binding "graduates" from being tentative to being locked in.
> I'm refraining from using the terms "staging" and "stable" here, since
> they have overloaded meaning in the kernel world that doesn't apply
> here.

+1

Tentative must still be reviewed to the same level bindings which have
been reviewed up to now were.

> One problem that needs to be solved is obviously how a binding
> graduates from tentative to locked. This work isn't going to be very
> interesting to most people, I suspect. Think standards committee type
> work.

I think a time based stabilization period would be better than a
separate directory to apply bindings too. Or time plus periodic review
perhaps.

> A possible way to handle this is to have exactly that: A group of
> people that essentially constitute the "standards committee" that meet
> on a regular basis to review and approve new bindings. They should be
> people not just doing ARM Linux work, but other stakeholders in these
> bindings too. One of the things they should probably do is sift
> through our current in-kernel bindings and select those who seem ready
> to be locked in, review/discuss/decide upon that and once the decision
> is made, that specific binding does become part of the static,
> never-ever-change ABI of firmware-to-kernel interfaces.   That might
> also be the time that the binding is moved from the kernel to a
> separate repo, but that's a technicality that we'll let the DT
> maintainers decide among themselves, IMHO.

Seems like a good plan.

Rob

>
> I think that captures most of what we discussed in person. Others
> might have changed their opinions since then, so I'm definitely not
> claiming that any of this was an official decision made by anybody.
> It's just one proposal on how to handle these things moving forward.
>
>
> -Olof
> --
> To unsubscribe from this list: send the line "unsubscribe devicetree" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 914 bytes --]

On Thu, Jul 25, 2013 at 11:25:30AM -0700, Olof Johansson wrote:

> To talk semi-specifics: What about USB PHY tunings for a specific
> board, where each of the three USB ports and their registers have
> slightly different layout? Sure, we can take 10 properties to describe
> each tunable field in each register, but at the end it will just be
> used to craft the contents blindly, so we might just stuff the 32-bit
> register value as a property instead. And in other cases we might
> indeed want to describe each property independently. Determining what
> is appropriate when is one of the most difficult parts of the review
> workflow.

There's also my constant question about why the driver can't just work
out the right value or decide to use the feature autonomously at runtime
- I'd say that the overwhelming proportion of properties I push back on
end up just getting removed and worked out automatically.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Thu, Jul 25, 2013 at 11:25:30AM -0700, Olof Johansson wrote:

> To talk semi-specifics: What about USB PHY tunings for a specific
> board, where each of the three USB ports and their registers have
> slightly different layout? Sure, we can take 10 properties to describe
> each tunable field in each register, but at the end it will just be
> used to craft the contents blindly, so we might just stuff the 32-bit
> register value as a property instead. And in other cases we might
> indeed want to describe each property independently. Determining what
> is appropriate when is one of the most difficult parts of the review
> workflow.

There's also my constant question about why the driver can't just work
out the right value or decide to use the feature autonomously at runtime
- I'd say that the overwhelming proportion of properties I push back on
end up just getting removed and worked out automatically.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130725/506c2a35/attachment.sig>

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
> On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:

> > One problem that needs to be solved is obviously how a binding
> > graduates from tentative to locked. This work isn't going to be very
> > interesting to most people, I suspect. Think standards committee type
> > work.
> 
> I think a time based stabilization period would be better than a
> separate directory to apply bindings too. Or time plus periodic review
> perhaps.

The only problem with a time-based versus separate directory is how do
users who've downloaded the tree determine which bindings are stable?
If they pull a tarball, or receive an SDK, there is most likely no git
history attached.

I think the idea of a 'tentative' directory (or 'locked') is churnish,
but necessary.  If I DL'd a tarball and had to type 'tentative' to get
to the binding doc I wanted, that would be a pretty clear clue to be
delicate about how I trust/use/plan with that binding.

thx,

Jason.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
> On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:

> > One problem that needs to be solved is obviously how a binding
> > graduates from tentative to locked. This work isn't going to be very
> > interesting to most people, I suspect. Think standards committee type
> > work.
> 
> I think a time based stabilization period would be better than a
> separate directory to apply bindings too. Or time plus periodic review
> perhaps.

The only problem with a time-based versus separate directory is how do
users who've downloaded the tree determine which bindings are stable?
If they pull a tarball, or receive an SDK, there is most likely no git
history attached.

I think the idea of a 'tentative' directory (or 'locked') is churnish,
but necessary.  If I DL'd a tarball and had to type 'tentative' to get
to the binding doc I wanted, that would be a pretty clear clue to be
delicate about how I trust/use/plan with that binding.

thx,

Jason.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 405 bytes --]

On Thu, Jul 25, 2013 at 03:31:35PM -0400, Jason Cooper wrote:

> I think the idea of a 'tentative' directory (or 'locked') is churnish,
> but necessary.  If I DL'd a tarball and had to type 'tentative' to get
> to the binding doc I wanted, that would be a pretty clear clue to be
> delicate about how I trust/use/plan with that binding.

Another option for this is to annotate within the document itself.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Thu, Jul 25, 2013 at 03:31:35PM -0400, Jason Cooper wrote:

> I think the idea of a 'tentative' directory (or 'locked') is churnish,
> but necessary.  If I DL'd a tarball and had to type 'tentative' to get
> to the binding doc I wanted, that would be a pretty clear clue to be
> delicate about how I trust/use/plan with that binding.

Another option for this is to annotate within the document itself.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130725/8a18c880/attachment-0001.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Thu, Jul 25, 2013 at 09:04:35PM +0100, Mark Brown wrote:
> On Thu, Jul 25, 2013 at 03:31:35PM -0400, Jason Cooper wrote:
> 
> > I think the idea of a 'tentative' directory (or 'locked') is churnish,
> > but necessary.  If I DL'd a tarball and had to type 'tentative' to get
> > to the binding doc I wanted, that would be a pretty clear clue to be
> > delicate about how I trust/use/plan with that binding.
> 
> Another option for this is to annotate within the document itself.

True, and this would make it easier to mark reg, interrupts, etc as
stable, and eg 'ti,hwmods' as unstable.  However, it could get pretty
messy.  Perhaps moving to a schema would help with that.

thx,

Jason.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Thu, Jul 25, 2013 at 09:04:35PM +0100, Mark Brown wrote:
> On Thu, Jul 25, 2013 at 03:31:35PM -0400, Jason Cooper wrote:
> 
> > I think the idea of a 'tentative' directory (or 'locked') is churnish,
> > but necessary.  If I DL'd a tarball and had to type 'tentative' to get
> > to the binding doc I wanted, that would be a pretty clear clue to be
> > delicate about how I trust/use/plan with that binding.
> 
> Another option for this is to annotate within the document itself.

True, and this would make it easier to mark reg, interrupts, etc as
stable, and eg 'ti,hwmods' as unstable.  However, it could get pretty
messy.  Perhaps moving to a schema would help with that.

thx,

Jason.

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On Thu, Jul 25, 2013 at 2:31 PM, Jason Cooper <jason@lakedaemon.net> wrote:
> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
>> On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
>
>> > One problem that needs to be solved is obviously how a binding
>> > graduates from tentative to locked. This work isn't going to be very
>> > interesting to most people, I suspect. Think standards committee type
>> > work.
>>
>> I think a time based stabilization period would be better than a
>> separate directory to apply bindings too. Or time plus periodic review
>> perhaps.
>
> The only problem with a time-based versus separate directory is how do
> users who've downloaded the tree determine which bindings are stable?
> If they pull a tarball, or receive an SDK, there is most likely no git
> history attached.

Well, if time based includes moving the binding out of the kernel,
then that is what defines it as stable or not. I guess that is a form
of a separate directory. I don't think we want to be moving bindings
twice: tentative -> stable and kernel -> DT repo.

The policy could be as simple as an binding without change in at least
N kernel releases is moved out and stable.

Rob

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On Thu, Jul 25, 2013 at 2:31 PM, Jason Cooper <jason@lakedaemon.net> wrote:
> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
>> On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
>
>> > One problem that needs to be solved is obviously how a binding
>> > graduates from tentative to locked. This work isn't going to be very
>> > interesting to most people, I suspect. Think standards committee type
>> > work.
>>
>> I think a time based stabilization period would be better than a
>> separate directory to apply bindings too. Or time plus periodic review
>> perhaps.
>
> The only problem with a time-based versus separate directory is how do
> users who've downloaded the tree determine which bindings are stable?
> If they pull a tarball, or receive an SDK, there is most likely no git
> history attached.

Well, if time based includes moving the binding out of the kernel,
then that is what defines it as stable or not. I guess that is a form
of a separate directory. I don't think we want to be moving bindings
twice: tentative -> stable and kernel -> DT repo.

The policy could be as simple as an binding without change in at least
N kernel releases is moved out and stable.

Rob

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Thu, Jul 25, 2013 at 03:16:14PM -0500, Rob Herring wrote:
> On Thu, Jul 25, 2013 at 2:31 PM, Jason Cooper <jason@lakedaemon.net> wrote:
> > On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
> >> On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
> >
> >> > One problem that needs to be solved is obviously how a binding
> >> > graduates from tentative to locked. This work isn't going to be very
> >> > interesting to most people, I suspect. Think standards committee type
> >> > work.
> >>
> >> I think a time based stabilization period would be better than a
> >> separate directory to apply bindings too. Or time plus periodic review
> >> perhaps.
> >
> > The only problem with a time-based versus separate directory is how do
> > users who've downloaded the tree determine which bindings are stable?
> > If they pull a tarball, or receive an SDK, there is most likely no git
> > history attached.
> 
> Well, if time based includes moving the binding out of the kernel,
> then that is what defines it as stable or not. I guess that is a form
> of a separate directory.

One of the things I've been trying to square up in my head is how to
retain the history of the binding when moving to the new tree.  My
current idea was to clone the kernel tree, add one patch deleting
everything but the bindings and dts files, and one more patch moving
things where we want them (arch/{powerpc,arm}/boot/dts -> dts).

Then, as needed, we could merge a kernel version tag and delete
everything we don't need (code) in the merge commit.

The downside of this is it would be messy, the upside is that we could
closely track the kernel tree (until the bindings and dts are moved
out), and retain the history of the bindings and dts files.

> I don't think we want to be moving bindings twice: tentative -> stable
> and kernel -> DT repo.

agreed.

> The policy could be as simple as an binding without change in at least
> N kernel releases is moved out and stable.

If the cloned tree idea works, we could just merge version tags that are
two or three behind the most recent version.

thx,

Jason.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Thu, Jul 25, 2013 at 03:16:14PM -0500, Rob Herring wrote:
> On Thu, Jul 25, 2013 at 2:31 PM, Jason Cooper <jason@lakedaemon.net> wrote:
> > On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
> >> On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
> >
> >> > One problem that needs to be solved is obviously how a binding
> >> > graduates from tentative to locked. This work isn't going to be very
> >> > interesting to most people, I suspect. Think standards committee type
> >> > work.
> >>
> >> I think a time based stabilization period would be better than a
> >> separate directory to apply bindings too. Or time plus periodic review
> >> perhaps.
> >
> > The only problem with a time-based versus separate directory is how do
> > users who've downloaded the tree determine which bindings are stable?
> > If they pull a tarball, or receive an SDK, there is most likely no git
> > history attached.
> 
> Well, if time based includes moving the binding out of the kernel,
> then that is what defines it as stable or not. I guess that is a form
> of a separate directory.

One of the things I've been trying to square up in my head is how to
retain the history of the binding when moving to the new tree.  My
current idea was to clone the kernel tree, add one patch deleting
everything but the bindings and dts files, and one more patch moving
things where we want them (arch/{powerpc,arm}/boot/dts -> dts).

Then, as needed, we could merge a kernel version tag and delete
everything we don't need (code) in the merge commit.

The downside of this is it would be messy, the upside is that we could
closely track the kernel tree (until the bindings and dts are moved
out), and retain the history of the bindings and dts files.

> I don't think we want to be moving bindings twice: tentative -> stable
> and kernel -> DT repo.

agreed.

> The policy could be as simple as an binding without change in at least
> N kernel releases is moved out and stable.

If the cloned tree idea works, we could just merge version tags that are
two or three behind the most recent version.

thx,

Jason.

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Thu, Jul 25, 2013 at 08:48:34PM +0200, Richard Cochran wrote:
> On Thu, Jul 25, 2013 at 07:29:20PM +0100, Mark Rutland wrote:
> > On Thu, Jul 25, 2013 at 07:05:48PM +0100, Stephen Warren wrote:
> > > 
> > > I don't think having people "rely" on the bindings is the issue so much
> > > as the awareness that if they do, there will be compatibility issues for
> > > unstable bindings.
> > 
> > As long as we can make sufficiently clear that trying to use an unstable
> > binding is going to be *very* painful, and not necessarily supported.
> 
> Oh, man.
> 
> The introduction of DT into ARM Linux was supposed to make everyone's
> life sooo much easier. Of course, based on experience with powerpc, I
> never believed it*, but still I would expect to hear that the DT
> bindings are, well, a *binding* contract between the board developer,
> boot loader, and the kernel.

To restate a perspective I've shared before (as someone shipping
embedded products with DT on PPC and ARM).

We use DT has a kernel configuration input. Our environment is
designed to guarantee 100% that the kernel and DT match exactly. DT
very deliberately isn't an ABI boundary in our systems.

We've been doing this for years and have a proven in the field track
record of upgrades from pre-2.6.16 to 3.7 and beyond with multiple
SOCs. The same bootloader that was shipped to support non-DT 2.6.16
boots DT 3.7 just fine.

For closed system embedded DT has proven *WONDERFUL*. It is a huge,
gigantic improvement over what we had before. The introduction of DT
carried with it an increase in generality and configurability that has
gone far beyond what was possible using just board.c files (back in
the 2.6.teens days).

This is where I see the value in DT. ABI stability is not something I
want/need from DT. As an ODM we have dramatically less board specific
code than ever before, and new code we need is upstreamable as DT
elements.

IMHO, this is a fine and very reasonable way to use DT in embedded.

To me, it is general purpose stuff (Chromebooks, ARM servers, etc)
where the main problem is. I think those cases need a different
solution: A subset of DT that is guarenteed ABI stable, firmware that
substantially sets up the entire machine, and the proper callback
hooks (eg through UEFI and AHCI) to let the firmware do low level
hardware specific work at runtime.

This is how x86 gets the kind of compatability it has. Trying to
describe and control every tiny detail (pin mux, regulator, clk) is
great for embedded, but fundamentally not future-proof enough for
general purpose.

Regards,
Jason

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Thu, Jul 25, 2013 at 08:48:34PM +0200, Richard Cochran wrote:
> On Thu, Jul 25, 2013 at 07:29:20PM +0100, Mark Rutland wrote:
> > On Thu, Jul 25, 2013 at 07:05:48PM +0100, Stephen Warren wrote:
> > > 
> > > I don't think having people "rely" on the bindings is the issue so much
> > > as the awareness that if they do, there will be compatibility issues for
> > > unstable bindings.
> > 
> > As long as we can make sufficiently clear that trying to use an unstable
> > binding is going to be *very* painful, and not necessarily supported.
> 
> Oh, man.
> 
> The introduction of DT into ARM Linux was supposed to make everyone's
> life sooo much easier. Of course, based on experience with powerpc, I
> never believed it*, but still I would expect to hear that the DT
> bindings are, well, a *binding* contract between the board developer,
> boot loader, and the kernel.

To restate a perspective I've shared before (as someone shipping
embedded products with DT on PPC and ARM).

We use DT has a kernel configuration input. Our environment is
designed to guarantee 100% that the kernel and DT match exactly. DT
very deliberately isn't an ABI boundary in our systems.

We've been doing this for years and have a proven in the field track
record of upgrades from pre-2.6.16 to 3.7 and beyond with multiple
SOCs. The same bootloader that was shipped to support non-DT 2.6.16
boots DT 3.7 just fine.

For closed system embedded DT has proven *WONDERFUL*. It is a huge,
gigantic improvement over what we had before. The introduction of DT
carried with it an increase in generality and configurability that has
gone far beyond what was possible using just board.c files (back in
the 2.6.teens days).

This is where I see the value in DT. ABI stability is not something I
want/need from DT. As an ODM we have dramatically less board specific
code than ever before, and new code we need is upstreamable as DT
elements.

IMHO, this is a fine and very reasonable way to use DT in embedded.

To me, it is general purpose stuff (Chromebooks, ARM servers, etc)
where the main problem is. I think those cases need a different
solution: A subset of DT that is guarenteed ABI stable, firmware that
substantially sets up the entire machine, and the proper callback
hooks (eg through UEFI and AHCI) to let the firmware do low level
hardware specific work at runtime.

This is how x86 gets the kind of compatability it has. Trying to
describe and control every tiny detail (pin mux, regulator, clk) is
great for embedded, but fundamentally not future-proof enough for
general purpose.

Regards,
Jason

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Domenico Andreoli]

On Thu, Jul 25, 2013 at 11:53:49AM -0700, Stephen Warren wrote:
> On 07/25/2013 11:48 AM, Richard Cochran wrote:
> > On Thu, Jul 25, 2013 at 07:29:20PM +0100, Mark Rutland wrote:
> >> On Thu, Jul 25, 2013 at 07:05:48PM +0100, Stephen Warren wrote:
> >>>
> >>> I don't think having people "rely" on the bindings is the issue so much
> >>> as the awareness that if they do, there will be compatibility issues for
> >>> unstable bindings.
> >>
> >> As long as we can make sufficiently clear that trying to use an unstable
> >> binding is going to be *very* painful, and not necessarily supported.
> > 
> > Oh, man.
> > 
> > The introduction of DT into ARM Linux was supposed to make everyone's
> > life sooo much easier. Of course, based on experience with powerpc, I
> > never believed it*, but still I would expect to hear that the DT
> > bindings are, well, a *binding* contract between the board developer,
> > boot loader, and the kernel.
> > 
> > Once it is working with a particular kernel, a DT board description
> > file should continue to work indefinitely with newer kernels. Anything
> > less is a regression, pure and simple.
> > 
> > If you go around changing the bindings willy nilly, then what is point
> > of having DT at all?
> 
> That's exactly why we're starting to think about which bindings should
> be considered stable and immutable, and when that should happen. As Olof
> pointed out, we haven't fully enforced that yet. Preferably bindings
> will be marked stable very fast, but mistakes are always going to happen
> in early development. ABIs are very hard.

If it was possible to clearly distinguish stable and unstable bindings,
then the kernel could allow unstable bindings only from appended DTBs.

This would prevent unwanted DT ABIs while leaving plenty of room for
the developers.

2c

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Domenico Andreoli]

On Thu, Jul 25, 2013 at 11:53:49AM -0700, Stephen Warren wrote:
> On 07/25/2013 11:48 AM, Richard Cochran wrote:
> > On Thu, Jul 25, 2013 at 07:29:20PM +0100, Mark Rutland wrote:
> >> On Thu, Jul 25, 2013 at 07:05:48PM +0100, Stephen Warren wrote:
> >>>
> >>> I don't think having people "rely" on the bindings is the issue so much
> >>> as the awareness that if they do, there will be compatibility issues for
> >>> unstable bindings.
> >>
> >> As long as we can make sufficiently clear that trying to use an unstable
> >> binding is going to be *very* painful, and not necessarily supported.
> > 
> > Oh, man.
> > 
> > The introduction of DT into ARM Linux was supposed to make everyone's
> > life sooo much easier. Of course, based on experience with powerpc, I
> > never believed it*, but still I would expect to hear that the DT
> > bindings are, well, a *binding* contract between the board developer,
> > boot loader, and the kernel.
> > 
> > Once it is working with a particular kernel, a DT board description
> > file should continue to work indefinitely with newer kernels. Anything
> > less is a regression, pure and simple.
> > 
> > If you go around changing the bindings willy nilly, then what is point
> > of having DT at all?
> 
> That's exactly why we're starting to think about which bindings should
> be considered stable and immutable, and when that should happen. As Olof
> pointed out, we haven't fully enforced that yet. Preferably bindings
> will be marked stable very fast, but mistakes are always going to happen
> in early development. ABIs are very hard.

If it was possible to clearly distinguish stable and unstable bindings,
then the kernel could allow unstable bindings only from appended DTBs.

This would prevent unwanted DT ABIs while leaving plenty of room for
the developers.

2c

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Ben Hutchings]

On Thu, Jul 25, 2013 at 04:32:28PM -0400, Jason Cooper wrote:
[...]
> One of the things I've been trying to square up in my head is how to
> retain the history of the binding when moving to the new tree.  My
> current idea was to clone the kernel tree, add one patch deleting
> everything but the bindings and dts files, and one more patch moving
> things where we want them (arch/{powerpc,arm}/boot/dts -> dts).
> 
> Then, as needed, we could merge a kernel version tag and delete
> everything we don't need (code) in the merge commit.
> 
> The downside of this is it would be messy, the upside is that we could
> closely track the kernel tree (until the bindings and dts are moved
> out), and retain the history of the bindings and dts files.
[...]

It's *extremely* messy, but 'git filter-branch' might help to make a
clean tracking branch preserving history of just DT files.  David
Woodhouse did something like that for the linux-firmware repo
initially.

Ben.

-- 
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
                                                              - Albert Camus

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Ben Hutchings]

On Thu, Jul 25, 2013 at 04:32:28PM -0400, Jason Cooper wrote:
[...]
> One of the things I've been trying to square up in my head is how to
> retain the history of the binding when moving to the new tree.  My
> current idea was to clone the kernel tree, add one patch deleting
> everything but the bindings and dts files, and one more patch moving
> things where we want them (arch/{powerpc,arm}/boot/dts -> dts).
> 
> Then, as needed, we could merge a kernel version tag and delete
> everything we don't need (code) in the merge commit.
> 
> The downside of this is it would be messy, the upside is that we could
> closely track the kernel tree (until the bindings and dts are moved
> out), and retain the history of the bindings and dts files.
[...]

It's *extremely* messy, but 'git filter-branch' might help to make a
clean tracking branch preserving history of just DT files.  David
Woodhouse did something like that for the linux-firmware repo
initially.

Ben.

-- 
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
                                                              - Albert Camus

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Thu, Jul 25, 2013 at 03:31:35PM -0400, Jason Cooper wrote:
> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
> > On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
> 
> > > One problem that needs to be solved is obviously how a binding
> > > graduates from tentative to locked. This work isn't going to be very
> > > interesting to most people, I suspect. Think standards committee type
> > > work.
> > 
> > I think a time based stabilization period would be better than a
> > separate directory to apply bindings too. Or time plus periodic review
> > perhaps.
> 
> The only problem with a time-based versus separate directory is how do
> users who've downloaded the tree determine which bindings are stable?
> If they pull a tarball, or receive an SDK, there is most likely no git
> history attached.
> 
> I think the idea of a 'tentative' directory (or 'locked') is churnish,
> but necessary.  If I DL'd a tarball and had to type 'tentative' to get
> to the binding doc I wanted, that would be a pretty clear clue to be
> delicate about how I trust/use/plan with that binding.

It's actually extremely simple.  If the bindings are in development,
they must not appear in a -final released kernel.  Anything that appears
in a -final kernel becomes part of the ABI at that point.

That obviously does not mean you remove them in the last -rc and put
them back during the merge window!

That's how we handle every other ABI thing in the kernel tree, why should
DT files be any different?  (I've added Linus and Grant to this discussion.)

As I've already stated, it is intended to eventually remove the DT files
from the kernel tree and have them as a separately maintained project,
which means they will be independent of the kernel version.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Thu, Jul 25, 2013 at 03:31:35PM -0400, Jason Cooper wrote:
> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
> > On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
> 
> > > One problem that needs to be solved is obviously how a binding
> > > graduates from tentative to locked. This work isn't going to be very
> > > interesting to most people, I suspect. Think standards committee type
> > > work.
> > 
> > I think a time based stabilization period would be better than a
> > separate directory to apply bindings too. Or time plus periodic review
> > perhaps.
> 
> The only problem with a time-based versus separate directory is how do
> users who've downloaded the tree determine which bindings are stable?
> If they pull a tarball, or receive an SDK, there is most likely no git
> history attached.
> 
> I think the idea of a 'tentative' directory (or 'locked') is churnish,
> but necessary.  If I DL'd a tarball and had to type 'tentative' to get
> to the binding doc I wanted, that would be a pretty clear clue to be
> delicate about how I trust/use/plan with that binding.

It's actually extremely simple.  If the bindings are in development,
they must not appear in a -final released kernel.  Anything that appears
in a -final kernel becomes part of the ABI at that point.

That obviously does not mean you remove them in the last -rc and put
them back during the merge window!

That's how we handle every other ABI thing in the kernel tree, why should
DT files be any different?  (I've added Linus and Grant to this discussion.)

As I've already stated, it is intended to eventually remove the DT files
from the kernel tree and have them as a separately maintained project,
which means they will be independent of the kernel version.

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Thu, Jul 25, 2013 at 7:18 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Thu, Jul 25, 2013 at 03:31:35PM -0400, Jason Cooper wrote:
>> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
>> > On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
>>
>> > > One problem that needs to be solved is obviously how a binding
>> > > graduates from tentative to locked. This work isn't going to be very
>> > > interesting to most people, I suspect. Think standards committee type
>> > > work.
>> >
>> > I think a time based stabilization period would be better than a
>> > separate directory to apply bindings too. Or time plus periodic review
>> > perhaps.
>>
>> The only problem with a time-based versus separate directory is how do
>> users who've downloaded the tree determine which bindings are stable?
>> If they pull a tarball, or receive an SDK, there is most likely no git
>> history attached.
>>
>> I think the idea of a 'tentative' directory (or 'locked') is churnish,
>> but necessary.  If I DL'd a tarball and had to type 'tentative' to get
>> to the binding doc I wanted, that would be a pretty clear clue to be
>> delicate about how I trust/use/plan with that binding.
>
> It's actually extremely simple.  If the bindings are in development,
> they must not appear in a -final released kernel.  Anything that appears
> in a -final kernel becomes part of the ABI at that point.
>
> That obviously does not mean you remove them in the last -rc and put
> them back during the merge window!
>
> That's how we handle every other ABI thing in the kernel tree, why should
> DT files be any different?  (I've added Linus and Grant to this discussion.)
>
> As I've already stated, it is intended to eventually remove the DT files
> from the kernel tree and have them as a separately maintained project,
> which means they will be independent of the kernel version.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Having a schema system for the device trees is closely related to this
discussion.  In this case the schema would probably be equal to the
stable set of nodes. This has been discussed before on the device tree
mailing list. The dtc compiler would take this schema and validate the
trees it compiles against it issuing warnings for 'non-standard'
usage. Over time the schema would be updated to allow these usages
when everyone agrees to it. Note that there would be a single schema
describing all possible legal Linux device trees.

The scheme is also quite useful for new tree developers since it will
show them the universe of device tree attributes that have already
been standardized. By using comments, you could probably turn the
device tree documentation into the schema source files.

-- 
Jon Smirl
jonsmirl@gmail.com

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Thu, Jul 25, 2013 at 7:18 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Thu, Jul 25, 2013 at 03:31:35PM -0400, Jason Cooper wrote:
>> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
>> > On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
>>
>> > > One problem that needs to be solved is obviously how a binding
>> > > graduates from tentative to locked. This work isn't going to be very
>> > > interesting to most people, I suspect. Think standards committee type
>> > > work.
>> >
>> > I think a time based stabilization period would be better than a
>> > separate directory to apply bindings too. Or time plus periodic review
>> > perhaps.
>>
>> The only problem with a time-based versus separate directory is how do
>> users who've downloaded the tree determine which bindings are stable?
>> If they pull a tarball, or receive an SDK, there is most likely no git
>> history attached.
>>
>> I think the idea of a 'tentative' directory (or 'locked') is churnish,
>> but necessary.  If I DL'd a tarball and had to type 'tentative' to get
>> to the binding doc I wanted, that would be a pretty clear clue to be
>> delicate about how I trust/use/plan with that binding.
>
> It's actually extremely simple.  If the bindings are in development,
> they must not appear in a -final released kernel.  Anything that appears
> in a -final kernel becomes part of the ABI at that point.
>
> That obviously does not mean you remove them in the last -rc and put
> them back during the merge window!
>
> That's how we handle every other ABI thing in the kernel tree, why should
> DT files be any different?  (I've added Linus and Grant to this discussion.)
>
> As I've already stated, it is intended to eventually remove the DT files
> from the kernel tree and have them as a separately maintained project,
> which means they will be independent of the kernel version.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Having a schema system for the device trees is closely related to this
discussion.  In this case the schema would probably be equal to the
stable set of nodes. This has been discussed before on the device tree
mailing list. The dtc compiler would take this schema and validate the
trees it compiles against it issuing warnings for 'non-standard'
usage. Over time the schema would be updated to allow these usages
when everyone agrees to it. Note that there would be a single schema
describing all possible legal Linux device trees.

The scheme is also quite useful for new tree developers since it will
show them the universe of device tree attributes that have already
been standardized. By using comments, you could probably turn the
device tree documentation into the schema source files.

-- 
Jon Smirl
jonsmirl at gmail.com

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 01:16 PM, Rob Herring wrote:
> On Thu, Jul 25, 2013 at 2:31 PM, Jason Cooper <jason@lakedaemon.net> wrote:
>> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
>>> On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
>>
>>>> One problem that needs to be solved is obviously how a binding
>>>> graduates from tentative to locked. This work isn't going to be very
>>>> interesting to most people, I suspect. Think standards committee type
>>>> work.
>>>
>>> I think a time based stabilization period would be better than a
>>> separate directory to apply bindings too. Or time plus periodic review
>>> perhaps.
>>
>> The only problem with a time-based versus separate directory is how do
>> users who've downloaded the tree determine which bindings are stable?
>> If they pull a tarball, or receive an SDK, there is most likely no git
>> history attached.
> 
> Well, if time based includes moving the binding out of the kernel,
> then that is what defines it as stable or not. I guess that is a form
> of a separate directory. I don't think we want to be moving bindings
> twice: tentative -> stable and kernel -> DT repo.
> 
> The policy could be as simple as an binding without change in at least
> N kernel releases is moved out and stable.

That might not be quite the right criteria. Just because something
didn't change doesn't mean it's "correct" and that any problems in the
binding have been addressed. As one example, on Tegra, we have a few
bindings that haven't changed in a while, yet rely on custom properties
for describing which DMA channel to use, rather than using the
fairly-recently-introduced standard DMA DT properties (this particular
example is being rectified now, but I'm sure there are plenty of similar
examples)

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 01:16 PM, Rob Herring wrote:
> On Thu, Jul 25, 2013 at 2:31 PM, Jason Cooper <jason@lakedaemon.net> wrote:
>> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
>>> On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
>>
>>>> One problem that needs to be solved is obviously how a binding
>>>> graduates from tentative to locked. This work isn't going to be very
>>>> interesting to most people, I suspect. Think standards committee type
>>>> work.
>>>
>>> I think a time based stabilization period would be better than a
>>> separate directory to apply bindings too. Or time plus periodic review
>>> perhaps.
>>
>> The only problem with a time-based versus separate directory is how do
>> users who've downloaded the tree determine which bindings are stable?
>> If they pull a tarball, or receive an SDK, there is most likely no git
>> history attached.
> 
> Well, if time based includes moving the binding out of the kernel,
> then that is what defines it as stable or not. I guess that is a form
> of a separate directory. I don't think we want to be moving bindings
> twice: tentative -> stable and kernel -> DT repo.
> 
> The policy could be as simple as an binding without change in at least
> N kernel releases is moved out and stable.

That might not be quite the right criteria. Just because something
didn't change doesn't mean it's "correct" and that any problems in the
binding have been addressed. As one example, on Tegra, we have a few
bindings that haven't changed in a while, yet rely on custom properties
for describing which DMA channel to use, rather than using the
fairly-recently-introduced standard DMA DT properties (this particular
example is being rectified now, but I'm sure there are plenty of similar
examples)

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 02:53 PM, Ben Hutchings wrote:
> On Thu, Jul 25, 2013 at 04:32:28PM -0400, Jason Cooper wrote:
> [...]
>> One of the things I've been trying to square up in my head is how to
>> retain the history of the binding when moving to the new tree.  My
>> current idea was to clone the kernel tree, add one patch deleting
>> everything but the bindings and dts files, and one more patch moving
>> things where we want them (arch/{powerpc,arm}/boot/dts -> dts).
>>
>> Then, as needed, we could merge a kernel version tag and delete
>> everything we don't need (code) in the merge commit.
>>
>> The downside of this is it would be messy, the upside is that we could
>> closely track the kernel tree (until the bindings and dts are moved
>> out), and retain the history of the bindings and dts files.
> [...]
> 
> It's *extremely* messy, but 'git filter-branch' might help to make a
> clean tracking branch preserving history of just DT files.  David
> Woodhouse did something like that for the linux-firmware repo
> initially.

Ian Campbell is already working on exactly this solution:
https://lkml.org/lkml/2013/7/23/547

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/25/2013 02:53 PM, Ben Hutchings wrote:
> On Thu, Jul 25, 2013 at 04:32:28PM -0400, Jason Cooper wrote:
> [...]
>> One of the things I've been trying to square up in my head is how to
>> retain the history of the binding when moving to the new tree.  My
>> current idea was to clone the kernel tree, add one patch deleting
>> everything but the bindings and dts files, and one more patch moving
>> things where we want them (arch/{powerpc,arm}/boot/dts -> dts).
>>
>> Then, as needed, we could merge a kernel version tag and delete
>> everything we don't need (code) in the merge commit.
>>
>> The downside of this is it would be messy, the upside is that we could
>> closely track the kernel tree (until the bindings and dts are moved
>> out), and retain the history of the bindings and dts files.
> [...]
> 
> It's *extremely* messy, but 'git filter-branch' might help to make a
> clean tracking branch preserving history of just DT files.  David
> Woodhouse did something like that for the linux-firmware repo
> initially.

Ian Campbell is already working on exactly this solution:
https://lkml.org/lkml/2013/7/23/547

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Thu, Jul 25, 2013 at 11:53:49AM -0700, Stephen Warren wrote:
> On 07/25/2013 11:48 AM, Richard Cochran wrote:
> > On Thu, Jul 25, 2013 at 07:29:20PM +0100, Mark Rutland wrote:
> >>
> >> As long as we can make sufficiently clear that trying to use an unstable
> >> binding is going to be *very* painful, and not necessarily supported.

IOW, its okay to break DT setups with every release, as long as we
tell people first.

Well, at least you are being honest about it now.

> That's exactly why we're starting to think about which bindings should
> be considered stable and immutable, and when that should happen. As Olof
> pointed out, we haven't fully enforced that yet. Preferably bindings
> will be marked stable very fast, but mistakes are always going to happen
> in early development. ABIs are very hard.

They are even harder if you cannot decide what the ABI is in the first
place.

Thanks,
Richard

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Thu, Jul 25, 2013 at 11:53:49AM -0700, Stephen Warren wrote:
> On 07/25/2013 11:48 AM, Richard Cochran wrote:
> > On Thu, Jul 25, 2013 at 07:29:20PM +0100, Mark Rutland wrote:
> >>
> >> As long as we can make sufficiently clear that trying to use an unstable
> >> binding is going to be *very* painful, and not necessarily supported.

IOW, its okay to break DT setups with every release, as long as we
tell people first.

Well, at least you are being honest about it now.

> That's exactly why we're starting to think about which bindings should
> be considered stable and immutable, and when that should happen. As Olof
> pointed out, we haven't fully enforced that yet. Preferably bindings
> will be marked stable very fast, but mistakes are always going to happen
> in early development. ABIs are very hard.

They are even harder if you cannot decide what the ABI is in the first
place.

Thanks,
Richard

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:

> We use DT has a kernel configuration input. Our environment is
> designed to guarantee 100% that the kernel and DT match exactly. DT
> very deliberately isn't an ABI boundary in our systems.

It is nice that you use DT in that way, but that is not how DT is
supposed to work. If you must keep your DT in sync with the kernel,
then there is no advantage over the old platfrom device method. At
least that had the virtue of being a C language interface (ABI), and
some mistakes were be caught by the compiler.

> We've been doing this for years and have a proven in the field track
> record of upgrades from pre-2.6.16 to 3.7 and beyond with multiple
> SOCs. The same bootloader that was shipped to support non-DT 2.6.16
> boots DT 3.7 just fine.

Try that with Freescale PowerPCs.  Good luck.

Heck, even Paul's OMAP test reports have been spoiled by his not
deleting old dtb files. Of course, that is his fault (and not DT's, no
never).
 
> For closed system embedded DT has proven *WONDERFUL*.

I too work on commercial embedded systems, and DT has proven to be
one gigantic *PITA*.

Thanks,
Richard

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:

> We use DT has a kernel configuration input. Our environment is
> designed to guarantee 100% that the kernel and DT match exactly. DT
> very deliberately isn't an ABI boundary in our systems.

It is nice that you use DT in that way, but that is not how DT is
supposed to work. If you must keep your DT in sync with the kernel,
then there is no advantage over the old platfrom device method. At
least that had the virtue of being a C language interface (ABI), and
some mistakes were be caught by the compiler.

> We've been doing this for years and have a proven in the field track
> record of upgrades from pre-2.6.16 to 3.7 and beyond with multiple
> SOCs. The same bootloader that was shipped to support non-DT 2.6.16
> boots DT 3.7 just fine.

Try that with Freescale PowerPCs.  Good luck.

Heck, even Paul's OMAP test reports have been spoiled by his not
deleting old dtb files. Of course, that is his fault (and not DT's, no
never).
 
> For closed system embedded DT has proven *WONDERFUL*.

I too work on commercial embedded systems, and DT has proven to be
one gigantic *PITA*.

Thanks,
Richard

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> 
> We use DT has a kernel configuration input. Our environment is
> designed to guarantee 100% that the kernel and DT match exactly. DT
> very deliberately isn't an ABI boundary in our systems.

Think about what you just said.

The DT describes the *hardware* not the kernel. Why should you ever
need to update your DT at all?

(Hint: the answer is, because the DT system is broken.)

Thanks,
Richard

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> 
> We use DT has a kernel configuration input. Our environment is
> designed to guarantee 100% that the kernel and DT match exactly. DT
> very deliberately isn't an ABI boundary in our systems.

Think about what you just said.

The DT describes the *hardware* not the kernel. Why should you ever
need to update your DT at all?

(Hint: the answer is, because the DT system is broken.)

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1509 bytes --]

On Fri, 2013-07-26 at 10:01 +0200, Richard Cochran wrote:
> On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> > 
> > We use DT has a kernel configuration input. Our environment is
> > designed to guarantee 100% that the kernel and DT match exactly. DT
> > very deliberately isn't an ABI boundary in our systems.
> 
> Think about what you just said.
> 
> The DT describes the *hardware* not the kernel. Why should you ever
> need to update your DT at all?

Well, the nodes which describe hardware devices, according to the
bindings which form an ABI contract between DT and drivers, should not
normally change. Although they *can* change, if for example you change
the MAC address and that's stored there. Or you change the PHY you want
it to use. Or something like that. The *ABI* doesn't change, but the
data you express *using* that ABI can change. That's kind of the point.

And the nodes in /chosen aren't describing hardware at all. Those are
*absolutely* expected to be configuration things. It's quite normal for
those to vary from one boot to the next, if your bootloader is
sophisticated enough to do that.

And if Jason wants to hard-code that kind of configuration data into his
device-tree and ship it as part of the overall system image that gets
installed, I see no particular problem with that.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

On Fri, 2013-07-26 at 10:01 +0200, Richard Cochran wrote:
> On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> > 
> > We use DT has a kernel configuration input. Our environment is
> > designed to guarantee 100% that the kernel and DT match exactly. DT
> > very deliberately isn't an ABI boundary in our systems.
> 
> Think about what you just said.
> 
> The DT describes the *hardware* not the kernel. Why should you ever
> need to update your DT at all?

Well, the nodes which describe hardware devices, according to the
bindings which form an ABI contract between DT and drivers, should not
normally change. Although they *can* change, if for example you change
the MAC address and that's stored there. Or you change the PHY you want
it to use. Or something like that. The *ABI* doesn't change, but the
data you express *using* that ABI can change. That's kind of the point.

And the nodes in /chosen aren't describing hardware at all. Those are
*absolutely* expected to be configuration things. It's quite normal for
those to vary from one boot to the next, if your bootloader is
sophisticated enough to do that.

And if Jason wants to hard-code that kind of configuration data into his
device-tree and ship it as part of the overall system image that gets
installed, I see no particular problem with that.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse at intel.com                              Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5745 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130726/b2ffc9da/attachment.bin>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:

> I would argue that little has gone in (if reviewed) with the
> expectation that the binding will change. Many of the common bindings
> have had many rounds of review and they have not changed. My position
> any time I see an incompatible change has been that it require anyone
> affected to be okay with the change. This has pretty much been limited
> to specific platforms from what I've seen.

The big exception I've seen has been the OMAP hwmod stuff.  Plus of
course the case where things get moved into the DT bindings from the
board file as things are built up, but this should be becoming a non
issue as people move onto DT only boards (though I am worried about
people on systems where the SoC is described in DT).

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:

> I would argue that little has gone in (if reviewed) with the
> expectation that the binding will change. Many of the common bindings
> have had many rounds of review and they have not changed. My position
> any time I see an incompatible change has been that it require anyone
> affected to be okay with the change. This has pretty much been limited
> to specific platforms from what I've seen.

The big exception I've seen has been the OMAP hwmod stuff.  Plus of
course the case where things get moved into the DT bindings from the
board file as things are built up, but this should be becoming a non
issue as people move onto DT only boards (though I am worried about
people on systems where the SoC is described in DT).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130726/b7ddc32a/attachment-0001.sig>

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Thu, Jul 25, 2013 at 08:27:15PM -0400, jonsmirl@gmail.com wrote:
> On Thu, Jul 25, 2013 at 7:18 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > On Thu, Jul 25, 2013 at 03:31:35PM -0400, Jason Cooper wrote:
> >> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
> >> > On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
> >>
> >> > > One problem that needs to be solved is obviously how a binding
> >> > > graduates from tentative to locked. This work isn't going to be very
> >> > > interesting to most people, I suspect. Think standards committee type
> >> > > work.
> >> >
> >> > I think a time based stabilization period would be better than a
> >> > separate directory to apply bindings too. Or time plus periodic review
> >> > perhaps.
> >>
> >> The only problem with a time-based versus separate directory is how do
> >> users who've downloaded the tree determine which bindings are stable?
> >> If they pull a tarball, or receive an SDK, there is most likely no git
> >> history attached.
> >>
> >> I think the idea of a 'tentative' directory (or 'locked') is churnish,
> >> but necessary.  If I DL'd a tarball and had to type 'tentative' to get
> >> to the binding doc I wanted, that would be a pretty clear clue to be
> >> delicate about how I trust/use/plan with that binding.
> >
> > It's actually extremely simple.  If the bindings are in development,
> > they must not appear in a -final released kernel.  Anything that appears
> > in a -final kernel becomes part of the ABI at that point.
> >
> > That obviously does not mean you remove them in the last -rc and put
> > them back during the merge window!
> >
> > That's how we handle every other ABI thing in the kernel tree, why should
> > DT files be any different?  (I've added Linus and Grant to this discussion.)
> >
> > As I've already stated, it is intended to eventually remove the DT files
> > from the kernel tree and have them as a separately maintained project,
> > which means they will be independent of the kernel version.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> 
> Having a schema system for the device trees is closely related to this
> discussion.  In this case the schema would probably be equal to the
> stable set of nodes. This has been discussed before on the device tree
> mailing list. The dtc compiler would take this schema and validate the
> trees it compiles against it issuing warnings for 'non-standard'
> usage. Over time the schema would be updated to allow these usages
> when everyone agrees to it. Note that there would be a single schema
> describing all possible legal Linux device trees.

s/Linux// ?

> The scheme is also quite useful for new tree developers since it will
> show them the universe of device tree attributes that have already
> been standardized. By using comments, you could probably turn the
> device tree documentation into the schema source files.

One more note on schema, since DT is a description of hardware, it would
be useful to have two comments, a url to the datasheet, and a canonical
name of the datasheet suitable for $searchengine.  Where available, of
course.

thx,

Jason.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Thu, Jul 25, 2013 at 08:27:15PM -0400, jonsmirl at gmail.com wrote:
> On Thu, Jul 25, 2013 at 7:18 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > On Thu, Jul 25, 2013 at 03:31:35PM -0400, Jason Cooper wrote:
> >> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
> >> > On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
> >>
> >> > > One problem that needs to be solved is obviously how a binding
> >> > > graduates from tentative to locked. This work isn't going to be very
> >> > > interesting to most people, I suspect. Think standards committee type
> >> > > work.
> >> >
> >> > I think a time based stabilization period would be better than a
> >> > separate directory to apply bindings too. Or time plus periodic review
> >> > perhaps.
> >>
> >> The only problem with a time-based versus separate directory is how do
> >> users who've downloaded the tree determine which bindings are stable?
> >> If they pull a tarball, or receive an SDK, there is most likely no git
> >> history attached.
> >>
> >> I think the idea of a 'tentative' directory (or 'locked') is churnish,
> >> but necessary.  If I DL'd a tarball and had to type 'tentative' to get
> >> to the binding doc I wanted, that would be a pretty clear clue to be
> >> delicate about how I trust/use/plan with that binding.
> >
> > It's actually extremely simple.  If the bindings are in development,
> > they must not appear in a -final released kernel.  Anything that appears
> > in a -final kernel becomes part of the ABI at that point.
> >
> > That obviously does not mean you remove them in the last -rc and put
> > them back during the merge window!
> >
> > That's how we handle every other ABI thing in the kernel tree, why should
> > DT files be any different?  (I've added Linus and Grant to this discussion.)
> >
> > As I've already stated, it is intended to eventually remove the DT files
> > from the kernel tree and have them as a separately maintained project,
> > which means they will be independent of the kernel version.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo at vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> 
> Having a schema system for the device trees is closely related to this
> discussion.  In this case the schema would probably be equal to the
> stable set of nodes. This has been discussed before on the device tree
> mailing list. The dtc compiler would take this schema and validate the
> trees it compiles against it issuing warnings for 'non-standard'
> usage. Over time the schema would be updated to allow these usages
> when everyone agrees to it. Note that there would be a single schema
> describing all possible legal Linux device trees.

s/Linux// ?

> The scheme is also quite useful for new tree developers since it will
> show them the universe of device tree attributes that have already
> been standardized. By using comments, you could probably turn the
> device tree documentation into the schema source files.

One more note on schema, since DT is a description of hardware, it would
be useful to have two comments, a url to the datasheet, and a canonical
name of the datasheet suitable for $searchengine.  Where available, of
course.

thx,

Jason.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Thu, Jul 25, 2013 at 10:53:15PM +0100, Ben Hutchings wrote:
> On Thu, Jul 25, 2013 at 04:32:28PM -0400, Jason Cooper wrote:
> [...]
> > One of the things I've been trying to square up in my head is how to
> > retain the history of the binding when moving to the new tree.  My
> > current idea was to clone the kernel tree, add one patch deleting
> > everything but the bindings and dts files, and one more patch moving
> > things where we want them (arch/{powerpc,arm}/boot/dts -> dts).
> > 
> > Then, as needed, we could merge a kernel version tag and delete
> > everything we don't need (code) in the merge commit.
> > 
> > The downside of this is it would be messy, the upside is that we could
> > closely track the kernel tree (until the bindings and dts are moved
> > out), and retain the history of the bindings and dts files.
> [...]
> 
> It's *extremely* messy, but 'git filter-branch' might help to make a
> clean tracking branch preserving history of just DT files.  David
> Woodhouse did something like that for the linux-firmware repo
> initially.

ok, but once that is setup, asking patch submitters to keep binding doc
and dts file changes in separate patches from code changes would allow
us to put those patches in a separate branch, mergable into both trees.

thx,

Jason.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Thu, Jul 25, 2013 at 10:53:15PM +0100, Ben Hutchings wrote:
> On Thu, Jul 25, 2013 at 04:32:28PM -0400, Jason Cooper wrote:
> [...]
> > One of the things I've been trying to square up in my head is how to
> > retain the history of the binding when moving to the new tree.  My
> > current idea was to clone the kernel tree, add one patch deleting
> > everything but the bindings and dts files, and one more patch moving
> > things where we want them (arch/{powerpc,arm}/boot/dts -> dts).
> > 
> > Then, as needed, we could merge a kernel version tag and delete
> > everything we don't need (code) in the merge commit.
> > 
> > The downside of this is it would be messy, the upside is that we could
> > closely track the kernel tree (until the bindings and dts are moved
> > out), and retain the history of the bindings and dts files.
> [...]
> 
> It's *extremely* messy, but 'git filter-branch' might help to make a
> clean tracking branch preserving history of just DT files.  David
> Woodhouse did something like that for the linux-firmware repo
> initially.

ok, but once that is setup, asking patch submitters to keep binding doc
and dts file changes in separate patches from code changes would allow
us to put those patches in a separate branch, mergable into both trees.

thx,

Jason.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Fri, Jul 26, 2013 at 10:42:24AM +0100, David Woodhouse wrote:
> On Fri, 2013-07-26 at 10:01 +0200, Richard Cochran wrote:
> > On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> > > 
> > > We use DT has a kernel configuration input. Our environment is
> > > designed to guarantee 100% that the kernel and DT match exactly. DT
> > > very deliberately isn't an ABI boundary in our systems.
> > 
> > Think about what you just said.
> > 
> > The DT describes the *hardware* not the kernel. Why should you ever
> > need to update your DT at all?
> 
> Well, the nodes which describe hardware devices, according to the
> bindings which form an ABI contract between DT and drivers, should not
> normally change. Although they *can* change, if for example you change
> the MAC address and that's stored there. Or you change the PHY you want
> it to use. Or something like that. The *ABI* doesn't change, but the
> data you express *using* that ABI can change. That's kind of the point.

Unless I totally misunderstood, the thread is talking about letting
established bindings change with each new kernel version.  I am
opposed to that.

Of course, a user may want to change the values of his MAC addresses,
if he needs to. But he should never have to change *how* he specifies
those addresses.

So, as long as everyone can agree to the principle that a working DT
should remain working after a kernel upgrade, then I'll shut up. In
actual fact, this is not the case today, nor was it ever so in the
past, AFAICT, but it never hurts to set goals.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Fri, Jul 26, 2013 at 10:42:24AM +0100, David Woodhouse wrote:
> On Fri, 2013-07-26 at 10:01 +0200, Richard Cochran wrote:
> > On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> > > 
> > > We use DT has a kernel configuration input. Our environment is
> > > designed to guarantee 100% that the kernel and DT match exactly. DT
> > > very deliberately isn't an ABI boundary in our systems.
> > 
> > Think about what you just said.
> > 
> > The DT describes the *hardware* not the kernel. Why should you ever
> > need to update your DT at all?
> 
> Well, the nodes which describe hardware devices, according to the
> bindings which form an ABI contract between DT and drivers, should not
> normally change. Although they *can* change, if for example you change
> the MAC address and that's stored there. Or you change the PHY you want
> it to use. Or something like that. The *ABI* doesn't change, but the
> data you express *using* that ABI can change. That's kind of the point.

Unless I totally misunderstood, the thread is talking about letting
established bindings change with each new kernel version.  I am
opposed to that.

Of course, a user may want to change the values of his MAC addresses,
if he needs to. But he should never have to change *how* he specifies
those addresses.

So, as long as everyone can agree to the principle that a working DT
should remain working after a kernel upgrade, then I'll shut up. In
actual fact, this is not the case today, nor was it ever so in the
past, AFAICT, but it never hurts to set goals.

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> On Fri, Jul 26, 2013 at 10:42:24AM +0100, David Woodhouse wrote:
> > On Fri, 2013-07-26 at 10:01 +0200, Richard Cochran wrote:
> > > On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> > > > 
> > > > We use DT has a kernel configuration input. Our environment is
> > > > designed to guarantee 100% that the kernel and DT match exactly. DT
> > > > very deliberately isn't an ABI boundary in our systems.
> > > 
> > > Think about what you just said.
> > > 
> > > The DT describes the *hardware* not the kernel. Why should you ever
> > > need to update your DT at all?
> > 
> > Well, the nodes which describe hardware devices, according to the
> > bindings which form an ABI contract between DT and drivers, should not
> > normally change. Although they *can* change, if for example you change
> > the MAC address and that's stored there. Or you change the PHY you want
> > it to use. Or something like that. The *ABI* doesn't change, but the
> > data you express *using* that ABI can change. That's kind of the point.
> 
> Unless I totally misunderstood, the thread is talking about letting
> established bindings change with each new kernel version.  I am
> opposed to that.
> 
> Of course, a user may want to change the values of his MAC addresses,
> if he needs to. But he should never have to change *how* he specifies
> those addresses.
> 
> So, as long as everyone can agree to the principle that a working DT
> should remain working after a kernel upgrade, then I'll shut up. In
> actual fact, this is not the case today, nor was it ever so in the
> past, AFAICT, but it never hurts to set goals.

We agree.

That's precisely why I'm saying that once a binding appears in a -final
released kernel, it is _stable_ by that very fact.  If there were any
doubts about it, it should never have been accepted into the kernel tree
in the first place.

It's no different from how we treat syscalls.  Once we accept a syscall
and it's published in a -final kernel, it can't then be altered - it can
be supplemented by a new syscall with a different interface, but the old
one remains and keeps its semantics for a great many years.

There's no reason for DT to be any different in this regard.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> On Fri, Jul 26, 2013 at 10:42:24AM +0100, David Woodhouse wrote:
> > On Fri, 2013-07-26 at 10:01 +0200, Richard Cochran wrote:
> > > On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> > > > 
> > > > We use DT has a kernel configuration input. Our environment is
> > > > designed to guarantee 100% that the kernel and DT match exactly. DT
> > > > very deliberately isn't an ABI boundary in our systems.
> > > 
> > > Think about what you just said.
> > > 
> > > The DT describes the *hardware* not the kernel. Why should you ever
> > > need to update your DT at all?
> > 
> > Well, the nodes which describe hardware devices, according to the
> > bindings which form an ABI contract between DT and drivers, should not
> > normally change. Although they *can* change, if for example you change
> > the MAC address and that's stored there. Or you change the PHY you want
> > it to use. Or something like that. The *ABI* doesn't change, but the
> > data you express *using* that ABI can change. That's kind of the point.
> 
> Unless I totally misunderstood, the thread is talking about letting
> established bindings change with each new kernel version.  I am
> opposed to that.
> 
> Of course, a user may want to change the values of his MAC addresses,
> if he needs to. But he should never have to change *how* he specifies
> those addresses.
> 
> So, as long as everyone can agree to the principle that a working DT
> should remain working after a kernel upgrade, then I'll shut up. In
> actual fact, this is not the case today, nor was it ever so in the
> past, AFAICT, but it never hurts to set goals.

We agree.

That's precisely why I'm saying that once a binding appears in a -final
released kernel, it is _stable_ by that very fact.  If there were any
doubts about it, it should never have been accepted into the kernel tree
in the first place.

It's no different from how we treat syscalls.  Once we accept a syscall
and it's published in a -final kernel, it can't then be altered - it can
be supplemented by a new syscall with a different interface, but the old
one remains and keeps its semantics for a great many years.

There's no reason for DT to be any different in this regard.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> On Fri, Jul 26, 2013 at 10:42:24AM +0100, David Woodhouse wrote:
> > On Fri, 2013-07-26 at 10:01 +0200, Richard Cochran wrote:
> > > On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> > > > 
> > > > We use DT has a kernel configuration input. Our environment is
> > > > designed to guarantee 100% that the kernel and DT match exactly. DT
> > > > very deliberately isn't an ABI boundary in our systems.
> > > 
> > > Think about what you just said.
> > > 
> > > The DT describes the *hardware* not the kernel. Why should you ever
> > > need to update your DT at all?
> > 
> > Well, the nodes which describe hardware devices, according to the
> > bindings which form an ABI contract between DT and drivers, should not
> > normally change. Although they *can* change, if for example you change
> > the MAC address and that's stored there. Or you change the PHY you want
> > it to use. Or something like that. The *ABI* doesn't change, but the
> > data you express *using* that ABI can change. That's kind of the point.
> 
> Unless I totally misunderstood, the thread is talking about letting
> established bindings change with each new kernel version.  I am
> opposed to that.
> 
> Of course, a user may want to change the values of his MAC addresses,
> if he needs to. But he should never have to change *how* he specifies
> those addresses.

The other dynamic change that bears mentioning here is attributes which
have been configured by the bootloader.  For example, in mvebu, we have
the Schrodinger's Cat register.  It allows you to reconfigure the base
address of the registers from *within* that register range.  If the
bootloader does this, the DT needs to be updated to reflect the current
hardware configuration.  Otherwise, the kernel is stuck poking around at
memory addresses hoping to find something sane.

But this falls into the same category as you mentioned, but outside of
chosen {};.

> So, as long as everyone can agree to the principle that a working DT
> should remain working after a kernel upgrade, then I'll shut up. In
> actual fact, this is not the case today, nor was it ever so in the
> past, AFAICT, but it never hurts to set goals.

Some instability is necessary as we figure out what 'stable' is.
Otherwise, things would've been so rigid, we'd've never made any
progress.  I'd say we now a sufficient body of code and experience to
start enforcing the goal.

thx,

Jason.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> On Fri, Jul 26, 2013 at 10:42:24AM +0100, David Woodhouse wrote:
> > On Fri, 2013-07-26 at 10:01 +0200, Richard Cochran wrote:
> > > On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> > > > 
> > > > We use DT has a kernel configuration input. Our environment is
> > > > designed to guarantee 100% that the kernel and DT match exactly. DT
> > > > very deliberately isn't an ABI boundary in our systems.
> > > 
> > > Think about what you just said.
> > > 
> > > The DT describes the *hardware* not the kernel. Why should you ever
> > > need to update your DT at all?
> > 
> > Well, the nodes which describe hardware devices, according to the
> > bindings which form an ABI contract between DT and drivers, should not
> > normally change. Although they *can* change, if for example you change
> > the MAC address and that's stored there. Or you change the PHY you want
> > it to use. Or something like that. The *ABI* doesn't change, but the
> > data you express *using* that ABI can change. That's kind of the point.
> 
> Unless I totally misunderstood, the thread is talking about letting
> established bindings change with each new kernel version.  I am
> opposed to that.
> 
> Of course, a user may want to change the values of his MAC addresses,
> if he needs to. But he should never have to change *how* he specifies
> those addresses.

The other dynamic change that bears mentioning here is attributes which
have been configured by the bootloader.  For example, in mvebu, we have
the Schrodinger's Cat register.  It allows you to reconfigure the base
address of the registers from *within* that register range.  If the
bootloader does this, the DT needs to be updated to reflect the current
hardware configuration.  Otherwise, the kernel is stuck poking around at
memory addresses hoping to find something sane.

But this falls into the same category as you mentioned, but outside of
chosen {};.

> So, as long as everyone can agree to the principle that a working DT
> should remain working after a kernel upgrade, then I'll shut up. In
> actual fact, this is not the case today, nor was it ever so in the
> past, AFAICT, but it never hurts to set goals.

Some instability is necessary as we figure out what 'stable' is.
Otherwise, things would've been so rigid, we'd've never made any
progress.  I'd say we now a sufficient body of code and experience to
start enforcing the goal.

thx,

Jason.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Fri, Jul 26, 2013 at 09:27:09AM -0400, Jason Cooper wrote:
> On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> > Unless I totally misunderstood, the thread is talking about letting
> > established bindings change with each new kernel version.  I am
> > opposed to that.
> > 
> > Of course, a user may want to change the values of his MAC addresses,
> > if he needs to. But he should never have to change *how* he specifies
> > those addresses.
> 
> The other dynamic change that bears mentioning here is attributes which
> have been configured by the bootloader.  For example, in mvebu, we have
> the Schrodinger's Cat register.  It allows you to reconfigure the base
> address of the registers from *within* that register range.  If the
> bootloader does this, the DT needs to be updated to reflect the current
> hardware configuration.  Otherwise, the kernel is stuck poking around at
> memory addresses hoping to find something sane.
> 
> But this falls into the same category as you mentioned, but outside of
> chosen {};.

No, this falls within the remit of "describing the hardware" and it is
certainly something that is free to change.

What should not "change" once a kernel is the method by which hardware is
described in DT.  "change" there in the sense that how it was described by
kernel 3.X should still be accepted by 3.X+n, even if 3.X+n comes up with
a much better way to describe it.

The actual data associated with those descriptions is free to change in
whatever way is necessary if the hardware itself changes due to things
being programmed differently.

Think of it as the difference between the design of an interface, and the
interface being used.  We don't mandate that the write() syscall shall
always be called for fd=1 with length=5 and bytes "Hello" in the buffer.
We mandate that the write() syscall shall be passed an integer fd, a
buffer pointer, and a length and we don't change that ever.

Think of "a better way to describe it" as introducing the writev() syscall
to supplement write() so that applications can do writes from scattered
memory locations.  We don't get rid of the write() syscall - we add to
the ABI that's already there leaving the existing interfaces with exactly
the same semantics, so that all the existing stuff continues to work as-is.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Fri, Jul 26, 2013 at 09:27:09AM -0400, Jason Cooper wrote:
> On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> > Unless I totally misunderstood, the thread is talking about letting
> > established bindings change with each new kernel version.  I am
> > opposed to that.
> > 
> > Of course, a user may want to change the values of his MAC addresses,
> > if he needs to. But he should never have to change *how* he specifies
> > those addresses.
> 
> The other dynamic change that bears mentioning here is attributes which
> have been configured by the bootloader.  For example, in mvebu, we have
> the Schrodinger's Cat register.  It allows you to reconfigure the base
> address of the registers from *within* that register range.  If the
> bootloader does this, the DT needs to be updated to reflect the current
> hardware configuration.  Otherwise, the kernel is stuck poking around at
> memory addresses hoping to find something sane.
> 
> But this falls into the same category as you mentioned, but outside of
> chosen {};.

No, this falls within the remit of "describing the hardware" and it is
certainly something that is free to change.

What should not "change" once a kernel is the method by which hardware is
described in DT.  "change" there in the sense that how it was described by
kernel 3.X should still be accepted by 3.X+n, even if 3.X+n comes up with
a much better way to describe it.

The actual data associated with those descriptions is free to change in
whatever way is necessary if the hardware itself changes due to things
being programmed differently.

Think of it as the difference between the design of an interface, and the
interface being used.  We don't mandate that the write() syscall shall
always be called for fd=1 with length=5 and bytes "Hello" in the buffer.
We mandate that the write() syscall shall be passed an integer fd, a
buffer pointer, and a length and we don't change that ever.

Think of "a better way to describe it" as introducing the writev() syscall
to supplement write() so that applications can do writes from scattered
memory locations.  We don't get rid of the write() syscall - we add to
the ABI that's already there leaving the existing interfaces with exactly
the same semantics, so that all the existing stuff continues to work as-is.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

On Fri, 2013-07-26 at 09:27 -0400, Jason Cooper wrote:
> 
> The other dynamic change that bears mentioning here is attributes which
> have been configured by the bootloader.  For example, in mvebu, we have
> the Schrodinger's Cat register.  It allows you to reconfigure the base
> address of the registers from *within* that register range.  If the
> bootloader does this, the DT needs to be updated to reflect the current
> hardware configuration.  Otherwise, the kernel is stuck poking around at
> memory addresses hoping to find something sane.
> 
> But this falls into the same category as you mentioned, but outside of
> chosen {};.

Yeah, /chosen was given as an example of stuff that's almost
*exclusively* "configuration" stuff.

But there's plenty outside there that can reasonably change.

It's OK to change the data, and of *course* the base address reported in
the DT should actually match reality *today*, if it changes on the fly.

It's not OK to change the *schema*  in which those data are expressed.
That's the ABI we're talking about.

-- 
dwmw2

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

On Fri, 2013-07-26 at 09:27 -0400, Jason Cooper wrote:
> 
> The other dynamic change that bears mentioning here is attributes which
> have been configured by the bootloader.  For example, in mvebu, we have
> the Schrodinger's Cat register.  It allows you to reconfigure the base
> address of the registers from *within* that register range.  If the
> bootloader does this, the DT needs to be updated to reflect the current
> hardware configuration.  Otherwise, the kernel is stuck poking around at
> memory addresses hoping to find something sane.
> 
> But this falls into the same category as you mentioned, but outside of
> chosen {};.

Yeah, /chosen was given as an example of stuff that's almost
*exclusively* "configuration" stuff.

But there's plenty outside there that can reasonably change.

It's OK to change the data, and of *course* the base address reported in
the DT should actually match reality *today*, if it changes on the fly.

It's not OK to change the *schema*  in which those data are expressed.
That's the ABI we're talking about.

-- 
dwmw2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5745 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130726/9f7f5720/attachment.bin>

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On 07/25/2013 11:54 PM, Richard Cochran wrote:
> On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> 
>> We use DT has a kernel configuration input. Our environment is
>> designed to guarantee 100% that the kernel and DT match exactly. DT
>> very deliberately isn't an ABI boundary in our systems.
> 
> It is nice that you use DT in that way, but that is not how DT is
> supposed to work. If you must keep your DT in sync with the kernel,
> then there is no advantage over the old platfrom device method. At
> least that had the virtue of being a C language interface (ABI), and
> some mistakes were be caught by the compiler.

I agree, but there is one advantage in that how the data in the DT is
described is unified. Before, how every platform defined and created
devices was slightly different. Now you know where to go look and don't
have things all over the place and hidden by custom macros. We could
have fixed all that in C without DT, but DT conversion forced the clean-up.

>> We've been doing this for years and have a proven in the field track
>> record of upgrades from pre-2.6.16 to 3.7 and beyond with multiple
>> SOCs. The same bootloader that was shipped to support non-DT 2.6.16
>> boots DT 3.7 just fine.
> 
> Try that with Freescale PowerPCs.  Good luck.
> 
> Heck, even Paul's OMAP test reports have been spoiled by his not
> deleting old dtb files. Of course, that is his fault (and not DT's, no
> never).
>  
>> For closed system embedded DT has proven *WONDERFUL*.
> 
> I too work on commercial embedded systems, and DT has proven to be
> one gigantic *PITA*.

Due to ABI breakage or other reasons?

Rob

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On 07/25/2013 11:54 PM, Richard Cochran wrote:
> On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> 
>> We use DT has a kernel configuration input. Our environment is
>> designed to guarantee 100% that the kernel and DT match exactly. DT
>> very deliberately isn't an ABI boundary in our systems.
> 
> It is nice that you use DT in that way, but that is not how DT is
> supposed to work. If you must keep your DT in sync with the kernel,
> then there is no advantage over the old platfrom device method. At
> least that had the virtue of being a C language interface (ABI), and
> some mistakes were be caught by the compiler.

I agree, but there is one advantage in that how the data in the DT is
described is unified. Before, how every platform defined and created
devices was slightly different. Now you know where to go look and don't
have things all over the place and hidden by custom macros. We could
have fixed all that in C without DT, but DT conversion forced the clean-up.

>> We've been doing this for years and have a proven in the field track
>> record of upgrades from pre-2.6.16 to 3.7 and beyond with multiple
>> SOCs. The same bootloader that was shipped to support non-DT 2.6.16
>> boots DT 3.7 just fine.
> 
> Try that with Freescale PowerPCs.  Good luck.
> 
> Heck, even Paul's OMAP test reports have been spoiled by his not
> deleting old dtb files. Of course, that is his fault (and not DT's, no
> never).
>  
>> For closed system embedded DT has proven *WONDERFUL*.
> 
> I too work on commercial embedded systems, and DT has proven to be
> one gigantic *PITA*.

Due to ABI breakage or other reasons?

Rob

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Fri, Jul 26, 2013 at 02:38:02PM +0100, Russell King - ARM Linux wrote:
> On Fri, Jul 26, 2013 at 09:27:09AM -0400, Jason Cooper wrote:
> > On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> > > Unless I totally misunderstood, the thread is talking about letting
> > > established bindings change with each new kernel version.  I am
> > > opposed to that.
> > > 
> > > Of course, a user may want to change the values of his MAC addresses,
> > > if he needs to. But he should never have to change *how* he specifies
> > > those addresses.
> > 
> > The other dynamic change that bears mentioning here is attributes which
> > have been configured by the bootloader.  For example, in mvebu, we have
> > the Schrodinger's Cat register.  It allows you to reconfigure the base
> > address of the registers from *within* that register range.  If the
> > bootloader does this, the DT needs to be updated to reflect the current
> > hardware configuration.  Otherwise, the kernel is stuck poking around at
> > memory addresses hoping to find something sane.
> > 
> > But this falls into the same category as you mentioned, but outside of
> > chosen {};.
> 
> No, this falls within the remit of "describing the hardware" and it is
> certainly something that is free to change.

We agree, I was just highlighting that attributes outside of chosen can
and need to be rewritten by the bootloader.

> What should not "change" once a kernel is the method by which hardware is
> described in DT.  "change" there in the sense that how it was described by
> kernel 3.X should still be accepted by 3.X+n, even if 3.X+n comes up with
> a much better way to describe it.
> 
> The actual data associated with those descriptions is free to change in
> whatever way is necessary if the hardware itself changes due to things
> being programmed differently.
> 
> Think of it as the difference between the design of an interface, and the
> interface being used.  We don't mandate that the write() syscall shall
> always be called for fd=1 with length=5 and bytes "Hello" in the buffer.
> We mandate that the write() syscall shall be passed an integer fd, a
> buffer pointer, and a length and we don't change that ever.
> 
> Think of "a better way to describe it" as introducing the writev() syscall
> to supplement write() so that applications can do writes from scattered
> memory locations.  We don't get rid of the write() syscall - we add to
> the ABI that's already there leaving the existing interfaces with exactly
> the same semantics, so that all the existing stuff continues to work as-is.

Yes, the manner in which the bootloader writes the changes should adhere
to the binding.  In my example, it shouldn't replace the reg property
with reg-mod.  It should just change the addresses to reflect the
current state of the hardware the kernel will see.

thx,

Jason.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Fri, Jul 26, 2013 at 02:38:02PM +0100, Russell King - ARM Linux wrote:
> On Fri, Jul 26, 2013 at 09:27:09AM -0400, Jason Cooper wrote:
> > On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> > > Unless I totally misunderstood, the thread is talking about letting
> > > established bindings change with each new kernel version.  I am
> > > opposed to that.
> > > 
> > > Of course, a user may want to change the values of his MAC addresses,
> > > if he needs to. But he should never have to change *how* he specifies
> > > those addresses.
> > 
> > The other dynamic change that bears mentioning here is attributes which
> > have been configured by the bootloader.  For example, in mvebu, we have
> > the Schrodinger's Cat register.  It allows you to reconfigure the base
> > address of the registers from *within* that register range.  If the
> > bootloader does this, the DT needs to be updated to reflect the current
> > hardware configuration.  Otherwise, the kernel is stuck poking around at
> > memory addresses hoping to find something sane.
> > 
> > But this falls into the same category as you mentioned, but outside of
> > chosen {};.
> 
> No, this falls within the remit of "describing the hardware" and it is
> certainly something that is free to change.

We agree, I was just highlighting that attributes outside of chosen can
and need to be rewritten by the bootloader.

> What should not "change" once a kernel is the method by which hardware is
> described in DT.  "change" there in the sense that how it was described by
> kernel 3.X should still be accepted by 3.X+n, even if 3.X+n comes up with
> a much better way to describe it.
> 
> The actual data associated with those descriptions is free to change in
> whatever way is necessary if the hardware itself changes due to things
> being programmed differently.
> 
> Think of it as the difference between the design of an interface, and the
> interface being used.  We don't mandate that the write() syscall shall
> always be called for fd=1 with length=5 and bytes "Hello" in the buffer.
> We mandate that the write() syscall shall be passed an integer fd, a
> buffer pointer, and a length and we don't change that ever.
> 
> Think of "a better way to describe it" as introducing the writev() syscall
> to supplement write() so that applications can do writes from scattered
> memory locations.  We don't get rid of the write() syscall - we add to
> the ABI that's already there leaving the existing interfaces with exactly
> the same semantics, so that all the existing stuff continues to work as-is.

Yes, the manner in which the bootloader writes the changes should adhere
to the binding.  In my example, it shouldn't replace the reg property
with reg-mod.  It should just change the addresses to reflect the
current state of the hardware the kernel will see.

thx,

Jason.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Fri, Jul 26, 2013 at 02:41:10PM +0100, David Woodhouse wrote:
> On Fri, 2013-07-26 at 09:27 -0400, Jason Cooper wrote:
> > 
> > The other dynamic change that bears mentioning here is attributes which
> > have been configured by the bootloader.  For example, in mvebu, we have
> > the Schrodinger's Cat register.  It allows you to reconfigure the base
> > address of the registers from *within* that register range.  If the
> > bootloader does this, the DT needs to be updated to reflect the current
> > hardware configuration.  Otherwise, the kernel is stuck poking around at
> > memory addresses hoping to find something sane.
> > 
> > But this falls into the same category as you mentioned, but outside of
> > chosen {};.
> 
> Yeah, /chosen was given as an example of stuff that's almost
> *exclusively* "configuration" stuff.
> 
> But there's plenty outside there that can reasonably change.
> 
> It's OK to change the data, and of *course* the base address reported in
> the DT should actually match reality *today*, if it changes on the fly.
> 
> It's not OK to change the *schema*  in which those data are expressed.
> That's the ABI we're talking about.

Agreed, much more clear than what I was saying.

thx,

Jason.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Fri, Jul 26, 2013 at 02:41:10PM +0100, David Woodhouse wrote:
> On Fri, 2013-07-26 at 09:27 -0400, Jason Cooper wrote:
> > 
> > The other dynamic change that bears mentioning here is attributes which
> > have been configured by the bootloader.  For example, in mvebu, we have
> > the Schrodinger's Cat register.  It allows you to reconfigure the base
> > address of the registers from *within* that register range.  If the
> > bootloader does this, the DT needs to be updated to reflect the current
> > hardware configuration.  Otherwise, the kernel is stuck poking around at
> > memory addresses hoping to find something sane.
> > 
> > But this falls into the same category as you mentioned, but outside of
> > chosen {};.
> 
> Yeah, /chosen was given as an example of stuff that's almost
> *exclusively* "configuration" stuff.
> 
> But there's plenty outside there that can reasonably change.
> 
> It's OK to change the data, and of *course* the base address reported in
> the DT should actually match reality *today*, if it changes on the fly.
> 
> It's not OK to change the *schema*  in which those data are expressed.
> That's the ABI we're talking about.

Agreed, much more clear than what I was saying.

thx,

Jason.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 577 bytes --]

On Fri, 2013-07-26 at 09:45 -0400, Jason Cooper wrote:
> 
> We agree, I was just highlighting that attributes outside of chosen can
> and need to be rewritten by the bootloader.

Yes, absolutely. I had actually edited and re-ordered my message
<1374831744.2923.42.camel@shinybook.infradead.org> to try to make that
clearer by explicitly saying that the nodes which describe hardware
devices *can* change, before I ever made any mention of /chosen as
something which is almost *exclusively* dynamic.

Evidently that wasn't sufficient clarification. :)

-- 
dwmw2

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

On Fri, 2013-07-26 at 09:45 -0400, Jason Cooper wrote:
> 
> We agree, I was just highlighting that attributes outside of chosen can
> and need to be rewritten by the bootloader.

Yes, absolutely. I had actually edited and re-ordered my message
<1374831744.2923.42.camel@shinybook.infradead.org> to try to make that
clearer by explicitly saying that the nodes which describe hardware
devices *can* change, before I ever made any mention of /chosen as
something which is almost *exclusively* dynamic.

Evidently that wasn't sufficient clarification. :)

-- 
dwmw2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5745 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130726/93200ee8/attachment.bin>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 513 bytes --]

On Fri, Jul 26, 2013 at 12:18:48AM +0100, Russell King - ARM Linux wrote:

> It's actually extremely simple.  If the bindings are in development,
> they must not appear in a -final released kernel.  Anything that appears
> in a -final kernel becomes part of the ABI at that point.

Of course from that point of view there's also an argument that once
people have shipped something on a device or generally out of the review
process then we have to support it no matter how depressing their
choices may have been.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Fri, Jul 26, 2013 at 12:18:48AM +0100, Russell King - ARM Linux wrote:

> It's actually extremely simple.  If the bindings are in development,
> they must not appear in a -final released kernel.  Anything that appears
> in a -final kernel becomes part of the ABI at that point.

Of course from that point of view there's also an argument that once
people have shipped something on a device or generally out of the review
process then we have to support it no matter how depressing their
choices may have been.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130726/09c3df80/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Fri, Jul 26, 2013 at 02:50:53PM +0100, David Woodhouse wrote:
> On Fri, 2013-07-26 at 09:45 -0400, Jason Cooper wrote:
> > 
> > We agree, I was just highlighting that attributes outside of chosen can
> > and need to be rewritten by the bootloader.
> 
> Yes, absolutely. I had actually edited and re-ordered my message
> <1374831744.2923.42.camel@shinybook.infradead.org> to try to make that
> clearer by explicitly saying that the nodes which describe hardware
> devices *can* change, before I ever made any mention of /chosen as
> something which is almost *exclusively* dynamic.
> 
> Evidently that wasn't sufficient clarification. :)

It was, I just wanted to spell out an actual use-case for folks who are
new to devicetree.  Also, I was trying to gently poke everyone to take a
look at the mvebu-mbus binding Ezequiel just posted ;-)

Arnd Bergman and Jason Gunthorpe (and others) have reviewed it to death,
but with this new push for review by the devicetree maintainers, we
thought it best to repost it here (devicetree ml) before merging it.

thx,

Jason.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Cooper]

On Fri, Jul 26, 2013 at 02:50:53PM +0100, David Woodhouse wrote:
> On Fri, 2013-07-26 at 09:45 -0400, Jason Cooper wrote:
> > 
> > We agree, I was just highlighting that attributes outside of chosen can
> > and need to be rewritten by the bootloader.
> 
> Yes, absolutely. I had actually edited and re-ordered my message
> <1374831744.2923.42.camel@shinybook.infradead.org> to try to make that
> clearer by explicitly saying that the nodes which describe hardware
> devices *can* change, before I ever made any mention of /chosen as
> something which is almost *exclusively* dynamic.
> 
> Evidently that wasn't sufficient clarification. :)

It was, I just wanted to spell out an actual use-case for folks who are
new to devicetree.  Also, I was trying to gently poke everyone to take a
look at the mvebu-mbus binding Ezequiel just posted ;-)

Arnd Bergman and Jason Gunthorpe (and others) have reviewed it to death,
but with this new push for review by the devicetree maintainers, we
thought it best to repost it here (devicetree ml) before merging it.

thx,

Jason.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1064 bytes --]

On Fri, 2013-07-26 at 07:38 -0400, Jason Cooper wrote:
> > The scheme is also quite useful for new tree developers since it will
> > show them the universe of device tree attributes that have already
> > been standardized. By using comments, you could probably turn the
> > device tree documentation into the schema source files.
> 
> One more note on schema, since DT is a description of hardware, it would
> be useful to have two comments, a url to the datasheet, and a canonical
> name of the datasheet suitable for $searchengine.  Where available, of
> course.

Yes, that's a very good idea. After all, the behaviour described in (a
specific version of) the datasheet is an *implicit* part of the DT
binding. If I tell you "there is an 8390 <here>", then we damn well
ought to agree about how an 8390 behaves. And what revision of the chip
can be assumed from a given 'compatible' string.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

On Fri, 2013-07-26 at 07:38 -0400, Jason Cooper wrote:
> > The scheme is also quite useful for new tree developers since it will
> > show them the universe of device tree attributes that have already
> > been standardized. By using comments, you could probably turn the
> > device tree documentation into the schema source files.
> 
> One more note on schema, since DT is a description of hardware, it would
> be useful to have two comments, a url to the datasheet, and a canonical
> name of the datasheet suitable for $searchengine.  Where available, of
> course.

Yes, that's a very good idea. After all, the behaviour described in (a
specific version of) the datasheet is an *implicit* part of the DT
binding. If I tell you "there is an 8390 <here>", then we damn well
ought to agree about how an 8390 behaves. And what revision of the chip
can be assumed from a given 'compatible' string.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse at intel.com                              Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5745 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130726/88a97073/attachment.bin>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:

> Unless I totally misunderstood, the thread is talking about letting
> established bindings change with each new kernel version.  I am
> opposed to that.

No, nobody is really saying that is a particularly good idea.  There is
some debate about how we work out what an established binding is but
there's no serious suggestion that we don't want stable bindings.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:

> Unless I totally misunderstood, the thread is talking about letting
> established bindings change with each new kernel version.  I am
> opposed to that.

No, nobody is really saying that is a particularly good idea.  There is
some debate about how we work out what an established binding is but
there's no serious suggestion that we don't want stable bindings.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130726/b4884404/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Fri, Jul 26, 2013 at 9:41 AM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Fri, 2013-07-26 at 09:27 -0400, Jason Cooper wrote:
>>
>> The other dynamic change that bears mentioning here is attributes which
>> have been configured by the bootloader.  For example, in mvebu, we have
>> the Schrodinger's Cat register.  It allows you to reconfigure the base
>> address of the registers from *within* that register range.  If the
>> bootloader does this, the DT needs to be updated to reflect the current
>> hardware configuration.  Otherwise, the kernel is stuck poking around at
>> memory addresses hoping to find something sane.
>>
>> But this falls into the same category as you mentioned, but outside of
>> chosen {};.
>
> Yeah, /chosen was given as an example of stuff that's almost
> *exclusively* "configuration" stuff.
>
> But there's plenty outside there that can reasonably change.
>
> It's OK to change the data, and of *course* the base address reported in
> the DT should actually match reality *today*, if it changes on the fly.
>
> It's not OK to change the *schema*  in which those data are expressed.
> That's the ABI we're talking about.

Yes, yes - that's why the schema should be written down and used as a
validation input to dtc. Then dtc can spit out errors for non-standard
items. There would be two versions - the standard one and a legacy one
that includes the standard one plus the hacks that can't be undone.

But more importantly it provides a framework for people creating new
node definitions. Now they can't work in a vacuum and come up with
random names and structure for everything.

Most of the problems express in the thread would go away if the schema
was written down and discussed. The rule going forward would be no new
nodes that aren't part of the standard schema.

>
> --
> dwmw2
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Fri, Jul 26, 2013 at 9:41 AM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Fri, 2013-07-26 at 09:27 -0400, Jason Cooper wrote:
>>
>> The other dynamic change that bears mentioning here is attributes which
>> have been configured by the bootloader.  For example, in mvebu, we have
>> the Schrodinger's Cat register.  It allows you to reconfigure the base
>> address of the registers from *within* that register range.  If the
>> bootloader does this, the DT needs to be updated to reflect the current
>> hardware configuration.  Otherwise, the kernel is stuck poking around at
>> memory addresses hoping to find something sane.
>>
>> But this falls into the same category as you mentioned, but outside of
>> chosen {};.
>
> Yeah, /chosen was given as an example of stuff that's almost
> *exclusively* "configuration" stuff.
>
> But there's plenty outside there that can reasonably change.
>
> It's OK to change the data, and of *course* the base address reported in
> the DT should actually match reality *today*, if it changes on the fly.
>
> It's not OK to change the *schema*  in which those data are expressed.
> That's the ABI we're talking about.

Yes, yes - that's why the schema should be written down and used as a
validation input to dtc. Then dtc can spit out errors for non-standard
items. There would be two versions - the standard one and a legacy one
that includes the standard one plus the hacks that can't be undone.

But more importantly it provides a framework for people creating new
node definitions. Now they can't work in a vacuum and come up with
random names and structure for everything.

Most of the problems express in the thread would go away if the schema
was written down and discussed. The rule going forward would be no new
nodes that aren't part of the standard schema.

>
> --
> dwmw2
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

On Fri, 2013-07-26 at 10:14 -0400, jonsmirl@gmail.com wrote:
> 
> Yes, yes - that's why the schema should be written down and used as a
> validation input to dtc. Then dtc can spit out errors for non-standard
> items. There would be two versions - the standard one and a legacy one
> that includes the standard one plus the hacks that can't be undone.
> 
> But more importantly it provides a framework for people creating new
> node definitions. Now they can't work in a vacuum and come up with
> random names and structure for everything.
> 
> Most of the problems express in the thread would go away if the schema
> was written down and discussed. The rule going forward would be no new
> nodes that aren't part of the standard schema.

Yes, that seems eminently sensible.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

On Fri, 2013-07-26 at 10:14 -0400, jonsmirl at gmail.com wrote:
> 
> Yes, yes - that's why the schema should be written down and used as a
> validation input to dtc. Then dtc can spit out errors for non-standard
> items. There would be two versions - the standard one and a legacy one
> that includes the standard one plus the hacks that can't be undone.
> 
> But more importantly it provides a framework for people creating new
> node definitions. Now they can't work in a vacuum and come up with
> random names and structure for everything.
> 
> Most of the problems express in the thread would go away if the schema
> was written down and discussed. The rule going forward would be no new
> nodes that aren't part of the standard schema.

Yes, that seems eminently sensible.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse at intel.com                              Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5745 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130726/27cb750a/attachment.bin>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Fri, Jul 26, 2013 at 10:14:32AM -0400, jonsmirl@gmail.com wrote:
> Yes, yes - that's why the schema should be written down and used as a
> validation input to dtc. Then dtc can spit out errors for non-standard
> items. There would be two versions - the standard one and a legacy one
> that includes the standard one plus the hacks that can't be undone.
> 
> But more importantly it provides a framework for people creating new
> node definitions. Now they can't work in a vacuum and come up with
> random names and structure for everything.
> 
> Most of the problems express in the thread would go away if the schema
> was written down and discussed. The rule going forward would be no new
> nodes that aren't part of the standard schema.

So this is why I'm seeing patches just a short time ago removing existing
compatible strings from the DT descriptions and associated driver, and
replacing them with new ones... meaning that the old DT files won't work
with newer kernels.

What that means is using the descriptions as the schema won't catch that
because they're changing those as well to match.

There's a solution to that: dtc becomes a separate project external to
the kernel which also contains the schemas that it verifies against.
That way, if you want to make changes such as that above, you need to
get it past not only kernel people but also past dtc maintainers -
which increases the chances of such stuff being caught.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Fri, Jul 26, 2013 at 10:14:32AM -0400, jonsmirl at gmail.com wrote:
> Yes, yes - that's why the schema should be written down and used as a
> validation input to dtc. Then dtc can spit out errors for non-standard
> items. There would be two versions - the standard one and a legacy one
> that includes the standard one plus the hacks that can't be undone.
> 
> But more importantly it provides a framework for people creating new
> node definitions. Now they can't work in a vacuum and come up with
> random names and structure for everything.
> 
> Most of the problems express in the thread would go away if the schema
> was written down and discussed. The rule going forward would be no new
> nodes that aren't part of the standard schema.

So this is why I'm seeing patches just a short time ago removing existing
compatible strings from the DT descriptions and associated driver, and
replacing them with new ones... meaning that the old DT files won't work
with newer kernels.

What that means is using the descriptions as the schema won't catch that
because they're changing those as well to match.

There's a solution to that: dtc becomes a separate project external to
the kernel which also contains the schemas that it verifies against.
That way, if you want to make changes such as that above, you need to
get it past not only kernel people but also past dtc maintainers -
which increases the chances of such stuff being caught.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Fri, Jul 26, 2013 at 10:21 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Fri, Jul 26, 2013 at 10:14:32AM -0400, jonsmirl@gmail.com wrote:
>> Yes, yes - that's why the schema should be written down and used as a
>> validation input to dtc. Then dtc can spit out errors for non-standard
>> items. There would be two versions - the standard one and a legacy one
>> that includes the standard one plus the hacks that can't be undone.
>>
>> But more importantly it provides a framework for people creating new
>> node definitions. Now they can't work in a vacuum and come up with
>> random names and structure for everything.
>>
>> Most of the problems express in the thread would go away if the schema
>> was written down and discussed. The rule going forward would be no new
>> nodes that aren't part of the standard schema.
>
> So this is why I'm seeing patches just a short time ago removing existing
> compatible strings from the DT descriptions and associated driver, and
> replacing them with new ones... meaning that the old DT files won't work
> with newer kernels.
>
> What that means is using the descriptions as the schema won't catch that
> because they're changing those as well to match.
>
> There's a solution to that: dtc becomes a separate project external to
> the kernel which also contains the schemas that it verifies against.
> That way, if you want to make changes such as that above, you need to
> get it past not only kernel people but also past dtc maintainers -
> which increases the chances of such stuff being caught.

I don't see that it has to be outside of the kernel. It could still be
inside and just have different maintainers.

The real value of having that schema inside of the kernel is so that
people writing new drivers have some rules to look at. It would really
help in stopping 37 different descriptions of DMA controllers that are
all slightly different. You'd have to match you DMA controller
description to the generic form or making a convincing argument as to
why you need the master schema to be changed.



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Fri, Jul 26, 2013 at 10:21 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Fri, Jul 26, 2013 at 10:14:32AM -0400, jonsmirl at gmail.com wrote:
>> Yes, yes - that's why the schema should be written down and used as a
>> validation input to dtc. Then dtc can spit out errors for non-standard
>> items. There would be two versions - the standard one and a legacy one
>> that includes the standard one plus the hacks that can't be undone.
>>
>> But more importantly it provides a framework for people creating new
>> node definitions. Now they can't work in a vacuum and come up with
>> random names and structure for everything.
>>
>> Most of the problems express in the thread would go away if the schema
>> was written down and discussed. The rule going forward would be no new
>> nodes that aren't part of the standard schema.
>
> So this is why I'm seeing patches just a short time ago removing existing
> compatible strings from the DT descriptions and associated driver, and
> replacing them with new ones... meaning that the old DT files won't work
> with newer kernels.
>
> What that means is using the descriptions as the schema won't catch that
> because they're changing those as well to match.
>
> There's a solution to that: dtc becomes a separate project external to
> the kernel which also contains the schemas that it verifies against.
> That way, if you want to make changes such as that above, you need to
> get it past not only kernel people but also past dtc maintainers -
> which increases the chances of such stuff being caught.

I don't see that it has to be outside of the kernel. It could still be
inside and just have different maintainers.

The real value of having that schema inside of the kernel is so that
people writing new drivers have some rules to look at. It would really
help in stopping 37 different descriptions of DMA controllers that are
all slightly different. You'd have to match you DMA controller
description to the generic form or making a convincing argument as to
why you need the master schema to be changed.



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 621 bytes --]

On Fri, Jul 26, 2013 at 03:21:40PM +0100, Russell King - ARM Linux wrote:

> So this is why I'm seeing patches just a short time ago removing existing
> compatible strings from the DT descriptions and associated driver, and
> replacing them with new ones... meaning that the old DT files won't work
> with newer kernels.

The big question is if you're seeing such patches merged.  People making
mistakes on submissions is totally normal.

If it's the case I think you mean TBH I'm not sure anyone cares, I don't
think anyone is using that stuff in production yet as those chips go
almost exclusively into Android phones.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Fri, Jul 26, 2013 at 03:21:40PM +0100, Russell King - ARM Linux wrote:

> So this is why I'm seeing patches just a short time ago removing existing
> compatible strings from the DT descriptions and associated driver, and
> replacing them with new ones... meaning that the old DT files won't work
> with newer kernels.

The big question is if you're seeing such patches merged.  People making
mistakes on submissions is totally normal.

If it's the case I think you mean TBH I'm not sure anyone cares, I don't
think anyone is using that stuff in production yet as those chips go
almost exclusively into Android phones.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130726/235a3d4f/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Olof Johansson]

On Fri, Jul 26, 2013 at 7:10 AM, Mark Brown <broonie@kernel.org> wrote:
> On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
>
>> Unless I totally misunderstood, the thread is talking about letting
>> established bindings change with each new kernel version.  I am
>> opposed to that.
>
> No, nobody is really saying that is a particularly good idea.  There is
> some debate about how we work out what an established binding is but
> there's no serious suggestion that we don't want stable bindings.

Yes, what Mark said -- _today_ all bindings are subject to change and
can be changed in lockstep with the kernel. This has been necessary as
part of development to sort out all of the various bootstrapping
issues across platforms.

What we're talking about is to end that mode of operation, and moving
over to locking in bindings. Device tree contents, as mentioned
elsewhere, might still be changed just like code is -- bugs are fixed,
etc. But it's time to start locking down the bindings, in particular
no longer change the established ones.

Long term, final goal is likely to be close to what Russell is saying
-- nothing should go into the kernel tree unless the binding is in a
fully stable state. However, we have a transitional period between now
and then, and even when we're at the final state there will be need to
have some sort of sandbox for development and test of future bindings.
Dealing with all that, as well as the actual process for locking in
bindings, is what needs to be sorted out.

I think we're all in agreement that bindings that change over time are
nothing but pain, but we're arguing that in circles anyway.


-Olof

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Olof Johansson]

On Fri, Jul 26, 2013 at 7:10 AM, Mark Brown <broonie@kernel.org> wrote:
> On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
>
>> Unless I totally misunderstood, the thread is talking about letting
>> established bindings change with each new kernel version.  I am
>> opposed to that.
>
> No, nobody is really saying that is a particularly good idea.  There is
> some debate about how we work out what an established binding is but
> there's no serious suggestion that we don't want stable bindings.

Yes, what Mark said -- _today_ all bindings are subject to change and
can be changed in lockstep with the kernel. This has been necessary as
part of development to sort out all of the various bootstrapping
issues across platforms.

What we're talking about is to end that mode of operation, and moving
over to locking in bindings. Device tree contents, as mentioned
elsewhere, might still be changed just like code is -- bugs are fixed,
etc. But it's time to start locking down the bindings, in particular
no longer change the established ones.

Long term, final goal is likely to be close to what Russell is saying
-- nothing should go into the kernel tree unless the binding is in a
fully stable state. However, we have a transitional period between now
and then, and even when we're at the final state there will be need to
have some sort of sandbox for development and test of future bindings.
Dealing with all that, as well as the actual process for locking in
bindings, is what needs to be sorted out.

I think we're all in agreement that bindings that change over time are
nothing but pain, but we're arguing that in circles anyway.


-Olof

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On 07/26/2013 10:49 AM, Olof Johansson wrote:
> On Fri, Jul 26, 2013 at 7:10 AM, Mark Brown <broonie@kernel.org> wrote:
>> On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
>>
>>> Unless I totally misunderstood, the thread is talking about letting
>>> established bindings change with each new kernel version.  I am
>>> opposed to that.
>>
>> No, nobody is really saying that is a particularly good idea.  There is
>> some debate about how we work out what an established binding is but
>> there's no serious suggestion that we don't want stable bindings.
> 
> Yes, what Mark said -- _today_ all bindings are subject to change and
> can be changed in lockstep with the kernel. This has been necessary as
> part of development to sort out all of the various bootstrapping
> issues across platforms.

This is absolutely not true on a global basis. Any binding used on
powerpc or sparc is not subject to change. Furthermore, there are ARM
platforms such as highbank where the bindings are expected to be stable.
That's not saying they don't change (new properties for SATA just
today), but they only change in a backwards compatible way.

Rob

> 
> What we're talking about is to end that mode of operation, and moving
> over to locking in bindings. Device tree contents, as mentioned
> elsewhere, might still be changed just like code is -- bugs are fixed,
> etc. But it's time to start locking down the bindings, in particular
> no longer change the established ones.
> 
> Long term, final goal is likely to be close to what Russell is saying
> -- nothing should go into the kernel tree unless the binding is in a
> fully stable state. However, we have a transitional period between now
> and then, and even when we're at the final state there will be need to
> have some sort of sandbox for development and test of future bindings.
> Dealing with all that, as well as the actual process for locking in
> bindings, is what needs to be sorted out.
> 
> I think we're all in agreement that bindings that change over time are
> nothing but pain, but we're arguing that in circles anyway.
> 
> 
> -Olof
> 

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On 07/26/2013 10:49 AM, Olof Johansson wrote:
> On Fri, Jul 26, 2013 at 7:10 AM, Mark Brown <broonie@kernel.org> wrote:
>> On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
>>
>>> Unless I totally misunderstood, the thread is talking about letting
>>> established bindings change with each new kernel version.  I am
>>> opposed to that.
>>
>> No, nobody is really saying that is a particularly good idea.  There is
>> some debate about how we work out what an established binding is but
>> there's no serious suggestion that we don't want stable bindings.
> 
> Yes, what Mark said -- _today_ all bindings are subject to change and
> can be changed in lockstep with the kernel. This has been necessary as
> part of development to sort out all of the various bootstrapping
> issues across platforms.

This is absolutely not true on a global basis. Any binding used on
powerpc or sparc is not subject to change. Furthermore, there are ARM
platforms such as highbank where the bindings are expected to be stable.
That's not saying they don't change (new properties for SATA just
today), but they only change in a backwards compatible way.

Rob

> 
> What we're talking about is to end that mode of operation, and moving
> over to locking in bindings. Device tree contents, as mentioned
> elsewhere, might still be changed just like code is -- bugs are fixed,
> etc. But it's time to start locking down the bindings, in particular
> no longer change the established ones.
> 
> Long term, final goal is likely to be close to what Russell is saying
> -- nothing should go into the kernel tree unless the binding is in a
> fully stable state. However, we have a transitional period between now
> and then, and even when we're at the final state there will be need to
> have some sort of sandbox for development and test of future bindings.
> Dealing with all that, as well as the actual process for locking in
> bindings, is what needs to be sorted out.
> 
> I think we're all in agreement that bindings that change over time are
> nothing but pain, but we're arguing that in circles anyway.
> 
> 
> -Olof
> 

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Fri, Jul 26, 2013 at 06:54:33AM +0200, Richard Cochran wrote:
> On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> 
> > We use DT has a kernel configuration input. Our environment is
> > designed to guarantee 100% that the kernel and DT match exactly. DT
> > very deliberately isn't an ABI boundary in our systems.
> 
> It is nice that you use DT in that way, but that is not how DT is
> supposed to work. If you must keep your DT in sync with the kernel,
> then there is no advantage over the old platfrom device method. At
> least that had the virtue of being a C language interface (ABI), and
> some mistakes were be caught by the compiler.

Over the years we have done it both ways, the platform device/open
coded method was not very flexible and it was hard to get changes into
the kernel. Even something as simple as pre-declaring I2C devices was
(back then) impossible/PITA.

Further, every other kernel release tended to break the board.c files,
just due to the usual kernel churn.

DT is much better, the stuff that can be described in DT is broader
and more thought tends to have been put into DT bindings than was put
into the old C methods. It has less churn, and what churn there is
seems simpler to follow.

Further, I've observed our usages tend to converge on something stable
after a few releases. I think our PPC platform went from 2.6.2x to 3.7
with only very minor DT changes, for instance.

Of course all this could have been done in C, but it wasn't/hasn't been..

> > We've been doing this for years and have a proven in the field track
> > record of upgrades from pre-2.6.16 to 3.7 and beyond with multiple
> > SOCs. The same bootloader that was shipped to support non-DT 2.6.16
> > boots DT 3.7 just fine.
> 
> Try that with Freescale PowerPCs.  Good luck.

Why would Freescale PPC be any different from the IBM PPC we use? We'd
use exactly the same techniques we use on ARM and PPC today and the
churn to the DT wouldn't be an operational problem in the field.

Hint: We don't use uboot on any production systems.

> I too work on commercial embedded systems, and DT has proven to be
> one gigantic *PITA*.

Why do you think our experiences are so different? I have now four
boards on two architectures that have run at least 8 different kernel
versions over 5-6 years (DT and non-DT) and DT has never once been a
problem for my systems. It all just works, and frankly, it works
really well.

[...]

>> We use DT has a kernel configuration input. Our environment is
>> designed to guarantee 100% that the kernel and DT match exactly. DT
>> very deliberately isn't an ABI boundary in our systems.

> Think about what you just said.

*shrug* For embedded I am being *very pragmatic*. I don't need/want an
ABI boundary between the DT and the kernel. I don't need the DT to
statically describe the hardware.

What I need, fundamentally, is a way to describe my board to the
kernel. IMHO, DT is doing that job today better than the old board.c
method.

> The DT describes the *hardware* not the kernel. Why should you ever
> need to update your DT at all?

Well, you know why. The DT schema used by the kernel changes over
time. Kirkwood just went through a massive change in schema in the
past few releases, and the same was true on our PPC platforms when
that was new.

Even if the mainline kernel was perfect, we'd still need to change the
DT schema in our products. Using kirkwood as an example - we have
various parts of the new schemas, and some variations of the
now-mainlined schemas as local patches in our tree to get the
functionality we need. This was all done about 8 months ago - final
upstream versions might ultimately land in 3.12..

I can't delay shipping while upstream sorts this out - so I know
*absolutely* that the DT schema will change. This has been planned for
and designed into the boot system and won't be a problem.

Why would anyone do embedded any other way?

Jason

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Fri, Jul 26, 2013 at 06:54:33AM +0200, Richard Cochran wrote:
> On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> 
> > We use DT has a kernel configuration input. Our environment is
> > designed to guarantee 100% that the kernel and DT match exactly. DT
> > very deliberately isn't an ABI boundary in our systems.
> 
> It is nice that you use DT in that way, but that is not how DT is
> supposed to work. If you must keep your DT in sync with the kernel,
> then there is no advantage over the old platfrom device method. At
> least that had the virtue of being a C language interface (ABI), and
> some mistakes were be caught by the compiler.

Over the years we have done it both ways, the platform device/open
coded method was not very flexible and it was hard to get changes into
the kernel. Even something as simple as pre-declaring I2C devices was
(back then) impossible/PITA.

Further, every other kernel release tended to break the board.c files,
just due to the usual kernel churn.

DT is much better, the stuff that can be described in DT is broader
and more thought tends to have been put into DT bindings than was put
into the old C methods. It has less churn, and what churn there is
seems simpler to follow.

Further, I've observed our usages tend to converge on something stable
after a few releases. I think our PPC platform went from 2.6.2x to 3.7
with only very minor DT changes, for instance.

Of course all this could have been done in C, but it wasn't/hasn't been..

> > We've been doing this for years and have a proven in the field track
> > record of upgrades from pre-2.6.16 to 3.7 and beyond with multiple
> > SOCs. The same bootloader that was shipped to support non-DT 2.6.16
> > boots DT 3.7 just fine.
> 
> Try that with Freescale PowerPCs.  Good luck.

Why would Freescale PPC be any different from the IBM PPC we use? We'd
use exactly the same techniques we use on ARM and PPC today and the
churn to the DT wouldn't be an operational problem in the field.

Hint: We don't use uboot on any production systems.

> I too work on commercial embedded systems, and DT has proven to be
> one gigantic *PITA*.

Why do you think our experiences are so different? I have now four
boards on two architectures that have run at least 8 different kernel
versions over 5-6 years (DT and non-DT) and DT has never once been a
problem for my systems. It all just works, and frankly, it works
really well.

[...]

>> We use DT has a kernel configuration input. Our environment is
>> designed to guarantee 100% that the kernel and DT match exactly. DT
>> very deliberately isn't an ABI boundary in our systems.

> Think about what you just said.

*shrug* For embedded I am being *very pragmatic*. I don't need/want an
ABI boundary between the DT and the kernel. I don't need the DT to
statically describe the hardware.

What I need, fundamentally, is a way to describe my board to the
kernel. IMHO, DT is doing that job today better than the old board.c
method.

> The DT describes the *hardware* not the kernel. Why should you ever
> need to update your DT at all?

Well, you know why. The DT schema used by the kernel changes over
time. Kirkwood just went through a massive change in schema in the
past few releases, and the same was true on our PPC platforms when
that was new.

Even if the mainline kernel was perfect, we'd still need to change the
DT schema in our products. Using kirkwood as an example - we have
various parts of the new schemas, and some variations of the
now-mainlined schemas as local patches in our tree to get the
functionality we need. This was all done about 8 months ago - final
upstream versions might ultimately land in 3.12..

I can't delay shipping while upstream sorts this out - so I know
*absolutely* that the DT schema will change. This has been planned for
and designed into the boot system and won't be a problem.

Why would anyone do embedded any other way?

Jason

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On Thu, Jul 25, 2013 at 7:34 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 07/25/2013 01:16 PM, Rob Herring wrote:
>> On Thu, Jul 25, 2013 at 2:31 PM, Jason Cooper <jason@lakedaemon.net> wrote:
>>> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
>>>> On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
>>>
>>>>> One problem that needs to be solved is obviously how a binding
>>>>> graduates from tentative to locked. This work isn't going to be very
>>>>> interesting to most people, I suspect. Think standards committee type
>>>>> work.
>>>>
>>>> I think a time based stabilization period would be better than a
>>>> separate directory to apply bindings too. Or time plus periodic review
>>>> perhaps.
>>>
>>> The only problem with a time-based versus separate directory is how do
>>> users who've downloaded the tree determine which bindings are stable?
>>> If they pull a tarball, or receive an SDK, there is most likely no git
>>> history attached.
>>
>> Well, if time based includes moving the binding out of the kernel,
>> then that is what defines it as stable or not. I guess that is a form
>> of a separate directory. I don't think we want to be moving bindings
>> twice: tentative -> stable and kernel -> DT repo.
>>
>> The policy could be as simple as an binding without change in at least
>> N kernel releases is moved out and stable.
>
> That might not be quite the right criteria. Just because something
> didn't change doesn't mean it's "correct" and that any problems in the
> binding have been addressed. As one example, on Tegra, we have a few
> bindings that haven't changed in a while, yet rely on custom properties
> for describing which DMA channel to use, rather than using the
> fairly-recently-introduced standard DMA DT properties (this particular
> example is being rectified now, but I'm sure there are plenty of similar
> examples)

In this case, I would suggest that both bindings are stable and the
conversion from one to the other is a platform decision. You could
make the kernel support both if you wanted. Or you know the platform
is not stable and are okay with having to keep dtb and kernel in sync.

I think this highlights the need to separate stable bindings and
stable dts files.

Rob

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On Thu, Jul 25, 2013 at 7:34 PM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 07/25/2013 01:16 PM, Rob Herring wrote:
>> On Thu, Jul 25, 2013 at 2:31 PM, Jason Cooper <jason@lakedaemon.net> wrote:
>>> On Thu, Jul 25, 2013 at 02:11:31PM -0500, Rob Herring wrote:
>>>> On Thu, Jul 25, 2013 at 11:09 AM, Olof Johansson <olof@lixom.net> wrote:
>>>
>>>>> One problem that needs to be solved is obviously how a binding
>>>>> graduates from tentative to locked. This work isn't going to be very
>>>>> interesting to most people, I suspect. Think standards committee type
>>>>> work.
>>>>
>>>> I think a time based stabilization period would be better than a
>>>> separate directory to apply bindings too. Or time plus periodic review
>>>> perhaps.
>>>
>>> The only problem with a time-based versus separate directory is how do
>>> users who've downloaded the tree determine which bindings are stable?
>>> If they pull a tarball, or receive an SDK, there is most likely no git
>>> history attached.
>>
>> Well, if time based includes moving the binding out of the kernel,
>> then that is what defines it as stable or not. I guess that is a form
>> of a separate directory. I don't think we want to be moving bindings
>> twice: tentative -> stable and kernel -> DT repo.
>>
>> The policy could be as simple as an binding without change in at least
>> N kernel releases is moved out and stable.
>
> That might not be quite the right criteria. Just because something
> didn't change doesn't mean it's "correct" and that any problems in the
> binding have been addressed. As one example, on Tegra, we have a few
> bindings that haven't changed in a while, yet rely on custom properties
> for describing which DMA channel to use, rather than using the
> fairly-recently-introduced standard DMA DT properties (this particular
> example is being rectified now, but I'm sure there are plenty of similar
> examples)

In this case, I would suggest that both bindings are stable and the
conversion from one to the other is a platform decision. You could
make the kernel support both if you wanted. Or you know the platform
is not stable and are okay with having to keep dtb and kernel in sync.

I think this highlights the need to separate stable bindings and
stable dts files.

Rob

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Fri, Jul 26, 2013 at 11:36:13AM -0500, Rob Herring wrote:
> On 07/26/2013 10:49 AM, Olof Johansson wrote:
> > On Fri, Jul 26, 2013 at 7:10 AM, Mark Brown <broonie@kernel.org> wrote:
> >> On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> >>
> >>> Unless I totally misunderstood, the thread is talking about letting
> >>> established bindings change with each new kernel version.  I am
> >>> opposed to that.
> >>
> >> No, nobody is really saying that is a particularly good idea.  There is
> >> some debate about how we work out what an established binding is but
> >> there's no serious suggestion that we don't want stable bindings.
> > 
> > Yes, what Mark said -- _today_ all bindings are subject to change and
> > can be changed in lockstep with the kernel. This has been necessary as
> > part of development to sort out all of the various bootstrapping
> > issues across platforms.

This statement is an incredible piece of doublespeak. "Of course we
want stable bindings. That is why 'all bindings are subject to change
and can be changed in lockstep with the kernel.'"

If you want to get away from the DT churn, then you have got to tell
people in no uncertain terms that bindings in a released kernel are a
stable ABI and must be supported into the future.

If you need a playground for new ideas, refactoring platforms, etc,
then go right ahead and create one, but please don't do this with
released kernels.

> This is absolutely not true on a global basis. Any binding used on
> powerpc or sparc is not subject to change. Furthermore, there are ARM
> platforms such as highbank where the bindings are expected to be stable.
> That's not saying they don't change (new properties for SATA just
> today), but they only change in a backwards compatible way.

Right, and lets hope the arm tree can also take this stand.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Fri, Jul 26, 2013 at 11:36:13AM -0500, Rob Herring wrote:
> On 07/26/2013 10:49 AM, Olof Johansson wrote:
> > On Fri, Jul 26, 2013 at 7:10 AM, Mark Brown <broonie@kernel.org> wrote:
> >> On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> >>
> >>> Unless I totally misunderstood, the thread is talking about letting
> >>> established bindings change with each new kernel version.  I am
> >>> opposed to that.
> >>
> >> No, nobody is really saying that is a particularly good idea.  There is
> >> some debate about how we work out what an established binding is but
> >> there's no serious suggestion that we don't want stable bindings.
> > 
> > Yes, what Mark said -- _today_ all bindings are subject to change and
> > can be changed in lockstep with the kernel. This has been necessary as
> > part of development to sort out all of the various bootstrapping
> > issues across platforms.

This statement is an incredible piece of doublespeak. "Of course we
want stable bindings. That is why 'all bindings are subject to change
and can be changed in lockstep with the kernel.'"

If you want to get away from the DT churn, then you have got to tell
people in no uncertain terms that bindings in a released kernel are a
stable ABI and must be supported into the future.

If you need a playground for new ideas, refactoring platforms, etc,
then go right ahead and create one, but please don't do this with
released kernels.

> This is absolutely not true on a global basis. Any binding used on
> powerpc or sparc is not subject to change. Furthermore, there are ARM
> platforms such as highbank where the bindings are expected to be stable.
> That's not saying they don't change (new properties for SATA just
> today), but they only change in a backwards compatible way.

Right, and lets hope the arm tree can also take this stand.

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
> 
> Long term, final goal is likely to be close to what Russell is saying

Why is this a long term goal? Start today.

> -- nothing should go into the kernel tree unless the binding is in a
> fully stable state. However, we have a transitional period between now
> and then, and even when we're at the final state there will be need to
> have some sort of sandbox for development and test of future bindings.

Why not just set up a git tree right away?

> Dealing with all that, as well as the actual process for locking in
> bindings, is what needs to be sorted out.
> 
> I think we're all in agreement that bindings that change over time are
> nothing but pain, but we're arguing that in circles anyway.

No.

I keep saying, the bindings must be stable ABI, *today*.

You keep saying, maybe later, but until then we will make things up as
we go along.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
> 
> Long term, final goal is likely to be close to what Russell is saying

Why is this a long term goal? Start today.

> -- nothing should go into the kernel tree unless the binding is in a
> fully stable state. However, we have a transitional period between now
> and then, and even when we're at the final state there will be need to
> have some sort of sandbox for development and test of future bindings.

Why not just set up a git tree right away?

> Dealing with all that, as well as the actual process for locking in
> bindings, is what needs to be sorted out.
> 
> I think we're all in agreement that bindings that change over time are
> nothing but pain, but we're arguing that in circles anyway.

No.

I keep saying, the bindings must be stable ABI, *today*.

You keep saying, maybe later, but until then we will make things up as
we go along.

Thanks,
Richard

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Fri, Jul 26, 2013 at 11:15:24AM -0600, Jason Gunthorpe wrote:
> 
> Further, every other kernel release tended to break the board.c files,
> just due to the usual kernel churn.

Yes...

> DT is much better, the stuff that can be described in DT is broader
> and more thought tends to have been put into DT bindings than was put
> into the old C methods. It has less churn, and what churn there is
> seems simpler to follow.

...
 
> Of course all this could have been done in C, but it wasn't/hasn't been..

You have identified the real issue: poor quality of the ARM board
support process within the kernel. Churn is still churn, whether in DT
or in platform devices, but the DT people promised to end the churn.

At least with the platform code, I knew where to look to resolve
issues. Thanks to DT, I now have three haystacks to search, namely
boot loader, DT, and kernel.
 
[ I disagree about the "more thought" part. The current discussion,
  coming years too late after the introduction of DT to ARM Linux, is
  contrary evidence enough. ]

> Why would Freescale PPC be any different from the IBM PPC we use? We'd
> use exactly the same techniques we use on ARM and PPC today and the
> churn to the DT wouldn't be an operational problem in the field.

I always suspected that the IBM powerpc Linux code was of higher
quality. Freescale has been just awful.

And that is just the point. No one is saying that DT cannot work. In
theory (and in your experience) it is really wonderful. However, all
this talk about "one day we will offer stable binding on ARM" already
tells me what kind of quality to expect.

> Why do you think our experiences are so different?

Because the "embedded mindset" was used to develop the code on the
platforms that I have been using.

> *shrug* For embedded I am being *very pragmatic*. I don't need/want an
> ABI boundary between the DT and the kernel. I don't need the DT to
> statically describe the hardware.

Yes, I know the kind of quality to expect from the embedded vendors.
But that doesn't mean that mainline Linux should also stink.

> Well, you know why. The DT schema used by the kernel changes over
> time. Kirkwood just went through a massive change in schema in the
> past few releases, and the same was true on our PPC platforms when
> that was new.

I find the very idea to be total unacceptable. This should never
happen in the stable kernel.

> I can't delay shipping while upstream sorts this out - so I know
> *absolutely* that the DT schema will change. This has been planned for
> and designed into the boot system and won't be a problem.
> 
> Why would anyone do embedded any other way?

Yep, that is the embedded way: ship it!

We can do better than that.

Thanks,
Richard

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Fri, Jul 26, 2013 at 11:15:24AM -0600, Jason Gunthorpe wrote:
> 
> Further, every other kernel release tended to break the board.c files,
> just due to the usual kernel churn.

Yes...

> DT is much better, the stuff that can be described in DT is broader
> and more thought tends to have been put into DT bindings than was put
> into the old C methods. It has less churn, and what churn there is
> seems simpler to follow.

...
 
> Of course all this could have been done in C, but it wasn't/hasn't been..

You have identified the real issue: poor quality of the ARM board
support process within the kernel. Churn is still churn, whether in DT
or in platform devices, but the DT people promised to end the churn.

At least with the platform code, I knew where to look to resolve
issues. Thanks to DT, I now have three haystacks to search, namely
boot loader, DT, and kernel.
 
[ I disagree about the "more thought" part. The current discussion,
  coming years too late after the introduction of DT to ARM Linux, is
  contrary evidence enough. ]

> Why would Freescale PPC be any different from the IBM PPC we use? We'd
> use exactly the same techniques we use on ARM and PPC today and the
> churn to the DT wouldn't be an operational problem in the field.

I always suspected that the IBM powerpc Linux code was of higher
quality. Freescale has been just awful.

And that is just the point. No one is saying that DT cannot work. In
theory (and in your experience) it is really wonderful. However, all
this talk about "one day we will offer stable binding on ARM" already
tells me what kind of quality to expect.

> Why do you think our experiences are so different?

Because the "embedded mindset" was used to develop the code on the
platforms that I have been using.

> *shrug* For embedded I am being *very pragmatic*. I don't need/want an
> ABI boundary between the DT and the kernel. I don't need the DT to
> statically describe the hardware.

Yes, I know the kind of quality to expect from the embedded vendors.
But that doesn't mean that mainline Linux should also stink.

> Well, you know why. The DT schema used by the kernel changes over
> time. Kirkwood just went through a massive change in schema in the
> past few releases, and the same was true on our PPC platforms when
> that was new.

I find the very idea to be total unacceptable. This should never
happen in the stable kernel.

> I can't delay shipping while upstream sorts this out - so I know
> *absolutely* that the DT schema will change. This has been planned for
> and designed into the boot system and won't be a problem.
> 
> Why would anyone do embedded any other way?

Yep, that is the embedded way: ship it!

We can do better than that.

Thanks,
Richard

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Fri, Jul 26, 2013 at 11:15:24AM -0600, Jason Gunthorpe wrote:
> On Fri, Jul 26, 2013 at 06:54:33AM +0200, Richard Cochran wrote:
> > I too work on commercial embedded systems, and DT has proven to be
> > one gigantic *PITA*.
> 
> Why do you think our experiences are so different?

Here are a few recent examples:

* What happens when one wants to boot vanilla kernel on the beaglebone?

  http://www.spinics.net/lists/arm-kernel/msg198431.html

* Wanting already merged code to work is too much to ask.

  http://www.spinics.net/lists/linux-omap/msg79731.html

* When people try in good faith to conduct methodical boot tests,
  DT is working against them.

  http://www.spinics.net/lists/linux-omap/msg79960.html

Thanks,
Richard

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Fri, Jul 26, 2013 at 11:15:24AM -0600, Jason Gunthorpe wrote:
> On Fri, Jul 26, 2013 at 06:54:33AM +0200, Richard Cochran wrote:
> > I too work on commercial embedded systems, and DT has proven to be
> > one gigantic *PITA*.
> 
> Why do you think our experiences are so different?

Here are a few recent examples:

* What happens when one wants to boot vanilla kernel on the beaglebone?

  http://www.spinics.net/lists/arm-kernel/msg198431.html

* Wanting already merged code to work is too much to ask.

  http://www.spinics.net/lists/linux-omap/msg79731.html

* When people try in good faith to conduct methodical boot tests,
  DT is working against them.

  http://www.spinics.net/lists/linux-omap/msg79960.html

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Ming Lei]

On Fri, Jul 26, 2013 at 10:14 PM, jonsmirl@gmail.com <jonsmirl@gmail.com> wrote:
>
> Yes, yes - that's why the schema should be written down and used as a
> validation input to dtc. Then dtc can spit out errors for non-standard
> items. There would be two versions - the standard one and a legacy one
> that includes the standard one plus the hacks that can't be undone.
>
> But more importantly it provides a framework for people creating new
> node definitions. Now they can't work in a vacuum and come up with
> random names and structure for everything.
>
> Most of the problems express in the thread would go away if the schema
> was written down and discussed. The rule going forward would be no new
> nodes that aren't part of the standard schema.

+1.

If we want to keep the schema stable, it has to be defined and described
explicitly with one language, just like syscall ABI: C type/API exported to
userspace header file.


Thanks,
-- 
Ming Lei

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Ming Lei]

On Fri, Jul 26, 2013 at 10:14 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
>
> Yes, yes - that's why the schema should be written down and used as a
> validation input to dtc. Then dtc can spit out errors for non-standard
> items. There would be two versions - the standard one and a legacy one
> that includes the standard one plus the hacks that can't be undone.
>
> But more importantly it provides a framework for people creating new
> node definitions. Now they can't work in a vacuum and come up with
> random names and structure for everything.
>
> Most of the problems express in the thread would go away if the schema
> was written down and discussed. The rule going forward would be no new
> nodes that aren't part of the standard schema.

+1.

If we want to keep the schema stable, it has to be defined and described
explicitly with one language, just like syscall ABI: C type/API exported to
userspace header file.


Thanks,
-- 
Ming Lei

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Saturday 27 of July 2013 07:04:08 Richard Cochran wrote:
> On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
> > Long term, final goal is likely to be close to what Russell is saying
> 
> Why is this a long term goal? Start today.
> 
> > -- nothing should go into the kernel tree unless the binding is in a
> > fully stable state. However, we have a transitional period between now
> > and then, and even when we're at the final state there will be need to
> > have some sort of sandbox for development and test of future bindings.
> 
> Why not just set up a git tree right away?
> 
> > Dealing with all that, as well as the actual process for locking in
> > bindings, is what needs to be sorted out.
> > 
> > I think we're all in agreement that bindings that change over time are
> > nothing but pain, but we're arguing that in circles anyway.
> 
> No.
> 
> I keep saying, the bindings must be stable ABI, *today*.
> 
> You keep saying, maybe later, but until then we will make things up as
> we go along.

We have currently a lot of broken bindings, because people didn't know how 
to define ones and those they defined have not been properly reviewed. Do 
you really want such broken ABI in the kernel?

Sure, there are many existing bindings that can be just made stable and 
well they probably are already de facto stable. This is mostly about 
subsystem bindings and whatever already has many users, both made them get 
more thought when designing and more review before merging.

Still, a lot of device and platform-specific bindings are simply broken. 
Take max8925 backlight driver, that Olof started this whole discussion 
with, as an example. We need to sort them out before they can be 
stabilized.

This is the whole point of having distinction between stable and staging 
bindings.

Best regards,
Tomasz

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Saturday 27 of July 2013 07:04:08 Richard Cochran wrote:
> On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
> > Long term, final goal is likely to be close to what Russell is saying
> 
> Why is this a long term goal? Start today.
> 
> > -- nothing should go into the kernel tree unless the binding is in a
> > fully stable state. However, we have a transitional period between now
> > and then, and even when we're at the final state there will be need to
> > have some sort of sandbox for development and test of future bindings.
> 
> Why not just set up a git tree right away?
> 
> > Dealing with all that, as well as the actual process for locking in
> > bindings, is what needs to be sorted out.
> > 
> > I think we're all in agreement that bindings that change over time are
> > nothing but pain, but we're arguing that in circles anyway.
> 
> No.
> 
> I keep saying, the bindings must be stable ABI, *today*.
> 
> You keep saying, maybe later, but until then we will make things up as
> we go along.

We have currently a lot of broken bindings, because people didn't know how 
to define ones and those they defined have not been properly reviewed. Do 
you really want such broken ABI in the kernel?

Sure, there are many existing bindings that can be just made stable and 
well they probably are already de facto stable. This is mostly about 
subsystem bindings and whatever already has many users, both made them get 
more thought when designing and more review before merging.

Still, a lot of device and platform-specific bindings are simply broken. 
Take max8925 backlight driver, that Olof started this whole discussion 
with, as an example. We need to sort them out before they can be 
stabilized.

This is the whole point of having distinction between stable and staging 
bindings.

Best regards,
Tomasz

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Saturday 27 of July 2013 10:53:01 Richard Cochran wrote:
> On Fri, Jul 26, 2013 at 11:15:24AM -0600, Jason Gunthorpe wrote:
> > On Fri, Jul 26, 2013 at 06:54:33AM +0200, Richard Cochran wrote:
> > > I too work on commercial embedded systems, and DT has proven to be
> > > one gigantic *PITA*.
> > 
> > Why do you think our experiences are so different?
> 
> Here are a few recent examples:
> 
> * What happens when one wants to boot vanilla kernel on the beaglebone?
> 
>   http://www.spinics.net/lists/arm-kernel/msg198431.html

Not really understand point of quoting this thread here. Lack of support 
for Beaglebone at that time was not really related to device tree in any 
way, just not enough support for it was merged at that time (as I could 
read here: http://www.spinics.net/lists/arm-kernel/msg199863.html)

> * Wanting already merged code to work is too much to ask.
> 
>   http://www.spinics.net/lists/linux-omap/msg79731.html

This is about not enough synchronization between OMAP people and TI not 
pushing enough and quickly enough to mainline...

> * When people try in good faith to conduct methodical boot tests,
>   DT is working against them.
> 
>   http://www.spinics.net/lists/linux-omap/msg79960.html

I don't really see any relation with DT in this thread. Again this looks 
like support for this platform was not yet developed enough at that time 
in mainline kernel. Still, you managed to boot 3.7-rc2 
(http://thread.gmane.org/gmane.linux.ports.arm.omap/86719/focus=86952), 
didn't you?

I wouldn't really connect all the problems we are having currently with DT 
alone. For many ARM people this is still something new, something that 
they don't fully know how to work with yet. We don't have proper 
processes, examples, best practices, etc. This is what we are trying to 
fix and a key point of all those discussions like this one.

Best regards,
Tomasz

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Saturday 27 of July 2013 10:53:01 Richard Cochran wrote:
> On Fri, Jul 26, 2013 at 11:15:24AM -0600, Jason Gunthorpe wrote:
> > On Fri, Jul 26, 2013 at 06:54:33AM +0200, Richard Cochran wrote:
> > > I too work on commercial embedded systems, and DT has proven to be
> > > one gigantic *PITA*.
> > 
> > Why do you think our experiences are so different?
> 
> Here are a few recent examples:
> 
> * What happens when one wants to boot vanilla kernel on the beaglebone?
> 
>   http://www.spinics.net/lists/arm-kernel/msg198431.html

Not really understand point of quoting this thread here. Lack of support 
for Beaglebone at that time was not really related to device tree in any 
way, just not enough support for it was merged at that time (as I could 
read here: http://www.spinics.net/lists/arm-kernel/msg199863.html)

> * Wanting already merged code to work is too much to ask.
> 
>   http://www.spinics.net/lists/linux-omap/msg79731.html

This is about not enough synchronization between OMAP people and TI not 
pushing enough and quickly enough to mainline...

> * When people try in good faith to conduct methodical boot tests,
>   DT is working against them.
> 
>   http://www.spinics.net/lists/linux-omap/msg79960.html

I don't really see any relation with DT in this thread. Again this looks 
like support for this platform was not yet developed enough at that time 
in mainline kernel. Still, you managed to boot 3.7-rc2 
(http://thread.gmane.org/gmane.linux.ports.arm.omap/86719/focus=86952), 
didn't you?

I wouldn't really connect all the problems we are having currently with DT 
alone. For many ARM people this is still something new, something that 
they don't fully know how to work with yet. We don't have proper 
processes, examples, best practices, etc. This is what we are trying to 
fix and a key point of all those discussions like this one.

Best regards,
Tomasz

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/27/2013 11:51 AM, Tomasz Figa wrote:
> On Saturday 27 of July 2013 07:04:08 Richard Cochran wrote:
>> On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
>>> Long term, final goal is likely to be close to what Russell is saying
>>
>> Why is this a long term goal? Start today.
>>
>>> -- nothing should go into the kernel tree unless the binding is in a
>>> fully stable state. However, we have a transitional period between now
>>> and then, and even when we're at the final state there will be need to
>>> have some sort of sandbox for development and test of future bindings.
>>
>> Why not just set up a git tree right away?
>>
>>> Dealing with all that, as well as the actual process for locking in
>>> bindings, is what needs to be sorted out.
>>>
>>> I think we're all in agreement that bindings that change over time are
>>> nothing but pain, but we're arguing that in circles anyway.
>>
>> No.
>>
>> I keep saying, the bindings must be stable ABI, *today*.
>>
>> You keep saying, maybe later, but until then we will make things up as
>> we go along.
>
> We have currently a lot of broken bindings, because people didn't know how
> to define ones and those they defined have not been properly reviewed. Do
> you really want such broken ABI in the kernel?
>
> Sure, there are many existing bindings that can be just made stable and
> well they probably are already de facto stable. This is mostly about
> subsystem bindings and whatever already has many users, both made them get
> more thought when designing and more review before merging.
>
> Still, a lot of device and platform-specific bindings are simply broken.
> Take max8925 backlight driver, that Olof started this whole discussion
> with, as an example. We need to sort them out before they can be
> stabilized.

That is a nice summary of how we got from null to now and Richard seems 
to be simply saying: let's stop mucking about and make this a project 
with a well-defined process of dealing with staging and stable bindings 
and keep stable bindings stable. Whether it should be within the kernel 
repo as a separate subsystem or in an entire different repo is a trivial 
decision, but still a decision that needs to be made.

Apart from stable DT bindings I would love to see a DT compiler that 
that next to DT syntax detects mistakes in properties used for the 
selfish reason that I spent hours debugging regulator code, because I 
typed vmmc_supply iso vmmc-supply. I did not go through all the 
bindings, but this may require a more formal description so it could be 
compiled/read in the DT compiler.

Regards,
Arend

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/27/2013 11:51 AM, Tomasz Figa wrote:
> On Saturday 27 of July 2013 07:04:08 Richard Cochran wrote:
>> On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
>>> Long term, final goal is likely to be close to what Russell is saying
>>
>> Why is this a long term goal? Start today.
>>
>>> -- nothing should go into the kernel tree unless the binding is in a
>>> fully stable state. However, we have a transitional period between now
>>> and then, and even when we're at the final state there will be need to
>>> have some sort of sandbox for development and test of future bindings.
>>
>> Why not just set up a git tree right away?
>>
>>> Dealing with all that, as well as the actual process for locking in
>>> bindings, is what needs to be sorted out.
>>>
>>> I think we're all in agreement that bindings that change over time are
>>> nothing but pain, but we're arguing that in circles anyway.
>>
>> No.
>>
>> I keep saying, the bindings must be stable ABI, *today*.
>>
>> You keep saying, maybe later, but until then we will make things up as
>> we go along.
>
> We have currently a lot of broken bindings, because people didn't know how
> to define ones and those they defined have not been properly reviewed. Do
> you really want such broken ABI in the kernel?
>
> Sure, there are many existing bindings that can be just made stable and
> well they probably are already de facto stable. This is mostly about
> subsystem bindings and whatever already has many users, both made them get
> more thought when designing and more review before merging.
>
> Still, a lot of device and platform-specific bindings are simply broken.
> Take max8925 backlight driver, that Olof started this whole discussion
> with, as an example. We need to sort them out before they can be
> stabilized.

That is a nice summary of how we got from null to now and Richard seems 
to be simply saying: let's stop mucking about and make this a project 
with a well-defined process of dealing with staging and stable bindings 
and keep stable bindings stable. Whether it should be within the kernel 
repo as a separate subsystem or in an entire different repo is a trivial 
decision, but still a decision that needs to be made.

Apart from stable DT bindings I would love to see a DT compiler that 
that next to DT syntax detects mistakes in properties used for the 
selfish reason that I spent hours debugging regulator code, because I 
typed vmmc_supply iso vmmc-supply. I did not go through all the 
bindings, but this may require a more formal description so it could be 
compiled/read in the DT compiler.

Regards,
Arend

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/27/2013 12:24 PM, Arend van Spriel wrote:
> On 07/27/2013 11:51 AM, Tomasz Figa wrote:
>> On Saturday 27 of July 2013 07:04:08 Richard Cochran wrote:
>>> On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
>>>> Long term, final goal is likely to be close to what Russell is saying
>>>
>>> Why is this a long term goal? Start today.
>>>
>>>> -- nothing should go into the kernel tree unless the binding is in a
>>>> fully stable state. However, we have a transitional period between now
>>>> and then, and even when we're at the final state there will be need to
>>>> have some sort of sandbox for development and test of future bindings.
>>>
>>> Why not just set up a git tree right away?
>>>
>>>> Dealing with all that, as well as the actual process for locking in
>>>> bindings, is what needs to be sorted out.
>>>>
>>>> I think we're all in agreement that bindings that change over time are
>>>> nothing but pain, but we're arguing that in circles anyway.
>>>
>>> No.
>>>
>>> I keep saying, the bindings must be stable ABI, *today*.
>>>
>>> You keep saying, maybe later, but until then we will make things up as
>>> we go along.
>>
>> We have currently a lot of broken bindings, because people didn't know
>> how
>> to define ones and those they defined have not been properly reviewed. Do
>> you really want such broken ABI in the kernel?
>>
>> Sure, there are many existing bindings that can be just made stable and
>> well they probably are already de facto stable. This is mostly about
>> subsystem bindings and whatever already has many users, both made them
>> get
>> more thought when designing and more review before merging.
>>
>> Still, a lot of device and platform-specific bindings are simply broken.
>> Take max8925 backlight driver, that Olof started this whole discussion
>> with, as an example. We need to sort them out before they can be
>> stabilized.
>
> That is a nice summary of how we got from null to now and Richard seems
> to be simply saying: let's stop mucking about and make this a project
> with a well-defined process of dealing with staging and stable bindings
> and keep stable bindings stable. Whether it should be within the kernel
> repo as a separate subsystem or in an entire different repo is a trivial
> decision, but still a decision that needs to be made.
>
> Apart from stable DT bindings I would love to see a DT compiler that
> that next to DT syntax detects mistakes in properties used for the
> selfish reason that I spent hours debugging regulator code, because I
> typed vmmc_supply iso vmmc-supply. I did not go through all the
> bindings, but this may require a more formal description so it could be
> compiled/read in the DT compiler.

Oh, and the reason for my tinkering on dts is here:

http://mid.gmane.org/51E7AA24.6080600@broadcom.com

Happily using Pandaboard for my driver testing and than *kaboom*. 
board-omap4panda.c is gone although the device tree does not provide the 
same functionality. Of course, this is not about DT bindings.

Regards,
Arend

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/27/2013 12:24 PM, Arend van Spriel wrote:
> On 07/27/2013 11:51 AM, Tomasz Figa wrote:
>> On Saturday 27 of July 2013 07:04:08 Richard Cochran wrote:
>>> On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
>>>> Long term, final goal is likely to be close to what Russell is saying
>>>
>>> Why is this a long term goal? Start today.
>>>
>>>> -- nothing should go into the kernel tree unless the binding is in a
>>>> fully stable state. However, we have a transitional period between now
>>>> and then, and even when we're at the final state there will be need to
>>>> have some sort of sandbox for development and test of future bindings.
>>>
>>> Why not just set up a git tree right away?
>>>
>>>> Dealing with all that, as well as the actual process for locking in
>>>> bindings, is what needs to be sorted out.
>>>>
>>>> I think we're all in agreement that bindings that change over time are
>>>> nothing but pain, but we're arguing that in circles anyway.
>>>
>>> No.
>>>
>>> I keep saying, the bindings must be stable ABI, *today*.
>>>
>>> You keep saying, maybe later, but until then we will make things up as
>>> we go along.
>>
>> We have currently a lot of broken bindings, because people didn't know
>> how
>> to define ones and those they defined have not been properly reviewed. Do
>> you really want such broken ABI in the kernel?
>>
>> Sure, there are many existing bindings that can be just made stable and
>> well they probably are already de facto stable. This is mostly about
>> subsystem bindings and whatever already has many users, both made them
>> get
>> more thought when designing and more review before merging.
>>
>> Still, a lot of device and platform-specific bindings are simply broken.
>> Take max8925 backlight driver, that Olof started this whole discussion
>> with, as an example. We need to sort them out before they can be
>> stabilized.
>
> That is a nice summary of how we got from null to now and Richard seems
> to be simply saying: let's stop mucking about and make this a project
> with a well-defined process of dealing with staging and stable bindings
> and keep stable bindings stable. Whether it should be within the kernel
> repo as a separate subsystem or in an entire different repo is a trivial
> decision, but still a decision that needs to be made.
>
> Apart from stable DT bindings I would love to see a DT compiler that
> that next to DT syntax detects mistakes in properties used for the
> selfish reason that I spent hours debugging regulator code, because I
> typed vmmc_supply iso vmmc-supply. I did not go through all the
> bindings, but this may require a more formal description so it could be
> compiled/read in the DT compiler.

Oh, and the reason for my tinkering on dts is here:

http://mid.gmane.org/51E7AA24.6080600 at broadcom.com

Happily using Pandaboard for my driver testing and than *kaboom*. 
board-omap4panda.c is gone although the device tree does not provide the 
same functionality. Of course, this is not about DT bindings.

Regards,
Arend

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Saturday 27 of July 2013 12:24:24 Arend van Spriel wrote:
> On 07/27/2013 11:51 AM, Tomasz Figa wrote:
> > On Saturday 27 of July 2013 07:04:08 Richard Cochran wrote:
> >> On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
> >>> Long term, final goal is likely to be close to what Russell is
> >>> saying
> >> 
> >> Why is this a long term goal? Start today.
> >> 
> >>> -- nothing should go into the kernel tree unless the binding is in a
> >>> fully stable state. However, we have a transitional period between
> >>> now
> >>> and then, and even when we're at the final state there will be need
> >>> to
> >>> have some sort of sandbox for development and test of future
> >>> bindings.
> >> 
> >> Why not just set up a git tree right away?
> >> 
> >>> Dealing with all that, as well as the actual process for locking in
> >>> bindings, is what needs to be sorted out.
> >>> 
> >>> I think we're all in agreement that bindings that change over time
> >>> are
> >>> nothing but pain, but we're arguing that in circles anyway.
> >> 
> >> No.
> >> 
> >> I keep saying, the bindings must be stable ABI, *today*.
> >> 
> >> You keep saying, maybe later, but until then we will make things up
> >> as
> >> we go along.
> > 
> > We have currently a lot of broken bindings, because people didn't know
> > how to define ones and those they defined have not been properly
> > reviewed. Do you really want such broken ABI in the kernel?
> > 
> > Sure, there are many existing bindings that can be just made stable
> > and
> > well they probably are already de facto stable. This is mostly about
> > subsystem bindings and whatever already has many users, both made them
> > get more thought when designing and more review before merging.
> > 
> > Still, a lot of device and platform-specific bindings are simply
> > broken. Take max8925 backlight driver, that Olof started this whole
> > discussion with, as an example. We need to sort them out before they
> > can be stabilized.
> 
> That is a nice summary of how we got from null to now and Richard seems
> to be simply saying: let's stop mucking about and make this a project
> with a well-defined process of dealing with staging and stable bindings
> and keep stable bindings stable. Whether it should be within the kernel
> repo as a separate subsystem or in an entire different repo is a trivial
> decision, but still a decision that needs to be made.

Yes, basically that's our current situation.

Still, I would disagree about the decision being trivial, as each choice 
will have further, and likely pretty significant, consequences on binding 
maintenance, submission, review and for dependent things, like drivers or 
platforms using such bindings. This needs to be discussed enough.

> Apart from stable DT bindings I would love to see a DT compiler that
> that next to DT syntax detects mistakes in properties used for the
> selfish reason that I spent hours debugging regulator code, because I
> typed vmmc_supply iso vmmc-supply. I did not go through all the
> bindings, but this may require a more formal description so it could be
> compiled/read in the DT compiler.

This bothered me as well and that's why I'm working on this. I still can't 
get myself to write a very long mail (I'm more a coder than writer...) 
about the whole idea, my proposal of how it could look and problems we 
need to solve, but I'll try better this evening.

Best regards,
Tomasz

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Saturday 27 of July 2013 12:24:24 Arend van Spriel wrote:
> On 07/27/2013 11:51 AM, Tomasz Figa wrote:
> > On Saturday 27 of July 2013 07:04:08 Richard Cochran wrote:
> >> On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
> >>> Long term, final goal is likely to be close to what Russell is
> >>> saying
> >> 
> >> Why is this a long term goal? Start today.
> >> 
> >>> -- nothing should go into the kernel tree unless the binding is in a
> >>> fully stable state. However, we have a transitional period between
> >>> now
> >>> and then, and even when we're at the final state there will be need
> >>> to
> >>> have some sort of sandbox for development and test of future
> >>> bindings.
> >> 
> >> Why not just set up a git tree right away?
> >> 
> >>> Dealing with all that, as well as the actual process for locking in
> >>> bindings, is what needs to be sorted out.
> >>> 
> >>> I think we're all in agreement that bindings that change over time
> >>> are
> >>> nothing but pain, but we're arguing that in circles anyway.
> >> 
> >> No.
> >> 
> >> I keep saying, the bindings must be stable ABI, *today*.
> >> 
> >> You keep saying, maybe later, but until then we will make things up
> >> as
> >> we go along.
> > 
> > We have currently a lot of broken bindings, because people didn't know
> > how to define ones and those they defined have not been properly
> > reviewed. Do you really want such broken ABI in the kernel?
> > 
> > Sure, there are many existing bindings that can be just made stable
> > and
> > well they probably are already de facto stable. This is mostly about
> > subsystem bindings and whatever already has many users, both made them
> > get more thought when designing and more review before merging.
> > 
> > Still, a lot of device and platform-specific bindings are simply
> > broken. Take max8925 backlight driver, that Olof started this whole
> > discussion with, as an example. We need to sort them out before they
> > can be stabilized.
> 
> That is a nice summary of how we got from null to now and Richard seems
> to be simply saying: let's stop mucking about and make this a project
> with a well-defined process of dealing with staging and stable bindings
> and keep stable bindings stable. Whether it should be within the kernel
> repo as a separate subsystem or in an entire different repo is a trivial
> decision, but still a decision that needs to be made.

Yes, basically that's our current situation.

Still, I would disagree about the decision being trivial, as each choice 
will have further, and likely pretty significant, consequences on binding 
maintenance, submission, review and for dependent things, like drivers or 
platforms using such bindings. This needs to be discussed enough.

> Apart from stable DT bindings I would love to see a DT compiler that
> that next to DT syntax detects mistakes in properties used for the
> selfish reason that I spent hours debugging regulator code, because I
> typed vmmc_supply iso vmmc-supply. I did not go through all the
> bindings, but this may require a more formal description so it could be
> compiled/read in the DT compiler.

This bothered me as well and that's why I'm working on this. I still can't 
get myself to write a very long mail (I'm more a coder than writer...) 
about the whole idea, my proposal of how it could look and problems we 
need to solve, but I'll try better this evening.

Best regards,
Tomasz

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

On Sat, Jul 27, 2013 at 10:48:26AM +0200, Richard Cochran wrote:

> [ I disagree about the "more thought" part. The current discussion,
>   coming years too late after the introduction of DT to ARM Linux, is
>   contrary evidence enough. ]

We did have exactly the same discussion when the DT transition was
started - this isn't something that people only just realised might be
an issue.  There was a deliberate decision to focus on getting the
technology deployed to the point where it could be used as a straight
replacement for board files and accept that sometimes the results won't
be perfect and that we may need to rework as a result.

To be clear the idea has always been to try for bindings that can be
stable but we didn't have the process to guarantee that and there were
substantial barriers in place to creating it while there was no critical
mass behind adoption.  What people are saying now is that we're now at a
point where it looks like those barriers can be overcome.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Sat, Jul 27, 2013 at 10:48:26AM +0200, Richard Cochran wrote:

> [ I disagree about the "more thought" part. The current discussion,
>   coming years too late after the introduction of DT to ARM Linux, is
>   contrary evidence enough. ]

We did have exactly the same discussion when the DT transition was
started - this isn't something that people only just realised might be
an issue.  There was a deliberate decision to focus on getting the
technology deployed to the point where it could be used as a straight
replacement for board files and accept that sometimes the results won't
be perfect and that we may need to rework as a result.

To be clear the idea has always been to try for bindings that can be
stable but we didn't have the process to guarantee that and there were
substantial barriers in place to creating it while there was no critical
mass behind adoption.  What people are saying now is that we're now at a
point where it looks like those barriers can be overcome.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130727/8d414b08/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/27/2013 12:36 PM, Tomasz Figa wrote:
> On Saturday 27 of July 2013 12:24:24 Arend van Spriel wrote:
>> On 07/27/2013 11:51 AM, Tomasz Figa wrote:
>>> On Saturday 27 of July 2013 07:04:08 Richard Cochran wrote:
>>>> On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
>>>>> Long term, final goal is likely to be close to what Russell is
>>>>> saying
>>>>
>>>> Why is this a long term goal? Start today.
>>>>
>>>>> -- nothing should go into the kernel tree unless the binding is in a
>>>>> fully stable state. However, we have a transitional period between
>>>>> now
>>>>> and then, and even when we're at the final state there will be need
>>>>> to
>>>>> have some sort of sandbox for development and test of future
>>>>> bindings.
>>>>
>>>> Why not just set up a git tree right away?
>>>>
>>>>> Dealing with all that, as well as the actual process for locking in
>>>>> bindings, is what needs to be sorted out.
>>>>>
>>>>> I think we're all in agreement that bindings that change over time
>>>>> are
>>>>> nothing but pain, but we're arguing that in circles anyway.
>>>>
>>>> No.
>>>>
>>>> I keep saying, the bindings must be stable ABI, *today*.
>>>>
>>>> You keep saying, maybe later, but until then we will make things up
>>>> as
>>>> we go along.
>>>
>>> We have currently a lot of broken bindings, because people didn't know
>>> how to define ones and those they defined have not been properly
>>> reviewed. Do you really want such broken ABI in the kernel?
>>>
>>> Sure, there are many existing bindings that can be just made stable
>>> and
>>> well they probably are already de facto stable. This is mostly about
>>> subsystem bindings and whatever already has many users, both made them
>>> get more thought when designing and more review before merging.
>>>
>>> Still, a lot of device and platform-specific bindings are simply
>>> broken. Take max8925 backlight driver, that Olof started this whole
>>> discussion with, as an example. We need to sort them out before they
>>> can be stabilized.
>>
>> That is a nice summary of how we got from null to now and Richard seems
>> to be simply saying: let's stop mucking about and make this a project
>> with a well-defined process of dealing with staging and stable bindings
>> and keep stable bindings stable. Whether it should be within the kernel
>> repo as a separate subsystem or in an entire different repo is a trivial
>> decision, but still a decision that needs to be made.
>
> Yes, basically that's our current situation.
>
> Still, I would disagree about the decision being trivial, as each choice
> will have further, and likely pretty significant, consequences on binding
> maintenance, submission, review and for dependent things, like drivers or
> platforms using such bindings. This needs to be discussed enough.
>
>> Apart from stable DT bindings I would love to see a DT compiler that
>> that next to DT syntax detects mistakes in properties used for the
>> selfish reason that I spent hours debugging regulator code, because I
>> typed vmmc_supply iso vmmc-supply. I did not go through all the
>> bindings, but this may require a more formal description so it could be
>> compiled/read in the DT compiler.
>
> This bothered me as well and that's why I'm working on this. I still can't
> get myself to write a very long mail (I'm more a coder than writer...)
> about the whole idea, my proposal of how it could look and problems we
> need to solve, but I'll try better this evening.

Let's see how many people go and scream if I say this: Too bad .dts 
files are not done using XML format as DT bindings could be described 
using XML Schema.

Regards,
Arend

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/27/2013 12:36 PM, Tomasz Figa wrote:
> On Saturday 27 of July 2013 12:24:24 Arend van Spriel wrote:
>> On 07/27/2013 11:51 AM, Tomasz Figa wrote:
>>> On Saturday 27 of July 2013 07:04:08 Richard Cochran wrote:
>>>> On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
>>>>> Long term, final goal is likely to be close to what Russell is
>>>>> saying
>>>>
>>>> Why is this a long term goal? Start today.
>>>>
>>>>> -- nothing should go into the kernel tree unless the binding is in a
>>>>> fully stable state. However, we have a transitional period between
>>>>> now
>>>>> and then, and even when we're at the final state there will be need
>>>>> to
>>>>> have some sort of sandbox for development and test of future
>>>>> bindings.
>>>>
>>>> Why not just set up a git tree right away?
>>>>
>>>>> Dealing with all that, as well as the actual process for locking in
>>>>> bindings, is what needs to be sorted out.
>>>>>
>>>>> I think we're all in agreement that bindings that change over time
>>>>> are
>>>>> nothing but pain, but we're arguing that in circles anyway.
>>>>
>>>> No.
>>>>
>>>> I keep saying, the bindings must be stable ABI, *today*.
>>>>
>>>> You keep saying, maybe later, but until then we will make things up
>>>> as
>>>> we go along.
>>>
>>> We have currently a lot of broken bindings, because people didn't know
>>> how to define ones and those they defined have not been properly
>>> reviewed. Do you really want such broken ABI in the kernel?
>>>
>>> Sure, there are many existing bindings that can be just made stable
>>> and
>>> well they probably are already de facto stable. This is mostly about
>>> subsystem bindings and whatever already has many users, both made them
>>> get more thought when designing and more review before merging.
>>>
>>> Still, a lot of device and platform-specific bindings are simply
>>> broken. Take max8925 backlight driver, that Olof started this whole
>>> discussion with, as an example. We need to sort them out before they
>>> can be stabilized.
>>
>> That is a nice summary of how we got from null to now and Richard seems
>> to be simply saying: let's stop mucking about and make this a project
>> with a well-defined process of dealing with staging and stable bindings
>> and keep stable bindings stable. Whether it should be within the kernel
>> repo as a separate subsystem or in an entire different repo is a trivial
>> decision, but still a decision that needs to be made.
>
> Yes, basically that's our current situation.
>
> Still, I would disagree about the decision being trivial, as each choice
> will have further, and likely pretty significant, consequences on binding
> maintenance, submission, review and for dependent things, like drivers or
> platforms using such bindings. This needs to be discussed enough.
>
>> Apart from stable DT bindings I would love to see a DT compiler that
>> that next to DT syntax detects mistakes in properties used for the
>> selfish reason that I spent hours debugging regulator code, because I
>> typed vmmc_supply iso vmmc-supply. I did not go through all the
>> bindings, but this may require a more formal description so it could be
>> compiled/read in the DT compiler.
>
> This bothered me as well and that's why I'm working on this. I still can't
> get myself to write a very long mail (I'm more a coder than writer...)
> about the whole idea, my proposal of how it could look and problems we
> need to solve, but I'll try better this evening.

Let's see how many people go and scream if I say this: Too bad .dts 
files are not done using XML format as DT bindings could be described 
using XML Schema.

Regards,
Arend

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Sat, Jul 27, 2013 at 12:34:32PM +0200, Arend van Spriel wrote:
> Oh, and the reason for my tinkering on dts is here:
>
> http://mid.gmane.org/51E7AA24.6080600@broadcom.com
>
> Happily using Pandaboard for my driver testing and than *kaboom*.  
> board-omap4panda.c is gone although the device tree does not provide the  
> same functionality. Of course, this is not about DT bindings.

Similar happened to the 4430SDP board - and I took it out of the build
system for over a month because of that.  I knew it would be _painful_
to get DT and running on the board, and I was right, not only because of
the different build process, but also trying to find which kernel config
options need to be added to a previously working configuration for what
is essentially a black box board.

The MMC device for the rootfs magically changed too - which meant finding
out how to tell the kernel which device to use as the rootfs without
giving it an explicit device.  (Using UUIDs, which aren't real UUIDs and
aren't related to the filesystem UUID either - I've still no idea where
this magic number comes from or how to identify it from a running system.)

It's also made a total mess of the boot log.  I've no idea if all the
devices were successfully probed because there's now soo much noise from
the deferred probing that it's like trying to work out if there's any
serious warnings in a kernel build where every file spits out a warning.
It's almost a pen and paper job to work it out.  There's just too much
noise in the kernel boot now that I'm just not going to bother reading
the boot log anymore unless there's something which has noticably failed.

On the plus side, the DT description for the SDP board _may_ have helped
to solve a problem I've had for over two years: that is why the LCDs don't
seem to work on the board.  It seems I need the TWL PWM driver.  I don't
yet know if this has solved the problem or not, because I've not seen the
board boot with that driver in it.  However, it's not clear to me from the
DT description whether that covers both of the LCDs or just one though.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Sat, Jul 27, 2013 at 12:34:32PM +0200, Arend van Spriel wrote:
> Oh, and the reason for my tinkering on dts is here:
>
> http://mid.gmane.org/51E7AA24.6080600 at broadcom.com
>
> Happily using Pandaboard for my driver testing and than *kaboom*.  
> board-omap4panda.c is gone although the device tree does not provide the  
> same functionality. Of course, this is not about DT bindings.

Similar happened to the 4430SDP board - and I took it out of the build
system for over a month because of that.  I knew it would be _painful_
to get DT and running on the board, and I was right, not only because of
the different build process, but also trying to find which kernel config
options need to be added to a previously working configuration for what
is essentially a black box board.

The MMC device for the rootfs magically changed too - which meant finding
out how to tell the kernel which device to use as the rootfs without
giving it an explicit device.  (Using UUIDs, which aren't real UUIDs and
aren't related to the filesystem UUID either - I've still no idea where
this magic number comes from or how to identify it from a running system.)

It's also made a total mess of the boot log.  I've no idea if all the
devices were successfully probed because there's now soo much noise from
the deferred probing that it's like trying to work out if there's any
serious warnings in a kernel build where every file spits out a warning.
It's almost a pen and paper job to work it out.  There's just too much
noise in the kernel boot now that I'm just not going to bother reading
the boot log anymore unless there's something which has noticably failed.

On the plus side, the DT description for the SDP board _may_ have helped
to solve a problem I've had for over two years: that is why the LCDs don't
seem to work on the board.  It seems I need the TWL PWM driver.  I don't
yet know if this has solved the problem or not, because I've not seen the
board boot with that driver in it.  However, it's not clear to me from the
DT description whether that covers both of the LCDs or just one though.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2066 bytes --]

On Sat, Jul 27, 2013 at 10:53:01AM +0200, Richard Cochran wrote:
> On Fri, Jul 26, 2013 at 11:15:24AM -0600, Jason Gunthorpe wrote:

> > Why do you think our experiences are so different?

> Here are a few recent examples:

OK, let's go through these...

> * What happens when one wants to boot vanilla kernel on the beaglebone?

>   http://www.spinics.net/lists/arm-kernel/msg198431.html

This actually sounds pretty good - glancing over the thread it seems you
were trying to boot a shiny new board that people were in the process of
trying to upstream support for and were just a bit too early.  Device
tree doesn't seem to have made a difference either way here.

> * Wanting already merged code to work is too much to ask.

>   http://www.spinics.net/lists/linux-omap/msg79731.html

Paul's reply here seems fairly clear - someone had merged a driver which
had been developed in an out of tree or pre-DT environment without DT
support so they just hadn't added DT support.  Sadly doing that is new
feature development and so not appropriate for the stabalisation phase
of development.

> * When people try in good faith to conduct methodical boot tests,
>   DT is working against them.

>   http://www.spinics.net/lists/linux-omap/msg79960.html

Again I don't see anything particularly related to DT here but instead
do with using a SoC and board that are in the early phases of mainline
integration.

I think what you're seeing here is not to do with DT but rather with the
way most SoC vendors engage with mainline - typically they do their SoC
bringup out of tree and then if things do get submitted to mainline that
happens after the SoC is released.  In this respect things seem to have
been going relatively well here, there were clearly active efforts to
get things integrated and facilitate mainline use.

It'd be much better if SoC vendors were to change the way they engage
with the kernel so that support was already in mainline at about the
time the SoC was released but that's a bigger commercial discussion
which isn't really relevant to this one.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Sat, Jul 27, 2013 at 10:53:01AM +0200, Richard Cochran wrote:
> On Fri, Jul 26, 2013 at 11:15:24AM -0600, Jason Gunthorpe wrote:

> > Why do you think our experiences are so different?

> Here are a few recent examples:

OK, let's go through these...

> * What happens when one wants to boot vanilla kernel on the beaglebone?

>   http://www.spinics.net/lists/arm-kernel/msg198431.html

This actually sounds pretty good - glancing over the thread it seems you
were trying to boot a shiny new board that people were in the process of
trying to upstream support for and were just a bit too early.  Device
tree doesn't seem to have made a difference either way here.

> * Wanting already merged code to work is too much to ask.

>   http://www.spinics.net/lists/linux-omap/msg79731.html

Paul's reply here seems fairly clear - someone had merged a driver which
had been developed in an out of tree or pre-DT environment without DT
support so they just hadn't added DT support.  Sadly doing that is new
feature development and so not appropriate for the stabalisation phase
of development.

> * When people try in good faith to conduct methodical boot tests,
>   DT is working against them.

>   http://www.spinics.net/lists/linux-omap/msg79960.html

Again I don't see anything particularly related to DT here but instead
do with using a SoC and board that are in the early phases of mainline
integration.

I think what you're seeing here is not to do with DT but rather with the
way most SoC vendors engage with mainline - typically they do their SoC
bringup out of tree and then if things do get submitted to mainline that
happens after the SoC is released.  In this respect things seem to have
been going relatively well here, there were clearly active efforts to
get things integrated and facilitate mainline use.

It'd be much better if SoC vendors were to change the way they engage
with the kernel so that support was already in mainline at about the
time the SoC was released but that's a bigger commercial discussion
which isn't really relevant to this one.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130727/e72bb460/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2153 bytes --]

On Fri, Jul 26, 2013 at 03:21:40PM +0100, Russell King - ARM Linux wrote:
> On Fri, Jul 26, 2013 at 10:14:32AM -0400, jonsmirl@gmail.com wrote:
> > Yes, yes - that's why the schema should be written down and used as a
> > validation input to dtc. Then dtc can spit out errors for non-standard
> > items. There would be two versions - the standard one and a legacy one
> > that includes the standard one plus the hacks that can't be undone.
> > 
> > But more importantly it provides a framework for people creating new
> > node definitions. Now they can't work in a vacuum and come up with
> > random names and structure for everything.
> > 
> > Most of the problems express in the thread would go away if the schema
> > was written down and discussed. The rule going forward would be no new
> > nodes that aren't part of the standard schema.
> 
> So this is why I'm seeing patches just a short time ago removing existing
> compatible strings from the DT descriptions and associated driver, and
> replacing them with new ones... meaning that the old DT files won't work
> with newer kernels.
> 
> What that means is using the descriptions as the schema won't catch that
> because they're changing those as well to match.
> 
> There's a solution to that: dtc becomes a separate project external to
> the kernel which also contains the schemas that it verifies against.

dtc has always been a separate project external to the kernel.  Though
we do have to keep reminding people to send their patches to upstream
first.

It's had minimal support for validation of the tree contents for a
long time.  It has no more than minimal support (including "schemas"
as such for bindings), simply because no-one's implemented it yet.

> That way, if you want to make changes such as that above, you need to
> get it past not only kernel people but also past dtc maintainers -
> which increases the chances of such stuff being caught.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

On Fri, Jul 26, 2013 at 03:21:40PM +0100, Russell King - ARM Linux wrote:
> On Fri, Jul 26, 2013 at 10:14:32AM -0400, jonsmirl at gmail.com wrote:
> > Yes, yes - that's why the schema should be written down and used as a
> > validation input to dtc. Then dtc can spit out errors for non-standard
> > items. There would be two versions - the standard one and a legacy one
> > that includes the standard one plus the hacks that can't be undone.
> > 
> > But more importantly it provides a framework for people creating new
> > node definitions. Now they can't work in a vacuum and come up with
> > random names and structure for everything.
> > 
> > Most of the problems express in the thread would go away if the schema
> > was written down and discussed. The rule going forward would be no new
> > nodes that aren't part of the standard schema.
> 
> So this is why I'm seeing patches just a short time ago removing existing
> compatible strings from the DT descriptions and associated driver, and
> replacing them with new ones... meaning that the old DT files won't work
> with newer kernels.
> 
> What that means is using the descriptions as the schema won't catch that
> because they're changing those as well to match.
> 
> There's a solution to that: dtc becomes a separate project external to
> the kernel which also contains the schemas that it verifies against.

dtc has always been a separate project external to the kernel.  Though
we do have to keep reminding people to send their patches to upstream
first.

It's had minimal support for validation of the tree contents for a
long time.  It has no more than minimal support (including "schemas"
as such for bindings), simply because no-one's implemented it yet.

> That way, if you want to make changes such as that above, you need to
> get it past not only kernel people but also past dtc maintainers -
> which increases the chances of such stuff being caught.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130728/d749095c/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3407 bytes --]

On Fri, Jul 26, 2013 at 02:14:23PM +0100, Russell King - ARM Linux wrote:
> On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> > On Fri, Jul 26, 2013 at 10:42:24AM +0100, David Woodhouse wrote:
> > > On Fri, 2013-07-26 at 10:01 +0200, Richard Cochran wrote:
> > > > On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> > > > > 
> > > > > We use DT has a kernel configuration input. Our environment is
> > > > > designed to guarantee 100% that the kernel and DT match exactly. DT
> > > > > very deliberately isn't an ABI boundary in our systems.
> > > > 
> > > > Think about what you just said.
> > > > 
> > > > The DT describes the *hardware* not the kernel. Why should you ever
> > > > need to update your DT at all?
> > > 
> > > Well, the nodes which describe hardware devices, according to the
> > > bindings which form an ABI contract between DT and drivers, should not
> > > normally change. Although they *can* change, if for example you change
> > > the MAC address and that's stored there. Or you change the PHY you want
> > > it to use. Or something like that. The *ABI* doesn't change, but the
> > > data you express *using* that ABI can change. That's kind of the point.
> > 
> > Unless I totally misunderstood, the thread is talking about letting
> > established bindings change with each new kernel version.  I am
> > opposed to that.
> > 
> > Of course, a user may want to change the values of his MAC addresses,
> > if he needs to. But he should never have to change *how* he specifies
> > those addresses.
> > 
> > So, as long as everyone can agree to the principle that a working DT
> > should remain working after a kernel upgrade, then I'll shut up. In
> > actual fact, this is not the case today, nor was it ever so in the
> > past, AFAICT, but it never hurts to set goals.
> 
> We agree.
> 
> That's precisely why I'm saying that once a binding appears in a -final
> released kernel, it is _stable_ by that very fact.  If there were any
> doubts about it, it should never have been accepted into the kernel tree
> in the first place.
> 
> It's no different from how we treat syscalls.  Once we accept a syscall
> and it's published in a -final kernel, it can't then be altered - it can
> be supplemented by a new syscall with a different interface, but the old
> one remains and keeps its semantics for a great many years.
> 
> There's no reason for DT to be any different in this regard.

I agree, and the fact that we have a bunch of bad bindings in the wild
is a mess which is difficult but necessary to clean up.

Something that helps, is that just as it's possible to support old,
badly-thought out syscalls with a compat shim over newer
implementations, we can in many cases localize handling of bad device
tree bindings to early boot code, so that old bad ideas don't have to
infect every part of every driver.

It would probably be a good idea to implement a DT quirks mechanism,
to be run immediately after unflattening.  That way a driver can just
register a fixup function to patch the DT up to a newer standard,
instead of scattering if (old-style-representation) throughout the
code.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

On Fri, Jul 26, 2013 at 02:14:23PM +0100, Russell King - ARM Linux wrote:
> On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> > On Fri, Jul 26, 2013 at 10:42:24AM +0100, David Woodhouse wrote:
> > > On Fri, 2013-07-26 at 10:01 +0200, Richard Cochran wrote:
> > > > On Thu, Jul 25, 2013 at 03:37:53PM -0600, Jason Gunthorpe wrote:
> > > > > 
> > > > > We use DT has a kernel configuration input. Our environment is
> > > > > designed to guarantee 100% that the kernel and DT match exactly. DT
> > > > > very deliberately isn't an ABI boundary in our systems.
> > > > 
> > > > Think about what you just said.
> > > > 
> > > > The DT describes the *hardware* not the kernel. Why should you ever
> > > > need to update your DT at all?
> > > 
> > > Well, the nodes which describe hardware devices, according to the
> > > bindings which form an ABI contract between DT and drivers, should not
> > > normally change. Although they *can* change, if for example you change
> > > the MAC address and that's stored there. Or you change the PHY you want
> > > it to use. Or something like that. The *ABI* doesn't change, but the
> > > data you express *using* that ABI can change. That's kind of the point.
> > 
> > Unless I totally misunderstood, the thread is talking about letting
> > established bindings change with each new kernel version.  I am
> > opposed to that.
> > 
> > Of course, a user may want to change the values of his MAC addresses,
> > if he needs to. But he should never have to change *how* he specifies
> > those addresses.
> > 
> > So, as long as everyone can agree to the principle that a working DT
> > should remain working after a kernel upgrade, then I'll shut up. In
> > actual fact, this is not the case today, nor was it ever so in the
> > past, AFAICT, but it never hurts to set goals.
> 
> We agree.
> 
> That's precisely why I'm saying that once a binding appears in a -final
> released kernel, it is _stable_ by that very fact.  If there were any
> doubts about it, it should never have been accepted into the kernel tree
> in the first place.
> 
> It's no different from how we treat syscalls.  Once we accept a syscall
> and it's published in a -final kernel, it can't then be altered - it can
> be supplemented by a new syscall with a different interface, but the old
> one remains and keeps its semantics for a great many years.
> 
> There's no reason for DT to be any different in this regard.

I agree, and the fact that we have a bunch of bad bindings in the wild
is a mess which is difficult but necessary to clean up.

Something that helps, is that just as it's possible to support old,
badly-thought out syscalls with a compat shim over newer
implementations, we can in many cases localize handling of bad device
tree bindings to early boot code, so that old bad ideas don't have to
infect every part of every driver.

It would probably be a good idea to implement a DT quirks mechanism,
to be run immediately after unflattening.  That way a driver can just
register a fixup function to patch the DT up to a newer standard,
instead of scattering if (old-style-representation) throughout the
code.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130728/54e30f87/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sat, Jul 27, 2013 at 11:40:18AM +0100, Mark Brown wrote:
> On Sat, Jul 27, 2013 at 10:48:26AM +0200, Richard Cochran wrote:
> 
> > [ I disagree about the "more thought" part. The current discussion,
> >   coming years too late after the introduction of DT to ARM Linux, is
> >   contrary evidence enough. ]
> 
> We did have exactly the same discussion when the DT transition was
> started - this isn't something that people only just realised might be
> an issue.  There was a deliberate decision to focus on getting the
> technology deployed to the point where it could be used as a straight
> replacement for board files and accept that sometimes the results won't
> be perfect and that we may need to rework as a result.

Can you tell a bit more about this decision? When was it made? Who
made it? How was it made public?

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sat, Jul 27, 2013 at 11:40:18AM +0100, Mark Brown wrote:
> On Sat, Jul 27, 2013 at 10:48:26AM +0200, Richard Cochran wrote:
> 
> > [ I disagree about the "more thought" part. The current discussion,
> >   coming years too late after the introduction of DT to ARM Linux, is
> >   contrary evidence enough. ]
> 
> We did have exactly the same discussion when the DT transition was
> started - this isn't something that people only just realised might be
> an issue.  There was a deliberate decision to focus on getting the
> technology deployed to the point where it could be used as a straight
> replacement for board files and accept that sometimes the results won't
> be perfect and that we may need to rework as a result.

Can you tell a bit more about this decision? When was it made? Who
made it? How was it made public?

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Lang]

On Sat, 27 Jul 2013, Richard Cochran wrote:

> On Sat, Jul 27, 2013 at 11:40:18AM +0100, Mark Brown wrote:
>> On Sat, Jul 27, 2013 at 10:48:26AM +0200, Richard Cochran wrote:
>>
>>> [ I disagree about the "more thought" part. The current discussion,
>>>   coming years too late after the introduction of DT to ARM Linux, is
>>>   contrary evidence enough. ]
>>
>> We did have exactly the same discussion when the DT transition was
>> started - this isn't something that people only just realised might be
>> an issue.  There was a deliberate decision to focus on getting the
>> technology deployed to the point where it could be used as a straight
>> replacement for board files and accept that sometimes the results won't
>> be perfect and that we may need to rework as a result.
>
> Can you tell a bit more about this decision? When was it made? Who
> made it? How was it made public?

I remember seeing some of the discussion on linux-kernel at the time. I believe 
there was also a LWN article.

David Lang

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Lang]

On Sat, 27 Jul 2013, Richard Cochran wrote:

> On Sat, Jul 27, 2013 at 11:40:18AM +0100, Mark Brown wrote:
>> On Sat, Jul 27, 2013 at 10:48:26AM +0200, Richard Cochran wrote:
>>
>>> [ I disagree about the "more thought" part. The current discussion,
>>>   coming years too late after the introduction of DT to ARM Linux, is
>>>   contrary evidence enough. ]
>>
>> We did have exactly the same discussion when the DT transition was
>> started - this isn't something that people only just realised might be
>> an issue.  There was a deliberate decision to focus on getting the
>> technology deployed to the point where it could be used as a straight
>> replacement for board files and accept that sometimes the results won't
>> be perfect and that we may need to rework as a result.
>
> Can you tell a bit more about this decision? When was it made? Who
> made it? How was it made public?

I remember seeing some of the discussion on linux-kernel at the time. I believe 
there was also a LWN article.

David Lang

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sat, Jul 27, 2013 at 12:36:02PM +0100, Mark Brown wrote:
> On Sat, Jul 27, 2013 at 10:53:01AM +0200, Richard Cochran wrote:
> > On Fri, Jul 26, 2013 at 11:15:24AM -0600, Jason Gunthorpe wrote:
> 
> > > Why do you think our experiences are so different?
> 
> > Here are a few recent examples:
> 
> OK, let's go through these...

Yes, lets, and remember the question was, why do I say that dealing
with DT is such a PITA.
 
> > * What happens when one wants to boot vanilla kernel on the beaglebone?
> 
> >   http://www.spinics.net/lists/arm-kernel/msg198431.html
> 
> This actually sounds pretty good - glancing over the thread it seems you
> were trying to boot a shiny new board that people were in the process of
> trying to upstream support for and were just a bit too early.  Device
> tree doesn't seem to have made a difference either way here.

Did you miss the part about CONFIG_ARM_APPENDED_DTB?

> > * Wanting already merged code to work is too much to ask.
> 
> >   http://www.spinics.net/lists/linux-omap/msg79731.html
> 
> Paul's reply here seems fairly clear - someone had merged a driver which
> had been developed in an out of tree or pre-DT environment without DT
> support so they just hadn't added DT support.  Sadly doing that is new
> feature development and so not appropriate for the stabalisation phase
> of development.

This is me asking for maintainers to take patches to fix a driver in
version v3.7 where the driver merged in v3.4. 

The patches contain the missing DT part, and the maintainer rejects
them, saying, no new features!

Q: What the heck kind of process is that?
A: DT process.

Seriously, why is it too hard for y'all to insist on merging drivers
only when they are really, truly ready (and don't forget the DT part,
please).

> > * When people try in good faith to conduct methodical boot tests,
> >   DT is working against them.
> 
> >   http://www.spinics.net/lists/linux-omap/msg79960.html
> 
> Again I don't see anything particularly related to DT here but instead
> do with using a SoC and board that are in the early phases of mainline
> integration.

It is ring around the rosie, DT, boot loader, and kernel.

Understandably, Paul doesn't want to upgrade his boot loader. He says
he is "not interested" in testing the boot loader, just the kernel.
And, if you follow the sage of Paul's test reports, you will find him
being told to update his boot loader and not to forget the delete old
dtb files.

So, like I said, it is a pity PITA.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sat, Jul 27, 2013 at 12:36:02PM +0100, Mark Brown wrote:
> On Sat, Jul 27, 2013 at 10:53:01AM +0200, Richard Cochran wrote:
> > On Fri, Jul 26, 2013 at 11:15:24AM -0600, Jason Gunthorpe wrote:
> 
> > > Why do you think our experiences are so different?
> 
> > Here are a few recent examples:
> 
> OK, let's go through these...

Yes, lets, and remember the question was, why do I say that dealing
with DT is such a PITA.
 
> > * What happens when one wants to boot vanilla kernel on the beaglebone?
> 
> >   http://www.spinics.net/lists/arm-kernel/msg198431.html
> 
> This actually sounds pretty good - glancing over the thread it seems you
> were trying to boot a shiny new board that people were in the process of
> trying to upstream support for and were just a bit too early.  Device
> tree doesn't seem to have made a difference either way here.

Did you miss the part about CONFIG_ARM_APPENDED_DTB?

> > * Wanting already merged code to work is too much to ask.
> 
> >   http://www.spinics.net/lists/linux-omap/msg79731.html
> 
> Paul's reply here seems fairly clear - someone had merged a driver which
> had been developed in an out of tree or pre-DT environment without DT
> support so they just hadn't added DT support.  Sadly doing that is new
> feature development and so not appropriate for the stabalisation phase
> of development.

This is me asking for maintainers to take patches to fix a driver in
version v3.7 where the driver merged in v3.4. 

The patches contain the missing DT part, and the maintainer rejects
them, saying, no new features!

Q: What the heck kind of process is that?
A: DT process.

Seriously, why is it too hard for y'all to insist on merging drivers
only when they are really, truly ready (and don't forget the DT part,
please).

> > * When people try in good faith to conduct methodical boot tests,
> >   DT is working against them.
> 
> >   http://www.spinics.net/lists/linux-omap/msg79960.html
> 
> Again I don't see anything particularly related to DT here but instead
> do with using a SoC and board that are in the early phases of mainline
> integration.

It is ring around the rosie, DT, boot loader, and kernel.

Understandably, Paul doesn't want to upgrade his boot loader. He says
he is "not interested" in testing the boot loader, just the kernel.
And, if you follow the sage of Paul's test reports, you will find him
being told to update his boot loader and not to forget the delete old
dtb files.

So, like I said, it is a pity PITA.

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sat, Jul 27, 2013 at 10:57:09AM -0700, David Lang wrote:
> On Sat, 27 Jul 2013, Richard Cochran wrote:
> 
> >On Sat, Jul 27, 2013 at 11:40:18AM +0100, Mark Brown wrote:
> >>On Sat, Jul 27, 2013 at 10:48:26AM +0200, Richard Cochran wrote:
> >>
> >>>[ I disagree about the "more thought" part. The current discussion,
> >>>  coming years too late after the introduction of DT to ARM Linux, is
> >>>  contrary evidence enough. ]
> >>
> >>We did have exactly the same discussion when the DT transition was
> >>started - this isn't something that people only just realised might be
> >>an issue.  There was a deliberate decision to focus on getting the
> >>technology deployed to the point where it could be used as a straight
> >>replacement for board files and accept that sometimes the results won't
> >>be perfect and that we may need to rework as a result.
> >
> >Can you tell a bit more about this decision? When was it made? Who
> >made it? How was it made public?
> 
> I remember seeing some of the discussion on linux-kernel at the
> time. I believe there was also a LWN article.

I must have missed it on lkml, although I do try to keep an eye on
this topic. I did find

   http://lwn.net/Articles/414016/
   http://lwn.net/Articles/426606/

but no word about unstable bindings. Maybe this was decided by the
modern method of secret committee?

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sat, Jul 27, 2013 at 10:57:09AM -0700, David Lang wrote:
> On Sat, 27 Jul 2013, Richard Cochran wrote:
> 
> >On Sat, Jul 27, 2013 at 11:40:18AM +0100, Mark Brown wrote:
> >>On Sat, Jul 27, 2013 at 10:48:26AM +0200, Richard Cochran wrote:
> >>
> >>>[ I disagree about the "more thought" part. The current discussion,
> >>>  coming years too late after the introduction of DT to ARM Linux, is
> >>>  contrary evidence enough. ]
> >>
> >>We did have exactly the same discussion when the DT transition was
> >>started - this isn't something that people only just realised might be
> >>an issue.  There was a deliberate decision to focus on getting the
> >>technology deployed to the point where it could be used as a straight
> >>replacement for board files and accept that sometimes the results won't
> >>be perfect and that we may need to rework as a result.
> >
> >Can you tell a bit more about this decision? When was it made? Who
> >made it? How was it made public?
> 
> I remember seeing some of the discussion on linux-kernel at the
> time. I believe there was also a LWN article.

I must have missed it on lkml, although I do try to keep an eye on
this topic. I did find

   http://lwn.net/Articles/414016/
   http://lwn.net/Articles/426606/

but no word about unstable bindings. Maybe this was decided by the
modern method of secret committee?

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sat, Jul 27, 2013 at 12:24:24PM +0200, Arend van Spriel wrote:
> 
> That is a nice summary of how we got from null to now and Richard
> seems to be simply saying: let's stop mucking about and make this a
> project with a well-defined process of dealing with staging and
> stable bindings and keep stable bindings stable.

Yes, that is right.

Frankly, I am really surprised and shocked at the cavalier attitude
expressed here WRT DT bindings in released kernels. Think about the
*users* of this code. Not everyone working with ARM Linux is a kernel
developer or a DT guru. There is really no indication at all that the
ARM Linux DT stuff released so far are not stable and trustworthy.

It is not nice to provide such a mess, and the idea that we *must*
have a mess because the whole system in still in development is bogus,
IMHO. Just make sure that the mainline kernel is really working and
that the DT bindings *there* are for keeps.

It is your job as kernel developers.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sat, Jul 27, 2013 at 12:24:24PM +0200, Arend van Spriel wrote:
> 
> That is a nice summary of how we got from null to now and Richard
> seems to be simply saying: let's stop mucking about and make this a
> project with a well-defined process of dealing with staging and
> stable bindings and keep stable bindings stable.

Yes, that is right.

Frankly, I am really surprised and shocked at the cavalier attitude
expressed here WRT DT bindings in released kernels. Think about the
*users* of this code. Not everyone working with ARM Linux is a kernel
developer or a DT guru. There is really no indication at all that the
ARM Linux DT stuff released so far are not stable and trustworthy.

It is not nice to provide such a mess, and the idea that we *must*
have a mess because the whole system in still in development is bogus,
IMHO. Just make sure that the mainline kernel is really working and
that the DT bindings *there* are for keeps.

It is your job as kernel developers.

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Saturday 27 of July 2013 20:31:01 Richard Cochran wrote:
> On Sat, Jul 27, 2013 at 12:24:24PM +0200, Arend van Spriel wrote:
> > That is a nice summary of how we got from null to now and Richard
> > seems to be simply saying: let's stop mucking about and make this a
> > project with a well-defined process of dealing with staging and
> > stable bindings and keep stable bindings stable.
> 
> Yes, that is right.
> 
> Frankly, I am really surprised and shocked at the cavalier attitude
> expressed here WRT DT bindings in released kernels. Think about the
> *users* of this code. Not everyone working with ARM Linux is a kernel
> developer or a DT guru. There is really no indication at all that the
> ARM Linux DT stuff released so far are not stable and trustworthy.
> 
> It is not nice to provide such a mess, and the idea that we *must*
> have a mess because the whole system in still in development is bogus,
> IMHO. Just make sure that the mainline kernel is really working and
> that the DT bindings *there* are for keeps.
> 
> It is your job as kernel developers.

Well, it depends on how we use the DT. There are (at least) two possible 
usage scenarios:

 a) using DT as direct replacement for board files - this means that you 
    are free to say that DTSes are strictly coupled with kernel version 
    and you are free to modify the bindings - see the analogy to board 
    files, where you could modify the platform data structures and could 
    not directly copy board file from one kernel version to another,

 b) using DT as an ABI - this is the original way, i.e. define stable 
    bindings and make sure that anu DTB built for older kernel will work,
    with equal or greater set of functionality on newer kernels.

Now I believe in this thread the point whether we should use a) or b) or a 
combination of both has been raised.

As for current situation, from users' perspective it's almost no 
difference. With a) they can use appended DTB feature (or hack, whatever 
you prefer). With b) they just upload new zImage/uImage/whatever, leaving 
old DTB as is, if they don't want any new functionality. If there is 
support added for new functionality, they must update their DTBs anyway, 
so there is no clear advantage of b) over a) in this matter.

So, basically, this is not an obvious issue. Without analyzing (and 
discussing) possible use cases, issues, consequences and whatever, there 
is no obvious answer, such as "just stabilize whatever we have now!" or 
"go back to board files!".

We have what we have, it is not perfect, some things have been screwed up, 
but we can't just leave that behind and say "now we'll be doing everything 
correctly", we must fix that up. People don't do everything correctly from 
the start, this is what the whole staging vs stable topic is about.

Best regards,
Tomasz

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Saturday 27 of July 2013 20:31:01 Richard Cochran wrote:
> On Sat, Jul 27, 2013 at 12:24:24PM +0200, Arend van Spriel wrote:
> > That is a nice summary of how we got from null to now and Richard
> > seems to be simply saying: let's stop mucking about and make this a
> > project with a well-defined process of dealing with staging and
> > stable bindings and keep stable bindings stable.
> 
> Yes, that is right.
> 
> Frankly, I am really surprised and shocked at the cavalier attitude
> expressed here WRT DT bindings in released kernels. Think about the
> *users* of this code. Not everyone working with ARM Linux is a kernel
> developer or a DT guru. There is really no indication at all that the
> ARM Linux DT stuff released so far are not stable and trustworthy.
> 
> It is not nice to provide such a mess, and the idea that we *must*
> have a mess because the whole system in still in development is bogus,
> IMHO. Just make sure that the mainline kernel is really working and
> that the DT bindings *there* are for keeps.
> 
> It is your job as kernel developers.

Well, it depends on how we use the DT. There are (at least) two possible 
usage scenarios:

 a) using DT as direct replacement for board files - this means that you 
    are free to say that DTSes are strictly coupled with kernel version 
    and you are free to modify the bindings - see the analogy to board 
    files, where you could modify the platform data structures and could 
    not directly copy board file from one kernel version to another,

 b) using DT as an ABI - this is the original way, i.e. define stable 
    bindings and make sure that anu DTB built for older kernel will work,
    with equal or greater set of functionality on newer kernels.

Now I believe in this thread the point whether we should use a) or b) or a 
combination of both has been raised.

As for current situation, from users' perspective it's almost no 
difference. With a) they can use appended DTB feature (or hack, whatever 
you prefer). With b) they just upload new zImage/uImage/whatever, leaving 
old DTB as is, if they don't want any new functionality. If there is 
support added for new functionality, they must update their DTBs anyway, 
so there is no clear advantage of b) over a) in this matter.

So, basically, this is not an obvious issue. Without analyzing (and 
discussing) possible use cases, issues, consequences and whatever, there 
is no obvious answer, such as "just stabilize whatever we have now!" or 
"go back to board files!".

We have what we have, it is not perfect, some things have been screwed up, 
but we can't just leave that behind and say "now we'll be doing everything 
correctly", we must fix that up. People don't do everything correctly from 
the start, this is what the whole staging vs stable topic is about.

Best regards,
Tomasz

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Sat, Jul 27, 2013 at 2:51 PM, Tomasz Figa <tomasz.figa@gmail.com> wrote:
> On Saturday 27 of July 2013 20:31:01 Richard Cochran wrote:
>> On Sat, Jul 27, 2013 at 12:24:24PM +0200, Arend van Spriel wrote:
>> > That is a nice summary of how we got from null to now and Richard
>> > seems to be simply saying: let's stop mucking about and make this a
>> > project with a well-defined process of dealing with staging and
>> > stable bindings and keep stable bindings stable.
>>
>> Yes, that is right.
>>
>> Frankly, I am really surprised and shocked at the cavalier attitude
>> expressed here WRT DT bindings in released kernels. Think about the
>> *users* of this code. Not everyone working with ARM Linux is a kernel
>> developer or a DT guru. There is really no indication at all that the
>> ARM Linux DT stuff released so far are not stable and trustworthy.
>>
>> It is not nice to provide such a mess, and the idea that we *must*
>> have a mess because the whole system in still in development is bogus,
>> IMHO. Just make sure that the mainline kernel is really working and
>> that the DT bindings *there* are for keeps.
>>
>> It is your job as kernel developers.
>
> Well, it depends on how we use the DT. There are (at least) two possible
> usage scenarios:
>
>  a) using DT as direct replacement for board files - this means that you
>     are free to say that DTSes are strictly coupled with kernel version
>     and you are free to modify the bindings - see the analogy to board
>     files, where you could modify the platform data structures and could
>     not directly copy board file from one kernel version to another,
>
>  b) using DT as an ABI - this is the original way, i.e. define stable
>     bindings and make sure that anu DTB built for older kernel will work,
>     with equal or greater set of functionality on newer kernels.

c) There was very good suggestion earlier about using quirks to fix up
legacy DTBs.  I really like that since the DTB can both be an ABI and
we can continue to evolve the kernel. The quirk code will have little
impact since it will be run at boot time and then discarded.

So there would be an evolving schema that represents the device tree
as understood by a particular kernel.  And then there would be a bunch
of quirks for converting the 'legacy' deployed DTBs into this
tree.These deployed DTBs are only the ones in boot loaders since they
are the only ones that can be seen by multiple kernels.

There are a lot of good places in this process to attach error
checking. For example the quirk code could complain about any DTBs it
is unable to fix up. And verifying things against the schema provides
a single point of contact for device tree design. The universal schema
will force discussion of how to create generic bindings for things
like DMA controllers.

Plus it is still possible to experiment in this model. Bind your DTB
to the kernel. Those DTBs won't be run through the quirk code. You'll
be able to have experimental elements that dtc will flag as not
matching the schema but they'll still be there.

Evolution also works very well. Each release of the kernel will
archive the schema and quirks at the point in time when it is
released.


>
> Now I believe in this thread the point whether we should use a) or b) or a
> combination of both has been raised.
>
> As for current situation, from users' perspective it's almost no
> difference. With a) they can use appended DTB feature (or hack, whatever
> you prefer). With b) they just upload new zImage/uImage/whatever, leaving
> old DTB as is, if they don't want any new functionality. If there is
> support added for new functionality, they must update their DTBs anyway,
> so there is no clear advantage of b) over a) in this matter.
>
> So, basically, this is not an obvious issue. Without analyzing (and
> discussing) possible use cases, issues, consequences and whatever, there
> is no obvious answer, such as "just stabilize whatever we have now!" or
> "go back to board files!".
>
> We have what we have, it is not perfect, some things have been screwed up,
> but we can't just leave that behind and say "now we'll be doing everything
> correctly", we must fix that up. People don't do everything correctly from
> the start, this is what the whole staging vs stable topic is about.
>
> Best regards,
> Tomasz
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Sat, Jul 27, 2013 at 2:51 PM, Tomasz Figa <tomasz.figa@gmail.com> wrote:
> On Saturday 27 of July 2013 20:31:01 Richard Cochran wrote:
>> On Sat, Jul 27, 2013 at 12:24:24PM +0200, Arend van Spriel wrote:
>> > That is a nice summary of how we got from null to now and Richard
>> > seems to be simply saying: let's stop mucking about and make this a
>> > project with a well-defined process of dealing with staging and
>> > stable bindings and keep stable bindings stable.
>>
>> Yes, that is right.
>>
>> Frankly, I am really surprised and shocked at the cavalier attitude
>> expressed here WRT DT bindings in released kernels. Think about the
>> *users* of this code. Not everyone working with ARM Linux is a kernel
>> developer or a DT guru. There is really no indication at all that the
>> ARM Linux DT stuff released so far are not stable and trustworthy.
>>
>> It is not nice to provide such a mess, and the idea that we *must*
>> have a mess because the whole system in still in development is bogus,
>> IMHO. Just make sure that the mainline kernel is really working and
>> that the DT bindings *there* are for keeps.
>>
>> It is your job as kernel developers.
>
> Well, it depends on how we use the DT. There are (at least) two possible
> usage scenarios:
>
>  a) using DT as direct replacement for board files - this means that you
>     are free to say that DTSes are strictly coupled with kernel version
>     and you are free to modify the bindings - see the analogy to board
>     files, where you could modify the platform data structures and could
>     not directly copy board file from one kernel version to another,
>
>  b) using DT as an ABI - this is the original way, i.e. define stable
>     bindings and make sure that anu DTB built for older kernel will work,
>     with equal or greater set of functionality on newer kernels.

c) There was very good suggestion earlier about using quirks to fix up
legacy DTBs.  I really like that since the DTB can both be an ABI and
we can continue to evolve the kernel. The quirk code will have little
impact since it will be run at boot time and then discarded.

So there would be an evolving schema that represents the device tree
as understood by a particular kernel.  And then there would be a bunch
of quirks for converting the 'legacy' deployed DTBs into this
tree.These deployed DTBs are only the ones in boot loaders since they
are the only ones that can be seen by multiple kernels.

There are a lot of good places in this process to attach error
checking. For example the quirk code could complain about any DTBs it
is unable to fix up. And verifying things against the schema provides
a single point of contact for device tree design. The universal schema
will force discussion of how to create generic bindings for things
like DMA controllers.

Plus it is still possible to experiment in this model. Bind your DTB
to the kernel. Those DTBs won't be run through the quirk code. You'll
be able to have experimental elements that dtc will flag as not
matching the schema but they'll still be there.

Evolution also works very well. Each release of the kernel will
archive the schema and quirks at the point in time when it is
released.


>
> Now I believe in this thread the point whether we should use a) or b) or a
> combination of both has been raised.
>
> As for current situation, from users' perspective it's almost no
> difference. With a) they can use appended DTB feature (or hack, whatever
> you prefer). With b) they just upload new zImage/uImage/whatever, leaving
> old DTB as is, if they don't want any new functionality. If there is
> support added for new functionality, they must update their DTBs anyway,
> so there is no clear advantage of b) over a) in this matter.
>
> So, basically, this is not an obvious issue. Without analyzing (and
> discussing) possible use cases, issues, consequences and whatever, there
> is no obvious answer, such as "just stabilize whatever we have now!" or
> "go back to board files!".
>
> We have what we have, it is not perfect, some things have been screwed up,
> but we can't just leave that behind and say "now we'll be doing everything
> correctly", we must fix that up. People don't do everything correctly from
> the start, this is what the whole staging vs stable topic is about.
>
> Best regards,
> Tomasz
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Fri, Jul 26, 2013 at 8:21 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Fri, Jul 26, 2013 at 10:14:32AM -0400, jonsmirl@gmail.com wrote:
>> Yes, yes - that's why the schema should be written down and used as a
>> validation input to dtc. Then dtc can spit out errors for non-standard
>> items. There would be two versions - the standard one and a legacy one
>> that includes the standard one plus the hacks that can't be undone.
>>
>> But more importantly it provides a framework for people creating new
>> node definitions. Now they can't work in a vacuum and come up with
>> random names and structure for everything.
>>
>> Most of the problems express in the thread would go away if the schema
>> was written down and discussed. The rule going forward would be no new
>> nodes that aren't part of the standard schema.
>
> So this is why I'm seeing patches just a short time ago removing existing
> compatible strings from the DT descriptions and associated driver, and
> replacing them with new ones... meaning that the old DT files won't work
> with newer kernels.
>
> What that means is using the descriptions as the schema won't catch that
> because they're changing those as well to match.
>
> There's a solution to that: dtc becomes a separate project external to
> the kernel which also contains the schemas that it verifies against.
> That way, if you want to make changes such as that above, you need to
> get it past not only kernel people but also past dtc maintainers -
> which increases the chances of such stuff being caught.

+1

dtc has always been a separate project that happens to be mirrored in
the kernel tree, but the bindings need to come out be turned into
schema that can be validated.

g.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Fri, Jul 26, 2013 at 8:21 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Fri, Jul 26, 2013 at 10:14:32AM -0400, jonsmirl at gmail.com wrote:
>> Yes, yes - that's why the schema should be written down and used as a
>> validation input to dtc. Then dtc can spit out errors for non-standard
>> items. There would be two versions - the standard one and a legacy one
>> that includes the standard one plus the hacks that can't be undone.
>>
>> But more importantly it provides a framework for people creating new
>> node definitions. Now they can't work in a vacuum and come up with
>> random names and structure for everything.
>>
>> Most of the problems express in the thread would go away if the schema
>> was written down and discussed. The rule going forward would be no new
>> nodes that aren't part of the standard schema.
>
> So this is why I'm seeing patches just a short time ago removing existing
> compatible strings from the DT descriptions and associated driver, and
> replacing them with new ones... meaning that the old DT files won't work
> with newer kernels.
>
> What that means is using the descriptions as the schema won't catch that
> because they're changing those as well to match.
>
> There's a solution to that: dtc becomes a separate project external to
> the kernel which also contains the schemas that it verifies against.
> That way, if you want to make changes such as that above, you need to
> get it past not only kernel people but also past dtc maintainers -
> which increases the chances of such stuff being caught.

+1

dtc has always been a separate project that happens to be mirrored in
the kernel tree, but the bindings need to come out be turned into
schema that can be validated.

g.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
> Let's see how many people go and scream if I say this: Too bad .dts files
> are not done using XML format as DT bindings could be described using XML
> Schema.

Draft an example and show us how it would look!  :-)  There is
absolutely nothing preventing us from expressing a DT in XML format,
or even using XSLT to define DT schema while still using our current
.dts syntax. It would be trivial to do lossless translation between
.dts syntax and xml.

The problem that I have with XML and XSLT is that it is very verbose
and not entirely friendly to mere-mortals. However, I'm more than
willing to be proved wrong on this point.

g.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
> Let's see how many people go and scream if I say this: Too bad .dts files
> are not done using XML format as DT bindings could be described using XML
> Schema.

Draft an example and show us how it would look!  :-)  There is
absolutely nothing preventing us from expressing a DT in XML format,
or even using XSLT to define DT schema while still using our current
.dts syntax. It would be trivial to do lossless translation between
.dts syntax and xml.

The problem that I have with XML and XSLT is that it is very verbose
and not entirely friendly to mere-mortals. However, I'm more than
willing to be proved wrong on this point.

g.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
>> Let's see how many people go and scream if I say this: Too bad .dts files
>> are not done using XML format as DT bindings could be described using XML
>> Schema.
>
> Draft an example and show us how it would look!  :-)  There is
> absolutely nothing preventing us from expressing a DT in XML format,
> or even using XSLT to define DT schema while still using our current
> .dts syntax. It would be trivial to do lossless translation between
> .dts syntax and xml.
>
> The problem that I have with XML and XSLT is that it is very verbose
> and not entirely friendly to mere-mortals. However, I'm more than
> willing to be proved wrong on this point.

I considered this approach a while ago and discarded it. It would work
but it is just too much of a Frankenstein monster.

Much cleaner to modify dtc to take a schema as part of the compilation
process. The schema language itself has no requirement to look like
DTS syntax. Whoever wrote dtc probably has a favorite language that
would be good for writing schemas in.


>
> g.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
>> Let's see how many people go and scream if I say this: Too bad .dts files
>> are not done using XML format as DT bindings could be described using XML
>> Schema.
>
> Draft an example and show us how it would look!  :-)  There is
> absolutely nothing preventing us from expressing a DT in XML format,
> or even using XSLT to define DT schema while still using our current
> .dts syntax. It would be trivial to do lossless translation between
> .dts syntax and xml.
>
> The problem that I have with XML and XSLT is that it is very verbose
> and not entirely friendly to mere-mortals. However, I'm more than
> willing to be proved wrong on this point.

I considered this approach a while ago and discarded it. It would work
but it is just too much of a Frankenstein monster.

Much cleaner to modify dtc to take a schema as part of the compilation
process. The schema language itself has no requirement to look like
DTS syntax. Whoever wrote dtc probably has a favorite language that
would be good for writing schemas in.


>
> g.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

On Sat, 2013-07-27 at 16:01 -0400, jonsmirl@gmail.com wrote:
> 
> Much cleaner to modify dtc to take a schema as part of the compilation
> process. The schema language itself has no requirement to look like
> DTS syntax. Whoever wrote dtc probably has a favorite language that
> would be good for writing schemas in.

Yes, we *definitely* want dtc to be doing the validation when it builds
trees. And warning (and needing an additional and suitably scarily named
command line option) if you try to build something *without* a schema.

-- 
dwmw2


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

On Sat, 2013-07-27 at 16:01 -0400, jonsmirl at gmail.com wrote:
> 
> Much cleaner to modify dtc to take a schema as part of the compilation
> process. The schema language itself has no requirement to look like
> DTS syntax. Whoever wrote dtc probably has a favorite language that
> would be good for writing schemas in.

Yes, we *definitely* want dtc to be doing the validation when it builds
trees. And warning (and needing an additional and suitably scarily named
command line option) if you try to build something *without* a schema.

-- 
dwmw2

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5745 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130727/664eb53a/attachment-0001.bin>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl@gmail.com <jonsmirl@gmail.com> wrote:
> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
>>> Let's see how many people go and scream if I say this: Too bad .dts files
>>> are not done using XML format as DT bindings could be described using XML
>>> Schema.
>>
>> Draft an example and show us how it would look!  :-)  There is
>> absolutely nothing preventing us from expressing a DT in XML format,
>> or even using XSLT to define DT schema while still using our current
>> .dts syntax. It would be trivial to do lossless translation between
>> .dts syntax and xml.
>>
>> The problem that I have with XML and XSLT is that it is very verbose
>> and not entirely friendly to mere-mortals. However, I'm more than
>> willing to be proved wrong on this point.
>
> I considered this approach a while ago and discarded it. It would work
> but it is just too much of a Frankenstein monster.
>
> Much cleaner to modify dtc to take a schema as part of the compilation
> process. The schema language itself has no requirement to look like
> DTS syntax. Whoever wrote dtc probably has a favorite language that
> would be good for writing schemas in.

Making it part of dtc is a required feature as far as I'm concerned.
Using XML/XSLT and dtc-integration are not mutually exclusive, but I
digress.

g.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
>>> Let's see how many people go and scream if I say this: Too bad .dts files
>>> are not done using XML format as DT bindings could be described using XML
>>> Schema.
>>
>> Draft an example and show us how it would look!  :-)  There is
>> absolutely nothing preventing us from expressing a DT in XML format,
>> or even using XSLT to define DT schema while still using our current
>> .dts syntax. It would be trivial to do lossless translation between
>> .dts syntax and xml.
>>
>> The problem that I have with XML and XSLT is that it is very verbose
>> and not entirely friendly to mere-mortals. However, I'm more than
>> willing to be proved wrong on this point.
>
> I considered this approach a while ago and discarded it. It would work
> but it is just too much of a Frankenstein monster.
>
> Much cleaner to modify dtc to take a schema as part of the compilation
> process. The schema language itself has no requirement to look like
> DTS syntax. Whoever wrote dtc probably has a favorite language that
> would be good for writing schemas in.

Making it part of dtc is a required feature as far as I'm concerned.
Using XML/XSLT and dtc-integration are not mutually exclusive, but I
digress.

g.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Sat, Jul 27, 2013 at 2:25 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl@gmail.com <jonsmirl@gmail.com> wrote:
>> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
>>>> Let's see how many people go and scream if I say this: Too bad .dts files
>>>> are not done using XML format as DT bindings could be described using XML
>>>> Schema.
>>>
>>> Draft an example and show us how it would look!  :-)  There is
>>> absolutely nothing preventing us from expressing a DT in XML format,
>>> or even using XSLT to define DT schema while still using our current
>>> .dts syntax. It would be trivial to do lossless translation between
>>> .dts syntax and xml.
>>>
>>> The problem that I have with XML and XSLT is that it is very verbose
>>> and not entirely friendly to mere-mortals. However, I'm more than
>>> willing to be proved wrong on this point.
>>
>> I considered this approach a while ago and discarded it. It would work
>> but it is just too much of a Frankenstein monster.
>>
>> Much cleaner to modify dtc to take a schema as part of the compilation
>> process. The schema language itself has no requirement to look like
>> DTS syntax. Whoever wrote dtc probably has a favorite language that
>> would be good for writing schemas in.
>
> Making it part of dtc is a required feature as far as I'm concerned.
> Using XML/XSLT and dtc-integration are not mutually exclusive, but I
> digress.

Oops, ignore the XSLT bit. XSLT isn't schema and has no bearing on the
discussion of schema. Sorry for the noise.

g.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Sat, Jul 27, 2013 at 2:25 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
>> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
>>>> Let's see how many people go and scream if I say this: Too bad .dts files
>>>> are not done using XML format as DT bindings could be described using XML
>>>> Schema.
>>>
>>> Draft an example and show us how it would look!  :-)  There is
>>> absolutely nothing preventing us from expressing a DT in XML format,
>>> or even using XSLT to define DT schema while still using our current
>>> .dts syntax. It would be trivial to do lossless translation between
>>> .dts syntax and xml.
>>>
>>> The problem that I have with XML and XSLT is that it is very verbose
>>> and not entirely friendly to mere-mortals. However, I'm more than
>>> willing to be proved wrong on this point.
>>
>> I considered this approach a while ago and discarded it. It would work
>> but it is just too much of a Frankenstein monster.
>>
>> Much cleaner to modify dtc to take a schema as part of the compilation
>> process. The schema language itself has no requirement to look like
>> DTS syntax. Whoever wrote dtc probably has a favorite language that
>> would be good for writing schemas in.
>
> Making it part of dtc is a required feature as far as I'm concerned.
> Using XML/XSLT and dtc-integration are not mutually exclusive, but I
> digress.

Oops, ignore the XSLT bit. XSLT isn't schema and has no bearing on the
discussion of schema. Sorry for the noise.

g.

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Thu, 25 Jul 2013 09:09:19 -0700, Olof Johansson <olof@lixom.net> wrote:
> [I'm adding LKML and ksummit-discuss to this thread, since the ACPI/DT
> discussions have been covered there and this overlaps some with that]
> 
> On Thu, Jul 25, 2013 at 7:38 AM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > On Thu, Jul 25, 2013 at 03:27:02PM +0100, Russell King - ARM Linux wrote:
> >> Remember the stated assertion when DT was first added to the kernel: the
> >> DT descriptions _will_ become a separately maintained project which is
> >> independent of the kernel.  They will _not_ be kernel version specific.
> >
> > I'm looking forward to this.
> >
> > Question for the DT guys: what are the plans here? Are we going to get
> > rid of the .dts files inside the kernel tree?
> 
> In the discussions we had in Dublin, a couple of options on how to
> lock in bindings were discussed. I'm a little surprised that Grant
> didn't cover them in his initial emails on the new maintainership
> model, but maybe he wanted the new group to handle it. And they didn't
> bring it up yet either. So I am. :-)

Excellent! My plan to avoid responsibility for stuff is proceeding.
Actually, I think I touched on it, but the details got lost in my head
among all the other stuff we discussed two weeks ago. Thanks for getting
this detail down into an email.

g.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Thu, 25 Jul 2013 09:09:19 -0700, Olof Johansson <olof@lixom.net> wrote:
> [I'm adding LKML and ksummit-discuss to this thread, since the ACPI/DT
> discussions have been covered there and this overlaps some with that]
> 
> On Thu, Jul 25, 2013 at 7:38 AM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > On Thu, Jul 25, 2013 at 03:27:02PM +0100, Russell King - ARM Linux wrote:
> >> Remember the stated assertion when DT was first added to the kernel: the
> >> DT descriptions _will_ become a separately maintained project which is
> >> independent of the kernel.  They will _not_ be kernel version specific.
> >
> > I'm looking forward to this.
> >
> > Question for the DT guys: what are the plans here? Are we going to get
> > rid of the .dts files inside the kernel tree?
> 
> In the discussions we had in Dublin, a couple of options on how to
> lock in bindings were discussed. I'm a little surprised that Grant
> didn't cover them in his initial emails on the new maintainership
> model, but maybe he wanted the new group to handle it. And they didn't
> bring it up yet either. So I am. :-)

Excellent! My plan to avoid responsibility for stuff is proceeding.
Actually, I think I touched on it, but the details got lost in my head
among all the other stuff we discussed two weeks ago. Thanks for getting
this detail down into an email.

g.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[James Bottomley]

On Sat, 2013-07-27 at 21:28 -0600, Grant Likely wrote:
> On Sat, Jul 27, 2013 at 2:25 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> > On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl@gmail.com <jonsmirl@gmail.com> wrote:
> >> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> >>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
> >>>> Let's see how many people go and scream if I say this: Too bad .dts files
> >>>> are not done using XML format as DT bindings could be described using XML
> >>>> Schema.
> >>>
> >>> Draft an example and show us how it would look!  :-)  There is
> >>> absolutely nothing preventing us from expressing a DT in XML format,
> >>> or even using XSLT to define DT schema while still using our current
> >>> .dts syntax. It would be trivial to do lossless translation between
> >>> .dts syntax and xml.
> >>>
> >>> The problem that I have with XML and XSLT is that it is very verbose
> >>> and not entirely friendly to mere-mortals. However, I'm more than
> >>> willing to be proved wrong on this point.
> >>
> >> I considered this approach a while ago and discarded it. It would work
> >> but it is just too much of a Frankenstein monster.
> >>
> >> Much cleaner to modify dtc to take a schema as part of the compilation
> >> process. The schema language itself has no requirement to look like
> >> DTS syntax. Whoever wrote dtc probably has a favorite language that
> >> would be good for writing schemas in.
> >
> > Making it part of dtc is a required feature as far as I'm concerned.
> > Using XML/XSLT and dtc-integration are not mutually exclusive, but I
> > digress.
> 
> Oops, ignore the XSLT bit. XSLT isn't schema and has no bearing on the
> discussion of schema. Sorry for the noise.

XSLT is a transform language ... you'd use it say to transform xml to
dtc, so it would be an integral component of an xml/xslt based schema.

If you want actually to describe and have validated the xml schema
itself, then you'd use xsd (XML schema description language) and its
associated tools.

I'm not saying you *should* do this, just that it's possible (plus I've
just blown my kernel cred by knowing about xml, sigh).

James

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[James Bottomley]

On Sat, 2013-07-27 at 21:28 -0600, Grant Likely wrote:
> On Sat, Jul 27, 2013 at 2:25 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> > On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
> >> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> >>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
> >>>> Let's see how many people go and scream if I say this: Too bad .dts files
> >>>> are not done using XML format as DT bindings could be described using XML
> >>>> Schema.
> >>>
> >>> Draft an example and show us how it would look!  :-)  There is
> >>> absolutely nothing preventing us from expressing a DT in XML format,
> >>> or even using XSLT to define DT schema while still using our current
> >>> .dts syntax. It would be trivial to do lossless translation between
> >>> .dts syntax and xml.
> >>>
> >>> The problem that I have with XML and XSLT is that it is very verbose
> >>> and not entirely friendly to mere-mortals. However, I'm more than
> >>> willing to be proved wrong on this point.
> >>
> >> I considered this approach a while ago and discarded it. It would work
> >> but it is just too much of a Frankenstein monster.
> >>
> >> Much cleaner to modify dtc to take a schema as part of the compilation
> >> process. The schema language itself has no requirement to look like
> >> DTS syntax. Whoever wrote dtc probably has a favorite language that
> >> would be good for writing schemas in.
> >
> > Making it part of dtc is a required feature as far as I'm concerned.
> > Using XML/XSLT and dtc-integration are not mutually exclusive, but I
> > digress.
> 
> Oops, ignore the XSLT bit. XSLT isn't schema and has no bearing on the
> discussion of schema. Sorry for the noise.

XSLT is a transform language ... you'd use it say to transform xml to
dtc, so it would be an integral component of an xml/xslt based schema.

If you want actually to describe and have validated the xml schema
itself, then you'd use xsd (XML schema description language) and its
associated tools.

I'm not saying you *should* do this, just that it's possible (plus I've
just blown my kernel cred by knowing about xml, sigh).

James

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Sat, Jul 27, 2013 at 11:11 PM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> If you want actually to describe and have validated the xml schema
> itself, then you'd use xsd (XML schema description language) and its
> associated tools.
>
> I'm not saying you *should* do this, just that it's possible (plus I've
> just blown my kernel cred by knowing about xml, sigh).

:)

g.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Sat, Jul 27, 2013 at 11:11 PM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> If you want actually to describe and have validated the xml schema
> itself, then you'd use xsd (XML schema description language) and its
> associated tools.
>
> I'm not saying you *should* do this, just that it's possible (plus I've
> just blown my kernel cred by knowing about xml, sigh).

:)

g.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sat, Jul 27, 2013 at 11:51:06AM -0700, Tomasz Figa wrote:
> On Saturday 27 of July 2013 20:31:01 Richard Cochran wrote:
> > 
> > Frankly, I am really surprised and shocked at the cavalier attitude
> > expressed here WRT DT bindings in released kernels. Think about the
> > *users* of this code. Not everyone working with ARM Linux is a kernel
> > developer or a DT guru. There is really no indication at all that the
> > ARM Linux DT stuff released so far are not stable and trustworthy.

Read the above again, please.

> Well, it depends on how we use the DT. There are (at least) two possible 
> usage scenarios:
> 
>  a) using DT as direct replacement for board files - this means that you 
>     are free to say that DTSes are strictly coupled with kernel version 
>     and you are free to modify the bindings - see the analogy to board 
>     files, where you could modify the platform data structures and could 
>     not directly copy board file from one kernel version to another,
> 
>  b) using DT as an ABI - this is the original way, i.e. define stable 
>     bindings and make sure that anu DTB built for older kernel will work,
>     with equal or greater set of functionality on newer kernels.
> 
> Now I believe in this thread the point whether we should use a) or b) or a 
> combination of both has been raised.

If you seriously want to pursue a) then you are thinking only of
yourself.

Please consider the needs of the people trying to use your code in
actual practice.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sat, Jul 27, 2013 at 11:51:06AM -0700, Tomasz Figa wrote:
> On Saturday 27 of July 2013 20:31:01 Richard Cochran wrote:
> > 
> > Frankly, I am really surprised and shocked at the cavalier attitude
> > expressed here WRT DT bindings in released kernels. Think about the
> > *users* of this code. Not everyone working with ARM Linux is a kernel
> > developer or a DT guru. There is really no indication at all that the
> > ARM Linux DT stuff released so far are not stable and trustworthy.

Read the above again, please.

> Well, it depends on how we use the DT. There are (at least) two possible 
> usage scenarios:
> 
>  a) using DT as direct replacement for board files - this means that you 
>     are free to say that DTSes are strictly coupled with kernel version 
>     and you are free to modify the bindings - see the analogy to board 
>     files, where you could modify the platform data structures and could 
>     not directly copy board file from one kernel version to another,
> 
>  b) using DT as an ABI - this is the original way, i.e. define stable 
>     bindings and make sure that anu DTB built for older kernel will work,
>     with equal or greater set of functionality on newer kernels.
> 
> Now I believe in this thread the point whether we should use a) or b) or a 
> combination of both has been raised.

If you seriously want to pursue a) then you are thinking only of
yourself.

Please consider the needs of the people trying to use your code in
actual practice.

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Sunday 28 of July 2013 10:56:52 Richard Cochran wrote:
> On Sat, Jul 27, 2013 at 11:51:06AM -0700, Tomasz Figa wrote:
> > On Saturday 27 of July 2013 20:31:01 Richard Cochran wrote:
> > > Frankly, I am really surprised and shocked at the cavalier attitude
> > > expressed here WRT DT bindings in released kernels. Think about the
> > > *users* of this code. Not everyone working with ARM Linux is a
> > > kernel
> > > developer or a DT guru. There is really no indication at all that
> > > the
> > > ARM Linux DT stuff released so far are not stable and trustworthy.
> 
> Read the above again, please.

I'm not really sure what effect on users this has. Maybe you should define 
"users".

Sure, if you don't like the fact that it is not cleary said, whether we 
use a) or b) (see below), then it's hard to disagree with you - it should 
have been said. We are trying to fix this now.

> > Well, it depends on how we use the DT. There are (at least) two
> > possible> 
> > usage scenarios:
> >  a) using DT as direct replacement for board files - this means that
> >  you
> >  
> >     are free to say that DTSes are strictly coupled with kernel
> >     version
> >     and you are free to modify the bindings - see the analogy to board
> >     files, where you could modify the platform data structures and
> >     could
> >     not directly copy board file from one kernel version to another,
> >  
> >  b) using DT as an ABI - this is the original way, i.e. define stable
> >  
> >     bindings and make sure that anu DTB built for older kernel will
> >     work,
> >     with equal or greater set of functionality on newer kernels.
> > 
> > Now I believe in this thread the point whether we should use a) or b)
> > or a combination of both has been raised.
> 
> If you seriously want to pursue a) then you are thinking only of
> yourself.
> 
> Please consider the needs of the people trying to use your code in
> actual practice.

Care to explain this reasoning?

Let's say I'm not strongly for neither of above and I'd like you to 
convince me to one of the options.

Best regards,
Tomasz

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Sunday 28 of July 2013 10:56:52 Richard Cochran wrote:
> On Sat, Jul 27, 2013 at 11:51:06AM -0700, Tomasz Figa wrote:
> > On Saturday 27 of July 2013 20:31:01 Richard Cochran wrote:
> > > Frankly, I am really surprised and shocked at the cavalier attitude
> > > expressed here WRT DT bindings in released kernels. Think about the
> > > *users* of this code. Not everyone working with ARM Linux is a
> > > kernel
> > > developer or a DT guru. There is really no indication at all that
> > > the
> > > ARM Linux DT stuff released so far are not stable and trustworthy.
> 
> Read the above again, please.

I'm not really sure what effect on users this has. Maybe you should define 
"users".

Sure, if you don't like the fact that it is not cleary said, whether we 
use a) or b) (see below), then it's hard to disagree with you - it should 
have been said. We are trying to fix this now.

> > Well, it depends on how we use the DT. There are (at least) two
> > possible> 
> > usage scenarios:
> >  a) using DT as direct replacement for board files - this means that
> >  you
> >  
> >     are free to say that DTSes are strictly coupled with kernel
> >     version
> >     and you are free to modify the bindings - see the analogy to board
> >     files, where you could modify the platform data structures and
> >     could
> >     not directly copy board file from one kernel version to another,
> >  
> >  b) using DT as an ABI - this is the original way, i.e. define stable
> >  
> >     bindings and make sure that anu DTB built for older kernel will
> >     work,
> >     with equal or greater set of functionality on newer kernels.
> > 
> > Now I believe in this thread the point whether we should use a) or b)
> > or a combination of both has been raised.
> 
> If you seriously want to pursue a) then you are thinking only of
> yourself.
> 
> Please consider the needs of the people trying to use your code in
> actual practice.

Care to explain this reasoning?

Let's say I'm not strongly for neither of above and I'd like you to 
convince me to one of the options.

Best regards,
Tomasz

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

On Sat, Jul 27, 2013 at 07:37:48PM +0200, Richard Cochran wrote:
> On Sat, Jul 27, 2013 at 11:40:18AM +0100, Mark Brown wrote:

> > We did have exactly the same discussion when the DT transition was
> > started - this isn't something that people only just realised might be
> > an issue.  There was a deliberate decision to focus on getting the
> > technology deployed to the point where it could be used as a straight
> > replacement for board files and accept that sometimes the results won't
> > be perfect and that we may need to rework as a result.

> Can you tell a bit more about this decision? When was it made? Who
> made it? How was it made public?

I honestly can't remember exactly - it was part of the discussion about
keeping the .dtb files in the kernel IIRC.  Given the timing and the
fact that I remember it it'd have been a mailing list discussion but I
couldn't point you at it without searching for a needle in a haystack,
sorry.  Perhaps there's a LWN writeup or something.

It's not the "we don't care" that you're seeing, it's more a product of
"we're not sure what we're doing yet, let's give ourselves a pass if we
mess things up while we're still learning" so it's probably buried in
some thread.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Sat, Jul 27, 2013 at 07:37:48PM +0200, Richard Cochran wrote:
> On Sat, Jul 27, 2013 at 11:40:18AM +0100, Mark Brown wrote:

> > We did have exactly the same discussion when the DT transition was
> > started - this isn't something that people only just realised might be
> > an issue.  There was a deliberate decision to focus on getting the
> > technology deployed to the point where it could be used as a straight
> > replacement for board files and accept that sometimes the results won't
> > be perfect and that we may need to rework as a result.

> Can you tell a bit more about this decision? When was it made? Who
> made it? How was it made public?

I honestly can't remember exactly - it was part of the discussion about
keeping the .dtb files in the kernel IIRC.  Given the timing and the
fact that I remember it it'd have been a mailing list discussion but I
couldn't point you at it without searching for a needle in a haystack,
sorry.  Perhaps there's a LWN writeup or something.

It's not the "we don't care" that you're seeing, it's more a product of
"we're not sure what we're doing yet, let's give ourselves a pass if we
mess things up while we're still learning" so it's probably buried in
some thread.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130728/4f4a3fe8/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 5033 bytes --]

On Sat, Jul 27, 2013 at 08:07:54PM +0200, Richard Cochran wrote:
> On Sat, Jul 27, 2013 at 12:36:02PM +0100, Mark Brown wrote:

> Yes, lets, and remember the question was, why do I say that dealing
> with DT is such a PITA.

There are definite issues with DT (getting a good process for quality
bindings together being the major one, and we're still working on some
of the subsystem bindings) but we shouldn't conflate other things with
those.  What you and others are saying about the need for a stable ABI
for DT is correct but we're going off on tangents here.

My read on what you're saying is that it's more to do with bleeding edge
stuff being painful than it is to do with DT itself; DT gets mentioned a
lot because it is an active area of development and gets used a lot
during bringup but it's not the cause.

> > >   http://www.spinics.net/lists/arm-kernel/msg198431.html

> > This actually sounds pretty good - glancing over the thread it seems you
> > were trying to boot a shiny new board that people were in the process of
> > trying to upstream support for and were just a bit too early.  Device
> > tree doesn't seem to have made a difference either way here.

> Did you miss the part about CONFIG_ARM_APPENDED_DTB?

It didn't seem terribly relevant - it's a feature if people and/or
systems want to use it.

> > >   http://www.spinics.net/lists/linux-omap/msg79731.html

> > Paul's reply here seems fairly clear - someone had merged a driver which
> > had been developed in an out of tree or pre-DT environment without DT
> > support so they just hadn't added DT support.  Sadly doing that is new
> > feature development and so not appropriate for the stabalisation phase
> > of development.

> This is me asking for maintainers to take patches to fix a driver in
> version v3.7 where the driver merged in v3.4. 

> The patches contain the missing DT part, and the maintainer rejects
> them, saying, no new features!

> Q: What the heck kind of process is that?
> A: DT process.

No, this is nothing to do with DT - this is just the general kernel
development process.  DT was a new feature for that driver (which
would've been merged a year or so before we got our first DT only ARM
platform...), it's unfortunate that you needed it for your system since
it was a DT only system but it's still a new feature with respect to the
driver.

> Seriously, why is it too hard for y'all to insist on merging drivers
> only when they are really, truly ready (and don't forget the DT part,
> please).

There's a couple of things there.  One is that subsystem maintainers
have no real way of telling if a driver actually works unless they
happen to have a system they can test it on.  For the most part that's
just not the case so we have to trust that someone will say something if
there's a problem.  We can spot some problems through review but there's
a huge set that just won't be obvious without knowledge of the hardware.

The other is that a partially featured driver in the kernel tends to be
a better thing for development than something out of tree - if it's
useful for someone that's great and it provides a starting point for
someone else to come along and add more features.  So long as it's not
causing problems at a subsystem level it's generally better to have it
than not, that way other people can come along and incrementally improve
the driver without having to work out some way of collaborating out of
tree.

You mention DT specifically here - obviously a lot of platforms don't
require DT (only some architectures use it at all) so it can't be a
requirement for drivers and if someone's done the work to get the
hardware working it seems more useful to get that merged and then let
someone who cares about DT add the bindings later.

> > > * When people try in good faith to conduct methodical boot tests,
> > >   DT is working against them.

> > >   http://www.spinics.net/lists/linux-omap/msg79960.html

> > Again I don't see anything particularly related to DT here but instead
> > do with using a SoC and board that are in the early phases of mainline
> > integration.

> It is ring around the rosie, DT, boot loader, and kernel.

> Understandably, Paul doesn't want to upgrade his boot loader. He says
> he is "not interested" in testing the boot loader, just the kernel.
> And, if you follow the sage of Paul's test reports, you will find him
> being told to update his boot loader and not to forget the delete old
> dtb files.

> So, like I said, it is a pity PITA.

New systems especially those using new SoCs generally are a pain; had
the issues not mentioned DT they might well have been something else
like missing pinmux in the bootloader, some bit of kernel functionality
not being merged yet or something else.  The flow of discussion there
looks very familiar to me from way before DT was talked about for ARM -
it reminds me a bit of some work I did before even PowerPC adopted DT!

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Sat, Jul 27, 2013 at 08:07:54PM +0200, Richard Cochran wrote:
> On Sat, Jul 27, 2013 at 12:36:02PM +0100, Mark Brown wrote:

> Yes, lets, and remember the question was, why do I say that dealing
> with DT is such a PITA.

There are definite issues with DT (getting a good process for quality
bindings together being the major one, and we're still working on some
of the subsystem bindings) but we shouldn't conflate other things with
those.  What you and others are saying about the need for a stable ABI
for DT is correct but we're going off on tangents here.

My read on what you're saying is that it's more to do with bleeding edge
stuff being painful than it is to do with DT itself; DT gets mentioned a
lot because it is an active area of development and gets used a lot
during bringup but it's not the cause.

> > >   http://www.spinics.net/lists/arm-kernel/msg198431.html

> > This actually sounds pretty good - glancing over the thread it seems you
> > were trying to boot a shiny new board that people were in the process of
> > trying to upstream support for and were just a bit too early.  Device
> > tree doesn't seem to have made a difference either way here.

> Did you miss the part about CONFIG_ARM_APPENDED_DTB?

It didn't seem terribly relevant - it's a feature if people and/or
systems want to use it.

> > >   http://www.spinics.net/lists/linux-omap/msg79731.html

> > Paul's reply here seems fairly clear - someone had merged a driver which
> > had been developed in an out of tree or pre-DT environment without DT
> > support so they just hadn't added DT support.  Sadly doing that is new
> > feature development and so not appropriate for the stabalisation phase
> > of development.

> This is me asking for maintainers to take patches to fix a driver in
> version v3.7 where the driver merged in v3.4. 

> The patches contain the missing DT part, and the maintainer rejects
> them, saying, no new features!

> Q: What the heck kind of process is that?
> A: DT process.

No, this is nothing to do with DT - this is just the general kernel
development process.  DT was a new feature for that driver (which
would've been merged a year or so before we got our first DT only ARM
platform...), it's unfortunate that you needed it for your system since
it was a DT only system but it's still a new feature with respect to the
driver.

> Seriously, why is it too hard for y'all to insist on merging drivers
> only when they are really, truly ready (and don't forget the DT part,
> please).

There's a couple of things there.  One is that subsystem maintainers
have no real way of telling if a driver actually works unless they
happen to have a system they can test it on.  For the most part that's
just not the case so we have to trust that someone will say something if
there's a problem.  We can spot some problems through review but there's
a huge set that just won't be obvious without knowledge of the hardware.

The other is that a partially featured driver in the kernel tends to be
a better thing for development than something out of tree - if it's
useful for someone that's great and it provides a starting point for
someone else to come along and add more features.  So long as it's not
causing problems at a subsystem level it's generally better to have it
than not, that way other people can come along and incrementally improve
the driver without having to work out some way of collaborating out of
tree.

You mention DT specifically here - obviously a lot of platforms don't
require DT (only some architectures use it at all) so it can't be a
requirement for drivers and if someone's done the work to get the
hardware working it seems more useful to get that merged and then let
someone who cares about DT add the bindings later.

> > > * When people try in good faith to conduct methodical boot tests,
> > >   DT is working against them.

> > >   http://www.spinics.net/lists/linux-omap/msg79960.html

> > Again I don't see anything particularly related to DT here but instead
> > do with using a SoC and board that are in the early phases of mainline
> > integration.

> It is ring around the rosie, DT, boot loader, and kernel.

> Understandably, Paul doesn't want to upgrade his boot loader. He says
> he is "not interested" in testing the boot loader, just the kernel.
> And, if you follow the sage of Paul's test reports, you will find him
> being told to update his boot loader and not to forget the delete old
> dtb files.

> So, like I said, it is a pity PITA.

New systems especially those using new SoCs generally are a pain; had
the issues not mentioned DT they might well have been something else
like missing pinmux in the bootloader, some bit of kernel functionality
not being merged yet or something else.  The flow of discussion there
looks very familiar to me from way before DT was talked about for ARM -
it reminds me a bit of some work I did before even PowerPC adopted DT!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130728/59f819a2/attachment-0001.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sun, Jul 28, 2013 at 11:12:53AM +0200, Tomasz Figa wrote:

> I'm not really sure what effect on users this has. Maybe you should define 
> "users".

...

> Care to explain this reasoning?

Use Case
~~~~~~~~

User acquires a machine running ARM Linux version 3.x, with u-boot
and dtb in a read only flash partition. The board boots and works just
fine. However, for his application, the user requires a new kernel
feature that appeared in version 3.y where y > x. He compiles the new
kernel, and it also works.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sun, Jul 28, 2013 at 11:12:53AM +0200, Tomasz Figa wrote:

> I'm not really sure what effect on users this has. Maybe you should define 
> "users".

...

> Care to explain this reasoning?

Use Case
~~~~~~~~

User acquires a machine running ARM Linux version 3.x, with u-boot
and dtb in a read only flash partition. The board boots and works just
fine. However, for his application, the user requires a new kernel
feature that appeared in version 3.y where y > x. He compiles the new
kernel, and it also works.

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Sunday 28 of July 2013 15:19:03 Richard Cochran wrote:
> On Sun, Jul 28, 2013 at 11:12:53AM +0200, Tomasz Figa wrote:
> > I'm not really sure what effect on users this has. Maybe you should
> > define "users".
> 
> ...
> 
> > Care to explain this reasoning?
> 
> Use Case
> ~~~~~~~~
> 
> User acquires a machine running ARM Linux version 3.x, with u-boot
> and dtb in a read only flash partition. The board boots and works just
> fine. However, for his application, the user requires a new kernel
> feature that appeared in version 3.y where y > x. He compiles the new
> kernel, and it also works.

Generally the user does not care where the dtb is stored. He just want to 
upgrade the kernel without thinking about internals.

There are many possible options:

a) The BSP packaging script he received from board vendor, or even kernel 
build system, builds dtb from kernel sources and appends it to built 
zImage. He just flashes the zImage and everything is working correctly.

  This is pretty common case today, as many boards still use legacy 
  bootloaders without native support for DT. See the analogy to board 
  files, being compiled as a part of kernel sources.

b) The user always compiles the kernel and dtb and flashes both at the 
same time.

  This does not differ at all to flashing the kernel alone, except two 
  files, not one, need to be flashed.

By the way, in use case you are describing, changes that dtb wouldn't have 
to be updated are very low, unless the one present in read only memory of 
the board is complete, i.e. fully describes all the available hardware, 
which is unlikely for a dtb built at time of 3.x availability, whenever 
the feature showed up in 3.y (y > x). The user will most likely have to 
update the dtb anyway.

Please note, though, I'm _not_ trying to convince you that this kind of 
solutions is good, as I'm not convinced either. That's why we are 
discussing this.

Best regards,
Tomasz

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Sunday 28 of July 2013 15:19:03 Richard Cochran wrote:
> On Sun, Jul 28, 2013 at 11:12:53AM +0200, Tomasz Figa wrote:
> > I'm not really sure what effect on users this has. Maybe you should
> > define "users".
> 
> ...
> 
> > Care to explain this reasoning?
> 
> Use Case
> ~~~~~~~~
> 
> User acquires a machine running ARM Linux version 3.x, with u-boot
> and dtb in a read only flash partition. The board boots and works just
> fine. However, for his application, the user requires a new kernel
> feature that appeared in version 3.y where y > x. He compiles the new
> kernel, and it also works.

Generally the user does not care where the dtb is stored. He just want to 
upgrade the kernel without thinking about internals.

There are many possible options:

a) The BSP packaging script he received from board vendor, or even kernel 
build system, builds dtb from kernel sources and appends it to built 
zImage. He just flashes the zImage and everything is working correctly.

  This is pretty common case today, as many boards still use legacy 
  bootloaders without native support for DT. See the analogy to board 
  files, being compiled as a part of kernel sources.

b) The user always compiles the kernel and dtb and flashes both at the 
same time.

  This does not differ at all to flashing the kernel alone, except two 
  files, not one, need to be flashed.

By the way, in use case you are describing, changes that dtb wouldn't have 
to be updated are very low, unless the one present in read only memory of 
the board is complete, i.e. fully describes all the available hardware, 
which is unlikely for a dtb built at time of 3.x availability, whenever 
the feature showed up in 3.y (y > x). The user will most likely have to 
update the dtb anyway.

Please note, though, I'm _not_ trying to convince you that this kind of 
solutions is good, as I'm not convinced either. That's why we are 
discussing this.

Best regards,
Tomasz

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Sun, Jul 28, 2013 at 9:39 AM, Tomasz Figa <tomasz.figa@gmail.com> wrote:
> On Sunday 28 of July 2013 15:19:03 Richard Cochran wrote:
>> On Sun, Jul 28, 2013 at 11:12:53AM +0200, Tomasz Figa wrote:
>> > I'm not really sure what effect on users this has. Maybe you should
>> > define "users".
>>
>> ...
>>
>> > Care to explain this reasoning?
>>
>> Use Case
>> ~~~~~~~~
>>
>> User acquires a machine running ARM Linux version 3.x, with u-boot
>> and dtb in a read only flash partition. The board boots and works just
>> fine. However, for his application, the user requires a new kernel
>> feature that appeared in version 3.y where y > x. He compiles the new
>> kernel, and it also works.
>
> Generally the user does not care where the dtb is stored. He just want to
> upgrade the kernel without thinking about internals.
>
> There are many possible options:
>
> a) The BSP packaging script he received from board vendor, or even kernel
> build system, builds dtb from kernel sources and appends it to built
> zImage. He just flashes the zImage and everything is working correctly.
>
>   This is pretty common case today, as many boards still use legacy
>   bootloaders without native support for DT. See the analogy to board
>   files, being compiled as a part of kernel sources.
>
> b) The user always compiles the kernel and dtb and flashes both at the
> same time.
>
>   This does not differ at all to flashing the kernel alone, except two
>   files, not one, need to be flashed.

c) Use a quirk system to map ROM based DTB's.

3.x kernel was matched to dtb that is flashed into boot rom.
Everything was initially ok.

The 3.y kernel is built with init time quirks that map the dtb from
3.x format  into the 3.y format. These quirks can be easily built
since we know the generic schema from both 3.x and 3.y. Mapping ten
year old power PC dtbs may involve more code.

These quirks aren't going to be large, it's not like the entire schema
is going to change on each kernel release.  They are just going to
make a best effort to map the dtb in ROM into something the 3.y kernel
can understand. Owners of these ROMs will complain loudly if the
quirks aren't right.

3.z kernel is free to alter the schema. But it will have to supply the
necessary quirks needed to keep those old dtb's functioning.

>
> By the way, in use case you are describing, changes that dtb wouldn't have
> to be updated are very low, unless the one present in read only memory of
> the board is complete, i.e. fully describes all the available hardware,
> which is unlikely for a dtb built at time of 3.x availability, whenever
> the feature showed up in 3.y (y > x). The user will most likely have to
> update the dtb anyway.
>
> Please note, though, I'm _not_ trying to convince you that this kind of
> solutions is good, as I'm not convinced either. That's why we are
> discussing this.
>
> Best regards,
> Tomasz
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Sun, Jul 28, 2013 at 9:39 AM, Tomasz Figa <tomasz.figa@gmail.com> wrote:
> On Sunday 28 of July 2013 15:19:03 Richard Cochran wrote:
>> On Sun, Jul 28, 2013 at 11:12:53AM +0200, Tomasz Figa wrote:
>> > I'm not really sure what effect on users this has. Maybe you should
>> > define "users".
>>
>> ...
>>
>> > Care to explain this reasoning?
>>
>> Use Case
>> ~~~~~~~~
>>
>> User acquires a machine running ARM Linux version 3.x, with u-boot
>> and dtb in a read only flash partition. The board boots and works just
>> fine. However, for his application, the user requires a new kernel
>> feature that appeared in version 3.y where y > x. He compiles the new
>> kernel, and it also works.
>
> Generally the user does not care where the dtb is stored. He just want to
> upgrade the kernel without thinking about internals.
>
> There are many possible options:
>
> a) The BSP packaging script he received from board vendor, or even kernel
> build system, builds dtb from kernel sources and appends it to built
> zImage. He just flashes the zImage and everything is working correctly.
>
>   This is pretty common case today, as many boards still use legacy
>   bootloaders without native support for DT. See the analogy to board
>   files, being compiled as a part of kernel sources.
>
> b) The user always compiles the kernel and dtb and flashes both at the
> same time.
>
>   This does not differ at all to flashing the kernel alone, except two
>   files, not one, need to be flashed.

c) Use a quirk system to map ROM based DTB's.

3.x kernel was matched to dtb that is flashed into boot rom.
Everything was initially ok.

The 3.y kernel is built with init time quirks that map the dtb from
3.x format  into the 3.y format. These quirks can be easily built
since we know the generic schema from both 3.x and 3.y. Mapping ten
year old power PC dtbs may involve more code.

These quirks aren't going to be large, it's not like the entire schema
is going to change on each kernel release.  They are just going to
make a best effort to map the dtb in ROM into something the 3.y kernel
can understand. Owners of these ROMs will complain loudly if the
quirks aren't right.

3.z kernel is free to alter the schema. But it will have to supply the
necessary quirks needed to keep those old dtb's functioning.

>
> By the way, in use case you are describing, changes that dtb wouldn't have
> to be updated are very low, unless the one present in read only memory of
> the board is complete, i.e. fully describes all the available hardware,
> which is unlikely for a dtb built at time of 3.x availability, whenever
> the feature showed up in 3.y (y > x). The user will most likely have to
> update the dtb anyway.
>
> Please note, though, I'm _not_ trying to convince you that this kind of
> solutions is good, as I'm not convinced either. That's why we are
> discussing this.
>
> Best regards,
> Tomasz
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sun, Jul 28, 2013 at 03:39:56PM +0200, Tomasz Figa wrote:
 
> There are many possible options:

...

Wow, you totally ignored the use case.

> Please note, though, I'm _not_ trying to convince you that this kind of 
> solutions is good, as I'm not convinced either. That's why we are 
> discussing this.

And next you are going to convince me that updating my BIOS is a
normal, expected, and necessary step in upgrading my kernel.

No, thank you.

Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sun, Jul 28, 2013 at 03:39:56PM +0200, Tomasz Figa wrote:
 
> There are many possible options:

...

Wow, you totally ignored the use case.

> Please note, though, I'm _not_ trying to convince you that this kind of 
> solutions is good, as I'm not convinced either. That's why we are 
> discussing this.

And next you are going to convince me that updating my BIOS is a
normal, expected, and necessary step in upgrading my kernel.

No, thank you.

Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sun, Jul 28, 2013 at 10:09:57AM -0400, jonsmirl@gmail.com wrote:
> 
> 3.z kernel is free to alter the schema. But it will have to supply the
> necessary quirks needed to keep those old dtb's functioning.

The quirks idea sounds okay to me, if it can really provide forward
compatibility. In practice, I doubt anyone will really spend the
effort to make this work. I think it would be much easier to make sure
the bindings are "future proof" in the first place.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Sun, Jul 28, 2013 at 10:09:57AM -0400, jonsmirl at gmail.com wrote:
> 
> 3.z kernel is free to alter the schema. But it will have to supply the
> necessary quirks needed to keep those old dtb's functioning.

The quirks idea sounds okay to me, if it can really provide forward
compatibility. In practice, I doubt anyone will really spend the
effort to make this work. I think it would be much easier to make sure
the bindings are "future proof" in the first place.

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Sun, Jul 28, 2013 at 11:35 AM, Richard Cochran
<richardcochran@gmail.com> wrote:
> On Sun, Jul 28, 2013 at 10:09:57AM -0400, jonsmirl@gmail.com wrote:
>>
>> 3.z kernel is free to alter the schema. But it will have to supply the
>> necessary quirks needed to keep those old dtb's functioning.
>
> The quirks idea sounds okay to me, if it can really provide forward
> compatibility. In practice, I doubt anyone will really spend the
> effort to make this work. I think it would be much easier to make sure
> the bindings are "future proof" in the first place.

"furture proof" is much easier to say that it is to do. We've been
messing around with the audio bindings for three years and still don't
have a really good scheme. It is pretty easy to come up with the first
90% of a device tree. It is really hard to work out that last 10%.

You can easily get the chips into the tree. Doing that will load the
correct device drivers. But now how are these chips wired together? Is
the appropriate button, LED, etc attached to all the IO pins offered
by the chip? Those answers vary by the PCB the chip was used in..
Trying to figure out a scheme for this has lead to some volatility in
the device trees.  The whole concept of pin mapping was missing from
the early device trees.


>
> Thanks,
> Richard



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Sun, Jul 28, 2013 at 11:35 AM, Richard Cochran
<richardcochran@gmail.com> wrote:
> On Sun, Jul 28, 2013 at 10:09:57AM -0400, jonsmirl at gmail.com wrote:
>>
>> 3.z kernel is free to alter the schema. But it will have to supply the
>> necessary quirks needed to keep those old dtb's functioning.
>
> The quirks idea sounds okay to me, if it can really provide forward
> compatibility. In practice, I doubt anyone will really spend the
> effort to make this work. I think it would be much easier to make sure
> the bindings are "future proof" in the first place.

"furture proof" is much easier to say that it is to do. We've been
messing around with the audio bindings for three years and still don't
have a really good scheme. It is pretty easy to come up with the first
90% of a device tree. It is really hard to work out that last 10%.

You can easily get the chips into the tree. Doing that will load the
correct device drivers. But now how are these chips wired together? Is
the appropriate button, LED, etc attached to all the IO pins offered
by the chip? Those answers vary by the PCB the chip was used in..
Trying to figure out a scheme for this has lead to some volatility in
the device trees.  The whole concept of pin mapping was missing from
the early device trees.


>
> Thanks,
> Richard



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

On Sun, Jul 28, 2013 at 11:50:19AM -0400, jonsmirl@gmail.com wrote:

> "furture proof" is much easier to say that it is to do. We've been
> messing around with the audio bindings for three years and still don't
> have a really good scheme. It is pretty easy to come up with the first
> 90% of a device tree. It is really hard to work out that last 10%.

I wouldn't say that for audio - we've got a pretty solid idea of what's
going on and little if any churn.  If some more of the subsystems that
we rely on were working well we could do a bit more but it's mostly just
extensions of where we're at now.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Sun, Jul 28, 2013 at 11:50:19AM -0400, jonsmirl at gmail.com wrote:

> "furture proof" is much easier to say that it is to do. We've been
> messing around with the audio bindings for three years and still don't
> have a really good scheme. It is pretty easy to come up with the first
> 90% of a device tree. It is really hard to work out that last 10%.

I wouldn't say that for audio - we've got a pretty solid idea of what's
going on and little if any churn.  If some more of the subsystems that
we rely on were working well we could do a bit more but it's mostly just
extensions of where we're at now.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130728/4fb8f828/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

On Sun, Jul 28, 2013 at 05:35:46PM +0200, Richard Cochran wrote:
> On Sun, Jul 28, 2013 at 10:09:57AM -0400, jonsmirl@gmail.com wrote:
> > 
> > 3.z kernel is free to alter the schema. But it will have to supply the
> > necessary quirks needed to keep those old dtb's functioning.
> 
> The quirks idea sounds okay to me, if it can really provide forward
> compatibility. In practice, I doubt anyone will really spend the
> effort to make this work. I think it would be much easier to make sure
> the bindings are "future proof" in the first place.

I should clarify.  The idea of DT quirks is not to remove the need to
properly design and review bindings.  It's to limit the damage when
there are, inevitably, failings in that process.  And when, also
inevitably, firmware vendors ship DTs that don't follow the bindings
correctly, even when there is a good one available.

I think it's more likely that people will create, and get right, a
well localized bit of quirk code, than they will get backwards compat
code correct for every place in which a driver wants info from the
device tree.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

On Sun, Jul 28, 2013 at 05:35:46PM +0200, Richard Cochran wrote:
> On Sun, Jul 28, 2013 at 10:09:57AM -0400, jonsmirl at gmail.com wrote:
> > 
> > 3.z kernel is free to alter the schema. But it will have to supply the
> > necessary quirks needed to keep those old dtb's functioning.
> 
> The quirks idea sounds okay to me, if it can really provide forward
> compatibility. In practice, I doubt anyone will really spend the
> effort to make this work. I think it would be much easier to make sure
> the bindings are "future proof" in the first place.

I should clarify.  The idea of DT quirks is not to remove the need to
properly design and review bindings.  It's to limit the damage when
there are, inevitably, failings in that process.  And when, also
inevitably, firmware vendors ship DTs that don't follow the bindings
correctly, even when there is a good one available.

I think it's more likely that people will create, and get right, a
well localized bit of quirk code, than they will get backwards compat
code correct for every place in which a driver wants info from the
device tree.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130729/fd78ddc3/attachment-0001.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]

Hi,

On Sun, Jul 28, 2013 at 03:19:03PM +0200, Richard Cochran wrote:
> On Sun, Jul 28, 2013 at 11:12:53AM +0200, Tomasz Figa wrote:
> 
> > I'm not really sure what effect on users this has. Maybe you should define 
> > "users".
> 
> ...
> 
> > Care to explain this reasoning?
> 
> Use Case
> ~~~~~~~~
> 
> User acquires a machine running ARM Linux version 3.x, with u-boot
> and dtb in a read only flash partition. The board boots and works just
> fine. However, for his application, the user requires a new kernel
> feature that appeared in version 3.y where y > x. He compiles the new
> kernel, and it also works.

I'm afraid this kind of use case will never be properly supported, DT
stable ABI or not.

Think about this: what kernel will actually be shipped in that board?
Most likely, it will be a BSP kernel from the vendor. Does the vendor
will have made that commitment to have a stable ABI for the DT? Will it
use the same bindings than mainline? Do we want to support all the crazy
bindings every vendor will come up with?

I'm afraid the answer to these three questions will most of the time be
"no.".

That doesn't mean we shouldn't aim for *mainline* having a stable DT
ABI, but that kind of use case doesn't seem very realistic to me.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Maxime Ripard]

Hi,

On Sun, Jul 28, 2013 at 03:19:03PM +0200, Richard Cochran wrote:
> On Sun, Jul 28, 2013 at 11:12:53AM +0200, Tomasz Figa wrote:
> 
> > I'm not really sure what effect on users this has. Maybe you should define 
> > "users".
> 
> ...
> 
> > Care to explain this reasoning?
> 
> Use Case
> ~~~~~~~~
> 
> User acquires a machine running ARM Linux version 3.x, with u-boot
> and dtb in a read only flash partition. The board boots and works just
> fine. However, for his application, the user requires a new kernel
> feature that appeared in version 3.y where y > x. He compiles the new
> kernel, and it also works.

I'm afraid this kind of use case will never be properly supported, DT
stable ABI or not.

Think about this: what kernel will actually be shipped in that board?
Most likely, it will be a BSP kernel from the vendor. Does the vendor
will have made that commitment to have a stable ABI for the DT? Will it
use the same bindings than mainline? Do we want to support all the crazy
bindings every vendor will come up with?

I'm afraid the answer to these three questions will most of the time be
"no.".

That doesn't mean we shouldn't aim for *mainline* having a stable DT
ABI, but that kind of use case doesn't seem very realistic to me.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130729/3a51fbe2/attachment-0001.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/27/2013 10:01 PM, jonsmirl@gmail.com wrote:
> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
>>> Let's see how many people go and scream if I say this: Too bad .dts files
>>> are not done using XML format as DT bindings could be described using XML
>>> Schema.
>>
>> Draft an example and show us how it would look!  :-)  There is
>> absolutely nothing preventing us from expressing a DT in XML format,
>> or even using XSLT to define DT schema while still using our current
>> .dts syntax. It would be trivial to do lossless translation between
>> .dts syntax and xml.
>>
>> The problem that I have with XML and XSLT is that it is very verbose
>> and not entirely friendly to mere-mortals. However, I'm more than
>> willing to be proved wrong on this point.
>
> I considered this approach a while ago and discarded it. It would work
> but it is just too much of a Frankenstein monster.

Ah, but he is so cute. At least I do not think it is more monstrous as 
the bindings files. I just browsed through a couple of arm binding files 
as I felt challenged to come up with an example. I did not get the 
impression that there is some kind of template in place to get consitent 
bindings descriptions.

> Much cleaner to modify dtc to take a schema as part of the compilation
> process. The schema language itself has no requirement to look like
> DTS syntax. Whoever wrote dtc probably has a favorite language that
> would be good for writing schemas in.

Not sure if I can follow here. I guess you mean the dts compilation, 
right? There are tools freely available to validate XML files against 
XSD specification files. As an example libxml2 has support for it. I 
suspect it is not desired to have a dependency for DTC with an 
out-of-tree library, but it could be incorporated and have DTC spew the 
validation results.

Regards,
Arend

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/27/2013 10:01 PM, jonsmirl at gmail.com wrote:
> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
>>> Let's see how many people go and scream if I say this: Too bad .dts files
>>> are not done using XML format as DT bindings could be described using XML
>>> Schema.
>>
>> Draft an example and show us how it would look!  :-)  There is
>> absolutely nothing preventing us from expressing a DT in XML format,
>> or even using XSLT to define DT schema while still using our current
>> .dts syntax. It would be trivial to do lossless translation between
>> .dts syntax and xml.
>>
>> The problem that I have with XML and XSLT is that it is very verbose
>> and not entirely friendly to mere-mortals. However, I'm more than
>> willing to be proved wrong on this point.
>
> I considered this approach a while ago and discarded it. It would work
> but it is just too much of a Frankenstein monster.

Ah, but he is so cute. At least I do not think it is more monstrous as 
the bindings files. I just browsed through a couple of arm binding files 
as I felt challenged to come up with an example. I did not get the 
impression that there is some kind of template in place to get consitent 
bindings descriptions.

> Much cleaner to modify dtc to take a schema as part of the compilation
> process. The schema language itself has no requirement to look like
> DTS syntax. Whoever wrote dtc probably has a favorite language that
> would be good for writing schemas in.

Not sure if I can follow here. I guess you mean the dts compilation, 
right? There are tools freely available to validate XML files against 
XSD specification files. As an example libxml2 has support for it. I 
suspect it is not desired to have a dependency for DTC with an 
out-of-tree library, but it could be incorporated and have DTC spew the 
validation results.

Regards,
Arend

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/29/2013 11:19 AM, Arend van Spriel wrote:
> On 07/27/2013 10:01 PM, jonsmirl@gmail.com wrote:
>> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely
>> <grant.likely@secretlab.ca> wrote:
>>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel
>>> <arend@broadcom.com> wrote:
>>>> Let's see how many people go and scream if I say this: Too bad .dts
>>>> files
>>>> are not done using XML format as DT bindings could be described
>>>> using XML
>>>> Schema.
>>>
>>> Draft an example and show us how it would look!  :-)  There is
>>> absolutely nothing preventing us from expressing a DT in XML format,
>>> or even using XSLT to define DT schema while still using our current
>>> .dts syntax. It would be trivial to do lossless translation between
>>> .dts syntax and xml.
>>>
>>> The problem that I have with XML and XSLT is that it is very verbose
>>> and not entirely friendly to mere-mortals. However, I'm more than
>>> willing to be proved wrong on this point.
>>
>> I considered this approach a while ago and discarded it. It would work
>> but it is just too much of a Frankenstein monster.
>
> Ah, but he is so cute. At least I do not think it is more monstrous as
> the bindings files. I just browsed through a couple of arm binding files
> as I felt challenged to come up with an example. I did not get the
> impression that there is some kind of template in place to get consitent
> bindings descriptions.
>
>> Much cleaner to modify dtc to take a schema as part of the compilation
>> process. The schema language itself has no requirement to look like
>> DTS syntax. Whoever wrote dtc probably has a favorite language that
>> would be good for writing schemas in.
>
> Not sure if I can follow here. I guess you mean the dts compilation,
> right? There are tools freely available to validate XML files against
> XSD specification files. As an example libxml2 has support for it. I
> suspect it is not desired to have a dependency for DTC with an
> out-of-tree library, but it could be incorporated and have DTC spew the
> validation results.

crap. Probably not as libxml2 has MIT-license.

Regards,
Arend

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/29/2013 11:19 AM, Arend van Spriel wrote:
> On 07/27/2013 10:01 PM, jonsmirl at gmail.com wrote:
>> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely
>> <grant.likely@secretlab.ca> wrote:
>>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel
>>> <arend@broadcom.com> wrote:
>>>> Let's see how many people go and scream if I say this: Too bad .dts
>>>> files
>>>> are not done using XML format as DT bindings could be described
>>>> using XML
>>>> Schema.
>>>
>>> Draft an example and show us how it would look!  :-)  There is
>>> absolutely nothing preventing us from expressing a DT in XML format,
>>> or even using XSLT to define DT schema while still using our current
>>> .dts syntax. It would be trivial to do lossless translation between
>>> .dts syntax and xml.
>>>
>>> The problem that I have with XML and XSLT is that it is very verbose
>>> and not entirely friendly to mere-mortals. However, I'm more than
>>> willing to be proved wrong on this point.
>>
>> I considered this approach a while ago and discarded it. It would work
>> but it is just too much of a Frankenstein monster.
>
> Ah, but he is so cute. At least I do not think it is more monstrous as
> the bindings files. I just browsed through a couple of arm binding files
> as I felt challenged to come up with an example. I did not get the
> impression that there is some kind of template in place to get consitent
> bindings descriptions.
>
>> Much cleaner to modify dtc to take a schema as part of the compilation
>> process. The schema language itself has no requirement to look like
>> DTS syntax. Whoever wrote dtc probably has a favorite language that
>> would be good for writing schemas in.
>
> Not sure if I can follow here. I guess you mean the dts compilation,
> right? There are tools freely available to validate XML files against
> XSD specification files. As an example libxml2 has support for it. I
> suspect it is not desired to have a dependency for DTC with an
> out-of-tree library, but it could be incorporated and have DTC spew the
> validation results.

crap. Probably not as libxml2 has MIT-license.

Regards,
Arend

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Matt Porter]

On Fri, Jul 26, 2013 at 03:40:47PM +0100, Mark Brown wrote:
> On Fri, Jul 26, 2013 at 03:21:40PM +0100, Russell King - ARM Linux wrote:
> 
> > So this is why I'm seeing patches just a short time ago removing existing
> > compatible strings from the DT descriptions and associated driver, and
> > replacing them with new ones... meaning that the old DT files won't work
> > with newer kernels.
> 
> The big question is if you're seeing such patches merged.  People making
> mistakes on submissions is totally normal.

Over in the bcm53xx thread, we've discussed such a patch to fix
inconsistencies. The problem here is that there is no canonical answer
to what a mistake is. I can make a strong argument that the support for
these parts is in such an early stage that the bindings (in this case
specifically the two different compatible strings for one vendor) should
be considered unstable and ok to fix the consistency issue. Mark Rutland
suggests we should change nothing and possibly add a third vendor prefix
for new bindings. I'm having trouble accepting that just because these
bindings made it into a final kernel during a period where
scrutinization of these things was not very high that we need to forever
carry this inconsistency in the "specification".

http://www.spinics.net/lists/arm-kernel/msg262059.html

> 
> If it's the case I think you mean TBH I'm not sure anyone cares, I don't
> think anyone is using that stuff in production yet as those chips go
> almost exclusively into Android phones.

At least the bcm281xx stuff falls into this category as you describe.
There's simply nobody that would be upset if its bindings changed from
bcm-><something_else> as there's just nobody using the DT-based upstream
work except for internal Broadcom and a couple people external working
closely with them.

This situation seems to illustrate the strong need for an unstable
binding period that's longer than just inclusion in a final kernel
release.

-Matt

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Matt Porter]

On Fri, Jul 26, 2013 at 03:40:47PM +0100, Mark Brown wrote:
> On Fri, Jul 26, 2013 at 03:21:40PM +0100, Russell King - ARM Linux wrote:
> 
> > So this is why I'm seeing patches just a short time ago removing existing
> > compatible strings from the DT descriptions and associated driver, and
> > replacing them with new ones... meaning that the old DT files won't work
> > with newer kernels.
> 
> The big question is if you're seeing such patches merged.  People making
> mistakes on submissions is totally normal.

Over in the bcm53xx thread, we've discussed such a patch to fix
inconsistencies. The problem here is that there is no canonical answer
to what a mistake is. I can make a strong argument that the support for
these parts is in such an early stage that the bindings (in this case
specifically the two different compatible strings for one vendor) should
be considered unstable and ok to fix the consistency issue. Mark Rutland
suggests we should change nothing and possibly add a third vendor prefix
for new bindings. I'm having trouble accepting that just because these
bindings made it into a final kernel during a period where
scrutinization of these things was not very high that we need to forever
carry this inconsistency in the "specification".

http://www.spinics.net/lists/arm-kernel/msg262059.html

> 
> If it's the case I think you mean TBH I'm not sure anyone cares, I don't
> think anyone is using that stuff in production yet as those chips go
> almost exclusively into Android phones.

At least the bcm281xx stuff falls into this category as you describe.
There's simply nobody that would be upset if its bindings changed from
bcm-><something_else> as there's just nobody using the DT-based upstream
work except for internal Broadcom and a couple people external working
closely with them.

This situation seems to illustrate the strong need for an unstable
binding period that's longer than just inclusion in a final kernel
release.

-Matt

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Matt Porter]

On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
> On Fri, Jul 26, 2013 at 7:10 AM, Mark Brown <broonie@kernel.org> wrote:
> > On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> >
> >> Unless I totally misunderstood, the thread is talking about letting
> >> established bindings change with each new kernel version.  I am
> >> opposed to that.
> >
> > No, nobody is really saying that is a particularly good idea.  There is
> > some debate about how we work out what an established binding is but
> > there's no serious suggestion that we don't want stable bindings.
> 
> Yes, what Mark said -- _today_ all bindings are subject to change and
> can be changed in lockstep with the kernel. This has been necessary as
> part of development to sort out all of the various bootstrapping
> issues across platforms.

However, we still have people arguing that we can't (or should not) change
a binding right now even to fix inconsistency issues that are discovered
after the fact. I'm hearing a different story depending on who is
telling it at the moment.

Getting quickly to a definitive answer on the criteria for an
"established" binding is will help end ongoing discussions as to whether
we can fix a currently broken one or just have to leave it.

> What we're talking about is to end that mode of operation, and moving
> over to locking in bindings. Device tree contents, as mentioned
> elsewhere, might still be changed just like code is -- bugs are fixed,
> etc. But it's time to start locking down the bindings, in particular
> no longer change the established ones.
> 
> Long term, final goal is likely to be close to what Russell is saying
> -- nothing should go into the kernel tree unless the binding is in a
> fully stable state. However, we have a transitional period between now
> and then, and even when we're at the final state there will be need to
> have some sort of sandbox for development and test of future bindings.
> Dealing with all that, as well as the actual process for locking in
> bindings, is what needs to be sorted out.
> 
> I think we're all in agreement that bindings that change over time are
> nothing but pain, but we're arguing that in circles anyway.

+1

-Matt

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Matt Porter]

On Fri, Jul 26, 2013 at 08:49:43AM -0700, Olof Johansson wrote:
> On Fri, Jul 26, 2013 at 7:10 AM, Mark Brown <broonie@kernel.org> wrote:
> > On Fri, Jul 26, 2013 at 03:09:29PM +0200, Richard Cochran wrote:
> >
> >> Unless I totally misunderstood, the thread is talking about letting
> >> established bindings change with each new kernel version.  I am
> >> opposed to that.
> >
> > No, nobody is really saying that is a particularly good idea.  There is
> > some debate about how we work out what an established binding is but
> > there's no serious suggestion that we don't want stable bindings.
> 
> Yes, what Mark said -- _today_ all bindings are subject to change and
> can be changed in lockstep with the kernel. This has been necessary as
> part of development to sort out all of the various bootstrapping
> issues across platforms.

However, we still have people arguing that we can't (or should not) change
a binding right now even to fix inconsistency issues that are discovered
after the fact. I'm hearing a different story depending on who is
telling it at the moment.

Getting quickly to a definitive answer on the criteria for an
"established" binding is will help end ongoing discussions as to whether
we can fix a currently broken one or just have to leave it.

> What we're talking about is to end that mode of operation, and moving
> over to locking in bindings. Device tree contents, as mentioned
> elsewhere, might still be changed just like code is -- bugs are fixed,
> etc. But it's time to start locking down the bindings, in particular
> no longer change the established ones.
> 
> Long term, final goal is likely to be close to what Russell is saying
> -- nothing should go into the kernel tree unless the binding is in a
> fully stable state. However, we have a transitional period between now
> and then, and even when we're at the final state there will be need to
> have some sort of sandbox for development and test of future bindings.
> Dealing with all that, as well as the actual process for locking in
> bindings, is what needs to be sorted out.
> 
> I think we're all in agreement that bindings that change over time are
> nothing but pain, but we're arguing that in circles anyway.

+1

-Matt

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Sat, Jul 27, 2013 at 10:48:26AM +0200, Richard Cochran wrote:

> > Of course all this could have been done in C, but it wasn't/hasn't been..
> 
> You have identified the real issue: poor quality of the ARM board
> support process within the kernel. Churn is still churn, whether in DT
> or in platform devices, but the DT people promised to end the churn.

Actually my experience in this area was mainly from PPC, and granted
it spanned the PPC32/PPC64 merge.

> [ I disagree about the "more thought" part. The current discussion,
>   coming years too late after the introduction of DT to ARM Linux, is
>   contrary evidence enough. ]

The current discussion has little to do with the quality of the
bindings, look at the new kirkwood bindings - they have had much more
thought put into them than the old board.c stuff they are replacing.

> > I can't delay shipping while upstream sorts this out - so I know
> > *absolutely* that the DT schema will change. This has been planned for
> > and designed into the boot system and won't be a problem.
> > 
> > Why would anyone do embedded any other way?
> 
> Yep, that is the embedded way: ship it!
> 
> We can do better than that.

I think you missed the point.

There is no way I can possibly ship a product with a DT that is
finished. I can't tie my company's product release cycles to the whims
of the kernel community.

So embedded people are going to ship with unfinished DT and upgrade
later. They have to. There is no choice. Stable DT doesn't change
anything unless you can create perfect stable bindings for a new SOC
instantaneously.

This is where I see the whole disconnect in this
discussion. General-purpose and embedded are *DIFFERENT* users, and
they have different use-cases and different needs.

Jason

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Sat, Jul 27, 2013 at 10:48:26AM +0200, Richard Cochran wrote:

> > Of course all this could have been done in C, but it wasn't/hasn't been..
> 
> You have identified the real issue: poor quality of the ARM board
> support process within the kernel. Churn is still churn, whether in DT
> or in platform devices, but the DT people promised to end the churn.

Actually my experience in this area was mainly from PPC, and granted
it spanned the PPC32/PPC64 merge.

> [ I disagree about the "more thought" part. The current discussion,
>   coming years too late after the introduction of DT to ARM Linux, is
>   contrary evidence enough. ]

The current discussion has little to do with the quality of the
bindings, look at the new kirkwood bindings - they have had much more
thought put into them than the old board.c stuff they are replacing.

> > I can't delay shipping while upstream sorts this out - so I know
> > *absolutely* that the DT schema will change. This has been planned for
> > and designed into the boot system and won't be a problem.
> > 
> > Why would anyone do embedded any other way?
> 
> Yep, that is the embedded way: ship it!
> 
> We can do better than that.

I think you missed the point.

There is no way I can possibly ship a product with a DT that is
finished. I can't tie my company's product release cycles to the whims
of the kernel community.

So embedded people are going to ship with unfinished DT and upgrade
later. They have to. There is no choice. Stable DT doesn't change
anything unless you can create perfect stable bindings for a new SOC
instantaneously.

This is where I see the whole disconnect in this
discussion. General-purpose and embedded are *DIFFERENT* users, and
they have different use-cases and different needs.

Jason

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Sat, Jul 27, 2013 at 11:51:06AM -0700, Tomasz Figa wrote:

> Well, it depends on how we use the DT. There are (at least) two possible 
> usage scenarios:
> 
>  a) using DT as direct replacement for board files - this means that you 
>     are free to say that DTSes are strictly coupled with kernel version 
>     and you are free to modify the bindings - see the analogy to board 
>     files, where you could modify the platform data structures and could 
>     not directly copy board file from one kernel version to another,
> 
>  b) using DT as an ABI - this is the original way, i.e. define stable 
>     bindings and make sure that anu DTB built for older kernel will work,
>     with equal or greater set of functionality on newer kernels.
> 
> Now I believe in this thread the point whether we should use a) or b) or a 
> combination of both has been raised.

Right, and I think it is very important to consider that different
systems can legitimately fall into those categories.

Clearly general purpose systems (eg servers, workstations, etc) with
*full featured firmware* fall into category b. Linux already basically
has stable DT for those systems - but the firmware is expected to do
lots of work and hide all the low level details from the
kernel. Basically, the DT should stick to approximately ePAR and
everything else is hidden by the firmware. This is essentially how x86
works and achieves its compatibility.

Systems that have minimalist firmware, where firmware functions are
pushed into the kernel and the DT is now required to describe
intricate and unique SOC specific functions are clearly very
different, and I think it makes sense to encourage those environments
to be case 'a'.

Basically, minimalist firmware should have a boot process design that
*can* couple the DT and kernel, while full featured firmware should
keep them seperate. IMHO that should be the clear message to people
implementing this stuff.

After enough time the DT for 'a' should become stable and churn free,
but expecting/demanding that from day 0 is not helping anyone, IMHO.

Jason

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Sat, Jul 27, 2013 at 11:51:06AM -0700, Tomasz Figa wrote:

> Well, it depends on how we use the DT. There are (at least) two possible 
> usage scenarios:
> 
>  a) using DT as direct replacement for board files - this means that you 
>     are free to say that DTSes are strictly coupled with kernel version 
>     and you are free to modify the bindings - see the analogy to board 
>     files, where you could modify the platform data structures and could 
>     not directly copy board file from one kernel version to another,
> 
>  b) using DT as an ABI - this is the original way, i.e. define stable 
>     bindings and make sure that anu DTB built for older kernel will work,
>     with equal or greater set of functionality on newer kernels.
> 
> Now I believe in this thread the point whether we should use a) or b) or a 
> combination of both has been raised.

Right, and I think it is very important to consider that different
systems can legitimately fall into those categories.

Clearly general purpose systems (eg servers, workstations, etc) with
*full featured firmware* fall into category b. Linux already basically
has stable DT for those systems - but the firmware is expected to do
lots of work and hide all the low level details from the
kernel. Basically, the DT should stick to approximately ePAR and
everything else is hidden by the firmware. This is essentially how x86
works and achieves its compatibility.

Systems that have minimalist firmware, where firmware functions are
pushed into the kernel and the DT is now required to describe
intricate and unique SOC specific functions are clearly very
different, and I think it makes sense to encourage those environments
to be case 'a'.

Basically, minimalist firmware should have a boot process design that
*can* couple the DT and kernel, while full featured firmware should
keep them seperate. IMHO that should be the clear message to people
implementing this stuff.

After enough time the DT for 'a' should become stable and churn free,
but expecting/demanding that from day 0 is not helping anyone, IMHO.

Jason

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Mon, Jul 29, 2013 at 11:54:00AM -0600, Jason Gunthorpe wrote:
> There is no way I can possibly ship a product with a DT that is
> finished. I can't tie my company's product release cycles to the whims
> of the kernel community.
> 
> So embedded people are going to ship with unfinished DT and upgrade
> later. They have to. There is no choice. Stable DT doesn't change
> anything unless you can create perfect stable bindings for a new SOC
> instantaneously.
> 
> This is where I see the whole disconnect in this
> discussion. General-purpose and embedded are *DIFFERENT* users, and
> they have different use-cases and different needs.

We have had for the last 20 odd years a stable kernel syscall ABI, in
that syscalls which were around 20 years ago are still around today,
and still function according to how they're meant to.

However, some syscalls were found to be insufficient for todays needs,
so they got augmented.  For example, the uid/gid IDs used to be 16-bit.
We now have 32-bit versions, but the 16-bit versions are still there.
(Since ARM started out from early 1.0 times, it too has the 16-bit
syscalls, and _still_ does.)

So, "stable" is possible.

What has to be realised to achieve that it requires nothing more than
a "keep existing stuff working" attitude towards it.  When you need to
change the interface, you supplement it, leaving the old version to
work in the same manner that it used to (yes, you should choose to
implement the old interfaces using the new stuff, just like we support
the 16-bit UID calls but internally deal with 32-bit UIDs.)

What does it take?  Good practice, care, thought and planning.  All
the qualities which should already be present for kernel _engineers_.
Not an "lets create something for me, I don't care about anyone else"
attitude.

We can draw the line at an interface becoming stable in exactly the
same way that we do every other "stable" interface like syscalls - if
it's in a -final kernel, then it has been released at that point as a
stable interface to the world.  That's how syscalls are dealt with -
if a syscall is supported by a -final kernel, then that interface
immediately becomes part of the userland ABI and can't be changed or
deleted without some really serious thought, a migration path, and
sufficient (which means many years) warning to users that it's obsolete.

If that is followed, then there is absolutely no reason why a "Stable
DT" is not possible - one which it's possible to write a DT file today,
and it should still work in 20 years time with updated kernels.  That's
what a stable interface _should_ allow, and this is what DT _should_ be.

It should be possible to say "I have a XYZ ethernet device, it is
hooked up like this, uses this interrupt" etc and that same way work
in 20 years time to describe exactly the same connections.

Sure, we can create a "staging" directory for stuff which we want to
merge but is not deemed to be properly fit for the kernel, and deem
_that_ stuff to be unstable - just like drivers/stable.  Maybe some
of it can live in drivers/stable too...

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Mon, Jul 29, 2013 at 11:54:00AM -0600, Jason Gunthorpe wrote:
> There is no way I can possibly ship a product with a DT that is
> finished. I can't tie my company's product release cycles to the whims
> of the kernel community.
> 
> So embedded people are going to ship with unfinished DT and upgrade
> later. They have to. There is no choice. Stable DT doesn't change
> anything unless you can create perfect stable bindings for a new SOC
> instantaneously.
> 
> This is where I see the whole disconnect in this
> discussion. General-purpose and embedded are *DIFFERENT* users, and
> they have different use-cases and different needs.

We have had for the last 20 odd years a stable kernel syscall ABI, in
that syscalls which were around 20 years ago are still around today,
and still function according to how they're meant to.

However, some syscalls were found to be insufficient for todays needs,
so they got augmented.  For example, the uid/gid IDs used to be 16-bit.
We now have 32-bit versions, but the 16-bit versions are still there.
(Since ARM started out from early 1.0 times, it too has the 16-bit
syscalls, and _still_ does.)

So, "stable" is possible.

What has to be realised to achieve that it requires nothing more than
a "keep existing stuff working" attitude towards it.  When you need to
change the interface, you supplement it, leaving the old version to
work in the same manner that it used to (yes, you should choose to
implement the old interfaces using the new stuff, just like we support
the 16-bit UID calls but internally deal with 32-bit UIDs.)

What does it take?  Good practice, care, thought and planning.  All
the qualities which should already be present for kernel _engineers_.
Not an "lets create something for me, I don't care about anyone else"
attitude.

We can draw the line at an interface becoming stable in exactly the
same way that we do every other "stable" interface like syscalls - if
it's in a -final kernel, then it has been released at that point as a
stable interface to the world.  That's how syscalls are dealt with -
if a syscall is supported by a -final kernel, then that interface
immediately becomes part of the userland ABI and can't be changed or
deleted without some really serious thought, a migration path, and
sufficient (which means many years) warning to users that it's obsolete.

If that is followed, then there is absolutely no reason why a "Stable
DT" is not possible - one which it's possible to write a DT file today,
and it should still work in 20 years time with updated kernels.  That's
what a stable interface _should_ allow, and this is what DT _should_ be.

It should be possible to say "I have a XYZ ethernet device, it is
hooked up like this, uses this interrupt" etc and that same way work
in 20 years time to describe exactly the same connections.

Sure, we can create a "staging" directory for stuff which we want to
merge but is not deemed to be properly fit for the kernel, and deem
_that_ stuff to be unstable - just like drivers/stable.  Maybe some
of it can live in drivers/stable too...

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Mon, Jul 29, 2013 at 12:05:13PM -0600, Jason Gunthorpe wrote:
> Clearly general purpose systems (eg servers, workstations, etc) with
> *full featured firmware* fall into category b. Linux already basically
> has stable DT for those systems - but the firmware is expected to do
> lots of work and hide all the low level details from the
> kernel. Basically, the DT should stick to approximately ePAR and
> everything else is hidden by the firmware. This is essentially how x86
> works and achieves its compatibility.

and the downside of placing that trust in firmware is that it very often
leads to bugs which can't be solved.

I have a set of patches for my laptop/docking station which gets the
serial/printer ports working.  I've had them for many years now - and
I thought I had a problem sorted.  I was wrong - the nice helpful ACPI
reports on dock that the serial port is enabled, but the port is
actually disabled in hardware.

If I have ACPI/PNP stuff setup to always believe that this port is
disabled, then on boot with the docking station connected, it comes up
initially as ttyS0, and then, because the port is "disabled", the
resources are busy so it gets reassigned to a different IO port.
Undocking/redocking works because the port is now properly re-enabled
after a dock event.

If I report the ACPI status to the kernel, then on boot, it correctly
remains as ttyS0.  However, after an undock/redock event, Linux
believes that the port is back, but any attempt to use it gets the
"LSR safety check engaged" message, because the port is actually
disabled.  The only way to get it working again is to manually unbind
the device and rebind it - so the ACPI disable + enable methods get
called.

I've had this problem for years, and it's basically unfixable for me -
it's only fixable if you are IBM and can reflash the BIOS with alternative
ACPI tables which are correct.  And I won't submit these patches all the
time that I don't know they work correctly.

That's the problem with "compatible systems" - you give away trust that
someone else will do the right thing, and if they don't, you may be
screwed in a way that is can not be worked around by quirks or similar
- just like the above.

Firmware is swings and roundabouts.  You give up some control to a
third party so you don't have to think about XYZ, and you hope that
they get it right.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Mon, Jul 29, 2013 at 12:05:13PM -0600, Jason Gunthorpe wrote:
> Clearly general purpose systems (eg servers, workstations, etc) with
> *full featured firmware* fall into category b. Linux already basically
> has stable DT for those systems - but the firmware is expected to do
> lots of work and hide all the low level details from the
> kernel. Basically, the DT should stick to approximately ePAR and
> everything else is hidden by the firmware. This is essentially how x86
> works and achieves its compatibility.

and the downside of placing that trust in firmware is that it very often
leads to bugs which can't be solved.

I have a set of patches for my laptop/docking station which gets the
serial/printer ports working.  I've had them for many years now - and
I thought I had a problem sorted.  I was wrong - the nice helpful ACPI
reports on dock that the serial port is enabled, but the port is
actually disabled in hardware.

If I have ACPI/PNP stuff setup to always believe that this port is
disabled, then on boot with the docking station connected, it comes up
initially as ttyS0, and then, because the port is "disabled", the
resources are busy so it gets reassigned to a different IO port.
Undocking/redocking works because the port is now properly re-enabled
after a dock event.

If I report the ACPI status to the kernel, then on boot, it correctly
remains as ttyS0.  However, after an undock/redock event, Linux
believes that the port is back, but any attempt to use it gets the
"LSR safety check engaged" message, because the port is actually
disabled.  The only way to get it working again is to manually unbind
the device and rebind it - so the ACPI disable + enable methods get
called.

I've had this problem for years, and it's basically unfixable for me -
it's only fixable if you are IBM and can reflash the BIOS with alternative
ACPI tables which are correct.  And I won't submit these patches all the
time that I don't know they work correctly.

That's the problem with "compatible systems" - you give away trust that
someone else will do the right thing, and if they don't, you may be
screwed in a way that is can not be worked around by quirks or similar
- just like the above.

Firmware is swings and roundabouts.  You give up some control to a
third party so you don't have to think about XYZ, and you hope that
they get it right.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Mon, Jul 29, 2013 at 09:31:23AM +0200, Maxime Ripard wrote:
> 
> I'm afraid this kind of use case will never be properly supported, DT
> stable ABI or not.
> 
> Think about this: what kernel will actually be shipped in that board?
> Most likely, it will be a BSP kernel from the vendor. Does the vendor
> will have made that commitment to have a stable ABI for the DT? Will it
> use the same bindings than mainline? Do we want to support all the crazy
> bindings every vendor will come up with?
> 
> I'm afraid the answer to these three questions will most of the time be
> "no.".

Yes, I know, and it is sad but true.
 
We can't stop the vendors from shipping half-baked BSPs. I really
don't mind that they do that. After all, they want to get *something*
working when they launch their chips.

> That doesn't mean we shouldn't aim for *mainline* having a stable DT
> ABI, but that kind of use case doesn't seem very realistic to me.

Right, we can and should do better. I got the beaglebone Ethernet
working in mainline (not by writing the driver, but by complaining
over and over again). I except that it will continue to work and not
fall victim to some random DT change.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Mon, Jul 29, 2013 at 09:31:23AM +0200, Maxime Ripard wrote:
> 
> I'm afraid this kind of use case will never be properly supported, DT
> stable ABI or not.
> 
> Think about this: what kernel will actually be shipped in that board?
> Most likely, it will be a BSP kernel from the vendor. Does the vendor
> will have made that commitment to have a stable ABI for the DT? Will it
> use the same bindings than mainline? Do we want to support all the crazy
> bindings every vendor will come up with?
> 
> I'm afraid the answer to these three questions will most of the time be
> "no.".

Yes, I know, and it is sad but true.
 
We can't stop the vendors from shipping half-baked BSPs. I really
don't mind that they do that. After all, they want to get *something*
working when they launch their chips.

> That doesn't mean we shouldn't aim for *mainline* having a stable DT
> ABI, but that kind of use case doesn't seem very realistic to me.

Right, we can and should do better. I got the beaglebone Ethernet
working in mainline (not by writing the driver, but by complaining
over and over again). I except that it will continue to work and not
fall victim to some random DT change.

Thanks,
Richard

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Mon, Jul 29, 2013 at 07:16:07PM +0100, Russell King - ARM Linux wrote:

> What does it take?  Good practice, care, thought and planning.  All
> the qualities which should already be present for kernel _engineers_.
> Not an "lets create something for me, I don't care about anyone else"
> attitude.

I agree with what you've written, but we are looking at this from
different ends of the problem.

I fully agree you can create a main line kernel GIT tree that has a
stable DT ABI.

However, I as an ODM, with time pressure, cannot wait for the kernel
folks to finish this work. So from my perspective the DT will not be
stable, as I will put whatever interm stuff I choose to have a
shippable product.

Thus I have to design my systems for an unstable DT, and the message
from the kernel community to people in my posistion should be:

  When you ship early with non-mainlined DT schema, design your boot
  system around an unstable DT. Plan to migrate your DT to upstream
  once it becomes finalized.

Here is the rub: Once I design for an unstable DT I simply don't
derive value from the kernel communities work to create a stable DT.

So who is getting the benefit of this work, and is it worth the cost?

Jason

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Mon, Jul 29, 2013 at 07:16:07PM +0100, Russell King - ARM Linux wrote:

> What does it take?  Good practice, care, thought and planning.  All
> the qualities which should already be present for kernel _engineers_.
> Not an "lets create something for me, I don't care about anyone else"
> attitude.

I agree with what you've written, but we are looking at this from
different ends of the problem.

I fully agree you can create a main line kernel GIT tree that has a
stable DT ABI.

However, I as an ODM, with time pressure, cannot wait for the kernel
folks to finish this work. So from my perspective the DT will not be
stable, as I will put whatever interm stuff I choose to have a
shippable product.

Thus I have to design my systems for an unstable DT, and the message
from the kernel community to people in my posistion should be:

  When you ship early with non-mainlined DT schema, design your boot
  system around an unstable DT. Plan to migrate your DT to upstream
  once it becomes finalized.

Here is the rub: Once I design for an unstable DT I simply don't
derive value from the kernel communities work to create a stable DT.

So who is getting the benefit of this work, and is it worth the cost?

Jason

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Mon, Jul 29, 2013 at 08:38:52PM +0200, Richard Cochran wrote:
> 
> Right, we can and should do better. I got the beaglebone Ethernet
> working in mainline (not by writing the driver, but by complaining
> over and over again). I except that it will continue to work and not
                          ^^^^^^
expect

> fall victim to some random DT change.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Mon, Jul 29, 2013 at 08:38:52PM +0200, Richard Cochran wrote:
> 
> Right, we can and should do better. I got the beaglebone Ethernet
> working in mainline (not by writing the driver, but by complaining
> over and over again). I except that it will continue to work and not
                          ^^^^^^
expect

> fall victim to some random DT change.

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3070 bytes --]

On Mon, Jul 29, 2013 at 12:05:13PM -0600, Jason Gunthorpe wrote:
> On Sat, Jul 27, 2013 at 11:51:06AM -0700, Tomasz Figa wrote:
> 
> > Well, it depends on how we use the DT. There are (at least) two possible 
> > usage scenarios:
> > 
> >  a) using DT as direct replacement for board files - this means that you 
> >     are free to say that DTSes are strictly coupled with kernel version 
> >     and you are free to modify the bindings - see the analogy to board 
> >     files, where you could modify the platform data structures and could 
> >     not directly copy board file from one kernel version to another,
> > 
> >  b) using DT as an ABI - this is the original way, i.e. define stable 
> >     bindings and make sure that anu DTB built for older kernel will work,
> >     with equal or greater set of functionality on newer kernels.
> > 
> > Now I believe in this thread the point whether we should use a) or b) or a 
> > combination of both has been raised.
> 
> Right, and I think it is very important to consider that different
> systems can legitimately fall into those categories.
> 
> Clearly general purpose systems (eg servers, workstations, etc) with
> *full featured firmware* fall into category b. Linux already basically
> has stable DT for those systems - but the firmware is expected to do
> lots of work and hide all the low level details from the
> kernel.  Basically, the DT should stick to approximately ePAR and
> everything else is hidden by the firmware.

No.  With the exception of the hypervisor/virtualization extensions,
ePAPR describes (for now) an entirely passive firmware interface.
That is, once the handover to the OS has happened, there is *no*
further firmware interaction.  It is not capable of hiding any details
from the OS, except those which can be done by one-time
initialization.

In fact, a guiding principle of ePAPR's design was that except in
cases where it's *much* easier for the firmware to do things, the OS
should be expected to do it, because the OS is usually easier to fix
than the firmware.

> This is essentially how x86
> works and achieves its compatibility.
> 
> Systems that have minimalist firmware, where firmware functions are
> pushed into the kernel and the DT is now required to describe
> intricate and unique SOC specific functions are clearly very
> different, and I think it makes sense to encourage those environments
> to be case 'a'.
> 
> Basically, minimalist firmware should have a boot process design that
> *can* couple the DT and kernel, while full featured firmware should
> keep them seperate. IMHO that should be the clear message to people
> implementing this stuff.
> 
> After enough time the DT for 'a' should become stable and churn free,
> but expecting/demanding that from day 0 is not helping anyone, IMHO.
> 
> Jason

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

On Mon, Jul 29, 2013 at 12:05:13PM -0600, Jason Gunthorpe wrote:
> On Sat, Jul 27, 2013 at 11:51:06AM -0700, Tomasz Figa wrote:
> 
> > Well, it depends on how we use the DT. There are (at least) two possible 
> > usage scenarios:
> > 
> >  a) using DT as direct replacement for board files - this means that you 
> >     are free to say that DTSes are strictly coupled with kernel version 
> >     and you are free to modify the bindings - see the analogy to board 
> >     files, where you could modify the platform data structures and could 
> >     not directly copy board file from one kernel version to another,
> > 
> >  b) using DT as an ABI - this is the original way, i.e. define stable 
> >     bindings and make sure that anu DTB built for older kernel will work,
> >     with equal or greater set of functionality on newer kernels.
> > 
> > Now I believe in this thread the point whether we should use a) or b) or a 
> > combination of both has been raised.
> 
> Right, and I think it is very important to consider that different
> systems can legitimately fall into those categories.
> 
> Clearly general purpose systems (eg servers, workstations, etc) with
> *full featured firmware* fall into category b. Linux already basically
> has stable DT for those systems - but the firmware is expected to do
> lots of work and hide all the low level details from the
> kernel.  Basically, the DT should stick to approximately ePAR and
> everything else is hidden by the firmware.

No.  With the exception of the hypervisor/virtualization extensions,
ePAPR describes (for now) an entirely passive firmware interface.
That is, once the handover to the OS has happened, there is *no*
further firmware interaction.  It is not capable of hiding any details
from the OS, except those which can be done by one-time
initialization.

In fact, a guiding principle of ePAPR's design was that except in
cases where it's *much* easier for the firmware to do things, the OS
should be expected to do it, because the OS is usually easier to fix
than the firmware.

> This is essentially how x86
> works and achieves its compatibility.
> 
> Systems that have minimalist firmware, where firmware functions are
> pushed into the kernel and the DT is now required to describe
> intricate and unique SOC specific functions are clearly very
> different, and I think it makes sense to encourage those environments
> to be case 'a'.
> 
> Basically, minimalist firmware should have a boot process design that
> *can* couple the DT and kernel, while full featured firmware should
> keep them seperate. IMHO that should be the clear message to people
> implementing this stuff.
> 
> After enough time the DT for 'a' should become stable and churn free,
> but expecting/demanding that from day 0 is not helping anyone, IMHO.
> 
> Jason

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130730/1061cd59/attachment-0001.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Tue, Jul 30, 2013 at 08:20:39AM +1000, David Gibson wrote:

> > Clearly general purpose systems (eg servers, workstations, etc) with
> > *full featured firmware* fall into category b. Linux already basically
> > has stable DT for those systems - but the firmware is expected to do
> > lots of work and hide all the low level details from the
> > kernel.  Basically, the DT should stick to approximately ePAR and
> > everything else is hidden by the firmware.
> 
> No.  With the exception of the hypervisor/virtualization extensions,
> ePAPR describes (for now) an entirely passive firmware interface.
> That is, once the handover to the OS has happened, there is *no*
> further firmware interaction.  It is not capable of hiding any details
> from the OS, except those which can be done by one-time
> initialization.

Well, one-time initialization details are actually exactly one of the
areas I am thinking about. Some of the embedded SOCs have extensive
one time initization code in the kernel that is highly SOC specific
and on x86 it would live in the firmware.

But I see what you mean, ePAPR was the wrong reference, I didn't
carefully check it. I ment the kind of DT use we've seen in SPARC,
POWER servers, Apple stuff, etc - systems explicitly designed so that
new hardware will boot old OSs.

Jason

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Jason Gunthorpe]

On Tue, Jul 30, 2013 at 08:20:39AM +1000, David Gibson wrote:

> > Clearly general purpose systems (eg servers, workstations, etc) with
> > *full featured firmware* fall into category b. Linux already basically
> > has stable DT for those systems - but the firmware is expected to do
> > lots of work and hide all the low level details from the
> > kernel.  Basically, the DT should stick to approximately ePAR and
> > everything else is hidden by the firmware.
> 
> No.  With the exception of the hypervisor/virtualization extensions,
> ePAPR describes (for now) an entirely passive firmware interface.
> That is, once the handover to the OS has happened, there is *no*
> further firmware interaction.  It is not capable of hiding any details
> from the OS, except those which can be done by one-time
> initialization.

Well, one-time initialization details are actually exactly one of the
areas I am thinking about. Some of the embedded SOCs have extensive
one time initization code in the kernel that is highly SOC specific
and on x86 it would live in the firmware.

But I see what you mean, ePAPR was the wrong reference, I didn't
carefully check it. I ment the kind of DT use we've seen in SPARC,
POWER servers, Apple stuff, etc - systems explicitly designed so that
new hardware will boot old OSs.

Jason

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2052 bytes --]

On Mon, Jul 29, 2013 at 05:14:25PM -0600, Jason Gunthorpe wrote:
> On Tue, Jul 30, 2013 at 08:20:39AM +1000, David Gibson wrote:
> 
> > > Clearly general purpose systems (eg servers, workstations, etc) with
> > > *full featured firmware* fall into category b. Linux already basically
> > > has stable DT for those systems - but the firmware is expected to do
> > > lots of work and hide all the low level details from the
> > > kernel.  Basically, the DT should stick to approximately ePAR and
> > > everything else is hidden by the firmware.
> > 
> > No.  With the exception of the hypervisor/virtualization extensions,
> > ePAPR describes (for now) an entirely passive firmware interface.
> > That is, once the handover to the OS has happened, there is *no*
> > further firmware interaction.  It is not capable of hiding any details
> > from the OS, except those which can be done by one-time
> > initialization.
> 
> Well, one-time initialization details are actually exactly one of the
> areas I am thinking about. Some of the embedded SOCs have extensive
> one time initization code in the kernel that is highly SOC specific
> and on x86 it would live in the firmware.
> 
> But I see what you mean, ePAPR was the wrong reference, I didn't
> carefully check it. I ment the kind of DT use we've seen in SPARC,
> POWER servers, Apple stuff, etc - systems explicitly designed so that
> new hardware will boot old OSs.

Yeah, and at least for POWER servers and Apples, like every other
attempt at this, ever, it at best sorta-kinda worked.  It's not *as*
bad as the mess of broken BIOSes on x86 (mostly due to smaller
variety), but there's still plenty of broken crap in Apple workstation
and IBM server firmwares and device trees.

You see a clear line between "full featured" and "minimal" firmwares
where none exists.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

On Mon, Jul 29, 2013 at 05:14:25PM -0600, Jason Gunthorpe wrote:
> On Tue, Jul 30, 2013 at 08:20:39AM +1000, David Gibson wrote:
> 
> > > Clearly general purpose systems (eg servers, workstations, etc) with
> > > *full featured firmware* fall into category b. Linux already basically
> > > has stable DT for those systems - but the firmware is expected to do
> > > lots of work and hide all the low level details from the
> > > kernel.  Basically, the DT should stick to approximately ePAR and
> > > everything else is hidden by the firmware.
> > 
> > No.  With the exception of the hypervisor/virtualization extensions,
> > ePAPR describes (for now) an entirely passive firmware interface.
> > That is, once the handover to the OS has happened, there is *no*
> > further firmware interaction.  It is not capable of hiding any details
> > from the OS, except those which can be done by one-time
> > initialization.
> 
> Well, one-time initialization details are actually exactly one of the
> areas I am thinking about. Some of the embedded SOCs have extensive
> one time initization code in the kernel that is highly SOC specific
> and on x86 it would live in the firmware.
> 
> But I see what you mean, ePAPR was the wrong reference, I didn't
> carefully check it. I ment the kind of DT use we've seen in SPARC,
> POWER servers, Apple stuff, etc - systems explicitly designed so that
> new hardware will boot old OSs.

Yeah, and at least for POWER servers and Apples, like every other
attempt at this, ever, it at best sorta-kinda worked.  It's not *as*
bad as the mess of broken BIOSes on x86 (mostly due to smaller
variety), but there's still plenty of broken crap in Apple workstation
and IBM server firmwares and device trees.

You see a clear line between "full featured" and "minimal" firmwares
where none exists.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130730/8890d07e/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5028 bytes --]

On Sat, Jul 27, 2013 at 10:11:16PM -0700, James Bottomley wrote:
> On Sat, 2013-07-27 at 21:28 -0600, Grant Likely wrote:
> > On Sat, Jul 27, 2013 at 2:25 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> > > On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl@gmail.com <jonsmirl@gmail.com> wrote:
> > >> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> > >>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
> > >>>> Let's see how many people go and scream if I say this: Too bad .dts files
> > >>>> are not done using XML format as DT bindings could be described using XML
> > >>>> Schema.
> > >>>
> > >>> Draft an example and show us how it would look!  :-)  There is
> > >>> absolutely nothing preventing us from expressing a DT in XML format,
> > >>> or even using XSLT to define DT schema while still using our current
> > >>> .dts syntax. It would be trivial to do lossless translation between
> > >>> .dts syntax and xml.
> > >>>
> > >>> The problem that I have with XML and XSLT is that it is very verbose
> > >>> and not entirely friendly to mere-mortals. However, I'm more than
> > >>> willing to be proved wrong on this point.
> > >>
> > >> I considered this approach a while ago and discarded it. It would work
> > >> but it is just too much of a Frankenstein monster.
> > >>
> > >> Much cleaner to modify dtc to take a schema as part of the compilation
> > >> process. The schema language itself has no requirement to look like
> > >> DTS syntax. Whoever wrote dtc probably has a favorite language that
> > >> would be good for writing schemas in.
> > >
> > > Making it part of dtc is a required feature as far as I'm concerned.
> > > Using XML/XSLT and dtc-integration are not mutually exclusive, but I
> > > digress.
> > 
> > Oops, ignore the XSLT bit. XSLT isn't schema and has no bearing on the
> > discussion of schema. Sorry for the noise.
> 
> XSLT is a transform language ... you'd use it say to transform xml to
> dtc, so it would be an integral component of an xml/xslt based schema.
> 
> If you want actually to describe and have validated the xml schema
> itself, then you'd use xsd (XML schema description language) and its
> associated tools.
> 
> I'm not saying you *should* do this, just that it's possible (plus I've
> just blown my kernel cred by knowing about xml, sigh).

Heh.  So, it was said in jest, but that actually raises an important
point.

There are basically two criteria to keep in mind for our
representation of schemas:
   1) Adequate expressiveness to validate a sufficiently large part,
of a sufficiently large number of bindings to be useful.
   2) Ease of use and ease of learning **for the target audience**.

To the best of my knowledge xsd would do well on (1), but I'm not
convinced it does very well on (2).  In an environment where XML was
already widely used, XSD would make perfect sense.  Here, I think it
would be pretty ugly to wire onto the existing DT tools and
infrastructure, and unpleasantly unfamiliar for many kernel and board
developers trying to work with DT schemas.


So, by way of investigation, let me propose an alternative expression
of schemas, that I'm also not convinced we should do, but is possible
and expressive.  It's illustrative, because it's kind of the polar
opposite approach to XSD: just use C.

dtc already has a (so far limited) "checks" mechanism which verifies
various aspects of DT content.  These are implemented by C functions
in checks.c.  There's obviously ample expressiveness - you can express
any constraint you want that way.  It can be pretty verbose, and
fiddly.  A good library of helper functions can mitigate that, but
it's not clear how much.  On the other hand, a very good fraction of
people working with this will already be familiar with C, which is a
big plus.  This is, after all, the reason that the dts syntax is
chiefly C inspired.

Now, in practice, I think we will want a more convenient schema
language (just as we wanted dts, rather than manually constructing
FDTs as C structures).  But I absolutely do think, that the schema
handling should be handled as plugins to the checks mechanism -
basically we'd have a validate_schemas() check function.

I also think we should consider the option of having a simple and
straightforward schema language which handles, say, 80% of cases with
a fall back to C for the 20% of curly cases.  That might actually be
simpler to work with in practice than a schema language which can
express absolutely anything, at the cost of being awkward for simple
cases or difficult to get your head around.

Remember, a schema language is only a win if it is - for the target
audience - more convenient to express schemas in than C.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

On Sat, Jul 27, 2013 at 10:11:16PM -0700, James Bottomley wrote:
> On Sat, 2013-07-27 at 21:28 -0600, Grant Likely wrote:
> > On Sat, Jul 27, 2013 at 2:25 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> > > On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
> > >> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> > >>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
> > >>>> Let's see how many people go and scream if I say this: Too bad .dts files
> > >>>> are not done using XML format as DT bindings could be described using XML
> > >>>> Schema.
> > >>>
> > >>> Draft an example and show us how it would look!  :-)  There is
> > >>> absolutely nothing preventing us from expressing a DT in XML format,
> > >>> or even using XSLT to define DT schema while still using our current
> > >>> .dts syntax. It would be trivial to do lossless translation between
> > >>> .dts syntax and xml.
> > >>>
> > >>> The problem that I have with XML and XSLT is that it is very verbose
> > >>> and not entirely friendly to mere-mortals. However, I'm more than
> > >>> willing to be proved wrong on this point.
> > >>
> > >> I considered this approach a while ago and discarded it. It would work
> > >> but it is just too much of a Frankenstein monster.
> > >>
> > >> Much cleaner to modify dtc to take a schema as part of the compilation
> > >> process. The schema language itself has no requirement to look like
> > >> DTS syntax. Whoever wrote dtc probably has a favorite language that
> > >> would be good for writing schemas in.
> > >
> > > Making it part of dtc is a required feature as far as I'm concerned.
> > > Using XML/XSLT and dtc-integration are not mutually exclusive, but I
> > > digress.
> > 
> > Oops, ignore the XSLT bit. XSLT isn't schema and has no bearing on the
> > discussion of schema. Sorry for the noise.
> 
> XSLT is a transform language ... you'd use it say to transform xml to
> dtc, so it would be an integral component of an xml/xslt based schema.
> 
> If you want actually to describe and have validated the xml schema
> itself, then you'd use xsd (XML schema description language) and its
> associated tools.
> 
> I'm not saying you *should* do this, just that it's possible (plus I've
> just blown my kernel cred by knowing about xml, sigh).

Heh.  So, it was said in jest, but that actually raises an important
point.

There are basically two criteria to keep in mind for our
representation of schemas:
   1) Adequate expressiveness to validate a sufficiently large part,
of a sufficiently large number of bindings to be useful.
   2) Ease of use and ease of learning **for the target audience**.

To the best of my knowledge xsd would do well on (1), but I'm not
convinced it does very well on (2).  In an environment where XML was
already widely used, XSD would make perfect sense.  Here, I think it
would be pretty ugly to wire onto the existing DT tools and
infrastructure, and unpleasantly unfamiliar for many kernel and board
developers trying to work with DT schemas.


So, by way of investigation, let me propose an alternative expression
of schemas, that I'm also not convinced we should do, but is possible
and expressive.  It's illustrative, because it's kind of the polar
opposite approach to XSD: just use C.

dtc already has a (so far limited) "checks" mechanism which verifies
various aspects of DT content.  These are implemented by C functions
in checks.c.  There's obviously ample expressiveness - you can express
any constraint you want that way.  It can be pretty verbose, and
fiddly.  A good library of helper functions can mitigate that, but
it's not clear how much.  On the other hand, a very good fraction of
people working with this will already be familiar with C, which is a
big plus.  This is, after all, the reason that the dts syntax is
chiefly C inspired.

Now, in practice, I think we will want a more convenient schema
language (just as we wanted dts, rather than manually constructing
FDTs as C structures).  But I absolutely do think, that the schema
handling should be handled as plugins to the checks mechanism -
basically we'd have a validate_schemas() check function.

I also think we should consider the option of having a simple and
straightforward schema language which handles, say, 80% of cases with
a fall back to C for the 20% of curly cases.  That might actually be
simpler to work with in practice than a schema language which can
express absolutely anything, at the cost of being awkward for simple
cases or difficult to get your head around.

Remember, a schema language is only a win if it is - for the target
audience - more convenient to express schemas in than C.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130730/508642a1/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Mon, Jul 29, 2013 at 9:44 PM, David Gibson
<david@gibson.dropbear.id.au> wrote:
> On Sat, Jul 27, 2013 at 10:11:16PM -0700, James Bottomley wrote:
>> On Sat, 2013-07-27 at 21:28 -0600, Grant Likely wrote:
>> > On Sat, Jul 27, 2013 at 2:25 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> > > On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl@gmail.com <jonsmirl@gmail.com> wrote:
>> > >> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> > >>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
>> > >>>> Let's see how many people go and scream if I say this: Too bad .dts files
>> > >>>> are not done using XML format as DT bindings could be described using XML
>> > >>>> Schema.
>> > >>>
>> > >>> Draft an example and show us how it would look!  :-)  There is
>> > >>> absolutely nothing preventing us from expressing a DT in XML format,
>> > >>> or even using XSLT to define DT schema while still using our current
>> > >>> .dts syntax. It would be trivial to do lossless translation between
>> > >>> .dts syntax and xml.
>> > >>>
>> > >>> The problem that I have with XML and XSLT is that it is very verbose
>> > >>> and not entirely friendly to mere-mortals. However, I'm more than
>> > >>> willing to be proved wrong on this point.
>> > >>
>> > >> I considered this approach a while ago and discarded it. It would work
>> > >> but it is just too much of a Frankenstein monster.
>> > >>
>> > >> Much cleaner to modify dtc to take a schema as part of the compilation
>> > >> process. The schema language itself has no requirement to look like
>> > >> DTS syntax. Whoever wrote dtc probably has a favorite language that
>> > >> would be good for writing schemas in.
>> > >
>> > > Making it part of dtc is a required feature as far as I'm concerned.
>> > > Using XML/XSLT and dtc-integration are not mutually exclusive, but I
>> > > digress.
>> >
>> > Oops, ignore the XSLT bit. XSLT isn't schema and has no bearing on the
>> > discussion of schema. Sorry for the noise.
>>
>> XSLT is a transform language ... you'd use it say to transform xml to
>> dtc, so it would be an integral component of an xml/xslt based schema.
>>
>> If you want actually to describe and have validated the xml schema
>> itself, then you'd use xsd (XML schema description language) and its
>> associated tools.
>>
>> I'm not saying you *should* do this, just that it's possible (plus I've
>> just blown my kernel cred by knowing about xml, sigh).
>
> Heh.  So, it was said in jest, but that actually raises an important
> point.
>
> There are basically two criteria to keep in mind for our
> representation of schemas:
>    1) Adequate expressiveness to validate a sufficiently large part,
> of a sufficiently large number of bindings to be useful.
>    2) Ease of use and ease of learning **for the target audience**.
>
> To the best of my knowledge xsd would do well on (1), but I'm not
> convinced it does very well on (2).  In an environment where XML was
> already widely used, XSD would make perfect sense.  Here, I think it
> would be pretty ugly to wire onto the existing DT tools and
> infrastructure, and unpleasantly unfamiliar for many kernel and board
> developers trying to work with DT schemas.
>
>
> So, by way of investigation, let me propose an alternative expression
> of schemas, that I'm also not convinced we should do, but is possible
> and expressive.  It's illustrative, because it's kind of the polar
> opposite approach to XSD: just use C.
>
> dtc already has a (so far limited) "checks" mechanism which verifies
> various aspects of DT content.  These are implemented by C functions
> in checks.c.  There's obviously ample expressiveness - you can express
> any constraint you want that way.  It can be pretty verbose, and
> fiddly.  A good library of helper functions can mitigate that, but
> it's not clear how much.  On the other hand, a very good fraction of
> people working with this will already be familiar with C, which is a
> big plus.  This is, after all, the reason that the dts syntax is
> chiefly C inspired.
>
> Now, in practice, I think we will want a more convenient schema
> language (just as we wanted dts, rather than manually constructing
> FDTs as C structures).  But I absolutely do think, that the schema
> handling should be handled as plugins to the checks mechanism -
> basically we'd have a validate_schemas() check function.
>
> I also think we should consider the option of having a simple and
> straightforward schema language which handles, say, 80% of cases with
> a fall back to C for the 20% of curly cases.  That might actually be
> simpler to work with in practice than a schema language which can
> express absolutely anything, at the cost of being awkward for simple
> cases or difficult to get your head around.

Would C++ work? You can use operating overloading and templates to
change the syntax into something that doesn't even resemble C any
more.

>
> Remember, a schema language is only a win if it is - for the target
> audience - more convenient to express schemas in than C.
>
> --
> David Gibson                    | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
>                                 | _way_ _around_!
> http://www.ozlabs.org/~dgibson



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Mon, Jul 29, 2013 at 9:44 PM, David Gibson
<david@gibson.dropbear.id.au> wrote:
> On Sat, Jul 27, 2013 at 10:11:16PM -0700, James Bottomley wrote:
>> On Sat, 2013-07-27 at 21:28 -0600, Grant Likely wrote:
>> > On Sat, Jul 27, 2013 at 2:25 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> > > On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
>> > >> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> > >>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
>> > >>>> Let's see how many people go and scream if I say this: Too bad .dts files
>> > >>>> are not done using XML format as DT bindings could be described using XML
>> > >>>> Schema.
>> > >>>
>> > >>> Draft an example and show us how it would look!  :-)  There is
>> > >>> absolutely nothing preventing us from expressing a DT in XML format,
>> > >>> or even using XSLT to define DT schema while still using our current
>> > >>> .dts syntax. It would be trivial to do lossless translation between
>> > >>> .dts syntax and xml.
>> > >>>
>> > >>> The problem that I have with XML and XSLT is that it is very verbose
>> > >>> and not entirely friendly to mere-mortals. However, I'm more than
>> > >>> willing to be proved wrong on this point.
>> > >>
>> > >> I considered this approach a while ago and discarded it. It would work
>> > >> but it is just too much of a Frankenstein monster.
>> > >>
>> > >> Much cleaner to modify dtc to take a schema as part of the compilation
>> > >> process. The schema language itself has no requirement to look like
>> > >> DTS syntax. Whoever wrote dtc probably has a favorite language that
>> > >> would be good for writing schemas in.
>> > >
>> > > Making it part of dtc is a required feature as far as I'm concerned.
>> > > Using XML/XSLT and dtc-integration are not mutually exclusive, but I
>> > > digress.
>> >
>> > Oops, ignore the XSLT bit. XSLT isn't schema and has no bearing on the
>> > discussion of schema. Sorry for the noise.
>>
>> XSLT is a transform language ... you'd use it say to transform xml to
>> dtc, so it would be an integral component of an xml/xslt based schema.
>>
>> If you want actually to describe and have validated the xml schema
>> itself, then you'd use xsd (XML schema description language) and its
>> associated tools.
>>
>> I'm not saying you *should* do this, just that it's possible (plus I've
>> just blown my kernel cred by knowing about xml, sigh).
>
> Heh.  So, it was said in jest, but that actually raises an important
> point.
>
> There are basically two criteria to keep in mind for our
> representation of schemas:
>    1) Adequate expressiveness to validate a sufficiently large part,
> of a sufficiently large number of bindings to be useful.
>    2) Ease of use and ease of learning **for the target audience**.
>
> To the best of my knowledge xsd would do well on (1), but I'm not
> convinced it does very well on (2).  In an environment where XML was
> already widely used, XSD would make perfect sense.  Here, I think it
> would be pretty ugly to wire onto the existing DT tools and
> infrastructure, and unpleasantly unfamiliar for many kernel and board
> developers trying to work with DT schemas.
>
>
> So, by way of investigation, let me propose an alternative expression
> of schemas, that I'm also not convinced we should do, but is possible
> and expressive.  It's illustrative, because it's kind of the polar
> opposite approach to XSD: just use C.
>
> dtc already has a (so far limited) "checks" mechanism which verifies
> various aspects of DT content.  These are implemented by C functions
> in checks.c.  There's obviously ample expressiveness - you can express
> any constraint you want that way.  It can be pretty verbose, and
> fiddly.  A good library of helper functions can mitigate that, but
> it's not clear how much.  On the other hand, a very good fraction of
> people working with this will already be familiar with C, which is a
> big plus.  This is, after all, the reason that the dts syntax is
> chiefly C inspired.
>
> Now, in practice, I think we will want a more convenient schema
> language (just as we wanted dts, rather than manually constructing
> FDTs as C structures).  But I absolutely do think, that the schema
> handling should be handled as plugins to the checks mechanism -
> basically we'd have a validate_schemas() check function.
>
> I also think we should consider the option of having a simple and
> straightforward schema language which handles, say, 80% of cases with
> a fall back to C for the 20% of curly cases.  That might actually be
> simpler to work with in practice than a schema language which can
> express absolutely anything, at the cost of being awkward for simple
> cases or difficult to get your head around.

Would C++ work? You can use operating overloading and templates to
change the syntax into something that doesn't even resemble C any
more.

>
> Remember, a schema language is only a win if it is - for the target
> audience - more convenient to express schemas in than C.
>
> --
> David Gibson                    | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
>                                 | _way_ _around_!
> http://www.ozlabs.org/~dgibson



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5619 bytes --]

On Mon, Jul 29, 2013 at 10:15:12PM -0400, jonsmirl@gmail.com wrote:
> On Mon, Jul 29, 2013 at 9:44 PM, David Gibson
> <david@gibson.dropbear.id.au> wrote:
> > On Sat, Jul 27, 2013 at 10:11:16PM -0700, James Bottomley wrote:
> >> On Sat, 2013-07-27 at 21:28 -0600, Grant Likely wrote:
> >> > On Sat, Jul 27, 2013 at 2:25 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> >> > > On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl@gmail.com <jonsmirl@gmail.com> wrote:
> >> > >> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> >> > >>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
> >> > >>>> Let's see how many people go and scream if I say this: Too bad .dts files
> >> > >>>> are not done using XML format as DT bindings could be described using XML
> >> > >>>> Schema.
> >> > >>>
> >> > >>> Draft an example and show us how it would look!  :-)  There is
> >> > >>> absolutely nothing preventing us from expressing a DT in XML format,
> >> > >>> or even using XSLT to define DT schema while still using our current
> >> > >>> .dts syntax. It would be trivial to do lossless translation between
> >> > >>> .dts syntax and xml.
> >> > >>>
> >> > >>> The problem that I have with XML and XSLT is that it is very verbose
> >> > >>> and not entirely friendly to mere-mortals. However, I'm more than
> >> > >>> willing to be proved wrong on this point.
> >> > >>
> >> > >> I considered this approach a while ago and discarded it. It would work
> >> > >> but it is just too much of a Frankenstein monster.
> >> > >>
> >> > >> Much cleaner to modify dtc to take a schema as part of the compilation
> >> > >> process. The schema language itself has no requirement to look like
> >> > >> DTS syntax. Whoever wrote dtc probably has a favorite language that
> >> > >> would be good for writing schemas in.
> >> > >
> >> > > Making it part of dtc is a required feature as far as I'm concerned.
> >> > > Using XML/XSLT and dtc-integration are not mutually exclusive, but I
> >> > > digress.
> >> >
> >> > Oops, ignore the XSLT bit. XSLT isn't schema and has no bearing on the
> >> > discussion of schema. Sorry for the noise.
> >>
> >> XSLT is a transform language ... you'd use it say to transform xml to
> >> dtc, so it would be an integral component of an xml/xslt based schema.
> >>
> >> If you want actually to describe and have validated the xml schema
> >> itself, then you'd use xsd (XML schema description language) and its
> >> associated tools.
> >>
> >> I'm not saying you *should* do this, just that it's possible (plus I've
> >> just blown my kernel cred by knowing about xml, sigh).
> >
> > Heh.  So, it was said in jest, but that actually raises an important
> > point.
> >
> > There are basically two criteria to keep in mind for our
> > representation of schemas:
> >    1) Adequate expressiveness to validate a sufficiently large part,
> > of a sufficiently large number of bindings to be useful.
> >    2) Ease of use and ease of learning **for the target audience**.
> >
> > To the best of my knowledge xsd would do well on (1), but I'm not
> > convinced it does very well on (2).  In an environment where XML was
> > already widely used, XSD would make perfect sense.  Here, I think it
> > would be pretty ugly to wire onto the existing DT tools and
> > infrastructure, and unpleasantly unfamiliar for many kernel and board
> > developers trying to work with DT schemas.
> >
> >
> > So, by way of investigation, let me propose an alternative expression
> > of schemas, that I'm also not convinced we should do, but is possible
> > and expressive.  It's illustrative, because it's kind of the polar
> > opposite approach to XSD: just use C.
> >
> > dtc already has a (so far limited) "checks" mechanism which verifies
> > various aspects of DT content.  These are implemented by C functions
> > in checks.c.  There's obviously ample expressiveness - you can express
> > any constraint you want that way.  It can be pretty verbose, and
> > fiddly.  A good library of helper functions can mitigate that, but
> > it's not clear how much.  On the other hand, a very good fraction of
> > people working with this will already be familiar with C, which is a
> > big plus.  This is, after all, the reason that the dts syntax is
> > chiefly C inspired.
> >
> > Now, in practice, I think we will want a more convenient schema
> > language (just as we wanted dts, rather than manually constructing
> > FDTs as C structures).  But I absolutely do think, that the schema
> > handling should be handled as plugins to the checks mechanism -
> > basically we'd have a validate_schemas() check function.
> >
> > I also think we should consider the option of having a simple and
> > straightforward schema language which handles, say, 80% of cases with
> > a fall back to C for the 20% of curly cases.  That might actually be
> > simpler to work with in practice than a schema language which can
> > express absolutely anything, at the cost of being awkward for simple
> > cases or difficult to get your head around.
> 
> Would C++ work? You can use operating overloading and templates to
> change the syntax into something that doesn't even resemble C any
> more.

Well, in theory.  But given that dtc and the kernel are both in plain
C, I don't think it's a good idea.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

On Mon, Jul 29, 2013 at 10:15:12PM -0400, jonsmirl at gmail.com wrote:
> On Mon, Jul 29, 2013 at 9:44 PM, David Gibson
> <david@gibson.dropbear.id.au> wrote:
> > On Sat, Jul 27, 2013 at 10:11:16PM -0700, James Bottomley wrote:
> >> On Sat, 2013-07-27 at 21:28 -0600, Grant Likely wrote:
> >> > On Sat, Jul 27, 2013 at 2:25 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> >> > > On Sat, Jul 27, 2013 at 2:01 PM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
> >> > >> On Sat, Jul 27, 2013 at 3:45 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> >> > >>> On Sat, Jul 27, 2013 at 4:59 AM, Arend van Spriel <arend@broadcom.com> wrote:
> >> > >>>> Let's see how many people go and scream if I say this: Too bad .dts files
> >> > >>>> are not done using XML format as DT bindings could be described using XML
> >> > >>>> Schema.
> >> > >>>
> >> > >>> Draft an example and show us how it would look!  :-)  There is
> >> > >>> absolutely nothing preventing us from expressing a DT in XML format,
> >> > >>> or even using XSLT to define DT schema while still using our current
> >> > >>> .dts syntax. It would be trivial to do lossless translation between
> >> > >>> .dts syntax and xml.
> >> > >>>
> >> > >>> The problem that I have with XML and XSLT is that it is very verbose
> >> > >>> and not entirely friendly to mere-mortals. However, I'm more than
> >> > >>> willing to be proved wrong on this point.
> >> > >>
> >> > >> I considered this approach a while ago and discarded it. It would work
> >> > >> but it is just too much of a Frankenstein monster.
> >> > >>
> >> > >> Much cleaner to modify dtc to take a schema as part of the compilation
> >> > >> process. The schema language itself has no requirement to look like
> >> > >> DTS syntax. Whoever wrote dtc probably has a favorite language that
> >> > >> would be good for writing schemas in.
> >> > >
> >> > > Making it part of dtc is a required feature as far as I'm concerned.
> >> > > Using XML/XSLT and dtc-integration are not mutually exclusive, but I
> >> > > digress.
> >> >
> >> > Oops, ignore the XSLT bit. XSLT isn't schema and has no bearing on the
> >> > discussion of schema. Sorry for the noise.
> >>
> >> XSLT is a transform language ... you'd use it say to transform xml to
> >> dtc, so it would be an integral component of an xml/xslt based schema.
> >>
> >> If you want actually to describe and have validated the xml schema
> >> itself, then you'd use xsd (XML schema description language) and its
> >> associated tools.
> >>
> >> I'm not saying you *should* do this, just that it's possible (plus I've
> >> just blown my kernel cred by knowing about xml, sigh).
> >
> > Heh.  So, it was said in jest, but that actually raises an important
> > point.
> >
> > There are basically two criteria to keep in mind for our
> > representation of schemas:
> >    1) Adequate expressiveness to validate a sufficiently large part,
> > of a sufficiently large number of bindings to be useful.
> >    2) Ease of use and ease of learning **for the target audience**.
> >
> > To the best of my knowledge xsd would do well on (1), but I'm not
> > convinced it does very well on (2).  In an environment where XML was
> > already widely used, XSD would make perfect sense.  Here, I think it
> > would be pretty ugly to wire onto the existing DT tools and
> > infrastructure, and unpleasantly unfamiliar for many kernel and board
> > developers trying to work with DT schemas.
> >
> >
> > So, by way of investigation, let me propose an alternative expression
> > of schemas, that I'm also not convinced we should do, but is possible
> > and expressive.  It's illustrative, because it's kind of the polar
> > opposite approach to XSD: just use C.
> >
> > dtc already has a (so far limited) "checks" mechanism which verifies
> > various aspects of DT content.  These are implemented by C functions
> > in checks.c.  There's obviously ample expressiveness - you can express
> > any constraint you want that way.  It can be pretty verbose, and
> > fiddly.  A good library of helper functions can mitigate that, but
> > it's not clear how much.  On the other hand, a very good fraction of
> > people working with this will already be familiar with C, which is a
> > big plus.  This is, after all, the reason that the dts syntax is
> > chiefly C inspired.
> >
> > Now, in practice, I think we will want a more convenient schema
> > language (just as we wanted dts, rather than manually constructing
> > FDTs as C structures).  But I absolutely do think, that the schema
> > handling should be handled as plugins to the checks mechanism -
> > basically we'd have a validate_schemas() check function.
> >
> > I also think we should consider the option of having a simple and
> > straightforward schema language which handles, say, 80% of cases with
> > a fall back to C for the 20% of curly cases.  That might actually be
> > simpler to work with in practice than a schema language which can
> > express absolutely anything, at the cost of being awkward for simple
> > cases or difficult to get your head around.
> 
> Would C++ work? You can use operating overloading and templates to
> change the syntax into something that doesn't even resemble C any
> more.

Well, in theory.  But given that dtc and the kernel are both in plain
C, I don't think it's a good idea.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130730/ea6e26b9/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Mon, Jul 29, 2013 at 1:31 AM, Maxime Ripard
<maxime.ripard@free-electrons.com> wrote:
> Hi,
>
> On Sun, Jul 28, 2013 at 03:19:03PM +0200, Richard Cochran wrote:
>> On Sun, Jul 28, 2013 at 11:12:53AM +0200, Tomasz Figa wrote:
>>
>> > I'm not really sure what effect on users this has. Maybe you should define
>> > "users".
>>
>> ...
>>
>> > Care to explain this reasoning?
>>
>> Use Case
>> ~~~~~~~~
>>
>> User acquires a machine running ARM Linux version 3.x, with u-boot
>> and dtb in a read only flash partition. The board boots and works just
>> fine. However, for his application, the user requires a new kernel
>> feature that appeared in version 3.y where y > x. He compiles the new
>> kernel, and it also works.
>
> I'm afraid this kind of use case will never be properly supported, DT
> stable ABI or not.

Why? New kernel features should be no problem at all.

New driver features /might/ not be available, but only if the new
feature requires additional data that isn't present in the tree and
cannot be obtained elsewhere.

>
> Think about this: what kernel will actually be shipped in that board?
> Most likely, it will be a BSP kernel from the vendor. Does the vendor
> will have made that commitment to have a stable ABI for the DT? Will it
> use the same bindings than mainline? Do we want to support all the crazy
> bindings every vendor will come up with?

That's not a DT issue. That an out-of-tree board/SoC support issue. DT
doesn't make that any better or worse.

g.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Grant Likely]

On Mon, Jul 29, 2013 at 1:31 AM, Maxime Ripard
<maxime.ripard@free-electrons.com> wrote:
> Hi,
>
> On Sun, Jul 28, 2013 at 03:19:03PM +0200, Richard Cochran wrote:
>> On Sun, Jul 28, 2013 at 11:12:53AM +0200, Tomasz Figa wrote:
>>
>> > I'm not really sure what effect on users this has. Maybe you should define
>> > "users".
>>
>> ...
>>
>> > Care to explain this reasoning?
>>
>> Use Case
>> ~~~~~~~~
>>
>> User acquires a machine running ARM Linux version 3.x, with u-boot
>> and dtb in a read only flash partition. The board boots and works just
>> fine. However, for his application, the user requires a new kernel
>> feature that appeared in version 3.y where y > x. He compiles the new
>> kernel, and it also works.
>
> I'm afraid this kind of use case will never be properly supported, DT
> stable ABI or not.

Why? New kernel features should be no problem at all.

New driver features /might/ not be available, but only if the new
feature requires additional data that isn't present in the tree and
cannot be obtained elsewhere.

>
> Think about this: what kernel will actually be shipped in that board?
> Most likely, it will be a BSP kernel from the vendor. Does the vendor
> will have made that commitment to have a stable ABI for the DT? Will it
> use the same bindings than mainline? Do we want to support all the crazy
> bindings every vendor will come up with?

That's not a DT issue. That an out-of-tree board/SoC support issue. DT
doesn't make that any better or worse.

g.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 2623 bytes --]

On Mon, Jul 29, 2013 at 10:35:03PM -0600, Grant Likely wrote:
> >> User acquires a machine running ARM Linux version 3.x, with u-boot
> >> and dtb in a read only flash partition. The board boots and works just
> >> fine. However, for his application, the user requires a new kernel
> >> feature that appeared in version 3.y where y > x. He compiles the new
> >> kernel, and it also works.
> >
> > I'm afraid this kind of use case will never be properly supported, DT
> > stable ABI or not.
> 
> Why? New kernel features should be no problem at all.
> 
> New driver features /might/ not be available, but only if the new
> feature requires additional data that isn't present in the tree and
> cannot be obtained elsewhere.
> 
> >
> > Think about this: what kernel will actually be shipped in that board?
> > Most likely, it will be a BSP kernel from the vendor. Does the vendor
> > will have made that commitment to have a stable ABI for the DT? Will it
> > use the same bindings than mainline? Do we want to support all the crazy
> > bindings every vendor will come up with?
> 
> That's not a DT issue. That an out-of-tree board/SoC support issue. DT
> doesn't make that any better or worse.

Yet, with the DT switch, the two are bound together.

Before the DT, the only convention we had basically was that we had to
be loaded in memory and executed.

Now, the bootloader has to pass a complex data structure to the kernel.
If this data structure doesn't follow the same convention than we do, we
are screwed, and if we can't update either the bootloader or the DT that
is loaded by it, we end up screwed.

One example that comes to my mind right now is this one: we've been
arguing over spidev probing for quite some time. The easy solution would
be to add a "linux,spidev" compatible. Obviously, this has been ruled
out numerous time. Yet, in the beaglebone black out-of-tree kernel,
there is actually a patch that adds this compatible, that is later used
in the DTs.

These spidev part of the DTs will never work with vanilla. And if you
can't update the DT and want to use mainline, you'll never ever have the
same feature set.

That's of course just a small example, with not much impact on a system,
but that could be way worse.

But saying that vendors will follow the same convention we do is just
unrealistic. I mean, even us haven't be able to work out a stable ABI
for more than 2 years now, yet we should expect anyone else to do it?

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Maxime Ripard]

On Mon, Jul 29, 2013 at 10:35:03PM -0600, Grant Likely wrote:
> >> User acquires a machine running ARM Linux version 3.x, with u-boot
> >> and dtb in a read only flash partition. The board boots and works just
> >> fine. However, for his application, the user requires a new kernel
> >> feature that appeared in version 3.y where y > x. He compiles the new
> >> kernel, and it also works.
> >
> > I'm afraid this kind of use case will never be properly supported, DT
> > stable ABI or not.
> 
> Why? New kernel features should be no problem at all.
> 
> New driver features /might/ not be available, but only if the new
> feature requires additional data that isn't present in the tree and
> cannot be obtained elsewhere.
> 
> >
> > Think about this: what kernel will actually be shipped in that board?
> > Most likely, it will be a BSP kernel from the vendor. Does the vendor
> > will have made that commitment to have a stable ABI for the DT? Will it
> > use the same bindings than mainline? Do we want to support all the crazy
> > bindings every vendor will come up with?
> 
> That's not a DT issue. That an out-of-tree board/SoC support issue. DT
> doesn't make that any better or worse.

Yet, with the DT switch, the two are bound together.

Before the DT, the only convention we had basically was that we had to
be loaded in memory and executed.

Now, the bootloader has to pass a complex data structure to the kernel.
If this data structure doesn't follow the same convention than we do, we
are screwed, and if we can't update either the bootloader or the DT that
is loaded by it, we end up screwed.

One example that comes to my mind right now is this one: we've been
arguing over spidev probing for quite some time. The easy solution would
be to add a "linux,spidev" compatible. Obviously, this has been ruled
out numerous time. Yet, in the beaglebone black out-of-tree kernel,
there is actually a patch that adds this compatible, that is later used
in the DTs.

These spidev part of the DTs will never work with vanilla. And if you
can't update the DT and want to use mainline, you'll never ever have the
same feature set.

That's of course just a small example, with not much impact on a system,
but that could be way worse.

But saying that vendors will follow the same convention we do is just
unrealistic. I mean, even us haven't be able to work out a stable ABI
for more than 2 years now, yet we should expect anyone else to do it?

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130730/9def5cff/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/29/2013 08:15 PM, jonsmirl@gmail.com wrote:
> On Mon, Jul 29, 2013 at 9:44 PM, David Gibson
> <david@gibson.dropbear.id.au> wrote:
...
>> I also think we should consider the option of having a simple and
>> straightforward schema language which handles, say, 80% of cases with
>> a fall back to C for the 20% of curly cases.  That might actually be
>> simpler to work with in practice than a schema language which can
>> express absolutely anything, at the cost of being awkward for simple
>> cases or difficult to get your head around.
> 
> Would C++ work? You can use operating overloading and templates to
> change the syntax into something that doesn't even resemble C any
> more.

>From my perspective, that's precisely why C++ should /not/ be used.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/29/2013 08:15 PM, jonsmirl at gmail.com wrote:
> On Mon, Jul 29, 2013 at 9:44 PM, David Gibson
> <david@gibson.dropbear.id.au> wrote:
...
>> I also think we should consider the option of having a simple and
>> straightforward schema language which handles, say, 80% of cases with
>> a fall back to C for the 20% of curly cases.  That might actually be
>> simpler to work with in practice than a schema language which can
>> express absolutely anything, at the cost of being awkward for simple
>> cases or difficult to get your head around.
> 
> Would C++ work? You can use operating overloading and templates to
> change the syntax into something that doesn't even resemble C any
> more.

>From my perspective, that's precisely why C++ should /not/ be used.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/29/2013 07:44 PM, David Gibson wrote:
...
> So, by way of investigation, let me propose an alternative
> expression of schemas, that I'm also not convinced we should do,
> but is possible and expressive.  It's illustrative, because it's
> kind of the polar opposite approach to XSD: just use C.

That actually sounds reasonable.

But, I think there's a difference between a schema, which defines
what's legal in an abstract fashion, and a validator, which defines
that the data conforms to the schema.

I can see that codding a validator in C would be pretty easy, and even
potentially quite simple/clean given a suitable set of library
functions as you say.

However, I'm not so convinced that expressing the schema itself as C
would work out so well. I think the code/data-structures would end up
being pretty stylized so they could actually be read as a schema. At
that point, a specific schema language would probably be simpler? Of
course, this is all somewhat conjecture; perhaps the only way to tell
would be to prototype various ideas and see how well they work.

Equally, I'm not sure I want to bind the schema definition to a
particular language. I'd like to be able to programatically generate
or manipulate *.dts files in arbitrary languages. Integrating a C
interpreter into those languages in order to handle the schema would
be annoying, whereas directly handling a special-purpose schema
language seems like it'd be more tangible.

As an aside, in the past I toyed with the idea of using Python data
structures as a *.dts syntax, and hence allowing
macros/auto-generation using custom Python code, and I'm sure Forth
will come up sometime in this thread:-)

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Stephen Warren]

On 07/29/2013 07:44 PM, David Gibson wrote:
...
> So, by way of investigation, let me propose an alternative
> expression of schemas, that I'm also not convinced we should do,
> but is possible and expressive.  It's illustrative, because it's
> kind of the polar opposite approach to XSD: just use C.

That actually sounds reasonable.

But, I think there's a difference between a schema, which defines
what's legal in an abstract fashion, and a validator, which defines
that the data conforms to the schema.

I can see that codding a validator in C would be pretty easy, and even
potentially quite simple/clean given a suitable set of library
functions as you say.

However, I'm not so convinced that expressing the schema itself as C
would work out so well. I think the code/data-structures would end up
being pretty stylized so they could actually be read as a schema. At
that point, a specific schema language would probably be simpler? Of
course, this is all somewhat conjecture; perhaps the only way to tell
would be to prototype various ideas and see how well they work.

Equally, I'm not sure I want to bind the schema definition to a
particular language. I'd like to be able to programatically generate
or manipulate *.dts files in arbitrary languages. Integrating a C
interpreter into those languages in order to handle the schema would
be annoying, whereas directly handling a special-purpose schema
language seems like it'd be more tangible.

As an aside, in the past I toyed with the idea of using Python data
structures as a *.dts syntax, and hence allowing
macros/auto-generation using custom Python code, and I'm sure Forth
will come up sometime in this thread:-)

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[John W. Linville]

On Tue, Jul 30, 2013 at 10:30:45AM -0600, Stephen Warren wrote:
> On 07/29/2013 08:15 PM, jonsmirl@gmail.com wrote:
> > On Mon, Jul 29, 2013 at 9:44 PM, David Gibson
> > <david@gibson.dropbear.id.au> wrote:
> ...
> >> I also think we should consider the option of having a simple and
> >> straightforward schema language which handles, say, 80% of cases with
> >> a fall back to C for the 20% of curly cases.  That might actually be
> >> simpler to work with in practice than a schema language which can
> >> express absolutely anything, at the cost of being awkward for simple
> >> cases or difficult to get your head around.
> > 
> > Would C++ work? You can use operating overloading and templates to
> > change the syntax into something that doesn't even resemble C any
> > more.
> 
> From my perspective, that's precisely why C++ should /not/ be used.

Amen.

-- 
John W. Linville		Someday the world will need a hero, and you
linville@tuxdriver.com			might be all we have.  Be ready.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[John W. Linville]

On Tue, Jul 30, 2013 at 10:30:45AM -0600, Stephen Warren wrote:
> On 07/29/2013 08:15 PM, jonsmirl at gmail.com wrote:
> > On Mon, Jul 29, 2013 at 9:44 PM, David Gibson
> > <david@gibson.dropbear.id.au> wrote:
> ...
> >> I also think we should consider the option of having a simple and
> >> straightforward schema language which handles, say, 80% of cases with
> >> a fall back to C for the 20% of curly cases.  That might actually be
> >> simpler to work with in practice than a schema language which can
> >> express absolutely anything, at the cost of being awkward for simple
> >> cases or difficult to get your head around.
> > 
> > Would C++ work? You can use operating overloading and templates to
> > change the syntax into something that doesn't even resemble C any
> > more.
> 
> From my perspective, that's precisely why C++ should /not/ be used.

Amen.

-- 
John W. Linville		Someday the world will need a hero, and you
linville at tuxdriver.com			might be all we have.  Be ready.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Maxime Bizon]

On Sat, 2013-07-27 at 11:51 -0700, Tomasz Figa wrote:

> Well, it depends on how we use the DT. There are (at least) two possible 
> usage scenarios:
> 
>  a) using DT as direct replacement for board files - this means that you 
>     are free to say that DTSes are strictly coupled with kernel version 
>     and you are free to modify the bindings - see the analogy to board 
>     files, where you could modify the platform data structures and could 
>     not directly copy board file from one kernel version to another,

I'm shocked to see this as a possible option.

Board files are C code anyone has the skill to edit/understand/refactor.
Moving to DT and keep them in tree tightly coupled with the kernel
version just adds another layer of indirection for *no purpose*.

The fact that we loose compiler syntax/type checking (as highlighted
somewhere else in this thread) even looks like a regression.


>  b) using DT as an ABI - this is the original way, i.e. define stable 
>     bindings and make sure that anu DTB built for older kernel will
> work, with equal or greater set of functionality on newer kernels.
> 
Linus started the whole thing some years ago by refusing to pull ARM
tree [1]. Reread his post, what he wants is clearly b).

Going a) does not solve any problem. You are just moving churn to
somewhere else. We had board files churn, then defconfigs churn, DTS
files (and associated drivers) will be next.

DT is self inflicted pain. It has to be for the greater good.

Going b) *might* allow what some people here dream about, a kernel free
of hardware knowledge. A new board could boot a kernel compiled before
it was even designed.

Now since I do "embedded" stuff everyday, I don't think b) can apply to
the whole ARM world. There is just to much hardware peculiarity.


[1] https://lkml.org/lkml/2011/3/30/525

-- 
Maxime

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Maxime Bizon]

On Sat, 2013-07-27 at 11:51 -0700, Tomasz Figa wrote:

> Well, it depends on how we use the DT. There are (at least) two possible 
> usage scenarios:
> 
>  a) using DT as direct replacement for board files - this means that you 
>     are free to say that DTSes are strictly coupled with kernel version 
>     and you are free to modify the bindings - see the analogy to board 
>     files, where you could modify the platform data structures and could 
>     not directly copy board file from one kernel version to another,

I'm shocked to see this as a possible option.

Board files are C code anyone has the skill to edit/understand/refactor.
Moving to DT and keep them in tree tightly coupled with the kernel
version just adds another layer of indirection for *no purpose*.

The fact that we loose compiler syntax/type checking (as highlighted
somewhere else in this thread) even looks like a regression.


>  b) using DT as an ABI - this is the original way, i.e. define stable 
>     bindings and make sure that anu DTB built for older kernel will
> work, with equal or greater set of functionality on newer kernels.
> 
Linus started the whole thing some years ago by refusing to pull ARM
tree [1]. Reread his post, what he wants is clearly b).

Going a) does not solve any problem. You are just moving churn to
somewhere else. We had board files churn, then defconfigs churn, DTS
files (and associated drivers) will be next.

DT is self inflicted pain. It has to be for the greater good.

Going b) *might* allow what some people here dream about, a kernel free
of hardware knowledge. A new board could boot a kernel compiled before
it was even designed.

Now since I do "embedded" stuff everyday, I don't think b) can apply to
the whole ARM world. There is just to much hardware peculiarity.


[1] https://lkml.org/lkml/2011/3/30/525

-- 
Maxime

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Wednesday 31 of July 2013 12:37:37 Maxime Bizon wrote:
> On Sat, 2013-07-27 at 11:51 -0700, Tomasz Figa wrote:
> > Well, it depends on how we use the DT. There are (at least) two possible
> > 
> > usage scenarios:
> >  a) using DT as direct replacement for board files - this means that you
> >  
> >     are free to say that DTSes are strictly coupled with kernel version
> >     and you are free to modify the bindings - see the analogy to board
> >     files, where you could modify the platform data structures and could
> >     not directly copy board file from one kernel version to another,
> 
> I'm shocked to see this as a possible option.
> 
> Board files are C code anyone has the skill to edit/understand/refactor.
> Moving to DT and keep them in tree tightly coupled with the kernel
> version just adds another layer of indirection for *no purpose*.

Well, I agree that it's not the most fortunate scenario, but it is not that 
bad as it might seem. It still has the benefit of separating hardware data 
(including board-specific data) from kernel code and has all the benefits of DT, 
e.g. easy way to specify relations between devices (phandles), no need to 
manually register any resources, platform data or platform devices, in other 
words, when describing hardware you just care about the hardware, not how the 
kernel works.

> The fact that we loose compiler syntax/type checking (as highlighted
> somewhere else in this thread) even looks like a regression.

Syntax checking is already there. We don't have semantic checking yet, but we 
are working on this.

> >  b) using DT as an ABI - this is the original way, i.e. define stable
> >  
> >     bindings and make sure that anu DTB built for older kernel will
> > 
> > work, with equal or greater set of functionality on newer kernels.
> 
> Linus started the whole thing some years ago by refusing to pull ARM
> tree [1]. Reread his post, what he wants is clearly b).
> 
> Going a) does not solve any problem. You are just moving churn to
> somewhere else. We had board files churn, then defconfigs churn, DTS
> files (and associated drivers) will be next.
> 
> DT is self inflicted pain. It has to be for the greater good.

It has several benefits over board files that I mentioned above, possible 
without fully separating them from kernel tree.

> Going b) *might* allow what some people here dream about, a kernel free
> of hardware knowledge. A new board could boot a kernel compiled before
> it was even designed.

This is very unlikely to happen. You can already see problems with defining 
bindings for existing hardware and what to say about hardware that is not even 
designed yet? Unless all the board uses is a set of already supported 
components, that have bindings defined and drivers merged, you would still need 
to provide remaining drivers and bindings for them.

> Now since I do "embedded" stuff everyday, I don't think b) can apply to
> the whole ARM world. There is just to much hardware peculiarity.

That's why I think we need a hybrid solution. Staging bindings could use a) 
and those stable one would use b). Staging bindings would be fixed over time if 
needed and if they turn out to be fine, they can be stabilized and move to b).

So basically, you can get all the stable functionality with one DTB on any 
kernel released after the functionality got stable, but during development of 
some functionality you might need to change the DTB to test the development.

Best regards,
Tomasz
 
> [1] https://lkml.org/lkml/2011/3/30/525

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Wednesday 31 of July 2013 12:37:37 Maxime Bizon wrote:
> On Sat, 2013-07-27 at 11:51 -0700, Tomasz Figa wrote:
> > Well, it depends on how we use the DT. There are (at least) two possible
> > 
> > usage scenarios:
> >  a) using DT as direct replacement for board files - this means that you
> >  
> >     are free to say that DTSes are strictly coupled with kernel version
> >     and you are free to modify the bindings - see the analogy to board
> >     files, where you could modify the platform data structures and could
> >     not directly copy board file from one kernel version to another,
> 
> I'm shocked to see this as a possible option.
> 
> Board files are C code anyone has the skill to edit/understand/refactor.
> Moving to DT and keep them in tree tightly coupled with the kernel
> version just adds another layer of indirection for *no purpose*.

Well, I agree that it's not the most fortunate scenario, but it is not that 
bad as it might seem. It still has the benefit of separating hardware data 
(including board-specific data) from kernel code and has all the benefits of DT, 
e.g. easy way to specify relations between devices (phandles), no need to 
manually register any resources, platform data or platform devices, in other 
words, when describing hardware you just care about the hardware, not how the 
kernel works.

> The fact that we loose compiler syntax/type checking (as highlighted
> somewhere else in this thread) even looks like a regression.

Syntax checking is already there. We don't have semantic checking yet, but we 
are working on this.

> >  b) using DT as an ABI - this is the original way, i.e. define stable
> >  
> >     bindings and make sure that anu DTB built for older kernel will
> > 
> > work, with equal or greater set of functionality on newer kernels.
> 
> Linus started the whole thing some years ago by refusing to pull ARM
> tree [1]. Reread his post, what he wants is clearly b).
> 
> Going a) does not solve any problem. You are just moving churn to
> somewhere else. We had board files churn, then defconfigs churn, DTS
> files (and associated drivers) will be next.
> 
> DT is self inflicted pain. It has to be for the greater good.

It has several benefits over board files that I mentioned above, possible 
without fully separating them from kernel tree.

> Going b) *might* allow what some people here dream about, a kernel free
> of hardware knowledge. A new board could boot a kernel compiled before
> it was even designed.

This is very unlikely to happen. You can already see problems with defining 
bindings for existing hardware and what to say about hardware that is not even 
designed yet? Unless all the board uses is a set of already supported 
components, that have bindings defined and drivers merged, you would still need 
to provide remaining drivers and bindings for them.

> Now since I do "embedded" stuff everyday, I don't think b) can apply to
> the whole ARM world. There is just to much hardware peculiarity.

That's why I think we need a hybrid solution. Staging bindings could use a) 
and those stable one would use b). Staging bindings would be fixed over time if 
needed and if they turn out to be fine, they can be stabilized and move to b).

So basically, you can get all the stable functionality with one DTB on any 
kernel released after the functionality got stable, but during development of 
some functionality you might need to change the DTB to test the development.

Best regards,
Tomasz
 
> [1] https://lkml.org/lkml/2011/3/30/525

Re: DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Ian Campbell]

On Thu, 2013-07-25 at 18:57 +0100, Mark Rutland wrote:
> > A possible way to handle this is to have exactly that: A group of
> > people that essentially constitute the "standards committee" that meet
> > on a regular basis to review and approve new bindings. They should be
> > people not just doing ARM Linux work, but other stakeholders in these
> > bindings too. One of the things they should probably do is sift
> > through our current in-kernel bindings and select those who seem ready
> > to be locked in, review/discuss/decide upon that and once the decision
> > is made, that specific binding does become part of the static,
> > never-ever-change ABI of firmware-to-kernel interfaces.   That might
> > also be the time that the binding is moved from the kernel to a
> > separate repo, but that's a technicality that we'll let the DT
> > maintainers decide among themselves, IMHO.
> 
> We're going to need input from other OSs too, or the bindings will
> remain Linux-specific regardless of how far away the bindings and dts
> repo(s) is/are.

FWIW my primary interest in stable DT ABIs is because Xen would like to
consume the same device trees on boot, as well as be able to do other
clever things with DT at runtime. Specifically:
      * Take the host device tree passed to Xen at boot, filter out the
        stuff which Xen uses for itself (which is for the most part core
        architectural stuff) and pass the remainder to the dom0 Linux
        kernel (or perhaps in the future another kernel) to drive the
        remainder of the hardware (mostly the SoC specific stuff, but
        some architectural and some Xen defined stuff too)
      * Generate a suitable DT for a guest domain on the fly depending
        on the guest configuration.

Obviously both of those need a stable ABI for us to understand,
manipulate and generate.

Ian.

DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Ian Campbell]

On Thu, 2013-07-25 at 18:57 +0100, Mark Rutland wrote:
> > A possible way to handle this is to have exactly that: A group of
> > people that essentially constitute the "standards committee" that meet
> > on a regular basis to review and approve new bindings. They should be
> > people not just doing ARM Linux work, but other stakeholders in these
> > bindings too. One of the things they should probably do is sift
> > through our current in-kernel bindings and select those who seem ready
> > to be locked in, review/discuss/decide upon that and once the decision
> > is made, that specific binding does become part of the static,
> > never-ever-change ABI of firmware-to-kernel interfaces.   That might
> > also be the time that the binding is moved from the kernel to a
> > separate repo, but that's a technicality that we'll let the DT
> > maintainers decide among themselves, IMHO.
> 
> We're going to need input from other OSs too, or the bindings will
> remain Linux-specific regardless of how far away the bindings and dts
> repo(s) is/are.

FWIW my primary interest in stable DT ABIs is because Xen would like to
consume the same device trees on boot, as well as be able to do other
clever things with DT at runtime. Specifically:
      * Take the host device tree passed to Xen at boot, filter out the
        stuff which Xen uses for itself (which is for the most part core
        architectural stuff) and pass the remainder to the dom0 Linux
        kernel (or perhaps in the future another kernel) to drive the
        remainder of the hardware (mostly the SoC specific stuff, but
        some architectural and some Xen defined stuff too)
      * Generate a suitable DT for a guest domain on the fly depending
        on the guest configuration.

Obviously both of those need a stable ABI for us to understand,
manipulate and generate.

Ian.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Wed, Jul 31, 2013 at 12:59:59PM +0200, Tomasz Figa wrote:
> On Wednesday 31 of July 2013 12:37:37 Maxime Bizon wrote:
> > 
> > Board files are C code anyone has the skill to edit/understand/refactor.
> > Moving to DT and keep them in tree tightly coupled with the kernel
> > version just adds another layer of indirection for *no purpose*.

+1

That is exactly what I tried to say.

> > Linus started the whole thing some years ago by refusing to pull ARM
> > tree [1]. Reread his post, what he wants is clearly b).
> > 
> > Going a) does not solve any problem. You are just moving churn to
> > somewhere else. We had board files churn, then defconfigs churn, DTS
> > files (and associated drivers) will be next.

And at this rate, we are headed for another Linus ultimatum, sooner or
later.

> > DT is self inflicted pain. It has to be for the greater good.
> 
> It has several benefits over board files that I mentioned above, possible 
> without fully separating them from kernel tree.

Every time a criticism is voiced about DT, the DT people stick their
fingers in their ears and say, "NAH, NAH, NAH, I CAN'T HEAR YOU!"

WRT to DT-as-platform-device, we would rather stick with the C code,
please. Just pushing the configuration tables into an external form
does not simplify the problem. In fact, it creates new problems by
inviting the possibility of a bootloader/DT/kernel mismatch.

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Wed, Jul 31, 2013 at 12:59:59PM +0200, Tomasz Figa wrote:
> On Wednesday 31 of July 2013 12:37:37 Maxime Bizon wrote:
> > 
> > Board files are C code anyone has the skill to edit/understand/refactor.
> > Moving to DT and keep them in tree tightly coupled with the kernel
> > version just adds another layer of indirection for *no purpose*.

+1

That is exactly what I tried to say.

> > Linus started the whole thing some years ago by refusing to pull ARM
> > tree [1]. Reread his post, what he wants is clearly b).
> > 
> > Going a) does not solve any problem. You are just moving churn to
> > somewhere else. We had board files churn, then defconfigs churn, DTS
> > files (and associated drivers) will be next.

And at this rate, we are headed for another Linus ultimatum, sooner or
later.

> > DT is self inflicted pain. It has to be for the greater good.
> 
> It has several benefits over board files that I mentioned above, possible 
> without fully separating them from kernel tree.

Every time a criticism is voiced about DT, the DT people stick their
fingers in their ears and say, "NAH, NAH, NAH, I CAN'T HEAR YOU!"

WRT to DT-as-platform-device, we would rather stick with the C code,
please. Just pushing the configuration tables into an external form
does not simplify the problem. In fact, it creates new problems by
inviting the possibility of a bootloader/DT/kernel mismatch.

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Wednesday 31 of July 2013 17:07:19 Richard Cochran wrote:
> On Wed, Jul 31, 2013 at 12:59:59PM +0200, Tomasz Figa wrote:
> > On Wednesday 31 of July 2013 12:37:37 Maxime Bizon wrote:
> > > Board files are C code anyone has the skill to edit/understand/refactor.
> > > Moving to DT and keep them in tree tightly coupled with the kernel
> > > version just adds another layer of indirection for *no purpose*.
> 
> +1
> 
> That is exactly what I tried to say.

I agree with you to some extent. Don't be so extreme though. As I already 
said, this is not entirely "no purpose", as there are more benefits of having 
device tree than just separation from kernel tree.

> > > Linus started the whole thing some years ago by refusing to pull ARM
> > > tree [1]. Reread his post, what he wants is clearly b).
> > > 
> > > Going a) does not solve any problem. You are just moving churn to
> > > somewhere else. We had board files churn, then defconfigs churn, DTS
> > > files (and associated drivers) will be next.
> 
> And at this rate, we are headed for another Linus ultimatum, sooner or
> later.

(see end of the message)

> > > DT is self inflicted pain. It has to be for the greater good.
> > 
> > It has several benefits over board files that I mentioned above, possible
> > without fully separating them from kernel tree.
> 
> Every time a criticism is voiced about DT, the DT people stick their
> fingers in their ears and say, "NAH, NAH, NAH, I CAN'T HEAR YOU!"

I won't comment this...

> WRT to DT-as-platform-device, we would rather stick with the C code,
> please. Just pushing the configuration tables into an external form
> does not simplify the problem. In fact, it creates new problems by
> inviting the possibility of a bootloader/DT/kernel mismatch.

Care to stop ignoring my points other than those defending ideas (nowhere 
stated as good or bad) from extreme opinions?

I said it many, many times, that a) and b) I proposed are just two extremes. 
It is unlikely that an extreme solution will be the best option to choose. I 
am strongly for something in the middle, just like I wrote in several of my 
previous replies.

This is something that should be commented, not those extreme options.

Best regards,
Tomasz

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Wednesday 31 of July 2013 17:07:19 Richard Cochran wrote:
> On Wed, Jul 31, 2013 at 12:59:59PM +0200, Tomasz Figa wrote:
> > On Wednesday 31 of July 2013 12:37:37 Maxime Bizon wrote:
> > > Board files are C code anyone has the skill to edit/understand/refactor.
> > > Moving to DT and keep them in tree tightly coupled with the kernel
> > > version just adds another layer of indirection for *no purpose*.
> 
> +1
> 
> That is exactly what I tried to say.

I agree with you to some extent. Don't be so extreme though. As I already 
said, this is not entirely "no purpose", as there are more benefits of having 
device tree than just separation from kernel tree.

> > > Linus started the whole thing some years ago by refusing to pull ARM
> > > tree [1]. Reread his post, what he wants is clearly b).
> > > 
> > > Going a) does not solve any problem. You are just moving churn to
> > > somewhere else. We had board files churn, then defconfigs churn, DTS
> > > files (and associated drivers) will be next.
> 
> And at this rate, we are headed for another Linus ultimatum, sooner or
> later.

(see end of the message)

> > > DT is self inflicted pain. It has to be for the greater good.
> > 
> > It has several benefits over board files that I mentioned above, possible
> > without fully separating them from kernel tree.
> 
> Every time a criticism is voiced about DT, the DT people stick their
> fingers in their ears and say, "NAH, NAH, NAH, I CAN'T HEAR YOU!"

I won't comment this...

> WRT to DT-as-platform-device, we would rather stick with the C code,
> please. Just pushing the configuration tables into an external form
> does not simplify the problem. In fact, it creates new problems by
> inviting the possibility of a bootloader/DT/kernel mismatch.

Care to stop ignoring my points other than those defending ideas (nowhere 
stated as good or bad) from extreme opinions?

I said it many, many times, that a) and b) I proposed are just two extremes. 
It is unlikely that an extreme solution will be the best option to choose. I 
am strongly for something in the middle, just like I wrote in several of my 
previous replies.

This is something that should be commented, not those extreme options.

Best regards,
Tomasz

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Wed, Jul 31, 2013 at 05:23:35PM +0200, Tomasz Figa wrote:
> 
> I said it many, many times, that a) and b) I proposed are just two extremes. 
> It is unlikely that an extreme solution will be the best option to choose. I 
> am strongly for something in the middle, just like I wrote in several of my 
> previous replies.
> 
> This is something that should be commented, not those extreme options.

We are saying that pursuing a) is useless because it adds pain and
complexity without adding benefit. I simply don't buy your argument
that DT makes a better platform data, but that is besides the point.

I had said, think about the users.  You said, what users?  I wrote a
clear and concise use case.  You said, lets think about a) and b) and
all the shades of gray in between.

In order to support the use case, you will have to provide a stable
ABI. You can't have a compromise solution. At the end of the day,
either you have a stable ABI, or you don't.

It was apparent to me that the arm/dt thing has been meandering around
since its inception, but what was surprising is that people were doing
this on purpose, and now they are defending this. Why can't we get a
firm commitment on having a stable ABI?

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Wed, Jul 31, 2013 at 05:23:35PM +0200, Tomasz Figa wrote:
> 
> I said it many, many times, that a) and b) I proposed are just two extremes. 
> It is unlikely that an extreme solution will be the best option to choose. I 
> am strongly for something in the middle, just like I wrote in several of my 
> previous replies.
> 
> This is something that should be commented, not those extreme options.

We are saying that pursuing a) is useless because it adds pain and
complexity without adding benefit. I simply don't buy your argument
that DT makes a better platform data, but that is besides the point.

I had said, think about the users.  You said, what users?  I wrote a
clear and concise use case.  You said, lets think about a) and b) and
all the shades of gray in between.

In order to support the use case, you will have to provide a stable
ABI. You can't have a compromise solution. At the end of the day,
either you have a stable ABI, or you don't.

It was apparent to me that the arm/dt thing has been meandering around
since its inception, but what was surprising is that people were doing
this on purpose, and now they are defending this. Why can't we get a
firm commitment on having a stable ABI?

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Wednesday 31 of July 2013 21:12:09 Richard Cochran wrote:
> On Wed, Jul 31, 2013 at 05:23:35PM +0200, Tomasz Figa wrote:
> > I said it many, many times, that a) and b) I proposed are just two
> > extremes. It is unlikely that an extreme solution will be the best
> > option to choose. I am strongly for something in the middle, just
> > like I wrote in several of my previous replies.
> > 
> > This is something that should be commented, not those extreme options.
> 
> We are saying that pursuing a) is useless because it adds pain and
> complexity without adding benefit. I simply don't buy your argument
> that DT makes a better platform data, but that is besides the point.
> 
> I had said, think about the users.  You said, what users?  I wrote a
> clear and concise use case.  You said, lets think about a) and b) and
> all the shades of gray in between.

I showed you two example solutions that could handle this use case without 
stable binding ABI, just to prove that b) is not the only option (even if 
it's the best one, which I continue to agree on, don't get me wrong).

I also added that the use case is not fully valid, because you can't 
magically define bindings for all future hardware, which means you can't 
support the hardware using a DTB made before stable bindings for that 
hardware have ever been introduced.

With all of this, I agreed that a DTB made for kernel 3.x, when used with 
kernel 3.y (y > x) should provide the same or greater feature set than 
used with kernel 3.x, in other words, this should cause no regressions. 
Still, for new features, you will likely need to update the DTB.

> In order to support the use case, you will have to provide a stable
> ABI. You can't have a compromise solution. At the end of the day,
> either you have a stable ABI, or you don't.
> 
> It was apparent to me that the arm/dt thing has been meandering around
> since its inception, but what was surprising is that people were doing
> this on purpose, and now they are defending this. Why can't we get a
> firm commitment on having a stable ABI?

This is exactly what are we trying to do right now. But you don't get 
stable things right away. You need some kind of process, for design, 
review, staging, stabilization, etc. Without all of this we will again 
surely end up with something pretending to be good and stable, while being 
completely useless and only a burden to support in new code.

So, again, to summarize, my view on this is as follows:
 - there is a list of best practices for binding design and existing 
   stable bindings that can be used to help for designing new bindings,
 - new bindings go through review process,
 - after positive review, such bindings gets staging status, i.e. they are 
   marked as something that could change,
 - after some period of time (we need to define this precisely) they get 
   frozen and can't be changed in a way that breaks compatibility any
   more. In other words, they become ABI.

What do you think?

Best regards,
Tomasz

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tomasz Figa]

On Wednesday 31 of July 2013 21:12:09 Richard Cochran wrote:
> On Wed, Jul 31, 2013 at 05:23:35PM +0200, Tomasz Figa wrote:
> > I said it many, many times, that a) and b) I proposed are just two
> > extremes. It is unlikely that an extreme solution will be the best
> > option to choose. I am strongly for something in the middle, just
> > like I wrote in several of my previous replies.
> > 
> > This is something that should be commented, not those extreme options.
> 
> We are saying that pursuing a) is useless because it adds pain and
> complexity without adding benefit. I simply don't buy your argument
> that DT makes a better platform data, but that is besides the point.
> 
> I had said, think about the users.  You said, what users?  I wrote a
> clear and concise use case.  You said, lets think about a) and b) and
> all the shades of gray in between.

I showed you two example solutions that could handle this use case without 
stable binding ABI, just to prove that b) is not the only option (even if 
it's the best one, which I continue to agree on, don't get me wrong).

I also added that the use case is not fully valid, because you can't 
magically define bindings for all future hardware, which means you can't 
support the hardware using a DTB made before stable bindings for that 
hardware have ever been introduced.

With all of this, I agreed that a DTB made for kernel 3.x, when used with 
kernel 3.y (y > x) should provide the same or greater feature set than 
used with kernel 3.x, in other words, this should cause no regressions. 
Still, for new features, you will likely need to update the DTB.

> In order to support the use case, you will have to provide a stable
> ABI. You can't have a compromise solution. At the end of the day,
> either you have a stable ABI, or you don't.
> 
> It was apparent to me that the arm/dt thing has been meandering around
> since its inception, but what was surprising is that people were doing
> this on purpose, and now they are defending this. Why can't we get a
> firm commitment on having a stable ABI?

This is exactly what are we trying to do right now. But you don't get 
stable things right away. You need some kind of process, for design, 
review, staging, stabilization, etc. Without all of this we will again 
surely end up with something pretending to be good and stable, while being 
completely useless and only a burden to support in new code.

So, again, to summarize, my view on this is as follows:
 - there is a list of best practices for binding design and existing 
   stable bindings that can be used to help for designing new bindings,
 - new bindings go through review process,
 - after positive review, such bindings gets staging status, i.e. they are 
   marked as something that could change,
 - after some period of time (we need to define this precisely) they get 
   frozen and can't be changed in a way that breaks compatibility any
   more. In other words, they become ABI.

What do you think?

Best regards,
Tomasz

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Wed, Jul 31, 2013 at 09:29:35PM +0200, Tomasz Figa wrote:
> 
> I showed you two example solutions that could handle this use case without 
> stable binding ABI, just to prove that b) is not the only option (even if 
> it's the best one, which I continue to agree on, don't get me wrong).

You only showed *one* solution (b) that satisfies the use case. The
use case was:

   User acquires a machine running ARM Linux version 3.x, with u-boot
   and dtb in a read only flash partition. The board boots and works just
                ^^^^^^^^^^^^^^^^^^^^^^^^^
   fine. However, for his application, the user requires a new kernel
   feature that appeared in version 3.y where y > x. He compiles the new
   kernel, and it also works.

But you suggested:

 a) using DT as direct replacement for board files - this means that you
    are free to say that DTSes are strictly coupled with kernel version
    and you are free to modify the bindings - see the analogy to board
    files, where you could modify the platform data structures and could
    not directly copy board file from one kernel version to another,

In the use case, the kernel is upgraded, but not the DTB. So this
solution makes no sense.

> I also added that the use case is not fully valid,

The use case *is* valid.  I am not inventing this just to be a pain.
There are plenty of people unable or unwilling to upgrade their boot
loader or DTB.  You can say that you won't support it, but it is a use
case from actual real life.

> because you can't 
> magically define bindings for all future hardware, which means you can't 
> support the hardware using a DTB made before stable bindings for that 
> hardware have ever been introduced.

Surely it is possible to develop and release a stable kernel and DT
ABI for a given hardware? Once this is present, it is reasonable for
users to expect forward compatibility from the DT system.
 
> With all of this, I agreed that a DTB made for kernel 3.x, when used with 
> kernel 3.y (y > x) should provide the same or greater feature set than 
> used with kernel 3.x, in other words, this should cause no regressions. 
> Still, for new features, you will likely need to update the DTB.

Yes, that is the idea.

> So, again, to summarize, my view on this is as follows:
>  - there is a list of best practices for binding design and existing 
>    stable bindings that can be used to help for designing new bindings,
>  - new bindings go through review process,
>  - after positive review, such bindings gets staging status, i.e. they are 
>    marked as something that could change,
>  - after some period of time (we need to define this precisely) they get 
>    frozen and can't be changed in a way that breaks compatibility any
>    more. In other words, they become ABI.
> 
> What do you think?

Sounds okay to me, but why bother with this marking business? Why not
just have staging or development trees like every other subsystem?

Thanks,
Richard

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Richard Cochran]

On Wed, Jul 31, 2013 at 09:29:35PM +0200, Tomasz Figa wrote:
> 
> I showed you two example solutions that could handle this use case without 
> stable binding ABI, just to prove that b) is not the only option (even if 
> it's the best one, which I continue to agree on, don't get me wrong).

You only showed *one* solution (b) that satisfies the use case. The
use case was:

   User acquires a machine running ARM Linux version 3.x, with u-boot
   and dtb in a read only flash partition. The board boots and works just
                ^^^^^^^^^^^^^^^^^^^^^^^^^
   fine. However, for his application, the user requires a new kernel
   feature that appeared in version 3.y where y > x. He compiles the new
   kernel, and it also works.

But you suggested:

 a) using DT as direct replacement for board files - this means that you
    are free to say that DTSes are strictly coupled with kernel version
    and you are free to modify the bindings - see the analogy to board
    files, where you could modify the platform data structures and could
    not directly copy board file from one kernel version to another,

In the use case, the kernel is upgraded, but not the DTB. So this
solution makes no sense.

> I also added that the use case is not fully valid,

The use case *is* valid.  I am not inventing this just to be a pain.
There are plenty of people unable or unwilling to upgrade their boot
loader or DTB.  You can say that you won't support it, but it is a use
case from actual real life.

> because you can't 
> magically define bindings for all future hardware, which means you can't 
> support the hardware using a DTB made before stable bindings for that 
> hardware have ever been introduced.

Surely it is possible to develop and release a stable kernel and DT
ABI for a given hardware? Once this is present, it is reasonable for
users to expect forward compatibility from the DT system.
 
> With all of this, I agreed that a DTB made for kernel 3.x, when used with 
> kernel 3.y (y > x) should provide the same or greater feature set than 
> used with kernel 3.x, in other words, this should cause no regressions. 
> Still, for new features, you will likely need to update the DTB.

Yes, that is the idea.

> So, again, to summarize, my view on this is as follows:
>  - there is a list of best practices for binding design and existing 
>    stable bindings that can be used to help for designing new bindings,
>  - new bindings go through review process,
>  - after positive review, such bindings gets staging status, i.e. they are 
>    marked as something that could change,
>  - after some period of time (we need to define this precisely) they get 
>    frozen and can't be changed in a way that breaks compatibility any
>    more. In other words, they become ABI.
> 
> What do you think?

Sounds okay to me, but why bother with this marking business? Why not
just have staging or development trees like every other subsystem?

Thanks,
Richard

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Wed, Jul 31, 2013 at 10:00:17PM +0200, Richard Cochran wrote:
> On Wed, Jul 31, 2013 at 09:29:35PM +0200, Tomasz Figa wrote:
> > 
> > I showed you two example solutions that could handle this use case without 
> > stable binding ABI, just to prove that b) is not the only option (even if 
> > it's the best one, which I continue to agree on, don't get me wrong).
> 
> You only showed *one* solution (b) that satisfies the use case. The
> use case was:
> 
>    User acquires a machine running ARM Linux version 3.x, with u-boot
>    and dtb in a read only flash partition. The board boots and works just
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^
>    fine. However, for his application, the user requires a new kernel
>    feature that appeared in version 3.y where y > x. He compiles the new
>    kernel, and it also works.
> 
> But you suggested:
> 
>  a) using DT as direct replacement for board files - this means that you
>     are free to say that DTSes are strictly coupled with kernel version
>     and you are free to modify the bindings - see the analogy to board
>     files, where you could modify the platform data structures and could
>     not directly copy board file from one kernel version to another,
> 
> In the use case, the kernel is upgraded, but not the DTB. So this
> solution makes no sense.

That's also crap for another reason.  DT on the whole is _supposed_ to
describe what the hardware is, and how it is wired up in a WELL DEFINED
and STABLE manner.  Unfortunately, there's a *BIG* mistake that was made
- having this /chosen node in DT which gets used for "software"
configuration - eg, the command line and so forth.

That was a mistake because it means that the DT isn't purely a
description of the hardware.  Such stuff should have been left in ATAGs,
and DT should have been placed into its own ATAG entry.  There would
not have been any conflict between ATAGs and DT, because ATAGs generally
don't deal with hardware configuration - the only real bit of hardware
configuration conveyed via ATAGs is the location of memory and size of
memory.

However, if we go back to the idea that DT is supposed to describe the
hardware, _and_ that the way to describe that hardware is well defined
and _stable_ (as it should be) then there should be no reason what so
ever that your old DT blob should not continue working in later kernels.
If it doesn't, then the contract that DT promised when we first moved
over to using DT has been _broken_.

Quite frankly, the fact that this discussion has gone on soo far, and
everyone is saying that the existing DT descriptions to date (for what,
two years) are all unstable is really extremely sad.

I've stated in my previous postings what I think very clearly - and it
comes down to the fact that every DT binding which is in existence in
a released kernel _is_ by that very nature a stable binding.  If it
wasn't a stable binding, it should have been clearly marked as such
and been subject to CONFIG_EXPERIMENTAL (when it existed) or
CONFIG_STAGING or similar.

We even went as far as creating documentation for the bindings - directly
along side the documentation for PPC bindings, with no distinction.

Given all that, it's down right insulting to those who have been using
ARM kernels to now start off a discussion about making these things
stable.  Those people who think that these bindings are not stable need
to wake up to reality - we have USERS of this stuff over the last two
years.  It's found its way into products.  If we're going to keep this
stuff "unstable" then we are actively _hurting_ our users.

I'll say it again: if a binding has been in a final kernel, it is by
definition a *stable* binding, and compatibility with that binding
*must* be maintained.

If we're not going to do that, then we owe all the users of the ARM
kernel a VERY BIG apology.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Wed, Jul 31, 2013 at 10:00:17PM +0200, Richard Cochran wrote:
> On Wed, Jul 31, 2013 at 09:29:35PM +0200, Tomasz Figa wrote:
> > 
> > I showed you two example solutions that could handle this use case without 
> > stable binding ABI, just to prove that b) is not the only option (even if 
> > it's the best one, which I continue to agree on, don't get me wrong).
> 
> You only showed *one* solution (b) that satisfies the use case. The
> use case was:
> 
>    User acquires a machine running ARM Linux version 3.x, with u-boot
>    and dtb in a read only flash partition. The board boots and works just
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^
>    fine. However, for his application, the user requires a new kernel
>    feature that appeared in version 3.y where y > x. He compiles the new
>    kernel, and it also works.
> 
> But you suggested:
> 
>  a) using DT as direct replacement for board files - this means that you
>     are free to say that DTSes are strictly coupled with kernel version
>     and you are free to modify the bindings - see the analogy to board
>     files, where you could modify the platform data structures and could
>     not directly copy board file from one kernel version to another,
> 
> In the use case, the kernel is upgraded, but not the DTB. So this
> solution makes no sense.

That's also crap for another reason.  DT on the whole is _supposed_ to
describe what the hardware is, and how it is wired up in a WELL DEFINED
and STABLE manner.  Unfortunately, there's a *BIG* mistake that was made
- having this /chosen node in DT which gets used for "software"
configuration - eg, the command line and so forth.

That was a mistake because it means that the DT isn't purely a
description of the hardware.  Such stuff should have been left in ATAGs,
and DT should have been placed into its own ATAG entry.  There would
not have been any conflict between ATAGs and DT, because ATAGs generally
don't deal with hardware configuration - the only real bit of hardware
configuration conveyed via ATAGs is the location of memory and size of
memory.

However, if we go back to the idea that DT is supposed to describe the
hardware, _and_ that the way to describe that hardware is well defined
and _stable_ (as it should be) then there should be no reason what so
ever that your old DT blob should not continue working in later kernels.
If it doesn't, then the contract that DT promised when we first moved
over to using DT has been _broken_.

Quite frankly, the fact that this discussion has gone on soo far, and
everyone is saying that the existing DT descriptions to date (for what,
two years) are all unstable is really extremely sad.

I've stated in my previous postings what I think very clearly - and it
comes down to the fact that every DT binding which is in existence in
a released kernel _is_ by that very nature a stable binding.  If it
wasn't a stable binding, it should have been clearly marked as such
and been subject to CONFIG_EXPERIMENTAL (when it existed) or
CONFIG_STAGING or similar.

We even went as far as creating documentation for the bindings - directly
along side the documentation for PPC bindings, with no distinction.

Given all that, it's down right insulting to those who have been using
ARM kernels to now start off a discussion about making these things
stable.  Those people who think that these bindings are not stable need
to wake up to reality - we have USERS of this stuff over the last two
years.  It's found its way into products.  If we're going to keep this
stuff "unstable" then we are actively _hurting_ our users.

I'll say it again: if a binding has been in a final kernel, it is by
definition a *stable* binding, and compatibility with that binding
*must* be maintained.

If we're not going to do that, then we owe all the users of the ARM
kernel a VERY BIG apology.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Wed, Jul 31, 2013 at 4:14 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Jul 31, 2013 at 10:00:17PM +0200, Richard Cochran wrote:
>> On Wed, Jul 31, 2013 at 09:29:35PM +0200, Tomasz Figa wrote:
>> >
>> > I showed you two example solutions that could handle this use case without
>> > stable binding ABI, just to prove that b) is not the only option (even if
>> > it's the best one, which I continue to agree on, don't get me wrong).
>>
>> You only showed *one* solution (b) that satisfies the use case. The
>> use case was:
>>
>>    User acquires a machine running ARM Linux version 3.x, with u-boot
>>    and dtb in a read only flash partition. The board boots and works just
>>                 ^^^^^^^^^^^^^^^^^^^^^^^^^
>>    fine. However, for his application, the user requires a new kernel
>>    feature that appeared in version 3.y where y > x. He compiles the new
>>    kernel, and it also works.
>>
>> But you suggested:
>>
>>  a) using DT as direct replacement for board files - this means that you
>>     are free to say that DTSes are strictly coupled with kernel version
>>     and you are free to modify the bindings - see the analogy to board
>>     files, where you could modify the platform data structures and could
>>     not directly copy board file from one kernel version to another,
>>
>> In the use case, the kernel is upgraded, but not the DTB. So this
>> solution makes no sense.
>
> That's also crap for another reason.  DT on the whole is _supposed_ to
> describe what the hardware is, and how it is wired up in a WELL DEFINED
> and STABLE manner.  Unfortunately, there's a *BIG* mistake that was made
> - having this /chosen node in DT which gets used for "software"
> configuration - eg, the command line and so forth.
>
> That was a mistake because it means that the DT isn't purely a
> description of the hardware.  Such stuff should have been left in ATAGs,
> and DT should have been placed into its own ATAG entry.  There would
> not have been any conflict between ATAGs and DT, because ATAGs generally
> don't deal with hardware configuration - the only real bit of hardware
> configuration conveyed via ATAGs is the location of memory and size of
> memory.
>
> However, if we go back to the idea that DT is supposed to describe the
> hardware, _and_ that the way to describe that hardware is well defined
> and _stable_ (as it should be) then there should be no reason what so
> ever that your old DT blob should not continue working in later kernels.
> If it doesn't, then the contract that DT promised when we first moved
> over to using DT has been _broken_.

Suppose your DT predates PINCTRL and the CPU needs the pins set to
function.  But setting the pins up is a board specific function. How
is this going to get implemented in a generic kernel that doesn't have
any board specific code in it?

I would propose that we only need to worry about DTs that got put into
boot loaders and not worry about ones that were only used when
appended to a kernel.

For those deployed bootloader based DTs we'd use a quirk system to
catch them at early boot time and add in the missing PINCTRL
information.  The function of this quirk system is to update deployed
DTs with the needed information to allow a completely generic kernel
to run.

Now we can have a very clean generic kernel with all of the board
specific fixups localized to a single spot - the quirk for those
deployed DTs. And hopefully new boards won't need any quirks.

>
> Quite frankly, the fact that this discussion has gone on soo far, and
> everyone is saying that the existing DT descriptions to date (for what,
> two years) are all unstable is really extremely sad.
>
> I've stated in my previous postings what I think very clearly - and it
> comes down to the fact that every DT binding which is in existence in
> a released kernel _is_ by that very nature a stable binding.  If it
> wasn't a stable binding, it should have been clearly marked as such
> and been subject to CONFIG_EXPERIMENTAL (when it existed) or
> CONFIG_STAGING or similar.
>
> We even went as far as creating documentation for the bindings - directly
> along side the documentation for PPC bindings, with no distinction.
>
> Given all that, it's down right insulting to those who have been using
> ARM kernels to now start off a discussion about making these things
> stable.  Those people who think that these bindings are not stable need
> to wake up to reality - we have USERS of this stuff over the last two
> years.  It's found its way into products.  If we're going to keep this
> stuff "unstable" then we are actively _hurting_ our users.
>
> I'll say it again: if a binding has been in a final kernel, it is by
> definition a *stable* binding, and compatibility with that binding
> *must* be maintained.
>
> If we're not going to do that, then we owe all the users of the ARM
> kernel a VERY BIG apology.
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Wed, Jul 31, 2013 at 4:14 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Jul 31, 2013 at 10:00:17PM +0200, Richard Cochran wrote:
>> On Wed, Jul 31, 2013 at 09:29:35PM +0200, Tomasz Figa wrote:
>> >
>> > I showed you two example solutions that could handle this use case without
>> > stable binding ABI, just to prove that b) is not the only option (even if
>> > it's the best one, which I continue to agree on, don't get me wrong).
>>
>> You only showed *one* solution (b) that satisfies the use case. The
>> use case was:
>>
>>    User acquires a machine running ARM Linux version 3.x, with u-boot
>>    and dtb in a read only flash partition. The board boots and works just
>>                 ^^^^^^^^^^^^^^^^^^^^^^^^^
>>    fine. However, for his application, the user requires a new kernel
>>    feature that appeared in version 3.y where y > x. He compiles the new
>>    kernel, and it also works.
>>
>> But you suggested:
>>
>>  a) using DT as direct replacement for board files - this means that you
>>     are free to say that DTSes are strictly coupled with kernel version
>>     and you are free to modify the bindings - see the analogy to board
>>     files, where you could modify the platform data structures and could
>>     not directly copy board file from one kernel version to another,
>>
>> In the use case, the kernel is upgraded, but not the DTB. So this
>> solution makes no sense.
>
> That's also crap for another reason.  DT on the whole is _supposed_ to
> describe what the hardware is, and how it is wired up in a WELL DEFINED
> and STABLE manner.  Unfortunately, there's a *BIG* mistake that was made
> - having this /chosen node in DT which gets used for "software"
> configuration - eg, the command line and so forth.
>
> That was a mistake because it means that the DT isn't purely a
> description of the hardware.  Such stuff should have been left in ATAGs,
> and DT should have been placed into its own ATAG entry.  There would
> not have been any conflict between ATAGs and DT, because ATAGs generally
> don't deal with hardware configuration - the only real bit of hardware
> configuration conveyed via ATAGs is the location of memory and size of
> memory.
>
> However, if we go back to the idea that DT is supposed to describe the
> hardware, _and_ that the way to describe that hardware is well defined
> and _stable_ (as it should be) then there should be no reason what so
> ever that your old DT blob should not continue working in later kernels.
> If it doesn't, then the contract that DT promised when we first moved
> over to using DT has been _broken_.

Suppose your DT predates PINCTRL and the CPU needs the pins set to
function.  But setting the pins up is a board specific function. How
is this going to get implemented in a generic kernel that doesn't have
any board specific code in it?

I would propose that we only need to worry about DTs that got put into
boot loaders and not worry about ones that were only used when
appended to a kernel.

For those deployed bootloader based DTs we'd use a quirk system to
catch them at early boot time and add in the missing PINCTRL
information.  The function of this quirk system is to update deployed
DTs with the needed information to allow a completely generic kernel
to run.

Now we can have a very clean generic kernel with all of the board
specific fixups localized to a single spot - the quirk for those
deployed DTs. And hopefully new boards won't need any quirks.

>
> Quite frankly, the fact that this discussion has gone on soo far, and
> everyone is saying that the existing DT descriptions to date (for what,
> two years) are all unstable is really extremely sad.
>
> I've stated in my previous postings what I think very clearly - and it
> comes down to the fact that every DT binding which is in existence in
> a released kernel _is_ by that very nature a stable binding.  If it
> wasn't a stable binding, it should have been clearly marked as such
> and been subject to CONFIG_EXPERIMENTAL (when it existed) or
> CONFIG_STAGING or similar.
>
> We even went as far as creating documentation for the bindings - directly
> along side the documentation for PPC bindings, with no distinction.
>
> Given all that, it's down right insulting to those who have been using
> ARM kernels to now start off a discussion about making these things
> stable.  Those people who think that these bindings are not stable need
> to wake up to reality - we have USERS of this stuff over the last two
> years.  It's found its way into products.  If we're going to keep this
> stuff "unstable" then we are actively _hurting_ our users.
>
> I'll say it again: if a binding has been in a final kernel, it is by
> definition a *stable* binding, and compatibility with that binding
> *must* be maintained.
>
> If we're not going to do that, then we owe all the users of the ARM
> kernel a VERY BIG apology.
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Wed, Jul 31, 2013 at 04:37:36PM -0400, jonsmirl@gmail.com wrote:
> On Wed, Jul 31, 2013 at 4:14 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > However, if we go back to the idea that DT is supposed to describe the
> > hardware, _and_ that the way to describe that hardware is well defined
> > and _stable_ (as it should be) then there should be no reason what so
> > ever that your old DT blob should not continue working in later kernels.
> > If it doesn't, then the contract that DT promised when we first moved
> > over to using DT has been _broken_.
> 
> Suppose your DT predates PINCTRL and the CPU needs the pins set to
> function.  But setting the pins up is a board specific function. How
> is this going to get implemented in a generic kernel that doesn't have
> any board specific code in it?
> 
> I would propose that we only need to worry about DTs that got put into
> boot loaders and not worry about ones that were only used when
> appended to a kernel.

That is - again - our mistake.  We should have made it clear from the
start that the use of an _appended_ DT, or a DT provided with the kernel
being booted was more or less mandatory for the time being.  We actually
did exactly the opposite - we advertised the appended DT as a stop-gap
measure just to allow a DT kernel to be booted with a boot loader which
didn't support DT.

That in itself _encouraged_ boot loader support for DT on ARM (which in
itself is not a bad thing) but also leads people down the path of
potentially separating the DT from the kernel.

Had we encouraged the use of an appended DT instead, but still encouraged
boot loaders to update, and also made a clear statement that DT is
unstable (which everyone can see - for example by making our DT stuff
depend on CONFIG_EXPERIMENTAL when it existed) then everyone would have
been clear about its status.

The fact that we did not - the fact that we ignored this process means
that it is _our_ problem that people like Richard are blowing up such a
storm over this.  We need to admit that we got it wrong, and commit to
doing it the right way in future.  What that means is starting off today
with a commitment to keep as much of the stuff which works today working
_tomorrow_, and the day after, and so forth.

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Russell King - ARM Linux]

On Wed, Jul 31, 2013 at 04:37:36PM -0400, jonsmirl at gmail.com wrote:
> On Wed, Jul 31, 2013 at 4:14 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > However, if we go back to the idea that DT is supposed to describe the
> > hardware, _and_ that the way to describe that hardware is well defined
> > and _stable_ (as it should be) then there should be no reason what so
> > ever that your old DT blob should not continue working in later kernels.
> > If it doesn't, then the contract that DT promised when we first moved
> > over to using DT has been _broken_.
> 
> Suppose your DT predates PINCTRL and the CPU needs the pins set to
> function.  But setting the pins up is a board specific function. How
> is this going to get implemented in a generic kernel that doesn't have
> any board specific code in it?
> 
> I would propose that we only need to worry about DTs that got put into
> boot loaders and not worry about ones that were only used when
> appended to a kernel.

That is - again - our mistake.  We should have made it clear from the
start that the use of an _appended_ DT, or a DT provided with the kernel
being booted was more or less mandatory for the time being.  We actually
did exactly the opposite - we advertised the appended DT as a stop-gap
measure just to allow a DT kernel to be booted with a boot loader which
didn't support DT.

That in itself _encouraged_ boot loader support for DT on ARM (which in
itself is not a bad thing) but also leads people down the path of
potentially separating the DT from the kernel.

Had we encouraged the use of an appended DT instead, but still encouraged
boot loaders to update, and also made a clear statement that DT is
unstable (which everyone can see - for example by making our DT stuff
depend on CONFIG_EXPERIMENTAL when it existed) then everyone would have
been clear about its status.

The fact that we did not - the fact that we ignored this process means
that it is _our_ problem that people like Richard are blowing up such a
storm over this.  We need to admit that we got it wrong, and commit to
doing it the right way in future.  What that means is starting off today
with a commitment to keep as much of the stuff which works today working
_tomorrow_, and the day after, and so forth.

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Wed, Jul 31, 2013 at 4:48 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Jul 31, 2013 at 04:37:36PM -0400, jonsmirl@gmail.com wrote:
>> On Wed, Jul 31, 2013 at 4:14 PM, Russell King - ARM Linux
>> <linux@arm.linux.org.uk> wrote:
>> > However, if we go back to the idea that DT is supposed to describe the
>> > hardware, _and_ that the way to describe that hardware is well defined
>> > and _stable_ (as it should be) then there should be no reason what so
>> > ever that your old DT blob should not continue working in later kernels.
>> > If it doesn't, then the contract that DT promised when we first moved
>> > over to using DT has been _broken_.
>>
>> Suppose your DT predates PINCTRL and the CPU needs the pins set to
>> function.  But setting the pins up is a board specific function. How
>> is this going to get implemented in a generic kernel that doesn't have
>> any board specific code in it?
>>
>> I would propose that we only need to worry about DTs that got put into
>> boot loaders and not worry about ones that were only used when
>> appended to a kernel.
>
> That is - again - our mistake.  We should have made it clear from the
> start that the use of an _appended_ DT, or a DT provided with the kernel
> being booted was more or less mandatory for the time being.  We actually
> did exactly the opposite - we advertised the appended DT as a stop-gap
> measure just to allow a DT kernel to be booted with a boot loader which
> didn't support DT.
>
> That in itself _encouraged_ boot loader support for DT on ARM (which in
> itself is not a bad thing) but also leads people down the path of
> potentially separating the DT from the kernel.
>
> Had we encouraged the use of an appended DT instead, but still encouraged
> boot loaders to update, and also made a clear statement that DT is
> unstable (which everyone can see - for example by making our DT stuff
> depend on CONFIG_EXPERIMENTAL when it existed) then everyone would have
> been clear about its status.
>
> The fact that we did not - the fact that we ignored this process means
> that it is _our_ problem that people like Richard are blowing up such a
> storm over this.  We need to admit that we got it wrong, and commit to
> doing it the right way in future.  What that means is starting off today
> with a commitment to keep as much of the stuff which works today working
> _tomorrow_, and the day after, and so forth.

What do you think about using a quirk layer to decouple deployed DTs
from whatever is implemented in the kernel? I still think there is
going to be volatility in the the kernel DT usage simply because we
haven't figured out all of the issues with it yet.  For example
defining a global schema system is going to expose a lot of irregular
DT syntax when you line up all of the platform DTs in a system where
it is easy to compare them. One the plus side the kernel is certainly
evolving to a point where this volatility will stop in the future, we
just aren't there yet.  If we don't use a quirk fixup system, then
board specific code is going to continue to exist scattered around in
the arch directories.

My preference would be to contain the board specific DT fixup code
inside a quirk system and have the goal of making the arch directories
free of board specific code. Any new board would have a good enough DT
that they don't need any board specific DT fixup code. Of course
there's quite a ways to go before reaching that goal.

Alternatively you may be of the belief that it is impossible to get
rid of the board specific code. But x86 doesn't have any of it, why
should ARM?

-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Wed, Jul 31, 2013 at 4:48 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Jul 31, 2013 at 04:37:36PM -0400, jonsmirl at gmail.com wrote:
>> On Wed, Jul 31, 2013 at 4:14 PM, Russell King - ARM Linux
>> <linux@arm.linux.org.uk> wrote:
>> > However, if we go back to the idea that DT is supposed to describe the
>> > hardware, _and_ that the way to describe that hardware is well defined
>> > and _stable_ (as it should be) then there should be no reason what so
>> > ever that your old DT blob should not continue working in later kernels.
>> > If it doesn't, then the contract that DT promised when we first moved
>> > over to using DT has been _broken_.
>>
>> Suppose your DT predates PINCTRL and the CPU needs the pins set to
>> function.  But setting the pins up is a board specific function. How
>> is this going to get implemented in a generic kernel that doesn't have
>> any board specific code in it?
>>
>> I would propose that we only need to worry about DTs that got put into
>> boot loaders and not worry about ones that were only used when
>> appended to a kernel.
>
> That is - again - our mistake.  We should have made it clear from the
> start that the use of an _appended_ DT, or a DT provided with the kernel
> being booted was more or less mandatory for the time being.  We actually
> did exactly the opposite - we advertised the appended DT as a stop-gap
> measure just to allow a DT kernel to be booted with a boot loader which
> didn't support DT.
>
> That in itself _encouraged_ boot loader support for DT on ARM (which in
> itself is not a bad thing) but also leads people down the path of
> potentially separating the DT from the kernel.
>
> Had we encouraged the use of an appended DT instead, but still encouraged
> boot loaders to update, and also made a clear statement that DT is
> unstable (which everyone can see - for example by making our DT stuff
> depend on CONFIG_EXPERIMENTAL when it existed) then everyone would have
> been clear about its status.
>
> The fact that we did not - the fact that we ignored this process means
> that it is _our_ problem that people like Richard are blowing up such a
> storm over this.  We need to admit that we got it wrong, and commit to
> doing it the right way in future.  What that means is starting off today
> with a commitment to keep as much of the stuff which works today working
> _tomorrow_, and the day after, and so forth.

What do you think about using a quirk layer to decouple deployed DTs
from whatever is implemented in the kernel? I still think there is
going to be volatility in the the kernel DT usage simply because we
haven't figured out all of the issues with it yet.  For example
defining a global schema system is going to expose a lot of irregular
DT syntax when you line up all of the platform DTs in a system where
it is easy to compare them. One the plus side the kernel is certainly
evolving to a point where this volatility will stop in the future, we
just aren't there yet.  If we don't use a quirk fixup system, then
board specific code is going to continue to exist scattered around in
the arch directories.

My preference would be to contain the board specific DT fixup code
inside a quirk system and have the goal of making the arch directories
free of board specific code. Any new board would have a good enough DT
that they don't need any board specific DT fixup code. Of course
there's quite a ways to go before reaching that goal.

Alternatively you may be of the belief that it is impossible to get
rid of the board specific code. But x86 doesn't have any of it, why
should ARM?

-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/31/2013 11:26 PM, jonsmirl@gmail.com wrote:
> Alternatively you may be of the belief that it is impossible to get
> rid of the board specific code. But x86 doesn't have any of it, why
> should ARM?

Well, I am curious whether that will stay that way once x86 is truly 
moving into the embedded arena (if ever).

Regards,
Arend

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Arend van Spriel]

On 07/31/2013 11:26 PM, jonsmirl at gmail.com wrote:
> Alternatively you may be of the belief that it is impossible to get
> rid of the board specific code. But x86 doesn't have any of it, why
> should ARM?

Well, I am curious whether that will stay that way once x86 is truly 
moving into the embedded arena (if ever).

Regards,
Arend

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 486 bytes --]

On Thu, Aug 01, 2013 at 11:57:43AM +0200, Arend van Spriel wrote:
> On 07/31/2013 11:26 PM, jonsmirl@gmail.com wrote:

> >Alternatively you may be of the belief that it is impossible to get
> >rid of the board specific code. But x86 doesn't have any of it, why
> >should ARM?

> Well, I am curious whether that will stay that way once x86 is truly
> moving into the embedded arena (if ever).

There's shipping phones using x86, they do actually have board code due
to their use of SFI.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Mark Brown]

On Thu, Aug 01, 2013 at 11:57:43AM +0200, Arend van Spriel wrote:
> On 07/31/2013 11:26 PM, jonsmirl at gmail.com wrote:

> >Alternatively you may be of the belief that it is impossible to get
> >rid of the board specific code. But x86 doesn't have any of it, why
> >should ARM?

> Well, I am curious whether that will stay that way once x86 is truly
> moving into the embedded arena (if ever).

There's shipping phones using x86, they do actually have board code due
to their use of SFI.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130801/b3df6834/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]

On Wed, 2013-07-31 at 17:26 -0400, jonsmirl@gmail.com wrote:
> Alternatively you may be of the belief that it is impossible to get
> rid of the board specific code. But x86 doesn't have any of it, why
> should ARM?

The reason x86 doesn't have it is because it carries three decades worth
of legacy baggage so that it can still look like a 1980s IBM PC when
necessary.

There *have* been some x86 platforms which abandon that legacy crap, and
for those we *do* have board-specific code. (Is James still maintaining
Voyager support? It feels very strange to talk about Voyager with it
*not* being the 'legacy crap' in question...)

We've even seen *recent* attempts to abandon the legacy crap in the
embedded x86 space, which backtracked and added it all back again — in
part because x86 lacked any sane way to describe the hardware if it
wasn't pretending to be a PC. ACPI doesn't cut it, and DT "wasn't
invented here"...

Unless you want the ARM world to settle on a strategy of "all the world
is an Assabet", I'd be careful what you wish for...

-- 
dwmw2


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Woodhouse]

On Wed, 2013-07-31 at 17:26 -0400, jonsmirl at gmail.com wrote:
> Alternatively you may be of the belief that it is impossible to get
> rid of the board specific code. But x86 doesn't have any of it, why
> should ARM?

The reason x86 doesn't have it is because it carries three decades worth
of legacy baggage so that it can still look like a 1980s IBM PC when
necessary.

There *have* been some x86 platforms which abandon that legacy crap, and
for those we *do* have board-specific code. (Is James still maintaining
Voyager support? It feels very strange to talk about Voyager with it
*not* being the 'legacy crap' in question...)

We've even seen *recent* attempts to abandon the legacy crap in the
embedded x86 space, which backtracked and added it all back again ? in
part because x86 lacked any sane way to describe the hardware if it
wasn't pretending to be a PC. ACPI doesn't cut it, and DT "wasn't
invented here"...

Unless you want the ARM world to settle on a strategy of "all the world
is an Assabet", I'd be careful what you wish for...

-- 
dwmw2

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5745 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130801/627e5f5e/attachment.bin>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Thu, Aug 1, 2013 at 6:18 AM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Wed, 2013-07-31 at 17:26 -0400, jonsmirl@gmail.com wrote:
>> Alternatively you may be of the belief that it is impossible to get
>> rid of the board specific code. But x86 doesn't have any of it, why
>> should ARM?
>
> The reason x86 doesn't have it is because it carries three decades worth
> of legacy baggage so that it can still look like a 1980s IBM PC when
> necessary.
>
> There *have* been some x86 platforms which abandon that legacy crap, and
> for those we *do* have board-specific code. (Is James still maintaining
> Voyager support? It feels very strange to talk about Voyager with it
> *not* being the 'legacy crap' in question...)
>
> We've even seen *recent* attempts to abandon the legacy crap in the
> embedded x86 space, which backtracked and added it all back again — in
> part because x86 lacked any sane way to describe the hardware if it
> wasn't pretending to be a PC. ACPI doesn't cut it, and DT "wasn't
> invented here"...
>
> Unless you want the ARM world to settle on a strategy of "all the world
> is an Assabet", I'd be careful what you wish for...

So there should be shades for gray in this area. Try to eliminate all
of the board specific code you can, and then only if that fails add
board specific support to the kernel.

But you take device trees pretty far. I believe Grant has even used
one to describe an FPGA that he can dynamically load code into and
change its function.  Not sure how he did it, I wasn't paying too
close of attention when he was talking about it.

I do believe a great deal of this simple plumbing can be eliminated.
Like how to hook up GPIO, I2C, SPI, etc. We're pretty far down that
road.

A path like this seems like it would be beneficial...
1) Implement DT schemas. Use those to enforce as much regularity as
possible into the device tree descriptions for common classes of
things (controllers especially - DMA, I2C, I2S, SPI, Uarts, etc)..
Form a group to review any changes to the common schema.
2) Cleaning up the controller classes is going to cause some DT churn.
Hide backward compatibility in a quirks layer.
3) Continue the process of removing all possible board specific code
that can be reasonably covered in device trees.
4) There will be some board specific code left at the end of this
process. But anyone who looks at should agree that the functions
handled by the code are something that is unreasonable to address in
the DT system.


>
> --
> dwmw2
>



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Thu, Aug 1, 2013 at 6:18 AM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Wed, 2013-07-31 at 17:26 -0400, jonsmirl at gmail.com wrote:
>> Alternatively you may be of the belief that it is impossible to get
>> rid of the board specific code. But x86 doesn't have any of it, why
>> should ARM?
>
> The reason x86 doesn't have it is because it carries three decades worth
> of legacy baggage so that it can still look like a 1980s IBM PC when
> necessary.
>
> There *have* been some x86 platforms which abandon that legacy crap, and
> for those we *do* have board-specific code. (Is James still maintaining
> Voyager support? It feels very strange to talk about Voyager with it
> *not* being the 'legacy crap' in question...)
>
> We've even seen *recent* attempts to abandon the legacy crap in the
> embedded x86 space, which backtracked and added it all back again ? in
> part because x86 lacked any sane way to describe the hardware if it
> wasn't pretending to be a PC. ACPI doesn't cut it, and DT "wasn't
> invented here"...
>
> Unless you want the ARM world to settle on a strategy of "all the world
> is an Assabet", I'd be careful what you wish for...

So there should be shades for gray in this area. Try to eliminate all
of the board specific code you can, and then only if that fails add
board specific support to the kernel.

But you take device trees pretty far. I believe Grant has even used
one to describe an FPGA that he can dynamically load code into and
change its function.  Not sure how he did it, I wasn't paying too
close of attention when he was talking about it.

I do believe a great deal of this simple plumbing can be eliminated.
Like how to hook up GPIO, I2C, SPI, etc. We're pretty far down that
road.

A path like this seems like it would be beneficial...
1) Implement DT schemas. Use those to enforce as much regularity as
possible into the device tree descriptions for common classes of
things (controllers especially - DMA, I2C, I2S, SPI, Uarts, etc)..
Form a group to review any changes to the common schema.
2) Cleaning up the controller classes is going to cause some DT churn.
Hide backward compatibility in a quirks layer.
3) Continue the process of removing all possible board specific code
that can be reasonably covered in device trees.
4) There will be some board specific code left at the end of this
process. But anyone who looks at should agree that the functions
handled by the code are something that is unreasonable to address in
the DT system.


>
> --
> dwmw2
>



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl]

On Thu, Aug 1, 2013 at 9:34 AM, jonsmirl@gmail.com <jonsmirl@gmail.com> wrote:
> On Thu, Aug 1, 2013 at 6:18 AM, David Woodhouse <dwmw2@infradead.org> wrote:
>> On Wed, 2013-07-31 at 17:26 -0400, jonsmirl@gmail.com wrote:
>>> Alternatively you may be of the belief that it is impossible to get
>>> rid of the board specific code. But x86 doesn't have any of it, why
>>> should ARM?
>>
>> The reason x86 doesn't have it is because it carries three decades worth
>> of legacy baggage so that it can still look like a 1980s IBM PC when
>> necessary.
>>
>> There *have* been some x86 platforms which abandon that legacy crap, and
>> for those we *do* have board-specific code. (Is James still maintaining
>> Voyager support? It feels very strange to talk about Voyager with it
>> *not* being the 'legacy crap' in question...)
>>
>> We've even seen *recent* attempts to abandon the legacy crap in the
>> embedded x86 space, which backtracked and added it all back again — in
>> part because x86 lacked any sane way to describe the hardware if it
>> wasn't pretending to be a PC. ACPI doesn't cut it, and DT "wasn't
>> invented here"...
>>
>> Unless you want the ARM world to settle on a strategy of "all the world
>> is an Assabet", I'd be careful what you wish for...
>
> So there should be shades for gray in this area. Try to eliminate all
> of the board specific code you can, and then only if that fails add
> board specific support to the kernel.
>
> But you take device trees pretty far. I believe Grant has even used
> one to describe an FPGA that he can dynamically load code into and
> change its function.  Not sure how he did it, I wasn't paying too
> close of attention when he was talking about it.
>
> I do believe a great deal of this simple plumbing can be eliminated.
> Like how to hook up GPIO, I2C, SPI, etc. We're pretty far down that
> road.
>
> A path like this seems like it would be beneficial...
> 1) Implement DT schemas. Use those to enforce as much regularity as
> possible into the device tree descriptions for common classes of
> things (controllers especially - DMA, I2C, I2S, SPI, Uarts, etc)..
> Form a group to review any changes to the common schema.
> 2) Cleaning up the controller classes is going to cause some DT churn.
> Hide backward compatibility in a quirks layer.
> 3) Continue the process of removing all possible board specific code
> that can be reasonably covered in device trees.
> 4) There will be some board specific code left at the end of this
> process. But anyone who looks at should agree that the functions
> handled by the code are something that is unreasonable to address in
> the DT system.

5) this scheme supports future improvements in the DT schema. Lets say
initially we had punted power management to board specific code.
Then in a later kernel version implemented it using device trees. The
quirk system lets you delete the board specific code and replace it
with a DT quirk. That DT quirk will see your deployed DTs at boot time
and add in the new, fancy power management DT attributes.  The new
generic DT based power power management code will see these attributes
added by the quirk and do the right thing. This gives us a way to
slowly remove move board specific code if we choose to.



>
>
>>
>> --
>> dwmw2
>>
>
>
>
> --
> Jon Smirl
> jonsmirl@gmail.com



-- 
Jon Smirl
jonsmirl@gmail.com

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[jonsmirl at gmail.com]

On Thu, Aug 1, 2013 at 9:34 AM, jonsmirl at gmail.com <jonsmirl@gmail.com> wrote:
> On Thu, Aug 1, 2013 at 6:18 AM, David Woodhouse <dwmw2@infradead.org> wrote:
>> On Wed, 2013-07-31 at 17:26 -0400, jonsmirl at gmail.com wrote:
>>> Alternatively you may be of the belief that it is impossible to get
>>> rid of the board specific code. But x86 doesn't have any of it, why
>>> should ARM?
>>
>> The reason x86 doesn't have it is because it carries three decades worth
>> of legacy baggage so that it can still look like a 1980s IBM PC when
>> necessary.
>>
>> There *have* been some x86 platforms which abandon that legacy crap, and
>> for those we *do* have board-specific code. (Is James still maintaining
>> Voyager support? It feels very strange to talk about Voyager with it
>> *not* being the 'legacy crap' in question...)
>>
>> We've even seen *recent* attempts to abandon the legacy crap in the
>> embedded x86 space, which backtracked and added it all back again ? in
>> part because x86 lacked any sane way to describe the hardware if it
>> wasn't pretending to be a PC. ACPI doesn't cut it, and DT "wasn't
>> invented here"...
>>
>> Unless you want the ARM world to settle on a strategy of "all the world
>> is an Assabet", I'd be careful what you wish for...
>
> So there should be shades for gray in this area. Try to eliminate all
> of the board specific code you can, and then only if that fails add
> board specific support to the kernel.
>
> But you take device trees pretty far. I believe Grant has even used
> one to describe an FPGA that he can dynamically load code into and
> change its function.  Not sure how he did it, I wasn't paying too
> close of attention when he was talking about it.
>
> I do believe a great deal of this simple plumbing can be eliminated.
> Like how to hook up GPIO, I2C, SPI, etc. We're pretty far down that
> road.
>
> A path like this seems like it would be beneficial...
> 1) Implement DT schemas. Use those to enforce as much regularity as
> possible into the device tree descriptions for common classes of
> things (controllers especially - DMA, I2C, I2S, SPI, Uarts, etc)..
> Form a group to review any changes to the common schema.
> 2) Cleaning up the controller classes is going to cause some DT churn.
> Hide backward compatibility in a quirks layer.
> 3) Continue the process of removing all possible board specific code
> that can be reasonably covered in device trees.
> 4) There will be some board specific code left at the end of this
> process. But anyone who looks at should agree that the functions
> handled by the code are something that is unreasonable to address in
> the DT system.

5) this scheme supports future improvements in the DT schema. Lets say
initially we had punted power management to board specific code.
Then in a later kernel version implemented it using device trees. The
quirk system lets you delete the board specific code and replace it
with a DT quirk. That DT quirk will see your deployed DTs at boot time
and add in the new, fancy power management DT attributes.  The new
generic DT based power power management code will see these attributes
added by the quirk and do the right thing. This gives us a way to
slowly remove move board specific code if we choose to.



>
>
>>
>> --
>> dwmw2
>>
>
>
>
> --
> Jon Smirl
> jonsmirl at gmail.com



-- 
Jon Smirl
jonsmirl at gmail.com

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On 08/01/2013 05:18 AM, David Woodhouse wrote:
> On Wed, 2013-07-31 at 17:26 -0400, jonsmirl@gmail.com wrote:
>> Alternatively you may be of the belief that it is impossible to get
>> rid of the board specific code. But x86 doesn't have any of it, why
>> should ARM?
> 
> The reason x86 doesn't have it is because it carries three decades worth
> of legacy baggage so that it can still look like a 1980s IBM PC when
> necessary.
> 
> There *have* been some x86 platforms which abandon that legacy crap, and
> for those we *do* have board-specific code. (Is James still maintaining
> Voyager support? It feels very strange to talk about Voyager with it
> *not* being the 'legacy crap' in question...)
> 
> We've even seen *recent* attempts to abandon the legacy crap in the
> embedded x86 space, which backtracked and added it all back again — in
> part because x86 lacked any sane way to describe the hardware if it
> wasn't pretending to be a PC. ACPI doesn't cut it, and DT "wasn't
> invented here"...
>
> Unless you want the ARM world to settle on a strategy of "all the world
> is an Assabet", I'd be careful what you wish for...

There is some level of belief that ACPI will enable running this years
OS on next years h/w. This idea is completely flawed as long as ARM
vendors don't design for compatibility, spin the Si for compatibility
issues, and have some mechanism to emulate legacy h/w. All the
discussions and issues around DT bindings and processes will apply to
ACPI bindings as well.

Rob

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Rob Herring]

On 08/01/2013 05:18 AM, David Woodhouse wrote:
> On Wed, 2013-07-31 at 17:26 -0400, jonsmirl at gmail.com wrote:
>> Alternatively you may be of the belief that it is impossible to get
>> rid of the board specific code. But x86 doesn't have any of it, why
>> should ARM?
> 
> The reason x86 doesn't have it is because it carries three decades worth
> of legacy baggage so that it can still look like a 1980s IBM PC when
> necessary.
> 
> There *have* been some x86 platforms which abandon that legacy crap, and
> for those we *do* have board-specific code. (Is James still maintaining
> Voyager support? It feels very strange to talk about Voyager with it
> *not* being the 'legacy crap' in question...)
> 
> We've even seen *recent* attempts to abandon the legacy crap in the
> embedded x86 space, which backtracked and added it all back again ? in
> part because x86 lacked any sane way to describe the hardware if it
> wasn't pretending to be a PC. ACPI doesn't cut it, and DT "wasn't
> invented here"...
>
> Unless you want the ARM world to settle on a strategy of "all the world
> is an Assabet", I'd be careful what you wish for...

There is some level of belief that ACPI will enable running this years
OS on next years h/w. This idea is completely flawed as long as ARM
vendors don't design for compatibility, spin the Si for compatibility
issues, and have some mechanism to emulate legacy h/w. All the
discussions and issues around DT bindings and processes will apply to
ACPI bindings as well.

Rob

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 695 bytes --]

On Wed, Jul 31, 2013 at 05:26:47PM -0400, jonsmirl@gmail.com wrote:
> On Wed, Jul 31, 2013 at 4:48 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > On Wed, Jul 31, 2013 at 04:37:36PM -0400, jonsmirl@gmail.com
> wrote:
[snip]
> Alternatively you may be of the belief that it is impossible to get
> rid of the board specific code. But x86 doesn't have any of it, why
> should ARM?

Sure x86 has board specific code.  It's just that x86 basically only
has one board - PC.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[David Gibson]

On Wed, Jul 31, 2013 at 05:26:47PM -0400, jonsmirl at gmail.com wrote:
> On Wed, Jul 31, 2013 at 4:48 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > On Wed, Jul 31, 2013 at 04:37:36PM -0400, jonsmirl at gmail.com
> wrote:
[snip]
> Alternatively you may be of the belief that it is impossible to get
> rid of the board specific code. But x86 doesn't have any of it, why
> should ARM?

Sure x86 has board specific code.  It's just that x86 basically only
has one board - PC.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130802/5f25dc97/attachment.sig>

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tony Lindgren]

* Russell King - ARM Linux <linux@arm.linux.org.uk> [130731 13:22]:
> On Wed, Jul 31, 2013 at 10:00:17PM +0200, Richard Cochran wrote:
> > On Wed, Jul 31, 2013 at 09:29:35PM +0200, Tomasz Figa wrote:
> > > 
> > > I showed you two example solutions that could handle this use case without 
> > > stable binding ABI, just to prove that b) is not the only option (even if 
> > > it's the best one, which I continue to agree on, don't get me wrong).
> > 
> > You only showed *one* solution (b) that satisfies the use case. The
> > use case was:
> > 
> >    User acquires a machine running ARM Linux version 3.x, with u-boot
> >    and dtb in a read only flash partition. The board boots and works just
> >                 ^^^^^^^^^^^^^^^^^^^^^^^^^
> >    fine. However, for his application, the user requires a new kernel
> >    feature that appeared in version 3.y where y > x. He compiles the new
> >    kernel, and it also works.
> > 
> > But you suggested:
> > 
> >  a) using DT as direct replacement for board files - this means that you
> >     are free to say that DTSes are strictly coupled with kernel version
> >     and you are free to modify the bindings - see the analogy to board
> >     files, where you could modify the platform data structures and could
> >     not directly copy board file from one kernel version to another,
> > 
> > In the use case, the kernel is upgraded, but not the DTB. So this
> > solution makes no sense.
> 
> That's also crap for another reason.  DT on the whole is _supposed_ to
> describe what the hardware is, and how it is wired up in a WELL DEFINED
> and STABLE manner.  Unfortunately, there's a *BIG* mistake that was made
> - having this /chosen node in DT which gets used for "software"
> configuration - eg, the command line and so forth.
> 
> That was a mistake because it means that the DT isn't purely a
> description of the hardware.  Such stuff should have been left in ATAGs,
> and DT should have been placed into its own ATAG entry.  There would
> not have been any conflict between ATAGs and DT, because ATAGs generally
> don't deal with hardware configuration - the only real bit of hardware
> configuration conveyed via ATAGs is the location of memory and size of
> memory.

This I completely agree with. And I'd go a bit further requiring the DT
binding should describe the _types_ of registers the hardware has so the
device driver does not have to contain data for each similar supported
register instances for things like clocks and muxes and timers.

In the worst case, platform_data is just being replaced by device tree
and driver hacks in a confusing way that requires constant updating of
both the .dts files and the device driver.

Regards,

Tony

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Tony Lindgren]

* Russell King - ARM Linux <linux@arm.linux.org.uk> [130731 13:22]:
> On Wed, Jul 31, 2013 at 10:00:17PM +0200, Richard Cochran wrote:
> > On Wed, Jul 31, 2013 at 09:29:35PM +0200, Tomasz Figa wrote:
> > > 
> > > I showed you two example solutions that could handle this use case without 
> > > stable binding ABI, just to prove that b) is not the only option (even if 
> > > it's the best one, which I continue to agree on, don't get me wrong).
> > 
> > You only showed *one* solution (b) that satisfies the use case. The
> > use case was:
> > 
> >    User acquires a machine running ARM Linux version 3.x, with u-boot
> >    and dtb in a read only flash partition. The board boots and works just
> >                 ^^^^^^^^^^^^^^^^^^^^^^^^^
> >    fine. However, for his application, the user requires a new kernel
> >    feature that appeared in version 3.y where y > x. He compiles the new
> >    kernel, and it also works.
> > 
> > But you suggested:
> > 
> >  a) using DT as direct replacement for board files - this means that you
> >     are free to say that DTSes are strictly coupled with kernel version
> >     and you are free to modify the bindings - see the analogy to board
> >     files, where you could modify the platform data structures and could
> >     not directly copy board file from one kernel version to another,
> > 
> > In the use case, the kernel is upgraded, but not the DTB. So this
> > solution makes no sense.
> 
> That's also crap for another reason.  DT on the whole is _supposed_ to
> describe what the hardware is, and how it is wired up in a WELL DEFINED
> and STABLE manner.  Unfortunately, there's a *BIG* mistake that was made
> - having this /chosen node in DT which gets used for "software"
> configuration - eg, the command line and so forth.
> 
> That was a mistake because it means that the DT isn't purely a
> description of the hardware.  Such stuff should have been left in ATAGs,
> and DT should have been placed into its own ATAG entry.  There would
> not have been any conflict between ATAGs and DT, because ATAGs generally
> don't deal with hardware configuration - the only real bit of hardware
> configuration conveyed via ATAGs is the location of memory and size of
> memory.

This I completely agree with. And I'd go a bit further requiring the DT
binding should describe the _types_ of registers the hardware has so the
device driver does not have to contain data for each similar supported
register instances for things like clocks and muxes and timers.

In the worst case, platform_data is just being replaced by device tree
and driver hacks in a confusing way that requires constant updating of
both the .dts files and the device driver.

Regards,

Tony

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[H. Peter Anvin]

On 08/01/2013 08:50 PM, David Gibson wrote:
> On Wed, Jul 31, 2013 at 05:26:47PM -0400, jonsmirl@gmail.com
> wrote:
>> On Wed, Jul 31, 2013 at 4:48 PM, Russell King - ARM Linux 
>> <linux@arm.linux.org.uk> wrote:
>>> On Wed, Jul 31, 2013 at 04:37:36PM -0400, jonsmirl@gmail.com
>> wrote:
> [snip]
>> Alternatively you may be of the belief that it is impossible to
>> get rid of the board specific code. But x86 doesn't have any of
>> it, why should ARM?
> 
> Sure x86 has board specific code.  It's just that x86 basically
> only has one board - PC.
> 

That is one aspect (hardware standardization)... but it is more to it
than that.

	-hpa

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[H. Peter Anvin]

On 08/01/2013 08:50 PM, David Gibson wrote:
> On Wed, Jul 31, 2013 at 05:26:47PM -0400, jonsmirl at gmail.com
> wrote:
>> On Wed, Jul 31, 2013 at 4:48 PM, Russell King - ARM Linux 
>> <linux@arm.linux.org.uk> wrote:
>>> On Wed, Jul 31, 2013 at 04:37:36PM -0400, jonsmirl at gmail.com
>> wrote:
> [snip]
>> Alternatively you may be of the belief that it is impossible to
>> get rid of the board specific code. But x86 doesn't have any of
>> it, why should ARM?
> 
> Sure x86 has board specific code.  It's just that x86 basically
> only has one board - PC.
> 

That is one aspect (hardware standardization)... but it is more to it
than that.

	-hpa

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Guenter Roeck]

On 08/13/2013 04:32 PM, H. Peter Anvin wrote:
> On 08/01/2013 08:50 PM, David Gibson wrote:
>> On Wed, Jul 31, 2013 at 05:26:47PM -0400, jonsmirl@gmail.com
>> wrote:
>>> On Wed, Jul 31, 2013 at 4:48 PM, Russell King - ARM Linux
>>> <linux@arm.linux.org.uk> wrote:
>>>> On Wed, Jul 31, 2013 at 04:37:36PM -0400, jonsmirl@gmail.com
>>> wrote:
>> [snip]
>>> Alternatively you may be of the belief that it is impossible to
>>> get rid of the board specific code. But x86 doesn't have any of
>>> it, why should ARM?
>>
>> Sure x86 has board specific code.  It's just that x86 basically
>> only has one board - PC.
>>
>
> That is one aspect (hardware standardization)... but it is more to it
> than that.
>
I have to deal with lots of embedded / non-PC x86 based systems. Worst one
I encountered so far was a board where the VGA memory space was re-used
for an eeprom. The upcoming next generation hardware I'll have to support
is so far off-standard that I'll probably have to define a new platform
type (similar to OLPC or CE4100).

No, it is not all PC. Not anymore. Intel has started to sell into
the embedded space, where PC compatibility is not a requirement.

Guenter

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[Guenter Roeck]

On 08/13/2013 04:32 PM, H. Peter Anvin wrote:
> On 08/01/2013 08:50 PM, David Gibson wrote:
>> On Wed, Jul 31, 2013 at 05:26:47PM -0400, jonsmirl at gmail.com
>> wrote:
>>> On Wed, Jul 31, 2013 at 4:48 PM, Russell King - ARM Linux
>>> <linux@arm.linux.org.uk> wrote:
>>>> On Wed, Jul 31, 2013 at 04:37:36PM -0400, jonsmirl at gmail.com
>>> wrote:
>> [snip]
>>> Alternatively you may be of the belief that it is impossible to
>>> get rid of the board specific code. But x86 doesn't have any of
>>> it, why should ARM?
>>
>> Sure x86 has board specific code.  It's just that x86 basically
>> only has one board - PC.
>>
>
> That is one aspect (hardware standardization)... but it is more to it
> than that.
>
I have to deal with lots of embedded / non-PC x86 based systems. Worst one
I encountered so far was a board where the VGA memory space was re-used
for an eeprom. The upcoming next generation hardware I'll have to support
is so far off-standard that I'll probably have to define a new platform
type (similar to OLPC or CE4100).

No, it is not all PC. Not anymore. Intel has started to sell into
the embedded space, where PC compatibility is not a requirement.

Guenter

Re: [Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[H. Peter Anvin]

On 08/13/2013 06:49 PM, Guenter Roeck wrote:
>
>> That is one aspect (hardware standardization)... but it is more to it
>> than that.
>
> I have to deal with lots of embedded / non-PC x86 based systems. Worst one
> I encountered so far was a board where the VGA memory space was re-used
> for an eeprom. The upcoming next generation hardware I'll have to support
> is so far off-standard that I'll probably have to define a new platform
> type (similar to OLPC or CE4100).
> 
> No, it is not all PC. Not anymore. Intel has started to sell into
> the embedded space, where PC compatibility is not a requirement.
> 

We try to encourage vendors to do the right thing, where "the right
thing" means, among other things, not to do gratuitously different
things.  With increasing amounts of the platform moving into the CPU and
PCH this is of course also becoming more and more rare.

Even on actual PCs we have seen some truly "special" facepalms.  There
was the Olivetti machine which used the GPIO for "fast A20" to control
the screen backlight, for example.  That one was fun to deal with.

	-hpa

[Ksummit-2013-discuss] DT bindings as ABI [was: Do we have people interested in device tree janitoring / cleanup?]

[H. Peter Anvin]

On 08/13/2013 06:49 PM, Guenter Roeck wrote:
>
>> That is one aspect (hardware standardization)... but it is more to it
>> than that.
>
> I have to deal with lots of embedded / non-PC x86 based systems. Worst one
> I encountered so far was a board where the VGA memory space was re-used
> for an eeprom. The upcoming next generation hardware I'll have to support
> is so far off-standard that I'll probably have to define a new platform
> type (similar to OLPC or CE4100).
> 
> No, it is not all PC. Not anymore. Intel has started to sell into
> the embedded space, where PC compatibility is not a requirement.
> 

We try to encourage vendors to do the right thing, where "the right
thing" means, among other things, not to do gratuitously different
things.  With increasing amounts of the platform moving into the CPU and
PCH this is of course also becoming more and more rare.

Even on actual PCs we have seen some truly "special" facepalms.  There
was the Olivetti machine which used the GPIO for "fast A20" to control
the screen backlight, for example.  That one was fun to deal with.

	-hpa