Issue with fw_devlink=on on 5.14

Issue with fw_devlink=on on 5.14

[Jernej Škrabec]

Hi Saravana!

I noticed that in kernel 5.14, at least on Allwinner H3, deinterlace driver 
doesn't load anymore if fw_devlink is set to on. If I set it to off, driver 
loads and works without problems. If I remove interconnects property in DT 
node and use driver hack instead, it also loads and work ok.

For reference, I speak about this node:
https://elixir.bootlin.com/linux/v5.14/source/arch/arm/boot/dts/sun8i-h3.dtsi#L153

Do you have any clue why it doesn't work?

Best regards,
Jernej

Re: Issue with fw_devlink=on on 5.14

[Saravana Kannan]

On Wed, Sep 15, 2021 at 11:07 AM Jernej Škrabec
<jernej.skrabec@gmail.com> wrote:
>
> Hi Saravana!
>
> I noticed that in kernel 5.14, at least on Allwinner H3, deinterlace driver
> doesn't load anymore if fw_devlink is set to on. If I set it to off, driver
> loads and works without problems. If I remove interconnects property in DT
> node and use driver hack instead, it also loads and work ok.
>
> For reference, I speak about this node:
> https://elixir.bootlin.com/linux/v5.14/source/arch/arm/boot/dts/sun8i-h3.dtsi#L153
>
> Do you have any clue why it doesn't work?

Kinda busy today and I'll take a close look later, but if you delete
"interconnect" but it works, then it seems like you have no driver for
your "mbus" interconnect? Is that expected?

If you have a driver, then make it use the proper driver model (probe
a platform device).

If you don't have a driver for it, I'm assuming you have
CONFIG_MODULES enabled. Otherwise, fw_devlink=on is smart enough to
know not to block on devices that'll never be probed.

If you do need CONFIG_MODULES enabled, then use
deferred_probe_timeout=xxx so that fw_devlink doesn't wait for more
modules to be loaded past xxx seconds.

Also I like to look up emails on lore, so when emailing me about
fw_devlink (or any Linux stuff in general I suppose), it'd be nice if
you can cc LKML.

Thanks,
Saravana

Re: Issue with fw_devlink=on on 5.14

[Saravana Kannan]

On Wed, Sep 15, 2021 at 11:23 AM Saravana Kannan <saravanak@google.com> wrote:
>
> On Wed, Sep 15, 2021 at 11:07 AM Jernej Škrabec
> <jernej.skrabec@gmail.com> wrote:
> >
> > Hi Saravana!
> >
> > I noticed that in kernel 5.14, at least on Allwinner H3, deinterlace driver
> > doesn't load anymore if fw_devlink is set to on. If I set it to off, driver
> > loads and works without problems. If I remove interconnects property in DT
> > node and use driver hack instead, it also loads and work ok.
> >
> > For reference, I speak about this node:
> > https://elixir.bootlin.com/linux/v5.14/source/arch/arm/boot/dts/sun8i-h3.dtsi#L153

Oh the dts file please. Not dtsi. I need the full picture of dependencies.

-Saravana

> >
> > Do you have any clue why it doesn't work?
>
> Kinda busy today and I'll take a close look later, but if you delete
> "interconnect" but it works, then it seems like you have no driver for
> your "mbus" interconnect? Is that expected?
>
> If you have a driver, then make it use the proper driver model (probe
> a platform device).
>
> If you don't have a driver for it, I'm assuming you have
> CONFIG_MODULES enabled. Otherwise, fw_devlink=on is smart enough to
> know not to block on devices that'll never be probed.
>
> If you do need CONFIG_MODULES enabled, then use
> deferred_probe_timeout=xxx so that fw_devlink doesn't wait for more
> modules to be loaded past xxx seconds.
>
> Also I like to look up emails on lore, so when emailing me about
> fw_devlink (or any Linux stuff in general I suppose), it'd be nice if
> you can cc LKML.
>
> Thanks,
> Saravana

Re: Re: Issue with fw_devlink=on on 5.14

[Jernej Škrabec]

/cc linux-kernel@vger.kernel.org

Dne sreda, 15. september 2021 ob 20:36:11 CEST je Saravana Kannan napisal(a):
> On Wed, Sep 15, 2021 at 11:23 AM Saravana Kannan <saravanak@google.com> 
wrote:
> >
> > On Wed, Sep 15, 2021 at 11:07 AM Jernej Škrabec
> > <jernej.skrabec@gmail.com> wrote:
> > >
> > > Hi Saravana!
> > >
> > > I noticed that in kernel 5.14, at least on Allwinner H3, deinterlace 
driver
> > > doesn't load anymore if fw_devlink is set to on. If I set it to off, 
driver
> > > loads and works without problems. If I remove interconnects property in 
DT
> > > node and use driver hack instead, it also loads and work ok.
> > >
> > > For reference, I speak about this node:
> > > https://elixir.bootlin.com/linux/v5.14/source/arch/arm/boot/dts/sun8i-h3.dtsi#L153
> 
> Oh the dts file please. Not dtsi. I need the full picture of dependencies.

I just dumped DTB from the board I use for testing this:
http://ix.io/3yZB

> 
> -Saravana
> 
> > >
> > > Do you have any clue why it doesn't work?
> >
> > Kinda busy today and I'll take a close look later, but if you delete
> > "interconnect" but it works, then it seems like you have no driver for
> > your "mbus" interconnect? Is that expected?

Ah, you're completely right. There is no MBUS driver (yet), since current 
purpose of that node is to determine DMA memory offsets and for that you don't 
need a driver.

There is already WIP driver for it, though:
https://github.com/crust-firmware/linux/commit/
db34316a0194dbf0fcb8eba677ef11085a06a340

> >
> > If you have a driver, then make it use the proper driver model (probe
> > a platform device).
> >
> > If you don't have a driver for it, I'm assuming you have
> > CONFIG_MODULES enabled. Otherwise, fw_devlink=on is smart enough to
> > know not to block on devices that'll never be probed.

Correct, modules are enabled.

> >
> > If you do need CONFIG_MODULES enabled, then use
> > deferred_probe_timeout=xxx so that fw_devlink doesn't wait for more
> > modules to be loaded past xxx seconds.

It's not what I want, but more that we broke default configuration and 
CONFIG_MODULES=y should really work. Is there any other way to fix this besides 
providing dummy MBUS driver? I doubt it would be accepted for backporting.

> >
> > Also I like to look up emails on lore, so when emailing me about
> > fw_devlink (or any Linux stuff in general I suppose), it'd be nice if
> > you can cc LKML.

Noted.

Best regards,
Jernej

> >
> > Thanks,
> > Saravana
> 

Re: Re: Issue with fw_devlink=on on 5.14

[Saravana Kannan]

On Wed, Sep 15, 2021 at 11:59 AM Jernej Škrabec
<jernej.skrabec@gmail.com> wrote:
>
> /cc linux-kernel@vger.kernel.org
>
> Dne sreda, 15. september 2021 ob 20:36:11 CEST je Saravana Kannan napisal(a):
> > On Wed, Sep 15, 2021 at 11:23 AM Saravana Kannan <saravanak@google.com>
> wrote:
> > >
> > > On Wed, Sep 15, 2021 at 11:07 AM Jernej Škrabec
> > > <jernej.skrabec@gmail.com> wrote:
> > > >
> > > > Hi Saravana!
> > > >
> > > > I noticed that in kernel 5.14, at least on Allwinner H3, deinterlace
> driver
> > > > doesn't load anymore if fw_devlink is set to on. If I set it to off,
> driver
> > > > loads and works without problems. If I remove interconnects property in
> DT
> > > > node and use driver hack instead, it also loads and work ok.
> > > >
> > > > For reference, I speak about this node:
> > > > https://elixir.bootlin.com/linux/v5.14/source/arch/arm/boot/dts/sun8i-h3.dtsi#L153
> >
> > Oh the dts file please. Not dtsi. I need the full picture of dependencies.
>
> I just dumped DTB from the board I use for testing this:
> http://ix.io/3yZB

I was basically asking for this:
arch/arm/boot/dts/sun8i-h3-orangepi-plus2e.dts

>
> >
> > -Saravana
> >
> > > >
> > > > Do you have any clue why it doesn't work?
> > >
> > > Kinda busy today and I'll take a close look later, but if you delete
> > > "interconnect" but it works, then it seems like you have no driver for
> > > your "mbus" interconnect? Is that expected?
>
> Ah, you're completely right. There is no MBUS driver (yet), since current
> purpose of that node is to determine DMA memory offsets and for that you don't
> need a driver.

Can you explain more how you get the DMA memory offsets in the code
from this node?

You might be able to add status="disabled" to the mbus node and this
would go away as fw_devlink doesn't cause a consumer to wait on a
disabled supplier. See more below -- you have a bunch of options.

>
> There is already WIP driver for it, though:
> https://github.com/crust-firmware/linux/commit/
> db34316a0194dbf0fcb8eba677ef11085a06a340
>
> > >
> > > If you have a driver, then make it use the proper driver model (probe
> > > a platform device).
> > >
> > > If you don't have a driver for it, I'm assuming you have
> > > CONFIG_MODULES enabled. Otherwise, fw_devlink=on is smart enough to
> > > know not to block on devices that'll never be probed.
>
> Correct, modules are enabled.
>
> > >
> > > If you do need CONFIG_MODULES enabled, then use
> > > deferred_probe_timeout=xxx so that fw_devlink doesn't wait for more
> > > modules to be loaded past xxx seconds.
>
> It's not what I want, but more that we broke default configuration and
> CONFIG_MODULES=y should really work.

Well, fw_devlink=on can't tell if you don't have a driver or if you
are going to load it soon (since you have CONFIG_MODULES enabled). So
if it's a board you are actively bringing up, one of the other options
below might be good to use.

> Is there any other way to fix this besides
> providing dummy MBUS driver? I doubt it would be accepted for backporting.

You could use fw_devlink=permissive in the kernel commandline, but it
will disable a bunch of benefits of fw_devlink=on (no pointless
deferred probes, no suspend/resume ordering based on dependencies
listed in DT, etc).

Another option would be to use deferred_probe_timeout=1 in the kernel
commandline, and this will cause this to be unblocked 1 second after
late_initcall() and you'll get the benefit of fw_devlink=on for all
the devices that have a driver registered by then (and some more
devices too -- see commit text of
d46f3e3ed5276e756caf40f760d4902d15c12dcb).

-Saravana

>
> > >
> > > Also I like to look up emails on lore, so when emailing me about
> > > fw_devlink (or any Linux stuff in general I suppose), it'd be nice if
> > > you can cc LKML.
>
> Noted.
>
> Best regards,
> Jernej
>
> > >
> > > Thanks,
> > > Saravana
> >
>
>

Re: Re: Issue with fw_devlink=on on 5.14

[Saravana Kannan]

On Wed, Sep 15, 2021 at 11:59 AM Jernej Škrabec
<jernej.skrabec@gmail.com> wrote:
>
> /cc linux-kernel@vger.kernel.org
>
> Dne sreda, 15. september 2021 ob 20:36:11 CEST je Saravana Kannan napisal(a):
> > On Wed, Sep 15, 2021 at 11:23 AM Saravana Kannan <saravanak@google.com>
> wrote:
> > >
> > > On Wed, Sep 15, 2021 at 11:07 AM Jernej Škrabec
> > > <jernej.skrabec@gmail.com> wrote:
> > > >
> > > > Hi Saravana!
> > > >
> > > > I noticed that in kernel 5.14, at least on Allwinner H3, deinterlace
> driver
> > > > doesn't load anymore if fw_devlink is set to on. If I set it to off,
> driver
> > > > loads and works without problems. If I remove interconnects property in
> DT
> > > > node and use driver hack instead, it also loads and work ok.
> > > >
> > > > For reference, I speak about this node:
> > > > https://elixir.bootlin.com/linux/v5.14/source/arch/arm/boot/dts/sun8i-h3.dtsi#L153
> >
> > Oh the dts file please. Not dtsi. I need the full picture of dependencies.
>
> I just dumped DTB from the board I use for testing this:
> http://ix.io/3yZB

I was basically asking for this:
arch/arm/boot/dts/sun8i-h3-orangepi-plus2e.dts

>
> >
> > -Saravana
> >
> > > >
> > > > Do you have any clue why it doesn't work?
> > >
> > > Kinda busy today and I'll take a close look later, but if you delete
> > > "interconnect" but it works, then it seems like you have no driver for
> > > your "mbus" interconnect? Is that expected?
>
> Ah, you're completely right. There is no MBUS driver (yet), since current
> purpose of that node is to determine DMA memory offsets and for that you don't
> need a driver.

Can you explain more how you get the DMA memory offsets in the code
from this node?

You might be able to add status="disabled" to the mbus node and this
would go away as fw_devlink doesn't cause a consumer to wait on a
disabled supplier. See more below -- you have a bunch of options.

>
> There is already WIP driver for it, though:
> https://github.com/crust-firmware/linux/commit/
> db34316a0194dbf0fcb8eba677ef11085a06a340
>
> > >
> > > If you have a driver, then make it use the proper driver model (probe
> > > a platform device).
> > >
> > > If you don't have a driver for it, I'm assuming you have
> > > CONFIG_MODULES enabled. Otherwise, fw_devlink=on is smart enough to
> > > know not to block on devices that'll never be probed.
>
> Correct, modules are enabled.
>
> > >
> > > If you do need CONFIG_MODULES enabled, then use
> > > deferred_probe_timeout=xxx so that fw_devlink doesn't wait for more
> > > modules to be loaded past xxx seconds.
>
> It's not what I want, but more that we broke default configuration and
> CONFIG_MODULES=y should really work.

Well, fw_devlink=on can't tell if you don't have a driver or if you
are going to load it soon (since you have CONFIG_MODULES enabled). So
if it's a board you are actively bringing up, one of the other options
below might be good to use.

> Is there any other way to fix this besides
> providing dummy MBUS driver? I doubt it would be accepted for backporting.

You could use fw_devlink=permissive in the kernel commandline, but it
will disable a bunch of benefits of fw_devlink=on (no pointless
deferred probes, no suspend/resume ordering based on dependencies
listed in DT, etc).

Another option would be to use deferred_probe_timeout=1 in the kernel
commandline, and this will cause this to be unblocked 1 second after
late_initcall() and you'll get the benefit of fw_devlink=on for all
the devices that have a driver registered by then (and some more
devices too -- see commit text of
d46f3e3ed5276e756caf40f760d4902d15c12dcb).

-Saravana

>
> > >
> > > Also I like to look up emails on lore, so when emailing me about
> > > fw_devlink (or any Linux stuff in general I suppose), it'd be nice if
> > > you can cc LKML.
>
> Noted.
>
> Best regards,
> Jernej
>
> > >
> > > Thanks,
> > > Saravana
> >
>
>

Re: Issue with fw_devlink=on on 5.14

[Jernej Škrabec]

/cc Maxime

Dne petek, 17. september 2021 ob 04:55:22 CEST je Saravana Kannan napisal(a):
> On Wed, Sep 15, 2021 at 11:59 AM Jernej Škrabec
> 
> <jernej.skrabec@gmail.com> wrote:
> > /cc linux-kernel@vger.kernel.org
> > 
> > Dne sreda, 15. september 2021 ob 20:36:11 CEST je Saravana Kannan 
napisal(a):
> > > On Wed, Sep 15, 2021 at 11:23 AM Saravana Kannan <saravanak@google.com>
> > 
> > wrote:
> > > > On Wed, Sep 15, 2021 at 11:07 AM Jernej Škrabec
> > > > 
> > > > <jernej.skrabec@gmail.com> wrote:
> > > > > Hi Saravana!
> > > > > 
> > > > > I noticed that in kernel 5.14, at least on Allwinner H3, deinterlace
> > 
> > driver
> > 
> > > > > doesn't load anymore if fw_devlink is set to on. If I set it to off,
> > 
> > driver
> > 
> > > > > loads and works without problems. If I remove interconnects property
> > > > > in
> > 
> > DT
> > 
> > > > > node and use driver hack instead, it also loads and work ok.
> > > > > 
> > > > > For reference, I speak about this node:
> > > > > https://elixir.bootlin.com/linux/v5.14/source/arch/arm/boot/dts/sun8
> > > > > i-h3.dtsi#L153> > 
> > > Oh the dts file please. Not dtsi. I need the full picture of
> > > dependencies.
> > 
> > I just dumped DTB from the board I use for testing this:
> > http://ix.io/3yZB
> 
> I was basically asking for this:
> arch/arm/boot/dts/sun8i-h3-orangepi-plus2e.dts
> 
> > > -Saravana
> > > 
> > > > > Do you have any clue why it doesn't work?
> > > > 
> > > > Kinda busy today and I'll take a close look later, but if you delete
> > > > "interconnect" but it works, then it seems like you have no driver for
> > > > your "mbus" interconnect? Is that expected?
> > 
> > Ah, you're completely right. There is no MBUS driver (yet), since current
> > purpose of that node is to determine DMA memory offsets and for that you
> > don't need a driver.
> 
> Can you explain more how you get the DMA memory offsets in the code
> from this node?

Maxime, can you help with this?

> 
> You might be able to add status="disabled" to the mbus node and this
> would go away as fw_devlink doesn't cause a consumer to wait on a
> disabled supplier. See more below -- you have a bunch of options.
> 
> > There is already WIP driver for it, though:
> > https://github.com/crust-firmware/linux/commit/
> > db34316a0194dbf0fcb8eba677ef11085a06a340
> > 
> > > > If you have a driver, then make it use the proper driver model (probe
> > > > a platform device).
> > > > 
> > > > If you don't have a driver for it, I'm assuming you have
> > > > CONFIG_MODULES enabled. Otherwise, fw_devlink=on is smart enough to
> > > > know not to block on devices that'll never be probed.
> > 
> > Correct, modules are enabled.
> > 
> > > > If you do need CONFIG_MODULES enabled, then use
> > > > deferred_probe_timeout=xxx so that fw_devlink doesn't wait for more
> > > > modules to be loaded past xxx seconds.
> > 
> > It's not what I want, but more that we broke default configuration and
> > CONFIG_MODULES=y should really work.
> 
> Well, fw_devlink=on can't tell if you don't have a driver or if you
> are going to load it soon (since you have CONFIG_MODULES enabled). So
> if it's a board you are actively bringing up, one of the other options
> below might be good to use.
> 
> > Is there any other way to fix this besides
> > providing dummy MBUS driver? I doubt it would be accepted for backporting.
> 
> You could use fw_devlink=permissive in the kernel commandline, but it
> will disable a bunch of benefits of fw_devlink=on (no pointless
> deferred probes, no suspend/resume ordering based on dependencies
> listed in DT, etc).

I tried this one, but for some reason network driver didn't show up and there 
was crash when doing reboot. Even if it would work, I don't consider it as 
proper solution because... (see below)

> 
> Another option would be to use deferred_probe_timeout=1 in the kernel
> commandline, and this will cause this to be unblocked 1 second after
> late_initcall() and you'll get the benefit of fw_devlink=on for all
> the devices that have a driver registered by then (and some more
> devices too -- see commit text of
> d46f3e3ed5276e756caf40f760d4902d15c12dcb).

That would probably work (can't test atm), but as I said, I want solution 
without adding any argument. Users are not familiar with any of this. From 
their perspective, it looks like as the driver (not just this one, but also 
others which use interconnects) was never enabled in kernel config in first 
place.

So, if "status = disabled" works, then great. If not, something else must be 
found. 

Note: I won't be able to test anything until next week.

Best regards,
Jernej

> 
> -Saravana
> 
> > > > Also I like to look up emails on lore, so when emailing me about
> > > > fw_devlink (or any Linux stuff in general I suppose), it'd be nice if
> > > > you can cc LKML.
> > 
> > Noted.
> > 
> > Best regards,
> > Jernej
> > 
> > > > Thanks,
> > > > Saravana

Re: Issue with fw_devlink=on on 5.14

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 2747 bytes --]

On Fri, Sep 17, 2021 at 06:32:45AM +0200, Jernej Škrabec wrote:
> Dne petek, 17. september 2021 ob 04:55:22 CEST je Saravana Kannan napisal(a):
> > On Wed, Sep 15, 2021 at 11:59 AM Jernej Škrabec
> > 
> > <jernej.skrabec@gmail.com> wrote:
> > > /cc linux-kernel@vger.kernel.org
> > > 
> > > Dne sreda, 15. september 2021 ob 20:36:11 CEST je Saravana Kannan 
> napisal(a):
> > > > On Wed, Sep 15, 2021 at 11:23 AM Saravana Kannan <saravanak@google.com>
> > > 
> > > wrote:
> > > > > On Wed, Sep 15, 2021 at 11:07 AM Jernej Škrabec
> > > > > 
> > > > > <jernej.skrabec@gmail.com> wrote:
> > > > > > Hi Saravana!
> > > > > > 
> > > > > > I noticed that in kernel 5.14, at least on Allwinner H3, deinterlace
> > > 
> > > driver
> > > 
> > > > > > doesn't load anymore if fw_devlink is set to on. If I set it to off,
> > > 
> > > driver
> > > 
> > > > > > loads and works without problems. If I remove interconnects property
> > > > > > in
> > > 
> > > DT
> > > 
> > > > > > node and use driver hack instead, it also loads and work ok.
> > > > > > 
> > > > > > For reference, I speak about this node:
> > > > > > https://elixir.bootlin.com/linux/v5.14/source/arch/arm/boot/dts/sun8
> > > > > > i-h3.dtsi#L153> > 
> > > > Oh the dts file please. Not dtsi. I need the full picture of
> > > > dependencies.
> > > 
> > > I just dumped DTB from the board I use for testing this:
> > > http://ix.io/3yZB
> > 
> > I was basically asking for this:
> > arch/arm/boot/dts/sun8i-h3-orangepi-plus2e.dts
> > 
> > > > -Saravana
> > > > 
> > > > > > Do you have any clue why it doesn't work?
> > > > > 
> > > > > Kinda busy today and I'll take a close look later, but if you delete
> > > > > "interconnect" but it works, then it seems like you have no driver for
> > > > > your "mbus" interconnect? Is that expected?
> > > 
> > > Ah, you're completely right. There is no MBUS driver (yet), since current
> > > purpose of that node is to determine DMA memory offsets and for that you
> > > don't need a driver.
> > 
> > Can you explain more how you get the DMA memory offsets in the code
> > from this node?
> 
> Maxime, can you help with this?

Sure, Jernej is right, the interconnects binding with the dma-mem name
is here to describe the path DMA accesses take between devices.

In our case, the registers accesses go through the usual busses, but the
DMA accesses go through a separate one controlled by the mbus controller
Jernej was talking about.

The driver core will then take this into account and apply any DMA
mapping the mbus controller has:
https://elixir.bootlin.com/linux/latest/source/drivers/of/address.c#L582

So it's entirely transparent to the consumer driver.

Maxime

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]