prueth: IEP driver doesn't probe anymore

prueth: IEP driver doesn't probe anymore

[Romain Naour]

Hi All,

I'm able to use the prueth driver on a AM5749 cpu using the TI vendor branch
(ti-linux-5.10.y). But I need a more recent kernel to support other device on my
custom board. So I had to rebase all the TI work related to the out of tree
prueth driver on a the latest kernel.

I hope this driver will be merged in the kernel soon but the series [1] doesn't
include the prueth driver for AM57xx cpus (only the icssg_prueth for AM65xx is
included).

For some reason the IEP driver [2] [3] doesn't probe anymore. Is a side effect
of recent ti-sysc or legacy platform data work ? I'm using a 5.19.8 stable kernel.

The prueth_probe() error out with "unable to get IEP" log message.

Do you have any clue?

Danish, if you can provide a new version of the "PRUSS Remoteproc, Platform
APIS, and Ethernet Driver" series including the prueth and iep driver for
AM57xx, I will be able to test it [4].

Thanks!

[1] https://lore.kernel.org/netdev/20220406094358.7895-1-p-mohan@ti.com/
[2]
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/?h=ti-linux-5.10.y&id=d3bf5a58c356a90fe14d34f213d6195b9a946dc5
[3]
https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/?h=ti-linux-5.10.y&id=7cb095866c8b436176dd783878a9e9c3c0bada0b
[4]
https://lore.kernel.org/linux-remoteproc/992019ad-5c58-d420-8a18-a82228f8e086@smile.fr/

Best regards,
Romain

Re: prueth: IEP driver doesn't probe anymore

[Romain Naour]

Hi All,

Le 16/09/2022 à 10:55, Romain Naour a écrit :
> Hi All,
> 
> I'm able to use the prueth driver on a AM5749 cpu using the TI vendor branch
> (ti-linux-5.10.y). But I need a more recent kernel to support other device on my
> custom board. So I had to rebase all the TI work related to the out of tree
> prueth driver on a the latest kernel.
> 
> I hope this driver will be merged in the kernel soon but the series [1] doesn't
> include the prueth driver for AM57xx cpus (only the icssg_prueth for AM65xx is
> included).
> 
> For some reason the IEP driver [2] [3] doesn't probe anymore. Is a side effect
> of recent ti-sysc or legacy platform data work ? I'm using a 5.19.8 stable kernel.
> 
> The prueth_probe() error out with "unable to get IEP" log message.
> 
> Do you have any clue?

I tested several kernel releases between the 5.10 and 5.19 and It turn out that
the issue appear in the 5.13.y kernel.

After a long and time consuming git bisect, the first bad commit is:

Merge tag 'arm-soc-5.13' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=01d7136894410a71932096e0fb9f1d301b6ccf07

Yes this is a merge commit from Linus.

I tested the kernel from each commit parent and the prueth sill works.

So I rebased the branch arm-soc-5.13 merged by Linus to have a fast-forward
history. The second git bisect return this commit as first bad commit:

ARM: dts: Move dra7 l3 noc to a separate node:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=7f2659ce657e87cb7f47e6b15099608eaa1349ac

The switch to ti-sysc seems to have some side effect on some driver like pci-dra7xx:

"After updating pci-dra7xx driver to probe with ti-sysc and genpd, I
noticed that dra7xx_pcie_probe() would not run if a power-domains property
was configured for the interconnect target module."

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=e259c2926c016dd815e5547412356d378fc1f589

This is my current understanding of the issue... but I don't understand the side
effect of the move of the dra7 l3 noc node. Also, there is something merged
before the 'arm-soc-5.13' branch that is a second cause of the issue.

Best regards,
Romain

> 
> Danish, if you can provide a new version of the "PRUSS Remoteproc, Platform
> APIS, and Ethernet Driver" series including the prueth and iep driver for
> AM57xx, I will be able to test it [4].
> 
> Thanks!
> 
> [1] https://lore.kernel.org/netdev/20220406094358.7895-1-p-mohan@ti.com/
> [2]
> https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/?h=ti-linux-5.10.y&id=d3bf5a58c356a90fe14d34f213d6195b9a946dc5
> [3]
> https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/?h=ti-linux-5.10.y&id=7cb095866c8b436176dd783878a9e9c3c0bada0b
> [4]
> https://lore.kernel.org/linux-remoteproc/992019ad-5c58-d420-8a18-a82228f8e086@smile.fr/
> 
> Best regards,
> Romain

Re: prueth: IEP driver doesn't probe anymore

[Romain Naour]

Hi All,

Le 19/09/2022 à 10:07, Romain Naour a écrit :
> Hi All,
> 
> Le 16/09/2022 à 10:55, Romain Naour a écrit :
>> Hi All,
>>
>> I'm able to use the prueth driver on a AM5749 cpu using the TI vendor branch
>> (ti-linux-5.10.y). But I need a more recent kernel to support other device on my
>> custom board. So I had to rebase all the TI work related to the out of tree
>> prueth driver on a the latest kernel.
>>
>> I hope this driver will be merged in the kernel soon but the series [1] doesn't
>> include the prueth driver for AM57xx cpus (only the icssg_prueth for AM65xx is
>> included).
>>
>> For some reason the IEP driver [2] [3] doesn't probe anymore. Is a side effect
>> of recent ti-sysc or legacy platform data work ? I'm using a 5.19.8 stable kernel.
>>
>> The prueth_probe() error out with "unable to get IEP" log message.
>>
>> Do you have any clue?
> 
> I tested several kernel releases between the 5.10 and 5.19 and It turn out that
> the issue appear in the 5.13.y kernel.
> 
> After a long and time consuming git bisect, the first bad commit is:
> 
> Merge tag 'arm-soc-5.13' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=01d7136894410a71932096e0fb9f1d301b6ccf07
> 
> Yes this is a merge commit from Linus.
> 
> I tested the kernel from each commit parent and the prueth sill works.
> 
> So I rebased the branch arm-soc-5.13 merged by Linus to have a fast-forward
> history. The second git bisect return this commit as first bad commit:
> 
> ARM: dts: Move dra7 l3 noc to a separate node:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=7f2659ce657e87cb7f47e6b15099608eaa1349ac
> 
> The switch to ti-sysc seems to have some side effect on some driver like pci-dra7xx:
> 
> "After updating pci-dra7xx driver to probe with ti-sysc and genpd, I
> noticed that dra7xx_pcie_probe() would not run if a power-domains property
> was configured for the interconnect target module."
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=e259c2926c016dd815e5547412356d378fc1f589
> 
> This is my current understanding of the issue... but I don't understand the side
> effect of the move of the dra7 l3 noc node. Also, there is something merged
> before the 'arm-soc-5.13' branch that is a second cause of the issue.

Ok, I understand what's going on...

The issue appear on the merge commit since on a omap side there was the switch
to ti-sysc (devicetree interconnect description) and on upstream side there was
a change on driver core behavior with fw_devlink=on being set by default:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=ea718c699055c8566eb64432388a04974c43b2ea

Actually the issue is really on the TI's prueth and IEP driver whith
fw_devlink=on. The IEP driver probe correctly with fw_devlink=permissive.

Best regards,
Romain


> 
> Best regards,
> Romain
> 
>>
>> Danish, if you can provide a new version of the "PRUSS Remoteproc, Platform
>> APIS, and Ethernet Driver" series including the prueth and iep driver for
>> AM57xx, I will be able to test it [4].
>>
>> Thanks!
>>
>> [1] https://lore.kernel.org/netdev/20220406094358.7895-1-p-mohan@ti.com/
>> [2]
>> https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/?h=ti-linux-5.10.y&id=d3bf5a58c356a90fe14d34f213d6195b9a946dc5
>> [3]
>> https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/?h=ti-linux-5.10.y&id=7cb095866c8b436176dd783878a9e9c3c0bada0b
>> [4]
>> https://lore.kernel.org/linux-remoteproc/992019ad-5c58-d420-8a18-a82228f8e086@smile.fr/
>>
>> Best regards,
>> Romain
> 

Re: prueth: IEP driver doesn't probe anymore

[Tony Lindgren]

Hi,

* Romain Naour <romain.naour@smile.fr> [220921 08:13]:
> Ok, I understand what's going on...
> 
> The issue appear on the merge commit since on a omap side there was the switch
> to ti-sysc (devicetree interconnect description) and on upstream side there was
> a change on driver core behavior with fw_devlink=on being set by default:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=ea718c699055c8566eb64432388a04974c43b2ea

OK good to hear it's not the change to ti-sysc. That should be all pretty
much standard Linux driver related changes.

> Actually the issue is really on the TI's prueth and IEP driver whith
> fw_devlink=on. The IEP driver probe correctly with fw_devlink=permissive.

Argh not again, the fw_devlink changes seem to be causing all kind of
issues. Any ideas what the issue here might be? It might be worth testing
with v6.0-rc7 too as it has a series of fixes to related issues.

Regards,

Tony

Re: prueth: IEP driver doesn't probe anymore

[Vignesh Raghavendra]

On 27/09/22 12:10, Tony Lindgren wrote:
> Hi,
> 
> * Romain Naour <romain.naour@smile.fr> [220921 08:13]:
>> Ok, I understand what's going on...
>>
>> The issue appear on the merge commit since on a omap side there was the switch
>> to ti-sysc (devicetree interconnect description) and on upstream side there was
>> a change on driver core behavior with fw_devlink=on being set by default:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=ea718c699055c8566eb64432388a04974c43b2ea
> 
> OK good to hear it's not the change to ti-sysc. That should be all pretty
> much standard Linux driver related changes.
> 
>> Actually the issue is really on the TI's prueth and IEP driver whith
>> fw_devlink=on. The IEP driver probe correctly with fw_devlink=permissive.
> 
> Argh not again, the fw_devlink changes seem to be causing all kind of
> issues. Any ideas what the issue here might be? It might be worth testing
> with v6.0-rc7 too as it has a series of fixes to related issues.
> 

Could you also try [1] from Puranjay's series? He did have to rework
pru_rproc_get()/pru_rproc_put() a bit to fix few probe failures seen
only on > 5.11 kernels.

[1] https://lore.kernel.org/netdev/20220406094358.7895-3-p-mohan@ti.com/

-- 
Regards
Vignesh

Re: prueth: IEP driver doesn't probe anymore

[Romain Naour]

Hi All,

Le 27/09/2022 à 10:04, Vignesh Raghavendra a écrit :
> 
> 
> On 27/09/22 12:10, Tony Lindgren wrote:
>> Hi,
>>
>> * Romain Naour <romain.naour@smile.fr> [220921 08:13]:
>>> Ok, I understand what's going on...
>>>
>>> The issue appear on the merge commit since on a omap side there was the switch
>>> to ti-sysc (devicetree interconnect description) and on upstream side there was
>>> a change on driver core behavior with fw_devlink=on being set by default:
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=ea718c699055c8566eb64432388a04974c43b2ea
>>
>> OK good to hear it's not the change to ti-sysc. That should be all pretty
>> much standard Linux driver related changes.
>>
>>> Actually the issue is really on the TI's prueth and IEP driver whith
>>> fw_devlink=on. The IEP driver probe correctly with fw_devlink=permissive.
>>
>> Argh not again, the fw_devlink changes seem to be causing all kind of
>> issues. Any ideas what the issue here might be? It might be worth testing
>> with v6.0-rc7 too as it has a series of fixes to related issues.

I rebased the prueth + IEP series on v6.0-rc7 and indeed the driver probe
correctly but with some delay due deferred probe.

[    9.568817] remoteproc remoteproc0: 4b234000.pru is available
[    9.580505] remoteproc remoteproc1: 4b238000.pru is available
[    9.625701] remoteproc remoteproc2: 4b2b4000.pru is available
[    9.632110] remoteproc remoteproc3: 4b2b8000.pru is available
[    9.932952] prueth pruss1_eth: unable to get IEP
[   10.238098] prueth pruss1_eth: unable to get IEP
[   10.469421] prueth pruss1_eth: unable to get IEP
[   10.675445] prueth pruss1_eth: unable to get IEP
[   10.885589] prueth pruss1_eth: unable to get IEP
[   11.117126] prueth pruss1_eth: unable to get IEP
[   11.755462] prueth pruss1_eth: unable to get IEP
[   12.316192] prueth pruss1_eth: unable to get IEP
[   13.387969] prueth pruss1_eth: unable to get IEP
[   14.175750] prueth pruss1_eth: unable to get IEP
[   14.388671] prueth pruss1_eth: unable to get IEP
[   14.648406] prueth pruss1_eth: unable to get IEP
[   16.946716] prueth pruss1_eth: unable to get IEP
[   25.037414] remoteproc remoteproc1: powering up 4b238000.pru
[   25.056793] remoteproc remoteproc1: Booting fw image
ti-pruss/am57xx-pru1-prueth-fw.elf, size 6952
[   25.065856] remoteproc remoteproc1: unsupported resource 5
[   25.071472] remoteproc remoteproc1: remote processor 4b238000.pru is now up
[   25.157104] remoteproc remoteproc0: powering up 4b234000.pru
[   25.165527] remoteproc remoteproc0: Booting fw image
ti-pruss/am57xx-pru0-prueth-fw.elf, size 6920
[   25.174713] remoteproc remoteproc0: unsupported resource 5
[   25.180328] remoteproc remoteproc0: remote processor 4b234000.pru is now up
[   27.047607] prueth pruss1_eth eth4: Link is Up - 100Mbps/Full - flow control off


>>
> 
> Could you also try [1] from Puranjay's series? He did have to rework
> pru_rproc_get()/pru_rproc_put() a bit to fix few probe failures seen
> only on > 5.11 kernels.
> 
> [1] https://lore.kernel.org/netdev/20220406094358.7895-3-p-mohan@ti.com/
> 

Thank for the link, I actually had a look to this series and updated the pru
remote proc driver.

Note: The RFC series "PRUSS Remoteproc, Platform APIS, and Ethernet Driver"
doesn't include yet the prueth driver for AM57xx so I had to fixup this driver
when needed. It would be great if the support for the AM57xx can be included in
the next submission [1].

[1]
https://lore.kernel.org/linux-remoteproc/992019ad-5c58-d420-8a18-a82228f8e086@smile.fr/

Best regards,
Romain

Re: prueth: IEP driver doesn't probe anymore

[Tony Lindgren]

* Romain Naour <romain.naour@smile.fr> [220927 10:03]:
> I rebased the prueth + IEP series on v6.0-rc7 and indeed the driver probe
> correctly but with some delay due deferred probe.

OK great. FYI, see commit 0b3acd1cc022 ("Merge tag 'driver-core-6.0-rc4' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core") for details
on what got merged.

> [    9.568817] remoteproc remoteproc0: 4b234000.pru is available
> [    9.580505] remoteproc remoteproc1: 4b238000.pru is available
> [    9.625701] remoteproc remoteproc2: 4b2b4000.pru is available
> [    9.632110] remoteproc remoteproc3: 4b2b8000.pru is available
> [    9.932952] prueth pruss1_eth: unable to get IEP
> [   10.238098] prueth pruss1_eth: unable to get IEP
> [   10.469421] prueth pruss1_eth: unable to get IEP
> [   10.675445] prueth pruss1_eth: unable to get IEP
> [   10.885589] prueth pruss1_eth: unable to get IEP
> [   11.117126] prueth pruss1_eth: unable to get IEP
> [   11.755462] prueth pruss1_eth: unable to get IEP
> [   12.316192] prueth pruss1_eth: unable to get IEP
> [   13.387969] prueth pruss1_eth: unable to get IEP
> [   14.175750] prueth pruss1_eth: unable to get IEP
> [   14.388671] prueth pruss1_eth: unable to get IEP
> [   14.648406] prueth pruss1_eth: unable to get IEP
> [   16.946716] prueth pruss1_eth: unable to get IEP
> [   25.037414] remoteproc remoteproc1: powering up 4b238000.pru
> [   25.056793] remoteproc remoteproc1: Booting fw image
> ti-pruss/am57xx-pru1-prueth-fw.elf, size 6952
> [   25.065856] remoteproc remoteproc1: unsupported resource 5
> [   25.071472] remoteproc remoteproc1: remote processor 4b238000.pru is now up
> [   25.157104] remoteproc remoteproc0: powering up 4b234000.pru
> [   25.165527] remoteproc remoteproc0: Booting fw image
> ti-pruss/am57xx-pru0-prueth-fw.elf, size 6920
> [   25.174713] remoteproc remoteproc0: unsupported resource 5
> [   25.180328] remoteproc remoteproc0: remote processor 4b234000.pru is now up
> [   27.047607] prueth pruss1_eth eth4: Link is Up - 100Mbps/Full - flow control off

Seem like this is still not ideal with delayed and noisy deferred probe :(

Regards,

Tony