XHCI bringup on the Raspberry Pi 4

XHCI bringup on the Raspberry Pi 4

[Nicolas Saenz Julienne]

Hi All,
I'm working on enabling the VIA805 XCHI controller found on the new Raspberry
Pi 4. The controller sits behind a PCIe bus, which I've already implemented[1]
and will submit once the XCHI issues are resolved, as it's worthless otherwise.

The XHCI initialization gets stuck after issuing the fist 'enable slot'
command. I've been reviewing the whole init process and comparing it to Linux's
for days without finding anything fishy. DMA constraints, which are important
on the RPi are mantained, and on top of that, I can confirm DMA trough PCIe
works fine as I see two 'port status change' events available in the ring
before completly stalling. Also note that, unsurprisingly, the CRR bit in the
CRCR register (which marks the running state of the Command Ring state machine)
is never set after ringing the relevant doorbell.

I'm clueless at this point, I figure the VIA805 is sensitive to the ordering of
some of the operations we perform in u-boot, or worse, the timing. For example,
I tried replicanting Linux's behaviour with regard to 'port status change'
events, processing them before calling the 'enable slot' command. I also tried
to mimic Linux by enabling port-0's power (the USB3 port) before starting the
HC. Again, no luck.

I attached the usb/xhci debug output, any ideas on where to look will be
apreciated.

Regards,
Nicolas

[1] https://github.com/vianpl/u-boot/commits/brcm-pcie

-------------- next part --------------
A non-text attachment was scrubbed...
Name: u-boot.log
Type: text/x-log
Size: 8114 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200401/deecda40/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200401/deecda40/attachment.sig>

XHCI bringup on the Raspberry Pi 4

[Marek Vasut]

On 4/1/20 7:30 PM, Nicolas Saenz Julienne wrote:
> Hi All,

Hi,

> I'm working on enabling the VIA805 XCHI controller found on the new Raspberry
> Pi 4. The controller sits behind a PCIe bus, which I've already implemented[1]
> and will submit once the XCHI issues are resolved, as it's worthless otherwise.
> 
> The XHCI initialization gets stuck after issuing the fist 'enable slot'
> command. I've been reviewing the whole init process and comparing it to Linux's
> for days without finding anything fishy. DMA constraints, which are important
> on the RPi are mantained, and on top of that, I can confirm DMA trough PCIe
> works fine as I see two 'port status change' events available in the ring
> before completly stalling. Also note that, unsurprisingly, the CRR bit in the
> CRCR register (which marks the running state of the Command Ring state machine)
> is never set after ringing the relevant doorbell.
> 
> I'm clueless at this point, I figure the VIA805 is sensitive to the ordering of
> some of the operations we perform in u-boot, or worse, the timing. For example,
> I tried replicanting Linux's behaviour with regard to 'port status change'
> events, processing them before calling the 'enable slot' command. I also tried
> to mimic Linux by enabling port-0's power (the USB3 port) before starting the
> HC. Again, no luck.
> 
> I attached the usb/xhci debug output, any ideas on where to look will be
> apreciated.

Try disabling caches (dcache off ; icache off) and see if it magically
starts working. That comes to mind.

XHCI bringup on the Raspberry Pi 4

[Nicolas Saenz Julienne]

On Wed, 2020-04-01 at 20:50 +0200, Marek Vasut wrote:
> On 4/1/20 7:30 PM, Nicolas Saenz Julienne wrote:
> > Hi All,
> 
> Hi,
> 
> > I'm working on enabling the VIA805 XCHI controller found on the new
> > Raspberry
> > Pi 4. The controller sits behind a PCIe bus, which I've already
> > implemented[1]
> > and will submit once the XCHI issues are resolved, as it's worthless
> > otherwise.
> > 
> > The XHCI initialization gets stuck after issuing the fist 'enable slot'
> > command. I've been reviewing the whole init process and comparing it to
> > Linux's
> > for days without finding anything fishy. DMA constraints, which are
> > important
> > on the RPi are mantained, and on top of that, I can confirm DMA trough PCIe
> > works fine as I see two 'port status change' events available in the ring
> > before completly stalling. Also note that, unsurprisingly, the CRR bit in
> > the
> > CRCR register (which marks the running state of the Command Ring state
> > machine)
> > is never set after ringing the relevant doorbell.
> > 
> > I'm clueless at this point, I figure the VIA805 is sensitive to the ordering
> > of
> > some of the operations we perform in u-boot, or worse, the timing. For
> > example,
> > I tried replicanting Linux's behaviour with regard to 'port status change'
> > events, processing them before calling the 'enable slot' command. I also
> > tried
> > to mimic Linux by enabling port-0's power (the USB3 port) before starting
> > the
> > HC. Again, no luck.
> > 
> > I attached the usb/xhci debug output, any ideas on where to look will be
> > apreciated.
> 
> Try disabling caches (dcache off ; icache off) and see if it magically
> starts working. That comes to mind.

Same outcome :(

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200401/6dc475c4/attachment.sig>

XHCI bringup on the Raspberry Pi 4

[Marek Vasut]

On 4/1/20 10:13 PM, Nicolas Saenz Julienne wrote:
> On Wed, 2020-04-01 at 20:50 +0200, Marek Vasut wrote:
>> On 4/1/20 7:30 PM, Nicolas Saenz Julienne wrote:
>>> Hi All,
>>
>> Hi,
>>
>>> I'm working on enabling the VIA805 XCHI controller found on the new
>>> Raspberry
>>> Pi 4. The controller sits behind a PCIe bus, which I've already
>>> implemented[1]
>>> and will submit once the XCHI issues are resolved, as it's worthless
>>> otherwise.
>>>
>>> The XHCI initialization gets stuck after issuing the fist 'enable slot'
>>> command. I've been reviewing the whole init process and comparing it to
>>> Linux's
>>> for days without finding anything fishy. DMA constraints, which are
>>> important
>>> on the RPi are mantained, and on top of that, I can confirm DMA trough PCIe
>>> works fine as I see two 'port status change' events available in the ring
>>> before completly stalling. Also note that, unsurprisingly, the CRR bit in
>>> the
>>> CRCR register (which marks the running state of the Command Ring state
>>> machine)
>>> is never set after ringing the relevant doorbell.
>>>
>>> I'm clueless at this point, I figure the VIA805 is sensitive to the ordering
>>> of
>>> some of the operations we perform in u-boot, or worse, the timing. For
>>> example,
>>> I tried replicanting Linux's behaviour with regard to 'port status change'
>>> events, processing them before calling the 'enable slot' command. I also
>>> tried
>>> to mimic Linux by enabling port-0's power (the USB3 port) before starting
>>> the
>>> HC. Again, no luck.
>>>
>>> I attached the usb/xhci debug output, any ideas on where to look will be
>>> apreciated.
>>
>> Try disabling caches (dcache off ; icache off) and see if it magically
>> starts working. That comes to mind.
> 
> Same outcome :(

Hmmm, no idea then, maybe others have some.

I presume PCIe works correctly , right?

XHCI bringup on the Raspberry Pi 4

[Nicolas Saenz Julienne]

On Wed, 2020-04-01 at 22:14 +0200, Marek Vasut wrote:
> On 4/1/20 10:13 PM, Nicolas Saenz Julienne wrote:
> > On Wed, 2020-04-01 at 20:50 +0200, Marek Vasut wrote:
> > > On 4/1/20 7:30 PM, Nicolas Saenz Julienne wrote:
> > > > Hi All,
> > > 
> > > Hi,
> > > 
> > > > I'm working on enabling the VIA805 XCHI controller found on the new
> > > > Raspberry
> > > > Pi 4. The controller sits behind a PCIe bus, which I've already
> > > > implemented[1]
> > > > and will submit once the XCHI issues are resolved, as it's worthless
> > > > otherwise.
> > > > 
> > > > The XHCI initialization gets stuck after issuing the fist 'enable slot'
> > > > command. I've been reviewing the whole init process and comparing it to
> > > > Linux's
> > > > for days without finding anything fishy. DMA constraints, which are
> > > > important
> > > > on the RPi are mantained, and on top of that, I can confirm DMA trough
> > > > PCIe
> > > > works fine as I see two 'port status change' events available in the
> > > > ring
> > > > before completly stalling. Also note that, unsurprisingly, the CRR bit
> > > > in
> > > > the
> > > > CRCR register (which marks the running state of the Command Ring state
> > > > machine)
> > > > is never set after ringing the relevant doorbell.
> > > > 
> > > > I'm clueless at this point, I figure the VIA805 is sensitive to the
> > > > ordering
> > > > of
> > > > some of the operations we perform in u-boot, or worse, the timing. For
> > > > example,
> > > > I tried replicanting Linux's behaviour with regard to 'port status
> > > > change'
> > > > events, processing them before calling the 'enable slot' command. I also
> > > > tried
> > > > to mimic Linux by enabling port-0's power (the USB3 port) before
> > > > starting
> > > > the
> > > > HC. Again, no luck.
> > > > 
> > > > I attached the usb/xhci debug output, any ideas on where to look will be
> > > > apreciated.
> > > 
> > > Try disabling caches (dcache off ; icache off) and see if it magically
> > > starts working. That comes to mind.
> > 
> > Same outcome :(
> 
> Hmmm, no idea then, maybe others have some.
> 
> I presume PCIe works correctly , right?

I was also responsible for the Linux version of the PCIe driver so I know it
fairly well. I also double-checked all pcie-brcmstb's registers have the
expected values.

From the USB perspective, I'm both able to map XHCI's registers into memory and
get some output out of the event ring, which implies XHCI accessed memory
trough the bus. On top of that if something went wrong with PCI there is a
register that should point it out, Host System Error (HSE), which so far has
been silent.

Overall I'm confident PCIe is fine.

Regards,
Nicolas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200401/df562ad7/attachment.sig>

XHCI bringup on the Raspberry Pi 4

[Marek Vasut]

On 4/1/20 10:47 PM, Nicolas Saenz Julienne wrote:
> On Wed, 2020-04-01 at 22:14 +0200, Marek Vasut wrote:
>> On 4/1/20 10:13 PM, Nicolas Saenz Julienne wrote:
>>> On Wed, 2020-04-01 at 20:50 +0200, Marek Vasut wrote:
>>>> On 4/1/20 7:30 PM, Nicolas Saenz Julienne wrote:
>>>>> Hi All,
>>>>
>>>> Hi,
>>>>
>>>>> I'm working on enabling the VIA805 XCHI controller found on the new
>>>>> Raspberry
>>>>> Pi 4. The controller sits behind a PCIe bus, which I've already
>>>>> implemented[1]
>>>>> and will submit once the XCHI issues are resolved, as it's worthless
>>>>> otherwise.
>>>>>
>>>>> The XHCI initialization gets stuck after issuing the fist 'enable slot'
>>>>> command. I've been reviewing the whole init process and comparing it to
>>>>> Linux's
>>>>> for days without finding anything fishy. DMA constraints, which are
>>>>> important
>>>>> on the RPi are mantained, and on top of that, I can confirm DMA trough
>>>>> PCIe
>>>>> works fine as I see two 'port status change' events available in the
>>>>> ring
>>>>> before completly stalling. Also note that, unsurprisingly, the CRR bit
>>>>> in
>>>>> the
>>>>> CRCR register (which marks the running state of the Command Ring state
>>>>> machine)
>>>>> is never set after ringing the relevant doorbell.
>>>>>
>>>>> I'm clueless at this point, I figure the VIA805 is sensitive to the
>>>>> ordering
>>>>> of
>>>>> some of the operations we perform in u-boot, or worse, the timing. For
>>>>> example,
>>>>> I tried replicanting Linux's behaviour with regard to 'port status
>>>>> change'
>>>>> events, processing them before calling the 'enable slot' command. I also
>>>>> tried
>>>>> to mimic Linux by enabling port-0's power (the USB3 port) before
>>>>> starting
>>>>> the
>>>>> HC. Again, no luck.
>>>>>
>>>>> I attached the usb/xhci debug output, any ideas on where to look will be
>>>>> apreciated.
>>>>
>>>> Try disabling caches (dcache off ; icache off) and see if it magically
>>>> starts working. That comes to mind.
>>>
>>> Same outcome :(
>>
>> Hmmm, no idea then, maybe others have some.
>>
>> I presume PCIe works correctly , right?
> 
> I was also responsible for the Linux version of the PCIe driver so I know it
> fairly well. I also double-checked all pcie-brcmstb's registers have the
> expected values.
> 
> From the USB perspective, I'm both able to map XHCI's registers into memory and
> get some output out of the event ring, which implies XHCI accessed memory
> trough the bus. On top of that if something went wrong with PCI there is a
> register that should point it out, Host System Error (HSE), which so far has
> been silent.
> 
> Overall I'm confident PCIe is fine.

OK, then I hope Bin can help here, he's the xHCI expert.

XHCI bringup on the Raspberry Pi 4

[Bin Meng]

Hi Nicolas,

On Thu, Apr 2, 2020 at 1:30 AM Nicolas Saenz Julienne
<nsaenzjulienne@suse.de> wrote:
>
> Hi All,
> I'm working on enabling the VIA805 XCHI controller found on the new Raspberry
> Pi 4. The controller sits behind a PCIe bus, which I've already implemented[1]
> and will submit once the XCHI issues are resolved, as it's worthless otherwise.
>
> The XHCI initialization gets stuck after issuing the fist 'enable slot'
> command. I've been reviewing the whole init process and comparing it to Linux's
> for days without finding anything fishy. DMA constraints, which are important
> on the RPi are mantained, and on top of that, I can confirm DMA trough PCIe
> works fine as I see two 'port status change' events available in the ring
> before completly stalling. Also note that, unsurprisingly, the CRR bit in the
> CRCR register (which marks the running state of the Command Ring state machine)
> is never set after ringing the relevant doorbell.

This is probably caused by the required structure setup by U-Boot is
viewed as buggy by the xHCI controller, hence there is no response to
the first "enable slot" command.

Could you please compare all the data structures, with the one set up
by the Linux kernel?

BTW: does Linux kernel xHCI driver work on this RPI 4 board?

>
> I'm clueless at this point, I figure the VIA805 is sensitive to the ordering of
> some of the operations we perform in u-boot, or worse, the timing. For example,
> I tried replicanting Linux's behaviour with regard to 'port status change'
> events, processing them before calling the 'enable slot' command. I also tried
> to mimic Linux by enabling port-0's power (the USB3 port) before starting the
> HC. Again, no luck.
>
> I attached the usb/xhci debug output, any ideas on where to look will be
> apreciated.
>

Regards,
Bin

XHCI bringup on the Raspberry Pi 4

[Peter Robinson]

> > I'm working on enabling the VIA805 XCHI controller found on the new Raspberry
> > Pi 4. The controller sits behind a PCIe bus, which I've already implemented[1]
> > and will submit once the XCHI issues are resolved, as it's worthless otherwise.
> >
> > The XHCI initialization gets stuck after issuing the fist 'enable slot'
> > command. I've been reviewing the whole init process and comparing it to Linux's
> > for days without finding anything fishy. DMA constraints, which are important
> > on the RPi are mantained, and on top of that, I can confirm DMA trough PCIe
> > works fine as I see two 'port status change' events available in the ring
> > before completly stalling. Also note that, unsurprisingly, the CRR bit in the
> > CRCR register (which marks the running state of the Command Ring state machine)
> > is never set after ringing the relevant doorbell.
>
> This is probably caused by the required structure setup by U-Boot is
> viewed as buggy by the xHCI controller, hence there is no response to
> the first "enable slot" command.
>
> Could you please compare all the data structures, with the one set up
> by the Linux kernel?
>
> BTW: does Linux kernel xHCI driver work on this RPI 4 board?

Yes, it uses the standard xhci-hcd controller driver, it's a fairly
vanilla VIA controller.

XHCI bringup on the Raspberry Pi 4

[Nicolas Saenz Julienne]

Hi Bin,
sorry for the late reply but I was off for easter.

On Fri, 2020-04-03 at 13:54 +0800, Bin Meng wrote:
> Hi Nicolas,
> This is probably caused by the required structure setup by U-Boot is
> viewed as buggy by the xHCI controller, hence there is no response to
> the first "enable slot" command.

Yes, that's my impression too. That said I can't seem to find the faulty
config.

Since I wasn't all that clear on my previous mail, this is a VIA805 chip,
directly soldered on the board and interfaced with trough RPi4's PCIe bus (the
PCIe bus isn't exposed, so it's the only user). The Linux version of this works
fine, and is already supported upstream.

> Could you please compare all the data structures, with the one set up
> by the Linux kernel?

I've attached some logs comparing u-boot's and Linux's view of the registers. I
dumped them before and after calling the 'enable slot' command. I left all the
debuging information just in case it's helpful and also attached some patches
to show my changes.

Note that I thoughfully compared them myself. The only odd thing I found was
"USBSTS: 0x10" (Port Detect Change is set), which isn't the case in Linux. I
tried to clear it at different places (before and after changing the individual
port status), without effect.

That's all. If the data I sent is a little bit too dense, please tell me and
I'll remove all the less important stuff :).

Regards,
Nicolas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: linux.log
Type: text/x-log
Size: 33624 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200413/4de8e04c/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: linux.patch
Type: text/x-patch
Size: 3435 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200413/4de8e04c/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: u-boot.log
Type: text/x-log
Size: 14260 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200413/4de8e04c/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: u-boot.patch
Type: text/x-patch
Size: 3895 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200413/4de8e04c/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200413/4de8e04c/attachment.sig>

XHCI bringup on the Raspberry Pi 4

[Bin Meng]

Hi Nicolas,

On Mon, Apr 13, 2020 at 11:45 PM Nicolas Saenz Julienne
<nsaenzjulienne@suse.de> wrote:
>
> Hi Bin,
> sorry for the late reply but I was off for easter.
>
> On Fri, 2020-04-03 at 13:54 +0800, Bin Meng wrote:
> > Hi Nicolas,
> > This is probably caused by the required structure setup by U-Boot is
> > viewed as buggy by the xHCI controller, hence there is no response to
> > the first "enable slot" command.
>
> Yes, that's my impression too. That said I can't seem to find the faulty
> config.
>
> Since I wasn't all that clear on my previous mail, this is a VIA805 chip,
> directly soldered on the board and interfaced with trough RPi4's PCIe bus (the
> PCIe bus isn't exposed, so it's the only user). The Linux version of this works
> fine, and is already supported upstream.
>
> > Could you please compare all the data structures, with the one set up
> > by the Linux kernel?
>
> I've attached some logs comparing u-boot's and Linux's view of the registers. I
> dumped them before and after calling the 'enable slot' command. I left all the
> debuging information just in case it's helpful and also attached some patches
> to show my changes.
>
> Note that I thoughfully compared them myself. The only odd thing I found was
> "USBSTS: 0x10" (Port Detect Change is set), which isn't the case in Linux. I
> tried to clear it at different places (before and after changing the individual
> port status), without effect.
>
> That's all. If the data I sent is a little bit too dense, please tell me and
> I'll remove all the less important stuff :).
>

The following patch should fix this issue you were seeing. Please try.
http://patchwork.ozlabs.org/project/uboot/patch/20200421165059.19394-2-s.nawrocki at samsung.com/

Regards,
Bin