[U-Boot] Jetson TX2 hangs during dhcp boot

[U-Boot] Jetson TX2 hangs during dhcp boot

[Stephen Warren]

On 6/14/19 11:09 AM, Mian Yousaf Kaukab wrote:
> Hi,
> I am seeing a hang with dhcp boot on Jetson TX2. Its running firmware from
> L4T 32.1 with U-Boot from master branch (c2ea87883e) with [1] applied.
> 
> I use following command sequence at u-boot prompt to boot:
> 
> Tegra186 (P2771-0000-500) # setenv ftdfile tegra186-p2771-0000.dtb
> Tegra186 (P2771-0000-500) # setenv boot_targets dhcp
> Tegra186 (P2771-0000-500) # boot
> 
> Last thing I see is the following error from TF-A and the board hangs:

What is TF-A?

> U-Boot 2019.07-rc4-00137-g6fefd9c475 (Jun 14 2019 - 16:27:34 +0200)
...
> TFTP from server xx.xx.xx.xx; our IP address is xx.xx.xx.xx
> Filename 'aarch64/grub.efi'.
> Load address: 0x80280000
...
> MMC: no card present
> Scanning disk sdhci at 3400000.blk...
> Disk sdhci at 3400000.blk not ready
> Scanning disk sdhci at 3460000.blk...
> Found 32 disks
> copying carveout for /host1x at 13e00000/display-hub at 15200000/display at 15200000...
> copying carveout for /host1x at 13e00000/display-hub at 15200000/display at 15210000...
> copying carveout for /host1x at 13e00000/display-hub at 15200000/display at 15220000...
> ERROR:   ARI request timed out: req 89 on CPU 4
> ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries != 0U

That file doesn't exist in U-Boot; I guess it's part of grub.efi? So, 
this isn't a hang in U-Boot at all. You'd have to get support for that 
from the authors of Grub I guess.

[U-Boot] Jetson TX2 hangs during dhcp boot

[Mian Yousaf Kaukab]

Hi,
I am seeing a hang with dhcp boot on Jetson TX2. Its running firmware from
L4T 32.1 with U-Boot from master branch (c2ea87883e) with [1] applied.

I use following command sequence at u-boot prompt to boot:

Tegra186 (P2771-0000-500) # setenv ftdfile tegra186-p2771-0000.dtb
Tegra186 (P2771-0000-500) # setenv boot_targets dhcp
Tegra186 (P2771-0000-500) # boot

Last thing I see is the following error from TF-A and the board hangs:

ERROR:   ARI request timed out: req 89 on CPU 4
ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries != 0U

Here is the complete log:

U-Boot 2019.07-rc4-00137-g6fefd9c475 (Jun 14 2019 - 16:27:34 +0200)

TEGRA186
Model: NVIDIA P2771-0000-500
Board: NVIDIA P2771-0000
DRAM:  7.8 GiB
MMC:   sdhci at 3400000: 1, sdhci at 3460000: 0
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet at 2490000
Hit any key to stop autoboot:  0 
Tegra186 (P2771-0000-500) # setenv boot_targets dhcp
Tegra186 (P2771-0000-500) # boot
starting USB...
No working controllers found
maximum number of regions parsed, aborting
ethernet at 2490000 Waiting for PHY auto negotiation to complete...... done
BOOTP broadcast 1
BOOTP broadcast 2
BOOTP broadcast 3
DHCP client bound to address xx.xx.xx.xx (1259 ms)
Using ethernet at 2490000 device
TFTP from server xx.xx.xx.xx; our IP address is xx.xx.xx.xx
Filename 'boot.scr.uimg'.
Load address: 0x82600000
Loading: *
TFTP error: 'File not found' (1)
Not retrying...
ethernet at 2490000 Waiting for PHY auto negotiation to complete........ done
BOOTP broadcast 1
DHCP client bound to address xx.xx.xx.xx (5 ms)
Using ethernet at 2490000 device
TFTP from server xx.xx.xx.xx; our IP address is xx.xx.xx.xx
Filename 'aarch64/grub.efi'.
Load address: 0x80280000
Loading: T #######T ##########################################################
	 #################################################################
	 ##
	 181.6 KiB/s
done
Bytes transferred = 1936896 (1d8e00 hex)
ethernet at 2490000 Waiting for PHY auto negotiation to complete........ done
Using ethernet at 2490000 device
TFTP from server xx.xx.xx.xx; our IP address is xx.xx.xx.xx
Filename 'dtb/tegra186-p2771-0000.dtb'.
Load address: 0x82400000
Loading: ###
	 2.6 MiB/s
done
Bytes transferred = 30196 (75f4 hex)
MMC: no card present
Scanning disk sdhci at 3400000.blk...
Disk sdhci at 3400000.blk not ready
Scanning disk sdhci at 3460000.blk...
Found 32 disks
copying carveout for /host1x at 13e00000/display-hub at 15200000/display at 15200000...
copying carveout for /host1x at 13e00000/display-hub at 15200000/display at 15210000...
copying carveout for /host1x at 13e00000/display-hub at 15200000/display at 15220000...
ERROR:   ARI request timed out: req 89 on CPU 4
ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries != 0U

It turned out that the last call to TF-A was from __asm_invalidate_l3_icache().
Following is the call trace: 

efi_load_image()
    efi_load_pe()
        invalidate_icache_all()
            __asm_invalidate_l3_icache() 
 
__asm_invalidate_l3_icache() has been called a couple of times before this point
without any issues.

If I boot the same EFI binary manually with bootefi it works fine. I can see that
__asm_invalidate_l3_icache() returns without the hang. Following is the working
command sequence:

Tegra186 (P2771-0000-500) # dhcp
Tegra186 (P2771-0000-500) # tftpboot $fdt_addr_r dtb/tegra186-p2771-0000.dtb
Tegra186 (P2771-0000-500) # tftpboot $kernel_addr_r aarch64/grub.efi
Tegra186 (P2771-0000-500) # bootefi $kernel_addr_r $fdt_addr_r

This issue is 100% reproducible. 

Any thoughts on what could be wrong or how to debug it further?

Thanks,
Yousaf

[1]: https://patchwork.ozlabs.org/patch/1115052/

[U-Boot] Jetson TX2 hangs during dhcp boot

[Andreas Färber]

Am 14.06.19 um 18:41 schrieb Stephen Warren:
> On 6/14/19 11:09 AM, Mian Yousaf Kaukab wrote:
>> Hi,
>> I am seeing a hang with dhcp boot on Jetson TX2. Its running firmware
>> from
>> L4T 32.1 with U-Boot from master branch (c2ea87883e) with [1] applied.
>>
>> I use following command sequence at u-boot prompt to boot:
>>
>> Tegra186 (P2771-0000-500) # setenv ftdfile tegra186-p2771-0000.dtb
>> Tegra186 (P2771-0000-500) # setenv boot_targets dhcp
>> Tegra186 (P2771-0000-500) # boot
>>
>> Last thing I see is the following error from TF-A and the board hangs:
> 
> What is TF-A?

The artist formerly known as Arm Trusted Firmware.

>> U-Boot 2019.07-rc4-00137-g6fefd9c475 (Jun 14 2019 - 16:27:34 +0200)
> ...
>> TFTP from server xx.xx.xx.xx; our IP address is xx.xx.xx.xx
>> Filename 'aarch64/grub.efi'.
>> Load address: 0x80280000
> ...
>> MMC: no card present
>> Scanning disk sdhci at 3400000.blk...
>> Disk sdhci at 3400000.blk not ready
>> Scanning disk sdhci at 3460000.blk...
>> Found 32 disks
>> copying carveout for
>> /host1x at 13e00000/display-hub at 15200000/display at 15200000...
>> copying carveout for
>> /host1x at 13e00000/display-hub at 15200000/display at 15210000...
>> copying carveout for
>> /host1x at 13e00000/display-hub at 15200000/display at 15220000...
>> ERROR:   ARI request timed out: req 89 on CPU 4
>> ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries
>> != 0U
> 
> That file doesn't exist in U-Boot; I guess it's part of grub.efi?

Looks rather like a TF-A path to me, CC'ing Varun.

As Yousaf says above, it's the TF-A you guys ship with latest R32.1.0.

Maybe the tftpboot operation is overwriting memory used by your TF-A, or
differing memory usage is uncovering a missing U-Boot EFI reservation?

(I recently wondered whether TX1 may be missing some, as I needed
multiple tries to boot the kernel successfully there - TF-A 2.1 didn't
work there, so some more investigations TBD. R28.3.0 still wasn't
booting U-Boot ~2019.04, I needed R24.2.2 still as you once suggested.)

Regards,
Andreas

-- 
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)

[U-Boot] Jetson TX2 hangs during dhcp boot

[Varun Wadekar]

>> ERROR:   ARI request timed out: req 89 on CPU 4
>> ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries 
>> != 0U

This time out means that TEGRA_ARI_ROC_FLUSH_CACHE_TRBITS ARI call timed out. Looks like the firmware responsible for handling this ARI call is stuck or overloaded and not servicing any calls.

-----Original Message-----
From: Andreas Färber <afaerber@suse.de> 
Sent: Friday, June 14, 2019 10:10 AM
To: Stephen Warren <swarren@wwwdotorg.org>; Mian Yousaf Kaukab <ykaukab@suse.de>
Cc: u-boot at lists.denx.de; Stephen Warren <swarren@nvidia.com>; Tom Warren <TWarren@nvidia.com>; Varun Wadekar <vwadekar@nvidia.com>
Subject: Re: [U-Boot] Jetson TX2 hangs during dhcp boot

Am 14.06.19 um 18:41 schrieb Stephen Warren:
> On 6/14/19 11:09 AM, Mian Yousaf Kaukab wrote:
>> Hi,
>> I am seeing a hang with dhcp boot on Jetson TX2. Its running firmware 
>> from L4T 32.1 with U-Boot from master branch (c2ea87883e) with [1] 
>> applied.
>>
>> I use following command sequence at u-boot prompt to boot:
>>
>> Tegra186 (P2771-0000-500) # setenv ftdfile tegra186-p2771-0000.dtb
>> Tegra186 (P2771-0000-500) # setenv boot_targets dhcp
>> Tegra186 (P2771-0000-500) # boot
>>
>> Last thing I see is the following error from TF-A and the board hangs:
> 
> What is TF-A?

The artist formerly known as Arm Trusted Firmware.

>> U-Boot 2019.07-rc4-00137-g6fefd9c475 (Jun 14 2019 - 16:27:34 +0200)
> ...
>> TFTP from server xx.xx.xx.xx; our IP address is xx.xx.xx.xx Filename 
>> 'aarch64/grub.efi'.
>> Load address: 0x80280000
> ...
>> MMC: no card present
>> Scanning disk sdhci at 3400000.blk...
>> Disk sdhci at 3400000.blk not ready
>> Scanning disk sdhci at 3460000.blk...
>> Found 32 disks
>> copying carveout for
>> /host1x at 13e00000/display-hub at 15200000/display at 15200000...
>> copying carveout for
>> /host1x at 13e00000/display-hub at 15200000/display at 15210000...
>> copying carveout for
>> /host1x at 13e00000/display-hub at 15200000/display at 15220000...
>> ERROR:   ARI request timed out: req 89 on CPU 4
>> ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries 
>> != 0U
> 
> That file doesn't exist in U-Boot; I guess it's part of grub.efi?

Looks rather like a TF-A path to me, CC'ing Varun.

As Yousaf says above, it's the TF-A you guys ship with latest R32.1.0.

Maybe the tftpboot operation is overwriting memory used by your TF-A, or differing memory usage is uncovering a missing U-Boot EFI reservation?

(I recently wondered whether TX1 may be missing some, as I needed multiple tries to boot the kernel successfully there - TF-A 2.1 didn't work there, so some more investigations TBD. R28.3.0 still wasn't booting U-Boot ~2019.04, I needed R24.2.2 still as you once suggested.)

Regards,
Andreas

--
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

[U-Boot] Jetson TX2 hangs during dhcp boot

[Stephen Warren]

On 6/14/19 11:09 AM, Andreas Färber wrote:
> Am 14.06.19 um 18:41 schrieb Stephen Warren:
>> On 6/14/19 11:09 AM, Mian Yousaf Kaukab wrote:
>>> Hi,
>>> I am seeing a hang with dhcp boot on Jetson TX2. Its running firmware
>>> from
>>> L4T 32.1 with U-Boot from master branch (c2ea87883e) with [1] applied.
>>>
>>> I use following command sequence at u-boot prompt to boot:
>>>
>>> Tegra186 (P2771-0000-500) # setenv ftdfile tegra186-p2771-0000.dtb
>>> Tegra186 (P2771-0000-500) # setenv boot_targets dhcp
>>> Tegra186 (P2771-0000-500) # boot
>>>
>>> Last thing I see is the following error from TF-A and the board hangs:
>>
>> What is TF-A?
> 
> The artist formerly known as Arm Trusted Firmware.

Ah OK. I had only ever heard that called ATF before.

>>> U-Boot 2019.07-rc4-00137-g6fefd9c475 (Jun 14 2019 - 16:27:34 +0200)
>> ...
>>> TFTP from server xx.xx.xx.xx; our IP address is xx.xx.xx.xx
>>> Filename 'aarch64/grub.efi'.
>>> Load address: 0x80280000
>> ...
>>> MMC: no card present
>>> Scanning disk sdhci at 3400000.blk...
>>> Disk sdhci at 3400000.blk not ready
>>> Scanning disk sdhci at 3460000.blk...
>>> Found 32 disks
>>> copying carveout for
>>> /host1x at 13e00000/display-hub at 15200000/display at 15200000...
>>> copying carveout for
>>> /host1x at 13e00000/display-hub at 15200000/display at 15210000...
>>> copying carveout for
>>> /host1x at 13e00000/display-hub at 15200000/display at 15220000...
>>> ERROR:   ARI request timed out: req 89 on CPU 4
>>> ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries
>>> != 0U
>>
>> That file doesn't exist in U-Boot; I guess it's part of grub.efi?
> 
> Looks rather like a TF-A path to me, CC'ing Varun.

Ah yes, I do see that path in our ATF code.

> As Yousaf says above, it's the TF-A you guys ship with latest R32.1.0.
> 
> Maybe the tftpboot operation is overwriting memory used by your TF-A, or

That's pretty unlikely since ATF code/data is in a protected memory 
region that non-secure software can't access.

> differing memory usage is uncovering a missing U-Boot EFI reservation?

Perhaps. I know almost nothing about U-Boot EFI though.

> (I recently wondered whether TX1 may be missing some, as I needed
> multiple tries to boot the kernel successfully there - TF-A 2.1 didn't
> work there, so some more investigations TBD. R28.3.0 still wasn't
> booting U-Boot ~2019.04, I needed R24.2.2 still as you once suggested.)

Hmm. Are you building your own copy of ATF (you mentioned 2.1; I can't 
recall what version we ship)?

FWIW, my usptream U-Boot test system uses L4T r28.1 to test Jetson TX1 
and TX2. It does test DHCP/TFTP, but not EFI or booting a kernel.

Can you provide complete instructions on how to reproduce this issue 
(e.g. which components in L4T you replaced, how to build those 
replacements, how to get/build/use EFI, any config/script/... files you 
used from U-Boot/EFI to trigger the issue above)?

Your best bet is likely to post all this information to the NVIDIA 
Jetson forum so our support people can track the process. I don't expect 
to have much time personally to look at EFI stuff on TX2 right now. Sorry.

[U-Boot] Jetson TX2 hangs during dhcp boot

[Andreas Färber]

Am 14.06.19 um 19:26 schrieb Varun Wadekar:
>>> ERROR:   ARI request timed out: req 89 on CPU 4
>>> ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries 
>>> != 0U
> 
> This time out means that TEGRA_ARI_ROC_FLUSH_CACHE_TRBITS ARI call timed out. Looks like the firmware responsible for handling this ARI call is stuck or overloaded and not servicing any calls.

Does this ARI firmware sit in the Arm cores' DRAM? If so, can you share
which memory region it needs?

Same question for BL31 - is that TZDRAM_BASE? If so, where to find the
size? The TF-A docs are not so telling of those layout details.
https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/plat/nvidia-tegra.rst

U-Boot only reserves any memrsv entries of the FDT, not other memory
nodes such as within your sysram at 30000000. Those may need C code in
U-Boot to reserve them from usage by UEFI apps such as GRUB and Linux.

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/arch/arm64/boot/dts/nvidia/tegra186.dtsi

Regards,
Andreas

-- 
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)

[U-Boot] Jetson TX2 hangs during dhcp boot

[Martin Husemann]

On Sun, Jun 16, 2019 at 09:27:12PM +0200, Mian Yousaf Kaukab wrote:
> > Hmm. Are you building your own copy of ATF (you mentioned 2.1; I can't
> > recall what version we ship)?
> No, I am using the one shipped with R32.1.

That did not work at all for me with mainline u-boot (u-boot would hang
early), so I went back to some older release (28.?) and all was fine
again.

Martin

[U-Boot] Jetson TX2 hangs during dhcp boot

[Andreas Färber]

Am 16.06.19 um 21:27 schrieb Mian Yousaf Kaukab:
> On Fri, Jun 14, 2019 at 11:40:49AM -0600, Stephen Warren wrote:
>> On 6/14/19 11:09 AM, Andreas Färber wrote:
>>> Am 14.06.19 um 18:41 schrieb Stephen Warren:
>>>> On 6/14/19 11:09 AM, Mian Yousaf Kaukab wrote:
>>>>> Hi,
>>>>> I am seeing a hang with dhcp boot on Jetson TX2. Its running firmware
>>>>> from
>>>>> L4T 32.1 with U-Boot from master branch (c2ea87883e) with [1] applied.
>>>>>
>>>>> I use following command sequence at u-boot prompt to boot:
>>>>>
>>>>> Tegra186 (P2771-0000-500) # setenv ftdfile tegra186-p2771-0000.dtb
>>>>> Tegra186 (P2771-0000-500) # setenv boot_targets dhcp
>>>>> Tegra186 (P2771-0000-500) # boot
>>>>>
>>>>> Last thing I see is the following error from TF-A and the board hangs:
[...]
>>>>> U-Boot 2019.07-rc4-00137-g6fefd9c475 (Jun 14 2019 - 16:27:34 +0200)
>>>> ...
>>>>> TFTP from server xx.xx.xx.xx; our IP address is xx.xx.xx.xx
>>>>> Filename 'aarch64/grub.efi'.
>>>>> Load address: 0x80280000
>>>> ...
>>>>> MMC: no card present
>>>>> Scanning disk sdhci at 3400000.blk...
>>>>> Disk sdhci at 3400000.blk not ready
>>>>> Scanning disk sdhci at 3460000.blk...
>>>>> Found 32 disks
>>>>> copying carveout for
>>>>> /host1x at 13e00000/display-hub at 15200000/display at 15200000...
>>>>> copying carveout for
>>>>> /host1x at 13e00000/display-hub at 15200000/display at 15210000...
>>>>> copying carveout for
>>>>> /host1x at 13e00000/display-hub at 15200000/display at 15220000...
>>>>> ERROR:   ARI request timed out: req 89 on CPU 4
>>>>> ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries
>>>>> != 0U
>>>>
>>>> That file doesn't exist in U-Boot; I guess it's part of grub.efi?
>>>
>>> Looks rather like a TF-A path to me, CC'ing Varun.
>>
>> Ah yes, I do see that path in our ATF code.
>>
>>> As Yousaf says above, it's the TF-A you guys ship with latest R32.1.0.

!!!

>>>
>>> Maybe the tftpboot operation is overwriting memory used by your TF-A, or
>>
>> That's pretty unlikely since ATF code/data is in a protected memory region
>> that non-secure software can't access.

Good point - I was playing with the rpi3 TF-A target, which does have
everything in unprotected DRAM.

>>
>>> differing memory usage is uncovering a missing U-Boot EFI reservation?
>>
>> Perhaps. I know almost nothing about U-Boot EFI though.
>>
>>> (I recently wondered whether TX1 may be missing some, as I needed
>>> multiple tries to boot the kernel successfully there - TF-A 2.1 didn't
>>> work there, so some more investigations TBD. R28.3.0 still wasn't
>>> booting U-Boot ~2019.04, I needed R24.2.2 still as you once suggested.)
>>
>> Hmm. Are you building your own copy of ATF (you mentioned 2.1; I can't
>> recall what version we ship)?
> No, I am using the one shipped with R32.1.

Careful: Yousaf is talking about his TX2 setup, whereas I am talking
about my TX1 setup. His answer does not apply to my quoted TX1 section.

Yes, for TX1 I am using an OBS-built upstream TF-A v2.1 / v1.5.
Will check Yousaf's LINUX_KERNEL_IMAGE_HEADER patch helps on TX1.

Let's split the two if they're not connected and focus on TX2 here.

Regards,
Andreas

-- 
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)

[U-Boot] Jetson TX2 hangs during dhcp boot

[Mian Yousaf Kaukab]

On Fri, Jun 14, 2019 at 11:40:49AM -0600, Stephen Warren wrote:
> On 6/14/19 11:09 AM, Andreas Färber wrote:
> > Am 14.06.19 um 18:41 schrieb Stephen Warren:
> > > On 6/14/19 11:09 AM, Mian Yousaf Kaukab wrote:
> > > > Hi,
> > > > I am seeing a hang with dhcp boot on Jetson TX2. Its running firmware
> > > > from
> > > > L4T 32.1 with U-Boot from master branch (c2ea87883e) with [1] applied.
> > > > 
> > > > I use following command sequence at u-boot prompt to boot:
> > > > 
> > > > Tegra186 (P2771-0000-500) # setenv ftdfile tegra186-p2771-0000.dtb
> > > > Tegra186 (P2771-0000-500) # setenv boot_targets dhcp
> > > > Tegra186 (P2771-0000-500) # boot
> > > > 
> > > > Last thing I see is the following error from TF-A and the board hangs:
> > > 
> > > What is TF-A?
> > 
> > The artist formerly known as Arm Trusted Firmware.
> 
> Ah OK. I had only ever heard that called ATF before.
4def07d5 Update Arm TF references to TF-A
> 
> > > > U-Boot 2019.07-rc4-00137-g6fefd9c475 (Jun 14 2019 - 16:27:34 +0200)
> > > ...
> > > > TFTP from server xx.xx.xx.xx; our IP address is xx.xx.xx.xx
> > > > Filename 'aarch64/grub.efi'.
> > > > Load address: 0x80280000
> > > ...
> > > > MMC: no card present
> > > > Scanning disk sdhci at 3400000.blk...
> > > > Disk sdhci at 3400000.blk not ready
> > > > Scanning disk sdhci at 3460000.blk...
> > > > Found 32 disks
> > > > copying carveout for
> > > > /host1x at 13e00000/display-hub at 15200000/display at 15200000...
> > > > copying carveout for
> > > > /host1x at 13e00000/display-hub at 15200000/display at 15210000...
> > > > copying carveout for
> > > > /host1x at 13e00000/display-hub at 15200000/display at 15220000...
> > > > ERROR:   ARI request timed out: req 89 on CPU 4
> > > > ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries
> > > > != 0U
> > > 
> > > That file doesn't exist in U-Boot; I guess it's part of grub.efi?
> > 
> > Looks rather like a TF-A path to me, CC'ing Varun.
> 
> Ah yes, I do see that path in our ATF code.
> 
> > As Yousaf says above, it's the TF-A you guys ship with latest R32.1.0.
> > 
> > Maybe the tftpboot operation is overwriting memory used by your TF-A, or
> 
> That's pretty unlikely since ATF code/data is in a protected memory region
> that non-secure software can't access.
> 
> > differing memory usage is uncovering a missing U-Boot EFI reservation?
> 
> Perhaps. I know almost nothing about U-Boot EFI though.
> 
> > (I recently wondered whether TX1 may be missing some, as I needed
> > multiple tries to boot the kernel successfully there - TF-A 2.1 didn't
> > work there, so some more investigations TBD. R28.3.0 still wasn't
> > booting U-Boot ~2019.04, I needed R24.2.2 still as you once suggested.)
> 
> Hmm. Are you building your own copy of ATF (you mentioned 2.1; I can't
> recall what version we ship)?
No, I am using the one shipped with R32.1.
> 
> FWIW, my usptream U-Boot test system uses L4T r28.1 to test Jetson TX1 and
> TX2. It does test DHCP/TFTP, but not EFI or booting a kernel.
> 
> Can you provide complete instructions on how to reproduce this issue (e.g.
> which components in L4T you replaced, how to build those replacements, how
> to get/build/use EFI, any config/script/... files you used from U-Boot/EFI
> to trigger the issue above)?
As shown in the u-boot log, I am using dhcp boot.
setenv boot_targets dhcp
boot

See if the instructions on the following link to setup the server works for you:
https://www.suse.com/documentation/suse-best-practices/singlehtml/sbp-multi-pxe-install/sbp-multi-pxe-install.html
and then populate it with Tumbleweed installation files from following:
http://download.opensuse.org/ports/aarch64/tumbleweed/iso/
Grub EFI binary can be found in EFI/BOOT/.

I am only replacing u-boot with the upstream u-boot using following
command:
./flash.sh -k kernel jetson-tx2 mmcblk0p1

Alternatively, I can do the testing this week if you provide me debug
builds.
> 
> Your best bet is likely to post all this information to the NVIDIA Jetson
> forum so our support people can track the process. I don't expect to have
> much time personally to look at EFI stuff on TX2 right now. Sorry.

[U-Boot] Jetson TX2 hangs during dhcp boot

[Mian Yousaf Kaukab]

On Fri, Jun 14, 2019 at 05:26:05PM +0000, Varun Wadekar wrote:
> >> ERROR:   ARI request timed out: req 89 on CPU 4
> >> ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries 
> >> != 0U
> 
> This time out means that TEGRA_ARI_ROC_FLUSH_CACHE_TRBITS ARI call timed out. Looks like the firmware responsible for handling this ARI call is stuck or overloaded and not servicing any calls.
Any thoughts on how to debug it?

I couldn't get upstream TF-A to work. Here is how I am building it:
make CROSS_COMPILE=aarch64-linux-gnu- PLAT=tegra TARGET_SOC=t186 DEBUG=1 all

then I build tos.img as following:
./gen_tos_part_img.py bl31.bin JetPack_4.2_Linux_P3310/Linux_for_Tegra/bootloader/tos.img

(try to)flash as following:
./flash.sh -k secure-os jetson-tx2 mmcblk0p1

However, I can't get flashing to work as soon as tos.img from R32.1 is
replaced with my build. Is upstream TF-A supposed to work with R32.1?

Alternatively, I can do the testing this week if you provide me any TF-A debug builds.

Thanks,
Yousaf

[U-Boot] Jetson TX1 boot problems

[Andreas Färber]

Am 14.06.19 um 19:40 schrieb Stephen Warren:
> On 6/14/19 11:09 AM, Andreas Färber wrote:
>> (I recently wondered whether TX1 may be missing some, as I needed
>> multiple tries to boot the kernel successfully there - TF-A 2.1 didn't
>> work there, so some more investigations TBD. R28.3.0 still wasn't
>> booting U-Boot ~2019.04, I needed R24.2.2 still as you once suggested.)
> 
> Hmm. Are you building your own copy of ATF (you mentioned 2.1; I can't
> recall what version we ship)?

v1.2

> FWIW, my usptream U-Boot test system uses L4T r28.1 to test Jetson TX1
> and TX2. It does test DHCP/TFTP, but not EFI or booting a kernel.

So it turned out that what I last tried with R28.3.0 was downstream
U-Boot plus upstream TF-A v2.1. I've been digging into this and have a
draft fix for upstream TF-A, a BL2 parameter layout change:

https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/1317

With this I can boot into -rc4 on R28.3.0. Haven't re-checked R24 yet.

Some U-Boot side EFI and non-EFI problems (as well as kernel deferred
probing problems) remain to be investigated.

Cheers,
Andreas

-- 
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)

[U-Boot] Jetson TX2 hangs during dhcp boot

[Mian Yousaf Kaukab]

On Sun, Jun 16, 2019 at 07:34:57PM +0200, Martin Husemann wrote:
> On Sun, Jun 16, 2019 at 09:27:12PM +0200, Mian Yousaf Kaukab wrote:
> > > Hmm. Are you building your own copy of ATF (you mentioned 2.1; I can't
> > > recall what version we ship)?
> > No, I am using the one shipped with R32.1.
> 
> That did not work at all for me with mainline u-boot (u-boot would hang
> early), so I went back to some older release (28.?) and all was fine
> again.
Have you tested with following patch applied to u-boot?
https://patchwork.ozlabs.org/patch/1115052/

> 
> Martin

BR,
Yousaf

[U-Boot] Jetson TX2 hangs during dhcp boot

[Stephen Warren]

On 6/16/19 1:46 PM, Mian Yousaf Kaukab wrote:
> On Fri, Jun 14, 2019 at 05:26:05PM +0000, Varun Wadekar wrote:
>>>> ERROR:   ARI request timed out: req 89 on CPU 4
>>>> ASSERT: plat/nvidia/tegra/soc/t186/drivers/mce/ari.c <127> : retries
>>>> != 0U
>>
>> This time out means that TEGRA_ARI_ROC_FLUSH_CACHE_TRBITS ARI call timed out. Looks like the firmware responsible for handling this ARI call is stuck or overloaded and not servicing any calls.
> Any thoughts on how to debug it?
> 
> I couldn't get upstream TF-A to work. Here is how I am building it:
> make CROSS_COMPILE=aarch64-linux-gnu- PLAT=tegra TARGET_SOC=t186 DEBUG=1 all
> 
> then I build tos.img as following:
> ./gen_tos_part_img.py bl31.bin JetPack_4.2_Linux_P3310/Linux_for_Tegra/bootloader/tos.img
> 
> (try to)flash as following:
> ./flash.sh -k secure-os jetson-tx2 mmcblk0p1
> 
> However, I can't get flashing to work as soon as tos.img from R32.1 is
> replaced with my build. Is upstream TF-A supposed to work with R32.1?

I personally wouldn't /expect/ anything except our downstream versions 
of ATF and Trusty to work, but they might if you're lucky. I believe 
we're quite close to (some version of) upstream for ATF so upstream ATF 
/might/ work (but I don't believe we test it, so no guarantees). For 
Trusty I'm less sure we're close to any particular upstream version 
these days, so I wouldn't count on much.