Xilinx ZynqMP SPL boot: psu_init_gpl.c code corrupts U-Boot memory

Xilinx ZynqMP SPL boot: psu_init_gpl.c code corrupts U-Boot memory

[Robert Hancock]

I've been trying to get the U-Boot SPL to work on a Xilinx ZCU102
development board. I have been testing with U-Boot 2021.01, and using
the generated psu_init_gpl HW initialization code generated by Vivado
2020.2, rather than the versions in the U-Boot tree, as we are
expecting to make changes to the configuration as we move to our own
board design.

The issue I was seeing is that the SPL seemingly locking up or crashing
early in the boot process, just after the SPL banner was printed. I've
CCed "Major A" as he reported something similar on the mailing list in
March ('ZynqMP boot: no messages from SPL other than "Debug uart
enabled"') but I didn't see a resoution posted.

After groveling around with the Xilinx JTAG debugger for a day or so, I
figured out that the problem was that the SPL ended up reading from an
invalid pointer address somewhere during the driver model
initialization process, and crashing. After using memory watchpoints to
identify where the bogus value was being written to the pointer storage
location, the memory write was actually coming from the Vivado 2020.2-
generated psu_init_gpl code. The offending code is in a function called
serdes_illcalib_pcie_gen1 and looks like this:

          if (gen2_calib != 1) 
          {
            ill1_val[loop] = ((0x04 + meancount[loop]*8) % 0x100);
            ill12_val[loop] = ((0x04 + meancount[loop]*8) >= 0x100) ?
0x10 : 0x00;
            Xil_Out32(0xFFFE0000+loop*4,iterresult[loop]);
            Xil_Out32(0xFFFE0010+loop*4,iterresult[loop+4]);
            Xil_Out32(0xFFFE0020+loop*4,bistpasscount[loop]);
            Xil_Out32(0xFFFE0030+loop*4,meancount[loop]);
          }
          if (gen2_calib == 1) 
          {
            ill1_val[loop] = ((0x104 + meancount[loop]*8) % 0x100);
            ill12_val[loop] = ((0x104 + meancount[loop]*8) >= 0x200) ?
0x02 : 0x01;
            Xil_Out32(0xFFFE0040+loop*4,iterresult[loop]);
            Xil_Out32(0xFFFE0050+loop*4,iterresult[loop+4]);
            Xil_Out32(0xFFFE0060+loop*4,bistpasscount[loop]);
            Xil_Out32(0xFFFE0070+loop*4,meancount[loop]);
          }

Those writes to 0xFFFExxxx are to on-chip memory addresses, not any
hardware register, so I have no idea what this is trying to achieve.
Possibly this is some debug code to store some calibration results that
was left in by mistake? Those addresses may not be used in the Xilinx
FSBL, but overwriting them in the U-Boot SPL wreaks havoc.

For now, I've worked around it by hacking the Xil_Out32 implementation
provided by U-Boot in board/xilinx/zynqmp/xil_io.h to trap and ignore
writes to memory addresses in the OCRAM range (0xFFFC0000 to
0xFFFFFFFF). With that in place, things seem to be working. Probably
this should be addressed in the Vivado psu_init code generator, but we
may need some workaround like this to avoid those using the affected
Xilinx tool versions from hitting this issue?

-- 
Robert Hancock
Senior Hardware Designer, Calian Advanced Technologies
www.calian.com

Xilinx ZynqMP SPL boot: psu_init_gpl.c code corrupts U-Boot memory

[Michal Simek]

Hi,

On 1/20/21 9:25 PM, Robert Hancock wrote:
> I've been trying to get the U-Boot SPL to work on a Xilinx ZCU102
> development board. I have been testing with U-Boot 2021.01, and using
> the generated psu_init_gpl HW initialization code generated by Vivado
> 2020.2, rather than the versions in the U-Boot tree, as we are
> expecting to make changes to the configuration as we move to our own
> board design.
> 
> The issue I was seeing is that the SPL seemingly locking up or crashing
> early in the boot process, just after the SPL banner was printed. I've
> CCed "Major A" as he reported something similar on the mailing list in
> March ('ZynqMP boot: no messages from SPL other than "Debug uart
> enabled"') but I didn't see a resoution posted.
> 
> After groveling around with the Xilinx JTAG debugger for a day or so, I
> figured out that the problem was that the SPL ended up reading from an
> invalid pointer address somewhere during the driver model
> initialization process, and crashing. After using memory watchpoints to
> identify where the bogus value was being written to the pointer storage
> location, the memory write was actually coming from the Vivado 2020.2-
> generated psu_init_gpl code. The offending code is in a function called
> serdes_illcalib_pcie_gen1 and looks like this:
> 
>           if (gen2_calib != 1) 
>           {
>             ill1_val[loop] = ((0x04 + meancount[loop]*8) % 0x100);
>             ill12_val[loop] = ((0x04 + meancount[loop]*8) >= 0x100) ?
> 0x10 : 0x00;
>             Xil_Out32(0xFFFE0000+loop*4,iterresult[loop]);
>             Xil_Out32(0xFFFE0010+loop*4,iterresult[loop+4]);
>             Xil_Out32(0xFFFE0020+loop*4,bistpasscount[loop]);
>             Xil_Out32(0xFFFE0030+loop*4,meancount[loop]);
>           }
>           if (gen2_calib == 1) 
>           {
>             ill1_val[loop] = ((0x104 + meancount[loop]*8) % 0x100);
>             ill12_val[loop] = ((0x104 + meancount[loop]*8) >= 0x200) ?
> 0x02 : 0x01;
>             Xil_Out32(0xFFFE0040+loop*4,iterresult[loop]);
>             Xil_Out32(0xFFFE0050+loop*4,iterresult[loop+4]);
>             Xil_Out32(0xFFFE0060+loop*4,bistpasscount[loop]);
>             Xil_Out32(0xFFFE0070+loop*4,meancount[loop]);
>           }
> 
> Those writes to 0xFFFExxxx are to on-chip memory addresses, not any
> hardware register, so I have no idea what this is trying to achieve.
> Possibly this is some debug code to store some calibration results that
> was left in by mistake? Those addresses may not be used in the Xilinx
> FSBL, but overwriting them in the U-Boot SPL wreaks havoc.
> 
> For now, I've worked around it by hacking the Xil_Out32 implementation
> provided by U-Boot in board/xilinx/zynqmp/xil_io.h to trap and ignore
> writes to memory addresses in the OCRAM range (0xFFFC0000 to
> 0xFFFFFFFF). With that in place, things seem to be working. Probably
> this should be addressed in the Vivado psu_init code generator, but we
> may need some workaround like this to avoid those using the affected
> Xilinx tool versions from hitting this issue?
> 

I can't see any code like this in u-boot code. I can't see context of
this code that's why simply don't know what it does and why.

And also not sure what you want me to do with this. Please suggest.

Thanks,
Michal

Xilinx ZynqMP SPL boot: psu_init_gpl.c code corrupts U-Boot memory

[Robert Hancock]

Hi Michal,

On Thu, 2021-01-21 at 09:41 +0100, Michal Simek wrote:
> Hi,
> 
> On 1/20/21 9:25 PM, Robert Hancock wrote:
> > I've been trying to get the U-Boot SPL to work on a Xilinx ZCU102
> > development board. I have been testing with U-Boot 2021.01, and using
> > the generated psu_init_gpl HW initialization code generated by Vivado
> > 2020.2, rather than the versions in the U-Boot tree, as we are
> > expecting to make changes to the configuration as we move to our own
> > board design.
> > 
> > The issue I was seeing is that the SPL seemingly locking up or crashing
> > early in the boot process, just after the SPL banner was printed. I've
> > CCed "Major A" as he reported something similar on the mailing list in
> > March ('ZynqMP boot: no messages from SPL other than "Debug uart
> > enabled"') but I didn't see a resoution posted.
> > 
> > After groveling around with the Xilinx JTAG debugger for a day or so, I
> > figured out that the problem was that the SPL ended up reading from an
> > invalid pointer address somewhere during the driver model
> > initialization process, and crashing. After using memory watchpoints to
> > identify where the bogus value was being written to the pointer storage
> > location, the memory write was actually coming from the Vivado 2020.2-
> > generated psu_init_gpl code. The offending code is in a function called
> > serdes_illcalib_pcie_gen1 and looks like this:
> > 
> >           if (gen2_calib != 1) 
> >           {
> >             ill1_val[loop] = ((0x04 + meancount[loop]*8) % 0x100);
> >             ill12_val[loop] = ((0x04 + meancount[loop]*8) >= 0x100) ?
> > 0x10 : 0x00;
> >             Xil_Out32(0xFFFE0000+loop*4,iterresult[loop]);
> >             Xil_Out32(0xFFFE0010+loop*4,iterresult[loop+4]);
> >             Xil_Out32(0xFFFE0020+loop*4,bistpasscount[loop]);
> >             Xil_Out32(0xFFFE0030+loop*4,meancount[loop]);
> >           }
> >           if (gen2_calib == 1) 
> >           {
> >             ill1_val[loop] = ((0x104 + meancount[loop]*8) % 0x100);
> >             ill12_val[loop] = ((0x104 + meancount[loop]*8) >= 0x200) ?
> > 0x02 : 0x01;
> >             Xil_Out32(0xFFFE0040+loop*4,iterresult[loop]);
> >             Xil_Out32(0xFFFE0050+loop*4,iterresult[loop+4]);
> >             Xil_Out32(0xFFFE0060+loop*4,bistpasscount[loop]);
> >             Xil_Out32(0xFFFE0070+loop*4,meancount[loop]);
> >           }
> > 
> > Those writes to 0xFFFExxxx are to on-chip memory addresses, not any
> > hardware register, so I have no idea what this is trying to achieve.
> > Possibly this is some debug code to store some calibration results that
> > was left in by mistake? Those addresses may not be used in the Xilinx
> > FSBL, but overwriting them in the U-Boot SPL wreaks havoc.
> > 
> > For now, I've worked around it by hacking the Xil_Out32 implementation
> > provided by U-Boot in board/xilinx/zynqmp/xil_io.h to trap and ignore
> > writes to memory addresses in the OCRAM range (0xFFFC0000 to
> > 0xFFFFFFFF). With that in place, things seem to be working. Probably
> > this should be addressed in the Vivado psu_init code generator, but we
> > may need some workaround like this to avoid those using the affected
> > Xilinx tool versions from hitting this issue?
> > 
> 
> I can't see any code like this in u-boot code. I can't see context of
> this code that's why simply don't know what it does and why.

I can email the psu_init_gpl code I have privately if you need it - the code is
almost 1MB so too large for the list or the usual Pastebin etc. it seems.
However, you can see similar code in this file from ZCU102 in the Xilinx
embeddedsw repo:

https://github.com/Xilinx/embeddedsw/blob/master/lib/sw_apps/zynqmp_fsbl/misc/zcu102/psu_init_gpl.c

That has writes to 0xFFFExxxx addresses in the serdes_illcalib_pcie_gen1
function, like mine has, as well as in its serdes_illcalib which don't seem to
be in the version I have. Looking at the git history, it appears those writes
were added in a commit that updated the file for the 2020.2 tools release, so
that may have been when this was introduced.

I don't see any such writes in the psu_init_gpl files in the U-Boot repo, I am
guessing because those were all generated using older tool versions that didn't
have this behavior.

Interestingly, looking at the 2020.2 Petalinux BSP for ZCU102 in its provided
xilinx-zcu102-2020.2/project-spec/hw-description/psu_init_gpl.c file, those
memory writes are there but have been commented out. Not sure if it somehow
generated that way or if someone realized that issue and did that manually?


> And also not sure what you want me to do with this. Please suggest.

The main purpose was to let the community know of this issue as others working
with ZynqMP hardware may run into this with the current Xilinx tools.

I have this patch locally to work around this issue right now. We may want
something like this in U-Boot as even if the Xilinx tools are eventually
updated to not do these writes, people using affected versions may still run
into issues. Let me know if something like this seems reasonable and I can
submit as a proper patch:

diff --git a/board/xilinx/zynqmp/xil_io.h b/board/xilinx/zynqmp/xil_io.h
index e6caa7c850..6c7c8829e9 100644
--- a/board/xilinx/zynqmp/xil_io.h
+++ b/board/xilinx/zynqmp/xil_io.h
@@ -12,6 +12,17 @@
 
 static void Xil_Out32(unsigned long addr, unsigned long val)
 {
+	/*
+	 * Vivado 2020.2 (at least) generates psu_init_gpl code that attempts
+	 * to write to OCM memory addresses. This may be some leftover debug
+	 * code meant for use with the Xilinx FSBL, but these memory addresses
+	 * are used by U-Boot SPL and overwriting them will corrupt our memory.
+	 * Ignore attempts to write to these addresses.
+	 */
+	if (addr >= 0xfffc0000 && addr <= 0xffffffff) {
+		printf("Ignoring psu_init_gpl write to 0x%08lx\n", addr);
+		return;
+	}
 	writel(val, addr);
 }
 

> 
> Thanks,
> Michal

Xilinx ZynqMP SPL boot: psu_init_gpl.c code corrupts U-Boot memory

[Robert Hancock]

On Thu, 2021-01-21 at 16:40 +0000, Robert Hancock wrote:
> On 1/20/21 9:25 PM, Robert Hancock wrote:
> > > I've been trying to get the U-Boot SPL to work on a Xilinx ZCU102
> > > development board. I have been testing with U-Boot 2021.01, and using
> > > the generated psu_init_gpl HW initialization code generated by Vivado
> > > 2020.2, rather than the versions in the U-Boot tree, as we are
> > > expecting to make changes to the configuration as we move to our own
> > > board design.
> > > 
> > > The issue I was seeing is that the SPL seemingly locking up or crashing
> > > early in the boot process, just after the SPL banner was printed. I've
> > > CCed "Major A" as he reported something similar on the mailing list in
> > > March ('ZynqMP boot: no messages from SPL other than "Debug uart
> > > enabled"') but I didn't see a resoution posted.
> > > 
> > > After groveling around with the Xilinx JTAG debugger for a day or so, I
> > > figured out that the problem was that the SPL ended up reading from an
> > > invalid pointer address somewhere during the driver model
> > > initialization process, and crashing. After using memory watchpoints to
> > > identify where the bogus value was being written to the pointer storage
> > > location, the memory write was actually coming from the Vivado 2020.2-
> > > generated psu_init_gpl code. The offending code is in a function called
> > > serdes_illcalib_pcie_gen1 and looks like this:
> > > 
> > >           if (gen2_calib != 1) 
> > >           {
> > >             ill1_val[loop] = ((0x04 + meancount[loop]*8) % 0x100);
> > >             ill12_val[loop] = ((0x04 + meancount[loop]*8) >= 0x100) ?
> > > 0x10 : 0x00;
> > >             Xil_Out32(0xFFFE0000+loop*4,iterresult[loop]);
> > >             Xil_Out32(0xFFFE0010+loop*4,iterresult[loop+4]);
> > >             Xil_Out32(0xFFFE0020+loop*4,bistpasscount[loop]);
> > >             Xil_Out32(0xFFFE0030+loop*4,meancount[loop]);
> > >           }
> > >           if (gen2_calib == 1) 
> > >           {
> > >             ill1_val[loop] = ((0x104 + meancount[loop]*8) % 0x100);
> > >             ill12_val[loop] = ((0x104 + meancount[loop]*8) >= 0x200) ?
> > > 0x02 : 0x01;
> > >             Xil_Out32(0xFFFE0040+loop*4,iterresult[loop]);
> > >             Xil_Out32(0xFFFE0050+loop*4,iterresult[loop+4]);
> > >             Xil_Out32(0xFFFE0060+loop*4,bistpasscount[loop]);
> > >             Xil_Out32(0xFFFE0070+loop*4,meancount[loop]);
> > >           }
> > > 
> > > Those writes to 0xFFFExxxx are to on-chip memory addresses, not any
> > > hardware register, so I have no idea what this is trying to achieve.
> > > Possibly this is some debug code to store some calibration results that
> > > was left in by mistake? Those addresses may not be used in the Xilinx
> > > FSBL, but overwriting them in the U-Boot SPL wreaks havoc.
> > > 
> > > For now, I've worked around it by hacking the Xil_Out32 implementation
> > > provided by U-Boot in board/xilinx/zynqmp/xil_io.h to trap and ignore
> > > writes to memory addresses in the OCRAM range (0xFFFC0000 to
> > > 0xFFFFFFFF). With that in place, things seem to be working. Probably
> > > this should be addressed in the Vivado psu_init code generator, but we
> > > may need some workaround like this to avoid those using the affected
> > > Xilinx tool versions from hitting this issue?
> > > 
> > 
> > I can't see any code like this in u-boot code. I can't see context of
> > this code that's why simply don't know what it does and why.
> 
> I can email the psu_init_gpl code I have privately if you need it - the code
> is
> almost 1MB so too large for the list or the usual Pastebin etc. it seems.
> However, you can see similar code in this file from ZCU102 in the Xilinx
> embeddedsw repo:
> 
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FXilinx%2Fembeddedsw%2Fblob%2Fmaster%2Flib%2Fsw_apps%2Fzynqmp_fsbl%2Fmisc%2Fzcu102%2Fpsu_init_gpl.c&data=04%7C01%7Crobert.hancock%40calian.com%7C55d03544d14741e8474808d8be4489a7%7C23b57807562f49ad92c43bb0f07a1fdf%7C0%7C0%7C637468548882351310%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nEiVgL1d%2F%2B4r9iqV8NYZ3JUVQPwCOPRFxEtU69FjH3Q%3D&reserved=0
> 
> That has writes to 0xFFFExxxx addresses in the serdes_illcalib_pcie_gen1
> function, like mine has, as well as in its serdes_illcalib which don't seem
> to
> be in the version I have. Looking at the git history, it appears those writes
> were added in a commit that updated the file for the 2020.2 tools release, so
> that may have been when this was introduced.
> 
> I don't see any such writes in the psu_init_gpl files in the U-Boot repo, I
> am
> guessing because those were all generated using older tool versions that
> didn't
> have this behavior.
> 
> Interestingly, looking at the 2020.2 Petalinux BSP for ZCU102 in its provided
> xilinx-zcu102-2020.2/project-spec/hw-description/psu_init_gpl.c file, those
> memory writes are there but have been commented out. Not sure if it somehow
> generated that way or if someone realized that issue and did that manually?
> 

From some more investigations on this, it appears that whether these writes end
up in the code or not may depend on exactly what state the Vivado project is in
when the XSA file is exported. If you just set up the block design and then
export the hardware at that point without bitstream included, these "debug"
writes are included. If the logic has been fully built up to bitstream
generation and then you export XSA with bitstream included, then the generated
code has the "debug" writes commented out.

That seems a bit unintuitive - I haven't tried to narrow down exactly what
stage of the build process causes that change to occur - but it's an
explanation for the difference at least. Hopefully if anyone else runs into a
similar issue they'll see this post and have some idea what has happened..

> 
> > And also not sure what you want me to do with this. Please suggest.
> 
> The main purpose was to let the community know of this issue as others
> working
> with ZynqMP hardware may run into this with the current Xilinx tools.
> 
> I have this patch locally to work around this issue right now. We may want
> something like this in U-Boot as even if the Xilinx tools are eventually
> updated to not do these writes, people using affected versions may still run
> into issues. Let me know if something like this seems reasonable and I can
> submit as a proper patch:
> 
> diff --git a/board/xilinx/zynqmp/xil_io.h b/board/xilinx/zynqmp/xil_io.h
> index e6caa7c850..6c7c8829e9 100644
> --- a/board/xilinx/zynqmp/xil_io.h
> +++ b/board/xilinx/zynqmp/xil_io.h
> @@ -12,6 +12,17 @@
>  
>  static void Xil_Out32(unsigned long addr, unsigned long val)
>  {
> +	/*
> +	 * Vivado 2020.2 (at least) generates psu_init_gpl code that attempts
> +	 * to write to OCM memory addresses. This may be some leftover debug
> +	 * code meant for use with the Xilinx FSBL, but these memory addresses
> +	 * are used by U-Boot SPL and overwriting them will corrupt our memory.
> +	 * Ignore attempts to write to these addresses.
> +	 */
> +	if (addr >= 0xfffc0000 && addr <= 0xffffffff) {
> +		printf("Ignoring psu_init_gpl write to 0x%08lx\n", addr);
> +		return;
> +	}
>  	writel(val, addr);
>  }
>  
> 
> > Thanks,
> > Michal
-- 
Robert Hancock
Senior Hardware Designer, Calian Advanced Technologies
www.calian.com

Xilinx ZynqMP SPL boot: psu_init_gpl.c code corrupts U-Boot memory

[Sean Anderson]

On 1/21/21 3:41 AM, Michal Simek wrote:
> Hi,
> 
> On 1/20/21 9:25 PM, Robert Hancock wrote:
>> I've been trying to get the U-Boot SPL to work on a Xilinx ZCU102
>> development board. I have been testing with U-Boot 2021.01, and using
>> the generated psu_init_gpl HW initialization code generated by Vivado
>> 2020.2, rather than the versions in the U-Boot tree, as we are
>> expecting to make changes to the configuration as we move to our own
>> board design.
>>
>> The issue I was seeing is that the SPL seemingly locking up or crashing
>> early in the boot process, just after the SPL banner was printed. I've
>> CCed "Major A" as he reported something similar on the mailing list in
>> March ('ZynqMP boot: no messages from SPL other than "Debug uart
>> enabled"') but I didn't see a resoution posted.
>>
>> After groveling around with the Xilinx JTAG debugger for a day or so, I
>> figured out that the problem was that the SPL ended up reading from an
>> invalid pointer address somewhere during the driver model
>> initialization process, and crashing. After using memory watchpoints to
>> identify where the bogus value was being written to the pointer storage
>> location, the memory write was actually coming from the Vivado 2020.2-
>> generated psu_init_gpl code. The offending code is in a function called
>> serdes_illcalib_pcie_gen1 and looks like this:
>>
>>            if (gen2_calib != 1)
>>            {
>>              ill1_val[loop] = ((0x04 + meancount[loop]*8) % 0x100);
>>              ill12_val[loop] = ((0x04 + meancount[loop]*8) >= 0x100) ?
>> 0x10 : 0x00;
>>              Xil_Out32(0xFFFE0000+loop*4,iterresult[loop]);
>>              Xil_Out32(0xFFFE0010+loop*4,iterresult[loop+4]);
>>              Xil_Out32(0xFFFE0020+loop*4,bistpasscount[loop]);
>>              Xil_Out32(0xFFFE0030+loop*4,meancount[loop]);
>>            }
>>            if (gen2_calib == 1)
>>            {
>>              ill1_val[loop] = ((0x104 + meancount[loop]*8) % 0x100);
>>              ill12_val[loop] = ((0x104 + meancount[loop]*8) >= 0x200) ?
>> 0x02 : 0x01;
>>              Xil_Out32(0xFFFE0040+loop*4,iterresult[loop]);
>>              Xil_Out32(0xFFFE0050+loop*4,iterresult[loop+4]);
>>              Xil_Out32(0xFFFE0060+loop*4,bistpasscount[loop]);
>>              Xil_Out32(0xFFFE0070+loop*4,meancount[loop]);
>>            }
>>
>> Those writes to 0xFFFExxxx are to on-chip memory addresses, not any
>> hardware register, so I have no idea what this is trying to achieve.
>> Possibly this is some debug code to store some calibration results that
>> was left in by mistake? Those addresses may not be used in the Xilinx
>> FSBL, but overwriting them in the U-Boot SPL wreaks havoc.
>>
>> For now, I've worked around it by hacking the Xil_Out32 implementation
>> provided by U-Boot in board/xilinx/zynqmp/xil_io.h to trap and ignore
>> writes to memory addresses in the OCRAM range (0xFFFC0000 to
>> 0xFFFFFFFF). With that in place, things seem to be working. Probably
>> this should be addressed in the Vivado psu_init code generator, but we
>> may need some workaround like this to avoid those using the affected
>> Xilinx tool versions from hitting this issue?
>>
> 
> I can't see any code like this in u-boot code. I can't see context of
> this code that's why simply don't know what it does and why.

I believe it's to fix [1]. See [2] for other people with this problem.
I'm not sure what Xilinx's stance on this behavior is.

--Sean

[1] https://www.xilinx.com/support/answers/72992.html
[2] https://forums.xilinx.com/t5/Embedded-Development-Tools/Custom-zcu102-2020-1-SATA-link-down-SStatus-1-SControl-330/td-p/1143816

> 
> And also not sure what you want me to do with this. Please suggest.
> 
> Thanks,
> Michal
> 

Xilinx ZynqMP SPL boot: psu_init_gpl.c code corrupts U-Boot memory

[Michal Simek]

Hi,

On 1/30/21 5:52 AM, Sean Anderson wrote:
> On 1/21/21 3:41 AM, Michal Simek wrote:
>> Hi,
>>
>> On 1/20/21 9:25 PM, Robert Hancock wrote:
>>> I've been trying to get the U-Boot SPL to work on a Xilinx ZCU102
>>> development board. I have been testing with U-Boot 2021.01, and using
>>> the generated psu_init_gpl HW initialization code generated by Vivado
>>> 2020.2, rather than the versions in the U-Boot tree, as we are
>>> expecting to make changes to the configuration as we move to our own
>>> board design.
>>>
>>> The issue I was seeing is that the SPL seemingly locking up or crashing
>>> early in the boot process, just after the SPL banner was printed. I've
>>> CCed "Major A" as he reported something similar on the mailing list in
>>> March ('ZynqMP boot: no messages from SPL other than "Debug uart
>>> enabled"') but I didn't see a resoution posted.
>>>
>>> After groveling around with the Xilinx JTAG debugger for a day or so, I
>>> figured out that the problem was that the SPL ended up reading from an
>>> invalid pointer address somewhere during the driver model
>>> initialization process, and crashing. After using memory watchpoints to
>>> identify where the bogus value was being written to the pointer storage
>>> location, the memory write was actually coming from the Vivado 2020.2-
>>> generated psu_init_gpl code. The offending code is in a function called
>>> serdes_illcalib_pcie_gen1 and looks like this:
>>>
>>> ?????????? if (gen2_calib != 1)
>>> ?????????? {
>>> ???????????? ill1_val[loop] = ((0x04 + meancount[loop]*8) % 0x100);
>>> ???????????? ill12_val[loop] = ((0x04 + meancount[loop]*8) >= 0x100) ?
>>> 0x10 : 0x00;
>>> ???????????? Xil_Out32(0xFFFE0000+loop*4,iterresult[loop]);
>>> ???????????? Xil_Out32(0xFFFE0010+loop*4,iterresult[loop+4]);
>>> ???????????? Xil_Out32(0xFFFE0020+loop*4,bistpasscount[loop]);
>>> ???????????? Xil_Out32(0xFFFE0030+loop*4,meancount[loop]);
>>> ?????????? }
>>> ?????????? if (gen2_calib == 1)
>>> ?????????? {
>>> ???????????? ill1_val[loop] = ((0x104 + meancount[loop]*8) % 0x100);
>>> ???????????? ill12_val[loop] = ((0x104 + meancount[loop]*8) >= 0x200) ?
>>> 0x02 : 0x01;
>>> ???????????? Xil_Out32(0xFFFE0040+loop*4,iterresult[loop]);
>>> ???????????? Xil_Out32(0xFFFE0050+loop*4,iterresult[loop+4]);
>>> ???????????? Xil_Out32(0xFFFE0060+loop*4,bistpasscount[loop]);
>>> ???????????? Xil_Out32(0xFFFE0070+loop*4,meancount[loop]);
>>> ?????????? }
>>>
>>> Those writes to 0xFFFExxxx are to on-chip memory addresses, not any
>>> hardware register, so I have no idea what this is trying to achieve.
>>> Possibly this is some debug code to store some calibration results that
>>> was left in by mistake? Those addresses may not be used in the Xilinx
>>> FSBL, but overwriting them in the U-Boot SPL wreaks havoc.
>>>
>>> For now, I've worked around it by hacking the Xil_Out32 implementation
>>> provided by U-Boot in board/xilinx/zynqmp/xil_io.h to trap and ignore
>>> writes to memory addresses in the OCRAM range (0xFFFC0000 to
>>> 0xFFFFFFFF). With that in place, things seem to be working. Probably
>>> this should be addressed in the Vivado psu_init code generator, but we
>>> may need some workaround like this to avoid those using the affected
>>> Xilinx tool versions from hitting this issue?
>>>
>>
>> I can't see any code like this in u-boot code. I can't see context of
>> this code that's why simply don't know what it does and why.
> 
> I believe it's to fix [1]. See [2] for other people with this problem.
> I'm not sure what Xilinx's stance on this behavior is.
> 
> --Sean
> 
> [1] https://www.xilinx.com/support/answers/72992.html
> [2]
> https://forums.xilinx.com/t5/Embedded-Development-Tools/Custom-zcu102-2020-1-SATA-link-down-SStatus-1-SControl-330/td-p/1143816
> 

I have let support team know about this but this is topic out of u-boot
mailing list because we simply don't have any code like this in the repo.
And it is just showing good decision to store psu_inits in u-boot repo
not to relay on specific design tool version.

Thanks,
Michal