Zynq MPSoC Dynamic DDR DIMM Configuration support

Zynq MPSoC Dynamic DDR DIMM Configuration support

[Jorge]

Hi Michal

We are doing some work on an MPSoC UZ3EG platform part of which
requires us to replace FSBL with SPL.

It seems the actual boot process is becoming an issue on these SoCs;
currently, 1) we embed the PMU firmware on SPL so the bootrom can
extract it and program it; 2) then SPL configures the PMU using a
platform specific binary that gets built also with SPL; and finally,
3) SPL sets up the DDR using its psu_init_gpl.c settings (also board
specific, part of the XSA).

It is this final step in the boot sequence that is being broken by the
Dynamic DDR DIMM configuration feature [1]

[1] https://www.xilinx.com/support/answers/75768.html

Are you aware of any work in progress to support this? Any thoughts on
how to work around it and train the DDR? will the functionality
required to implmenet Dynamic DDR DIMM configuration be added as a
separate file to the XSA tarball or will we need to do some native
implementation in SPL?

Becase without a change in the last link in the process chain
described earlier (calls to psu_init()), DDR just wont be accessible
to U-BOOT or OP-TEE.

In our case, we were able to boot from QSPI, boot SPL (in OCM), have
SPL unpack and validate the FIT image, execute TF-A(in OCM), but then
any jumps to OP-TEE or U-BOOT would obviously not progress since the
DDR wasnt properly trained/initialized.

so, any thoughts or plans you can share?

TIA!

jorge

Zynq MPSoC Dynamic DDR DIMM Configuration support

[Michal Simek]

Hi Jorge,

On 5/12/21 11:21 PM, Jorge Ramirez-Ortiz, Foundries wrote:
> Hi Michal
> 
> We are doing some work on an MPSoC UZ3EG platform part of which
> requires us to replace FSBL with SPL.

Just for curiosity what's the reason for this requirement that fsbl is
not enough?

> 
> It seems the actual boot process is becoming an issue on these SoCs;
> currently, 1) we embed the PMU firmware on SPL so the bootrom can
> extract it and program it;

Actually it is not working like this. PMU FW is own part of boot.bin and
it is not embed in SPL.


> 2) then SPL configures the PMU using a
> platform specific binary that gets built also with SPL; 

PMU cfg object.


and finally,
> 3) SPL sets up the DDR using its psu_init_gpl.c settings (also board
> specific, part of the XSA).

yes.

> 
> It is this final step in the boot sequence that is being broken by the
> Dynamic DDR DIMM configuration feature [1]
> 
> [1] https://www.xilinx.com/support/answers/75768.html

This was developed for zcu102 and maybe others boards which have DIMM
modules where origin part were EOL.


> Are you aware of any work in progress to support this? Any thoughts on
> how to work around it and train the DDR? will the functionality
> required to implmenet Dynamic DDR DIMM configuration be added as a
> separate file to the XSA tarball or will we need to do some native
> implementation in SPL?

I am not aware about any work on SPL side to support this. IIRC FSBL
didn't have generic DDR configuration. It was only by reading SPD and
aligned some parameters but it is quite a long time I have looked at it
last time.


> Becase without a change in the last link in the process chain
> described earlier (calls to psu_init()), DDR just wont be accessible
> to U-BOOT or OP-TEE.
> 
> In our case, we were able to boot from QSPI, boot SPL (in OCM), have
> SPL unpack and validate the FIT image, execute TF-A(in OCM), but then
> any jumps to OP-TEE or U-BOOT would obviously not progress since the
> DDR wasnt properly trained/initialized.
> 
> so, any thoughts or plans you can share?

The question is why you need this feature to be there. It is not working
for every DIMM module. And normally if you have custom boards you need
just one DDR configuration (or limited number based on HW versions) and
for that there is really no need to waste your boot up time on it.
You can add multiple configurations to psu_init_gpl() and based on any
information (board rev/pins/etc) decided which one should be used.

Thanks,
Michal

Zynq MPSoC Dynamic DDR DIMM Configuration support

[Jorge]

On 13/05/21, Michal Simek wrote:
> Hi Jorge,
> 
> On 5/12/21 11:21 PM, Jorge Ramirez-Ortiz, Foundries wrote:
> > Hi Michal
> > 
> > We are doing some work on an MPSoC UZ3EG platform part of which
> > requires us to replace FSBL with SPL.
> 
> Just for curiosity what's the reason for this requirement that fsbl is
> not enough?

we prefer to have a common boot strategy on all the boards we support
(whether it is from NXP, Xilinx, ST or TI); it just simplifies the OTA
process (including their firmwares); and of course our meta layers.

> 
> > 
> > It seems the actual boot process is becoming an issue on these SoCs;
> > currently, 1) we embed the PMU firmware on SPL so the bootrom can
> > extract it and program it;
> 
> Actually it is not working like this. PMU FW is own part of boot.bin and
> it is not embed in SPL.

yes, absolutely (boot.bin is what we have been flashing), holding SPL and FW 

> 
> 
> > 2) then SPL configures the PMU using a
> > platform specific binary that gets built also with SPL; 
> 
> PMU cfg object.

yep, pm_cfg_obj.c

> 
> 
> and finally,
> > 3) SPL sets up the DDR using its psu_init_gpl.c settings (also board
> > specific, part of the XSA).
> 
> yes.
> 
> > 
> > It is this final step in the boot sequence that is being broken by the
> > Dynamic DDR DIMM configuration feature [1]
> > 
> > [1] https://www.xilinx.com/support/answers/75768.html
> 
> This was developed for zcu102 and maybe others boards which have DIMM
> modules where origin part were EOL.

right.

but notice that there is also support on u-boot for
altera/imx/marvell/microchip; so just wondering if we should add
drivers/ddr/xilinx to this list.  

> 
> 
> > Are you aware of any work in progress to support this? Any thoughts on
> > how to work around it and train the DDR? will the functionality
> > required to implmenet Dynamic DDR DIMM configuration be added as a
> > separate file to the XSA tarball or will we need to do some native
> > implementation in SPL?
> 
> I am not aware about any work on SPL side to support this. IIRC FSBL
> didn't have generic DDR configuration. It was only by reading SPD and
> aligned some parameters but it is quite a long time I have looked at it
> last time.

ok

> 
> 
> > Becase without a change in the last link in the process chain
> > described earlier (calls to psu_init()), DDR just wont be accessible
> > to U-BOOT or OP-TEE.
> > 
> > In our case, we were able to boot from QSPI, boot SPL (in OCM), have
> > SPL unpack and validate the FIT image, execute TF-A(in OCM), but then
> > any jumps to OP-TEE or U-BOOT would obviously not progress since the
> > DDR wasnt properly trained/initialized.
> > 
> > so, any thoughts or plans you can share?
> 
> The question is why you need this feature to be there. It is not working
> for every DIMM module. And normally if you have custom boards you need
> just one DDR configuration (or limited number based on HW versions) and
> for that there is really no need to waste your boot up time on it.

not sure what you mean. we need this feature because it adds the
expected flexibility to the bootloader. Sure, we can hardcode DDR
configurations but why should we when it can be resolved by
software. Am I missing your point?

> You can add multiple configurations to psu_init_gpl() and based on any
> information (board rev/pins/etc) decided which one should be used.

so what you are suggesting is that we customize psu_init_gpl() to the
target (ie, have an updated xsa file with whatever config we need for
this system).  

what I fail to grasp is why we can't take a step forward and do what
we do for other architectures in u-boot. and what fsbl already does by
config.

I mean, why not? do you foresee any integration issues with the
current bootflow?

> 
> Thanks,
> Michal
> 

Zynq MPSoC Dynamic DDR DIMM Configuration support

[Jorge]

On 13/05/21, Jorge Ramirez-Ortiz, Foundries wrote:
> On 13/05/21, Michal Simek wrote:
> > Hi Jorge,
> > 
> > On 5/12/21 11:21 PM, Jorge Ramirez-Ortiz, Foundries wrote:
> > > Hi Michal
> > > 
> > > We are doing some work on an MPSoC UZ3EG platform part of which
> > > requires us to replace FSBL with SPL.
> > 
> > Just for curiosity what's the reason for this requirement that fsbl is
> > not enough?
> 
> we prefer to have a common boot strategy on all the boards we support
> (whether it is from NXP, Xilinx, ST or TI); it just simplifies the OTA
> process (including their firmwares); and of course our meta layers.
> 
> > 
> > > 
> > > It seems the actual boot process is becoming an issue on these SoCs;
> > > currently, 1) we embed the PMU firmware on SPL so the bootrom can
> > > extract it and program it;
> > 
> > Actually it is not working like this. PMU FW is own part of boot.bin and
> > it is not embed in SPL.
> 
> yes, absolutely (boot.bin is what we have been flashing), holding SPL and FW 
> 
> > 
> > 
> > > 2) then SPL configures the PMU using a
> > > platform specific binary that gets built also with SPL; 
> > 
> > PMU cfg object.
> 
> yep, pm_cfg_obj.c
> 
> > 
> > 
> > and finally,
> > > 3) SPL sets up the DDR using its psu_init_gpl.c settings (also board
> > > specific, part of the XSA).
> > 
> > yes.
> > 
> > > 
> > > It is this final step in the boot sequence that is being broken by the
> > > Dynamic DDR DIMM configuration feature [1]
> > > 
> > > [1] https://www.xilinx.com/support/answers/75768.html
> > 
> > This was developed for zcu102 and maybe others boards which have DIMM
> > modules where origin part were EOL.
> 
> right.
> 
> but notice that there is also support on u-boot for
> altera/imx/marvell/microchip; so just wondering if we should add
> drivers/ddr/xilinx to this list.  
> 
> > 
> > 
> > > Are you aware of any work in progress to support this? Any thoughts on
> > > how to work around it and train the DDR? will the functionality
> > > required to implmenet Dynamic DDR DIMM configuration be added as a
> > > separate file to the XSA tarball or will we need to do some native
> > > implementation in SPL?
> > 
> > I am not aware about any work on SPL side to support this. IIRC FSBL
> > didn't have generic DDR configuration. It was only by reading SPD and
> > aligned some parameters but it is quite a long time I have looked at it
> > last time.
> 
> ok
> 
> > 
> > 
> > > Becase without a change in the last link in the process chain
> > > described earlier (calls to psu_init()), DDR just wont be accessible
> > > to U-BOOT or OP-TEE.
> > > 
> > > In our case, we were able to boot from QSPI, boot SPL (in OCM), have
> > > SPL unpack and validate the FIT image, execute TF-A(in OCM), but then
> > > any jumps to OP-TEE or U-BOOT would obviously not progress since the
> > > DDR wasnt properly trained/initialized.
> > > 
> > > so, any thoughts or plans you can share?
> > 
> > The question is why you need this feature to be there. It is not working
> > for every DIMM module. And normally if you have custom boards you need
> > just one DDR configuration (or limited number based on HW versions) and
> > for that there is really no need to waste your boot up time on it.
> 
> not sure what you mean. we need this feature because it adds the
> expected flexibility to the bootloader. Sure, we can hardcode DDR
> configurations but why should we when it can be resolved by
> software. Am I missing your point?
> 
> > You can add multiple configurations to psu_init_gpl() and based on any
> > information (board rev/pins/etc) decided which one should be used.
> 
> so what you are suggesting is that we customize psu_init_gpl() to the
> target (ie, have an updated xsa file with whatever config we need for
> this system).  
> 
> what I fail to grasp is why we can't take a step forward and do what
> we do for other architectures in u-boot. and what fsbl already does by
> config.
> 
> I mean, why not? do you foresee any integration issues with the
> current bootflow?

this is the current code on fsbl_init

so if we dont do this change, there will be more designs encountering
the same issue that we had. IMO it should be addressed in SPL.


#ifdef XFSBL_PS_DDR
#ifdef XPAR_DYNAMIC_DDR_ENABLED
	/*
	 * This function is used for all the ZynqMP boards.
	 * This function initialize the DDR by fetching the SPD data from
	 * EEPROM. This function will determine the type of the DDR and decode
	 * the SPD structure accordingly. The SPD data is used to calculate the
	 * register values of DDR controller and DDR PHY.
	 */
	Status = XFsbl_DdrInit();
	if (XFSBL_SUCCESS != Status) {
		XFsbl_Printf(DEBUG_GENERAL,"XFSBL_DDR_INIT_FAILED\n\r");
		goto END;
	}
#endif
#endif


> 
> > 
> > Thanks,
> > Michal
> > 

Zynq MPSoC Dynamic DDR DIMM Configuration support

[Michal Simek]

Hi,

On 5/13/21 9:24 AM, Jorge Ramirez-Ortiz, Foundries wrote:
> On 13/05/21, Michal Simek wrote:
>> Hi Jorge,
>>
>> On 5/12/21 11:21 PM, Jorge Ramirez-Ortiz, Foundries wrote:
>>> Hi Michal
>>>
>>> We are doing some work on an MPSoC UZ3EG platform part of which
>>> requires us to replace FSBL with SPL.
>>
>> Just for curiosity what's the reason for this requirement that fsbl is
>> not enough?
> 
> we prefer to have a common boot strategy on all the boards we support
> (whether it is from NXP, Xilinx, ST or TI); it just simplifies the OTA
> process (including their firmwares); and of course our meta layers.
> 
>>
>>>
>>> It seems the actual boot process is becoming an issue on these SoCs;
>>> currently, 1) we embed the PMU firmware on SPL so the bootrom can
>>> extract it and program it;
>>
>> Actually it is not working like this. PMU FW is own part of boot.bin and
>> it is not embed in SPL.
> 
> yes, absolutely (boot.bin is what we have been flashing), holding SPL and FW 
> 
>>
>>
>>> 2) then SPL configures the PMU using a
>>> platform specific binary that gets built also with SPL; 
>>
>> PMU cfg object.
> 
> yep, pm_cfg_obj.c
> 
>>
>>
>> and finally,
>>> 3) SPL sets up the DDR using its psu_init_gpl.c settings (also board
>>> specific, part of the XSA).
>>
>> yes.
>>
>>>
>>> It is this final step in the boot sequence that is being broken by the
>>> Dynamic DDR DIMM configuration feature [1]
>>>
>>> [1] https://www.xilinx.com/support/answers/75768.html
>>
>> This was developed for zcu102 and maybe others boards which have DIMM
>> modules where origin part were EOL.
> 
> right.
> 
> but notice that there is also support on u-boot for
> altera/imx/marvell/microchip; so just wondering if we should add
> drivers/ddr/xilinx to this list.  

Also SPD code is around too.



>>
>>
>>> Are you aware of any work in progress to support this? Any thoughts on
>>> how to work around it and train the DDR? will the functionality
>>> required to implmenet Dynamic DDR DIMM configuration be added as a
>>> separate file to the XSA tarball or will we need to do some native
>>> implementation in SPL?
>>
>> I am not aware about any work on SPL side to support this. IIRC FSBL
>> didn't have generic DDR configuration. It was only by reading SPD and
>> aligned some parameters but it is quite a long time I have looked at it
>> last time.
> 
> ok
> 
>>
>>
>>> Becase without a change in the last link in the process chain
>>> described earlier (calls to psu_init()), DDR just wont be accessible
>>> to U-BOOT or OP-TEE.
>>>
>>> In our case, we were able to boot from QSPI, boot SPL (in OCM), have
>>> SPL unpack and validate the FIT image, execute TF-A(in OCM), but then
>>> any jumps to OP-TEE or U-BOOT would obviously not progress since the
>>> DDR wasnt properly trained/initialized.
>>>
>>> so, any thoughts or plans you can share?
>>
>> The question is why you need this feature to be there. It is not working
>> for every DIMM module. And normally if you have custom boards you need
>> just one DDR configuration (or limited number based on HW versions) and
>> for that there is really no need to waste your boot up time on it.
> 
> not sure what you mean. we need this feature because it adds the
> expected flexibility to the bootloader. Sure, we can hardcode DDR
> configurations but why should we when it can be resolved by
> software. Am I missing your point?

Ok. One is support for evaluation boards and the second for a product.
In product you don't need this because I would say most of projects are
not using DIMMs.
If you want to support generic evaluations boards then of course it can
be done.

> 
>> You can add multiple configurations to psu_init_gpl() and based on any
>> information (board rev/pins/etc) decided which one should be used.
> 
> so what you are suggesting is that we customize psu_init_gpl() to the
> target (ie, have an updated xsa file with whatever config we need for
> this system).  

In products yes.

> 
> what I fail to grasp is why we can't take a step forward and do what
> we do for other architectures in u-boot. and what fsbl already does by
> config.
> 
> I mean, why not? do you foresee any integration issues with the
> current bootflow?

Not really if you are willing to write that code just go for it.
I think it is definitely feasible to do so but I haven't seen any
algorithm to do it in generic way even there must be one behind in
xilinx tools. There are a lot of types of memories which you can
configure in xilinx design tools but it won't be any problem to limit it
to some.

Thanks,
Michal

Zynq MPSoC Dynamic DDR DIMM Configuration support

[Jorge]

On 13/05/21, Michal Simek wrote:
> Hi,
> 
> On 5/13/21 9:24 AM, Jorge Ramirez-Ortiz, Foundries wrote:
> > On 13/05/21, Michal Simek wrote:
> >> Hi Jorge,
> >>
> >> On 5/12/21 11:21 PM, Jorge Ramirez-Ortiz, Foundries wrote:
> >>> Hi Michal
> >>>
> >>> We are doing some work on an MPSoC UZ3EG platform part of which
> >>> requires us to replace FSBL with SPL.
> >>
> >> Just for curiosity what's the reason for this requirement that fsbl is
> >> not enough?
> > 
> > we prefer to have a common boot strategy on all the boards we support
> > (whether it is from NXP, Xilinx, ST or TI); it just simplifies the OTA
> > process (including their firmwares); and of course our meta layers.
> > 
> >>
> >>>
> >>> It seems the actual boot process is becoming an issue on these SoCs;
> >>> currently, 1) we embed the PMU firmware on SPL so the bootrom can
> >>> extract it and program it;
> >>
> >> Actually it is not working like this. PMU FW is own part of boot.bin and
> >> it is not embed in SPL.
> > 
> > yes, absolutely (boot.bin is what we have been flashing), holding SPL and FW 
> > 
> >>
> >>
> >>> 2) then SPL configures the PMU using a
> >>> platform specific binary that gets built also with SPL; 
> >>
> >> PMU cfg object.
> > 
> > yep, pm_cfg_obj.c
> > 
> >>
> >>
> >> and finally,
> >>> 3) SPL sets up the DDR using its psu_init_gpl.c settings (also board
> >>> specific, part of the XSA).
> >>
> >> yes.
> >>
> >>>
> >>> It is this final step in the boot sequence that is being broken by the
> >>> Dynamic DDR DIMM configuration feature [1]
> >>>
> >>> [1] https://www.xilinx.com/support/answers/75768.html
> >>
> >> This was developed for zcu102 and maybe others boards which have DIMM
> >> modules where origin part were EOL.
> > 
> > right.
> > 
> > but notice that there is also support on u-boot for
> > altera/imx/marvell/microchip; so just wondering if we should add
> > drivers/ddr/xilinx to this list.  
> 
> Also SPD code is around too.
> 
> 
> 
> >>
> >>
> >>> Are you aware of any work in progress to support this? Any thoughts on
> >>> how to work around it and train the DDR? will the functionality
> >>> required to implmenet Dynamic DDR DIMM configuration be added as a
> >>> separate file to the XSA tarball or will we need to do some native
> >>> implementation in SPL?
> >>
> >> I am not aware about any work on SPL side to support this. IIRC FSBL
> >> didn't have generic DDR configuration. It was only by reading SPD and
> >> aligned some parameters but it is quite a long time I have looked at it
> >> last time.
> > 
> > ok
> > 
> >>
> >>
> >>> Becase without a change in the last link in the process chain
> >>> described earlier (calls to psu_init()), DDR just wont be accessible
> >>> to U-BOOT or OP-TEE.
> >>>
> >>> In our case, we were able to boot from QSPI, boot SPL (in OCM), have
> >>> SPL unpack and validate the FIT image, execute TF-A(in OCM), but then
> >>> any jumps to OP-TEE or U-BOOT would obviously not progress since the
> >>> DDR wasnt properly trained/initialized.
> >>>
> >>> so, any thoughts or plans you can share?
> >>
> >> The question is why you need this feature to be there. It is not working
> >> for every DIMM module. And normally if you have custom boards you need
> >> just one DDR configuration (or limited number based on HW versions) and
> >> for that there is really no need to waste your boot up time on it.
> > 
> > not sure what you mean. we need this feature because it adds the
> > expected flexibility to the bootloader. Sure, we can hardcode DDR
> > configurations but why should we when it can be resolved by
> > software. Am I missing your point?
> 
> Ok. One is support for evaluation boards and the second for a product.
> In product you don't need this because I would say most of projects are
> not using DIMMs.
> If you want to support generic evaluations boards then of course it can
> be done.
> 
> > 
> >> You can add multiple configurations to psu_init_gpl() and based on any
> >> information (board rev/pins/etc) decided which one should be used.
> > 
> > so what you are suggesting is that we customize psu_init_gpl() to the
> > target (ie, have an updated xsa file with whatever config we need for
> > this system).  
> 
> In products yes.
> 
> > 
> > what I fail to grasp is why we can't take a step forward and do what
> > we do for other architectures in u-boot. and what fsbl already does by
> > config.
> > 
> > I mean, why not? do you foresee any integration issues with the
> > current bootflow?
> 
> Not really if you are willing to write that code just go for it.
> I think it is definitely feasible to do so but I haven't seen any
> algorithm to do it in generic way even there must be one behind in
> xilinx tools. There are a lot of types of memories which you can
> configure in xilinx design tools but it won't be any problem to limit it
> to some.

ok. unfortunately I am not sure when - we have tight deadlines on this
product so we will just update psu_init().

In fact, and this is relevant as well to keep in mind for a future
TO-DO list, the issue we were having was not due to the dynamic
configuration of DDR but to the fact that the DDR on this product is
ECC (and ECC, when enabled in the DDR controller must be initialized
_after_ psu_init() has run - which is something that FSBL does but SPL
doesnt).

So we will just disable ECC for the time being and will send a patch
once we come around to do it (or with a bit of luck someone will beat
me to it and I will just pick it up from the next u-boot release)

> 
> Thanks,
> Michal
> 

Zynq MPSoC Dynamic DDR DIMM Configuration support

[Michal Simek]

On 5/14/21 9:38 AM, Jorge Ramirez-Ortiz, Foundries wrote:
> On 13/05/21, Michal Simek wrote:
>> Hi,
>>
>> On 5/13/21 9:24 AM, Jorge Ramirez-Ortiz, Foundries wrote:
>>> On 13/05/21, Michal Simek wrote:
>>>> Hi Jorge,
>>>>
>>>> On 5/12/21 11:21 PM, Jorge Ramirez-Ortiz, Foundries wrote:
>>>>> Hi Michal
>>>>>
>>>>> We are doing some work on an MPSoC UZ3EG platform part of which
>>>>> requires us to replace FSBL with SPL.
>>>>
>>>> Just for curiosity what's the reason for this requirement that fsbl is
>>>> not enough?
>>>
>>> we prefer to have a common boot strategy on all the boards we support
>>> (whether it is from NXP, Xilinx, ST or TI); it just simplifies the OTA
>>> process (including their firmwares); and of course our meta layers.
>>>
>>>>
>>>>>
>>>>> It seems the actual boot process is becoming an issue on these SoCs;
>>>>> currently, 1) we embed the PMU firmware on SPL so the bootrom can
>>>>> extract it and program it;
>>>>
>>>> Actually it is not working like this. PMU FW is own part of boot.bin and
>>>> it is not embed in SPL.
>>>
>>> yes, absolutely (boot.bin is what we have been flashing), holding SPL and FW 
>>>
>>>>
>>>>
>>>>> 2) then SPL configures the PMU using a
>>>>> platform specific binary that gets built also with SPL; 
>>>>
>>>> PMU cfg object.
>>>
>>> yep, pm_cfg_obj.c
>>>
>>>>
>>>>
>>>> and finally,
>>>>> 3) SPL sets up the DDR using its psu_init_gpl.c settings (also board
>>>>> specific, part of the XSA).
>>>>
>>>> yes.
>>>>
>>>>>
>>>>> It is this final step in the boot sequence that is being broken by the
>>>>> Dynamic DDR DIMM configuration feature [1]
>>>>>
>>>>> [1] https://www.xilinx.com/support/answers/75768.html
>>>>
>>>> This was developed for zcu102 and maybe others boards which have DIMM
>>>> modules where origin part were EOL.
>>>
>>> right.
>>>
>>> but notice that there is also support on u-boot for
>>> altera/imx/marvell/microchip; so just wondering if we should add
>>> drivers/ddr/xilinx to this list.  
>>
>> Also SPD code is around too.
>>
>>
>>
>>>>
>>>>
>>>>> Are you aware of any work in progress to support this? Any thoughts on
>>>>> how to work around it and train the DDR? will the functionality
>>>>> required to implmenet Dynamic DDR DIMM configuration be added as a
>>>>> separate file to the XSA tarball or will we need to do some native
>>>>> implementation in SPL?
>>>>
>>>> I am not aware about any work on SPL side to support this. IIRC FSBL
>>>> didn't have generic DDR configuration. It was only by reading SPD and
>>>> aligned some parameters but it is quite a long time I have looked at it
>>>> last time.
>>>
>>> ok
>>>
>>>>
>>>>
>>>>> Becase without a change in the last link in the process chain
>>>>> described earlier (calls to psu_init()), DDR just wont be accessible
>>>>> to U-BOOT or OP-TEE.
>>>>>
>>>>> In our case, we were able to boot from QSPI, boot SPL (in OCM), have
>>>>> SPL unpack and validate the FIT image, execute TF-A(in OCM), but then
>>>>> any jumps to OP-TEE or U-BOOT would obviously not progress since the
>>>>> DDR wasnt properly trained/initialized.
>>>>>
>>>>> so, any thoughts or plans you can share?
>>>>
>>>> The question is why you need this feature to be there. It is not working
>>>> for every DIMM module. And normally if you have custom boards you need
>>>> just one DDR configuration (or limited number based on HW versions) and
>>>> for that there is really no need to waste your boot up time on it.
>>>
>>> not sure what you mean. we need this feature because it adds the
>>> expected flexibility to the bootloader. Sure, we can hardcode DDR
>>> configurations but why should we when it can be resolved by
>>> software. Am I missing your point?
>>
>> Ok. One is support for evaluation boards and the second for a product.
>> In product you don't need this because I would say most of projects are
>> not using DIMMs.
>> If you want to support generic evaluations boards then of course it can
>> be done.
>>
>>>
>>>> You can add multiple configurations to psu_init_gpl() and based on any
>>>> information (board rev/pins/etc) decided which one should be used.
>>>
>>> so what you are suggesting is that we customize psu_init_gpl() to the
>>> target (ie, have an updated xsa file with whatever config we need for
>>> this system).  
>>
>> In products yes.
>>
>>>
>>> what I fail to grasp is why we can't take a step forward and do what
>>> we do for other architectures in u-boot. and what fsbl already does by
>>> config.
>>>
>>> I mean, why not? do you foresee any integration issues with the
>>> current bootflow?
>>
>> Not really if you are willing to write that code just go for it.
>> I think it is definitely feasible to do so but I haven't seen any
>> algorithm to do it in generic way even there must be one behind in
>> xilinx tools. There are a lot of types of memories which you can
>> configure in xilinx design tools but it won't be any problem to limit it
>> to some.
> 
> ok. unfortunately I am not sure when - we have tight deadlines on this
> product so we will just update psu_init().
> 
> In fact, and this is relevant as well to keep in mind for a future
> TO-DO list, the issue we were having was not due to the dynamic
> configuration of DDR but to the fact that the DDR on this product is
> ECC (and ECC, when enabled in the DDR controller must be initialized
> _after_ psu_init() has run - which is something that FSBL does but SPL
> doesnt).
> 
> So we will just disable ECC for the time being and will send a patch
> once we come around to do it (or with a bit of luck someone will beat
> me to it and I will just pick it up from the next u-boot release)

On Zynq ECC should be supported already. Take a look at ddrc_init() but
I would have to look at details how is DDR initialized because this is
clearing just 1MB which is shadow by OCM at this stage.

I would expect that procedure will be the same. Just find out that DDR
controller is working in ECC mode and I hope that for initialization any
DMA is used.

Thanks,
Michal

Zynq MPSoC Dynamic DDR DIMM Configuration support

[Jorge]

On 14/05/21, Michal Simek wrote:
> 
> 
> On 5/14/21 9:38 AM, Jorge Ramirez-Ortiz, Foundries wrote:
> > On 13/05/21, Michal Simek wrote:
> >> Hi,
> >>
> >> On 5/13/21 9:24 AM, Jorge Ramirez-Ortiz, Foundries wrote:
> >>> On 13/05/21, Michal Simek wrote:
> >>>> Hi Jorge,
> >>>>
> >>>> On 5/12/21 11:21 PM, Jorge Ramirez-Ortiz, Foundries wrote:
> >>>>> Hi Michal
> >>>>>
> >>>>> We are doing some work on an MPSoC UZ3EG platform part of which
> >>>>> requires us to replace FSBL with SPL.
> >>>>
> >>>> Just for curiosity what's the reason for this requirement that fsbl is
> >>>> not enough?
> >>>
> >>> we prefer to have a common boot strategy on all the boards we support
> >>> (whether it is from NXP, Xilinx, ST or TI); it just simplifies the OTA
> >>> process (including their firmwares); and of course our meta layers.
> >>>
> >>>>
> >>>>>
> >>>>> It seems the actual boot process is becoming an issue on these SoCs;
> >>>>> currently, 1) we embed the PMU firmware on SPL so the bootrom can
> >>>>> extract it and program it;
> >>>>
> >>>> Actually it is not working like this. PMU FW is own part of boot.bin and
> >>>> it is not embed in SPL.
> >>>
> >>> yes, absolutely (boot.bin is what we have been flashing), holding SPL and FW 
> >>>
> >>>>
> >>>>
> >>>>> 2) then SPL configures the PMU using a
> >>>>> platform specific binary that gets built also with SPL; 
> >>>>
> >>>> PMU cfg object.
> >>>
> >>> yep, pm_cfg_obj.c
> >>>
> >>>>
> >>>>
> >>>> and finally,
> >>>>> 3) SPL sets up the DDR using its psu_init_gpl.c settings (also board
> >>>>> specific, part of the XSA).
> >>>>
> >>>> yes.
> >>>>
> >>>>>
> >>>>> It is this final step in the boot sequence that is being broken by the
> >>>>> Dynamic DDR DIMM configuration feature [1]
> >>>>>
> >>>>> [1] https://www.xilinx.com/support/answers/75768.html
> >>>>
> >>>> This was developed for zcu102 and maybe others boards which have DIMM
> >>>> modules where origin part were EOL.
> >>>
> >>> right.
> >>>
> >>> but notice that there is also support on u-boot for
> >>> altera/imx/marvell/microchip; so just wondering if we should add
> >>> drivers/ddr/xilinx to this list.  
> >>
> >> Also SPD code is around too.
> >>
> >>
> >>
> >>>>
> >>>>
> >>>>> Are you aware of any work in progress to support this? Any thoughts on
> >>>>> how to work around it and train the DDR? will the functionality
> >>>>> required to implmenet Dynamic DDR DIMM configuration be added as a
> >>>>> separate file to the XSA tarball or will we need to do some native
> >>>>> implementation in SPL?
> >>>>
> >>>> I am not aware about any work on SPL side to support this. IIRC FSBL
> >>>> didn't have generic DDR configuration. It was only by reading SPD and
> >>>> aligned some parameters but it is quite a long time I have looked at it
> >>>> last time.
> >>>
> >>> ok
> >>>
> >>>>
> >>>>
> >>>>> Becase without a change in the last link in the process chain
> >>>>> described earlier (calls to psu_init()), DDR just wont be accessible
> >>>>> to U-BOOT or OP-TEE.
> >>>>>
> >>>>> In our case, we were able to boot from QSPI, boot SPL (in OCM), have
> >>>>> SPL unpack and validate the FIT image, execute TF-A(in OCM), but then
> >>>>> any jumps to OP-TEE or U-BOOT would obviously not progress since the
> >>>>> DDR wasnt properly trained/initialized.
> >>>>>
> >>>>> so, any thoughts or plans you can share?
> >>>>
> >>>> The question is why you need this feature to be there. It is not working
> >>>> for every DIMM module. And normally if you have custom boards you need
> >>>> just one DDR configuration (or limited number based on HW versions) and
> >>>> for that there is really no need to waste your boot up time on it.
> >>>
> >>> not sure what you mean. we need this feature because it adds the
> >>> expected flexibility to the bootloader. Sure, we can hardcode DDR
> >>> configurations but why should we when it can be resolved by
> >>> software. Am I missing your point?
> >>
> >> Ok. One is support for evaluation boards and the second for a product.
> >> In product you don't need this because I would say most of projects are
> >> not using DIMMs.
> >> If you want to support generic evaluations boards then of course it can
> >> be done.
> >>
> >>>
> >>>> You can add multiple configurations to psu_init_gpl() and based on any
> >>>> information (board rev/pins/etc) decided which one should be used.
> >>>
> >>> so what you are suggesting is that we customize psu_init_gpl() to the
> >>> target (ie, have an updated xsa file with whatever config we need for
> >>> this system).  
> >>
> >> In products yes.
> >>
> >>>
> >>> what I fail to grasp is why we can't take a step forward and do what
> >>> we do for other architectures in u-boot. and what fsbl already does by
> >>> config.
> >>>
> >>> I mean, why not? do you foresee any integration issues with the
> >>> current bootflow?
> >>
> >> Not really if you are willing to write that code just go for it.
> >> I think it is definitely feasible to do so but I haven't seen any
> >> algorithm to do it in generic way even there must be one behind in
> >> xilinx tools. There are a lot of types of memories which you can
> >> configure in xilinx design tools but it won't be any problem to limit it
> >> to some.
> > 
> > ok. unfortunately I am not sure when - we have tight deadlines on this
> > product so we will just update psu_init().
> > 
> > In fact, and this is relevant as well to keep in mind for a future
> > TO-DO list, the issue we were having was not due to the dynamic
> > configuration of DDR but to the fact that the DDR on this product is
> > ECC (and ECC, when enabled in the DDR controller must be initialized
> > _after_ psu_init() has run - which is something that FSBL does but SPL
> > doesnt).
> > 
> > So we will just disable ECC for the time being and will send a patch
> > once we come around to do it (or with a bit of luck someone will beat
> > me to it and I will just pick it up from the next u-boot release)
> 
> On Zynq ECC should be supported already. Take a look at ddrc_init() but
> I would have to look at details how is DDR initialized because this is
> clearing just 1MB which is shadow by OCM at this stage.
> 
> I would expect that procedure will be the same. Just find out that DDR
> controller is working in ECC mode and I hope that for initialization any
> DMA is used.

yeah, awesome. thanks for the info.

> 
> Thanks,
> Michal