Bug: qemu_arm64: Cannot access the second flash bank

Bug: qemu_arm64: Cannot access the second flash bank

[Robin Randhawa]

Hi folks.

[CC'ing some hopefully relevant folks].

As of:

commit 0ba41ce1b7816c229cc19e0621148b98f990cb68
libfdt: return correct value if #size-cells property is not present

.. accesses to the second flash bank on the qemu_arm64 virtual board
appear broken.

To demonstrate, consider that the physical memory map for the 2 flash
banks is:

Bank 1: 0x0000_0000 - 0x03FC_0000
Bank 2: 0x0400_0000 - 0x7FC0_0000

Now, consider the abbreviated output of the flinfo command pre and post
the above commit:

Pre:
===

=> flinfo

Bank # 1: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
  Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
  Erase timeout: 16384 ms, write timeout: 3 ms
  Buffer write timeout: 3 ms, buffer size: 2048 bytes

  Sector Start Addresses:
  00000000   RO   00040000   RO   00080000   RO   000C0000        00100000
  00140000        00180000        001C0000        00200000        00240000
  .
  .
  03E80000        03EC0000        03F00000        03F40000        03F80000
  03FC0000

Bank # 2: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
  Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
  Erase timeout: 16384 ms, write timeout: 3 ms
  Buffer write timeout: 3 ms, buffer size: 2048 bytes

  Sector Start Addresses:
  04000000   RO   04040000        04080000        040C0000        04100000
  04140000        04180000        041C0000        04200000        04240000
  .
  .
  07E80000        07EC0000        07F00000        07F40000        07F80000
  07FC0000

Post:
====

=> flinfo

Bank # 1: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
  Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
  Erase timeout: 16384 ms, write timeout: 3 ms
  Buffer write timeout: 3 ms, buffer size: 2048 bytes

  Sector Start Addresses:
  00000000   RO   00040000   RO   00080000   RO   000C0000        00100000
  00140000        00180000        001C0000        00200000        00240000
  .
  .
  03E80000        03EC0000        03F00000        03F40000        03F80000
  03FC0000

Bank # 2: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
  Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
  Erase timeout: 16384 ms, write timeout: 3 ms
  Buffer write timeout: 3 ms, buffer size: 2048 bytes

  Sector Start Addresses:
  400000000000000        400000000040000        400000000080000        4000000000C0000        400000000100000
  400000000140000        400000000180000        4000000001C0000        400000000200000        400000000240000
  .
  .
 
400000003E80000        400000003EC0000        400000003F00000        40
0000003F40000        400000003F80000
  400000003FC0000

As a result, the second bank is unusable for environment stores
(CONFIG_ENV_ADDR is 0x4000000):

=> saveenv
Saving Environment to Flash... Error: start and/or end address not on
sector boundary
Error: start and/or end address not on sector boundary
Failed (1)

Rewinding the u-boot repo to before this commit fixes the problem.

Manually (uncleanly) reverting the commit and it's dependent commits
fixes the problem.

Here are the HEAD commits from the relevant repos that I used for the data above:

qemu: commit dd5b0f95490883cd8bc7d070db8de70d5c979cbc
u-boot: commit 6cb87cbb1475f668689f95911d1521ee6ba7f55c

Here is the qemu invocation I used:

$ dd if=/dev/zero of=./flash0-with-uboot.img bs=1M count=64 && dd if=/path/to/u-boot.bin of=./flash0-with-uboot.img conv=notrunc
$ qemu-system-aarch64 -M virt -cpu cortex-a53 -m 1024M -nographic -drive if=pflash,format=raw,index=0,file=flash0-with-uboot.img  -drive if=pflash,format=raw,index=1,file=flash1.img

I'm happy to help test any fixes if and as needed.

Cheers,
Robin

Bug: qemu_arm64: Cannot access the second flash bank

[Matthias Brugger]

Hi Robin,

On 01/01/2020 19:20, Robin Randhawa wrote:
> Hi folks.
> 
> [CC'ing some hopefully relevant folks].
> 
> As of:
> 
> commit 0ba41ce1b7816c229cc19e0621148b98f990cb68
> libfdt: return correct value if #size-cells property is not present
> 
> .. accesses to the second flash bank on the qemu_arm64 virtual board
> appear broken.
> 

Can you pinpoint me to where I can find the DTS used by U-boot.

> To demonstrate, consider that the physical memory map for the 2 flash
> banks is:
> 
> Bank 1: 0x0000_0000 - 0x03FC_0000
> Bank 2: 0x0400_0000 - 0x7FC0_0000
> 
> Now, consider the abbreviated output of the flinfo command pre and post
> the above commit:
> 
> Pre:
> ===
> 
> => flinfo
> 
> Bank # 1: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
>   Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
>   Erase timeout: 16384 ms, write timeout: 3 ms
>   Buffer write timeout: 3 ms, buffer size: 2048 bytes
> 
>   Sector Start Addresses:
>   00000000   RO   00040000   RO   00080000   RO   000C0000        00100000
>   00140000        00180000        001C0000        00200000        00240000
>   .
>   .
>   03E80000        03EC0000        03F00000        03F40000        03F80000
>   03FC0000
> 
> Bank # 2: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
>   Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
>   Erase timeout: 16384 ms, write timeout: 3 ms
>   Buffer write timeout: 3 ms, buffer size: 2048 bytes
> 
>   Sector Start Addresses:
>   04000000   RO   04040000        04080000        040C0000        04100000
>   04140000        04180000        041C0000        04200000        04240000
>   .
>   .
>   07E80000        07EC0000        07F00000        07F40000        07F80000
>   07FC0000
> 
> Post:
> ====
> 
> => flinfo
> 
> Bank # 1: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
>   Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
>   Erase timeout: 16384 ms, write timeout: 3 ms
>   Buffer write timeout: 3 ms, buffer size: 2048 bytes
> 
>   Sector Start Addresses:
>   00000000   RO   00040000   RO   00080000   RO   000C0000        00100000
>   00140000        00180000        001C0000        00200000        00240000
>   .
>   .
>   03E80000        03EC0000        03F00000        03F40000        03F80000
>   03FC0000
> 
> Bank # 2: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
>   Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
>   Erase timeout: 16384 ms, write timeout: 3 ms
>   Buffer write timeout: 3 ms, buffer size: 2048 bytes
> 
>   Sector Start Addresses:
>   400000000000000        400000000040000        400000000080000        4000000000C0000        400000000100000
>   400000000140000        400000000180000        4000000001C0000        400000000200000        400000000240000
>   .
>   .
>  
> 400000003E80000        400000003EC0000        400000003F00000        40
> 0000003F40000        400000003F80000
>   400000003FC0000
> 
> As a result, the second bank is unusable for environment stores
> (CONFIG_ENV_ADDR is 0x4000000):
> 
> => saveenv
> Saving Environment to Flash... Error: start and/or end address not on
> sector boundary
> Error: start and/or end address not on sector boundary
> Failed (1)
> 
> Rewinding the u-boot repo to before this commit fixes the problem.
> 
> Manually (uncleanly) reverting the commit and it's dependent commits
> fixes the problem.
> 
> Here are the HEAD commits from the relevant repos that I used for the data above:
> 
> qemu: commit dd5b0f95490883cd8bc7d070db8de70d5c979cbc
> u-boot: commit 6cb87cbb1475f668689f95911d1521ee6ba7f55c
> 
> Here is the qemu invocation I used:
> 
> $ dd if=/dev/zero of=./flash0-with-uboot.img bs=1M count=64 && dd if=/path/to/u-boot.bin of=./flash0-with-uboot.img conv=notrunc
> $ qemu-system-aarch64 -M virt -cpu cortex-a53 -m 1024M -nographic -drive if=pflash,format=raw,index=0,file=flash0-with-uboot.img  -drive if=pflash,format=raw,index=1,file=flash1.img
> 

I tried to run that myself but wasn't able to see any output. Which U-Boot
config do you use? rpi_3_defconfig?

Regards,
Matthias

> I'm happy to help test any fixes if and as needed.
> 
> Cheers,
> Robin
> 

Bug: qemu_arm64: Cannot access the second flash bank

[Robin Randhawa]

Hi Matthias.

On Thu, 2020-01-09 at 12:12 +0100, Matthias Brugger wrote:

[...]

> Can you pinpoint me to where I can find the DTS used by U-boot.

As per my understanding the DTB for this virtual platform is generated
by qemu and handed to u-boot.

I dumped the DTB to the host filesystem using GDB and 'dump mem' then
dtc'd it to get the DTS. 

It's at:
https://pastebin.ubuntu.com/p/2KzPKdxddx/


> I tried to run that myself but wasn't able to see any output. Which
> U-Boot
> config do you use? rpi_3_defconfig?

I hit this problem for the qemu_arm64 u-boot platform target for which
I used qemu_arm64_defconfig [1].

Let me know if you need more info.

Cheers,
Robin

[1]: https://github.com/u-boot/u-boot/blob/master/configs/qemu_arm64_defconfig

Bug: qemu_arm64: Cannot access the second flash bank

[Matthias Brugger]

[adding Stefan as the maintainer of drivers/mtd/cfi_*]

On 09/01/2020 14:11, Robin Randhawa wrote:
> Hi Matthias.
> 
> On Thu, 2020-01-09 at 12:12 +0100, Matthias Brugger wrote:
> 
> [...]
> 
>> Can you pinpoint me to where I can find the DTS used by U-boot.
> 
> As per my understanding the DTB for this virtual platform is generated
> by qemu and handed to u-boot.
> 
> I dumped the DTB to the host filesystem using GDB and 'dump mem' then
> dtc'd it to get the DTS. 

Site note, I think you can do that in U-Boot like this:
fdt addr <address-of-your-dtb>
fdt print

> 
> It's at:
> https://pastebin.ubuntu.com/p/2KzPKdxddx/
> 

Ok, what I found so far:
in hw/arm/virt.c virt_flash_fdt() it creates the reg property like this [0]:
qemu_fdt_setprop_sized_cells(vms->fdt, nodename, "reg",
          2, flashbase, 2, flashsize,
          2, flashbase + flashsize, 2, flashsize);

Which creates the flash node we see in the dts you shared:
reg = <0x0 0x0 0x0 0x4000000 0x0 0x4000000 0x0 0x4000000>;

Which is:
Bank1: Flashbase 0x0 0x0         Flashsize 0x0 0x4000000
Bank2: Flashbase 0x0 0x4000000   Flashsize 0x0 0x4000000

The property expects size-cells to be two, but U-Boot will use one cell if no
size-cells are defined in the device node (which is not the case) and therefor
will see

Bank1: Flashbase 0x0 0x0         Flashsize 0x4000000
Bank2: Flashbase 0x4000000 0x0   Flashsize 0x4000000

Which maches with the flinfo you mentioned.

Now the question is if this is a problem of U-Boots implementation of cfi_flash,
which expects size-cells to be present in the flash node as it otherwise assumes
one [1]. Or if this is a problem of qemu which creates a device tree specifying
the size-cells only in the parent node.

Unfortunately the device tree specification as I understand it, is not clear [2]
on this.

Peter, Stefan any thoughts?

Regards,
Matthias

[0]
https://git.qemu.org/?p=qemu.git;a=blob;f=hw/arm/virt.c;h=39ab5f47e0bdc33b9bcb6320fdc10d35396579b1;hb=HEAD#l991
[1] https://gitlab.denx.de/u-boot/u-boot/blob/master/drivers/mtd/cfi_flash.c#L2472
[2]
https://github.com/devicetree-org/devicetree-specification/blob/master/source/devicetree-basics.rst#address-cells-and-size-cells

> 
>> I tried to run that myself but wasn't able to see any output. Which
>> U-Boot
>> config do you use? rpi_3_defconfig?
> 
> I hit this problem for the qemu_arm64 u-boot platform target for which
> I used qemu_arm64_defconfig [1].
> 
> Let me know if you need more info.
> 
> Cheers,
> Robin
> 
> [1]: https://github.com/u-boot/u-boot/blob/master/configs/qemu_arm64_defconfig
> 
> 
> 

Bug: qemu_arm64: Cannot access the second flash bank

[Robin Randhawa]

On Thu, 2020-01-09 at 15:57 +0100, Matthias Brugger wrote:

[...]

> The property expects size-cells to be two, but U-Boot will use one
> cell if no
> size-cells are defined in the device node (which is not the case) and
> therefor
> will see
> 
> Bank1: Flashbase 0x0 0x0         Flashsize 0x4000000
> Bank2: Flashbase 0x4000000 0x0   Flashsize 0x4000000

My knowledge of DT is superficial. However, looking at the following
lines from the spec:

- A |spec|-compliant boot program shall supply #address-cells and
#size-cells on all nodes that have children.

- If missing, a client program should assume a default value of 2 for
#address-cells, and a value of 1 for #size-cells.

.. and contrasting with the root node and device node in question from
the DTS for this platform:

/ {
	interrupt-parent = <0x8001>;
	#size-cells = <0x2>;
	#address-cells = <0x2>;
	compatible = "linux,dummy-virt";
.
.

	flash at 0 {
		bank-width = <0x4>;
		reg = <0x0 0x0 0x0 0x4000000 0x0 0x4000000 0x0 0x4000000>;
		compatible = "cfi-flash";
	};

.. it seems to me that while the flash node is missing #size-cells,
given that #size-cells _is_ defined in the parent node ("the node that
has children") then that value (0x2) is the one u-boot should have used
but didn't.

Maybe the u-boot DT interpreting logic needs to check if the parent
node also does not specify #size-cells before making the assumption
that the value 1 is to be used ?

Thanks,
Robin

Bug: qemu_arm64: Cannot access the second flash bank

[Peter Maydell]

On Thu, 9 Jan 2020 at 13:11, Robin Randhawa <Robin.Randhawa@arm.com> wrote:
> I dumped the DTB to the host filesystem using GDB and 'dump mem' then
> dtc'd it to get the DTS.

The easier way to dump the DTB is to pass QEMU the extra
command line option -machine dumpdtb=file.dtb (plus all the
other arguments for the configuration you want). Instead of starting
the VM QEMU will write the dtb to the specified file and exit.

thanks
-- PMM

Bug: qemu_arm64: Cannot access the second flash bank

[Matthias Brugger]

Hi Robin,

On 09/01/2020 16:21, Robin Randhawa wrote:
> On Thu, 2020-01-09 at 15:57 +0100, Matthias Brugger wrote:
> 
> [...]
> 
>> The property expects size-cells to be two, but U-Boot will use one
>> cell if no
>> size-cells are defined in the device node (which is not the case) and
>> therefor
>> will see
>>
>> Bank1: Flashbase 0x0 0x0         Flashsize 0x4000000
>> Bank2: Flashbase 0x4000000 0x0   Flashsize 0x4000000
> 
> My knowledge of DT is superficial. However, looking at the following
> lines from the spec:
> 
> - A |spec|-compliant boot program shall supply #address-cells and
> #size-cells on all nodes that have children.
> 
> - If missing, a client program should assume a default value of 2 for
> #address-cells, and a value of 1 for #size-cells.
> 
> .. and contrasting with the root node and device node in question from
> the DTS for this platform:
> 
> / {
> 	interrupt-parent = <0x8001>;
> 	#size-cells = <0x2>;
> 	#address-cells = <0x2>;
> 	compatible = "linux,dummy-virt";
> .
> .
> 
> 	flash at 0 {
> 		bank-width = <0x4>;
> 		reg = <0x0 0x0 0x0 0x4000000 0x0 0x4000000 0x0 0x4000000>;
> 		compatible = "cfi-flash";
> 	};
> 
> .. it seems to me that while the flash node is missing #size-cells,
> given that #size-cells _is_ defined in the parent node ("the node that
> has children") then that value (0x2) is the one u-boot should have used
> but didn't.
> 
> Maybe the u-boot DT interpreting logic needs to check if the parent
> node also does not specify #size-cells before making the assumption
> that the value 1 is to be used ?
> 

Sorry for taking so long. Yes you are correct, I found the issue in U-Boot and
I'm working on a patch. Please stay tuned.

Regards,
Matthias

Re: Bug: qemu_arm64: Cannot access the second flash bank

[Matthias Brugger]

Hi Robin,

It's a long long time that you reported this issue.

I prepared a fix in qemu for it. Would you mind to try it out? You can find a 
branch with the fix on top here:
https://github.com/mbgg/qemu/tree/vrit-flash-dtb-bug

Basically I fix the reg property to reflect the fact that the size-cell is one.

Please let me know if that fixes the issue for you and I'll send the fix upstream.

Regards,
Matthias

On 01/01/2020 19:20, Robin Randhawa wrote:
> Hi folks.
> 
> [CC'ing some hopefully relevant folks].
> 
> As of:
> 
> commit 0ba41ce1b7816c229cc19e0621148b98f990cb68
> libfdt: return correct value if #size-cells property is not present
> 
> .. accesses to the second flash bank on the qemu_arm64 virtual board
> appear broken.
> 
> To demonstrate, consider that the physical memory map for the 2 flash
> banks is:
> 
> Bank 1: 0x0000_0000 - 0x03FC_0000
> Bank 2: 0x0400_0000 - 0x7FC0_0000
> 
> Now, consider the abbreviated output of the flinfo command pre and post
> the above commit:
> 
> Pre:
> ===
> 
> => flinfo
> 
> Bank # 1: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
>    Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
>    Erase timeout: 16384 ms, write timeout: 3 ms
>    Buffer write timeout: 3 ms, buffer size: 2048 bytes
> 
>    Sector Start Addresses:
>    00000000   RO   00040000   RO   00080000   RO   000C0000        00100000
>    00140000        00180000        001C0000        00200000        00240000
>    .
>    .
>    03E80000        03EC0000        03F00000        03F40000        03F80000
>    03FC0000
> 
> Bank # 2: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
>    Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
>    Erase timeout: 16384 ms, write timeout: 3 ms
>    Buffer write timeout: 3 ms, buffer size: 2048 bytes
> 
>    Sector Start Addresses:
>    04000000   RO   04040000        04080000        040C0000        04100000
>    04140000        04180000        041C0000        04200000        04240000
>    .
>    .
>    07E80000        07EC0000        07F00000        07F40000        07F80000
>    07FC0000
> 
> Post:
> ====
> 
> => flinfo
> 
> Bank # 1: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
>    Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
>    Erase timeout: 16384 ms, write timeout: 3 ms
>    Buffer write timeout: 3 ms, buffer size: 2048 bytes
> 
>    Sector Start Addresses:
>    00000000   RO   00040000   RO   00080000   RO   000C0000        00100000
>    00140000        00180000        001C0000        00200000        00240000
>    .
>    .
>    03E80000        03EC0000        03F00000        03F40000        03F80000
>    03FC0000
> 
> Bank # 2: CFI conformant flash (32 x 16)  Size: 64 MB in 256 Sectors
>    Intel Extended command set, Manufacturer ID: 0x89, Device ID: 0x0018
>    Erase timeout: 16384 ms, write timeout: 3 ms
>    Buffer write timeout: 3 ms, buffer size: 2048 bytes
> 
>    Sector Start Addresses:
>    400000000000000        400000000040000        400000000080000        4000000000C0000        400000000100000
>    400000000140000        400000000180000        4000000001C0000        400000000200000        400000000240000
>    .
>    .
>   
> 400000003E80000        400000003EC0000        400000003F00000        40
> 0000003F40000        400000003F80000
>    400000003FC0000
> 
> As a result, the second bank is unusable for environment stores
> (CONFIG_ENV_ADDR is 0x4000000):
> 
> => saveenv
> Saving Environment to Flash... Error: start and/or end address not on
> sector boundary
> Error: start and/or end address not on sector boundary
> Failed (1)
> 
> Rewinding the u-boot repo to before this commit fixes the problem.
> 
> Manually (uncleanly) reverting the commit and it's dependent commits
> fixes the problem.
> 
> Here are the HEAD commits from the relevant repos that I used for the data above:
> 
> qemu: commit dd5b0f95490883cd8bc7d070db8de70d5c979cbc
> u-boot: commit 6cb87cbb1475f668689f95911d1521ee6ba7f55c
> 
> Here is the qemu invocation I used:
> 
> $ dd if=/dev/zero of=./flash0-with-uboot.img bs=1M count=64 && dd if=/path/to/u-boot.bin of=./flash0-with-uboot.img conv=notrunc
> $ qemu-system-aarch64 -M virt -cpu cortex-a53 -m 1024M -nographic -drive if=pflash,format=raw,index=0,file=flash0-with-uboot.img  -drive if=pflash,format=raw,index=1,file=flash1.img
> 
> I'm happy to help test any fixes if and as needed.
> 
> Cheers,
> Robin
> 

Re: Bug: qemu_arm64: Cannot access the second flash bank

[Peter Maydell]

On Mon, 13 Sept 2021 at 10:31, Matthias Brugger <mbrugger@suse.com> wrote:
>
> Hi Robin,
>
> It's a long long time that you reported this issue.
>
> I prepared a fix in qemu for it. Would you mind to try it out? You can find a
> branch with the fix on top here:
> https://github.com/mbgg/qemu/tree/vrit-flash-dtb-bug
>
> Basically I fix the reg property to reflect the fact that the size-cell is one.

Isn't #size-cells here inherited from the fdt root node (where we
set it to 2) ?

thanks
-- PMM

Re: Bug: qemu_arm64: Cannot access the second flash bank

[Matthias Brugger]

On 13/09/2021 11:40, Peter Maydell wrote:
> On Mon, 13 Sept 2021 at 10:31, Matthias Brugger <mbrugger@suse.com> wrote:
>>
>> Hi Robin,
>>
>> It's a long long time that you reported this issue.
>>
>> I prepared a fix in qemu for it. Would you mind to try it out? You can find a
>> branch with the fix on top here:
>> https://github.com/mbgg/qemu/tree/vrit-flash-dtb-bug
>>
>> Basically I fix the reg property to reflect the fact that the size-cell is one.
> 
> Isn't #size-cells here inherited from the fdt root node (where we
> set it to 2) ?
> 

Yes, you are right. After a lot of digging I realized that this should be fixed, by:
492b9917c68 ("cfi_flash: Fix devicetree address determination")

ae6b33dcc34 ("dm: fix ofnode_read_addr/size_cells()")


Sorry for the noise.

Matthias