kwboot: Testing latest kwboot with Kirkwood SoC boards

kwboot: Testing latest kwboot with Kirkwood SoC boards

[Tony Dinh]

Hi Marek and Pali,

First off all, thanks for your hughe work on this. I have a few Armada
38x boards that I had quite a hard time running kwboot in the past.
Hopefully, I will have more luck with the new kwboot.

Here is a problem I've found running kwboot on the Zyxel NSA310S NAS
(Kirkwood 6702). The target is a Pogoplug V4 (Kirkwood 6192 SoC). Same
behavior was observed running kwboot on this Pogoplug V4 to boot the
NSA310S target.

- kwboot build version

./kwboot-2021/kwboot
kwboot version 2022.01-rc1-00034-gbc18582a14

- kwboot session log

./kwboot-2021/kwboot -t -p -B 115200 /dev/ttyUSB0 -b
uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
Patching image boot signature to UART
Aligning image header to Xmodem block size
Sending boot message. Please reboot the target...\
Waiting 2s and flushing tty
Sending boot image header (480 bytes)...
 26 % [....                                                                  ]
Done

U-Boot 2021.04-tld-1 (Jun 10 2021 - 20:46:14 -0700)
Pogoplug V4

SoC:   Kirkwood 88F6192_A1
DRAM:  128 MiB
NAND:  128 MiB
MMC:
Loading Environment from NAND... *** Warning - bad CRC, using default
environment
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet-controller@72000
88E1116 Initialized on ethernet-controller@72000
Hit any key to stop autoboot:  9
 8
 7

As you can see above, it looks like the u-boot header transfer was
completed but the u-boot image transfer did not start. Seems like
BootROM did not recognize the header is UART. The Pogoplug V4
continued to boot with the image on NAND (U-Boot 2021.04-tld-1). So
kwboot_xm_finish() was never executed. At this point we lost the
terminal, and could not interrupt the countdown. The system continued
booting and at one point, serial console output stopped with an error
message that indicated kwboot thinks it is still in xmodem mode (but
the BootROM actually started the NAND image).

xmodem: Connection timed out

Please let me know if there is any idea that you want me to try.

Thanks,
Tony

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Pali Rohár]

Hello!

On Thursday 04 November 2021 23:27:32 Tony Dinh wrote:
> Hi Marek and Pali,
> 
> First off all, thanks for your hughe work on this. I have a few Armada
> 38x boards that I had quite a hard time running kwboot in the past.
> Hopefully, I will have more luck with the new kwboot.
> 
> Here is a problem I've found running kwboot on the Zyxel NSA310S NAS
> (Kirkwood 6702). The target is a Pogoplug V4 (Kirkwood 6192 SoC). Same
> behavior was observed running kwboot on this Pogoplug V4 to boot the
> NSA310S target.
> 
> - kwboot build version
> 
> ./kwboot-2021/kwboot
> kwboot version 2022.01-rc1-00034-gbc18582a14
> 
> - kwboot session log
> 
> ./kwboot-2021/kwboot -t -p -B 115200 /dev/ttyUSB0 -b
> uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> Patching image boot signature to UART
> Aligning image header to Xmodem block size
> Sending boot message. Please reboot the target...\
> Waiting 2s and flushing tty
> Sending boot image header (480 bytes)...
>  26 % [....                                                                  ]
> Done
> 
> U-Boot 2021.04-tld-1 (Jun 10 2021 - 20:46:14 -0700)
> Pogoplug V4
> 
> SoC:   Kirkwood 88F6192_A1
> DRAM:  128 MiB
> NAND:  128 MiB
> MMC:
> Loading Environment from NAND... *** Warning - bad CRC, using default
> environment
> In:    serial
> Out:   serial
> Err:   serial
> Net:   eth0: ethernet-controller@72000
> 88E1116 Initialized on ethernet-controller@72000
> Hit any key to stop autoboot:  9
>  8
>  7
> 
> As you can see above, it looks like the u-boot header transfer was
> completed but the u-boot image transfer did not start. Seems like
> BootROM did not recognize the header is UART.

Yes, it looks like that kwbimage header was refused by the bootroom.

Do you have some older U-Boot binary in .kwb format which you can
successfully boot? Or are you able to boot that U-Boot binary via some
older version of kwboot?

I would like to inspect some working setup, so need some U-Boot binary
which can be successfully transferred.

> The Pogoplug V4
> continued to boot with the image on NAND (U-Boot 2021.04-tld-1). So
> kwboot_xm_finish() was never executed. At this point we lost the
> terminal, and could not interrupt the countdown. The system continued
> booting and at one point, serial console output stopped with an error
> message that indicated kwboot thinks it is still in xmodem mode (but
> the BootROM actually started the NAND image).
> 
> xmodem: Connection timed out
> 
> Please let me know if there is any idea that you want me to try.
> 
> Thanks,
> Tony

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Pali Rohár]

On Friday 05 November 2021 09:38:28 Pali Rohár wrote:
> Hello!
> 
> On Thursday 04 November 2021 23:27:32 Tony Dinh wrote:
> > Hi Marek and Pali,
> > 
> > First off all, thanks for your hughe work on this. I have a few Armada
> > 38x boards that I had quite a hard time running kwboot in the past.
> > Hopefully, I will have more luck with the new kwboot.
> > 
> > Here is a problem I've found running kwboot on the Zyxel NSA310S NAS
> > (Kirkwood 6702). The target is a Pogoplug V4 (Kirkwood 6192 SoC). Same
> > behavior was observed running kwboot on this Pogoplug V4 to boot the
> > NSA310S target.
> > 
> > - kwboot build version
> > 
> > ./kwboot-2021/kwboot
> > kwboot version 2022.01-rc1-00034-gbc18582a14
> > 
> > - kwboot session log
> > 
> > ./kwboot-2021/kwboot -t -p -B 115200 /dev/ttyUSB0 -b
> > uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> > Patching image boot signature to UART
> > Aligning image header to Xmodem block size
> > Sending boot message. Please reboot the target...\
> > Waiting 2s and flushing tty
> > Sending boot image header (480 bytes)...
> >  26 % [....                                                                  ]
> > Done
> > 
> > U-Boot 2021.04-tld-1 (Jun 10 2021 - 20:46:14 -0700)
> > Pogoplug V4
> > 
> > SoC:   Kirkwood 88F6192_A1
> > DRAM:  128 MiB
> > NAND:  128 MiB
> > MMC:
> > Loading Environment from NAND... *** Warning - bad CRC, using default
> > environment
> > In:    serial
> > Out:   serial
> > Err:   serial
> > Net:   eth0: ethernet-controller@72000
> > 88E1116 Initialized on ethernet-controller@72000
> > Hit any key to stop autoboot:  9
> >  8
> >  7
> > 
> > As you can see above, it looks like the u-boot header transfer was
> > completed but the u-boot image transfer did not start. Seems like
> > BootROM did not recognize the header is UART.
> 
> Yes, it looks like that kwbimage header was refused by the bootroom.

Tony, could you try following patch?

diff --git a/tools/kwboot.c b/tools/kwboot.c
index bacca1530110..2470906a38d1 100644
--- a/tools/kwboot.c
+++ b/tools/kwboot.c
@@ -1072,6 +1072,7 @@ kwboot_xmodem(int tty, const void *_img, size_t size, int baudrate)
 	size_t hdrsz;
 
 	hdrsz = kwbheader_size(img);
+	hdrsz += (KWBOOT_XM_BLKSZ - hdrsz % KWBOOT_XM_BLKSZ) % KWBOOT_XM_BLKSZ;
 
 	kwboot_printv("Waiting 2s and flushing tty\n");
 	sleep(2); /* flush isn't effective without it */
@@ -1083,12 +1084,13 @@ kwboot_xmodem(int tty, const void *_img, size_t size, int baudrate)
 	if (rc)
 		return rc;
 
-	img += hdrsz;
-	size -= hdrsz;
-
-	rc = kwboot_xmodem_one(tty, &pnum, 0, img, size, 0);
-	if (rc)
-		return rc;
+	if (hdrsz < size) {
+		img += hdrsz;
+		size -= hdrsz;
+		rc = kwboot_xmodem_one(tty, &pnum, 0, img, size, 0);
+		if (rc)
+			return rc;
+	}
 
 	rc = kwboot_xm_finish(tty);
 	if (rc)

> Do you have some older U-Boot binary in .kwb format which you can
> successfully boot? Or are you able to boot that U-Boot binary via some
> older version of kwboot?
> 
> I would like to inspect some working setup, so need some U-Boot binary
> which can be successfully transferred.
> 
> > The Pogoplug V4
> > continued to boot with the image on NAND (U-Boot 2021.04-tld-1). So
> > kwboot_xm_finish() was never executed. At this point we lost the
> > terminal, and could not interrupt the countdown. The system continued
> > booting and at one point, serial console output stopped with an error
> > message that indicated kwboot thinks it is still in xmodem mode (but
> > the BootROM actually started the NAND image).
> > 
> > xmodem: Connection timed out
> > 
> > Please let me know if there is any idea that you want me to try.
> > 
> > Thanks,
> > Tony

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Tony Dinh]

Hi Pali,

On Fri, Nov 5, 2021 at 3:19 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Friday 05 November 2021 09:38:28 Pali Rohár wrote:
> > Hello!
> >
> > On Thursday 04 November 2021 23:27:32 Tony Dinh wrote:
> > > Hi Marek and Pali,
> > >
> > > First off all, thanks for your hughe work on this. I have a few Armada
> > > 38x boards that I had quite a hard time running kwboot in the past.
> > > Hopefully, I will have more luck with the new kwboot.
> > >
> > > Here is a problem I've found running kwboot on the Zyxel NSA310S NAS
> > > (Kirkwood 6702). The target is a Pogoplug V4 (Kirkwood 6192 SoC). Same
> > > behavior was observed running kwboot on this Pogoplug V4 to boot the
> > > NSA310S target.
> > >
> > > - kwboot build version
> > >
> > > ./kwboot-2021/kwboot
> > > kwboot version 2022.01-rc1-00034-gbc18582a14
> > >
> > > - kwboot session log
> > >
> > > ./kwboot-2021/kwboot -t -p -B 115200 /dev/ttyUSB0 -b
> > > uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> > > Patching image boot signature to UART
> > > Aligning image header to Xmodem block size
> > > Sending boot message. Please reboot the target...\
> > > Waiting 2s and flushing tty
> > > Sending boot image header (480 bytes)...
> > >  26 % [....                                                                  ]
> > > Done
> > >
> > > U-Boot 2021.04-tld-1 (Jun 10 2021 - 20:46:14 -0700)
> > > Pogoplug V4
> > >
> > > SoC:   Kirkwood 88F6192_A1
> > > DRAM:  128 MiB
> > > NAND:  128 MiB
> > > MMC:
> > > Loading Environment from NAND... *** Warning - bad CRC, using default
> > > environment
> > > In:    serial
> > > Out:   serial
> > > Err:   serial
> > > Net:   eth0: ethernet-controller@72000
> > > 88E1116 Initialized on ethernet-controller@72000
> > > Hit any key to stop autoboot:  9
> > >  8
> > >  7
> > >
> > > As you can see above, it looks like the u-boot header transfer was
> > > completed but the u-boot image transfer did not start. Seems like
> > > BootROM did not recognize the header is UART.
> >
> > Yes, it looks like that kwbimage header was refused by the bootroom.
>
> Tony, could you try following patch?
>
> diff --git a/tools/kwboot.c b/tools/kwboot.c
> index bacca1530110..2470906a38d1 100644
> --- a/tools/kwboot.c
> +++ b/tools/kwboot.c
> @@ -1072,6 +1072,7 @@ kwboot_xmodem(int tty, const void *_img, size_t size, int baudrate)
>         size_t hdrsz;
>
>         hdrsz = kwbheader_size(img);
> +       hdrsz += (KWBOOT_XM_BLKSZ - hdrsz % KWBOOT_XM_BLKSZ) % KWBOOT_XM_BLKSZ;
>
>         kwboot_printv("Waiting 2s and flushing tty\n");
>         sleep(2); /* flush isn't effective without it */
> @@ -1083,12 +1084,13 @@ kwboot_xmodem(int tty, const void *_img, size_t size, int baudrate)
>         if (rc)
>                 return rc;
>
> -       img += hdrsz;
> -       size -= hdrsz;
> -
> -       rc = kwboot_xmodem_one(tty, &pnum, 0, img, size, 0);
> -       if (rc)
> -               return rc;
> +       if (hdrsz < size) {
> +               img += hdrsz;
> +               size -= hdrsz;
> +               rc = kwboot_xmodem_one(tty, &pnum, 0, img, size, 0);
> +               if (rc)
> +                       return rc;
> +       }
>
>         rc = kwboot_xm_finish(tty);
>         if (rc)
>

Thanks, the patch works great! Here is the log.

./kwboot-2021/kwboot
kwboot version 2022.01-rc1-00041-g2a5ad542e6-dirty

./kwboot-2021/kwboot -t -B 115200 /dev/ttyUSB0 -b
uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
Patching image boot signature to UART
Aligning image header to Xmodem block size
Sending boot message. Please reboot the target...\
Waiting 2s and flushing tty
Sending boot image header (512 bytes)...
 25 % [....                                                                  ]
Done
Sending boot image data (534200 bytes)...
  0 % [......................................................................]
<snip>
 98 % [............................................                          ]
Done
Finishing transfer
[Type Ctrl-\ + c to quit]


U-Boot 2021.07-tld-1 (Nov 02 2021 - 19:48:03 -0700)
Pogoplug V4
SoC:   Kirkwood 88F6192_A1
Model: Pogoplug v4
DRAM:  128 MiB
NAND:  128 MiB
MMC:   mvsdio@90000: 0
Loading Environment from NAND... *** Warning - bad CRC, using default
environment
In:    serial
Out:   serial
Err:   serial
Net:
Warning: ethernet-controller@72000 (eth0) using random MAC address -
12:1c:5b:05:ed:15
eth0: ethernet-controller@72000
Hit any key to stop autoboot:  0
Pogo_V4>

Also, I have several Kirkwood boards (with various old BootROM
versions) that I can run the kwboot tests on. Will keep you posted.

Thanks,
Tony




> > Do you have some older U-Boot binary in .kwb format which you can
> > successfully boot? Or are you able to boot that U-Boot binary via some
> > older version of kwboot?
> >
> > I would like to inspect some working setup, so need some U-Boot binary
> > which can be successfully transferred.
> >
> > > The Pogoplug V4
> > > continued to boot with the image on NAND (U-Boot 2021.04-tld-1). So
> > > kwboot_xm_finish() was never executed. At this point we lost the
> > > terminal, and could not interrupt the countdown. The system continued
> > > booting and at one point, serial console output stopped with an error
> > > message that indicated kwboot thinks it is still in xmodem mode (but
> > > the BootROM actually started the NAND image).
> > >
> > > xmodem: Connection timed out
> > >
> > > Please let me know if there is any idea that you want me to try.
> > >
> > > Thanks,
> > > Tony

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Pali Rohár]

Hello!

On Friday 05 November 2021 14:25:14 Tony Dinh wrote:
> Hi Pali,
> 
> On Fri, Nov 5, 2021 at 3:19 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Friday 05 November 2021 09:38:28 Pali Rohár wrote:
> > > Hello!
> > >
> > > On Thursday 04 November 2021 23:27:32 Tony Dinh wrote:
> > > > Hi Marek and Pali,
> > > >
> > > > First off all, thanks for your hughe work on this. I have a few Armada
> > > > 38x boards that I had quite a hard time running kwboot in the past.
> > > > Hopefully, I will have more luck with the new kwboot.
> > > >
> > > > Here is a problem I've found running kwboot on the Zyxel NSA310S NAS
> > > > (Kirkwood 6702). The target is a Pogoplug V4 (Kirkwood 6192 SoC). Same
> > > > behavior was observed running kwboot on this Pogoplug V4 to boot the
> > > > NSA310S target.
> > > >
> > > > - kwboot build version
> > > >
> > > > ./kwboot-2021/kwboot
> > > > kwboot version 2022.01-rc1-00034-gbc18582a14
> > > >
> > > > - kwboot session log
> > > >
> > > > ./kwboot-2021/kwboot -t -p -B 115200 /dev/ttyUSB0 -b
> > > > uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> > > > Patching image boot signature to UART
> > > > Aligning image header to Xmodem block size
> > > > Sending boot message. Please reboot the target...\
> > > > Waiting 2s and flushing tty
> > > > Sending boot image header (480 bytes)...
> > > >  26 % [....                                                                  ]
> > > > Done
> > > >
> > > > U-Boot 2021.04-tld-1 (Jun 10 2021 - 20:46:14 -0700)
> > > > Pogoplug V4
> > > >
> > > > SoC:   Kirkwood 88F6192_A1
> > > > DRAM:  128 MiB
> > > > NAND:  128 MiB
> > > > MMC:
> > > > Loading Environment from NAND... *** Warning - bad CRC, using default
> > > > environment
> > > > In:    serial
> > > > Out:   serial
> > > > Err:   serial
> > > > Net:   eth0: ethernet-controller@72000
> > > > 88E1116 Initialized on ethernet-controller@72000
> > > > Hit any key to stop autoboot:  9
> > > >  8
> > > >  7
> > > >
> > > > As you can see above, it looks like the u-boot header transfer was
> > > > completed but the u-boot image transfer did not start. Seems like
> > > > BootROM did not recognize the header is UART.
> > >
> > > Yes, it looks like that kwbimage header was refused by the bootroom.
> >
> > Tony, could you try following patch?
> >
> > diff --git a/tools/kwboot.c b/tools/kwboot.c
> > index bacca1530110..2470906a38d1 100644
> > --- a/tools/kwboot.c
> > +++ b/tools/kwboot.c
> > @@ -1072,6 +1072,7 @@ kwboot_xmodem(int tty, const void *_img, size_t size, int baudrate)
> >         size_t hdrsz;
> >
> >         hdrsz = kwbheader_size(img);
> > +       hdrsz += (KWBOOT_XM_BLKSZ - hdrsz % KWBOOT_XM_BLKSZ) % KWBOOT_XM_BLKSZ;
> >
> >         kwboot_printv("Waiting 2s and flushing tty\n");
> >         sleep(2); /* flush isn't effective without it */
> > @@ -1083,12 +1084,13 @@ kwboot_xmodem(int tty, const void *_img, size_t size, int baudrate)
> >         if (rc)
> >                 return rc;
> >
> > -       img += hdrsz;
> > -       size -= hdrsz;
> > -
> > -       rc = kwboot_xmodem_one(tty, &pnum, 0, img, size, 0);
> > -       if (rc)
> > -               return rc;
> > +       if (hdrsz < size) {
> > +               img += hdrsz;
> > +               size -= hdrsz;
> > +               rc = kwboot_xmodem_one(tty, &pnum, 0, img, size, 0);
> > +               if (rc)
> > +                       return rc;
> > +       }
> >
> >         rc = kwboot_xm_finish(tty);
> >         if (rc)
> >
> 
> Thanks, the patch works great! Here is the log.

Perfect! I did not think that I will detect and fix this issue at the
first shot.

I will send this patch with proper commit message.

> ./kwboot-2021/kwboot
> kwboot version 2022.01-rc1-00041-g2a5ad542e6-dirty
> 
> ./kwboot-2021/kwboot -t -B 115200 /dev/ttyUSB0 -b
> uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> Patching image boot signature to UART
> Aligning image header to Xmodem block size
> Sending boot message. Please reboot the target...\
> Waiting 2s and flushing tty
> Sending boot image header (512 bytes)...
>  25 % [....                                                                  ]
> Done
> Sending boot image data (534200 bytes)...
>   0 % [......................................................................]
> <snip>
>  98 % [............................................                          ]
> Done
> Finishing transfer
> [Type Ctrl-\ + c to quit]
> 
> 
> U-Boot 2021.07-tld-1 (Nov 02 2021 - 19:48:03 -0700)
> Pogoplug V4
> SoC:   Kirkwood 88F6192_A1
> Model: Pogoplug v4
> DRAM:  128 MiB
> NAND:  128 MiB
> MMC:   mvsdio@90000: 0
> Loading Environment from NAND... *** Warning - bad CRC, using default
> environment
> In:    serial
> Out:   serial
> Err:   serial
> Net:
> Warning: ethernet-controller@72000 (eth0) using random MAC address -
> 12:1c:5b:05:ed:15
> eth0: ethernet-controller@72000
> Hit any key to stop autoboot:  0
> Pogo_V4>
> 
> Also, I have several Kirkwood boards (with various old BootROM
> versions) that I can run the kwboot tests on. Will keep you posted.

Ok! Do you have some Kirkwood board with PCIe slot? If yes, I would like
to see dumps from config space of Kirkwood PCIe Root Port, see:
https://lore.kernel.org/u-boot/20211104154921.b6zxjpczj7t6qlct@pali/

> Thanks,
> Tony
> 
> 
> 
> 
> > > Do you have some older U-Boot binary in .kwb format which you can
> > > successfully boot? Or are you able to boot that U-Boot binary via some
> > > older version of kwboot?
> > >
> > > I would like to inspect some working setup, so need some U-Boot binary
> > > which can be successfully transferred.
> > >
> > > > The Pogoplug V4
> > > > continued to boot with the image on NAND (U-Boot 2021.04-tld-1). So
> > > > kwboot_xm_finish() was never executed. At this point we lost the
> > > > terminal, and could not interrupt the countdown. The system continued
> > > > booting and at one point, serial console output stopped with an error
> > > > message that indicated kwboot thinks it is still in xmodem mode (but
> > > > the BootROM actually started the NAND image).
> > > >
> > > > xmodem: Connection timed out
> > > >
> > > > Please let me know if there is any idea that you want me to try.
> > > >
> > > > Thanks,
> > > > Tony

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Tony Dinh]

Hi Pali,

On Fri, Nov 5, 2021 at 2:39 PM Pali Rohár <pali@kernel.org> wrote:
>
> Hello!
>
> On Friday 05 November 2021 14:25:14 Tony Dinh wrote:
> > Hi Pali,
> >
> > On Fri, Nov 5, 2021 at 3:19 AM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Friday 05 November 2021 09:38:28 Pali Rohár wrote:
> > > > Hello!
> > > >
> > > > On Thursday 04 November 2021 23:27:32 Tony Dinh wrote:
> > > > > Hi Marek and Pali,
> > > > >
> > > > > First off all, thanks for your hughe work on this. I have a few Armada
> > > > > 38x boards that I had quite a hard time running kwboot in the past.
> > > > > Hopefully, I will have more luck with the new kwboot.
> > > > >
> > > > > Here is a problem I've found running kwboot on the Zyxel NSA310S NAS
> > > > > (Kirkwood 6702). The target is a Pogoplug V4 (Kirkwood 6192 SoC). Same
> > > > > behavior was observed running kwboot on this Pogoplug V4 to boot the
> > > > > NSA310S target.
> > > > >
> > > > > - kwboot build version
> > > > >
> > > > > ./kwboot-2021/kwboot
> > > > > kwboot version 2022.01-rc1-00034-gbc18582a14
> > > > >
> > > > > - kwboot session log
> > > > >
> > > > > ./kwboot-2021/kwboot -t -p -B 115200 /dev/ttyUSB0 -b
> > > > > uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> > > > > Patching image boot signature to UART
> > > > > Aligning image header to Xmodem block size
> > > > > Sending boot message. Please reboot the target...\
> > > > > Waiting 2s and flushing tty
> > > > > Sending boot image header (480 bytes)...
> > > > >  26 % [....                                                                  ]
> > > > > Done
> > > > >
> > > > > U-Boot 2021.04-tld-1 (Jun 10 2021 - 20:46:14 -0700)
> > > > > Pogoplug V4
> > > > >
> > > > > SoC:   Kirkwood 88F6192_A1
> > > > > DRAM:  128 MiB
> > > > > NAND:  128 MiB
> > > > > MMC:
> > > > > Loading Environment from NAND... *** Warning - bad CRC, using default
> > > > > environment
> > > > > In:    serial
> > > > > Out:   serial
> > > > > Err:   serial
> > > > > Net:   eth0: ethernet-controller@72000
> > > > > 88E1116 Initialized on ethernet-controller@72000
> > > > > Hit any key to stop autoboot:  9
> > > > >  8
> > > > >  7
> > > > >
> > > > > As you can see above, it looks like the u-boot header transfer was
> > > > > completed but the u-boot image transfer did not start. Seems like
> > > > > BootROM did not recognize the header is UART.
> > > >
> > > > Yes, it looks like that kwbimage header was refused by the bootroom.
> > >
> > > Tony, could you try following patch?
> > >
> > > diff --git a/tools/kwboot.c b/tools/kwboot.c
> > > index bacca1530110..2470906a38d1 100644
> > > --- a/tools/kwboot.c
> > > +++ b/tools/kwboot.c
> > > @@ -1072,6 +1072,7 @@ kwboot_xmodem(int tty, const void *_img, size_t size, int baudrate)
> > >         size_t hdrsz;
> > >
> > >         hdrsz = kwbheader_size(img);
> > > +       hdrsz += (KWBOOT_XM_BLKSZ - hdrsz % KWBOOT_XM_BLKSZ) % KWBOOT_XM_BLKSZ;
> > >
> > >         kwboot_printv("Waiting 2s and flushing tty\n");
> > >         sleep(2); /* flush isn't effective without it */
> > > @@ -1083,12 +1084,13 @@ kwboot_xmodem(int tty, const void *_img, size_t size, int baudrate)
> > >         if (rc)
> > >                 return rc;
> > >
> > > -       img += hdrsz;
> > > -       size -= hdrsz;
> > > -
> > > -       rc = kwboot_xmodem_one(tty, &pnum, 0, img, size, 0);
> > > -       if (rc)
> > > -               return rc;
> > > +       if (hdrsz < size) {
> > > +               img += hdrsz;
> > > +               size -= hdrsz;
> > > +               rc = kwboot_xmodem_one(tty, &pnum, 0, img, size, 0);
> > > +               if (rc)
> > > +                       return rc;
> > > +       }
> > >
> > >         rc = kwboot_xm_finish(tty);
> > >         if (rc)
> > >
> >
> > Thanks, the patch works great! Here is the log.
>
> Perfect! I did not think that I will detect and fix this issue at the
> first shot.
>
> I will send this patch with proper commit message.
>
> > ./kwboot-2021/kwboot
> > kwboot version 2022.01-rc1-00041-g2a5ad542e6-dirty
> >
> > ./kwboot-2021/kwboot -t -B 115200 /dev/ttyUSB0 -b
> > uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> > Patching image boot signature to UART
> > Aligning image header to Xmodem block size
> > Sending boot message. Please reboot the target...\
> > Waiting 2s and flushing tty
> > Sending boot image header (512 bytes)...
> >  25 % [....                                                                  ]
> > Done
> > Sending boot image data (534200 bytes)...
> >   0 % [......................................................................]
> > <snip>
> >  98 % [............................................                          ]
> > Done
> > Finishing transfer
> > [Type Ctrl-\ + c to quit]
> >
> >
> > U-Boot 2021.07-tld-1 (Nov 02 2021 - 19:48:03 -0700)
> > Pogoplug V4
> > SoC:   Kirkwood 88F6192_A1
> > Model: Pogoplug v4
> > DRAM:  128 MiB
> > NAND:  128 MiB
> > MMC:   mvsdio@90000: 0
> > Loading Environment from NAND... *** Warning - bad CRC, using default
> > environment
> > In:    serial
> > Out:   serial
> > Err:   serial
> > Net:
> > Warning: ethernet-controller@72000 (eth0) using random MAC address -
> > 12:1c:5b:05:ed:15
> > eth0: ethernet-controller@72000
> > Hit any key to stop autoboot:  0
> > Pogo_V4>
> >
> > Also, I have several Kirkwood boards (with various old BootROM
> > versions) that I can run the kwboot tests on. Will keep you posted.
>
> Ok! Do you have some Kirkwood board with PCIe slot? If yes, I would like
> to see dumps from config space of Kirkwood PCIe Root Port, see:
> https://lore.kernel.org/u-boot/20211104154921.b6zxjpczj7t6qlct@pali/

I have these Kirwood boards with PCI:
- No slot (host bus for USB 3.0): Pogoplug V4 (6192), Zyxel NSA325v2
(6282). These 2 boards can be kwboot.
- Iomega iConnect (6281), with PCIe slot for Wifi card. This board
does not have kwboot booting support.

I'll take a look at your link above and get back to you about the
config space dumps.

By the way, I'm starting to look at the driver/pci/pci_mvebu.c to see
if it can be made to work with Kirkwood SoCs. I think there are many
differences in the addresses and memory space. I would appreciate it
if you have a general assessment whether I can use that driver for
Kirkwood.

Thanks,
Tony

> >
> >
> >
> >
> > > > Do you have some older U-Boot binary in .kwb format which you can
> > > > successfully boot? Or are you able to boot that U-Boot binary via some
> > > > older version of kwboot?
> > > >
> > > > I would like to inspect some working setup, so need some U-Boot binary
> > > > which can be successfully transferred.
> > > >
> > > > > The Pogoplug V4
> > > > > continued to boot with the image on NAND (U-Boot 2021.04-tld-1). So
> > > > > kwboot_xm_finish() was never executed. At this point we lost the
> > > > > terminal, and could not interrupt the countdown. The system continued
> > > > > booting and at one point, serial console output stopped with an error
> > > > > message that indicated kwboot thinks it is still in xmodem mode (but
> > > > > the BootROM actually started the NAND image).
> > > > >
> > > > > xmodem: Connection timed out
> > > > >
> > > > > Please let me know if there is any idea that you want me to try.
> > > > >
> > > > > Thanks,
> > > > > Tony

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Pali Rohár]

On Friday 05 November 2021 15:07:17 Tony Dinh wrote:
> > > Also, I have several Kirkwood boards (with various old BootROM
> > > versions) that I can run the kwboot tests on. Will keep you posted.
> >
> > Ok! Do you have some Kirkwood board with PCIe slot? If yes, I would like
> > to see dumps from config space of Kirkwood PCIe Root Port, see:
> > https://lore.kernel.org/u-boot/20211104154921.b6zxjpczj7t6qlct@pali/
> 
> I have these Kirwood boards with PCI:
> - No slot (host bus for USB 3.0): Pogoplug V4 (6192), Zyxel NSA325v2
> (6282). These 2 boards can be kwboot.
> - Iomega iConnect (6281), with PCIe slot for Wifi card. This board
> does not have kwboot booting support.

What do you mean that it 'does not have kwboot booting support'?
88F6281 is also Kirkwood and UART booting with kwboot should work.

> I'll take a look at your link above and get back to you about the
> config space dumps.
> 
> By the way, I'm starting to look at the driver/pci/pci_mvebu.c to see
> if it can be made to work with Kirkwood SoCs. I think there are many
> differences in the addresses and memory space. I would appreciate it
> if you have a general assessment whether I can use that driver for
> Kirkwood.

pci_mvebu.c should work with Kirkwood SoCs and also with all these
32-bit Marvell SoCs: Orion, Discovery, Kirkwood, Dove, A370, AXP, A375,
A38x and A39x. According to Functional Specifications all these SoCs
have common PCIe register set.

If there is any issue with it, I could try to look at it.

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Tony Dinh]

Hi Pali,

On Fri, Nov 5, 2021 at 3:15 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Friday 05 November 2021 15:07:17 Tony Dinh wrote:
> > > > Also, I have several Kirkwood boards (with various old BootROM
> > > > versions) that I can run the kwboot tests on. Will keep you posted.
> > >
> > > Ok! Do you have some Kirkwood board with PCIe slot? If yes, I would like
> > > to see dumps from config space of Kirkwood PCIe Root Port, see:
> > > https://lore.kernel.org/u-boot/20211104154921.b6zxjpczj7t6qlct@pali/
> >
> > I have these Kirwood boards with PCI:
> > - No slot (host bus for USB 3.0): Pogoplug V4 (6192), Zyxel NSA325v2
> > (6282). These 2 boards can be kwboot.
> > - Iomega iConnect (6281), with PCIe slot for Wifi card. This board
> > does not have kwboot booting support.
>
> What do you mean that it 'does not have kwboot booting support'?
> 88F6281 is also Kirkwood and UART booting with kwboot should work.

Most of the Kirkwood boards do have UART booting support. However, in
my past experience, some Kirkwood boxes did not work with kwboot
booting. It was observed experimentally that certain BootROM versions
(depending on the time of manufacturing) on the 88F6281 SoC have
problems with UART booting. But we have not proven this to be the real
reason. These boards failed UART booting (the behavior is like the
UART magic string handshake never occur):

Seagate Dockstar (all), Iomega iConnect (all), Sheevaplug (some models
probably do work), Seagate GoFlex Net (most boxes work, but a few
models don't, and they have a different BootROM version from ones that
do work).

> > I'll take a look at your link above and get back to you about the
> > config space dumps.
> >
> > By the way, I'm starting to look at the driver/pci/pci_mvebu.c to see
> > if it can be made to work with Kirkwood SoCs. I think there are many
> > differences in the addresses and memory space. I would appreciate it
> > if you have a general assessment whether I can use that driver for
> > Kirkwood.
>
> pci_mvebu.c should work with Kirkwood SoCs and also with all these
> 32-bit Marvell SoCs: Orion, Discovery, Kirkwood, Dove, A370, AXP, A375,
> A38x and A39x. According to Functional Specifications all these SoCs
> have common PCIe register set.

That's great to hear!

>
> If there is any issue with it, I could try to look at it.

At the moment, pci_mvebu.c is not included in the build for Kirkwood
boards because ./drivers/pci/Kconfig excludes it:

config PCI_MVEBU
bool "Enable Armada XP/38x PCIe driver"
depends on ARCH_MVEBU

When I removed the above  dependency, the build had errors. Because
different soc.h and cpu.h are brought into pci_mvebu.c when
ARCH_KIRWOOD is enabled and ARCH_MVEBU is disabled.

#include <asm/arch/cpu.h>
#include <asm/arch/soc.h>


Thanks,
Tony

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Pali Rohár]

On Friday 05 November 2021 16:36:47 Tony Dinh wrote:
> Hi Pali,
> 
> On Fri, Nov 5, 2021 at 3:15 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Friday 05 November 2021 15:07:17 Tony Dinh wrote:
> > > > > Also, I have several Kirkwood boards (with various old BootROM
> > > > > versions) that I can run the kwboot tests on. Will keep you posted.
> > > >
> > > > Ok! Do you have some Kirkwood board with PCIe slot? If yes, I would like
> > > > to see dumps from config space of Kirkwood PCIe Root Port, see:
> > > > https://lore.kernel.org/u-boot/20211104154921.b6zxjpczj7t6qlct@pali/
> > >
> > > I have these Kirwood boards with PCI:
> > > - No slot (host bus for USB 3.0): Pogoplug V4 (6192), Zyxel NSA325v2
> > > (6282). These 2 boards can be kwboot.
> > > - Iomega iConnect (6281), with PCIe slot for Wifi card. This board
> > > does not have kwboot booting support.
> >
> > What do you mean that it 'does not have kwboot booting support'?
> > 88F6281 is also Kirkwood and UART booting with kwboot should work.
> 
> Most of the Kirkwood boards do have UART booting support. However, in
> my past experience, some Kirkwood boxes did not work with kwboot
> booting. It was observed experimentally that certain BootROM versions
> (depending on the time of manufacturing) on the 88F6281 SoC have
> problems with UART booting. But we have not proven this to be the real
> reason. These boards failed UART booting (the behavior is like the
> UART magic string handshake never occur):
> 
> Seagate Dockstar (all), Iomega iConnect (all), Sheevaplug (some models
> probably do work), Seagate GoFlex Net (most boxes work, but a few
> models don't, and they have a different BootROM version from ones that
> do work).

Hmm... ok. Maybe some bootrom versions have broken UART booting.

During experiment with A385 I observed that it is needed to send
continuous stream of boot pattern without delay, so bootrom can properly
detect it and enter UART booting. But after bootrom is in UART booting
mode, it responds only when do not transmitting anything for some few
milliseconds.

So it is needed to solve two timing issues. First with upper bound (you
cannot use large delay as bootrom does not detect boot pattern) and
second with lower bound (you cannot use small delay as bootrom does not
answer). Plus another issue that linux kernel does provide asynchronous
tty API which could tell when output buffer was transmitted via UART.

If some bootrom versions are too much timing sensitive and you do not
know exact characteristic of it (and also of UART HW on the host) then
it could be hard or impossible to enter that UART boot mode.

I'm planning to rewrite kwboot code which is sending boot pattern to be
more precise on timings... So if you are interested in testing it, I
could do it in a way with more configurable delays... once I would have
some time for it.

You could try to use tools/mrvl_uart.sh instead of kwboot. It implements
also code for sending boot pattern. But it requires valid image with
UART signature, it does not support on-the-fly patching like kwboot.

> > > I'll take a look at your link above and get back to you about the
> > > config space dumps.
> > >
> > > By the way, I'm starting to look at the driver/pci/pci_mvebu.c to see
> > > if it can be made to work with Kirkwood SoCs. I think there are many
> > > differences in the addresses and memory space. I would appreciate it
> > > if you have a general assessment whether I can use that driver for
> > > Kirkwood.
> >
> > pci_mvebu.c should work with Kirkwood SoCs and also with all these
> > 32-bit Marvell SoCs: Orion, Discovery, Kirkwood, Dove, A370, AXP, A375,
> > A38x and A39x. According to Functional Specifications all these SoCs
> > have common PCIe register set.
> 
> That's great to hear!
> 
> >
> > If there is any issue with it, I could try to look at it.
> 
> At the moment, pci_mvebu.c is not included in the build for Kirkwood
> boards because ./drivers/pci/Kconfig excludes it:
> 
> config PCI_MVEBU
> bool "Enable Armada XP/38x PCIe driver"
> depends on ARCH_MVEBU
> 
> When I removed the above  dependency, the build had errors. Because
> different soc.h and cpu.h are brought into pci_mvebu.c when
> ARCH_KIRWOOD is enabled and ARCH_MVEBU is disabled.
> 
> #include <asm/arch/cpu.h>
> #include <asm/arch/soc.h>

So, it it needed to do some adjustment of SoC related code and defines.
I think the relevant parts are mapping of mbus windows.

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Tony Dinh]

Hi Pali,

On Fri, Nov 5, 2021 at 5:10 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Friday 05 November 2021 16:36:47 Tony Dinh wrote:
> > Hi Pali,
> >
> > On Fri, Nov 5, 2021 at 3:15 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Friday 05 November 2021 15:07:17 Tony Dinh wrote:
> > > > > > Also, I have several Kirkwood boards (with various old BootROM
> > > > > > versions) that I can run the kwboot tests on. Will keep you posted.
> > > > >
> > > > > Ok! Do you have some Kirkwood board with PCIe slot? If yes, I would like
> > > > > to see dumps from config space of Kirkwood PCIe Root Port, see:
> > > > > https://lore.kernel.org/u-boot/20211104154921.b6zxjpczj7t6qlct@pali/
> > > >
> > > > I have these Kirwood boards with PCI:
> > > > - No slot (host bus for USB 3.0): Pogoplug V4 (6192), Zyxel NSA325v2
> > > > (6282). These 2 boards can be kwboot.
> > > > - Iomega iConnect (6281), with PCIe slot for Wifi card. This board
> > > > does not have kwboot booting support.
> > >
> > > What do you mean that it 'does not have kwboot booting support'?
> > > 88F6281 is also Kirkwood and UART booting with kwboot should work.
> >
> > Most of the Kirkwood boards do have UART booting support. However, in
> > my past experience, some Kirkwood boxes did not work with kwboot
> > booting. It was observed experimentally that certain BootROM versions
> > (depending on the time of manufacturing) on the 88F6281 SoC have
> > problems with UART booting. But we have not proven this to be the real
> > reason. These boards failed UART booting (the behavior is like the
> > UART magic string handshake never occur):
> >
> > Seagate Dockstar (all), Iomega iConnect (all), Sheevaplug (some models
> > probably do work), Seagate GoFlex Net (most boxes work, but a few
> > models don't, and they have a different BootROM version from ones that
> > do work).
>
> Hmm... ok. Maybe some bootrom versions have broken UART booting.
>
> During experiment with A385 I observed that it is needed to send
> continuous stream of boot pattern without delay, so bootrom can properly
> detect it and enter UART booting. But after bootrom is in UART booting
> mode, it responds only when do not transmitting anything for some few
> milliseconds.
> So it is needed to solve two timing issues. First with upper bound (you
> cannot use large delay as bootrom does not detect boot pattern) and
> second with lower bound (you cannot use small delay as bootrom does not
> answer). Plus another issue that linux kernel does provide asynchronous
> tty API which could tell when output buffer was transmitted via UART.

That's exactly what I've found trying to boot the Thecus N2350 (Armada
385). I've tried various -q -s parameters but could not find the right
combination! OTOH, the Zyxel NAS326 (Armada 380) is OK with just the
default timing (still more work on my part in the u-boot image, but
kwboot started OK).

>
> If some bootrom versions are too much timing sensitive and you do not
> know exact characteristic of it (and also of UART HW on the host) then
> it could be hard or impossible to enter that UART boot mode.

I've always suspected the box UART HW is the reason for failure to
handshake, not the BootROM. But I'll try testing the old Kirkwood
boxes again with the new kwboot to be sure.

> I'm planning to rewrite kwboot code which is sending boot pattern to be
> more precise on timings... So if you are interested in testing it, I
> could do it in a way with more configurable delays... once I would have
> some time for it.

I'll be glad to test any new kwboot code you will have. My main
interest is the Armada 38x and all Kirkwood boards.

>
> You could try to use tools/mrvl_uart.sh instead of kwboot. It implements
> also code for sending boot pattern. But it requires valid image with
> UART signature, it does not support on-the-fly patching like kwboot.

That's what I did to boot the stock Thecus N2350 u-boot UART version.
An old version of this mrvl_uart.sh script has been on the net for
quite some time. But kwboot is more robust in the timing setup and
allows us to boot the final version that will be flashed.

> > > > I'll take a look at your link above and get back to you about the
> > > > config space dumps.
> > > >
> > > > By the way, I'm starting to look at the driver/pci/pci_mvebu.c to see
> > > > if it can be made to work with Kirkwood SoCs. I think there are many
> > > > differences in the addresses and memory space. I would appreciate it
> > > > if you have a general assessment whether I can use that driver for
> > > > Kirkwood.
> > >
> > > pci_mvebu.c should work with Kirkwood SoCs and also with all these
> > > 32-bit Marvell SoCs: Orion, Discovery, Kirkwood, Dove, A370, AXP, A375,
> > > A38x and A39x. According to Functional Specifications all these SoCs
> > > have common PCIe register set.
> >
> > That's great to hear!
> >
> > >
> > > If there is any issue with it, I could try to look at it.
> >
> > At the moment, pci_mvebu.c is not included in the build for Kirkwood
> > boards because ./drivers/pci/Kconfig excludes it:
> >
> > config PCI_MVEBU
> > bool "Enable Armada XP/38x PCIe driver"
> > depends on ARCH_MVEBU
> >
> > When I removed the above  dependency, the build had errors. Because
> > different soc.h and cpu.h are brought into pci_mvebu.c when
> > ARCH_KIRWOOD is enabled and ARCH_MVEBU is disabled.
> >
> > #include <asm/arch/cpu.h>
> > #include <asm/arch/soc.h>
>
> So, it it needed to do some adjustment of SoC related code and defines.
> I think the relevant parts are mapping of mbus windows.

I did look a bit at the mbus windows. There are some differences from
MVEBU, such as in arch/arm/dts/kirkwood.dtsi

pcie-mem-aperture = <0xe0000000 0x10000000>; /* 256 MiB memory space */
pcie-io-aperture  = <0xf2000000 0x100000>;   /*   1 MiB    I/O space */

But my knowledge in PCI drivers is practically nothing, just hacking
it :) So if you plan to make this driver work for Kirkwood, I'd be
happy to volunteer to be a tester.

Thanks,
Tony

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Pali Rohár]

Hello!

On Friday 05 November 2021 20:50:18 Tony Dinh wrote:
> Hi Pali,
> 
> On Fri, Nov 5, 2021 at 5:10 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Friday 05 November 2021 16:36:47 Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > On Fri, Nov 5, 2021 at 3:15 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Friday 05 November 2021 15:07:17 Tony Dinh wrote:
> > > > > > > Also, I have several Kirkwood boards (with various old BootROM
> > > > > > > versions) that I can run the kwboot tests on. Will keep you posted.
> > > > > >
> > > > > > Ok! Do you have some Kirkwood board with PCIe slot? If yes, I would like
> > > > > > to see dumps from config space of Kirkwood PCIe Root Port, see:
> > > > > > https://lore.kernel.org/u-boot/20211104154921.b6zxjpczj7t6qlct@pali/
> > > > >
> > > > > I have these Kirwood boards with PCI:
> > > > > - No slot (host bus for USB 3.0): Pogoplug V4 (6192), Zyxel NSA325v2
> > > > > (6282). These 2 boards can be kwboot.
> > > > > - Iomega iConnect (6281), with PCIe slot for Wifi card. This board
> > > > > does not have kwboot booting support.
> > > >
> > > > What do you mean that it 'does not have kwboot booting support'?
> > > > 88F6281 is also Kirkwood and UART booting with kwboot should work.
> > >
> > > Most of the Kirkwood boards do have UART booting support. However, in
> > > my past experience, some Kirkwood boxes did not work with kwboot
> > > booting. It was observed experimentally that certain BootROM versions
> > > (depending on the time of manufacturing) on the 88F6281 SoC have
> > > problems with UART booting. But we have not proven this to be the real
> > > reason. These boards failed UART booting (the behavior is like the
> > > UART magic string handshake never occur):
> > >
> > > Seagate Dockstar (all), Iomega iConnect (all), Sheevaplug (some models
> > > probably do work), Seagate GoFlex Net (most boxes work, but a few
> > > models don't, and they have a different BootROM version from ones that
> > > do work).
> >
> > Hmm... ok. Maybe some bootrom versions have broken UART booting.
> >
> > During experiment with A385 I observed that it is needed to send
> > continuous stream of boot pattern without delay, so bootrom can properly
> > detect it and enter UART booting. But after bootrom is in UART booting
> > mode, it responds only when do not transmitting anything for some few
> > milliseconds.
> > So it is needed to solve two timing issues. First with upper bound (you
> > cannot use large delay as bootrom does not detect boot pattern) and
> > second with lower bound (you cannot use small delay as bootrom does not
> > answer). Plus another issue that linux kernel does provide asynchronous
> > tty API which could tell when output buffer was transmitted via UART.
> 
> That's exactly what I've found trying to boot the Thecus N2350 (Armada
> 385). I've tried various -q -s parameters but could not find the right
> combination! OTOH, the Zyxel NAS326 (Armada 380) is OK with just the
> default timing (still more work on my part in the u-boot image, but
> kwboot started OK).
> 
> >
> > If some bootrom versions are too much timing sensitive and you do not
> > know exact characteristic of it (and also of UART HW on the host) then
> > it could be hard or impossible to enter that UART boot mode.
> 
> I've always suspected the box UART HW is the reason for failure to
> handshake, not the BootROM. But I'll try testing the old Kirkwood
> boxes again with the new kwboot to be sure.
> 
> > I'm planning to rewrite kwboot code which is sending boot pattern to be
> > more precise on timings... So if you are interested in testing it, I
> > could do it in a way with more configurable delays... once I would have
> > some time for it.
> 
> I'll be glad to test any new kwboot code you will have. My main
> interest is the Armada 38x and all Kirkwood boards.
> 
> >
> > You could try to use tools/mrvl_uart.sh instead of kwboot. It implements
> > also code for sending boot pattern. But it requires valid image with
> > UART signature, it does not support on-the-fly patching like kwboot.
> 
> That's what I did to boot the stock Thecus N2350 u-boot UART version.
> An old version of this mrvl_uart.sh script has been on the net for
> quite some time. But kwboot is more robust in the timing setup and
> allows us to boot the final version that will be flashed.

What could be interested is to try to use tools/mrvl_uart.sh script for
booting those Kirkwood boards which was said that have broken UART
booting. If tools/mrvl_uart.sh is able to trigger bootrom to switch to
UART or not. Last version of tools/mrvl_uart.sh is still in U-Boot
repository.

But as you said, kwboot is more robust and once kwboot would be to
transfer all images which tools/mrvl_uart.sh is able then I send request
to removal of tools/mrvl_uart.sh from U-Boot. As in U-Boot there is no
need to have two different tools which implements same functionality.

> > > > > I'll take a look at your link above and get back to you about the
> > > > > config space dumps.
> > > > >
> > > > > By the way, I'm starting to look at the driver/pci/pci_mvebu.c to see
> > > > > if it can be made to work with Kirkwood SoCs. I think there are many
> > > > > differences in the addresses and memory space. I would appreciate it
> > > > > if you have a general assessment whether I can use that driver for
> > > > > Kirkwood.
> > > >
> > > > pci_mvebu.c should work with Kirkwood SoCs and also with all these
> > > > 32-bit Marvell SoCs: Orion, Discovery, Kirkwood, Dove, A370, AXP, A375,
> > > > A38x and A39x. According to Functional Specifications all these SoCs
> > > > have common PCIe register set.
> > >
> > > That's great to hear!
> > >
> > > >
> > > > If there is any issue with it, I could try to look at it.
> > >
> > > At the moment, pci_mvebu.c is not included in the build for Kirkwood
> > > boards because ./drivers/pci/Kconfig excludes it:
> > >
> > > config PCI_MVEBU
> > > bool "Enable Armada XP/38x PCIe driver"
> > > depends on ARCH_MVEBU
> > >
> > > When I removed the above  dependency, the build had errors. Because
> > > different soc.h and cpu.h are brought into pci_mvebu.c when
> > > ARCH_KIRWOOD is enabled and ARCH_MVEBU is disabled.
> > >
> > > #include <asm/arch/cpu.h>
> > > #include <asm/arch/soc.h>
> >
> > So, it it needed to do some adjustment of SoC related code and defines.
> > I think the relevant parts are mapping of mbus windows.
> 
> I did look a bit at the mbus windows. There are some differences from
> MVEBU, such as in arch/arm/dts/kirkwood.dtsi
> 
> pcie-mem-aperture = <0xe0000000 0x10000000>; /* 256 MiB memory space */
> pcie-io-aperture  = <0xf2000000 0x100000>;   /*   1 MiB    I/O space */
> 
> But my knowledge in PCI drivers is practically nothing, just hacking
> it :) So if you plan to make this driver work for Kirkwood, I'd be
> happy to volunteer to be a tester.

I do not have Kirkwood hardware for testing. But it looks like that
mach-kirkwood has just different names for mbus window constants.

Here is simple (untested) patch which allows me to compile pci_mvebu.c
for ARCH_KIRKWOOD:

diff --git a/arch/arm/mach-kirkwood/cpu.c b/arch/arm/mach-kirkwood/cpu.c
index e9571298a824..80f893ab369a 100644
--- a/arch/arm/mach-kirkwood/cpu.c
+++ b/arch/arm/mach-kirkwood/cpu.c
@@ -54,11 +54,11 @@ unsigned int kw_winctrl_calcsize(unsigned int sizeval)
 
 static struct mbus_win windows[] = {
 	/* Window 0: PCIE MEM address space */
-	{ KW_DEFADR_PCI_MEM, 1024 * 1024 * 256,
+	{ KW_DEFADR_PCI_MEM, KW_DEFADR_PCI_MEM_SIZE,
 	  KWCPU_TARGET_PCIE, KWCPU_ATTR_PCIE_MEM },
 
 	/* Window 1: PCIE IO address space */
-	{ KW_DEFADR_PCI_IO, 1024 * 64,
+	{ KW_DEFADR_PCI_IO, KW_DEFADR_PCI_IO_SIZE,
 	  KWCPU_TARGET_PCIE, KWCPU_ATTR_PCIE_IO },
 
 	/* Window 2: NAND Flash address space */
diff --git a/arch/arm/mach-kirkwood/include/mach/cpu.h b/arch/arm/mach-kirkwood/include/mach/cpu.h
index ea42182cf9c6..71c546f9acf6 100644
--- a/arch/arm/mach-kirkwood/include/mach/cpu.h
+++ b/arch/arm/mach-kirkwood/include/mach/cpu.h
@@ -68,6 +68,9 @@ enum kwcpu_attrib {
 #define KW_DEFADR_SPIF		0xE8000000
 #define KW_DEFADR_BOOTROM	0xF8000000
 
+#define KW_DEFADR_PCI_MEM_SIZE	(1024 * 1024 * 256)
+#define KW_DEFADR_PCI_IO_SIZE	(1024 * 64)
+
 struct mbus_win {
 	u32 base;
 	u32 size;
diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index cc139af6cb57..71fac12257ad 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -256,12 +256,12 @@ config PCIE_IPROC
 	  Say Y here if you want to enable Broadcom iProc PCIe controller,
 
 config PCI_MVEBU
-	bool "Enable Armada XP/38x PCIe driver"
-	depends on ARCH_MVEBU
+	bool "Enable Kirkwood / Armada 370/XP/375/38x PCIe driver"
+	depends on (ARCH_KIRKWOOD || ARCH_MVEBU)
 	select MISC
 	help
 	  Say Y here if you want to enable PCIe controller support on
-	  Armada XP/38x SoCs.
+	  Kirkwood and Armada 370/XP/375/38x SoCs.
 
 config PCIE_DW_COMMON
 	bool
diff --git a/drivers/pci/pci_mvebu.c b/drivers/pci/pci_mvebu.c
index c575e9412b2a..4cc8d2014052 100644
--- a/drivers/pci/pci_mvebu.c
+++ b/drivers/pci/pci_mvebu.c
@@ -26,6 +26,14 @@
 #include <linux/ioport.h>
 #include <linux/mbus.h>
 
+#ifdef CONFIG_ARCH_KIRKWOOD
+#define SOC_REGS_PHY_BASE KW_REGS_PHY_BASE
+#define MBUS_PCI_MEM_BASE KW_DEFADR_PCI_MEM
+#define MBUS_PCI_IO_BASE KW_DEFADR_PCI_IO
+#define MBUS_PCI_MEM_SIZE KW_DEFADR_PCI_MEM_SIZE
+#define MBUS_PCI_IO_SIZE KW_DEFADR_PCI_IO_SIZE
+#endif
+
 DECLARE_GLOBAL_DATA_PTR;
 
 /* PCIe unit register offsets */
@@ -97,7 +105,6 @@ struct mvebu_pcie {
  * and 64K of I/O space when registered.
  */
 static void __iomem *mvebu_pcie_membase = (void __iomem *)MBUS_PCI_MEM_BASE;
-#define PCIE_MEM_SIZE	(128 << 20)
 static void __iomem *mvebu_pcie_iobase = (void __iomem *)MBUS_PCI_IO_BASE;
 
 static inline bool mvebu_pcie_link_up(struct mvebu_pcie *pcie)
@@ -433,14 +440,14 @@ static int mvebu_pcie_probe(struct udevice *dev)
 	mvebu_pcie_set_local_dev_nr(pcie, 1);
 
 	pcie->mem.start = (u32)mvebu_pcie_membase;
-	pcie->mem.end = pcie->mem.start + PCIE_MEM_SIZE - 1;
-	mvebu_pcie_membase += PCIE_MEM_SIZE;
+	pcie->mem.end = pcie->mem.start + MBUS_PCI_MEM_SIZE - 1;
+	mvebu_pcie_membase += MBUS_PCI_MEM_SIZE;
 
 	if (mvebu_mbus_add_window_by_id(pcie->mem_target, pcie->mem_attr,
 					(phys_addr_t)pcie->mem.start,
-					PCIE_MEM_SIZE)) {
+					MBUS_PCI_MEM_SIZE)) {
 		printf("PCIe unable to add mbus window for mem at %08x+%08x\n",
-		       (u32)pcie->mem.start, PCIE_MEM_SIZE);
+		       (u32)pcie->mem.start, MBUS_PCI_MEM_SIZE);
 	}
 
 	pcie->io.start = (u32)mvebu_pcie_iobase;
@@ -459,7 +466,7 @@ static int mvebu_pcie_probe(struct udevice *dev)
 
 	/* PCI memory space */
 	pci_set_region(hose->regions + 0, pcie->mem.start,
-		       pcie->mem.start, PCIE_MEM_SIZE, PCI_REGION_MEM);
+		       pcie->mem.start, MBUS_PCI_MEM_SIZE, PCI_REGION_MEM);
 	pci_set_region(hose->regions + 1,
 		       0, 0,
 		       gd->ram_size,
@@ -659,6 +666,7 @@ static int mvebu_pcie_bind(struct udevice *parent)
 static const struct udevice_id mvebu_pcie_ids[] = {
 	{ .compatible = "marvell,armada-xp-pcie" },
 	{ .compatible = "marvell,armada-370-pcie" },
+	{ .compatible = "marvell,kirkwood-pcie" },
 	{ }
 };
 

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Tony Dinh]

Hi Pali,

On Sat, Nov 6, 2021 at 3:57 AM Pali Rohár <pali@kernel.org> wrote:
>
> Hello!
>
> On Friday 05 November 2021 20:50:18 Tony Dinh wrote:
> > Hi Pali,
> >
> > On Fri, Nov 5, 2021 at 5:10 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Friday 05 November 2021 16:36:47 Tony Dinh wrote:
> > > > Hi Pali,
> > > >
> > > > On Fri, Nov 5, 2021 at 3:15 PM Pali Rohár <pali@kernel.org> wrote:
> > > > >
> > > > > On Friday 05 November 2021 15:07:17 Tony Dinh wrote:
> > > > > > > > Also, I have several Kirkwood boards (with various old BootROM
> > > > > > > > versions) that I can run the kwboot tests on. Will keep you posted.
> > > > > > >
> > > > > > > Ok! Do you have some Kirkwood board with PCIe slot? If yes, I would like
> > > > > > > to see dumps from config space of Kirkwood PCIe Root Port, see:
> > > > > > > https://lore.kernel.org/u-boot/20211104154921.b6zxjpczj7t6qlct@pali/
> > > > > >
> > > > > > I have these Kirwood boards with PCI:
> > > > > > - No slot (host bus for USB 3.0): Pogoplug V4 (6192), Zyxel NSA325v2
> > > > > > (6282). These 2 boards can be kwboot.
> > > > > > - Iomega iConnect (6281), with PCIe slot for Wifi card. This board
> > > > > > does not have kwboot booting support.
> > > > >
> > > > > What do you mean that it 'does not have kwboot booting support'?
> > > > > 88F6281 is also Kirkwood and UART booting with kwboot should work.
> > > >
> > > > Most of the Kirkwood boards do have UART booting support. However, in
> > > > my past experience, some Kirkwood boxes did not work with kwboot
> > > > booting. It was observed experimentally that certain BootROM versions
> > > > (depending on the time of manufacturing) on the 88F6281 SoC have
> > > > problems with UART booting. But we have not proven this to be the real
> > > > reason. These boards failed UART booting (the behavior is like the
> > > > UART magic string handshake never occur):
> > > >
> > > > Seagate Dockstar (all), Iomega iConnect (all), Sheevaplug (some models
> > > > probably do work), Seagate GoFlex Net (most boxes work, but a few
> > > > models don't, and they have a different BootROM version from ones that
> > > > do work).
> > >
> > > Hmm... ok. Maybe some bootrom versions have broken UART booting.
> > >
> > > During experiment with A385 I observed that it is needed to send
> > > continuous stream of boot pattern without delay, so bootrom can properly
> > > detect it and enter UART booting. But after bootrom is in UART booting
> > > mode, it responds only when do not transmitting anything for some few
> > > milliseconds.
> > > So it is needed to solve two timing issues. First with upper bound (you
> > > cannot use large delay as bootrom does not detect boot pattern) and
> > > second with lower bound (you cannot use small delay as bootrom does not
> > > answer). Plus another issue that linux kernel does provide asynchronous
> > > tty API which could tell when output buffer was transmitted via UART.
> >
> > That's exactly what I've found trying to boot the Thecus N2350 (Armada
> > 385). I've tried various -q -s parameters but could not find the right
> > combination! OTOH, the Zyxel NAS326 (Armada 380) is OK with just the
> > default timing (still more work on my part in the u-boot image, but
> > kwboot started OK).
> >
> > >
> > > If some bootrom versions are too much timing sensitive and you do not
> > > know exact characteristic of it (and also of UART HW on the host) then
> > > it could be hard or impossible to enter that UART boot mode.
> >
> > I've always suspected the box UART HW is the reason for failure to
> > handshake, not the BootROM. But I'll try testing the old Kirkwood
> > boxes again with the new kwboot to be sure.
> >
> > > I'm planning to rewrite kwboot code which is sending boot pattern to be
> > > more precise on timings... So if you are interested in testing it, I
> > > could do it in a way with more configurable delays... once I would have
> > > some time for it.
> >
> > I'll be glad to test any new kwboot code you will have. My main
> > interest is the Armada 38x and all Kirkwood boards.
> >
> > >
> > > You could try to use tools/mrvl_uart.sh instead of kwboot. It implements
> > > also code for sending boot pattern. But it requires valid image with
> > > UART signature, it does not support on-the-fly patching like kwboot.
> >
> > That's what I did to boot the stock Thecus N2350 u-boot UART version.
> > An old version of this mrvl_uart.sh script has been on the net for
> > quite some time. But kwboot is more robust in the timing setup and
> > allows us to boot the final version that will be flashed.
>
> What could be interested is to try to use tools/mrvl_uart.sh script for
> booting those Kirkwood boards which was said that have broken UART
> booting. If tools/mrvl_uart.sh is able to trigger bootrom to switch to
> UART or not. Last version of tools/mrvl_uart.sh is still in U-Boot
> repository.

Good point. I will try that script to boot the iConnect and other boards.

>
> But as you said, kwboot is more robust and once kwboot would be to
> transfer all images which tools/mrvl_uart.sh is able then I send request
> to removal of tools/mrvl_uart.sh from U-Boot. As in U-Boot there is no
> need to have two different tools which implements same functionality.
>
> > > > > > I'll take a look at your link above and get back to you about the
> > > > > > config space dumps.
> > > > > >
> > > > > > By the way, I'm starting to look at the driver/pci/pci_mvebu.c to see
> > > > > > if it can be made to work with Kirkwood SoCs. I think there are many
> > > > > > differences in the addresses and memory space. I would appreciate it
> > > > > > if you have a general assessment whether I can use that driver for
> > > > > > Kirkwood.
> > > > >
> > > > > pci_mvebu.c should work with Kirkwood SoCs and also with all these
> > > > > 32-bit Marvell SoCs: Orion, Discovery, Kirkwood, Dove, A370, AXP, A375,
> > > > > A38x and A39x. According to Functional Specifications all these SoCs
> > > > > have common PCIe register set.
> > > >
> > > > That's great to hear!
> > > >
> > > > >
> > > > > If there is any issue with it, I could try to look at it.
> > > >
> > > > At the moment, pci_mvebu.c is not included in the build for Kirkwood
> > > > boards because ./drivers/pci/Kconfig excludes it:
> > > >
> > > > config PCI_MVEBU
> > > > bool "Enable Armada XP/38x PCIe driver"
> > > > depends on ARCH_MVEBU
> > > >
> > > > When I removed the above  dependency, the build had errors. Because
> > > > different soc.h and cpu.h are brought into pci_mvebu.c when
> > > > ARCH_KIRWOOD is enabled and ARCH_MVEBU is disabled.
> > > >
> > > > #include <asm/arch/cpu.h>
> > > > #include <asm/arch/soc.h>
> > >
> > > So, it it needed to do some adjustment of SoC related code and defines.
> > > I think the relevant parts are mapping of mbus windows.
> >
> > I did look a bit at the mbus windows. There are some differences from
> > MVEBU, such as in arch/arm/dts/kirkwood.dtsi
> >
> > pcie-mem-aperture = <0xe0000000 0x10000000>; /* 256 MiB memory space */
> > pcie-io-aperture  = <0xf2000000 0x100000>;   /*   1 MiB    I/O space */
> >
> > But my knowledge in PCI drivers is practically nothing, just hacking
> > it :) So if you plan to make this driver work for Kirkwood, I'd be
> > happy to volunteer to be a tester.
>
> I do not have Kirkwood hardware for testing. But it looks like that
> mach-kirkwood has just different names for mbus window constants.
>
> Here is simple (untested) patch which allows me to compile pci_mvebu.c
> for ARCH_KIRKWOOD:
>
> diff --git a/arch/arm/mach-kirkwood/cpu.c b/arch/arm/mach-kirkwood/cpu.c
> index e9571298a824..80f893ab369a 100644
> --- a/arch/arm/mach-kirkwood/cpu.c
> +++ b/arch/arm/mach-kirkwood/cpu.c
> @@ -54,11 +54,11 @@ unsigned int kw_winctrl_calcsize(unsigned int sizeval)
>
>  static struct mbus_win windows[] = {
>         /* Window 0: PCIE MEM address space */
> -       { KW_DEFADR_PCI_MEM, 1024 * 1024 * 256,
> +       { KW_DEFADR_PCI_MEM, KW_DEFADR_PCI_MEM_SIZE,
>           KWCPU_TARGET_PCIE, KWCPU_ATTR_PCIE_MEM },
>
>         /* Window 1: PCIE IO address space */
> -       { KW_DEFADR_PCI_IO, 1024 * 64,
> +       { KW_DEFADR_PCI_IO, KW_DEFADR_PCI_IO_SIZE,
>           KWCPU_TARGET_PCIE, KWCPU_ATTR_PCIE_IO },
>
>         /* Window 2: NAND Flash address space */
> diff --git a/arch/arm/mach-kirkwood/include/mach/cpu.h b/arch/arm/mach-kirkwood/include/mach/cpu.h
> index ea42182cf9c6..71c546f9acf6 100644
> --- a/arch/arm/mach-kirkwood/include/mach/cpu.h
> +++ b/arch/arm/mach-kirkwood/include/mach/cpu.h
> @@ -68,6 +68,9 @@ enum kwcpu_attrib {
>  #define KW_DEFADR_SPIF         0xE8000000
>  #define KW_DEFADR_BOOTROM      0xF8000000
>
> +#define KW_DEFADR_PCI_MEM_SIZE (1024 * 1024 * 256)
> +#define KW_DEFADR_PCI_IO_SIZE  (1024 * 64)
> +
>  struct mbus_win {
>         u32 base;
>         u32 size;
> diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
> index cc139af6cb57..71fac12257ad 100644
> --- a/drivers/pci/Kconfig
> +++ b/drivers/pci/Kconfig
> @@ -256,12 +256,12 @@ config PCIE_IPROC
>           Say Y here if you want to enable Broadcom iProc PCIe controller,
>
>  config PCI_MVEBU
> -       bool "Enable Armada XP/38x PCIe driver"
> -       depends on ARCH_MVEBU
> +       bool "Enable Kirkwood / Armada 370/XP/375/38x PCIe driver"
> +       depends on (ARCH_KIRKWOOD || ARCH_MVEBU)
>         select MISC
>         help
>           Say Y here if you want to enable PCIe controller support on
> -         Armada XP/38x SoCs.
> +         Kirkwood and Armada 370/XP/375/38x SoCs.
>
>  config PCIE_DW_COMMON
>         bool
> diff --git a/drivers/pci/pci_mvebu.c b/drivers/pci/pci_mvebu.c
> index c575e9412b2a..4cc8d2014052 100644
> --- a/drivers/pci/pci_mvebu.c
> +++ b/drivers/pci/pci_mvebu.c
> @@ -26,6 +26,14 @@
>  #include <linux/ioport.h>
>  #include <linux/mbus.h>
>
> +#ifdef CONFIG_ARCH_KIRKWOOD
> +#define SOC_REGS_PHY_BASE KW_REGS_PHY_BASE
> +#define MBUS_PCI_MEM_BASE KW_DEFADR_PCI_MEM
> +#define MBUS_PCI_IO_BASE KW_DEFADR_PCI_IO
> +#define MBUS_PCI_MEM_SIZE KW_DEFADR_PCI_MEM_SIZE
> +#define MBUS_PCI_IO_SIZE KW_DEFADR_PCI_IO_SIZE
> +#endif
> +
>  DECLARE_GLOBAL_DATA_PTR;
>
>  /* PCIe unit register offsets */
> @@ -97,7 +105,6 @@ struct mvebu_pcie {
>   * and 64K of I/O space when registered.
>   */
>  static void __iomem *mvebu_pcie_membase = (void __iomem *)MBUS_PCI_MEM_BASE;
> -#define PCIE_MEM_SIZE  (128 << 20)
>  static void __iomem *mvebu_pcie_iobase = (void __iomem *)MBUS_PCI_IO_BASE;
>
>  static inline bool mvebu_pcie_link_up(struct mvebu_pcie *pcie)
> @@ -433,14 +440,14 @@ static int mvebu_pcie_probe(struct udevice *dev)
>         mvebu_pcie_set_local_dev_nr(pcie, 1);
>
>         pcie->mem.start = (u32)mvebu_pcie_membase;
> -       pcie->mem.end = pcie->mem.start + PCIE_MEM_SIZE - 1;
> -       mvebu_pcie_membase += PCIE_MEM_SIZE;
> +       pcie->mem.end = pcie->mem.start + MBUS_PCI_MEM_SIZE - 1;
> +       mvebu_pcie_membase += MBUS_PCI_MEM_SIZE;
>
>         if (mvebu_mbus_add_window_by_id(pcie->mem_target, pcie->mem_attr,
>                                         (phys_addr_t)pcie->mem.start,
> -                                       PCIE_MEM_SIZE)) {
> +                                       MBUS_PCI_MEM_SIZE)) {
>                 printf("PCIe unable to add mbus window for mem at %08x+%08x\n",
> -                      (u32)pcie->mem.start, PCIE_MEM_SIZE);
> +                      (u32)pcie->mem.start, MBUS_PCI_MEM_SIZE);
>         }
>
>         pcie->io.start = (u32)mvebu_pcie_iobase;
> @@ -459,7 +466,7 @@ static int mvebu_pcie_probe(struct udevice *dev)
>
>         /* PCI memory space */
>         pci_set_region(hose->regions + 0, pcie->mem.start,
> -                      pcie->mem.start, PCIE_MEM_SIZE, PCI_REGION_MEM);
> +                      pcie->mem.start, MBUS_PCI_MEM_SIZE, PCI_REGION_MEM);
>         pci_set_region(hose->regions + 1,
>                        0, 0,
>                        gd->ram_size,
> @@ -659,6 +666,7 @@ static int mvebu_pcie_bind(struct udevice *parent)
>  static const struct udevice_id mvebu_pcie_ids[] = {
>         { .compatible = "marvell,armada-xp-pcie" },
>         { .compatible = "marvell,armada-370-pcie" },
> +       { .compatible = "marvell,kirkwood-pcie" },
>         { }
>  };
>

Cool! I will give it a try and let you know.

Thanks,
Tony

Re: kwboot: Testing latest kwboot with Kirkwood SoC boards

[Tony Dinh]

Hi Pali,

I've applied your patch, and ran some tests. Looks like we got to the
bind, but it was not probing (or the probe died silently). Later, I
tried pci enum and got some errors during probing.

Here is the log (I added some debug printf to drivers/pci/pci_mvebu.c
and drivers/core/device.c to see the progress).

- U-Boot boot log:

U-Boot 2022.01-tld-1-00054-g52207514ba-dirty (Nov 06 2021 - 18:11:41 -0700)
Pogoplug V4

SoC:   Kirkwood 88F6281_A1
DRAM:  128 MiB
mvebu_pcie_bind: begin
mvebu_pcie_bind: got an available node
Bound device  to pcie@82000000
mvebu_pcie_bind: bound
mvebu_pcie_bind: exit 0
Bound device pcie@82000000 to mbus@f1000000
Bound device mbus@f1000000 to root_driver
Bound device ehci@50000 to ocp@f1000000
Bound device ethernet-controller@72000 to ocp@f1000000
Bound device sata@80000 to ocp@f1000000
Bound device mvsdio@90000.blk to mvsdio@90000
Bound device mvsdio@90000 to ocp@f1000000
Bound device ocp@f1000000 to root_driver
NAND:  128 MiB
MMC:   mvsdio@90000: 0
Loading Environment from NAND... *** Warning - bad CRC, using default
environment

In:    serial
Out:   serial
Err:   serial
Net:
Warning: ethernet-controller@72000 (eth0) using random MAC address -
66:43:1f:50:f0:e7
eth0: ethernet-controller@72000
Hit any key to stop autoboot:  0

Pogo_V4> pci
No such bus
Pogo_V4> pci 0
No such bus
Pogo_V4> pci 1
No such bus

Pogo_V4> pci enum
mvebu_pcie_probe:
Cannot add window '4:e8', conflicts with another window
PCIe unable to add mbus window for mem at 90000000+10000000
Cannot add window '4:e0', conflicts with another window
PCIe unable to add mbus window for IO at c0000000+00010000
mvebu_pcie_probe: exit 0
Bound device pci_0:0.0 to pcie0.0
Bound device xhci_pci to pci_0:0.0

Pogo_V4> pci regions
#   Bus start          Phys start         Size                Flags
0   0x0000000090000000 0x0000000090000000 0x0000000010000000  mem
1   0x0000000000000000 0x0000000000000000 0x0000000008000000  mem sysmem
2   0x00000000c0000000 0x00000000c0000000 0x0000000000010000  io io

- Linux dmesg (grep for pci and xhci)

[   13.163011][    T1] mvebu-pcie mbus@f1000000:pcie@82000000: host
bridge /mbus@f1000000/pcie@82000000 ranges:
[   13.163147][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
MEM 0x00f1040000..0x00f1041fff -> 0x0000040000
[   13.163217][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
[   13.163268][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
IO 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
[   13.163754][    T1] mvebu-pcie mbus@f1000000:pcie@82000000: PCI
host bridge to bus 0000:00
[   13.163791][    T1] pci_bus 0000:00: root bus resource [bus 00-ff]
[   13.163829][    T1] pci_bus 0000:00: root bus resource [mem
0xf1040000-0xf1041fff] (bus address [0x00040000-0x00041fff])
[   13.163862][    T1] pci_bus 0000:00: root bus resource [mem
0xe0000000-0xefffffff]
[   13.163892][    T1] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
[   13.164075][    T1] pci 0000:00:01.0: [11ab:6281] type 01 class 0x060400
[   13.164133][    T1] pci 0000:00:01.0: reg 0x38: [mem
0x00000000-0x000007ff pref]
[   13.165893][    T1] PCI: bus0: Fast back to back transfers disabled
[   13.165949][    T1] pci 0000:00:01.0: bridge configuration invalid
([bus 00-00]), reconfiguring
[   13.166276][    T1] pci 0000:01:00.0: [1b73:1009] type 00 class 0x0c0330
[   13.166334][    T1] pci 0000:01:00.0: reg 0x10: [mem
0x00000000-0x0000ffff 64bit]
[   13.166381][    T1] pci 0000:01:00.0: reg 0x18: [mem
0x00000000-0x00000fff 64bit]
[   13.166424][    T1] pci 0000:01:00.0: reg 0x20: [mem
0x00000000-0x00000fff 64bit]
[   13.166561][    T1] pci 0000:01:00.0: supports D1
[   13.166588][    T1] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
[   13.166685][    T1] pci 0000:01:00.0: 2.000 Gb/s available PCIe
bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable
of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
[   13.197158][    T1] PCI: bus1: Fast back to back transfers disabled
[   13.197208][    T1] pci_bus 0000:01: busn_res: [bus 01-ff] end is
updated to 01
[   13.197538][    T1] pci 0000:00:01.0: BAR 14: assigned [mem
0xe0000000-0xe00fffff]
[   13.197581][    T1] pci 0000:00:01.0: BAR 6: assigned [mem
0xe0100000-0xe01007ff pref]
[   13.197624][    T1] pci 0000:01:00.0: BAR 0: assigned [mem
0xe0000000-0xe000ffff 64bit]
[   13.197670][    T1] pci 0000:01:00.0: BAR 2: assigned [mem
0xe0010000-0xe0010fff 64bit]
[   13.197715][    T1] pci 0000:01:00.0: BAR 4: assigned [mem
0xe0011000-0xe0011fff 64bit]
[   13.197759][    T1] pci 0000:00:01.0: PCI bridge to [bus 01]
[   13.197792][    T1] pci 0000:00:01.0:   bridge window [mem
0xe0000000-0xe00fffff]
[   13.197985][    T1] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
[   13.198135][    T1] pci 0000:01:00.0: enabling device (0140 -> 0142)

[   15.619756][    T1] xhci_hcd 0000:01:00.0: xHCI Host Controller
[   15.625694][    T1] xhci_hcd 0000:01:00.0: new USB bus registered,
assigned bus number 2
[   15.635643][    T1] xhci_hcd 0000:01:00.0: hcc params 0x200073a1
hci version 0x100 quirks 0x0000000000080010
[   15.646477][    T1] xhci_hcd 0000:01:00.0: xHCI Host Controller
[   15.652432][    T1] xhci_hcd 0000:01:00.0: new USB bus registered,
assigned bus number 3
[   15.660560][    T1] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed

Note that this board PCI and USB 3.0 have been working in the Linux
kernel for quite many years.

- These are relevant configs used in the build (not sure if I need to
use CONFIG_DM_PCI_COMPAT?)

CONFIG_DM=y
CONFIG_CMD_PCI=y
CONFIG_PCI=y
CONFIG_PCI_MVEBU=y
CONFIG_PCI_ENDPOINT=y
CONFIG_DM_PCI_COMPAT=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_PCI=y
CONFIG_USB_XHCI_MVEBU=y

BTW, the topic is no longer kwboot, should we move this to another new
thread? i.e. Testing PCI MVEBU with Kirkwood SoCs.

Thanks,
Tony

On Sat, Nov 6, 2021 at 4:26 PM Tony Dinh <mibodhi@gmail.com> wrote:
>
> Hi Pali,
>
> On Sat, Nov 6, 2021 at 3:57 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > Hello!
> >
> > On Friday 05 November 2021 20:50:18 Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > On Fri, Nov 5, 2021 at 5:10 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Friday 05 November 2021 16:36:47 Tony Dinh wrote:
> > > > > Hi Pali,
> > > > >
> > > > > On Fri, Nov 5, 2021 at 3:15 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > >
> > > > > > On Friday 05 November 2021 15:07:17 Tony Dinh wrote:
> > > > > > > > > Also, I have several Kirkwood boards (with various old BootROM
> > > > > > > > > versions) that I can run the kwboot tests on. Will keep you posted.
> > > > > > > >
> > > > > > > > Ok! Do you have some Kirkwood board with PCIe slot? If yes, I would like
> > > > > > > > to see dumps from config space of Kirkwood PCIe Root Port, see:
> > > > > > > > https://lore.kernel.org/u-boot/20211104154921.b6zxjpczj7t6qlct@pali/
> > > > > > >
> > > > > > > I have these Kirwood boards with PCI:
> > > > > > > - No slot (host bus for USB 3.0): Pogoplug V4 (6192), Zyxel NSA325v2
> > > > > > > (6282). These 2 boards can be kwboot.
> > > > > > > - Iomega iConnect (6281), with PCIe slot for Wifi card. This board
> > > > > > > does not have kwboot booting support.
> > > > > >
> > > > > > What do you mean that it 'does not have kwboot booting support'?
> > > > > > 88F6281 is also Kirkwood and UART booting with kwboot should work.
> > > > >
> > > > > Most of the Kirkwood boards do have UART booting support. However, in
> > > > > my past experience, some Kirkwood boxes did not work with kwboot
> > > > > booting. It was observed experimentally that certain BootROM versions
> > > > > (depending on the time of manufacturing) on the 88F6281 SoC have
> > > > > problems with UART booting. But we have not proven this to be the real
> > > > > reason. These boards failed UART booting (the behavior is like the
> > > > > UART magic string handshake never occur):
> > > > >
> > > > > Seagate Dockstar (all), Iomega iConnect (all), Sheevaplug (some models
> > > > > probably do work), Seagate GoFlex Net (most boxes work, but a few
> > > > > models don't, and they have a different BootROM version from ones that
> > > > > do work).
> > > >
> > > > Hmm... ok. Maybe some bootrom versions have broken UART booting.
> > > >
> > > > During experiment with A385 I observed that it is needed to send
> > > > continuous stream of boot pattern without delay, so bootrom can properly
> > > > detect it and enter UART booting. But after bootrom is in UART booting
> > > > mode, it responds only when do not transmitting anything for some few
> > > > milliseconds.
> > > > So it is needed to solve two timing issues. First with upper bound (you
> > > > cannot use large delay as bootrom does not detect boot pattern) and
> > > > second with lower bound (you cannot use small delay as bootrom does not
> > > > answer). Plus another issue that linux kernel does provide asynchronous
> > > > tty API which could tell when output buffer was transmitted via UART.
> > >
> > > That's exactly what I've found trying to boot the Thecus N2350 (Armada
> > > 385). I've tried various -q -s parameters but could not find the right
> > > combination! OTOH, the Zyxel NAS326 (Armada 380) is OK with just the
> > > default timing (still more work on my part in the u-boot image, but
> > > kwboot started OK).
> > >
> > > >
> > > > If some bootrom versions are too much timing sensitive and you do not
> > > > know exact characteristic of it (and also of UART HW on the host) then
> > > > it could be hard or impossible to enter that UART boot mode.
> > >
> > > I've always suspected the box UART HW is the reason for failure to
> > > handshake, not the BootROM. But I'll try testing the old Kirkwood
> > > boxes again with the new kwboot to be sure.
> > >
> > > > I'm planning to rewrite kwboot code which is sending boot pattern to be
> > > > more precise on timings... So if you are interested in testing it, I
> > > > could do it in a way with more configurable delays... once I would have
> > > > some time for it.
> > >
> > > I'll be glad to test any new kwboot code you will have. My main
> > > interest is the Armada 38x and all Kirkwood boards.
> > >
> > > >
> > > > You could try to use tools/mrvl_uart.sh instead of kwboot. It implements
> > > > also code for sending boot pattern. But it requires valid image with
> > > > UART signature, it does not support on-the-fly patching like kwboot.
> > >
> > > That's what I did to boot the stock Thecus N2350 u-boot UART version.
> > > An old version of this mrvl_uart.sh script has been on the net for
> > > quite some time. But kwboot is more robust in the timing setup and
> > > allows us to boot the final version that will be flashed.
> >
> > What could be interested is to try to use tools/mrvl_uart.sh script for
> > booting those Kirkwood boards which was said that have broken UART
> > booting. If tools/mrvl_uart.sh is able to trigger bootrom to switch to
> > UART or not. Last version of tools/mrvl_uart.sh is still in U-Boot
> > repository.
>
> Good point. I will try that script to boot the iConnect and other boards.
>
> >
> > But as you said, kwboot is more robust and once kwboot would be to
> > transfer all images which tools/mrvl_uart.sh is able then I send request
> > to removal of tools/mrvl_uart.sh from U-Boot. As in U-Boot there is no
> > need to have two different tools which implements same functionality.
> >
> > > > > > > I'll take a look at your link above and get back to you about the
> > > > > > > config space dumps.
> > > > > > >
> > > > > > > By the way, I'm starting to look at the driver/pci/pci_mvebu.c to see
> > > > > > > if it can be made to work with Kirkwood SoCs. I think there are many
> > > > > > > differences in the addresses and memory space. I would appreciate it
> > > > > > > if you have a general assessment whether I can use that driver for
> > > > > > > Kirkwood.
> > > > > >
> > > > > > pci_mvebu.c should work with Kirkwood SoCs and also with all these
> > > > > > 32-bit Marvell SoCs: Orion, Discovery, Kirkwood, Dove, A370, AXP, A375,
> > > > > > A38x and A39x. According to Functional Specifications all these SoCs
> > > > > > have common PCIe register set.
> > > > >
> > > > > That's great to hear!
> > > > >
> > > > > >
> > > > > > If there is any issue with it, I could try to look at it.
> > > > >
> > > > > At the moment, pci_mvebu.c is not included in the build for Kirkwood
> > > > > boards because ./drivers/pci/Kconfig excludes it:
> > > > >
> > > > > config PCI_MVEBU
> > > > > bool "Enable Armada XP/38x PCIe driver"
> > > > > depends on ARCH_MVEBU
> > > > >
> > > > > When I removed the above  dependency, the build had errors. Because
> > > > > different soc.h and cpu.h are brought into pci_mvebu.c when
> > > > > ARCH_KIRWOOD is enabled and ARCH_MVEBU is disabled.
> > > > >
> > > > > #include <asm/arch/cpu.h>
> > > > > #include <asm/arch/soc.h>
> > > >
> > > > So, it it needed to do some adjustment of SoC related code and defines.
> > > > I think the relevant parts are mapping of mbus windows.
> > >
> > > I did look a bit at the mbus windows. There are some differences from
> > > MVEBU, such as in arch/arm/dts/kirkwood.dtsi
> > >
> > > pcie-mem-aperture = <0xe0000000 0x10000000>; /* 256 MiB memory space */
> > > pcie-io-aperture  = <0xf2000000 0x100000>;   /*   1 MiB    I/O space */
> > >
> > > But my knowledge in PCI drivers is practically nothing, just hacking
> > > it :) So if you plan to make this driver work for Kirkwood, I'd be
> > > happy to volunteer to be a tester.
> >
> > I do not have Kirkwood hardware for testing. But it looks like that
> > mach-kirkwood has just different names for mbus window constants.
> >
> > Here is simple (untested) patch which allows me to compile pci_mvebu.c
> > for ARCH_KIRKWOOD:
> >
> > diff --git a/arch/arm/mach-kirkwood/cpu.c b/arch/arm/mach-kirkwood/cpu.c
> > index e9571298a824..80f893ab369a 100644
> > --- a/arch/arm/mach-kirkwood/cpu.c
> > +++ b/arch/arm/mach-kirkwood/cpu.c
> > @@ -54,11 +54,11 @@ unsigned int kw_winctrl_calcsize(unsigned int sizeval)
> >
> >  static struct mbus_win windows[] = {
> >         /* Window 0: PCIE MEM address space */
> > -       { KW_DEFADR_PCI_MEM, 1024 * 1024 * 256,
> > +       { KW_DEFADR_PCI_MEM, KW_DEFADR_PCI_MEM_SIZE,
> >           KWCPU_TARGET_PCIE, KWCPU_ATTR_PCIE_MEM },
> >
> >         /* Window 1: PCIE IO address space */
> > -       { KW_DEFADR_PCI_IO, 1024 * 64,
> > +       { KW_DEFADR_PCI_IO, KW_DEFADR_PCI_IO_SIZE,
> >           KWCPU_TARGET_PCIE, KWCPU_ATTR_PCIE_IO },
> >
> >         /* Window 2: NAND Flash address space */
> > diff --git a/arch/arm/mach-kirkwood/include/mach/cpu.h b/arch/arm/mach-kirkwood/include/mach/cpu.h
> > index ea42182cf9c6..71c546f9acf6 100644
> > --- a/arch/arm/mach-kirkwood/include/mach/cpu.h
> > +++ b/arch/arm/mach-kirkwood/include/mach/cpu.h
> > @@ -68,6 +68,9 @@ enum kwcpu_attrib {
> >  #define KW_DEFADR_SPIF         0xE8000000
> >  #define KW_DEFADR_BOOTROM      0xF8000000
> >
> > +#define KW_DEFADR_PCI_MEM_SIZE (1024 * 1024 * 256)
> > +#define KW_DEFADR_PCI_IO_SIZE  (1024 * 64)
> > +
> >  struct mbus_win {
> >         u32 base;
> >         u32 size;
> > diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
> > index cc139af6cb57..71fac12257ad 100644
> > --- a/drivers/pci/Kconfig
> > +++ b/drivers/pci/Kconfig
> > @@ -256,12 +256,12 @@ config PCIE_IPROC
> >           Say Y here if you want to enable Broadcom iProc PCIe controller,
> >
> >  config PCI_MVEBU
> > -       bool "Enable Armada XP/38x PCIe driver"
> > -       depends on ARCH_MVEBU
> > +       bool "Enable Kirkwood / Armada 370/XP/375/38x PCIe driver"
> > +       depends on (ARCH_KIRKWOOD || ARCH_MVEBU)
> >         select MISC
> >         help
> >           Say Y here if you want to enable PCIe controller support on
> > -         Armada XP/38x SoCs.
> > +         Kirkwood and Armada 370/XP/375/38x SoCs.
> >
> >  config PCIE_DW_COMMON
> >         bool
> > diff --git a/drivers/pci/pci_mvebu.c b/drivers/pci/pci_mvebu.c
> > index c575e9412b2a..4cc8d2014052 100644
> > --- a/drivers/pci/pci_mvebu.c
> > +++ b/drivers/pci/pci_mvebu.c
> > @@ -26,6 +26,14 @@
> >  #include <linux/ioport.h>
> >  #include <linux/mbus.h>
> >
> > +#ifdef CONFIG_ARCH_KIRKWOOD
> > +#define SOC_REGS_PHY_BASE KW_REGS_PHY_BASE
> > +#define MBUS_PCI_MEM_BASE KW_DEFADR_PCI_MEM
> > +#define MBUS_PCI_IO_BASE KW_DEFADR_PCI_IO
> > +#define MBUS_PCI_MEM_SIZE KW_DEFADR_PCI_MEM_SIZE
> > +#define MBUS_PCI_IO_SIZE KW_DEFADR_PCI_IO_SIZE
> > +#endif
> > +
> >  DECLARE_GLOBAL_DATA_PTR;
> >
> >  /* PCIe unit register offsets */
> > @@ -97,7 +105,6 @@ struct mvebu_pcie {
> >   * and 64K of I/O space when registered.
> >   */
> >  static void __iomem *mvebu_pcie_membase = (void __iomem *)MBUS_PCI_MEM_BASE;
> > -#define PCIE_MEM_SIZE  (128 << 20)
> >  static void __iomem *mvebu_pcie_iobase = (void __iomem *)MBUS_PCI_IO_BASE;
> >
> >  static inline bool mvebu_pcie_link_up(struct mvebu_pcie *pcie)
> > @@ -433,14 +440,14 @@ static int mvebu_pcie_probe(struct udevice *dev)
> >         mvebu_pcie_set_local_dev_nr(pcie, 1);
> >
> >         pcie->mem.start = (u32)mvebu_pcie_membase;
> > -       pcie->mem.end = pcie->mem.start + PCIE_MEM_SIZE - 1;
> > -       mvebu_pcie_membase += PCIE_MEM_SIZE;
> > +       pcie->mem.end = pcie->mem.start + MBUS_PCI_MEM_SIZE - 1;
> > +       mvebu_pcie_membase += MBUS_PCI_MEM_SIZE;
> >
> >         if (mvebu_mbus_add_window_by_id(pcie->mem_target, pcie->mem_attr,
> >                                         (phys_addr_t)pcie->mem.start,
> > -                                       PCIE_MEM_SIZE)) {
> > +                                       MBUS_PCI_MEM_SIZE)) {
> >                 printf("PCIe unable to add mbus window for mem at %08x+%08x\n",
> > -                      (u32)pcie->mem.start, PCIE_MEM_SIZE);
> > +                      (u32)pcie->mem.start, MBUS_PCI_MEM_SIZE);
> >         }
> >
> >         pcie->io.start = (u32)mvebu_pcie_iobase;
> > @@ -459,7 +466,7 @@ static int mvebu_pcie_probe(struct udevice *dev)
> >
> >         /* PCI memory space */
> >         pci_set_region(hose->regions + 0, pcie->mem.start,
> > -                      pcie->mem.start, PCIE_MEM_SIZE, PCI_REGION_MEM);
> > +                      pcie->mem.start, MBUS_PCI_MEM_SIZE, PCI_REGION_MEM);
> >         pci_set_region(hose->regions + 1,
> >                        0, 0,
> >                        gd->ram_size,
> > @@ -659,6 +666,7 @@ static int mvebu_pcie_bind(struct udevice *parent)
> >  static const struct udevice_id mvebu_pcie_ids[] = {
> >         { .compatible = "marvell,armada-xp-pcie" },
> >         { .compatible = "marvell,armada-370-pcie" },
> > +       { .compatible = "marvell,kirkwood-pcie" },
> >         { }
> >  };
> >
>
> Cool! I will give it a try and let you know.
>
> Thanks,
> Tony

Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

Hello!

On Sunday 07 November 2021 12:56:37 Tony Dinh wrote:
> Hi Pali,
> 
> I've applied your patch, and ran some tests. Looks like we got to the
> bind, but it was not probing (or the probe died silently). Later, I
> tried pci enum and got some errors during probing.

It looks like that arch/arm/mach-kirkwood/cpu.c already setup PCIe
windows. Therefore when pci_mvebu.c tries it too, it fails on error
"conflicts with another window" which you see in the log.

Could you try to comment calling of "mvebu_mbus_add_window_by_id"
function in pci_mvebu.c? As PCIe window setup should be done exactly
once.

Also try to call 'pci 0' and 'pci 1' after the 'pci enum'.

> Here is the log (I added some debug printf to drivers/pci/pci_mvebu.c
> and drivers/core/device.c to see the progress).
> 
> - U-Boot boot log:
> 
> U-Boot 2022.01-tld-1-00054-g52207514ba-dirty (Nov 06 2021 - 18:11:41 -0700)
> Pogoplug V4
> 
> SoC:   Kirkwood 88F6281_A1
> DRAM:  128 MiB
> mvebu_pcie_bind: begin
> mvebu_pcie_bind: got an available node
> Bound device  to pcie@82000000
> mvebu_pcie_bind: bound
> mvebu_pcie_bind: exit 0
> Bound device pcie@82000000 to mbus@f1000000
> Bound device mbus@f1000000 to root_driver
> Bound device ehci@50000 to ocp@f1000000
> Bound device ethernet-controller@72000 to ocp@f1000000
> Bound device sata@80000 to ocp@f1000000
> Bound device mvsdio@90000.blk to mvsdio@90000
> Bound device mvsdio@90000 to ocp@f1000000
> Bound device ocp@f1000000 to root_driver
> NAND:  128 MiB
> MMC:   mvsdio@90000: 0
> Loading Environment from NAND... *** Warning - bad CRC, using default
> environment
> 
> In:    serial
> Out:   serial
> Err:   serial
> Net:
> Warning: ethernet-controller@72000 (eth0) using random MAC address -
> 66:43:1f:50:f0:e7
> eth0: ethernet-controller@72000
> Hit any key to stop autoboot:  0
> 
> Pogo_V4> pci
> No such bus
> Pogo_V4> pci 0
> No such bus
> Pogo_V4> pci 1
> No such bus
> 
> Pogo_V4> pci enum
> mvebu_pcie_probe:
> Cannot add window '4:e8', conflicts with another window
> PCIe unable to add mbus window for mem at 90000000+10000000
> Cannot add window '4:e0', conflicts with another window
> PCIe unable to add mbus window for IO at c0000000+00010000
> mvebu_pcie_probe: exit 0
> Bound device pci_0:0.0 to pcie0.0
> Bound device xhci_pci to pci_0:0.0
> 
> Pogo_V4> pci regions
> #   Bus start          Phys start         Size                Flags
> 0   0x0000000090000000 0x0000000090000000 0x0000000010000000  mem
> 1   0x0000000000000000 0x0000000000000000 0x0000000008000000  mem sysmem
> 2   0x00000000c0000000 0x00000000c0000000 0x0000000000010000  io io
> 
> - Linux dmesg (grep for pci and xhci)
> 
> [   13.163011][    T1] mvebu-pcie mbus@f1000000:pcie@82000000: host
> bridge /mbus@f1000000/pcie@82000000 ranges:
> [   13.163147][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
> MEM 0x00f1040000..0x00f1041fff -> 0x0000040000
> [   13.163217][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
> MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
> [   13.163268][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
> IO 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
> [   13.163754][    T1] mvebu-pcie mbus@f1000000:pcie@82000000: PCI
> host bridge to bus 0000:00
> [   13.163791][    T1] pci_bus 0000:00: root bus resource [bus 00-ff]
> [   13.163829][    T1] pci_bus 0000:00: root bus resource [mem
> 0xf1040000-0xf1041fff] (bus address [0x00040000-0x00041fff])
> [   13.163862][    T1] pci_bus 0000:00: root bus resource [mem
> 0xe0000000-0xefffffff]
> [   13.163892][    T1] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
> [   13.164075][    T1] pci 0000:00:01.0: [11ab:6281] type 01 class 0x060400
> [   13.164133][    T1] pci 0000:00:01.0: reg 0x38: [mem
> 0x00000000-0x000007ff pref]
> [   13.165893][    T1] PCI: bus0: Fast back to back transfers disabled
> [   13.165949][    T1] pci 0000:00:01.0: bridge configuration invalid
> ([bus 00-00]), reconfiguring
> [   13.166276][    T1] pci 0000:01:00.0: [1b73:1009] type 00 class 0x0c0330
> [   13.166334][    T1] pci 0000:01:00.0: reg 0x10: [mem
> 0x00000000-0x0000ffff 64bit]
> [   13.166381][    T1] pci 0000:01:00.0: reg 0x18: [mem
> 0x00000000-0x00000fff 64bit]
> [   13.166424][    T1] pci 0000:01:00.0: reg 0x20: [mem
> 0x00000000-0x00000fff 64bit]
> [   13.166561][    T1] pci 0000:01:00.0: supports D1
> [   13.166588][    T1] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
> [   13.166685][    T1] pci 0000:01:00.0: 2.000 Gb/s available PCIe
> bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable
> of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
> [   13.197158][    T1] PCI: bus1: Fast back to back transfers disabled
> [   13.197208][    T1] pci_bus 0000:01: busn_res: [bus 01-ff] end is
> updated to 01
> [   13.197538][    T1] pci 0000:00:01.0: BAR 14: assigned [mem
> 0xe0000000-0xe00fffff]
> [   13.197581][    T1] pci 0000:00:01.0: BAR 6: assigned [mem
> 0xe0100000-0xe01007ff pref]
> [   13.197624][    T1] pci 0000:01:00.0: BAR 0: assigned [mem
> 0xe0000000-0xe000ffff 64bit]
> [   13.197670][    T1] pci 0000:01:00.0: BAR 2: assigned [mem
> 0xe0010000-0xe0010fff 64bit]
> [   13.197715][    T1] pci 0000:01:00.0: BAR 4: assigned [mem
> 0xe0011000-0xe0011fff 64bit]
> [   13.197759][    T1] pci 0000:00:01.0: PCI bridge to [bus 01]
> [   13.197792][    T1] pci 0000:00:01.0:   bridge window [mem
> 0xe0000000-0xe00fffff]
> [   13.197985][    T1] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
> [   13.198135][    T1] pci 0000:01:00.0: enabling device (0140 -> 0142)
> 
> [   15.619756][    T1] xhci_hcd 0000:01:00.0: xHCI Host Controller
> [   15.625694][    T1] xhci_hcd 0000:01:00.0: new USB bus registered,
> assigned bus number 2
> [   15.635643][    T1] xhci_hcd 0000:01:00.0: hcc params 0x200073a1
> hci version 0x100 quirks 0x0000000000080010
> [   15.646477][    T1] xhci_hcd 0000:01:00.0: xHCI Host Controller
> [   15.652432][    T1] xhci_hcd 0000:01:00.0: new USB bus registered,
> assigned bus number 3
> [   15.660560][    T1] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
> 
> Note that this board PCI and USB 3.0 have been working in the Linux
> kernel for quite many years.
> 
> - These are relevant configs used in the build (not sure if I need to
> use CONFIG_DM_PCI_COMPAT?)
> 
> CONFIG_DM=y
> CONFIG_CMD_PCI=y
> CONFIG_PCI=y
> CONFIG_PCI_MVEBU=y
> CONFIG_PCI_ENDPOINT=y
> CONFIG_DM_PCI_COMPAT=y
> CONFIG_USB_XHCI_HCD=y
> CONFIG_USB_XHCI_PCI=y
> CONFIG_USB_XHCI_MVEBU=y

Following options should be enough:

CONFIG_CMD_PCI=y
CONFIG_PCI=y
# CONFIG_DM_PCI_COMPAT is not set
CONFIG_PCI_PNP=y
CONFIG_PCI_MVEBU=y

Option CONFIG_PCI_PNP=y is required to see endpoint cards. Without it
U-Boot would see only PCIe Root Ports. But for first basic tests it
should be enough.

> BTW, the topic is no longer kwboot, should we move this to another new
> thread? i.e. Testing PCI MVEBU with Kirkwood SoCs.

Changed :-)

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

Looks like it is working! But in a weird way. Please see my responses
in between, and also in the test log below.

On Sun, Nov 7, 2021 at 3:45 PM Pali Rohár <pali@kernel.org> wrote:
>
> Hello!
>
> On Sunday 07 November 2021 12:56:37 Tony Dinh wrote:
> > Hi Pali,
> >
> > I've applied your patch, and ran some tests. Looks like we got to the
> > bind, but it was not probing (or the probe died silently). Later, I
> > tried pci enum and got some errors during probing.
>
> It looks like that arch/arm/mach-kirkwood/cpu.c already setup PCIe
> windows. Therefore when pci_mvebu.c tries it too, it fails on error
> "conflicts with another window" which you see in the log.
>
> Could you try to comment calling of "mvebu_mbus_add_window_by_id"
> function in pci_mvebu.c? As PCIe window setup should be done exactly
> once.
>
> Also try to call 'pci 0' and 'pci 1' after the 'pci enum'.
>
> > Here is the log (I added some debug printf to drivers/pci/pci_mvebu.c
> > and drivers/core/device.c to see the progress).
> >
> > - U-Boot boot log:
> >
> > U-Boot 2022.01-tld-1-00054-g52207514ba-dirty (Nov 06 2021 - 18:11:41 -0700)
> > Pogoplug V4
> >
> > SoC:   Kirkwood 88F6281_A1
> > DRAM:  128 MiB
> > mvebu_pcie_bind: begin
> > mvebu_pcie_bind: got an available node
> > Bound device  to pcie@82000000
> > mvebu_pcie_bind: bound
> > mvebu_pcie_bind: exit 0
> > Bound device pcie@82000000 to mbus@f1000000
> > Bound device mbus@f1000000 to root_driver
> > Bound device ehci@50000 to ocp@f1000000
> > Bound device ethernet-controller@72000 to ocp@f1000000
> > Bound device sata@80000 to ocp@f1000000
> > Bound device mvsdio@90000.blk to mvsdio@90000
> > Bound device mvsdio@90000 to ocp@f1000000
> > Bound device ocp@f1000000 to root_driver
> > NAND:  128 MiB
> > MMC:   mvsdio@90000: 0
> > Loading Environment from NAND... *** Warning - bad CRC, using default
> > environment
> >
> > In:    serial
> > Out:   serial
> > Err:   serial
> > Net:
> > Warning: ethernet-controller@72000 (eth0) using random MAC address -
> > 66:43:1f:50:f0:e7
> > eth0: ethernet-controller@72000
> > Hit any key to stop autoboot:  0
> >
> > Pogo_V4> pci
> > No such bus
> > Pogo_V4> pci 0
> > No such bus
> > Pogo_V4> pci 1
> > No such bus
> >
> > Pogo_V4> pci enum
> > mvebu_pcie_probe:
> > Cannot add window '4:e8', conflicts with another window
> > PCIe unable to add mbus window for mem at 90000000+10000000
> > Cannot add window '4:e0', conflicts with another window
> > PCIe unable to add mbus window for IO at c0000000+00010000
> > mvebu_pcie_probe: exit 0
> > Bound device pci_0:0.0 to pcie0.0
> > Bound device xhci_pci to pci_0:0.0
> >
> > Pogo_V4> pci regions
> > #   Bus start          Phys start         Size                Flags
> > 0   0x0000000090000000 0x0000000090000000 0x0000000010000000  mem
> > 1   0x0000000000000000 0x0000000000000000 0x0000000008000000  mem sysmem
> > 2   0x00000000c0000000 0x00000000c0000000 0x0000000000010000  io io
> >
> > - Linux dmesg (grep for pci and xhci)
> >
> > [   13.163011][    T1] mvebu-pcie mbus@f1000000:pcie@82000000: host
> > bridge /mbus@f1000000/pcie@82000000 ranges:
> > [   13.163147][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
> > MEM 0x00f1040000..0x00f1041fff -> 0x0000040000
> > [   13.163217][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
> > MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
> > [   13.163268][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
> > IO 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
> > [   13.163754][    T1] mvebu-pcie mbus@f1000000:pcie@82000000: PCI
> > host bridge to bus 0000:00
> > [   13.163791][    T1] pci_bus 0000:00: root bus resource [bus 00-ff]
> > [   13.163829][    T1] pci_bus 0000:00: root bus resource [mem
> > 0xf1040000-0xf1041fff] (bus address [0x00040000-0x00041fff])
> > [   13.163862][    T1] pci_bus 0000:00: root bus resource [mem
> > 0xe0000000-0xefffffff]
> > [   13.163892][    T1] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
> > [   13.164075][    T1] pci 0000:00:01.0: [11ab:6281] type 01 class 0x060400
> > [   13.164133][    T1] pci 0000:00:01.0: reg 0x38: [mem
> > 0x00000000-0x000007ff pref]
> > [   13.165893][    T1] PCI: bus0: Fast back to back transfers disabled
> > [   13.165949][    T1] pci 0000:00:01.0: bridge configuration invalid
> > ([bus 00-00]), reconfiguring
> > [   13.166276][    T1] pci 0000:01:00.0: [1b73:1009] type 00 class 0x0c0330
> > [   13.166334][    T1] pci 0000:01:00.0: reg 0x10: [mem
> > 0x00000000-0x0000ffff 64bit]
> > [   13.166381][    T1] pci 0000:01:00.0: reg 0x18: [mem
> > 0x00000000-0x00000fff 64bit]
> > [   13.166424][    T1] pci 0000:01:00.0: reg 0x20: [mem
> > 0x00000000-0x00000fff 64bit]
> > [   13.166561][    T1] pci 0000:01:00.0: supports D1
> > [   13.166588][    T1] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
> > [   13.166685][    T1] pci 0000:01:00.0: 2.000 Gb/s available PCIe
> > bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable
> > of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
> > [   13.197158][    T1] PCI: bus1: Fast back to back transfers disabled
> > [   13.197208][    T1] pci_bus 0000:01: busn_res: [bus 01-ff] end is
> > updated to 01
> > [   13.197538][    T1] pci 0000:00:01.0: BAR 14: assigned [mem
> > 0xe0000000-0xe00fffff]
> > [   13.197581][    T1] pci 0000:00:01.0: BAR 6: assigned [mem
> > 0xe0100000-0xe01007ff pref]
> > [   13.197624][    T1] pci 0000:01:00.0: BAR 0: assigned [mem
> > 0xe0000000-0xe000ffff 64bit]
> > [   13.197670][    T1] pci 0000:01:00.0: BAR 2: assigned [mem
> > 0xe0010000-0xe0010fff 64bit]
> > [   13.197715][    T1] pci 0000:01:00.0: BAR 4: assigned [mem
> > 0xe0011000-0xe0011fff 64bit]
> > [   13.197759][    T1] pci 0000:00:01.0: PCI bridge to [bus 01]
> > [   13.197792][    T1] pci 0000:00:01.0:   bridge window [mem
> > 0xe0000000-0xe00fffff]
> > [   13.197985][    T1] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
> > [   13.198135][    T1] pci 0000:01:00.0: enabling device (0140 -> 0142)
> >
> > [   15.619756][    T1] xhci_hcd 0000:01:00.0: xHCI Host Controller
> > [   15.625694][    T1] xhci_hcd 0000:01:00.0: new USB bus registered,
> > assigned bus number 2
> > [   15.635643][    T1] xhci_hcd 0000:01:00.0: hcc params 0x200073a1
> > hci version 0x100 quirks 0x0000000000080010
> > [   15.646477][    T1] xhci_hcd 0000:01:00.0: xHCI Host Controller
> > [   15.652432][    T1] xhci_hcd 0000:01:00.0: new USB bus registered,
> > assigned bus number 3
> > [   15.660560][    T1] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
> >
> > Note that this board PCI and USB 3.0 have been working in the Linux
> > kernel for quite many years.
> >
> > - These are relevant configs used in the build (not sure if I need to
> > use CONFIG_DM_PCI_COMPAT?)
> >
> > CONFIG_DM=y
> > CONFIG_CMD_PCI=y
> > CONFIG_PCI=y
> > CONFIG_PCI_MVEBU=y
> > CONFIG_PCI_ENDPOINT=y
> > CONFIG_DM_PCI_COMPAT=y
> > CONFIG_USB_XHCI_HCD=y
> > CONFIG_USB_XHCI_PCI=y
> > CONFIG_USB_XHCI_MVEBU=y
>
> Following options should be enough:
>
> CONFIG_CMD_PCI=y
> CONFIG_PCI=y
> # CONFIG_DM_PCI_COMPAT is not set
> CONFIG_PCI_PNP=y
> CONFIG_PCI_MVEBU=y

Since the PCI bus in this box is used for USB 3.0, I've included
various XHCI configs to see if I can probe this USB 3.0 flash drive
also (that's the ultimate test).

>
> Option CONFIG_PCI_PNP=y is required to see endpoint cards. Without it
> U-Boot would see only PCIe Root Ports. But for first basic tests it
> should be enough.
>
> > BTW, the topic is no longer kwboot, should we move this to another new
> > thread? i.e. Testing PCI MVEBU with Kirkwood SoCs.
>
> Changed :-)

Thanks :)

Here is the log of a successful spin up for the PCI bus and USB 3.0
drive.  Please see my commentary ### in between the logs.

U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 16:03:46 -0800)
Pogoplug V4

SoC:   Kirkwood 88F6281_A1
DRAM:  128 MiB
device_probe:
device_probe: exit 0


### The device_probe above seems to have nothing happening.


mvebu_pcie_bind: begin
mvebu_pcie_bind: got an available node
Bound device  to pcie@82000000
mvebu_pcie_bind: bound
mvebu_pcie_bind: exit 0
Bound device pcie@82000000 to mbus@f1000000
Bound device mbus@f1000000 to root_driver
Bound device ehci@50000 to ocp@f1000000
Bound device ethernet-controller@72000 to ocp@f1000000
Bound device sata@80000 to ocp@f1000000
Bound device mvsdio@90000.blk to mvsdio@90000
Bound device mvsdio@90000 to ocp@f1000000
Bound device ocp@f1000000 to root_driver
NAND:  128 MiB
MMC:   device_probe:
device_probe:
device_probe:
device_probe: exit 0
device_probe: exit 0
device_probe:
device_probe:
mvsdio@90000: 0
Loading Environment from NAND... *** Warning - bad CRC, using default
environment

In:    serial
Out:   serial
Err:   serial
Net:   device_probe:
device_probe:

Warning: ethernet-controller@72000 (eth0) using random MAC address -
86:b8:84:ee:a9:a5
device_probe: exit 0
eth0: ethernet-controller@72000
Hit any key to stop autoboot:  0
Pogo_V4> usb reset
resetting USB...
Bus ehci@50000: device_probe:
device_probe:
USB EHCI 1.00
device_probe: exit 0
scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
device_probe:
device_probe:
Bound device usb_hub to usb_hub
device_probe:
device_probe:
device_probe: exit 0
Bound device usb_mass_storage to usb_hub
device_probe:
device_probe:
Bound device usb_mass_storage.lun0 to usb_mass_storage
device_probe: exit 0
device_probe: exit 0
3 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found


### Try to bring up both USB drives (2.0 and 3.0). But only the USB
2.0 was up as shown above.


Pogo_V4> pci enum
device_probe:
device_probe:
device_probe:
device_probe:
device_probe: exit 0
device_probe: exit 0
mvebu_pcie_probe:
Cannot add window '4:e8', conflicts with another window
PCIe unable to add mbus window for mem at 90000000+10000000
Cannot add window '4:e0', conflicts with another window
PCIe unable to add mbus window for IO at c0000000+00010000
mvebu_pcie_probe: exit 0
Bound device pci_0:0.0 to pcie0.0
device_probe:
device_probe:
Bound device xhci_pci to pci_0:0.0
device_probe: exit 0
device_probe: exit 0
device_probe:

### So I did the pci enum above to get the XHCI bound to PCIe.

Pogo_V4> usb reset
resetting USB...
Bus ehci@50000: device_probe:
device_probe:
USB EHCI 1.00
device_probe: exit 0
Bus xhci_pci: device_probe:
device_probe:
Register 400081f NbrPorts 4
Starting the controller
USB XHCI 1.00
device_probe: exit 0
scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
device_probe:
device_probe:
Bound device usb_hub to usb_hub
device_probe:
device_probe:
device_probe: exit 0
Bound device usb_mass_storage to usb_hub
device_probe:
device_probe:
Bound device usb_mass_storage.lun0 to usb_mass_storage
device_probe: exit 0
device_probe: exit 0
3 USB Device(s) found
scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
device_probe:
device_probe:
Bound device usb_mass_storage to usb_hub
device_probe:
device_probe:
Bound device usb_mass_storage.lun0 to usb_mass_storage
device_probe: exit 0
device_probe: exit 0
2 USB Device(s) found
       scanning usb for storage devices... 2 Storage Device(s) found


### Now both drives are up!


Pogo_V4> pci 0
device_probe:
Scanning PCI devices on bus 0
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
00.00.00   0x11ab     0x6281     Bridge device           0x04
Pogo_V4> pci 1
device_probe:
Scanning PCI devices on bus 1
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
01.00.00   0x1b73     0x1009     Serial bus controller   0x03


### And the pci command has meaningful output at this point as shown above.


Pogo_V4> usb tree
USB device tree:
  1  Hub (480 Mb/s, 0mA)
  |  u-boot EHCI Host Controller
  |
  +-2  Hub (480 Mb/s, 100mA)
    |  GenesysLogic USB2.0 Hub
    |
    +-3  Mass Storage (480 Mb/s, 224mA)
         SanDisk Ultra Fit 4C531001560827107320

  1  Hub (5 Gb/s, 0mA)
  |  U-Boot XHCI Host Controller
  |
  +-2  Mass Storage (5 Gb/s, 224mA)
       SanDisk Cruzer Glide 3.0 4C530000130418116112

Pogo_V4> usb info
1: Hub,  USB Revision 2.0
 - u-boot EHCI Host Controller
 - Class: Hub
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x0000  Product 0x0000 Version 1.0
   Configuration: 1
   - Interfaces: 1 Self Powered 0mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms

2: Hub,  USB Revision 2.0
 - GenesysLogic USB2.0 Hub
 - Class: Hub
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x05e3  Product 0x0610 Version 144.48
   Configuration: 1
   - Interfaces: 1 Self Powered Remote Wakeup 100mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
     - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms

3: Mass Storage,  USB Revision 2.10
 - SanDisk Ultra Fit 4C531001560827107320
 - Class: (from Interface) Mass Storage
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x0781  Product 0x5583 Version 1.0
   Configuration: 1
   - Interfaces: 1 Bus Powered 224mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 2
     - Class Mass Storage, Transp. SCSI, Bulk only
     - Endpoint 1 In Bulk MaxPacket 512
     - Endpoint 2 Out Bulk MaxPacket 512

1: Hub,  USB Revision 3.0
 - U-Boot XHCI Host Controller
 - Class: Hub
 - PacketSize: 512  Configurations: 1
 - Vendor: 0x0000  Product 0x0000 Version 1.0
   Configuration: 1
   - Interfaces: 1 Self Powered 0mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms

2: Mass Storage,  USB Revision 3.0
 - SanDisk Cruzer Glide 3.0 4C530000130418116112
 - Class: (from Interface) Mass Storage
 - PacketSize: 512  Configurations: 1
 - Vendor: 0x0781  Product 0x5597 Version 1.0
   Configuration: 1
   - Interfaces: 1 Bus Powered 224mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 2
     - Class Mass Storage, Transp. SCSI, Bulk only
     - Endpoint 1 In Bulk MaxPacket 1024
     - Endpoint 2 Out Bulk MaxPacket 1024

### The XHCI Host Controller and the SanDisk Cruzer Glide are now
active as shown above.

### So list the files on this SanDisk Cruzer Glide to make sure we can read it.


Pogo_V4> ext2ls usb 1:1 /
device_probe:
device_probe:
device_probe: exit 0
<DIR>       4096 .
<DIR>       4096 ..
          522666 uboot.2021.07-tld-1.pogo_v4.bin
          524288 uboot.2021.07-tld-1.pogo_v4.mtd0.kwb

Great works Pali!

I think some more investigation is needed. Why did we need to do "pci
enum", and then "usb reset", in that order, to get the PCI bus and the
XHCI controller probed? there must be something missing in the process
somewhere between the Device uclass, the PCI uclass, and the pci_mvebu
uclass?

Thanks,
Tony

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

On Sun, Nov 7, 2021 at 4:58 PM Tony Dinh <mibodhi@gmail.com> wrote:
>
> Hi Pali,
>
> Looks like it is working! But in a weird way. Please see my responses
> in between, and also in the test log below.
>
> On Sun, Nov 7, 2021 at 3:45 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > Hello!
> >
> > On Sunday 07 November 2021 12:56:37 Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > I've applied your patch, and ran some tests. Looks like we got to the
> > > bind, but it was not probing (or the probe died silently). Later, I
> > > tried pci enum and got some errors during probing.
> >
> > It looks like that arch/arm/mach-kirkwood/cpu.c already setup PCIe
> > windows. Therefore when pci_mvebu.c tries it too, it fails on error
> > "conflicts with another window" which you see in the log.
> >
> > Could you try to comment calling of "mvebu_mbus_add_window_by_id"
> > function in pci_mvebu.c? As PCIe window setup should be done exactly
> > once.
> >
> > Also try to call 'pci 0' and 'pci 1' after the 'pci enum'.
> >
> > > Here is the log (I added some debug printf to drivers/pci/pci_mvebu.c
> > > and drivers/core/device.c to see the progress).
> > >
> > > - U-Boot boot log:
> > >
> > > U-Boot 2022.01-tld-1-00054-g52207514ba-dirty (Nov 06 2021 - 18:11:41 -0700)
> > > Pogoplug V4
> > >
> > > SoC:   Kirkwood 88F6281_A1
> > > DRAM:  128 MiB
> > > mvebu_pcie_bind: begin
> > > mvebu_pcie_bind: got an available node
> > > Bound device  to pcie@82000000
> > > mvebu_pcie_bind: bound
> > > mvebu_pcie_bind: exit 0
> > > Bound device pcie@82000000 to mbus@f1000000
> > > Bound device mbus@f1000000 to root_driver
> > > Bound device ehci@50000 to ocp@f1000000
> > > Bound device ethernet-controller@72000 to ocp@f1000000
> > > Bound device sata@80000 to ocp@f1000000
> > > Bound device mvsdio@90000.blk to mvsdio@90000
> > > Bound device mvsdio@90000 to ocp@f1000000
> > > Bound device ocp@f1000000 to root_driver
> > > NAND:  128 MiB
> > > MMC:   mvsdio@90000: 0
> > > Loading Environment from NAND... *** Warning - bad CRC, using default
> > > environment
> > >
> > > In:    serial
> > > Out:   serial
> > > Err:   serial
> > > Net:
> > > Warning: ethernet-controller@72000 (eth0) using random MAC address -
> > > 66:43:1f:50:f0:e7
> > > eth0: ethernet-controller@72000
> > > Hit any key to stop autoboot:  0
> > >
> > > Pogo_V4> pci
> > > No such bus
> > > Pogo_V4> pci 0
> > > No such bus
> > > Pogo_V4> pci 1
> > > No such bus
> > >
> > > Pogo_V4> pci enum
> > > mvebu_pcie_probe:
> > > Cannot add window '4:e8', conflicts with another window
> > > PCIe unable to add mbus window for mem at 90000000+10000000
> > > Cannot add window '4:e0', conflicts with another window
> > > PCIe unable to add mbus window for IO at c0000000+00010000
> > > mvebu_pcie_probe: exit 0
> > > Bound device pci_0:0.0 to pcie0.0
> > > Bound device xhci_pci to pci_0:0.0
> > >
> > > Pogo_V4> pci regions
> > > #   Bus start          Phys start         Size                Flags
> > > 0   0x0000000090000000 0x0000000090000000 0x0000000010000000  mem
> > > 1   0x0000000000000000 0x0000000000000000 0x0000000008000000  mem sysmem
> > > 2   0x00000000c0000000 0x00000000c0000000 0x0000000000010000  io io
> > >
> > > - Linux dmesg (grep for pci and xhci)
> > >
> > > [   13.163011][    T1] mvebu-pcie mbus@f1000000:pcie@82000000: host
> > > bridge /mbus@f1000000/pcie@82000000 ranges:
> > > [   13.163147][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
> > > MEM 0x00f1040000..0x00f1041fff -> 0x0000040000
> > > [   13.163217][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
> > > MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
> > > [   13.163268][    T1] mvebu-pcie mbus@f1000000:pcie@82000000:
> > > IO 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
> > > [   13.163754][    T1] mvebu-pcie mbus@f1000000:pcie@82000000: PCI
> > > host bridge to bus 0000:00
> > > [   13.163791][    T1] pci_bus 0000:00: root bus resource [bus 00-ff]
> > > [   13.163829][    T1] pci_bus 0000:00: root bus resource [mem
> > > 0xf1040000-0xf1041fff] (bus address [0x00040000-0x00041fff])
> > > [   13.163862][    T1] pci_bus 0000:00: root bus resource [mem
> > > 0xe0000000-0xefffffff]
> > > [   13.163892][    T1] pci_bus 0000:00: root bus resource [io  0x1000-0xeffff]
> > > [   13.164075][    T1] pci 0000:00:01.0: [11ab:6281] type 01 class 0x060400
> > > [   13.164133][    T1] pci 0000:00:01.0: reg 0x38: [mem
> > > 0x00000000-0x000007ff pref]
> > > [   13.165893][    T1] PCI: bus0: Fast back to back transfers disabled
> > > [   13.165949][    T1] pci 0000:00:01.0: bridge configuration invalid
> > > ([bus 00-00]), reconfiguring
> > > [   13.166276][    T1] pci 0000:01:00.0: [1b73:1009] type 00 class 0x0c0330
> > > [   13.166334][    T1] pci 0000:01:00.0: reg 0x10: [mem
> > > 0x00000000-0x0000ffff 64bit]
> > > [   13.166381][    T1] pci 0000:01:00.0: reg 0x18: [mem
> > > 0x00000000-0x00000fff 64bit]
> > > [   13.166424][    T1] pci 0000:01:00.0: reg 0x20: [mem
> > > 0x00000000-0x00000fff 64bit]
> > > [   13.166561][    T1] pci 0000:01:00.0: supports D1
> > > [   13.166588][    T1] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
> > > [   13.166685][    T1] pci 0000:01:00.0: 2.000 Gb/s available PCIe
> > > bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable
> > > of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
> > > [   13.197158][    T1] PCI: bus1: Fast back to back transfers disabled
> > > [   13.197208][    T1] pci_bus 0000:01: busn_res: [bus 01-ff] end is
> > > updated to 01
> > > [   13.197538][    T1] pci 0000:00:01.0: BAR 14: assigned [mem
> > > 0xe0000000-0xe00fffff]
> > > [   13.197581][    T1] pci 0000:00:01.0: BAR 6: assigned [mem
> > > 0xe0100000-0xe01007ff pref]
> > > [   13.197624][    T1] pci 0000:01:00.0: BAR 0: assigned [mem
> > > 0xe0000000-0xe000ffff 64bit]
> > > [   13.197670][    T1] pci 0000:01:00.0: BAR 2: assigned [mem
> > > 0xe0010000-0xe0010fff 64bit]
> > > [   13.197715][    T1] pci 0000:01:00.0: BAR 4: assigned [mem
> > > 0xe0011000-0xe0011fff 64bit]
> > > [   13.197759][    T1] pci 0000:00:01.0: PCI bridge to [bus 01]
> > > [   13.197792][    T1] pci 0000:00:01.0:   bridge window [mem
> > > 0xe0000000-0xe00fffff]
> > > [   13.197985][    T1] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
> > > [   13.198135][    T1] pci 0000:01:00.0: enabling device (0140 -> 0142)
> > >
> > > [   15.619756][    T1] xhci_hcd 0000:01:00.0: xHCI Host Controller
> > > [   15.625694][    T1] xhci_hcd 0000:01:00.0: new USB bus registered,
> > > assigned bus number 2
> > > [   15.635643][    T1] xhci_hcd 0000:01:00.0: hcc params 0x200073a1
> > > hci version 0x100 quirks 0x0000000000080010
> > > [   15.646477][    T1] xhci_hcd 0000:01:00.0: xHCI Host Controller
> > > [   15.652432][    T1] xhci_hcd 0000:01:00.0: new USB bus registered,
> > > assigned bus number 3
> > > [   15.660560][    T1] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
> > >
> > > Note that this board PCI and USB 3.0 have been working in the Linux
> > > kernel for quite many years.
> > >
> > > - These are relevant configs used in the build (not sure if I need to
> > > use CONFIG_DM_PCI_COMPAT?)
> > >
> > > CONFIG_DM=y
> > > CONFIG_CMD_PCI=y
> > > CONFIG_PCI=y
> > > CONFIG_PCI_MVEBU=y
> > > CONFIG_PCI_ENDPOINT=y
> > > CONFIG_DM_PCI_COMPAT=y
> > > CONFIG_USB_XHCI_HCD=y
> > > CONFIG_USB_XHCI_PCI=y
> > > CONFIG_USB_XHCI_MVEBU=y
> >
> > Following options should be enough:
> >
> > CONFIG_CMD_PCI=y
> > CONFIG_PCI=y
> > # CONFIG_DM_PCI_COMPAT is not set
> > CONFIG_PCI_PNP=y
> > CONFIG_PCI_MVEBU=y
>
> Since the PCI bus in this box is used for USB 3.0, I've included
> various XHCI configs to see if I can probe this USB 3.0 flash drive
> also (that's the ultimate test).
>
> >
> > Option CONFIG_PCI_PNP=y is required to see endpoint cards. Without it
> > U-Boot would see only PCIe Root Ports. But for first basic tests it
> > should be enough.
> >
> > > BTW, the topic is no longer kwboot, should we move this to another new
> > > thread? i.e. Testing PCI MVEBU with Kirkwood SoCs.
> >
> > Changed :-)
>
> Thanks :)
>
> Here is the log of a successful spin up for the PCI bus and USB 3.0
> drive.  Please see my commentary ### in between the logs.
>
> U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 16:03:46 -0800)
> Pogoplug V4
>
> SoC:   Kirkwood 88F6281_A1
> DRAM:  128 MiB
> device_probe:
> device_probe: exit 0
>
>
> ### The device_probe above seems to have nothing happening.
>
>
> mvebu_pcie_bind: begin
> mvebu_pcie_bind: got an available node
> Bound device  to pcie@82000000
> mvebu_pcie_bind: bound
> mvebu_pcie_bind: exit 0
> Bound device pcie@82000000 to mbus@f1000000
> Bound device mbus@f1000000 to root_driver
> Bound device ehci@50000 to ocp@f1000000
> Bound device ethernet-controller@72000 to ocp@f1000000
> Bound device sata@80000 to ocp@f1000000
> Bound device mvsdio@90000.blk to mvsdio@90000
> Bound device mvsdio@90000 to ocp@f1000000
> Bound device ocp@f1000000 to root_driver
> NAND:  128 MiB
> MMC:   device_probe:
> device_probe:
> device_probe:
> device_probe: exit 0
> device_probe: exit 0
> device_probe:
> device_probe:
> mvsdio@90000: 0
> Loading Environment from NAND... *** Warning - bad CRC, using default
> environment
>
> In:    serial
> Out:   serial
> Err:   serial
> Net:   device_probe:
> device_probe:
>
> Warning: ethernet-controller@72000 (eth0) using random MAC address -
> 86:b8:84:ee:a9:a5
> device_probe: exit 0
> eth0: ethernet-controller@72000
> Hit any key to stop autoboot:  0
> Pogo_V4> usb reset
> resetting USB...
> Bus ehci@50000: device_probe:
> device_probe:
> USB EHCI 1.00
> device_probe: exit 0
> scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> device_probe:
> device_probe:
> Bound device usb_hub to usb_hub
> device_probe:
> device_probe:
> device_probe: exit 0
> Bound device usb_mass_storage to usb_hub
> device_probe:
> device_probe:
> Bound device usb_mass_storage.lun0 to usb_mass_storage
> device_probe: exit 0
> device_probe: exit 0
> 3 USB Device(s) found
>        scanning usb for storage devices... 1 Storage Device(s) found
>
>
> ### Try to bring up both USB drives (2.0 and 3.0). But only the USB
> 2.0 was up as shown above.
>
>
> Pogo_V4> pci enum
> device_probe:
> device_probe:
> device_probe:
> device_probe:
> device_probe: exit 0
> device_probe: exit 0
> mvebu_pcie_probe:
> Cannot add window '4:e8', conflicts with another window
> PCIe unable to add mbus window for mem at 90000000+10000000
> Cannot add window '4:e0', conflicts with another window
> PCIe unable to add mbus window for IO at c0000000+00010000
> mvebu_pcie_probe: exit 0
> Bound device pci_0:0.0 to pcie0.0
> device_probe:
> device_probe:
> Bound device xhci_pci to pci_0:0.0
> device_probe: exit 0
> device_probe: exit 0
> device_probe:
>
> ### So I did the pci enum above to get the XHCI bound to PCIe.
>
> Pogo_V4> usb reset
> resetting USB...
> Bus ehci@50000: device_probe:
> device_probe:
> USB EHCI 1.00
> device_probe: exit 0
> Bus xhci_pci: device_probe:
> device_probe:
> Register 400081f NbrPorts 4
> Starting the controller
> USB XHCI 1.00
> device_probe: exit 0
> scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> device_probe:
> device_probe:
> Bound device usb_hub to usb_hub
> device_probe:
> device_probe:
> device_probe: exit 0
> Bound device usb_mass_storage to usb_hub
> device_probe:
> device_probe:
> Bound device usb_mass_storage.lun0 to usb_mass_storage
> device_probe: exit 0
> device_probe: exit 0
> 3 USB Device(s) found
> scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
> device_probe:
> device_probe:
> Bound device usb_mass_storage to usb_hub
> device_probe:
> device_probe:
> Bound device usb_mass_storage.lun0 to usb_mass_storage
> device_probe: exit 0
> device_probe: exit 0
> 2 USB Device(s) found
>        scanning usb for storage devices... 2 Storage Device(s) found
>
>
> ### Now both drives are up!
>
>
> Pogo_V4> pci 0
> device_probe:
> Scanning PCI devices on bus 0
> BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
> _____________________________________________________________
> 00.00.00   0x11ab     0x6281     Bridge device           0x04
> Pogo_V4> pci 1
> device_probe:
> Scanning PCI devices on bus 1
> BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
> _____________________________________________________________
> 01.00.00   0x1b73     0x1009     Serial bus controller   0x03
>
>
> ### And the pci command has meaningful output at this point as shown above.
>
>
> Pogo_V4> usb tree
> USB device tree:
>   1  Hub (480 Mb/s, 0mA)
>   |  u-boot EHCI Host Controller
>   |
>   +-2  Hub (480 Mb/s, 100mA)
>     |  GenesysLogic USB2.0 Hub
>     |
>     +-3  Mass Storage (480 Mb/s, 224mA)
>          SanDisk Ultra Fit 4C531001560827107320
>
>   1  Hub (5 Gb/s, 0mA)
>   |  U-Boot XHCI Host Controller
>   |
>   +-2  Mass Storage (5 Gb/s, 224mA)
>        SanDisk Cruzer Glide 3.0 4C530000130418116112
>
> Pogo_V4> usb info
> 1: Hub,  USB Revision 2.0
>  - u-boot EHCI Host Controller
>  - Class: Hub
>  - PacketSize: 64  Configurations: 1
>  - Vendor: 0x0000  Product 0x0000 Version 1.0
>    Configuration: 1
>    - Interfaces: 1 Self Powered 0mA
>      Interface: 0
>      - Alternate Setting 0, Endpoints: 1
>      - Class Hub
>      - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
>
> 2: Hub,  USB Revision 2.0
>  - GenesysLogic USB2.0 Hub
>  - Class: Hub
>  - PacketSize: 64  Configurations: 1
>  - Vendor: 0x05e3  Product 0x0610 Version 144.48
>    Configuration: 1
>    - Interfaces: 1 Self Powered Remote Wakeup 100mA
>      Interface: 0
>      - Alternate Setting 0, Endpoints: 1
>      - Class Hub
>      - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
>      - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
>
> 3: Mass Storage,  USB Revision 2.10
>  - SanDisk Ultra Fit 4C531001560827107320
>  - Class: (from Interface) Mass Storage
>  - PacketSize: 64  Configurations: 1
>  - Vendor: 0x0781  Product 0x5583 Version 1.0
>    Configuration: 1
>    - Interfaces: 1 Bus Powered 224mA
>      Interface: 0
>      - Alternate Setting 0, Endpoints: 2
>      - Class Mass Storage, Transp. SCSI, Bulk only
>      - Endpoint 1 In Bulk MaxPacket 512
>      - Endpoint 2 Out Bulk MaxPacket 512
>
> 1: Hub,  USB Revision 3.0
>  - U-Boot XHCI Host Controller
>  - Class: Hub
>  - PacketSize: 512  Configurations: 1
>  - Vendor: 0x0000  Product 0x0000 Version 1.0
>    Configuration: 1
>    - Interfaces: 1 Self Powered 0mA
>      Interface: 0
>      - Alternate Setting 0, Endpoints: 1
>      - Class Hub
>      - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
>
> 2: Mass Storage,  USB Revision 3.0
>  - SanDisk Cruzer Glide 3.0 4C530000130418116112
>  - Class: (from Interface) Mass Storage
>  - PacketSize: 512  Configurations: 1
>  - Vendor: 0x0781  Product 0x5597 Version 1.0
>    Configuration: 1
>    - Interfaces: 1 Bus Powered 224mA
>      Interface: 0
>      - Alternate Setting 0, Endpoints: 2
>      - Class Mass Storage, Transp. SCSI, Bulk only
>      - Endpoint 1 In Bulk MaxPacket 1024
>      - Endpoint 2 Out Bulk MaxPacket 1024
>
> ### The XHCI Host Controller and the SanDisk Cruzer Glide are now
> active as shown above.
>
> ### So list the files on this SanDisk Cruzer Glide to make sure we can read it.
>
>
> Pogo_V4> ext2ls usb 1:1 /
> device_probe:
> device_probe:
> device_probe: exit 0
> <DIR>       4096 .
> <DIR>       4096 ..
>           522666 uboot.2021.07-tld-1.pogo_v4.bin
>           524288 uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
>
> Great works Pali!
>
> I think some more investigation is needed. Why did we need to do "pci
> enum", and then "usb reset", in that order, to get the PCI bus and the
> XHCI controller probed? there must be something missing in the process
> somewhere between the Device uclass, the PCI uclass, and the pci_mvebu
> uclass?

I think I can see the order of enumeration. PCI must be enumerated
first, and then XHCI being the controller on this host bus would come
alive? I think we can live with 'pci enum' and 'usb reset' to get the
USB 3.0 drives enumerated. However, it seems just a little bit
unintuitive.

Thanks,
Tony

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

Hello!

On Sunday 07 November 2021 18:08:56 Tony Dinh wrote:
> Hi Pali,
> 
> On Sun, Nov 7, 2021 at 4:58 PM Tony Dinh <mibodhi@gmail.com> wrote:
> >
> > Hi Pali,
> >
> > Looks like it is working! But in a weird way. Please see my responses
> > in between, and also in the test log below.
> >
> > On Sun, Nov 7, 2021 at 3:45 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > Hello!
> > >
> > > On Sunday 07 November 2021 12:56:37 Tony Dinh wrote:
> > > > Hi Pali,
> > > >
> > > > I've applied your patch, and ran some tests. Looks like we got to the
> > > > bind, but it was not probing (or the probe died silently). Later, I
> > > > tried pci enum and got some errors during probing.
> > >
> > > It looks like that arch/arm/mach-kirkwood/cpu.c already setup PCIe
> > > windows. Therefore when pci_mvebu.c tries it too, it fails on error
> > > "conflicts with another window" which you see in the log.
> > >
> > > Could you try to comment calling of "mvebu_mbus_add_window_by_id"
> > > function in pci_mvebu.c? As PCIe window setup should be done exactly
> > > once.
> > >
> > > Also try to call 'pci 0' and 'pci 1' after the 'pci enum'.
> > >
...
> > > >
> > > > - These are relevant configs used in the build (not sure if I need to
> > > > use CONFIG_DM_PCI_COMPAT?)
> > > >
> > > > CONFIG_DM=y
> > > > CONFIG_CMD_PCI=y
> > > > CONFIG_PCI=y
> > > > CONFIG_PCI_MVEBU=y
> > > > CONFIG_PCI_ENDPOINT=y
> > > > CONFIG_DM_PCI_COMPAT=y
> > > > CONFIG_USB_XHCI_HCD=y
> > > > CONFIG_USB_XHCI_PCI=y
> > > > CONFIG_USB_XHCI_MVEBU=y
> > >
> > > Following options should be enough:
> > >
> > > CONFIG_CMD_PCI=y
> > > CONFIG_PCI=y
> > > # CONFIG_DM_PCI_COMPAT is not set
> > > CONFIG_PCI_PNP=y
> > > CONFIG_PCI_MVEBU=y
> >
> > Since the PCI bus in this box is used for USB 3.0, I've included
> > various XHCI configs to see if I can probe this USB 3.0 flash drive
> > also (that's the ultimate test).

OK!

> > >
> > > Option CONFIG_PCI_PNP=y is required to see endpoint cards. Without it
> > > U-Boot would see only PCIe Root Ports. But for first basic tests it
> > > should be enough.
> > >
> > > > BTW, the topic is no longer kwboot, should we move this to another new
> > > > thread? i.e. Testing PCI MVEBU with Kirkwood SoCs.
> > >
> > > Changed :-)
> >
> > Thanks :)
> >
> > Here is the log of a successful spin up for the PCI bus and USB 3.0
> > drive.  Please see my commentary ### in between the logs.
> >
> > U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 16:03:46 -0800)
> > Pogoplug V4
> >
> > SoC:   Kirkwood 88F6281_A1
> > DRAM:  128 MiB
> > device_probe:
> > device_probe: exit 0
> >
> >
> > ### The device_probe above seems to have nothing happening.
> >
> >
> > mvebu_pcie_bind: begin
> > mvebu_pcie_bind: got an available node
> > Bound device  to pcie@82000000
> > mvebu_pcie_bind: bound
> > mvebu_pcie_bind: exit 0
> > Bound device pcie@82000000 to mbus@f1000000
> > Bound device mbus@f1000000 to root_driver
> > Bound device ehci@50000 to ocp@f1000000
> > Bound device ethernet-controller@72000 to ocp@f1000000
> > Bound device sata@80000 to ocp@f1000000
> > Bound device mvsdio@90000.blk to mvsdio@90000
> > Bound device mvsdio@90000 to ocp@f1000000
> > Bound device ocp@f1000000 to root_driver
> > NAND:  128 MiB
> > MMC:   device_probe:
> > device_probe:
> > device_probe:
> > device_probe: exit 0
> > device_probe: exit 0
> > device_probe:
> > device_probe:
> > mvsdio@90000: 0
> > Loading Environment from NAND... *** Warning - bad CRC, using default
> > environment
> >
> > In:    serial
> > Out:   serial
> > Err:   serial
> > Net:   device_probe:
> > device_probe:
> >
> > Warning: ethernet-controller@72000 (eth0) using random MAC address -
> > 86:b8:84:ee:a9:a5
> > device_probe: exit 0
> > eth0: ethernet-controller@72000
> > Hit any key to stop autoboot:  0
> > Pogo_V4> usb reset
> > resetting USB...
> > Bus ehci@50000: device_probe:
> > device_probe:
> > USB EHCI 1.00
> > device_probe: exit 0
> > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> > device_probe:
> > device_probe:
> > Bound device usb_hub to usb_hub
> > device_probe:
> > device_probe:
> > device_probe: exit 0
> > Bound device usb_mass_storage to usb_hub
> > device_probe:
> > device_probe:
> > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > device_probe: exit 0
> > device_probe: exit 0
> > 3 USB Device(s) found
> >        scanning usb for storage devices... 1 Storage Device(s) found
> >
> >
> > ### Try to bring up both USB drives (2.0 and 3.0). But only the USB
> > 2.0 was up as shown above.
> >
> >
> > Pogo_V4> pci enum
> > device_probe:
> > device_probe:
> > device_probe:
> > device_probe:
> > device_probe: exit 0
> > device_probe: exit 0
> > mvebu_pcie_probe:
> > Cannot add window '4:e8', conflicts with another window
> > PCIe unable to add mbus window for mem at 90000000+10000000
> > Cannot add window '4:e0', conflicts with another window
> > PCIe unable to add mbus window for IO at c0000000+00010000

Error is still there...

Have you tried to comment and disable calling
"mvebu_mbus_add_window_by_id" function from pci_mvebu.c?

> > mvebu_pcie_probe: exit 0
> > Bound device pci_0:0.0 to pcie0.0
> > device_probe:
> > device_probe:
> > Bound device xhci_pci to pci_0:0.0
> > device_probe: exit 0
> > device_probe: exit 0
> > device_probe:
> >
> > ### So I did the pci enum above to get the XHCI bound to PCIe.
> >
> > Pogo_V4> usb reset
> > resetting USB...
> > Bus ehci@50000: device_probe:
> > device_probe:
> > USB EHCI 1.00
> > device_probe: exit 0
> > Bus xhci_pci: device_probe:
> > device_probe:
> > Register 400081f NbrPorts 4
> > Starting the controller
> > USB XHCI 1.00
> > device_probe: exit 0
> > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> > device_probe:
> > device_probe:
> > Bound device usb_hub to usb_hub
> > device_probe:
> > device_probe:
> > device_probe: exit 0
> > Bound device usb_mass_storage to usb_hub
> > device_probe:
> > device_probe:
> > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > device_probe: exit 0
> > device_probe: exit 0
> > 3 USB Device(s) found
> > scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
> > device_probe:
> > device_probe:
> > Bound device usb_mass_storage to usb_hub
> > device_probe:
> > device_probe:
> > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > device_probe: exit 0
> > device_probe: exit 0
> > 2 USB Device(s) found
> >        scanning usb for storage devices... 2 Storage Device(s) found
> >
> >
> > ### Now both drives are up!
> >
> >
> > Pogo_V4> pci 0
> > device_probe:
> > Scanning PCI devices on bus 0
> > BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
> > _____________________________________________________________
> > 00.00.00   0x11ab     0x6281     Bridge device           0x04
> > Pogo_V4> pci 1
> > device_probe:
> > Scanning PCI devices on bus 1
> > BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
> > _____________________________________________________________
> > 01.00.00   0x1b73     0x1009     Serial bus controller   0x03
> >
> >
> > ### And the pci command has meaningful output at this point as shown above.
> >
> >
> > Pogo_V4> usb tree
> > USB device tree:
> >   1  Hub (480 Mb/s, 0mA)
> >   |  u-boot EHCI Host Controller
> >   |
> >   +-2  Hub (480 Mb/s, 100mA)
> >     |  GenesysLogic USB2.0 Hub
> >     |
> >     +-3  Mass Storage (480 Mb/s, 224mA)
> >          SanDisk Ultra Fit 4C531001560827107320
> >
> >   1  Hub (5 Gb/s, 0mA)
> >   |  U-Boot XHCI Host Controller
> >   |
> >   +-2  Mass Storage (5 Gb/s, 224mA)
> >        SanDisk Cruzer Glide 3.0 4C530000130418116112
> >
> > Pogo_V4> usb info
> > 1: Hub,  USB Revision 2.0
> >  - u-boot EHCI Host Controller
> >  - Class: Hub
> >  - PacketSize: 64  Configurations: 1
> >  - Vendor: 0x0000  Product 0x0000 Version 1.0
> >    Configuration: 1
> >    - Interfaces: 1 Self Powered 0mA
> >      Interface: 0
> >      - Alternate Setting 0, Endpoints: 1
> >      - Class Hub
> >      - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
> >
> > 2: Hub,  USB Revision 2.0
> >  - GenesysLogic USB2.0 Hub
> >  - Class: Hub
> >  - PacketSize: 64  Configurations: 1
> >  - Vendor: 0x05e3  Product 0x0610 Version 144.48
> >    Configuration: 1
> >    - Interfaces: 1 Self Powered Remote Wakeup 100mA
> >      Interface: 0
> >      - Alternate Setting 0, Endpoints: 1
> >      - Class Hub
> >      - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
> >      - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
> >
> > 3: Mass Storage,  USB Revision 2.10
> >  - SanDisk Ultra Fit 4C531001560827107320
> >  - Class: (from Interface) Mass Storage
> >  - PacketSize: 64  Configurations: 1
> >  - Vendor: 0x0781  Product 0x5583 Version 1.0
> >    Configuration: 1
> >    - Interfaces: 1 Bus Powered 224mA
> >      Interface: 0
> >      - Alternate Setting 0, Endpoints: 2
> >      - Class Mass Storage, Transp. SCSI, Bulk only
> >      - Endpoint 1 In Bulk MaxPacket 512
> >      - Endpoint 2 Out Bulk MaxPacket 512
> >
> > 1: Hub,  USB Revision 3.0
> >  - U-Boot XHCI Host Controller
> >  - Class: Hub
> >  - PacketSize: 512  Configurations: 1
> >  - Vendor: 0x0000  Product 0x0000 Version 1.0
> >    Configuration: 1
> >    - Interfaces: 1 Self Powered 0mA
> >      Interface: 0
> >      - Alternate Setting 0, Endpoints: 1
> >      - Class Hub
> >      - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
> >
> > 2: Mass Storage,  USB Revision 3.0
> >  - SanDisk Cruzer Glide 3.0 4C530000130418116112
> >  - Class: (from Interface) Mass Storage
> >  - PacketSize: 512  Configurations: 1
> >  - Vendor: 0x0781  Product 0x5597 Version 1.0
> >    Configuration: 1
> >    - Interfaces: 1 Bus Powered 224mA
> >      Interface: 0
> >      - Alternate Setting 0, Endpoints: 2
> >      - Class Mass Storage, Transp. SCSI, Bulk only
> >      - Endpoint 1 In Bulk MaxPacket 1024
> >      - Endpoint 2 Out Bulk MaxPacket 1024
> >
> > ### The XHCI Host Controller and the SanDisk Cruzer Glide are now
> > active as shown above.
> >
> > ### So list the files on this SanDisk Cruzer Glide to make sure we can read it.
> >
> >
> > Pogo_V4> ext2ls usb 1:1 /
> > device_probe:
> > device_probe:
> > device_probe: exit 0
> > <DIR>       4096 .
> > <DIR>       4096 ..
> >           522666 uboot.2021.07-tld-1.pogo_v4.bin
> >           524288 uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> >
> > Great works Pali!
> >
> > I think some more investigation is needed. Why did we need to do "pci
> > enum", and then "usb reset", in that order, to get the PCI bus and the
> > XHCI controller probed? there must be something missing in the process
> > somewhere between the Device uclass, the PCI uclass, and the pci_mvebu
> > uclass?
> 
> I think I can see the order of enumeration. PCI must be enumerated
> first, and then XHCI being the controller on this host bus would come
> alive? I think we can live with 'pci enum' and 'usb reset' to get the
> USB 3.0 drives enumerated. However, it seems just a little bit
> unintuitive.

'pci enum' should be called internally by U-Boot during loading. So only
'usb start' would be needed.

But from your boot log it looks like that PCI enumaration was not done
and so calling 'pci enum' manually is needed.

I will look into U-Boot code why it happens...

Anyway, based on your test, PCIe must work correctly :)

Could you send config space dump of PCIe Root Port? Following U-Boot
command prints it on terminal: 'pci display.b 0.0.0 0 200'

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

On Mon, Nov 8, 2021 at 3:11 AM Pali Rohár <pali@kernel.org> wrote:
>
> Hello!
>
> On Sunday 07 November 2021 18:08:56 Tony Dinh wrote:
> > Hi Pali,
> >
> > On Sun, Nov 7, 2021 at 4:58 PM Tony Dinh <mibodhi@gmail.com> wrote:
> > >
> > > Hi Pali,
> > >
> > > Looks like it is working! But in a weird way. Please see my responses
> > > in between, and also in the test log below.
> > >
> > > On Sun, Nov 7, 2021 at 3:45 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > Hello!
> > > >
> > > > On Sunday 07 November 2021 12:56:37 Tony Dinh wrote:
> > > > > Hi Pali,
> > > > >
> > > > > I've applied your patch, and ran some tests. Looks like we got to the
> > > > > bind, but it was not probing (or the probe died silently). Later, I
> > > > > tried pci enum and got some errors during probing.
> > > >
> > > > It looks like that arch/arm/mach-kirkwood/cpu.c already setup PCIe
> > > > windows. Therefore when pci_mvebu.c tries it too, it fails on error
> > > > "conflicts with another window" which you see in the log.
> > > >
> > > > Could you try to comment calling of "mvebu_mbus_add_window_by_id"
> > > > function in pci_mvebu.c? As PCIe window setup should be done exactly
> > > > once.
> > > >
> > > > Also try to call 'pci 0' and 'pci 1' after the 'pci enum'.
> > > >
> ...
> > > > >
> > > > > - These are relevant configs used in the build (not sure if I need to
> > > > > use CONFIG_DM_PCI_COMPAT?)
> > > > >
> > > > > CONFIG_DM=y
> > > > > CONFIG_CMD_PCI=y
> > > > > CONFIG_PCI=y
> > > > > CONFIG_PCI_MVEBU=y
> > > > > CONFIG_PCI_ENDPOINT=y
> > > > > CONFIG_DM_PCI_COMPAT=y
> > > > > CONFIG_USB_XHCI_HCD=y
> > > > > CONFIG_USB_XHCI_PCI=y
> > > > > CONFIG_USB_XHCI_MVEBU=y
> > > >
> > > > Following options should be enough:
> > > >
> > > > CONFIG_CMD_PCI=y
> > > > CONFIG_PCI=y
> > > > # CONFIG_DM_PCI_COMPAT is not set
> > > > CONFIG_PCI_PNP=y
> > > > CONFIG_PCI_MVEBU=y
> > >
> > > Since the PCI bus in this box is used for USB 3.0, I've included
> > > various XHCI configs to see if I can probe this USB 3.0 flash drive
> > > also (that's the ultimate test).
>
> OK!
>
> > > >
> > > > Option CONFIG_PCI_PNP=y is required to see endpoint cards. Without it
> > > > U-Boot would see only PCIe Root Ports. But for first basic tests it
> > > > should be enough.
> > > >
> > > > > BTW, the topic is no longer kwboot, should we move this to another new
> > > > > thread? i.e. Testing PCI MVEBU with Kirkwood SoCs.
> > > >
> > > > Changed :-)
> > >
> > > Thanks :)
> > >
> > > Here is the log of a successful spin up for the PCI bus and USB 3.0
> > > drive.  Please see my commentary ### in between the logs.
> > >
> > > U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 16:03:46 -0800)
> > > Pogoplug V4
> > >
> > > SoC:   Kirkwood 88F6281_A1
> > > DRAM:  128 MiB
> > > device_probe:
> > > device_probe: exit 0
> > >
> > >
> > > ### The device_probe above seems to have nothing happening.
> > >
> > >
> > > mvebu_pcie_bind: begin
> > > mvebu_pcie_bind: got an available node
> > > Bound device  to pcie@82000000
> > > mvebu_pcie_bind: bound
> > > mvebu_pcie_bind: exit 0
> > > Bound device pcie@82000000 to mbus@f1000000
> > > Bound device mbus@f1000000 to root_driver
> > > Bound device ehci@50000 to ocp@f1000000
> > > Bound device ethernet-controller@72000 to ocp@f1000000
> > > Bound device sata@80000 to ocp@f1000000
> > > Bound device mvsdio@90000.blk to mvsdio@90000
> > > Bound device mvsdio@90000 to ocp@f1000000
> > > Bound device ocp@f1000000 to root_driver
> > > NAND:  128 MiB
> > > MMC:   device_probe:
> > > device_probe:
> > > device_probe:
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > device_probe:
> > > device_probe:
> > > mvsdio@90000: 0
> > > Loading Environment from NAND... *** Warning - bad CRC, using default
> > > environment
> > >
> > > In:    serial
> > > Out:   serial
> > > Err:   serial
> > > Net:   device_probe:
> > > device_probe:
> > >
> > > Warning: ethernet-controller@72000 (eth0) using random MAC address -
> > > 86:b8:84:ee:a9:a5
> > > device_probe: exit 0
> > > eth0: ethernet-controller@72000
> > > Hit any key to stop autoboot:  0
> > > Pogo_V4> usb reset
> > > resetting USB...
> > > Bus ehci@50000: device_probe:
> > > device_probe:
> > > USB EHCI 1.00
> > > device_probe: exit 0
> > > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> > > device_probe:
> > > device_probe:
> > > Bound device usb_hub to usb_hub
> > > device_probe:
> > > device_probe:
> > > device_probe: exit 0
> > > Bound device usb_mass_storage to usb_hub
> > > device_probe:
> > > device_probe:
> > > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > 3 USB Device(s) found
> > >        scanning usb for storage devices... 1 Storage Device(s) found
> > >
> > >
> > > ### Try to bring up both USB drives (2.0 and 3.0). But only the USB
> > > 2.0 was up as shown above.
> > >
> > >
> > > Pogo_V4> pci enum
> > > device_probe:
> > > device_probe:
> > > device_probe:
> > > device_probe:
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > mvebu_pcie_probe:
> > > Cannot add window '4:e8', conflicts with another window
> > > PCIe unable to add mbus window for mem at 90000000+10000000
> > > Cannot add window '4:e0', conflicts with another window
> > > PCIe unable to add mbus window for IO at c0000000+00010000
>
> Error is still there...
>
> Have you tried to comment and disable calling
> "mvebu_mbus_add_window_by_id" function from pci_mvebu.c?
>
> > > mvebu_pcie_probe: exit 0
> > > Bound device pci_0:0.0 to pcie0.0
> > > device_probe:
> > > device_probe:
> > > Bound device xhci_pci to pci_0:0.0
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > device_probe:
> > >
> > > ### So I did the pci enum above to get the XHCI bound to PCIe.
> > >
> > > Pogo_V4> usb reset
> > > resetting USB...
> > > Bus ehci@50000: device_probe:
> > > device_probe:
> > > USB EHCI 1.00
> > > device_probe: exit 0
> > > Bus xhci_pci: device_probe:
> > > device_probe:
> > > Register 400081f NbrPorts 4
> > > Starting the controller
> > > USB XHCI 1.00
> > > device_probe: exit 0
> > > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> > > device_probe:
> > > device_probe:
> > > Bound device usb_hub to usb_hub
> > > device_probe:
> > > device_probe:
> > > device_probe: exit 0
> > > Bound device usb_mass_storage to usb_hub
> > > device_probe:
> > > device_probe:
> > > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > 3 USB Device(s) found
> > > scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
> > > device_probe:
> > > device_probe:
> > > Bound device usb_mass_storage to usb_hub
> > > device_probe:
> > > device_probe:
> > > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > > device_probe: exit 0
> > > device_probe: exit 0
> > > 2 USB Device(s) found
> > >        scanning usb for storage devices... 2 Storage Device(s) found
> > >
> > >
> > > ### Now both drives are up!
> > >
> > >
> > > Pogo_V4> pci 0
> > > device_probe:
> > > Scanning PCI devices on bus 0
> > > BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
> > > _____________________________________________________________
> > > 00.00.00   0x11ab     0x6281     Bridge device           0x04
> > > Pogo_V4> pci 1
> > > device_probe:
> > > Scanning PCI devices on bus 1
> > > BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
> > > _____________________________________________________________
> > > 01.00.00   0x1b73     0x1009     Serial bus controller   0x03
> > >
> > >
> > > ### And the pci command has meaningful output at this point as shown above.
> > >
> > >
> > > Pogo_V4> usb tree
> > > USB device tree:
> > >   1  Hub (480 Mb/s, 0mA)
> > >   |  u-boot EHCI Host Controller
> > >   |
> > >   +-2  Hub (480 Mb/s, 100mA)
> > >     |  GenesysLogic USB2.0 Hub
> > >     |
> > >     +-3  Mass Storage (480 Mb/s, 224mA)
> > >          SanDisk Ultra Fit 4C531001560827107320
> > >
> > >   1  Hub (5 Gb/s, 0mA)
> > >   |  U-Boot XHCI Host Controller
> > >   |
> > >   +-2  Mass Storage (5 Gb/s, 224mA)
> > >        SanDisk Cruzer Glide 3.0 4C530000130418116112
> > >
> > > Pogo_V4> usb info
> > > 1: Hub,  USB Revision 2.0
> > >  - u-boot EHCI Host Controller
> > >  - Class: Hub
> > >  - PacketSize: 64  Configurations: 1
> > >  - Vendor: 0x0000  Product 0x0000 Version 1.0
> > >    Configuration: 1
> > >    - Interfaces: 1 Self Powered 0mA
> > >      Interface: 0
> > >      - Alternate Setting 0, Endpoints: 1
> > >      - Class Hub
> > >      - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
> > >
> > > 2: Hub,  USB Revision 2.0
> > >  - GenesysLogic USB2.0 Hub
> > >  - Class: Hub
> > >  - PacketSize: 64  Configurations: 1
> > >  - Vendor: 0x05e3  Product 0x0610 Version 144.48
> > >    Configuration: 1
> > >    - Interfaces: 1 Self Powered Remote Wakeup 100mA
> > >      Interface: 0
> > >      - Alternate Setting 0, Endpoints: 1
> > >      - Class Hub
> > >      - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
> > >      - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
> > >
> > > 3: Mass Storage,  USB Revision 2.10
> > >  - SanDisk Ultra Fit 4C531001560827107320
> > >  - Class: (from Interface) Mass Storage
> > >  - PacketSize: 64  Configurations: 1
> > >  - Vendor: 0x0781  Product 0x5583 Version 1.0
> > >    Configuration: 1
> > >    - Interfaces: 1 Bus Powered 224mA
> > >      Interface: 0
> > >      - Alternate Setting 0, Endpoints: 2
> > >      - Class Mass Storage, Transp. SCSI, Bulk only
> > >      - Endpoint 1 In Bulk MaxPacket 512
> > >      - Endpoint 2 Out Bulk MaxPacket 512
> > >
> > > 1: Hub,  USB Revision 3.0
> > >  - U-Boot XHCI Host Controller
> > >  - Class: Hub
> > >  - PacketSize: 512  Configurations: 1
> > >  - Vendor: 0x0000  Product 0x0000 Version 1.0
> > >    Configuration: 1
> > >    - Interfaces: 1 Self Powered 0mA
> > >      Interface: 0
> > >      - Alternate Setting 0, Endpoints: 1
> > >      - Class Hub
> > >      - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms
> > >
> > > 2: Mass Storage,  USB Revision 3.0
> > >  - SanDisk Cruzer Glide 3.0 4C530000130418116112
> > >  - Class: (from Interface) Mass Storage
> > >  - PacketSize: 512  Configurations: 1
> > >  - Vendor: 0x0781  Product 0x5597 Version 1.0
> > >    Configuration: 1
> > >    - Interfaces: 1 Bus Powered 224mA
> > >      Interface: 0
> > >      - Alternate Setting 0, Endpoints: 2
> > >      - Class Mass Storage, Transp. SCSI, Bulk only
> > >      - Endpoint 1 In Bulk MaxPacket 1024
> > >      - Endpoint 2 Out Bulk MaxPacket 1024
> > >
> > > ### The XHCI Host Controller and the SanDisk Cruzer Glide are now
> > > active as shown above.
> > >
> > > ### So list the files on this SanDisk Cruzer Glide to make sure we can read it.
> > >
> > >
> > > Pogo_V4> ext2ls usb 1:1 /
> > > device_probe:
> > > device_probe:
> > > device_probe: exit 0
> > > <DIR>       4096 .
> > > <DIR>       4096 ..
> > >           522666 uboot.2021.07-tld-1.pogo_v4.bin
> > >           524288 uboot.2021.07-tld-1.pogo_v4.mtd0.kwb
> > >
> > > Great works Pali!
> > >
> > > I think some more investigation is needed. Why did we need to do "pci
> > > enum", and then "usb reset", in that order, to get the PCI bus and the
> > > XHCI controller probed? there must be something missing in the process
> > > somewhere between the Device uclass, the PCI uclass, and the pci_mvebu
> > > uclass?
> >
> > I think I can see the order of enumeration. PCI must be enumerated
> > first, and then XHCI being the controller on this host bus would come
> > alive? I think we can live with 'pci enum' and 'usb reset' to get the
> > USB 3.0 drives enumerated. However, it seems just a little bit
> > unintuitive.
>
> 'pci enum' should be called internally by U-Boot during loading. So only
> 'usb start' would be needed.
>
> But from your boot log it looks like that PCI enumaration was not done
> and so calling 'pci enum' manually is needed.
>
> I will look into U-Boot code why it happens...
>
> Anyway, based on your test, PCIe must work correctly :)

Indeed, it's working perfectly :)  I've also commented out the 2 calls
to mvebu_mbus_add_window_by_id(), and no longer see the conflicts
error.

-       if (mvebu_mbus_add_window_by_id(pcie->mem_target, pcie->mem_attr,
-                                       (phys_addr_t)pcie->mem.start,
-                                       PCIE_MEM_SIZE)) {

-       if (mvebu_mbus_add_window_by_id(pcie->io_target, pcie->io_attr,
-                                       (phys_addr_t)pcie->io.start,
-                                       MBUS_PCI_IO_SIZE)) {


> Could you send config space dump of PCIe Root Port? Following U-Boot
> command prints it on terminal: 'pci display.b 0.0.0 0 200'

Sure, here is the log with the dump.


U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 22:52:05 -0800)
Pogoplug V4

SoC:   Kirkwood 88F6281_A1
DRAM:  128 MiB
NAND:  128 MiB
MMC:   mvsdio@90000: 0
Loading Environment from NAND... *** Warning - bad CRC, using default
environment

In:    serial
Out:   serial
Err:   serial
Net:
Warning: ethernet-controller@72000 (eth0) using random MAC address -
56:f0:8e:db:76:2f
eth0: ethernet-controller@72000
Hit any key to stop autoboot:  0
Pogo_V4> pci enum
Pogo_V4> pci 0
Scanning PCI devices on bus 0
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
00.00.00   0x11ab     0x6281     Bridge device           0x04
Pogo_V4> pci 1
Scanning PCI devices on bus 1
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
01.00.00   0x1b73     0x1009     Serial bus controller   0x03
Pogo_V4> pci display.b 0.0.0 0 200
00000000: ab 11 81 62 07 00 10 00 03 00 04 06 00 00 01 00
00000010: 00 00 00 00 00 00 00 00 00 01 01 00 01 f1 00 00
00000020: 00 90 00 90 01 10 01 00 00 00 00 00 00 00 00 00
00000030: 00 c0 ff bf 40 00 00 00 00 00 00 00 00 01 00 00
00000040: 01 50 03 06 00 00 00 00 00 00 00 00 00 00 00 00
00000050: 05 60 80 00 00 00 00 00 00 00 00 00 00 00 00 00
00000060: 10 00 41 00 80 80 00 00 00 20 00 00 11 ac 07 00
00000070: 08 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00
00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000100: 01 00 01 00 00 00 00 00 00 00 00 00 10 00 06 00
00000110: 00 00 00 00 00 20 00 00 00 00 00 00 01 00 00 4a
00000120: 04 00 00 01 00 00 08 01 0c 03 30 02 00 00 00 00
00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Pogo_V4>

Thanks,
Tony

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

On Monday 08 November 2021 12:54:39 Tony Dinh wrote:
> On Mon, Nov 8, 2021 at 3:11 AM Pali Rohár <pali@kernel.org> wrote:
> > On Sunday 07 November 2021 18:08:56 Tony Dinh wrote:
...
> > > > I think some more investigation is needed. Why did we need to do "pci
> > > > enum", and then "usb reset", in that order, to get the PCI bus and the
> > > > XHCI controller probed? there must be something missing in the process
> > > > somewhere between the Device uclass, the PCI uclass, and the pci_mvebu
> > > > uclass?
> > >
> > > I think I can see the order of enumeration. PCI must be enumerated
> > > first, and then XHCI being the controller on this host bus would come
> > > alive? I think we can live with 'pci enum' and 'usb reset' to get the
> > > USB 3.0 drives enumerated. However, it seems just a little bit
> > > unintuitive.
> >
> > 'pci enum' should be called internally by U-Boot during loading. So only
> > 'usb start' would be needed.
> >
> > But from your boot log it looks like that PCI enumaration was not done
> > and so calling 'pci enum' manually is needed.
> >
> > I will look into U-Boot code why it happens...

Ok. Command 'pci enum' is just calling pci_init() function.

So to avoid calling 'pci enum' manually, you need to put pci_init();
function call into your board board_early_init_r() function.

Which files in board/** are you using for your Kirkwood board? I do not
see any 6281 in board/Marvell/*.

> > Anyway, based on your test, PCIe must work correctly :)
> 
> Indeed, it's working perfectly :)  I've also commented out the 2 calls
> to mvebu_mbus_add_window_by_id(), and no longer see the conflicts
> error.

Perfect!

> 
> -       if (mvebu_mbus_add_window_by_id(pcie->mem_target, pcie->mem_attr,
> -                                       (phys_addr_t)pcie->mem.start,
> -                                       PCIE_MEM_SIZE)) {
> 
> -       if (mvebu_mbus_add_window_by_id(pcie->io_target, pcie->io_attr,
> -                                       (phys_addr_t)pcie->io.start,
> -                                       MBUS_PCI_IO_SIZE)) {
> 
> 
> > Could you send config space dump of PCIe Root Port? Following U-Boot
> > command prints it on terminal: 'pci display.b 0.0.0 0 200'
> 
> Sure, here is the log with the dump.

Thanks! Checked and seems to be correct.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

On Mon, Nov 8, 2021 at 2:02 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Monday 08 November 2021 12:54:39 Tony Dinh wrote:
> > On Mon, Nov 8, 2021 at 3:11 AM Pali Rohár <pali@kernel.org> wrote:
> > > On Sunday 07 November 2021 18:08:56 Tony Dinh wrote:
> ...
> > > > > I think some more investigation is needed. Why did we need to do "pci
> > > > > enum", and then "usb reset", in that order, to get the PCI bus and the
> > > > > XHCI controller probed? there must be something missing in the process
> > > > > somewhere between the Device uclass, the PCI uclass, and the pci_mvebu
> > > > > uclass?
> > > >
> > > > I think I can see the order of enumeration. PCI must be enumerated
> > > > first, and then XHCI being the controller on this host bus would come
> > > > alive? I think we can live with 'pci enum' and 'usb reset' to get the
> > > > USB 3.0 drives enumerated. However, it seems just a little bit
> > > > unintuitive.
> > >
> > > 'pci enum' should be called internally by U-Boot during loading. So only
> > > 'usb start' would be needed.
> > >
> > > But from your boot log it looks like that PCI enumaration was not done
> > > and so calling 'pci enum' manually is needed.
> > >
> > > I will look into U-Boot code why it happens...
>
> Ok. Command 'pci enum' is just calling pci_init() function.
>
> So to avoid calling 'pci enum' manually, you need to put pci_init();
> function call into your board board_early_init_r() function.

Thanks! will do that.

> Which files in board/** are you using for your Kirkwood board? I do not
> see any 6281 in board/Marvell/*.

This Pogoplug V4 board is still out-of-tree. I have not submitted
patches to add support for it. I plan to do that in the near future,
and it will be under board/cloudengines/.

However, there is a show stopper right now that prevents me from going
ahead to add this Pogoplug V4 board. A previous fdt patch has not been
accepted, because Simon wanted this to be implemented with livetree
calls:
https://lists.denx.de/pipermail/u-boot/2021-August/457311.html
It is working fine as a flat tree implementation.

The other 2 Kirkwood boards that could be useful in PCIe testing are
the Iomega iConnect (in mainline at board/iomega/iconnect/), and the
Zyxel NAS325 (still out-of-tree). I am also stuck on this NAS325 board
because of the fdt_support flattree vs. livetree implementation.

Thanks,
Tony




> > > Anyway, based on your test, PCIe must work correctly :)
> >
> > Indeed, it's working perfectly :)  I've also commented out the 2 calls
> > to mvebu_mbus_add_window_by_id(), and no longer see the conflicts
> > error.
>
> Perfect!
>
> >
> > -       if (mvebu_mbus_add_window_by_id(pcie->mem_target, pcie->mem_attr,
> > -                                       (phys_addr_t)pcie->mem.start,
> > -                                       PCIE_MEM_SIZE)) {
> >
> > -       if (mvebu_mbus_add_window_by_id(pcie->io_target, pcie->io_attr,
> > -                                       (phys_addr_t)pcie->io.start,
> > -                                       MBUS_PCI_IO_SIZE)) {
> >
> >
> > > Could you send config space dump of PCIe Root Port? Following U-Boot
> > > command prints it on terminal: 'pci display.b 0.0.0 0 200'
> >
> > Sure, here is the log with the dump.
>
> Thanks! Checked and seems to be correct.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Marek Behún]

On Mon, 8 Nov 2021 14:48:03 -0800
Tony Dinh <mibodhi@gmail.com> wrote:

> > So to avoid calling 'pci enum' manually, you need to put pci_init();
> > function call into your board board_early_init_r() function.  
> 
> Thanks! will do that.

Don't do that.

Instead enable CONFIG_PCI_INIT_R and CONFIG_SYS_EARLY_PCI_INIT, and the
code in common/board_r.c will do it itself.

Look at:

https://source.denx.de/u-boot/u-boot/-/blob/f8ed9059001d803b0eae4b49178789aa0e29edec/common/board_r.c#L675

Marek

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Marek and Pali,

On Mon, Nov 8, 2021 at 3:04 PM Marek Behún <marek.behun@nic.cz> wrote:
>
> On Mon, 8 Nov 2021 14:48:03 -0800
> Tony Dinh <mibodhi@gmail.com> wrote:
>
> > > So to avoid calling 'pci enum' manually, you need to put pci_init();
> > > function call into your board board_early_init_r() function.
> >
> > Thanks! will do that.
>
> Don't do that.
>
> Instead enable CONFIG_PCI_INIT_R and CONFIG_SYS_EARLY_PCI_INIT, and the
> code in common/board_r.c will do it itself.
>
> Look at:
>
> https://source.denx.de/u-boot/u-boot/-/blob/f8ed9059001d803b0eae4b49178789aa0e29edec/common/board_r.c#L675

Thanks for that advice. I also compared this code with the  "pci enum"
command. Indeed, like Pali said, it just calls pci_init() like in
board_r.c.

However, after enabling CONFIG_PCI_INIT_R and
CONFIG_SYS_EARLY_PCI_INIT, it hit a bug-on while scanning the USB bus!

U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 08 2021 - 22:17:10 -0800)
Pogoplug V4

SoC:   Kirkwood 88F6192_A1
DRAM:  128 MiB
device_probe:
device_probe: exit 0
mvebu_pcie_bind: begin
mvebu_pcie_bind: got an available node
Bound device  to pcie@82000000
mvebu_pcie_bind: bound
mvebu_pcie_bind: exit 0
Bound device pcie@82000000 to mbus@f1000000
Bound device mbus@f1000000 to root_driver
Bound device ehci@50000 to ocp@f1000000
Bound device ethernet-controller@72000 to ocp@f1000000
Bound device sata@80000 to ocp@f1000000
Bound device mvsdio@90000.blk to mvsdio@90000
Bound device mvsdio@90000 to ocp@f1000000
Bound device ocp@f1000000 to root_driver
NAND:  128 MiB
MMC:   device_probe:
device_probe:
device_probe:
device_probe: exit 0
device_probe: exit 0
device_probe:
device_probe:
mvsdio@90000: 0
Loading Environment from NAND... *** Warning - bad CRC, using default
environment

device_probe:
device_probe:
device_probe:
device_probe:
device_probe: exit 0
device_probe: exit 0
mvebu_pcie_probe:
mvebu_pcie_probe: exit 0
Bound device pci_0:0.0 to pcie0.0
device_probe:
device_probe:
Bound device xhci_pci to pci_0:0.0
device_probe: exit 0
device_probe: exit 0
device_probe:
In:    serial
Out:   serial
Err:   serial
Net:   device_probe:
device_probe:

Warning: ethernet-controller@72000 (eth0) using random MAC address -
82:09:1e:66:49:21
device_probe: exit 0
eth0: ethernet-controller@72000
Hit any key to stop autoboot:  0
Pogo_V4> pci 0
device_probe:
Scanning PCI devices on bus 0
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
00.00.00   0x11ab     0x6281     Bridge device           0x04
Pogo_V4> pci 1
device_probe:
Scanning PCI devices on bus 1
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
01.00.00   0x1b73     0x1009     Serial bus controller   0x03
Pogo_V4> usb start
starting USB...
Bus ehci@50000: device_probe:
device_probe:
USB EHCI 1.00
device_probe: exit 0
Bus xhci_pci: device_probe:
device_probe:
Register 400081f NbrPorts 4
Starting the controller
USB XHCI 1.00
device_probe: exit 0
scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
device_probe:
device_probe:
Bound device usb_hub to usb_hub
device_probe:
device_probe:
device_probe: exit 0
Bound device usb_mass_storage to usb_hub
device_probe:
device_probe:
Bound device usb_mass_storage.lun0 to usb_mass_storage
device_probe: exit 0
device_probe: exit 0
3 USB Device(s) found
scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
device_probe:
device_probe:
XHCI timeout on event type 33... cannot recover.
BUG at drivers/usb/host/xhci-ring.c:481/xhci_wait_for_event()!
BUG!
resetting ...

The above log was the build with the following configs:
CONFIG_CMD_PCI=y
CONFIG_PCI=y
CONFIG_PCI_MVEBU=y
CONFIG_PCI_PNP=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_PCI=y
CONFIG_USB_XHCI_MVEBU=y
CONFIG_PCI_INIT_R=y
CONFIG_SYS_EARLY_PCI_INIT=y


Any idea how to debug this? there must be some differences between the
pci_init call early on, and much later with "pci enum".

Thanks,
Tony

>
> Marek

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

On Monday 08 November 2021 22:34:51 Tony Dinh wrote:
> The above log was the build with the following configs:
> CONFIG_CMD_PCI=y
> CONFIG_PCI=y
> CONFIG_PCI_MVEBU=y
> CONFIG_PCI_PNP=y
> CONFIG_USB_XHCI_HCD=y
> CONFIG_USB_XHCI_PCI=y
> CONFIG_USB_XHCI_MVEBU=y
> CONFIG_PCI_INIT_R=y
> CONFIG_SYS_EARLY_PCI_INIT=y
> 
> 
> Any idea how to debug this? there must be some differences between the
> pci_init call early on, and much later with "pci enum".

Hello! The only difference is when pci_init() function is called. So it
looks like that it needs to be called later (perhaps because PCIe needs
some other steps for initialization?).

Could you try to disable CONFIG_SYS_EARLY_PCI_INIT option if it helps?

And could you try to put pci_init() into board_early_init_r() and
disable both CONFIG_SYS_EARLY_PCI_INIT and CONFIG_PCI_INIT_R if it
helps?

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

On Tue, Nov 9, 2021 at 7:08 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Monday 08 November 2021 22:34:51 Tony Dinh wrote:
> > The above log was the build with the following configs:
> > CONFIG_CMD_PCI=y
> > CONFIG_PCI=y
> > CONFIG_PCI_MVEBU=y
> > CONFIG_PCI_PNP=y
> > CONFIG_USB_XHCI_HCD=y
> > CONFIG_USB_XHCI_PCI=y
> > CONFIG_USB_XHCI_MVEBU=y
> > CONFIG_PCI_INIT_R=y
> > CONFIG_SYS_EARLY_PCI_INIT=y
> >
> >
> > Any idea how to debug this? there must be some differences between the
> > pci_init call early on, and much later with "pci enum".
>
> Hello! The only difference is when pci_init() function is called. So it
> looks like that it needs to be called later (perhaps because PCIe needs
> some other steps for initialization?).
>
> Could you try to disable CONFIG_SYS_EARLY_PCI_INIT option if it helps?

Please correct me if I'm wrong. It is a compound condition in
board_r.c, so it would not be executed anyway?

#if defined(CONFIG_PCI_INIT_R) && defined(CONFIG_SYS_EARLY_PCI_INIT)


> And could you try to put pci_init() into board_early_init_r() and
> disable both CONFIG_SYS_EARLY_PCI_INIT and CONFIG_PCI_INIT_R if it
> helps?

This did not help. I've added board_early_init_r to the board file,
and called pci_init() there. In the debug output, I can see pci_init()
executed early. But later, "usb start" still hangs at EHCI enumeration
like before.

scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000

Strangely, on a hunch, I ran it a second time and then did "pci enum"
and "usb start", I got a bit further. Now it scans the XHCI bus, and
gets 3 USB devices, but then hangs.

scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
Bound device usb_hub to usb_hub
Bound device usb_mass_storage to usb_hub
Bound device usb_mass_storage.lun0 to usb_mass_storage
3 USB Device(s) found
scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci

So it seems that after the board has finished initialization, calling
pci_init() again from "pci enum" will result in more probing
processing.

Thanks,
Tony

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

Hello!

On Tuesday 09 November 2021 14:51:33 Tony Dinh wrote:
> Hi Pali,
> 
> On Tue, Nov 9, 2021 at 7:08 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Monday 08 November 2021 22:34:51 Tony Dinh wrote:
> > > The above log was the build with the following configs:
> > > CONFIG_CMD_PCI=y
> > > CONFIG_PCI=y
> > > CONFIG_PCI_MVEBU=y
> > > CONFIG_PCI_PNP=y
> > > CONFIG_USB_XHCI_HCD=y
> > > CONFIG_USB_XHCI_PCI=y
> > > CONFIG_USB_XHCI_MVEBU=y
> > > CONFIG_PCI_INIT_R=y
> > > CONFIG_SYS_EARLY_PCI_INIT=y
> > >
> > >
> > > Any idea how to debug this? there must be some differences between the
> > > pci_init call early on, and much later with "pci enum".
> >
> > Hello! The only difference is when pci_init() function is called. So it
> > looks like that it needs to be called later (perhaps because PCIe needs
> > some other steps for initialization?).
> >
> > Could you try to disable CONFIG_SYS_EARLY_PCI_INIT option if it helps?
> 
> Please correct me if I'm wrong. It is a compound condition in
> board_r.c, so it would not be executed anyway?
> 
> #if defined(CONFIG_PCI_INIT_R) && defined(CONFIG_SYS_EARLY_PCI_INIT)
> 

And few lines below is section:

#if defined(CONFIG_PCI_INIT_R) && !defined(CONFIG_SYS_EARLY_PCI_INIT)
	/*
	 * Do pci configuration
	 */
	pci_init,
#endif

So CONFIG_SYS_EARLY_PCI_INIT controls if pci_init() is called earlier or
later.

> > And could you try to put pci_init() into board_early_init_r() and
> > disable both CONFIG_SYS_EARLY_PCI_INIT and CONFIG_PCI_INIT_R if it
> > helps?
> 
> This did not help. I've added board_early_init_r to the board file,
> and called pci_init() there. In the debug output, I can see pci_init()
> executed early. But later, "usb start" still hangs at EHCI enumeration
> like before.
> 
> scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> 
> Strangely, on a hunch, I ran it a second time and then did "pci enum"
> and "usb start", I got a bit further. Now it scans the XHCI bus, and
> gets 3 USB devices, but then hangs.
> 
> scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> Bound device usb_hub to usb_hub
> Bound device usb_mass_storage to usb_hub
> Bound device usb_mass_storage.lun0 to usb_mass_storage
> 3 USB Device(s) found
> scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
> 
> So it seems that after the board has finished initialization, calling
> pci_init() again from "pci enum" will result in more probing
> processing.

So it means that there are some timing issues.

Could you try to add "mdelay(200);" at the end of mvebu_pcie_probe()
function? For mdelay you probably need also "#include <linux/delay.h>".

Or could you try to add "int board_late_init(void) { pci_init(); return 0; }"
into your board code?

In every test ensure that pci_init() is called only once. So if adding
pci_init() in board code then disable CONFIG_PCI_INIT_R.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

On Tue, Nov 9, 2021 at 3:05 PM Pali Rohár <pali@kernel.org> wrote:
>
> Hello!
>
> On Tuesday 09 November 2021 14:51:33 Tony Dinh wrote:
> > Hi Pali,
> >
> > On Tue, Nov 9, 2021 at 7:08 AM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Monday 08 November 2021 22:34:51 Tony Dinh wrote:
> > > > The above log was the build with the following configs:
> > > > CONFIG_CMD_PCI=y
> > > > CONFIG_PCI=y
> > > > CONFIG_PCI_MVEBU=y
> > > > CONFIG_PCI_PNP=y
> > > > CONFIG_USB_XHCI_HCD=y
> > > > CONFIG_USB_XHCI_PCI=y
> > > > CONFIG_USB_XHCI_MVEBU=y
> > > > CONFIG_PCI_INIT_R=y
> > > > CONFIG_SYS_EARLY_PCI_INIT=y
> > > >
> > > >
> > > > Any idea how to debug this? there must be some differences between the
> > > > pci_init call early on, and much later with "pci enum".
> > >
> > > Hello! The only difference is when pci_init() function is called. So it
> > > looks like that it needs to be called later (perhaps because PCIe needs
> > > some other steps for initialization?).
> > >
> > > Could you try to disable CONFIG_SYS_EARLY_PCI_INIT option if it helps?
> >
> > Please correct me if I'm wrong. It is a compound condition in
> > board_r.c, so it would not be executed anyway?
> >
> > #if defined(CONFIG_PCI_INIT_R) && defined(CONFIG_SYS_EARLY_PCI_INIT)
> >
>
> And few lines below is section:
>
> #if defined(CONFIG_PCI_INIT_R) && !defined(CONFIG_SYS_EARLY_PCI_INIT)
>         /*
>          * Do pci configuration
>          */
>         pci_init,
> #endif
>
> So CONFIG_SYS_EARLY_PCI_INIT controls if pci_init() is called earlier or
> later.
>
> > > And could you try to put pci_init() into board_early_init_r() and
> > > disable both CONFIG_SYS_EARLY_PCI_INIT and CONFIG_PCI_INIT_R if it
> > > helps?
> >
> > This did not help. I've added board_early_init_r to the board file,
> > and called pci_init() there. In the debug output, I can see pci_init()
> > executed early. But later, "usb start" still hangs at EHCI enumeration
> > like before.
> >
> > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> >
> > Strangely, on a hunch, I ran it a second time and then did "pci enum"
> > and "usb start", I got a bit further. Now it scans the XHCI bus, and
> > gets 3 USB devices, but then hangs.
> >
> > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> > Bound device usb_hub to usb_hub
> > Bound device usb_mass_storage to usb_hub
> > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > 3 USB Device(s) found
> > scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
> >
> > So it seems that after the board has finished initialization, calling
> > pci_init() again from "pci enum" will result in more probing
> > processing.
>
> So it means that there are some timing issues.
>
> Could you try to add "mdelay(200);" at the end of mvebu_pcie_probe()
> function? For mdelay you probably need also "#include <linux/delay.h>".
>

I have not tried this, given we are successful with board_late_init now.

> Or could you try to add "int board_late_init(void) { pci_init(); return 0; }"
> into your board code?

board_late_init is working! Here is the log.


U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 09 2021 - 18:28:37 -0800)
Pogoplug V4

SoC:   Kirkwood 88F6192_A1
DRAM:  128 MiB
device_probe:
device_probe: exit 0
mvebu_pcie_bind: begin
mvebu_pcie_bind: got an available node
Bound device  to pcie@82000000
mvebu_pcie_bind: bound
mvebu_pcie_bind: exit 0
Bound device pcie@82000000 to mbus@f1000000
Bound device mbus@f1000000 to root_driver
Bound device ehci@50000 to ocp@f1000000
Bound device ethernet-controller@72000 to ocp@f1000000
Bound device sata@80000 to ocp@f1000000
Bound device mvsdio@90000.blk to mvsdio@90000
Bound device mvsdio@90000 to ocp@f1000000
Bound device ocp@f1000000 to root_driver
NAND:  128 MiB
MMC:   device_probe:
device_probe:
device_probe:
device_probe: exit 0
device_probe: exit 0
device_probe:
device_probe:
mvsdio@90000: 0
Loading Environment from NAND... *** Warning - bad CRC, using default
environment

In:    serial
Out:   serial
Err:   serial
pci_init:
device_probe:
device_probe:
device_probe:
device_probe:
device_probe: exit 0
device_probe: exit 0
mvebu_pcie_probe:
mvebu_pcie_probe: exit 0
Bound device pci_0:0.0 to pcie0.0
device_probe:
device_probe:
Bound device xhci_pci to pci_0:0.0
device_probe: exit 0
device_probe: exit 0
device_probe:
pci_init: exit 0
Net:   device_probe:
device_probe:

Warning: ethernet-controller@72000 (eth0) using random MAC address -
4a:8f:e6:9d:8b:79
device_probe: exit 0
eth0: ethernet-controller@72000
Hit any key to stop autoboot:  0

Pogo_V4> pci 0
device_probe:
Scanning PCI devices on bus 0
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
00.00.00   0x11ab     0x6281     Bridge device           0x04
Pogo_V4> pci 1
device_probe:
Scanning PCI devices on bus 1
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
01.00.00   0x1b73     0x1009     Serial bus controller   0x03

Pogo_V4> usb start
starting USB...
Bus ehci@50000: device_probe:
device_probe:
USB EHCI 1.00
device_probe: exit 0
Bus xhci_pci: device_probe:
device_probe:
Register 400081f NbrPorts 4
Starting the controller
USB XHCI 1.00
device_probe: exit 0
scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
device_probe:
device_probe:
Bound device usb_hub to usb_hub
device_probe:
device_probe:
device_probe: exit 0
Bound device usb_mass_storage to usb_hub
device_probe:
device_probe:
Bound device usb_mass_storage.lun0 to usb_mass_storage
device_probe: exit 0
device_probe: exit 0
3 USB Device(s) found
scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
device_probe:
device_probe:
Bound device usb_mass_storage to usb_hub
device_probe:
device_probe:
Bound device usb_mass_storage.lun0 to usb_mass_storage
device_probe: exit 0
device_probe: exit 0
2 USB Device(s) found
       scanning usb for storage devices... 2 Storage Device(s) found

Pogo_V4> usb tree
USB device tree:
  1  Hub (480 Mb/s, 0mA)
  |  u-boot EHCI Host Controller
  |
  +-2  Hub (480 Mb/s, 100mA)
    |  GenesysLogic USB2.0 Hub
    |
    +-3  Mass Storage (480 Mb/s, 224mA)
         SanDisk Ultra Fit 4C531001560827107320

  1  Hub (5 Gb/s, 0mA)
  |  U-Boot XHCI Host Controller
  |
  +-2  Mass Storage (5 Gb/s, 224mA)
       SanDisk Cruzer Glide 3.0 4C530000130418116112

I think we got the problem identified! Most of other boards, the
pci_init is called in early_init. Perhaps Kirwood is different. or
perhaps there is some timing problem lurking in there somewhere?. I
will try this new patch with the Zyxel NSA325 u-boot to see if we can
observe the same behavior.

And thanks for all the great work! I'll submit patches to add support
for this Pogoplug V4 board if you think we are done enough for now.
This board is a really good vehicle for Kirkwood SoCs testing (SATA,
USB, PCIe, MMC, Gbit Ethernet), even more than the Sheevaplug can do.

Thanks,
Tony


> In every test ensure that pci_init() is called only once. So if adding
> pci_init() in board code then disable CONFIG_PCI_INIT_R.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

I've tried the test with mdelay(200) at the end of mvebu_pcie_probe().
Please see below.

On Tue, Nov 9, 2021 at 7:17 PM Tony Dinh <mibodhi@gmail.com> wrote:
>
> Hi Pali,
>
> On Tue, Nov 9, 2021 at 3:05 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > Hello!
> >
> > On Tuesday 09 November 2021 14:51:33 Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > On Tue, Nov 9, 2021 at 7:08 AM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Monday 08 November 2021 22:34:51 Tony Dinh wrote:
> > > > > The above log was the build with the following configs:
> > > > > CONFIG_CMD_PCI=y
> > > > > CONFIG_PCI=y
> > > > > CONFIG_PCI_MVEBU=y
> > > > > CONFIG_PCI_PNP=y
> > > > > CONFIG_USB_XHCI_HCD=y
> > > > > CONFIG_USB_XHCI_PCI=y
> > > > > CONFIG_USB_XHCI_MVEBU=y
> > > > > CONFIG_PCI_INIT_R=y
> > > > > CONFIG_SYS_EARLY_PCI_INIT=y
> > > > >
> > > > >
> > > > > Any idea how to debug this? there must be some differences between the
> > > > > pci_init call early on, and much later with "pci enum".
> > > >
> > > > Hello! The only difference is when pci_init() function is called. So it
> > > > looks like that it needs to be called later (perhaps because PCIe needs
> > > > some other steps for initialization?).
> > > >
> > > > Could you try to disable CONFIG_SYS_EARLY_PCI_INIT option if it helps?
> > >
> > > Please correct me if I'm wrong. It is a compound condition in
> > > board_r.c, so it would not be executed anyway?
> > >
> > > #if defined(CONFIG_PCI_INIT_R) && defined(CONFIG_SYS_EARLY_PCI_INIT)
> > >
> >
> > And few lines below is section:
> >
> > #if defined(CONFIG_PCI_INIT_R) && !defined(CONFIG_SYS_EARLY_PCI_INIT)
> >         /*
> >          * Do pci configuration
> >          */
> >         pci_init,
> > #endif
> >
> > So CONFIG_SYS_EARLY_PCI_INIT controls if pci_init() is called earlier or
> > later.
> >
> > > > And could you try to put pci_init() into board_early_init_r() and
> > > > disable both CONFIG_SYS_EARLY_PCI_INIT and CONFIG_PCI_INIT_R if it
> > > > helps?
> > >
> > > This did not help. I've added board_early_init_r to the board file,
> > > and called pci_init() there. In the debug output, I can see pci_init()
> > > executed early. But later, "usb start" still hangs at EHCI enumeration
> > > like before.
> > >
> > > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> > >
> > > Strangely, on a hunch, I ran it a second time and then did "pci enum"
> > > and "usb start", I got a bit further. Now it scans the XHCI bus, and
> > > gets 3 USB devices, but then hangs.
> > >
> > > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> > > Bound device usb_hub to usb_hub
> > > Bound device usb_mass_storage to usb_hub
> > > Bound device usb_mass_storage.lun0 to usb_mass_storage
> > > 3 USB Device(s) found
> > > scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
> > >
> > > So it seems that after the board has finished initialization, calling
> > > pci_init() again from "pci enum" will result in more probing
> > > processing.
> >
> > So it means that there are some timing issues.
> >
> > Could you try to add "mdelay(200);" at the end of mvebu_pcie_probe()
> > function? For mdelay you probably need also "#include <linux/delay.h>".

I've tried the mdelay(200) at the end of mvebu_pcie_probe(). In this
test I set the configs is back to:

CONFIG_CMD_PCI=y
CONFIG_PCI=y
CONFIG_PCI_MVEBU=y
CONFIG_PCI_PNP=y
CONFIG_USB_XHCI_HCD=y
CONFIG_USB_XHCI_PCI=y
CONFIG_USB_XHCI_MVEBU=y
CONFIG_PCI_INIT_R=y
CONFIG_SYS_EARLY_PCI_INIT=y

The behavior is the same as before. It hangs at "usb start".

U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 10 2021 - 13:56:09 -0800)
Pogoplug V4

SoC:   Kirkwood 88F6192_A1
DRAM:  128 MiB
device_probe:
device_probe: exit 0
mvebu_pcie_bind: begin
mvebu_pcie_bind: got an available node
Bound device  to pcie@82000000
mvebu_pcie_bind: bound
mvebu_pcie_bind: exit 0
Bound device pcie@82000000 to mbus@f1000000
Bound device mbus@f1000000 to root_driver
Bound device ehci@50000 to ocp@f1000000
Bound device ethernet-controller@72000 to ocp@f1000000
Bound device sata@80000 to ocp@f1000000
Bound device mvsdio@90000.blk to mvsdio@90000
Bound device mvsdio@90000 to ocp@f1000000
Bound device ocp@f1000000 to root_driver
NAND:  128 MiB
MMC:   device_probe:
device_probe:
device_probe:
device_probe: exit 0
device_probe: exit 0
device_probe:
device_probe:
mvsdio@90000: 0
Loading Environment from NAND... *** Warning - bad CRC, using default
environment

pci_init:
device_probe:
device_probe:
device_probe:
device_probe:
device_probe: exit 0
device_probe: exit 0
mvebu_pcie_probe:
mvebu_pcie_probe: exit 0
Bound device pci_0:0.0 to pcie0.0
device_probe:
device_probe:
Bound device xhci_pci to pci_0:0.0
device_probe: exit 0
device_probe: exit 0
device_probe:
pci_init: exit 0
In:    serial
Out:   serial
Err:   serial
Net:   device_probe:
device_probe:

Warning: ethernet-controller@72000 (eth0) using random MAC address -
a2:f2:2b:f9:74:ac
device_probe: exit 0
eth0: ethernet-controller@72000
Hit any key to stop autoboot:  0
Pogo_V4> pci 0
device_probe:
Scanning PCI devices on bus 0
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
00.00.00   0x11ab     0x6281     Bridge device           0x04
Pogo_V4> pci 1
device_probe:
Scanning PCI devices on bus 1
BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
_____________________________________________________________
01.00.00   0x1b73     0x1009     Serial bus controller   0x03
Pogo_V4> usb start
starting USB...
Bus ehci@50000: device_probe:
device_probe:
USB EHCI 1.00
device_probe: exit 0
Bus xhci_pci: device_probe:
device_probe:
Register 400081f NbrPorts 4
Starting the controller
USB XHCI 1.00
device_probe: exit 0
scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
device_probe:
device_probe:

I've also tried mdelay(3000), just to be sure. The result was the same
hang in 'usb start'.

Thanks,
Tony



> >
>
> I have not tried this, given we are successful with board_late_init now.
>
> > Or could you try to add "int board_late_init(void) { pci_init(); return 0; }"
> > into your board code?
>
> board_late_init is working! Here is the log.
>
>
> U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 09 2021 - 18:28:37 -0800)
> Pogoplug V4
>
> SoC:   Kirkwood 88F6192_A1
> DRAM:  128 MiB
> device_probe:
> device_probe: exit 0
> mvebu_pcie_bind: begin
> mvebu_pcie_bind: got an available node
> Bound device  to pcie@82000000
> mvebu_pcie_bind: bound
> mvebu_pcie_bind: exit 0
> Bound device pcie@82000000 to mbus@f1000000
> Bound device mbus@f1000000 to root_driver
> Bound device ehci@50000 to ocp@f1000000
> Bound device ethernet-controller@72000 to ocp@f1000000
> Bound device sata@80000 to ocp@f1000000
> Bound device mvsdio@90000.blk to mvsdio@90000
> Bound device mvsdio@90000 to ocp@f1000000
> Bound device ocp@f1000000 to root_driver
> NAND:  128 MiB
> MMC:   device_probe:
> device_probe:
> device_probe:
> device_probe: exit 0
> device_probe: exit 0
> device_probe:
> device_probe:
> mvsdio@90000: 0
> Loading Environment from NAND... *** Warning - bad CRC, using default
> environment
>
> In:    serial
> Out:   serial
> Err:   serial
> pci_init:
> device_probe:
> device_probe:
> device_probe:
> device_probe:
> device_probe: exit 0
> device_probe: exit 0
> mvebu_pcie_probe:
> mvebu_pcie_probe: exit 0
> Bound device pci_0:0.0 to pcie0.0
> device_probe:
> device_probe:
> Bound device xhci_pci to pci_0:0.0
> device_probe: exit 0
> device_probe: exit 0
> device_probe:
> pci_init: exit 0
> Net:   device_probe:
> device_probe:
>
> Warning: ethernet-controller@72000 (eth0) using random MAC address -
> 4a:8f:e6:9d:8b:79
> device_probe: exit 0
> eth0: ethernet-controller@72000
> Hit any key to stop autoboot:  0
>
> Pogo_V4> pci 0
> device_probe:
> Scanning PCI devices on bus 0
> BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
> _____________________________________________________________
> 00.00.00   0x11ab     0x6281     Bridge device           0x04
> Pogo_V4> pci 1
> device_probe:
> Scanning PCI devices on bus 1
> BusDevFun  VendorId   DeviceId   Device Class       Sub-Class
> _____________________________________________________________
> 01.00.00   0x1b73     0x1009     Serial bus controller   0x03
>
> Pogo_V4> usb start
> starting USB...
> Bus ehci@50000: device_probe:
> device_probe:
> USB EHCI 1.00
> device_probe: exit 0
> Bus xhci_pci: device_probe:
> device_probe:
> Register 400081f NbrPorts 4
> Starting the controller
> USB XHCI 1.00
> device_probe: exit 0
> scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000
> device_probe:
> device_probe:
> Bound device usb_hub to usb_hub
> device_probe:
> device_probe:
> device_probe: exit 0
> Bound device usb_mass_storage to usb_hub
> device_probe:
> device_probe:
> Bound device usb_mass_storage.lun0 to usb_mass_storage
> device_probe: exit 0
> device_probe: exit 0
> 3 USB Device(s) found
> scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci
> device_probe:
> device_probe:
> Bound device usb_mass_storage to usb_hub
> device_probe:
> device_probe:
> Bound device usb_mass_storage.lun0 to usb_mass_storage
> device_probe: exit 0
> device_probe: exit 0
> 2 USB Device(s) found
>        scanning usb for storage devices... 2 Storage Device(s) found
>
> Pogo_V4> usb tree
> USB device tree:
>   1  Hub (480 Mb/s, 0mA)
>   |  u-boot EHCI Host Controller
>   |
>   +-2  Hub (480 Mb/s, 100mA)
>     |  GenesysLogic USB2.0 Hub
>     |
>     +-3  Mass Storage (480 Mb/s, 224mA)
>          SanDisk Ultra Fit 4C531001560827107320
>
>   1  Hub (5 Gb/s, 0mA)
>   |  U-Boot XHCI Host Controller
>   |
>   +-2  Mass Storage (5 Gb/s, 224mA)
>        SanDisk Cruzer Glide 3.0 4C530000130418116112
>
> I think we got the problem identified! Most of other boards, the
> pci_init is called in early_init. Perhaps Kirwood is different. or
> perhaps there is some timing problem lurking in there somewhere?. I
> will try this new patch with the Zyxel NSA325 u-boot to see if we can
> observe the same behavior.
>
> And thanks for all the great work! I'll submit patches to add support
> for this Pogoplug V4 board if you think we are done enough for now.
> This board is a really good vehicle for Kirkwood SoCs testing (SATA,
> USB, PCIe, MMC, Gbit Ethernet), even more than the Sheevaplug can do.
>
> Thanks,
> Tony
>
>
> > In every test ensure that pci_init() is called only once. So if adding
> > pci_init() in board code then disable CONFIG_PCI_INIT_R.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

On Wednesday 10 November 2021 16:04:20 Tony Dinh wrote:
> I've also tried mdelay(3000), just to be sure. The result was the same
> hang in 'usb start'.

Ok. So put pci_init() into board_late_init().

There are some more cleanup and fixes patches for pci_mvebu.c on mailing
list. After they are merged I will prepare and send final PCI Kirkwood
patch for testing.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Stefan Roese]

On 11/11/21 22:10, Pali Rohár wrote:
> On Wednesday 10 November 2021 16:04:20 Tony Dinh wrote:
>> I've also tried mdelay(3000), just to be sure. The result was the same
>> hang in 'usb start'.
> 
> Ok. So put pci_init() into board_late_init().
> 
> There are some more cleanup and fixes patches for pci_mvebu.c on mailing
> list. After they are merged I will prepare and send final PCI Kirkwood
> patch for testing.

Chiming in a bit late in this discussion:

Is an explicit call to pci_init() necessary in this Kirwood case? IIRC,
the DM infrastructure should make sure that all device are probed - but
only when really needed. So if you don't need PCI in the boot process
at all, then it's not probed at all. Or if you set CONFIG_PCI_PNP this
will be done always. Then there should be no need for the additional
"pci enum".

I might be missing something - did not check in depth.

Thanks,
Stefan

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

On Friday 12 November 2021 15:36:31 Stefan Roese wrote:
> On 11/11/21 22:10, Pali Rohár wrote:
> > On Wednesday 10 November 2021 16:04:20 Tony Dinh wrote:
> > > I've also tried mdelay(3000), just to be sure. The result was the same
> > > hang in 'usb start'.
> > 
> > Ok. So put pci_init() into board_late_init().
> > 
> > There are some more cleanup and fixes patches for pci_mvebu.c on mailing
> > list. After they are merged I will prepare and send final PCI Kirkwood
> > patch for testing.
> 
> Chiming in a bit late in this discussion:
> 
> Is an explicit call to pci_init() necessary in this Kirwood case? IIRC,
> the DM infrastructure should make sure that all device are probed - but
> only when really needed. So if you don't need PCI in the boot process
> at all, then it's not probed at all. Or if you set CONFIG_PCI_PNP this
> will be done always. Then there should be no need for the additional
> "pci enum".

CONFIG_PCI_PNP probe and initialize all devices behind host bridge.
CONFIG_PCI_INIT_R probe and initialize host bridge (this is done by
calling pci_init()). Based on tests (see discussion in this thread) it
looks like that CONFIG_PCI_INIT_R calls pci_init() too early and
initialization is failing. So unsetting CONFIG_PCI_INIT_R and calling
pci_init() manually from board_late_init() (which is called later than
CONFIG_PCI_INIT_R) seems to fix these issues. No idea why...

> I might be missing something - did not check in depth.
> 
> Thanks,
> Stefan

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Stefan & Pali,

On Fri, Nov 12, 2021 at 7:06 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Friday 12 November 2021 15:36:31 Stefan Roese wrote:
> > On 11/11/21 22:10, Pali Rohár wrote:
> > > On Wednesday 10 November 2021 16:04:20 Tony Dinh wrote:
> > > > I've also tried mdelay(3000), just to be sure. The result was the same
> > > > hang in 'usb start'.
> > >
> > > Ok. So put pci_init() into board_late_init().
> > >
> > > There are some more cleanup and fixes patches for pci_mvebu.c on mailing
> > > list. After they are merged I will prepare and send final PCI Kirkwood
> > > patch for testing.
> >
> > Chiming in a bit late in this discussion:
> >
> > Is an explicit call to pci_init() necessary in this Kirwood case? IIRC,
> > the DM infrastructure should make sure that all device are probed - but
> > only when really needed. So if you don't need PCI in the boot process
> > at all, then it's not probed at all. Or if you set CONFIG_PCI_PNP this
> > will be done always. Then there should be no need for the additional
> > "pci enum".
>
> CONFIG_PCI_PNP probe and initialize all devices behind host bridge.
> CONFIG_PCI_INIT_R probe and initialize host bridge (this is done by
> calling pci_init()). Based on tests (see discussion in this thread) it
> looks like that CONFIG_PCI_INIT_R calls pci_init() too early and
> initialization is failing. So unsetting CONFIG_PCI_INIT_R and calling
> pci_init() manually from board_late_init() (which is called later than
> CONFIG_PCI_INIT_R) seems to fix these issues. No idea why...
> > I might be missing something - did not check in depth.

I think this must be the problem. Sorry I've missed this:

config USB_XHCI_MVEBU
        bool "MVEBU USB 3.0 support"
        default y
        depends on ARCH_MVEBU
        select DM_REGULATOR
        help
          Choose this option to add support for USB 3.0 driver on mvebu
          SoCs, which includes Armada8K, Armada3700 and other Armada
          family SoCs.

So I'll need rebuild with "depends on (ARCH_MVEBU || ARCH_KIRKWOOD)"
and do some more testing.

Thanks,
Tony

> >
> > Thanks,
> > Stefan

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Also probably a new Kirkwood udevice is needed in
drivers/usb/host/xhci-mvebu.c (if armada-380-xhci is not appropriate).

static const struct udevice_id xhci_usb_ids[] = {
        { .compatible = "marvell,armada3700-xhci" },
        { .compatible = "marvell,armada-380-xhci" },
        { .compatible = "marvell,armada-8k-xhci" },
        { }
};

But I guess the fact that it works during board_late_init must be
because of a generic USB_XHCI driver being used?

Thanks,
Tony

On Fri, Nov 12, 2021 at 1:46 PM Tony Dinh <mibodhi@gmail.com> wrote:
>
> Hi Stefan & Pali,
>
> On Fri, Nov 12, 2021 at 7:06 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Friday 12 November 2021 15:36:31 Stefan Roese wrote:
> > > On 11/11/21 22:10, Pali Rohár wrote:
> > > > On Wednesday 10 November 2021 16:04:20 Tony Dinh wrote:
> > > > > I've also tried mdelay(3000), just to be sure. The result was the same
> > > > > hang in 'usb start'.
> > > >
> > > > Ok. So put pci_init() into board_late_init().
> > > >
> > > > There are some more cleanup and fixes patches for pci_mvebu.c on mailing
> > > > list. After they are merged I will prepare and send final PCI Kirkwood
> > > > patch for testing.
> > >
> > > Chiming in a bit late in this discussion:
> > >
> > > Is an explicit call to pci_init() necessary in this Kirwood case? IIRC,
> > > the DM infrastructure should make sure that all device are probed - but
> > > only when really needed. So if you don't need PCI in the boot process
> > > at all, then it's not probed at all. Or if you set CONFIG_PCI_PNP this
> > > will be done always. Then there should be no need for the additional
> > > "pci enum".
> >
> > CONFIG_PCI_PNP probe and initialize all devices behind host bridge.
> > CONFIG_PCI_INIT_R probe and initialize host bridge (this is done by
> > calling pci_init()). Based on tests (see discussion in this thread) it
> > looks like that CONFIG_PCI_INIT_R calls pci_init() too early and
> > initialization is failing. So unsetting CONFIG_PCI_INIT_R and calling
> > pci_init() manually from board_late_init() (which is called later than
> > CONFIG_PCI_INIT_R) seems to fix these issues. No idea why...
> > > I might be missing something - did not check in depth.
>
> I think this must be the problem. Sorry I've missed this:
>
> config USB_XHCI_MVEBU
>         bool "MVEBU USB 3.0 support"
>         default y
>         depends on ARCH_MVEBU
>         select DM_REGULATOR
>         help
>           Choose this option to add support for USB 3.0 driver on mvebu
>           SoCs, which includes Armada8K, Armada3700 and other Armada
>           family SoCs.
>
> So I'll need rebuild with "depends on (ARCH_MVEBU || ARCH_KIRKWOOD)"
> and do some more testing.
>
> Thanks,
> Tony
>
> > >
> > > Thanks,
> > > Stefan

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

On Friday 12 November 2021 13:46:29 Tony Dinh wrote:
> Hi Stefan & Pali,
> 
> On Fri, Nov 12, 2021 at 7:06 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Friday 12 November 2021 15:36:31 Stefan Roese wrote:
> > > On 11/11/21 22:10, Pali Rohár wrote:
> > > > On Wednesday 10 November 2021 16:04:20 Tony Dinh wrote:
> > > > > I've also tried mdelay(3000), just to be sure. The result was the same
> > > > > hang in 'usb start'.
> > > >
> > > > Ok. So put pci_init() into board_late_init().
> > > >
> > > > There are some more cleanup and fixes patches for pci_mvebu.c on mailing
> > > > list. After they are merged I will prepare and send final PCI Kirkwood
> > > > patch for testing.
> > >
> > > Chiming in a bit late in this discussion:
> > >
> > > Is an explicit call to pci_init() necessary in this Kirwood case? IIRC,
> > > the DM infrastructure should make sure that all device are probed - but
> > > only when really needed. So if you don't need PCI in the boot process
> > > at all, then it's not probed at all. Or if you set CONFIG_PCI_PNP this
> > > will be done always. Then there should be no need for the additional
> > > "pci enum".
> >
> > CONFIG_PCI_PNP probe and initialize all devices behind host bridge.
> > CONFIG_PCI_INIT_R probe and initialize host bridge (this is done by
> > calling pci_init()). Based on tests (see discussion in this thread) it
> > looks like that CONFIG_PCI_INIT_R calls pci_init() too early and
> > initialization is failing. So unsetting CONFIG_PCI_INIT_R and calling
> > pci_init() manually from board_late_init() (which is called later than
> > CONFIG_PCI_INIT_R) seems to fix these issues. No idea why...
> > > I might be missing something - did not check in depth.
> 
> I think this must be the problem. Sorry I've missed this:
> 
> config USB_XHCI_MVEBU
>         bool "MVEBU USB 3.0 support"
>         default y
>         depends on ARCH_MVEBU
>         select DM_REGULATOR
>         help
>           Choose this option to add support for USB 3.0 driver on mvebu
>           SoCs, which includes Armada8K, Armada3700 and other Armada
>           family SoCs.

This is used for native USB 3.0 controller in Marvell SoC (connected via
serdes).

But you wrote and sent pci output, that you have PCIe-based USB
controller connected via PCIe.

> So I'll need rebuild with "depends on (ARCH_MVEBU || ARCH_KIRKWOOD)"
> and do some more testing.
> 
> Thanks,
> Tony
> 
> > >
> > > Thanks,
> > > Stefan

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

On Fri, Nov 12, 2021 at 2:42 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Friday 12 November 2021 13:46:29 Tony Dinh wrote:
> > Hi Stefan & Pali,
> >
> > On Fri, Nov 12, 2021 at 7:06 AM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Friday 12 November 2021 15:36:31 Stefan Roese wrote:
> > > > On 11/11/21 22:10, Pali Rohár wrote:
> > > > > On Wednesday 10 November 2021 16:04:20 Tony Dinh wrote:
> > > > > > I've also tried mdelay(3000), just to be sure. The result was the same
> > > > > > hang in 'usb start'.
> > > > >
> > > > > Ok. So put pci_init() into board_late_init().
> > > > >
> > > > > There are some more cleanup and fixes patches for pci_mvebu.c on mailing
> > > > > list. After they are merged I will prepare and send final PCI Kirkwood
> > > > > patch for testing.
> > > >
> > > > Chiming in a bit late in this discussion:
> > > >
> > > > Is an explicit call to pci_init() necessary in this Kirwood case? IIRC,
> > > > the DM infrastructure should make sure that all device are probed - but
> > > > only when really needed. So if you don't need PCI in the boot process
> > > > at all, then it's not probed at all. Or if you set CONFIG_PCI_PNP this
> > > > will be done always. Then there should be no need for the additional
> > > > "pci enum".
> > >
> > > CONFIG_PCI_PNP probe and initialize all devices behind host bridge.
> > > CONFIG_PCI_INIT_R probe and initialize host bridge (this is done by
> > > calling pci_init()). Based on tests (see discussion in this thread) it
> > > looks like that CONFIG_PCI_INIT_R calls pci_init() too early and
> > > initialization is failing. So unsetting CONFIG_PCI_INIT_R and calling
> > > pci_init() manually from board_late_init() (which is called later than
> > > CONFIG_PCI_INIT_R) seems to fix these issues. No idea why...
> > > > I might be missing something - did not check in depth.
> >
> > I think this must be the problem. Sorry I've missed this:
> >
> > config USB_XHCI_MVEBU
> >         bool "MVEBU USB 3.0 support"
> >         default y
> >         depends on ARCH_MVEBU
> >         select DM_REGULATOR
> >         help
> >           Choose this option to add support for USB 3.0 driver on mvebu
> >           SoCs, which includes Armada8K, Armada3700 and other Armada
> >           family SoCs.
>
> This is used for native USB 3.0 controller in Marvell SoC (connected via
> serdes).
>
> But you wrote and sent pci output, that you have PCIe-based USB
> controller connected via PCIe.

Ah, got it.

Thanks,
Tony

> > So I'll need rebuild with "depends on (ARCH_MVEBU || ARCH_KIRKWOOD)"
> > and do some more testing.
> >
> > Thanks,
> > Tony
> >
> > > >
> > > > Thanks,
> > > > Stefan

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

While we are at it, this default should be changed in drivers/usb/host/Kconfig.

config USB_XHCI_MVEBU
        bool "MVEBU USB 3.0 support"
        default y

Setting this default correctly will save a couple Kbytes for other boards.
        default y if ARCH_MVEBU

Thanks,
Tony

On Fri, Nov 12, 2021 at 3:24 PM Tony Dinh <mibodhi@gmail.com> wrote:
>
> On Fri, Nov 12, 2021 at 2:42 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Friday 12 November 2021 13:46:29 Tony Dinh wrote:
> > > Hi Stefan & Pali,
> > >
> > > On Fri, Nov 12, 2021 at 7:06 AM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Friday 12 November 2021 15:36:31 Stefan Roese wrote:
> > > > > On 11/11/21 22:10, Pali Rohár wrote:
> > > > > > On Wednesday 10 November 2021 16:04:20 Tony Dinh wrote:
> > > > > > > I've also tried mdelay(3000), just to be sure. The result was the same
> > > > > > > hang in 'usb start'.
> > > > > >
> > > > > > Ok. So put pci_init() into board_late_init().
> > > > > >
> > > > > > There are some more cleanup and fixes patches for pci_mvebu.c on mailing
> > > > > > list. After they are merged I will prepare and send final PCI Kirkwood
> > > > > > patch for testing.
> > > > >
> > > > > Chiming in a bit late in this discussion:
> > > > >
> > > > > Is an explicit call to pci_init() necessary in this Kirwood case? IIRC,
> > > > > the DM infrastructure should make sure that all device are probed - but
> > > > > only when really needed. So if you don't need PCI in the boot process
> > > > > at all, then it's not probed at all. Or if you set CONFIG_PCI_PNP this
> > > > > will be done always. Then there should be no need for the additional
> > > > > "pci enum".
> > > >
> > > > CONFIG_PCI_PNP probe and initialize all devices behind host bridge.
> > > > CONFIG_PCI_INIT_R probe and initialize host bridge (this is done by
> > > > calling pci_init()). Based on tests (see discussion in this thread) it
> > > > looks like that CONFIG_PCI_INIT_R calls pci_init() too early and
> > > > initialization is failing. So unsetting CONFIG_PCI_INIT_R and calling
> > > > pci_init() manually from board_late_init() (which is called later than
> > > > CONFIG_PCI_INIT_R) seems to fix these issues. No idea why...
> > > > > I might be missing something - did not check in depth.
> > >
> > > I think this must be the problem. Sorry I've missed this:
> > >
> > > config USB_XHCI_MVEBU
> > >         bool "MVEBU USB 3.0 support"
> > >         default y
> > >         depends on ARCH_MVEBU
> > >         select DM_REGULATOR
> > >         help
> > >           Choose this option to add support for USB 3.0 driver on mvebu
> > >           SoCs, which includes Armada8K, Armada3700 and other Armada
> > >           family SoCs.
> >
> > This is used for native USB 3.0 controller in Marvell SoC (connected via
> > serdes).
> >
> > But you wrote and sent pci output, that you have PCIe-based USB
> > controller connected via PCIe.
>
> Ah, got it.
>
> Thanks,
> Tony
>
> > > So I'll need rebuild with "depends on (ARCH_MVEBU || ARCH_KIRKWOOD)"
> > > and do some more testing.
> > >
> > > Thanks,
> > > Tony
> > >
> > > > >
> > > > > Thanks,
> > > > > Stefan

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

On Tuesday 16 November 2021 14:26:56 Tony Dinh wrote:
> Hi Pali,
> 
> While we are at it, this default should be changed in drivers/usb/host/Kconfig.
> 
> config USB_XHCI_MVEBU
>         bool "MVEBU USB 3.0 support"
>         default y
> 
> Setting this default correctly will save a couple Kbytes for other boards.
>         default y if ARCH_MVEBU

Hello! USB_XHCI_MVEBU has currently "depends on ARCH_MVEBU" which means
that USB_XHCI_MVEBU cannot be enabled when ARCH_MVEBU is disabled. So
"default y if ARCH_MVEBU" is same as "default y".

If you have mvebu board when mvebu xhci controller is unused, you can
disable this option and save some space. I see that other xhci drivers
have also "default y" so seems that this is the standard default option.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

On Tue, Nov 16, 2021 at 2:37 PM Pali Rohár <pali@kernel.org> wrote:
>
> On Tuesday 16 November 2021 14:26:56 Tony Dinh wrote:
> > Hi Pali,
> >
> > While we are at it, this default should be changed in drivers/usb/host/Kconfig.
> >
> > config USB_XHCI_MVEBU
> >         bool "MVEBU USB 3.0 support"
> >         default y
> >
> > Setting this default correctly will save a couple Kbytes for other boards.
> >         default y if ARCH_MVEBU
>
> Hello! USB_XHCI_MVEBU has currently "depends on ARCH_MVEBU" which means
> that USB_XHCI_MVEBU cannot be enabled when ARCH_MVEBU is disabled. So
> "default y if ARCH_MVEBU" is same as "default y".

Ah, thanks. This "depends on ARCH_MVEBU" was added in the latest
master tree. I'm switching back and forth between the 2021.10 tag and
master so I did not see this in the 2021.10 tree.

Thanks,
Tony

> If you have mvebu board when mvebu xhci controller is unused, you can
> disable this option and save some space. I see that other xhci drivers
> have also "default y" so seems that this is the standard default option.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

On Tuesday 16 November 2021 15:02:48 Tony Dinh wrote:
> Hi Pali,
> 
> On Tue, Nov 16, 2021 at 2:37 PM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Tuesday 16 November 2021 14:26:56 Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > While we are at it, this default should be changed in drivers/usb/host/Kconfig.
> > >
> > > config USB_XHCI_MVEBU
> > >         bool "MVEBU USB 3.0 support"
> > >         default y
> > >
> > > Setting this default correctly will save a couple Kbytes for other boards.
> > >         default y if ARCH_MVEBU
> >
> > Hello! USB_XHCI_MVEBU has currently "depends on ARCH_MVEBU" which means
> > that USB_XHCI_MVEBU cannot be enabled when ARCH_MVEBU is disabled. So
> > "default y if ARCH_MVEBU" is same as "default y".
> 
> Ah, thanks. This "depends on ARCH_MVEBU" was added in the latest
> master tree. I'm switching back and forth between the 2021.10 tag and
> master so I did not see this in the 2021.10 tree.

Hello! Are you planning to send patches which add support for your
kirkwood board into mainline U-Boot?

I would like to know if we should take care about that PCIe kirkwood
support as currently there is no board in mainline U-Boot which use PCIe
on kirkwood SoC.

> Thanks,
> Tony
> 
> > If you have mvebu board when mvebu xhci controller is unused, you can
> > disable this option and save some space. I see that other xhci drivers
> > have also "default y" so seems that this is the standard default option.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

On Sat, Dec 11, 2021 at 7:39 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Tuesday 16 November 2021 15:02:48 Tony Dinh wrote:
> > Hi Pali,
> >
> > On Tue, Nov 16, 2021 at 2:37 PM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Tuesday 16 November 2021 14:26:56 Tony Dinh wrote:
> > > > Hi Pali,
> > > >
> > > > While we are at it, this default should be changed in drivers/usb/host/Kconfig.
> > > >
> > > > config USB_XHCI_MVEBU
> > > >         bool "MVEBU USB 3.0 support"
> > > >         default y
> > > >
> > > > Setting this default correctly will save a couple Kbytes for other boards.
> > > >         default y if ARCH_MVEBU
> > >
> > > Hello! USB_XHCI_MVEBU has currently "depends on ARCH_MVEBU" which means
> > > that USB_XHCI_MVEBU cannot be enabled when ARCH_MVEBU is disabled. So
> > > "default y if ARCH_MVEBU" is same as "default y".
> >
> > Ah, thanks. This "depends on ARCH_MVEBU" was added in the latest
> > master tree. I'm switching back and forth between the 2021.10 tag and
> > master so I did not see this in the 2021.10 tree.
>
> Hello! Are you planning to send patches which add support for your
> kirkwood board into mainline U-Boot?
>
> I would like to know if we should take care about that PCIe kirkwood
> support as currently there is no board in mainline U-Boot which use PCIe
> on kirkwood SoC.

Yes, I plan to submit patches for the Pogoplug V4 (new board), and the
Iomega iConnect board (update). So far the Pogoplug V4 PCIe is working
perfectly with your in-progress patches.

Thanks,
Tony

> > Thanks,
> > Tony
> >
> > > If you have mvebu board when mvebu xhci controller is unused, you can
> > > disable this option and save some space. I see that other xhci drivers
> > > have also "default y" so seems that this is the standard default option.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

On Saturday 11 December 2021 13:24:07 Tony Dinh wrote:
> Hi Pali,
> 
> On Sat, Dec 11, 2021 at 7:39 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Tuesday 16 November 2021 15:02:48 Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > On Tue, Nov 16, 2021 at 2:37 PM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Tuesday 16 November 2021 14:26:56 Tony Dinh wrote:
> > > > > Hi Pali,
> > > > >
> > > > > While we are at it, this default should be changed in drivers/usb/host/Kconfig.
> > > > >
> > > > > config USB_XHCI_MVEBU
> > > > >         bool "MVEBU USB 3.0 support"
> > > > >         default y
> > > > >
> > > > > Setting this default correctly will save a couple Kbytes for other boards.
> > > > >         default y if ARCH_MVEBU
> > > >
> > > > Hello! USB_XHCI_MVEBU has currently "depends on ARCH_MVEBU" which means
> > > > that USB_XHCI_MVEBU cannot be enabled when ARCH_MVEBU is disabled. So
> > > > "default y if ARCH_MVEBU" is same as "default y".
> > >
> > > Ah, thanks. This "depends on ARCH_MVEBU" was added in the latest
> > > master tree. I'm switching back and forth between the 2021.10 tag and
> > > master so I did not see this in the 2021.10 tree.
> >
> > Hello! Are you planning to send patches which add support for your
> > kirkwood board into mainline U-Boot?
> >
> > I would like to know if we should take care about that PCIe kirkwood
> > support as currently there is no board in mainline U-Boot which use PCIe
> > on kirkwood SoC.
> 
> Yes, I plan to submit patches for the Pogoplug V4 (new board), and the
> Iomega iConnect board (update). So far the Pogoplug V4 PCIe is working
> perfectly with your in-progress patches.

Ok! Because I'm going to do more changes in pci_mvebu.c driver and
kirkwood patch which I have sent to you would not work anymore.

> Thanks,
> Tony
> 
> > > Thanks,
> > > Tony
> > >
> > > > If you have mvebu board when mvebu xhci controller is unused, you can
> > > > disable this option and save some space. I see that other xhci drivers
> > > > have also "default y" so seems that this is the standard default option.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Tony Dinh]

Hi Pali,

On Tue, Dec 14, 2021 at 6:33 AM Pali Rohár <pali@kernel.org> wrote:
>
> On Saturday 11 December 2021 13:24:07 Tony Dinh wrote:
> > Hi Pali,
> >
> > On Sat, Dec 11, 2021 at 7:39 AM Pali Rohár <pali@kernel.org> wrote:
> > >
> > > On Tuesday 16 November 2021 15:02:48 Tony Dinh wrote:
> > > > Hi Pali,
> > > >
> > > > On Tue, Nov 16, 2021 at 2:37 PM Pali Rohár <pali@kernel.org> wrote:
> > > > >
> > > > > On Tuesday 16 November 2021 14:26:56 Tony Dinh wrote:
> > > > > > Hi Pali,
> > > > > >
> > > > > > While we are at it, this default should be changed in drivers/usb/host/Kconfig.
> > > > > >
> > > > > > config USB_XHCI_MVEBU
> > > > > >         bool "MVEBU USB 3.0 support"
> > > > > >         default y
> > > > > >
> > > > > > Setting this default correctly will save a couple Kbytes for other boards.
> > > > > >         default y if ARCH_MVEBU
> > > > >
> > > > > Hello! USB_XHCI_MVEBU has currently "depends on ARCH_MVEBU" which means
> > > > > that USB_XHCI_MVEBU cannot be enabled when ARCH_MVEBU is disabled. So
> > > > > "default y if ARCH_MVEBU" is same as "default y".
> > > >
> > > > Ah, thanks. This "depends on ARCH_MVEBU" was added in the latest
> > > > master tree. I'm switching back and forth between the 2021.10 tag and
> > > > master so I did not see this in the 2021.10 tree.
> > >
> > > Hello! Are you planning to send patches which add support for your
> > > kirkwood board into mainline U-Boot?
> > >
> > > I would like to know if we should take care about that PCIe kirkwood
> > > support as currently there is no board in mainline U-Boot which use PCIe
> > > on kirkwood SoC.
> >
> > Yes, I plan to submit patches for the Pogoplug V4 (new board), and the
> > Iomega iConnect board (update). So far the Pogoplug V4 PCIe is working
> > perfectly with your in-progress patches.
>
> Ok! Because I'm going to do more changes in pci_mvebu.c driver and
> kirkwood patch which I have sent to you would not work anymore.

OK! I have been testing my Pogoplug V4 patch series (almost done).
Sounds like I should wait for your new patch to do more testing?

Thanks,
Tony
> > > >
> > > > > If you have mvebu board when mvebu xhci controller is unused, you can
> > > > > disable this option and save some space. I see that other xhci drivers
> > > > > have also "default y" so seems that this is the standard default option.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

On Tuesday 14 December 2021 14:36:48 Tony Dinh wrote:
> Hi Pali,
> 
> On Tue, Dec 14, 2021 at 6:33 AM Pali Rohár <pali@kernel.org> wrote:
> >
> > On Saturday 11 December 2021 13:24:07 Tony Dinh wrote:
> > > Hi Pali,
> > >
> > > On Sat, Dec 11, 2021 at 7:39 AM Pali Rohár <pali@kernel.org> wrote:
> > > >
> > > > On Tuesday 16 November 2021 15:02:48 Tony Dinh wrote:
> > > > > Hi Pali,
> > > > >
> > > > > On Tue, Nov 16, 2021 at 2:37 PM Pali Rohár <pali@kernel.org> wrote:
> > > > > >
> > > > > > On Tuesday 16 November 2021 14:26:56 Tony Dinh wrote:
> > > > > > > Hi Pali,
> > > > > > >
> > > > > > > While we are at it, this default should be changed in drivers/usb/host/Kconfig.
> > > > > > >
> > > > > > > config USB_XHCI_MVEBU
> > > > > > >         bool "MVEBU USB 3.0 support"
> > > > > > >         default y
> > > > > > >
> > > > > > > Setting this default correctly will save a couple Kbytes for other boards.
> > > > > > >         default y if ARCH_MVEBU
> > > > > >
> > > > > > Hello! USB_XHCI_MVEBU has currently "depends on ARCH_MVEBU" which means
> > > > > > that USB_XHCI_MVEBU cannot be enabled when ARCH_MVEBU is disabled. So
> > > > > > "default y if ARCH_MVEBU" is same as "default y".
> > > > >
> > > > > Ah, thanks. This "depends on ARCH_MVEBU" was added in the latest
> > > > > master tree. I'm switching back and forth between the 2021.10 tag and
> > > > > master so I did not see this in the 2021.10 tree.
> > > >
> > > > Hello! Are you planning to send patches which add support for your
> > > > kirkwood board into mainline U-Boot?
> > > >
> > > > I would like to know if we should take care about that PCIe kirkwood
> > > > support as currently there is no board in mainline U-Boot which use PCIe
> > > > on kirkwood SoC.
> > >
> > > Yes, I plan to submit patches for the Pogoplug V4 (new board), and the
> > > Iomega iConnect board (update). So far the Pogoplug V4 PCIe is working
> > > perfectly with your in-progress patches.
> >
> > Ok! Because I'm going to do more changes in pci_mvebu.c driver and
> > kirkwood patch which I have sent to you would not work anymore.
> 
> OK! I have been testing my Pogoplug V4 patch series (almost done).
> Sounds like I should wait for your new patch to do more testing?

Well, you do not need to wait. If you know that it is working fine, you
can send patches. I will then modify both mvebu and kirkwood code and
send you my new modifications / cleanups.

> Thanks,
> Tony
> > > > >
> > > > > > If you have mvebu board when mvebu xhci controller is unused, you can
> > > > > > disable this option and save some space. I see that other xhci drivers
> > > > > > have also "default y" so seems that this is the standard default option.

Re: Testing pci_mvebu.c with Kirkwood SoCs

[Pali Rohár]

On Monday 08 November 2021 14:48:03 Tony Dinh wrote:
> This Pogoplug V4 board is still out-of-tree. I have not submitted
> patches to add support for it. I plan to do that in the near future,
> and it will be under board/cloudengines/.
> 
> However, there is a show stopper right now that prevents me from going
> ahead to add this Pogoplug V4 board. A previous fdt patch has not been
> accepted, because Simon wanted this to be implemented with livetree
> calls:
> https://lists.denx.de/pipermail/u-boot/2021-August/457311.html
> It is working fine as a flat tree implementation.

You can at least send patches which adds base code for this board, even
without network support, which can be added later.

> The other 2 Kirkwood boards that could be useful in PCIe testing are
> the Iomega iConnect (in mainline at board/iomega/iconnect/), and the
> Zyxel NAS325 (still out-of-tree). I am also stuck on this NAS325 board
> because of the fdt_support flattree vs. livetree implementation.