* Reproducibility regression with mx6cuboxi
@ 2020-07-19 18:23 Vagrant Cascadian
2020-07-21 22:24 ` Tom Rini
2020-07-22 18:32 ` Heinrich Schuchardt
0 siblings, 2 replies; 15+ messages in thread
From: Vagrant Cascadian @ 2020-07-19 18:23 UTC (permalink / raw)
To: u-boot
The mx6cuboxi target no longer builds reproducibility on Debian. I've
bisected it down to:
eb9124f5748c96ffd548e50fd6989c3b5395b353 mx6cuboxi: enable OF_CONTROL with DM_MMC and DM_USB
Based on the diffoscope output, it *might* have something to do with the
changes to how board detection uses gpios:
--- /tmp/tmpj_t0iqs7/control
+++ /tmp/tmpj_t0iqs7/experiment-time
? --- /tmp/tmpj_t0iqs7/control/source-root
??? +++ /tmp/tmpj_t0iqs7/experiment-time/source-root
? ? --- /tmp/tmpj_t0iqs7/control/source-root/u-boot-with-spl.imx
? ??? +++ /tmp/tmpj_t0iqs7/experiment-time/source-root/u-boot-with-spl.imx
? ? @@ -4346,16 +4346,16 @@
? ? 00010f90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? 00010fa0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? 00010fb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? 00010fc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? 00010fd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? 00010fe0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? 00010ff0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? -00011000: 2705 1956 7088 eb01 5f14 7b49 0008 565c '..Vp..._.{I..V\
? ? -00011010: 1780 0000 1780 0000 9463 4032 1102 0500 .........c at 2....
? ? +00011000: 2705 1956 36e7 7190 5f14 7b49 0008 565c '..V6.q._.{I..V\
? ? +00011010: 1780 0000 1780 0000 de74 408a 1102 0500 .........t at .....
? ? 00011020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? 00011030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? 00011040: b800 00ea 14f0 9fe5 14f0 9fe5 14f0 9fe5 ................
? ? 00011050: 14f0 9fe5 14f0 9fe5 14f0 9fe5 14f0 9fe5 ................
? ? 00011060: 6000 8017 c000 8017 2001 8017 8001 8017 `....... .......
? ? 00011070: e001 8017 4002 8017 a002 8017 efbe adde .... at ...........
? ? 00011080: 2000 9000 00f0 20e3 00f0 20e3 00f0 20e3 ..... ... ... .
? ? @@ -33330,15 +33330,15 @@
? ? 00082310: 6c6f 636b 2d6d 6173 7465 7200 7369 6d70 lock-master.simp
? ? 00082320: 6c65 2d61 7564 696f 2d63 6172 642c 6672 le-audio-card,fr
? ? 00082330: 616d 652d 6d61 7374 6572 0073 696d 706c ame-master.simpl
? ? 00082340: 652d 6175 6469 6f2d 6361 7264 2c77 6964 e-audio-card,wid
? ? 00082350: 6765 7473 0073 696d 706c 652d 6175 6469 gets.simple-audi
? ? 00082360: 6f2d 6361 7264 2c72 6f75 7469 6e67 0073 o-card,routing.s
? ? 00082370: 6f75 6e64 2d64 6169 0064 6574 6563 742d ound-dai.detect-
? ? -00082380: 6770 696f 7300 6275 d00d feed 0000 9e9e gpios.bu........
? ? +00082380: 6770 696f 7300 0023 d00d feed 0000 9e9e gpios..#........
? ? 00082390: 0000 0038 0000 9600 0000 0028 0000 0011 ...8.......(....
? ? 000823a0: 0000 0010 0000 0000 0000 089e 0000 95c8 ................
? ? 000823b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? 000823c0: 0000 0001 0000 0000 0000 0003 0000 0004 ................
? ? 000823d0: 0000 0000 0000 0001 0000 0003 0000 0004 ................
? ? 000823e0: 0000 000f 0000 0001 0000 0003 0000 0033 ...............3
? ? 000823f0: 0000 001b 536f 6c69 6452 756e 2048 756d ....SolidRun Hum
? ? @@ -35868,15 +35868,15 @@
? ? 0008c1b0: 6c6f 636b 2d6d 6173 7465 7200 7369 6d70 lock-master.simp
? ? 0008c1c0: 6c65 2d61 7564 696f 2d63 6172 642c 6672 le-audio-card,fr
? ? 0008c1d0: 616d 652d 6d61 7374 6572 0073 696d 706c ame-master.simpl
? ? 0008c1e0: 652d 6175 6469 6f2d 6361 7264 2c77 6964 e-audio-card,wid
? ? 0008c1f0: 6765 7473 0073 696d 706c 652d 6175 6469 gets.simple-audi
? ? 0008c200: 6f2d 6361 7264 2c72 6f75 7469 6e67 0073 o-card,routing.s
? ? 0008c210: 6f75 6e64 2d64 6169 0064 6574 6563 742d ound-dai.detect-
? ? -0008c220: 6770 696f 7300 0023 d00d feed 0000 a473 gpios..#.......s
? ? +0008c220: 6770 696f 7300 3230 d00d feed 0000 a473 gpios.20.......s
? ? 0008c230: 0000 0038 0000 9b6c 0000 0028 0000 0011 ...8...l...(....
? ? 0008c240: 0000 0010 0000 0000 0000 0907 0000 9b34 ...............4
? ? 0008c250: 0000 0000 0000 0000 0000 0000 0000 0000 ................
? ? 0008c260: 0000 0001 0000 0000 0000 0003 0000 0004 ................
? ? 0008c270: 0000 0000 0000 0001 0000 0003 0000 0004 ................
? ? 0008c280: 0000 000f 0000 0001 0000 0003 0000 002f .............../
? ? 0008c290: 0000 001b 536f 6c69 6452 756e 2048 756d ....SolidRun Hum
? ? @@ -38499,8 +38499,8 @@
? ? 00096620: 2c62 6974 636c 6f63 6b2d 6d61 7374 6572 ,bitclock-master
? ? 00096630: 0073 696d 706c 652d 6175 6469 6f2d 6361 .simple-audio-ca
? ? 00096640: 7264 2c66 7261 6d65 2d6d 6173 7465 7200 rd,frame-master.
? ? 00096650: 7369 6d70 6c65 2d61 7564 696f 2d63 6172 simple-audio-car
? ? 00096660: 642c 7769 6467 6574 7300 7369 6d70 6c65 d,widgets.simple
? ? 00096670: 2d61 7564 696f 2d63 6172 642c 726f 7574 -audio-card,rout
? ? 00096680: 696e 6700 736f 756e 642d 6461 6900 6465 ing.sound-dai.de
? ? -00096690: 7465 6374 2d67 7069 6f73 0000 tect-gpios..
? ? +00096690: 7465 6374 2d67 7069 6f73 0061 tect-gpios.a
A similar diffoscope output is also available here:
https://tests.reproducible-builds.org/debian/rb-pkg/bullseye/armhf/diffoscope-results/u-boot.html
Using reprotest to diagnose the issue, it suggests the issue may be
related to the time of the build.
Is there a generated file embedded in the resulting image, or a
compressed archive of some kind (e.g. gzip, tar, zip, etc.) that might
include the build time?
https://diffoscope.org/
https://packages.debian.org/unstable/reprotest
live well,
vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200719/bba7be46/attachment.sig>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-19 18:23 Reproducibility regression with mx6cuboxi Vagrant Cascadian
@ 2020-07-21 22:24 ` Tom Rini
2020-07-21 22:54 ` Vagrant Cascadian
2020-07-22 18:32 ` Heinrich Schuchardt
1 sibling, 1 reply; 15+ messages in thread
From: Tom Rini @ 2020-07-21 22:24 UTC (permalink / raw)
To: u-boot
On Sun, Jul 19, 2020 at 11:23:05AM -0700, Vagrant Cascadian wrote:
> The mx6cuboxi target no longer builds reproducibility on Debian. I've
> bisected it down to:
>
> eb9124f5748c96ffd548e50fd6989c3b5395b353 mx6cuboxi: enable OF_CONTROL with DM_MMC and DM_USB
Are you tracking any other platforms which use OF_CONTROL? Thanks!
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200721/ac741d6f/attachment.sig>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-21 22:24 ` Tom Rini
@ 2020-07-21 22:54 ` Vagrant Cascadian
0 siblings, 0 replies; 15+ messages in thread
From: Vagrant Cascadian @ 2020-07-21 22:54 UTC (permalink / raw)
To: u-boot
On 2020-07-21, Tom Rini wrote:
> On Sun, Jul 19, 2020 at 11:23:05AM -0700, Vagrant Cascadian wrote:
>
>> The mx6cuboxi target no longer builds reproducibility on Debian. I've
>> bisected it down to:
>>
>> eb9124f5748c96ffd548e50fd6989c3b5395b353 mx6cuboxi: enable OF_CONTROL with DM_MMC and DM_USB
>
> Are you tracking any other platforms which use OF_CONTROL? Thanks!
Several in the imx family, all of which appear to be building
reproducibly:
$ zgrep CONFIG_OF_CONTROL=y usr/share/doc/u-boot-imx/configs/config.*
usr/share/doc/u-boot-imx/configs/config.dh_imx6.gz:CONFIG_OF_CONTROL=y
usr/share/doc/u-boot-imx/configs/config.mx6cuboxi.gz:CONFIG_OF_CONTROL=y
usr/share/doc/u-boot-imx/configs/config.mx6qsabrelite.gz:CONFIG_OF_CONTROL=y
usr/share/doc/u-boot-imx/configs/config.nitrogen6q.gz:CONFIG_OF_CONTROL=y
usr/share/doc/u-boot-imx/configs/config.novena-rawsd.gz:CONFIG_OF_CONTROL=y
usr/share/doc/u-boot-imx/configs/config.novena.gz:CONFIG_OF_CONTROL=y
usr/share/doc/u-boot-imx/configs/config.wandboard.gz:CONFIG_OF_CONTROL=y
dh_imx6 did have reproducibility issues around u-boot 2020.04, but
haven't seen them in 2020.07-rc1 or newer, so I haven't bothered to
bisect or look into it further.
CONFIG_OF_CONTROL is enabled in many other boards in the Debian packages
without issue.
The dra7xx_evm platform has reproducibility issues introduced
approximately the same time, but I haven't been able to trigger the
issue in my local test environment (which is a little less comprehensive
than what tests.reproducible-builds.org does). I'll try to do some more
testing of that one soon, but that should probably be it's own thread as
it seems unrelated.
live well,
vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200721/59b2b26d/attachment.sig>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-19 18:23 Reproducibility regression with mx6cuboxi Vagrant Cascadian
2020-07-21 22:24 ` Tom Rini
@ 2020-07-22 18:32 ` Heinrich Schuchardt
2020-07-22 18:44 ` Tom Rini
1 sibling, 1 reply; 15+ messages in thread
From: Heinrich Schuchardt @ 2020-07-22 18:32 UTC (permalink / raw)
To: u-boot
On 19.07.20 20:23, Vagrant Cascadian wrote:
> The mx6cuboxi target no longer builds reproducibility on Debian. I've
> bisected it down to:
>
> eb9124f5748c96ffd548e50fd6989c3b5395b353 mx6cuboxi: enable OF_CONTROL with DM_MMC and DM_USB
>
>
> Based on the diffoscope output, it *might* have something to do with the
> changes to how board detection uses gpios:
>
> --- /tmp/tmpj_t0iqs7/control
> +++ /tmp/tmpj_t0iqs7/experiment-time
> ? --- /tmp/tmpj_t0iqs7/control/source-root
> ??? +++ /tmp/tmpj_t0iqs7/experiment-time/source-root
> ? ? --- /tmp/tmpj_t0iqs7/control/source-root/u-boot-with-spl.imx
> ? ??? +++ /tmp/tmpj_t0iqs7/experiment-time/source-root/u-boot-with-spl.imx
> ? ? @@ -4346,16 +4346,16 @@
> ? ? 00010f90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? 00010fa0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? 00010fb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? 00010fc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? 00010fd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? 00010fe0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? 00010ff0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? -00011000: 2705 1956 7088 eb01 5f14 7b49 0008 565c '..Vp..._.{I..V\
> ? ? -00011010: 1780 0000 1780 0000 9463 4032 1102 0500 .........c at 2....
> ? ? +00011000: 2705 1956 36e7 7190 5f14 7b49 0008 565c '..V6.q._.{I..V\
> ? ? +00011010: 1780 0000 1780 0000 de74 408a 1102 0500 .........t at .....
> ? ? 00011020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? 00011030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? 00011040: b800 00ea 14f0 9fe5 14f0 9fe5 14f0 9fe5 ................
> ? ? 00011050: 14f0 9fe5 14f0 9fe5 14f0 9fe5 14f0 9fe5 ................
> ? ? 00011060: 6000 8017 c000 8017 2001 8017 8001 8017 `....... .......
> ? ? 00011070: e001 8017 4002 8017 a002 8017 efbe adde .... at ...........
> ? ? 00011080: 2000 9000 00f0 20e3 00f0 20e3 00f0 20e3 ..... ... ... .
> ? ? @@ -33330,15 +33330,15 @@
> ? ? 00082310: 6c6f 636b 2d6d 6173 7465 7200 7369 6d70 lock-master.simp
> ? ? 00082320: 6c65 2d61 7564 696f 2d63 6172 642c 6672 le-audio-card,fr
> ? ? 00082330: 616d 652d 6d61 7374 6572 0073 696d 706c ame-master.simpl
> ? ? 00082340: 652d 6175 6469 6f2d 6361 7264 2c77 6964 e-audio-card,wid
> ? ? 00082350: 6765 7473 0073 696d 706c 652d 6175 6469 gets.simple-audi
> ? ? 00082360: 6f2d 6361 7264 2c72 6f75 7469 6e67 0073 o-card,routing.s
> ? ? 00082370: 6f75 6e64 2d64 6169 0064 6574 6563 742d ound-dai.detect-
> ? ? -00082380: 6770 696f 7300 6275 d00d feed 0000 9e9e gpios.bu........
> ? ? +00082380: 6770 696f 7300 0023 d00d feed 0000 9e9e gpios..#........
In file 'u-boot' there is only one occurence of string "detect-gpios"
followed by "\0HUSH_NO_EVAL".
In file 'u-boot.bin' I see a second occurence not followed by
"\0HUSH_NO_EVAL".
The second instance is introduced fit-dtb.blob.
fit-dtb.blob is created from dts/dt.dtb using the mkimage tool.
The last bytes of dts/dtb are
"routing.sound-dai\0detect-gpios\0"
So it seems the mkimage tool is introducing the variable bytes after
"detect-gpios\0". Possibly it is not clearing buffers.
Best regards
Heinich
> ? ? 00082390: 0000 0038 0000 9600 0000 0028 0000 0011 ...8.......(....
> ? ? 000823a0: 0000 0010 0000 0000 0000 089e 0000 95c8 ................
> ? ? 000823b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? 000823c0: 0000 0001 0000 0000 0000 0003 0000 0004 ................
> ? ? 000823d0: 0000 0000 0000 0001 0000 0003 0000 0004 ................
> ? ? 000823e0: 0000 000f 0000 0001 0000 0003 0000 0033 ...............3
> ? ? 000823f0: 0000 001b 536f 6c69 6452 756e 2048 756d ....SolidRun Hum
> ? ? @@ -35868,15 +35868,15 @@
> ? ? 0008c1b0: 6c6f 636b 2d6d 6173 7465 7200 7369 6d70 lock-master.simp
> ? ? 0008c1c0: 6c65 2d61 7564 696f 2d63 6172 642c 6672 le-audio-card,fr
> ? ? 0008c1d0: 616d 652d 6d61 7374 6572 0073 696d 706c ame-master.simpl
> ? ? 0008c1e0: 652d 6175 6469 6f2d 6361 7264 2c77 6964 e-audio-card,wid
> ? ? 0008c1f0: 6765 7473 0073 696d 706c 652d 6175 6469 gets.simple-audi
> ? ? 0008c200: 6f2d 6361 7264 2c72 6f75 7469 6e67 0073 o-card,routing.s
> ? ? 0008c210: 6f75 6e64 2d64 6169 0064 6574 6563 742d ound-dai.detect-
> ? ? -0008c220: 6770 696f 7300 0023 d00d feed 0000 a473 gpios..#.......s
> ? ? +0008c220: 6770 696f 7300 3230 d00d feed 0000 a473 gpios.20.......s
> ? ? 0008c230: 0000 0038 0000 9b6c 0000 0028 0000 0011 ...8...l...(....
> ? ? 0008c240: 0000 0010 0000 0000 0000 0907 0000 9b34 ...............4
> ? ? 0008c250: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> ? ? 0008c260: 0000 0001 0000 0000 0000 0003 0000 0004 ................
> ? ? 0008c270: 0000 0000 0000 0001 0000 0003 0000 0004 ................
> ? ? 0008c280: 0000 000f 0000 0001 0000 0003 0000 002f .............../
> ? ? 0008c290: 0000 001b 536f 6c69 6452 756e 2048 756d ....SolidRun Hum
> ? ? @@ -38499,8 +38499,8 @@
> ? ? 00096620: 2c62 6974 636c 6f63 6b2d 6d61 7374 6572 ,bitclock-master
> ? ? 00096630: 0073 696d 706c 652d 6175 6469 6f2d 6361 .simple-audio-ca
> ? ? 00096640: 7264 2c66 7261 6d65 2d6d 6173 7465 7200 rd,frame-master.
> ? ? 00096650: 7369 6d70 6c65 2d61 7564 696f 2d63 6172 simple-audio-car
> ? ? 00096660: 642c 7769 6467 6574 7300 7369 6d70 6c65 d,widgets.simple
> ? ? 00096670: 2d61 7564 696f 2d63 6172 642c 726f 7574 -audio-card,rout
> ? ? 00096680: 696e 6700 736f 756e 642d 6461 6900 6465 ing.sound-dai.de
> ? ? -00096690: 7465 6374 2d67 7069 6f73 0000 tect-gpios..
> ? ? +00096690: 7465 6374 2d67 7069 6f73 0061 tect-gpios.a
>
> A similar diffoscope output is also available here:
>
> https://tests.reproducible-builds.org/debian/rb-pkg/bullseye/armhf/diffoscope-results/u-boot.html
>
> Using reprotest to diagnose the issue, it suggests the issue may be
> related to the time of the build.
>
> Is there a generated file embedded in the resulting image, or a
> compressed archive of some kind (e.g. gzip, tar, zip, etc.) that might
> include the build time?
>
>
> https://diffoscope.org/
> https://packages.debian.org/unstable/reprotest
>
>
> live well,
> vagrant
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-22 18:32 ` Heinrich Schuchardt
@ 2020-07-22 18:44 ` Tom Rini
2020-07-22 20:40 ` Fabio Estevam
0 siblings, 1 reply; 15+ messages in thread
From: Tom Rini @ 2020-07-22 18:44 UTC (permalink / raw)
To: u-boot
On Wed, Jul 22, 2020 at 08:32:28PM +0200, Heinrich Schuchardt wrote:
> On 19.07.20 20:23, Vagrant Cascadian wrote:
> > The mx6cuboxi target no longer builds reproducibility on Debian. I've
> > bisected it down to:
> >
> > eb9124f5748c96ffd548e50fd6989c3b5395b353 mx6cuboxi: enable OF_CONTROL with DM_MMC and DM_USB
> >
> >
> > Based on the diffoscope output, it *might* have something to do with the
> > changes to how board detection uses gpios:
> >
> > --- /tmp/tmpj_t0iqs7/control
> > +++ /tmp/tmpj_t0iqs7/experiment-time
> > ? --- /tmp/tmpj_t0iqs7/control/source-root
> > ??? +++ /tmp/tmpj_t0iqs7/experiment-time/source-root
> > ? ? --- /tmp/tmpj_t0iqs7/control/source-root/u-boot-with-spl.imx
> > ? ??? +++ /tmp/tmpj_t0iqs7/experiment-time/source-root/u-boot-with-spl.imx
> > ? ? @@ -4346,16 +4346,16 @@
> > ? ? 00010f90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > ? ? 00010fa0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > ? ? 00010fb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > ? ? 00010fc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > ? ? 00010fd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > ? ? 00010fe0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > ? ? 00010ff0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > ? ? -00011000: 2705 1956 7088 eb01 5f14 7b49 0008 565c '..Vp..._.{I..V\
> > ? ? -00011010: 1780 0000 1780 0000 9463 4032 1102 0500 .........c at 2....
> > ? ? +00011000: 2705 1956 36e7 7190 5f14 7b49 0008 565c '..V6.q._.{I..V\
> > ? ? +00011010: 1780 0000 1780 0000 de74 408a 1102 0500 .........t at .....
> > ? ? 00011020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > ? ? 00011030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > ? ? 00011040: b800 00ea 14f0 9fe5 14f0 9fe5 14f0 9fe5 ................
> > ? ? 00011050: 14f0 9fe5 14f0 9fe5 14f0 9fe5 14f0 9fe5 ................
> > ? ? 00011060: 6000 8017 c000 8017 2001 8017 8001 8017 `....... .......
> > ? ? 00011070: e001 8017 4002 8017 a002 8017 efbe adde .... at ...........
> > ? ? 00011080: 2000 9000 00f0 20e3 00f0 20e3 00f0 20e3 ..... ... ... .
> > ? ? @@ -33330,15 +33330,15 @@
> > ? ? 00082310: 6c6f 636b 2d6d 6173 7465 7200 7369 6d70 lock-master.simp
> > ? ? 00082320: 6c65 2d61 7564 696f 2d63 6172 642c 6672 le-audio-card,fr
> > ? ? 00082330: 616d 652d 6d61 7374 6572 0073 696d 706c ame-master.simpl
> > ? ? 00082340: 652d 6175 6469 6f2d 6361 7264 2c77 6964 e-audio-card,wid
> > ? ? 00082350: 6765 7473 0073 696d 706c 652d 6175 6469 gets.simple-audi
> > ? ? 00082360: 6f2d 6361 7264 2c72 6f75 7469 6e67 0073 o-card,routing.s
> > ? ? 00082370: 6f75 6e64 2d64 6169 0064 6574 6563 742d ound-dai.detect-
> > ? ? -00082380: 6770 696f 7300 6275 d00d feed 0000 9e9e gpios.bu........
> > ? ? +00082380: 6770 696f 7300 0023 d00d feed 0000 9e9e gpios..#........
>
> In file 'u-boot' there is only one occurence of string "detect-gpios"
> followed by "\0HUSH_NO_EVAL".
>
> In file 'u-boot.bin' I see a second occurence not followed by
> "\0HUSH_NO_EVAL".
>
> The second instance is introduced fit-dtb.blob.
>
> fit-dtb.blob is created from dts/dt.dtb using the mkimage tool.
>
> The last bytes of dts/dtb are
>
> "routing.sound-dai\0detect-gpios\0"
>
> So it seems the mkimage tool is introducing the variable bytes after
> "detect-gpios\0". Possibly it is not clearing buffers.
Possibly the data leak that Marek pointed out a while back but we never
cycled back on fixing only that and not changing padding stuff at the
same time.
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200722/5342d62c/attachment.sig>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-22 18:44 ` Tom Rini
@ 2020-07-22 20:40 ` Fabio Estevam
2020-07-22 21:10 ` Tom Rini
0 siblings, 1 reply; 15+ messages in thread
From: Fabio Estevam @ 2020-07-22 20:40 UTC (permalink / raw)
To: u-boot
Hi Tom,
[Adding Marek]
On Wed, Jul 22, 2020 at 3:45 PM Tom Rini <trini@konsulko.com> wrote:
>
> On Wed, Jul 22, 2020 at 08:32:28PM +0200, Heinrich Schuchardt wrote:
> > On 19.07.20 20:23, Vagrant Cascadian wrote:
> > > The mx6cuboxi target no longer builds reproducibility on Debian. I've
> > > bisected it down to:
> > >
> > > eb9124f5748c96ffd548e50fd6989c3b5395b353 mx6cuboxi: enable OF_CONTROL with DM_MMC and DM_USB
> > >
> > >
> > > Based on the diffoscope output, it *might* have something to do with the
> > > changes to how board detection uses gpios:
> > >
> > > --- /tmp/tmpj_t0iqs7/control
> > > +++ /tmp/tmpj_t0iqs7/experiment-time
> > > ? --- /tmp/tmpj_t0iqs7/control/source-root
> > > ??? +++ /tmp/tmpj_t0iqs7/experiment-time/source-root
> > > ? ? --- /tmp/tmpj_t0iqs7/control/source-root/u-boot-with-spl.imx
> > > ? ??? +++ /tmp/tmpj_t0iqs7/experiment-time/source-root/u-boot-with-spl.imx
> > > ? ? @@ -4346,16 +4346,16 @@
> > > ? ? 00010f90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > ? ? 00010fa0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > ? ? 00010fb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > ? ? 00010fc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > ? ? 00010fd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > ? ? 00010fe0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > ? ? 00010ff0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > ? ? -00011000: 2705 1956 7088 eb01 5f14 7b49 0008 565c '..Vp..._.{I..V\
> > > ? ? -00011010: 1780 0000 1780 0000 9463 4032 1102 0500 .........c at 2....
> > > ? ? +00011000: 2705 1956 36e7 7190 5f14 7b49 0008 565c '..V6.q._.{I..V\
> > > ? ? +00011010: 1780 0000 1780 0000 de74 408a 1102 0500 .........t at .....
> > > ? ? 00011020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > ? ? 00011030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > ? ? 00011040: b800 00ea 14f0 9fe5 14f0 9fe5 14f0 9fe5 ................
> > > ? ? 00011050: 14f0 9fe5 14f0 9fe5 14f0 9fe5 14f0 9fe5 ................
> > > ? ? 00011060: 6000 8017 c000 8017 2001 8017 8001 8017 `....... .......
> > > ? ? 00011070: e001 8017 4002 8017 a002 8017 efbe adde .... at ...........
> > > ? ? 00011080: 2000 9000 00f0 20e3 00f0 20e3 00f0 20e3 ..... ... ... .
> > > ? ? @@ -33330,15 +33330,15 @@
> > > ? ? 00082310: 6c6f 636b 2d6d 6173 7465 7200 7369 6d70 lock-master.simp
> > > ? ? 00082320: 6c65 2d61 7564 696f 2d63 6172 642c 6672 le-audio-card,fr
> > > ? ? 00082330: 616d 652d 6d61 7374 6572 0073 696d 706c ame-master.simpl
> > > ? ? 00082340: 652d 6175 6469 6f2d 6361 7264 2c77 6964 e-audio-card,wid
> > > ? ? 00082350: 6765 7473 0073 696d 706c 652d 6175 6469 gets.simple-audi
> > > ? ? 00082360: 6f2d 6361 7264 2c72 6f75 7469 6e67 0073 o-card,routing.s
> > > ? ? 00082370: 6f75 6e64 2d64 6169 0064 6574 6563 742d ound-dai.detect-
> > > ? ? -00082380: 6770 696f 7300 6275 d00d feed 0000 9e9e gpios.bu........
> > > ? ? +00082380: 6770 696f 7300 0023 d00d feed 0000 9e9e gpios..#........
> >
> > In file 'u-boot' there is only one occurence of string "detect-gpios"
> > followed by "\0HUSH_NO_EVAL".
> >
> > In file 'u-boot.bin' I see a second occurence not followed by
> > "\0HUSH_NO_EVAL".
> >
> > The second instance is introduced fit-dtb.blob.
> >
> > fit-dtb.blob is created from dts/dt.dtb using the mkimage tool.
> >
> > The last bytes of dts/dtb are
> >
> > "routing.sound-dai\0detect-gpios\0"
> >
> > So it seems the mkimage tool is introducing the variable bytes after
> > "detect-gpios\0". Possibly it is not clearing buffers.
>
> Possibly the data leak that Marek pointed out a while back but we never
> cycled back on fixing only that and not changing padding stuff at the
> same time.
Do you mean this one?
http://u-boot.10912.n7.nabble.com/PATCH-V2-mkimage-fit-Do-not-tail-pad-fitImage-with-external-data-td409920.html
Thanks
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-22 20:40 ` Fabio Estevam
@ 2020-07-22 21:10 ` Tom Rini
2020-07-26 14:57 ` Fabio Estevam
0 siblings, 1 reply; 15+ messages in thread
From: Tom Rini @ 2020-07-22 21:10 UTC (permalink / raw)
To: u-boot
On Wed, Jul 22, 2020 at 05:40:12PM -0300, Fabio Estevam wrote:
> Hi Tom,
>
> [Adding Marek]
>
> On Wed, Jul 22, 2020 at 3:45 PM Tom Rini <trini@konsulko.com> wrote:
> >
> > On Wed, Jul 22, 2020 at 08:32:28PM +0200, Heinrich Schuchardt wrote:
> > > On 19.07.20 20:23, Vagrant Cascadian wrote:
> > > > The mx6cuboxi target no longer builds reproducibility on Debian. I've
> > > > bisected it down to:
> > > >
> > > > eb9124f5748c96ffd548e50fd6989c3b5395b353 mx6cuboxi: enable OF_CONTROL with DM_MMC and DM_USB
> > > >
> > > >
> > > > Based on the diffoscope output, it *might* have something to do with the
> > > > changes to how board detection uses gpios:
> > > >
> > > > --- /tmp/tmpj_t0iqs7/control
> > > > +++ /tmp/tmpj_t0iqs7/experiment-time
> > > > ? --- /tmp/tmpj_t0iqs7/control/source-root
> > > > ??? +++ /tmp/tmpj_t0iqs7/experiment-time/source-root
> > > > ? ? --- /tmp/tmpj_t0iqs7/control/source-root/u-boot-with-spl.imx
> > > > ? ??? +++ /tmp/tmpj_t0iqs7/experiment-time/source-root/u-boot-with-spl.imx
> > > > ? ? @@ -4346,16 +4346,16 @@
> > > > ? ? 00010f90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > > ? ? 00010fa0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > > ? ? 00010fb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > > ? ? 00010fc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > > ? ? 00010fd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > > ? ? 00010fe0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > > ? ? 00010ff0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > > ? ? -00011000: 2705 1956 7088 eb01 5f14 7b49 0008 565c '..Vp..._.{I..V\
> > > > ? ? -00011010: 1780 0000 1780 0000 9463 4032 1102 0500 .........c at 2....
> > > > ? ? +00011000: 2705 1956 36e7 7190 5f14 7b49 0008 565c '..V6.q._.{I..V\
> > > > ? ? +00011010: 1780 0000 1780 0000 de74 408a 1102 0500 .........t at .....
> > > > ? ? 00011020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > > ? ? 00011030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > > > ? ? 00011040: b800 00ea 14f0 9fe5 14f0 9fe5 14f0 9fe5 ................
> > > > ? ? 00011050: 14f0 9fe5 14f0 9fe5 14f0 9fe5 14f0 9fe5 ................
> > > > ? ? 00011060: 6000 8017 c000 8017 2001 8017 8001 8017 `....... .......
> > > > ? ? 00011070: e001 8017 4002 8017 a002 8017 efbe adde .... at ...........
> > > > ? ? 00011080: 2000 9000 00f0 20e3 00f0 20e3 00f0 20e3 ..... ... ... .
> > > > ? ? @@ -33330,15 +33330,15 @@
> > > > ? ? 00082310: 6c6f 636b 2d6d 6173 7465 7200 7369 6d70 lock-master.simp
> > > > ? ? 00082320: 6c65 2d61 7564 696f 2d63 6172 642c 6672 le-audio-card,fr
> > > > ? ? 00082330: 616d 652d 6d61 7374 6572 0073 696d 706c ame-master.simpl
> > > > ? ? 00082340: 652d 6175 6469 6f2d 6361 7264 2c77 6964 e-audio-card,wid
> > > > ? ? 00082350: 6765 7473 0073 696d 706c 652d 6175 6469 gets.simple-audi
> > > > ? ? 00082360: 6f2d 6361 7264 2c72 6f75 7469 6e67 0073 o-card,routing.s
> > > > ? ? 00082370: 6f75 6e64 2d64 6169 0064 6574 6563 742d ound-dai.detect-
> > > > ? ? -00082380: 6770 696f 7300 6275 d00d feed 0000 9e9e gpios.bu........
> > > > ? ? +00082380: 6770 696f 7300 0023 d00d feed 0000 9e9e gpios..#........
> > >
> > > In file 'u-boot' there is only one occurence of string "detect-gpios"
> > > followed by "\0HUSH_NO_EVAL".
> > >
> > > In file 'u-boot.bin' I see a second occurence not followed by
> > > "\0HUSH_NO_EVAL".
> > >
> > > The second instance is introduced fit-dtb.blob.
> > >
> > > fit-dtb.blob is created from dts/dt.dtb using the mkimage tool.
> > >
> > > The last bytes of dts/dtb are
> > >
> > > "routing.sound-dai\0detect-gpios\0"
> > >
> > > So it seems the mkimage tool is introducing the variable bytes after
> > > "detect-gpios\0". Possibly it is not clearing buffers.
> >
> > Possibly the data leak that Marek pointed out a while back but we never
> > cycled back on fixing only that and not changing padding stuff at the
> > same time.
>
> Do you mean this one?
> http://u-boot.10912.n7.nabble.com/PATCH-V2-mkimage-fit-Do-not-tail-pad-fitImage-with-external-data-td409920.html
Yes, that one.
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200722/c84553bc/attachment.sig>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-22 21:10 ` Tom Rini
@ 2020-07-26 14:57 ` Fabio Estevam
2020-07-26 15:05 ` Tom Rini
0 siblings, 1 reply; 15+ messages in thread
From: Fabio Estevam @ 2020-07-26 14:57 UTC (permalink / raw)
To: u-boot
Hi Vagrant,
On Wed, Jul 22, 2020 at 6:10 PM Tom Rini <trini@konsulko.com> wrote:
> > Do you mean this one?
> > http://u-boot.10912.n7.nabble.com/PATCH-V2-mkimage-fit-Do-not-tail-pad-fitImage-with-external-data-td409920.html
>
> Yes, that one.
I know Marek's patch caused a regression on other platforms, but could
you please give it a try and see if it fixes the reproducibility
problem?
If it works for you, then we need to come up with a solution that does
not cause the boot regression on other platforms.
Thanks
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-26 14:57 ` Fabio Estevam
@ 2020-07-26 15:05 ` Tom Rini
2020-07-26 17:00 ` Fabio Estevam
0 siblings, 1 reply; 15+ messages in thread
From: Tom Rini @ 2020-07-26 15:05 UTC (permalink / raw)
To: u-boot
On Sun, Jul 26, 2020 at 11:57:43AM -0300, Fabio Estevam wrote:
> Hi Vagrant,
>
> On Wed, Jul 22, 2020 at 6:10 PM Tom Rini <trini@konsulko.com> wrote:
>
> > > Do you mean this one?
> > > http://u-boot.10912.n7.nabble.com/PATCH-V2-mkimage-fit-Do-not-tail-pad-fitImage-with-external-data-td409920.html
> >
> > Yes, that one.
>
> I know Marek's patch caused a regression on other platforms, but could
> you please give it a try and see if it fixes the reproducibility
> problem?
>
> If it works for you, then we need to come up with a solution that does
> not cause the boot regression on other platforms.
We just need to use calloc() in the tool and not mess with alignment.
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200726/672ff2e7/attachment.sig>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-26 15:05 ` Tom Rini
@ 2020-07-26 17:00 ` Fabio Estevam
2020-07-26 17:15 ` Tom Rini
0 siblings, 1 reply; 15+ messages in thread
From: Fabio Estevam @ 2020-07-26 17:00 UTC (permalink / raw)
To: u-boot
On Sun, Jul 26, 2020 at 12:05 PM Tom Rini <trini@konsulko.com> wrote:
> We just need to use calloc() in the tool and not mess with alignment.
Like this?
--- a/tools/fit_image.c
+++ b/tools/fit_image.c
@@ -446,7 +446,6 @@ static int fit_extract_data(struct
image_tool_params *params, const char *fname)
int ret;
int images;
int node;
- int image_number;
int align_size;
align_size = params->bl_len ? params->bl_len : 4;
@@ -461,13 +460,12 @@ static int fit_extract_data(struct
image_tool_params *params, const char *fname)
ret = -EINVAL;
goto err_munmap;
}
- image_number = fdtdec_get_child_count(fdt, images);
/*
* Allocate space to hold the image data we will extract,
* extral space allocate for image alignment to prevent overflow.
*/
- buf = malloc(fit_size + (align_size * image_number));
+ buf = calloc(1, fit_size);
if (!buf) {
ret = -ENOMEM;
goto err_munmap;
If this is not the right approach, care to propose a patch?
Thanks
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-26 17:00 ` Fabio Estevam
@ 2020-07-26 17:15 ` Tom Rini
2020-07-26 17:56 ` Fabio Estevam
0 siblings, 1 reply; 15+ messages in thread
From: Tom Rini @ 2020-07-26 17:15 UTC (permalink / raw)
To: u-boot
On Sun, Jul 26, 2020 at 02:00:33PM -0300, Fabio Estevam wrote:
> On Sun, Jul 26, 2020 at 12:05 PM Tom Rini <trini@konsulko.com> wrote:
>
> > We just need to use calloc() in the tool and not mess with alignment.
>
> Like this?
>
> --- a/tools/fit_image.c
> +++ b/tools/fit_image.c
> @@ -446,7 +446,6 @@ static int fit_extract_data(struct
> image_tool_params *params, const char *fname)
> int ret;
> int images;
> int node;
> - int image_number;
> int align_size;
>
> align_size = params->bl_len ? params->bl_len : 4;
> @@ -461,13 +460,12 @@ static int fit_extract_data(struct
> image_tool_params *params, const char *fname)
> ret = -EINVAL;
> goto err_munmap;
> }
> - image_number = fdtdec_get_child_count(fdt, images);
>
> /*
> * Allocate space to hold the image data we will extract,
> * extral space allocate for image alignment to prevent overflow.
> */
> - buf = malloc(fit_size + (align_size * image_number));
> + buf = calloc(1, fit_size);
> if (!buf) {
> ret = -ENOMEM;
> goto err_munmap;
>
> If this is not the right approach, care to propose a patch?
I mean just literally changing the malloc(...) to calloc(1, ...), audit any
other malloc(...) calls in the file and change nothing else. Thanks!
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200726/a9791d76/attachment.sig>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-26 17:15 ` Tom Rini
@ 2020-07-26 17:56 ` Fabio Estevam
2020-07-26 18:17 ` Vagrant Cascadian
0 siblings, 1 reply; 15+ messages in thread
From: Fabio Estevam @ 2020-07-26 17:56 UTC (permalink / raw)
To: u-boot
On Sun, Jul 26, 2020 at 2:16 PM Tom Rini <trini@konsulko.com> wrote:
> I mean just literally changing the malloc(...) to calloc(1, ...), audit any
> other malloc(...) calls in the file and change nothing else. Thanks!
Thanks for the clarification, Tom
Vagrant,
Does the patch below fix the reproducibility regression?
https://pastebin.com/raw/MWDUDrJ2
Thanks
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-26 17:56 ` Fabio Estevam
@ 2020-07-26 18:17 ` Vagrant Cascadian
2020-07-26 18:22 ` Fabio Estevam
2020-07-28 11:23 ` Fabio Estevam
0 siblings, 2 replies; 15+ messages in thread
From: Vagrant Cascadian @ 2020-07-26 18:17 UTC (permalink / raw)
To: u-boot
On 2020-07-26, Fabio Estevam wrote:
> On Sun, Jul 26, 2020 at 2:16 PM Tom Rini <trini@konsulko.com> wrote:
>
>> I mean just literally changing the malloc(...) to calloc(1, ...), audit any
>> other malloc(...) calls in the file and change nothing else. Thanks!
>
> Thanks for the clarification, Tom
>
> Vagrant,
>
> Does the patch below fix the reproducibility regression?
> https://pastebin.com/raw/MWDUDrJ2
That appears to build reproducibly for me on top of v2020.07. Haven't
tested if the resulting image boots.
FWIW, re-applying Marek's old patch also appeared to fix the
reproducibility issues:
20a154f95bfe0a3b5bfba90bea7f001c58217536 mkimage: fit: Do not tail-pad
fitImage with external data
Thanks all, seems a fix is near!
live well,
vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200726/12932f05/attachment.sig>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-26 18:17 ` Vagrant Cascadian
@ 2020-07-26 18:22 ` Fabio Estevam
2020-07-28 11:23 ` Fabio Estevam
1 sibling, 0 replies; 15+ messages in thread
From: Fabio Estevam @ 2020-07-26 18:22 UTC (permalink / raw)
To: u-boot
Hi Vagrant,
On Sun, Jul 26, 2020 at 3:17 PM Vagrant Cascadian
<vagrant@reproducible-builds.org> wrote:
> That appears to build reproducibly for me on top of v2020.07. Haven't
> tested if the resulting image boots.
I can confirm it boots fine on a imx6 humming board.
I will submit it as a formal patch.
Thanks
^ permalink raw reply [flat|nested] 15+ messages in thread
* Reproducibility regression with mx6cuboxi
2020-07-26 18:17 ` Vagrant Cascadian
2020-07-26 18:22 ` Fabio Estevam
@ 2020-07-28 11:23 ` Fabio Estevam
1 sibling, 0 replies; 15+ messages in thread
From: Fabio Estevam @ 2020-07-28 11:23 UTC (permalink / raw)
To: u-boot
Hi Vagrant,
On Sun, Jul 26, 2020 at 3:17 PM Vagrant Cascadian
<vagrant@reproducible-builds.org> wrote:
> That appears to build reproducibly for me on top of v2020.07. Haven't
> tested if the resulting image boots.
When you have a chance to test it, please reply with your Tested-by on
my v2 patch.
Thanks
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2020-07-28 11:23 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-19 18:23 Reproducibility regression with mx6cuboxi Vagrant Cascadian
2020-07-21 22:24 ` Tom Rini
2020-07-21 22:54 ` Vagrant Cascadian
2020-07-22 18:32 ` Heinrich Schuchardt
2020-07-22 18:44 ` Tom Rini
2020-07-22 20:40 ` Fabio Estevam
2020-07-22 21:10 ` Tom Rini
2020-07-26 14:57 ` Fabio Estevam
2020-07-26 15:05 ` Tom Rini
2020-07-26 17:00 ` Fabio Estevam
2020-07-26 17:15 ` Tom Rini
2020-07-26 17:56 ` Fabio Estevam
2020-07-26 18:17 ` Vagrant Cascadian
2020-07-26 18:22 ` Fabio Estevam
2020-07-28 11:23 ` Fabio Estevam
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.