Re: [OE-core] [PATCH 1/7] rsync: Fix a file sorting determinism issue

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Mon, 2021-02-22 at 17:40 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of
> > Richard Purdie
> > Sent: den 21 februari 2021 23:17
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [PATCH 1/7] rsync: Fix a file sorting determinism issue
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  .../rsync/files/determism.patch               | 28 +++++++++++++++++++
> >  meta/recipes-devtools/rsync/rsync_3.2.3.bb    |  1 +
> >  2 files changed, 29 insertions(+)
> >  create mode 100644 meta/recipes-devtools/rsync/files/determism.patch
> > 
> > diff --git a/meta/recipes-devtools/rsync/files/determism.patch b/meta/recipes-
> > devtools/rsync/files/determism.patch
> > new file mode 100644
> > index 00000000000..53a4ca75058
> > --- /dev/null
> > +++ b/meta/recipes-devtools/rsync/files/determism.patch
> > @@ -0,0 +1,28 @@
> > +The Makefile calls awk on a "*.c" glob. The results of this glob are sorted
> > +but the order depends on the locale settings, particularly whether
> > +"util.c" and "util2.c" sort before or after each other. In en_US.UTF-8
> > +they sort one way, in C, they sort the other. The sorting order changes
> > +the output binaries. The behaviour also changes dependning on whether
> > +SHELL (/bin/sh) is dash or bash.
> > +
> > +Specify a C locale setting to be deterministic.
> > +
> > +Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > +Upstream-Status: Pending
> > +
> > +Index: rsync-3.2.3/Makefile.in
> > +===================================================================
> > +--- rsync-3.2.3.orig/Makefile.in
> > ++++ rsync-3.2.3/Makefile.in
> > +@@ -26,6 +26,11 @@ MKDIR_P=@MKDIR_P@
> > + VPATH=$(srcdir)
> > + SHELL=/bin/sh
> > +
> > ++# We use globbing in commands, need to be deterministic
> > ++unexport LC_ALL
> > ++LC_COLLATE=C
> > ++export LC_COLLATE
> 
> Rather than using a patch, can't this be achieved by doing:
> 
> export LC_COLLATE=C
> 
> or:
> 
> export LC_ALL=C
> 
> in the recipe?

I think its all a bit of a mess. bitbake.conf sets:

export LC_ALL = "en_US.UTF-8"

which seems odd at first but going from memory, its because we need utf8 support 
and was the best way to get it as distros disagree over what an utf8 C locale 
is called (of if it exists).

I think bash sees and honours that, dash does not but both will work
with LC_COLLATE=C as long as LC_ALL is unset.

More testing would be needed to see which alternatives might work. 
LC_COLLATE=C in the recipe may break python for example or conflict with the 
LC_ALL setting :(

I'm open for someone to look more into this...

Cheers,

Richard