* Question about mkfs.erofs and reproducible builds
@ 2021-10-27 11:11 Todor Ivanov
2021-10-27 11:27 ` Gao Xiang
0 siblings, 1 reply; 6+ messages in thread
From: Todor Ivanov @ 2021-10-27 11:11 UTC (permalink / raw)
To: linux-erofs
[-- Attachment #1: Type: text/plain, Size: 968 bytes --]
Hello,
We are trying to replace squashfs with erofs and face an issue with
reproducing the build from one and the same source folder. The source
folder is "/etc" actually taken from an offline ubuntu 20.04 image and
mounted as read-only.
I managed to narrow down the scope and it turns out that the issue
is when you have a file starting with "." (dot) in this folder. I.e.:
etc/.anyfilename
If I remove this file the erofs image of "etc" is reproducible (-T and -U
are used as well)
The issue is somehow related to the other 76 subfolders of etc and this
file starting with dot. For example if I create such .anyfilename in usr or
var, there is no issue. Also if I create this file under
etc/xdg/.anyfilename, this is fine as well.
I also tried with etc from debian10 and the result is the same. Removing
any file that starts with dot directly under etc, makes the erofs build
reproducible.
Do you have any advice on this?
Regards,
Todor
[-- Attachment #2: Type: text/html, Size: 1217 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Question about mkfs.erofs and reproducible builds 2021-10-27 11:11 Question about mkfs.erofs and reproducible builds Todor Ivanov @ 2021-10-27 11:27 ` Gao Xiang 2021-10-27 11:41 ` Todor Ivanov 0 siblings, 1 reply; 6+ messages in thread From: Gao Xiang @ 2021-10-27 11:27 UTC (permalink / raw) To: Todor Ivanov; +Cc: linux-erofs Hi Todor, On Wed, Oct 27, 2021 at 02:11:24PM +0300, Todor Ivanov wrote: > Hello, > > We are trying to replace squashfs with erofs and face an issue with > reproducing the build from one and the same source folder. The source > folder is "/etc" actually taken from an offline ubuntu 20.04 image and > mounted as read-only. > I managed to narrow down the scope and it turns out that the issue > is when you have a file starting with "." (dot) in this folder. I.e.: > > etc/.anyfilename > > If I remove this file the erofs image of "etc" is reproducible (-T and -U > are used as well) > > The issue is somehow related to the other 76 subfolders of etc and this > file starting with dot. For example if I create such .anyfilename in usr or > var, there is no issue. Also if I create this file under > etc/xdg/.anyfilename, this is fine as well. > I also tried with etc from debian10 and the result is the same. Removing > any file that starts with dot directly under etc, makes the erofs build > reproducible. > Do you have any advice on this? In principle filenames starting with '.' won't impact anything about reproducible builds... Let me investigate it now... But may I ask which erofs-utils version is used? Does it still happen on the latest dev branch? Thanks, Gao Xiang > > Regards, > Todor ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question about mkfs.erofs and reproducible builds 2021-10-27 11:27 ` Gao Xiang @ 2021-10-27 11:41 ` Todor Ivanov 2021-10-27 11:51 ` Gao Xiang 0 siblings, 1 reply; 6+ messages in thread From: Todor Ivanov @ 2021-10-27 11:41 UTC (permalink / raw) To: Gao Xiang; +Cc: linux-erofs [-- Attachment #1: Type: text/plain, Size: 1886 bytes --] Hi, Gao, This is how I installed mkfs.erofs on debian10: apt-get install pkg-config liblz4-dev gawk wget http://ftp.debian.org/debian/pool/main/e/erofs-utils/erofs-utils_1.3.orig.tar.gz tar xvzpf erofs-utils_1.3.orig.tar.gz cd erofs-utils-1.3/ ./autogen.sh ./configure make make install Can you tell me where do I clone it from and if build instructions are different? Kind regards, Todor On Wed, Oct 27, 2021 at 2:27 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote: > Hi Todor, > > On Wed, Oct 27, 2021 at 02:11:24PM +0300, Todor Ivanov wrote: > > Hello, > > > > We are trying to replace squashfs with erofs and face an issue > with > > reproducing the build from one and the same source folder. The source > > folder is "/etc" actually taken from an offline ubuntu 20.04 image and > > mounted as read-only. > > I managed to narrow down the scope and it turns out that the > issue > > is when you have a file starting with "." (dot) in this folder. I.e.: > > > > etc/.anyfilename > > > > If I remove this file the erofs image of "etc" is reproducible (-T and -U > > are used as well) > > > > The issue is somehow related to the other 76 subfolders of etc and this > > file starting with dot. For example if I create such .anyfilename in usr > or > > var, there is no issue. Also if I create this file under > > etc/xdg/.anyfilename, this is fine as well. > > I also tried with etc from debian10 and the result is the same. Removing > > any file that starts with dot directly under etc, makes the erofs build > > reproducible. > > Do you have any advice on this? > > In principle filenames starting with '.' won't impact anything about > reproducible builds... > > Let me investigate it now... But may I ask which erofs-utils version > is used? Does it still happen on the latest dev branch? > > Thanks, > Gao Xiang > > > > > Regards, > > Todor > [-- Attachment #2: Type: text/html, Size: 2692 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question about mkfs.erofs and reproducible builds 2021-10-27 11:41 ` Todor Ivanov @ 2021-10-27 11:51 ` Gao Xiang 2021-10-27 13:40 ` Todor Ivanov 0 siblings, 1 reply; 6+ messages in thread From: Gao Xiang @ 2021-10-27 11:51 UTC (permalink / raw) To: Todor Ivanov; +Cc: linux-erofs On Wed, Oct 27, 2021 at 02:41:54PM +0300, Todor Ivanov wrote: > Hi, Gao, > > This is how I installed mkfs.erofs on debian10: > > apt-get install pkg-config liblz4-dev gawk > wget > http://ftp.debian.org/debian/pool/main/e/erofs-utils/erofs-utils_1.3.orig.tar.gz > tar xvzpf erofs-utils_1.3.orig.tar.gz > cd erofs-utils-1.3/ > ./autogen.sh > ./configure > make > make install > > Can you tell me where do I clone it from and if build instructions are > different? You could get the latest dev branch from: git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b dev We fixed some reproducable build issues recently, I think it might be related to https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/commit/?id=6324fac820c28c6a946f595fa58a0abba0f48eb4 Thanks, Gao Xiang > > Kind regards, > Todor > > On Wed, Oct 27, 2021 at 2:27 PM Gao Xiang <hsiangkao@linux.alibaba.com> > wrote: > > > Hi Todor, > > > > On Wed, Oct 27, 2021 at 02:11:24PM +0300, Todor Ivanov wrote: > > > Hello, > > > > > > We are trying to replace squashfs with erofs and face an issue > > with > > > reproducing the build from one and the same source folder. The source > > > folder is "/etc" actually taken from an offline ubuntu 20.04 image and > > > mounted as read-only. > > > I managed to narrow down the scope and it turns out that the > > issue > > > is when you have a file starting with "." (dot) in this folder. I.e.: > > > > > > etc/.anyfilename > > > > > > If I remove this file the erofs image of "etc" is reproducible (-T and -U > > > are used as well) > > > > > > The issue is somehow related to the other 76 subfolders of etc and this > > > file starting with dot. For example if I create such .anyfilename in usr > > or > > > var, there is no issue. Also if I create this file under > > > etc/xdg/.anyfilename, this is fine as well. > > > I also tried with etc from debian10 and the result is the same. Removing > > > any file that starts with dot directly under etc, makes the erofs build > > > reproducible. > > > Do you have any advice on this? > > > > In principle filenames starting with '.' won't impact anything about > > reproducible builds... > > > > Let me investigate it now... But may I ask which erofs-utils version > > is used? Does it still happen on the latest dev branch? > > > > Thanks, > > Gao Xiang > > > > > > > > Regards, > > > Todor > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question about mkfs.erofs and reproducible builds 2021-10-27 11:51 ` Gao Xiang @ 2021-10-27 13:40 ` Todor Ivanov 2021-10-27 14:04 ` Gao Xiang 0 siblings, 1 reply; 6+ messages in thread From: Todor Ivanov @ 2021-10-27 13:40 UTC (permalink / raw) To: Gao Xiang; +Cc: linux-erofs [-- Attachment #1: Type: text/plain, Size: 2944 bytes --] Hi, Gao, Indeed c_version:[1.3-g9fe440d0], seems to fix the problem :) Thank you very much for the prompt and accurate feedback! Do you think we can use the dev branch for our images or is it better until it is merged into master? What do you think? Kind regards, Todor On Wed, Oct 27, 2021 at 2:51 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote: > On Wed, Oct 27, 2021 at 02:41:54PM +0300, Todor Ivanov wrote: > > Hi, Gao, > > > > This is how I installed mkfs.erofs on debian10: > > > > apt-get install pkg-config liblz4-dev gawk > > wget > > > http://ftp.debian.org/debian/pool/main/e/erofs-utils/erofs-utils_1.3.orig.tar.gz > > tar xvzpf erofs-utils_1.3.orig.tar.gz > > cd erofs-utils-1.3/ > > ./autogen.sh > > ./configure > > make > > make install > > > > Can you tell me where do I clone it from and if build instructions are > > different? > > You could get the latest dev branch from: > git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b dev > > We fixed some reproducable build issues recently, I think it might > be related to > > https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/commit/?id=6324fac820c28c6a946f595fa58a0abba0f48eb4 > > Thanks, > Gao Xiang > > > > > Kind regards, > > Todor > > > > On Wed, Oct 27, 2021 at 2:27 PM Gao Xiang <hsiangkao@linux.alibaba.com> > > wrote: > > > > > Hi Todor, > > > > > > On Wed, Oct 27, 2021 at 02:11:24PM +0300, Todor Ivanov wrote: > > > > Hello, > > > > > > > > We are trying to replace squashfs with erofs and face an > issue > > > with > > > > reproducing the build from one and the same source folder. The source > > > > folder is "/etc" actually taken from an offline ubuntu 20.04 image > and > > > > mounted as read-only. > > > > I managed to narrow down the scope and it turns out that the > > > issue > > > > is when you have a file starting with "." (dot) in this folder. I.e.: > > > > > > > > etc/.anyfilename > > > > > > > > If I remove this file the erofs image of "etc" is reproducible (-T > and -U > > > > are used as well) > > > > > > > > The issue is somehow related to the other 76 subfolders of etc and > this > > > > file starting with dot. For example if I create such .anyfilename in > usr > > > or > > > > var, there is no issue. Also if I create this file under > > > > etc/xdg/.anyfilename, this is fine as well. > > > > I also tried with etc from debian10 and the result is the same. > Removing > > > > any file that starts with dot directly under etc, makes the erofs > build > > > > reproducible. > > > > Do you have any advice on this? > > > > > > In principle filenames starting with '.' won't impact anything about > > > reproducible builds... > > > > > > Let me investigate it now... But may I ask which erofs-utils version > > > is used? Does it still happen on the latest dev branch? > > > > > > Thanks, > > > Gao Xiang > > > > > > > > > > > Regards, > > > > Todor > > > > [-- Attachment #2: Type: text/html, Size: 4584 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question about mkfs.erofs and reproducible builds 2021-10-27 13:40 ` Todor Ivanov @ 2021-10-27 14:04 ` Gao Xiang 0 siblings, 0 replies; 6+ messages in thread From: Gao Xiang @ 2021-10-27 14:04 UTC (permalink / raw) To: Todor Ivanov; +Cc: linux-erofs On Wed, Oct 27, 2021 at 04:40:28PM +0300, Todor Ivanov wrote: > Hi, Gao, > > Indeed c_version:[1.3-g9fe440d0], seems to fix the problem :) Thank you > very much for the prompt and accurate feedback! Do you think we can use the > dev branch for our images or is it better until it is merged into master? > What do you think? Many thanks for your feedback too in time! ;) I'm fine to fast-forward the main branch without bumping up the version (since other folks also needed that as well...) But I'd like to postpone new erofs-utils v1.4 for a while since I'm waiting for merging some other on-going features too~ Thanks, Gao Xiang > > Kind regards, > Todor > > > On Wed, Oct 27, 2021 at 2:51 PM Gao Xiang <hsiangkao@linux.alibaba.com> > wrote: > > > On Wed, Oct 27, 2021 at 02:41:54PM +0300, Todor Ivanov wrote: > > > Hi, Gao, > > > > > > This is how I installed mkfs.erofs on debian10: > > > > > > apt-get install pkg-config liblz4-dev gawk > > > wget > > > > > http://ftp.debian.org/debian/pool/main/e/erofs-utils/erofs-utils_1.3.orig.tar.gz > > > tar xvzpf erofs-utils_1.3.orig.tar.gz > > > cd erofs-utils-1.3/ > > > ./autogen.sh > > > ./configure > > > make > > > make install > > > > > > Can you tell me where do I clone it from and if build instructions are > > > different? > > > > You could get the latest dev branch from: > > git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b dev > > > > We fixed some reproducable build issues recently, I think it might > > be related to > > > > https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/commit/?id=6324fac820c28c6a946f595fa58a0abba0f48eb4 > > > > Thanks, > > Gao Xiang > > > > > > > > Kind regards, > > > Todor > > > > > > On Wed, Oct 27, 2021 at 2:27 PM Gao Xiang <hsiangkao@linux.alibaba.com> > > > wrote: > > > > > > > Hi Todor, > > > > > > > > On Wed, Oct 27, 2021 at 02:11:24PM +0300, Todor Ivanov wrote: > > > > > Hello, > > > > > > > > > > We are trying to replace squashfs with erofs and face an > > issue > > > > with > > > > > reproducing the build from one and the same source folder. The source > > > > > folder is "/etc" actually taken from an offline ubuntu 20.04 image > > and > > > > > mounted as read-only. > > > > > I managed to narrow down the scope and it turns out that the > > > > issue > > > > > is when you have a file starting with "." (dot) in this folder. I.e.: > > > > > > > > > > etc/.anyfilename > > > > > > > > > > If I remove this file the erofs image of "etc" is reproducible (-T > > and -U > > > > > are used as well) > > > > > > > > > > The issue is somehow related to the other 76 subfolders of etc and > > this > > > > > file starting with dot. For example if I create such .anyfilename in > > usr > > > > or > > > > > var, there is no issue. Also if I create this file under > > > > > etc/xdg/.anyfilename, this is fine as well. > > > > > I also tried with etc from debian10 and the result is the same. > > Removing > > > > > any file that starts with dot directly under etc, makes the erofs > > build > > > > > reproducible. > > > > > Do you have any advice on this? > > > > > > > > In principle filenames starting with '.' won't impact anything about > > > > reproducible builds... > > > > > > > > Let me investigate it now... But may I ask which erofs-utils version > > > > is used? Does it still happen on the latest dev branch? > > > > > > > > Thanks, > > > > Gao Xiang > > > > > > > > > > > > > > Regards, > > > > > Todor > > > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-10-27 14:04 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-10-27 11:11 Question about mkfs.erofs and reproducible builds Todor Ivanov 2021-10-27 11:27 ` Gao Xiang 2021-10-27 11:41 ` Todor Ivanov 2021-10-27 11:51 ` Gao Xiang 2021-10-27 13:40 ` Todor Ivanov 2021-10-27 14:04 ` Gao Xiang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).