* e2fsprogs: building ext2simg
@ 2017-10-08 20:04 Nicolas Dechesne
2017-10-08 21:27 ` Theodore Ts'o
0 siblings, 1 reply; 4+ messages in thread
From: Nicolas Dechesne @ 2017-10-08 20:04 UTC (permalink / raw)
To: linux-ext4
hi,
ext2simg.c file was moved from android (system/extras) into e2fsprogs.
however it seems that it cannot be compiled outside of android (it
uses sparse/sparse.h). some distro used to build/package this utility
(e.g. debian has it in android-tools-fsutils).
what was the main reason why it was moved into e2fsprogs in the first
place, given that it cannot be compiled (easily) outside of AOSP?
Also, the commit that brought this file has the following reference:
From AOSP commit: c1b7d19958dc3dbe53810811ea3dcc4f04f85c73
However I don't think this commit in AOSP.
thanks!
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: e2fsprogs: building ext2simg 2017-10-08 20:04 e2fsprogs: building ext2simg Nicolas Dechesne @ 2017-10-08 21:27 ` Theodore Ts'o 2017-10-09 8:24 ` Nicolas Dechesne 0 siblings, 1 reply; 4+ messages in thread From: Theodore Ts'o @ 2017-10-08 21:27 UTC (permalink / raw) To: Nicolas Dechesne; +Cc: linux-ext4 On Sun, Oct 08, 2017 at 10:04:21PM +0200, Nicolas Dechesne wrote: > hi, > > ext2simg.c file was moved from android (system/extras) into e2fsprogs. > however it seems that it cannot be compiled outside of android (it > uses sparse/sparse.h). some distro used to build/package this utility > (e.g. debian has it in android-tools-fsutils). > > what was the main reason why it was moved into e2fsprogs in the first > place, given that it cannot be compiled (easily) outside of AOSP? I've been pushing a long-term effort to try to rid the world from make_ext4fs, which only exists at all because there was a time when there were people who greatly feared the GPLv2. So the idea was that AOSP would never use any GPLv2 code, and AOSP would use make_ext4fs, and Android devices would *never* need to do silly things like run e2fsck. (This was a long time ago, but we've been still paying for that design decision, with interest.) Unfortunately, make_ext4fs was a clean room implementation of mke2fs, which means it had gotten a number of things wrong. Worse, it was originally meant to work (only) as part of the build process using sparse files or later, libsparse, and so it assumes that an unwritten block is all zeros. This even mostly worked on flash devices, assuming that the device actually bothered to honor the discard/trim "hint" --- but it was a complete nightmare for devices using dm-crypt, since the trim/discard would result in the underlying blocks to be zero, but then when dm-crypt read back the blocks that were supposed to be zeroed, the decryption would cause the block to have some random data. Hilarity would then ensue --- but only in very random, edge cases. I've lost track of the number of times I've been asked to assist when make_ext4fs has screwed up in some wonderous fashion, and there has been some internal bug, usually but not always involving a user-initiated factory reset, that had stumped the good folks on the Android team, and then I would get asked to assist. Unfortunately, make_ext4fs and its associated tools have accumulated a number of extra functionality (populating the empty file system, adding SELinux attributes, etc.), so trying to make replacements that don't depend on a clean-room implementation of e2fsprogs and its library has been a long, and not completely finished, journey. Still, thanks to a summer intern last year, and some assistance from folks in the Android team, we're almost there. Anyway, the version of ext2simg in e2fsprogs uses the ext2fs library, and since the cutover from make_ext4fs to mke2fs has been done on a device by device basis (to minimize the risks of breakage), it was simpler to make the e2fsprogs-dependent version of ext2simg live in the e2fsprogs git tree. As far as why it's in the upstream e2fsprogs tree, I've been making an effort to keep the upstream and AOSP versions of e2fsprogs more closely aligned, so it's easier for AOSP to update to a newer version of e2fsprogs. You can now upgrade to a newer version of e2fsprogs by doing a "git merge", which used to not work at all. Now, it might result in some merge conflicts, but at least there is a common git ancestor so the merge has a chance of working. As far as it being hard to compile libsparse outside of AOSP, it would be great if someone were to package libsparse and libsparse-dev for Debian. If they do so, I'll be glad (as debian maintainer) to package up the e2fsprogs variants of these tools. We'll probably need to give them a different package name so that both versions of the packages can exist at least in the package namespace, if not installed at the same time on a Debian system. But we can deal with that problem when we get there. > Also, the commit that brought this file has the following reference: > > From AOSP commit: c1b7d19958dc3dbe53810811ea3dcc4f04f85c73 > > However I don't think this commit in AOSP. Hmm... I don't have easy access to AOSP at the moment, since I'm at home and it's not on my upstream development laptop, but is it possible that the contents of the git commit exists, but with a different git commit ID? My understanding is that changes to external/e2fsprogs, once code reviewed in Gerrit, do get pushed out to the AOSP master branch right away. Android source trees have a massively complex merging scheme, and the most likely explanation is that I used the git commit ID belonging to some merge into a tree that isn't published in AOSP. Either that, or the commit ID got somehow corrupted while it was in my editor doing the cherry-pick into the upstream tree. Cheers, - Ted P.S. One of the other really nice things you can do as a result of a lot of this effort is you can now build tools such as debugfs for Android, built against bionic. So it becomes a lot easier to build a userdebug android image that includes debugfs, which is helpful if you need to try to debug some interesting new failure. And using mke2fs instead of make_ext4fs means that if we do add some new file system feature into ext4/e2fsprogs which Android wants to depend upon, it will be a lot easier to keep things in sync with the external upstream world. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: e2fsprogs: building ext2simg 2017-10-08 21:27 ` Theodore Ts'o @ 2017-10-09 8:24 ` Nicolas Dechesne 2017-10-09 12:59 ` Theodore Ts'o 0 siblings, 1 reply; 4+ messages in thread From: Nicolas Dechesne @ 2017-10-09 8:24 UTC (permalink / raw) To: Theodore Ts'o; +Cc: linux-ext4 hi Theo, thanks for the very detailed answer! On Sun, Oct 8, 2017 at 11:27 PM, Theodore Ts'o <tytso@mit.edu> wrote: > On Sun, Oct 08, 2017 at 10:04:21PM +0200, Nicolas Dechesne wrote: >> hi, >> >> ext2simg.c file was moved from android (system/extras) into e2fsprogs. >> however it seems that it cannot be compiled outside of android (it >> uses sparse/sparse.h). some distro used to build/package this utility >> (e.g. debian has it in android-tools-fsutils). >> >> what was the main reason why it was moved into e2fsprogs in the first >> place, given that it cannot be compiled (easily) outside of AOSP? > > I've been pushing a long-term effort to try to rid the world from > make_ext4fs, which only exists at all because there was a time when > there were people who greatly feared the GPLv2. So the idea was that > AOSP would never use any GPLv2 code, and AOSP would use make_ext4fs, > and Android devices would *never* need to do silly things like run > e2fsck. (This was a long time ago, but we've been still paying for > that design decision, with interest.) > > Unfortunately, make_ext4fs was a clean room implementation of mke2fs, > which means it had gotten a number of things wrong. Worse, it was > originally meant to work (only) as part of the build process using > sparse files or later, libsparse, and so it assumes that an unwritten > block is all zeros. > > This even mostly worked on flash devices, assuming that the device > actually bothered to honor the discard/trim "hint" --- but it was a > complete nightmare for devices using dm-crypt, since the trim/discard > would result in the underlying blocks to be zero, but then when > dm-crypt read back the blocks that were supposed to be zeroed, the > decryption would cause the block to have some random data. Hilarity > would then ensue --- but only in very random, edge cases. > > I've lost track of the number of times I've been asked to assist when > make_ext4fs has screwed up in some wonderous fashion, and there has > been some internal bug, usually but not always involving a > user-initiated factory reset, that had stumped the good folks on the > Android team, and then I would get asked to assist. > > Unfortunately, make_ext4fs and its associated tools have accumulated a > number of extra functionality (populating the empty file system, > adding SELinux attributes, etc.), so trying to make replacements that > don't depend on a clean-room implementation of e2fsprogs and its > library has been a long, and not completely finished, journey. Still, > thanks to a summer intern last year, and some assistance from folks in > the Android team, we're almost there. > > Anyway, the version of ext2simg in e2fsprogs uses the ext2fs library, > and since the cutover from make_ext4fs to mke2fs has been done on a > device by device basis (to minimize the risks of breakage), it was > simpler to make the e2fsprogs-dependent version of ext2simg live in > the e2fsprogs git tree. > > As far as why it's in the upstream e2fsprogs tree, I've been making an > effort to keep the upstream and AOSP versions of e2fsprogs more > closely aligned, so it's easier for AOSP to update to a newer version > of e2fsprogs. > > You can now upgrade to a newer version of e2fsprogs by doing a "git > merge", which used to not work at all. Now, it might result in some > merge conflicts, but at least there is a common git ancestor so the > merge has a chance of working. I agree that getting ext2simg closer to upstream and ext2fs was the right thing to do, i have faced issues with make_ext4fs as well, so I am aware of what you are discussing here! > > As far as it being hard to compile libsparse outside of AOSP, it would > be great if someone were to package libsparse and libsparse-dev for > Debian. If they do so, I'll be glad (as debian maintainer) to package > up the e2fsprogs variants of these tools. We'll probably need to give > them a different package name so that both versions of the packages > can exist at least in the package namespace, if not installed at the > same time on a Debian system. But we can deal with that problem when > we get there. libsparse is already available in Debian, in android-libsparse-dev package, but it's being packaged from the full system/core, not standalone. In fact, building the 'new' ext2simg is exactly what I needed to do, and the reason why i started to investigate what happened to ext2simg.. the real reason why i've been looking into that , is that I have noticed that my file system gets corrupted when doing ext2simg -> simg2img. we are building ext4 file system for small dev boards that are meant to be flashed with fastboot, so our workflow is: mkfs.ext4 xxx -> fill in the image ext2simg What i've noticed is that if we upgrade our servers running Debian Jessie , this no longer works and after doing simg2img (or flashing with fasboot) the files in the file system are corrupted (md5sum differ). The problems appears as soon as I upgrade from e2fsprogs 1.42.12-2+b1 (the version in Jessie) to either the version in Stretch or the version in jessie-backport. So it looks like e2fsprogrs 1.43, breaks something with existing ext2simg (I am using ext2simg from android-tools-fsutils, so the 'old' one). if i use img2simg, then of course, there is no corruption. > >> Also, the commit that brought this file has the following reference: >> >> From AOSP commit: c1b7d19958dc3dbe53810811ea3dcc4f04f85c73 >> >> However I don't think this commit in AOSP. > > Hmm... I don't have easy access to AOSP at the moment, since I'm at > home and it's not on my upstream development laptop, but is it > possible that the contents of the git commit exists, but with a > different git commit ID? My understanding is that changes to > external/e2fsprogs, once code reviewed in Gerrit, do get pushed out to > the AOSP master branch right away. Android source trees have a > massively complex merging scheme, and the most likely explanation is > that I used the git commit ID belonging to some merge into a tree that > isn't published in AOSP. Either that, or the commit ID got somehow > corrupted while it was in my editor doing the cherry-pick into the > upstream tree. argh. too bad, we lost the back link.. > > Cheers, > > - Ted > > P.S. One of the other really nice things you can do as a result of a > lot of this effort is you can now build tools such as debugfs for > Android, built against bionic. So it becomes a lot easier to build a > userdebug android image that includes debugfs, which is helpful if you > need to try to debug some interesting new failure. > > And using mke2fs instead of make_ext4fs means that if we do add some > new file system feature into ext4/e2fsprogs which Android wants to > depend upon, it will be a lot easier to keep things in sync with the > external upstream world. [1] https://packages.debian.org/sid/android-libsparse ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: e2fsprogs: building ext2simg 2017-10-09 8:24 ` Nicolas Dechesne @ 2017-10-09 12:59 ` Theodore Ts'o 0 siblings, 0 replies; 4+ messages in thread From: Theodore Ts'o @ 2017-10-09 12:59 UTC (permalink / raw) To: Nicolas Dechesne; +Cc: linux-ext4 On Mon, Oct 09, 2017 at 10:24:46AM +0200, Nicolas Dechesne wrote: > > The problems appears as soon as I upgrade from e2fsprogs 1.42.12-2+b1 > (the version in Jessie) to either the version in Stretch or the > version in jessie-backport. So it looks like e2fsprogrs 1.43, breaks > something with existing ext2simg (I am using ext2simg from > android-tools-fsutils, so the 'old' one). So *that* shouldn't have happened. I do try to keep the ABI/API of the libext2fs shared library stable, even across major version numbers. That being said, I suspect ext2simg was doing something that made it highly dependent on the internal implementation detalis of libext2fs. It would be interesting to know what in the world that was, so I can try to avoid that in the future. But I've always viewed many of these tools to be quite highly fragile, so I'm not sure there's much we can do to fix this in terms of changing what libext2fs is doing. But if I can fix/improve the backwards compatibility problem from libext2fs's side, I'm certainly willing to entertain that notion. - Ted ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-10-09 12:59 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-10-08 20:04 e2fsprogs: building ext2simg Nicolas Dechesne 2017-10-08 21:27 ` Theodore Ts'o 2017-10-09 8:24 ` Nicolas Dechesne 2017-10-09 12:59 ` Theodore Ts'o
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).