Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores

Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores

[Phillip Lougher]

On Wed, 31 Jul 2019 17:43:44 +0200 Alexander Couzens <lynxis@fe80.eu> wrote:
> Hi László,
>
> > On Wed, Jul 31, 2019 at 3:06 AM Steven Shiau <steven@nchc.org.tw>
> > wrote: [...]
> >  As mentioned in the past, it's the
> > 0016-remove-frag_deflator_thread.patch that needs either reverted or
> > fixed by it's original author, Alexander Couzens (added to this mail).
>
> Since I've full weeks ahead, I can not make any estimation when
> I've time to look onto the performance issue.
>

That patch is a laughable piece of rubbish.  I believe both you
people (the Debian maintainer and author) are in total denial
about your incompetence in this matter.  This is obviously just my
opinion I've formed over the last couple of months, in case you want to
claim that it is libellous.

It is, obvious, from the patch where the problem lies.  You *remove*
the parallel fragment compressor threads, and move the work onto the
main thread.

Now what do you think that does?  It completely removes a significant
amount of the parallelism of Mksquashfs and makes the main thread
a bottleneck.

This is your fault and your problem, and it lies in your lack of
understanding.

Yet you continually blame your inability to make Mksquashfs work
correctly on *my* code being old and unmaintained and badly written.

This can be seen from the following excerpt from a post in this
Debian bug thread made by the Debian maintainer.

"> First of all, as squashfs-tools wasn't written in the last years, when
> reproducible builds became more famous. So it's not written
> with reproducible building in mind.
> For me is reproducible builds more important than using all cpu cores.
> But I don't use it with gigabytes images.
 Yeah, it's quite an old software without real development in the recent years."

and

" This sounds more complex work than it can be achieved in this week.
Maybe a complete rewrite would be better then on the long run."

Constantly pointing the blame at my tools and my code.

This is typical of the poor worker who chooses to blame everyone
else but himself.

None of that is the case.  50% of the code-base of Squashfs-tools
was (re-)written in the last 9 years as part of on-going improvements.
It has also been maintained across that period.

As for your specific claim that Mksquashfs can't be made to produce
reproducible builds because it is old and written before reproducible
builds became popular.  That is abject nonsense.

I added reproducible builds to Mksquashfs.  It took one weekend.

https://github.com/plougher/squashfs-tools/commit/24da0c63c80be64e1adc3f24c27459ebe18a19af

> > > Or shall we gradually switch to squashfs-tools-ng is the upstream
> > > is more active:
> > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965
> >  It's under investigation. I'm traveling and will only arrive back
> > home on Sunday evening. Expect more details on this next week.
>
> I also seen the upcoming squashfs-tools-ng rising and quite interested
> to test it and reading the code.
> Depending on tests & the code, I could imagine I'll put my effort and
> time towards squashfs-tools-ng.
>
> @László It should be your call to revert or not. For sure there are
> also downstream users who need reproducible builds (e.g. tails) who
> may also change to squashfs-tools-ng if -ng is the only reproducible
> squashfs generator in debian.
>

Upstream Squashfs-tools now produces reproducible builds.

Squashfs-tools-ng, this is a rogue project masquerading as an
official new upstream .  It is neither official nor a new upstream.  As
Squashfs author and maintainer I completely disassociate that project
with the Squashfs project.

I also have publicly stated that this project is spreading falsehoods and
that it is defamatory to me as the Squashfs author and maintainer.

See:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965#29

and

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965#34

I have lived for a couple of months with you two people bad-mouthing
Squashfs tools, it's code-base and my maintenance.

I have absolutely had enough.

I have CC'd this to the Debian project leader and the Debian technical
commitee, and also to linux-kernel, linux-fsdevel and linux-embedded
for wider attention.

What else do I have to do to make you stop bad-mouthing Squashfs?  Sue?

Yours

Dr. Phillip Lougher

> Hopefully I'll find time to look into it.
>
> Best Regards,
> lynxis
> --
> Alexander Couzens
>
> mail: lynxis@fe80.eu
> jabber: lynxis@fe80.eu
> gpg: 390D CF78 8BF9 AA50 4F8F  F1E2 C29E 9DA6 A0DF 8604

Re: Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores

[László Böszörményi (GCS)]

On Thu, Aug 1, 2019 at 5:27 PM Phillip Lougher
<phillip.lougher@gmail.com> wrote:
> On Wed, 31 Jul 2019 17:43:44 +0200 Alexander Couzens <lynxis@fe80.eu> wrote:
> > > On Wed, Jul 31, 2019 at 3:06 AM Steven Shiau <steven@nchc.org.tw>
> > > wrote: [...]
> > >  As mentioned in the past, it's the
> > > 0016-remove-frag_deflator_thread.patch that needs either reverted or
> > > fixed by it's original author, Alexander Couzens (added to this mail).
> >
> > Since I've full weeks ahead, I can not make any estimation when
> > I've time to look onto the performance issue.
>
> That patch is a laughable piece of rubbish.  I believe both you
> people (the Debian maintainer and author) are in total denial
> about your incompetence in this matter.  This is obviously just my
> opinion I've formed over the last couple of months, in case you want to
> claim that it is libellous.
 Good to know that you could form your opinion in the last couple of
months as on the other side I didn't know you until last week.

> It is, obvious, from the patch where the problem lies.  You *remove*
> the parallel fragment compressor threads, and move the work onto the
> main thread.
>
> Now what do you think that does?  It completely removes a significant
> amount of the parallelism of Mksquashfs and makes the main thread
> a bottleneck.
>
> This is your fault and your problem, and it lies in your lack of
> understanding.
 Who said we don't know where is the bottleneck? Alexander only stated
he can't work on fixing the bottleneck for some weeks.
While I don't know Alexander in person, according to my information he
contributes to 227 projects. Those programmed in C, C++, Python and
Rust; I wouldn't say he lacks any understanding in code.

> Yet you continually blame your inability to make Mksquashfs work
> correctly on *my* code being old and unmaintained and badly written.
 Never said that 1) mksquashfs doesn't work correctly, 2) it has a
problem which you don't fix and not possible to fix by others because
your code quality is bad. Please prove it where I have said such
things.

> This can be seen from the following excerpt from a post in this
> Debian bug thread made by the Debian maintainer.
>
> "> First of all, as squashfs-tools wasn't written in the last years, when
> > reproducible builds became more famous. So it's not written
> > with reproducible building in mind.
> > For me is reproducible builds more important than using all cpu cores.
> > But I don't use it with gigabytes images.
>  Yeah, it's quite an old software without real development in the recent years."
 Just for the record, this is Debian bugreport #919207 [1] started by
a third person (not me or Alexander), Chris Lamb. Neither written by
me, but Alexander[2]. More importantly it only states that development
goals was not that the result image should always be the same (be
reproducible). It doesn't say anything about code quality.

> " This sounds more complex work than it can be achieved in this week.
> Maybe a complete rewrite would be better then on the long run."
 Written by me, talking _about the patch_ used for the
reproducibility[3] and _not_ about your code, see which part of the
previous email was referred above.

> Constantly pointing the blame at my tools and my code.
 Again, where is the blame on your code? You didn't code it with
reproducibility in mind, but it doesn't mean in any way that your code
would be bad.

> This is typical of the poor worker who chooses to blame everyone
> else but himself.
 Let me ask, who is the one who puff it up, cross post to unrelated
mailing lists (like LKML, which doesn't involved with Debian matters
or squashfs-tools application development)?

> None of that is the case.  50% of the code-base of Squashfs-tools
> was (re-)written in the last 9 years as part of on-going improvements.
> It has also been maintained across that period.
 Until recently the know VCS location was at SourceForge[4]. The last
commit there from 2017-08-15, adding zstd support by Sean Purcell. In
2017 there were eight commits, the one mentioned previously and thinks
like "Make all compressor functions static", "Fix pseudo format error
message", "add pseudo definition format to the help text" and "improve
the error message when filenames with spaces are used" etc. Doesn't
look like a rewrite.

> As for your specific claim that Mksquashfs can't be made to produce
> reproducible builds because it is old and written before reproducible
> builds became popular.  That is abject nonsense.
 Where do you see the statement it can't be made to produce
reproducible images? Again, it only said when you originally coded
squashfs-tools this was not in your goals.

> I added reproducible builds to Mksquashfs.  It took one weekend.
>
> https://github.com/plougher/squashfs-tools/commit/24da0c63c80be64e1adc3f24c27459ebe18a19af
 On June the 30th, 2019 just for the record. For some reason you
didn't notice anyone about this, that you have the correct solution
when talking about this matter[5].

> Squashfs-tools-ng, this is a rogue project masquerading as an
> official new upstream .  It is neither official nor a new upstream.  As
> Squashfs author and maintainer I completely disassociate that project
> with the Squashfs project.
 It has a different name, immediately stating it's "a new set of tools
for working with SquashFS images"[6]. Where do you see it saying it's
the main project for squashfs images and/or your code is no more or
just shouldn't be used? Then your code is under the GPL-2 or later
license meaning it can be changed, extended and/or reimplemented with
a different name. Nothing happened against this license or against
your personal life / promotion.
What else David should add to prevent all of your misunderstandings?

> I also have publicly stated that this project is spreading falsehoods and
> that it is defamatory to me as the Squashfs author and maintainer.
 I still don't see these nor a personal offense against you.

> I have lived for a couple of months with you two people bad-mouthing
> Squashfs tools, it's code-base and my maintenance.
 As for me, I've stated only public facts. Like that two security
vulnerabilities, CVE-2015-4645 and CVE-2015-4646 were reported on July
20, 2015 [7] (maybe even before) and you only fixed these on July 15,
2019 [8] with last bits on July 20, 2019 [9] (exactly) four years
later. Didn't judged you on this or questioned why it took so long if
you are such an active maintainer of squashfs-tools.

> I have absolutely had enough.
>
> I have CC'd this to the Debian project leader and the Debian technical
> commitee, and also to linux-kernel, linux-fsdevel and linux-embedded
> for wider attention.
 Let me add Chris Lamb then the previous Debian Project Leader (also
British just like you [as I know] and you may sit down and talk about
this in person) who asked for the reproducibility patch / build in the
first place.

> What else do I have to do to make you stop bad-mouthing Squashfs?  Sue?
 If you feel yourself better with that, be my guest. I don't know who
is the lawyer of Debian, but I'm sure s/he can show you that it's only
you who dance this storm.

Regards,
Laszlo/GCS
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#5
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#83
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#88
[4] https://sourceforge.net/p/squashfs/code/ci/e38956b92f738518c29734399629e7cdb33072d3/log/?path=
[5] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921146#75
[6] https://github.com/AgentD/squashfs-tools-ng
[7] https://lwn.net/Articles/651775/
[8] https://github.com/plougher/squashfs-tools/commit/f95864afe8833fe3ad782d714b41378e860977b1
[9] https://github.com/plougher/squashfs-tools/commit/ba215d73e153a6f237088b4ecb88c702bb4d4183

Re: Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores

[Phillip Lougher]

On Thu, Aug 1, 2019 at 11:26 PM Don Armstrong <don@debian.org> wrote:
>
> On Thu, 01 Aug 2019, Phillip Lougher wrote:
> > That patch is a laughable piece of rubbish.  I believe both you
> > people (the Debian maintainer and author) are in total denial
> > about your incompetence in this matter.  This is obviously just my
> > opinion I've formed over the last couple of months, in case you want to
> > claim that it is libellous.
>
> This isn't an appropriate tone for Debian mailing lists or the our bug
> tracking system.
>
> It's fine to disagree on technical matters, but it's not appropriate to
> claim that people are incompetent or that they are making rubbish.
>

I am only defending myself from the slurs and false information being
spread by your maintainer.

I would not be doing this otherwise.

Cheers

Phillip

> Please stop.
>
> --
> Don Armstrong                      https://www.donarmstrong.com
>
> To punish me for my contempt of authority, Fate has made me an
> authority myself
>  -- Albert Einstein

Re: Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores

[Theodore Y. Ts'o]

Phillip,

Peace.

You may not like the fact that David Oberhollenzer (GitHub username
AgentD) started an effort to implement a new set of tools to generate
squashfs images on April 30th, 2019, and called it squashfs-tools-ng.

However, it's really not fair to complain that there is a "violation
of copyright" given that all of squashfs-tools was released is under
the GPL.  Using some text from squashfs-tools in the package
description or documentation of squashfs-tools-ng is totally allowed
under the GPL.  You could complain that they didn't include an
acknowledgement that text was taken from your program.  But then give
them time to fix up the acknowledgements.  Assuming good faith is
always a good default.

The other thing that you've complained about is that some folks have
(inaccurately, in your view) described squashfs-tools as not being
maintained.  I'd encourage you to take a step back, and consider how
this might be quite understandable how they might have gotten that
impression.

First of all, let's look on the documentation in kernel source tree,
located at Documentation/filesystems/squashfs.txt.  It states that
squashfs-tools's web site is www.squashfs.org, and the git tree is at
git://git.kernel.org/pub/scm/fs/squashfs/squashfs-tools.git.

The web site www.squashfs.org is not currently responding, but
according to the Internet Archive, it was redirecting to
http://squashfs.sourceforge.net/.  This web site describes the latest
version of squashfs-tools is 4.2, released in February 2011, It
apparently wasn't updated when squashfs-tools 4.3 was released in May
2014.

The git.kernel.org tree is identical to the sourceforge.net's git
tree.  That tree's most recent commit is from August 2017, e38956b9
("squashfs-tools: Add zstd support").  Now, the fascinating thing is
that the github tree has a completely different commit-id for the same
commit is 61133613 ("squashfs-tools: Add zstd support").  The git
commit that the two trees have in common is 9c1db6d1 from September
2014.

Reconstructing the git history, you didn't make any commits between
September 2014 and March 2017.  At that time, you merged a number of
github pull requests between 2014 and 2017, but then exported them as
patches and applied them on the kernel.org/sourceforge git trees.
Why, I'm not sure.

In August 2017, you stopped updating the kernel.org and sourceforge
git trees, and abandoned them.  After that for the rest of 2017, you
merged one more pull request, and applied one commit to add the -nold
option.  In 2018, there were only two commits, in February and June.
And then nothing until April 2019 (about the time that squash-tools-ng
was started/announced), there has been a flurry of activity, including
merging github pull requests from 2017 and 2018, antd you've done a
lot of work since then.

I say this not to criticize the amount of attention you've paid to
squashfs-tools, but to point out that when David started work on
squashfs-tools-ng, it's not unreasonable that he might have gotten the
impression that development had ceased --- especially if he followed
the documentation in the kernel sources, and found an extremely
cobwebby website, and a git tree on git.kernel.org that hadn't been
updated since 2017, with substantive heavy development basically
ending in 2014 (which is also when the last release of squashfs-tools
was cut).  You don't need to ascribe malice to what might just simply
be an impression by looking in the official locations in the official
kernel documentation!

As a fellow kernel file system developer, let me make a few
suggestions.

* Don't worry about with "competing" software projects.  For a while,
  a multi-billion dollar company attempted to maintain a BSD-licensed
  "competition" to some of the programs in e2fsprogs.  This was
  because Andy Rubin was highly allergic to the GPL way back when.  I
  pointed the independent implementation was creating invalid file
  systems, and was buggy, and in general was making that billion
  dollar company's life harder, not easier.  They eventually gave up
  on it, and Android uses e2fsprogs these days.

  The whole point of open source is "may the best code win".  If
  you're convinced that you, as the upstream kernel developer, can do
  a better job maintaining the userspace tools, then instead of
  complaining and threatening to sue, just keep your head down, and
  keep improving your code, and in the end, the best code will win.

* I'd suggest that you make sure there is a single canonical git tree.
  It appears it's the github version of your git tree.  So... starting
  with your github tree, do a "git merge" of the master branch from
  git.kernel.org, and then push updates to github, git.kernel.org, and
  git.sf.code.net.  It's fine to have multiple mirrors of your git
  tree.  I maintain multiple copies of git e2fsprogs repo on
  git.kernel.org, github, and sourceforge.

* Please consider tagging your releases.  There are git tags for
  squashfs 3.1 and 3.2, but none of your 4.x releases.  It's going to
  make it much easier for other people to know which git commits
  correspond to your official releases.  For bonus points, you could
  use signed git tags.  If you need help getting that set up, please
  contact me off-line.  I'd be delighted to help you with that.

* Please consider updating the squashfs web site.  I acutally keep a
  copy of the e2fsprogs.sourceforge.net web site in the e2fsprogs git
  tree, under the "web" branch.  You can see it here:
  https://github.com/tytso/e2fsprogs/tree/web

* It can be handy to audit and apply/merge patches being carried by
  distributions.  Before I took over the Debian maintainership of
  e2fsprogs, I would periodically scan the debian patches (and I still
  keep an eye to see what Ubuntu and Fedora have in their local
  patches).  If any of those patches fix real bugs, I'll pull them
  into the e2fsprogs git repo, and then send a note to the
  distribution maintainer that I've done so, and let them know that in
  the next release, I've included their patch, and so they can drop it
  from their tree.  That way, I can find and fix bugs introduced by
  distribution patches.

In general, I've found that keeping on good terms with the
distribution packagers is really good thing from the perspective of
the upstream author.

Best regards,

						- Ted
						

Re: Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores

[Phillip Lougher]

On Thu, Aug 1, 2019 at 11:41 PM László Böszörményi (GCS) <gcs@debian.org> wrote:

>  Let me add Chris Lamb then the previous Debian Project Leader (also
> British just like you [as I know] and you may sit down and talk about
> this in person) who asked for the reproducibility patch / build in the
> first place.
>

If Chris Lamb or anyone else wants a face-to-face meeting I'm more
than happy to do so.

I coincidentally have a week's holiday (vacation) next week, and I'm
happy to spend a day of it travelling and meeting to discuss the
situation.

I do want to de-escalate this situation if possible.

Phillip

> > What else do I have to do to make you stop bad-mouthing Squashfs?  Sue?
>  If you feel yourself better with that, be my guest. I don't know who
> is the lawyer of Debian, but I'm sure s/he can show you that it's only
> you who dance this storm.
>
> Regards,
> Laszlo/GCS
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#5
> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#83
> [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919207#88
> [4] https://sourceforge.net/p/squashfs/code/ci/e38956b92f738518c29734399629e7cdb33072d3/log/?path=
> [5] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921146#75
> [6] https://github.com/AgentD/squashfs-tools-ng
> [7] https://lwn.net/Articles/651775/
> [8] https://github.com/plougher/squashfs-tools/commit/f95864afe8833fe3ad782d714b41378e860977b1
> [9] https://github.com/plougher/squashfs-tools/commit/ba215d73e153a6f237088b4ecb88c702bb4d4183

Re: Bug#921146: Program mksquashfs from squashfs-tools 1:4.3-11 does not make use all CPU cores

[Phillip Lougher]

On Fri, Aug 2, 2019 at 1:35 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> Phillip,
>

Hi Theodore (Ted),

Thank-you for your kind and well intentioned email.  It has
de-escalated tensions on my side.

It is very late here, and so I can only be brief, but, I want this
said as soon as possible.

> Peace.

Yes, I would very much like that to happen.

>
> You may not like the fact that David Oberhollenzer (GitHub username
> AgentD) started an effort to implement a new set of tools to generate
> squashfs images on April 30th, 2019, and called it squashfs-tools-ng.
>
> However, it's really not fair to complain that there is a "violation
> of copyright" given that all of squashfs-tools was released is under
> the GPL.  Using some text from squashfs-tools in the package
> description or documentation of squashfs-tools-ng is totally allowed
> under the GPL.  You could complain that they didn't include an
> acknowledgement that text was taken from your program.  But then give
> them time to fix up the acknowledgements.  Assuming good faith is
> always a good default.
>
> The other thing that you've complained about is that some folks have
> (inaccurately, in your view) described squashfs-tools as not being
> maintained.  I'd encourage you to take a step back, and consider how
> this might be quite understandable how they might have gotten that
> impression.
>
> First of all, let's look on the documentation in kernel source tree,
> located at Documentation/filesystems/squashfs.txt.  It states that
> squashfs-tools's web site is www.squashfs.org, and the git tree is at
> git://git.kernel.org/pub/scm/fs/squashfs/squashfs-tools.git.
>
> The web site www.squashfs.org is not currently responding, but
> according to the Internet Archive, it was redirecting to
> http://squashfs.sourceforge.net/.  This web site describes the latest
> version of squashfs-tools is 4.2, released in February 2011, It
> apparently wasn't updated when squashfs-tools 4.3 was released in May
> 2014.
>

I gave up on Sourceforge many years ago when it came unusable (in my
opinion) due to too many adverts.

If I knew how to remove Squashfs from Sourceforge to save confusion I
would have done so.

> The git.kernel.org tree is identical to the sourceforge.net's git
> tree.  That tree's most recent commit is from August 2017, e38956b9
> ("squashfs-tools: Add zstd support").  Now, the fascinating thing is
> that the github tree has a completely different commit-id for the same
> commit is 61133613 ("squashfs-tools: Add zstd support").  The git
> commit that the two trees have in common is 9c1db6d1 from September
> 2014.
>
> Reconstructing the git history, you didn't make any commits between
> September 2014 and March 2017.  At that time, you merged a number of
> github pull requests between 2014 and 2017, but then exported them as
> patches and applied them on the kernel.org/sourceforge git trees.
> Why, I'm not sure.
>

I clicked the web merge button on GitHub, and then ended up with the
patches in the GitHUb repository (which I didn't use at the time), and
synced manually with kernel.org/sourceforge.

Blame lack of knowledge of GitHub. on my behalf.  I tend to prefer
command-interfaces which can be scripted.

> In August 2017, you stopped updating the kernel.org and sourceforge
> git trees, and abandoned them.  After that for the rest of 2017, you
> merged one more pull request, and applied one commit to add the -nold
> option.  In 2018, there were only two commits, in February and June.
> And then nothing until April 2019 (about the time that squash-tools-ng
> was started/announced), there has been a flurry of activity, including
> merging github pull requests from 2017 and 2018, antd you've done a
> lot of work since then.
>

The start of development in April and the co-incidence with the
squashfs-tools-ng project is purely coincidental.   I didn't know
anything about squashfs-tools-ng until Matt Turner of Gentoo mentioned
it in an issue on GitHub
(https://github.com/plougher/squashfs-tools/issues/25) nine days ago.

I didn't know about this co-incidence until you pointed it out.  This
in fact may explain some of the mis-understanding.

I meant to do some development on Squashfs-tools over the last
Christmas vacation.  I don't want to go into private family details.
but two deaths and a stroke in the family meant I had more important
things to deal with.  April was then the first opportunity I got to do
some development.

I try to keep people up to date with my intentions and commitments,
and I mentioned all this in another issue on GitHub
(https://github.com/plougher/squashfs-tools/issues/54).

> I say this not to criticize the amount of attention you've paid to
> squashfs-tools, but to point out that when David started work on
> squashfs-tools-ng, it's not unreasonable that he might have gotten the
> impression that development had ceased --- especially if he followed
> the documentation in the kernel sources, and found an extremely
> cobwebby website, and a git tree on git.kernel.org that hadn't been
> updated since 2017, with substantive heavy development basically
> ending in 2014 (which is also when the last release of squashfs-tools
> was cut).

In 2012-2014 I was put into a difficult position.  The previous year I
had started work @ Redhat as a kernel maintainer, which was leaving me
very little free time.

But the tools (especially Mksquashfs) which I had written in my spare
time between 2002-2009, when Squashfs was purely a hobbyist
filesystem, were increasingly breaking.  There were issues with stack
size, overflows, and basically dozens of things which fuzzers and
others like to exploit to produce crashes etc.

I decided I had to do a major rewrite of Mksquashfs.  It took over two
years, working all evenings available (from work) and most weekends
and at the end of it I felt completely burnt out.  My intention was
this should give me space to concentrate on other things for a while,
having cleared up all the issues.

I went on vacation a couple of months after release, deliberately
where there was no internet (off grid).  When I came back I found a
number of highly abusive emails from people, that had gotten more
abusive as I'd not answered them for a week.  I then had what I now
recognise to be a nervous breakdown.  I destroyed all my development
files and notes, and I couldn't bear to look at a line of Squashfs
code for a couple of months.

I obviously came back after a couple of months.  But, I took the
decision to not let Squashfs become the nightmare it had been for a
couple of years.  I would step back and cease to do pro-active
development and concentrate on security issues, bug fixes and
correspondence.

I genuinely felt things were going well and I was getting a good
balance between my life, my job, and Squashfs, until now.

Perhaps that is why I have reacted "so badly" now, it is called dismay.

I can't write anymore as it is already very late.

Best Regards

Phillip



> You don't need to ascribe malice to what might just simply
> be an impression by looking in the official locations in the official
> kernel documentation!
>
> As a fellow kernel file system developer, let me make a few
> suggestions.
>
> * Don't worry about with "competing" software projects.  For a while,
>   a multi-billion dollar company attempted to maintain a BSD-licensed
>   "competition" to some of the programs in e2fsprogs.  This was
>   because Andy Rubin was highly allergic to the GPL way back when.  I
>   pointed the independent implementation was creating invalid file
>   systems, and was buggy, and in general was making that billion
>   dollar company's life harder, not easier.  They eventually gave up
>   on it, and Android uses e2fsprogs these days.
>
>   The whole point of open source is "may the best code win".  If
>   you're convinced that you, as the upstream kernel developer, can do
>   a better job maintaining the userspace tools, then instead of
>   complaining and threatening to sue, just keep your head down, and
>   keep improving your code, and in the end, the best code will win.
>
> * I'd suggest that you make sure there is a single canonical git tree.
>   It appears it's the github version of your git tree.  So... starting
>   with your github tree, do a "git merge" of the master branch from
>   git.kernel.org, and then push updates to github, git.kernel.org, and
>   git.sf.code.net.  It's fine to have multiple mirrors of your git
>   tree.  I maintain multiple copies of git e2fsprogs repo on
>   git.kernel.org, github, and sourceforge.
>
> * Please consider tagging your releases.  There are git tags for
>   squashfs 3.1 and 3.2, but none of your 4.x releases.  It's going to
>   make it much easier for other people to know which git commits
>   correspond to your official releases.  For bonus points, you could
>   use signed git tags.  If you need help getting that set up, please
>   contact me off-line.  I'd be delighted to help you with that.
>
> * Please consider updating the squashfs web site.  I acutally keep a
>   copy of the e2fsprogs.sourceforge.net web site in the e2fsprogs git
>   tree, under the "web" branch.  You can see it here:
>   https://github.com/tytso/e2fsprogs/tree/web
>
> * It can be handy to audit and apply/merge patches being carried by
>   distributions.  Before I took over the Debian maintainership of
>   e2fsprogs, I would periodically scan the debian patches (and I still
>   keep an eye to see what Ubuntu and Fedora have in their local
>   patches).  If any of those patches fix real bugs, I'll pull them
>   into the e2fsprogs git repo, and then send a note to the
>   distribution maintainer that I've done so, and let them know that in
>   the next release, I've included their patch, and so they can drop it
>   from their tree.  That way, I can find and fix bugs introduced by
>   distribution patches.
>
> In general, I've found that keeping on good terms with the
> distribution packagers is really good thing from the perspective of
> the upstream author.
>
> Best regards,
>
>                                                 - Ted
>