Linux-4.X-rcY patches can't be applied with git?

Linux-4.X-rcY patches can't be applied with git?

[Jarod Wilson]

It's entirely possible that we (Red Hat and the Fedora kernel team) are
doing something wrong here, but to the best of our knowledge, it seems
that the canonical upstream RC snap patches can't be applied to a tree
using either git or old-fashioned patch.

For example:
$ cd /tmp
$ wget http://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.8.tar.xz
$ tar xf linux-4.8.xz
$ cd linux-4.8
$ wget http://cdn.kernel.org/pub/linux/kernel/v4.x/testing/patch-4.9-rc2.xz
$ unxz patch-4.9-rc2.xz
$ git --version
git version 2.10.0
$ git apply patch-4.9-rc2
patch-4.9-rc2:15822: space before tab in indent.
                                IRQ_AUDIO pin
patch-4.9-rc2:15832: space before tab in indent.
                                IRQ_RTC pin
patch-4.9-rc2:19120: space before tab in indent.
                        };
patch-4.9-rc2:20847: space before tab in indent.
        - "moxa,moxart-timer"
patch-4.9-rc2:20848: space before tab in indent.
        - "aspeed,ast2400-timer"
error: cannot apply binary patch to 'Documentation/media/media_api_files/typical_media_device.pdf' without full index line
error: Documentation/media/media_api_files/typical_media_device.pdf: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/dev-raw-vbi_files/vbi_525.pdf' without full index line
error: Documentation/media/uapi/v4l/dev-raw-vbi_files/vbi_525.pdf: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/dev-raw-vbi_files/vbi_625.pdf' without full index line
error: Documentation/media/uapi/v4l/dev-raw-vbi_files/vbi_625.pdf: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt.gif' without full index line
error: Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt.gif: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt.png' without full index line
error: Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt.png: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt_example.gif' without full index line
error: Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt_example.gif: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt_example.png' without full index line
error: Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt_example.png: patch does not apply
$ echo $?
1

Most of the patch applies with 'patch -p1 < patch-4.9-rc2', but those same
binary files fail. No big deal, except that there's an error return from
patch, which means an rpm build using standard rpm macro stuff fails.
That's actually one of the reasons we switched to using git to apply any
patches in the Fedora kernel rpm spec files. But as per above, that doesn't
work all the tme either, which has resulted in the addition of a perl
script that gets called to strip binary patch data out entirely, which of
course, makes the resulting rpm omit the binary changes entirely, and thus
not *quite* 4.9-rc2 that gets spit out here.

Perl script:
 http://pkgs.fedoraproject.org/cgit/rpms/kernel.git/tree/remove-binary-diff.pl

However, if the RC patch is generated like so:

$ cd ~/src/linux
$ git diff --binary v4.8..v4.9-rc2 > /tmp/linux-4.8/patch-4.9-rc2

Then:

$ cd /tmp/linux-4.8
$ git apply patch-4.9-rc2
patch-4.9-rc2:8359: space before tab in indent.
        ./configure
patch-4.9-rc2:8719: space before tab in indent.
        alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER,
patch-4.9-rc2:8720: space before tab in indent.
                                  CRYPTO_ALG_TYPE_MASK);
patch-4.9-rc2:8721: space before tab in indent.
        if (IS_ERR(alg))
patch-4.9-rc2:8725: space before tab in indent.
        /* Block size must be >= 4 bytes. */
warning: squelched 104 whitespace errors
warning: 109 lines add whitespace errors.
$ echo $?
0

I just noticed that the 'space before tab in indent' noise is different
between the two, but otherwise, things seem to work much better. I know
it results in a slightly larger patch file to download (unxz size is 2MB
larger), and it still will not apply with plain old patch, but at least
it applies with git.

Or am I missing something blindingly obvious in the git apply options
that would work with the existing patches?

-- 
Jarod Wilson
jarod@redhat.com

Re: Linux-4.X-rcY patches can't be applied with git?

[Linus Torvalds]

On Mon, Oct 24, 2016 at 11:25 AM, Jarod Wilson <jarod@redhat.com> wrote:
> It's entirely possible that we (Red Hat and the Fedora kernel team) are
> doing something wrong here, but to the best of our knowledge, it seems
> that the canonical upstream RC snap patches can't be applied to a tree
> using either git or old-fashioned patch.

No, you're not imagining it, it's definitely happening.

What is going on is that I generate patches without the "--binary"
flag, which means that git skips the binary diffs entirely. So the
diff just contains

   Binary files
a/Documentation/media/media_api_files/typical_media_device.pdf and
b/Documentation/media/media_api_files/typical_media_device.pdf differ

Then, when you do "git apply", "git apply" will see that, and try to
use the index lines to regenerate the thing, which obviously only
works in a repository that already _has_ those objects.

This is actually not new. I've skipped binary files for the last ten
years or so in the diffs, because the diffs are completely illegible,
and nobody has ever cared - and non-git users haven't been able to use
them anyway.

Obviously, part of it is too that we simply don't have very many
binary files, so it almost never ends up being a problem. The
documentation changes made them happen now.

But quite frankly, I see the tar-balls and diffs as a way to have
non-git people have a minimally working system, and their *only* point
is for people who don't have git.

And since plain "patch" cannot handle git binary diffs anyway, there
was never any valid reason to include the binary diffs in the diff.

Btw, this is why the diffs don't have renames either.

The diffs would often be much smaller, and certainly much more
legible, if I used the "-M" or "-C" flags, but since the primary
reason for the tarballs and diffs existing is for non-git users, and
traditional "patch" doesn't understand rename diffs, I don't do it.

(Yes, modern GNU patch has actually grown support for rename diffs,
although last I looked it gets it wrong for some of the more complex
cases - notably cross renames).

Summary:

 - if you have git, you shouldn't use the tar-balls and patches

 - if you don't have git, binary patches and renames wouldn't work for
you anyway, so generating them is pointless and would potentially keep
you from getting a working tree.

I could easily add "--binary" to my script, because I _think_
traditional "patch" will just ignore it as noise, but I'd honestly
rather discourage people from downloading full tar-balls in the first
place.

Hmm?

               Linus

Re: Linux-4.X-rcY patches can't be applied with git?

[Josh Boyer]

On Mon, Oct 24, 2016 at 3:24 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Mon, Oct 24, 2016 at 11:25 AM, Jarod Wilson <jarod@redhat.com> wrote:
>> It's entirely possible that we (Red Hat and the Fedora kernel team) are
>> doing something wrong here, but to the best of our knowledge, it seems
>> that the canonical upstream RC snap patches can't be applied to a tree
>> using either git or old-fashioned patch.
>
> No, you're not imagining it, it's definitely happening.
>
> What is going on is that I generate patches without the "--binary"
> flag, which means that git skips the binary diffs entirely. So the
> diff just contains
>
>    Binary files
> a/Documentation/media/media_api_files/typical_media_device.pdf and
> b/Documentation/media/media_api_files/typical_media_device.pdf differ
>
> Then, when you do "git apply", "git apply" will see that, and try to
> use the index lines to regenerate the thing, which obviously only
> works in a repository that already _has_ those objects.
>
> This is actually not new. I've skipped binary files for the last ten
> years or so in the diffs, because the diffs are completely illegible,
> and nobody has ever cared - and non-git users haven't been able to use
> them anyway.
>
> Obviously, part of it is too that we simply don't have very many
> binary files, so it almost never ends up being a problem. The
> documentation changes made them happen now.
>
> But quite frankly, I see the tar-balls and diffs as a way to have
> non-git people have a minimally working system, and their *only* point
> is for people who don't have git.
>
> And since plain "patch" cannot handle git binary diffs anyway, there
> was never any valid reason to include the binary diffs in the diff.
>
> Btw, this is why the diffs don't have renames either.
>
> The diffs would often be much smaller, and certainly much more
> legible, if I used the "-M" or "-C" flags, but since the primary
> reason for the tarballs and diffs existing is for non-git users, and
> traditional "patch" doesn't understand rename diffs, I don't do it.
>
> (Yes, modern GNU patch has actually grown support for rename diffs,
> although last I looked it gets it wrong for some of the more complex
> cases - notably cross renames).
>
> Summary:
>
>  - if you have git, you shouldn't use the tar-balls and patches
>
>  - if you don't have git, binary patches and renames wouldn't work for
> you anyway, so generating them is pointless and would potentially keep
> you from getting a working tree.
>
> I could easily add "--binary" to my script, because I _think_
> traditional "patch" will just ignore it as noise, but I'd honestly
> rather discourage people from downloading full tar-balls in the first
> place.
>
> Hmm?

The benefit of tarballs and patches from a distribution standpoint is
purely size.  And yes, disk is very cheap but the size implications
are magnified by the fact that e.g. SRPMs are required and stored for
every binary build.  Today, the Fedora kernel SRPMs are ~94.5MB each.
If we just used git-archive or something to produce a tarball of a git
tree, it would explode to 664MB.  That's from a fresh clone of your
tree, run through 'git archive -o ../linux.tar.xz master'.  If we just
tarred it up without xz compression, a fresh tree is 1.8GB.  That's a
lot of storage, particularly when you're talking about daily builds.

You could just tar up the source files themselves from a git tree and
skip .git, but that means you lose the history and more importantly
you lose any semblance of verified sources because all the signed
commits are then gone.  Essentially, it's recreating what is uploaded
on kernel.org *minus* the fact that it is signed.  That just seems
dumb.

Perhaps I'm not thinking of something obvious, but it seems
tarballs+patches still have legitimate use cases from a distribution
point of view.  Making the patches provided on kernel.org usable by
the tooling we have available seems like a worthwhile goal to me...

josh

Re: Linux-4.X-rcY patches can't be applied with git?

[Linus Torvalds]

On Mon, Oct 24, 2016 at 2:02 PM, Josh Boyer <jwboyer@fedoraproject.org> wrote:
>
> The benefit of tarballs and patches from a distribution standpoint is
> purely size.

I wonder if you wouldn't be better off just generating your own diffs.

If you know the other end uses "git apply", then not only can you use
"--binary", but you can also enable rename detection.

That will often shrink the diffs a *lot*.

So for _me_, the primary use of patches is for non-git people.

They are hopefully rare by now, but quite frankly, if they go away as
an issue, then the solution for me isn't "start using fancier patches
that rely on git", but "don't even bother with tar-balls and patches
at all".

See?

But if you want to see patches as a distribution format for a distro,
then your issues are different, and you may well want to use
"--binary" and "-M" to generate patches.

It's just that your reasons to use patches sound very different from
_my_ reasons to use patches..

                Linus

Re: Linux-4.X-rcY patches can't be applied with git?

[Jarod Wilson]

On Mon, Oct 24, 2016 at 02:10:30PM -0700, Linus Torvalds wrote:
> On Mon, Oct 24, 2016 at 2:02 PM, Josh Boyer <jwboyer@fedoraproject.org> wrote:
> >
> > The benefit of tarballs and patches from a distribution standpoint is
> > purely size.
> 
> I wonder if you wouldn't be better off just generating your own diffs.
> 
> If you know the other end uses "git apply", then not only can you use
> "--binary", but you can also enable rename detection.
> 
> That will often shrink the diffs a *lot*.
> 
> So for _me_, the primary use of patches is for non-git people.
> 
> They are hopefully rare by now, but quite frankly, if they go away as
> an issue, then the solution for me isn't "start using fancier patches
> that rely on git", but "don't even bother with tar-balls and patches
> at all".
> 
> See?

Yes and no. What are these non-git people doing with the patches? Since
patch won't apply them, are they just reading them for personal
enlightenment? i.e., who really actually uses tarballs and the patches
anymore outside of distros for packaging, and for what? From a distro
standpoint, I think we'd like to at least continue to see release
tarballs, but we could quite easily make do with the "just don't even
bother" approach for RC patches.

> But if you want to see patches as a distribution format for a distro,
> then your issues are different, and you may well want to use
> "--binary" and "-M" to generate patches.
> 
> It's just that your reasons to use patches sound very different from
> _my_ reasons to use patches..

If "just generate your own" is the best answer, I'm okay with that for my
own use case, and I'll let Josh speak for the Fedora case. Just wanted to
make sure we weren't missing something obvious, and figured it couldn't
hurt to ask if it were simply a case of nobody having said there were
issues with them for what I imagine is one of their few remaining uses. :)

-- 
Jarod Wilson
jarod@redhat.com

Re: Linux-4.X-rcY patches can't be applied with git?

[Linus Torvalds]

On Mon, Oct 24, 2016 at 2:27 PM, Jarod Wilson <jarod@redhat.com> wrote:
>
> Yes and no. What are these non-git people doing with the patches? Since
> patch won't apply them, are they just reading them for personal
> enlightenment?

Oh, patch applies them fine.

They might end up not seeing some of the updates to binary files,
that's all. So things like the logo updates or some pdf file might not
work out for them, but that doesn't mean that they can't use the patch
and the resulting kernel.

             Linus

Re: Linux-4.X-rcY patches can't be applied with git?

[Jarod Wilson]

On Mon, Oct 24, 2016 at 02:45:01PM -0700, Linus Torvalds wrote:
> On Mon, Oct 24, 2016 at 2:27 PM, Jarod Wilson <jarod@redhat.com> wrote:
> >
> > Yes and no. What are these non-git people doing with the patches? Since
> > patch won't apply them, are they just reading them for personal
> > enlightenment?
> 
> Oh, patch applies them fine.
> 
> They might end up not seeing some of the updates to binary files,
> that's all. So things like the logo updates or some pdf file might not
> work out for them, but that doesn't mean that they can't use the patch
> and the resulting kernel.

Ah, you're right, of course. I was conflating "patch won't apply them"
with "rpmbuild fails, because patch returns an error code due to the
binary bits failing to apply".

But in that case, what if your patch generation script used a filter to
exclude those binary files? No harm to that target audience, and it would
actually make them behave better for distro builds. Though that might be
counter to the goal of making them disappear entirely. :)

-- 
Jarod Wilson
jarod@redhat.com

Re: Linux-4.X-rcY patches can't be applied with git?

[Linus Torvalds]

On Mon, Oct 24, 2016 at 4:18 PM, Jarod Wilson <jarod@redhat.com> wrote:
>
> But in that case, what if your patch generation script used a filter to
> exclude those binary files? No harm to that target audience, and it would
> actually make them behave better for distro builds. Though that might be
> counter to the goal of making them disappear entirely. :)

Heh, I'd rather people get the warning that "oops, something is
incomplete". They can still work with the end result, but at least
they got some indication that hey, that patch didn't work wonderfully
well...

To be honest, I really would like to not do the tar-balls and patches at all.

But maybe rather than saying "it's only for legacy 'patch' users", I
could just say that it's getting phased out, and say "you have to use
'git apply' to apply them".

Then I could just enable "--binary" and "-M", and see what happens.

I suspect that these days, git is so ubiquitous that it's ok.

And then in a few years, maybe I can just stop doing patches entirely,
having proved the point that everybody already has git ;)

                   Linus

Re: Linux-4.X-rcY patches can't be applied with git?

[Jarod Wilson]

On Mon, Oct 24, 2016 at 05:06:42PM -0700, Linus Torvalds wrote:
> On Mon, Oct 24, 2016 at 4:18 PM, Jarod Wilson <jarod@redhat.com> wrote:
> >
> > But in that case, what if your patch generation script used a filter to
> > exclude those binary files? No harm to that target audience, and it would
> > actually make them behave better for distro builds. Though that might be
> > counter to the goal of making them disappear entirely. :)
> 
> Heh, I'd rather people get the warning that "oops, something is
> incomplete". They can still work with the end result, but at least
> they got some indication that hey, that patch didn't work wonderfully
> well...
> 
> To be honest, I really would like to not do the tar-balls and patches at all.
> 
> But maybe rather than saying "it's only for legacy 'patch' users", I
> could just say that it's getting phased out, and say "you have to use
> 'git apply' to apply them".
> 
> Then I could just enable "--binary" and "-M", and see what happens.

I like this idea!

> I suspect that these days, git is so ubiquitous that it's ok.
> 
> And then in a few years, maybe I can just stop doing patches entirely,
> having proved the point that everybody already has git ;)

Honestly, the only people that don't have access to git to pull down
kernel sources? People who haven't yet got a kernel up and running, who
will probably get there via a distro kernel. ;)

Side note in favor of tarballs: I know Fedora likes upstream to have
tarballs, with checksums provided, so that packages can be verified to
contain a legitimate upstream source tarball, rather than a random tarball
created by the packager that could have some extraneous bits (possibly
malicious) added to them. One can certainly examine and validate a
generated tarball too, but it's a fair bit more work than just "does the
checksum match?" and not as easily automated.

-- 
Jarod Wilson
jarod@redhat.com

Re: Linux-4.X-rcY patches can't be applied with git?

[Josh Boyer]

On Mon, Oct 24, 2016 at 10:49 PM, Jarod Wilson <jarod@redhat.com> wrote:
> On Mon, Oct 24, 2016 at 05:06:42PM -0700, Linus Torvalds wrote:
>> On Mon, Oct 24, 2016 at 4:18 PM, Jarod Wilson <jarod@redhat.com> wrote:
>> >
>> > But in that case, what if your patch generation script used a filter to
>> > exclude those binary files? No harm to that target audience, and it would
>> > actually make them behave better for distro builds. Though that might be
>> > counter to the goal of making them disappear entirely. :)
>>
>> Heh, I'd rather people get the warning that "oops, something is
>> incomplete". They can still work with the end result, but at least
>> they got some indication that hey, that patch didn't work wonderfully
>> well...
>>
>> To be honest, I really would like to not do the tar-balls and patches at all.
>>
>> But maybe rather than saying "it's only for legacy 'patch' users", I
>> could just say that it's getting phased out, and say "you have to use
>> 'git apply' to apply them".
>>
>> Then I could just enable "--binary" and "-M", and see what happens.
>
> I like this idea!
>
>> I suspect that these days, git is so ubiquitous that it's ok.
>>
>> And then in a few years, maybe I can just stop doing patches entirely,
>> having proved the point that everybody already has git ;)
>
> Honestly, the only people that don't have access to git to pull down
> kernel sources? People who haven't yet got a kernel up and running, who
> will probably get there via a distro kernel. ;)

Possibly.  And to your earlier idea of generating our own diffs, we do
that for the subset of git snapshot builds we do in Rawhide between
the -rcX releases.  The problem with doing that all the time for
everything is that today the -rcX patches are signed by Linus.  Doing
it ourselves means they aren't.  Fedora/other distros could have the
maintainers sign their generated diffs but it loses some of the
verification chain.

> Side note in favor of tarballs: I know Fedora likes upstream to have
> tarballs, with checksums provided, so that packages can be verified to
> contain a legitimate upstream source tarball, rather than a random tarball
> created by the packager that could have some extraneous bits (possibly
> malicious) added to them. One can certainly examine and validate a
> generated tarball too, but it's a fair bit more work than just "does the
> checksum match?" and not as easily automated.

Right, this and the signing issue I pointed to above.

josh

Re: Linux-4.X-rcY patches can't be applied with git?

[Linus Torvalds]

On Mon, Oct 24, 2016 at 5:06 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> Then I could just enable "--binary" and "-M", and see what happens.

Let's see what happens with 4.9-rc3. Will anybody notice? Did I do it right?

             Linus

Re: Linux-4.X-rcY patches can't be applied with git?

[Laura Abbott]

On 10/29/2016 02:08 PM, Linus Torvalds wrote:
> On Mon, Oct 24, 2016 at 5:06 PM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>>
>> Then I could just enable "--binary" and "-M", and see what happens.
>
> Let's see what happens with 4.9-rc3. Will anybody notice? Did I do it right?
>
>              Linus
>

'git apply' works at least. I'll have to spend some time fixing up the
Fedora spec file to use git am instead of just patch.

Thanks,
Laura

Re: Linux-4.X-rcY patches can't be applied with git?

[Jarod Wilson]

On Sat, Oct 29, 2016 at 02:08:12PM -0700, Linus Torvalds wrote:
> On Mon, Oct 24, 2016 at 5:06 PM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > Then I could just enable "--binary" and "-M", and see what happens.
> 
> Let's see what happens with 4.9-rc3. Will anybody notice? Did I do it right?

Looks good to me, works perfectly for my use case. Thanks much!

-- 
Jarod Wilson
jarod@redhat.com

Re: Linux-4.X-rcY patches can't be applied with git?

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1664 bytes --]

Hi!

> > But in that case, what if your patch generation script used a filter to
> > exclude those binary files? No harm to that target audience, and it would
> > actually make them behave better for distro builds. Though that might be
> > counter to the goal of making them disappear entirely. :)
> 
> Heh, I'd rather people get the warning that "oops, something is
> incomplete". They can still work with the end result, but at least
> they got some indication that hey, that patch didn't work wonderfully
> well...
> 
> To be honest, I really would like to not do the tar-balls and patches at all.
> 
> But maybe rather than saying "it's only for legacy 'patch' users", I
> could just say that it's getting phased out, and say "you have to use
> 'git apply' to apply them".
> 
> Then I could just enable "--binary" and "-M", and see what happens.
> 
> I suspect that these days, git is so ubiquitous that it's ok.
> 
> And then in a few years, maybe I can just stop doing patches entirely,
> having proved the point that everybody already has git ;)

Well, having git is not quite the same thing as "having big enough
machine to run git on tree as big as kernel". Handheld zaurus was
powerful enough to compile kernel (in few hours) but git was pretty
painful to use there.

I'm now cross-compiling for n900, so problem is gone there, but there
is difference between downloading single release, and downloading full
history when you are on a slow line...

Best regards,

								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]