[StGit PATCH] add option to import series directly from a tar archive

[StGit PATCH] add option to import series directly from a tar archive

[Clark Williams]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Catalin/Karl,

Attached is my first cut at adding the ability to import a patch series by specifying
the tarball. For example, the following command:

	$ stg import --tarfile patch-2.6.26.3-rt6.bz2

will apply the latest -rt patch series to your current kernel tree.

No Karl, I haven't developed a test for it (yet).  I wanted to see what you guys
thought first :)

Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEYEARECAAYFAkjDTscACgkQqA4JVb61b9dNRgCZAW+tOCgz5Y+A0IdomcOA4X7v
u8MAnRvFWMXRJ0Kxv1rAnBRnheq6Iidi
=W7Dl
-----END PGP SIGNATURE-----

[-- Attachment #2: tarfiles.patch --]
[-- Type: text/plain, Size: 2562 bytes --]

patch to allow importing a series from a tar archive

From: Clark Williams <williams@redhat.com>

Signed-off-by: Clark Williams <williams@redhat.com>
---
 stgit/commands/imprt.py |   46 +++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 45 insertions(+), 1 deletions(-)

diff --git a/stgit/commands/imprt.py b/stgit/commands/imprt.py
index 227743f..0e9bb73 100644
--- a/stgit/commands/imprt.py
+++ b/stgit/commands/imprt.py
@@ -87,7 +87,10 @@ options = [make_option('-m', '--mail',
            make_option('--commname',
                        help = 'use COMMNAME as the committer name'),
            make_option('--commemail',
-                       help = 'use COMMEMAIL as the committer e-mail')
+                       help = 'use COMMEMAIL as the committer e-mail'),
+           make_option('--tarfile',
+                       help = 'import a series from a tar archive',
+                       action = "store_true"),
            ] + make_sign_options()
 
 
@@ -287,6 +290,45 @@ def __import_url(url, options):
     urllib.urlretrieve(url, filename)
     __import_file(filename, options)
 
+def __import_tarfile(tar, options):
+    """Import patch series from a tar archive
+    """
+    import tarfile
+    import tempfile
+
+    if not tarfile.is_tarfile(tar):
+        raise CmdException, "%s is not a tarfile!" % tar
+
+
+    t = tarfile.open(tar, 'r')
+    names = t.getnames()
+
+    # verify paths in the tarfile are safe
+    for n in names:
+        if n.startswith('/'):
+            raise CmdException, "Absolute path found in %s" % tar
+        if n.startswith("../"):
+            raise CmdException, "Relative path found in %s" % tar
+
+    # find the series file
+    seriesfile = '';
+    for m in names:
+        if m.endswith('/series') or m == 'series':
+            seriesfile = m
+            break
+    if seriesfile == '':
+        raise CmdException, "no series file found in %s" % tar
+
+    # unpack into a tmp dir
+    tmpdir = tempfile.mkdtemp('.stg')
+    t.extractall(tmpdir)
+
+    # apply the series
+    __import_series(os.path.join(tmpdir, seriesfile), options)
+
+    # cleanup the tmpdir
+    os.system('rm -rf %s' % tmpdir)
+
 def func(parser, options, args):
     """Import a GNU diff file as a new patch
     """
@@ -308,6 +350,8 @@ def func(parser, options, args):
         __import_mbox(filename, options)
     elif options.url:
         __import_url(filename, options)
+    elif options.tarfile:
+        __import_tarfile(filename, options)
     else:
         __import_file(filename, options)
 

Re: [StGit PATCH] add option to import series directly from a tar archive

[Karl Hasselström]

On 2008-09-06 22:47:19 -0500, Clark Williams wrote:

> Attached is my first cut at adding the ability to import a patch
> series by specifying the tarball.

Thanks!

> No Karl, I haven't developed a test for it (yet). I wanted to see
> what you guys thought first :)

I don't see a problem with it, and if you took the time to code it
there is obviously at least one user (I have no idea how common patch
series tarballs are). I do have some comments below, but nothing that
would prevent you from writing a test or two right away. ;-)

> +           make_option('--tarfile',
> +                       help = 'import a series from a tar archive',
> +                       action = "store_true"),

As I hint below, you might want to autodetect tarballs with --series
instead, since a tarball is just a tarred series directory.

> +        if n.startswith("../"):
> +            raise CmdException, "Relative path found in %s" % tar

I guess any occurrence of /../ in the middle of n should be caught as
well? Or can't that happen?

By the way, is the separator always '/' in tarfile? Or should you use
os.sep? (There is also os.pardir which you could use instead of '..',
but that might be overdoing it a little ...)

> +        raise CmdException, "no series file found in %s" % tar

Perhaps "no 'series' file ...", to make it clear what the name should
be?

> +    # unpack into a tmp dir
> +    tmpdir = tempfile.mkdtemp('.stg')
> +    t.extractall(tmpdir)
> +
> +    # apply the series
> +    __import_series(os.path.join(tmpdir, seriesfile), options)

Hmm. It seems like such a waste to go via the file system here, when
tarfile has such nice file extraction methods.

What you could do is something like this:

  1. Make two small classes with the same interface, one backed by a
     tarfile and one backed by a directory, that have two methods:
     get_series() and get_file(filename). Both methods return
     file-like objects (created by either open() or
     tarfile.extractfile()).

  2. Change __import_series() to use objects of this class rather than
     a directory directly -- starting with creating an instance of one
     or the other depending on tarfile.is_tarfile(). This will involve
     teaching __import_file to accept a file-like object instead of
     just a file name, but that's a one-liner.

  3. Drop the --tarfile flag, since you've just taught the --series
     flag to handle tarballs!

That said, if you don't feel like doing it the hard way, I won't
insist. The way you coded it is in no way bad (in particular, you
chose the right function to create a temp dir).

> +    # cleanup the tmpdir
> +    os.system('rm -rf %s' % tmpdir)

Aaah! My eyes! My _eyes_!!!!!

Seriously, though, you'd want to use something like shutil.rmtree
here.

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

Re: [StGit PATCH] add option to import series directly from a tar archive

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Karl Hasselström wrote:
> On 2008-09-06 22:47:19 -0500, Clark Williams wrote:
> 
>> Attached is my first cut at adding the ability to import a patch
>> series by specifying the tarball.
> 
> Thanks!
> 
>> No Karl, I haven't developed a test for it (yet). I wanted to see
>> what you guys thought first :)
> 
> I don't see a problem with it, and if you took the time to code it
> there is obviously at least one user (I have no idea how common patch
> series tarballs are). I do have some comments below, but nothing that
> would prevent you from writing a test or two right away. ;-)
> 

Patch series tarballs are quite common from people who use quilt (e.g. many of the
kernel -rt series developers). My biggest problem (now that I can directly import
them) is to see if I can ease StGit's patch import rules a bit, since quilt accepts
pretty much anything as long as there's a diff in there somewhere. I bomb out
regularly importing the -rt series using StGit, because some people don't put
complete email addresses in their patches.

As to the test, I'll get right on that...:)

>> +           make_option('--tarfile',
>> +                       help = 'import a series from a tar archive',
>> +                       action = "store_true"),
> 
> As I hint below, you might want to autodetect tarballs with --series
> instead, since a tarball is just a tarred series directory.

Yeah I thought about that, as well as auto-detecting it in the file case. I'll look
into that a bit more.

> 
>> +        if n.startswith("../"):
>> +            raise CmdException, "Relative path found in %s" % tar
> 
> I guess any occurrence of /../ in the middle of n should be caught as
> well? Or can't that happen?
> 

Hence the "would you guys look at this?". Yeah, I need to detect sneaky stuff like
that too.


> By the way, is the separator always '/' in tarfile? Or should you use
> os.sep? (There is also os.pardir which you could use instead of '..',
> but that might be overdoing it a little ...)

I doubt there are many Windows-generated tarballs out there (except for the Cygwin
case; I believe they use '/'), but I shouldn't be so Unix-centric. I'll work on
cleaning it up.

I did consider adding Zipfile support as well, but didn't get a very good match-up
between tar functionality and zip functionality. Maybe later...

> 
>> +        raise CmdException, "no series file found in %s" % tar
> 
> Perhaps "no 'series' file ...", to make it clear what the name should
> be?
> 

Yeah, that makes sense.

>> +    # unpack into a tmp dir
>> +    tmpdir = tempfile.mkdtemp('.stg')
>> +    t.extractall(tmpdir)
>> +
>> +    # apply the series
>> +    __import_series(os.path.join(tmpdir, seriesfile), options)
> 
> Hmm. It seems like such a waste to go via the file system here, when
> tarfile has such nice file extraction methods.
> 
> What you could do is something like this:
> 
>   1. Make two small classes with the same interface, one backed by a
>      tarfile and one backed by a directory, that have two methods:
>      get_series() and get_file(filename). Both methods return
>      file-like objects (created by either open() or
>      tarfile.extractfile()).
> 
>   2. Change __import_series() to use objects of this class rather than
>      a directory directly -- starting with creating an instance of one
>      or the other depending on tarfile.is_tarfile(). This will involve
>      teaching __import_file to accept a file-like object instead of
>      just a file name, but that's a one-liner.
> 
>   3. Drop the --tarfile flag, since you've just taught the --series
>      flag to handle tarballs!
> 
> That said, if you don't feel like doing it the hard way, I won't
> insist. The way you coded it is in no way bad (in particular, you
> chose the right function to create a temp dir).

I did consider pulling directly from the tarball. I'll look into it.

> 
>> +    # cleanup the tmpdir
>> +    os.system('rm -rf %s' % tmpdir)
> 
> Aaah! My eyes! My _eyes_!!!!!
> 
> Seriously, though, you'd want to use something like shutil.rmtree
> here.
> 

Man, I could not for the life of me remember which module had that in it. To be fair
I wasn't up at work with my Python Essential Reference, which would have pointed me
directly at it, but I would have thought I could have gotten there through the Python
docs. Sigh...

You can dock my StGit pay for the visit to the eye doctor :)

Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEYEARECAAYFAkjFatgACgkQqA4JVb61b9fMRQCeLfK0zhPNEq3t5M4HW+vbRtaG
VhgAn0rtszqVLbd1bz12MS0b/3r0OkT2
=gkf1
-----END PGP SIGNATURE-----

Re: [StGit PATCH] add option to import series directly from a tar archive

[Karl Hasselström]

On 2008-09-08 13:11:37 -0500, Clark Williams wrote:

> Patch series tarballs are quite common from people who use quilt
> (e.g. many of the kernel -rt series developers).

Ah.

> My biggest problem (now that I can directly import them) is to see
> if I can ease StGit's patch import rules a bit, since quilt accepts
> pretty much anything as long as there's a diff in there somewhere. I
> bomb out regularly importing the -rt series using StGit, because
> some people don't put complete email addresses in their patches.

Yes, that would be a welcome addition.

> As to the test, I'll get right on that...:)

Swell!

> Karl Hasselström wrote:
>
> > By the way, is the separator always '/' in tarfile? Or should you
> > use os.sep? (There is also os.pardir which you could use instead
> > of '..', but that might be overdoing it a little ...)
>
> I doubt there are many Windows-generated tarballs out there (except
> for the Cygwin case; I believe they use '/'), but I shouldn't be so
> Unix-centric. I'll work on cleaning it up.

Well, it's no big deal, really. Just thought I'd mention it.

> I did consider adding Zipfile support as well, but didn't get a very
> good match-up between tar functionality and zip functionality. Maybe
> later...

I had a quick look at the zipfile module, and it looks like it too
could easily be wrapped in a small class like I suggested in point
(1).

> I did consider pulling directly from the tarball. I'll look into it.

Just don't let my suggestions take all the fun out of contributing ...
only do it my way if you really think it's better.

> > On 2008-09-06 22:47:19 -0500, Clark Williams wrote:
> >
> > > +    # cleanup the tmpdir
> > > +    os.system('rm -rf %s' % tmpdir)
> >
> > Aaah! My eyes! My _eyes_!!!!!
> >
> > Seriously, though, you'd want to use something like shutil.rmtree
> > here.
>
> Man, I could not for the life of me remember which module had that
> in it. To be fair I wasn't up at work with my Python Essential
> Reference, which would have pointed me directly at it, but I would
> have thought I could have gotten there through the Python docs.
> Sigh...
>
> You can dock my StGit pay for the visit to the eye doctor :)

:-)

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

Re: [StGit PATCH] add option to import series directly from a tar archive

[Samuel Tardieu]

>>>>> "Clark" == Clark Williams <clark.williams@gmail.com> writes:

Clark> [...] is to see if I can ease StGit's patch
Clark> import rules a bit, since quilt accepts pretty much anything as
Clark> long as there's a diff in there somewhere. I bomb out regularly
Clark> importing the -rt series using StGit, because some people don't
Clark> put complete email addresses in their patches.

Two things that would be great would be:

  - to be able to import patches with "-p0" (people not using git
    often sends such patches)

  - to be able to find where the patch should be applied; I sometimes
    receive patches for GCC directory "gcc/ada/", diffed from there,
    and if StGit could see that the patch only makes sense there and
    not at the top-level it would be great as well

 Sam
-- 
Samuel Tardieu -- sam@rfc1149.net -- http://www.rfc1149.net/

Re: [StGit PATCH] add option to import series directly from a tar archive

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Samuel Tardieu wrote:
>>>>>> "Clark" == Clark Williams <clark.williams@gmail.com> writes:
> 
> Clark> [...] is to see if I can ease StGit's patch
> Clark> import rules a bit, since quilt accepts pretty much anything as
> Clark> long as there's a diff in there somewhere. I bomb out regularly
> Clark> importing the -rt series using StGit, because some people don't
> Clark> put complete email addresses in their patches.
> 
> Two things that would be great would be:
> 
>   - to be able to import patches with "-p0" (people not using git
>     often sends such patches)

I'm not sure how easy this is going to be. It looks like the patch is applied with
'git --apply' from the file stgit/git.py:apply_patch(). The default '-p' value is 1,
so we'd have to figure out how to pass the 0 along and then get it into the
apply_patch() function.

> 
>   - to be able to find where the patch should be applied; I sometimes
>     receive patches for GCC directory "gcc/ada/", diffed from there,
>     and if StGit could see that the patch only makes sense there and
>     not at the top-level it would be great as well
> 

Zowie, I thought I only had to worry about folks sending patches with incomplete
information. So you get patches to the ada compiler that are rooted in gcc/ada (e.g.
patch in tarball says "./ChangeLog", instead of gcc/ada/ChangeLog) rather than at a
top level? Only way I could see to deal with that would be to try and pass in the
appropriate prefix from the command line.

My current plans are to clean up the first cut at the tarfile logic, then write a
test to keep Karl happy, then try to come up with a way to deal with importing
patches that don't have complete email addresses, no descriptions, etc. Once I get
through that, I'll see if we can deal with weirdly rooted patch series.

Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEYEARECAAYFAkjKZ0sACgkQqA4JVb61b9eg2ACffDv+FXsL1NifMvxr1tbO2c3s
Hc4AoJPb/RZJrpqT6QybeZrj8rNFJg1y
=ccj/
-----END PGP SIGNATURE-----

Re: [StGit PATCH] add option to import series directly from a tar archive

[Karl Hasselström]

On 2008-09-12 14:21:13 +0200, Samuel Tardieu wrote:

> Two things that would be great would be:
>
>   - to be able to import patches with "-p0" (people not using git
>     often sends such patches)

This should be trivial to implement, since git-apply (pardon the dash)
has a -p flag with precisely this meaning.

>   - to be able to find where the patch should be applied; I
>     sometimes receive patches for GCC directory "gcc/ada/", diffed
>     from there, and if StGit could see that the patch only makes
>     sense there and not at the top-level it would be great as well

I don't believe git-apply can do this (please correct me if I'm
wrong), and the right way to teach StGit to do it would arguably be to
teach it to git-apply and then make StGit use it. It'd be _possible_
to do it directly in StGit, but it wouldn't be quite the right level,
and git users wouldn't benefit.

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

Re: [StGit PATCH] add option to import series directly from a tar archive

[Karl Hasselström]

On 2008-09-12 15:07:03 +0200, Karl Hasselström wrote:

> On 2008-09-12 14:21:13 +0200, Samuel Tardieu wrote:
>
> >   - to be able to find where the patch should be applied; I
> >     sometimes receive patches for GCC directory "gcc/ada/", diffed
> >     from there, and if StGit could see that the patch only makes
> >     sense there and not at the top-level it would be great as well
>
> I don't believe git-apply can do this (please correct me if I'm
> wrong)

It does have a --directory flag, but that requires the user to specify
the path manually.

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

Re: [StGit PATCH] add option to import series directly from a tar archive

[Samuel Tardieu]

>>>>> "Clark" == Clark Williams <clark.williams@gmail.com> writes:

Clark> Zowie, I thought I only had to worry about folks sending
Clark> patches with incomplete information. So you get patches to the
Clark> ada compiler that are rooted in gcc/ada (e.g.  patch in tarball
Clark> says "./ChangeLog", instead of gcc/ada/ChangeLog) rather than
Clark> at a top level? Only way I could see to deal with that would be
Clark> to try and pass in the appropriate prefix from the command
Clark> line.

Yes, passing the prefix and strip levels would be fine.

  Sam
-- 
Samuel Tardieu -- sam@rfc1149.net -- http://www.rfc1149.net/

Re: [StGit PATCH] add option to import series directly from a tar archive

[Karl Hasselström]

On 2008-09-12 07:57:47 -0500, Clark Williams wrote:

> then write a test to keep Karl happy,

Technically, you write the test to make sure that your new feature
works as intended and won't break in the future. But since that's
rather a mouthful, I guess "Karl" will do as an acronym. ;-)

> then try to come up with a way to deal with importing patches that
> don't have complete email addresses, no descriptions, etc. Once I
> get through that, I'll see if we can deal with weirdly rooted patch
> series.

Nice.

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle