RE: git archive setting user and group

RE: git archive setting user and group

[Jason Pyeron]

> From: Jason Pyeron <jpyeron@pdinc.us>
> Sent: Friday, January 22, 2021 3:09 PM
> 
> I am about to make a release for logwatch tonight. Historically the files are owned by logwatch in the
> tgz file. When I use git archive it is owned by uid 0, is there an option to set the uid/uname,
> gid/gname owner of the files?

Answer: not at this time, as it is hard coded in the source.

archive-tar.c:
static void prepare_header(struct archiver_args *args,
                           struct ustar_header *header,
                           unsigned int mode, unsigned long size)
{
        xsnprintf(header->mode, sizeof(header->mode), "%07o", mode & 07777);
        xsnprintf(header->size, sizeof(header->size), "%011"PRIoMAX , S_ISREG(mode) ? (uintmax_t)size : (uintmax_t)0);
        xsnprintf(header->mtime, sizeof(header->mtime), "%011lo", (unsigned long) args->time);

        xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
        xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
        strlcpy(header->uname, "root", sizeof(header->uname));
        strlcpy(header->gname, "root", sizeof(header->gname));


meh.

Re: git archive setting user and group

[René Scharfe]

Am 22.01.21 um 21:40 schrieb Jason Pyeron:
>> From: Jason Pyeron <jpyeron@pdinc.us>
>> Sent: Friday, January 22, 2021 3:09 PM
>>
>> I am about to make a release for logwatch tonight. Historically the files are owned by logwatch in the
>> tgz file. When I use git archive it is owned by uid 0, is there an option to set the uid/uname,
>> gid/gname owner of the files?
>
> Answer: not at this time, as it is hard coded in the source.
>
> archive-tar.c:
> static void prepare_header(struct archiver_args *args,
>                            struct ustar_header *header,
>                            unsigned int mode, unsigned long size)
> {
>         xsnprintf(header->mode, sizeof(header->mode), "%07o", mode & 07777);
>         xsnprintf(header->size, sizeof(header->size), "%011"PRIoMAX , S_ISREG(mode) ? (uintmax_t)size : (uintmax_t)0);
>         xsnprintf(header->mtime, sizeof(header->mtime), "%011lo", (unsigned long) args->time);
>
>         xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
>         xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
>         strlcpy(header->uname, "root", sizeof(header->uname));
>         strlcpy(header->gname, "root", sizeof(header->gname));
>
>
> meh.
>

Adding support for using a custom user and group should be easy.  Is
this just a cosmetic thing?  Regular users would ignore the user info in
the archive, and root should not be used for extracting, and on systems
that don't have a logwatch user this wouldn't make a difference anyway,
right?

René

RE: git archive setting user and group

[Jason Pyeron]

> From: René Scharfe 
> Sent: Friday, January 22, 2021 4:00 PM
> 
> Am 22.01.21 um 21:40 schrieb Jason Pyeron:
> >> From: Jason Pyeron
> >> Sent: Friday, January 22, 2021 3:09 PM
> >>
> >> I am about to make a release for logwatch tonight. Historically the files are owned by logwatch in
> the
> >> tgz file. When I use git archive it is owned by uid 0, is there an option to set the uid/uname,
> >> gid/gname owner of the files?
> >
> > Answer: not at this time, as it is hard coded in the source.
> >
> > archive-tar.c:
> > static void prepare_header(struct archiver_args *args,
> >                            struct ustar_header *header,
> >                            unsigned int mode, unsigned long size)
> > {
> >         xsnprintf(header->mode, sizeof(header->mode), "%07o", mode & 07777);
> >         xsnprintf(header->size, sizeof(header->size), "%011"PRIoMAX , S_ISREG(mode) ?
> (uintmax_t)size : (uintmax_t)0);
> >         xsnprintf(header->mtime, sizeof(header->mtime), "%011lo", (unsigned long) args->time);
> >
> >         xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
> >         xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
> >         strlcpy(header->uname, "root", sizeof(header->uname));
> >         strlcpy(header->gname, "root", sizeof(header->gname));
> >
> >
> > meh.
> >
> 
> Adding support for using a custom user and group should be easy.  Is
> this just a cosmetic thing?  Regular users would ignore the user info in

In this case, likely a cosmetic thing. 

> the archive, and root should not be used for extracting, and on systems

But I can think of situations, where it would matter - and I do not agree that "root should not be used for extraction".

> that don't have a logwatch user this wouldn't make a difference anyway,
> right?

I updated the hard coded values to match what was needed, did a make and presto all was happy.

The backend should take:

--group=NAME
--group-id=GID
--owner=NAME
--owner-id=UID

Patch wanted?

Re: git archive setting user and group

[Konstantin Ryabitsev]

On Fri, Jan 22, 2021 at 10:00:04PM +0100, René Scharfe wrote:
> Adding support for using a custom user and group should be easy.  Is
> this just a cosmetic thing?  Regular users would ignore the user info in
> the archive, and root should not be used for extracting, and on systems
> that don't have a logwatch user this wouldn't make a difference anyway,
> right?

Right now, "git archive" operations are bit-for-bit identical across all
versions going back at least 8+ years. In fact, we've been relying on this to
support bundling tarball signatures with git tags themselves (via git notes).
E.g. you can see this in action here:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9

If you click on "(sig)", you will download a signature that can be used to
verify the tarball generated using "git archive".

I would argue that adding user/group support to "git archive" operation is
not really solving any problems other than "it's different from when I run it
as a regular user" -- and can introduce potential compatibility problems if
implemented.

So, I would selfishly vote not to implement this.

-K

RE: git archive setting user and group

[Jason Pyeron]

> From: Konstantin Ryabitsev
> Sent: Friday, January 22, 2021 4:40 PM
> Subject: Re: git archive setting user and group
> 
> On Fri, Jan 22, 2021 at 10:00:04PM +0100, René Scharfe wrote:
> > Adding support for using a custom user and group should be easy.  Is
> > this just a cosmetic thing?  Regular users would ignore the user info in
> > the archive, and root should not be used for extracting, and on systems
> > that don't have a logwatch user this wouldn't make a difference anyway,
> > right?
> 
> Right now, "git archive" operations are bit-for-bit identical across all
> versions going back at least 8+ years. In fact, we've been relying on this to
> support bundling tarball signatures with git tags themselves (via git notes).
> E.g. you can see this in action here:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9
> 
> If you click on "(sig)", you will download a signature that can be used to
> verify the tarball generated using "git archive".
> 
> I would argue that adding user/group support to "git archive" operation is
> not really solving any problems other than "it's different from when I run it
> as a regular user" -- and can introduce potential compatibility problems if

Being pedantic here, it is different than when I run it as any user - including root.

Don’t confuse tar x with tar c.

tar c captures the current owner of the files, or allows override with a single user / map file.

> implemented.
> 
> So, I would selfishly vote not to implement this.
> 
> -K

Re: git archive setting user and group

[Ævar Arnfjörð Bjarmason]

On Fri, Jan 22 2021, Konstantin Ryabitsev wrote:

> On Fri, Jan 22, 2021 at 10:00:04PM +0100, René Scharfe wrote:
>> Adding support for using a custom user and group should be easy.  Is
>> this just a cosmetic thing?  Regular users would ignore the user info in
>> the archive, and root should not be used for extracting, and on systems
>> that don't have a logwatch user this wouldn't make a difference anyway,
>> right?
>
> Right now, "git archive" operations are bit-for-bit identical across all
> versions going back at least 8+ years. In fact, we've been relying on this to
> support bundling tarball signatures with git tags themselves (via git notes).
> E.g. you can see this in action here:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9
>
> If you click on "(sig)", you will download a signature that can be used to
> verify the tarball generated using "git archive".
>
> I would argue that adding user/group support to "git archive" operation is
> not really solving any problems other than "it's different from when I run it
> as a regular user" -- and can introduce potential compatibility problems if
> implemented.
>
> So, I would selfishly vote not to implement this.

It seems "logwatch" has the same situation, except their backwards
compatibility is with non-"git archive" tool that used user != root.

If it solves a problem for some users and someone comes up with a patch
I don't see why it shouldn't be implemented, but I think it's important
that "root" is the default unless configured or the relevant option is
invoked.

Or do you mean that the kernel.org use-case is that users are expected
to run "git archive" on their own machines and upload the result to
kernel.org, and that kernel.org relies on two such files bit-for-bit
identical?

Re: git archive setting user and group

[Junio C Hamano]

René Scharfe <l.s.r@web.de> writes:

> Am 22.01.21 um 21:40 schrieb Jason Pyeron:
> ...
>>         xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
>>         xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
>>         strlcpy(header->uname, "root", sizeof(header->uname));
>>         strlcpy(header->gname, "root", sizeof(header->gname));
>
> Adding support for using a custom user and group should be easy.  Is
> this just a cosmetic thing?  Regular users would ignore the user info in
> the archive, and root should not be used for extracting, and on systems
> that don't have a logwatch user this wouldn't make a difference anyway,
> right?

I am not particularly interested in cosmetics, but it probably is OK
to make uname/gname overridable.  I do not see any point in uid/gid
numeric values overridable, though.  Just like user names and group
names do not name the same user and group on every machine, uid/gid
are even less so.

RE: git archive setting user and group

[Jason Pyeron]

> From: Junio C Hamano
> Sent: Friday, January 22, 2021 5:29 PM
> 
> René Scharfe <l.s.r@web.de> writes:
> 
> > Am 22.01.21 um 21:40 schrieb Jason Pyeron:
> > ...
> >>         xsnprintf(header->uid, sizeof(header->uid), "%07o", 0);
> >>         xsnprintf(header->gid, sizeof(header->gid), "%07o", 0);
> >>         strlcpy(header->uname, "root", sizeof(header->uname));
> >>         strlcpy(header->gname, "root", sizeof(header->gname));
> >
> > Adding support for using a custom user and group should be easy.  Is
> > this just a cosmetic thing?  Regular users would ignore the user info in
> > the archive, and root should not be used for extracting, and on systems
> > that don't have a logwatch user this wouldn't make a difference anyway,
> > right?
> 
> I am not particularly interested in cosmetics, but it probably is OK
> to make uname/gname overridable.  I do not see any point in uid/gid
> numeric values overridable, though.  Just like user names and group

Interesting. So logwatch(0) vs logwatch(1100) ? I guess it is as good as any value, but tar never uses 0.

On a system without the logwatch user/group tar behaves as follows:

$ tar czf /tmp/logwatch.tgz --owner=logwatch --group=logwatch .

$ tar tvzf /tmp/logwatch.tgz | head -n 1
drwxrwxr-x logwatch/logwatch 0 2020-03-14 15:01 ./

$ tar tvzf /tmp/logwatch.tgz --numeric-owner | head -n 1
drwxrwxr-x 1049681/1049088   0 2020-03-14 15:01 ./

$ id
uid=1049681(myuser) gid=1049088(mygroup) groups=no one cares...

but when the user / group is known...

$ tar czf /tmp/logwatch.tgz --owner=Guest --group=Users .

$ tar tvzf /tmp/logwatch.tgz | head -n 1
drwxrwxr-x Guest/Users       0 2020-03-14 15:01 ./

$ tar tvzf /tmp/logwatch.tgz --numeric-owner | head -n 1
drwxrwxr-x 1049077/545       0 2020-03-14 15:01 ./

$ getent passwd Guest | cut -f 3 -d :
1049077

$ getent group Users | cut -f 3 -d :
545


> names do not name the same user and group on every machine, uid/gid
> are even less so.
> 
> 

Re: git archive setting user and group

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]

On 2021-01-22 at 21:39:54, Konstantin Ryabitsev wrote:
> On Fri, Jan 22, 2021 at 10:00:04PM +0100, René Scharfe wrote:
> > Adding support for using a custom user and group should be easy.  Is
> > this just a cosmetic thing?  Regular users would ignore the user info in
> > the archive, and root should not be used for extracting, and on systems
> > that don't have a logwatch user this wouldn't make a difference anyway,
> > right?
> 
> Right now, "git archive" operations are bit-for-bit identical across all
> versions going back at least 8+ years. In fact, we've been relying on this to
> support bundling tarball signatures with git tags themselves (via git notes).
> E.g. you can see this in action here:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9
> 
> If you click on "(sig)", you will download a signature that can be used to
> verify the tarball generated using "git archive".

Please do not rely on this behavior.  I want to state in the strongest
possible terms that this is not guaranteed behavior and it may change at
any time.  We have explicitly said so on the list multiple times.  If
you need reproducible archives, you need to add a tool to canonicalize
them in a suitable format and not rely on Git to never change things.

If you are relying on this behavior right now, I urge you to change that
at your earliest possible convenience.  I don't want to break
kernel.org's infrastructure again, but I'm also not going to tiptoe
around sending patches in fear of that, nor feel bad if it happens again
for this reason.

> I would argue that adding user/group support to "git archive" operation is
> not really solving any problems other than "it's different from when I run it
> as a regular user" -- and can introduce potential compatibility problems if
> implemented.

I agree that this feature isn't really something we want.  Git produces
tar archives for software interchange, in which case producing an
intentionally anonymous tarball is the desired behavior.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: git archive setting user and group

[Jeff King]

On Sat, Jan 23, 2021 at 01:05:00AM +0000, brian m. carlson wrote:

> > Right now, "git archive" operations are bit-for-bit identical across all
> > versions going back at least 8+ years. In fact, we've been relying on this to
> > support bundling tarball signatures with git tags themselves (via git notes).
> > E.g. you can see this in action here:
> > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9
> > 
> > If you click on "(sig)", you will download a signature that can be used to
> > verify the tarball generated using "git archive".
> 
> Please do not rely on this behavior.  I want to state in the strongest
> possible terms that this is not guaranteed behavior and it may change at
> any time.  We have explicitly said so on the list multiple times.  If
> you need reproducible archives, you need to add a tool to canonicalize
> them in a suitable format and not rely on Git to never change things.

I strongly second this. :)

It's also not quite true that things have remained bit-for-bit identical
for all that time. We have fixed bugs in that time, although they do not
always cause a change in every output tarball (they often depend on
corner cases like having long pathnames). Two off the top of my head
(that have indeed caused people to complain about changing checksums):

  - 22f0dcd963 (archive-tar: split long paths more carefully,
    2013-01-05)

  - 82a46af13e (archive-tar: fix pax extended header length calculation,
    2019-08-17)

We also rely on system gzip. That's pretty stable, but I have heard tell
that even `gzip -n` may differ on platforms.

Another fun one I saw recently: using export-subst with $Format:%h$ will
produce different results depending on how many objects are present in
the repository running git-archive.

-Peff

Re: git archive setting user and group

[Konstantin Ryabitsev]

On Sat, Jan 23, 2021 at 01:05:00AM +0000, brian m. carlson wrote:
> > Right now, "git archive" operations are bit-for-bit identical across all
> > versions going back at least 8+ years. In fact, we've been relying on this to
> > support bundling tarball signatures with git tags themselves (via git notes).
> > E.g. you can see this in action here:
> > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tag/?h=v5.10.9
> > 
> > If you click on "(sig)", you will download a signature that can be used to
> > verify the tarball generated using "git archive".
> 
> Please do not rely on this behavior.  I want to state in the strongest
> possible terms that this is not guaranteed behavior and it may change at
> any time.  We have explicitly said so on the list multiple times.  If
> you need reproducible archives, you need to add a tool to canonicalize
> them in a suitable format and not rely on Git to never change things.

It doesn't need to be perpetually the same. This is the reason the comments
mention git version -- "to generate an archive that matches this signature,
run this command (with this git version)." I know there is no guarantee that
this won't ever change across versions, but I don't see why we can't expect
git-archive to return the same bit-for-bit output when using the same version
of git. Adding --owner and --group would potentially change that, especially
if that's something settable via git-config.

> If you are relying on this behavior right now, I urge you to change that
> at your earliest possible convenience.  I don't want to break
> kernel.org's infrastructure again, but I'm also not going to tiptoe
> around sending patches in fear of that, nor feel bad if it happens again
> for this reason.

It won't break, it will just inconvenience folks who are using this perk --
Greg KH will just have to use the same git version that we have on the server
when he generates the signatures. This has happened before.

-K

Re: git archive setting user and group

[Konstantin Ryabitsev]

On Fri, Jan 22, 2021 at 11:58:19PM -0500, Jeff King wrote:
> We also rely on system gzip. That's pretty stable, but I have heard tell
> that even `gzip -n` may differ on platforms.

The signatures are made against uncompressed .tar output, so this is not a
consideration.

> Another fun one I saw recently: using export-subst with $Format:%h$ will
> produce different results depending on how many objects are present in
> the repository running git-archive.

As long as the output is the same with thhe flags we specified in the
comment, we're still okay. E.g.:

    -----BEGIN PGP SIGNATURE-----
    Comment: This signature is for the .tar version of the archive
    Comment: git archive --format tar --prefix=linux-5.10.9/ v5.10.9
    Comment: git version 2.30.0

If running "git archive --format tar --prefix=linux-5.10.9/ v5.10.9" becomes
non-deterministic within the same version of git, *then* I'm in trouble. It's
been remarkably stable within the past 8 years, so I don't expect there's a
dramatic reason why "--format tar" output would need to change -- it's not
like the tar spec is much of a moving target.

-K

git archive setting user and group

[Jason Pyeron]

I am about to make a release for logwatch tonight. Historically the files are owned by logwatch in the tgz file. When I use git archive it is owned by uid 0, is there an option to set the uid/uname, gid/gname owner of the files?

--
Jason Pyeron  | Architect
PD Inc        |
10 w 24th St  |
Baltimore, MD |
 
.mil: jason.j.pyeron.ctr@mail.mil
.com: jpyeron@pdinc.us
tel : 202-741-9397