About pseudo's chmod

About pseudo's chmod

[Robert Yang]

Hi,

When run "chmod 0444 <file>" under pseudo, it would always adds
write permission for real file (and w + x for dir), which means that
it runs as "chmod 0644 <file>". It does this on real file, not record
this on pseudo's database. Here are the code from pseudo:

/* Root can read and write files, and enter directories which have no
  * read, write, or execute permissions.  (But can't execute files without
  * execute permissions!)
  *
  * A non-root user can't.
  *
  * When doing anything which actually writes to the filesystem, we add in
  * the user read/write/execute bits.  When storing to the database, though,
  * we mask out any such bits which weren't in the original mode.
  */
#define PSEUDO_FS_MODE(mode, isdir) (((mode) | S_IRUSR | S_IWUSR | ((isdir) ? 
S_IXUSR : 0)) & ~(S_IWGRP | S_IWOTH))
#define PSEUDO_DB_MODE(fs_mode, user_mode) (((fs_mode) & ~0722) | ((user_mode & 
0722)))

It has a side effect for -dbg pkgs if the source files foo.c's mode is 0444:
1) bitbake foo
2) Edit rpm-native
3) bitbake foo

After the first bitake foo, we will see that foo.c in foo-dbg is 0444, but
after the second bitbake foo, foo.c in foo-dbg will be 0644, because the first
build has changed src file foo.c's mode to 0644, this is incorrect.

I have two suggestions on it:
1) Don't add more permissions when chmod(e.g., don't change 0444 -> 0644),
    The user can add it clearly if a file/dir really needs that.
2) This mainly affects do_package task AFAIK, the code is:
             if not cpath.islink(file):
                 os.link(file, fpath)
                 fstat = cpath.stat(file)
                 os.chmod(fpath, fstat.st_mode)
                 os.chown(fpath, fstat.st_uid, fstat.st_gid)

    Another solution is checking mode before run chmod, if we really need
    run chmod, then copy the file rather than link.

Any suggestion is appreciated.

The following recipes in oe-core have this issue:
blktool
coreutils
e2fsprogs
gnutls
guile
gzip
less
lsof
mtools
opensp
parted
screen
tcp-wrappers

-- 
Thanks

Robert

Re: About pseudo's chmod

[Mark Hatle]

On 7/5/16 5:23 AM, Robert Yang wrote:
> Hi,
> 
> When run "chmod 0444 <file>" under pseudo, it would always adds
> write permission for real file (and w + x for dir), which means that
> it runs as "chmod 0644 <file>". It does this on real file, not record
> this on pseudo's database. Here are the code from pseudo:
> 
> /* Root can read and write files, and enter directories which have no
>   * read, write, or execute permissions.  (But can't execute files without
>   * execute permissions!)
>   *
>   * A non-root user can't.
>   *
>   * When doing anything which actually writes to the filesystem, we add in
>   * the user read/write/execute bits.  When storing to the database, though,
>   * we mask out any such bits which weren't in the original mode.
>   */
> #define PSEUDO_FS_MODE(mode, isdir) (((mode) | S_IRUSR | S_IWUSR | ((isdir) ? 
> S_IXUSR : 0)) & ~(S_IWGRP | S_IWOTH))
> #define PSEUDO_DB_MODE(fs_mode, user_mode) (((fs_mode) & ~0722) | ((user_mode & 
> 0722)))
> 
> It has a side effect for -dbg pkgs if the source files foo.c's mode is 0444:
> 1) bitbake foo
> 2) Edit rpm-native
> 3) bitbake foo
> 
> After the first bitake foo, we will see that foo.c in foo-dbg is 0444, but
> after the second bitbake foo, foo.c in foo-dbg will be 0644, because the first
> build has changed src file foo.c's mode to 0644, this is incorrect.
> 
> I have two suggestions on it:
> 1) Don't add more permissions when chmod(e.g., don't change 0444 -> 0644),
>     The user can add it clearly if a file/dir really needs that.

As noted above, we have to adjust the permissions to writable, or we can not
make various changes later.  When working as the 'root' user, permissions are
basically ignored.  So a non-writable file is still writable.  The only way to
emulate this is to make the actual file writable.

I don't understand how/why on a second run the 0644 is showing up though.
Unless the pseudo database is wiped -- or the debug commands are not clearing
the split directories before writing into them, it should be creating new files
with the new pseudo database that follows the same semantics.

> 2) This mainly affects do_package task AFAIK, the code is:
>              if not cpath.islink(file):
>                  os.link(file, fpath)
>                  fstat = cpath.stat(file)
>                  os.chmod(fpath, fstat.st_mode)
>                  os.chown(fpath, fstat.st_uid, fstat.st_gid)
> 
>     Another solution is checking mode before run chmod, if we really need
>     run chmod, then copy the file rather than link.
> 
> Any suggestion is appreciated.
> 
> The following recipes in oe-core have this issue:
> blktool
> coreutils
> e2fsprogs
> gnutls
> guile
> gzip
> less
> lsof
> mtools
> opensp
> parted
> screen
> tcp-wrappers
> 

Re: About pseudo's chmod

[Robert Yang]

On 07/05/2016 09:10 PM, Mark Hatle wrote:
> On 7/5/16 5:23 AM, Robert Yang wrote:
>> Hi,
>>
>> When run "chmod 0444 <file>" under pseudo, it would always adds
>> write permission for real file (and w + x for dir), which means that
>> it runs as "chmod 0644 <file>". It does this on real file, not record
>> this on pseudo's database. Here are the code from pseudo:
>>
>> /* Root can read and write files, and enter directories which have no
>>    * read, write, or execute permissions.  (But can't execute files without
>>    * execute permissions!)
>>    *
>>    * A non-root user can't.
>>    *
>>    * When doing anything which actually writes to the filesystem, we add in
>>    * the user read/write/execute bits.  When storing to the database, though,
>>    * we mask out any such bits which weren't in the original mode.
>>    */
>> #define PSEUDO_FS_MODE(mode, isdir) (((mode) | S_IRUSR | S_IWUSR | ((isdir) ?
>> S_IXUSR : 0)) & ~(S_IWGRP | S_IWOTH))
>> #define PSEUDO_DB_MODE(fs_mode, user_mode) (((fs_mode) & ~0722) | ((user_mode &
>> 0722)))
>>
>> It has a side effect for -dbg pkgs if the source files foo.c's mode is 0444:
>> 1) bitbake foo
>> 2) Edit rpm-native
>> 3) bitbake foo
>>
>> After the first bitake foo, we will see that foo.c in foo-dbg is 0444, but
>> after the second bitbake foo, foo.c in foo-dbg will be 0644, because the first
>> build has changed src file foo.c's mode to 0644, this is incorrect.
>>
>> I have two suggestions on it:
>> 1) Don't add more permissions when chmod(e.g., don't change 0444 -> 0644),
>>      The user can add it clearly if a file/dir really needs that.
>
> As noted above, we have to adjust the permissions to writable, or we can not
> make various changes later.  When working as the 'root' user, permissions are
> basically ignored.  So a non-writable file is still writable.  The only way to
> emulate this is to make the actual file writable.
>
> I don't understand how/why on a second run the 0644 is showing up though.
> Unless the pseudo database is wiped -- or the debug commands are not clearing
> the split directories before writing into them, it should be creating new files
> with the new pseudo database that follows the same semantics.
>

I had talked with Mark via IM, why it gets 0644 is because of the code.
fstat = cpath.stat(file) in package.bbclass, this command would get mode
from the real file rather than pseudo's database, and this is expected.
And in the second run, the file's mode had been changed from 0444 to 0644
in the first run, so it would get 0644.

The real problem might be that filesystem such as ext4 doesn't allow
different items of hardlinks have different permissions.

// Robert

>> 2) This mainly affects do_package task AFAIK, the code is:
>>               if not cpath.islink(file):
>>                   os.link(file, fpath)
>>                   fstat = cpath.stat(file)
>>                   os.chmod(fpath, fstat.st_mode)
>>                   os.chown(fpath, fstat.st_uid, fstat.st_gid)
>>
>>      Another solution is checking mode before run chmod, if we really need
>>      run chmod, then copy the file rather than link.
>>
>> Any suggestion is appreciated.
>>
>> The following recipes in oe-core have this issue:
>> blktool
>> coreutils
>> e2fsprogs
>> gnutls
>> guile
>> gzip
>> less
>> lsof
>> mtools
>> opensp
>> parted
>> screen
>> tcp-wrappers
>>
>
>

Re: About pseudo's chmod

[Robert Yang]

On 07/05/2016 09:10 PM, Mark Hatle wrote:
> On 7/5/16 5:23 AM, Robert Yang wrote:
>> Hi,
>>
>> When run "chmod 0444 <file>" under pseudo, it would always adds
>> write permission for real file (and w + x for dir), which means that
>> it runs as "chmod 0644 <file>". It does this on real file, not record
>> this on pseudo's database. Here are the code from pseudo:
>>
>> /* Root can read and write files, and enter directories which have no
>>    * read, write, or execute permissions.  (But can't execute files without
>>    * execute permissions!)
>>    *
>>    * A non-root user can't.
>>    *
>>    * When doing anything which actually writes to the filesystem, we add in
>>    * the user read/write/execute bits.  When storing to the database, though,
>>    * we mask out any such bits which weren't in the original mode.
>>    */
>> #define PSEUDO_FS_MODE(mode, isdir) (((mode) | S_IRUSR | S_IWUSR | ((isdir) ?
>> S_IXUSR : 0)) & ~(S_IWGRP | S_IWOTH))
>> #define PSEUDO_DB_MODE(fs_mode, user_mode) (((fs_mode) & ~0722) | ((user_mode &
>> 0722)))
>>
>> It has a side effect for -dbg pkgs if the source files foo.c's mode is 0444:
>> 1) bitbake foo
>> 2) Edit rpm-native
>> 3) bitbake foo
>>
>> After the first bitake foo, we will see that foo.c in foo-dbg is 0444, but
>> after the second bitbake foo, foo.c in foo-dbg will be 0644, because the first
>> build has changed src file foo.c's mode to 0644, this is incorrect.
>>
>> I have two suggestions on it:
>> 1) Don't add more permissions when chmod(e.g., don't change 0444 -> 0644),
>>      The user can add it clearly if a file/dir really needs that.
>
> As noted above, we have to adjust the permissions to writable, or we can not
> make various changes later.  When working as the 'root' user, permissions are
> basically ignored.  So a non-writable file is still writable.  The only way to
> emulate this is to make the actual file writable.
>
> I don't understand how/why on a second run the 0644 is showing up though.

Hi Mark,

It got 0644 but not 0444 in the second build was because pseudo's unlink
doesn't take core of hard links, for example:
$ umask 0022
$ touch file1
$ ln file1 file2
$ chmod 0777 file1
$ ls -l file1 file2
We can see that both file1 and file2's mode is 0777.

But if we remove file1:
$ rm -f file1
$ ls file2
Now file2's mode is 0644 since the info had been removed from database.

After talked with RP online, there isn't any file systems that can support
different modes on different references for the same inode, so pseudo's
chmod should update all the references' mode, in another word, it should
not remove the info from database if links count is greater than 1.

Here is a patch for pseudo to fix the problem, I'm testing it locally now,
will send out sooner:

diff --git a/ports/unix/guts/unlinkat.c b/ports/unix/guts/unlinkat.c
index e723a01..a0ff685 100644
--- a/ports/unix/guts/unlinkat.c
+++ b/ports/unix/guts/unlinkat.c
@@ -36,8 +36,9 @@
     if (rc == -1) {
         return rc;
     }
+
     msg = pseudo_client_op(OP_MAY_UNLINK, 0, -1, dirfd, path, &buf);
-   if (msg && msg->result == RESULT_SUCCEED)
+   if (msg && msg->result == RESULT_SUCCEED && buf.st_nlink == 1)
         old_db_entry = 1;
  #ifdef PSEUDO_NO_REAL_AT_FUNCTIONS
     rc = real_unlink(path);
@@ -52,6 +53,8 @@
         } else {
             pseudo_client_op(OP_DID_UNLINK, 0, -1, -1, path, &buf);
         }
+   } else if (buf.st_nlink > 1) {
+       pseudo_debug(PDBGF_FILE, "not remove ino %lu from database since its 
links count is greater than 1.\n", buf.st_ino);
     } else {
         pseudo_debug(PDBGF_FILE, "unlink on <%s>, not in database, no 
effect.\n", path);
     }

// Robert

> Unless the pseudo database is wiped -- or the debug commands are not clearing
> the split directories before writing into them, it should be creating new files
> with the new pseudo database that follows the same semantics.
>
>> 2) This mainly affects do_package task AFAIK, the code is:
>>               if not cpath.islink(file):
>>                   os.link(file, fpath)
>>                   fstat = cpath.stat(file)
>>                   os.chmod(fpath, fstat.st_mode)
>>                   os.chown(fpath, fstat.st_uid, fstat.st_gid)
>>
>>      Another solution is checking mode before run chmod, if we really need
>>      run chmod, then copy the file rather than link.
>>
>> Any suggestion is appreciated.
>>
>> The following recipes in oe-core have this issue:
>> blktool
>> coreutils
>> e2fsprogs
>> gnutls
>> guile
>> gzip
>> less
>> lsof
>> mtools
>> opensp
>> parted
>> screen
>> tcp-wrappers
>>
>
>

Re: About pseudo's chmod

[Robert Yang]

On 07/29/2016 03:38 PM, Robert Yang wrote:
>
>
> On 07/05/2016 09:10 PM, Mark Hatle wrote:
>> On 7/5/16 5:23 AM, Robert Yang wrote:
>>> Hi,
>>>
>>> When run "chmod 0444 <file>" under pseudo, it would always adds
>>> write permission for real file (and w + x for dir), which means that
>>> it runs as "chmod 0644 <file>". It does this on real file, not record
>>> this on pseudo's database. Here are the code from pseudo:
>>>
>>> /* Root can read and write files, and enter directories which have no
>>>    * read, write, or execute permissions.  (But can't execute files without
>>>    * execute permissions!)
>>>    *
>>>    * A non-root user can't.
>>>    *
>>>    * When doing anything which actually writes to the filesystem, we add in
>>>    * the user read/write/execute bits.  When storing to the database, though,
>>>    * we mask out any such bits which weren't in the original mode.
>>>    */
>>> #define PSEUDO_FS_MODE(mode, isdir) (((mode) | S_IRUSR | S_IWUSR | ((isdir) ?
>>> S_IXUSR : 0)) & ~(S_IWGRP | S_IWOTH))
>>> #define PSEUDO_DB_MODE(fs_mode, user_mode) (((fs_mode) & ~0722) | ((user_mode &
>>> 0722)))
>>>
>>> It has a side effect for -dbg pkgs if the source files foo.c's mode is 0444:
>>> 1) bitbake foo
>>> 2) Edit rpm-native
>>> 3) bitbake foo
>>>
>>> After the first bitake foo, we will see that foo.c in foo-dbg is 0444, but
>>> after the second bitbake foo, foo.c in foo-dbg will be 0644, because the first
>>> build has changed src file foo.c's mode to 0644, this is incorrect.
>>>
>>> I have two suggestions on it:
>>> 1) Don't add more permissions when chmod(e.g., don't change 0444 -> 0644),
>>>      The user can add it clearly if a file/dir really needs that.
>>
>> As noted above, we have to adjust the permissions to writable, or we can not
>> make various changes later.  When working as the 'root' user, permissions are
>> basically ignored.  So a non-writable file is still writable.  The only way to
>> emulate this is to make the actual file writable.
>>
>> I don't understand how/why on a second run the 0644 is showing up though.
>
> Hi Mark,
>
> It got 0644 but not 0444 in the second build was because pseudo's unlink
> doesn't take core of hard links, for example:
> $ umask 0022
> $ touch file1
> $ ln file1 file2
> $ chmod 0777 file1
> $ ls -l file1 file2
> We can see that both file1 and file2's mode is 0777.
>
> But if we remove file1:
> $ rm -f file1
> $ ls file2
> Now file2's mode is 0644 since the info had been removed from database.
>
> After talked with RP online, there isn't any file systems that can support
> different modes on different references for the same inode, so pseudo's

Here should be: "there isn't any file systems can ... *as far as we know*"

// Robert

> chmod should update all the references' mode, in another word, it should
> not remove the info from database if links count is greater than 1.
>
> Here is a patch for pseudo to fix the problem, I'm testing it locally now,
> will send out sooner:
>
> diff --git a/ports/unix/guts/unlinkat.c b/ports/unix/guts/unlinkat.c
> index e723a01..a0ff685 100644
> --- a/ports/unix/guts/unlinkat.c
> +++ b/ports/unix/guts/unlinkat.c
> @@ -36,8 +36,9 @@
>      if (rc == -1) {
>          return rc;
>      }
> +
>      msg = pseudo_client_op(OP_MAY_UNLINK, 0, -1, dirfd, path, &buf);
> -   if (msg && msg->result == RESULT_SUCCEED)
> +   if (msg && msg->result == RESULT_SUCCEED && buf.st_nlink == 1)
>          old_db_entry = 1;
>   #ifdef PSEUDO_NO_REAL_AT_FUNCTIONS
>      rc = real_unlink(path);
> @@ -52,6 +53,8 @@
>          } else {
>              pseudo_client_op(OP_DID_UNLINK, 0, -1, -1, path, &buf);
>          }
> +   } else if (buf.st_nlink > 1) {
> +       pseudo_debug(PDBGF_FILE, "not remove ino %lu from database since its
> links count is greater than 1.\n", buf.st_ino);
>      } else {
>          pseudo_debug(PDBGF_FILE, "unlink on <%s>, not in database, no
> effect.\n", path);
>      }
>
> // Robert
>
>> Unless the pseudo database is wiped -- or the debug commands are not clearing
>> the split directories before writing into them, it should be creating new files
>> with the new pseudo database that follows the same semantics.
>>
>>> 2) This mainly affects do_package task AFAIK, the code is:
>>>               if not cpath.islink(file):
>>>                   os.link(file, fpath)
>>>                   fstat = cpath.stat(file)
>>>                   os.chmod(fpath, fstat.st_mode)
>>>                   os.chown(fpath, fstat.st_uid, fstat.st_gid)
>>>
>>>      Another solution is checking mode before run chmod, if we really need
>>>      run chmod, then copy the file rather than link.
>>>
>>> Any suggestion is appreciated.
>>>
>>> The following recipes in oe-core have this issue:
>>> blktool
>>> coreutils
>>> e2fsprogs
>>> gnutls
>>> guile
>>> gzip
>>> less
>>> lsof
>>> mtools
>>> opensp
>>> parted
>>> screen
>>> tcp-wrappers
>>>
>>
>>

Re: About pseudo's chmod

[Seebs]

On 29 Jul 2016, at 2:38, Robert Yang wrote:

> It got 0644 but not 0444 in the second build was because pseudo's 
> unlink
> doesn't take core of hard links, for example:
> $ umask 0022
> $ touch file1
> $ ln file1 file2
> $ chmod 0777 file1
> $ ls -l file1 file2
> We can see that both file1 and file2's mode is 0777.

> But if we remove file1:
> $ rm -f file1
> $ ls file2
> Now file2's mode is 0644 since the info had been removed from 
> database.

I don't get that result. Before the remove, I see:

sqlite> select * from files;
1|/home/seebs/pseudo/f1|2054|2757274|0|0|33279|0|0
2|/home/seebs/pseudo/f2|2054|2757274|0|0|33279|0|0

So both files have the correct information stored for them. This is
because chmod-type operations on files update everything with the same
device and inode.

> After talked with RP online, there isn't any file systems that can 
> support
> different modes on different references for the same inode, so 
> pseudo's
> chmod should update all the references' mode

Yes.

> in another word, it should
> not remove the info from database if links count is greater than 1.

This doesn't seem right to me. I can't produce any failures from the
current code that match the behavior you're describing; updates like
chmod always propogate to all the links.

On the other hand, if f1 *already exists*, and was not being tracked
by pseudo, then we do indeed see a similar problem. The underlying 
file's
mode is actually changed.

I do not think it would be a good idea at all to stop removing entries 
from the database when they become invalid. It seems to me that if we're 
hard linking to a file inside pseudo, we should probably be tracking 
that file. Although if we track the file, now we'll just continue to see 
it as mode 777, because we changed the mode of the file.

Consider, for a moment, what happens if you do this *without* pseudo 
involved. Unpack an archive containing a file "foo", mode 444.

$ ln foo bar
$ chmod 777 bar
$ rm bar
$ ls -l foo

You'll find that foo's mode is now 0777. So it'd change either way; the
difference is that, when pseudo's doing this, it masks out group and
other write bits on the filesystem. (Because we don't want other people 
to be able to overwrite things in the build tree just because they'd be 
writeable on the target filesystem.)

-s

Re: About pseudo's chmod

[Robert Yang]

Hi Peter,

Thanks for the reply.

On 07/30/2016 12:02 AM, Seebs wrote:
> On 29 Jul 2016, at 2:38, Robert Yang wrote:
>
>> It got 0644 but not 0444 in the second build was because pseudo's unlink
>> doesn't take core of hard links, for example:
>> $ umask 0022
>> $ touch file1
>> $ ln file1 file2
>> $ chmod 0777 file1
>> $ ls -l file1 file2
>> We can see that both file1 and file2's mode is 0777.
>
>> But if we remove file1:
>> $ rm -f file1
>> $ ls file2
>> Now file2's mode is 0644 since the info had been removed from database.
>
> I don't get that result. Before the remove, I see:
>
> sqlite> select * from files;
> 1|/home/seebs/pseudo/f1|2054|2757274|0|0|33279|0|0
> 2|/home/seebs/pseudo/f2|2054|2757274|0|0|33279|0|0
>
> So both files have the correct information stored for them. This is
> because chmod-type operations on files update everything with the same
> device and inode.

Sorry, the steps were wrong, they should be:
1) Out of pseudo:
    $ umask 0022
    $ touch file1
2) Under pseudo:
    $ ln file1 file2
    $ chmod 777 file2
    $ ls -l file1 file2
    We can see that both file1 and file2's mode is 0777.
    But if we remove file2:
    $ rm -f file2
    $ ls file1
    Now file1's permission is 0755, not 0777 or 0644, it should be 0777 here.

>
>> After talked with RP online, there isn't any file systems that can support
>> different modes on different references for the same inode, so pseudo's
>> chmod should update all the references' mode
>
> Yes.
>
>> in another word, it should
>> not remove the info from database if links count is greater than 1.
>
> This doesn't seem right to me. I can't produce any failures from the
> current code that match the behavior you're describing; updates like
> chmod always propogate to all the links.
>
> On the other hand, if f1 *already exists*, and was not being tracked
> by pseudo, then we do indeed see a similar problem. The underlying file's
> mode is actually changed.
>
> I do not think it would be a good idea at all to stop removing entries from the
> database when they become invalid. It seems to me that if we're hard linking to
> a file inside pseudo, we should probably be tracking that file. Although if we

Yes, I agree that we need track the src file when hard link.

// Robert

> track the file, now we'll just continue to see it as mode 777, because we
> changed the mode of the file.
>
> Consider, for a moment, what happens if you do this *without* pseudo involved.
> Unpack an archive containing a file "foo", mode 444.
>
> $ ln foo bar
> $ chmod 777 bar
> $ rm bar
> $ ls -l foo
>
> You'll find that foo's mode is now 0777. So it'd change either way; the
> difference is that, when pseudo's doing this, it masks out group and
> other write bits on the filesystem. (Because we don't want other people to be
> able to overwrite things in the build tree just because they'd be writeable on
> the target filesystem.)
>
> -s

Re: About pseudo's chmod

[Seebs]

On 1 Aug 2016, at 0:57, Robert Yang wrote:

> Sorry, the steps were wrong, they should be:
> 1) Out of pseudo:
>    $ umask 0022
>    $ touch file1
> 2) Under pseudo:
>    $ ln file1 file2
>    $ chmod 777 file2
>    $ ls -l file1 file2
>    We can see that both file1 and file2's mode is 0777.
>    But if we remove file2:
>    $ rm -f file2
>    $ ls file1
>    Now file1's permission is 0755, not 0777 or 0644, it should be 0777 
> here.

The short answer is: If you aren't tracking a file in pseudo, we don't 
make promises about its behavior. Normally, we don't touch them, but if 
there's hard links to them that are being touched, well. And having a 
hard link that is tracked, and another that isn't, is probably 
impossible to support. I definitely don't want to keep database entries 
for files that have been deleted, that way lies madness and possible 
database corruption; for instance, if we do that, and you make a new 
file of the same type, it'll show up as having those permissions, with 
only a path-mismatch warning in the database to suggest what went wrong.

I would say that the probable correct answer is either "make the 
original file be tracked" or "don't use hard links in this case".

Note that, under older pseudo, the file would have ended up 0777. The 
behavior of silently masking out 022 from underlying filesystem 
permissions is entirely intentional. During some debugging quite a while 
back, we discovered a quirk in oe-core, plus a strange behavior of
GNU tar, which between them resulted in some sstage directories getting
unpacked with mode 0777.

And we *really* don't want the build system generating files which other 
users can modify, especially not in stuff that's intended to go into a 
root filesystem. So stripping out those bits in the underlying 
filesystem is intentional.

If you were actually root, yes, the original file would have its mode 
changed to 0777. But we should never be *caring* about the mode bits on
raw files outside of pseudo, except that we want them to allow owner
write permissions and not allow group or other write permissions. If the
file's permissions matter to the build system or generated stuff, the
file should be tracked by pseudo.

-s

Re: About pseudo's chmod

[Robert Yang]

On 08/01/2016 04:42 PM, Seebs wrote:
> On 1 Aug 2016, at 0:57, Robert Yang wrote:
>
>> Sorry, the steps were wrong, they should be:
>> 1) Out of pseudo:
>>    $ umask 0022
>>    $ touch file1
>> 2) Under pseudo:
>>    $ ln file1 file2
>>    $ chmod 777 file2
>>    $ ls -l file1 file2
>>    We can see that both file1 and file2's mode is 0777.
>>    But if we remove file2:
>>    $ rm -f file2
>>    $ ls file1
>>    Now file1's permission is 0755, not 0777 or 0644, it should be 0777 here.
>
> The short answer is: If you aren't tracking a file in pseudo, we don't make
> promises about its behavior. Normally, we don't touch them, but if there's hard
> links to them that are being touched, well. And having a hard link that is
> tracked, and another that isn't, is probably impossible to support. I definitely
> don't want to keep database entries for files that have been deleted, that way
> lies madness and possible database corruption; for instance, if we do that, and
> you make a new file of the same type, it'll show up as having those permissions,
> with only a path-mismatch warning in the database to suggest what went wrong.
>
> I would say that the probable correct answer is either "make the original file
> be tracked" or "don't use hard links in this case".

Hi Peter,

How about we track the src when hardlink, for example:

$ ln oldpath newpath

Track both oldpath and newpath. The patch for pseudo is:

diff --git a/ports/unix/guts/linkat.c b/ports/unix/guts/linkat.c
index ec27e47..521b8fa 100644
--- a/ports/unix/guts/linkat.c
+++ b/ports/unix/guts/linkat.c
@@ -62,6 +62,10 @@
           * if the thing linked is a symlink.
           */
          pseudo_client_op(OP_LINK, 0, -1, -1, newpath, &buf);
+        /*
+         * Track oldpath in case it isn't tracked.
+         */
+        pseudo_client_op(OP_LINK, 0, -1, -1, oldpath, &buf);

          errno = save_errno;

// Robert

>
> Note that, under older pseudo, the file would have ended up 0777. The behavior
> of silently masking out 022 from underlying filesystem permissions is entirely
> intentional. During some debugging quite a while back, we discovered a quirk in
> oe-core, plus a strange behavior of
> GNU tar, which between them resulted in some sstage directories getting
> unpacked with mode 0777.
>
> And we *really* don't want the build system generating files which other users
> can modify, especially not in stuff that's intended to go into a root
> filesystem. So stripping out those bits in the underlying filesystem is
> intentional.
>
> If you were actually root, yes, the original file would have its mode changed to
> 0777. But we should never be *caring* about the mode bits on
> raw files outside of pseudo, except that we want them to allow owner
> write permissions and not allow group or other write permissions. If the
> file's permissions matter to the build system or generated stuff, the
> file should be tracked by pseudo.
>
> -s

Re: About pseudo's chmod

[Seebs]

On 1 Aug 2016, at 3:57, Robert Yang wrote:

> How about we track the src when hardlink, for example:

I don't think I like this.

Fundamentally, the problem here is making a hard link to a file outside 
of pseudo, and then modifying the file.

Consider, if you will, what happens if you replace "chmod 0777 file2"
with:
	# echo "# last line" >> file2

The problem here isn't pseudo's tracking; it's that we're making a hard 
link to a file, modifying the hard link, deleting the hard link, and 
expecting the original file to have the same attributes it had before
this all happened.

-s

Re: About pseudo's chmod

[Richard Purdie]

On Mon, 2016-08-01 at 13:17 -0500, Seebs wrote:
> On 1 Aug 2016, at 3:57, Robert Yang wrote:
> 
> > How about we track the src when hardlink, for example:
> 
> I don't think I like this.
> 
> Fundamentally, the problem here is making a hard link to a file
> outside 
> of pseudo, and then modifying the file.
> 
> Consider, if you will, what happens if you replace "chmod 0777 file2"
> with:
> 	# echo "# last line" >> file2
> 
> The problem here isn't pseudo's tracking; it's that we're making a
> hard 
> link to a file, modifying the hard link, deleting the hard link, and 
> expecting the original file to have the same attributes it had before
> this all happened.

No, we're actually expecting it to retain the mode it was given via the
hardlink under pseudo. 

This is what a real world system would do and in this case we could
track it via pseudo since pseudo is loaded when the hardlink is
created. It would be unreasonable for pseudo to track all hardlinks but
tracking ones created under it does seem reasonable?

Cheers,

Richard

Re: About pseudo's chmod

[Seebs]

On 1 Aug 2016, at 15:01, Richard Purdie wrote:

> No, we're actually expecting it to retain the mode it was given via 
> the
> hardlink under pseudo.
>
> This is what a real world system would do and in this case we could
> track it via pseudo since pseudo is loaded when the hardlink is
> created. It would be unreasonable for pseudo to track all hardlinks 
> but
> tracking ones created under it does seem reasonable?

Hmm. Well, strictly speaking, the link created under pseudo *does* get 
tracked. Hmm. But an implicit request to track also the thing linked to 
is not a horrible idea. Although you'd still be able to beat it:

	$ touch file1
	$ ln file1 file2
	$ pseudo
	# ln file2 file3
	# chmod 777 file3
	# rm file2 file3
	# ls -l file1

The general case of "find everything this link also refers to" is 
clearly out of scope. That said... Hmm. I think my main feeling is, if 
we want
to link to the file, and we want the changes to the linked file to
survive, we should probably either create that file under pseudo, or 
explicitly claim it with pseudo when we start wanting to do the 
tracking.
(You can trivially do this to a tree with chown -R root tree).

-s

Re: About pseudo's chmod

[Richard Purdie]

On Mon, 2016-08-01 at 15:17 -0500, Seebs wrote:
> On 1 Aug 2016, at 15:01, Richard Purdie wrote:
> 
> > No, we're actually expecting it to retain the mode it was given via
> > the
> > hardlink under pseudo.
> > 
> > This is what a real world system would do and in this case we could
> > track it via pseudo since pseudo is loaded when the hardlink is
> > created. It would be unreasonable for pseudo to track all hardlinks
> > but
> > tracking ones created under it does seem reasonable?
> 
> Hmm. Well, strictly speaking, the link created under pseudo *does*
> get 
> tracked. Hmm. But an implicit request to track also the thing linked
> to 
> is not a horrible idea. Although you'd still be able to beat it:
> 
> 	$ touch file1
> 	$ ln file1 file2
> 	$ pseudo
> 	# ln file2 file3
> 	# chmod 777 file3
> 	# rm file2 file3
> 	# ls -l file1
> 
> The general case of "find everything this link also refers to" is 
> clearly out of scope.

Agreed.

>  That said... Hmm. I think my main feeling is, if we want
> to link to the file, and we want the changes to the linked file to
> survive, we should probably either create that file under pseudo, or 
> explicitly claim it with pseudo when we start wanting to do the 
> tracking.
> (You can trivially do this to a tree with chown -R root tree).

The trouble is that for speed, we do create trees of hardlinked files
and play games with those and sstate amongst other things. Its
obviously faster to do this than make physical copies of the files.
Given what I know of the code paths, I suspect that tracking the source
of a hardlink would make life much easier for us. Obviously we can go
and start adding "chown -R" calls everywhere but that seems a little
ugly to me and doesn't do performance any favours.

Is there any significant downside if we do track the source of
hardlinks?

Cheers,

Richard

Re: About pseudo's chmod

[Mark Hatle]

On 8/1/16 5:55 PM, Richard Purdie wrote:
> On Mon, 2016-08-01 at 15:17 -0500, Seebs wrote:
>> On 1 Aug 2016, at 15:01, Richard Purdie wrote:
>>
>>> No, we're actually expecting it to retain the mode it was given via
>>> the
>>> hardlink under pseudo.
>>>
>>> This is what a real world system would do and in this case we could
>>> track it via pseudo since pseudo is loaded when the hardlink is
>>> created. It would be unreasonable for pseudo to track all hardlinks
>>> but
>>> tracking ones created under it does seem reasonable?
>>
>> Hmm. Well, strictly speaking, the link created under pseudo *does*
>> get 
>> tracked. Hmm. But an implicit request to track also the thing linked
>> to 
>> is not a horrible idea. Although you'd still be able to beat it:
>>
>> 	$ touch file1
>> 	$ ln file1 file2
>> 	$ pseudo
>> 	# ln file2 file3
>> 	# chmod 777 file3
>> 	# rm file2 file3
>> 	# ls -l file1
>>
>> The general case of "find everything this link also refers to" is 
>> clearly out of scope.
> 
> Agreed.
> 
>>  That said... Hmm. I think my main feeling is, if we want
>> to link to the file, and we want the changes to the linked file to
>> survive, we should probably either create that file under pseudo, or 
>> explicitly claim it with pseudo when we start wanting to do the 
>> tracking.
>> (You can trivially do this to a tree with chown -R root tree).
> 
> The trouble is that for speed, we do create trees of hardlinked files
> and play games with those and sstate amongst other things. Its
> obviously faster to do this than make physical copies of the files.
> Given what I know of the code paths, I suspect that tracking the source
> of a hardlink would make life much easier for us. Obviously we can go
> and start adding "chown -R" calls everywhere but that seems a little
> ugly to me and doesn't do performance any favours.
> 
> Is there any significant downside if we do track the source of
> hardlinks?

Would it makes sense to track the xattrs and linked files and such using some
type of inode reference (virtual or otherwise)?

Since in the case of a hard link, on a normal Linux style filesystem, there will
be a single inode that has a reference count higher then 1.  Thus you can know
the modes, xattrs, etc for that inode.. then the file points to the inode with
reference counts.  (this might require a rework on internal structures.. but
also might solve the problem.)

--Mark

> Cheers,
> 
> Richard
> 
> 
> 
> 

Re: About pseudo's chmod

[Seebs]

On 1 Aug 2016, at 18:36, Mark Hatle wrote:

> Would it makes sense to track the xattrs and linked files and such 
> using some
> type of inode reference (virtual or otherwise)?
>
> Since in the case of a hard link, on a normal Linux style filesystem, 
> there will
> be a single inode that has a reference count higher then 1.  Thus you 
> can know
> the modes, xattrs, etc for that inode.. then the file points to the 
> inode with
> reference counts.  (this might require a rework on internal 
> structures.. but
> also might solve the problem.)

Well, that *is* how we track xattrs. And everything else, we do use 
device and inode, but, we maintain one row for each path, and delete the 
individual path rows.

This gets back to one of the original design goals, which was to avoid
all the horrific things that happen to fakeroot because it's only 
tracking
device/inode. We *want* the multiple path entries so we can report 
apparent database problems.

(Side note: There's some performance optimizations that have reduced 
stability/robustness, and I have plans involving making pseudo smarter
about those, and/or allowing configuration to prefer being more 
cautious.)

-s

Re: About pseudo's chmod

[Robert Yang]

On 08/02/2016 06:55 AM, Richard Purdie wrote:
> On Mon, 2016-08-01 at 15:17 -0500, Seebs wrote:
>> On 1 Aug 2016, at 15:01, Richard Purdie wrote:
>>
>>> No, we're actually expecting it to retain the mode it was given via
>>> the
>>> hardlink under pseudo.
>>>
>>> This is what a real world system would do and in this case we could
>>> track it via pseudo since pseudo is loaded when the hardlink is
>>> created. It would be unreasonable for pseudo to track all hardlinks
>>> but
>>> tracking ones created under it does seem reasonable?
>>
>> Hmm. Well, strictly speaking, the link created under pseudo *does*
>> get
>> tracked. Hmm. But an implicit request to track also the thing linked
>> to
>> is not a horrible idea. Although you'd still be able to beat it:
>>
>> 	$ touch file1
>> 	$ ln file1 file2
>> 	$ pseudo
>> 	# ln file2 file3
>> 	# chmod 777 file3
>> 	# rm file2 file3
>> 	# ls -l file1
>>
>> The general case of "find everything this link also refers to" is
>> clearly out of scope.
>
> Agreed.
>
>>   That said... Hmm. I think my main feeling is, if we want
>> to link to the file, and we want the changes to the linked file to
>> survive, we should probably either create that file under pseudo, or
>> explicitly claim it with pseudo when we start wanting to do the
>> tracking.
>> (You can trivially do this to a tree with chown -R root tree).
>
> The trouble is that for speed, we do create trees of hardlinked files
> and play games with those and sstate amongst other things. Its
> obviously faster to do this than make physical copies of the files.
> Given what I know of the code paths, I suspect that tracking the source
> of a hardlink would make life much easier for us. Obviously we can go
> and start adding "chown -R" calls everywhere but that seems a little
> ugly to me and doesn't do performance any favours.
>
> Is there any significant downside if we do track the source of
> hardlinks?

AFAIK, no.

And when remove file2, but file1's permission is changed, it should
be considered as a bug.

// Robert

>
> Cheers,
>
> Richard
>
>
>
>

Re: About pseudo's chmod

[Seebs]

On 1 Aug 2016, at 20:52, Robert Yang wrote:

> And when remove file2, but file1's permission is changed, it should
> be considered as a bug.

I'm not sure of that. My interpretation would be that hard linking under 
pseudo to files which weren't created under the same pseudo database is
user error; that's not how the database is intended to work. That said, 
it's pretty trivial to add the things to it.

Although I'd like to know more about the use cases for these, because it
occurs to me that the qualifier "same pseudo database" points out 
another
possible failure mode: Would any of those files that are being linked to
be getting linked to from *more than one* pseudo database? Because if 
they were, that would be a thing I haven't been planning for and I don't 
know
whether it'd work sanely.

-s

Re: About pseudo's chmod

[Robert Yang]

On 08/02/2016 11:43 AM, Seebs wrote:
> On 1 Aug 2016, at 20:52, Robert Yang wrote:
>
>> And when remove file2, but file1's permission is changed, it should
>> be considered as a bug.
>
> I'm not sure of that. My interpretation would be that hard linking under pseudo
> to files which weren't created under the same pseudo database is
> user error; that's not how the database is intended to work. That said, it's
> pretty trivial to add the things to it.
>
> Although I'd like to know more about the use cases for these, because it
> occurs to me that the qualifier "same pseudo database" points out another

Currently, the problem in oe-core is:

       1) bitbake gzip
       2) Edit rpm-native or package.bbclass to make do_package re-run.
       3) bitbake gzip
       After the first build, build/version.c in gzip-dbg is 0444, but after
       the second build, it will be 0644, this is because do_package does:
       $ ln ${B}/version.c gzip-dbg/version.c,
       $ chmod 0444 gzip-dbg/version.c (it runs chmod 0644 on the real filesystem)
       And in the second build, the gzip-dbg/version.c will be removed and
       created again, so that stat() can't get 0444 but 0644 since
       ${B}/version.c is not tracked by pseudo.

// Robert

> possible failure mode: Would any of those files that are being linked to
> be getting linked to from *more than one* pseudo database? Because if they were,
> that would be a thing I haven't been planning for and I don't know
> whether it'd work sanely.
>
> -s

Re: About pseudo's chmod

[Robert Yang]

On 08/02/2016 02:07 PM, Robert Yang wrote:
>
>
> On 08/02/2016 11:43 AM, Seebs wrote:
>> On 1 Aug 2016, at 20:52, Robert Yang wrote:
>>
>>> And when remove file2, but file1's permission is changed, it should
>>> be considered as a bug.
>>
>> I'm not sure of that. My interpretation would be that hard linking under pseudo
>> to files which weren't created under the same pseudo database is
>> user error; that's not how the database is intended to work. That said, it's
>> pretty trivial to add the things to it.
>>
>> Although I'd like to know more about the use cases for these, because it
>> occurs to me that the qualifier "same pseudo database" points out another
>
> Currently, the problem in oe-core is:
>
>        1) bitbake gzip
>        2) Edit rpm-native or package.bbclass to make do_package re-run.
>        3) bitbake gzip
>        After the first build, build/version.c in gzip-dbg is 0444, but after
>        the second build, it will be 0644, this is because do_package does:
>        $ ln ${B}/version.c gzip-dbg/version.c,
>        $ chmod 0444 gzip-dbg/version.c (it runs chmod 0644 on the real filesystem)
>        And in the second build, the gzip-dbg/version.c will be removed and
>        created again, so that stat() can't get 0444 but 0644 since
>        ${B}/version.c is not tracked by pseudo.


And please see the first email in the thread for more details.

// Robert

>
> // Robert
>
>> possible failure mode: Would any of those files that are being linked to
>> be getting linked to from *more than one* pseudo database? Because if they were,
>> that would be a thing I haven't been planning for and I don't know
>> whether it'd work sanely.
>>
>> -s

Re: About pseudo's chmod

[Seebs]

On 2 Aug 2016, at 1:07, Robert Yang wrote:

> Currently, the problem in oe-core is:
>
>       1) bitbake gzip
>       2) Edit rpm-native or package.bbclass to make do_package re-run.
>       3) bitbake gzip
>       After the first build, build/version.c in gzip-dbg is 0444, but 
> after
>       the second build, it will be 0644, this is because do_package 
> does:
>       $ ln ${B}/version.c gzip-dbg/version.c,
>       $ chmod 0444 gzip-dbg/version.c (it runs chmod 0644 on the real 
> filesystem)
>       And in the second build, the gzip-dbg/version.c will be removed 
> and
>       created again, so that stat() can't get 0444 but 0644 since
>       ${B}/version.c is not tracked by pseudo.

Hmm. I'm a bit confused by this. Wouldn't the second build also do a
"chmod 0444" on gzip-dbg/version.c? Why is it doing that chmod the first
time and not the second? If it does it the second time, the fact that
the underlying file's mode changed won't matter.

But in this case... While I'm fine with modifying things to track the
file linked-to, it still feels like this is a usage error. 
Fundamentally,
we're unpacking a file (${B}/version.c), then linking to it, changing
the link in some way, deleting the link, and expecting the original file
to be unchanged. That doesn't seem right to me.

-s

Re: About pseudo's chmod

[Robert Yang]

On 08/02/2016 02:30 PM, Seebs wrote:
> On 2 Aug 2016, at 1:07, Robert Yang wrote:
>
>> Currently, the problem in oe-core is:
>>
>>       1) bitbake gzip
>>       2) Edit rpm-native or package.bbclass to make do_package re-run.
>>       3) bitbake gzip
>>       After the first build, build/version.c in gzip-dbg is 0444, but after
>>       the second build, it will be 0644, this is because do_package does:
>>       $ ln ${B}/version.c gzip-dbg/version.c,
>>       $ chmod 0444 gzip-dbg/version.c (it runs chmod 0644 on the real filesystem)
>>       And in the second build, the gzip-dbg/version.c will be removed and
>>       created again, so that stat() can't get 0444 but 0644 since
>>       ${B}/version.c is not tracked by pseudo.
>
> Hmm. I'm a bit confused by this. Wouldn't the second build also do a
> "chmod 0444" on gzip-dbg/version.c? Why is it doing that chmod the first

Because the stat() gets 0644 on ${B}/version.c in the second run, so it
would run chmod 0644 rather than 0444 on gzip-dbg/version.c. And why it
gets 0644 ? pseudo's chmod automatically adds "w" on the real file, so:
if -e gzip-dbg/version.c; then
	${B}/version.c = 0444
else
	${B}/version.c = 0644
fi

And in the second run, gzip-dbg/version.c is removed and will be recreated,
so that it got 0644.

> time and not the second? If it does it the second time, the fact that
> the underlying file's mode changed won't matter.
>
> But in this case... While I'm fine with modifying things to track the

Thanks, I will send a patch for it.

> file linked-to, it still feels like this is a usage error. Fundamentally,
> we're unpacking a file (${B}/version.c), then linking to it, changing
> the link in some way, deleting the link, and expecting the original file
> to be unchanged. That doesn't seem right to me.

But that is what the real filesystem works without pseudo:
$ touch file1
$ ln file1 file2
$ chmod 777 file2
$ rm -f file2

file1 will be 777 on the real filesystem.

// Robert

>
> -s

Re: About pseudo's chmod

[Seebs]

On 2 Aug 2016, at 1:44, Robert Yang wrote:

> Because the stat() gets 0644 on ${B}/version.c in the second run, so 
> it
> would run chmod 0644 rather than 0444 on gzip-dbg/version.c.

Why is it calling chmod at all? The goal is apparently to give
gzip-dbg/version.c the same mode that ${B}/version.c has. Since it's a
hard link, no chmod call is needed at all.

>> time and not the second? If it does it the second time, the fact that
>> the underlying file's mode changed won't matter.
>>
>> But in this case... While I'm fine with modifying things to track the
>
> Thanks, I will send a patch for it.

I already have a patch for this.

>> file linked-to, it still feels like this is a usage error. 
>> Fundamentally,
>> we're unpacking a file (${B}/version.c), then linking to it, changing
>> the link in some way, deleting the link, and expecting the original 
>> file
>> to be unchanged. That doesn't seem right to me.

> But that is what the real filesystem works without pseudo:
> $ touch file1
> $ ln file1 file2
> $ chmod 777 file2
> $ rm -f file2
>
> file1 will be 777 on the real filesystem.

Yes. But it seems that the mode the file is being changed to is 
dependant on the original reported mode. And that's the part that I'm 
confused by;
we're relying on the mode of the original file, but we're also changing
the mode of a hard link to it. This makes no sense to me.

So, I have a likely fix handy. (The difference between mine and the
version you proposed earlier is that, as I recall, yours does the LINK
operation on the original file unconditionally, which is in error; it
should only be done when no existing data was found in the database.) 
I'll double-check that again and go through looking for loose ends, 
because
I have a vague feeling that I'm overlooking a thing, but I haven't 
figured out what it would be yet.

-s

Re: About pseudo's chmod

[Robert Yang]

On 08/02/2016 02:50 PM, Seebs wrote:
> On 2 Aug 2016, at 1:44, Robert Yang wrote:
>
>> Because the stat() gets 0644 on ${B}/version.c in the second run, so it
>> would run chmod 0644 rather than 0444 on gzip-dbg/version.c.
>
> Why is it calling chmod at all? The goal is apparently to give
> gzip-dbg/version.c the same mode that ${B}/version.c has. Since it's a
> hard link, no chmod call is needed at all.

It is worth trying, I will try to remove os.chown() and os.chmod() from
package.bbclass to see what will happen:

             if not cpath.islink(file):
                 os.link(file, fpath)
                 fstat = cpath.stat(file)
                 os.chmod(fpath, fstat.st_mode)
                 os.chown(fpath, fstat.st_uid, fstat.st_gid)

>
>>> time and not the second? If it does it the second time, the fact that
>>> the underlying file's mode changed won't matter.
>>>
>>> But in this case... While I'm fine with modifying things to track the
>>
>> Thanks, I will send a patch for it.
>
> I already have a patch for this.
>
>>> file linked-to, it still feels like this is a usage error. Fundamentally,
>>> we're unpacking a file (${B}/version.c), then linking to it, changing
>>> the link in some way, deleting the link, and expecting the original file
>>> to be unchanged. That doesn't seem right to me.
>
>> But that is what the real filesystem works without pseudo:
>> $ touch file1
>> $ ln file1 file2
>> $ chmod 777 file2
>> $ rm -f file2
>>
>> file1 will be 777 on the real filesystem.
>
> Yes. But it seems that the mode the file is being changed to is dependant on the
> original reported mode. And that's the part that I'm confused by;
> we're relying on the mode of the original file, but we're also changing
> the mode of a hard link to it. This makes no sense to me.
>
> So, I have a likely fix handy. (The difference between mine and the
> version you proposed earlier is that, as I recall, yours does the LINK
> operation on the original file unconditionally, which is in error; it

I had checked pseudo's source code, I didn't find any function that can check
this, and I appreciated that if you can fix it:-).

// Robert

> should only be done when no existing data was found in the database.) I'll
> double-check that again and go through looking for loose ends, because
> I have a vague feeling that I'm overlooking a thing, but I haven't figured out
> what it would be yet.
>
> -s

Re: About pseudo's chmod

[Seebs]

On 2 Aug 2016, at 3:32, Robert Yang wrote:

> It is worth trying, I will try to remove os.chown() and os.chmod() from
> package.bbclass to see what will happen:
>
>             if not cpath.islink(file):
>                 os.link(file, fpath)
>                 fstat = cpath.stat(file)
>                 os.chmod(fpath, fstat.st_mode)
>                 os.chown(fpath, fstat.st_uid, fstat.st_gid)

... Okay, I admit, I can't figure this one out. Can anyone explain
this?

Does python's os.link work across devices maybe?

-s

Re: About pseudo's chmod

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 200 bytes --]

On 2 August 2016 at 20:16, Seebs <seebs@seebs.net> wrote:

> Does python's os.link work across devices maybe?
>

The documentation at least says hard link explicitly, so presumably not.

Ross

[-- Attachment #2: Type: text/html, Size: 594 bytes --]

Re: About pseudo's chmod

[Mark Hatle]

On 8/2/16 1:50 AM, Seebs wrote:
> On 2 Aug 2016, at 1:44, Robert Yang wrote:
> 
>> Because the stat() gets 0644 on ${B}/version.c in the second run, so 
>> it
>> would run chmod 0644 rather than 0444 on gzip-dbg/version.c.
> 
> Why is it calling chmod at all? The goal is apparently to give
> gzip-dbg/version.c the same mode that ${B}/version.c has. Since it's a
> hard link, no chmod call is needed at all.
> 
>>> time and not the second? If it does it the second time, the fact that
>>> the underlying file's mode changed won't matter.
>>>
>>> But in this case... While I'm fine with modifying things to track the
>>
>> Thanks, I will send a patch for it.
> 
> I already have a patch for this.
> 
>>> file linked-to, it still feels like this is a usage error. 
>>> Fundamentally,
>>> we're unpacking a file (${B}/version.c), then linking to it, changing
>>> the link in some way, deleting the link, and expecting the original 
>>> file
>>> to be unchanged. That doesn't seem right to me.
> 
>> But that is what the real filesystem works without pseudo:
>> $ touch file1
>> $ ln file1 file2
>> $ chmod 777 file2
>> $ rm -f file2
>>
>> file1 will be 777 on the real filesystem.
> 
> Yes. But it seems that the mode the file is being changed to is 
> dependant on the original reported mode. And that's the part that I'm 
> confused by;
> we're relying on the mode of the original file, but we're also changing
> the mode of a hard link to it. This makes no sense to me.
> 
> So, I have a likely fix handy. (The difference between mine and the
> version you proposed earlier is that, as I recall, yours does the LINK
> operation on the original file unconditionally, which is in error; it
> should only be done when no existing data was found in the database.) 
> I'll double-check that again and go through looking for loose ends, 
> because
> I have a vague feeling that I'm overlooking a thing, but I haven't 
> figured out what it would be yet.

A better way to describe the problem:

I create a file:

$ touch my-special-file
$ chmod 0444 my-special-file

I then run pseudo

$ pseudo ln my-special-file file1
$ stat file1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
$ pseudo rm -f file1
$ # remove pseudo database

At this point the original 'my-special-file' is now 0644, not 0444.  This is
because pseudo changes the perms on the file so that it could pretend to be root.

So if we repeat the last steps:

$ pseudo ln my-special-file file1
$ stat file1
Access: (0644/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
$ pseudo rm -f file1
$ # remove pseudo database

So the problem is -- pseudo is modifying the 'original' file perms, which on a
new instance of the pseudo database then gets inherited.  So we get
unpredictable results if this is the first time through -- or not the first time
through.

The problem is 'my-special-file' is never owned by pseudo -- but that isn't
really what the problem is -- the problem is because it's a hardlink -- and
pseudo is changing perms to emulate root -- it triggers a QA fault because the
file is 'changing'.

--Mark

> -s
> 

Re: About pseudo's chmod

[Seebs]

On 2 Aug 2016, at 10:12, Mark Hatle wrote:

> So the problem is -- pseudo is modifying the 'original' file perms, 
> which on a
> new instance of the pseudo database then gets inherited.  So we get
> unpredictable results if this is the first time through -- or not the 
> first time
> through.

Yeah. And the problem here is in part that we're calling chmod on a file 
linked to a file external to pseudo, but we're also checking that file's 
mode to find out what permissions we want it to have. Which is probably 
an error.

So, it does look like we need the tracking, but also if we stopped 
doing:

	stat file1 # obtain mode
	ln file1 file2
	chmod <mode> file2

we'd save significant time AND avoid the problem.

-s

Re: About pseudo's chmod

[Mark Hatle]

On 8/2/16 2:19 PM, Seebs wrote:
> On 2 Aug 2016, at 10:12, Mark Hatle wrote:
> 
>> So the problem is -- pseudo is modifying the 'original' file perms, 
>> which on a
>> new instance of the pseudo database then gets inherited.  So we get
>> unpredictable results if this is the first time through -- or not the 
>> first time
>> through.
> 
> Yeah. And the problem here is in part that we're calling chmod on a file 
> linked to a file external to pseudo, but we're also checking that file's 
> mode to find out what permissions we want it to have. Which is probably 
> an error.
> 
> So, it does look like we need the tracking, but also if we stopped 
> doing:
> 
> 	stat file1 # obtain mode
> 	ln file1 file2
> 	chmod <mode> file2
> 
> we'd save significant time AND avoid the problem.
> 
> -s
> 

The alternative to the 'ln' is a 'cp' operation.  This is how it used to work
until optimizations were added a few releases ago.  It was observed that this
saves a large amount of space -- but it had the unintended consequence of
suddenly files are now changes as the do_install/do_package processing
manipulates files and modes change...

I almost wonder if in pseudo we should avoid changing any modes internal to the
file -- until something tries to write to the file.  (Can we even capture that
and try to fix it?  If not, what we're doing is likely the only answer.)

--Mark

Re: About pseudo's chmod

[Seebs]

On 2 Aug 2016, at 14:39, Mark Hatle wrote:

> The alternative to the 'ln' is a 'cp' operation.  This is how it used 
> to work
> until optimizations were added a few releases ago.  It was observed 
> that this
> saves a large amount of space -- but it had the unintended consequence 
> of
> suddenly files are now changes as the do_install/do_package processing
> manipulates files and modes change...

Yeah, and I think if we just dropped the copy of permissions, this
would at least temporarily go away.

> I almost wonder if in pseudo we should avoid changing any modes 
> internal to the
> file -- until something tries to write to the file.  (Can we even 
> capture that
> and try to fix it?  If not, what we're doing is likely the only 
> answer.)

I don't think we usefully can.

Hmm.

So it occurs to me, there's only a few cases where we actually have any
reason to make on-filesystem permissions changes at all. Say, execute 
bits for executables. Otherwise, we could in theory just not bother 
writing
changes to the disk, because the disk permissions are irrelevant.

-s

Re: About pseudo's chmod

[Seebs]

On 1 Aug 2016, at 17:55, Richard Purdie wrote:

> The trouble is that for speed, we do create trees of hardlinked files
> and play games with those and sstate amongst other things.

Yeah. One concern would be ownership: If the things we're hardlinking to
weren't created/tracked by pseudo initially, they're going to show up as 
owned by the actual user doing the build. So hardlinks to them will 
also.

So it may be more reasonable to just create those things under pseudo
also, in which case they're already tracked.

> Its
> obviously faster to do this than make physical copies of the files.
> Given what I know of the code paths, I suspect that tracking the 
> source
> of a hardlink would make life much easier for us. Obviously we can go
> and start adding "chown -R" calls everywhere but that seems a little
> ugly to me and doesn't do performance any favours.

> Is there any significant downside if we do track the source of
> hardlinks?

Hmm. I'm not sure. It'll make things more complicated, but only 
marginally. The net impact would be fewer surprises in common cases, but 
the uncommon cases would get *really* weird. But it seems reasonable.
(At least for the case where there isn't an existing entry.) I'll
poke at this a bit, I think.

-s