Reading/changing projid of a symlink

Reading/changing projid of a symlink

[Ilya Pronin]

Hello,

I am trying to recursively check/change projid on a directory that
contains symlinks among other files. The way I do it is like this
(attributes reading part as an example, modulo error handling):

oflags = O_NOFOLLOW | O_RDONLY;
if (S_ISDIR(stat->st_mode))
        oflags |= O_DIRECTORY;
if (S_ISLNK(stat->st_mode))
        oflags |= O_PATH;
fd = open(path, oflags, 0);
xfsctl(NULL, fd, XFS_IOC_FSGETXATTR, &attr);

The xfsctl() call fails with EBADF. Apparently this is because
xfsctl() is implemented as ioctl(), at least on Linux, which doesn't
work with file descriptors obtained with O_PATH. Is there any way to
manipulate projid on symlinks?

Thanks!

-- 
Ilya Pronin

Re: Reading/changing projid of a symlink

[Carlos Maiolino]

On Wed, May 30, 2018 at 05:26:46PM -0700, Ilya Pronin wrote:
> Hello,
> 
> I am trying to recursively check/change projid on a directory that
> contains symlinks among other files. The way I do it is like this
> (attributes reading part as an example, modulo error handling):
> 
> oflags = O_NOFOLLOW | O_RDONLY;
> if (S_ISDIR(stat->st_mode))
>         oflags |= O_DIRECTORY;
> if (S_ISLNK(stat->st_mode))
>         oflags |= O_PATH;
> fd = open(path, oflags, 0);
> xfsctl(NULL, fd, XFS_IOC_FSGETXATTR, &attr);
> 
> The xfsctl() call fails with EBADF. Apparently this is because
> xfsctl() is implemented as ioctl(), at least on Linux, which doesn't
> work with file descriptors obtained with O_PATH. Is there any way to
> manipulate projid on symlinks?
> 

Not sure if somebody already replied you, I couldn't find anything in my mbox,
but you can't.

Out of curiosity, why would you do that though?


> Thanks!
> 
> -- 
> Ilya Pronin
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Carlos

Re: Reading/changing projid of a symlink

[Ilya Pronin]

So the only way to remove a symlink from a project is to remove it?

Here's my use case. In Apache Mesos we use XFS project quotas for disk
space isolation. Every container sandbox is allocated its own project
ID. When the container terminates its project ID is unset from the
sandbox and files inside it, and can be used for other containers.
Sandboxes of terminated containers stay on the host for some time for
debugging, etc. With such approach the inability to remove projid from
symlinks presents a small problem: when a project ID is reused for a
new container, any lingering symlinks that still have that project ID
will contribute to disk usage of the new container. Typically not
much, but still that's inaccurate. I guess we'll need to change when
we "deallocate" projids.

On Tue, Jun 12, 2018 at 7:36 AM, Carlos Maiolino <cmaiolino@redhat.com> wrote:
> On Wed, May 30, 2018 at 05:26:46PM -0700, Ilya Pronin wrote:
>> Hello,
>>
>> I am trying to recursively check/change projid on a directory that
>> contains symlinks among other files. The way I do it is like this
>> (attributes reading part as an example, modulo error handling):
>>
>> oflags = O_NOFOLLOW | O_RDONLY;
>> if (S_ISDIR(stat->st_mode))
>>         oflags |= O_DIRECTORY;
>> if (S_ISLNK(stat->st_mode))
>>         oflags |= O_PATH;
>> fd = open(path, oflags, 0);
>> xfsctl(NULL, fd, XFS_IOC_FSGETXATTR, &attr);
>>
>> The xfsctl() call fails with EBADF. Apparently this is because
>> xfsctl() is implemented as ioctl(), at least on Linux, which doesn't
>> work with file descriptors obtained with O_PATH. Is there any way to
>> manipulate projid on symlinks?
>>
>
> Not sure if somebody already replied you, I couldn't find anything in my mbox,
> but you can't.
>
> Out of curiosity, why would you do that though?
>
>
>> Thanks!
>>
>> --
>> Ilya Pronin
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> Carlos



-- 
Ilya Pronin

Re: Reading/changing projid of a symlink

[Christoph Hellwig]

On Tue, Jun 12, 2018 at 03:17:27PM -0700, Ilya Pronin wrote:
> So the only way to remove a symlink from a project is to remove it?

Yes.

> Here's my use case. In Apache Mesos we use XFS project quotas for disk
> space isolation. Every container sandbox is allocated its own project
> ID. When the container terminates its project ID is unset from the
> sandbox and files inside it, and can be used for other containers.
> Sandboxes of terminated containers stay on the host for some time for
> debugging, etc. With such approach the inability to remove projid from
> symlinks presents a small problem: when a project ID is reused for a
> new container, any lingering symlinks that still have that project ID
> will contribute to disk usage of the new container. Typically not
> much, but still that's inaccurate. I guess we'll need to change when
> we "deallocate" projids.

There isn't really much we can do in XFS about it.  Traditionally in
all Unix variants you can never get a file descriptor for a symlink,
and all normal file operations always follow symlinks.  So you need
special new syscalls to handle the symlink.  Now Linux added the
O_PATH mount option a while ago which actually allows you to get
a file descriptor on symlinks, but just very limited ones that do
not actually support calls into the underlying file system.

So to support your use case we'd either need a new lioctl system
call, or some way to support limited ioctls on O_PATH fds.  Either
would be a VFS change and would turn into a huge discussion.  Once
that is done however wiring up the XFS bits would be trivial.

Re: Reading/changing projid of a symlink

[Carlos Maiolino]

On Tue, Jun 12, 2018 at 03:17:27PM -0700, Ilya Pronin wrote:
> So the only way to remove a symlink from a project is to remove it?
> 

Well, basically, yes.


> Here's my use case. In Apache Mesos we use XFS project quotas for disk
> space isolation. Every container sandbox is allocated its own project
> ID. When the container terminates its project ID is unset from the
> sandbox and files inside it, and can be used for other containers.
> Sandboxes of terminated containers stay on the host for some time for
> debugging, etc.

> With such approach the inability to remove projid from
> symlinks presents a small problem: when a project ID is reused for a
> new container, any lingering symlinks that still have that project ID
> will contribute to disk usage of the new container. Typically not
> much, but still that's inaccurate. I guess we'll need to change when
> we "deallocate" projids.

Hm, well, I see what you mean, but, as you said, xfs uses special xattrs to
manage inode quota information, handled by ioctl()s. Which, are not possible to
be issued in a symlink inode.

A few comments from my POV about it:

- Most of the space used in the project's quota by a symlink, is the space used
by the inode itself, unless you have really long links which don't fit into the
inode's literal area.
- Also, the space used by its directory entry in symlink's parent directory.

The symlink's project id, will always be related to the project id of its parent
directory.


At the end, Project quota, is a way to create a directory tree based quota,
where all inodes corresponding to a single project id, should be under the same
subtree. If you are creating a new subtree, under an already existing project
id, IMHO, the right approach would be to remove the project id from the old
directory tree, and add it to the new tree, and use user/group quotas for any
quota accounting that should be done filesystem-wide (instead of directory tree
wise). Still, this is just my opinion.

But at the end, you can't manually change symlink's project ids.

Cheers.

> 
> On Tue, Jun 12, 2018 at 7:36 AM, Carlos Maiolino <cmaiolino@redhat.com> wrote:
> > On Wed, May 30, 2018 at 05:26:46PM -0700, Ilya Pronin wrote:
> >> Hello,
> >>
> >> I am trying to recursively check/change projid on a directory that
> >> contains symlinks among other files. The way I do it is like this
> >> (attributes reading part as an example, modulo error handling):
> >>
> >> oflags = O_NOFOLLOW | O_RDONLY;
> >> if (S_ISDIR(stat->st_mode))
> >>         oflags |= O_DIRECTORY;
> >> if (S_ISLNK(stat->st_mode))
> >>         oflags |= O_PATH;
> >> fd = open(path, oflags, 0);
> >> xfsctl(NULL, fd, XFS_IOC_FSGETXATTR, &attr);
> >>
> >> The xfsctl() call fails with EBADF. Apparently this is because
> >> xfsctl() is implemented as ioctl(), at least on Linux, which doesn't
> >> work with file descriptors obtained with O_PATH. Is there any way to
> >> manipulate projid on symlinks?
> >>
> >
> > Not sure if somebody already replied you, I couldn't find anything in my mbox,
> > but you can't.
> >
> > Out of curiosity, why would you do that though?
> >
> >
> >> Thanks!
> >>
> >> --
> >> Ilya Pronin
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> > --
> > Carlos
> 
> 
> 
> -- 
> Ilya Pronin

-- 
Carlos

Re: Reading/changing projid of a symlink

[Eric Sandeen]

On 6/13/18 1:23 AM, Christoph Hellwig wrote:
> On Tue, Jun 12, 2018 at 03:17:27PM -0700, Ilya Pronin wrote:
>> So the only way to remove a symlink from a project is to remove it?
> 
> Yes.

Dumb question, should symlinks even /be/ in projects?

If it can't be manipulated from userspace, should it be inherited
in kernelspace (I assume that's how it got set?)

I ... guess it's needed for inode accounting, still?

-Eric

Re: Reading/changing projid of a symlink

[Dave Chinner]

On Tue, Jun 12, 2018 at 03:17:27PM -0700, Ilya Pronin wrote:
> So the only way to remove a symlink from a project is to remove it?
> 
> Here's my use case. In Apache Mesos we use XFS project quotas for disk
> space isolation. Every container sandbox is allocated its own project
> ID. When the container terminates its project ID is unset from the
> sandbox and files inside it, and can be used for other containers.
> Sandboxes of terminated containers stay on the host for some time for
> debugging, etc. With such approach the inability to remove projid from
> symlinks presents a small problem: when a project ID is reused for a
> new container, any lingering symlinks that still have that project ID
> will contribute to disk usage of the new container. Typically not
> much, but still that's inaccurate. I guess we'll need to change when
> we "deallocate" projids.

XFS has a 32bit project ID space, so you really don't need to reuse
project IDs at all. Just keep incrementing the new container ID, and
maybe after a couple of billion project IDs have been consumed you
can look at reclaiming/recycling/resetting the new ID counter?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Reading/changing projid of a symlink

[Ilya Pronin]

Thanks everyone for the answers!

On Wed, Jun 13, 2018 at 2:40 AM, Carlos Maiolino <cmaiolino@redhat.com> wrote:
> The symlink's project id, will always be related to the project id of its parent
> directory.
>
> At the end, Project quota, is a way to create a directory tree based quota,
> where all inodes corresponding to a single project id, should be under the same
> subtree. If you are creating a new subtree, under an already existing project
> id, IMHO, the right approach would be to remove the project id from the old
> directory tree, and add it to the new tree, and use user/group quotas for any
> quota accounting that should be done filesystem-wide (instead of directory tree
> wise).

It's a bit weird though that a symlink can inherit project ID from its
parent directory, but then cannot get rid of it :) And thus doesn't
allow to completely remove the project ID from the directory tree it's
in.

-- 
Ilya Pronin

Re: Reading/changing projid of a symlink

[Carlos Maiolino]

On Thu, Jun 14, 2018 at 05:14:34PM -0700, Ilya Pronin wrote:
> Thanks everyone for the answers!
> 
> On Wed, Jun 13, 2018 at 2:40 AM, Carlos Maiolino <cmaiolino@redhat.com> wrote:
> > The symlink's project id, will always be related to the project id of its parent
> > directory.
> >
> > At the end, Project quota, is a way to create a directory tree based quota,
> > where all inodes corresponding to a single project id, should be under the same
> > subtree. If you are creating a new subtree, under an already existing project
> > id, IMHO, the right approach would be to remove the project id from the old
> > directory tree, and add it to the new tree, and use user/group quotas for any
> > quota accounting that should be done filesystem-wide (instead of directory tree
> > wise).
> 
> It's a bit weird though that a symlink can inherit project ID from its
> parent directory,

Well, yes, a symlink occupy an inode, and so, we need to account for it, in case
for example, you limit inode creation in the project id.

> but then cannot get rid of it :) And thus doesn't
> allow to completely remove the project ID from the directory tree it's
> in.

Once you move the symlink out of the project subtree, the project id will be
removed or updated to the new project id (in case its new place belongs to
another project id).

As I said before, IMHO, I believe the way you are using project quotas is wrong.
Project quotas is supposed to be used on a sub-tree based granularity, not on a
file based granularity. But again, that's just my opinion.

> 
> -- 
> Ilya Pronin

-- 
Carlos

Re: Reading/changing projid of a symlink

[Christoph Hellwig]

On Fri, Jun 15, 2018 at 09:08:29AM +0200, Carlos Maiolino wrote:
> As I said before, IMHO, I believe the way you are using project quotas is wrong.
> Project quotas is supposed to be used on a sub-tree based granularity, not on a
> file based granularity. But again, that's just my opinion.

That is not true.  The per-file usage was indeed the original use
case.  The tree quoats implemented using the inheritance were added
much latter.  They are the common use case now, but that doesn't make
other use cases wrong.

Re: Reading/changing projid of a symlink

[Carlos Maiolino]

On Fri, Jun 15, 2018 at 12:12:36AM -0700, Christoph Hellwig wrote:
> On Fri, Jun 15, 2018 at 09:08:29AM +0200, Carlos Maiolino wrote:
> > As I said before, IMHO, I believe the way you are using project quotas is wrong.
> > Project quotas is supposed to be used on a sub-tree based granularity, not on a
> > file based granularity. But again, that's just my opinion.
> 
> That is not true.  The per-file usage was indeed the original use
> case.  The tree quoats implemented using the inheritance were added
> much latter.  They are the common use case now, but that doesn't make
> other use cases wrong.

Thanks for the info. I always thought project quotas were designed for
directory-tree granularity, and user/group quotas as a file-based granularity.
But, out of curiosity, if this is not a stupid question...
If project quotas was initially designed as a per-file usage use case in mind,
what would make it different from group quota for example?

> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Carlos

Re: Reading/changing projid of a symlink

[Christoph Hellwig]

On Fri, Jun 15, 2018 at 10:57:24AM +0200, Carlos Maiolino wrote:
> Thanks for the info. I always thought project quotas were designed for
> directory-tree granularity, and user/group quotas as a file-based granularity.
> But, out of curiosity, if this is not a stupid question...
> If project quotas was initially designed as a per-file usage use case in mind,
> what would make it different from group quota for example?

IRIX had projects integrated into the whole OS, that these did fit into.
Group quotas were only added to XFS after the Linux port.