Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients.

Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients.

[Michael Wyraz]

Hi,

I do rsync based backups against a cephfs. If I do sync xattrs, rsync 
tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails 
with an error. xattrs does not get synchronized in this case.
IMO the problem is, that ceph's internal attributes are exposed to the 
mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the 
mounted cephfs and get results). So rsync sees these xattrs and tries to 
remove it.

Example output:

    rsync: rsync_xal_set:
    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.entries")
    failed: Operation not supported (95)
    rsync: rsync_xal_set:
    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.files")
    failed: Operation not supported (95)
    rsync: rsync_xal_set:
    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.subdirs")
    failed: Operation not supported (95)
    rsync: rsync_xal_set:
    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rentries")
    failed: Operation not supported (95)
    rsync: rsync_xal_set:
    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rfiles")
    failed: Operation not supported (95)
    rsync: rsync_xal_set:
    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rsubdirs")
    failed: Operation not supported (95)
    rsync: rsync_xal_set:
    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rbytes")
    failed: Operation not supported (95)
    rsync: rsync_xal_set:
    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rctime")
    failed: Operation not supported (95)

Kind regards,
Michael.

Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients.

[Sage Weil]

On Tue, 21 Jun 2016, Michael Wyraz wrote:
> Hi,
> 
> I do rsync based backups against a cephfs. If I do sync xattrs, rsync tries to
> delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an error.
> xattrs does not get synchronized in this case.
> IMO the problem is, that ceph's internal attributes are exposed to the mounted
> fileystem (I can do "getfattr -n ceph.dir.entries *" on the mounted cephfs and
> get results). So rsync sees these xattrs and tries to remove it.
> 
> Example output:
> 
>    rsync: rsync_xal_set:
>    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.entries")
>    failed: Operation not supported (95)
>    rsync: rsync_xal_set:
>    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.files")
>    failed: Operation not supported (95)
>    rsync: rsync_xal_set:
>    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.subdirs")
>    failed: Operation not supported (95)
>    rsync: rsync_xal_set:
>    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rentries")
>    failed: Operation not supported (95)
>    rsync: rsync_xal_set:
>    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rfiles")
>    failed: Operation not supported (95)
>    rsync: rsync_xal_set:
>    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rsubdirs")
>    failed: Operation not supported (95)
>    rsync: rsync_xal_set:
>    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rbytes")
>    failed: Operation not supported (95)
>    rsync: rsync_xal_set:
>    lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rctime")
>    failed: Operation not supported (95)

I think the right fix here is to patch rsync to ignore ceph.* xattrs.  
Does that seem reasonable?

Other possible workarounds might be:

 - ignore setxattr and removexattr requests on these xattrs (return 
success but do nothing)
 - never show these xattrs in listxattr.  this makes them less useful or 
friendly :(

sage

RE: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients.

[Allen Samuels]

I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand.

What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information?

There's nothing to prevent having a debug-ish toggle to expose them when you really want to see them.

Allen Samuels
SanDisk |a Western Digital brand
2880 Junction Avenue, San Jose, CA 95134
T: +1 408 801 7030| M: +1 408 780 6416
allen.samuels@SanDisk.com


> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Tuesday, June 21, 2016 2:30 PM
> To: Michael Wyraz <michael@wyraz.de>
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal
> xattrs are exposed to clients.
> 
> On Tue, 21 Jun 2016, Michael Wyraz wrote:
> > Hi,
> >
> > I do rsync based backups against a cephfs. If I do sync xattrs, rsync
> > tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an
> error.
> > xattrs does not get synchronized in this case.
> > IMO the problem is, that ceph's internal attributes are exposed to the
> > mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the
> > mounted cephfs and get results). So rsync sees these xattrs and tries to
> remove it.
> >
> > Example output:
> >
> >    rsync: rsync_xal_set:
> >    lremovexattr(""/backups/2016-05-31_00-09-
> 32/etc/acpi/events"","ceph.dir.entries")
> >    failed: Operation not supported (95)
> >    rsync: rsync_xal_set:
> >    lremovexattr(""/backups/2016-05-31_00-09-
> 32/etc/acpi/events"","ceph.dir.files")
> >    failed: Operation not supported (95)
> >    rsync: rsync_xal_set:
> >    lremovexattr(""/backups/2016-05-31_00-09-
> 32/etc/acpi/events"","ceph.dir.subdirs")
> >    failed: Operation not supported (95)
> >    rsync: rsync_xal_set:
> >    lremovexattr(""/backups/2016-05-31_00-09-
> 32/etc/acpi/events"","ceph.dir.rentries")
> >    failed: Operation not supported (95)
> >    rsync: rsync_xal_set:
> >    lremovexattr(""/backups/2016-05-31_00-09-
> 32/etc/acpi/events"","ceph.dir.rfiles")
> >    failed: Operation not supported (95)
> >    rsync: rsync_xal_set:
> >    lremovexattr(""/backups/2016-05-31_00-09-
> 32/etc/acpi/events"","ceph.dir.rsubdirs")
> >    failed: Operation not supported (95)
> >    rsync: rsync_xal_set:
> >    lremovexattr(""/backups/2016-05-31_00-09-
> 32/etc/acpi/events"","ceph.dir.rbytes")
> >    failed: Operation not supported (95)
> >    rsync: rsync_xal_set:
> >    lremovexattr(""/backups/2016-05-31_00-09-
> 32/etc/acpi/events"","ceph.dir.rctime")
> >    failed: Operation not supported (95)
> 
> I think the right fix here is to patch rsync to ignore ceph.* xattrs.
> Does that seem reasonable?
> 
> Other possible workarounds might be:
> 
>  - ignore setxattr and removexattr requests on these xattrs (return success
> but do nothing)
>  - never show these xattrs in listxattr.  this makes them less useful or friendly
> :(
> 
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients.

[Michael Wyraz]

That's a good point, Allen. Exposing of this special xattrs should be
enabled on demand, like it is already done with the "dirstat" mount option.
> I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand.
>
> What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information?
>
> There's nothing to prevent having a debug-ish toggle to expose them when you really want to see them.
>
> Allen Samuels
> SanDisk |a Western Digital brand
> 2880 Junction Avenue, San Jose, CA 95134
> T: +1 408 801 7030| M: +1 408 780 6416
> allen.samuels@SanDisk.com
>
>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>> owner@vger.kernel.org] On Behalf Of Sage Weil
>> Sent: Tuesday, June 21, 2016 2:30 PM
>> To: Michael Wyraz <michael@wyraz.de>
>> Cc: ceph-devel@vger.kernel.org
>> Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal
>> xattrs are exposed to clients.
>>
>> On Tue, 21 Jun 2016, Michael Wyraz wrote:
>>> Hi,
>>>
>>> I do rsync based backups against a cephfs. If I do sync xattrs, rsync
>>> tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an
>> error.
>>> xattrs does not get synchronized in this case.
>>> IMO the problem is, that ceph's internal attributes are exposed to the
>>> mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the
>>> mounted cephfs and get results). So rsync sees these xattrs and tries to
>> remove it.
>>> Example output:
>>>
>>>    rsync: rsync_xal_set:
>>>    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.entries")
>>>    failed: Operation not supported (95)
>>>    rsync: rsync_xal_set:
>>>    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.files")
>>>    failed: Operation not supported (95)
>>>    rsync: rsync_xal_set:
>>>    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.subdirs")
>>>    failed: Operation not supported (95)
>>>    rsync: rsync_xal_set:
>>>    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.rentries")
>>>    failed: Operation not supported (95)
>>>    rsync: rsync_xal_set:
>>>    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.rfiles")
>>>    failed: Operation not supported (95)
>>>    rsync: rsync_xal_set:
>>>    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.rsubdirs")
>>>    failed: Operation not supported (95)
>>>    rsync: rsync_xal_set:
>>>    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.rbytes")
>>>    failed: Operation not supported (95)
>>>    rsync: rsync_xal_set:
>>>    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.rctime")
>>>    failed: Operation not supported (95)
>> I think the right fix here is to patch rsync to ignore ceph.* xattrs.
>> Does that seem reasonable?
>>
>> Other possible workarounds might be:
>>
>>  - ignore setxattr and removexattr requests on these xattrs (return success
>> but do nothing)
>>  - never show these xattrs in listxattr.  this makes them less useful or friendly
>> :(
>>
>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
>> body of a message to majordomo@vger.kernel.org More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients.

[Gregory Farnum]

On Tue, Jun 21, 2016 at 3:30 PM, Allen Samuels
<Allen.Samuels@sandisk.com> wrote:
> I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand.
>
> What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information?

This is something I think we've talked about before, and I keep on
being surprised to learn we *do* list them without a specific query.
So, what are the upsides of listing them without being asked?

*) Easier to view for users
*) If syncing between Ceph clusters with the same pool names, you can
transfer layouts properly (except...probably xattrs are set after data
is written in most systems, so that doesn't work?)
*) ...I've got nothing else

Downsides:
*) Other tools get very confused by viewing xattrs they can't set

Personally, I'd rather only list them when the Ceph namespace is
specifically queried (or the xattr itself is, at whatever granularity
we can do). This isn't like directory sizes; xattrs have defined
semantics that we're breaking, *and* exposing them all the time just
doesn't seem that useful. Why would anybody care about the data
they're sharing, who doesn't already know that it's available?
-Greg

>
> There's nothing to prevent having a debug-ish toggle to expose them when you really want to see them.
>
> Allen Samuels
> SanDisk |a Western Digital brand
> 2880 Junction Avenue, San Jose, CA 95134
> T: +1 408 801 7030| M: +1 408 780 6416
> allen.samuels@SanDisk.com
>
>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>> owner@vger.kernel.org] On Behalf Of Sage Weil
>> Sent: Tuesday, June 21, 2016 2:30 PM
>> To: Michael Wyraz <michael@wyraz.de>
>> Cc: ceph-devel@vger.kernel.org
>> Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal
>> xattrs are exposed to clients.
>>
>> On Tue, 21 Jun 2016, Michael Wyraz wrote:
>> > Hi,
>> >
>> > I do rsync based backups against a cephfs. If I do sync xattrs, rsync
>> > tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an
>> error.
>> > xattrs does not get synchronized in this case.
>> > IMO the problem is, that ceph's internal attributes are exposed to the
>> > mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the
>> > mounted cephfs and get results). So rsync sees these xattrs and tries to
>> remove it.
>> >
>> > Example output:
>> >
>> >    rsync: rsync_xal_set:
>> >    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.entries")
>> >    failed: Operation not supported (95)
>> >    rsync: rsync_xal_set:
>> >    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.files")
>> >    failed: Operation not supported (95)
>> >    rsync: rsync_xal_set:
>> >    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.subdirs")
>> >    failed: Operation not supported (95)
>> >    rsync: rsync_xal_set:
>> >    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.rentries")
>> >    failed: Operation not supported (95)
>> >    rsync: rsync_xal_set:
>> >    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.rfiles")
>> >    failed: Operation not supported (95)
>> >    rsync: rsync_xal_set:
>> >    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.rsubdirs")
>> >    failed: Operation not supported (95)
>> >    rsync: rsync_xal_set:
>> >    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.rbytes")
>> >    failed: Operation not supported (95)
>> >    rsync: rsync_xal_set:
>> >    lremovexattr(""/backups/2016-05-31_00-09-
>> 32/etc/acpi/events"","ceph.dir.rctime")
>> >    failed: Operation not supported (95)
>>
>> I think the right fix here is to patch rsync to ignore ceph.* xattrs.
>> Does that seem reasonable?
>>
>> Other possible workarounds might be:
>>
>>  - ignore setxattr and removexattr requests on these xattrs (return success
>> but do nothing)
>>  - never show these xattrs in listxattr.  this makes them less useful or friendly
>> :(
>>
>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
>> body of a message to majordomo@vger.kernel.org More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients.

[Allen Samuels]

> -----Original Message-----
> From: Gregory Farnum [mailto:gfarnum@redhat.com]
> Sent: Wednesday, June 22, 2016 1:16 PM
> To: Allen Samuels <Allen.Samuels@sandisk.com>
> Cc: Sage Weil <sage@newdream.net>; Michael Wyraz <michael@wyraz.de>;
> ceph-devel@vger.kernel.org
> Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal
> xattrs are exposed to clients.
> 
> On Tue, Jun 21, 2016 at 3:30 PM, Allen Samuels
> <Allen.Samuels@sandisk.com> wrote:
> > I doubt that rsync is the only application that's going to be confused by the
> presence of attributes it doesn't understand.
> >
> > What is the value in having these attributes be visible to user-space
> applications? Is there a better mechanism for conveying this information?
> 
> This is something I think we've talked about before, and I keep on being
> surprised to learn we *do* list them without a specific query.
> So, what are the upsides of listing them without being asked?
> 
> *) Easier to view for users

If users == Ceph Cognoscenti, you're right. But if users != Ceph Cognoscenti, then this is actually a downside -- not really different from confusing the tools.

> *) If syncing between Ceph clusters with the same pool names, you can
> transfer layouts properly (except...probably xattrs are set after data is
> written in most systems, so that doesn't work?)

If it doesn't work, it's probably not a plus ;-)

> *) ...I've got nothing else
> 
> Downsides:
> *) Other tools get very confused by viewing xattrs they can't set
> 
> Personally, I'd rather only list them when the Ceph namespace is specifically
> queried (or the xattr itself is, at whatever granularity we can do). This isn't
> like directory sizes; xattrs have defined semantics that we're breaking, *and*
> exposing them all the time just doesn't seem that useful. Why would
> anybody care about the data they're sharing, who doesn't already know that
> it's available?
> -Greg
> 
> >
> > There's nothing to prevent having a debug-ish toggle to expose them when
> you really want to see them.
> >
> > Allen Samuels
> > SanDisk |a Western Digital brand
> > 2880 Junction Avenue, San Jose, CA 95134
> > T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@SanDisk.com
> >
> >
> >> -----Original Message-----
> >> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> >> owner@vger.kernel.org] On Behalf Of Sage Weil
> >> Sent: Tuesday, June 21, 2016 2:30 PM
> >> To: Michael Wyraz <michael@wyraz.de>
> >> Cc: ceph-devel@vger.kernel.org
> >> Subject: Re: Problems with cephfs, rsync and xattr because cephfs
> internal
> >> xattrs are exposed to clients.
> >>
> >> On Tue, 21 Jun 2016, Michael Wyraz wrote:
> >> > Hi,
> >> >
> >> > I do rsync based backups against a cephfs. If I do sync xattrs, rsync
> >> > tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails
> with an
> >> error.
> >> > xattrs does not get synchronized in this case.
> >> > IMO the problem is, that ceph's internal attributes are exposed to the
> >> > mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the
> >> > mounted cephfs and get results). So rsync sees these xattrs and tries to
> >> remove it.
> >> >
> >> > Example output:
> >> >
> >> >    rsync: rsync_xal_set:
> >> >    lremovexattr(""/backups/2016-05-31_00-09-
> >> 32/etc/acpi/events"","ceph.dir.entries")
> >> >    failed: Operation not supported (95)
> >> >    rsync: rsync_xal_set:
> >> >    lremovexattr(""/backups/2016-05-31_00-09-
> >> 32/etc/acpi/events"","ceph.dir.files")
> >> >    failed: Operation not supported (95)
> >> >    rsync: rsync_xal_set:
> >> >    lremovexattr(""/backups/2016-05-31_00-09-
> >> 32/etc/acpi/events"","ceph.dir.subdirs")
> >> >    failed: Operation not supported (95)
> >> >    rsync: rsync_xal_set:
> >> >    lremovexattr(""/backups/2016-05-31_00-09-
> >> 32/etc/acpi/events"","ceph.dir.rentries")
> >> >    failed: Operation not supported (95)
> >> >    rsync: rsync_xal_set:
> >> >    lremovexattr(""/backups/2016-05-31_00-09-
> >> 32/etc/acpi/events"","ceph.dir.rfiles")
> >> >    failed: Operation not supported (95)
> >> >    rsync: rsync_xal_set:
> >> >    lremovexattr(""/backups/2016-05-31_00-09-
> >> 32/etc/acpi/events"","ceph.dir.rsubdirs")
> >> >    failed: Operation not supported (95)
> >> >    rsync: rsync_xal_set:
> >> >    lremovexattr(""/backups/2016-05-31_00-09-
> >> 32/etc/acpi/events"","ceph.dir.rbytes")
> >> >    failed: Operation not supported (95)
> >> >    rsync: rsync_xal_set:
> >> >    lremovexattr(""/backups/2016-05-31_00-09-
> >> 32/etc/acpi/events"","ceph.dir.rctime")
> >> >    failed: Operation not supported (95)
> >>
> >> I think the right fix here is to patch rsync to ignore ceph.* xattrs.
> >> Does that seem reasonable?
> >>
> >> Other possible workarounds might be:
> >>
> >>  - ignore setxattr and removexattr requests on these xattrs (return
> success
> >> but do nothing)
> >>  - never show these xattrs in listxattr.  this makes them less useful or
> friendly
> >> :(
> >>
> >> sage
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the
> >> body of a message to majordomo@vger.kernel.org More majordomo info
> at
> >> http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients.

[John Spray]

On Wed, Jun 22, 2016 at 9:15 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> On Tue, Jun 21, 2016 at 3:30 PM, Allen Samuels
> <Allen.Samuels@sandisk.com> wrote:
>> I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand.
>>
>> What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information?
>
> This is something I think we've talked about before, and I keep on
> being surprised to learn we *do* list them without a specific query.
> So, what are the upsides of listing them without being asked?
>
> *) Easier to view for users
> *) If syncing between Ceph clusters with the same pool names, you can
> transfer layouts properly (except...probably xattrs are set after data
> is written in most systems, so that doesn't work?)
> *) ...I've got nothing else
>
> Downsides:
> *) Other tools get very confused by viewing xattrs they can't set
>
> Personally, I'd rather only list them when the Ceph namespace is
> specifically queried (or the xattr itself is, at whatever granularity
> we can do). This isn't like directory sizes; xattrs have defined
> semantics that we're breaking, *and* exposing them all the time just
> doesn't seem that useful. Why would anybody care about the data
> they're sharing, who doesn't already know that it's available?
> -Greg

I'm inclined to agree that we shouldn't include ceph.* xattrs in the
normal listing of xattrs.  Created tickets:
http://tracker.ceph.com/issues/16468
http://tracker.ceph.com/issues/16467

I do think it would be more correct for rsync to avoid xattrs in any
namespace it doesn't know about though; given that xattrs are meant to
be for potentially-filesystem-specific ones, it should assume that any
other namespaces are non-standard and are not necessarily safe to
copy.  Realistically though, rsync probably won't be the only tool
that ever tries to do this, so we should try to be safe against people
running any other tools that make the same assumption that listed
xattrs are copyable.

John

>>
>> There's nothing to prevent having a debug-ish toggle to expose them when you really want to see them.
>>
>> Allen Samuels
>> SanDisk |a Western Digital brand
>> 2880 Junction Avenue, San Jose, CA 95134
>> T: +1 408 801 7030| M: +1 408 780 6416
>> allen.samuels@SanDisk.com
>>
>>
>>> -----Original Message-----
>>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>>> owner@vger.kernel.org] On Behalf Of Sage Weil
>>> Sent: Tuesday, June 21, 2016 2:30 PM
>>> To: Michael Wyraz <michael@wyraz.de>
>>> Cc: ceph-devel@vger.kernel.org
>>> Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal
>>> xattrs are exposed to clients.
>>>
>>> On Tue, 21 Jun 2016, Michael Wyraz wrote:
>>> > Hi,
>>> >
>>> > I do rsync based backups against a cephfs. If I do sync xattrs, rsync
>>> > tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an
>>> error.
>>> > xattrs does not get synchronized in this case.
>>> > IMO the problem is, that ceph's internal attributes are exposed to the
>>> > mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the
>>> > mounted cephfs and get results). So rsync sees these xattrs and tries to
>>> remove it.
>>> >
>>> > Example output:
>>> >
>>> >    rsync: rsync_xal_set:
>>> >    lremovexattr(""/backups/2016-05-31_00-09-
>>> 32/etc/acpi/events"","ceph.dir.entries")
>>> >    failed: Operation not supported (95)
>>> >    rsync: rsync_xal_set:
>>> >    lremovexattr(""/backups/2016-05-31_00-09-
>>> 32/etc/acpi/events"","ceph.dir.files")
>>> >    failed: Operation not supported (95)
>>> >    rsync: rsync_xal_set:
>>> >    lremovexattr(""/backups/2016-05-31_00-09-
>>> 32/etc/acpi/events"","ceph.dir.subdirs")
>>> >    failed: Operation not supported (95)
>>> >    rsync: rsync_xal_set:
>>> >    lremovexattr(""/backups/2016-05-31_00-09-
>>> 32/etc/acpi/events"","ceph.dir.rentries")
>>> >    failed: Operation not supported (95)
>>> >    rsync: rsync_xal_set:
>>> >    lremovexattr(""/backups/2016-05-31_00-09-
>>> 32/etc/acpi/events"","ceph.dir.rfiles")
>>> >    failed: Operation not supported (95)
>>> >    rsync: rsync_xal_set:
>>> >    lremovexattr(""/backups/2016-05-31_00-09-
>>> 32/etc/acpi/events"","ceph.dir.rsubdirs")
>>> >    failed: Operation not supported (95)
>>> >    rsync: rsync_xal_set:
>>> >    lremovexattr(""/backups/2016-05-31_00-09-
>>> 32/etc/acpi/events"","ceph.dir.rbytes")
>>> >    failed: Operation not supported (95)
>>> >    rsync: rsync_xal_set:
>>> >    lremovexattr(""/backups/2016-05-31_00-09-
>>> 32/etc/acpi/events"","ceph.dir.rctime")
>>> >    failed: Operation not supported (95)
>>>
>>> I think the right fix here is to patch rsync to ignore ceph.* xattrs.
>>> Does that seem reasonable?
>>>
>>> Other possible workarounds might be:
>>>
>>>  - ignore setxattr and removexattr requests on these xattrs (return success
>>> but do nothing)
>>>  - never show these xattrs in listxattr.  this makes them less useful or friendly
>>> :(
>>>
>>> sage
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
>>> body of a message to majordomo@vger.kernel.org More majordomo info at
>>> http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients.

[Sage Weil]

On Fri, 24 Jun 2016, John Spray wrote:
> On Wed, Jun 22, 2016 at 9:15 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> > On Tue, Jun 21, 2016 at 3:30 PM, Allen Samuels
> > <Allen.Samuels@sandisk.com> wrote:
> >> I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand.
> >>
> >> What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information?
> >
> > This is something I think we've talked about before, and I keep on
> > being surprised to learn we *do* list them without a specific query.
> > So, what are the upsides of listing them without being asked?
> >
> > *) Easier to view for users
> > *) If syncing between Ceph clusters with the same pool names, you can
> > transfer layouts properly (except...probably xattrs are set after data
> > is written in most systems, so that doesn't work?)
> > *) ...I've got nothing else
> >
> > Downsides:
> > *) Other tools get very confused by viewing xattrs they can't set
> >
> > Personally, I'd rather only list them when the Ceph namespace is
> > specifically queried (or the xattr itself is, at whatever granularity
> > we can do). This isn't like directory sizes; xattrs have defined
> > semantics that we're breaking, *and* exposing them all the time just
> > doesn't seem that useful. Why would anybody care about the data
> > they're sharing, who doesn't already know that it's available?
> > -Greg
> 
> I'm inclined to agree that we shouldn't include ceph.* xattrs in the
> normal listing of xattrs.  Created tickets:
> http://tracker.ceph.com/issues/16468
> http://tracker.ceph.com/issues/16467
> 
> I do think it would be more correct for rsync to avoid xattrs in any
> namespace it doesn't know about though; given that xattrs are meant to
> be for potentially-filesystem-specific ones, it should assume that any
> other namespaces are non-standard and are not necessarily safe to
> copy.  Realistically though, rsync probably won't be the only tool
> that ever tries to do this, so we should try to be safe against people
> running any other tools that make the same assumption that listed
> xattrs are copyable.

Yeah.

My main concern is that currently getfattr -d or attr -l are an 
easy/convenient way for users to discover what interesting information 
Ceph is exposing, without any special tools.  Without this, the user has 
to know what information is there, and remember the exact name of the 
xattr(s).  It's also one way to discover whether a quota or layout is set 
on a particular directory.

Maybe we could make a ceph.all vxattr that, when you get it, will dump a 
list of key/value pairs with everything?  e.g.,

$ getfattr -n ceph.all .
# file: .
ceph.all="ceph.dir.entries=14832
ceph.dir.files=755
ceph.dir.rbytes=18300954645291
ceph.dir.rctime=1466771118.09752118291
ceph.dir.rentries=22378800
ceph.dir.rfiles=21586687
ceph.dir.rsubdirs=792113
ceph.dir.subdirs=14077
"

?  Then it's just one magic xattr to remember to discover the rest 
(including anything new we add in the future).

sage





> John
> 
> >>
> >> There's nothing to prevent having a debug-ish toggle to expose them when you really want to see them.
> >>
> >> Allen Samuels
> >> SanDisk |a Western Digital brand
> >> 2880 Junction Avenue, San Jose, CA 95134
> >> T: +1 408 801 7030| M: +1 408 780 6416
> >> allen.samuels@SanDisk.com
> >>
> >>
> >>> -----Original Message-----
> >>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> >>> owner@vger.kernel.org] On Behalf Of Sage Weil
> >>> Sent: Tuesday, June 21, 2016 2:30 PM
> >>> To: Michael Wyraz <michael@wyraz.de>
> >>> Cc: ceph-devel@vger.kernel.org
> >>> Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal
> >>> xattrs are exposed to clients.
> >>>
> >>> On Tue, 21 Jun 2016, Michael Wyraz wrote:
> >>> > Hi,
> >>> >
> >>> > I do rsync based backups against a cephfs. If I do sync xattrs, rsync
> >>> > tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an
> >>> error.
> >>> > xattrs does not get synchronized in this case.
> >>> > IMO the problem is, that ceph's internal attributes are exposed to the
> >>> > mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the
> >>> > mounted cephfs and get results). So rsync sees these xattrs and tries to
> >>> remove it.
> >>> >
> >>> > Example output:
> >>> >
> >>> >    rsync: rsync_xal_set:
> >>> >    lremovexattr(""/backups/2016-05-31_00-09-
> >>> 32/etc/acpi/events"","ceph.dir.entries")
> >>> >    failed: Operation not supported (95)
> >>> >    rsync: rsync_xal_set:
> >>> >    lremovexattr(""/backups/2016-05-31_00-09-
> >>> 32/etc/acpi/events"","ceph.dir.files")
> >>> >    failed: Operation not supported (95)
> >>> >    rsync: rsync_xal_set:
> >>> >    lremovexattr(""/backups/2016-05-31_00-09-
> >>> 32/etc/acpi/events"","ceph.dir.subdirs")
> >>> >    failed: Operation not supported (95)
> >>> >    rsync: rsync_xal_set:
> >>> >    lremovexattr(""/backups/2016-05-31_00-09-
> >>> 32/etc/acpi/events"","ceph.dir.rentries")
> >>> >    failed: Operation not supported (95)
> >>> >    rsync: rsync_xal_set:
> >>> >    lremovexattr(""/backups/2016-05-31_00-09-
> >>> 32/etc/acpi/events"","ceph.dir.rfiles")
> >>> >    failed: Operation not supported (95)
> >>> >    rsync: rsync_xal_set:
> >>> >    lremovexattr(""/backups/2016-05-31_00-09-
> >>> 32/etc/acpi/events"","ceph.dir.rsubdirs")
> >>> >    failed: Operation not supported (95)
> >>> >    rsync: rsync_xal_set:
> >>> >    lremovexattr(""/backups/2016-05-31_00-09-
> >>> 32/etc/acpi/events"","ceph.dir.rbytes")
> >>> >    failed: Operation not supported (95)
> >>> >    rsync: rsync_xal_set:
> >>> >    lremovexattr(""/backups/2016-05-31_00-09-
> >>> 32/etc/acpi/events"","ceph.dir.rctime")
> >>> >    failed: Operation not supported (95)
> >>>
> >>> I think the right fix here is to patch rsync to ignore ceph.* xattrs.
> >>> Does that seem reasonable?
> >>>
> >>> Other possible workarounds might be:
> >>>
> >>>  - ignore setxattr and removexattr requests on these xattrs (return success
> >>> but do nothing)
> >>>  - never show these xattrs in listxattr.  this makes them less useful or friendly
> >>> :(
> >>>
> >>> sage
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
> >>> body of a message to majordomo@vger.kernel.org More majordomo info at
> >>> http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients.

[Gregory Farnum]

On Fri, Jun 24, 2016 at 5:27 AM, Sage Weil <sweil@redhat.com> wrote:
> On Fri, 24 Jun 2016, John Spray wrote:
>> On Wed, Jun 22, 2016 at 9:15 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>> > On Tue, Jun 21, 2016 at 3:30 PM, Allen Samuels
>> > <Allen.Samuels@sandisk.com> wrote:
>> >> I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand.
>> >>
>> >> What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information?
>> >
>> > This is something I think we've talked about before, and I keep on
>> > being surprised to learn we *do* list them without a specific query.
>> > So, what are the upsides of listing them without being asked?
>> >
>> > *) Easier to view for users
>> > *) If syncing between Ceph clusters with the same pool names, you can
>> > transfer layouts properly (except...probably xattrs are set after data
>> > is written in most systems, so that doesn't work?)
>> > *) ...I've got nothing else
>> >
>> > Downsides:
>> > *) Other tools get very confused by viewing xattrs they can't set
>> >
>> > Personally, I'd rather only list them when the Ceph namespace is
>> > specifically queried (or the xattr itself is, at whatever granularity
>> > we can do). This isn't like directory sizes; xattrs have defined
>> > semantics that we're breaking, *and* exposing them all the time just
>> > doesn't seem that useful. Why would anybody care about the data
>> > they're sharing, who doesn't already know that it's available?
>> > -Greg
>>
>> I'm inclined to agree that we shouldn't include ceph.* xattrs in the
>> normal listing of xattrs.  Created tickets:
>> http://tracker.ceph.com/issues/16468
>> http://tracker.ceph.com/issues/16467
>>
>> I do think it would be more correct for rsync to avoid xattrs in any
>> namespace it doesn't know about though; given that xattrs are meant to
>> be for potentially-filesystem-specific ones, it should assume that any
>> other namespaces are non-standard and are not necessarily safe to
>> copy.  Realistically though, rsync probably won't be the only tool
>> that ever tries to do this, so we should try to be safe against people
>> running any other tools that make the same assumption that listed
>> xattrs are copyable.
>
> Yeah.
>
> My main concern is that currently getfattr -d or attr -l are an
> easy/convenient way for users to discover what interesting information
> Ceph is exposing, without any special tools.  Without this, the user has
> to know what information is there, and remember the exact name of the
> xattr(s).  It's also one way to discover whether a quota or layout is set
> on a particular directory.
>
> Maybe we could make a ceph.all vxattr that, when you get it, will dump a
> list of key/value pairs with everything?  e.g.,
>
> $ getfattr -n ceph.all .
> # file: .
> ceph.all="ceph.dir.entries=14832
> ceph.dir.files=755
> ceph.dir.rbytes=18300954645291
> ceph.dir.rctime=1466771118.09752118291
> ceph.dir.rentries=22378800
> ceph.dir.rfiles=21586687
> ceph.dir.rsubdirs=792113
> ceph.dir.subdirs=14077
> "
>
> ?  Then it's just one magic xattr to remember to discover the rest
> (including anything new we add in the future).

That's analogous to how we handle snapshot directories and is what I'd
like to see. (Unless there's some way of requesting listing of a
specific xattr namespace?)
-Greg