* Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients. @ 2016-06-21 20:34 Michael Wyraz 2016-06-21 21:30 ` Sage Weil 0 siblings, 1 reply; 9+ messages in thread From: Michael Wyraz @ 2016-06-21 20:34 UTC (permalink / raw) To: ceph-devel Hi, I do rsync based backups against a cephfs. If I do sync xattrs, rsync tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an error. xattrs does not get synchronized in this case. IMO the problem is, that ceph's internal attributes are exposed to the mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the mounted cephfs and get results). So rsync sees these xattrs and tries to remove it. Example output: rsync: rsync_xal_set: lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.entries") failed: Operation not supported (95) rsync: rsync_xal_set: lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.files") failed: Operation not supported (95) rsync: rsync_xal_set: lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.subdirs") failed: Operation not supported (95) rsync: rsync_xal_set: lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rentries") failed: Operation not supported (95) rsync: rsync_xal_set: lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rfiles") failed: Operation not supported (95) rsync: rsync_xal_set: lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rsubdirs") failed: Operation not supported (95) rsync: rsync_xal_set: lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rbytes") failed: Operation not supported (95) rsync: rsync_xal_set: lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rctime") failed: Operation not supported (95) Kind regards, Michael. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients. 2016-06-21 20:34 Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients Michael Wyraz @ 2016-06-21 21:30 ` Sage Weil 2016-06-21 22:30 ` Allen Samuels 0 siblings, 1 reply; 9+ messages in thread From: Sage Weil @ 2016-06-21 21:30 UTC (permalink / raw) To: Michael Wyraz; +Cc: ceph-devel On Tue, 21 Jun 2016, Michael Wyraz wrote: > Hi, > > I do rsync based backups against a cephfs. If I do sync xattrs, rsync tries to > delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an error. > xattrs does not get synchronized in this case. > IMO the problem is, that ceph's internal attributes are exposed to the mounted > fileystem (I can do "getfattr -n ceph.dir.entries *" on the mounted cephfs and > get results). So rsync sees these xattrs and tries to remove it. > > Example output: > > rsync: rsync_xal_set: > lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.entries") > failed: Operation not supported (95) > rsync: rsync_xal_set: > lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.files") > failed: Operation not supported (95) > rsync: rsync_xal_set: > lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.subdirs") > failed: Operation not supported (95) > rsync: rsync_xal_set: > lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rentries") > failed: Operation not supported (95) > rsync: rsync_xal_set: > lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rfiles") > failed: Operation not supported (95) > rsync: rsync_xal_set: > lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rsubdirs") > failed: Operation not supported (95) > rsync: rsync_xal_set: > lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rbytes") > failed: Operation not supported (95) > rsync: rsync_xal_set: > lremovexattr(""/backups/2016-05-31_00-09-32/etc/acpi/events"","ceph.dir.rctime") > failed: Operation not supported (95) I think the right fix here is to patch rsync to ignore ceph.* xattrs. Does that seem reasonable? Other possible workarounds might be: - ignore setxattr and removexattr requests on these xattrs (return success but do nothing) - never show these xattrs in listxattr. this makes them less useful or friendly :( sage ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients. 2016-06-21 21:30 ` Sage Weil @ 2016-06-21 22:30 ` Allen Samuels 2016-06-22 4:46 ` Michael Wyraz 2016-06-22 20:15 ` Gregory Farnum 0 siblings, 2 replies; 9+ messages in thread From: Allen Samuels @ 2016-06-21 22:30 UTC (permalink / raw) To: Sage Weil, Michael Wyraz; +Cc: ceph-devel I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand. What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information? There's nothing to prevent having a debug-ish toggle to expose them when you really want to see them. Allen Samuels SanDisk |a Western Digital brand 2880 Junction Avenue, San Jose, CA 95134 T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@SanDisk.com > -----Original Message----- > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- > owner@vger.kernel.org] On Behalf Of Sage Weil > Sent: Tuesday, June 21, 2016 2:30 PM > To: Michael Wyraz <michael@wyraz.de> > Cc: ceph-devel@vger.kernel.org > Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal > xattrs are exposed to clients. > > On Tue, 21 Jun 2016, Michael Wyraz wrote: > > Hi, > > > > I do rsync based backups against a cephfs. If I do sync xattrs, rsync > > tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an > error. > > xattrs does not get synchronized in this case. > > IMO the problem is, that ceph's internal attributes are exposed to the > > mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the > > mounted cephfs and get results). So rsync sees these xattrs and tries to > remove it. > > > > Example output: > > > > rsync: rsync_xal_set: > > lremovexattr(""/backups/2016-05-31_00-09- > 32/etc/acpi/events"","ceph.dir.entries") > > failed: Operation not supported (95) > > rsync: rsync_xal_set: > > lremovexattr(""/backups/2016-05-31_00-09- > 32/etc/acpi/events"","ceph.dir.files") > > failed: Operation not supported (95) > > rsync: rsync_xal_set: > > lremovexattr(""/backups/2016-05-31_00-09- > 32/etc/acpi/events"","ceph.dir.subdirs") > > failed: Operation not supported (95) > > rsync: rsync_xal_set: > > lremovexattr(""/backups/2016-05-31_00-09- > 32/etc/acpi/events"","ceph.dir.rentries") > > failed: Operation not supported (95) > > rsync: rsync_xal_set: > > lremovexattr(""/backups/2016-05-31_00-09- > 32/etc/acpi/events"","ceph.dir.rfiles") > > failed: Operation not supported (95) > > rsync: rsync_xal_set: > > lremovexattr(""/backups/2016-05-31_00-09- > 32/etc/acpi/events"","ceph.dir.rsubdirs") > > failed: Operation not supported (95) > > rsync: rsync_xal_set: > > lremovexattr(""/backups/2016-05-31_00-09- > 32/etc/acpi/events"","ceph.dir.rbytes") > > failed: Operation not supported (95) > > rsync: rsync_xal_set: > > lremovexattr(""/backups/2016-05-31_00-09- > 32/etc/acpi/events"","ceph.dir.rctime") > > failed: Operation not supported (95) > > I think the right fix here is to patch rsync to ignore ceph.* xattrs. > Does that seem reasonable? > > Other possible workarounds might be: > > - ignore setxattr and removexattr requests on these xattrs (return success > but do nothing) > - never show these xattrs in listxattr. this makes them less useful or friendly > :( > > sage > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the > body of a message to majordomo@vger.kernel.org More majordomo info at > http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients. 2016-06-21 22:30 ` Allen Samuels @ 2016-06-22 4:46 ` Michael Wyraz 2016-06-22 20:15 ` Gregory Farnum 1 sibling, 0 replies; 9+ messages in thread From: Michael Wyraz @ 2016-06-22 4:46 UTC (permalink / raw) Cc: ceph-devel That's a good point, Allen. Exposing of this special xattrs should be enabled on demand, like it is already done with the "dirstat" mount option. > I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand. > > What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information? > > There's nothing to prevent having a debug-ish toggle to expose them when you really want to see them. > > Allen Samuels > SanDisk |a Western Digital brand > 2880 Junction Avenue, San Jose, CA 95134 > T: +1 408 801 7030| M: +1 408 780 6416 > allen.samuels@SanDisk.com > > >> -----Original Message----- >> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- >> owner@vger.kernel.org] On Behalf Of Sage Weil >> Sent: Tuesday, June 21, 2016 2:30 PM >> To: Michael Wyraz <michael@wyraz.de> >> Cc: ceph-devel@vger.kernel.org >> Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal >> xattrs are exposed to clients. >> >> On Tue, 21 Jun 2016, Michael Wyraz wrote: >>> Hi, >>> >>> I do rsync based backups against a cephfs. If I do sync xattrs, rsync >>> tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an >> error. >>> xattrs does not get synchronized in this case. >>> IMO the problem is, that ceph's internal attributes are exposed to the >>> mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the >>> mounted cephfs and get results). So rsync sees these xattrs and tries to >> remove it. >>> Example output: >>> >>> rsync: rsync_xal_set: >>> lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.entries") >>> failed: Operation not supported (95) >>> rsync: rsync_xal_set: >>> lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.files") >>> failed: Operation not supported (95) >>> rsync: rsync_xal_set: >>> lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.subdirs") >>> failed: Operation not supported (95) >>> rsync: rsync_xal_set: >>> lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.rentries") >>> failed: Operation not supported (95) >>> rsync: rsync_xal_set: >>> lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.rfiles") >>> failed: Operation not supported (95) >>> rsync: rsync_xal_set: >>> lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.rsubdirs") >>> failed: Operation not supported (95) >>> rsync: rsync_xal_set: >>> lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.rbytes") >>> failed: Operation not supported (95) >>> rsync: rsync_xal_set: >>> lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.rctime") >>> failed: Operation not supported (95) >> I think the right fix here is to patch rsync to ignore ceph.* xattrs. >> Does that seem reasonable? >> >> Other possible workarounds might be: >> >> - ignore setxattr and removexattr requests on these xattrs (return success >> but do nothing) >> - never show these xattrs in listxattr. this makes them less useful or friendly >> :( >> >> sage >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the >> body of a message to majordomo@vger.kernel.org More majordomo info at >> http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients. 2016-06-21 22:30 ` Allen Samuels 2016-06-22 4:46 ` Michael Wyraz @ 2016-06-22 20:15 ` Gregory Farnum 2016-06-22 20:24 ` Allen Samuels 2016-06-24 10:08 ` John Spray 1 sibling, 2 replies; 9+ messages in thread From: Gregory Farnum @ 2016-06-22 20:15 UTC (permalink / raw) To: Allen Samuels; +Cc: Sage Weil, Michael Wyraz, ceph-devel On Tue, Jun 21, 2016 at 3:30 PM, Allen Samuels <Allen.Samuels@sandisk.com> wrote: > I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand. > > What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information? This is something I think we've talked about before, and I keep on being surprised to learn we *do* list them without a specific query. So, what are the upsides of listing them without being asked? *) Easier to view for users *) If syncing between Ceph clusters with the same pool names, you can transfer layouts properly (except...probably xattrs are set after data is written in most systems, so that doesn't work?) *) ...I've got nothing else Downsides: *) Other tools get very confused by viewing xattrs they can't set Personally, I'd rather only list them when the Ceph namespace is specifically queried (or the xattr itself is, at whatever granularity we can do). This isn't like directory sizes; xattrs have defined semantics that we're breaking, *and* exposing them all the time just doesn't seem that useful. Why would anybody care about the data they're sharing, who doesn't already know that it's available? -Greg > > There's nothing to prevent having a debug-ish toggle to expose them when you really want to see them. > > Allen Samuels > SanDisk |a Western Digital brand > 2880 Junction Avenue, San Jose, CA 95134 > T: +1 408 801 7030| M: +1 408 780 6416 > allen.samuels@SanDisk.com > > >> -----Original Message----- >> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- >> owner@vger.kernel.org] On Behalf Of Sage Weil >> Sent: Tuesday, June 21, 2016 2:30 PM >> To: Michael Wyraz <michael@wyraz.de> >> Cc: ceph-devel@vger.kernel.org >> Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal >> xattrs are exposed to clients. >> >> On Tue, 21 Jun 2016, Michael Wyraz wrote: >> > Hi, >> > >> > I do rsync based backups against a cephfs. If I do sync xattrs, rsync >> > tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an >> error. >> > xattrs does not get synchronized in this case. >> > IMO the problem is, that ceph's internal attributes are exposed to the >> > mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the >> > mounted cephfs and get results). So rsync sees these xattrs and tries to >> remove it. >> > >> > Example output: >> > >> > rsync: rsync_xal_set: >> > lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.entries") >> > failed: Operation not supported (95) >> > rsync: rsync_xal_set: >> > lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.files") >> > failed: Operation not supported (95) >> > rsync: rsync_xal_set: >> > lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.subdirs") >> > failed: Operation not supported (95) >> > rsync: rsync_xal_set: >> > lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.rentries") >> > failed: Operation not supported (95) >> > rsync: rsync_xal_set: >> > lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.rfiles") >> > failed: Operation not supported (95) >> > rsync: rsync_xal_set: >> > lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.rsubdirs") >> > failed: Operation not supported (95) >> > rsync: rsync_xal_set: >> > lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.rbytes") >> > failed: Operation not supported (95) >> > rsync: rsync_xal_set: >> > lremovexattr(""/backups/2016-05-31_00-09- >> 32/etc/acpi/events"","ceph.dir.rctime") >> > failed: Operation not supported (95) >> >> I think the right fix here is to patch rsync to ignore ceph.* xattrs. >> Does that seem reasonable? >> >> Other possible workarounds might be: >> >> - ignore setxattr and removexattr requests on these xattrs (return success >> but do nothing) >> - never show these xattrs in listxattr. this makes them less useful or friendly >> :( >> >> sage >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the >> body of a message to majordomo@vger.kernel.org More majordomo info at >> http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients. 2016-06-22 20:15 ` Gregory Farnum @ 2016-06-22 20:24 ` Allen Samuels 2016-06-24 10:08 ` John Spray 1 sibling, 0 replies; 9+ messages in thread From: Allen Samuels @ 2016-06-22 20:24 UTC (permalink / raw) To: Gregory Farnum; +Cc: Sage Weil, Michael Wyraz, ceph-devel > -----Original Message----- > From: Gregory Farnum [mailto:gfarnum@redhat.com] > Sent: Wednesday, June 22, 2016 1:16 PM > To: Allen Samuels <Allen.Samuels@sandisk.com> > Cc: Sage Weil <sage@newdream.net>; Michael Wyraz <michael@wyraz.de>; > ceph-devel@vger.kernel.org > Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal > xattrs are exposed to clients. > > On Tue, Jun 21, 2016 at 3:30 PM, Allen Samuels > <Allen.Samuels@sandisk.com> wrote: > > I doubt that rsync is the only application that's going to be confused by the > presence of attributes it doesn't understand. > > > > What is the value in having these attributes be visible to user-space > applications? Is there a better mechanism for conveying this information? > > This is something I think we've talked about before, and I keep on being > surprised to learn we *do* list them without a specific query. > So, what are the upsides of listing them without being asked? > > *) Easier to view for users If users == Ceph Cognoscenti, you're right. But if users != Ceph Cognoscenti, then this is actually a downside -- not really different from confusing the tools. > *) If syncing between Ceph clusters with the same pool names, you can > transfer layouts properly (except...probably xattrs are set after data is > written in most systems, so that doesn't work?) If it doesn't work, it's probably not a plus ;-) > *) ...I've got nothing else > > Downsides: > *) Other tools get very confused by viewing xattrs they can't set > > Personally, I'd rather only list them when the Ceph namespace is specifically > queried (or the xattr itself is, at whatever granularity we can do). This isn't > like directory sizes; xattrs have defined semantics that we're breaking, *and* > exposing them all the time just doesn't seem that useful. Why would > anybody care about the data they're sharing, who doesn't already know that > it's available? > -Greg > > > > > There's nothing to prevent having a debug-ish toggle to expose them when > you really want to see them. > > > > Allen Samuels > > SanDisk |a Western Digital brand > > 2880 Junction Avenue, San Jose, CA 95134 > > T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@SanDisk.com > > > > > >> -----Original Message----- > >> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- > >> owner@vger.kernel.org] On Behalf Of Sage Weil > >> Sent: Tuesday, June 21, 2016 2:30 PM > >> To: Michael Wyraz <michael@wyraz.de> > >> Cc: ceph-devel@vger.kernel.org > >> Subject: Re: Problems with cephfs, rsync and xattr because cephfs > internal > >> xattrs are exposed to clients. > >> > >> On Tue, 21 Jun 2016, Michael Wyraz wrote: > >> > Hi, > >> > > >> > I do rsync based backups against a cephfs. If I do sync xattrs, rsync > >> > tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails > with an > >> error. > >> > xattrs does not get synchronized in this case. > >> > IMO the problem is, that ceph's internal attributes are exposed to the > >> > mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the > >> > mounted cephfs and get results). So rsync sees these xattrs and tries to > >> remove it. > >> > > >> > Example output: > >> > > >> > rsync: rsync_xal_set: > >> > lremovexattr(""/backups/2016-05-31_00-09- > >> 32/etc/acpi/events"","ceph.dir.entries") > >> > failed: Operation not supported (95) > >> > rsync: rsync_xal_set: > >> > lremovexattr(""/backups/2016-05-31_00-09- > >> 32/etc/acpi/events"","ceph.dir.files") > >> > failed: Operation not supported (95) > >> > rsync: rsync_xal_set: > >> > lremovexattr(""/backups/2016-05-31_00-09- > >> 32/etc/acpi/events"","ceph.dir.subdirs") > >> > failed: Operation not supported (95) > >> > rsync: rsync_xal_set: > >> > lremovexattr(""/backups/2016-05-31_00-09- > >> 32/etc/acpi/events"","ceph.dir.rentries") > >> > failed: Operation not supported (95) > >> > rsync: rsync_xal_set: > >> > lremovexattr(""/backups/2016-05-31_00-09- > >> 32/etc/acpi/events"","ceph.dir.rfiles") > >> > failed: Operation not supported (95) > >> > rsync: rsync_xal_set: > >> > lremovexattr(""/backups/2016-05-31_00-09- > >> 32/etc/acpi/events"","ceph.dir.rsubdirs") > >> > failed: Operation not supported (95) > >> > rsync: rsync_xal_set: > >> > lremovexattr(""/backups/2016-05-31_00-09- > >> 32/etc/acpi/events"","ceph.dir.rbytes") > >> > failed: Operation not supported (95) > >> > rsync: rsync_xal_set: > >> > lremovexattr(""/backups/2016-05-31_00-09- > >> 32/etc/acpi/events"","ceph.dir.rctime") > >> > failed: Operation not supported (95) > >> > >> I think the right fix here is to patch rsync to ignore ceph.* xattrs. > >> Does that seem reasonable? > >> > >> Other possible workarounds might be: > >> > >> - ignore setxattr and removexattr requests on these xattrs (return > success > >> but do nothing) > >> - never show these xattrs in listxattr. this makes them less useful or > friendly > >> :( > >> > >> sage > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the > >> body of a message to majordomo@vger.kernel.org More majordomo info > at > >> http://vger.kernel.org/majordomo-info.html > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients. 2016-06-22 20:15 ` Gregory Farnum 2016-06-22 20:24 ` Allen Samuels @ 2016-06-24 10:08 ` John Spray 2016-06-24 12:27 ` Sage Weil 1 sibling, 1 reply; 9+ messages in thread From: John Spray @ 2016-06-24 10:08 UTC (permalink / raw) To: Gregory Farnum; +Cc: Allen Samuels, Sage Weil, Michael Wyraz, ceph-devel On Wed, Jun 22, 2016 at 9:15 PM, Gregory Farnum <gfarnum@redhat.com> wrote: > On Tue, Jun 21, 2016 at 3:30 PM, Allen Samuels > <Allen.Samuels@sandisk.com> wrote: >> I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand. >> >> What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information? > > This is something I think we've talked about before, and I keep on > being surprised to learn we *do* list them without a specific query. > So, what are the upsides of listing them without being asked? > > *) Easier to view for users > *) If syncing between Ceph clusters with the same pool names, you can > transfer layouts properly (except...probably xattrs are set after data > is written in most systems, so that doesn't work?) > *) ...I've got nothing else > > Downsides: > *) Other tools get very confused by viewing xattrs they can't set > > Personally, I'd rather only list them when the Ceph namespace is > specifically queried (or the xattr itself is, at whatever granularity > we can do). This isn't like directory sizes; xattrs have defined > semantics that we're breaking, *and* exposing them all the time just > doesn't seem that useful. Why would anybody care about the data > they're sharing, who doesn't already know that it's available? > -Greg I'm inclined to agree that we shouldn't include ceph.* xattrs in the normal listing of xattrs. Created tickets: http://tracker.ceph.com/issues/16468 http://tracker.ceph.com/issues/16467 I do think it would be more correct for rsync to avoid xattrs in any namespace it doesn't know about though; given that xattrs are meant to be for potentially-filesystem-specific ones, it should assume that any other namespaces are non-standard and are not necessarily safe to copy. Realistically though, rsync probably won't be the only tool that ever tries to do this, so we should try to be safe against people running any other tools that make the same assumption that listed xattrs are copyable. John >> >> There's nothing to prevent having a debug-ish toggle to expose them when you really want to see them. >> >> Allen Samuels >> SanDisk |a Western Digital brand >> 2880 Junction Avenue, San Jose, CA 95134 >> T: +1 408 801 7030| M: +1 408 780 6416 >> allen.samuels@SanDisk.com >> >> >>> -----Original Message----- >>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- >>> owner@vger.kernel.org] On Behalf Of Sage Weil >>> Sent: Tuesday, June 21, 2016 2:30 PM >>> To: Michael Wyraz <michael@wyraz.de> >>> Cc: ceph-devel@vger.kernel.org >>> Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal >>> xattrs are exposed to clients. >>> >>> On Tue, 21 Jun 2016, Michael Wyraz wrote: >>> > Hi, >>> > >>> > I do rsync based backups against a cephfs. If I do sync xattrs, rsync >>> > tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an >>> error. >>> > xattrs does not get synchronized in this case. >>> > IMO the problem is, that ceph's internal attributes are exposed to the >>> > mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the >>> > mounted cephfs and get results). So rsync sees these xattrs and tries to >>> remove it. >>> > >>> > Example output: >>> > >>> > rsync: rsync_xal_set: >>> > lremovexattr(""/backups/2016-05-31_00-09- >>> 32/etc/acpi/events"","ceph.dir.entries") >>> > failed: Operation not supported (95) >>> > rsync: rsync_xal_set: >>> > lremovexattr(""/backups/2016-05-31_00-09- >>> 32/etc/acpi/events"","ceph.dir.files") >>> > failed: Operation not supported (95) >>> > rsync: rsync_xal_set: >>> > lremovexattr(""/backups/2016-05-31_00-09- >>> 32/etc/acpi/events"","ceph.dir.subdirs") >>> > failed: Operation not supported (95) >>> > rsync: rsync_xal_set: >>> > lremovexattr(""/backups/2016-05-31_00-09- >>> 32/etc/acpi/events"","ceph.dir.rentries") >>> > failed: Operation not supported (95) >>> > rsync: rsync_xal_set: >>> > lremovexattr(""/backups/2016-05-31_00-09- >>> 32/etc/acpi/events"","ceph.dir.rfiles") >>> > failed: Operation not supported (95) >>> > rsync: rsync_xal_set: >>> > lremovexattr(""/backups/2016-05-31_00-09- >>> 32/etc/acpi/events"","ceph.dir.rsubdirs") >>> > failed: Operation not supported (95) >>> > rsync: rsync_xal_set: >>> > lremovexattr(""/backups/2016-05-31_00-09- >>> 32/etc/acpi/events"","ceph.dir.rbytes") >>> > failed: Operation not supported (95) >>> > rsync: rsync_xal_set: >>> > lremovexattr(""/backups/2016-05-31_00-09- >>> 32/etc/acpi/events"","ceph.dir.rctime") >>> > failed: Operation not supported (95) >>> >>> I think the right fix here is to patch rsync to ignore ceph.* xattrs. >>> Does that seem reasonable? >>> >>> Other possible workarounds might be: >>> >>> - ignore setxattr and removexattr requests on these xattrs (return success >>> but do nothing) >>> - never show these xattrs in listxattr. this makes them less useful or friendly >>> :( >>> >>> sage >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the >>> body of a message to majordomo@vger.kernel.org More majordomo info at >>> http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients. 2016-06-24 10:08 ` John Spray @ 2016-06-24 12:27 ` Sage Weil 2016-06-24 13:53 ` Gregory Farnum 0 siblings, 1 reply; 9+ messages in thread From: Sage Weil @ 2016-06-24 12:27 UTC (permalink / raw) To: John Spray; +Cc: Gregory Farnum, Allen Samuels, Michael Wyraz, ceph-devel On Fri, 24 Jun 2016, John Spray wrote: > On Wed, Jun 22, 2016 at 9:15 PM, Gregory Farnum <gfarnum@redhat.com> wrote: > > On Tue, Jun 21, 2016 at 3:30 PM, Allen Samuels > > <Allen.Samuels@sandisk.com> wrote: > >> I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand. > >> > >> What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information? > > > > This is something I think we've talked about before, and I keep on > > being surprised to learn we *do* list them without a specific query. > > So, what are the upsides of listing them without being asked? > > > > *) Easier to view for users > > *) If syncing between Ceph clusters with the same pool names, you can > > transfer layouts properly (except...probably xattrs are set after data > > is written in most systems, so that doesn't work?) > > *) ...I've got nothing else > > > > Downsides: > > *) Other tools get very confused by viewing xattrs they can't set > > > > Personally, I'd rather only list them when the Ceph namespace is > > specifically queried (or the xattr itself is, at whatever granularity > > we can do). This isn't like directory sizes; xattrs have defined > > semantics that we're breaking, *and* exposing them all the time just > > doesn't seem that useful. Why would anybody care about the data > > they're sharing, who doesn't already know that it's available? > > -Greg > > I'm inclined to agree that we shouldn't include ceph.* xattrs in the > normal listing of xattrs. Created tickets: > http://tracker.ceph.com/issues/16468 > http://tracker.ceph.com/issues/16467 > > I do think it would be more correct for rsync to avoid xattrs in any > namespace it doesn't know about though; given that xattrs are meant to > be for potentially-filesystem-specific ones, it should assume that any > other namespaces are non-standard and are not necessarily safe to > copy. Realistically though, rsync probably won't be the only tool > that ever tries to do this, so we should try to be safe against people > running any other tools that make the same assumption that listed > xattrs are copyable. Yeah. My main concern is that currently getfattr -d or attr -l are an easy/convenient way for users to discover what interesting information Ceph is exposing, without any special tools. Without this, the user has to know what information is there, and remember the exact name of the xattr(s). It's also one way to discover whether a quota or layout is set on a particular directory. Maybe we could make a ceph.all vxattr that, when you get it, will dump a list of key/value pairs with everything? e.g., $ getfattr -n ceph.all . # file: . ceph.all="ceph.dir.entries=14832 ceph.dir.files=755 ceph.dir.rbytes=18300954645291 ceph.dir.rctime=1466771118.09752118291 ceph.dir.rentries=22378800 ceph.dir.rfiles=21586687 ceph.dir.rsubdirs=792113 ceph.dir.subdirs=14077 " ? Then it's just one magic xattr to remember to discover the rest (including anything new we add in the future). sage > John > > >> > >> There's nothing to prevent having a debug-ish toggle to expose them when you really want to see them. > >> > >> Allen Samuels > >> SanDisk |a Western Digital brand > >> 2880 Junction Avenue, San Jose, CA 95134 > >> T: +1 408 801 7030| M: +1 408 780 6416 > >> allen.samuels@SanDisk.com > >> > >> > >>> -----Original Message----- > >>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- > >>> owner@vger.kernel.org] On Behalf Of Sage Weil > >>> Sent: Tuesday, June 21, 2016 2:30 PM > >>> To: Michael Wyraz <michael@wyraz.de> > >>> Cc: ceph-devel@vger.kernel.org > >>> Subject: Re: Problems with cephfs, rsync and xattr because cephfs internal > >>> xattrs are exposed to clients. > >>> > >>> On Tue, 21 Jun 2016, Michael Wyraz wrote: > >>> > Hi, > >>> > > >>> > I do rsync based backups against a cephfs. If I do sync xattrs, rsync > >>> > tries to delete ceph-internal xattrs (e.g. ceph.dir.entries) which fails with an > >>> error. > >>> > xattrs does not get synchronized in this case. > >>> > IMO the problem is, that ceph's internal attributes are exposed to the > >>> > mounted fileystem (I can do "getfattr -n ceph.dir.entries *" on the > >>> > mounted cephfs and get results). So rsync sees these xattrs and tries to > >>> remove it. > >>> > > >>> > Example output: > >>> > > >>> > rsync: rsync_xal_set: > >>> > lremovexattr(""/backups/2016-05-31_00-09- > >>> 32/etc/acpi/events"","ceph.dir.entries") > >>> > failed: Operation not supported (95) > >>> > rsync: rsync_xal_set: > >>> > lremovexattr(""/backups/2016-05-31_00-09- > >>> 32/etc/acpi/events"","ceph.dir.files") > >>> > failed: Operation not supported (95) > >>> > rsync: rsync_xal_set: > >>> > lremovexattr(""/backups/2016-05-31_00-09- > >>> 32/etc/acpi/events"","ceph.dir.subdirs") > >>> > failed: Operation not supported (95) > >>> > rsync: rsync_xal_set: > >>> > lremovexattr(""/backups/2016-05-31_00-09- > >>> 32/etc/acpi/events"","ceph.dir.rentries") > >>> > failed: Operation not supported (95) > >>> > rsync: rsync_xal_set: > >>> > lremovexattr(""/backups/2016-05-31_00-09- > >>> 32/etc/acpi/events"","ceph.dir.rfiles") > >>> > failed: Operation not supported (95) > >>> > rsync: rsync_xal_set: > >>> > lremovexattr(""/backups/2016-05-31_00-09- > >>> 32/etc/acpi/events"","ceph.dir.rsubdirs") > >>> > failed: Operation not supported (95) > >>> > rsync: rsync_xal_set: > >>> > lremovexattr(""/backups/2016-05-31_00-09- > >>> 32/etc/acpi/events"","ceph.dir.rbytes") > >>> > failed: Operation not supported (95) > >>> > rsync: rsync_xal_set: > >>> > lremovexattr(""/backups/2016-05-31_00-09- > >>> 32/etc/acpi/events"","ceph.dir.rctime") > >>> > failed: Operation not supported (95) > >>> > >>> I think the right fix here is to patch rsync to ignore ceph.* xattrs. > >>> Does that seem reasonable? > >>> > >>> Other possible workarounds might be: > >>> > >>> - ignore setxattr and removexattr requests on these xattrs (return success > >>> but do nothing) > >>> - never show these xattrs in listxattr. this makes them less useful or friendly > >>> :( > >>> > >>> sage > >>> -- > >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the > >>> body of a message to majordomo@vger.kernel.org More majordomo info at > >>> http://vger.kernel.org/majordomo-info.html > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients. 2016-06-24 12:27 ` Sage Weil @ 2016-06-24 13:53 ` Gregory Farnum 0 siblings, 0 replies; 9+ messages in thread From: Gregory Farnum @ 2016-06-24 13:53 UTC (permalink / raw) To: Sage Weil; +Cc: John Spray, Allen Samuels, Michael Wyraz, ceph-devel On Fri, Jun 24, 2016 at 5:27 AM, Sage Weil <sweil@redhat.com> wrote: > On Fri, 24 Jun 2016, John Spray wrote: >> On Wed, Jun 22, 2016 at 9:15 PM, Gregory Farnum <gfarnum@redhat.com> wrote: >> > On Tue, Jun 21, 2016 at 3:30 PM, Allen Samuels >> > <Allen.Samuels@sandisk.com> wrote: >> >> I doubt that rsync is the only application that's going to be confused by the presence of attributes it doesn't understand. >> >> >> >> What is the value in having these attributes be visible to user-space applications? Is there a better mechanism for conveying this information? >> > >> > This is something I think we've talked about before, and I keep on >> > being surprised to learn we *do* list them without a specific query. >> > So, what are the upsides of listing them without being asked? >> > >> > *) Easier to view for users >> > *) If syncing between Ceph clusters with the same pool names, you can >> > transfer layouts properly (except...probably xattrs are set after data >> > is written in most systems, so that doesn't work?) >> > *) ...I've got nothing else >> > >> > Downsides: >> > *) Other tools get very confused by viewing xattrs they can't set >> > >> > Personally, I'd rather only list them when the Ceph namespace is >> > specifically queried (or the xattr itself is, at whatever granularity >> > we can do). This isn't like directory sizes; xattrs have defined >> > semantics that we're breaking, *and* exposing them all the time just >> > doesn't seem that useful. Why would anybody care about the data >> > they're sharing, who doesn't already know that it's available? >> > -Greg >> >> I'm inclined to agree that we shouldn't include ceph.* xattrs in the >> normal listing of xattrs. Created tickets: >> http://tracker.ceph.com/issues/16468 >> http://tracker.ceph.com/issues/16467 >> >> I do think it would be more correct for rsync to avoid xattrs in any >> namespace it doesn't know about though; given that xattrs are meant to >> be for potentially-filesystem-specific ones, it should assume that any >> other namespaces are non-standard and are not necessarily safe to >> copy. Realistically though, rsync probably won't be the only tool >> that ever tries to do this, so we should try to be safe against people >> running any other tools that make the same assumption that listed >> xattrs are copyable. > > Yeah. > > My main concern is that currently getfattr -d or attr -l are an > easy/convenient way for users to discover what interesting information > Ceph is exposing, without any special tools. Without this, the user has > to know what information is there, and remember the exact name of the > xattr(s). It's also one way to discover whether a quota or layout is set > on a particular directory. > > Maybe we could make a ceph.all vxattr that, when you get it, will dump a > list of key/value pairs with everything? e.g., > > $ getfattr -n ceph.all . > # file: . > ceph.all="ceph.dir.entries=14832 > ceph.dir.files=755 > ceph.dir.rbytes=18300954645291 > ceph.dir.rctime=1466771118.09752118291 > ceph.dir.rentries=22378800 > ceph.dir.rfiles=21586687 > ceph.dir.rsubdirs=792113 > ceph.dir.subdirs=14077 > " > > ? Then it's just one magic xattr to remember to discover the rest > (including anything new we add in the future). That's analogous to how we handle snapshot directories and is what I'd like to see. (Unless there's some way of requesting listing of a specific xattr namespace?) -Greg ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-06-24 13:53 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-06-21 20:34 Problems with cephfs, rsync and xattr because cephfs internal xattrs are exposed to clients Michael Wyraz 2016-06-21 21:30 ` Sage Weil 2016-06-21 22:30 ` Allen Samuels 2016-06-22 4:46 ` Michael Wyraz 2016-06-22 20:15 ` Gregory Farnum 2016-06-22 20:24 ` Allen Samuels 2016-06-24 10:08 ` John Spray 2016-06-24 12:27 ` Sage Weil 2016-06-24 13:53 ` Gregory Farnum
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.