XFS fiemap issue with Linux 3.10.0-327.el7.x86_64 (CentOS 7)

XFS fiemap issue with Linux 3.10.0-327.el7.x86_64 (CentOS 7)

[Jim Wilcoxson]

I'm developing a backup program, HashBackup, and recently added sparse
"hole skipping" using fiemap.  Today I ran into a weird issue with xfs
on Linux 3.10.

A test program creates a sparse file with 10000*(4K data, 4M hole).
HB calls fiemap with start=0, length=0xFFFF...FFFF, mapped_extents=0
to get the number of extents.    Fiemap is coming back with
mapped_extents=1364 instead of 10000.

The fiemap.txt file says:

"If fm_extent_count is zero, then the fm_extents[] array is ignored
(no extents will be returned), and the fm_mapped_extents count will
hold the number of extents needed in fm_extents[] to hold the file's
current mapping."

It doesn't say the filesystem can choose to return fewer extents if it
wants, but maybe xfs interprets it this way and fiemap has to be
called in a loop until the extent_last flag is set.  If that's the
case, fiemap.txt should be updated.

Thanks,
Jim
(not on the xfs mailing list)
-- 
HashBackup: easy onsite and offsite Unix backup
http://www.hashbackup.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS fiemap issue with Linux 3.10.0-327.el7.x86_64 (CentOS 7)

[Eric Sandeen]

On 2/15/16 11:26 AM, Jim Wilcoxson wrote:
> I'm developing a backup program, HashBackup, and recently added sparse
> "hole skipping" using fiemap.  

That is probably not safe.  Mapping can change immediately after fiemap
returns, and you could miss data.  cp went down that path a few years ago,
and it caused pain.

SEEK_HOLE/SEEK_DATA would be a better choice.

> Today I ran into a weird issue with xfs
> on Linux 3.10.
> 
> A test program creates a sparse file with 10000*(4K data, 4M hole).
> HB calls fiemap with start=0, length=0xFFFF...FFFF, mapped_extents=0
> to get the number of extents.    Fiemap is coming back with
> mapped_extents=1364 instead of 10000.

Take a look at xfs_bmap -v output to get a clear picture of what
is actually on disk.  It's possible that some combination of
speculative preallocation or other things has merged extents.

In general, the kernel has plenty of leeway in terms of how it
chooses to lay out a file.  "seek/write" doesn't necessarily
guarantee a hole.

> The fiemap.txt file says:
> 
> "If fm_extent_count is zero, then the fm_extents[] array is ignored
> (no extents will be returned), and the fm_mapped_extents count will
> hold the number of extents needed in fm_extents[] to hold the file's
> current mapping."
> 
> It doesn't say the filesystem can choose to return fewer extents if it
> wants, but maybe xfs interprets it this way and fiemap has to be
> called in a loop until the extent_last flag is set.  If that's the
> case, fiemap.txt should be updated.

It's not clear that it is returning fewer extents.  See what xfs_bmap
says, and compare.

But also, don't use fiemap results for anything that requires
data integrity...

-Eric

> Thanks,
> Jim
> (not on the xfs mailing list)
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS fiemap issue with Linux 3.10.0-327.el7.x86_64 (CentOS 7)

[Jim Wilcoxson]

Thanks Eric.

I ran xfs_bmap -v and it returned extents 0-19999, alternating data
with holes.  The holes and data were various sizes, I suppose for xfs
alignment reasons, but everything was there.

Running fiemap again after xfs_bmap still returned 1364 extents.
Fiemap only returns data extents, with holes implied, and comparing
this to a seek_data/seek_hole map, they are identical as far as they
go.  The fiemap just poops out early, around the 5GB mark as I recall,
but the file is 41GB.  It appears broken to me.

I appreciate the advice about things changing after the fiemap call.
That's true of any backup that isn't of a snapshot, so it's usually a
crap shoot in that regard.

Jim

On 2/15/16, Eric Sandeen <sandeen@sandeen.net> wrote:
> On 2/15/16 11:26 AM, Jim Wilcoxson wrote:
>> I'm developing a backup program, HashBackup, and recently added sparse
>> "hole skipping" using fiemap.
>
> That is probably not safe.  Mapping can change immediately after fiemap
> returns, and you could miss data.  cp went down that path a few years ago,
> and it caused pain.
>
> SEEK_HOLE/SEEK_DATA would be a better choice.
>
>> Today I ran into a weird issue with xfs
>> on Linux 3.10.
>>
>> A test program creates a sparse file with 10000*(4K data, 4M hole).
>> HB calls fiemap with start=0, length=0xFFFF...FFFF, mapped_extents=0
>> to get the number of extents.    Fiemap is coming back with
>> mapped_extents=1364 instead of 10000.
>
> Take a look at xfs_bmap -v output to get a clear picture of what
> is actually on disk.  It's possible that some combination of
> speculative preallocation or other things has merged extents.
>
> In general, the kernel has plenty of leeway in terms of how it
> chooses to lay out a file.  "seek/write" doesn't necessarily
> guarantee a hole.
>
>> The fiemap.txt file says:
>>
>> "If fm_extent_count is zero, then the fm_extents[] array is ignored
>> (no extents will be returned), and the fm_mapped_extents count will
>> hold the number of extents needed in fm_extents[] to hold the file's
>> current mapping."
>>
>> It doesn't say the filesystem can choose to return fewer extents if it
>> wants, but maybe xfs interprets it this way and fiemap has to be
>> called in a loop until the extent_last flag is set.  If that's the
>> case, fiemap.txt should be updated.
>
> It's not clear that it is returning fewer extents.  See what xfs_bmap
> says, and compare.
>
> But also, don't use fiemap results for anything that requires
> data integrity...
>
> -Eric
>
>> Thanks,
>> Jim
>> (not on the xfs mailing list)

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS fiemap issue with Linux 3.10.0-327.el7.x86_64 (CentOS 7)

[Dave Chinner]

On Mon, Feb 15, 2016 at 02:28:36PM -0500, Jim Wilcoxson wrote:
> Thanks Eric.
> 
> I ran xfs_bmap -v and it returned extents 0-19999, alternating data
> with holes.  The holes and data were various sizes, I suppose for xfs
> alignment reasons, but everything was there.
> 
> Running fiemap again after xfs_bmap still returned 1364 extents.

Yes, fiemap in XFS uses a buffer size of:

        bm.bmv_count = min_t(__s32, bm.bmv_count,
                             (PAGE_SIZE * 16 / sizeof(struct getbmapx)));

i.e. limits a single fiemap fetch to a maximum of 64k of extent
data.

I think you have an incorrect assumption about fiemap behaviour.
fiemap is not designed to return or even count all the extents in a
file in a single call; on XFS it returns how many extents it can
return in a single call. When you then map the file, if the
FIEMAP_EXTENT_LAST is not set on the last extent returned, then the
application needs to make another FIEMAP call from the end offset of
last extent mapping returned in the last call, and it will then
return the next N extents in the file.

IOWs, you have to keep calling FIEMAP to map the entire file, not
assume a single call will return an arbitrary amount of data to you
in a single call.

But, as Eric said - fiemap is a diagnosis tool, not an interface you
can rely on for anything involving data movement.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS fiemap issue with Linux 3.10.0-327.el7.x86_64 (CentOS 7)

[Eric Sandeen]

On 2/15/16 2:40 PM, Dave Chinner wrote:
> On Mon, Feb 15, 2016 at 02:28:36PM -0500, Jim Wilcoxson wrote:
>> Thanks Eric.
>>
>> I ran xfs_bmap -v and it returned extents 0-19999, alternating data
>> with holes.  The holes and data were various sizes, I suppose for xfs
>> alignment reasons, but everything was there.
>>
>> Running fiemap again after xfs_bmap still returned 1364 extents.
> 
> Yes, fiemap in XFS uses a buffer size of:
> 
>         bm.bmv_count = min_t(__s32, bm.bmv_count,
>                              (PAGE_SIZE * 16 / sizeof(struct getbmapx)));
> 
> i.e. limits a single fiemap fetch to a maximum of 64k of extent
> data.
> 
> I think you have an incorrect assumption about fiemap behaviour.
> fiemap is not designed to return or even count all the extents in a
> file in a single call; 

I think it is; as was quoted earlier,

"If fm_extent_count is zero, then the
fm_extents[] array is ignored (no extents will be returned), and the
fm_mapped_extents count will hold the number of extents needed in
fm_extents[] to hold the file's current mapping."

and that's in there:

        /* only count the extents */
        if (fieinfo->fi_extents_max == 0) {
                fieinfo->fi_extents_mapped++;
                return (flags & FIEMAP_EXTENT_LAST) ? 1 : 0;
        }

> on XFS it returns how many extents it can
> return in a single call. When you then map the file, if the
> FIEMAP_EXTENT_LAST is not set on the last extent returned, then the
> application needs to make another FIEMAP call from the end offset of
> last extent mapping returned in the last call, and it will then
> return the next N extents in the file.
> 
> IOWs, you have to keep calling FIEMAP to map the entire file, not
> assume a single call will return an arbitrary amount of data to you
> in a single call.

He's not trying to get all data in one call, just a count.

-Eric

> But, as Eric said - fiemap is a diagnosis tool, not an interface you
> can rely on for anything involving data movement.
> 
> Cheers,
> 
> Dave.
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS fiemap issue with Linux 3.10.0-327.el7.x86_64 (CentOS 7)

[Dave Chinner]

On Mon, Feb 15, 2016 at 04:47:16PM -0600, Eric Sandeen wrote:
> 
> 
> On 2/15/16 2:40 PM, Dave Chinner wrote:
> > On Mon, Feb 15, 2016 at 02:28:36PM -0500, Jim Wilcoxson wrote:
> >> Thanks Eric.
> >>
> >> I ran xfs_bmap -v and it returned extents 0-19999, alternating data
> >> with holes.  The holes and data were various sizes, I suppose for xfs
> >> alignment reasons, but everything was there.
> >>
> >> Running fiemap again after xfs_bmap still returned 1364 extents.
> > 
> > Yes, fiemap in XFS uses a buffer size of:
> > 
> >         bm.bmv_count = min_t(__s32, bm.bmv_count,
> >                              (PAGE_SIZE * 16 / sizeof(struct getbmapx)));
> > 
> > i.e. limits a single fiemap fetch to a maximum of 64k of extent
> > data.
> > 
> > I think you have an incorrect assumption about fiemap behaviour.
> > fiemap is not designed to return or even count all the extents in a
> > file in a single call; 
> 
> I think it is; as was quoted earlier,
> 
> "If fm_extent_count is zero, then the
> fm_extents[] array is ignored (no extents will be returned), and the
> fm_mapped_extents count will hold the number of extents needed in
> fm_extents[] to hold the file's current mapping."
> 
> and that's in there:
> 
>         /* only count the extents */
>         if (fieinfo->fi_extents_max == 0) {
>                 fieinfo->fi_extents_mapped++;
>                 return (flags & FIEMAP_EXTENT_LAST) ? 1 : 0;
>         }

Right, but I think we only pass one /internal/ buffer's extents to
the fill function. When we run out of extents to process (i.e we hit
the limit in the above extent count passed to xfs_getbmapx), it
returns to userspace.

This isn't a problem when iterating extent lists, because of the
FIEMAP_EXTENT_LAST is exported to userspace. The problem appears to
be that counting has different semantics, and I bet nobody wrote a
regression test that covered this case....

> > on XFS it returns how many extents it can
> > return in a single call. When you then map the file, if the
> > FIEMAP_EXTENT_LAST is not set on the last extent returned, then the
> > application needs to make another FIEMAP call from the end offset of
> > last extent mapping returned in the last call, and it will then
> > return the next N extents in the file.
> > 
> > IOWs, you have to keep calling FIEMAP to map the entire file, not
> > assume a single call will return an arbitrary amount of data to you
> > in a single call.
> 
> He's not trying to get all data in one call, just a count.

I was under the impression that counting was a ranged operation,
too. i.e. you can ask for the number of extents within a certain
file offset. Maybe we have got counting wrong as FIEMAP_EXTENT_LAST
is never exposed to userspace, nor does the fiemap code record the
offset we counted up to, so it would be hard to iterate.

Patches to fix the kernel, to support extent counting in xfs_io
and a regression test, please!

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS fiemap issue with Linux 3.10.0-327.el7.x86_64 (CentOS 7)

[Eric Sandeen]

On 2/15/16 6:20 PM, Dave Chinner wrote:
> On Mon, Feb 15, 2016 at 04:47:16PM -0600, Eric Sandeen wrote:
>>
>>
>> On 2/15/16 2:40 PM, Dave Chinner wrote:
>>> On Mon, Feb 15, 2016 at 02:28:36PM -0500, Jim Wilcoxson wrote:
>>>> Thanks Eric.
>>>>
>>>> I ran xfs_bmap -v and it returned extents 0-19999, alternating data
>>>> with holes.  The holes and data were various sizes, I suppose for xfs
>>>> alignment reasons, but everything was there.
>>>>
>>>> Running fiemap again after xfs_bmap still returned 1364 extents.
>>>
>>> Yes, fiemap in XFS uses a buffer size of:
>>>
>>>         bm.bmv_count = min_t(__s32, bm.bmv_count,
>>>                              (PAGE_SIZE * 16 / sizeof(struct getbmapx)));
>>>
>>> i.e. limits a single fiemap fetch to a maximum of 64k of extent
>>> data.
>>>
>>> I think you have an incorrect assumption about fiemap behaviour.
>>> fiemap is not designed to return or even count all the extents in a
>>> file in a single call; 
>>
>> I think it is; as was quoted earlier,
>>
>> "If fm_extent_count is zero, then the
>> fm_extents[] array is ignored (no extents will be returned), and the
>> fm_mapped_extents count will hold the number of extents needed in
>> fm_extents[] to hold the file's current mapping."
>>
>> and that's in there:
>>
>>         /* only count the extents */
>>         if (fieinfo->fi_extents_max == 0) {
>>                 fieinfo->fi_extents_mapped++;
>>                 return (flags & FIEMAP_EXTENT_LAST) ? 1 : 0;
>>         }
> 
> Right, but I think we only pass one /internal/ buffer's extents to
> the fill function. When we run out of extents to process (i.e we hit
> the limit in the above extent count passed to xfs_getbmapx), it
> returns to userspace.
> 
> This isn't a problem when iterating extent lists, because of the
> FIEMAP_EXTENT_LAST is exported to userspace. The problem appears to
> be that counting has different semantics, and I bet nobody wrote a
> regression test that covered this case....

Yeah, I looked at xfs_io's fiemap, and there's no way to pass it
a count-only set of options.

>>> on XFS it returns how many extents it can
>>> return in a single call. When you then map the file, if the
>>> FIEMAP_EXTENT_LAST is not set on the last extent returned, then the
>>> application needs to make another FIEMAP call from the end offset of
>>> last extent mapping returned in the last call, and it will then
>>> return the next N extents in the file.
>>>
>>> IOWs, you have to keep calling FIEMAP to map the entire file, not
>>> assume a single call will return an arbitrary amount of data to you
>>> in a single call.
>>
>> He's not trying to get all data in one call, just a count.
> 
> I was under the impression that counting was a ranged operation,
> too. i.e. you can ask for the number of extents within a certain
> file offset. Maybe we have got counting wrong as FIEMAP_EXTENT_LAST
> is never exposed to userspace, nor does the fiemap code record the
> offset we counted up to, so it would be hard to iterate.
> 
> Patches to fix the kernel, to support extent counting in xfs_io
> and a regression test, please!

Yep....

-Eric

> Cheers,
> 
> Dave.
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs