An question for FICLONERANGE ioctl

An question for FICLONERANGE ioctl

[Sidong Yang]

Hi, All.
I've tried to handle btrfs-progs issue.
(https://github.com/kdave/btrfs-progs/issues/396)

And I tested some code like below.

src_fd = open(src_path, O_RDONLY);
if (src_fd < 0) {
	error("cannot open src path %s", src_path);
	return 1;
}

dest_fd = open(dest_path, O_WRONLY|O_CREAT, 0666);
if (dest_fd < 0) {
    close(src_fd);
    error("cannot open dest path %s", dest_path);
    return 1;
}

range.src_fd = src_fd;
range.src_offset = src_offset;
range.src_length = length;
range.dest_offset = dest_offset;

ret = ioctl(dest_fd, FICLONERANGE, &range);

And this ioctl call failed with error code invalid arguments when length!=0.
I tried to understand FICLONERANGE man page but I think there is no clue
about this. I traced kernel code and found out it goes fail in
generic_remap_checks(). There is an condition checks if req_count is
correct size and it makes test code fails. 

I don't know about this condition but it seems that it can be passed for
setting REMAP_FILE_CAN_SHORTEN. Is there any way to setting remap_flags
in FICLONERANGE ioctl call?

Also it would be pleased that if you provide some documentation about
this.

Sorry for writing without thinking deeply.

Thanks,
Sidong

Re: An question for FICLONERANGE ioctl

[Qu Wenruo]

On 2021/9/5 下午8:14, Sidong Yang wrote:
> Hi, All.
> I've tried to handle btrfs-progs issue.
> (https://github.com/kdave/btrfs-progs/issues/396)
>
> And I tested some code like below.
>
> src_fd = open(src_path, O_RDONLY);
> if (src_fd < 0) {
> 	error("cannot open src path %s", src_path);
> 	return 1;
> }
>
> dest_fd = open(dest_path, O_WRONLY|O_CREAT, 0666);
> if (dest_fd < 0) {
>      close(src_fd);
>      error("cannot open dest path %s", dest_path);
>      return 1;
> }
>
> range.src_fd = src_fd;
> range.src_offset = src_offset;
> range.src_length = length;
> range.dest_offset = dest_offset;

Mind to give an example of the value?

One quick hint to the invalid arguments is:

- Range alignment
   The src/dst offset must be aligned to the block size of the
   filesystem.
   For btrfs, the sectorsize is currently the same as page size,
   thus both src/dest and length must be aligned to 4K for x86.

Thus a more detailed example can be much better for us to understand the
problem.

Thanks,
Qu
>
> ret = ioctl(dest_fd, FICLONERANGE, &range);
>
> And this ioctl call failed with error code invalid arguments when length!=0.
> I tried to understand FICLONERANGE man page but I think there is no clue
> about this. I traced kernel code and found out it goes fail in
> generic_remap_checks(). There is an condition checks if req_count is
> correct size and it makes test code fails.
>
> I don't know about this condition but it seems that it can be passed for
> setting REMAP_FILE_CAN_SHORTEN. Is there any way to setting remap_flags
> in FICLONERANGE ioctl call?
>
> Also it would be pleased that if you provide some documentation about
> this.
>
> Sorry for writing without thinking deeply.
>
> Thanks,
> Sidong
>

Re: An question for FICLONERANGE ioctl

[Sidong Yang]

On Mon, Sep 06, 2021 at 09:13:06AM +0800, Qu Wenruo wrote:
> 
> 
> On 2021/9/5 下午8:14, Sidong Yang wrote:
> > Hi, All.
> > I've tried to handle btrfs-progs issue.
> > (https://github.com/kdave/btrfs-progs/issues/396)
> > 
> > And I tested some code like below.
> > 
> > src_fd = open(src_path, O_RDONLY);
> > if (src_fd < 0) {
> > 	error("cannot open src path %s", src_path);
> > 	return 1;
> > }
> > 
> > dest_fd = open(dest_path, O_WRONLY|O_CREAT, 0666);
> > if (dest_fd < 0) {
> >      close(src_fd);
> >      error("cannot open dest path %s", dest_path);
> >      return 1;
> > }
> > 
> > range.src_fd = src_fd;
> > range.src_offset = src_offset;
> > range.src_length = length;
> > range.dest_offset = dest_offset;
> 
> Mind to give an example of the value?
It was src_offset = 0, src_length = 10, dest_offset = 0.

> 
> One quick hint to the invalid arguments is:
> 
> - Range alignment
>   The src/dst offset must be aligned to the block size of the
>   filesystem.
>   For btrfs, the sectorsize is currently the same as page size,
>   thus both src/dest and length must be aligned to 4K for x86.

I think it's because of this. I set too small value. It works with
length 4K. If reflink cmd in btrfs-progs exists, Users should set the
length aligned?

Thanks,
Sidong
> 
> Thus a more detailed example can be much better for us to understand the
> problem.
> 
> Thanks,
> Qu
> > 
> > ret = ioctl(dest_fd, FICLONERANGE, &range);
> > 
> > And this ioctl call failed with error code invalid arguments when length!=0.
> > I tried to understand FICLONERANGE man page but I think there is no clue
> > about this. I traced kernel code and found out it goes fail in
> > generic_remap_checks(). There is an condition checks if req_count is
> > correct size and it makes test code fails.
> > 
> > I don't know about this condition but it seems that it can be passed for
> > setting REMAP_FILE_CAN_SHORTEN. Is there any way to setting remap_flags
> > in FICLONERANGE ioctl call?
> > 
> > Also it would be pleased that if you provide some documentation about
> > this.
> > 
> > Sorry for writing without thinking deeply.
> > 
> > Thanks,
> > Sidong
> > 

Re: An question for FICLONERANGE ioctl

[Qu Wenruo]

On 2021/9/6 下午1:57, Sidong Yang wrote:
> On Mon, Sep 06, 2021 at 09:13:06AM +0800, Qu Wenruo wrote:
>>
>>
>> On 2021/9/5 下午8:14, Sidong Yang wrote:
>>> Hi, All.
>>> I've tried to handle btrfs-progs issue.
>>> (https://github.com/kdave/btrfs-progs/issues/396)
>>>
>>> And I tested some code like below.
>>>
>>> src_fd = open(src_path, O_RDONLY);
>>> if (src_fd < 0) {
>>> 	error("cannot open src path %s", src_path);
>>> 	return 1;
>>> }
>>>
>>> dest_fd = open(dest_path, O_WRONLY|O_CREAT, 0666);
>>> if (dest_fd < 0) {
>>>       close(src_fd);
>>>       error("cannot open dest path %s", dest_path);
>>>       return 1;
>>> }
>>>
>>> range.src_fd = src_fd;
>>> range.src_offset = src_offset;
>>> range.src_length = length;
>>> range.dest_offset = dest_offset;
>>
>> Mind to give an example of the value?
> It was src_offset = 0, src_length = 10, dest_offset = 0.
Oh, that's the case.

>
>>
>> One quick hint to the invalid arguments is:
>>
>> - Range alignment
>>    The src/dst offset must be aligned to the block size of the
>>    filesystem.
>>    For btrfs, the sectorsize is currently the same as page size,
>>    thus both src/dest and length must be aligned to 4K for x86.
>
> I think it's because of this. I set too small value. It works with
> length 4K. If reflink cmd in btrfs-progs exists, Users should set the
> length aligned?

Reflink/clone/dedupe all work based on block size.

Currently all these major files systems (ext*/xfs/btrfs/f2fs) in linux
are block filesystems, which means block size is their minimal unit of
read/write.

For data stored on-disk, they are all aligned to block size and smaller
ranges are padded to meet block alignment.

So is such ioctls (and things like direct IO).

Thus no matter what the user-space wrapper is, the kernel ioctl still
requires block size alignment.

Thanks,
Qu

>
> Thanks,
> Sidong
>>
>> Thus a more detailed example can be much better for us to understand the
>> problem.
>>
>> Thanks,
>> Qu
>>>
>>> ret = ioctl(dest_fd, FICLONERANGE, &range);
>>>
>>> And this ioctl call failed with error code invalid arguments when length!=0.
>>> I tried to understand FICLONERANGE man page but I think there is no clue
>>> about this. I traced kernel code and found out it goes fail in
>>> generic_remap_checks(). There is an condition checks if req_count is
>>> correct size and it makes test code fails.
>>>
>>> I don't know about this condition but it seems that it can be passed for
>>> setting REMAP_FILE_CAN_SHORTEN. Is there any way to setting remap_flags
>>> in FICLONERANGE ioctl call?
>>>
>>> Also it would be pleased that if you provide some documentation about
>>> this.
>>>
>>> Sorry for writing without thinking deeply.
>>>
>>> Thanks,
>>> Sidong
>>>
>

Re: An question for FICLONERANGE ioctl

[Sidong Yang]

On Mon, Sep 06, 2021 at 02:30:21PM +0800, Qu Wenruo wrote:
> 
> 
> On 2021/9/6 下午1:57, Sidong Yang wrote:
> > On Mon, Sep 06, 2021 at 09:13:06AM +0800, Qu Wenruo wrote:
> > > 
> > > 
> > > On 2021/9/5 下午8:14, Sidong Yang wrote:
> > > > Hi, All.
> > > > I've tried to handle btrfs-progs issue.
> > > > (https://github.com/kdave/btrfs-progs/issues/396)
> > > > 
> > > > And I tested some code like below.
> > > > 
> > > > src_fd = open(src_path, O_RDONLY);
> > > > if (src_fd < 0) {
> > > > 	error("cannot open src path %s", src_path);
> > > > 	return 1;
> > > > }
> > > > 
> > > > dest_fd = open(dest_path, O_WRONLY|O_CREAT, 0666);
> > > > if (dest_fd < 0) {
> > > >       close(src_fd);
> > > >       error("cannot open dest path %s", dest_path);
> > > >       return 1;
> > > > }
> > > > 
> > > > range.src_fd = src_fd;
> > > > range.src_offset = src_offset;
> > > > range.src_length = length;
> > > > range.dest_offset = dest_offset;
> > > 
> > > Mind to give an example of the value?
> > It was src_offset = 0, src_length = 10, dest_offset = 0.
> Oh, that's the case.
> 
> > 
> > > 
> > > One quick hint to the invalid arguments is:
> > > 
> > > - Range alignment
> > >    The src/dst offset must be aligned to the block size of the
> > >    filesystem.
> > >    For btrfs, the sectorsize is currently the same as page size,
> > >    thus both src/dest and length must be aligned to 4K for x86.
> > 
> > I think it's because of this. I set too small value. It works with
> > length 4K. If reflink cmd in btrfs-progs exists, Users should set the
> > length aligned?
> 
> Reflink/clone/dedupe all work based on block size.
> 
> Currently all these major files systems (ext*/xfs/btrfs/f2fs) in linux
> are block filesystems, which means block size is their minimal unit of
> read/write.
> 
> For data stored on-disk, they are all aligned to block size and smaller
> ranges are padded to meet block alignment.
> 
> So is such ioctls (and things like direct IO).
> 
> Thus no matter what the user-space wrapper is, the kernel ioctl still
> requires block size alignment.

I understood that all datas are aligned to block size.
Thanks a lot for detailed description!

Thanks,
Sidong
> 
> Thanks,
> Qu
> 
> > 
> > Thanks,
> > Sidong
> > > 
> > > Thus a more detailed example can be much better for us to understand the
> > > problem.
> > > 
> > > Thanks,
> > > Qu
> > > > 
> > > > ret = ioctl(dest_fd, FICLONERANGE, &range);
> > > > 
> > > > And this ioctl call failed with error code invalid arguments when length!=0.
> > > > I tried to understand FICLONERANGE man page but I think there is no clue
> > > > about this. I traced kernel code and found out it goes fail in
> > > > generic_remap_checks(). There is an condition checks if req_count is
> > > > correct size and it makes test code fails.
> > > > 
> > > > I don't know about this condition but it seems that it can be passed for
> > > > setting REMAP_FILE_CAN_SHORTEN. Is there any way to setting remap_flags
> > > > in FICLONERANGE ioctl call?
> > > > 
> > > > Also it would be pleased that if you provide some documentation about
> > > > this.
> > > > 
> > > > Sorry for writing without thinking deeply.
> > > > 
> > > > Thanks,
> > > > Sidong
> > > > 
> >