What's the difference between `btrfs sub del -c` and `btrfs fi sync`?

What's the difference between `btrfs sub del -c` and `btrfs fi sync`?

[intelfx]

Hi,

I've been trying to read btrfs-progs code to understand btrfs ioctls
and one thing evades my understanding.

A `btrfs subvolume delete --commit-{after,each}` operation involves
issuing two ioctls at the commit time: BTRFS_IOC_START_SYNC immediately
followed by BTRFS_IOC_WAIT_SYNC. Notably, the relevant comment says
"<...> issue SYNC ioctl <...>" and the function that encapsulates the
two ioctls is called `wait_for_commit()`.

On the other hand, a `btrfs filesystem sync` operation involves issuing
just one ioctl, BTRFS_IOC_SYNC (encapsulated in a function called
`btrfs_util_sync_fd()`).

I tried to look at the kernel code for the three ioctls but to my
untrained eye, they look like they are doing different things with
different side effects.

What is the difference, and why is it needed (i.e. why are there two
sets of sync-related ioctls)?

Cheers,
-- 
Ivan Shapovalov / intelfx /

Re: What's the difference between `btrfs sub del -c` and `btrfs fi sync`?

[Qu Wenruo]

在 2024/4/27 08:22, intelfx@intelfx.name 写道:
> Hi,
>
> I've been trying to read btrfs-progs code to understand btrfs ioctls
> and one thing evades my understanding.
>
> A `btrfs subvolume delete --commit-{after,each}` operation involves
> issuing two ioctls at the commit time: BTRFS_IOC_START_SYNC immediately
> followed by BTRFS_IOC_WAIT_SYNC. Notably, the relevant comment says
> "<...> issue SYNC ioctl <...>" and the function that encapsulates the
> two ioctls is called `wait_for_commit()`.
>
> On the other hand, a `btrfs filesystem sync` operation involves issuing
> just one ioctl, BTRFS_IOC_SYNC (encapsulated in a function called
> `btrfs_util_sync_fd()`).
>
> I tried to look at the kernel code for the three ioctls but to my
> untrained eye, they look like they are doing different things with
> different side effects.
>
> What is the difference, and why is it needed (i.e. why are there two
> sets of sync-related ioctls)?

IIRC --commit-after/each only commit the current transaction, and it's
just doing the same `btrfs fi sync` after all/each subvolume deletion.

The reason is to ensure the unlinking (not fully deleting) of the target
subvolume fully committed to disk, so a sudden powerloss after the
deletion won't lead to the re-appearing of the target subvolume(s)


However there is a another behavior involved, `btrfs subvolume sync`,
which is to wait for a deleted subvolume to be fully dropped.
In the case of btrfs subvolume deletion, it can be a heavy load, thus
btrfs only unlink the to-be-deleted subvolume, and mark it for
background deletion.
`btrfs subvolume sync` would wait for any such orphan subvolume to be
deleted.

Thanks,
Qu


>
> Cheers,

Re: What's the difference between `btrfs sub del -c` and `btrfs fi sync`?

[intelfx]

On 2024-04-27 at 08:36 +0930, Qu Wenruo wrote:
> 
> 在 2024/4/27 08:22, intelfx@intelfx.name 写道:
> > Hi,
> > 
> > I've been trying to read btrfs-progs code to understand btrfs ioctls
> > and one thing evades my understanding.
> > 
> > A `btrfs subvolume delete --commit-{after,each}` operation involves
> > issuing two ioctls at the commit time: BTRFS_IOC_START_SYNC immediately
> > followed by BTRFS_IOC_WAIT_SYNC. Notably, the relevant comment says
> > "<...> issue SYNC ioctl <...>" and the function that encapsulates the
> > two ioctls is called `wait_for_commit()`.
> > 
> > On the other hand, a `btrfs filesystem sync` operation involves issuing
> > just one ioctl, BTRFS_IOC_SYNC (encapsulated in a function called
> > `btrfs_util_sync_fd()`).
> > 
> > I tried to look at the kernel code for the three ioctls but to my
> > untrained eye, they look like they are doing different things with
> > different side effects.
> > 
> > What is the difference, and why is it needed (i.e. why are there two
> > sets of sync-related ioctls)?
> 
> IIRC --commit-after/each only commit the current transaction, and it's
> just doing the same `btrfs fi sync` after all/each subvolume deletion.
> 
> The reason is to ensure the unlinking (not fully deleting) of the target
> subvolume fully committed to disk, so a sudden powerloss after the
> deletion won't lead to the re-appearing of the target subvolume(s)
> 
> 
> However there is a another behavior involved, `btrfs subvolume sync`,
> which is to wait for a deleted subvolume to be fully dropped.
> In the case of btrfs subvolume deletion, it can be a heavy load, thus
> btrfs only unlink the to-be-deleted subvolume, and mark it for
> background deletion.
> `btrfs subvolume sync` would wait for any such orphan subvolume to be
> deleted.
> 
> Thanks,
> Qu
> 
> 
> > 
> > Cheers,

Thanks for the fast reply!

Yes, I'm aware about `btrfs sub sync`. I understand that's a totally
different operation.

What I was asking about was specifically the difference between
`btrfs _filesystem_ sync` and the operation that happens at the end of
a `btrfs subvolume delete --commit-after`.

Or, in kernel terms: what exactly is the difference between issuing a
BTRFS_IOC_SYNC and issuing a BTRFS_IOC_START_SYNC immediately followed
by a BTRFS_IOC_WAIT_SYNC?

It is not immediately obvious that the kernel code for the three ioctls
is equivalent (even if it is). For instance, BTRFS_IOC_SYNC begins with
a call to btrfs_start_delalloc_roots() whereas BTRFS_IOC_START_SYNC
begins with a call to btrfs_orphan_cleanup(), and the subsequent
transaction handling code seems subtly different.

-- 
Ivan Shapovalov / intelfx /

Re: What's the difference between `btrfs sub del -c` and `btrfs fi sync`?

[Qu Wenruo]

在 2024/4/27 08:44, intelfx@intelfx.name 写道:
> On 2024-04-27 at 08:36 +0930, Qu Wenruo wrote:
>>
>> 在 2024/4/27 08:22, intelfx@intelfx.name 写道:
>>> Hi,
>>>
>>> I've been trying to read btrfs-progs code to understand btrfs ioctls
>>> and one thing evades my understanding.
>>>
>>> A `btrfs subvolume delete --commit-{after,each}` operation involves
>>> issuing two ioctls at the commit time: BTRFS_IOC_START_SYNC immediately
>>> followed by BTRFS_IOC_WAIT_SYNC. Notably, the relevant comment says
>>> "<...> issue SYNC ioctl <...>" and the function that encapsulates the
>>> two ioctls is called `wait_for_commit()`.
>>>
>>> On the other hand, a `btrfs filesystem sync` operation involves issuing
>>> just one ioctl, BTRFS_IOC_SYNC (encapsulated in a function called
>>> `btrfs_util_sync_fd()`).
>>>
>>> I tried to look at the kernel code for the three ioctls but to my
>>> untrained eye, they look like they are doing different things with
>>> different side effects.
>>>
>>> What is the difference, and why is it needed (i.e. why are there two
>>> sets of sync-related ioctls)?
>>
>> IIRC --commit-after/each only commit the current transaction, and it's
>> just doing the same `btrfs fi sync` after all/each subvolume deletion.
>>
>> The reason is to ensure the unlinking (not fully deleting) of the target
>> subvolume fully committed to disk, so a sudden powerloss after the
>> deletion won't lead to the re-appearing of the target subvolume(s)
>>
>>
>> However there is a another behavior involved, `btrfs subvolume sync`,
>> which is to wait for a deleted subvolume to be fully dropped.
>> In the case of btrfs subvolume deletion, it can be a heavy load, thus
>> btrfs only unlink the to-be-deleted subvolume, and mark it for
>> background deletion.
>> `btrfs subvolume sync` would wait for any such orphan subvolume to be
>> deleted.
>>
>> Thanks,
>> Qu
>>
>>
>>>
>>> Cheers,
> 
> Thanks for the fast reply!
> 
> Yes, I'm aware about `btrfs sub sync`. I understand that's a totally
> different operation.
> 
> What I was asking about was specifically the difference between
> `btrfs _filesystem_ sync` and the operation that happens at the end of
> a `btrfs subvolume delete --commit-after`.
> 
> Or, in kernel terms: what exactly is the difference between issuing a
> BTRFS_IOC_SYNC and issuing a BTRFS_IOC_START_SYNC immediately followed
> by a BTRFS_IOC_WAIT_SYNC?

If you go really deep, there is some small difference, but overall you 
can consider them the same, despite the START/WAIT_SYNC is an async 
operation, while IOC_SYNC would wait for it.

> 
> It is not immediately obvious that the kernel code for the three ioctls
> is equivalent (even if it is). For instance, BTRFS_IOC_SYNC begins with
> a call to btrfs_start_delalloc_roots() whereas BTRFS_IOC_START_SYNC
> begins with a call to btrfs_orphan_cleanup(), and the subsequent
> transaction handling code seems subtly different.
>
There is a small difference, but not really effect end users.

The IOC_SYNC would start and wait for the writeback of all dirty files.
(AKA, the same behavior as `sync` command).
Meanwhile IOC_START_SYNC would not trigger the writeback, just commit 
the metadata which is already dirty.

For the --commit-after/each, IOC_START_SYNC is faster, since 
IOC_SNAP_DESTORY has already dirtied the necessary metadata, we only 
need to commit the dirtied metadata in current transaction, no need to 
wait for other data writeback.

Thanks,
Qu

Re: What's the difference between `btrfs sub del -c` and `btrfs fi sync`?

[intelfx]

On 2024-04-27 at 09:34 +0930, Qu Wenruo wrote:
> 
> 在 2024/4/27 08:44, intelfx@intelfx.name 写道:
> > On 2024-04-27 at 08:36 +0930, Qu Wenruo wrote:
> > > 
> > > 在 2024/4/27 08:22, intelfx@intelfx.name 写道:
> > > > Hi,
> > > > 
> > > > I've been trying to read btrfs-progs code to understand btrfs ioctls
> > > > and one thing evades my understanding.
> > > > 
> > > > A `btrfs subvolume delete --commit-{after,each}` operation involves
> > > > issuing two ioctls at the commit time: BTRFS_IOC_START_SYNC immediately
> > > > followed by BTRFS_IOC_WAIT_SYNC. Notably, the relevant comment says
> > > > "<...> issue SYNC ioctl <...>" and the function that encapsulates the
> > > > two ioctls is called `wait_for_commit()`.
> > > > 
> > > > On the other hand, a `btrfs filesystem sync` operation involves issuing
> > > > just one ioctl, BTRFS_IOC_SYNC (encapsulated in a function called
> > > > `btrfs_util_sync_fd()`).
> > > > 
> > > > I tried to look at the kernel code for the three ioctls but to my
> > > > untrained eye, they look like they are doing different things with
> > > > different side effects.
> > > > 
> > > > What is the difference, and why is it needed (i.e. why are there two
> > > > sets of sync-related ioctls)?
> > > 
> > > IIRC --commit-after/each only commit the current transaction, and it's
> > > just doing the same `btrfs fi sync` after all/each subvolume deletion.
> > > 
> > > The reason is to ensure the unlinking (not fully deleting) of the target
> > > subvolume fully committed to disk, so a sudden powerloss after the
> > > deletion won't lead to the re-appearing of the target subvolume(s)
> > > 
> > > 
> > > However there is a another behavior involved, `btrfs subvolume sync`,
> > > which is to wait for a deleted subvolume to be fully dropped.
> > > In the case of btrfs subvolume deletion, it can be a heavy load, thus
> > > btrfs only unlink the to-be-deleted subvolume, and mark it for
> > > background deletion.
> > > `btrfs subvolume sync` would wait for any such orphan subvolume to be
> > > deleted.
> > > 
> > > Thanks,
> > > Qu
> > > 
> > > 
> > > > 
> > > > Cheers,
> > 
> > Thanks for the fast reply!
> > 
> > Yes, I'm aware about `btrfs sub sync`. I understand that's a totally
> > different operation.
> > 
> > What I was asking about was specifically the difference between
> > `btrfs _filesystem_ sync` and the operation that happens at the end of
> > a `btrfs subvolume delete --commit-after`.
> > 
> > Or, in kernel terms: what exactly is the difference between issuing a
> > BTRFS_IOC_SYNC and issuing a BTRFS_IOC_START_SYNC immediately followed
> > by a BTRFS_IOC_WAIT_SYNC?
> 
> If you go really deep, there is some small difference, but overall you 
> can consider them the same, despite the START/WAIT_SYNC is an async 
> operation, while IOC_SYNC would wait for it.
> 
> > 
> > It is not immediately obvious that the kernel code for the three ioctls
> > is equivalent (even if it is). For instance, BTRFS_IOC_SYNC begins with
> > a call to btrfs_start_delalloc_roots() whereas BTRFS_IOC_START_SYNC
> > begins with a call to btrfs_orphan_cleanup(), and the subsequent
> > transaction handling code seems subtly different.
> > 
> There is a small difference, but not really effect end users.
> 
> The IOC_SYNC would start and wait for the writeback of all dirty files.
> (AKA, the same behavior as `sync` command).
> Meanwhile IOC_START_SYNC would not trigger the writeback, just commit 
> the metadata which is already dirty.
> 
> For the --commit-after/each, IOC_START_SYNC is faster, since 
> IOC_SNAP_DESTORY has already dirtied the necessary metadata, we only 
> need to commit the dirtied metadata in current transaction, no need to 
> wait for other data writeback.

I see, thanks. That's what I suspected, but wanted to check if my
(limited) understanding was correct.

It is unfortunate that ioctls that should be equivalent if judging by
the names alone, in fact have subtle semantic differences that aren't
documented anywhere. It's as if the ioctls were introduced specifically
for the relevant userspace tools, encoding specific assumptions valid
for their (only) consumers, without consideration of their wider
applicability...

Thanks,
-- 
Ivan Shapovalov / intelfx /

Re: What's the difference between `btrfs sub del -c` and `btrfs fi sync`?

[David Sterba]

On Sun, Apr 28, 2024 at 12:20:28PM +0200, intelfx@intelfx.name wrote:
> I see, thanks. That's what I suspected, but wanted to check if my
> (limited) understanding was correct.
> 
> It is unfortunate that ioctls that should be equivalent if judging by
> the names alone, in fact have subtle semantic differences that aren't
> documented anywhere.

Ioctls get documented incrementally, at least the FS SYNC is there,
https://btrfs.readthedocs.io/en/latest/btrfs-ioctl.html#btrfs-ioc-sync

> It's as if the ioctls were introduced specifically
> for the relevant userspace tools, encoding specific assumptions valid
> for their (only) consumers, without consideration of their wider
> applicability...

The start/wait ioctls can be used for cases when there's first phase
that generates some data or changes and second phase waits for the
commit. It was convenient to use that for the subvol sync, can be used
for other changes that eg. do only btrfs_end_transaction() in kernel
while some sort of sync semantics is wanted from user space (that would
be equivalent to btrfs_commit_transaction()).

You can read the discussions behind that in
https://www.spinics.net/lists/linux-btrfs/msg28354.html

The semantics of the start/wait ioctls were explained in the initial
patch https://git.kernel.org/linus/462045928bda777c86919a396a42991fcf235378
originally added for ceph.