Multiple files with the same name in one directory

Multiple files with the same name in one directory

[Martin Raiber]

Hi,

I have this in a btrfs directory. Linux kernel 5.10.16, no errors in dmesg, no scrub errors:

ls -lh
total 19G
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
-rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
...

disk_config.dat gets written to using fsync rename ( write new version to disk_config.dat.new, fsync disk_config.dat.new, then rename to disk_config.dat -- it is missing the parent directory fsync).

So far no negative consequences... (except that programs might get confused).

echo 3 > /proc/sys/vm/drop_caches doesn't help.

Regards,
Martin Raiber

Re: Multiple files with the same name in one directory

[Filipe Manana]

On Wed, Mar 10, 2021 at 5:18 PM Martin Raiber <martin@urbackup.org> wrote:
>
> Hi,
>
> I have this in a btrfs directory. Linux kernel 5.10.16, no errors in dmesg, no scrub errors:
>
> ls -lh
> total 19G
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> ...
>
> disk_config.dat gets written to using fsync rename ( write new version to disk_config.dat.new, fsync disk_config.dat.new, then rename to disk_config.dat -- it is missing the parent directory fsync).

That's interesting.

I've just tried something like the following on 5.10.15 (and 5.12-rc2):

create disk_config.dat
sync
for ((i = 0; i < 10; i++)); do
    create disk_config.dat.new
    write to disk_config.dat.new
    fsync disk_config.dat.new
    mv -f disk_config.dat.new disk_config.dat
done
<power fail>
mount fs
list directory

I only get one file with the name disk_config.dat and one file with
the name disk_config.dat.new.
File disk_config.dat has the data written at iteration 9 and
disk_config.dat.new has the data written at iteration 10 (expected).

You haven't mentioned, but I suppose you had a power failure / unclean
shutdown somewhere after an fsync, right?
Is this something you can reproduce at will?

>
> So far no negative consequences... (except that programs might get confused).
>
> echo 3 > /proc/sys/vm/drop_caches doesn't help.
>
> Regards,
> Martin Raiber
>


-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Multiple files with the same name in one directory

[Maksim Fomin]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, 10 March 2021 г., 20:15, Martin Raiber <martin@urbackup.org> wrote:

> Hi,
>
> I have this in a btrfs directory. Linux kernel 5.10.16, no errors in dmesg, no scrub errors:
>
> ls -lh
> total 19G
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> ...
>
> disk_config.dat gets written to using fsync rename ( write new version to disk_config.dat.new, fsync disk_config.dat.new, then rename to disk_config.dat -- it is missing the parent directory fsync).
>
> So far no negative consequences... (except that programs might get confused).
>
> echo 3 > /proc/sys/vm/drop_caches doesn't help.
>
> Regards,
> Martin Raiber

I can say that approx 1 month ago I also was surprised by having several files with duplicate names (listed by pcmanfm file manager and ls in console). In my case I had duplicates for several different files in one folder, not 10 copies of one file. I resolved the problem by renaming files and deleting duplicates. I decided not to report that issue because I thought I have unintentionally done something crazy. Btw, they likely occured also after power loss.

Regards,
Maxim Fomin

Re: Multiple files with the same name in one directory

[Martin Raiber]

On 11.03.2021 15:43 Filipe Manana wrote:
> On Wed, Mar 10, 2021 at 5:18 PM Martin Raiber <martin@urbackup.org> wrote:
>> Hi,
>>
>> I have this in a btrfs directory. Linux kernel 5.10.16, no errors in dmesg, no scrub errors:
>>
>> ls -lh
>> total 19G
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
>> ...
>>
>> disk_config.dat gets written to using fsync rename ( write new version to disk_config.dat.new, fsync disk_config.dat.new, then rename to disk_config.dat -- it is missing the parent directory fsync).
> That's interesting.
>
> I've just tried something like the following on 5.10.15 (and 5.12-rc2):
>
> create disk_config.dat
> sync
> for ((i = 0; i < 10; i++)); do
>     create disk_config.dat.new
>     write to disk_config.dat.new
>     fsync disk_config.dat.new
>     mv -f disk_config.dat.new disk_config.dat
> done
> <power fail>
> mount fs
> list directory
>
> I only get one file with the name disk_config.dat and one file with
> the name disk_config.dat.new.
> File disk_config.dat has the data written at iteration 9 and
> disk_config.dat.new has the data written at iteration 10 (expected).
>
> You haven't mentioned, but I suppose you had a power failure / unclean
> shutdown somewhere after an fsync, right?
> Is this something you can reproduce at will?

I think I rebooted via "echo b > /proc/sysrq-trigger". But at that point it probably didn't write to disk_config.dat anymore (for more than the commit interval). I'm also not sure about the delay of me noticing those multiple files (since it doesn't cause any problems) -- can't reproduce.

This is the same machine and file system with ENOSPC in btrfs_async_reclaim_metadata_space -> flush_space -> btrfs_run_delayed_refs. Could be that something went wrong with the error handling/remount-ro w.r.t. to the tree log?

>
>> So far no negative consequences... (except that programs might get confused).
>>
>> echo 3 > /proc/sys/vm/drop_caches doesn't help.
>>
>> Regards,
>> Martin Raiber

Re: Multiple files with the same name in one directory

[Filipe Manana]

On Thu, Mar 11, 2021 at 6:05 PM Martin Raiber <martin@urbackup.org> wrote:
>
> On 11.03.2021 15:43 Filipe Manana wrote:
> > On Wed, Mar 10, 2021 at 5:18 PM Martin Raiber <martin@urbackup.org> wrote:
> >> Hi,
> >>
> >> I have this in a btrfs directory. Linux kernel 5.10.16, no errors in dmesg, no scrub errors:
> >>
> >> ls -lh
> >> total 19G
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> >> ...
> >>
> >> disk_config.dat gets written to using fsync rename ( write new version to disk_config.dat.new, fsync disk_config.dat.new, then rename to disk_config.dat -- it is missing the parent directory fsync).
> > That's interesting.
> >
> > I've just tried something like the following on 5.10.15 (and 5.12-rc2):
> >
> > create disk_config.dat
> > sync
> > for ((i = 0; i < 10; i++)); do
> >     create disk_config.dat.new
> >     write to disk_config.dat.new
> >     fsync disk_config.dat.new
> >     mv -f disk_config.dat.new disk_config.dat
> > done
> > <power fail>
> > mount fs
> > list directory
> >
> > I only get one file with the name disk_config.dat and one file with
> > the name disk_config.dat.new.
> > File disk_config.dat has the data written at iteration 9 and
> > disk_config.dat.new has the data written at iteration 10 (expected).
> >
> > You haven't mentioned, but I suppose you had a power failure / unclean
> > shutdown somewhere after an fsync, right?
> > Is this something you can reproduce at will?
>
> I think I rebooted via "echo b > /proc/sysrq-trigger". But at that point it probably didn't write to disk_config.dat anymore (for more than the commit interval). I'm also not sure about the delay of me noticing those multiple files (since it doesn't cause any problems) -- can't reproduce.
>
> This is the same machine and file system with ENOSPC in btrfs_async_reclaim_metadata_space -> flush_space -> btrfs_run_delayed_refs. Could be that something went wrong with the error handling/remount-ro w.r.t. to the tree log?

It could be a bug between aborting a transaction (the error
handling/remount ro) and an fsync in progress yes.
I'll have to look at that and see if it's possible.

Thanks for the report.


>
> >
> >> So far no negative consequences... (except that programs might get confused).
> >>
> >> echo 3 > /proc/sys/vm/drop_caches doesn't help.
> >>
> >> Regards,
> >> Martin Raiber
>
>


-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: Multiple files with the same name in one directory

[Zygo Blaxell]

On Thu, Mar 11, 2021 at 02:43:26PM +0000, Filipe Manana wrote:
> On Wed, Mar 10, 2021 at 5:18 PM Martin Raiber <martin@urbackup.org> wrote:
> >
> > Hi,
> >
> > I have this in a btrfs directory. Linux kernel 5.10.16, no errors in dmesg, no scrub errors:
> >
> > ls -lh
> > total 19G
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > -rwxr-x--- 1 root     root      783 Mar 10 14:56 disk_config.dat
> > ...
> >
> > disk_config.dat gets written to using fsync rename ( write new version to disk_config.dat.new, fsync disk_config.dat.new, then rename to disk_config.dat -- it is missing the parent directory fsync).
> 
> That's interesting.
> 
> I've just tried something like the following on 5.10.15 (and 5.12-rc2):
> 
> create disk_config.dat
> sync
> for ((i = 0; i < 10; i++)); do
>     create disk_config.dat.new
>     write to disk_config.dat.new
>     fsync disk_config.dat.new
>     mv -f disk_config.dat.new disk_config.dat
> done
> <power fail>
> mount fs
> list directory
> 
> I only get one file with the name disk_config.dat and one file with
> the name disk_config.dat.new.
> File disk_config.dat has the data written at iteration 9 and
> disk_config.dat.new has the data written at iteration 10 (expected).
> 
> You haven't mentioned, but I suppose you had a power failure / unclean
> shutdown somewhere after an fsync, right?
> Is this something you can reproduce at will?

I've seen this off and on as far back as kernel 4.4.  Usually
(only?) happens on machines with metadata-heavy write loads at the
time of the fsync.  Very easy to reproduce if you hit a transaction
deadlock bug (which happened to be a lot more common a few years ago),
these inodeless filenames would show up almost every time if fsync was
called before rename.

Somewhere between 4.9 and 4.14, it became possible to create new files
with the same names as the inodeless filenames, and the bug became much
less annoying (the filenames would even go away by themselves sometimes).
Then the bug became harder to spot--earler versions of the bug would
stop an application dead with a filename that could never be used again.
Later versions didn't break anything, so they could only be found by
looking for errors when running e.g. 'find' or 'rsync'.

The issue never got near the top of my bug list, and until today I thought
it had been quietly fixed before 5.1.  I guess it's still active, or a
new bug has replaced it.

The last instance I recorded of this issue was found on a filesystem in
late 2019, although I don't know which kernel was running at the time
the inodeless filename was created (could have been anything between
4.4 and 5.0, though probably not before 4.14 because I would have been
forced to burn the subvol if it happened on an earlier kernel).

> > So far no negative consequences... (except that programs might get confused).
> >
> > echo 3 > /proc/sys/vm/drop_caches doesn't help.
> >
> > Regards,
> > Martin Raiber
> >
> 
> 
> -- 
> Filipe David Manana,
> 
> “Whether you think you can, or you think you can't — you're right.”