* yet another "out of space" on a filesystem with >100 GB free space, and strange files which exist but don't exist
@ 2017-10-04 11:13 Tomasz Chmielewski
2017-10-04 11:20 ` Austin S. Hemmelgarn
0 siblings, 1 reply; 4+ messages in thread
From: Tomasz Chmielewski @ 2017-10-04 11:13 UTC (permalink / raw)
To: linux-btrfs
Kernel: 4.13.4, btrfs RAID-1.
Disk usage more or less like below (yes, I know about btrfs fi df / show
/ usage):
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 424G 262G 161G 62% /var/lib/lxd
Balance would exit immediately with "out of space", but continues to run
after I've removed a few gigabytes from the filesystem.
Now, I'm seeing some files which exist, but don't. Strange, I know.
root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb
# ls *set
ls: cannot access 'WiredTiger.turtle.set': No such file or directory
root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb
# ls -l|grep set
ls: cannot access 'WiredTiger.turtle.set': No such file or directory
-????????? ? ? ? ? ?
WiredTiger.turtle.set
root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb
# mv WiredTiger.turtle.set WiredTiger.turtle.set.Ghost.File
mv: cannot stat 'WiredTiger.turtle.set': No such file or directory
root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb
# rm -v WiredTiger.turtle.set
rm: cannot remove 'WiredTiger.turtle.set': No such file or directory
What is this file, and why does it exist if it doesn't? How do I remove
it?
Tomasz Chmielewski
https://lxadm.com
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: yet another "out of space" on a filesystem with >100 GB free space, and strange files which exist but don't exist
2017-10-04 11:13 yet another "out of space" on a filesystem with >100 GB free space, and strange files which exist but don't exist Tomasz Chmielewski
@ 2017-10-04 11:20 ` Austin S. Hemmelgarn
2017-10-04 12:46 ` Tomasz Chmielewski
0 siblings, 1 reply; 4+ messages in thread
From: Austin S. Hemmelgarn @ 2017-10-04 11:20 UTC (permalink / raw)
To: Tomasz Chmielewski, linux-btrfs
On 2017-10-04 07:13, Tomasz Chmielewski wrote:
> Kernel: 4.13.4, btrfs RAID-1.
>
> Disk usage more or less like below (yes, I know about btrfs fi df / show
> / usage):
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 424G 262G 161G 62% /var/lib/lxd
>
>
> Balance would exit immediately with "out of space", but continues to run
> after I've removed a few gigabytes from the filesystem.
>
>
> Now, I'm seeing some files which exist, but don't. Strange, I know.
>
>
> root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb
> # ls *set
> ls: cannot access 'WiredTiger.turtle.set': No such file or directory
>
> root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb
> # ls -l|grep set
> ls: cannot access 'WiredTiger.turtle.set': No such file or directory
> -????????? ? ? ? ? ? WiredTiger.turtle.set
>
> root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb
> # mv WiredTiger.turtle.set WiredTiger.turtle.set.Ghost.File
> mv: cannot stat 'WiredTiger.turtle.set': No such file or directory
>
> root@lxd02 /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb
> # rm -v WiredTiger.turtle.set
> rm: cannot remove 'WiredTiger.turtle.set': No such file or directory
>
>
>
> What is this file, and why does it exist if it doesn't? How do I remove it?
It's got corrupted metadata, probably the inode itself (IIRC, the dentry
in BTRFS just matches the inode to the file name, and all the other data
reported by ls -l is stored in the inode). If you're running with a
replicated metadata profile (dup, raid1, or raid10), run a scrub, and it
may fix things. If not, you will likely have to run a check in repair
mode (though I would suggest waiting to hear from one of the developers
before doing so). Alternatively, if that's in a subvolume, and you can
afford to just nuke the subvolume and recreate it, deleting the
subvolume should get rid of it (though you should still run a check).
Either way, this is likely related to the balance issues you're seeing.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: yet another "out of space" on a filesystem with >100 GB free space, and strange files which exist but don't exist
2017-10-04 11:20 ` Austin S. Hemmelgarn
@ 2017-10-04 12:46 ` Tomasz Chmielewski
2017-10-04 13:25 ` Dmitrii Tcvetkov
0 siblings, 1 reply; 4+ messages in thread
From: Tomasz Chmielewski @ 2017-10-04 12:46 UTC (permalink / raw)
To: Austin S. Hemmelgarn; +Cc: linux-btrfs
On 2017-10-04 20:20, Austin S. Hemmelgarn wrote:
> On 2017-10-04 07:13, Tomasz Chmielewski wrote:
>> Kernel: 4.13.4, btrfs RAID-1.
>>
>> Disk usage more or less like below (yes, I know about btrfs fi df /
>> show / usage):
>>
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda3 424G 262G 161G 62% /var/lib/lxd
>>
>>
>> Balance would exit immediately with "out of space", but continues to
>> run after I've removed a few gigabytes from the filesystem.
>>
>>
>> Now, I'm seeing some files which exist, but don't. Strange, I know.
>>
>>
>> root@lxd02
>> /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb # ls *set
>> ls: cannot access 'WiredTiger.turtle.set': No such file or directory
>>
>> root@lxd02
>> /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb # ls
>> -l|grep set
>> ls: cannot access 'WiredTiger.turtle.set': No such file or directory
>> -????????? ? ? ? ? ?
>> WiredTiger.turtle.set
>>
>> root@lxd02
>> /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb # mv
>> WiredTiger.turtle.set WiredTiger.turtle.set.Ghost.File
>> mv: cannot stat 'WiredTiger.turtle.set': No such file or directory
>>
>> root@lxd02
>> /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb # rm -v
>> WiredTiger.turtle.set
>> rm: cannot remove 'WiredTiger.turtle.set': No such file or directory
>>
>>
>>
>> What is this file, and why does it exist if it doesn't? How do I
>> remove it?
> It's got corrupted metadata, probably the inode itself (IIRC, the
> dentry in BTRFS just matches the inode to the file name, and all the
> other data reported by ls -l is stored in the inode). If you're
> running with a replicated metadata profile (dup, raid1, or raid10),
> run a scrub, and it may fix things. If not, you will likely have to
> run a check in repair mode (though I would suggest waiting to hear
> from one of the developers before doing so). Alternatively, if that's
> in a subvolume, and you can afford to just nuke the subvolume and
> recreate it, deleting the subvolume should get rid of it (though you
> should still run a check).
>
> Either way, this is likely related to the balance issues you're seeing.
Unfortunately scrub didn't help:
# btrfs scrub status /var/lib/lxd
scrub status for 6340f5de-f635-4d09-bbb2-1e03b1e1b160
scrub started at Wed Oct 4 14:12:29 2017 and finished after
00:10:32
total bytes scrubbed: 525.57GiB with 0 errors
"Ghost file" is still there:
# ls -l
/var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb|grep set
ls: cannot access
'/var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb/WiredTiger.turtle.set':
No such file or directory
-????????? ? ? ? ? ?
WiredTiger.turtle.set
Tomasz Chmielewski
https://lxadm.com
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: yet another "out of space" on a filesystem with >100 GB free space, and strange files which exist but don't exist
2017-10-04 12:46 ` Tomasz Chmielewski
@ 2017-10-04 13:25 ` Dmitrii Tcvetkov
0 siblings, 0 replies; 4+ messages in thread
From: Dmitrii Tcvetkov @ 2017-10-04 13:25 UTC (permalink / raw)
To: Tomasz Chmielewski; +Cc: linux-btrfs
> "Ghost file" is still there:
>
> # ls -l
> /var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb|grep set
> ls: cannot access
> '/var/lib/lxd/containers/mongo-repl04b/rootfs/var/lib/mongodb/WiredTiger.turtle.set':
> No such file or directory
> -????????? ? ? ? ? ?
> WiredTiger.turtle.set
I had similiar issue couple of times, both after unclean shutdown
(power loss), only btrfs check --repair helped with the issue, but I'd
suggest to wait for someone's else input about that as I'm not a
developer.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-10-04 13:25 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-10-04 11:13 yet another "out of space" on a filesystem with >100 GB free space, and strange files which exist but don't exist Tomasz Chmielewski
2017-10-04 11:20 ` Austin S. Hemmelgarn
2017-10-04 12:46 ` Tomasz Chmielewski
2017-10-04 13:25 ` Dmitrii Tcvetkov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.