All of lore.kernel.org
 help / color / mirror / Atom feed
* partition 100% full No space left on device. looks like xfs is corrupted or a bug
@ 2016-07-29  9:01 Lista Unx
  2016-07-29 10:48 ` Carlos E. R.
                   ` (3 more replies)
  0 siblings, 4 replies; 29+ messages in thread
From: Lista Unx @ 2016-07-29  9:01 UTC (permalink / raw)
  To: xfs


[-- Attachment #1.1: Type: text/plain, Size: 7643 bytes --]

Hello xfs experts,

I am crawling in the dark from few days and I have no idea how to fix the following problem. On a centos 7 system:

# uname -a
Linux 1a 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

df is reporting 100% full of / and du is reporting only 1.7G usage from 50GB available (less than 4%). I want to mention that / is xfs. See below:

# df -a|grep ^/
/dev/mapper/centos-root  52403200 52400396      2804 100% /
                                     ^^^^^^^^^^   ^^^^^^^^^^
/dev/sda1                  503040   131876    371164  27% /boot
/dev/mapper/centos-home 210529792    35204 210494588   1% /home

du is estimating just 1.7G usage of /
# du -sch /* --exclude=home --exclude=boot
0       /bin
0       /dev
25M     /etc
0       /lib
0       /lib64
744K    /luarocks-2.3.0
0       /media
0       /mnt
125M    /openresty-1.9.7.4
0       /opt
420K    /root
49M     /run
0       /sbin
0       /srv
0       /sys
0       /tmp
1.3G    /usr
227M    /var
1.7G    total
[root@localhost ~]#

df is also reporting 80% of inode usage:

# df -i
Filesystem                 Inodes IUsed     IFree IUse% Mounted on
/dev/mapper/centos-root     78160 66218     11942   85% /
                                       ^^^^^^^^
devtmpfs                  8218272   519   8217753    1% /dev
tmpfs                     8221010     1   8221009    1% /dev/shm
tmpfs                     8221010   648   8220362    1% /run
tmpfs                     8221010    13   8220997    1% /sys/fs/cgroup
/dev/sda1                  509952   330    509622    1% /boot
/dev/mapper/centos-home 210632704    99 210632605    1% /home
tmpfs                     8221010     1   8221009    1% /run/user/0
#

/ partition is created on top of a LVM having also 50GB size.

# lvdisplay /dev/centos/root
  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos

  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

I've already checked against rootkit without finding anything wrong!

I have another system, identical with this one which is healthy. The only difference I found between those systems is regarding max number of inodes available on / (which has the same size, 50GB on booth servers). On the second one (healthy), max number of inodes are ~52 milions and not only just ~85.000 as are reported on "seek" server.

# df -i|grep ^/
/dev/mapper/centos-root  52424704 66137  52358567    1% /
                                   ^^^^^^^^^^^^^
/dev/sda1                                  509952   330    509622    1% /boot
/dev/mapper/centos-home 210632704    26 210632678    1% /home
[root@localhost ~]#

Suspected also large number of files on /. Counted total number of files and or booth servers are the same: ~180K. So no difference here.

Look to find also files larger than 100M and on booth servers and found just 1 (104M size):

find / -type f -size +100000k -exec ls -lh {} \;
#
/usr/lib/locale/locale-archive
#

Looking to find files larger than 10M, I found just ~20 on booth servers.

# find / -type f -size +10000k -exec ls -lh {} \; |wc -l
16
#

So for sure, there are NO files exhausting free space.

On booth servers, number of used inodes are identical: ~66K. Also xfs_info report is identical for booth. What is different is number of AVAILABLE inodes: 85K (on seek node) vs 52 milion (on healthy node)!!! How is possible that!!! Booth servers has the same size (50GB) for /!

#lsof -nP |grep -i delete|wc -l
0
#find /proc/*/fd -ls | grep -i dele|wc -l
0

so lsof and find does not report anything wrong (any file deleted and still open)!

reboot does not fix the problem, / remain 100% full

After reboot, on 25th July:

# df -ah|grep centos-root
/dev/mapper/centos-root   50G   50G  4.0M 100% /
#

Also max number of inodes = 67k:
# df -i
Filesystem                 Inodes IUsed     IFree IUse% Mounted on
/dev/mapper/centos-root     66960 66165       795   99% /
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devtmpfs                  8218272   519   8217753    1% /dev
tmpfs                     8221010     1   8221009    1% /dev/shm
tmpfs                     8221010   630   8220380    1% /run
tmpfs                     8221010    13   8220997    1% /sys/fs/cgroup
/dev/sda1                  509952   330    509622    1% /boot
/dev/mapper/centos-home 210632704    28 210632676    1% /home
tmpfs                     8221010     1   8221009    1% /run/user/0
#

Lets try to run intentionally xfs_grow (which normally should not produce any change)

# xfs_growfs /dev/mapper/centos-root
meta-data=/dev/mapper/centos-root isize=256    agcount=16, agsize=819136 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=13106176, imaxpct=25
         =                       sunit=64     swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 13106176 to 13107200
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#

Partition remain the same, 50GB size:
[root@nl-hvs-ov001a ~]# df -ah|grep centos-root
/dev/mapper/centos-root   50G   50G  4.0M 100% /

But number of inodes INCREASED with more tha 20%!!!
# df -i
Filesystem                 Inodes IUsed     IFree IUse% Mounted on
/dev/mapper/centos-root     83200 66165     17035   80% /
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devtmpfs                  8218272   519   8217753    1% /dev
tmpfs                     8221010     1   8221009    1% /dev/shm
tmpfs                     8221010   630   8220380    1% /run
tmpfs                     8221010    13   8220997    1% /sys/fs/cgroup
/dev/sda1                  509952   330    509622    1% /boot
/dev/mapper/centos-home 210632704    28 210632676    1% /home
tmpfs                     8221010     1   8221009    1% /run/user/0
#

On 27July without changing anything there, max number inodes available for / decreased to ~67k (the same size like 2 days ago, before xfs_grow)!

# df -i
Filesystem                 Inodes IUsed     IFree IUse% Mounted on
/dev/mapper/centos-root     67024 66225       799   99% /
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devtmpfs                  8218272   519   8217753    1% /dev
tmpfs                     8221010     1   8221009    1% /dev/shm
tmpfs                     8221010   632   8220378    1% /run
tmpfs                     8221010    13   8220997    1% /sys/fs/cgroup
/dev/mapper/centos-home 210632704    99 210632605    1% /home
/dev/sda1                  509952   330    509622    1% /boot
tmpfs                     8221010     1   8221009    1% /run/user/0
#

Please note that all that time, number of files remain unchanged ~180K, the same for inodes used, the number remain constant ~66K. Just max number of inodes available decreased which is an abnormal behavior.

How can be fixed? Looks like xfs is crrupted or like a bug.

Thanks in advance for help.
Alex

[-- Attachment #1.2: Type: text/html, Size: 18138 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 29+ messages in thread
* spam on this list
@ 2013-11-23 14:41 Pierre Frenkiel
  0 siblings, 0 replies; 29+ messages in thread
From: Pierre Frenkiel @ 2013-11-23 14:41 UTC (permalink / raw)
  To: linux-bluetooth

I'm surprised to recieve  spams rather often from this list. I'm
subscribed to a big number of lists, and it's the only for which
this occurs. Would it be possible to refuse posts from non-members ?

best regards,
-- 
Pierre Frenkiel

^ permalink raw reply	[flat|nested] 29+ messages in thread
[parent not found: <3E314A6E.2BFB8F88@plain.de>]

end of thread, other threads:[~2016-08-05  8:25 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-07-29  9:01 partition 100% full No space left on device. looks like xfs is corrupted or a bug Lista Unx
2016-07-29 10:48 ` Carlos E. R.
2016-07-29 14:27   ` partition 100% full No space left on device. looks like xfs iscorrupted " Lista Unx
2016-07-29 14:03 ` partition 100% full No space left on device. looks like xfs is corrupted " Brian Foster
2016-07-29 14:37   ` partition 100% full No space left on device. looks like xfs iscorrupted " Lista Unx
2016-07-29 15:20     ` Brian Foster
2016-07-29 21:49 ` partition 100% full No space left on device. looks like xfs is corrupted " Eric Sandeen
2016-08-01 11:24   ` partition 100% full No space left on device. looks like xfs iscorrupted " Lista Unx
2016-07-29 23:35 ` partition 100% full No space left on device. looks like xfs is corrupted " Dave Chinner
2016-08-01 12:00   ` partition 100% full No space left on device. looks like xfs iscorrupted " Lista Unx
2016-08-01 12:23     ` Carlos E. R.
2016-08-02 17:34       ` partition 100% full No space left on device. looks like xfsiscorrupted " Lista Unx
2016-08-02 17:34       ` Lista Unx
2016-08-01 16:51     ` partition 100% full No space left on device. looks like xfs iscorrupted " Chris Murphy
2016-08-02 17:58       ` partition 100% full No space left on device. looks like xfsiscorrupted " Lista Unx
2016-08-02 19:11         ` Troy McCorkell
2016-08-03 12:59     ` Spam on this list [Was: Re: partition 100% full No space left on device. looks like xfs iscorrupted or a bug] Carlos E. R.
2016-08-03 13:21       ` Martin Steigerwald
2016-08-03 13:34         ` Carlos E. R.
2016-08-03 23:15           ` Spam on this list Dave Chinner
2016-08-03 23:29             ` Darrick J. Wong
2016-08-04  0:51             ` Carlos E. R.
2016-08-04 11:34             ` Lista Unx
2016-08-04 13:40             ` Troy McCorkell
2016-08-04 15:49             ` Martin Steigerwald
2016-08-05  8:25               ` Carlos Eduardo Maiolino
  -- strict thread matches above, loose matches on Subject: below --
2013-11-23 14:41 spam " Pierre Frenkiel
     [not found] <3E314A6E.2BFB8F88@plain.de>
2005-05-19  6:23 ` Philip Edelbrock
2005-05-19  6:23 ` Jean Delvare

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.