Hello xfs experts, I am crawling in the dark from few days and I have no idea how to fix the following problem. On a centos 7 system: # uname -a Linux 1a 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux df is reporting 100% full of / and du is reporting only 1.7G usage from 50GB available (less than 4%). I want to mention that / is xfs. See below: # df -a|grep ^/ /dev/mapper/centos-root 52403200 52400396 2804 100% / ^^^^^^^^^^ ^^^^^^^^^^ /dev/sda1 503040 131876 371164 27% /boot /dev/mapper/centos-home 210529792 35204 210494588 1% /home du is estimating just 1.7G usage of / # du -sch /* --exclude=home --exclude=boot 0 /bin 0 /dev 25M /etc 0 /lib 0 /lib64 744K /luarocks-2.3.0 0 /media 0 /mnt 125M /openresty-1.9.7.4 0 /opt 420K /root 49M /run 0 /sbin 0 /srv 0 /sys 0 /tmp 1.3G /usr 227M /var 1.7G total [root@localhost ~]# df is also reporting 80% of inode usage: # df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/centos-root 78160 66218 11942 85% / ^^^^^^^^ devtmpfs 8218272 519 8217753 1% /dev tmpfs 8221010 1 8221009 1% /dev/shm tmpfs 8221010 648 8220362 1% /run tmpfs 8221010 13 8220997 1% /sys/fs/cgroup /dev/sda1 509952 330 509622 1% /boot /dev/mapper/centos-home 210632704 99 210632605 1% /home tmpfs 8221010 1 8221009 1% /run/user/0 # / partition is created on top of a LVM having also 50GB size. # lvdisplay /dev/centos/root --- Logical volume --- LV Path /dev/centos/root LV Name root VG Name centos LV Status available # open 1 LV Size 50.00 GiB Current LE 12800 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 I've already checked against rootkit without finding anything wrong! I have another system, identical with this one which is healthy. The only difference I found between those systems is regarding max number of inodes available on / (which has the same size, 50GB on booth servers). On the second one (healthy), max number of inodes are ~52 milions and not only just ~85.000 as are reported on "seek" server. # df -i|grep ^/ /dev/mapper/centos-root 52424704 66137 52358567 1% / ^^^^^^^^^^^^^ /dev/sda1 509952 330 509622 1% /boot /dev/mapper/centos-home 210632704 26 210632678 1% /home [root@localhost ~]# Suspected also large number of files on /. Counted total number of files and or booth servers are the same: ~180K. So no difference here. Look to find also files larger than 100M and on booth servers and found just 1 (104M size): find / -type f -size +100000k -exec ls -lh {} \; # /usr/lib/locale/locale-archive # Looking to find files larger than 10M, I found just ~20 on booth servers. # find / -type f -size +10000k -exec ls -lh {} \; |wc -l 16 # So for sure, there are NO files exhausting free space. On booth servers, number of used inodes are identical: ~66K. Also xfs_info report is identical for booth. What is different is number of AVAILABLE inodes: 85K (on seek node) vs 52 milion (on healthy node)!!! How is possible that!!! Booth servers has the same size (50GB) for /! #lsof -nP |grep -i delete|wc -l 0 #find /proc/*/fd -ls | grep -i dele|wc -l 0 so lsof and find does not report anything wrong (any file deleted and still open)! reboot does not fix the problem, / remain 100% full After reboot, on 25th July: # df -ah|grep centos-root /dev/mapper/centos-root 50G 50G 4.0M 100% / # Also max number of inodes = 67k: # df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/centos-root 66960 66165 795 99% / ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ devtmpfs 8218272 519 8217753 1% /dev tmpfs 8221010 1 8221009 1% /dev/shm tmpfs 8221010 630 8220380 1% /run tmpfs 8221010 13 8220997 1% /sys/fs/cgroup /dev/sda1 509952 330 509622 1% /boot /dev/mapper/centos-home 210632704 28 210632676 1% /home tmpfs 8221010 1 8221009 1% /run/user/0 # Lets try to run intentionally xfs_grow (which normally should not produce any change) # xfs_growfs /dev/mapper/centos-root meta-data=/dev/mapper/centos-root isize=256 agcount=16, agsize=819136 blks = sectsz=512 attr=2, projid32bit=1 = crc=0 finobt=0 data = bsize=4096 blocks=13106176, imaxpct=25 = sunit=64 swidth=64 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=0 log =internal bsize=4096 blocks=6400, version=2 = sectsz=512 sunit=64 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 data blocks changed from 13106176 to 13107200 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # Partition remain the same, 50GB size: [root@nl-hvs-ov001a ~]# df -ah|grep centos-root /dev/mapper/centos-root 50G 50G 4.0M 100% / But number of inodes INCREASED with more tha 20%!!! # df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/centos-root 83200 66165 17035 80% / ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ devtmpfs 8218272 519 8217753 1% /dev tmpfs 8221010 1 8221009 1% /dev/shm tmpfs 8221010 630 8220380 1% /run tmpfs 8221010 13 8220997 1% /sys/fs/cgroup /dev/sda1 509952 330 509622 1% /boot /dev/mapper/centos-home 210632704 28 210632676 1% /home tmpfs 8221010 1 8221009 1% /run/user/0 # On 27July without changing anything there, max number inodes available for / decreased to ~67k (the same size like 2 days ago, before xfs_grow)! # df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/centos-root 67024 66225 799 99% / ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ devtmpfs 8218272 519 8217753 1% /dev tmpfs 8221010 1 8221009 1% /dev/shm tmpfs 8221010 632 8220378 1% /run tmpfs 8221010 13 8220997 1% /sys/fs/cgroup /dev/mapper/centos-home 210632704 99 210632605 1% /home /dev/sda1 509952 330 509622 1% /boot tmpfs 8221010 1 8221009 1% /run/user/0 # Please note that all that time, number of files remain unchanged ~180K, the same for inodes used, the number remain constant ~66K. Just max number of inodes available decreased which is an abnormal behavior. How can be fixed? Looks like xfs is crrupted or like a bug. Thanks in advance for help. Alex