From: Thomas Walker <Thomas.Walker@twosigma.com>
To: Elana Hashman <Elana.Hashman@twosigma.com>
Cc: "Darrick J. Wong" <darrick.wong@oracle.com>,
"'tytso@mit.edu'" <tytso@mit.edu>,
"'linux-ext4@vger.kernel.org'" <linux-ext4@vger.kernel.org>
Subject: Re: Phantom full ext4 root filesystems on 4.1 through 4.14 kernels
Date: Wed, 23 Jan 2019 14:59:22 -0500 [thread overview]
Message-ID: <20190123195922.GA16927@twosigma.com> (raw)
In-Reply-To: <c7cfeaf451d7438781da95b01f21116e@exmbdft5.ad.twosigma.com>
Unfortunately this still continues to be a persistent problem for us. On another example system:
# uname -a
Linux <hostname> 4.14.67-ts1 #1 SMP Wed Aug 29 13:28:25 UTC 2018 x86_64 GNU/Linux
# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/disk/by-uuid/<uuid> 50G 46G 1.1G 98% /
# df -hi /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/disk/by-uuid/<uuid> 3.2M 306K 2.9M 10% /
# du -hsx /
14G /
And confirmed not to be due to sparse files or deleted but still open files.
The most interesting thing that I've been able to find so far is this:
# mount -o remount,ro /
mount: / is busy
# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/disk/by-uuid/<uuid> 50G 14G 33G 30% /
Something about attempting (and failing) to remount read-only frees up all of the phantom space usage.
Curious whether that sparks ideas in anyone's mind?
I've tried all manner of other things without success. Unmounting all of the overlays. Killing off virtually all of usersapce (dropping to single user). Dropping page/inode/dentry caches.Nothing else (short of a reboot) seems to give us the space back.
On Wed, Dec 05, 2018 at 11:26:19AM -0500, Elana Hashman wrote:
> Okay, let's take a look at another affected host. I have not drained it, just
> cordoned it, so it's still in Kubernetes service and has running, active pods.
>
next prev parent reply other threads:[~2019-01-23 20:05 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-08 17:59 Phantom full ext4 root filesystems on 4.1 through 4.14 kernels Elana Hashman
2018-11-08 18:13 ` Reindl Harald
2018-11-08 18:20 ` Elana Hashman
2018-11-08 18:47 ` Darrick J. Wong
2018-12-05 16:26 ` Elana Hashman
2019-01-23 19:59 ` Thomas Walker [this message]
2019-06-26 15:17 ` Thomas Walker
2019-07-11 9:23 ` Jan Kara
2019-07-11 14:40 ` Geoffrey Thomas
2019-07-11 15:23 ` Jan Kara
2019-07-11 17:10 ` Theodore Ts'o
2019-07-12 19:19 ` Thomas Walker
2019-07-12 20:28 ` Theodore Ts'o
2019-07-12 21:47 ` Geoffrey Thomas
2019-07-25 21:22 ` Geoffrey Thomas
2019-07-29 10:09 ` Jan Kara
2019-07-29 11:18 ` ext4 file system is constantly writing to the block device with no activity from the applications, is it a bug? Dmitrij Gusev
2019-07-29 12:55 ` Theodore Y. Ts'o
2019-07-29 21:12 ` Dmitrij Gusev
2019-01-24 1:54 ` Phantom full ext4 root filesystems on 4.1 through 4.14 kernels Liu Bo
2019-01-24 14:40 ` Elana Hashman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190123195922.GA16927@twosigma.com \
--to=thomas.walker@twosigma.com \
--cc=Elana.Hashman@twosigma.com \
--cc=darrick.wong@oracle.com \
--cc=linux-ext4@vger.kernel.org \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).