From: Jan Kara <jack@suse.cz>
To: Geoffrey Thomas <Geoffrey.Thomas@twosigma.com>
Cc: 'Jan Kara' <jack@suse.cz>,
Thomas Walker <Thomas.Walker@twosigma.com>,
"'linux-ext4@vger.kernel.org'" <linux-ext4@vger.kernel.org>,
"Darrick J. Wong" <darrick.wong@oracle.com>,
"'tytso@mit.edu'" <tytso@mit.edu>
Subject: Re: Phantom full ext4 root filesystems on 4.1 through 4.14 kernels
Date: Thu, 11 Jul 2019 17:23:28 +0200 [thread overview]
Message-ID: <20190711152328.GB2449@quack2.suse.cz> (raw)
In-Reply-To: <96c4e04f8d5146c49ee9f4478c161dcb@EXMBDFT10.ad.twosigma.com>
On Thu 11-07-19 14:40:43, Geoffrey Thomas wrote:
> On Thursday, July 11, 2019 5:23 AM, Jan Kara <jack@suse.cz> wrote:
> > On Wed 26-06-19 11:17:54, Thomas Walker wrote:
> > > Sorry to revive a rather old thread, but Elana mentioned that there might
> > > have been a related fix recently? Possibly something to do with
> > > truncate? A quick scan of the last month or so turned up
> > > https://www.spinics.net/lists/linux-ext4/msg65772.html but none of these
> > > seemed obviously applicable to me. We do still experience this phantom
> > > space usage quite frequently (although the remount workaround below has
> > > lowered the priority).
> >
> > I don't recall any fix for this. But seeing that remount "fixes" the issue
> > for you can you try whether one of the following has a similar effect?
> >
> > 1) Try "sync"
> > 2) Try "fsfreeze -f / && fsfreeze -u /"
> > 3) Try "echo 3 >/proc/sys/vm/drop_caches"
> >
> > Also what is the contents of
> > /sys/fs/ext4/<problematic-device>/delayed_allocation_blocks
> > when the issue happens?
>
> We just had one of these today, and no luck from any of those.
> delayed_allocation_blocks is 1:
...
This is very strange because failed remount read-only (with EBUSY) doesn't
really do more than what "sync; echo 3 >/proc/sys/vm/drop_caches" does. I
suspect there's really some userspace taking up space and cleaning up on
umount. Anyway once this happens again, can you do:
fsfreeze -f /
e2image -r /dev/disk/by-uuid/523c8243-5a25-40eb-8627-f3bbf98ec299 - | \
xz >some_storage.xz
fsfreeze -u /
some_storage.xz can be on some usb stick or so. It will dump ext4 metadata
to the file. Then please provide some_storage.xz for download somewhere.
Thanks! BTW I'll be on vacation next two weeks so it will take a while to
get to this...
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
next prev parent reply other threads:[~2019-07-11 15:23 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-08 17:59 Phantom full ext4 root filesystems on 4.1 through 4.14 kernels Elana Hashman
2018-11-08 18:13 ` Reindl Harald
2018-11-08 18:20 ` Elana Hashman
2018-11-08 18:47 ` Darrick J. Wong
2018-12-05 16:26 ` Elana Hashman
2019-01-23 19:59 ` Thomas Walker
2019-06-26 15:17 ` Thomas Walker
2019-07-11 9:23 ` Jan Kara
2019-07-11 14:40 ` Geoffrey Thomas
2019-07-11 15:23 ` Jan Kara [this message]
2019-07-11 17:10 ` Theodore Ts'o
2019-07-12 19:19 ` Thomas Walker
2019-07-12 20:28 ` Theodore Ts'o
2019-07-12 21:47 ` Geoffrey Thomas
2019-07-25 21:22 ` Geoffrey Thomas
2019-07-29 10:09 ` Jan Kara
2019-07-29 11:18 ` ext4 file system is constantly writing to the block device with no activity from the applications, is it a bug? Dmitrij Gusev
2019-07-29 12:55 ` Theodore Y. Ts'o
2019-07-29 21:12 ` Dmitrij Gusev
2019-01-24 1:54 ` Phantom full ext4 root filesystems on 4.1 through 4.14 kernels Liu Bo
2019-01-24 14:40 ` Elana Hashman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190711152328.GB2449@quack2.suse.cz \
--to=jack@suse.cz \
--cc=Geoffrey.Thomas@twosigma.com \
--cc=Thomas.Walker@twosigma.com \
--cc=darrick.wong@oracle.com \
--cc=linux-ext4@vger.kernel.org \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).