Re: POSIX violation by writeback error

From: Dave Chinner <david@fromorbit.com>
To: Vito Caputo <vcaputo@pengaru.com>
Cc: "Jeff Layton" <jlayton@redhat.com>,
	"J. Bruce Fields" <bfields@fieldses.org>,
	"Rogier Wolff" <R.E.Wolff@BitWizard.nl>,
	焦晓冬 <milestonejxd@gmail.com>,
	linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: POSIX violation by writeback error
Date: Wed, 5 Sep 2018 10:51:42 +1000	[thread overview]
Message-ID: <20180905005141.GD27618@dastard> (raw)
In-Reply-To: <20180904203534.yumaest6v5p6izln@shells.gnugeneration.com>

On Tue, Sep 04, 2018 at 01:35:34PM -0700, Vito Caputo wrote:
> On Tue, Sep 04, 2018 at 04:18:18PM -0400, Jeff Layton wrote:
> > On Tue, 2018-09-04 at 14:54 -0400, J. Bruce Fields wrote:
> > > On Tue, Sep 04, 2018 at 06:23:48PM +0200, Rogier Wolff wrote:
> > > > On Tue, Sep 04, 2018 at 12:12:03PM -0400, J. Bruce Fields wrote:
> > > > > Well, I think the point was that in the above examples you'd prefer that
> > > > > the read just fail--no need to keep the data.  A bit marking the file
> > > > > (or even the entire filesystem) unreadable would satisfy posix, I guess.
> > > > > Whether that's practical, I don't know.
> > > > 
> > > > When you would do it like that (mark the whole filesystem as "in
> > > > error") things go from bad to worse even faster. The Linux kernel 
> > > > tries to keep the system up even in the face of errors. 
> > > > 
> > > > With that suggestion, having one application run into a writeback
> > > > error would effectively crash the whole system because the filesystem
> > > > may be the root filesystem and stuff like "sshd" that you need to
> > > > diagnose the problem needs to be read from the disk.... 
> > > 
> > > Well, the absolutist position on posix compliance here would be that a
> > > crash is still preferable to returning the wrong data.  And for the
> > > cases 焦晓冬 gives, that sounds right?  Maybe it's the wrong balance in
> > > general, I don't know.  And we do already have filesystems with
> > > panic-on-error options, so if they aren't used maybe then maybe users
> > > have already voted against that level of strictness.
> > > 
> > 
> > Yeah, idk. The problem here is that this is squarely in the domain of
> > implementation defined behavior. I do think that the current "policy"
> > (if you call it that) of what to do after a wb error is weird and wrong.
> > What we probably ought to do is start considering how we'd like it to
> > behave.
> > 
> > How about something like this?
> > 
> > Mark the pages as "uncleanable" after a writeback error. We'll satisfy
> > reads from the cached data until someone calls fsync, at which point
> > we'd return the error and invalidate the uncleanable pages.
> > 
> > If no one calls fsync and scrapes the error, we'll hold on to it for as
> > long as we can (or up to some predefined limit) and then after that
> > we'll invalidate the uncleanable pages and start returning errors on
> > reads. If someone eventually calls fsync afterward, we can return to
> > normal operation.
> > 
> > As always though...what about mmap? Would we need to SIGBUS at the point
> > where we'd start returning errors on read()?
> > 
> > Would that approximate the current behavior enough and make sense?
> > Implementing it all sounds non-trivial though...
> > 
> 
> Here's a crazy and potentially stupid idea:
> 
> Implement a new class of swap space for backing dirty pages which fail
> to write back.  Pages in this space survive reboots, essentially backing
> the implicit commitment POSIX establishes in the face of asynchronous
> writeback errors.  Rather than evicting these pages as clean, they are
> swapped out to the persistent swap.

And when that "swap" area gets write errors, too? What then? We're
straight back to the same "what the hell do we do with the error"
problem.

Adding more turtles doesn't help solve this issue.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com