[ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Josef Bacik]

Hello,

I'd like to attend this years LSF.  I've been doing various file system work for
the last 6 years, most of that with my head down in btrfs.

I'd like to talk about what to do about O_DIRECT.  Nobody really owns it and
nobody really _wants_ to own it, and we've all been tacking on our own file
systems optimizations and work arounds to make the generic stuff work.  I'm to
the point now where I'm just going to do all the work ourselves inside of btrfs
since we need to have different waiting rules.  So the question is do we want to
just rm -f fs/direct-io.c and let everybody do their own thing, or is there some
way we can tease out the actual generic stuff that everybody is going to need to
do and adapt everybody to use that?  And then theres the question of what are
the things we want to do in the generic code, do we want to just do the get
pages thing, do we want to still have stuff to build and submit the bios?  What
about how AIO interacts with it?  And best of all can we convince Zach to do all
of it for us!  Thanks,

Josef

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Zach Brown]

> about how AIO interacts with it?  And best of all can we convince Zach to do all
> of it for us!  Thanks,

While I'll condede that you *can*, I look forward to finding out if you
will :).

- z

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Theodore Ts'o]

.... and can we get rid of this horrible hack where we have this
bastardized use of a struct buffer_head which is allocated on the
stack, which has nothing really to do with a buffer head, and is all
about the fact that no one wants to change the function signature for
get_block_t?

						- Ted

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Dave Chinner]

On Fri, Jan 18, 2013 at 06:01:28PM -0500, Theodore Ts'o wrote:
> .... and can we get rid of this horrible hack where we have this
> bastardized use of a struct buffer_head which is allocated on the
> stack, which has nothing really to do with a buffer head, and is all
> about the fact that no one wants to change the function signature for
> get_block_t?

I have patches to do that, but they are on the back burner right now
because it causes some kind of weird corruption in the pwritev case
that kvm uses to issue IO (i.e. large iovecs of 4k segments).

IMO, the direct IO code is that complex and convoluted now that t is
close to impossible to modify without introduce some weird, subtle
and almost impossible to debug issue. The code has been optimised to
the point of being unmaintainable, and I have seriously considered
just reimplementing the bits XFS needs just for XFS several times in
the past year....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Josef Bacik]

On Sun, Jan 20, 2013 at 03:35:21PM -0700, Dave Chinner wrote:
> On Fri, Jan 18, 2013 at 06:01:28PM -0500, Theodore Ts'o wrote:
> > .... and can we get rid of this horrible hack where we have this
> > bastardized use of a struct buffer_head which is allocated on the
> > stack, which has nothing really to do with a buffer head, and is all
> > about the fact that no one wants to change the function signature for
> > get_block_t?
> 
> I have patches to do that, but they are on the back burner right now
> because it causes some kind of weird corruption in the pwritev case
> that kvm uses to issue IO (i.e. large iovecs of 4k segments).
> 
> IMO, the direct IO code is that complex and convoluted now that t is
> close to impossible to modify without introduce some weird, subtle
> and almost impossible to debug issue. The code has been optimised to
> the point of being unmaintainable, and I have seriously considered
> just reimplementing the bits XFS needs just for XFS several times in
> the past year....
> 

Yeah this is the point that I'm at currently, and it's even worse for Btrfs
since we have to short circuit most of the work that the generic stuff does
already since we build bio's ourselves.  So maybe it would be best that we trim
the generic stuff down to it's most basic functionality and leave it in place
for things like ext2/3, and then for the more advanced file systems we just all
do our own things.  If we can all take a good look at what we would need to
replace maybe we can come up with some generic helpers.  Thanks,

Josef

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Jan Kara]

  Hello,

On Fri 18-01-13 17:10:07, Josef Bacik wrote:
> I'd like to talk about what to do about O_DIRECT.  Nobody really owns it
> and nobody really _wants_ to own it, and we've all been tacking on our
> own file systems optimizations and work arounds to make the generic stuff
> work.  I'm to the point now where I'm just going to do all the work
> ourselves inside of btrfs since we need to have different waiting rules.
> So the question is do we want to just rm -f fs/direct-io.c and let
> everybody do their own thing,
  I don't think we really can. Just grep for its uses. There are like 15
filesystems using it. That would be a huge amount of duplication.

> or is there some way we can tease out the
> actual generic stuff that everybody is going to need to do and adapt
> everybody to use that?  And then theres the question of what are the
> things we want to do in the generic code, do we want to just do the get
> pages thing, do we want to still have stuff to build and submit the bios?
> What about how AIO interacts with it? 
  I'm not sure what issues you are exactly facing but I can understand
blockdev_direct_IO() isn't doing what btrfs would need. And I also agree
with others that the code is rather complex and hard to maintain. E.g.  the
get_block_t insanity of using buffer_head is nagging me for a long time.
The handling of unaligned DIO which all filesystems just serialize (at least
for writes) because it causes data corruption. But these are mostly smaller
gradual improvements.

  IMHO the devil is in "show me the code that is flexible enough to work
for most, fast, and simpler than what we have". So I think we can speak
about what btrfs (or xfs or whoever else) would need and how we could
change (or whether it's worth to change) the generic code to accommodate
its needs. Hum?

								Honza

-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Dave Chinner]

On Tue, Jan 22, 2013 at 03:03:37PM +0100, Jan Kara wrote:
>   Hello,
> 
> On Fri 18-01-13 17:10:07, Josef Bacik wrote:
> > I'd like to talk about what to do about O_DIRECT.  Nobody really owns it
> > and nobody really _wants_ to own it, and we've all been tacking on our
> > own file systems optimizations and work arounds to make the generic stuff
> > work.  I'm to the point now where I'm just going to do all the work
> > ourselves inside of btrfs since we need to have different waiting rules.
> > So the question is do we want to just rm -f fs/direct-io.c and let
> > everybody do their own thing,
>   I don't think we really can. Just grep for its uses. There are like 15
> filesystems using it. That would be a huge amount of duplication.
> 
> > or is there some way we can tease out the
> > actual generic stuff that everybody is going to need to do and adapt
> > everybody to use that?  And then theres the question of what are the
> > things we want to do in the generic code, do we want to just do the get
> > pages thing, do we want to still have stuff to build and submit the bios?
> > What about how AIO interacts with it? 
>   I'm not sure what issues you are exactly facing but I can understand
> blockdev_direct_IO() isn't doing what btrfs would need. And I also agree
> with others that the code is rather complex and hard to maintain. E.g.  the
> get_block_t insanity of using buffer_head is nagging me for a long time.
> The handling of unaligned DIO which all filesystems just serialize (at least
> for writes) because it causes data corruption. But these are mostly smaller
> gradual improvements.

The problem I find is that small gradual improvements is that every
time I try to do one I end up with some wierd subtle problem that
I've been unable to debug. It's happened several times in the past
year, and each time I've given up on trying to make gradual
improvements because of this....

That fits my definition of unmaintainable code almost perfectly.

>   IMHO the devil is in "show me the code that is flexible enough to work
> for most, fast, and simpler than what we have". So I think we can speak
> about what btrfs (or xfs or whoever else) would need and how we could
> change (or whether it's worth to change) the generic code to accommodate
> its needs. Hum?

I'd say XFS needs very little outside help - AFAICT it still has >90%
of the infrastructure it needs to do direct IO itself....

Cheers,

Dave
-- 
Dave Chinner
david@fromorbit.com

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Jan Kara]

On Thu 31-01-13 10:16:00, Dave Chinner wrote:
> On Tue, Jan 22, 2013 at 03:03:37PM +0100, Jan Kara wrote:
> >   Hello,
> > 
> > On Fri 18-01-13 17:10:07, Josef Bacik wrote:
> > > I'd like to talk about what to do about O_DIRECT.  Nobody really owns it
> > > and nobody really _wants_ to own it, and we've all been tacking on our
> > > own file systems optimizations and work arounds to make the generic stuff
> > > work.  I'm to the point now where I'm just going to do all the work
> > > ourselves inside of btrfs since we need to have different waiting rules.
> > > So the question is do we want to just rm -f fs/direct-io.c and let
> > > everybody do their own thing,
> >   I don't think we really can. Just grep for its uses. There are like 15
> > filesystems using it. That would be a huge amount of duplication.
> > 
> > > or is there some way we can tease out the
> > > actual generic stuff that everybody is going to need to do and adapt
> > > everybody to use that?  And then theres the question of what are the
> > > things we want to do in the generic code, do we want to just do the get
> > > pages thing, do we want to still have stuff to build and submit the bios?
> > > What about how AIO interacts with it? 
> >   I'm not sure what issues you are exactly facing but I can understand
> > blockdev_direct_IO() isn't doing what btrfs would need. And I also agree
> > with others that the code is rather complex and hard to maintain. E.g.  the
> > get_block_t insanity of using buffer_head is nagging me for a long time.
> > The handling of unaligned DIO which all filesystems just serialize (at least
> > for writes) because it causes data corruption. But these are mostly smaller
> > gradual improvements.
> 
> The problem I find is that small gradual improvements is that every
> time I try to do one I end up with some wierd subtle problem that
> I've been unable to debug. It's happened several times in the past
> year, and each time I've given up on trying to make gradual
> improvements because of this....
> 
> That fits my definition of unmaintainable code almost perfectly.
> 
> >   IMHO the devil is in "show me the code that is flexible enough to work
> > for most, fast, and simpler than what we have". So I think we can speak
> > about what btrfs (or xfs or whoever else) would need and how we could
> > change (or whether it's worth to change) the generic code to accommodate
> > its needs. Hum?
> 
> I'd say XFS needs very little outside help - AFAICT it still has >90%
> of the infrastructure it needs to do direct IO itself....
  I'm not sure it's really about infrastructure. When I look into
fs/direct-io.c, it is theoretically a trivial thing - just a loop with get
pages, map blocks, submit bio, repeat. But then there are the details of
blocksize < pagesize or even DIO not aligned to blocksize, holes in files,
throttling so that we don't have too many bios in flight... and suddently
we have the beast it is now. And every filesystem will have to deal with
these special cases so doing it in the generic code looks like a good
thing to me (I can imagine those nasty subtle bugs when each filesystem has
to handle all the cases on its own - and I believe XFS may get it right
pretty quickly but world isn't just XFS)...

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Dave Chinner]

On Thu, Jan 31, 2013 at 11:41:09PM +0100, Jan Kara wrote:
> On Thu 31-01-13 10:16:00, Dave Chinner wrote:
> > On Tue, Jan 22, 2013 at 03:03:37PM +0100, Jan Kara wrote:
> > >   Hello,
> > > 
> > > On Fri 18-01-13 17:10:07, Josef Bacik wrote:
> > > > I'd like to talk about what to do about O_DIRECT.  Nobody really owns it
> > > > and nobody really _wants_ to own it, and we've all been tacking on our
> > > > own file systems optimizations and work arounds to make the generic stuff
> > > > work.  I'm to the point now where I'm just going to do all the work
> > > > ourselves inside of btrfs since we need to have different waiting rules.
> > > > So the question is do we want to just rm -f fs/direct-io.c and let
> > > > everybody do their own thing,
> > >   I don't think we really can. Just grep for its uses. There are like 15
> > > filesystems using it. That would be a huge amount of duplication.
> > > 
> > > > or is there some way we can tease out the
> > > > actual generic stuff that everybody is going to need to do and adapt
> > > > everybody to use that?  And then theres the question of what are the
> > > > things we want to do in the generic code, do we want to just do the get
> > > > pages thing, do we want to still have stuff to build and submit the bios?
> > > > What about how AIO interacts with it? 
> > >   I'm not sure what issues you are exactly facing but I can understand
> > > blockdev_direct_IO() isn't doing what btrfs would need. And I also agree
> > > with others that the code is rather complex and hard to maintain. E.g.  the
> > > get_block_t insanity of using buffer_head is nagging me for a long time.
> > > The handling of unaligned DIO which all filesystems just serialize (at least
> > > for writes) because it causes data corruption. But these are mostly smaller
> > > gradual improvements.
> > 
> > The problem I find is that small gradual improvements is that every
> > time I try to do one I end up with some wierd subtle problem that
> > I've been unable to debug. It's happened several times in the past
> > year, and each time I've given up on trying to make gradual
> > improvements because of this....
> > 
> > That fits my definition of unmaintainable code almost perfectly.
> > 
> > >   IMHO the devil is in "show me the code that is flexible enough to work
> > > for most, fast, and simpler than what we have". So I think we can speak
> > > about what btrfs (or xfs or whoever else) would need and how we could
> > > change (or whether it's worth to change) the generic code to accommodate
> > > its needs. Hum?
> > 
> > I'd say XFS needs very little outside help - AFAICT it still has >90%
> > of the infrastructure it needs to do direct IO itself....
>   I'm not sure it's really about infrastructure. When I look into
> fs/direct-io.c, it is theoretically a trivial thing - just a loop with get
> pages, map blocks, submit bio, repeat. But then there are the details of
> blocksize < pagesize or even DIO not aligned to blocksize, holes in files,
> throttling so that we don't have too many bios in flight...

Sure, that's relatively simple, but once you optimise it repeated to
the point where the order of single instructions is important even
simple code becomes a tangled, ugly mess.

> and suddently
> we have the beast it is now. And every filesystem will have to deal with
> these special cases so doing it in the generic code looks like a good
> thing to me (I can imagine those nasty subtle bugs when each filesystem has
> to handle all the cases on its own - and I believe XFS may get it right
> pretty quickly but world isn't just XFS)...

The advantage of using shared code is that it eases the burden of
maintenance and enhancement on individual filesystems.  Both Josef
and I are putting forward the argument that the shared direct IO
code provides neither of those advantages any more due to current
complexity and fragility that has resulted from the monolithic
"everything for everyone" approach we currently have.

What I'm trying to say is that maybe there's a better way of
providing generic direct IO support. Perhaps we are better served by
having smaller generic helpers similar to the buffered IO path to
allow filesystems to the simple stuff as optimally as possible
without all the overhead they don't need. One-size-fits-all has
never worked in the filesystems game, yet we seem to be stuck on
that approach here even when it appears to be collapsing under it's
own weight.... :/

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Joel Becker]

On Wed, Feb 06, 2013 at 08:51:12AM +1100, Dave Chinner wrote:
> The advantage of using shared code is that it eases the burden of
> maintenance and enhancement on individual filesystems.  Both Josef
> and I are putting forward the argument that the shared direct IO
> code provides neither of those advantages any more due to current
> complexity and fragility that has resulted from the monolithic
> "everything for everyone" approach we currently have.
> 
> What I'm trying to say is that maybe there's a better way of
> providing generic direct IO support. Perhaps we are better served by
> having smaller generic helpers similar to the buffered IO path to
> allow filesystems to the simple stuff as optimally as possible
> without all the overhead they don't need. One-size-fits-all has
> never worked in the filesystems game, yet we seem to be stuck on
> that approach here even when it appears to be collapsing under it's
> own weight.... :/

	I vote for trying the helper approach.  I think dropping generic
code altogether would be a disaster.  The corner cases of O_DIRECT are
legion; everyone has behavioral assumptions based on historical
implementations, etc.  Remember how badly some high-performance software
handles O_DIRECT alignments larger than 512B.
	We have a long history of successfully inverting the generic
code with helpers and (if necessary, I'm not saying it is) operations
structures.
	I don't think I'd try to shove down generic_aio_read/write.  Let
them handle the check for O_DIRECT and the fallback to buffered I/O.

Joel


-- 

"And yet I fight,
 And yet I fight this battle all alone.
 No one to cry to;
 No place to call home."

			http://www.jlbec.org/
			jlbec@evilplan.org

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Kent Overstreet]

On Tue, Feb 5, 2013 at 4:40 PM, Joel Becker <jlbec@evilplan.org> wrote:
>         I vote for trying the helper approach.  I think dropping generic
> code altogether would be a disaster.  The corner cases of O_DIRECT are
> legion; everyone has behavioral assumptions based on historical
> implementations, etc.  Remember how badly some high-performance software
> handles O_DIRECT alignments larger than 512B.
>         We have a long history of successfully inverting the generic
> code with helpers and (if necessary, I'm not saying it is) operations
> structures.
>         I don't think I'd try to shove down generic_aio_read/write.  Let
> them handle the check for O_DIRECT and the fallback to buffered I/O.

Tackling the dio code has been high on my list, if I ever get time -
in the profiles I've been doing, reading from a raw block device, the
dio code is the biggest single source of overhead by a good margin and
there's plenty of obvious inefficiency.

The helper approach was more or less what I had in mind - also,
splitting out the block device path out to start with, and once that's
cleaned up using that to guide how the helpers and such should be
structured.

The other thing that really frustrates me about the dio code is the
perverse flow of control between the generic code and the fs code, via
the getblk callbacks.

IMO, it'd be much cleaner to aim for something more like submitting a
bio to the block layer - just set up a bio, point it at inode:offset
and pass it off to the fs code. If I can finish off my immutable
bvec/efficient bio splitting code and get it in it should make that
approach work out nicely.

But currently my brain is stuffed full of other sutff :/

Re: [ATTEND] [LSF TOPIC] What to do about O_DIRECT?

[Jan Kara]

On Wed 06-02-13 08:51:12, Dave Chinner wrote:
> The advantage of using shared code is that it eases the burden of
> maintenance and enhancement on individual filesystems.  Both Josef
> and I are putting forward the argument that the shared direct IO
> code provides neither of those advantages any more due to current
> complexity and fragility that has resulted from the monolithic
> "everything for everyone" approach we currently have.
> 
> What I'm trying to say is that maybe there's a better way of
> providing generic direct IO support. Perhaps we are better served by
> having smaller generic helpers similar to the buffered IO path to
> allow filesystems to the simple stuff as optimally as possible
> without all the overhead they don't need. One-size-fits-all has
> never worked in the filesystems game, yet we seem to be stuck on
> that approach here even when it appears to be collapsing under it's
> own weight.... :/
  Yeah, the approach of providing smaller generic helpers could make the
code more readable so I guess it's worth a try.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR