Re: [PATCH 0/11] Update version of write stream ID patchset

From: Jens Axboe <axboe@fb.com>
To: Jeff Moyer <jmoyer@redhat.com>
Cc: <linux-fsdevel@vger.kernel.org>, <linux-block@vger.kernel.org>,
	<calvinowens@fb.com>, <hch@lst.de>, <adilger@dilger.ca>,
	"Martin K. Petersen" <martin.petersen@oracle.com>
Subject: Re: [PATCH 0/11] Update version of write stream ID patchset
Date: Fri, 4 Mar 2016 14:06:41 -0700	[thread overview]
Message-ID: <56D9F8E1.2080702@fb.com> (raw)
In-Reply-To: <x4960x2nejd.fsf@segfault.boston.devel.redhat.com>

On 03/04/2016 02:01 PM, Jeff Moyer wrote:
> Jens Axboe <axboe@fb.com> writes:
>
>> On 03/04/2016 12:42 PM, Jeff Moyer wrote:
>>> My main question is why expose this to userspace at all?  If we're
>>> keeping track of write streams per file, then why not implement that in
>>> the kernel, transparent to the application?  That would benefit all
>>> applications instead of requiring application developers to opt in.
>>
>> Because lots of different files could be the same write ID.
>
> Do you mean that the application will want to have multiple files that
> share the same write ID, or that there will be collisions due to the
> small ID space (I think the former)?  There's no way to avoid the
> latter, of course.  For the former, I agree that encoding a per-file
> policy in the kernel could be limiting.

The former - and it won't just be limiting, it'll be mostly useless...

>> It's not like we're going to have millions of streams available, you
>> have to group them more wisely. Unless the policy is
>> one-stream-per-file always, then we can't put that sort of thing in
>> the kernel. The kernel has no way of knowing.
>
> I know that hard-coding a one stream per file (or directory) scheme
> wouldn't be the most ideal situation, but I wonder if it covers 90% of
> the use cases without requiring application involvement.  Some numbers
> here would be very useful in supporting one scheme versus another.

It'd be even harder to convince applications to change their file 
layout, and it'd be a more involved change than simply opting in to add 
a system call to set a stream ID when you open a file. It's really not a 
hard change at all.

>>> I'm sure your argument will have something to do with how stream id's
>>> are allocated/freed (expensive/slow, limited resource, whatever), but
>>> that really just gets back to Martin's original questions about what we
>>> should expect from the hardware and what the programming model should
>>> look like (questions that are, afaik, still open).
>>
>> That's orthogonal, really. The open/close might be expensive, or it
>> might not be, it has no real bearing on how you assign specific writes
>> to specific stream IDs.
>
> It may be important if you plan to open and close the streams often.

Of course, I'm saying that the fact that if it's expensive or not, it 
has no impact on that part of the discussion.

> Again, it's not clear to me what the hardware supports or requires in
> this area, so I'm not sure if it's a relevant question.  -ENOSPEC.  :)
>
>>> I'm not against write streams, I think it's a neat idea.  I just think
>>> it will die on the vine if you require application developers to opt
>>> in.  Not all storage is SSDs, and I don't like that SSDs now have to be
>>> treated differently by the programmer.
>>
>> But that's why it's kept really simple. There are people that want to
>> make this more involved, and tie QoS criteria to streams. My argument
>> there has been what you are saying, it will never be used or get
>> adopted. For streams in general, the wins are big enough that
>> applications will care. And it's not difficult to use at all...
>
> I'm not convinced applications will care.  :)  How many developers will
> even know that they should use this interface?

You have to market it a bit, obviously. But if you consume flash 
devices, then you WILL be interested in cutting your write amplification 
down a lot. Especially if you have tons of them.

So applications might not care, the people that use them certainly will. 
And most people care about performance, too.

What I'm saying is that I disagree wildly. There's enough of a carrot 
here for people to care.

>> It's not just SSDs, either. Could be used for tiered storage in
>> general. That would mostly require going a bit further and assigning
>> performance characteristics to specific stream IDs, but there's
>> nothing preventing that from being done down the road. For now, this
>> is just a basic interface with a kernel managed stream ID space
>> attached.
>
> OK.  I'm still of the opinion that we should try to make this
> transparent.  I could be swayed by workload descriptions and numbers
> comparing approaches, though.

You can't just waive that flag and not have a solution. Any solution in 
that space would imply having policy in the kernel. A "just use a stream 
per file" is never going to work.

-- 
Jens Axboe