Write atomicity guarantees

Write atomicity guarantees

[Matthew Wilcox]

NVMe allows the drive to tell the host what atomicity guarantees it
provides for a write command.  At the moment, I don't think Linux has
a way for the driver to pass that information up to the filesystem.

The value that is most interesting to report is Atomic Write Unit Power
Fail ("if you send a write no larger than this, the drive guarantees to
write all of it or none of it"), minimum value 1 sector. [1]

There's a proposal before the NVMe workgroup to add a boundary size/offset
to modify AWUPF ("except if you cross this boundary, then AWUPF is not
guaranteed").  Think RAID stripe crossing.

So, three questions.  Is there somewhere already to pass boundary
information up to the filesystem?  Can filesystems make use of a larger
atomic write unit than a single sector?  And, if the device is internally
a RAID device, is knowing the boundary size/offset useful?


[1] There is also Atomic Write Unit Normal ("if you send two writes,
neither of which is larger than this, subsequent reads will get either
one or the other, not a mixture of both"), which I don't think we care
about because the page cache prevents us from sending two writes which
overlap with each other.

Re: Write atomicity guarantees

[Chris Mason]

On 04/24/2014 01:39 PM, Matthew Wilcox wrote:
>
> NVMe allows the drive to tell the host what atomicity guarantees it
> provides for a write command.  At the moment, I don't think Linux has
> a way for the driver to pass that information up to the filesystem.
>
> The value that is most interesting to report is Atomic Write Unit Power
> Fail ("if you send a write no larger than this, the drive guarantees to
> write all of it or none of it"), minimum value 1 sector. [1]
>
> There's a proposal before the NVMe workgroup to add a boundary size/offset
> to modify AWUPF ("except if you cross this boundary, then AWUPF is not
> guaranteed").  Think RAID stripe crossing.
>
> So, three questions.  Is there somewhere already to pass boundary
> information up to the filesystem?  Can filesystems make use of a larger
> atomic write unit than a single sector?  And, if the device is internally
> a RAID device, is knowing the boundary size/offset useful?
>
>
> [1] There is also Atomic Write Unit Normal ("if you send two writes,
> neither of which is larger than this, subsequent reads will get either
> one or the other, not a mixture of both"), which I don't think we care
> about because the page cache prevents us from sending two writes which
> overlap with each other.

I think we really need the atomics to be vectored.  Send N writes which 
as a unit are not larger than X, but which may span anywhere on device. 
  An array with writeback cache, or a log structured squirrel in the FTL 
should be able to provide this pretty easily?

The immediate use case is mysql (16K writes) on a fragmented filesystem. 
  The FS needs to be able to collect a single atomic write made up of N 
4K sectors.

-chris

Re: Write atomicity guarantees

[Dan Williams]

On Thu, Apr 24, 2014 at 11:03 AM, Chris Mason <clm@fb.com> wrote:
> On 04/24/2014 01:39 PM, Matthew Wilcox wrote:
>>
>>
>> NVMe allows the drive to tell the host what atomicity guarantees it
>> provides for a write command.  At the moment, I don't think Linux has
>> a way for the driver to pass that information up to the filesystem.
>>
>> The value that is most interesting to report is Atomic Write Unit Power
>> Fail ("if you send a write no larger than this, the drive guarantees to
>> write all of it or none of it"), minimum value 1 sector. [1]
>>
>> There's a proposal before the NVMe workgroup to add a boundary size/offset
>> to modify AWUPF ("except if you cross this boundary, then AWUPF is not
>> guaranteed").  Think RAID stripe crossing.
>>
>> So, three questions.  Is there somewhere already to pass boundary
>> information up to the filesystem?  Can filesystems make use of a larger
>> atomic write unit than a single sector?  And, if the device is internally
>> a RAID device, is knowing the boundary size/offset useful?
>>
>>
>> [1] There is also Atomic Write Unit Normal ("if you send two writes,
>> neither of which is larger than this, subsequent reads will get either
>> one or the other, not a mixture of both"), which I don't think we care
>> about because the page cache prevents us from sending two writes which
>> overlap with each other.
>
>
> I think we really need the atomics to be vectored.  Send N writes which as a
> unit are not larger than X, but which may span anywhere on device.  An array
> with writeback cache, or a log structured squirrel in the FTL should be able
> to provide this pretty easily?
>
> The immediate use case is mysql (16K writes) on a fragmented filesystem.
> The FS needs to be able to collect a single atomic write made up of N 4K
> sectors.

How big does N need to be before it starts to be generally useful?
Here it seems we're talking on the order to tens of writes, but for
the upper bound Dave said that N could be in the hundreds of thousands
[1].

--
Dan

[1]: http://marc.info/?l=linux-fsdevel&m=139262740324307&w=2

Re: Write atomicity guarantees

[Matthew Wilcox]

On Thu, Apr 24, 2014 at 11:10:09AM -0700, Purush Gupta wrote:
> That would require a new NVMe write command semantic with descriptors
> currently its not there.

Yes ... I don't want to start defining such a command on linux-fsdevel.
What I'm hearing is that there's no benefit to a device that can guarantee
to write multiple contiguous sectors in a non-torn manner over a device
that can write a single sector in a non-torn manner.

For any real benefit, filesystems need (and Linux needs to introduce
plumbing for) vectored atomic writes.

Re: Write atomicity guarantees

[Chris Mason]

On 04/24/2014 02:25 PM, Matthew Wilcox wrote:
> On Thu, Apr 24, 2014 at 11:10:09AM -0700, Purush Gupta wrote:
>> That would require a new NVMe write command semantic with descriptors
>> currently its not there.
>
> Yes ... I don't want to start defining such a command on linux-fsdevel.
> What I'm hearing is that there's no benefit to a device that can guarantee
> to write multiple contiguous sectors in a non-torn manner over a device
> that can write a single sector in a non-torn manner.
>
> For any real benefit, filesystems need (and Linux needs to introduce
> plumbing for) vectored atomic writes.
>

That's my feeling.  The non-vectored use case is pretty limited, mostly 
to help get a contiguous log entry on disk as a full unit.  But most of 
the time the filesystem log commits are pretty big.  Workloads with a 
very small number of latency sensitive writers would see improvements.

We could also the contiguous atomics for something like mysql by using a 
16K sector size in the filesystem.  At that point all of the db 16K 
units will be contig and the existing atomics proposals become interesting.

But I was really hoping for the vectors ;)

-chris

Re: Write atomicity guarantees

[Chris Mason]

On 04/24/2014 02:23 PM, Dan Williams wrote:
> On Thu, Apr 24, 2014 at 11:03 AM, Chris Mason <clm@fb.com> wrote:
>> On 04/24/2014 01:39 PM, Matthew Wilcox wrote:
>>>
>>>
>>> NVMe allows the drive to tell the host what atomicity guarantees it
>>> provides for a write command.  At the moment, I don't think Linux has
>>> a way for the driver to pass that information up to the filesystem.
>>>
>>> The value that is most interesting to report is Atomic Write Unit Power
>>> Fail ("if you send a write no larger than this, the drive guarantees to
>>> write all of it or none of it"), minimum value 1 sector. [1]
>>>
>>> There's a proposal before the NVMe workgroup to add a boundary size/offset
>>> to modify AWUPF ("except if you cross this boundary, then AWUPF is not
>>> guaranteed").  Think RAID stripe crossing.
>>>
>>> So, three questions.  Is there somewhere already to pass boundary
>>> information up to the filesystem?  Can filesystems make use of a larger
>>> atomic write unit than a single sector?  And, if the device is internally
>>> a RAID device, is knowing the boundary size/offset useful?
>>>
>>>
>>> [1] There is also Atomic Write Unit Normal ("if you send two writes,
>>> neither of which is larger than this, subsequent reads will get either
>>> one or the other, not a mixture of both"), which I don't think we care
>>> about because the page cache prevents us from sending two writes which
>>> overlap with each other.
>>
>>
>> I think we really need the atomics to be vectored.  Send N writes which as a
>> unit are not larger than X, but which may span anywhere on device.  An array
>> with writeback cache, or a log structured squirrel in the FTL should be able
>> to provide this pretty easily?
>>
>> The immediate use case is mysql (16K writes) on a fragmented filesystem.
>> The FS needs to be able to collect a single atomic write made up of N 4K
>> sectors.
>
> How big does N need to be before it starts to be generally useful?
> Here it seems we're talking on the order to tens of writes, but for
> the upper bound Dave said that N could be in the hundreds of thousands

Right, if you ask the filesystem guys, we'll want to dump the entire 
contents of ram down to the storage in atomic fashion.  I do agree with 
Dave here, bigger is definitely better.

16K and up are useful, depending on which workload you're targeting. 
The fusion devices can do 1MB.

-chris

Re: Write atomicity guarantees

[Dave Chinner]

On Thu, Apr 24, 2014 at 02:50:23PM -0400, Chris Mason wrote:
> 
> 
> On 04/24/2014 02:23 PM, Dan Williams wrote:
> >On Thu, Apr 24, 2014 at 11:03 AM, Chris Mason <clm@fb.com> wrote:
> >>On 04/24/2014 01:39 PM, Matthew Wilcox wrote:
> >>>
> >>>
> >>>NVMe allows the drive to tell the host what atomicity guarantees it
> >>>provides for a write command.  At the moment, I don't think Linux has
> >>>a way for the driver to pass that information up to the filesystem.
> >>>
> >>>The value that is most interesting to report is Atomic Write Unit Power
> >>>Fail ("if you send a write no larger than this, the drive guarantees to
> >>>write all of it or none of it"), minimum value 1 sector. [1]
> >>>
> >>>There's a proposal before the NVMe workgroup to add a boundary size/offset
> >>>to modify AWUPF ("except if you cross this boundary, then AWUPF is not
> >>>guaranteed").  Think RAID stripe crossing.
> >>>
> >>>So, three questions.  Is there somewhere already to pass boundary
> >>>information up to the filesystem?  Can filesystems make use of a larger
> >>>atomic write unit than a single sector?  And, if the device is internally
> >>>a RAID device, is knowing the boundary size/offset useful?
> >>>
> >>>
> >>>[1] There is also Atomic Write Unit Normal ("if you send two writes,
> >>>neither of which is larger than this, subsequent reads will get either
> >>>one or the other, not a mixture of both"), which I don't think we care
> >>>about because the page cache prevents us from sending two writes which
> >>>overlap with each other.
> >>
> >>
> >>I think we really need the atomics to be vectored.  Send N writes which as a
> >>unit are not larger than X, but which may span anywhere on device.  An array
> >>with writeback cache, or a log structured squirrel in the FTL should be able
> >>to provide this pretty easily?
> >>
> >>The immediate use case is mysql (16K writes) on a fragmented filesystem.
> >>The FS needs to be able to collect a single atomic write made up of N 4K
> >>sectors.
> >
> >How big does N need to be before it starts to be generally useful?
> >Here it seems we're talking on the order to tens of writes, but for
> >the upper bound Dave said that N could be in the hundreds of thousands
> 
> Right, if you ask the filesystem guys, we'll want to dump the entire
> contents of ram down to the storage in atomic fashion.  I do agree
> with Dave here, bigger is definitely better.

Right, bigger is better, but what about minimum requirements?

The minimum requirement I need for converting XFS is around 4MB of
discontiguous single sector IOs for the worst case event.  That
covers the largest *single* atomic transaction log reservation we
currently make on XFS at 64k block sizes.

> 16K and up are useful, depending on which workload you're targeting.
> The fusion devices can do 1MB.

User data workloads, yes. The moment we start thinking about
atomic filesystem metadata updates, the requirements go way, way up....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com