agcount for 2TB, 4TB and 8TB drives

agcount for 2TB, 4TB and 8TB drives

[Gandalf Corvotempesta]

Hi to all,
i'm new to XFS.

Which is the proper agcount for 2TB, 4TB and 8TB drives (not part of any RAID) ?

mkfs.xfs automatically choosen 4 AGs. Isn't this too low ?

Re: agcount for 2TB, 4TB and 8TB drives

[Darrick J. Wong]

On Fri, Oct 06, 2017 at 10:46:20AM +0200, Gandalf Corvotempesta wrote:
> Hi to all,
> i'm new to XFS.
> 
> Which is the proper agcount for 2TB, 4TB and 8TB drives (not part of any RAID) ?
> 
> mkfs.xfs automatically choosen 4 AGs. Isn't this too low ?

No.  Have a look at calc_default_ag_geometry in libxcmd/topology.c for
how we calculate the default AG count / size.  4TB single-disks and
smaller get 4 AGs; larger than that get 1AG per TB.  RAID arrays are
different.

Semirelated question: for a solid state disk on a machine with high CPU
counts do we prefer agcount == cpucount to take advantage of the
high(er) iops and lack of seek time to increase parallelism?

(Not that I've studied that in depth.)

--D

> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: agcount for 2TB, 4TB and 8TB drives

[Eric Sandeen]

On 10/6/17 10:38 AM, Darrick J. Wong wrote:
> On Fri, Oct 06, 2017 at 10:46:20AM +0200, Gandalf Corvotempesta wrote:
>> Hi to all,
>> i'm new to XFS.
>>
>> Which is the proper agcount for 2TB, 4TB and 8TB drives (not part of any RAID) ?
>>
>> mkfs.xfs automatically choosen 4 AGs. Isn't this too low ?
> 
> No.  Have a look at calc_default_ag_geometry in libxcmd/topology.c for
> how we calculate the default AG count / size.  4TB single-disks and
> smaller get 4 AGs; larger than that get 1AG per TB.  RAID arrays are
> different.

Right; max AG size is 1T (for a default mkfs):

        /*
         * For a single underlying storage device over 4TB in size
         * use the maximum AG size.  Between 128MB and 4TB, just use
         * 4 AGs and scale up smoothly between min/max AG sizes.
         */

But if there is a stripe unit, it goes into multi-disk mode, assumes
you have more parallelism than a single spindle, and makes more AGs.

        /*
         * For the multidisk configs we choose an AG count based on the number
         * of data blocks available, trying to keep the number of AGs higher
         * than the single disk configurations. This makes the assumption that
         * larger filesystems have more parallelism available to them.
         */

If you have a single 8T disk with only a handful of heads, you won't
benefit from more AGs.

> Semirelated question: for a solid state disk on a machine with high CPU
> counts do we prefer agcount == cpucount to take advantage of the
> high(er) iops and lack of seek time to increase parallelism?
> 
> (Not that I've studied that in depth.)

Interesting question.  :)  Maybe harder to answer for SSD black boxes?

-Eric

Re: agcount for 2TB, 4TB and 8TB drives

[Dave Chinner]

On Fri, Oct 06, 2017 at 11:18:39AM -0500, Eric Sandeen wrote:
> On 10/6/17 10:38 AM, Darrick J. Wong wrote:
> > On Fri, Oct 06, 2017 at 10:46:20AM +0200, Gandalf Corvotempesta wrote:
> > Semirelated question: for a solid state disk on a machine with high CPU
> > counts do we prefer agcount == cpucount to take advantage of the
> > high(er) iops and lack of seek time to increase parallelism?
> > 
> > (Not that I've studied that in depth.)
> 
> Interesting question.  :)  Maybe harder to answer for SSD black boxes?

Easy: switch to multidisk mode if /sys/block/<dev>/queue/rotational
is zero after doing all the other checks. Then SSDs will get larger
AG counts automatically.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: agcount for 2TB, 4TB and 8TB drives

[Eric Sandeen]

On 10/6/17 5:20 PM, Dave Chinner wrote:
> On Fri, Oct 06, 2017 at 11:18:39AM -0500, Eric Sandeen wrote:
>> On 10/6/17 10:38 AM, Darrick J. Wong wrote:
>>> On Fri, Oct 06, 2017 at 10:46:20AM +0200, Gandalf Corvotempesta wrote:
>>> Semirelated question: for a solid state disk on a machine with high CPU
>>> counts do we prefer agcount == cpucount to take advantage of the
>>> high(er) iops and lack of seek time to increase parallelism?
>>>
>>> (Not that I've studied that in depth.)
>>
>> Interesting question.  :)  Maybe harder to answer for SSD black boxes?
> 
> Easy: switch to multidisk mode if /sys/block/<dev>/queue/rotational
> is zero after doing all the other checks. Then SSDs will get larger
> AG counts automatically.

The "hard part" was knowing just how much parallelism is actually inside
the black box.  But "multidisk mode" doesn't go too overboard, so yeah
that's probably fine.

-Eric
 
> Cheers,
> 
> Dave.
> 

Re: agcount for 2TB, 4TB and 8TB drives

[Avi Kivity]

On 10/07/2017 01:21 AM, Eric Sandeen wrote:
>
> On 10/6/17 5:20 PM, Dave Chinner wrote:
>> On Fri, Oct 06, 2017 at 11:18:39AM -0500, Eric Sandeen wrote:
>>> On 10/6/17 10:38 AM, Darrick J. Wong wrote:
>>>> On Fri, Oct 06, 2017 at 10:46:20AM +0200, Gandalf Corvotempesta wrote:
>>>> Semirelated question: for a solid state disk on a machine with high CPU
>>>> counts do we prefer agcount == cpucount to take advantage of the
>>>> high(er) iops and lack of seek time to increase parallelism?
>>>>
>>>> (Not that I've studied that in depth.)
>>> Interesting question.  :)  Maybe harder to answer for SSD black boxes?
>> Easy: switch to multidisk mode if /sys/block/<dev>/queue/rotational
>> is zero after doing all the other checks. Then SSDs will get larger
>> AG counts automatically.
> The "hard part" was knowing just how much parallelism is actually inside
> the black box.

It's often > 100.

>    But "multidisk mode" doesn't go too overboard, so yeah
> that's probably fine.
>


Is there a penalty associated with having too many allocation groups?

Re: agcount for 2TB, 4TB and 8TB drives

[Dave Chinner]

On Mon, Oct 09, 2017 at 11:05:56AM +0300, Avi Kivity wrote:
> On 10/07/2017 01:21 AM, Eric Sandeen wrote:
> >On 10/6/17 5:20 PM, Dave Chinner wrote:
> >>On Fri, Oct 06, 2017 at 11:18:39AM -0500, Eric Sandeen wrote:
> >>>On 10/6/17 10:38 AM, Darrick J. Wong wrote:
> >>>>On Fri, Oct 06, 2017 at 10:46:20AM +0200, Gandalf Corvotempesta wrote:
> >>>>Semirelated question: for a solid state disk on a machine with high CPU
> >>>>counts do we prefer agcount == cpucount to take advantage of the
> >>>>high(er) iops and lack of seek time to increase parallelism?
> >>>>
> >>>>(Not that I've studied that in depth.)
> >>>Interesting question.  :)  Maybe harder to answer for SSD black boxes?
> >>Easy: switch to multidisk mode if /sys/block/<dev>/queue/rotational
> >>is zero after doing all the other checks. Then SSDs will get larger
> >>AG counts automatically.
> >The "hard part" was knowing just how much parallelism is actually inside
> >the black box.
> 
> It's often > 100.

Sure, that might be the IO concurrency the SSD sees and handles, but
you very rarely require that much allocation parallelism in the
workload. Only a small amount of the IO submission path is actually
allocation work, so a single AG can provide plenty of async IO
parallelism before an AG is the limiting factor.

i.e. A single AG can typically support tens of thousands of free
space manipulations per second before the AG locks become the
bottleneck. Hence by the time you get to 16 AGs there's concurrency
available for (runs a concurrent workload and measures) at least
350,000 allocation transactions per second on relatively slow 5 year
old 8-core server CPUs. And that's CPU bound (16 CPUs all at >95%),
so faster, more recent CPUs will run much higher numbers.

IOws, don't confuse allocation concurrency with IO concurrency or
application concurrency. It's not the same thing and it is rarely a
limiting factor for most workloads, even the most IO intensive
ones...

> >   But "multidisk mode" doesn't go too overboard, so yeah
> >that's probably fine.
> 
> Is there a penalty associated with having too many allocation groups?

Yes. You break up the large contiguous free spaces into many smaller
free spaces and so can induce premature onset of filesystem aging
related performance degradations. And for spinning disks, more than
4-8AGs per spindle causes excessive seeks in mixed workloads and
degrades performance that way....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: agcount for 2TB, 4TB and 8TB drives

[Avi Kivity]

On 10/09/2017 02:23 PM, Dave Chinner wrote:
> On Mon, Oct 09, 2017 at 11:05:56AM +0300, Avi Kivity wrote:
>> On 10/07/2017 01:21 AM, Eric Sandeen wrote:
>>> On 10/6/17 5:20 PM, Dave Chinner wrote:
>>>> On Fri, Oct 06, 2017 at 11:18:39AM -0500, Eric Sandeen wrote:
>>>>> On 10/6/17 10:38 AM, Darrick J. Wong wrote:
>>>>>> On Fri, Oct 06, 2017 at 10:46:20AM +0200, Gandalf Corvotempesta wrote:
>>>>>> Semirelated question: for a solid state disk on a machine with high CPU
>>>>>> counts do we prefer agcount == cpucount to take advantage of the
>>>>>> high(er) iops and lack of seek time to increase parallelism?
>>>>>>
>>>>>> (Not that I've studied that in depth.)
>>>>> Interesting question.  :)  Maybe harder to answer for SSD black boxes?
>>>> Easy: switch to multidisk mode if /sys/block/<dev>/queue/rotational
>>>> is zero after doing all the other checks. Then SSDs will get larger
>>>> AG counts automatically.
>>> The "hard part" was knowing just how much parallelism is actually inside
>>> the black box.
>> It's often > 100.
> Sure, that might be the IO concurrency the SSD sees and handles, but
> you very rarely require that much allocation parallelism in the
> workload. Only a small amount of the IO submission path is actually
> allocation work, so a single AG can provide plenty of async IO
> parallelism before an AG is the limiting factor.

Sure. Can a single AG issue multiple I/Os, or is it single-threaded?

I understand that XFS_XFLAG_EXTSIZE and XFS_IOC_FSSETXATTR can reduce 
the AG's load. Is there a downside? for example, when I truncate + close 
the file, will the preallocated data still remain allocated? Do I need 
to return it with an fallocate()?

>
> i.e. A single AG can typically support tens of thousands of free
> space manipulations per second before the AG locks become the
> bottleneck. Hence by the time you get to 16 AGs there's concurrency
> available for (runs a concurrent workload and measures) at least
> 350,000 allocation transactions per second on relatively slow 5 year
> old 8-core server CPUs. And that's CPU bound (16 CPUs all at >95%),
> so faster, more recent CPUs will run much higher numbers.
>
> IOws, don't confuse allocation concurrency with IO concurrency or
> application concurrency. It's not the same thing and it is rarely a
> limiting factor for most workloads, even the most IO intensive
> ones...

In my load, the allocation load is not very high, but the impact of 
iowait is. So if I can reduce the chance of io_submit() blocking because 
of AG contention, then I'm happy to increase the number of AGs even if 
it hurts other things.

>
>>>    But "multidisk mode" doesn't go too overboard, so yeah
>>> that's probably fine.
>> Is there a penalty associated with having too many allocation groups?
> Yes. You break up the large contiguous free spaces into many smaller
> free spaces and so can induce premature onset of filesystem aging
> related performance degradations. And for spinning disks, more than
> 4-8AGs per spindle causes excessive seeks in mixed workloads and
> degrades performance that way....

For an SSD, would an AG per 10GB be reasonable? per 100GB?

Machines with 60-100 logical cores and low-tens of terabytes of SSD are 
becoming common.  How many AGs would work for such a machine? Again the 
allocation load is not very high (allocating a few GB/s with 32MB hints, 
so < 100 allocs/sec), but the penalty for contention is pretty high.

Thanks for the info!

> Cheers,
>
> Dave.

Re: agcount for 2TB, 4TB and 8TB drives

[Dave Chinner]

On Mon, Oct 09, 2017 at 06:46:41PM +0300, Avi Kivity wrote:
> 
> 
> On 10/09/2017 02:23 PM, Dave Chinner wrote:
> >On Mon, Oct 09, 2017 at 11:05:56AM +0300, Avi Kivity wrote:
> >>On 10/07/2017 01:21 AM, Eric Sandeen wrote:
> >>>On 10/6/17 5:20 PM, Dave Chinner wrote:
> >>>>On Fri, Oct 06, 2017 at 11:18:39AM -0500, Eric Sandeen wrote:
> >>>>>On 10/6/17 10:38 AM, Darrick J. Wong wrote:
> >>>>>>On Fri, Oct 06, 2017 at 10:46:20AM +0200, Gandalf Corvotempesta wrote:
> >>>>>>Semirelated question: for a solid state disk on a machine with high CPU
> >>>>>>counts do we prefer agcount == cpucount to take advantage of the
> >>>>>>high(er) iops and lack of seek time to increase parallelism?
> >>>>>>
> >>>>>>(Not that I've studied that in depth.)
> >>>>>Interesting question.  :)  Maybe harder to answer for SSD black boxes?
> >>>>Easy: switch to multidisk mode if /sys/block/<dev>/queue/rotational
> >>>>is zero after doing all the other checks. Then SSDs will get larger
> >>>>AG counts automatically.
> >>>The "hard part" was knowing just how much parallelism is actually inside
> >>>the black box.
> >>It's often > 100.
> >Sure, that might be the IO concurrency the SSD sees and handles, but
> >you very rarely require that much allocation parallelism in the
> >workload. Only a small amount of the IO submission path is actually
> >allocation work, so a single AG can provide plenty of async IO
> >parallelism before an AG is the limiting factor.
> 
> Sure. Can a single AG issue multiple I/Os, or is it single-threaded?

AGs don't issue IO. Applications issue IO, the filesystem allocates
space from AGs according to the write IO that passes through it.

i.e. when you don't do allocation in the write IO path or you are
doing read IOs, then the number of AGs is /completely irrelevant/.
In those cases a single AG can "support" the entire IO load your
application and storage subsystem can handle.

The only time an AG lock is taken in the IO path is during extent
allocation (i.e. writes). And, as I've already said, a single AG can
easily handle tens of thousands of allocation transactions a second
before it becomes a bottleneck.

IOWs, the worse case is that you'll get tens of thousands of IOs per
second through an AG.

> I understand that XFS_XFLAG_EXTSIZE and XFS_IOC_FSSETXATTR can
> reduce the AG's load.

Not really. They change the allocation pattern on the inode. This
changes how the inode data is laid out on disk, but it doesn't
necessarily change the allocation overhead of the write IO path.
That's all dependent on what the application IO patterns are and how
they match the extent size hints.

In general, nobody ever notices what the "load" on an AG is and
that's because almost no-one ever drives AGs to their limits.  The
mkfs defaults and the allocation policies keep the load distributed
across the filesystem and so storage subsystems almost always run
out of IO and/or seek capability before the filesystem runs out of
allocation concurrency. And, in general, most machines run out of
CPU power before they drive enough concurrency and load through the
filesystem that it starts contending on internal locks.

Sure, I have plenty of artificial workloads that drive this sort
contention, but no-one has a production workload that requires those
sorts of behaviours or creates the same level of lock contention
that these artificial workloads drive.

> Is there a downside? for example, when I
> truncate + close the file, will the preallocated data still remain
> allocated? Do I need to return it with an fallocate()?

No. Yes.

> >space manipulations per second before the AG locks become the
> >bottleneck. Hence by the time you get to 16 AGs there's concurrency
> >available for (runs a concurrent workload and measures) at least
> >350,000 allocation transactions per second on relatively slow 5 year
> >old 8-core server CPUs. And that's CPU bound (16 CPUs all at >95%),
> >so faster, more recent CPUs will run much higher numbers.
> >
> >IOws, don't confuse allocation concurrency with IO concurrency or
> >application concurrency. It's not the same thing and it is rarely a
> >limiting factor for most workloads, even the most IO intensive
> >ones...
> 
> In my load, the allocation load is not very high, but the impact of
> iowait is. So if I can reduce the chance of io_submit() blocking
> because of AG contention, then I'm happy to increase the number of
> AGs even if it hurts other things.

That's what RWF_NOWAIT is for. It pushes any write IO that requires
allocation into a thread rather possibly blocking the submitting
thread on any lock or IO in the allocation path.

> >>>   But "multidisk mode" doesn't go too overboard, so yeah
> >>>that's probably fine.
> >>Is there a penalty associated with having too many allocation groups?
> >Yes. You break up the large contiguous free spaces into many smaller
> >free spaces and so can induce premature onset of filesystem aging
> >related performance degradations. And for spinning disks, more than
> >4-8AGs per spindle causes excessive seeks in mixed workloads and
> >degrades performance that way....
> 
> For an SSD, would an AG per 10GB be reasonable? per 100GB?

No. Maybe.

Like I said, we can use the multi-disk mode in mkfs for this - it
already selects an appropriate number of AGs according to the size
of the filesystem appropriately.

> Machines with 60-100 logical cores and low-tens of terabytes of SSD
> are becoming common.  How many AGs would work for such a machine?

Multidisk default, which will be 32 AGs for anything in the 1->32TB
range. And over 32TB, you get 1 AG per TB...

> Again the allocation load is not very high (allocating a few GB/s
> with 32MB hints, so < 100 allocs/sec), but the penalty for
> contention is pretty high.

I think you're worrying about a non-problem. Use RWF_NOWAIT for your
AIO, and most of your existing IO submission blocking problems will
go away.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: agcount for 2TB, 4TB and 8TB drives

[Avi Kivity]

On 10/10/2017 01:03 AM, Dave Chinner wrote:
> On Mon, Oct 09, 2017 at 06:46:41PM +0300, Avi Kivity wrote:
>>
>> On 10/09/2017 02:23 PM, Dave Chinner wrote:
>>> On Mon, Oct 09, 2017 at 11:05:56AM +0300, Avi Kivity wrote:
>>>> On 10/07/2017 01:21 AM, Eric Sandeen wrote:
>>>>> On 10/6/17 5:20 PM, Dave Chinner wrote:
>>>>>> On Fri, Oct 06, 2017 at 11:18:39AM -0500, Eric Sandeen wrote:
>>>>>>> On 10/6/17 10:38 AM, Darrick J. Wong wrote:
>>>>>>>> On Fri, Oct 06, 2017 at 10:46:20AM +0200, Gandalf Corvotempesta wrote:
>>>>>>>> Semirelated question: for a solid state disk on a machine with high CPU
>>>>>>>> counts do we prefer agcount == cpucount to take advantage of the
>>>>>>>> high(er) iops and lack of seek time to increase parallelism?
>>>>>>>>
>>>>>>>> (Not that I've studied that in depth.)
>>>>>>> Interesting question.  :)  Maybe harder to answer for SSD black boxes?
>>>>>> Easy: switch to multidisk mode if /sys/block/<dev>/queue/rotational
>>>>>> is zero after doing all the other checks. Then SSDs will get larger
>>>>>> AG counts automatically.
>>>>> The "hard part" was knowing just how much parallelism is actually inside
>>>>> the black box.
>>>> It's often > 100.
>>> Sure, that might be the IO concurrency the SSD sees and handles, but
>>> you very rarely require that much allocation parallelism in the
>>> workload. Only a small amount of the IO submission path is actually
>>> allocation work, so a single AG can provide plenty of async IO
>>> parallelism before an AG is the limiting factor.
>> Sure. Can a single AG issue multiple I/Os, or is it single-threaded?
> AGs don't issue IO. Applications issue IO, the filesystem allocates
> space from AGs according to the write IO that passes through it.

What I meant was I/O in order to satisfy an allocation (read from the 
free extent btree or whatever), not the application's I/O.

>
> i.e. when you don't do allocation in the write IO path or you are
> doing read IOs, then the number of AGs is /completely irrelevant/.
> In those cases a single AG can "support" the entire IO load your
> application and storage subsystem can handle.
>
> The only time an AG lock is taken in the IO path is during extent
> allocation (i.e. writes). And, as I've already said, a single AG can
> easily handle tens of thousands of allocation transactions a second
> before it becomes a bottleneck.

Well, my own workload has at most a hundred allocations per second (32MB 
hints, 3GB/s writes)*, so I'm asking more to increase my understanding 
of XFS. But for me locks become a problem a lot sooner than they become 
a bottleneck, because I am using AIO and blocking in io_submit() 
destroys performance for me.

*below I see that it may wrong, so perhaps I have about 23k allocs/sec 
(128k buffers, 3GB/s writes).

>
> IOWs, the worse case is that you'll get tens of thousands of IOs per
> second through an AG.

For me, the worst case is worse. If io_submit() blocks, then there is 
nothing to utilize the processor core, and thus generate more I/Os that 
may have utilized the disk (for example, reads that don't need that 
lock). My use case is much more sensitive to lock contention.


Does the new RWF_NOWAIT goodness extend to AG locks? In that case I'll 
punt the io_submit to a worker thread that can block.

Ah, below you say it does.

>
>> I understand that XFS_XFLAG_EXTSIZE and XFS_IOC_FSSETXATTR can
>> reduce the AG's load.
> Not really. They change the allocation pattern on the inode. This
> changes how the inode data is laid out on disk, but it doesn't
> necessarily change the allocation overhead of the write IO path.
> That's all dependent on what the application IO patterns are and how
> they match the extent size hints.

I write 128k naturally-aligned writes using aio, so I expect it will 
match. Will every write go into the AG allocator, or just writes that 
cross a 32MB boundary?


>
> In general, nobody ever notices what the "load" on an AG is and
> that's because almost no-one ever drives AGs to their limits.  The
> mkfs defaults and the allocation policies keep the load distributed
> across the filesystem and so storage subsystems almost always run
> out of IO and/or seek capability before the filesystem runs out of
> allocation concurrency. And, in general, most machines run out of
> CPU power before they drive enough concurrency and load through the
> filesystem that it starts contending on internal locks.
>
> Sure, I have plenty of artificial workloads that drive this sort
> contention, but no-one has a production workload that requires those
> sorts of behaviours or creates the same level of lock contention
> that these artificial workloads drive.

I've certainly seen lock contention in XFS, there was a recent thread 
(started by Tomasz) where a filesystem that was close to full was almost 
completely degraded for us.

Again we are more sensitive to contention than other workloads, because 
contention for us doesn't just block the work downstream from lock 
acquisition, it blocks all other work on that core for the duration.

>
>> Is there a downside? for example, when I
>> truncate + close the file, will the preallocated data still remain
>> allocated? Do I need to return it with an fallocate()?
> No. Yes.

Thanks. Most of my files are much larger, so the waste isn't too high, 
but it's still waste.

>
>>> space manipulations per second before the AG locks become the
>>> bottleneck. Hence by the time you get to 16 AGs there's concurrency
>>> available for (runs a concurrent workload and measures) at least
>>> 350,000 allocation transactions per second on relatively slow 5 year
>>> old 8-core server CPUs. And that's CPU bound (16 CPUs all at >95%),
>>> so faster, more recent CPUs will run much higher numbers.
>>>
>>> IOws, don't confuse allocation concurrency with IO concurrency or
>>> application concurrency. It's not the same thing and it is rarely a
>>> limiting factor for most workloads, even the most IO intensive
>>> ones...
>> In my load, the allocation load is not very high, but the impact of
>> iowait is. So if I can reduce the chance of io_submit() blocking
>> because of AG contention, then I'm happy to increase the number of
>> AGs even if it hurts other things.
> That's what RWF_NOWAIT is for. It pushes any write IO that requires
> allocation into a thread rather possibly blocking the submitting
> thread on any lock or IO in the allocation path.

Excellent, we'll use that, although it will be years before our users 
see the benefit.

>>>>>    But "multidisk mode" doesn't go too overboard, so yeah
>>>>> that's probably fine.
>>>> Is there a penalty associated with having too many allocation groups?
>>> Yes. You break up the large contiguous free spaces into many smaller
>>> free spaces and so can induce premature onset of filesystem aging
>>> related performance degradations. And for spinning disks, more than
>>> 4-8AGs per spindle causes excessive seeks in mixed workloads and
>>> degrades performance that way....
>> For an SSD, would an AG per 10GB be reasonable? per 100GB?
> No. Maybe.
>
> Like I said, we can use the multi-disk mode in mkfs for this - it
> already selects an appropriate number of AGs according to the size
> of the filesystem appropriately.
>
>> Machines with 60-100 logical cores and low-tens of terabytes of SSD
>> are becoming common.  How many AGs would work for such a machine?
> Multidisk default, which will be 32 AGs for anything in the 1->32TB
> range. And over 32TB, you get 1 AG per TB...


Ok. Then doubling it so that each logical core has an AG wouldn't be 
such a big change.

>
>> Again the allocation load is not very high (allocating a few GB/s
>> with 32MB hints, so < 100 allocs/sec), but the penalty for
>> contention is pretty high.
> I think you're worrying about a non-problem. Use RWF_NOWAIT for your
> AIO, and most of your existing IO submission blocking problems will
> go away.
>

We'll start using RWF_NOWAIT, but many of our users are on a 3.10 
derivative kernel and won't install 4.14-rc6 on their production 
clusters. If a mkfs tweak can help them, then I'll happily do it.

I don't have direct proof that too few AGs are causing problems for me, 
but I've seen many traces showing XFS blocking, and like I said, it's a 
disaster for us. Unfortunately these problems are hard to reproduce and 
are expensive to test.

Re: agcount for 2TB, 4TB and 8TB drives

[Dave Chinner]

On Tue, Oct 10, 2017 at 12:07:42PM +0300, Avi Kivity wrote:
> On 10/10/2017 01:03 AM, Dave Chinner wrote:
> >>On 10/09/2017 02:23 PM, Dave Chinner wrote:
> >>>On Mon, Oct 09, 2017 at 11:05:56AM +0300, Avi Kivity wrote:
> >>>Sure, that might be the IO concurrency the SSD sees and handles, but
> >>>you very rarely require that much allocation parallelism in the
> >>>workload. Only a small amount of the IO submission path is actually
> >>>allocation work, so a single AG can provide plenty of async IO
> >>>parallelism before an AG is the limiting factor.
> >>Sure. Can a single AG issue multiple I/Os, or is it single-threaded?
> >AGs don't issue IO. Applications issue IO, the filesystem allocates
> >space from AGs according to the write IO that passes through it.
> 
> What I meant was I/O in order to satisfy an allocation (read from
> the free extent btree or whatever), not the application's I/O.

Once you're in the per-AG allocator context, it is single threaded
until the allocation is complete. We do things like btree block
readahead to minimise IO wait times, but we can't completely hide
things like metadata read Io wait time when it is required to make
progress.

> >>I understand that XFS_XFLAG_EXTSIZE and XFS_IOC_FSSETXATTR can
> >>reduce the AG's load.
> >Not really. They change the allocation pattern on the inode. This
> >changes how the inode data is laid out on disk, but it doesn't
> >necessarily change the allocation overhead of the write IO path.
> >That's all dependent on what the application IO patterns are and how
> >they match the extent size hints.
> 
> I write 128k naturally-aligned writes using aio, so I expect it will
> match. Will every write go into the AG allocator, or just writes
> that cross a 32MB boundary?

It enters an allocation only when an allocation is required. i.e.
only when the write lands in a hole. If you're doing sequential 128k
writes and using 32MB extent size hints, then it only allocates once
every 32768/128 = 256 writes. If you are doing random IO into a
sparse file, then it all bets are off.

> >That's what RWF_NOWAIT is for. It pushes any write IO that requires
> >allocation into a thread rather possibly blocking the submitting
> >thread on any lock or IO in the allocation path.
> 
> Excellent, we'll use that, although it will be years before our
> users see the benefit.

Well, that's really in your control, not mine.

The disconnect between upstream progress and LTS production
systems is not something upstream can do anything about. Often the
problems LTS production systems see are already solved upstream and
so the only answer we can really give you here is "upgrade, backport
features your customers need yourself, or pay someone else to
maintain a backport with the features you need".

> >>Machines with 60-100 logical cores and low-tens of terabytes of SSD
> >>are becoming common.  How many AGs would work for such a machine?
> >Multidisk default, which will be 32 AGs for anything in the 1->32TB
> >range. And over 32TB, you get 1 AG per TB...
> 
> 
> Ok. Then doubling it so that each logical core has an AG wouldn't be
> such a big change.

But it won't make any difference to your workload because there's no
relationship between CPU cores and the AG selected for allocation.
The AG selection is based on filesystem relationships (e.g. local to
parent directory inode), and so if you have two files in the same
directory they will start trying to allocate from the same AG even
thought hey get written from different cores concurrently. The only
time they'll get moved into different AGs is if there is allocation
contention.

Yes, the allocator algorithms detect AG contention internally and
switch to uncontended AGs rather than blocking. There's /lots/ of
stuff inside the allocators to minimise blocking - that's one of the
reasons you see less submission blocking problems on XFS than other
filesytsems. If you're not getting threads blocking waiting to get
AGF locks, then you most certainly don't have allocator contention.
Even if you do have threads blocking on AGF locks, that could simply
be a sign you are running too close to ENOSPC, not contention...

The reality is, however, that even an uncontended AG can block if
the necessary metadata isn't in memory, or the log is full, or
memory cannot be immediately allocated, etc. RWF_NOWAIT avoids the
whole class of "allocator can block" problem...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: agcount for 2TB, 4TB and 8TB drives

[Avi Kivity]

On 10/11/2017 01:55 AM, Dave Chinner wrote:
> On Tue, Oct 10, 2017 at 12:07:42PM +0300, Avi Kivity wrote:
>> On 10/10/2017 01:03 AM, Dave Chinner wrote:
>>>> On 10/09/2017 02:23 PM, Dave Chinner wrote:
>>>>> On Mon, Oct 09, 2017 at 11:05:56AM +0300, Avi Kivity wrote:
>>>>> Sure, that might be the IO concurrency the SSD sees and handles, but
>>>>> you very rarely require that much allocation parallelism in the
>>>>> workload. Only a small amount of the IO submission path is actually
>>>>> allocation work, so a single AG can provide plenty of async IO
>>>>> parallelism before an AG is the limiting factor.
>>>> Sure. Can a single AG issue multiple I/Os, or is it single-threaded?
>>> AGs don't issue IO. Applications issue IO, the filesystem allocates
>>> space from AGs according to the write IO that passes through it.
>> What I meant was I/O in order to satisfy an allocation (read from
>> the free extent btree or whatever), not the application's I/O.
> Once you're in the per-AG allocator context, it is single threaded
> until the allocation is complete. We do things like btree block
> readahead to minimise IO wait times, but we can't completely hide
> things like metadata read Io wait time when it is required to make
> progress.

I see, thanks. Will RWF_NOWAIT detect the need to do I/O for the free 
space btree, or just contention? (I expect the latter from the patches 
I've seen, but perhaps I missed something).

I imagine I'll have a lot of amortization there: if a 32MB allocation 
fails, the subsequent 32MB allocation for the same file will likely hit 
the same location and be satisified from cache. My workload is pure 
O_DIRECT so no memory pressure in the kernel.

>>>> I understand that XFS_XFLAG_EXTSIZE and XFS_IOC_FSSETXATTR can
>>>> reduce the AG's load.
>>> Not really. They change the allocation pattern on the inode. This
>>> changes how the inode data is laid out on disk, but it doesn't
>>> necessarily change the allocation overhead of the write IO path.
>>> That's all dependent on what the application IO patterns are and how
>>> they match the extent size hints.
>> I write 128k naturally-aligned writes using aio, so I expect it will
>> match. Will every write go into the AG allocator, or just writes
>> that cross a 32MB boundary?
> It enters an allocation only when an allocation is required. i.e.
> only when the write lands in a hole. If you're doing sequential 128k
> writes and using 32MB extent size hints, then it only allocates once
> every 32768/128 = 256 writes. If you are doing random IO into a
> sparse file, then it all bets are off.

Pure sequential writes.


>
>>> That's what RWF_NOWAIT is for. It pushes any write IO that requires
>>> allocation into a thread rather possibly blocking the submitting
>>> thread on any lock or IO in the allocation path.
>> Excellent, we'll use that, although it will be years before our
>> users see the benefit.
> Well, that's really in your control, not mine.
>
> The disconnect between upstream progress and LTS production
> systems is not something upstream can do anything about. Often the
> problems LTS production systems see are already solved upstream and
> so the only answer we can really give you here is "upgrade, backport
> features your customers need yourself, or pay someone else to
> maintain a backport with the features you need".

I understand the situation. This was to explain why I'm looking for 
workarounds in deployed code when fixes in new code are available. My 
users/customers don't run kernels provided by me.

>>>> Machines with 60-100 logical cores and low-tens of terabytes of SSD
>>>> are becoming common.  How many AGs would work for such a machine?
>>> Multidisk default, which will be 32 AGs for anything in the 1->32TB
>>> range. And over 32TB, you get 1 AG per TB...
>>
>> Ok. Then doubling it so that each logical core has an AG wouldn't be
>> such a big change.
> But it won't make any difference to your workload because there's no
> relationship between CPU cores and the AG selected for allocation.
> The AG selection is based on filesystem relationships (e.g. local to
> parent directory inode), and so if you have two files in the same
> directory they will start trying to allocate from the same AG even
> thought hey get written from different cores concurrently. The only
> time they'll get moved into different AGs is if there is allocation
> contention.

Unfortunately, all cores writing files in the same directory is exactly 
my workload. I can change it, but there is a backwards compatibility 
cost to that change. I can probably also trick XFS by creating the file 
in a dedicated subdirectory and rename()ing it later.

>
> Yes, the allocator algorithms detect AG contention internally and
> switch to uncontended AGs rather than blocking. There's /lots/ of
> stuff inside the allocators to minimise blocking - that's one of the
> reasons you see less submission blocking problems on XFS than other
> filesytsems. If you're not getting threads blocking waiting to get
> AGF locks, then you most certainly don't have allocator contention.
> Even if you do have threads blocking on AGF locks, that could simply
> be a sign you are running too close to ENOSPC, not contention...
>
> The reality is, however, that even an uncontended AG can block if
> the necessary metadata isn't in memory, or the log is full, or
> memory cannot be immediately allocated, etc. RWF_NOWAIT avoids the
> whole class of "allocator can block" problem...


Thanks. I do have blocks from time to time, but we were not able to 
pinpoint the cause as I don't own those systems (and also lack knowledge 
about the internals). At least one issue _was_ related to free space 
running out, so that fits.

The vast majority of the time XFS AIO works very well. The problem is 
that when problems do happen, performance drops of sharply, and it's 
often in a situation that's hard to debug.

Re: agcount for 2TB, 4TB and 8TB drives

[Dave Chinner]

On Fri, Oct 13, 2017 at 11:13:24AM +0300, Avi Kivity wrote:
> On 10/11/2017 01:55 AM, Dave Chinner wrote:
> >On Tue, Oct 10, 2017 at 12:07:42PM +0300, Avi Kivity wrote:
> >>On 10/10/2017 01:03 AM, Dave Chinner wrote:
> >>>>On 10/09/2017 02:23 PM, Dave Chinner wrote:
> >>>>>On Mon, Oct 09, 2017 at 11:05:56AM +0300, Avi Kivity wrote:
> >>>>>Sure, that might be the IO concurrency the SSD sees and handles, but
> >>>>>you very rarely require that much allocation parallelism in the
> >>>>>workload. Only a small amount of the IO submission path is actually
> >>>>>allocation work, so a single AG can provide plenty of async IO
> >>>>>parallelism before an AG is the limiting factor.
> >>>>Sure. Can a single AG issue multiple I/Os, or is it single-threaded?
> >>>AGs don't issue IO. Applications issue IO, the filesystem allocates
> >>>space from AGs according to the write IO that passes through it.
> >>What I meant was I/O in order to satisfy an allocation (read from
> >>the free extent btree or whatever), not the application's I/O.
> >Once you're in the per-AG allocator context, it is single threaded
> >until the allocation is complete. We do things like btree block
> >readahead to minimise IO wait times, but we can't completely hide
> >things like metadata read Io wait time when it is required to make
> >progress.
> 
> I see, thanks. Will RWF_NOWAIT detect the need to do I/O for the
> free space btree, or just contention? (I expect the latter from the
> patches I've seen, but perhaps I missed something).

No, it checks at a high level whether allocation is needed (i.e. IO
into a hole) and if allocation is needed, it punts the IO
immediately to the background thread and returns to userspace. i.e.
it never gets near the allocator to begin with....

Like I said before, RWF_NOWAIT prevents entire classes of
AIO submission blocking issues from occuring. Use it and almost all
filesystem blocking concerns go away....

> The vast majority of the time XFS AIO works very well. The problem
> is that when problems do happen, performance drops of sharply, and
> it's often in a situation that's hard to debug.

Yes, and that's made worse by there being relatively few people
around with the knowledge to be able to find the the root cause when
it does happen. :/

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: agcount for 2TB, 4TB and 8TB drives

[Avi Kivity]

On 10/15/2017 01:42 AM, Dave Chinner wrote:
> On Fri, Oct 13, 2017 at 11:13:24AM +0300, Avi Kivity wrote:
>> On 10/11/2017 01:55 AM, Dave Chinner wrote:
>>> On Tue, Oct 10, 2017 at 12:07:42PM +0300, Avi Kivity wrote:
>>>> On 10/10/2017 01:03 AM, Dave Chinner wrote:
>>>>>> On 10/09/2017 02:23 PM, Dave Chinner wrote:
>>>>>>> On Mon, Oct 09, 2017 at 11:05:56AM +0300, Avi Kivity wrote:
>>>>>>> Sure, that might be the IO concurrency the SSD sees and handles, but
>>>>>>> you very rarely require that much allocation parallelism in the
>>>>>>> workload. Only a small amount of the IO submission path is actually
>>>>>>> allocation work, so a single AG can provide plenty of async IO
>>>>>>> parallelism before an AG is the limiting factor.
>>>>>> Sure. Can a single AG issue multiple I/Os, or is it single-threaded?
>>>>> AGs don't issue IO. Applications issue IO, the filesystem allocates
>>>>> space from AGs according to the write IO that passes through it.
>>>> What I meant was I/O in order to satisfy an allocation (read from
>>>> the free extent btree or whatever), not the application's I/O.
>>> Once you're in the per-AG allocator context, it is single threaded
>>> until the allocation is complete. We do things like btree block
>>> readahead to minimise IO wait times, but we can't completely hide
>>> things like metadata read Io wait time when it is required to make
>>> progress.
>> I see, thanks. Will RWF_NOWAIT detect the need to do I/O for the
>> free space btree, or just contention? (I expect the latter from the
>> patches I've seen, but perhaps I missed something).
> No, it checks at a high level whether allocation is needed (i.e. IO
> into a hole) and if allocation is needed, it punts the IO
> immediately to the background thread and returns to userspace. i.e.
> it never gets near the allocator to begin with....


Interesting, that's both good and bad. Good, because we avoided a 
potential stall. Bad, because if the stall would not actually have 
happened (lock not contended, btree nodes cached) then we got punted to 
the helper thread which is a more expensive path.

In fact we don't even need to try the write, we know that every 
32MB/128k = 256 writes we will hit an allocation. Perhaps we can 
fallocate() the next 32MB chunk while writing to the previous one. If 
fallocate() is fast enough, writes will both never block/fail. If it's 
not, then we'll block/fail, but the likelihood is reduced. We can even 
increase the chunk size if we see we're getting blocked.

Even better would be if XFS would detect the sequential write and start 
allocating ahead of it.

>
> Like I said before, RWF_NOWAIT prevents entire classes of
> AIO submission blocking issues from occuring. Use it and almost all
> filesystem blocking concerns go away....

I will indeed.

Re: agcount for 2TB, 4TB and 8TB drives

[Dave Chinner]

On Sun, Oct 15, 2017 at 12:36:03PM +0300, Avi Kivity wrote:
> 
> 
> On 10/15/2017 01:42 AM, Dave Chinner wrote:
> >On Fri, Oct 13, 2017 at 11:13:24AM +0300, Avi Kivity wrote:
> >>On 10/11/2017 01:55 AM, Dave Chinner wrote:
> >>>On Tue, Oct 10, 2017 at 12:07:42PM +0300, Avi Kivity wrote:
> >>>>On 10/10/2017 01:03 AM, Dave Chinner wrote:
> >>>>>>On 10/09/2017 02:23 PM, Dave Chinner wrote:
> >>>>>>>On Mon, Oct 09, 2017 at 11:05:56AM +0300, Avi Kivity wrote:
> >>>>>>>Sure, that might be the IO concurrency the SSD sees and handles, but
> >>>>>>>you very rarely require that much allocation parallelism in the
> >>>>>>>workload. Only a small amount of the IO submission path is actually
> >>>>>>>allocation work, so a single AG can provide plenty of async IO
> >>>>>>>parallelism before an AG is the limiting factor.
> >>>>>>Sure. Can a single AG issue multiple I/Os, or is it single-threaded?
> >>>>>AGs don't issue IO. Applications issue IO, the filesystem allocates
> >>>>>space from AGs according to the write IO that passes through it.
> >>>>What I meant was I/O in order to satisfy an allocation (read from
> >>>>the free extent btree or whatever), not the application's I/O.
> >>>Once you're in the per-AG allocator context, it is single threaded
> >>>until the allocation is complete. We do things like btree block
> >>>readahead to minimise IO wait times, but we can't completely hide
> >>>things like metadata read Io wait time when it is required to make
> >>>progress.
> >>I see, thanks. Will RWF_NOWAIT detect the need to do I/O for the
> >>free space btree, or just contention? (I expect the latter from the
> >>patches I've seen, but perhaps I missed something).
> >No, it checks at a high level whether allocation is needed (i.e. IO
> >into a hole) and if allocation is needed, it punts the IO
> >immediately to the background thread and returns to userspace. i.e.
> >it never gets near the allocator to begin with....
> 
> Interesting, that's both good and bad. Good, because we avoided a
> potential stall. Bad, because if the stall would not actually have
> happened (lock not contended, btree nodes cached) then we got punted
> to the helper thread which is a more expensive path.

Avoiding latency has costs in complexity, resources and CPU time.
That's why we've never ended up with a fully generic async syscall
interface in the kernel - every time someone tries, it dies the
death of complexity.

RWF_NOWAIT is simple, easy to maintain and has, in most cases, no
observable overhead.

> In fact we don't even need to try the write, we know that every
> 32MB/128k = 256 writes we will hit an allocation. Perhaps we can
> fallocate() the next 32MB chunk while writing to the previous one.

fallocate will block *all* IO and mmap faults on that file, not just
the ones that require allocation. fallocate creates a complete IO
submission pipeline stall, punting all new IO submissions to the
background worker where they will block until fallocate completes.

IOWs, in terms of overhead, IO submission efficiency and IO pipeline
bubbles, fallocate is close the worst thing you can possibly do.
Extent size hints are far more efficient and less intrusive than
manually using fallocate from userspace.

> If fallocate() is fast enough, writes will both never block/fail. If
> it's not, then we'll block/fail, but the likelihood is reduced. We
> can even increase the chunk size if we see we're getting blocked.

If you call fallocate, other AIO writes will always get blocked
because fallocate creates an IO submission barrier. fallocate might
be fast, but it's also a total IO submission serialisation point and
so has a much more significant effect on IO submission latency when
compared to doing allocation directly in the IO path via extent size
hints...

> Even better would be if XFS would detect the sequential write and
> start allocating ahead of it.

That's what delayed allocation does with buffered IO. We
specifically do not do that with direct IO because it's direct IO
and we only do exactly what the IO the user submits requires us to
do.

As it is, I'm not sure that it would gain us anything over extent
size hints because they are effectively doing exactly the same thing
(i.e.  allocate ahead) on every write that hits a hole beyond
EOF when extending the file....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: agcount for 2TB, 4TB and 8TB drives

[Avi Kivity]

On 10/16/2017 01:00 AM, Dave Chinner wrote:
> On Sun, Oct 15, 2017 at 12:36:03PM +0300, Avi Kivity wrote:
>>
>> On 10/15/2017 01:42 AM, Dave Chinner wrote:
>>> On Fri, Oct 13, 2017 at 11:13:24AM +0300, Avi Kivity wrote:
>>>> On 10/11/2017 01:55 AM, Dave Chinner wrote:
>>>>> On Tue, Oct 10, 2017 at 12:07:42PM +0300, Avi Kivity wrote:
>>>>>> On 10/10/2017 01:03 AM, Dave Chinner wrote:
>>>>>>>> On 10/09/2017 02:23 PM, Dave Chinner wrote:
>>>>>>>>> On Mon, Oct 09, 2017 at 11:05:56AM +0300, Avi Kivity wrote:
>>>>>>>>> Sure, that might be the IO concurrency the SSD sees and handles, but
>>>>>>>>> you very rarely require that much allocation parallelism in the
>>>>>>>>> workload. Only a small amount of the IO submission path is actually
>>>>>>>>> allocation work, so a single AG can provide plenty of async IO
>>>>>>>>> parallelism before an AG is the limiting factor.
>>>>>>>> Sure. Can a single AG issue multiple I/Os, or is it single-threaded?
>>>>>>> AGs don't issue IO. Applications issue IO, the filesystem allocates
>>>>>>> space from AGs according to the write IO that passes through it.
>>>>>> What I meant was I/O in order to satisfy an allocation (read from
>>>>>> the free extent btree or whatever), not the application's I/O.
>>>>> Once you're in the per-AG allocator context, it is single threaded
>>>>> until the allocation is complete. We do things like btree block
>>>>> readahead to minimise IO wait times, but we can't completely hide
>>>>> things like metadata read Io wait time when it is required to make
>>>>> progress.
>>>> I see, thanks. Will RWF_NOWAIT detect the need to do I/O for the
>>>> free space btree, or just contention? (I expect the latter from the
>>>> patches I've seen, but perhaps I missed something).
>>> No, it checks at a high level whether allocation is needed (i.e. IO
>>> into a hole) and if allocation is needed, it punts the IO
>>> immediately to the background thread and returns to userspace. i.e.
>>> it never gets near the allocator to begin with....
>> Interesting, that's both good and bad. Good, because we avoided a
>> potential stall. Bad, because if the stall would not actually have
>> happened (lock not contended, btree nodes cached) then we got punted
>> to the helper thread which is a more expensive path.
> Avoiding latency has costs in complexity, resources and CPU time.
> That's why we've never ended up with a fully generic async syscall
> interface in the kernel - every time someone tries, it dies the
> death of complexity.
>
> RWF_NOWAIT is simple, easy to maintain and has, in most cases, no
> observable overhead.

There is no observable overhead in the kernel, but there will be some 
for the application. As soon as we cross a hint boundary writes start to 
fail, and the application needs to move them to a helper thread and 
re-submit them. These duplicate submissions happen until the helper 
thread is able to respond, and the first write manages to allocate the 
space.

Without RWF_NOWAIT, there are two possibilities: either you get lucky 
and the first write to cross the boundary doesn't block, or you get  
unlucky and you stall. There's no doubt that RWF_NOWAIT is a lot better, 
but it does cause the system to do some more work. I guess it can be 
amortized away with larger hints.

>> In fact we don't even need to try the write, we know that every
>> 32MB/128k = 256 writes we will hit an allocation. Perhaps we can
>> fallocate() the next 32MB chunk while writing to the previous one.
> fallocate will block *all* IO and mmap faults on that file, not just
> the ones that require allocation. fallocate creates a complete IO
> submission pipeline stall, punting all new IO submissions to the
> background worker where they will block until fallocate completes.

Ok, I'll stay away from it, except during close time to remove unused 
extents.

> IOWs, in terms of overhead, IO submission efficiency and IO pipeline
> bubbles, fallocate is close the worst thing you can possibly do.
> Extent size hints are far more efficient and less intrusive than
> manually using fallocate from userspace.
>
>> If fallocate() is fast enough, writes will both never block/fail. If
>> it's not, then we'll block/fail, but the likelihood is reduced. We
>> can even increase the chunk size if we see we're getting blocked.
> If you call fallocate, other AIO writes will always get blocked
> because fallocate creates an IO submission barrier. fallocate might
> be fast, but it's also a total IO submission serialisation point and
> so has a much more significant effect on IO submission latency when
> compared to doing allocation directly in the IO path via extent size
> hints...

Got it.

>> Even better would be if XFS would detect the sequential write and
>> start allocating ahead of it.
> That's what delayed allocation does with buffered IO. We
> specifically do not do that with direct IO because it's direct IO
> and we only do exactly what the IO the user submits requires us to
> do.
>
> As it is, I'm not sure that it would gain us anything over extent
> size hints because they are effectively doing exactly the same thing
> (i.e.  allocate ahead) on every write that hits a hole beyond
> EOF when extending the file....

If I understand correctly, you do get momentary serialization when you 
cross a hint boundary, while with allocate ahead, you would not.

Re: agcount for 2TB, 4TB and 8TB drives

[Dave Chinner]

On Mon, Oct 16, 2017 at 01:00:32PM +0300, Avi Kivity wrote:
> On 10/16/2017 01:00 AM, Dave Chinner wrote:
> >On Sun, Oct 15, 2017 at 12:36:03PM +0300, Avi Kivity wrote:
> >>On 10/15/2017 01:42 AM, Dave Chinner wrote:
> >>>On Fri, Oct 13, 2017 at 11:13:24AM +0300, Avi Kivity wrote:
> >>Even better would be if XFS would detect the sequential write and
> >>start allocating ahead of it.
> >That's what delayed allocation does with buffered IO. We
> >specifically do not do that with direct IO because it's direct IO
> >and we only do exactly what the IO the user submits requires us to
> >do.
> >
> >As it is, I'm not sure that it would gain us anything over extent
> >size hints because they are effectively doing exactly the same thing
> >(i.e.  allocate ahead) on every write that hits a hole beyond
> >EOF when extending the file....
> 
> If I understand correctly, you do get momentary serialization when
> you cross a hint boundary, while with allocate ahead, you would not.

Allocate ahead still requires a threshold to be crossed to trigger
allocation. So it doesn't get rid of allocation, it just changes
what IO triggers it.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: agcount for 2TB, 4TB and 8TB drives

[Christoph Hellwig]

On Mon, Oct 16, 2017 at 09:00:19AM +1100, Dave Chinner wrote:
> fallocate will block *all* IO and mmap faults on that file, not just
> the ones that require allocation. fallocate creates a complete IO
> submission pipeline stall, punting all new IO submissions to the
> background worker where they will block until fallocate completes.

Not sure if it helps Avis case, but I think we could relax the
fallocate exclusive i_rwsem requirement a bit, similar to direct I/O
writes.  Basically as long as your fallocate is block aligned we
should be fine with a shared iolock for the "normal" allocating
fallocate (discounting things like hole punches and extent shifts
of course).