[LSF/MM ATTEND] memory allocation scope

[LSF/MM ATTEND] memory allocation scope

[Goldwyn Rodrigues]

Discussion with the memory folks towards scope based allocation
I am working on converting some of the GFP_NOFS memory allocation calls
to new scope API [1]. While other allocation types (noio, nofs,
noreclaim) are covered. Are there plans for identifying scope of
GFP_ATOMIC allocations? This should cover most (if not all) of the
allocation scope.

Transient Errors with direct I/O
In a large enough direct I/O, bios are split. If any of these bios get
an error, the whole I/O is marked as erroneous. What this means at the
application level is that part of your direct I/O data may be written
while part may not be. In the end, you can have an inconsistent write
with some parts of it written and some not. Currently the applications
need to overwrite the whole write() again.

Other things I am interested in:
 - new mount API
 - Online Filesystem Check
 - FS cache shrinking

[1] https://lwn.net/Articles/710545/


-- 
Goldwyn

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM ATTEND] memory allocation scope

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]

On Wed, Feb 14 2018, Goldwyn Rodrigues wrote:

> Discussion with the memory folks towards scope based allocation
> I am working on converting some of the GFP_NOFS memory allocation calls
> to new scope API [1]. While other allocation types (noio, nofs,
> noreclaim) are covered. Are there plans for identifying scope of
> GFP_ATOMIC allocations? This should cover most (if not all) of the
> allocation scope.
>
> Transient Errors with direct I/O
> In a large enough direct I/O, bios are split. If any of these bios get
> an error, the whole I/O is marked as erroneous. What this means at the
> application level is that part of your direct I/O data may be written
> while part may not be. In the end, you can have an inconsistent write
> with some parts of it written and some not. Currently the applications
> need to overwrite the whole write() again.

So?
If that is a problem for the application, maybe it should use smaller
writes.  If smaller writes cause higher latency, then use aio to submit
them.

I doubt that splitting bios is the only thing that can cause a write
that reported as EIO to have partially completed.  An application should
*always* assume that EIO from a write means that the data on the device
is indistinguishable from garbage - shouldn't it?

NeilBrown


>
> Other things I am interested in:
>  - new mount API
>  - Online Filesystem Check
>  - FS cache shrinking
>
> [1] https://lwn.net/Articles/710545/
>
>
> -- 
> Goldwyn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [LSF/MM ATTEND] memory allocation scope

[Goldwyn Rodrigues]

On 02/14/2018 09:53 PM, NeilBrown wrote:
> On Wed, Feb 14 2018, Goldwyn Rodrigues wrote:
> 
>> Discussion with the memory folks towards scope based allocation
>> I am working on converting some of the GFP_NOFS memory allocation calls
>> to new scope API [1]. While other allocation types (noio, nofs,
>> noreclaim) are covered. Are there plans for identifying scope of
>> GFP_ATOMIC allocations? This should cover most (if not all) of the
>> allocation scope.
>>
>> Transient Errors with direct I/O
>> In a large enough direct I/O, bios are split. If any of these bios get
>> an error, the whole I/O is marked as erroneous. What this means at the
>> application level is that part of your direct I/O data may be written
>> while part may not be. In the end, you can have an inconsistent write
>> with some parts of it written and some not. Currently the applications
>> need to overwrite the whole write() again.
> 
> So?
> If that is a problem for the application, maybe it should use smaller
> writes.  If smaller writes cause higher latency, then use aio to submit
> them.
> 
> I doubt that splitting bios is the only thing that can cause a write
> that reported as EIO to have partially completed.  An application should
> *always* assume that EIO from a write means that the data on the device
> is indistinguishable from garbage - shouldn't it?
> 

Yes, and that is what I got from others as well. The scenario is not
deterministic of the contents of the file in case of overwriting a file.
And no, splitting bios is not the only reason you can have partial
write. This is different from what buffered I/O would result in, where a
partial write may not be an error and returns the bytes written.

Perhaps this needs to be documented in the man pages. I will put in one
shortly.

-- 
Goldwyn

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/MM ATTEND] memory allocation scope

[Michal Hocko]

On Wed 14-02-18 16:51:53, Goldwyn Rodrigues wrote:
> 
> Discussion with the memory folks towards scope based allocation
> I am working on converting some of the GFP_NOFS memory allocation calls
> to new scope API [1]. While other allocation types (noio, nofs,
> noreclaim) are covered. Are there plans for identifying scope of
> GFP_ATOMIC allocations? This should cover most (if not all) of the
> allocation scope.

There was no explicit request for that but I can see how some users
might want it. I would have to double check but maybe this would allow
vmalloc(GFP_ATOMIC). There were some users but most of them could have
been changed in some way so the motivation is not very large.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Lsf-pc] [LSF/MM ATTEND] memory allocation scope

[James Bottomley]

On Thu, 2018-02-15 at 15:48 +0100, Michal Hocko wrote:
> On Wed 14-02-18 16:51:53, Goldwyn Rodrigues wrote:
> > 
> > 
> > Discussion with the memory folks towards scope based allocation
> > I am working on converting some of the GFP_NOFS memory allocation
> > calls to new scope API [1]. While other allocation types (noio,
> > nofs, noreclaim) are covered. Are there plans for identifying scope
> > of GFP_ATOMIC allocations? This should cover most (if not all) of
> > the allocation scope.
> 
> There was no explicit request for that but I can see how some users
> might want it. I would have to double check but maybe this would
> allow vmalloc(GFP_ATOMIC). There were some users but most of them
> could have been changed in some way so the motivation is not very
> large.

We have to be careful about that: most GFP_ATOMIC allocations are in
drivers and may be for DMA'able memory.  We can't currently use vmalloc
memory for DMA to kernel via block because bio_map_kern() uses
virt_to_page() which assumes offset mapping.  The latter is fixable,
obviously, but is it worth fixing?  Very few GFP_ATOMIC allocations in
drivers will be for large chunks.

James

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Lsf-pc] [LSF/MM ATTEND] memory allocation scope

[James Bottomley]

On Thu, 2018-02-15 at 15:48 +0100, Michal Hocko wrote:
> On Wed 14-02-18 16:51:53, Goldwyn Rodrigues wrote:
> > 
> > 
> > Discussion with the memory folks towards scope based allocation
> > I am working on converting some of the GFP_NOFS memory allocation
> > calls to new scope API [1]. While other allocation types (noio,
> > nofs, noreclaim) are covered. Are there plans for identifying scope
> > of GFP_ATOMIC allocations? This should cover most (if not all) of
> > the allocation scope.
> 
> There was no explicit request for that but I can see how some users
> might want it. I would have to double check but maybe this would
> allow vmalloc(GFP_ATOMIC). There were some users but most of them
> could have been changed in some way so the motivation is not very
> large.

We have to be careful about that: most GFP_ATOMIC allocations are in
drivers and may be for DMA'able memory. A We can't currently use vmalloc
memory for DMA to kernel via block because bio_map_kern() uses
virt_to_page() which assumes offset mapping. A The latter is fixable,
obviously, but is it worth fixing? A Very few GFP_ATOMIC allocations in
drivers will be for large chunks.

James

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Lsf-pc] [LSF/MM ATTEND] memory allocation scope

[Michal Hocko]

On Thu 15-02-18 07:57:37, James Bottomley wrote:
> On Thu, 2018-02-15 at 15:48 +0100, Michal Hocko wrote:
> > On Wed 14-02-18 16:51:53, Goldwyn Rodrigues wrote:
> > > 
> > > 
> > > Discussion with the memory folks towards scope based allocation
> > > I am working on converting some of the GFP_NOFS memory allocation
> > > calls to new scope API [1]. While other allocation types (noio,
> > > nofs, noreclaim) are covered. Are there plans for identifying scope
> > > of GFP_ATOMIC allocations? This should cover most (if not all) of
> > > the allocation scope.
> > 
> > There was no explicit request for that but I can see how some users
> > might want it. I would have to double check but maybe this would
> > allow vmalloc(GFP_ATOMIC). There were some users but most of them
> > could have been changed in some way so the motivation is not very
> > large.
> 
> We have to be careful about that: most GFP_ATOMIC allocations are in
> drivers and may be for DMA'able memory. �We can't currently use vmalloc
> memory for DMA to kernel via block because bio_map_kern() uses
> virt_to_page() which assumes offset mapping. �The latter is fixable,
> obviously, but is it worth fixing? �Very few GFP_ATOMIC allocations in
> drivers will be for large chunks.

Yes this might be not worth bothering. But from the conceptual POV
GFP_ATOMIC resp. GFP_NOWAIT is very often a scope context - IRQs,
preemption or RCU. So protecting all allocations from that context makes
some sense. Not sure this is really worth spending another context bit
though.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Lsf-pc] [LSF/MM ATTEND] memory allocation scope

[Michal Hocko]

On Thu 15-02-18 07:57:37, James Bottomley wrote:
> On Thu, 2018-02-15 at 15:48 +0100, Michal Hocko wrote:
> > On Wed 14-02-18 16:51:53, Goldwyn Rodrigues wrote:
> > > 
> > > 
> > > Discussion with the memory folks towards scope based allocation
> > > I am working on converting some of the GFP_NOFS memory allocation
> > > calls to new scope API [1]. While other allocation types (noio,
> > > nofs, noreclaim) are covered. Are there plans for identifying scope
> > > of GFP_ATOMIC allocations? This should cover most (if not all) of
> > > the allocation scope.
> > 
> > There was no explicit request for that but I can see how some users
> > might want it. I would have to double check but maybe this would
> > allow vmalloc(GFP_ATOMIC). There were some users but most of them
> > could have been changed in some way so the motivation is not very
> > large.
> 
> We have to be careful about that: most GFP_ATOMIC allocations are in
> drivers and may be for DMA'able memory.  We can't currently use vmalloc
> memory for DMA to kernel via block because bio_map_kern() uses
> virt_to_page() which assumes offset mapping.  The latter is fixable,
> obviously, but is it worth fixing?  Very few GFP_ATOMIC allocations in
> drivers will be for large chunks.

Yes this might be not worth bothering. But from the conceptual POV
GFP_ATOMIC resp. GFP_NOWAIT is very often a scope context - IRQs,
preemption or RCU. So protecting all allocations from that context makes
some sense. Not sure this is really worth spending another context bit
though.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Lsf-pc] [LSF/MM ATTEND] memory allocation scope

[Michal Hocko]

On Thu 15-02-18 17:02:41, Michal Hocko wrote:
> On Thu 15-02-18 07:57:37, James Bottomley wrote:
> > On Thu, 2018-02-15 at 15:48 +0100, Michal Hocko wrote:
> > > On Wed 14-02-18 16:51:53, Goldwyn Rodrigues wrote:
> > > > 
> > > > 
> > > > Discussion with the memory folks towards scope based allocation
> > > > I am working on converting some of the GFP_NOFS memory allocation
> > > > calls to new scope API [1]. While other allocation types (noio,
> > > > nofs, noreclaim) are covered. Are there plans for identifying scope
> > > > of GFP_ATOMIC allocations? This should cover most (if not all) of
> > > > the allocation scope.
> > > 
> > > There was no explicit request for that but I can see how some users
> > > might want it. I would have to double check but maybe this would
> > > allow vmalloc(GFP_ATOMIC). There were some users but most of them
> > > could have been changed in some way so the motivation is not very
> > > large.
> > 
> > We have to be careful about that: most GFP_ATOMIC allocations are in
> > drivers and may be for DMA'able memory. �We can't currently use vmalloc
> > memory for DMA to kernel via block because bio_map_kern() uses
> > virt_to_page() which assumes offset mapping. �The latter is fixable,
> > obviously, but is it worth fixing? �Very few GFP_ATOMIC allocations in
> > drivers will be for large chunks.
> 
> Yes this might be not worth bothering. But from the conceptual POV
> GFP_ATOMIC resp. GFP_NOWAIT is very often a scope context - IRQs,
> preemption or RCU. So protecting all allocations from that context makes
> some sense. Not sure this is really worth spending another context bit
> though.

And just to clarify why I've mentioned vmalloc explicitly. vmalloc
simply ignores the given gfp flags for pte allocations (those are
hardcoded GFP_KERNEL) and that is why it is not generally suitable for
atomic contexts. Maybe there are other obstacles (sleeping locks) but
scope gfp would solve at least the pte allocation side.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Lsf-pc] [LSF/MM ATTEND] memory allocation scope

[Michal Hocko]

On Thu 15-02-18 17:02:41, Michal Hocko wrote:
> On Thu 15-02-18 07:57:37, James Bottomley wrote:
> > On Thu, 2018-02-15 at 15:48 +0100, Michal Hocko wrote:
> > > On Wed 14-02-18 16:51:53, Goldwyn Rodrigues wrote:
> > > > 
> > > > 
> > > > Discussion with the memory folks towards scope based allocation
> > > > I am working on converting some of the GFP_NOFS memory allocation
> > > > calls to new scope API [1]. While other allocation types (noio,
> > > > nofs, noreclaim) are covered. Are there plans for identifying scope
> > > > of GFP_ATOMIC allocations? This should cover most (if not all) of
> > > > the allocation scope.
> > > 
> > > There was no explicit request for that but I can see how some users
> > > might want it. I would have to double check but maybe this would
> > > allow vmalloc(GFP_ATOMIC). There were some users but most of them
> > > could have been changed in some way so the motivation is not very
> > > large.
> > 
> > We have to be careful about that: most GFP_ATOMIC allocations are in
> > drivers and may be for DMA'able memory.  We can't currently use vmalloc
> > memory for DMA to kernel via block because bio_map_kern() uses
> > virt_to_page() which assumes offset mapping.  The latter is fixable,
> > obviously, but is it worth fixing?  Very few GFP_ATOMIC allocations in
> > drivers will be for large chunks.
> 
> Yes this might be not worth bothering. But from the conceptual POV
> GFP_ATOMIC resp. GFP_NOWAIT is very often a scope context - IRQs,
> preemption or RCU. So protecting all allocations from that context makes
> some sense. Not sure this is really worth spending another context bit
> though.

And just to clarify why I've mentioned vmalloc explicitly. vmalloc
simply ignores the given gfp flags for pte allocations (those are
hardcoded GFP_KERNEL) and that is why it is not generally suitable for
atomic contexts. Maybe there are other obstacles (sleeping locks) but
scope gfp would solve at least the pte allocation side.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>