From: Tom Lendacky <thomas.lendacky@amd.com>
To: Baoquan He <bhe@redhat.com>, Robin Murphy <robin.murphy@arm.com>,
Christoph Hellwig <hch@infradead.org>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org,
rientjes@google.com, rppt@linux.ibm.com, brijesh.singh@amd.com,
kexec@lists.infradead.org, iommu@lists.linux-foundation.org,
m.szyprowski@samsung.com
Subject: Re: [PATCH RFC 0/2] dma-pool: allow user to disable atomic pool
Date: Tue, 10 Aug 2021 15:52:25 -0500 [thread overview]
Message-ID: <5df93c4e-ffdd-88c5-db4a-1a09826d047a@amd.com> (raw)
In-Reply-To: <20210805065410.GA2051@MiWiFi-R3L-srv>
On 8/5/21 1:54 AM, Baoquan He wrote:
> On 06/24/21 at 11:47am, Robin Murphy wrote:
>> On 2021-06-24 10:29, Baoquan He wrote:
>>> On 06/24/21 at 08:40am, Christoph Hellwig wrote:
>>>> So reduce the amount allocated. But the pool is needed for proper
>>>> operation on systems with memory encryption. And please add the right
>>>> maintainer or at least mailing list for the code you're touching next
>>>> time.
>>>
>>> Oh, I thoutht it's memory issue only, should have run
>>> ./scripts/get_maintainer.pl. sorry.
>>>
>>> About reducing the amount allocated, it may not help. Because on x86_64,
>>> kdump kernel doesn't put any page of memory into buddy allocator of DMA
>>> zone. Means it will defenitely OOM for atomic_pool_dma initialization.
>>>
>>> Wondering in which case or on which device the atomic pool is needed on
>>> AMD system with mem encrytion enabled. As we can see, the OOM will
>>> happen too in kdump kernel on Intel system, even though it's not
>>> necessary.
>
> Sorry for very late response, and thank both for your comments.
>
>>
>> Hmm, I think the Kconfig reshuffle has actually left a slight wrinkle here.
>> For DMA_DIRECT_REMAP=y we can assume an atomic pool is always needed, since
>> that was the original behaviour anyway. However the implications of
>> AMD_MEM_ENCRYPT=y are different - even if support is enabled, it still
>> should only be relevant if mem_encrypt_active(), so it probably does make
>> sense to have an additional runtime gate on that.
>
>>
>> From a quick scan, use of dma_alloc_from_pool() already depends on
>> force_dma_unencrypted() so that's probably fine already, but I think we'd
>> need a bit of extra protection around dma_free_from_pool() to prevent
>> gen_pool_has_addr() dereferencing NULL if the pools are uninitialised, even
>> with your proposed patch as it is. Presumably nothing actually called
>> dma_direct_free() when you tested this?
>
> Yes, enforcing the conditional check of force_dma_unencrypted() around
> dma_free_from_pool sounds reasonable, just as we have done in
> dma_alloc_from_pool().
>
> I have tested this patchset on normal x86_64 systems and one amd system
> with SME support, disabling atomic pool can fix the issue that there's no
> managed pages in dma zone then requesting page from dma zone will cause
> allocation failure. And even disabling atomic pool in 1st kernel didn't
> cause any problem on one AMD EPYC system which supports SME. I am not
> expert of DMA area, wondering how atomic pool is supposed to do in
> SME/SEV system.
I think the atomic pool is used by the NVMe driver. My understanding is
that driver will do a dma_alloc_coherent() from interrupt context, so it
needs to use GFP_ATOMIC. The pool was created because dma_alloc_coherent()
would perform a set_memory_decrypted() call, which can sleep. The pool
eliminates that issue (David can correct me if I got that wrong).
Thanks,
Tom
>
> Besides, even though atomic pool is disabled, slub page for allocation
> of dma-kmalloc also triggers page allocation failure. So I change to
> take another way to fix them, please check v2 post. The atomic pool
> disabling an be a good to have change.
>
> Thanks
> Baoquan
>
WARNING: multiple messages have this Message-ID (diff)
From: Tom Lendacky via iommu <iommu@lists.linux-foundation.org>
To: Baoquan He <bhe@redhat.com>, Robin Murphy <robin.murphy@arm.com>,
Christoph Hellwig <hch@infradead.org>
Cc: brijesh.singh@amd.com, x86@kernel.org, kexec@lists.infradead.org,
linux-kernel@vger.kernel.org, rppt@linux.ibm.com,
linux-mm@kvack.org, iommu@lists.linux-foundation.org,
rientjes@google.com
Subject: Re: [PATCH RFC 0/2] dma-pool: allow user to disable atomic pool
Date: Tue, 10 Aug 2021 15:52:25 -0500 [thread overview]
Message-ID: <5df93c4e-ffdd-88c5-db4a-1a09826d047a@amd.com> (raw)
In-Reply-To: <20210805065410.GA2051@MiWiFi-R3L-srv>
On 8/5/21 1:54 AM, Baoquan He wrote:
> On 06/24/21 at 11:47am, Robin Murphy wrote:
>> On 2021-06-24 10:29, Baoquan He wrote:
>>> On 06/24/21 at 08:40am, Christoph Hellwig wrote:
>>>> So reduce the amount allocated. But the pool is needed for proper
>>>> operation on systems with memory encryption. And please add the right
>>>> maintainer or at least mailing list for the code you're touching next
>>>> time.
>>>
>>> Oh, I thoutht it's memory issue only, should have run
>>> ./scripts/get_maintainer.pl. sorry.
>>>
>>> About reducing the amount allocated, it may not help. Because on x86_64,
>>> kdump kernel doesn't put any page of memory into buddy allocator of DMA
>>> zone. Means it will defenitely OOM for atomic_pool_dma initialization.
>>>
>>> Wondering in which case or on which device the atomic pool is needed on
>>> AMD system with mem encrytion enabled. As we can see, the OOM will
>>> happen too in kdump kernel on Intel system, even though it's not
>>> necessary.
>
> Sorry for very late response, and thank both for your comments.
>
>>
>> Hmm, I think the Kconfig reshuffle has actually left a slight wrinkle here.
>> For DMA_DIRECT_REMAP=y we can assume an atomic pool is always needed, since
>> that was the original behaviour anyway. However the implications of
>> AMD_MEM_ENCRYPT=y are different - even if support is enabled, it still
>> should only be relevant if mem_encrypt_active(), so it probably does make
>> sense to have an additional runtime gate on that.
>
>>
>> From a quick scan, use of dma_alloc_from_pool() already depends on
>> force_dma_unencrypted() so that's probably fine already, but I think we'd
>> need a bit of extra protection around dma_free_from_pool() to prevent
>> gen_pool_has_addr() dereferencing NULL if the pools are uninitialised, even
>> with your proposed patch as it is. Presumably nothing actually called
>> dma_direct_free() when you tested this?
>
> Yes, enforcing the conditional check of force_dma_unencrypted() around
> dma_free_from_pool sounds reasonable, just as we have done in
> dma_alloc_from_pool().
>
> I have tested this patchset on normal x86_64 systems and one amd system
> with SME support, disabling atomic pool can fix the issue that there's no
> managed pages in dma zone then requesting page from dma zone will cause
> allocation failure. And even disabling atomic pool in 1st kernel didn't
> cause any problem on one AMD EPYC system which supports SME. I am not
> expert of DMA area, wondering how atomic pool is supposed to do in
> SME/SEV system.
I think the atomic pool is used by the NVMe driver. My understanding is
that driver will do a dma_alloc_coherent() from interrupt context, so it
needs to use GFP_ATOMIC. The pool was created because dma_alloc_coherent()
would perform a set_memory_decrypted() call, which can sleep. The pool
eliminates that issue (David can correct me if I got that wrong).
Thanks,
Tom
>
> Besides, even though atomic pool is disabled, slub page for allocation
> of dma-kmalloc also triggers page allocation failure. So I change to
> take another way to fix them, please check v2 post. The atomic pool
> disabling an be a good to have change.
>
> Thanks
> Baoquan
>
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
WARNING: multiple messages have this Message-ID (diff)
From: Tom Lendacky <thomas.lendacky@amd.com>
To: Baoquan He <bhe@redhat.com>, Robin Murphy <robin.murphy@arm.com>,
Christoph Hellwig <hch@infradead.org>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org,
rientjes@google.com, rppt@linux.ibm.com, brijesh.singh@amd.com,
kexec@lists.infradead.org, iommu@lists.linux-foundation.org,
m.szyprowski@samsung.com
Subject: Re: [PATCH RFC 0/2] dma-pool: allow user to disable atomic pool
Date: Tue, 10 Aug 2021 15:52:25 -0500 [thread overview]
Message-ID: <5df93c4e-ffdd-88c5-db4a-1a09826d047a@amd.com> (raw)
In-Reply-To: <20210805065410.GA2051@MiWiFi-R3L-srv>
On 8/5/21 1:54 AM, Baoquan He wrote:
> On 06/24/21 at 11:47am, Robin Murphy wrote:
>> On 2021-06-24 10:29, Baoquan He wrote:
>>> On 06/24/21 at 08:40am, Christoph Hellwig wrote:
>>>> So reduce the amount allocated. But the pool is needed for proper
>>>> operation on systems with memory encryption. And please add the right
>>>> maintainer or at least mailing list for the code you're touching next
>>>> time.
>>>
>>> Oh, I thoutht it's memory issue only, should have run
>>> ./scripts/get_maintainer.pl. sorry.
>>>
>>> About reducing the amount allocated, it may not help. Because on x86_64,
>>> kdump kernel doesn't put any page of memory into buddy allocator of DMA
>>> zone. Means it will defenitely OOM for atomic_pool_dma initialization.
>>>
>>> Wondering in which case or on which device the atomic pool is needed on
>>> AMD system with mem encrytion enabled. As we can see, the OOM will
>>> happen too in kdump kernel on Intel system, even though it's not
>>> necessary.
>
> Sorry for very late response, and thank both for your comments.
>
>>
>> Hmm, I think the Kconfig reshuffle has actually left a slight wrinkle here.
>> For DMA_DIRECT_REMAP=y we can assume an atomic pool is always needed, since
>> that was the original behaviour anyway. However the implications of
>> AMD_MEM_ENCRYPT=y are different - even if support is enabled, it still
>> should only be relevant if mem_encrypt_active(), so it probably does make
>> sense to have an additional runtime gate on that.
>
>>
>> From a quick scan, use of dma_alloc_from_pool() already depends on
>> force_dma_unencrypted() so that's probably fine already, but I think we'd
>> need a bit of extra protection around dma_free_from_pool() to prevent
>> gen_pool_has_addr() dereferencing NULL if the pools are uninitialised, even
>> with your proposed patch as it is. Presumably nothing actually called
>> dma_direct_free() when you tested this?
>
> Yes, enforcing the conditional check of force_dma_unencrypted() around
> dma_free_from_pool sounds reasonable, just as we have done in
> dma_alloc_from_pool().
>
> I have tested this patchset on normal x86_64 systems and one amd system
> with SME support, disabling atomic pool can fix the issue that there's no
> managed pages in dma zone then requesting page from dma zone will cause
> allocation failure. And even disabling atomic pool in 1st kernel didn't
> cause any problem on one AMD EPYC system which supports SME. I am not
> expert of DMA area, wondering how atomic pool is supposed to do in
> SME/SEV system.
I think the atomic pool is used by the NVMe driver. My understanding is
that driver will do a dma_alloc_coherent() from interrupt context, so it
needs to use GFP_ATOMIC. The pool was created because dma_alloc_coherent()
would perform a set_memory_decrypted() call, which can sleep. The pool
eliminates that issue (David can correct me if I got that wrong).
Thanks,
Tom
>
> Besides, even though atomic pool is disabled, slub page for allocation
> of dma-kmalloc also triggers page allocation failure. So I change to
> take another way to fix them, please check v2 post. The atomic pool
> disabling an be a good to have change.
>
> Thanks
> Baoquan
>
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
next prev parent reply other threads:[~2021-08-10 20:52 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-24 5:20 [PATCH RFC 0/2] dma-pool: allow user to disable atomic pool Baoquan He
2021-06-24 5:20 ` Baoquan He
2021-06-24 5:20 ` [PATCH 1/2] docs: kernel-parameters: Update to reflect the current default size of " Baoquan He
2021-06-24 5:20 ` Baoquan He
2021-06-24 5:20 ` [PATCH 2/2] dma-pool: allow user to disable " Baoquan He
2021-06-24 5:20 ` Baoquan He
2021-06-24 7:40 ` [PATCH RFC 0/2] " Christoph Hellwig
2021-06-24 7:40 ` Christoph Hellwig
2021-06-24 9:29 ` Baoquan He
2021-06-24 9:29 ` Baoquan He
2021-06-24 9:29 ` Baoquan He
2021-06-24 10:47 ` Robin Murphy
2021-06-24 10:47 ` Robin Murphy
2021-06-24 10:47 ` Robin Murphy
2021-06-24 12:10 ` Christoph Hellwig
2021-06-24 12:10 ` Christoph Hellwig
2021-06-24 12:10 ` Christoph Hellwig
2021-08-05 6:54 ` Baoquan He
2021-08-05 6:54 ` Baoquan He
2021-08-05 6:54 ` Baoquan He
2021-08-10 20:52 ` Tom Lendacky [this message]
2021-08-10 20:52 ` Tom Lendacky
2021-08-10 20:52 ` Tom Lendacky via iommu
2021-08-11 2:23 ` Baoquan He
2021-08-11 2:23 ` Baoquan He
2021-08-11 2:23 ` Baoquan He
2021-08-11 13:46 ` Tom Lendacky
2021-08-11 13:46 ` Tom Lendacky
2021-08-11 13:46 ` Tom Lendacky via iommu
2021-08-11 5:52 ` Christoph Hellwig
2021-08-11 5:52 ` Christoph Hellwig
2021-08-11 5:52 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5df93c4e-ffdd-88c5-db4a-1a09826d047a@amd.com \
--to=thomas.lendacky@amd.com \
--cc=bhe@redhat.com \
--cc=brijesh.singh@amd.com \
--cc=hch@infradead.org \
--cc=iommu@lists.linux-foundation.org \
--cc=kexec@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=m.szyprowski@samsung.com \
--cc=rientjes@google.com \
--cc=robin.murphy@arm.com \
--cc=rppt@linux.ibm.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.