* [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory
@ 2021-11-16 9:47 Ard Biesheuvel
2021-11-16 12:04 ` Russell King (Oracle)
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Ard Biesheuvel @ 2021-11-16 9:47 UTC (permalink / raw)
To: linux-mm
Cc: Ard Biesheuvel, Quanyang Wang, Linus Walleij, Thomas Gleixner,
Russell King, Andrew Morton
The kmap_local conversion broke the ARM architecture, because the new
code assumes that all PTEs used for creating kmaps form a linear array
in memory, and uses array indexing to look up the kmap PTE belonging to
a certain kmap index.
On ARM, this cannot work, not only because the PTE pages may be
non-adjacent in memory, but also because ARM/!LPAE interleaves hardware
entries and extended entries (carrying software-only bits) in a way that
is not compatible with array indexing.
Fortunately, this only seems to affect configurations with more than 8
CPUs, due to the way the per-CPU kmap slots are organized in memory.
Work around this by permitting an architecture to set a Kconfig symbol
that signifies that the kmap PTEs do not form a lineary array in memory,
and so the only way to locate the appropriate one is to walk the page
tables.
Link: https://lore.kernel.org/linux-arm-kernel/20211026131249.3731275-1-ardb@kernel.org/
Reported-by: Quanyang Wang <quanyang.wang@windriver.com>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
Resent to linux-mm on akpm's request
arch/arm/Kconfig | 1 +
mm/Kconfig | 3 ++
mm/highmem.c | 32 +++++++++++++-------
3 files changed, 25 insertions(+), 11 deletions(-)
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 899830259bed..5d4ab8e8477f 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1470,6 +1470,7 @@ config HIGHMEM
bool "High Memory Support"
depends on MMU
select KMAP_LOCAL
+ select KMAP_LOCAL_NON_LINEAR_PTE_ARRAY
help
The address space of ARM processors is only 4 Gigabytes large
and it has to accommodate user address space, kernel address
diff --git a/mm/Kconfig b/mm/Kconfig
index 068ce591a13a..28edafc820ad 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -890,6 +890,9 @@ config MAPPING_DIRTY_HELPERS
config KMAP_LOCAL
bool
+config KMAP_LOCAL_NON_LINEAR_PTE_ARRAY
+ bool
+
# struct io_mapping based helper. Selected by drivers that need them
config IO_MAPPING
bool
diff --git a/mm/highmem.c b/mm/highmem.c
index 88f65f155845..ca9fa8c92593 100644
--- a/mm/highmem.c
+++ b/mm/highmem.c
@@ -503,16 +503,22 @@ static inline int kmap_local_calc_idx(int idx)
static pte_t *__kmap_pte;
-static pte_t *kmap_get_pte(void)
+static pte_t *kmap_get_pte(unsigned long vaddr, int idx)
{
+ if (IS_ENABLED(CONFIG_KMAP_LOCAL_NON_LINEAR_PTE_ARRAY))
+ /*
+ * Set by the arch if __kmap_pte[-idx] does not produce
+ * the correct entry.
+ */
+ return virt_to_kpte(vaddr);
if (!__kmap_pte)
__kmap_pte = virt_to_kpte(__fix_to_virt(FIX_KMAP_BEGIN));
- return __kmap_pte;
+ return &__kmap_pte[-idx];
}
void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot)
{
- pte_t pteval, *kmap_pte = kmap_get_pte();
+ pte_t pteval, *kmap_pte;
unsigned long vaddr;
int idx;
@@ -524,9 +530,10 @@ void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot)
preempt_disable();
idx = arch_kmap_local_map_idx(kmap_local_idx_push(), pfn);
vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
- BUG_ON(!pte_none(*(kmap_pte - idx)));
+ kmap_pte = kmap_get_pte(vaddr, idx);
+ BUG_ON(!pte_none(*kmap_pte));
pteval = pfn_pte(pfn, prot);
- arch_kmap_local_set_pte(&init_mm, vaddr, kmap_pte - idx, pteval);
+ arch_kmap_local_set_pte(&init_mm, vaddr, kmap_pte, pteval);
arch_kmap_local_post_map(vaddr, pteval);
current->kmap_ctrl.pteval[kmap_local_idx()] = pteval;
preempt_enable();
@@ -559,7 +566,7 @@ EXPORT_SYMBOL(__kmap_local_page_prot);
void kunmap_local_indexed(void *vaddr)
{
unsigned long addr = (unsigned long) vaddr & PAGE_MASK;
- pte_t *kmap_pte = kmap_get_pte();
+ pte_t *kmap_pte;
int idx;
if (addr < __fix_to_virt(FIX_KMAP_END) ||
@@ -584,8 +591,9 @@ void kunmap_local_indexed(void *vaddr)
idx = arch_kmap_local_unmap_idx(kmap_local_idx(), addr);
WARN_ON_ONCE(addr != __fix_to_virt(FIX_KMAP_BEGIN + idx));
+ kmap_pte = kmap_get_pte(addr, idx);
arch_kmap_local_pre_unmap(addr);
- pte_clear(&init_mm, addr, kmap_pte - idx);
+ pte_clear(&init_mm, addr, kmap_pte);
arch_kmap_local_post_unmap(addr);
current->kmap_ctrl.pteval[kmap_local_idx()] = __pte(0);
kmap_local_idx_pop();
@@ -607,7 +615,7 @@ EXPORT_SYMBOL(kunmap_local_indexed);
void __kmap_local_sched_out(void)
{
struct task_struct *tsk = current;
- pte_t *kmap_pte = kmap_get_pte();
+ pte_t *kmap_pte;
int i;
/* Clear kmaps */
@@ -634,8 +642,9 @@ void __kmap_local_sched_out(void)
idx = arch_kmap_local_map_idx(i, pte_pfn(pteval));
addr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
+ kmap_pte = kmap_get_pte(addr, idx);
arch_kmap_local_pre_unmap(addr);
- pte_clear(&init_mm, addr, kmap_pte - idx);
+ pte_clear(&init_mm, addr, kmap_pte);
arch_kmap_local_post_unmap(addr);
}
}
@@ -643,7 +652,7 @@ void __kmap_local_sched_out(void)
void __kmap_local_sched_in(void)
{
struct task_struct *tsk = current;
- pte_t *kmap_pte = kmap_get_pte();
+ pte_t *kmap_pte;
int i;
/* Restore kmaps */
@@ -663,7 +672,8 @@ void __kmap_local_sched_in(void)
/* See comment in __kmap_local_sched_out() */
idx = arch_kmap_local_map_idx(i, pte_pfn(pteval));
addr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
- set_pte_at(&init_mm, addr, kmap_pte - idx, pteval);
+ kmap_pte = kmap_get_pte(addr, idx);
+ set_pte_at(&init_mm, addr, kmap_pte, pteval);
arch_kmap_local_post_map(addr, pteval);
}
}
--
2.30.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory
2021-11-16 9:47 [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory Ard Biesheuvel
@ 2021-11-16 12:04 ` Russell King (Oracle)
2021-11-16 22:12 ` Andrew Morton
2021-11-25 11:00 ` Thomas Gleixner
2 siblings, 0 replies; 6+ messages in thread
From: Russell King (Oracle) @ 2021-11-16 12:04 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: linux-mm, Quanyang Wang, Linus Walleij, Thomas Gleixner, Andrew Morton
On Tue, Nov 16, 2021 at 10:47:37AM +0100, Ard Biesheuvel wrote:
> The kmap_local conversion broke the ARM architecture, because the new
> code assumes that all PTEs used for creating kmaps form a linear array
> in memory, and uses array indexing to look up the kmap PTE belonging to
> a certain kmap index.
>
> On ARM, this cannot work, not only because the PTE pages may be
> non-adjacent in memory, but also because ARM/!LPAE interleaves hardware
> entries and extended entries (carrying software-only bits) in a way that
> is not compatible with array indexing.
>
> Fortunately, this only seems to affect configurations with more than 8
> CPUs, due to the way the per-CPU kmap slots are organized in memory.
>
> Work around this by permitting an architecture to set a Kconfig symbol
> that signifies that the kmap PTEs do not form a lineary array in memory,
> and so the only way to locate the appropriate one is to walk the page
> tables.
>
> Link: https://lore.kernel.org/linux-arm-kernel/20211026131249.3731275-1-ardb@kernel.org/
> Reported-by: Quanyang Wang <quanyang.wang@windriver.com>
> Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Russell King <linux@armlinux.org.uk>
Acked-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
Thanks.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory
2021-11-16 9:47 [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory Ard Biesheuvel
2021-11-16 12:04 ` Russell King (Oracle)
@ 2021-11-16 22:12 ` Andrew Morton
2021-11-16 22:13 ` Ard Biesheuvel
2021-11-25 11:00 ` Thomas Gleixner
2 siblings, 1 reply; 6+ messages in thread
From: Andrew Morton @ 2021-11-16 22:12 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: linux-mm, Quanyang Wang, Linus Walleij, Thomas Gleixner, Russell King
On Tue, 16 Nov 2021 10:47:37 +0100 Ard Biesheuvel <ardb@kernel.org> wrote:
> The kmap_local conversion broke the ARM architecture, because the new
> code assumes that all PTEs used for creating kmaps form a linear array
> in memory, and uses array indexing to look up the kmap PTE belonging to
> a certain kmap index.
>
> On ARM, this cannot work, not only because the PTE pages may be
> non-adjacent in memory, but also because ARM/!LPAE interleaves hardware
> entries and extended entries (carrying software-only bits) in a way that
> is not compatible with array indexing.
>
> Fortunately, this only seems to affect configurations with more than 8
> CPUs, due to the way the per-CPU kmap slots are organized in memory.
>
> Work around this by permitting an architecture to set a Kconfig symbol
> that signifies that the kmap PTEs do not form a lineary array in memory,
> and so the only way to locate the appropriate one is to walk the page
> tables.
A Fixes: link would be helpful.
kmap_local() was a year ago, so is a -stable backport desired here?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory
2021-11-16 22:12 ` Andrew Morton
@ 2021-11-16 22:13 ` Ard Biesheuvel
2021-11-16 22:15 ` Ard Biesheuvel
0 siblings, 1 reply; 6+ messages in thread
From: Ard Biesheuvel @ 2021-11-16 22:13 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux Memory Management List, Quanyang Wang, Linus Walleij,
Thomas Gleixner, Russell King
On Tue, 16 Nov 2021 at 23:12, Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Tue, 16 Nov 2021 10:47:37 +0100 Ard Biesheuvel <ardb@kernel.org> wrote:
>
> > The kmap_local conversion broke the ARM architecture, because the new
> > code assumes that all PTEs used for creating kmaps form a linear array
> > in memory, and uses array indexing to look up the kmap PTE belonging to
> > a certain kmap index.
> >
> > On ARM, this cannot work, not only because the PTE pages may be
> > non-adjacent in memory, but also because ARM/!LPAE interleaves hardware
> > entries and extended entries (carrying software-only bits) in a way that
> > is not compatible with array indexing.
> >
> > Fortunately, this only seems to affect configurations with more than 8
> > CPUs, due to the way the per-CPU kmap slots are organized in memory.
> >
> > Work around this by permitting an architecture to set a Kconfig symbol
> > that signifies that the kmap PTEs do not form a lineary array in memory,
> > and so the only way to locate the appropriate one is to walk the page
> > tables.
>
> A Fixes: link would be helpful.
>
> kmap_local() was a year ago, so is a -stable backport desired here?
Fixes: 1834eb1a6027 ("kmap_local: don't assume kmap PTEs are linear
arrays in memory")
Cc: <stable@vger.kernel.org>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory
2021-11-16 22:13 ` Ard Biesheuvel
@ 2021-11-16 22:15 ` Ard Biesheuvel
0 siblings, 0 replies; 6+ messages in thread
From: Ard Biesheuvel @ 2021-11-16 22:15 UTC (permalink / raw)
To: Andrew Morton
Cc: Linux Memory Management List, Quanyang Wang, Linus Walleij,
Thomas Gleixner, Russell King
On Tue, 16 Nov 2021 at 23:13, Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Tue, 16 Nov 2021 at 23:12, Andrew Morton <akpm@linux-foundation.org> wrote:
> >
> > On Tue, 16 Nov 2021 10:47:37 +0100 Ard Biesheuvel <ardb@kernel.org> wrote:
> >
> > > The kmap_local conversion broke the ARM architecture, because the new
> > > code assumes that all PTEs used for creating kmaps form a linear array
> > > in memory, and uses array indexing to look up the kmap PTE belonging to
> > > a certain kmap index.
> > >
> > > On ARM, this cannot work, not only because the PTE pages may be
> > > non-adjacent in memory, but also because ARM/!LPAE interleaves hardware
> > > entries and extended entries (carrying software-only bits) in a way that
> > > is not compatible with array indexing.
> > >
> > > Fortunately, this only seems to affect configurations with more than 8
> > > CPUs, due to the way the per-CPU kmap slots are organized in memory.
> > >
> > > Work around this by permitting an architecture to set a Kconfig symbol
> > > that signifies that the kmap PTEs do not form a lineary array in memory,
> > > and so the only way to locate the appropriate one is to walk the page
> > > tables.
> >
> > A Fixes: link would be helpful.
> >
> > kmap_local() was a year ago, so is a -stable backport desired here?
>
> Fixes: 1834eb1a6027 ("kmap_local: don't assume kmap PTEs are linear
> arrays in memory")
Oops that should be
Fixes: 2a15ba82fa6c ("ARM: highmem: Switch to generic kmap atomic")
> Cc: <stable@vger.kernel.org>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory
2021-11-16 9:47 [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory Ard Biesheuvel
2021-11-16 12:04 ` Russell King (Oracle)
2021-11-16 22:12 ` Andrew Morton
@ 2021-11-25 11:00 ` Thomas Gleixner
2 siblings, 0 replies; 6+ messages in thread
From: Thomas Gleixner @ 2021-11-25 11:00 UTC (permalink / raw)
To: Ard Biesheuvel, linux-mm
Cc: Ard Biesheuvel, Quanyang Wang, Linus Walleij, Russell King,
Andrew Morton
On Tue, Nov 16 2021 at 10:47, Ard Biesheuvel wrote:
> The kmap_local conversion broke the ARM architecture, because the new
> code assumes that all PTEs used for creating kmaps form a linear array
> in memory, and uses array indexing to look up the kmap PTE belonging to
> a certain kmap index.
>
> On ARM, this cannot work, not only because the PTE pages may be
> non-adjacent in memory, but also because ARM/!LPAE interleaves hardware
> entries and extended entries (carrying software-only bits) in a way that
> is not compatible with array indexing.
>
> Fortunately, this only seems to affect configurations with more than 8
> CPUs, due to the way the per-CPU kmap slots are organized in memory.
Ooops. I completely missed that detail. Sorry for the wreckage.
> Work around this by permitting an architecture to set a Kconfig symbol
> that signifies that the kmap PTEs do not form a lineary array in memory,
> and so the only way to locate the appropriate one is to walk the page
> tables.
> +static pte_t *kmap_get_pte(unsigned long vaddr, int idx)
> {
> + if (IS_ENABLED(CONFIG_KMAP_LOCAL_NON_LINEAR_PTE_ARRAY))
> + /*
> + * Set by the arch if __kmap_pte[-idx] does not produce
> + * the correct entry.
> + */
> + return virt_to_kpte(vaddr);
Nit. The above is not a one line statement (even if it is from a
compiler perspective).
/*
* Does the architecture have non-linear KMAP pte entries which
* cannot be accessed by index?
*/
if (IS_ENABLED(CONFIG_KMAP_LOCAL_NON_LINEAR_PTE_ARRAY))
return virt_to_kpte(vaddr);
or
if (IS_ENABLED(CONFIG_KMAP_LOCAL_NON_LINEAR_PTE_ARRAY)) {
/* PTE entries cannot be accessed by index. */
return virt_to_kpte(vaddr);
}
are readable but this is really breaking the brain OCR:
if (IS_ENABLED(CONFIG_KMAP_LOCAL_NON_LINEAR_PTE_ARRAY))
/*
* Set by the arch if __kmap_pte[-idx] does not produce
* the correct entry.
*/
return virt_to_kpte(vaddr);
if (!__kmap_pte)
Other than that.
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-11-25 11:02 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-16 9:47 [PATCH resend] kmap_local: don't assume kmap PTEs are linear arrays in memory Ard Biesheuvel
2021-11-16 12:04 ` Russell King (Oracle)
2021-11-16 22:12 ` Andrew Morton
2021-11-16 22:13 ` Ard Biesheuvel
2021-11-16 22:15 ` Ard Biesheuvel
2021-11-25 11:00 ` Thomas Gleixner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.