* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-29 20:49 ` Stuart Yoder
@ 2015-07-29 20:57 ` Arnd Bergmann
2015-07-29 20:58 ` Ard Biesheuvel
2015-07-30 10:13 ` Catalin Marinas
2 siblings, 0 replies; 30+ messages in thread
From: Arnd Bergmann @ 2015-07-29 20:57 UTC (permalink / raw)
To: linux-arm-kernel
On Wednesday 29 July 2015 20:49:57 Stuart Yoder wrote:
> >
> > Perhaps you should mention, for the benefit of those not following the
> > other thread, that the platform in question has 2 chunks of memory,
> > i.e., 2 GB and 14 GB, with a 508 GB hole in between.
>
> Yes, our physical memory layout for RAM looks like this:
> 2 GB at 0x8000_0000
> 510 GB at 0x80_8000_0000
Could we just ignore the first 2GB on this machine?
> > To be honest, I think this is poorly designed, and I am not sure we
> > should cater for such configurations in the defconfig.
>
> Agree, if this is a one-off weird platform then we shouldn't.
>
> But, the 'Principles of ARM Memory Maps' doc proposes this:
> 2 GB at 0x8000_0000
> 30 GB at 0x8_8000_0000
> 480 GB at 0x88_0000_0000
>
> ...i.e. if you have > 32 GB then your RAM regions are split into 3
> chunks. The aarch64 kernel will support > than 32GB right? A
> basic server will have that much or more.
>
> How will we deal with systems with > 32GB of memory that follow that
> map?
Burn them? ;-)
Hopefully the hardware on most machines should allow the memory map to
be configured so it can appear somewhere in a single chunk by modifying
the boot loader.
> When do we expect the default page size for the aarch64 kernel to be
> changed to 64KB?
No.
> Any workload that puts pressure on the TLBs will benefit
> from this.
I would assume that the only benchmarks that benefit from 64kb
pages are the ones that benefit more from using transparent hugepages.
64kb pages is crazy as a default because it sucks for almost every
workload you'd see in practice.
Arnd
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-29 20:49 ` Stuart Yoder
2015-07-29 20:57 ` Arnd Bergmann
@ 2015-07-29 20:58 ` Ard Biesheuvel
2015-07-30 10:13 ` Catalin Marinas
2 siblings, 0 replies; 30+ messages in thread
From: Ard Biesheuvel @ 2015-07-29 20:58 UTC (permalink / raw)
To: linux-arm-kernel
On 29 July 2015 at 22:49, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>
>> -----Original Message-----
>> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> Sent: Wednesday, July 29, 2015 2:51 PM
>> To: Yoder Stuart-B08248
>> Cc: Marc Zyngier; Catalin Marinas; Will Deacon; linux-arm-kernel at lists.infradead.org; Newton Peter-RA3823;
>> Mark Rutland
>> Subject: Re: [RFC] arm64: defconfig: enable 48-bit VA by default
>>
>> On 29 July 2015 at 21:27, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> >
>> >> -----Original Message-----
>> >> From: Marc Zyngier [mailto:marc.zyngier at arm.com]
>> >> Sent: Thursday, July 23, 2015 7:44 AM
>> >> To: Yoder Stuart-B08248; Catalin Marinas; Will Deacon
>> >> Cc: linux-arm-kernel at lists.infradead.org; Newton Peter-RA3823
>> >> Subject: Re: [RFC] arm64: defconfig: enable 48-bit VA by default
>> >>
>> >> On 22/07/15 20:49, Stuart Yoder wrote:
>> >> > Catalin/Will,
>> >> >
>> >> > This is not a patch mean to be applied, but a query about whether there
>> >> > is any reason to not enable 48-bit VA by default in the arm64 defconfig.
>> >> >
>> >> > The Freescale LS2085A physical memory map requires 48-bit VA in Linux for the
>> >> > reasons mentioned in [1].
>> >> >
>> >> > Based on the comment in [1] by Catalin, it seems that the intent
>> >> > is to turn this on by default.
>> >> >
>> >> > Is there any issues anyone sees with a patch that does this:
>> >> >
>> >> > diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
>> >> > index 4e17e7e..5acf75d 100644
>> >> > @@ -47,6 +47,7 @@ CONFIG_ARCH_ZYNQMP=y
>> >> > CONFIG_PCI=y
>> >> > CONFIG_PCI_MSI=y
>> >> > CONFIG_PCI_XGENE=y
>> >> > +CONFIG_ARM64_VA_BITS_48=y
>> >> > CONFIG_SMP=y
>> >> > CONFIG_PREEMPT=y
>> >> > CONFIG_KSM=y
>> >> >
>> >> > Thanks,
>> >> > Stuart
>> >> >
>> >> > [1] https://www.marc.info/?l=linux-arm-kernel&m=140965303205473&w=1
>> >> >
>> >>
>> >> Is that still a requirement now that our idmap can use 4 levels (as part
>> >> of dd006da)?
>> >
>> > So, yes it appears still to be a requirement. The idmap support is not
>> > the issue, it's the linear mapping.
>> >
>> > Has there been discussion or thinking about enabling 48-bit VA in the
>> > default defconfig? As mentioned before, it seemed that supporting 48-bit
>> > VA was the planned default (~1 year ago), and was waiting on KVM issues to get
>> > resolved.
>> >
>> > A related question is what the thinking around enabling 64KB pages
>> > by default. Any chance of that happening?
>> >
>> > I would like to see our platform work with the default defconfig, which
>> > is the reason for the questions.
>> >
>>
>> Perhaps you should mention, for the benefit of those not following the
>> other thread, that the platform in question has 2 chunks of memory,
>> i.e., 2 GB and 14 GB, with a 508 GB hole in between.
>
> Yes, our physical memory layout for RAM looks like this:
> 2 GB at 0x8000_0000
> 510 GB at 0x80_8000_0000
>
Once you use more than 256 GB of DRAM, you are going to need 4 levels
of page tables anyway for 4 KB pages.
>>
>> To be honest, I think this is poorly designed, and I am not sure we
>> should cater for such configurations in the defconfig.
>
> Agree, if this is a one-off weird platform then we shouldn't.
>
> But, the 'Principles of ARM Memory Maps' doc proposes this:
> 2 GB at 0x8000_0000
> 30 GB at 0x8_8000_0000
> 480 GB at 0x88_0000_0000
>
> ...i.e. if you have > 32 GB then your RAM regions are split into 3
> chunks. The aarch64 kernel will support > than 32GB right? A
> basic server will have that much or more.
>
> How will we deal with systems with > 32GB of memory that follow that
> map?
>
We will use either 64 KB pages or 4 levels of page tables, obviously.
But that does not mean it should be the default for everyone.
> When do we expect the default page size for the aarch64 kernel to be
> changed to 64KB? Any workload that puts pressure on the TLBs will benefit
> from this.
>
64 K pages are fully supported, but also, it is simply not the
default, and I don't expect it to be for the foreseeable future. TLB
pressure is a very artificial argument to make, since it is highly
workload dependent how it affects performance and there are downsides
to the higher granularity as well.
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-29 20:49 ` Stuart Yoder
2015-07-29 20:57 ` Arnd Bergmann
2015-07-29 20:58 ` Ard Biesheuvel
@ 2015-07-30 10:13 ` Catalin Marinas
2015-07-30 14:52 ` Stuart Yoder
2015-07-30 19:27 ` Ard Biesheuvel
2 siblings, 2 replies; 30+ messages in thread
From: Catalin Marinas @ 2015-07-30 10:13 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> > On 29 July 2015 at 21:27, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> > >> From: Marc Zyngier [mailto:marc.zyngier at arm.com]
> > >> On 22/07/15 20:49, Stuart Yoder wrote:
> > >> > This is not a patch mean to be applied, but a query about whether there
> > >> > is any reason to not enable 48-bit VA by default in the arm64 defconfig.
> > >> >
> > >> > The Freescale LS2085A physical memory map requires 48-bit VA in Linux for the
> > >> > reasons mentioned in [1].
> > >> >
> > >> > Based on the comment in [1] by Catalin, it seems that the intent
> > >> > is to turn this on by default.
> > >> >
> > >> > Is there any issues anyone sees with a patch that does this:
> > >> >
> > >> > diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
> > >> > index 4e17e7e..5acf75d 100644
> > >> > @@ -47,6 +47,7 @@ CONFIG_ARCH_ZYNQMP=y
> > >> > CONFIG_PCI=y
> > >> > CONFIG_PCI_MSI=y
> > >> > CONFIG_PCI_XGENE=y
> > >> > +CONFIG_ARM64_VA_BITS_48=y
> > >> > CONFIG_SMP=y
> > >> > CONFIG_PREEMPT=y
> > >> > CONFIG_KSM=y
> > >> >
> > >> > Thanks,
> > >> > Stuart
> > >> >
> > >> > [1] https://www.marc.info/?l=linux-arm-kernel&m=140965303205473&w=1
> > >>
> > >> Is that still a requirement now that our idmap can use 4 levels (as part
> > >> of dd006da)?
> > >
> > > So, yes it appears still to be a requirement. The idmap support is not
> > > the issue, it's the linear mapping.
> > >
> > > Has there been discussion or thinking about enabling 48-bit VA in the
> > > default defconfig? As mentioned before, it seemed that supporting 48-bit
> > > VA was the planned default (~1 year ago), and was waiting on KVM issues to get
> > > resolved.
> > >
> > > A related question is what the thinking around enabling 64KB pages
> > > by default. Any chance of that happening?
> > >
> > > I would like to see our platform work with the default defconfig, which
> > > is the reason for the questions.
> >
> > Perhaps you should mention, for the benefit of those not following the
> > other thread, that the platform in question has 2 chunks of memory,
> > i.e., 2 GB and 14 GB, with a 508 GB hole in between.
>
> Yes, our physical memory layout for RAM looks like this:
> 2 GB at 0x8000_0000
> 510 GB at 0x80_8000_0000
So your platform currently only has 16GB of RAM. Shouldn't the last 14GB
be placed at 34GB offset (according to the "Principles of ARM Memory
Maps")?
> > To be honest, I think this is poorly designed, and I am not sure we
> > should cater for such configurations in the defconfig.
>
> Agree, if this is a one-off weird platform then we shouldn't.
>
> But, the 'Principles of ARM Memory Maps' doc proposes this:
> 2 GB at 0x8000_0000
> 30 GB at 0x8_8000_0000
> 480 GB at 0x88_0000_0000
I'm not particularly recommending this layout, at least not without some
clarifications on DRAM aliases (I'll ping people internally about it
again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
and all the memory beyond 32-bit was highmem anyway. It was later
updated for AArch64 but only to allow 44/48-bit PA (a few sections
added).
How I understood it at the time is that in hardware you place the large
DRAM (chip select) at high addresses, e.g. 64GB would be at a 512GB
offset, but lower 2GB+30GB aliased in the lower space with the
corresponding high address remaining a hole (that produces external
aborts if accessed). The first alias at 2GB was meant for 32-bit
initialisation code before the MMU is enabled. With AArch64, you don't
really need this low alias (assuming your EL3/EL2/secure-EL1 code is
64-bit) since PC can use >32-bit addresses.
> ...i.e. if you have > 32 GB then your RAM regions are split into 3
> chunks. The aarch64 kernel will support > than 32GB right? A
> basic server will have that much or more.
>
> How will we deal with systems with > 32GB of memory that follow that
> map?
My recommendation would be to avoid the lower alias (in hardware or some
EL3 configuration change) and keep all the RAM at 0x80_0000_0000, just
boot the system in AArch64 mode.
> When do we expect the default page size for the aarch64 kernel to be
> changed to 64KB? Any workload that puts pressure on the TLBs will benefit
> from this.
This comes with its own set problems. 64KB is useful for specific
use-cases (e.g. large databases) but not general purpose where you waste
a large amount of RAM. So there are no plans to change the default page
size to 64KB.
--
Catalin
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-30 10:13 ` Catalin Marinas
@ 2015-07-30 14:52 ` Stuart Yoder
2015-07-30 16:12 ` Catalin Marinas
2015-07-30 19:27 ` Ard Biesheuvel
1 sibling, 1 reply; 30+ messages in thread
From: Stuart Yoder @ 2015-07-30 14:52 UTC (permalink / raw)
To: linux-arm-kernel
> -----Original Message-----
> From: Catalin Marinas [mailto:catalin.marinas at arm.com]
> >
[cut]
> > Yes, our physical memory layout for RAM looks like this:
> > 2 GB at 0x8000_0000
> > 510 GB at 0x80_8000_0000
>
> So your platform currently only has 16GB of RAM. Shouldn't the last 14GB
> be placed at 34GB offset (according to the "Principles of ARM Memory
> Maps")?
Yes, but it looks like our hardware designer didn't follow it exactly. And
instead created a variant by dropping the middle region. The reasoning
seems to be that our SoC architecture is 40-bit PA only and thus it seemed
unnecessary to have the middle region at 34GB offset. Plus, they wanted
that middle region for other I/O.
> > > To be honest, I think this is poorly designed, and I am not sure we
> > > should cater for such configurations in the defconfig.
> >
> > Agree, if this is a one-off weird platform then we shouldn't.
> >
> > But, the 'Principles of ARM Memory Maps' doc proposes this:
> > 2 GB at 0x8000_0000
> > 30 GB at 0x8_8000_0000
> > 480 GB at 0x88_0000_0000
>
> I'm not particularly recommending this layout, at least not without some
> clarifications on DRAM aliases (I'll ping people internally about it
> again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
> and all the memory beyond 32-bit was highmem anyway. It was later
> updated for AArch64 but only to allow 44/48-bit PA (a few sections
> added).
Published guidance from ARM I think is important here. If the aarch64
kernel maintainers have issues with the proposed map in the 'Principles' doc,
I consider that's a problem.
> How I understood it at the time is that in hardware you place the large
> DRAM (chip select) at high addresses, e.g. 64GB would be at a 512GB
> offset, but lower 2GB+30GB aliased in the lower space with the
> corresponding high address remaining a hole (that produces external
> aborts if accessed). The first alias at 2GB was meant for 32-bit
> initialisation code before the MMU is enabled. With AArch64, you don't
> really need this low alias (assuming your EL3/EL2/secure-EL1 code is
> 64-bit) since PC can use >32-bit addresses.
I think the intent with our SoC was to do that, but ommitting the
middle alias, because there was no obivous reason why it was needed.
> > ...i.e. if you have > 32 GB then your RAM regions are split into 3
> > chunks. The aarch64 kernel will support > than 32GB right? A
> > basic server will have that much or more.
> >
> > How will we deal with systems with > 32GB of memory that follow that
> > map?
>
> My recommendation would be to avoid the lower alias (in hardware or some
> EL3 configuration change) and keep all the RAM at 0x80_0000_0000, just
> boot the system in AArch64 mode.
Unfortunately, for our existing armv8 silicon I am not aware any mechanism to
change the aliasing and keeping all RAM at 512GB offset. So, we're stuck
for now with 2 discontiguous regions. And things are ok as long as we
turn on 48-bit VA or 64K pages, but that's not the case with the default
defconfig.
My interest in the default defconfig is related to distro support. When
a distro (Fedora, Debian, Ubuntu,etc) has a binary kernel image we want
it to run on our hardware. I don't know to what degree the upstream
defconfig informs distros on what defconfig options are enabled in their
kernels...but we want to be as compatible as possible with the default
defconfig.
(BTW, Fedora 22 for aarch64 appears to have 64KB pages enabled, which
is good for our memory map)
Thanks,
Stuart
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-30 14:52 ` Stuart Yoder
@ 2015-07-30 16:12 ` Catalin Marinas
2015-07-30 16:32 ` Stuart Yoder
0 siblings, 1 reply; 30+ messages in thread
From: Catalin Marinas @ 2015-07-30 16:12 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, Jul 30, 2015 at 02:52:40PM +0000, Stuart Yoder wrote:
> > -----Original Message-----
> > From: Catalin Marinas [mailto:catalin.marinas at arm.com]
> > > Yes, our physical memory layout for RAM looks like this:
> > > 2 GB at 0x8000_0000
> > > 510 GB at 0x80_8000_0000
> >
> > So your platform currently only has 16GB of RAM. Shouldn't the last 14GB
> > be placed at 34GB offset (according to the "Principles of ARM Memory
> > Maps")?
>
> Yes, but it looks like our hardware designer didn't follow it exactly. And
> instead created a variant by dropping the middle region. The reasoning
> seems to be that our SoC architecture is 40-bit PA only and thus it seemed
> unnecessary to have the middle region at 34GB offset. Plus, they wanted
> that middle region for other I/O.
Ah, so you can't blame the ARM Memory Maps document since you haven't
followed it properly ;).
> > > > To be honest, I think this is poorly designed, and I am not sure we
> > > > should cater for such configurations in the defconfig.
> > >
> > > Agree, if this is a one-off weird platform then we shouldn't.
> > >
> > > But, the 'Principles of ARM Memory Maps' doc proposes this:
> > > 2 GB at 0x8000_0000
> > > 30 GB at 0x8_8000_0000
> > > 480 GB at 0x88_0000_0000
> >
> > I'm not particularly recommending this layout, at least not without some
> > clarifications on DRAM aliases (I'll ping people internally about it
> > again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
> > and all the memory beyond 32-bit was highmem anyway. It was later
> > updated for AArch64 but only to allow 44/48-bit PA (a few sections
> > added).
>
> Published guidance from ARM I think is important here. If the aarch64
> kernel maintainers have issues with the proposed map in the 'Principles' doc,
> I consider that's a problem.
As I said, the problem I have with it is around the mandated aliases and
holes (and I'll raise it internally). But I won't complain about SoC
vendors following the recommendation (correctly; which doesn't seem to
be the case here).
> > How I understood it at the time is that in hardware you place the large
> > DRAM (chip select) at high addresses, e.g. 64GB would be at a 512GB
> > offset, but lower 2GB+30GB aliased in the lower space with the
> > corresponding high address remaining a hole (that produces external
> > aborts if accessed). The first alias at 2GB was meant for 32-bit
> > initialisation code before the MMU is enabled. With AArch64, you don't
> > really need this low alias (assuming your EL3/EL2/secure-EL1 code is
> > 64-bit) since PC can use >32-bit addresses.
>
> I think the intent with our SoC was to do that, but ommitting the
> middle alias, because there was no obivous reason why it was needed.
Can you avoid the low alias as well and keep all the RAM at
0x80_0000_0000? That's pretty much what we have on AMD's Seattle.
> > > ...i.e. if you have > 32 GB then your RAM regions are split into 3
> > > chunks. The aarch64 kernel will support > than 32GB right? A
> > > basic server will have that much or more.
> > >
> > > How will we deal with systems with > 32GB of memory that follow that
> > > map?
> >
> > My recommendation would be to avoid the lower alias (in hardware or some
> > EL3 configuration change) and keep all the RAM at 0x80_0000_0000, just
> > boot the system in AArch64 mode.
>
> Unfortunately, for our existing armv8 silicon I am not aware any mechanism to
> change the aliasing and keeping all RAM at 512GB offset. So, we're stuck
> for now with 2 discontiguous regions. And things are ok as long as we
> turn on 48-bit VA or 64K pages, but that's not the case with the default
> defconfig.
OK, so there is no hardware configuration you can change at this stage.
You could ignore the lower 2GB and only use 14GB (load the kernel at the
high address and it should ignore the lower memory block in the DT).
> My interest in the default defconfig is related to distro support. When
> a distro (Fedora, Debian, Ubuntu,etc) has a binary kernel image we want
> it to run on our hardware. I don't know to what degree the upstream
> defconfig informs distros on what defconfig options are enabled in their
> kernels...but we want to be as compatible as possible with the default
> defconfig.
For the time being, I would keep defconfig to 39-bit VA. If we start
seeing systems with lots of RAM (over 256GB), we'll probably change the
defconfig. Enabling another page table level has implications on
performance.
> (BTW, Fedora 22 for aarch64 appears to have 64KB pages enabled, which
> is good for our memory map)
Yes, Fedora should work. I'm not sure what other distros do but in
general they ship their own config.
--
Catalin
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-30 16:12 ` Catalin Marinas
@ 2015-07-30 16:32 ` Stuart Yoder
2015-07-30 16:41 ` Catalin Marinas
2015-07-30 17:45 ` Ard Biesheuvel
0 siblings, 2 replies; 30+ messages in thread
From: Stuart Yoder @ 2015-07-30 16:32 UTC (permalink / raw)
To: linux-arm-kernel
> -----Original Message-----
> From: Catalin Marinas [mailto:catalin.marinas at arm.com]
>
> For the time being, I would keep defconfig to 39-bit VA. If we start
> seeing systems with lots of RAM (over 256GB), we'll probably change the
> defconfig.
Somone who follows the "Principles" doc "correctly" will hit an issue
with the linear memory map after 32GB of RAM, right? 2 GB in 1st alias,
30 GB in 2nd alias, and rest at the 544 GB offset.
But, I guess it still remains to be seen who else will hit that
and when.
Thanks,
Stuart
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-30 16:32 ` Stuart Yoder
@ 2015-07-30 16:41 ` Catalin Marinas
2015-07-30 17:45 ` Ard Biesheuvel
1 sibling, 0 replies; 30+ messages in thread
From: Catalin Marinas @ 2015-07-30 16:41 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, Jul 30, 2015 at 04:32:19PM +0000, Stuart Yoder wrote:
> > -----Original Message-----
> > From: Catalin Marinas [mailto:catalin.marinas at arm.com]
> >
> > For the time being, I would keep defconfig to 39-bit VA. If we start
> > seeing systems with lots of RAM (over 256GB), we'll probably change the
> > defconfig.
>
> Somone who follows the "Principles" doc "correctly" will hit an issue
> with the linear memory map after 32GB of RAM, right? 2 GB in 1st alias,
> 30 GB in 2nd alias, and rest at the 544 GB offset.
Right (though I still don't like it but, well...).
--
Catalin
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-30 16:32 ` Stuart Yoder
2015-07-30 16:41 ` Catalin Marinas
@ 2015-07-30 17:45 ` Ard Biesheuvel
2015-07-30 18:10 ` Stuart Yoder
2015-08-07 19:01 ` Stuart Yoder
1 sibling, 2 replies; 30+ messages in thread
From: Ard Biesheuvel @ 2015-07-30 17:45 UTC (permalink / raw)
To: linux-arm-kernel
On 30 July 2015 at 18:32, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>
>
>> -----Original Message-----
>> From: Catalin Marinas [mailto:catalin.marinas at arm.com]
>>
>> For the time being, I would keep defconfig to 39-bit VA. If we start
>> seeing systems with lots of RAM (over 256GB), we'll probably change the
>> defconfig.
>
> Somone who follows the "Principles" doc "correctly" will hit an issue
> with the linear memory map after 32GB of RAM, right? 2 GB in 1st alias,
> 30 GB in 2nd alias, and rest at the 544 GB offset.
>
> But, I guess it still remains to be seen who else will hit that
> and when.
>
Hi Stuart,
I think you may be overestimating the significance of defconfig. As
you pointed out, distros have their own configs (and many out of tree
patches) for the binary kernels they ship. Fedora currently ships with
64k pages because it was the only way to support Seattle before the
4-level 4kb pages implementation was introduced. They will likely keep
doing that, or switch to 4-level 4 kB pages since, as you pointed out,
it is fairly likely that servers have more than 32 GB DRAM, which
means that they need more than 256 GB of virtual address space in the
linear region if they follow the ARM recommendation.
Whether defconfig supports your platform optimally has nothing to do
with that. Of course, we should deal with the unexpected memory layout
gracefully, which is why Mark Rutland and myself proposed patches to
fix the panic you reported. But in a development context, I think it
is perfectly acceptable to simply load the kernel at 0x80_8000_0000,
and be able to run defconfig fine while losing just 2 GB of your 16 GB
at the low end. Of course, you would never ship a system like that,
but that is not what defconfig is meant to cater for.
Regards,
Ard.
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-30 17:45 ` Ard Biesheuvel
@ 2015-07-30 18:10 ` Stuart Yoder
2015-08-07 19:01 ` Stuart Yoder
1 sibling, 0 replies; 30+ messages in thread
From: Stuart Yoder @ 2015-07-30 18:10 UTC (permalink / raw)
To: linux-arm-kernel
> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> Sent: Thursday, July 30, 2015 12:46 PM
> To: Yoder Stuart-B08248
> Cc: Catalin Marinas; Mark Rutland; Marc Zyngier; Will Deacon; Newton Peter-RA3823; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: [RFC] arm64: defconfig: enable 48-bit VA by default
>
> On 30 July 2015 at 18:32, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Catalin Marinas [mailto:catalin.marinas at arm.com]
> >>
> >> For the time being, I would keep defconfig to 39-bit VA. If we start
> >> seeing systems with lots of RAM (over 256GB), we'll probably change the
> >> defconfig.
> >
> > Somone who follows the "Principles" doc "correctly" will hit an issue
> > with the linear memory map after 32GB of RAM, right? 2 GB in 1st alias,
> > 30 GB in 2nd alias, and rest at the 544 GB offset.
> >
> > But, I guess it still remains to be seen who else will hit that
> > and when.
> >
>
> Hi Stuart,
>
> I think you may be overestimating the significance of defconfig.
I might be, it's just a paranoia point...which led to wondering
about what to expect with respect 48-bit VA and 64K pages going
forward.
> As you pointed out, distros have their own configs (and many out of tree
> patches) for the binary kernels they ship. Fedora currently ships with
> 64k pages because it was the only way to support Seattle before the
> 4-level 4kb pages implementation was introduced. They will likely keep
> doing that, or switch to 4-level 4 kB pages since, as you pointed out,
> it is fairly likely that servers have more than 32 GB DRAM, which
> means that they need more than 256 GB of virtual address space in the
> linear region if they follow the ARM recommendation.
>
> Whether defconfig supports your platform optimally has nothing to do
> with that. Of course, we should deal with the unexpected memory layout
> gracefully, which is why Mark Rutland and myself proposed patches to
> fix the panic you reported. But in a development context, I think it
> is perfectly acceptable to simply load the kernel at 0x80_8000_0000,
> and be able to run defconfig fine while losing just 2 GB of your 16 GB
> at the low end. Of course, you would never ship a system like that,
> but that is not what defconfig is meant to cater for.
Thanks for the patch and all your input, the picture and options are
quite a bit clearer to me now.
Thanks,
Stuart
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-30 17:45 ` Ard Biesheuvel
2015-07-30 18:10 ` Stuart Yoder
@ 2015-08-07 19:01 ` Stuart Yoder
2015-08-08 8:20 ` Ard Biesheuvel
1 sibling, 1 reply; 30+ messages in thread
From: Stuart Yoder @ 2015-08-07 19:01 UTC (permalink / raw)
To: linux-arm-kernel
> Whether defconfig supports your platform optimally has nothing to do
> with that. Of course, we should deal with the unexpected memory layout
> gracefully, which is why Mark Rutland and myself proposed patches to
> fix the panic you reported. But in a development context, I think it
> is perfectly acceptable to simply load the kernel at 0x80_8000_0000,
> and be able to run defconfig fine while losing just 2 GB of your 16 GB
> at the low end.
I've done this experiment-- loading/running the kernel at 0x80_8000_0000
and losing the lower 2GB of memory. And in fact I can see the kernel
ignoring the low memory:
[ 0.000000] Ignoring memory block 0x80000000 - 0x100000000
However, this only works with 48-bit VA enabled. With 39-bit VA
enabled the kernel crashes before the early console is working
and I see nothing.
Does the linear mapping we've been talking about just need to cover
all of _RAM_, or does it need to cover all of I/O as well in
the same mapping.
In our case we have various SoC I/O devices, like the UART
in the space below 4 GB... at 0x0100_0000.
Before debugging further I want to check whether this should
work in theory. I had the impression that the issue was
discontiguous RAM regions, and I/O was a separate issue.
Thanks,
Stuart
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-08-07 19:01 ` Stuart Yoder
@ 2015-08-08 8:20 ` Ard Biesheuvel
2015-08-13 19:24 ` Stuart Yoder
0 siblings, 1 reply; 30+ messages in thread
From: Ard Biesheuvel @ 2015-08-08 8:20 UTC (permalink / raw)
To: linux-arm-kernel
On 7 aug. 2015, at 21:01, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> Whether defconfig supports your platform optimally has nothing to do
>> with that. Of course, we should deal with the unexpected memory layout
>> gracefully, which is why Mark Rutland and myself proposed patches to
>> fix the panic you reported. But in a development context, I think it
>> is perfectly acceptable to simply load the kernel at 0x80_8000_0000,
>> and be able to run defconfig fine while losing just 2 GB of your 16 GB
>> at the low end.
>
> I've done this experiment-- loading/running the kernel at 0x80_8000_0000
> and losing the lower 2GB of memory. And in fact I can see the kernel
> ignoring the low memory:
>
> [ 0.000000] Ignoring memory block 0x80000000 - 0x100000000
>
Yes that looks correct
> However, this only works with 48-bit VA enabled. With 39-bit VA
> enabled the kernel crashes before the early console is working
> and I see nothing.
>
That may well be a kernel bug: the changes to allow 39-bit VA kernels
to execute on systems whose memory is outside the 39-bit VA range was
developed and tested on AMD Seattle only, which has its RAM at
0x80_0000_0000. But it is hard to be sure, of course, with no output
at all ...
> Does the linear mapping we've been talking about just need to cover
> all of _RAM_, or does it need to cover all of I/O as well in
> the same mapping.
>
> In our case we have various SoC I/O devices, like the UART
> in the space below 4 GB... at 0x0100_0000.
>
The linear mapping only covers system RAM, so that should not matter at all.
> Before debugging further I want to check whether this should
> work in theory. I had the impression that the issue was
> discontiguous RAM regions, and I/O was a separate issue.
Yes, it should work in theory. The linear region starts at the base of
the kernel Image, and extends all the way up to the end of DRAM (14 GB
in your case, IIRC)
--
Ard.
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-08-08 8:20 ` Ard Biesheuvel
@ 2015-08-13 19:24 ` Stuart Yoder
2015-08-14 12:15 ` Ard Biesheuvel
0 siblings, 1 reply; 30+ messages in thread
From: Stuart Yoder @ 2015-08-13 19:24 UTC (permalink / raw)
To: linux-arm-kernel
> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> Sent: Saturday, August 08, 2015 3:21 AM
> To: Yoder Stuart-B08248
> Cc: Catalin Marinas; Mark Rutland; Marc Zyngier; Will Deacon; Newton Peter-RA3823; linux-arm-
> kernel at lists.infradead.org; Sun York-R58495
> Subject: Re: [RFC] arm64: defconfig: enable 48-bit VA by default
>
> On 7 aug. 2015, at 21:01, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>
> >> Whether defconfig supports your platform optimally has nothing to do
> >> with that. Of course, we should deal with the unexpected memory layout
> >> gracefully, which is why Mark Rutland and myself proposed patches to
> >> fix the panic you reported. But in a development context, I think it
> >> is perfectly acceptable to simply load the kernel at 0x80_8000_0000,
> >> and be able to run defconfig fine while losing just 2 GB of your 16 GB
> >> at the low end.
> >
> > I've done this experiment-- loading/running the kernel at 0x80_8000_0000
> > and losing the lower 2GB of memory. And in fact I can see the kernel
> > ignoring the low memory:
> >
> > [ 0.000000] Ignoring memory block 0x80000000 - 0x100000000
> >
>
> Yes that looks correct
>
> > However, this only works with 48-bit VA enabled. With 39-bit VA
> > enabled the kernel crashes before the early console is working
> > and I see nothing.
> >
>
> That may well be a kernel bug: the changes to allow 39-bit VA kernels
> to execute on systems whose memory is outside the 39-bit VA range was
> developed and tested on AMD Seattle only, which has its RAM at
> 0x80_0000_0000. But it is hard to be sure, of course, with no output
> at all ...
>
> > Does the linear mapping we've been talking about just need to cover
> > all of _RAM_, or does it need to cover all of I/O as well in
> > the same mapping.
> >
> > In our case we have various SoC I/O devices, like the UART
> > in the space below 4 GB... at 0x0100_0000.
> >
>
> The linear mapping only covers system RAM, so that should not matter at all.
>
> > Before debugging further I want to check whether this should
> > work in theory. I had the impression that the issue was
> > discontiguous RAM regions, and I/O was a separate issue.
>
> Yes, it should work in theory. The linear region starts at the base of
> the kernel Image, and extends all the way up to the end of DRAM (14 GB
> in your case, IIRC)
An update on this... in my original test where the kernel did not run from
high memory I was using a 4.0.x kernel. I re-tried with a 4.2-rc4 kernel
and it booted fine. So, there is no kernel bug it in the latest kernel,
it appears.
Thanks,
Stuart
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-08-13 19:24 ` Stuart Yoder
@ 2015-08-14 12:15 ` Ard Biesheuvel
2015-08-14 13:24 ` Catalin Marinas
0 siblings, 1 reply; 30+ messages in thread
From: Ard Biesheuvel @ 2015-08-14 12:15 UTC (permalink / raw)
To: linux-arm-kernel
On 13 August 2015 at 21:24, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>
>> -----Original Message-----
>> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> Sent: Saturday, August 08, 2015 3:21 AM
>> To: Yoder Stuart-B08248
>> Cc: Catalin Marinas; Mark Rutland; Marc Zyngier; Will Deacon; Newton Peter-RA3823; linux-arm-
>> kernel at lists.infradead.org; Sun York-R58495
>> Subject: Re: [RFC] arm64: defconfig: enable 48-bit VA by default
>>
>> On 7 aug. 2015, at 21:01, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>>
>> >> Whether defconfig supports your platform optimally has nothing to do
>> >> with that. Of course, we should deal with the unexpected memory layout
>> >> gracefully, which is why Mark Rutland and myself proposed patches to
>> >> fix the panic you reported. But in a development context, I think it
>> >> is perfectly acceptable to simply load the kernel at 0x80_8000_0000,
>> >> and be able to run defconfig fine while losing just 2 GB of your 16 GB
>> >> at the low end.
>> >
>> > I've done this experiment-- loading/running the kernel at 0x80_8000_0000
>> > and losing the lower 2GB of memory. And in fact I can see the kernel
>> > ignoring the low memory:
>> >
>> > [ 0.000000] Ignoring memory block 0x80000000 - 0x100000000
>> >
>>
>> Yes that looks correct
>>
>> > However, this only works with 48-bit VA enabled. With 39-bit VA
>> > enabled the kernel crashes before the early console is working
>> > and I see nothing.
>> >
>>
>> That may well be a kernel bug: the changes to allow 39-bit VA kernels
>> to execute on systems whose memory is outside the 39-bit VA range was
>> developed and tested on AMD Seattle only, which has its RAM at
>> 0x80_0000_0000. But it is hard to be sure, of course, with no output
>> at all ...
>>
>> > Does the linear mapping we've been talking about just need to cover
>> > all of _RAM_, or does it need to cover all of I/O as well in
>> > the same mapping.
>> >
>> > In our case we have various SoC I/O devices, like the UART
>> > in the space below 4 GB... at 0x0100_0000.
>> >
>>
>> The linear mapping only covers system RAM, so that should not matter at all.
>>
>> > Before debugging further I want to check whether this should
>> > work in theory. I had the impression that the issue was
>> > discontiguous RAM regions, and I/O was a separate issue.
>>
>> Yes, it should work in theory. The linear region starts at the base of
>> the kernel Image, and extends all the way up to the end of DRAM (14 GB
>> in your case, IIRC)
>
> An update on this... in my original test where the kernel did not run from
> high memory I was using a 4.0.x kernel. I re-tried with a 4.2-rc4 kernel
> and it booted fine. So, there is no kernel bug it in the latest kernel,
> it appears.
>
OK, thanks for reporting back.
So we still need to decide how to fix the case where the linear region
is not of sufficient size to cover all of memory, considering that is
what got this discussion started in the first place.
RobH was not too happy about how I proposed to split off
early_init_dt_add_memory_arch () but there has been no movement on
this since.
--
Ard.
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-08-14 12:15 ` Ard Biesheuvel
@ 2015-08-14 13:24 ` Catalin Marinas
2015-08-14 13:55 ` Ard Biesheuvel
0 siblings, 1 reply; 30+ messages in thread
From: Catalin Marinas @ 2015-08-14 13:24 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, Aug 14, 2015 at 02:15:23PM +0200, Ard Biesheuvel wrote:
> >> On 7 aug. 2015, at 21:01, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >>
> >> >> Whether defconfig supports your platform optimally has nothing to do
> >> >> with that. Of course, we should deal with the unexpected memory layout
> >> >> gracefully, which is why Mark Rutland and myself proposed patches to
> >> >> fix the panic you reported. But in a development context, I think it
> >> >> is perfectly acceptable to simply load the kernel at 0x80_8000_0000,
> >> >> and be able to run defconfig fine while losing just 2 GB of your 16 GB
> >> >> at the low end.
[...]
> So we still need to decide how to fix the case where the linear region
> is not of sufficient size to cover all of memory, considering that is
> what got this discussion started in the first place.
We already have a solution, just enable 4-levels of page tables (only
that I don't think we should change defconfig as well).
If we want to do any tricks like compacting the memory range, it needs
to be backed by benchmarks to prove that it's worth compared to a full
48-bit VA.
--
Catalin
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-08-14 13:24 ` Catalin Marinas
@ 2015-08-14 13:55 ` Ard Biesheuvel
2015-08-14 15:37 ` Catalin Marinas
0 siblings, 1 reply; 30+ messages in thread
From: Ard Biesheuvel @ 2015-08-14 13:55 UTC (permalink / raw)
To: linux-arm-kernel
On 14 August 2015 at 15:24, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Fri, Aug 14, 2015 at 02:15:23PM +0200, Ard Biesheuvel wrote:
>> >> On 7 aug. 2015, at 21:01, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> >>
>> >> >> Whether defconfig supports your platform optimally has nothing to do
>> >> >> with that. Of course, we should deal with the unexpected memory layout
>> >> >> gracefully, which is why Mark Rutland and myself proposed patches to
>> >> >> fix the panic you reported. But in a development context, I think it
>> >> >> is perfectly acceptable to simply load the kernel at 0x80_8000_0000,
>> >> >> and be able to run defconfig fine while losing just 2 GB of your 16 GB
>> >> >> at the low end.
> [...]
>> So we still need to decide how to fix the case where the linear region
>> is not of sufficient size to cover all of memory, considering that is
>> what got this discussion started in the first place.
>
> We already have a solution, just enable 4-levels of page tables (only
> that I don't think we should change defconfig as well).
>
> If we want to do any tricks like compacting the memory range, it needs
> to be backed by benchmarks to prove that it's worth compared to a full
> 48-bit VA.
>
If you run a 39-bit VA kernel on a system whose DRAM configuration
covers a larger area than what the linear mapping can support, you
currently get a panic since phys_to_virt() overflows into the user
range. So at the very least, we need to clip the range to a size the
kernel can manage.
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-08-14 13:55 ` Ard Biesheuvel
@ 2015-08-14 15:37 ` Catalin Marinas
0 siblings, 0 replies; 30+ messages in thread
From: Catalin Marinas @ 2015-08-14 15:37 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, Aug 14, 2015 at 03:55:19PM +0200, Ard Biesheuvel wrote:
> On 14 August 2015 at 15:24, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Fri, Aug 14, 2015 at 02:15:23PM +0200, Ard Biesheuvel wrote:
> >> >> On 7 aug. 2015, at 21:01, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >> >>
> >> >> >> Whether defconfig supports your platform optimally has nothing to do
> >> >> >> with that. Of course, we should deal with the unexpected memory layout
> >> >> >> gracefully, which is why Mark Rutland and myself proposed patches to
> >> >> >> fix the panic you reported. But in a development context, I think it
> >> >> >> is perfectly acceptable to simply load the kernel at 0x80_8000_0000,
> >> >> >> and be able to run defconfig fine while losing just 2 GB of your 16 GB
> >> >> >> at the low end.
> > [...]
> >> So we still need to decide how to fix the case where the linear region
> >> is not of sufficient size to cover all of memory, considering that is
> >> what got this discussion started in the first place.
> >
> > We already have a solution, just enable 4-levels of page tables (only
> > that I don't think we should change defconfig as well).
> >
> > If we want to do any tricks like compacting the memory range, it needs
> > to be backed by benchmarks to prove that it's worth compared to a full
> > 48-bit VA.
>
> If you run a 39-bit VA kernel on a system whose DRAM configuration
> covers a larger area than what the linear mapping can support, you
> currently get a panic since phys_to_virt() overflows into the user
> range. So at the very least, we need to clip the range to a size the
> kernel can manage.
Yes, this needs fixing (clipping looks like the best option).
--
Catalin
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-30 10:13 ` Catalin Marinas
2015-07-30 14:52 ` Stuart Yoder
@ 2015-07-30 19:27 ` Ard Biesheuvel
2015-07-31 12:53 ` Catalin Marinas
1 sibling, 1 reply; 30+ messages in thread
From: Ard Biesheuvel @ 2015-07-30 19:27 UTC (permalink / raw)
To: linux-arm-kernel
On 30 July 2015 at 12:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
>> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
[...]
>> > To be honest, I think this is poorly designed, and I am not sure we
>> > should cater for such configurations in the defconfig.
>>
>> Agree, if this is a one-off weird platform then we shouldn't.
>>
>> But, the 'Principles of ARM Memory Maps' doc proposes this:
>> 2 GB at 0x8000_0000
>> 30 GB at 0x8_8000_0000
>> 480 GB at 0x88_0000_0000
>
> I'm not particularly recommending this layout, at least not without some
> clarifications on DRAM aliases (I'll ping people internally about it
> again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
> and all the memory beyond 32-bit was highmem anyway. It was later
> updated for AArch64 but only to allow 44/48-bit PA (a few sections
> added).
>
As an aside, is there any reason why the direct mapping *must* be a
linear mapping?
Other than the performance concerns regarding phys_to_virt/virt_to_phys, I mean?
> How I understood it at the time is that in hardware you place the large
> DRAM (chip select) at high addresses, e.g. 64GB would be at a 512GB
> offset, but lower 2GB+30GB aliased in the lower space with the
> corresponding high address remaining a hole (that produces external
> aborts if accessed). The first alias at 2GB was meant for 32-bit
> initialisation code before the MMU is enabled. With AArch64, you don't
> really need this low alias (assuming your EL3/EL2/secure-EL1 code is
> 64-bit) since PC can use >32-bit addresses.
>
>> ...i.e. if you have > 32 GB then your RAM regions are split into 3
>> chunks. The aarch64 kernel will support > than 32GB right? A
>> basic server will have that much or more.
>>
>> How will we deal with systems with > 32GB of memory that follow that
>> map?
>
> My recommendation would be to avoid the lower alias (in hardware or some
> EL3 configuration change) and keep all the RAM at 0x80_0000_0000, just
> boot the system in AArch64 mode.
>
>> When do we expect the default page size for the aarch64 kernel to be
>> changed to 64KB? Any workload that puts pressure on the TLBs will benefit
>> from this.
>
> This comes with its own set problems. 64KB is useful for specific
> use-cases (e.g. large databases) but not general purpose where you waste
> a large amount of RAM. So there are no plans to change the default page
> size to 64KB.
>
> --
> Catalin
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-30 19:27 ` Ard Biesheuvel
@ 2015-07-31 12:53 ` Catalin Marinas
2015-07-31 13:10 ` Ard Biesheuvel
0 siblings, 1 reply; 30+ messages in thread
From: Catalin Marinas @ 2015-07-31 12:53 UTC (permalink / raw)
To: linux-arm-kernel
On Thu, Jul 30, 2015 at 09:27:03PM +0200, Ard Biesheuvel wrote:
> On 30 July 2015 at 12:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
> >> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> [...]
> >> > To be honest, I think this is poorly designed, and I am not sure we
> >> > should cater for such configurations in the defconfig.
> >>
> >> Agree, if this is a one-off weird platform then we shouldn't.
> >>
> >> But, the 'Principles of ARM Memory Maps' doc proposes this:
> >> 2 GB at 0x8000_0000
> >> 30 GB at 0x8_8000_0000
> >> 480 GB at 0x88_0000_0000
> >
> > I'm not particularly recommending this layout, at least not without some
> > clarifications on DRAM aliases (I'll ping people internally about it
> > again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
> > and all the memory beyond 32-bit was highmem anyway. It was later
> > updated for AArch64 but only to allow 44/48-bit PA (a few sections
> > added).
>
> As an aside, is there any reason why the direct mapping *must* be a
> linear mapping?
> Other than the performance concerns regarding
> phys_to_virt/virt_to_phys, I mean?
Mostly performance concerns. You could compact the physical range into a
smaller virtual one but the conversion will be costly, especially if you
want to make it multi-platform (having to look-up memory ranges,
memblock offsets). This would affect page table entry setup, code that
requires a page structure (like virt_to_page) and anything else doing
the virt/phys conversion.
I tried something like that for RealView PBX in the past but it was
hard-coded (no multi-platform at the time). See
arch/arm/mach-realview/include/mach/memory.h.
As for the mem_map array, on arm64 we have enough VA space available
(with vmemmap it takes a few GB to cover a 512GB RAM space).
--
Catalin
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-31 12:53 ` Catalin Marinas
@ 2015-07-31 13:10 ` Ard Biesheuvel
2015-07-31 13:22 ` Catalin Marinas
0 siblings, 1 reply; 30+ messages in thread
From: Ard Biesheuvel @ 2015-07-31 13:10 UTC (permalink / raw)
To: linux-arm-kernel
On 31 July 2015 at 14:53, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Thu, Jul 30, 2015 at 09:27:03PM +0200, Ard Biesheuvel wrote:
>> On 30 July 2015 at 12:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
>> >> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> [...]
>> >> > To be honest, I think this is poorly designed, and I am not sure we
>> >> > should cater for such configurations in the defconfig.
>> >>
>> >> Agree, if this is a one-off weird platform then we shouldn't.
>> >>
>> >> But, the 'Principles of ARM Memory Maps' doc proposes this:
>> >> 2 GB at 0x8000_0000
>> >> 30 GB at 0x8_8000_0000
>> >> 480 GB at 0x88_0000_0000
>> >
>> > I'm not particularly recommending this layout, at least not without some
>> > clarifications on DRAM aliases (I'll ping people internally about it
>> > again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
>> > and all the memory beyond 32-bit was highmem anyway. It was later
>> > updated for AArch64 but only to allow 44/48-bit PA (a few sections
>> > added).
>>
>> As an aside, is there any reason why the direct mapping *must* be a
>> linear mapping?
>> Other than the performance concerns regarding
>> phys_to_virt/virt_to_phys, I mean?
>
> Mostly performance concerns. You could compact the physical range into a
> smaller virtual one but the conversion will be costly, especially if you
> want to make it multi-platform (having to look-up memory ranges,
> memblock offsets). This would affect page table entry setup, code that
> requires a page structure (like virt_to_page) and anything else doing
> the virt/phys conversion.
>
> I tried something like that for RealView PBX in the past but it was
> hard-coded (no multi-platform at the time). See
> arch/arm/mach-realview/include/mach/memory.h.
>
Yes, that looks vaguely like what I had in mind.
IOW, we could partition the direct mapping just like the ARM
recommendation, i.e.,
0 - 2 GB
2 - 32 GB
32 - 512 GB
but default to 1:1 correspondence, so that
PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
PHYS_OFFSET1 = memstart_addr + 2 GB
PHYS_OFFSET2 = memstart_addr + 32 GB
and only if the ARM recommended physical memory map is detected (with
memstart_addr @ 0x8000_0000), switch to
PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
PHYS_OFFSET1 = memstart_addr + 30 GB
PHYS_OFFSET2 = memstart_addr + 480 GB
I guess such a special case would be out of the question for one-off
crazy designs like Freescale, but since this is the layout recommended
by ARM itself, I suppose we could try and support it a bit better.
> As for the mem_map array, on arm64 we have enough VA space available
> (with vmemmap it takes a few GB to cover a 512GB RAM space).
>
> --
> Catalin
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-31 13:10 ` Ard Biesheuvel
@ 2015-07-31 13:22 ` Catalin Marinas
2015-07-31 13:30 ` Ard Biesheuvel
0 siblings, 1 reply; 30+ messages in thread
From: Catalin Marinas @ 2015-07-31 13:22 UTC (permalink / raw)
To: linux-arm-kernel
On Fri, Jul 31, 2015 at 03:10:39PM +0200, Ard Biesheuvel wrote:
> On 31 July 2015 at 14:53, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Jul 30, 2015 at 09:27:03PM +0200, Ard Biesheuvel wrote:
> >> On 30 July 2015 at 12:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> > On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
> >> >> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> >> [...]
> >> >> > To be honest, I think this is poorly designed, and I am not sure we
> >> >> > should cater for such configurations in the defconfig.
> >> >>
> >> >> Agree, if this is a one-off weird platform then we shouldn't.
> >> >>
> >> >> But, the 'Principles of ARM Memory Maps' doc proposes this:
> >> >> 2 GB at 0x8000_0000
> >> >> 30 GB at 0x8_8000_0000
> >> >> 480 GB at 0x88_0000_0000
> >> >
> >> > I'm not particularly recommending this layout, at least not without some
> >> > clarifications on DRAM aliases (I'll ping people internally about it
> >> > again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
> >> > and all the memory beyond 32-bit was highmem anyway. It was later
> >> > updated for AArch64 but only to allow 44/48-bit PA (a few sections
> >> > added).
> >>
> >> As an aside, is there any reason why the direct mapping *must* be a
> >> linear mapping?
> >> Other than the performance concerns regarding
> >> phys_to_virt/virt_to_phys, I mean?
> >
> > Mostly performance concerns. You could compact the physical range into a
> > smaller virtual one but the conversion will be costly, especially if you
> > want to make it multi-platform (having to look-up memory ranges,
> > memblock offsets). This would affect page table entry setup, code that
> > requires a page structure (like virt_to_page) and anything else doing
> > the virt/phys conversion.
> >
> > I tried something like that for RealView PBX in the past but it was
> > hard-coded (no multi-platform at the time). See
> > arch/arm/mach-realview/include/mach/memory.h.
>
> Yes, that looks vaguely like what I had in mind.
>
> IOW, we could partition the direct mapping just like the ARM
> recommendation, i.e.,
>
> 0 - 2 GB
> 2 - 32 GB
> 32 - 512 GB
>
> but default to 1:1 correspondence, so that
>
> PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> PHYS_OFFSET1 = memstart_addr + 2 GB
> PHYS_OFFSET2 = memstart_addr + 32 GB
>
> and only if the ARM recommended physical memory map is detected (with
> memstart_addr @ 0x8000_0000), switch to
>
> PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> PHYS_OFFSET1 = memstart_addr + 30 GB
> PHYS_OFFSET2 = memstart_addr + 480 GB
I don't really like such complexity when all you need on arm64 is to
enable 48-bit VA (though it would be interesting to benchmark it).
> I guess such a special case would be out of the question for one-off
> crazy designs like Freescale, but since this is the layout recommended
> by ARM itself, I suppose we could try and support it a bit better.
I'm trying to get the layout fixed before it spreads any further ;).
Interestingly, SBSA doesn't mention this document at all.
--
Catalin
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-31 13:22 ` Catalin Marinas
@ 2015-07-31 13:30 ` Ard Biesheuvel
2015-08-01 21:08 ` Arnd Bergmann
0 siblings, 1 reply; 30+ messages in thread
From: Ard Biesheuvel @ 2015-07-31 13:30 UTC (permalink / raw)
To: linux-arm-kernel
On 31 July 2015 at 15:22, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Fri, Jul 31, 2015 at 03:10:39PM +0200, Ard Biesheuvel wrote:
>> On 31 July 2015 at 14:53, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > On Thu, Jul 30, 2015 at 09:27:03PM +0200, Ard Biesheuvel wrote:
>> >> On 30 July 2015 at 12:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> >> > On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
>> >> >> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> >> [...]
>> >> >> > To be honest, I think this is poorly designed, and I am not sure we
>> >> >> > should cater for such configurations in the defconfig.
>> >> >>
>> >> >> Agree, if this is a one-off weird platform then we shouldn't.
>> >> >>
>> >> >> But, the 'Principles of ARM Memory Maps' doc proposes this:
>> >> >> 2 GB at 0x8000_0000
>> >> >> 30 GB at 0x8_8000_0000
>> >> >> 480 GB at 0x88_0000_0000
>> >> >
>> >> > I'm not particularly recommending this layout, at least not without some
>> >> > clarifications on DRAM aliases (I'll ping people internally about it
>> >> > again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
>> >> > and all the memory beyond 32-bit was highmem anyway. It was later
>> >> > updated for AArch64 but only to allow 44/48-bit PA (a few sections
>> >> > added).
>> >>
>> >> As an aside, is there any reason why the direct mapping *must* be a
>> >> linear mapping?
>> >> Other than the performance concerns regarding
>> >> phys_to_virt/virt_to_phys, I mean?
>> >
>> > Mostly performance concerns. You could compact the physical range into a
>> > smaller virtual one but the conversion will be costly, especially if you
>> > want to make it multi-platform (having to look-up memory ranges,
>> > memblock offsets). This would affect page table entry setup, code that
>> > requires a page structure (like virt_to_page) and anything else doing
>> > the virt/phys conversion.
>> >
>> > I tried something like that for RealView PBX in the past but it was
>> > hard-coded (no multi-platform at the time). See
>> > arch/arm/mach-realview/include/mach/memory.h.
>>
>> Yes, that looks vaguely like what I had in mind.
>>
>> IOW, we could partition the direct mapping just like the ARM
>> recommendation, i.e.,
>>
>> 0 - 2 GB
>> 2 - 32 GB
>> 32 - 512 GB
>>
>> but default to 1:1 correspondence, so that
>>
>> PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
>> PHYS_OFFSET1 = memstart_addr + 2 GB
>> PHYS_OFFSET2 = memstart_addr + 32 GB
>>
>> and only if the ARM recommended physical memory map is detected (with
>> memstart_addr @ 0x8000_0000), switch to
>>
>> PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
>> PHYS_OFFSET1 = memstart_addr + 30 GB
>> PHYS_OFFSET2 = memstart_addr + 480 GB
>
> I don't really like such complexity when all you need on arm64 is to
> enable 48-bit VA (though it would be interesting to benchmark it).
>
>> I guess such a special case would be out of the question for one-off
>> crazy designs like Freescale, but since this is the layout recommended
>> by ARM itself, I suppose we could try and support it a bit better.
>
> I'm trying to get the layout fixed before it spreads any further ;).
Yes, that would be even better.
> Interestingly, SBSA doesn't mention this document at all.
>
No, and the document itself never claims that its recommendations
apply universally, i.e., from the introduction
"""
This document describes the address maps used by ARM for A-class
systems, from models and emulators to development boards and complex
SoCs.
"""
where 'used by ARM' should probably be clarified to mean 'used by ARM
Ltd. for its own designs' rather than 'recommended by ARM Ltd. for all
designs by licensees'
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-07-31 13:30 ` Ard Biesheuvel
@ 2015-08-01 21:08 ` Arnd Bergmann
2015-08-02 6:19 ` Ard Biesheuvel
0 siblings, 1 reply; 30+ messages in thread
From: Arnd Bergmann @ 2015-08-01 21:08 UTC (permalink / raw)
To: linux-arm-kernel
On Friday 31 July 2015 15:30:23 Ard Biesheuvel wrote:
> On 31 July 2015 at 15:22, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> > I tried something like that for RealView PBX in the past but it was
> >> > hard-coded (no multi-platform at the time). See
> >> > arch/arm/mach-realview/include/mach/memory.h.
> >>
> >> Yes, that looks vaguely like what I had in mind.
> >>
> >> IOW, we could partition the direct mapping just like the ARM
> >> recommendation, i.e.,
> >>
> >> 0 - 2 GB
> >> 2 - 32 GB
> >> 32 - 512 GB
> >>
> >> but default to 1:1 correspondence, so that
> >>
> >> PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> >> PHYS_OFFSET1 = memstart_addr + 2 GB
> >> PHYS_OFFSET2 = memstart_addr + 32 GB
> >>
> >> and only if the ARM recommended physical memory map is detected (with
> >> memstart_addr @ 0x8000_0000), switch to
> >>
> >> PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> >> PHYS_OFFSET1 = memstart_addr + 30 GB
> >> PHYS_OFFSET2 = memstart_addr + 480 GB
> >
> > I don't really like such complexity when all you need on arm64 is to
> > enable 48-bit VA (though it would be interesting to benchmark it).
More importantly, hardwiring this in virt_to_phys() would mean we can
no longer run a kernel with this hack turned on with systems that have
contiguous memory or any other layout besides the one from the document.
I also don't think it's a huge issue. On systems with lots of RAM,
you want the 48-bit VA space anyway to run things like databases
(which tend to mmap large files) efficiently, so all distros will
use that anyway (except the crazy ones that use 64kb pages and
also don't have the problem), and anyone with less than 32GB can
live with 3-leval tables if they run a custom kernel.
> >> I guess such a special case would be out of the question for one-off
> >> crazy designs like Freescale, but since this is the layout recommended
> >> by ARM itself, I suppose we could try and support it a bit better.
> >
> > I'm trying to get the layout fixed before it spreads any further ;).
>
> Yes, that would be even better.
+1
Arnd
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-08-01 21:08 ` Arnd Bergmann
@ 2015-08-02 6:19 ` Ard Biesheuvel
2015-08-03 8:00 ` Arnd Bergmann
0 siblings, 1 reply; 30+ messages in thread
From: Ard Biesheuvel @ 2015-08-02 6:19 UTC (permalink / raw)
To: linux-arm-kernel
On 1 August 2015 at 23:08, Arnd Bergmann <arnd@arndb.de> wrote:
> On Friday 31 July 2015 15:30:23 Ard Biesheuvel wrote:
>> On 31 July 2015 at 15:22, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> >> > I tried something like that for RealView PBX in the past but it was
>> >> > hard-coded (no multi-platform at the time). See
>> >> > arch/arm/mach-realview/include/mach/memory.h.
>> >>
>> >> Yes, that looks vaguely like what I had in mind.
>> >>
>> >> IOW, we could partition the direct mapping just like the ARM
>> >> recommendation, i.e.,
>> >>
>> >> 0 - 2 GB
>> >> 2 - 32 GB
>> >> 32 - 512 GB
>> >>
>> >> but default to 1:1 correspondence, so that
>> >>
>> >> PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
>> >> PHYS_OFFSET1 = memstart_addr + 2 GB
>> >> PHYS_OFFSET2 = memstart_addr + 32 GB
>> >>
>> >> and only if the ARM recommended physical memory map is detected (with
>> >> memstart_addr @ 0x8000_0000), switch to
>> >>
>> >> PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
>> >> PHYS_OFFSET1 = memstart_addr + 30 GB
>> >> PHYS_OFFSET2 = memstart_addr + 480 GB
>> >
>> > I don't really like such complexity when all you need on arm64 is to
>> > enable 48-bit VA (though it would be interesting to benchmark it).
>
> More importantly, hardwiring this in virt_to_phys() would mean we can
> no longer run a kernel with this hack turned on with systems that have
> contiguous memory or any other layout besides the one from the document.
>
Well the hack, if enabled, would be turned on dynamically iff the
exact layout from the recommendation is detected, and the kernel is
loaded right at the base of DRAM.
Note that the first layout I presented is effectively the current
linear mapping, but with three steps.
> I also don't think it's a huge issue. On systems with lots of RAM,
> you want the 48-bit VA space anyway to run things like databases
> (which tend to mmap large files) efficiently, so all distros will
> use that anyway (except the crazy ones that use 64kb pages and
> also don't have the problem), and anyone with less than 32GB can
> live with 3-leval tables if they run a custom kernel.
>
>> >> I guess such a special case would be out of the question for one-off
>> >> crazy designs like Freescale, but since this is the layout recommended
>> >> by ARM itself, I suppose we could try and support it a bit better.
>> >
>> > I'm trying to get the layout fixed before it spreads any further ;).
>>
>> Yes, that would be even better.
>
> +1
>
> Arnd
^ permalink raw reply [flat|nested] 30+ messages in thread
* [RFC] arm64: defconfig: enable 48-bit VA by default
2015-08-02 6:19 ` Ard Biesheuvel
@ 2015-08-03 8:00 ` Arnd Bergmann
0 siblings, 0 replies; 30+ messages in thread
From: Arnd Bergmann @ 2015-08-03 8:00 UTC (permalink / raw)
To: linux-arm-kernel
On Sunday 02 August 2015 08:19:07 Ard Biesheuvel wrote:
> >> >>
> >> >> PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> >> >> PHYS_OFFSET1 = memstart_addr + 2 GB
> >> >> PHYS_OFFSET2 = memstart_addr + 32 GB
> >> >>
> >> >> and only if the ARM recommended physical memory map is detected (with
> >> >> memstart_addr @ 0x8000_0000), switch to
> >> >>
> >> >> PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> >> >> PHYS_OFFSET1 = memstart_addr + 30 GB
> >> >> PHYS_OFFSET2 = memstart_addr + 480 GB
> >> >
> >> > I don't really like such complexity when all you need on arm64 is to
> >> > enable 48-bit VA (though it would be interesting to benchmark it).
> >
> > More importantly, hardwiring this in virt_to_phys() would mean we can
> > no longer run a kernel with this hack turned on with systems that have
> > contiguous memory or any other layout besides the one from the document.
> >
>
> Well the hack, if enabled, would be turned on dynamically iff the
> exact layout from the recommendation is detected, and the kernel is
> loaded right at the base of DRAM.
> Note that the first layout I presented is effectively the current
> linear mapping, but with three steps.
Ok, I see. So it would work everywhere, at the price of another conditional
for each virt_to_phys, either runtime or by binary patching at boottime.
Arnd
^ permalink raw reply [flat|nested] 30+ messages in thread