All of lore.kernel.org
 help / color / mirror / Atom feed
* ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA by default)
@ 2016-06-10 16:34 Stuart Yoder
  2016-06-11 11:38 ` Ard Biesheuvel
  0 siblings, 1 reply; 7+ messages in thread
From: Stuart Yoder @ 2016-06-10 16:34 UTC (permalink / raw)
  To: linux-arm-kernel


> -----Original Message-----
> From: Catalin Marinas [mailto:catalin.marinas at arm.com]
> Sent: Friday, July 31, 2015 8:23 AM
> To: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Cc: Mark Rutland <mark.rutland@arm.com>; Marc Zyngier <marc.zyngier@arm.com>; Will Deacon
> <Will.Deacon@arm.com>; Stuart Yoder <stuart.yoder@freescale.com>; Peter Newton
> <Peter.Newton@freescale.com>; linux-arm-kernel at lists.infradead.org
> Subject: Re: [RFC] arm64: defconfig: enable 48-bit VA by default
> 
> On Fri, Jul 31, 2015 at 03:10:39PM +0200, Ard Biesheuvel wrote:
> > On 31 July 2015 at 14:53, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > On Thu, Jul 30, 2015 at 09:27:03PM +0200, Ard Biesheuvel wrote:
> > >> On 30 July 2015 at 12:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > >> > On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
> > >> >> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> > >> [...]
> > >> >> > To be honest, I think this is poorly designed, and I am not sure we
> > >> >> > should cater for such configurations in the defconfig.
> > >> >>
> > >> >> Agree, if this is a one-off weird platform then we shouldn't.
> > >> >>
> > >> >> But, the 'Principles of ARM Memory Maps' doc proposes this:
> > >> >> 2 GB at 0x8000_0000
> > >> >> 30 GB at 0x8_8000_0000
> > >> >> 480 GB at 0x88_0000_0000
> > >> >
> > >> > I'm not particularly recommending this layout, at least not without some
> > >> > clarifications on DRAM aliases (I'll ping people internally about it
> > >> > again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
> > >> > and all the memory beyond 32-bit was highmem anyway. It was later
> > >> > updated for AArch64 but only to allow 44/48-bit PA (a few sections
> > >> > added).
> > >>
> > >> As an aside, is there any reason why the direct mapping *must* be a
> > >> linear mapping?
> > >> Other than the performance concerns regarding
> > >> phys_to_virt/virt_to_phys, I mean?
> > >
> > > Mostly performance concerns. You could compact the physical range into a
> > > smaller virtual one but the conversion will be costly, especially if you
> > > want to make it multi-platform (having to look-up memory ranges,
> > > memblock offsets). This would affect page table entry setup, code that
> > > requires a page structure (like virt_to_page) and anything else doing
> > > the virt/phys conversion.
> > >
> > > I tried something like that for RealView PBX in the past but it was
> > > hard-coded (no multi-platform at the time). See
> > > arch/arm/mach-realview/include/mach/memory.h.
> >
> > Yes, that looks vaguely like what I had in mind.
> >
> > IOW, we could partition the direct mapping just like the ARM
> > recommendation, i.e.,
> >
> > 0 - 2 GB
> > 2 - 32 GB
> > 32 - 512 GB
> >
> > but default to 1:1 correspondence, so that
> >
> > PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> > PHYS_OFFSET1 = memstart_addr + 2 GB
> > PHYS_OFFSET2 = memstart_addr + 32 GB
> >
> > and only if the ARM recommended physical memory map is detected (with
> > memstart_addr @ 0x8000_0000), switch to
> >
> > PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> > PHYS_OFFSET1 = memstart_addr + 30 GB
> > PHYS_OFFSET2 = memstart_addr + 480 GB
> 
> I don't really like such complexity when all you need on arm64 is to
> enable 48-bit VA (though it would be interesting to benchmark it).
> 
> > I guess such a special case would be out of the question for one-off
> > crazy designs like Freescale, but since this is the layout recommended
> > by ARM itself, I suppose we could try and support it a bit better.
> 
> I'm trying to get the layout fixed before it spreads any further ;).

Hi Catalin, in this very old thread there was the intent on your side
to revisit the physical address layout recommended in the 'Principles of ARM
Memory Maps' whitepaper.  Has anything happened or changed in the
last ~1 year.  Does ARM have recommendations?  SBSA doesn't 
mandate anything as far as I can see.

We have the opportunity to influences some future designs and wanted to
see if ARM has recommendations.  In particular, what is the status of
the recommendations in 'Principles of ARM Memory Maps'?

Thanks,
Stuart

^ permalink raw reply	[flat|nested] 7+ messages in thread

* ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA by default)
  2016-06-10 16:34 ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA by default) Stuart Yoder
@ 2016-06-11 11:38 ` Ard Biesheuvel
  2016-06-13 18:41   ` Stuart Yoder
  0 siblings, 1 reply; 7+ messages in thread
From: Ard Biesheuvel @ 2016-06-11 11:38 UTC (permalink / raw)
  To: linux-arm-kernel

On 10 June 2016 at 18:34, Stuart Yoder <stuart.yoder@nxp.com> wrote:
>
>> -----Original Message-----
>> From: Catalin Marinas [mailto:catalin.marinas at arm.com]
>> Sent: Friday, July 31, 2015 8:23 AM
>> To: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>> Cc: Mark Rutland <mark.rutland@arm.com>; Marc Zyngier <marc.zyngier@arm.com>; Will Deacon
>> <Will.Deacon@arm.com>; Stuart Yoder <stuart.yoder@freescale.com>; Peter Newton
>> <Peter.Newton@freescale.com>; linux-arm-kernel at lists.infradead.org
>> Subject: Re: [RFC] arm64: defconfig: enable 48-bit VA by default
>>
>> On Fri, Jul 31, 2015 at 03:10:39PM +0200, Ard Biesheuvel wrote:
>> > On 31 July 2015 at 14:53, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > > On Thu, Jul 30, 2015 at 09:27:03PM +0200, Ard Biesheuvel wrote:
>> > >> On 30 July 2015 at 12:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > >> > On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
>> > >> >> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> > >> [...]
>> > >> >> > To be honest, I think this is poorly designed, and I am not sure we
>> > >> >> > should cater for such configurations in the defconfig.
>> > >> >>
>> > >> >> Agree, if this is a one-off weird platform then we shouldn't.
>> > >> >>
>> > >> >> But, the 'Principles of ARM Memory Maps' doc proposes this:
>> > >> >> 2 GB at 0x8000_0000
>> > >> >> 30 GB at 0x8_8000_0000
>> > >> >> 480 GB at 0x88_0000_0000
>> > >> >
>> > >> > I'm not particularly recommending this layout, at least not without some
>> > >> > clarifications on DRAM aliases (I'll ping people internally about it
>> > >> > again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
>> > >> > and all the memory beyond 32-bit was highmem anyway. It was later
>> > >> > updated for AArch64 but only to allow 44/48-bit PA (a few sections
>> > >> > added).
>> > >>
>> > >> As an aside, is there any reason why the direct mapping *must* be a
>> > >> linear mapping?
>> > >> Other than the performance concerns regarding
>> > >> phys_to_virt/virt_to_phys, I mean?
>> > >
>> > > Mostly performance concerns. You could compact the physical range into a
>> > > smaller virtual one but the conversion will be costly, especially if you
>> > > want to make it multi-platform (having to look-up memory ranges,
>> > > memblock offsets). This would affect page table entry setup, code that
>> > > requires a page structure (like virt_to_page) and anything else doing
>> > > the virt/phys conversion.
>> > >
>> > > I tried something like that for RealView PBX in the past but it was
>> > > hard-coded (no multi-platform at the time). See
>> > > arch/arm/mach-realview/include/mach/memory.h.
>> >
>> > Yes, that looks vaguely like what I had in mind.
>> >
>> > IOW, we could partition the direct mapping just like the ARM
>> > recommendation, i.e.,
>> >
>> > 0 - 2 GB
>> > 2 - 32 GB
>> > 32 - 512 GB
>> >
>> > but default to 1:1 correspondence, so that
>> >
>> > PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
>> > PHYS_OFFSET1 = memstart_addr + 2 GB
>> > PHYS_OFFSET2 = memstart_addr + 32 GB
>> >
>> > and only if the ARM recommended physical memory map is detected (with
>> > memstart_addr @ 0x8000_0000), switch to
>> >
>> > PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
>> > PHYS_OFFSET1 = memstart_addr + 30 GB
>> > PHYS_OFFSET2 = memstart_addr + 480 GB
>>
>> I don't really like such complexity when all you need on arm64 is to
>> enable 48-bit VA (though it would be interesting to benchmark it).
>>
>> > I guess such a special case would be out of the question for one-off
>> > crazy designs like Freescale, but since this is the layout recommended
>> > by ARM itself, I suppose we could try and support it a bit better.
>>
>> I'm trying to get the layout fixed before it spreads any further ;).
>
> Hi Catalin, in this very old thread there was the intent on your side
> to revisit the physical address layout recommended in the 'Principles of ARM
> Memory Maps' whitepaper.  Has anything happened or changed in the
> last ~1 year.  Does ARM have recommendations?  SBSA doesn't
> mandate anything as far as I can see.
>
> We have the opportunity to influences some future designs and wanted to
> see if ARM has recommendations.  In particular, what is the status of
> the recommendations in 'Principles of ARM Memory Maps'?
>

Frankly, I don't understand why anyone would use this document as a
definitive recommendation for how to lay out the physical address
space of a new SOC. From the doc itself:

"""
ARM creates a variety of development systems to support A-class cortex
CPUs, ranging from cycle accurate RTL models, to fast software models,
onto FPGAs and full custom SoCs. ARM has been harmonizing the memory
maps in these systems to provide internal consistency and software
portability, and to address the constraints that come with mixing
32-bit components within larger address spaces.
"""

IOW, the document explains why ARM systems are configured the way they
are, taking into consideration that they need to serve as references
for a variety of hardware and software development, using both 32-bit
and 64-bit components. The fact that neither the SBSA nor the document
itself present it as a recommendation for server platforms means that
it should not be mistaken for that.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA by default)
  2016-06-11 11:38 ` Ard Biesheuvel
@ 2016-06-13 18:41   ` Stuart Yoder
  2016-06-14  8:51     ` Ard Biesheuvel
  0 siblings, 1 reply; 7+ messages in thread
From: Stuart Yoder @ 2016-06-13 18:41 UTC (permalink / raw)
  To: linux-arm-kernel



> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> Sent: Saturday, June 11, 2016 6:38 AM
> To: Stuart Yoder <stuart.yoder@nxp.com>
> Cc: Catalin Marinas <catalin.marinas@arm.com>; Mark Rutland <mark.rutland@arm.com>; Marc Zyngier
> <marc.zyngier@arm.com>; Will Deacon <Will.Deacon@arm.com>; Peter Newton <Peter.Newton@freescale.com>;
> linux-arm-kernel at lists.infradead.org
> Subject: Re: ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA
> by default)
> 
> On 10 June 2016 at 18:34, Stuart Yoder <stuart.yoder@nxp.com> wrote:
> >
> >> -----Original Message-----
> >> From: Catalin Marinas [mailto:catalin.marinas at arm.com]
> >> Sent: Friday, July 31, 2015 8:23 AM
> >> To: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> >> Cc: Mark Rutland <mark.rutland@arm.com>; Marc Zyngier <marc.zyngier@arm.com>; Will Deacon
> >> <Will.Deacon@arm.com>; Stuart Yoder <stuart.yoder@freescale.com>; Peter Newton
> >> <Peter.Newton@freescale.com>; linux-arm-kernel at lists.infradead.org
> >> Subject: Re: [RFC] arm64: defconfig: enable 48-bit VA by default
> >>
> >> On Fri, Jul 31, 2015 at 03:10:39PM +0200, Ard Biesheuvel wrote:
> >> > On 31 July 2015 at 14:53, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> > > On Thu, Jul 30, 2015 at 09:27:03PM +0200, Ard Biesheuvel wrote:
> >> > >> On 30 July 2015 at 12:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> > >> > On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
> >> > >> >> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> >> > >> [...]
> >> > >> >> > To be honest, I think this is poorly designed, and I am not sure we
> >> > >> >> > should cater for such configurations in the defconfig.
> >> > >> >>
> >> > >> >> Agree, if this is a one-off weird platform then we shouldn't.
> >> > >> >>
> >> > >> >> But, the 'Principles of ARM Memory Maps' doc proposes this:
> >> > >> >> 2 GB at 0x8000_0000
> >> > >> >> 30 GB at 0x8_8000_0000
> >> > >> >> 480 GB at 0x88_0000_0000
> >> > >> >
> >> > >> > I'm not particularly recommending this layout, at least not without some
> >> > >> > clarifications on DRAM aliases (I'll ping people internally about it
> >> > >> > again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
> >> > >> > and all the memory beyond 32-bit was highmem anyway. It was later
> >> > >> > updated for AArch64 but only to allow 44/48-bit PA (a few sections
> >> > >> > added).
> >> > >>
> >> > >> As an aside, is there any reason why the direct mapping *must* be a
> >> > >> linear mapping?
> >> > >> Other than the performance concerns regarding
> >> > >> phys_to_virt/virt_to_phys, I mean?
> >> > >
> >> > > Mostly performance concerns. You could compact the physical range into a
> >> > > smaller virtual one but the conversion will be costly, especially if you
> >> > > want to make it multi-platform (having to look-up memory ranges,
> >> > > memblock offsets). This would affect page table entry setup, code that
> >> > > requires a page structure (like virt_to_page) and anything else doing
> >> > > the virt/phys conversion.
> >> > >
> >> > > I tried something like that for RealView PBX in the past but it was
> >> > > hard-coded (no multi-platform at the time). See
> >> > > arch/arm/mach-realview/include/mach/memory.h.
> >> >
> >> > Yes, that looks vaguely like what I had in mind.
> >> >
> >> > IOW, we could partition the direct mapping just like the ARM
> >> > recommendation, i.e.,
> >> >
> >> > 0 - 2 GB
> >> > 2 - 32 GB
> >> > 32 - 512 GB
> >> >
> >> > but default to 1:1 correspondence, so that
> >> >
> >> > PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> >> > PHYS_OFFSET1 = memstart_addr + 2 GB
> >> > PHYS_OFFSET2 = memstart_addr + 32 GB
> >> >
> >> > and only if the ARM recommended physical memory map is detected (with
> >> > memstart_addr @ 0x8000_0000), switch to
> >> >
> >> > PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> >> > PHYS_OFFSET1 = memstart_addr + 30 GB
> >> > PHYS_OFFSET2 = memstart_addr + 480 GB
> >>
> >> I don't really like such complexity when all you need on arm64 is to
> >> enable 48-bit VA (though it would be interesting to benchmark it).
> >>
> >> > I guess such a special case would be out of the question for one-off
> >> > crazy designs like Freescale, but since this is the layout recommended
> >> > by ARM itself, I suppose we could try and support it a bit better.
> >>
> >> I'm trying to get the layout fixed before it spreads any further ;).
> >
> > Hi Catalin, in this very old thread there was the intent on your side
> > to revisit the physical address layout recommended in the 'Principles of ARM
> > Memory Maps' whitepaper.  Has anything happened or changed in the
> > last ~1 year.  Does ARM have recommendations?  SBSA doesn't
> > mandate anything as far as I can see.
> >
> > We have the opportunity to influences some future designs and wanted to
> > see if ARM has recommendations.  In particular, what is the status of
> > the recommendations in 'Principles of ARM Memory Maps'?
> >
> 
> Frankly, I don't understand why anyone would use this document as a
> definitive recommendation for how to lay out the physical address
> space of a new SOC. From the doc itself:
> 
> """
> ARM creates a variety of development systems to support A-class cortex
> CPUs, ranging from cycle accurate RTL models, to fast software models,
> onto FPGAs and full custom SoCs. ARM has been harmonizing the memory
> maps in these systems to provide internal consistency and software
> portability, and to address the constraints that come with mixing
> 32-bit components within larger address spaces.
> """
> 
> IOW, the document explains why ARM systems are configured the way they
> are, taking into consideration that they need to serve as references
> for a variety of hardware and software development, using both 32-bit
> and 64-bit components. The fact that neither the SBSA nor the document
> itself present it as a recommendation for server platforms means that
> it should not be mistaken for that.

I guess how definitive someone might read it is open to interpretation
but our hardware guys intepreted it as the "ARM recommendation".
There are lot's of "must" statements in there, so its easy for me to
see how it can be intepreted to be fairly definitive.  It doesn't
say a 64GB DRAM part _could_ be sub-divided like...  It says "For example
a 64GB DRAM part _will_ be sub-divided into three regions."

In any case, we are not going to follow that document as definitive.
But, my question is whether there are any updated ARM recommendations?
Catalin hinted at taking some action to "get the layout fixed".
I'm simply wondering if there is any update.  I'm at the point where I can 
influence the physical memory layout of an SoC and wanted to check if the are
any recommendations I may not know about.

Thanks,
Stuart

^ permalink raw reply	[flat|nested] 7+ messages in thread

* ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA by default)
  2016-06-13 18:41   ` Stuart Yoder
@ 2016-06-14  8:51     ` Ard Biesheuvel
  2016-06-14 10:49       ` Catalin Marinas
                         ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Ard Biesheuvel @ 2016-06-14  8:51 UTC (permalink / raw)
  To: linux-arm-kernel

On 13 June 2016 at 20:41, Stuart Yoder <stuart.yoder@nxp.com> wrote:
>
>
>> -----Original Message-----
>> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> Sent: Saturday, June 11, 2016 6:38 AM
>> To: Stuart Yoder <stuart.yoder@nxp.com>
>> Cc: Catalin Marinas <catalin.marinas@arm.com>; Mark Rutland <mark.rutland@arm.com>; Marc Zyngier
>> <marc.zyngier@arm.com>; Will Deacon <Will.Deacon@arm.com>; Peter Newton <Peter.Newton@freescale.com>;
>> linux-arm-kernel at lists.infradead.org
>> Subject: Re: ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA
>> by default)
>>
>> On 10 June 2016 at 18:34, Stuart Yoder <stuart.yoder@nxp.com> wrote:
>> >
>> >> -----Original Message-----
>> >> From: Catalin Marinas [mailto:catalin.marinas at arm.com]
>> >> Sent: Friday, July 31, 2015 8:23 AM
>> >> To: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>> >> Cc: Mark Rutland <mark.rutland@arm.com>; Marc Zyngier <marc.zyngier@arm.com>; Will Deacon
>> >> <Will.Deacon@arm.com>; Stuart Yoder <stuart.yoder@freescale.com>; Peter Newton
>> >> <Peter.Newton@freescale.com>; linux-arm-kernel at lists.infradead.org
>> >> Subject: Re: [RFC] arm64: defconfig: enable 48-bit VA by default
>> >>
>> >> On Fri, Jul 31, 2015 at 03:10:39PM +0200, Ard Biesheuvel wrote:
>> >> > On 31 July 2015 at 14:53, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> >> > > On Thu, Jul 30, 2015 at 09:27:03PM +0200, Ard Biesheuvel wrote:
>> >> > >> On 30 July 2015 at 12:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> >> > >> > On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
>> >> > >> >> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> >> > >> [...]
>> >> > >> >> > To be honest, I think this is poorly designed, and I am not sure we
>> >> > >> >> > should cater for such configurations in the defconfig.
>> >> > >> >>
>> >> > >> >> Agree, if this is a one-off weird platform then we shouldn't.
>> >> > >> >>
>> >> > >> >> But, the 'Principles of ARM Memory Maps' doc proposes this:
>> >> > >> >> 2 GB at 0x8000_0000
>> >> > >> >> 30 GB at 0x8_8000_0000
>> >> > >> >> 480 GB at 0x88_0000_0000
>> >> > >> >
>> >> > >> > I'm not particularly recommending this layout, at least not without some
>> >> > >> > clarifications on DRAM aliases (I'll ping people internally about it
>> >> > >> > again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
>> >> > >> > and all the memory beyond 32-bit was highmem anyway. It was later
>> >> > >> > updated for AArch64 but only to allow 44/48-bit PA (a few sections
>> >> > >> > added).
>> >> > >>
>> >> > >> As an aside, is there any reason why the direct mapping *must* be a
>> >> > >> linear mapping?
>> >> > >> Other than the performance concerns regarding
>> >> > >> phys_to_virt/virt_to_phys, I mean?
>> >> > >
>> >> > > Mostly performance concerns. You could compact the physical range into a
>> >> > > smaller virtual one but the conversion will be costly, especially if you
>> >> > > want to make it multi-platform (having to look-up memory ranges,
>> >> > > memblock offsets). This would affect page table entry setup, code that
>> >> > > requires a page structure (like virt_to_page) and anything else doing
>> >> > > the virt/phys conversion.
>> >> > >
>> >> > > I tried something like that for RealView PBX in the past but it was
>> >> > > hard-coded (no multi-platform at the time). See
>> >> > > arch/arm/mach-realview/include/mach/memory.h.
>> >> >
>> >> > Yes, that looks vaguely like what I had in mind.
>> >> >
>> >> > IOW, we could partition the direct mapping just like the ARM
>> >> > recommendation, i.e.,
>> >> >
>> >> > 0 - 2 GB
>> >> > 2 - 32 GB
>> >> > 32 - 512 GB
>> >> >
>> >> > but default to 1:1 correspondence, so that
>> >> >
>> >> > PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
>> >> > PHYS_OFFSET1 = memstart_addr + 2 GB
>> >> > PHYS_OFFSET2 = memstart_addr + 32 GB
>> >> >
>> >> > and only if the ARM recommended physical memory map is detected (with
>> >> > memstart_addr @ 0x8000_0000), switch to
>> >> >
>> >> > PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
>> >> > PHYS_OFFSET1 = memstart_addr + 30 GB
>> >> > PHYS_OFFSET2 = memstart_addr + 480 GB
>> >>
>> >> I don't really like such complexity when all you need on arm64 is to
>> >> enable 48-bit VA (though it would be interesting to benchmark it).
>> >>
>> >> > I guess such a special case would be out of the question for one-off
>> >> > crazy designs like Freescale, but since this is the layout recommended
>> >> > by ARM itself, I suppose we could try and support it a bit better.
>> >>
>> >> I'm trying to get the layout fixed before it spreads any further ;).
>> >
>> > Hi Catalin, in this very old thread there was the intent on your side
>> > to revisit the physical address layout recommended in the 'Principles of ARM
>> > Memory Maps' whitepaper.  Has anything happened or changed in the
>> > last ~1 year.  Does ARM have recommendations?  SBSA doesn't
>> > mandate anything as far as I can see.
>> >
>> > We have the opportunity to influences some future designs and wanted to
>> > see if ARM has recommendations.  In particular, what is the status of
>> > the recommendations in 'Principles of ARM Memory Maps'?
>> >
>>
>> Frankly, I don't understand why anyone would use this document as a
>> definitive recommendation for how to lay out the physical address
>> space of a new SOC. From the doc itself:
>>
>> """
>> ARM creates a variety of development systems to support A-class cortex
>> CPUs, ranging from cycle accurate RTL models, to fast software models,
>> onto FPGAs and full custom SoCs. ARM has been harmonizing the memory
>> maps in these systems to provide internal consistency and software
>> portability, and to address the constraints that come with mixing
>> 32-bit components within larger address spaces.
>> """
>>
>> IOW, the document explains why ARM systems are configured the way they
>> are, taking into consideration that they need to serve as references
>> for a variety of hardware and software development, using both 32-bit
>> and 64-bit components. The fact that neither the SBSA nor the document
>> itself present it as a recommendation for server platforms means that
>> it should not be mistaken for that.
>
> I guess how definitive someone might read it is open to interpretation
> but our hardware guys intepreted it as the "ARM recommendation".
> There are lot's of "must" statements in there, so its easy for me to
> see how it can be intepreted to be fairly definitive.  It doesn't
> say a 64GB DRAM part _could_ be sub-divided like...  It says "For example
> a 64GB DRAM part _will_ be sub-divided into three regions."
>

Whether or not you interpret the tone as normative is only one side of
it. The reason I quoted the paragraph above is to emphasize that the
definition of an 'ARM system' is not clear either, i.e., whether it
means 'any system that implements the ARM architecture' or simply 'a
system created by ARM Ltd.'. The fact that the paragraph mentions
software models and FPGA implementations leans to the latter IMO.

> In any case, we are not going to follow that document as definitive.
> But, my question is whether there are any updated ARM recommendations?

That again suggests that the document in question is intended as a
general recommendation, which I believe may not be the case. So if we
could get some clarification on that as well from any of the ARM Ltd.
cc'ees, that would be great.

> Catalin hinted at taking some action to "get the layout fixed".
> I'm simply wondering if there is any update.  I'm at the point where I can
> influence the physical memory layout of an SoC and wanted to check if the are
> any recommendations I may not know about.
>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA by default)
  2016-06-14  8:51     ` Ard Biesheuvel
@ 2016-06-14 10:49       ` Catalin Marinas
  2016-06-14 11:04       ` Arnd Bergmann
  2016-06-14 13:55       ` Stuart Yoder
  2 siblings, 0 replies; 7+ messages in thread
From: Catalin Marinas @ 2016-06-14 10:49 UTC (permalink / raw)
  To: linux-arm-kernel

On Tue, Jun 14, 2016 at 10:51:57AM +0200, Ard Biesheuvel wrote:
> On 13 June 2016 at 20:41, Stuart Yoder <stuart.yoder@nxp.com> wrote:
> > I guess how definitive someone might read it is open to interpretation
> > but our hardware guys intepreted it as the "ARM recommendation".
> > There are lot's of "must" statements in there, so its easy for me to
> > see how it can be intepreted to be fairly definitive.  It doesn't
> > say a 64GB DRAM part _could_ be sub-divided like...  It says "For example
> > a 64GB DRAM part _will_ be sub-divided into three regions."
> 
> Whether or not you interpret the tone as normative is only one side of
> it. The reason I quoted the paragraph above is to emphasize that the
> definition of an 'ARM system' is not clear either, i.e., whether it
> means 'any system that implements the ARM architecture' or simply 'a
> system created by ARM Ltd.'. The fact that the paragraph mentions
> software models and FPGA implementations leans to the latter IMO.

This document describes the reasoning behind the memory map decision of
the ARM Ltd SoC/FPGA/model implementations (and I'm not aware of any
with more than 32GB of RAM) with proposed extensions for higher range
address space. It could be used as a guideline by SoC vendors but it's
not something mandated (and it also allows for flexibility, see below).

> > In any case, we are not going to follow that document as definitive.
> > But, my question is whether there are any updated ARM recommendations?
> 
> That again suggests that the document in question is intended as a
> general recommendation, which I believe may not be the case. So if we
> could get some clarification on that as well from any of the ARM Ltd.
> cc'ees, that would be great.

I started a discussion within ARM about a year ago but we didn't reach
a clear conclusion on whether/how this document should be updated. It
was written in a way to cope with both hardware requirements (chip
select) and potential 32-bit software/peripheral needs.

If you are targeting a server platform, I strongly recommend SBSA which,
in 4.1.3, states: "This specification does not mandate a standard memory
map. It is expected that the system memory map is described to system
software by system firmware data". There are further clarifications on
the 64K granule size but that's about it.

Now, this "Principles of ARM memory maps" document could still be used
as a broad guideline as it allows some flexibility based on what
software you target. At the time it was written and still valid
currently, 32-bit software and 32-bit peripherals support is required in
certain configurations (e.g. mobile). I'm not talking about a high level
OS like Linux here but secure firmware already verified/certified that
cannot be easily changed. Such software runs at secure EL1 and, at least
for enabling the MMU, needs to run in a 32-bit physical address space
(hence the low DRAM alias). As for 32-bit peripherals, if present, they
may be behind an SMMU but it depends on whether the secure firmware is
able to configure it (and the feedback I got so far was "no"). That's
additional work which your partners may not be prepared to do.

If you don't have such 32-bit firmware requirements and any 32-bit
peripherals can always be routed via an SMMU, the memory maps
recommendations can be followed *without* the low aliases. For example,
4.3 "40-bit Memory Map" allows up to 512GB RAM at 512G phys addr
(usually chip select requirements) with the range between phys address
512G to 544G being optionally "hole or DRAM". That's where you can
choose DRAM and avoid the low aliases (at 2GB and 32GB). This would fit
well with the hardware chip selects for DRAM and it won't create a
sparse memory range for Linux.

(BTW, arm64 defconfig now enables the 48-bit VA option)

-- 
Catalin

^ permalink raw reply	[flat|nested] 7+ messages in thread

* ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA by default)
  2016-06-14  8:51     ` Ard Biesheuvel
  2016-06-14 10:49       ` Catalin Marinas
@ 2016-06-14 11:04       ` Arnd Bergmann
  2016-06-14 13:55       ` Stuart Yoder
  2 siblings, 0 replies; 7+ messages in thread
From: Arnd Bergmann @ 2016-06-14 11:04 UTC (permalink / raw)
  To: linux-arm-kernel

On Tuesday, June 14, 2016 10:51:57 AM CEST Ard Biesheuvel wrote:
> > I guess how definitive someone might read it is open to interpretation
> > but our hardware guys intepreted it as the "ARM recommendation".
> > There are lot's of "must" statements in there, so its easy for me to
> > see how it can be intepreted to be fairly definitive.  It doesn't
> > say a 64GB DRAM part _could_ be sub-divided like...  It says "For example
> > a 64GB DRAM part _will_ be sub-divided into three regions."
> >
> 
> Whether or not you interpret the tone as normative is only one side of
> it. The reason I quoted the paragraph above is to emphasize that the
> definition of an 'ARM system' is not clear either, i.e., whether it
> means 'any system that implements the ARM architecture' or simply 'a
> system created by ARM Ltd.'. The fact that the paragraph mentions
> software models and FPGA implementations leans to the latter IMO.

Agreed.

> > In any case, we are not going to follow that document as definitive.
> > But, my question is whether there are any updated ARM recommendations?
> 
> That again suggests that the document in question is intended as a
> general recommendation, which I believe may not be the case. So if we
> could get some clarification on that as well from any of the ARM Ltd.
> cc'ees, that would be great.

The other question is to which degree we want to support this in Linux,
e.g. whether it's ok if we only run on systems with such extremely
sparsely populated DRAM areas at the same performance as on the
normal memory maps, or in the default configuration at all.

Do we know of there are any ARM reference systems that even support
more than 32GB of RAM?

	Arnd

^ permalink raw reply	[flat|nested] 7+ messages in thread

* ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA by default)
  2016-06-14  8:51     ` Ard Biesheuvel
  2016-06-14 10:49       ` Catalin Marinas
  2016-06-14 11:04       ` Arnd Bergmann
@ 2016-06-14 13:55       ` Stuart Yoder
  2 siblings, 0 replies; 7+ messages in thread
From: Stuart Yoder @ 2016-06-14 13:55 UTC (permalink / raw)
  To: linux-arm-kernel


> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> Sent: Tuesday, June 14, 2016 3:52 AM
> To: Stuart Yoder <stuart.yoder@nxp.com>
> Cc: Catalin Marinas <catalin.marinas@arm.com>; Mark Rutland <mark.rutland@arm.com>; Marc Zyngier
> <marc.zyngier@arm.com>; Will Deacon <Will.Deacon@arm.com>; Peter Newton <Peter.Newton@freescale.com>;
> linux-arm-kernel at lists.infradead.org
> Subject: Re: ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA
> by default)
> 
> On 13 June 2016 at 20:41, Stuart Yoder <stuart.yoder@nxp.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> >> Sent: Saturday, June 11, 2016 6:38 AM
> >> To: Stuart Yoder <stuart.yoder@nxp.com>
> >> Cc: Catalin Marinas <catalin.marinas@arm.com>; Mark Rutland <mark.rutland@arm.com>; Marc Zyngier
> >> <marc.zyngier@arm.com>; Will Deacon <Will.Deacon@arm.com>; Peter Newton <Peter.Newton@freescale.com>;
> >> linux-arm-kernel at lists.infradead.org
> >> Subject: Re: ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit
> VA
> >> by default)
> >>
> >> On 10 June 2016 at 18:34, Stuart Yoder <stuart.yoder@nxp.com> wrote:
> >> >
> >> >> -----Original Message-----
> >> >> From: Catalin Marinas [mailto:catalin.marinas at arm.com]
> >> >> Sent: Friday, July 31, 2015 8:23 AM
> >> >> To: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> >> >> Cc: Mark Rutland <mark.rutland@arm.com>; Marc Zyngier <marc.zyngier@arm.com>; Will Deacon
> >> >> <Will.Deacon@arm.com>; Stuart Yoder <stuart.yoder@freescale.com>; Peter Newton
> >> >> <Peter.Newton@freescale.com>; linux-arm-kernel at lists.infradead.org
> >> >> Subject: Re: [RFC] arm64: defconfig: enable 48-bit VA by default
> >> >>
> >> >> On Fri, Jul 31, 2015 at 03:10:39PM +0200, Ard Biesheuvel wrote:
> >> >> > On 31 July 2015 at 14:53, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> >> > > On Thu, Jul 30, 2015 at 09:27:03PM +0200, Ard Biesheuvel wrote:
> >> >> > >> On 30 July 2015 at 12:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> >> > >> > On Wed, Jul 29, 2015 at 08:49:57PM +0000, Stuart Yoder wrote:
> >> >> > >> >> > From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> >> >> > >> [...]
> >> >> > >> >> > To be honest, I think this is poorly designed, and I am not sure we
> >> >> > >> >> > should cater for such configurations in the defconfig.
> >> >> > >> >>
> >> >> > >> >> Agree, if this is a one-off weird platform then we shouldn't.
> >> >> > >> >>
> >> >> > >> >> But, the 'Principles of ARM Memory Maps' doc proposes this:
> >> >> > >> >> 2 GB at 0x8000_0000
> >> >> > >> >> 30 GB at 0x8_8000_0000
> >> >> > >> >> 480 GB at 0x88_0000_0000
> >> >> > >> >
> >> >> > >> > I'm not particularly recommending this layout, at least not without some
> >> >> > >> > clarifications on DRAM aliases (I'll ping people internally about it
> >> >> > >> > again). The original layout pre-dates ARMv8, it was meant for ARMv7/LPAE
> >> >> > >> > and all the memory beyond 32-bit was highmem anyway. It was later
> >> >> > >> > updated for AArch64 but only to allow 44/48-bit PA (a few sections
> >> >> > >> > added).
> >> >> > >>
> >> >> > >> As an aside, is there any reason why the direct mapping *must* be a
> >> >> > >> linear mapping?
> >> >> > >> Other than the performance concerns regarding
> >> >> > >> phys_to_virt/virt_to_phys, I mean?
> >> >> > >
> >> >> > > Mostly performance concerns. You could compact the physical range into a
> >> >> > > smaller virtual one but the conversion will be costly, especially if you
> >> >> > > want to make it multi-platform (having to look-up memory ranges,
> >> >> > > memblock offsets). This would affect page table entry setup, code that
> >> >> > > requires a page structure (like virt_to_page) and anything else doing
> >> >> > > the virt/phys conversion.
> >> >> > >
> >> >> > > I tried something like that for RealView PBX in the past but it was
> >> >> > > hard-coded (no multi-platform at the time). See
> >> >> > > arch/arm/mach-realview/include/mach/memory.h.
> >> >> >
> >> >> > Yes, that looks vaguely like what I had in mind.
> >> >> >
> >> >> > IOW, we could partition the direct mapping just like the ARM
> >> >> > recommendation, i.e.,
> >> >> >
> >> >> > 0 - 2 GB
> >> >> > 2 - 32 GB
> >> >> > 32 - 512 GB
> >> >> >
> >> >> > but default to 1:1 correspondence, so that
> >> >> >
> >> >> > PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> >> >> > PHYS_OFFSET1 = memstart_addr + 2 GB
> >> >> > PHYS_OFFSET2 = memstart_addr + 32 GB
> >> >> >
> >> >> > and only if the ARM recommended physical memory map is detected (with
> >> >> > memstart_addr @ 0x8000_0000), switch to
> >> >> >
> >> >> > PHYS_OFFSET = PHYS_OFFSET0 = memstart_addr
> >> >> > PHYS_OFFSET1 = memstart_addr + 30 GB
> >> >> > PHYS_OFFSET2 = memstart_addr + 480 GB
> >> >>
> >> >> I don't really like such complexity when all you need on arm64 is to
> >> >> enable 48-bit VA (though it would be interesting to benchmark it).
> >> >>
> >> >> > I guess such a special case would be out of the question for one-off
> >> >> > crazy designs like Freescale, but since this is the layout recommended
> >> >> > by ARM itself, I suppose we could try and support it a bit better.
> >> >>
> >> >> I'm trying to get the layout fixed before it spreads any further ;).
> >> >
> >> > Hi Catalin, in this very old thread there was the intent on your side
> >> > to revisit the physical address layout recommended in the 'Principles of ARM
> >> > Memory Maps' whitepaper.  Has anything happened or changed in the
> >> > last ~1 year.  Does ARM have recommendations?  SBSA doesn't
> >> > mandate anything as far as I can see.
> >> >
> >> > We have the opportunity to influences some future designs and wanted to
> >> > see if ARM has recommendations.  In particular, what is the status of
> >> > the recommendations in 'Principles of ARM Memory Maps'?
> >> >
> >>
> >> Frankly, I don't understand why anyone would use this document as a
> >> definitive recommendation for how to lay out the physical address
> >> space of a new SOC. From the doc itself:
> >>
> >> """
> >> ARM creates a variety of development systems to support A-class cortex
> >> CPUs, ranging from cycle accurate RTL models, to fast software models,
> >> onto FPGAs and full custom SoCs. ARM has been harmonizing the memory
> >> maps in these systems to provide internal consistency and software
> >> portability, and to address the constraints that come with mixing
> >> 32-bit components within larger address spaces.
> >> """
> >>
> >> IOW, the document explains why ARM systems are configured the way they
> >> are, taking into consideration that they need to serve as references
> >> for a variety of hardware and software development, using both 32-bit
> >> and 64-bit components. The fact that neither the SBSA nor the document
> >> itself present it as a recommendation for server platforms means that
> >> it should not be mistaken for that.
> >
> > I guess how definitive someone might read it is open to interpretation
> > but our hardware guys intepreted it as the "ARM recommendation".
> > There are lot's of "must" statements in there, so its easy for me to
> > see how it can be intepreted to be fairly definitive.  It doesn't
> > say a 64GB DRAM part _could_ be sub-divided like...  It says "For example
> > a 64GB DRAM part _will_ be sub-divided into three regions."
> >
> 
> Whether or not you interpret the tone as normative is only one side of
> it. The reason I quoted the paragraph above is to emphasize that the
> definition of an 'ARM system' is not clear either, i.e., whether it
> means 'any system that implements the ARM architecture' or simply 'a
> system created by ARM Ltd.'. The fact that the paragraph mentions
> software models and FPGA implementations leans to the latter IMO.
> 
> > In any case, we are not going to follow that document as definitive.
> > But, my question is whether there are any updated ARM recommendations?
> 
> That again suggests that the document in question is intended as a
> general recommendation, which I believe may not be the case. So if we
> could get some clarification on that as well from any of the ARM Ltd.
> cc'ees, that would be great.

One specific definitive guideline should be something related to
normal memory and systems using 39-bit VA.  If the Linux kernel cannot
create a mapping to span all your DDR you are kind of screwed.  IIUC, Ubuntu
and Debian use 39-bit VAs and fall into this case.

But, is that the only guideline?

Thanks,
Stuart

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2016-06-14 13:55 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-10 16:34 ARM physical memory map recommendation? (was RE: [RFC] arm64: defconfig: enable 48-bit VA by default) Stuart Yoder
2016-06-11 11:38 ` Ard Biesheuvel
2016-06-13 18:41   ` Stuart Yoder
2016-06-14  8:51     ` Ard Biesheuvel
2016-06-14 10:49       ` Catalin Marinas
2016-06-14 11:04       ` Arnd Bergmann
2016-06-14 13:55       ` Stuart Yoder

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.