Linux-ARM-Kernel Archive on lore.kernel.org
 help / color / Atom feed
From: Mohamed Mediouni <mohamed.mediouni@caramail.com>
To: Will Deacon <will@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>,
	Marc Zyngier <maz@kernel.org>, Hector Martin <marcan@marcan.st>,
	linux-kernel@vger.kernel.org,
	Catalin Marinas <catalin.marinas@arm.com>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>,
	Stan Skowronek <stan@corellium.com>
Subject: Re: [RFC PATCH 3/7] arm64: mm: use nGnRnE instead of nGnRE on Apple processors
Date: Thu, 21 Jan 2021 16:12:04 +0100
Message-ID: <39067FDD-F7D3-4F37-8B43-1A9FCF80EEE5@caramail.com> (raw)
In-Reply-To: <20210121124742.GA22123@willie-the-truck>



> On 21 Jan 2021, at 13:47, Will Deacon <will@kernel.org> wrote:
> 
> On Thu, Jan 21, 2021 at 11:44:23AM +0000, Marc Zyngier wrote:
>> On 2021-01-21 11:27, Will Deacon wrote:
>>> On Wed, Jan 20, 2021 at 02:27:13PM +0100, Mohamed Mediouni wrote:
>>>> Use nGnRnE instead of nGnRE on Apple SoCs to workaround a serious
>>>> hardware quirk.
>>>> 
>>>> On Apple processors, writes using the nGnRE device memory type get
>>>> dropped in flight,
>>>> getting to nowhere.
>>>> 
>>>> Signed-off-by: Stan Skowronek <stan@corellium.com>
>>>> Signed-off-by: Mohamed Mediouni <mohamed.mediouni@caramail.com>
>>>> ---
>>>> arch/arm64/mm/proc.S | 26 ++++++++++++++++++++++++++
>>>> 1 file changed, 26 insertions(+)
>>>> 
>>>> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
>>>> index 1f7ee8c8b7b8..06436916f137 100644
>>>> --- a/arch/arm64/mm/proc.S
>>>> +++ b/arch/arm64/mm/proc.S
>>>> @@ -51,6 +51,25 @@
>>>> #define TCR_KASAN_HW_FLAGS 0
>>>> #endif
>>>> 
>>>> +#ifdef CONFIG_ARCH_APPLE
>>>> +
>>>> +/*
>>>> + * Apple cores appear to black-hole writes done with nGnRE.
>>>> + * We settled on a work-around that uses MAIR vs changing every
>>>> single user of
>>>> + * nGnRE across the arm64 code.
>>>> + */
>>>> +
>>>> +#define MAIR_EL1_SET_APPLE						\
>>>> +	(MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRnE, MT_DEVICE_nGnRnE) |	\
>>>> +	 MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRnE, MT_DEVICE_nGnRE) |	\
>>>> +	 MAIR_ATTRIDX(MAIR_ATTR_DEVICE_GRE, MT_DEVICE_GRE) |		\
>>>> +	 MAIR_ATTRIDX(MAIR_ATTR_NORMAL_NC, MT_NORMAL_NC) |		\
>>>> +	 MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL) |			\
>>>> +	 MAIR_ATTRIDX(MAIR_ATTR_NORMAL_WT, MT_NORMAL_WT) |		\
>>>> +	 MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL_TAGGED))
>>>> +
>>>> +#endif
>>>> +
>>>> /*
>>>>  * Default MAIR_EL1. MT_NORMAL_TAGGED is initially mapped as Normal
>>>> memory and
>>>>  * changed during __cpu_setup to Normal Tagged if the system
>>>> supports MTE.
>>>> @@ -432,6 +451,13 @@ SYM_FUNC_START(__cpu_setup)
>>>> 	 * Memory region attributes
>>>> 	 */
>>>> 	mov_q	x5, MAIR_EL1_SET
>>>> +#ifdef CONFIG_ARCH_APPLE
>>>> +	mrs	x0, MIDR_EL1
>>>> +	lsr	w0, w0, #24
>>>> +	mov_q	x1, MAIR_EL1_SET_APPLE
>>>> +	cmp	x0, #0x61			// 0x61 = Implementer: Apple
>>>> +	csel	x5, x1, x5, eq
>>> 
>>> Why does this need to be done so early? It would be a lot cleaner if we
>>> could detect this in a similar fashion to other errata and update the
>>> MAIR
>>> appropriately. If that's not possible because of early IO mappings
>>> (which
>>> ones?), then we could instead initialise to nGnRnE unconditionally, but
>>> relax it to nGnRE if we detect that we _don't_ have the erratum.
>> 
>> Would that imply another round-trip into the idmap, much like we do
>> when we switch to non-global mappings? Or do you expect that we can change
>> the MAIR with live mappings?
> 
> I think we should be able to change it live and then invalidate the TLB. At
> least, my reading of the BBM requirements suggests that it isn't required
> for changing between different types of device memory. I can seek
> clarification from Arm if necessary.
> 
Please ignore that patch.

It turns out that the PCIe controller on Apple M1 expects posted writes and so the memory range for it ought to be set nGnRE. 
So, we need to use nGnRnE for on-chip MMIO and nGnRE for PCIe BARs.

The MAIR approach isn’t adequate for such a thing, so we’ll have to look elsewhere.

Thank you,
> Will


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

  reply index

Thread overview: 55+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-20 13:27 [RFC PATCH 0/7] Linux on Apple Silicon Mohamed Mediouni
2021-01-20 13:27 ` [RFC PATCH 1/7] arm64: kernel: FIQ support Mohamed Mediouni
2021-01-20 13:27 ` [RFC PATCH 2/7] arm64: kernel: Add a WFI hook Mohamed Mediouni
2021-01-20 16:46   ` Alexander Graf
     [not found]     ` <94C20F55-D3B8-4349-B26F-9EA8AAEBF639@caramail.com>
2021-01-21 12:33       ` Hector Martin 'marcan'
2021-01-21 10:52   ` Arnd Bergmann
2021-01-21 11:01     ` Mohamed Mediouni
2021-01-21 11:36       ` Arnd Bergmann
2021-01-20 13:27 ` [RFC PATCH 3/7] arm64: mm: use nGnRnE instead of nGnRE on Apple processors Mohamed Mediouni
2021-01-20 16:47   ` Alexander Graf
2021-01-20 18:06     ` Mohamed Mediouni
2021-01-20 18:10       ` Alexander Graf
2021-01-21 11:27   ` Will Deacon
2021-01-21 11:38     ` Arnd Bergmann
2021-01-21 11:44     ` Marc Zyngier
2021-01-21 12:47       ` Will Deacon
2021-01-21 15:12         ` Mohamed Mediouni [this message]
2021-01-21 16:25           ` Marc Zyngier
2021-01-21 17:55             ` Will Deacon
2021-01-21 18:15               ` Marc Zyngier
2021-01-21 18:22                 ` Mohamed Mediouni
2021-01-21 18:22                 ` Will Deacon
2021-01-20 13:27 ` [RFC PATCH 4/7] irqchip/apple-aic: Add support for Apple AIC Mohamed Mediouni
2021-01-20 17:11   ` Alexander Graf
2021-01-20 18:04     ` Mohamed Mediouni
2021-01-20 20:16       ` Andrew Lunn
2021-01-20 21:18   ` Stan Skowronek
2021-01-21  9:48   ` Linus Walleij
2021-01-21 10:37     ` Arnd Bergmann
2021-01-21 15:29       ` Hector Martin 'marcan'
2021-01-21 17:09         ` Rob Herring
2021-01-21 17:45       ` Rob Herring
2021-01-21 16:44     ` Rob Herring
2021-01-21 16:53   ` Hector Martin 'marcan'
2021-01-20 13:27 ` [RFC PATCH 5/7] arm64/Kconfig: Add Apple Silicon SoC platform Mohamed Mediouni
2021-01-20 13:27 ` [RFC PATCH 6/7] arm64: kernel: Apple CPU start driver Mohamed Mediouni
2021-01-21 11:14   ` Arnd Bergmann
2021-01-20 13:27 ` [RFC PATCH 7/7] irqchip/apple-aic: add SMP support to the Apple AIC driver Mohamed Mediouni
2021-01-21 12:44   ` Arnd Bergmann
2021-01-21 12:50     ` Mohamed Mediouni
2021-01-21 13:00       ` Arnd Bergmann
2021-01-21 13:01         ` Hector Martin 'marcan'
2021-01-21 13:22       ` Marc Zyngier
2021-01-21 13:32         ` Mark Rutland
2021-01-21 14:05           ` Marc Zyngier
2021-01-21 13:34         ` Mohamed Mediouni
2021-01-21 14:10           ` Marc Zyngier
2021-01-21 15:09             ` Arnd Bergmann
2021-01-21 15:18               ` Mohamed Mediouni
2021-01-21 16:40       ` Rob Herring
2021-01-21 16:43         ` Mohamed Mediouni
2021-01-21 17:37           ` Rob Herring
2021-01-21 18:08             ` Mohamed Mediouni
2021-01-21 18:57               ` Rob Herring
2021-02-02 19:15   ` Linus Walleij

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=39067FDD-F7D3-4F37-8B43-1A9FCF80EEE5@caramail.com \
    --to=mohamed.mediouni@caramail.com \
    --cc=catalin.marinas@arm.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=marcan@marcan.st \
    --cc=mark.rutland@arm.com \
    --cc=maz@kernel.org \
    --cc=stan@corellium.com \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-ARM-Kernel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-arm-kernel/0 linux-arm-kernel/git/0.git
	git clone --mirror https://lore.kernel.org/linux-arm-kernel/1 linux-arm-kernel/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-arm-kernel linux-arm-kernel/ https://lore.kernel.org/linux-arm-kernel \
		linux-arm-kernel@lists.infradead.org
	public-inbox-index linux-arm-kernel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.infradead.lists.linux-arm-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git