Re: [RFC 3/4] Add xen superpage splitting support to arm

From: Julien Grall <julien@xen.org>
To: Demi Marie Obenour <demi@invisiblethingslab.com>,
	"Smith, Jackson" <rsmith@riversideresearch.org>
Cc: "Brookes, Scott" <sbrookes@riversideresearch.org>,
	Xen-devel <xen-devel@lists.xenproject.org>,
	"Stefano Stabellini" <sstabellini@kernel.org>,
	"bertrand.marquis@arm.com" <bertrand.marquis@arm.com>,
	"jbeulich@suse.com" <jbeulich@suse.com>,
	"Andrew Cooper" <andrew.cooper3@citrix.com>,
	"Roger Pau Monné" <roger.pau@citrix.com>,
	"George Dunlap" <george.dunlap@citrix.com>,
	"Daniel P. Smith" <dpsmith@apertussolutions.com>,
	"christopher.w.clark@gmail.com" <christopher.w.clark@gmail.com>
Subject: Re: [RFC 3/4] Add xen superpage splitting support to arm
Date: Wed, 14 Dec 2022 09:09:58 +0000	[thread overview]
Message-ID: <ce49d2e8-294a-18d0-c094-dd352dca1936@xen.org> (raw)
In-Reply-To: <Y5kpFMp38Yg7If/Y@itl-email>

Hi Demi,

On 14/12/2022 01:38, Demi Marie Obenour wrote:
> On Tue, Dec 13, 2022 at 11:07:55PM +0000, Julien Grall wrote:
>> Hi Demi,
>>
>> On 13/12/2022 22:17, Demi Marie Obenour wrote:
>>> On Tue, Dec 13, 2022 at 09:15:49PM +0000, Julien Grall wrote:
> 
> [snip]
> 
>>>>> +
>>>>> +    /*
>>>>> +     * Generate the entry for this new table we created,
>>>>> +     * and write it back in place of the superpage entry.
>>>>> +     */
>>>>
>>>> I am afraid this is not compliant with the Arm Arm. If you want to update
>>>> valid entry (e.g. shattering a superpage), then you need to follow the
>>>> break-before-make sequence. This means that:
>>>>     1. Replace the valid entry with an entry with an invalid one
>>>>     2. Flush the TLBs
>>>>     3. Write the new entry
>>>>
>>>> Those steps will make your code compliant but it also means that a virtual
>>>> address will be temporarily invalid so you could take a fault in the middle
>>>> of your split if your stack or the table was part of the region. The same
>>>> could happen for the other running CPUs but this is less problematic as they
>>>> could spin on the page-table lock.
>>>
>>> Could this be worked around by writing the critical section in
>>> assembler?
>>
>> Everything is feasible. Is this worth it? I don't think so. There are way we
>> can avoid the shattering at first by simply not mapping all the RAM.
> 
> Good point.  I do wonder what would go wrong if one replaced one live
> PTE with another that pointed to the same physical address. 

It depends what you are modifying the PTE. If you only modify the 
permissions, then that's fine. But anything else could result to TLB 
conflict, loss of coherency...

> Is this
> merely a case of “spec doesn’t allow it”, or does it actually break on
> real hardware?
I have seen issues on real HW if the ordering is not respected. Recent 
version of the Arm Arm introduced the possibility to skip the sequence 
in certain conditions and if the HW supports it (reported via the ID 
registers).

I haven't yet seen such processor.

Cheers,

-- 
Julien Grall