Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated space

[Oleksandr <olekstysh@gmail.com>]

On 09.08.21 23:45, Julien Grall wrote:


Hi Julien

>
>
> On 09/08/2021 19:24, Oleksandr wrote:
>>
>> On 09.08.21 18:42, Julien Grall wrote:
>>> Hi Oleksandr,
>>
>>
>> Hi Julien.
>>
>>
>> Thank you for the input.
>>
>>
>>>
>>> On 07/08/2021 18:03, Oleksandr wrote:
>>>>
>>>> On 06.08.21 03:30, Stefano Stabellini wrote:
>>>>
>>>> Hi Stefano
>>>>
>>>>> On Wed, 4 Aug 2021, Julien Grall wrote:
>>>>>>> +#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* 
>>>>>>> 128GB */
>>>>>>> +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000)
>>>>>>>
>>>>>>> While the possible new DT bindings has not been agreed yet, I 
>>>>>>> re-used
>>>>>>> existing "reg" property under the hypervisor node to pass safe 
>>>>>>> range as a
>>>>>>> second region,
>>>>>>> https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10: 
>>>>>>>
>>>>>> So a single region works for a guest today, but for dom0 we will 
>>>>>> need multiple
>>>>>> regions because it is may be difficult to find enough contiguous 
>>>>>> space for a
>>>>>> single region.
>>>>>>
>>>>>> That said, as dom0 is mapped 1:1 (including some guest mapping), 
>>>>>> there is also
>>>>>> the question where to allocate the safe region. For grant table, 
>>>>>> we so far
>>>>>> re-use the Xen address space because it is assumed it will space 
>>>>>> will always
>>>>>> be bigger than the grant table.
>>>>>>
>>>>>> I am not sure yet where we could allocate the safe regions. 
>>>>>> Stefano, do you
>>>>>> have any ideas?
>>>>> The safest choice would be the address range corresponding to memory
>>>>> (/memory) not already allocated to Dom0.
>>>>>
>>>>> For instance from my last boot logs:
>>>>> (XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
>>>>> (XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB)
>>>>> (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)
>>>>>
>>>>> All the other ranges could be given as unallocated space:
>>>>>
>>>>> - 0x0 - 0x10000000
>>>>> - 0x70000000 - 0x78000000
>>>>> - 0x8_0000_0000 - 0x8_8000_0000
>>>>
>>>> Thank you for the ideas.
>>>>
>>>> If I got the idea correctly, yes, as these ranges represent the 
>>>> real RAM, so no I/O would be in conflict with them and as the 
>>>> result - no overlaps would be expected.
>>>> But, I wonder, would this work if we have IOMMU enabled for Dom0 
>>>> and need to establish 1:1 mapping for the DMA devices to work with 
>>>> grant mappings...
>>>> In arm_iommu_map_page() we call guest_physmap_add_entry() with gfn 
>>>> = mfn, so the question is could we end up with this new gfn 
>>>> replacing the valid mapping
>>>> (with gfn allocated from the safe region)?
>>>
>>> Right, when we enable the IOMMU for dom0, Xen will add an extra 
>>> mapping with GFN == MFN for foreign and grant pages. This is because 
>>> Linux is not aware that whether a device is protected by an IOMMU. 
>>> Therefore it is assuming it is not and will use the MFN to configure 
>>> for DMA transaction.
>>>
>>> We can't remove the mapping without significant changes in Linux and 
>>> Xen. I would not mandate them for this work.
>>>
>>> That said, I think it would be acceptable to have different way to 
>>> find the region depending on the dom0 configuration. So we could use 
>>> the RAM not used by dom0 when the IOMMU is turned off.
>>
>> OK
>>
>>
>>>
>>>>> The second best choice would be an hole: an address range not used by
>>>>> anybody else (no reg property) and also not even mappable by a bus 
>>>>> (not
>>>>> covered by a ranges property). This is not the best choice because 
>>>>> there
>>>>> can cases where physical resources appear afterwards.
>>>
>>> Are you saying that the original device-tree doesn't even describe 
>>> them in any way (i.e. reserved...)?
>>>
>>>>
>>>> Unfortunately, yes.
>>>
>>> So the decision where the safe region is located will be done by 
>>> Xen. There is no involvement of the domain (it will discover the 
>>> region from the DT). Therefore, I don't think we need to think about 
>>> everything right now as we could adapt this is exact region is not 
>>> part of the stable ABI.
>>>
>>> The hotplug is one I would defer because this is not supported (and 
>>> quite likely not working) in Xen upstream today.
>>
>> Sounds reasonable.
>>
>>
>>>
>>>
>>> Now regarding the case where dom0 is using the IOMMU. The assumption 
>>> is Xen will be able to figure out all the regions used from the 
>>> firmware table (ACPI or DT).
>>>
>>> AFAIK, this assumption would be correct for DT. However, for ACPI, I 
>>> remember we were not able to find all the MMIOs region in Xen (see 
>>> [1] and [2]). So even this solution would not work for ACPI.
>>>
>>> If I am not mistaken, we don't support IOMMU with ACPI yet. So we 
>>> could defer the problem to when this is going to be supported.
>>
>> Sounds reasonable.
>>
>>
>> To summarize:
>>
>> 0. Skip ACPI case for now, implement for DT case
>
> Just to be clear, I suggested to skip it when the IOMMU is enabled 
> with ACPI. We should still support the case without IOMMU. The 
> implementation would be the same as 2.

yes, sorry for not being precise


>
>
>>
>> 1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as 
>> safe ranges
>>
>> I would take into the account holes >= 1MB.
>
> May I ask why 1MB?

Nothing special, just thinking to not bother with small regions which 
would not be too useful overall, but could bloat resulting reg property.

Anyway, I would be ok with any sizes.


>
>> I am wondering, do we need a special alignment here other than a 
>> PAGE_SIZE?
>
> It needs to be 64KB aligned so a guest using 64KB pages can use it.

ok, sounds reasonable


>
>>
>> 2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to 
>> Dom0 as safe ranges
>>
>> We could even provide holes here as well.
>
> I would rather not. We likely need hack for the hotplug case. So I 
> want to keep them contained to IOMMU unless there is a strong reason 
> to do it.

ok, I got it


>
> Cheers,
>
-- 
Regards,

Oleksandr Tyshchenko