Re: [PATCH v4] VMX: use a single, global APIC access page

[Jan Beulich <jbeulich@suse.com>]

On 12.04.2021 17:31, Roger Pau Monné wrote:
> On Mon, Apr 12, 2021 at 12:40:48PM +0200, Jan Beulich wrote:
>> The address of this page is used by the CPU only to recognize when to
>> access the virtual APIC page instead. No accesses would ever go to this
>> page. It only needs to be present in the (CPU) page tables so that
>> address translation will produce its address as result for respective
>> accesses.
>>
>> By making this page global, we also eliminate the need to refcount it,
>> or to assign it to any domain in the first place.
>>
>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
>> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
>> ---
>> v4: Set PGC_extra on the page. Make shadow mode work.
>> v3: Split p2m insertion change to a separate patch.
>> v2: Avoid insertion when !has_vlapic(). Split off change to
>>     p2m_get_iommu_flags().
>> ---
>> I did further consider not allocating any real page at all, but just
>> using the address of some unpopulated space (which would require
>> announcing this page as reserved to Dom0, so it wouldn't put any PCI
>> MMIO BARs there). But I thought this would be too controversial, because
>> of the possible risks associated with this.
> 
> Really seems more trouble than reward. Also there are systems with
> MMIO regions in holes on the memory map, like the issue I had with the
> Intel pinctrl stuff that had an MMIO region in a hole on the memory
> map [0], so I'm not sure Xen would be in a position to select a
> suitable unpopulated page anyway.
> 
> [0] https://lore.kernel.org/xen-devel/YFx80wYt%2FKcHanC7@smile.fi.intel.com/

Yeah, I had seen that. What I'm having trouble to understand is how the
OS will know to avoid that range for e.g. placing BARs.

>> @@ -411,28 +411,22 @@ static int vmx_domain_initialise(struct
>>       */
>>      d->arch.hvm.vmx.exec_sp = is_hardware_domain(d) || opt_ept_exec_sp;
>>  
>> -    if ( !has_vlapic(d) )
>> -        return 0;
>> -
>> -    if ( (rc = vmx_alloc_vlapic_mapping(d)) != 0 )
>> -        return rc;
>> -
>>      return 0;
>>  }
>>  
>> -static void vmx_domain_relinquish_resources(struct domain *d)
>> +static void domain_creation_finished(struct domain *d)
>>  {
>> -    if ( !has_vlapic(d) )
>> +    gfn_t gfn = gaddr_to_gfn(APIC_DEFAULT_PHYS_BASE);
> 
> Worth making it const static?

The compiler ought to be able to fold this into a suitable constant
at the use site. Definitely not static imo, and I see little point
in making a local variable const, unless one really wants to
document something very special.

>> +    uint8_t ipat;
>> +
>> +    if ( !has_vlapic(d) || mfn_eq(apic_access_mfn, _mfn(0)) )
> 
> It would be better to use INVALID_MFN here, and init apic_access_mfn
> to that value.

Oh, yes, that's easier possible now that the variable is static.

>> --- a/xen/arch/x86/mm/shadow/set.c
>> +++ b/xen/arch/x86/mm/shadow/set.c
>> @@ -94,6 +94,22 @@ shadow_get_page_from_l1e(shadow_l1e_t sl
>>      ASSERT(!sh_l1e_is_magic(sl1e));
>>      ASSERT(shadow_mode_refcounts(d));
>>  
>> +    /*
>> +     * VMX'es APIC access MFN is just a surrogate page.  It doesn't actually
>> +     * get accessed, and hence there's no need to refcount it (and refcounting
>> +     * would fail, due to the page having no owner).
>> +     */
>> +    if ( mfn_valid(mfn = shadow_l1e_get_mfn(sl1e)) )
> 
> I find this assignment inside the parameter list quite ugly, I would
> rather split it on it's own line.

Well, okay. To be honest I'm not even sure why I did it this way, as
I could have expected a respective comment.

>> +    {
>> +        const struct page_info *pg = mfn_to_page(mfn);
>> +
>> +        if ( !page_get_owner(pg) && (pg->count_info & PGC_extra) )
>> +        {
>> +            ASSERT(type == p2m_mmio_direct);
>> +            return 0;
> 
> Are there any other pages that could pass this check? I don't think
> so, but wanted to assert.

"Normal" extra pages have an owner, so no, there aren't any others.
If and when any appear, this may need further customizing, albeit
generally I'd hope further pages matching this pattern would also
want similar treatment.

Jan