Re: [Xen-devel] [PATCH] x86/flush: use APIC ALLBUT destination shorthand when possible

From: Andrew Cooper <andrew.cooper3@citrix.com>
To: Jan Beulich <jbeulich@suse.com>, Roger Pau Monne <roger.pau@citrix.com>
Cc: xen-devel@lists.xenproject.org, Wei Liu <wl@xen.org>
Subject: Re: [Xen-devel] [PATCH] x86/flush: use APIC ALLBUT destination shorthand when possible
Date: Fri, 3 Jan 2020 12:24:45 +0000	[thread overview]
Message-ID: <901251c1-eb1e-b32c-6c3e-d09e3a3e4810@citrix.com> (raw)
In-Reply-To: <2215c6fe-4492-4494-af26-6754372d4c95@suse.com>

On 03/01/2020 12:08, Jan Beulich wrote:
> On 24.12.2019 13:44, Roger Pau Monne wrote:
>> @@ -227,14 +233,47 @@ void flush_area_mask(const cpumask_t *mask, const void *va, unsigned int flags)
>>      if ( (flags & ~FLUSH_ORDER_MASK) &&
>>           !cpumask_subset(mask, cpumask_of(cpu)) )
>>      {
>> +        bool cpus_locked = false;
>> +
>>          spin_lock(&flush_lock);
>>          cpumask_and(&flush_cpumask, mask, &cpu_online_map);
>>          cpumask_clear_cpu(cpu, &flush_cpumask);
>>          flush_va      = va;
>>          flush_flags   = flags;
>> -        send_IPI_mask(&flush_cpumask, INVALIDATE_TLB_VECTOR);
>> +
>> +        /*
>> +         * Prevent any CPU hot{un}plug while sending the IPIs if we are to use
>> +         * a shorthand, also refuse to use a shorthand if not all CPUs are
>> +         * online or have been parked.
>> +         */
>> +        if ( system_state > SYS_STATE_smp_boot && !cpu_overflow &&
>> +             (cpus_locked = get_cpu_maps()) &&
>> +             (park_offline_cpus ||
> Why is it relevant whether we park offline CPUs, or whether we've
> even brought up all of the ones a system has? An IPI, in particular
> a broadcast one, shouldn't have any issue getting delivered if some
> of the nominal recipients don't listen, should it?

TGLX had a hard time time making shorthands work correctly on Linux. 
ISTR target CPUs in wait-for-SIPI may cause the source side of the IPI
to hang indefinitely, or yield an APIC Send error.

Linux maintains a bitmap of "CPUs which have booted once and we know can
safely handle IPIs", and doesn't permit any broadcast shorthands until
this matches the cpus_available mask.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel