On Tue, 2015-11-03 at 16:43 +0800, Feng Wu wrote:

> diff --git a/docs/misc/vtd-pi.txt b/docs/misc/vtd-pi.txt
> new file mode 100644
> index 0000000..f9b4637
> --- /dev/null
> +++ b/docs/misc/vtd-pi.txt

> +With VT-d Posted-interrupt we can get the following advantages:
> +- Direct delivery of external interrupts to running vCPUs without
> VMM
> +intervention
> +- Decrease the interrupt migration complexity. On vCPU migration,
> software
> +can atomically co-migrate all interrupts targeting the migrating
> vCPU. For
> +virtual machines with assigned devices, migrating a vCPU across
> pCPUs
> +either incurs the overhead of forwarding interrupts in software
> (e.g. via VMM
> +generated IPIs), or complexity to independently migrate each
> interrupt targeting
> +the vCPU to the new pCPU. However, after enabling VT-d PI, the
> destination vCPU
> +of an external interrupt from assigned devices is stored in the IRTE
> (i.e.
> +Posted-interrupt Descriptor Address), when vCPU is migrated to
> another pCPU,
> +we will set this new pCPU in the 'NDST' filed of Posted-interrupt

                                           ^field

> +More information about APICv and CPU-side Posted-interrupt, please
> refer
> +to Chapter 29, and Section 29.6 in the Intel SDM:
> +http://www.intel.com/content/dam/www/public/us/en/documents/manuals/
> 64-ia-32-architectures-software-developer-manual-325462.pdf
> +
ISTR Andrew said something about chapter names and numbers, and about
what to link in these cases, during v8 review.

> +- Update posted-interrupt descriptor during vCPU scheduling
> +
> +The basic idea here is:
> +1. When vCPU is running
> +        - Set 'NV' to 'posted_intr_vector'.
> +        - Clear 'SN' to accept posted-interrupts.
> +        - Set 'NDST' to the pCPU on which the vCPU will be running.
> +2. When vCPU is blocked
> +        - Set 'NV' to ' pi_wakeup_vector ', so we can wake up the
> +          related vCPU when posted-interrupt happens for it.
> +          Please refer to the above section about the new global
> vector.
> +        - Clear 'SN' to accept posted-interrupts
> +3. When vCPU is preempted or sleeping
> +        - Set 'SN' to suppress non-urgent interrupts
> +          (Currently, we only support non-urgent interrupts)
> +         When vCPU is preempted or sleep, it doesn't need to accept
> +         posted-interrupt notification event since we don't change
> the behavior
> +         of scheduler when the interrupt occurs, we still need wait
> for the next
> +         scheduling of the vCPU. When external interrupts from
> assigned devices occur,
> +         the interrupts are recorded in PIR, and will be synced to
> IRR before VM-Entry.
> +        - Set 'NV' to 'posted_intr_vector'.
> +
> +- How to wakeup blocked vCPU when an interrupt is posted for it
> (wakeup notification handler).
> +
>
> [...]
>
> +Here are what we do for the blocked vCPU:
> +1. Define a per-cpu list 'pi_blocked_vcpu', which stored the blocked
> +vCPU on the pCPU.
> +2. When the vCPU is going to block, insert the vCPU
> +to the per-cpu list belonging to the pCPU it was running.
> +3. When the vCPU is unblocked, remove the vCPU from the related pCPU
> list.
> +
> +In the handler of 'pi_wakeup_vector', we do:
> +1. Get the physical CPU.
> +2. Iterate the list 'pi_blocked_vcpu' of the current pCPU, if 'ON'
> is set,
> +we unblock the associated vCPU.
> +
This is indeed more general, and the fact that it does no longer
mentions RUNSTATEs, makes it more adherent to the actual implementation
(and hence more useful), so thanks for doing this.

Personally, I'd add a quick comment about how this, despite being
related to blocking and unblocking, happens actually inside VMX
handlers, i.e., (quickly) what is the relationship within these sets of
events (blocking/unblocking VMENTER/EXIT) and why it is ok to do things
like they are done.

I'm not too fussed about this, though. So, if others don't think
something like this is necessary, I'm fine with what you have here.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)