Xen Platform QoS design discussion

["Xu, Dongxiao" <dongxiao.xu@intel.com>]

Hello maintainers,

As PQoS feature has already been reviewed for 10 rounds, and also considering that more QoS related features are published in new SDM, like MBM and CQE (refer to chapter 17.14 and 17.15 of Intel SDM Volume 3). Before sending out another version of patch, I'd like to make sure we are aligned on some basic design issues, making the final implementation simple and performance efficient.

1) The hypercall to query QoS monitoring data, should be sysctl or domctl?
Previous QoS monitoring data query hypercall is designed with sysctl, which returns the whole QoS data in a 2-dimenstion array format for all domains. However users don't like to handle such 2-dimension array themselves, but preferring the hypercall API to be issued per-domain, by using domain id as the input parameter and returns the domain related QoS data. Here I propose to use the domctl style hypercall to get QoS data for specific domain. This has the advantage of simplifying the libxl QoS APIs for user-space developers, and also make the QoS memory allocation in Xen much easier.

2) How much memory needs to be allocated in Xen for QoS monitoring data?
There are a lot of comments complaining the previous memory allocation for QoS monitoring data. But if we adopt the domctl style of hypercall proposed in 1), then the needed memory is much smaller since the data structure becomes 1-dimension. We only need to allocate "nr_sockets" related size in initialization time to present the domain's QoS monitoring data. Furthermore, no additional QoS resource allocation logic is needed when CPU online/offline.

3) Copy or share for QoS monitoring data?
In previous patches, sharing mechanism is used for passing data between Xen and dom0 toolstack. However when MBM feature comes out, we may need to re-consider this solution.
CQM and MBM features belong to L3 monitoring category, and they share the same CPUID enumeration method for its QoS data structure, which is 64bit width. From data structure design point of view, all L3 monitoring category features (CQM, MBM and others in L3 category) should better to share the following structure:

struct socket_l3 {
    unsigned int socket_id;
    unsigned int l3_type; // CQM or MBM or others
    unsigned long qm_data_l3;
}

In hypercall, user sets the "l3_type" as an input parameter (CQM or MBM or others in L3 category), and Xen returns the qm_data_l3 as the monitoring data for that certain feature.
In this case, sharing doesn't work since data may be collapsed if there are two xl instances running simultaneously to separately get CQM and MBM data. Besides, if we use the per-domain hypercall (domctl instead of sysctl), the data amount is very small for every hypercall. (saying 4 sockets, the data amount is 4*sizeof(struct socket_l3) = 64 bytes per hypercall). Even for data sharing mechanism, we need to pass the page address as the parameter, which also requires 8 or more bytes. Therefore in my opinion, data copy is acceptable in PQoS case.

Any feedback is welcome!

Thanks,
Dongxiao