RE: [PATCH v5 1/7] docs/migration: add qpl compression feature

["Liu, Yuan1" <yuan1.liu@intel.com>]

> -----Original Message-----
> From: Peter Xu <peterx@redhat.com>
> Sent: Wednesday, March 27, 2024 1:58 AM
> To: Liu, Yuan1 <yuan1.liu@intel.com>
> Cc: farosas@suse.de; qemu-devel@nongnu.org; hao.xiang@bytedance.com;
> bryan.zhang@bytedance.com; Zou, Nanhai <nanhai.zou@intel.com>
> Subject: Re: [PATCH v5 1/7] docs/migration: add qpl compression feature
> 
> On Wed, Mar 20, 2024 at 12:45:21AM +0800, Yuan Liu wrote:
> > add Intel Query Processing Library (QPL) compression method
> > introduction
> >
> > Signed-off-by: Yuan Liu <yuan1.liu@intel.com>
> > Reviewed-by: Nanhai Zou <nanhai.zou@intel.com>
> > ---
> >  docs/devel/migration/features.rst        |   1 +
> >  docs/devel/migration/qpl-compression.rst | 231 +++++++++++++++++++++++
> >  2 files changed, 232 insertions(+)
> >  create mode 100644 docs/devel/migration/qpl-compression.rst
> >
> > diff --git a/docs/devel/migration/features.rst
> b/docs/devel/migration/features.rst
> > index d5ca7b86d5..bc98b65075 100644
> > --- a/docs/devel/migration/features.rst
> > +++ b/docs/devel/migration/features.rst
> > @@ -12,3 +12,4 @@ Migration has plenty of features to support different
> use cases.
> >     virtio
> >     mapped-ram
> >     CPR
> > +   qpl-compression
> > diff --git a/docs/devel/migration/qpl-compression.rst
> b/docs/devel/migration/qpl-compression.rst
> > new file mode 100644
> > index 0000000000..42c7969d30
> > --- /dev/null
> > +++ b/docs/devel/migration/qpl-compression.rst
> > @@ -0,0 +1,231 @@
> > +===============
> > +QPL Compression
> > +===============
> > +The Intel Query Processing Library (Intel ``QPL``) is an open-source
> library to
> > +provide compression and decompression features and it is based on
> deflate
> > +compression algorithm (RFC 1951).
> > +
> > +The ``QPL`` compression relies on Intel In-Memory Analytics
> Accelerator(``IAA``)
> > +and Shared Virtual Memory(``SVM``) technology, they are new features
> supported
> > +from Intel 4th Gen Intel Xeon Scalable processors, codenamed Sapphire
> Rapids
> > +processor(``SPR``).
> > +
> > +For more ``QPL`` introduction, please refer to:
> > +
> >
> +https://intel.github.io/qpl/documentation/introduction_docs/introduction.
> html
> 
> There're a bunch of links in this page, please consider switching all of
> them to use the link formats of .rST:
> 
>   Please refer to `QPL introduction page <https://...>`_.

Sure, thanks for the suggestion

> > +
> > +QPL Compression Framework
> > +=========================
> > +
> > +::
> > +
> > +  +----------------+       +------------------+
> > +  | MultiFD Service|       |accel-config tool |
> > +  +-------+--------+       +--------+---------+
> > +          |                         |
> > +          |                         |
> > +  +-------+--------+                | Setup IAA
> > +  |  QPL library   |                | Resources
> > +  +-------+---+----+                |
> > +          |   |                     |
> > +          |   +-------------+-------+
> > +          |   Open IAA      |
> > +          |   Devices +-----+-----+
> > +          |           |idxd driver|
> > +          |           +-----+-----+
> > +          |                 |
> > +          |                 |
> > +          |           +-----+-----+
> > +          +-----------+IAA Devices|
> > +      Submit jobs     +-----------+
> > +      via enqcmd
> > +
> > +
> > +Intel In-Memory Analytics Accelerator (Intel IAA) Introduction
> > +================================================================
> > +
> > +Intel ``IAA`` is an accelerator that has been designed to help benefit
> > +in-memory databases and analytic workloads. There are three main areas
> > +that Intel ``IAA`` can assist with analytics primitives (scan, filter,
> etc.),
> > +sparse data compression and memory tiering.
> > +
> > +``IAA`` Manual Documentation:
> > +
> > +https://www.intel.com/content/www/us/en/content-details/721858/intel-
> in-memory-analytics-accelerator-architecture-specification
> > +
> > +IAA Device Enabling
> > +-------------------
> > +
> > +- Enabling ``IAA`` devices for platform configuration, please refer to:
> > +
> > +https://www.intel.com/content/www/us/en/content-details/780887/intel-
> in-memory-analytics-accelerator-intel-iaa.html
> > +
> > +- ``IAA`` device driver is ``Intel Data Accelerator Driver (idxd)``, it
> is
> > +  recommended that the minimum version of Linux kernel is 5.18.
> > +
> > +- Add ``"intel_iommu=on,sm_on"`` parameter to kernel command line
> > +  for ``SVM`` feature enabling.
> > +
> > +Here is an easy way to verify ``IAA`` device driver and ``SVM``, refer
> to:
> > +
> > +https://github.com/intel/idxd-config/tree/stable/test
> > +
> > +IAA Device Management
> > +---------------------
> > +
> > +The number of ``IAA`` devices will vary depending on the Xeon product
> model.
> > +On a ``SPR`` server, there can be a maximum of 8 ``IAA`` devices, with
> up to
> > +4 devices per socket.
> > +
> > +By default, all ``IAA`` devices are disabled and need to be configured
> and
> > +enabled by users manually.
> > +
> > +Check the number of devices through the following command
> > +
> > +.. code-block:: shell
> > +
> > +  # lspci -d 8086:0cfe
> > +  # 6a:02.0 System peripheral: Intel Corporation Device 0cfe
> > +  # 6f:02.0 System peripheral: Intel Corporation Device 0cfe
> > +  # 74:02.0 System peripheral: Intel Corporation Device 0cfe
> > +  # 79:02.0 System peripheral: Intel Corporation Device 0cfe
> > +  # e7:02.0 System peripheral: Intel Corporation Device 0cfe
> > +  # ec:02.0 System peripheral: Intel Corporation Device 0cfe
> > +  # f1:02.0 System peripheral: Intel Corporation Device 0cfe
> > +  # f6:02.0 System peripheral: Intel Corporation Device 0cfe
> > +
> > +IAA Device Configuration
> > +------------------------
> > +
> > +The ``accel-config`` tool is used to enable ``IAA`` devices and
> configure
> > +``IAA`` hardware resources(work queues and engines). One ``IAA`` device
> > +has 8 work queues and 8 processing engines, multiple engines can be
> assigned
> > +to a work queue via ``group`` attribute.
> > +
> > +One example of configuring and enabling an ``IAA`` device.
> > +
> > +.. code-block:: shell
> > +
> > +  # accel-config config-engine iax1/engine1.0 -g 0
> > +  # accel-config config-engine iax1/engine1.1 -g 0
> > +  # accel-config config-engine iax1/engine1.2 -g 0
> > +  # accel-config config-engine iax1/engine1.3 -g 0
> > +  # accel-config config-engine iax1/engine1.4 -g 0
> > +  # accel-config config-engine iax1/engine1.5 -g 0
> > +  # accel-config config-engine iax1/engine1.6 -g 0
> > +  # accel-config config-engine iax1/engine1.7 -g 0
> > +  # accel-config config-wq iax1/wq1.0 -g 0 -s 128 -p 10 -b 1 -t 128 -m
> shared -y user -n app1 -d user
> > +  # accel-config enable-device iax1
> > +  # accel-config enable-wq iax1/wq1.0
> > +
> > +.. note::
> > +   IAX is an early name for IAA
> > +
> > +- The ``IAA`` device index is 1, use ``ls -lh
> /sys/bus/dsa/devices/iax*``
> > +  command to query the ``IAA`` device index.
> > +
> > +- 8 engines and 1 work queue are configured in group 0, so all
> compression jobs
> > +  submitted to this work queue can be processed by all engines at the
> same time.
> > +
> > +- Set work queue attributes including the work mode, work queue size
> and so on.
> > +
> > +- Enable the ``IAA1`` device and work queue 1.0
> > +
> > +.. note::
> > +  Set work queue mode to shared mode, since ``QPL`` library only
> supports
> > +  shared mode
> > +
> > +For more detailed configuration, please refer to:
> > +
> > +https://github.com/intel/idxd-config/tree/stable/Documentation/accfg
> > +
> > +IAA Resources Allocation For Migration
> > +--------------------------------------
> > +
> > +There is no ``IAA`` resource configuration parameters for migration and
> > +``accel-config`` tool configuration cannot directly specify the ``IAA``
> > +resources used for migration.
> > +
> > +``QPL`` will use all work queues that are enabled and set to shared
> mode,
> > +and use all engines assigned to the work queues with shared mode.
> > +
> > +By default, ``QPL`` will only use the local ``IAA`` device for
> compression
> > +job processing. The local ``IAA`` device means that the CPU of the job
> > +submission and the ``IAA`` device are on the same socket, so one CPU
> > +can submit the jobs to up to 4 ``IAA`` devices.
> > +
> > +Shared Virtual Memory(SVM) Introduction
> > +=======================================
> > +
> > +An ability for an accelerator I/O device to operate in the same virtual
> > +memory space of applications on host processors. It also implies the
> > +ability to operate from pageable memory, avoiding functional
> requirements
> > +to pin memory for DMA operations.
> > +
> > +When using ``SVM`` technology, users do not need to reserve memory for
> the
> > +``IAA`` device and perform pin memory operation. The ``IAA`` device can
> > +directly access data using the virtual address of the process.
> > +
> > +For more ``SVM`` technology, please refer to:
> > +
> > +https://docs.kernel.org/next/x86/sva.html
> > +
> > +
> > +How To Use QPL Compression In Migration
> > +=======================================
> > +
> > +1 - Installation of ``accel-config`` tool and ``QPL`` library
> 
> We can drop "1 " and stick with:
> 
>   - item1
>     - item1.1
>     ...
>   - item2
> 

Yes, it is better.

> > +
> > +  - Install ``accel-config`` tool from https://github.com/intel/idxd-
> config
> > +  - Install ``QPL`` library from https://github.com/intel/qpl
> > +
> > +2 - Configure and enable ``IAA`` devices and work queues via ``accel-
> config``
> > +
> > +3 - Build ``Qemu`` with ``--enable-qpl`` parameter
> > +
> > +  E.g. configure --target-list=x86_64-softmmu --enable-kvm ``--enable-
> qpl``
> > +
> > +4 - Start VMs with ``sudo`` command or ``root`` permission
> > +
> > +  Use the ``sudo`` command or ``root`` privilege to start the source
> and
> > +  destination virtual machines, since migration service needs
> permission
> > +  to access ``IAA`` hardware resources.
> > +
> > +5 - Enable ``QPL`` compression during migration
> > +
> > +  Set ``migrate_set_parameter multifd-compression qpl`` when migrating,
> the
> > +  ``QPL`` compression does not support configuring the compression
> level, it
> > +  only supports one compression level.
> > +
> > +The Difference Between QPL And ZLIB
> > +===================================
> > +
> > +Although both ``QPL`` and ``ZLIB`` are based on the deflate compression
> > +algorithm, and ``QPL`` can support the header and tail of ``ZLIB``,
> ``QPL``
> > +is still not fully compatible with the ``ZLIB`` compression in the
> migration.
> > +
> > +``QPL`` only supports 4K history buffer, and ``ZLIB`` is 32K by
> default. The
> > +``ZLIB`` compressed data that ``QPL`` may not decompress correctly and
> > +vice versa.
> > +
> > +``QPL`` does not support the ``Z_SYNC_FLUSH`` operation in ``ZLIB``
> streaming
> > +compression, current ``ZLIB`` implementation uses ``Z_SYNC_FLUSH``, so
> each
> > +``multifd`` thread has a ``ZLIB`` streaming context, and all page
> compression
> > +and decompression are based on this stream. ``QPL`` cannot decompress
> such data
> > +and vice versa.
> > +
> > +The introduction for ``Z_SYNC_FLUSH``, please refer to:
> > +
> > +https://www.zlib.net/manual.html
> > +
> > +The Best Practices
> > +==================
> > +
> > +When the virtual machine's pages are not populated and the ``IAA``
> device is
> > +used, I/O page faults occur, which can impact performance due to a
> large number
> > +of flush ``IOTLB`` operations.
> 
> AFAIU the IOTLB issue is not expected and can be fixed, while per our
> discussion in the other thread, the DMA fault latency cannot.
> 
> I think we can mention the possibility of IOTLB flush issue but that
> shouldn't be the major cause of such suggestion.  Again, I think it'll be
> great to mention how slow a DMA page fault can be, it can be compared in
> migration performance difference, or just describe an average DMA page
> fault latency, v.s. a processor fault.  That might explain why we suggest
> prefault the pages (comparing to a generic setup where the pages are
> always
> faulted by processors).

Get it, I will explain this issue in depth and add some comparative data, 
including I/O page fault Vs. process fault performance comparison and the
performance impact of I/O page fault on the entire live migration. Help 
developers and users better understand why -mem- prealloc needs to be 
added currently.

> > +
> > +Since the normal pages on the source side are all populated, ``IOTLB``
> caused
> > +by I/O page fault will not occur. On the destination side, a large
> number
> > +of normal pages need to be loaded, so it is recommended to add ``-mem-
> prealloc``
> > +parameter on the destination side.
> > --
> > 2.39.3
> >
> 
> --
> Peter Xu