[PATCH 0/2] Documentation/x86: Update the dynamic XSTATE doc

[PATCH 0/2] Documentation/x86: Update the dynamic XSTATE doc

[Chang S. Bae]

Hi all,

This series is intended to update the documentation only. It is not ready
yet for x86 maintainers. So more acknowledgment from Intel reviewers is
needed. Any preliminary review from the cc'ed folks will be much
appreciated. But it is possibly a waste of maintainers' time to review this
draft at this stage.

=== Cover Letter ===

With the AMX support in the mainline, I realize a couple of new
arch_prctl(2) options that were added for KVM have been missing in the doc.
And recently I heard some folks had hard time to understand the AMX
enabling process. A code example is expected to clarify the steps.

Thus, this patch set includes the following two updates:
(1) Patch 1 adds AMX enabling code example.
(2) Patch 2 explains the arch_prctl(2) options for guest:
    ARCH_{GET|REQ}_XCOMP_GUEST_PERM

The arch_prctl(2) manual page [1] is also missing the above and even other
options that are already included in the doc. Perhaps, the man-page update
follows up after this.

These changes can be found in the repo:
  git://github.com/intel/amx-linux.git doc

And the compiled preview is available here:
  https://htmlpreview.github.io/?https://github.com/intel/amx-linux/blob/doc-web/x86/xstate.html

Thanks,
Chang

[1] arch_prctl(2): https://man7.org/linux/man-pages/man2/arch_prctl.2.html

Chang S. Bae (2):
  Documentation/x86: Add the AMX enabling example
  Documentation/x86: Explain guest XSTATE permission control

 Documentation/x86/xstate.rst | 69 ++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)


base-commit: b13baccc3850ca8b8cccbf8ed9912dbaa0fdf7f3
-- 
2.17.1

[PATCH 1/2] Documentation/x86: Add the AMX enabling example

[Chang S. Bae]

Explain steps to enable the dynamic feature with a code example.

Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 Documentation/x86/xstate.rst | 48 ++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/Documentation/x86/xstate.rst b/Documentation/x86/xstate.rst
index 5cec7fb558d6..9597e6caa30e 100644
--- a/Documentation/x86/xstate.rst
+++ b/Documentation/x86/xstate.rst
@@ -64,6 +64,54 @@ the handler allocates a larger xstate buffer for the task so the large
 state can be context switched. In the unlikely cases that the allocation
 fails, the kernel sends SIGSEGV.
 
+AMX TILE_DATA enabling example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following steps dynamically enable TILE_DATA:
+
+  1. **Check the feature availability**. AMX_TILE is enumerated in CPUID
+     leaf 7, sub-leaf 0, bit 24 of EDX. If available, ``/proc/cpuinfo``
+     shows ``amx_tile`` in the flag entry of the CPUs.  Given that, the
+     kernel may have set XSTATE component 18 in the XCR0 register. But a
+     user needs to ensure the kernel support via the ARCH_GET_XCOMP_SUPP
+     option::
+
+        #include <asm/prctl.h>
+        #include <sys/syscall.h>
+	#include <stdio.h>
+        #include <unistd.h>
+
+        #define ARCH_GET_XCOMP_SUPP  0x1021
+
+        #define XFEATURE_XTILECFG    17
+        #define XFEATURE_XTILEDATA   18
+        #define XFEATURE_MASK_XTILE ((1 << XFEATURE_XTILECFG) | (1 << XFEATURE_XFILEDATA))
+
+        unsigned long features;
+        long rc;
+
+        ...
+
+        rc = syscall(SYS_arch_prctl, ARCH_GET_XCOMP_SUPP, &features);
+
+        if (!rc && features & XFEATURE_MASK_XTILE == XFEATURE_MASK_XTILE)
+            printf("AMX is available.\n");
+
+  2. **Request permission**. Now it is found that the kernel supports the
+     feature. But the permission is not automatically given. A user needs
+     to explicitly request it via the ARCH_REQ_XCOMP_PERM option::
+
+        #define ARCH_REQ_XCOMP_PERM  0x1023
+
+        ...
+
+        rc = syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA);
+
+        if (!rc)
+            printf("AMX is ready for use.\n");
+
+Note this example does not include the sigaltstack preparation.
+
 Dynamic features in signal frames
 ---------------------------------
 
-- 
2.17.1

[PATCH 2/2] Documentation/x86: Explain guest XSTATE permission control

[Chang S. Bae]

Commit 980fe2fddcff ("x86/fpu: Extend fpu_xstate_prctl() with guest
permissions") extends a couple of arch_prctl(2) options for VCPU threads.
Add description for them.

Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Yang Zhong <yang.zhong@intel.com>
Cc: kvm@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 Documentation/x86/xstate.rst | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/Documentation/x86/xstate.rst b/Documentation/x86/xstate.rst
index 9597e6caa30e..55cbce580853 100644
--- a/Documentation/x86/xstate.rst
+++ b/Documentation/x86/xstate.rst
@@ -64,6 +64,27 @@ the handler allocates a larger xstate buffer for the task so the large
 state can be context switched. In the unlikely cases that the allocation
 fails, the kernel sends SIGSEGV.
 
+In addition, a couple of extended options are provided for a VCPU thread.
+The VCPU XSTATE permission is separately controlled.
+
+-ARCH_GET_XCOMP_GUEST_PERM
+
+ arch_prctl(ARCH_GET_XCOMP_GUEST_PERM, &features);
+
+ ARCH_GET_XCOMP_GUEST_PERM is a variant of ARCH_GET_XCOMP_PERM. So it
+ provides the same semantics and functionality but for VCPU.
+
+-ARCH_REQ_XCOMP_GUEST_PERM
+
+ arch_prctl(ARCH_REQ_XCOMP_GUEST_PERM, feature_nr);
+
+ ARCH_REQ_XCOMP_GUEST_PERM is a variant of ARCH_REQ_XCOMP_PERM. Like the
+ above, it has the same semantics for VCPU permission. It performs a
+ similar functionality but with a constraint. Permission is frozen when the
+ first VCPU is created. So any attempt to change permission after that
+ point is rejected. Thus, permission has to be requested before the first
+ VCPU creation.
+
 AMX TILE_DATA enabling example
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-- 
2.17.1

Re: [PATCH 1/2] Documentation/x86: Add the AMX enabling example

[Dave Hansen]

> +  1. **Check the feature availability**. AMX_TILE is enumerated in CPUID
> +     leaf 7, sub-leaf 0, bit 24 of EDX. If available, ``/proc/cpuinfo``
> +     shows ``amx_tile`` in the flag entry of the CPUs.  Given that, the
> +     kernel may have set XSTATE component 18 in the XCR0 register. But a
> +     user needs to ensure the kernel support via the ARCH_GET_XCOMP_SUPP
> +     option::

Why did you bother mentioning the XCR0 and CPUID specifics?  We don't
want applications doing that, right?

> +        #include <asm/prctl.h>
> +        #include <sys/syscall.h>
> +	#include <stdio.h>
> +        #include <unistd.h>

^ Just from the appearance here there looks to be some spaces vs. tabs
inconsistency.

> +        #define ARCH_GET_XCOMP_SUPP  0x1021
> +
> +        #define XFEATURE_XTILECFG    17
> +        #define XFEATURE_XTILEDATA   18
> +        #define XFEATURE_MASK_XTILE ((1 << XFEATURE_XTILECFG) | (1 << XFEATURE_XFILEDATA))
> +
> +        unsigned long features;
> +        long rc;
> +
> +        ...
> +
> +        rc = syscall(SYS_arch_prctl, ARCH_GET_XCOMP_SUPP, &features);
> +
> +        if (!rc && features & XFEATURE_MASK_XTILE == XFEATURE_MASK_XTILE)
> +            printf("AMX is available.\n");
> +
> +  2. **Request permission**. Now it is found that the kernel supports the
> +     feature. But the permission is not automatically given. A user needs
> +     to explicitly request it via the ARCH_REQ_XCOMP_PERM option::

That phrasing is a bit awkward.  How about:

	After determining support for AMX, an application must
	explicitly ask permission to use it:
	...

> +        #define ARCH_REQ_XCOMP_PERM  0x1023
> +
> +        ...
> +
> +        rc = syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA);
> +
> +        if (!rc)
> +            printf("AMX is ready for use.\n");
> +
> +Note this example does not include the sigaltstack preparation.
> +
>  Dynamic features in signal frames
>  ---------------------------------
>  

Re: [PATCH 2/2] Documentation/x86: Explain guest XSTATE permission control

[Dave Hansen]

On 6/16/22 14:22, Chang S. Bae wrote:
> +In addition, a couple of extended options are provided for a VCPU thread.
> +The VCPU XSTATE permission is separately controlled.
> +
> +-ARCH_GET_XCOMP_GUEST_PERM
> +
> + arch_prctl(ARCH_GET_XCOMP_GUEST_PERM, &features);
> +
> + ARCH_GET_XCOMP_GUEST_PERM is a variant of ARCH_GET_XCOMP_PERM. So it
> + provides the same semantics and functionality but for VCPU.

This touches on the "what", but not the "why".  Could you explain in
here both why this is needed and why an app might want to use it?

Re: [PATCH 1/2] Documentation/x86: Add the AMX enabling example

[Chang S. Bae]

On 6/16/2022 3:45 PM, Dave Hansen wrote:
>> +  1. **Check the feature availability**. AMX_TILE is enumerated in CPUID
>> +     leaf 7, sub-leaf 0, bit 24 of EDX. If available, ``/proc/cpuinfo``
>> +     shows ``amx_tile`` in the flag entry of the CPUs.  Given that, the
>> +     kernel may have set XSTATE component 18 in the XCR0 register. But a
>> +     user needs to ensure the kernel support via the ARCH_GET_XCOMP_SUPP
>> +     option::
> 
> Why did you bother mentioning the XCR0 and CPUID specifics?  We don't
> want applications doing that, right?

Without checking them, this arch_prctl(2) option can be tried. Then it 
will return either EINVAL or the feature bit off if unavailable. Yes, 
that's all wanted instead of that old way. So maybe something like this 
here:

	An application first needs to determine the feature support:

> 
>> +        #include <asm/prctl.h>
>> +        #include <sys/syscall.h>
>> +	#include <stdio.h>
>> +        #include <unistd.h>
> 
> ^ Just from the appearance here there looks to be some spaces vs. tabs
> inconsistency.

Sorry, a tab instead of spaces was added later to fix a compile error.

<snip>

>> +  2. **Request permission**. Now it is found that the kernel supports the
>> +     feature. But the permission is not automatically given. A user needs
>> +     to explicitly request it via the ARCH_REQ_XCOMP_PERM option::
> 
> That phrasing is a bit awkward.  How about:
> 
> 	After determining support for AMX, an application must
> 	explicitly ask permission to use it:
> 	...

Yeah, looks to be concise. Thanks!

Chang

Re: [PATCH 2/2] Documentation/x86: Explain guest XSTATE permission control

[Chang S. Bae]

On 6/16/2022 3:49 PM, Dave Hansen wrote:
> On 6/16/22 14:22, Chang S. Bae wrote:
>> +In addition, a couple of extended options are provided for a VCPU thread.
>> +The VCPU XSTATE permission is separately controlled.
>> +
>> +-ARCH_GET_XCOMP_GUEST_PERM
>> +
>> + arch_prctl(ARCH_GET_XCOMP_GUEST_PERM, &features);
>> +
>> + ARCH_GET_XCOMP_GUEST_PERM is a variant of ARCH_GET_XCOMP_PERM. So it
>> + provides the same semantics and functionality but for VCPU.
> 
> This touches on the "what", but not the "why".  Could you explain in
> here both why this is needed and why an app might want to use it?

[ while studying on this a bit further, found a few things here ]

They (ARCH_{REQ|GET}_XCOMP_GUEST_PERM) provide a userspace VMM to 
request & check guest permission.

In general, KVM looks to have an API as a set of ioctls [1]. A guest VMM 
uses KVM_GET_DEVICE_ATTR::KVM_X86_XCOMP_GUEST_SUPP to query the 
available features [2][3]. ARCH_GET_XCOMP_SUPP is not usable here 
because KVM wants to control those exposed features [4] (via 
KVM_SUPPORTED_XCR0).

But oddly this mask does not appear to be actively referenced by those 
two arch_prctl options. I can see this ioctl attribute is currently 
disconnected from these arch_prctl options.

Also I failed to find the documentation about this 
KVM_X86_XCOMP_GUEST_SUPP interface:

	$ git grep KVM_X86_XCOMP_GUEST_SUPP ./Documentation/
	$

I guess people will be confused with having these two options only. I 
think documenting this has to come along with these missing pieces (and 
potential fix). So I'm inclined to drop this one at the moment.

Thanks,
Chang

[1] https://kernel.org/doc/html/latest/virt/kvm/index.html
[2] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/kvm/lib/x86_64/processor.c#n641
[3] 
https://github.com/qemu/qemu/blob/58b53669e87fed0d70903e05cd42079fbbdbc195/target/i386/kvm/kvm.c#L428
[4] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/kvm/x86.c#n9008

Re: [PATCH 2/2] Documentation/x86: Explain guest XSTATE permission control

[Chang S. Bae]

On 6/23/2022 4:55 PM, Chang S. Bae wrote:
> On 6/16/2022 3:49 PM, Dave Hansen wrote:
>>
>> This touches on the "what", but not the "why".  Could you explain in
>> here both why this is needed and why an app might want to use it?
> 
> [ while studying on this a bit further, found a few things here ]
> 
> They (ARCH_{REQ|GET}_XCOMP_GUEST_PERM) provide a userspace VMM to 
> request & check guest permission.
> 
> In general, KVM looks to have an API as a set of ioctls [1]. A guest VMM 
> uses KVM_GET_DEVICE_ATTR::KVM_X86_XCOMP_GUEST_SUPP to query the 
> available features [2][3]. ARCH_GET_XCOMP_SUPP is not usable here 
> because KVM wants to control those exposed features [4] (via 
> KVM_SUPPORTED_XCR0).
> 
> But oddly this mask does not appear to be actively referenced by those 
> two arch_prctl options. I can see this ioctl attribute is currently 
> disconnected from these arch_prctl options.
> 
> Also I failed to find the documentation about this 
> KVM_X86_XCOMP_GUEST_SUPP interface:
> 
>      $ git grep KVM_X86_XCOMP_GUEST_SUPP ./Documentation/
>      $
> 
> I guess people will be confused with having these two options only. I 
> think documenting this has to come along with these missing pieces (and 
> potential fix). So I'm inclined to drop this one at the moment.

Posted this series as following up this:

https://lore.kernel.org/lkml/20220823231402.7839-1-chang.seok.bae@intel.com/

Thanks,
Chang

Re: [PATCH 1/2] Documentation/x86: Add the AMX enabling example

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 1352 bytes --]

On Thu, Jun 16, 2022 at 02:22:09PM -0700, Chang S. Bae wrote: 
> +  1. **Check the feature availability**. AMX_TILE is enumerated in CPUID
> +     leaf 7, sub-leaf 0, bit 24 of EDX. If available, ``/proc/cpuinfo``
> +     shows ``amx_tile`` in the flag entry of the CPUs.  Given that, the
> +     kernel may have set XSTATE component 18 in the XCR0 register. But a
> +     user needs to ensure the kernel support via the ARCH_GET_XCOMP_SUPP
> +     option::
> +
> +        #include <asm/prctl.h>
> +        #include <sys/syscall.h>
> +	#include <stdio.h>
> +        #include <unistd.h>
> +
> +        #define ARCH_GET_XCOMP_SUPP  0x1021
> +
> +        #define XFEATURE_XTILECFG    17
> +        #define XFEATURE_XTILEDATA   18
> +        #define XFEATURE_MASK_XTILE ((1 << XFEATURE_XTILECFG) | (1 << XFEATURE_XFILEDATA))
> +
> +        unsigned long features;
> +        long rc;
> +
> +        ...
> +
> +        rc = syscall(SYS_arch_prctl, ARCH_GET_XCOMP_SUPP, &features);
> +
> +        if (!rc && features & XFEATURE_MASK_XTILE == XFEATURE_MASK_XTILE)
> +            printf("AMX is available.\n");
> +

nit: stdio include line isn't aligned with the rest of code.

Otherwise LGTM (no new warnings).

Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]