[PATCH v2 0/7] x86: compat header generation and checking adjustments

[PATCH v2 0/7] x86: compat header generation and checking adjustments

[Jan Beulich]

As was pointed out by 0e2e54966af5 ("mm: fix public declaration of
struct xen_mem_acquire_resource"), we're not currently handling structs
correctly that have uint64_aligned_t fields. Patch 2 demonstrates that
there was also an issue with XEN_GUEST_HANDLE_64().

Only the 1st patch was previously sent, but the approach chosen has
been changed altogether. All later patches are new. For 4.14 I think
at least patch 1 should be considered.

1: x86: fix compat header generation
2: x86/mce: add compat struct checking for XEN_MC_inject_v2
3: x86/mce: bring hypercall subop compat checking in sync again
4: x86/dmop: add compat struct checking for XEN_DMOP_map_mem_type_to_ioreq_server
5: x86: generalize padding field handling
6: flask: drop dead compat translation code
7: x86: only generate compat headers actually needed

Jan

[PATCH v2 1/7] x86: fix compat header generation

[Jan Beulich]

As was pointed out by 0e2e54966af5 ("mm: fix public declaration of
struct xen_mem_acquire_resource"), we're not currently handling structs
correctly that have uint64_aligned_t fields. #pragma pack(4) suppresses
the necessary alignment even if the type did properly survive (which
it also didn't) in the process of generating the headers. Overall,
with the above mentioned change applied, there's only a latent issue
here afaict, i.e. no other of our interface structs is currently
affected.

As a result it is clear that using #pragma pack(4) is not an option.
Drop all uses from compat header generation. Make sure
{,u}int64_aligned_t actually survives, such that explicitly aligned
fields will remain aligned. Arrange for {,u}int64_t to be transformed
into a type that's 64 bits wide and 4-byte aligned, by utilizing that
in typedef-s the "aligned" attribute can be used to reduce alignment.
Additionally, for the cases where native structures get re-used,
enforce suitable alignment via typedef-s (which allow alignment to be
reduced).

This use of typedef-s makes necessary changes to CHECK_*() macro
generation: Previously get-fields.sh relied on finding struct/union
keywords when other compound types were used. We now need to use the
typedef-s (guaranteeing suitable alignment) now, and hence the script
has to recognize those cases, too. (Unfortunately there are a few
special cases to be dealt with, but this is really not much different
from e.g. the pre-existing compat_domain_handle_t special case.)

This need to use typedef-s is certainly somewhat fragile going forward,
as in similar future cases it is imperative to also use typedef-s, or
else the CHECK_*() macros won't check what they're supposed to check. I
don't currently see any means to avoid this fragility, though.

There's one change to generated code according to my observations: In
arch_compat_vcpu_op() the runstate area "area" variable would previously
have been put in a just 4-byte aligned stack slot (despite being 8 bytes
in size), whereas now it gets put in an 8-byte aligned location.

There also results some curious inconsistency in struct xen_mc from
these changes - I intend to clean this up later on. Otherwise unrelated
code would also need adjustment right here.

Signed-off-by: Jan Beulich <jbeulich@suse.com>
---
v2: Different approach, addressing the latent alignment issues in v1.

--- a/xen/include/Makefile
+++ b/xen/include/Makefile
@@ -34,15 +34,6 @@ headers-$(CONFIG_XSM_FLASK) += compat/xs
 cppflags-y                := -include public/xen-compat.h -DXEN_GENERATING_COMPAT_HEADERS
 cppflags-$(CONFIG_X86)    += -m32
 
-# 8-byte types are 4-byte aligned on x86_32 ...
-ifeq ($(CONFIG_CC_IS_CLANG),y)
-prefix-$(CONFIG_X86)      := \#pragma pack(push, 4)
-suffix-$(CONFIG_X86)      := \#pragma pack(pop)
-else
-prefix-$(CONFIG_X86)      := \#pragma pack(4)
-suffix-$(CONFIG_X86)      := \#pragma pack()
-endif
-
 endif
 
 public-$(CONFIG_X86) := $(wildcard public/arch-x86/*.h public/arch-x86/*/*.h)
@@ -57,10 +48,8 @@ compat/%.h: compat/%.i Makefile $(BASEDI
 	echo "#define $$id" >>$@.new; \
 	echo "#include <xen/compat.h>" >>$@.new; \
 	$(if $(filter-out compat/arch-%.h,$@),echo "#include <$(patsubst compat/%,public/%,$@)>" >>$@.new;) \
-	$(if $(prefix-y),echo "$(prefix-y)" >>$@.new;) \
 	grep -v '^# [0-9]' $< | \
 	$(PYTHON) $(BASEDIR)/tools/compat-build-header.py | uniq >>$@.new; \
-	$(if $(suffix-y),echo "$(suffix-y)" >>$@.new;) \
 	echo "#endif /* $$id */" >>$@.new
 	mv -f $@.new $@
 
--- a/xen/include/public/arch-x86/pmu.h
+++ b/xen/include/public/arch-x86/pmu.h
@@ -105,7 +105,7 @@ struct xen_pmu_arch {
          * Processor's registers at the time of interrupt.
          * WO for hypervisor, RO for guests.
          */
-        struct xen_pmu_regs regs;
+        xen_pmu_regs_t regs;
         /* Padding for adding new registers to xen_pmu_regs in the future */
 #define XENPMU_REGS_PAD_SZ  64
         uint8_t pad[XENPMU_REGS_PAD_SZ];
@@ -132,8 +132,8 @@ struct xen_pmu_arch {
      * hypervisor into hardware during XENPMU_flush
      */
     union {
-        struct xen_pmu_amd_ctxt amd;
-        struct xen_pmu_intel_ctxt intel;
+        xen_pmu_amd_ctxt_t amd;
+        xen_pmu_intel_ctxt_t intel;
 
         /*
          * Padding for contexts (fixed parts only, does not include MSR banks
--- a/xen/include/public/arch-x86/xen-mca.h
+++ b/xen/include/public/arch-x86/xen-mca.h
@@ -112,7 +112,7 @@ struct mcinfo_common {
     uint16_t type;      /* structure type */
     uint16_t size;      /* size of this struct in bytes */
 };
-
+typedef struct mcinfo_common xen_mcinfo_common_t;
 
 #define MC_FLAG_CORRECTABLE     (1 << 0)
 #define MC_FLAG_UNCORRECTABLE   (1 << 1)
@@ -123,7 +123,7 @@ struct mcinfo_common {
 #define MC_FLAG_MCE		(1 << 6)
 /* contains global x86 mc information */
 struct mcinfo_global {
-    struct mcinfo_common common;
+    xen_mcinfo_common_t common;
 
     /* running domain at the time in error (most likely the impacted one) */
     uint16_t mc_domid;
@@ -138,7 +138,7 @@ struct mcinfo_global {
 
 /* contains bank local x86 mc information */
 struct mcinfo_bank {
-    struct mcinfo_common common;
+    xen_mcinfo_common_t common;
 
     uint16_t mc_bank; /* bank nr */
     uint16_t mc_domid; /* Usecase 5: domain referenced by mc_addr on dom0
@@ -156,11 +156,12 @@ struct mcinfo_msr {
     uint64_t reg;   /* MSR */
     uint64_t value; /* MSR value */
 };
+typedef struct mcinfo_msr xen_mcinfo_msr_t;
 
 /* contains mc information from other
  * or additional mc MSRs */
 struct mcinfo_extended {
-    struct mcinfo_common common;
+    xen_mcinfo_common_t common;
 
     /* You can fill up to five registers.
      * If you need more, then use this structure
@@ -172,7 +173,7 @@ struct mcinfo_extended {
      * and E(R)FLAGS, E(R)IP, E(R)MISC, up to 11/19 of them might be
      * useful at present. So expand this array to 32 to leave room.
      */
-    struct mcinfo_msr mc_msr[32];
+    xen_mcinfo_msr_t mc_msr[32];
 };
 
 /* Recovery Action flags. Giving recovery result information to DOM0 */
@@ -208,6 +209,7 @@ struct page_offline_action
     uint64_t mfn;
     uint64_t status;
 };
+typedef struct page_offline_action xen_page_offline_action_t;
 
 struct cpu_offline_action
 {
@@ -216,17 +218,18 @@ struct cpu_offline_action
     uint16_t mc_coreid;
     uint16_t mc_core_threadid;
 };
+typedef struct cpu_offline_action xen_cpu_offline_action_t;
 
 #define MAX_UNION_SIZE 16
 struct mcinfo_recovery
 {
-    struct mcinfo_common common;
+    xen_mcinfo_common_t common;
     uint16_t mc_bank; /* bank nr */
     uint8_t action_flags;
     uint8_t action_types;
     union {
-        struct page_offline_action page_retire;
-        struct cpu_offline_action cpu_offline;
+        xen_page_offline_action_t page_retire;
+        xen_cpu_offline_action_t cpu_offline;
         uint8_t pad[MAX_UNION_SIZE];
     } action_info;
 };
@@ -279,7 +282,7 @@ struct mcinfo_logical_cpu {
     uint32_t mc_cache_size;
     uint32_t mc_cache_alignment;
     int32_t mc_nmsrvals;
-    struct mcinfo_msr mc_msrvalues[__MC_MSR_ARRAYSIZE];
+    xen_mcinfo_msr_t mc_msrvalues[__MC_MSR_ARRAYSIZE];
 };
 typedef struct mcinfo_logical_cpu xen_mc_logical_cpu_t;
 DEFINE_XEN_GUEST_HANDLE(xen_mc_logical_cpu_t);
@@ -399,8 +402,9 @@ struct xen_mc_msrinject {
     domid_t  mcinj_domid;           /* valid only if MC_MSRINJ_F_GPADDR is
                                        present in mcinj_flags */
     uint16_t _pad0;
-    struct mcinfo_msr mcinj_msr[MC_MSRINJ_MAXMSRS];
+    xen_mcinfo_msr_t mcinj_msr[MC_MSRINJ_MAXMSRS];
 };
+typedef struct xen_mc_msrinject xen_mc_msrinject_t;
 
 /* Flags for mcinj_flags above; bits 16-31 are reserved */
 #define MC_MSRINJ_F_INTERPOSE   0x1
@@ -410,6 +414,7 @@ struct xen_mc_msrinject {
 struct xen_mc_mceinject {
     unsigned int mceinj_cpunr;      /* target processor id */
 };
+typedef struct xen_mc_mceinject xen_mc_mceinject_t;
 
 #if defined(__XEN__) || defined(__XEN_TOOLS__)
 #define XEN_MC_inject_v2        6
@@ -422,7 +427,7 @@ struct xen_mc_mceinject {
 
 struct xen_mc_inject_v2 {
     uint32_t flags;
-    struct xenctl_bitmap cpumap;
+    xenctl_bitmap_t cpumap;
 };
 #endif
 
@@ -431,10 +436,10 @@ struct xen_mc {
     uint32_t interface_version; /* XEN_MCA_INTERFACE_VERSION */
     union {
         struct xen_mc_fetch        mc_fetch;
-        struct xen_mc_notifydomain mc_notifydomain;
+        xen_mc_notifydomain_t      mc_notifydomain;
         struct xen_mc_physcpuinfo  mc_physcpuinfo;
-        struct xen_mc_msrinject    mc_msrinject;
-        struct xen_mc_mceinject    mc_mceinject;
+        xen_mc_msrinject_t         mc_msrinject;
+        xen_mc_mceinject_t         mc_mceinject;
 #if defined(__XEN__) || defined(__XEN_TOOLS__)
         struct xen_mc_inject_v2    mc_inject_v2;
 #endif
--- a/xen/include/public/argo.h
+++ b/xen/include/public/argo.h
@@ -67,8 +67,8 @@ typedef struct xen_argo_addr
 
 typedef struct xen_argo_send_addr
 {
-    struct xen_argo_addr src;
-    struct xen_argo_addr dst;
+    xen_argo_addr_t src;
+    xen_argo_addr_t dst;
 } xen_argo_send_addr_t;
 
 typedef struct xen_argo_ring
@@ -121,7 +121,7 @@ typedef struct xen_argo_unregister_ring
 
 typedef struct xen_argo_ring_data_ent
 {
-    struct xen_argo_addr ring;
+    xen_argo_addr_t ring;
     uint16_t flags;
     uint16_t pad;
     uint32_t space_required;
@@ -132,13 +132,13 @@ typedef struct xen_argo_ring_data
 {
     uint32_t nent;
     uint32_t pad;
-    struct xen_argo_ring_data_ent data[XEN_FLEX_ARRAY_DIM];
+    xen_argo_ring_data_ent_t data[XEN_FLEX_ARRAY_DIM];
 } xen_argo_ring_data_t;
 
 struct xen_argo_ring_message_header
 {
     uint32_t len;
-    struct xen_argo_addr source;
+    xen_argo_addr_t source;
     uint32_t message_type;
     uint8_t data[XEN_FLEX_ARRAY_DIM];
 };
--- a/xen/include/public/event_channel.h
+++ b/xen/include/public/event_channel.h
@@ -321,16 +321,16 @@ typedef struct evtchn_set_priority evtch
 struct evtchn_op {
     uint32_t cmd; /* enum event_channel_op */
     union {
-        struct evtchn_alloc_unbound    alloc_unbound;
-        struct evtchn_bind_interdomain bind_interdomain;
-        struct evtchn_bind_virq        bind_virq;
-        struct evtchn_bind_pirq        bind_pirq;
-        struct evtchn_bind_ipi         bind_ipi;
-        struct evtchn_close            close;
-        struct evtchn_send             send;
-        struct evtchn_status           status;
-        struct evtchn_bind_vcpu        bind_vcpu;
-        struct evtchn_unmask           unmask;
+        evtchn_alloc_unbound_t    alloc_unbound;
+        evtchn_bind_interdomain_t bind_interdomain;
+        evtchn_bind_virq_t        bind_virq;
+        evtchn_bind_pirq_t        bind_pirq;
+        evtchn_bind_ipi_t         bind_ipi;
+        evtchn_close_t            close;
+        evtchn_send_t             send;
+        evtchn_status_t           status;
+        evtchn_bind_vcpu_t        bind_vcpu;
+        evtchn_unmask_t           unmask;
     } u;
 };
 typedef struct evtchn_op evtchn_op_t;
--- a/xen/include/public/hvm/dm_op.h
+++ b/xen/include/public/hvm/dm_op.h
@@ -74,6 +74,7 @@ struct xen_dm_op_create_ioreq_server {
     /* OUT - server id */
     ioservid_t id;
 };
+typedef struct xen_dm_op_create_ioreq_server xen_dm_op_create_ioreq_server_t;
 
 /*
  * XEN_DMOP_get_ioreq_server_info: Get all the information necessary to
@@ -113,6 +114,7 @@ struct xen_dm_op_get_ioreq_server_info {
     /* OUT - buffered ioreq gfn (see block comment above)*/
     uint64_aligned_t bufioreq_gfn;
 };
+typedef struct xen_dm_op_get_ioreq_server_info xen_dm_op_get_ioreq_server_info_t;
 
 /*
  * XEN_DMOP_map_io_range_to_ioreq_server: Register an I/O range for
@@ -148,6 +150,7 @@ struct xen_dm_op_ioreq_server_range {
     /* IN - inclusive start and end of range */
     uint64_aligned_t start, end;
 };
+typedef struct xen_dm_op_ioreq_server_range xen_dm_op_ioreq_server_range_t;
 
 #define XEN_DMOP_PCI_SBDF(s,b,d,f) \
 	((((s) & 0xffff) << 16) |  \
@@ -173,6 +176,7 @@ struct xen_dm_op_set_ioreq_server_state
     uint8_t enabled;
     uint8_t pad;
 };
+typedef struct xen_dm_op_set_ioreq_server_state xen_dm_op_set_ioreq_server_state_t;
 
 /*
  * XEN_DMOP_destroy_ioreq_server: Destroy the IOREQ Server <id>.
@@ -186,6 +190,7 @@ struct xen_dm_op_destroy_ioreq_server {
     ioservid_t id;
     uint16_t pad;
 };
+typedef struct xen_dm_op_destroy_ioreq_server xen_dm_op_destroy_ioreq_server_t;
 
 /*
  * XEN_DMOP_track_dirty_vram: Track modifications to the specified pfn
@@ -203,6 +208,7 @@ struct xen_dm_op_track_dirty_vram {
     /* IN - first pfn to track */
     uint64_aligned_t first_pfn;
 };
+typedef struct xen_dm_op_track_dirty_vram xen_dm_op_track_dirty_vram_t;
 
 /*
  * XEN_DMOP_set_pci_intx_level: Set the logical level of one of a domain's
@@ -217,6 +223,7 @@ struct xen_dm_op_set_pci_intx_level {
     /* IN - Level: 0 -> deasserted, 1 -> asserted */
     uint8_t  level;
 };
+typedef struct xen_dm_op_set_pci_intx_level xen_dm_op_set_pci_intx_level_t;
 
 /*
  * XEN_DMOP_set_isa_irq_level: Set the logical level of a one of a domain's
@@ -230,6 +237,7 @@ struct xen_dm_op_set_isa_irq_level {
     /* IN - Level: 0 -> deasserted, 1 -> asserted */
     uint8_t  level;
 };
+typedef struct xen_dm_op_set_isa_irq_level xen_dm_op_set_isa_irq_level_t;
 
 /*
  * XEN_DMOP_set_pci_link_route: Map a PCI INTx line to an IRQ line.
@@ -242,6 +250,7 @@ struct xen_dm_op_set_pci_link_route {
     /* ISA IRQ (1-15) or 0 -> disable link */
     uint8_t  isa_irq;
 };
+typedef struct xen_dm_op_set_pci_link_route xen_dm_op_set_pci_link_route_t;
 
 /*
  * XEN_DMOP_modified_memory: Notify that a set of pages were modified by
@@ -265,6 +274,7 @@ struct xen_dm_op_modified_memory {
     /* IN/OUT - Must be set to 0 */
     uint32_t opaque;
 };
+typedef struct xen_dm_op_modified_memory xen_dm_op_modified_memory_t;
 
 struct xen_dm_op_modified_memory_extent {
     /* IN - number of contiguous pages modified */
@@ -294,6 +304,7 @@ struct xen_dm_op_set_mem_type {
     /* IN - first pfn in region */
     uint64_aligned_t first_pfn;
 };
+typedef struct xen_dm_op_set_mem_type xen_dm_op_set_mem_type_t;
 
 /*
  * XEN_DMOP_inject_event: Inject an event into a VCPU, which will
@@ -327,6 +338,7 @@ struct xen_dm_op_inject_event {
     /* IN - type-specific extra data (%cr2 for #PF, pending_dbg for #DB) */
     uint64_aligned_t cr2;
 };
+typedef struct xen_dm_op_inject_event xen_dm_op_inject_event_t;
 
 /*
  * XEN_DMOP_inject_msi: Inject an MSI for an emulated device.
@@ -340,6 +352,7 @@ struct xen_dm_op_inject_msi {
     /* IN - MSI address (0xfeexxxxx) */
     uint64_aligned_t addr;
 };
+typedef struct xen_dm_op_inject_msi xen_dm_op_inject_msi_t;
 
 /*
  * XEN_DMOP_map_mem_type_to_ioreq_server : map or unmap the IOREQ Server <id>
@@ -366,6 +379,7 @@ struct xen_dm_op_map_mem_type_to_ioreq_s
     uint64_t opaque;    /* IN/OUT - only used for hypercall continuation,
                            has to be set to zero by the caller */
 };
+typedef struct xen_dm_op_map_mem_type_to_ioreq_server xen_dm_op_map_mem_type_to_ioreq_server_t;
 
 /*
  * XEN_DMOP_remote_shutdown : Declare a shutdown for another domain
@@ -377,6 +391,7 @@ struct xen_dm_op_remote_shutdown {
     uint32_t reason;       /* SHUTDOWN_* => enum sched_shutdown_reason */
                            /* (Other reason values are not blocked) */
 };
+typedef struct xen_dm_op_remote_shutdown xen_dm_op_remote_shutdown_t;
 
 /*
  * XEN_DMOP_relocate_memory : Relocate GFNs for the specified guest.
@@ -395,6 +410,7 @@ struct xen_dm_op_relocate_memory {
     /* Starting GFN where GFNs should be relocated. */
     uint64_aligned_t dst_gfn;
 };
+typedef struct xen_dm_op_relocate_memory xen_dm_op_relocate_memory_t;
 
 /*
  * XEN_DMOP_pin_memory_cacheattr : Pin caching type of RAM space.
@@ -416,30 +432,30 @@ struct xen_dm_op_pin_memory_cacheattr {
     uint32_t type;          /* XEN_DMOP_MEM_CACHEATTR_* */
     uint32_t pad;
 };
+typedef struct xen_dm_op_pin_memory_cacheattr xen_dm_op_pin_memory_cacheattr_t;
 
 struct xen_dm_op {
     uint32_t op;
     uint32_t pad;
     union {
-        struct xen_dm_op_create_ioreq_server create_ioreq_server;
-        struct xen_dm_op_get_ioreq_server_info get_ioreq_server_info;
-        struct xen_dm_op_ioreq_server_range map_io_range_to_ioreq_server;
-        struct xen_dm_op_ioreq_server_range unmap_io_range_from_ioreq_server;
-        struct xen_dm_op_set_ioreq_server_state set_ioreq_server_state;
-        struct xen_dm_op_destroy_ioreq_server destroy_ioreq_server;
-        struct xen_dm_op_track_dirty_vram track_dirty_vram;
-        struct xen_dm_op_set_pci_intx_level set_pci_intx_level;
-        struct xen_dm_op_set_isa_irq_level set_isa_irq_level;
-        struct xen_dm_op_set_pci_link_route set_pci_link_route;
-        struct xen_dm_op_modified_memory modified_memory;
-        struct xen_dm_op_set_mem_type set_mem_type;
-        struct xen_dm_op_inject_event inject_event;
-        struct xen_dm_op_inject_msi inject_msi;
-        struct xen_dm_op_map_mem_type_to_ioreq_server
-                map_mem_type_to_ioreq_server;
-        struct xen_dm_op_remote_shutdown remote_shutdown;
-        struct xen_dm_op_relocate_memory relocate_memory;
-        struct xen_dm_op_pin_memory_cacheattr pin_memory_cacheattr;
+        xen_dm_op_create_ioreq_server_t create_ioreq_server;
+        xen_dm_op_get_ioreq_server_info_t get_ioreq_server_info;
+        xen_dm_op_ioreq_server_range_t map_io_range_to_ioreq_server;
+        xen_dm_op_ioreq_server_range_t unmap_io_range_from_ioreq_server;
+        xen_dm_op_set_ioreq_server_state_t set_ioreq_server_state;
+        xen_dm_op_destroy_ioreq_server_t destroy_ioreq_server;
+        xen_dm_op_track_dirty_vram_t track_dirty_vram;
+        xen_dm_op_set_pci_intx_level_t set_pci_intx_level;
+        xen_dm_op_set_isa_irq_level_t set_isa_irq_level;
+        xen_dm_op_set_pci_link_route_t set_pci_link_route;
+        xen_dm_op_modified_memory_t modified_memory;
+        xen_dm_op_set_mem_type_t set_mem_type;
+        xen_dm_op_inject_event_t inject_event;
+        xen_dm_op_inject_msi_t inject_msi;
+        xen_dm_op_map_mem_type_to_ioreq_server_t map_mem_type_to_ioreq_server;
+        xen_dm_op_remote_shutdown_t remote_shutdown;
+        xen_dm_op_relocate_memory_t relocate_memory;
+        xen_dm_op_pin_memory_cacheattr_t pin_memory_cacheattr;
     } u;
 };
 
--- a/xen/include/public/hvm/hvm_vcpu.h
+++ b/xen/include/public/hvm/hvm_vcpu.h
@@ -69,6 +69,7 @@ struct vcpu_hvm_x86_32 {
 
     uint16_t pad2[3];
 };
+typedef struct vcpu_hvm_x86_32 xen_vcpu_hvm_x86_32_t;
 
 /*
  * The layout of the _ar fields of the segment registers is the
@@ -114,6 +115,7 @@ struct vcpu_hvm_x86_64 {
      * the 32-bit structure should be used instead.
      */
 };
+typedef struct vcpu_hvm_x86_64 xen_vcpu_hvm_x86_64_t;
 
 struct vcpu_hvm_context {
 #define VCPU_HVM_MODE_32B 0  /* 32bit fields of the structure will be used. */
@@ -124,8 +126,8 @@ struct vcpu_hvm_context {
 
     /* CPU registers. */
     union {
-        struct vcpu_hvm_x86_32 x86_32;
-        struct vcpu_hvm_x86_64 x86_64;
+        xen_vcpu_hvm_x86_32_t x86_32;
+        xen_vcpu_hvm_x86_64_t x86_64;
     } cpu_regs;
 };
 typedef struct vcpu_hvm_context vcpu_hvm_context_t;
--- a/xen/include/public/hypfs.h
+++ b/xen/include/public/hypfs.h
@@ -53,9 +53,10 @@ struct xen_hypfs_direntry {
     uint32_t content_len;      /* Current length of data. */
     uint32_t max_write_len;    /* Max. length for writes (0 if read-only). */
 };
+typedef struct xen_hypfs_direntry xen_hypfs_direntry_t;
 
 struct xen_hypfs_dirlistentry {
-    struct xen_hypfs_direntry e;
+    xen_hypfs_direntry_t e;
     /* Offset in bytes to next entry (0 == this is the last entry). */
     uint16_t off_next;
     /* Zero terminated entry name, possibly with some padding for alignment. */
--- a/xen/include/public/memory.h
+++ b/xen/include/public/memory.h
@@ -604,7 +604,7 @@ struct xen_reserved_device_memory_map {
     XEN_GUEST_HANDLE(xen_reserved_device_memory_t) buffer;
     /* IN */
     union {
-        struct physdev_pci_device pci;
+        physdev_pci_device_t pci;
     } dev;
 };
 typedef struct xen_reserved_device_memory_map xen_reserved_device_memory_map_t;
--- a/xen/include/public/physdev.h
+++ b/xen/include/public/physdev.h
@@ -229,11 +229,11 @@ DEFINE_XEN_GUEST_HANDLE(physdev_manage_p
 struct physdev_op {
     uint32_t cmd;
     union {
-        struct physdev_irq_status_query      irq_status_query;
-        struct physdev_set_iopl              set_iopl;
-        struct physdev_set_iobitmap          set_iobitmap;
-        struct physdev_apic                  apic_op;
-        struct physdev_irq                   irq_op;
+        physdev_irq_status_query_t irq_status_query;
+        physdev_set_iopl_t         set_iopl;
+        physdev_set_iobitmap_t     set_iobitmap;
+        physdev_apic_t             apic_op;
+        physdev_irq_t              irq_op;
     } u;
 };
 typedef struct physdev_op physdev_op_t;
@@ -334,7 +334,7 @@ struct physdev_dbgp_op {
     uint8_t op;
     uint8_t bus;
     union {
-        struct physdev_pci_device pci;
+        physdev_pci_device_t pci;
     } u;
 };
 typedef struct physdev_dbgp_op physdev_dbgp_op_t;
--- a/xen/include/public/platform.h
+++ b/xen/include/public/platform.h
@@ -42,6 +42,7 @@ struct xenpf_settime32 {
     uint32_t nsecs;
     uint64_t system_time;
 };
+typedef struct xenpf_settime32 xenpf_settime32_t;
 #define XENPF_settime64           62
 struct xenpf_settime64 {
     /* IN variables. */
@@ -50,6 +51,7 @@ struct xenpf_settime64 {
     uint32_t mbz;
     uint64_t system_time;
 };
+typedef struct xenpf_settime64 xenpf_settime64_t;
 #if __XEN_INTERFACE_VERSION__ < 0x00040600
 #define XENPF_settime XENPF_settime32
 #define xenpf_settime xenpf_settime32
@@ -529,6 +531,7 @@ struct xenpf_cpu_hotadd
 	uint32_t acpi_id;
 	uint32_t pxm;
 };
+typedef struct xenpf_cpu_hotadd xenpf_cpu_hotadd_t;
 
 #define XENPF_mem_hotadd    59
 struct xenpf_mem_hotadd
@@ -538,6 +541,7 @@ struct xenpf_mem_hotadd
     uint32_t pxm;
     uint32_t flags;
 };
+typedef struct xenpf_mem_hotadd xenpf_mem_hotadd_t;
 
 #define XENPF_core_parking  60
 
@@ -622,29 +626,29 @@ struct xen_platform_op {
     uint32_t cmd;
     uint32_t interface_version; /* XENPF_INTERFACE_VERSION */
     union {
-        struct xenpf_settime           settime;
-        struct xenpf_settime32         settime32;
-        struct xenpf_settime64         settime64;
-        struct xenpf_add_memtype       add_memtype;
-        struct xenpf_del_memtype       del_memtype;
-        struct xenpf_read_memtype      read_memtype;
-        struct xenpf_microcode_update  microcode;
-        struct xenpf_platform_quirk    platform_quirk;
-        struct xenpf_efi_runtime_call  efi_runtime_call;
-        struct xenpf_firmware_info     firmware_info;
-        struct xenpf_enter_acpi_sleep  enter_acpi_sleep;
-        struct xenpf_change_freq       change_freq;
-        struct xenpf_getidletime       getidletime;
-        struct xenpf_set_processor_pminfo set_pminfo;
-        struct xenpf_pcpuinfo          pcpu_info;
-        struct xenpf_pcpu_version      pcpu_version;
-        struct xenpf_cpu_ol            cpu_ol;
-        struct xenpf_cpu_hotadd        cpu_add;
-        struct xenpf_mem_hotadd        mem_add;
-        struct xenpf_core_parking      core_parking;
-        struct xenpf_resource_op       resource_op;
-        struct xenpf_symdata           symdata;
-        uint8_t                        pad[128];
+        xenpf_settime_t               settime;
+        xenpf_settime32_t             settime32;
+        xenpf_settime64_t             settime64;
+        xenpf_add_memtype_t           add_memtype;
+        xenpf_del_memtype_t           del_memtype;
+        xenpf_read_memtype_t          read_memtype;
+        xenpf_microcode_update_t      microcode;
+        xenpf_platform_quirk_t        platform_quirk;
+        xenpf_efi_runtime_call_t      efi_runtime_call;
+        xenpf_firmware_info_t         firmware_info;
+        xenpf_enter_acpi_sleep_t      enter_acpi_sleep;
+        xenpf_change_freq_t           change_freq;
+        xenpf_getidletime_t           getidletime;
+        xenpf_set_processor_pminfo_t  set_pminfo;
+        xenpf_pcpuinfo_t              pcpu_info;
+        xenpf_pcpu_version_t          pcpu_version;
+        xenpf_cpu_ol_t                cpu_ol;
+        xenpf_cpu_hotadd_t            cpu_add;
+        xenpf_mem_hotadd_t            mem_add;
+        xenpf_core_parking_t          core_parking;
+        xenpf_resource_op_t           resource_op;
+        xenpf_symdata_t               symdata;
+        uint8_t                       pad[128];
     } u;
 };
 typedef struct xen_platform_op xen_platform_op_t;
--- a/xen/include/public/pmu.h
+++ b/xen/include/public/pmu.h
@@ -127,7 +127,7 @@ struct xen_pmu_data {
     uint8_t pad[6];
 
     /* Architecture-specific information */
-    struct xen_pmu_arch pmu;
+    xen_pmu_arch_t pmu;
 };
 
 #endif /* __XEN_PUBLIC_PMU_H__ */
--- a/xen/include/public/xen.h
+++ b/xen/include/public/xen.h
@@ -726,7 +726,7 @@ struct vcpu_info {
 #endif /* XEN_HAVE_PV_UPCALL_MASK */
     xen_ulong_t evtchn_pending_sel;
     struct arch_vcpu_info arch;
-    struct vcpu_time_info time;
+    vcpu_time_info_t time;
 }; /* 64 bytes (x86) */
 #ifndef __XEN__
 typedef struct vcpu_info vcpu_info_t;
@@ -1031,6 +1031,7 @@ struct xenctl_bitmap {
     XEN_GUEST_HANDLE_64(uint8) bitmap;
     uint32_t nr_bits;
 };
+typedef struct xenctl_bitmap xenctl_bitmap_t;
 #endif
 
 #endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
--- a/xen/include/public/xsm/flask_op.h
+++ b/xen/include/public/xsm/flask_op.h
@@ -33,10 +33,12 @@ struct xen_flask_load {
     XEN_GUEST_HANDLE(char) buffer;
     uint32_t size;
 };
+typedef struct xen_flask_load xen_flask_load_t;
 
 struct xen_flask_setenforce {
     uint32_t enforcing;
 };
+typedef struct xen_flask_setenforce xen_flask_setenforce_t;
 
 struct xen_flask_sid_context {
     /* IN/OUT: sid to convert to/from string */
@@ -47,6 +49,7 @@ struct xen_flask_sid_context {
     uint32_t size;
     XEN_GUEST_HANDLE(char) context;
 };
+typedef struct xen_flask_sid_context xen_flask_sid_context_t;
 
 struct xen_flask_access {
     /* IN: access request */
@@ -60,6 +63,7 @@ struct xen_flask_access {
     uint32_t audit_deny;
     uint32_t seqno;
 };
+typedef struct xen_flask_access xen_flask_access_t;
 
 struct xen_flask_transition {
     /* IN: transition SIDs and class */
@@ -69,6 +73,7 @@ struct xen_flask_transition {
     /* OUT: new SID */
     uint32_t newsid;
 };
+typedef struct xen_flask_transition xen_flask_transition_t;
 
 #if __XEN_INTERFACE_VERSION__ < 0x00040800
 struct xen_flask_userlist {
@@ -106,11 +111,13 @@ struct xen_flask_boolean {
      */
     XEN_GUEST_HANDLE(char) name;
 };
+typedef struct xen_flask_boolean xen_flask_boolean_t;
 
 struct xen_flask_setavc_threshold {
     /* IN */
     uint32_t threshold;
 };
+typedef struct xen_flask_setavc_threshold xen_flask_setavc_threshold_t;
 
 struct xen_flask_hash_stats {
     /* OUT */
@@ -119,6 +126,7 @@ struct xen_flask_hash_stats {
     uint32_t buckets_total;
     uint32_t max_chain_len;
 };
+typedef struct xen_flask_hash_stats xen_flask_hash_stats_t;
 
 struct xen_flask_cache_stats {
     /* IN */
@@ -131,6 +139,7 @@ struct xen_flask_cache_stats {
     uint32_t reclaims;
     uint32_t frees;
 };
+typedef struct xen_flask_cache_stats xen_flask_cache_stats_t;
 
 struct xen_flask_ocontext {
     /* IN */
@@ -138,6 +147,7 @@ struct xen_flask_ocontext {
     uint32_t sid;
     uint64_t low, high;
 };
+typedef struct xen_flask_ocontext xen_flask_ocontext_t;
 
 struct xen_flask_peersid {
     /* IN */
@@ -145,12 +155,14 @@ struct xen_flask_peersid {
     /* OUT */
     uint32_t sid;
 };
+typedef struct xen_flask_peersid xen_flask_peersid_t;
 
 struct xen_flask_relabel {
     /* IN */
     uint32_t domid;
     uint32_t sid;
 };
+typedef struct xen_flask_relabel xen_flask_relabel_t;
 
 struct xen_flask_devicetree_label {
     /* IN */
@@ -158,6 +170,7 @@ struct xen_flask_devicetree_label {
     uint32_t length;
     XEN_GUEST_HANDLE(char) path;
 };
+typedef struct xen_flask_devicetree_label xen_flask_devicetree_label_t;
 
 struct xen_flask_op {
     uint32_t cmd;
@@ -188,26 +201,26 @@ struct xen_flask_op {
 #define FLASK_DEVICETREE_LABEL  25
     uint32_t interface_version; /* XEN_FLASK_INTERFACE_VERSION */
     union {
-        struct xen_flask_load load;
-        struct xen_flask_setenforce enforce;
+        xen_flask_load_t load;
+        xen_flask_setenforce_t enforce;
         /* FLASK_CONTEXT_TO_SID and FLASK_SID_TO_CONTEXT */
-        struct xen_flask_sid_context sid_context;
-        struct xen_flask_access access;
+        xen_flask_sid_context_t sid_context;
+        xen_flask_access_t access;
         /* FLASK_CREATE, FLASK_RELABEL, FLASK_MEMBER */
-        struct xen_flask_transition transition;
+        xen_flask_transition_t transition;
 #if __XEN_INTERFACE_VERSION__ < 0x00040800
         struct xen_flask_userlist userlist;
 #endif
         /* FLASK_GETBOOL, FLASK_SETBOOL */
-        struct xen_flask_boolean boolean;
-        struct xen_flask_setavc_threshold setavc_threshold;
-        struct xen_flask_hash_stats hash_stats;
-        struct xen_flask_cache_stats cache_stats;
+        xen_flask_boolean_t boolean;
+        xen_flask_setavc_threshold_t setavc_threshold;
+        xen_flask_hash_stats_t hash_stats;
+        xen_flask_cache_stats_t cache_stats;
         /* FLASK_ADD_OCONTEXT, FLASK_DEL_OCONTEXT */
-        struct xen_flask_ocontext ocontext;
-        struct xen_flask_peersid peersid;
-        struct xen_flask_relabel relabel;
-        struct xen_flask_devicetree_label devicetree_label;
+        xen_flask_ocontext_t ocontext;
+        xen_flask_peersid_t peersid;
+        xen_flask_relabel_t relabel;
+        xen_flask_devicetree_label_t devicetree_label;
     } u;
 };
 typedef struct xen_flask_op xen_flask_op_t;
--- a/xen/tools/compat-build-header.py
+++ b/xen/tools/compat-build-header.py
@@ -3,7 +3,7 @@
 import re,sys
 
 pats = [
- [ r"__InClUdE__(.*)", r"#include\1\n#pragma pack(4)" ],
+ [ r"__InClUdE__(.*)", r"#include\1" ],
  [ r"__IfDeF__ (XEN_HAVE.*)", r"#ifdef \1" ],
  [ r"__ElSe__", r"#else" ],
  [ r"__EnDif__", r"#endif" ],
@@ -11,9 +11,11 @@ pats = [
  [ r"__UnDeF__", r"#undef" ],
  [ r"\"xen-compat.h\"", r"<public/xen-compat.h>" ],
  [ r"(struct|union|enum)\s+(xen_?)?(\w)", r"\1 compat_\3" ],
- [ r"@KeeP@", r"" ],
+ [ r"typedef(.*)@KeeP@(xen_?)?([\w]+)([^\w])",
+   r"typedef\1\2\3 __attribute__((__aligned__(__alignof(\1compat_\3))))\4" ],
  [ r"_t([^\w]|$)", r"_compat_t\1" ],
- [ r"(8|16|32|64)_compat_t([^\w]|$)", r"\1_t\2" ],
+ [ r"int(8|16|32|64_aligned)_compat_t([^\w]|$)", r"int\1_t\2" ],
+ [ r"(\su?int64(_compat)?)_T([^\w]|$)", r"\1_t\3" ],
  [ r"(^|[^\w])xen_?(\w*)_compat_t([^\w]|$$)", r"\1compat_\2_t\3" ],
  [ r"(^|[^\w])XEN_?", r"\1COMPAT_" ],
  [ r"(^|[^\w])Xen_?", r"\1Compat_" ],
--- a/xen/tools/compat-build-source.py
+++ b/xen/tools/compat-build-source.py
@@ -9,6 +9,7 @@ pats = [
  [ r"^\s*#\s*endif /\* (XEN_HAVE.*) \*/\s+", r"__EnDif__" ],
  [ r"^\s*#\s*define\s+([A-Z_]*_GUEST_HANDLE)", r"#define HIDE_\1" ],
  [ r"^\s*#\s*define\s+([a-z_]*_guest_handle)", r"#define hide_\1" ],
+ [ r"^\s*#\s*define\s+(u?int64)_aligned_t\s.*", r"typedef \1_T __attribute__((aligned(4))) \1_compat_T;" ],
  [ r"XEN_GUEST_HANDLE(_[0-9A-Fa-f]+)?", r"COMPAT_HANDLE" ],
 ];
 
--- a/xen/tools/get-fields.sh
+++ b/xen/tools/get-fields.sh
@@ -418,6 +418,21 @@ check_field ()
 			"}")
 				level=$(expr $level - 1) id=
 				;;
+			compat_*_t)
+				if [ $level = 2 ]
+				then
+					fields=" "
+					token="${token%_t}"
+					token="${token#compat_}"
+				fi
+				;;
+			evtchn_*_compat_t)
+				if [ $level = 2 -a $token != evtchn_port_compat_t ]
+				then
+					fields=" "
+					token="${token%_compat_t}"
+				fi
+				;;
 			[a-zA-Z]*)
 				id=$token
 				;;
@@ -464,6 +479,14 @@ build_check ()
 		"]")
 			arrlvl=$(expr $arrlvl - 1)
 			;;
+		compat_*_t)
+			if [ $level = 2 -a $token != compat_argo_port_t ]
+			then
+				fields=" "
+				token="${token%_t}"
+				token="${token#compat_}"
+			fi
+			;;
 		[a-zA-Z_]*)
 			test $level != 2 -o $arrlvl != 1 || id=$token
 			;;

Re: [PATCH v2 1/7] x86: fix compat header generation

[Roger Pau Monné]

On Wed, Jul 01, 2020 at 12:25:15PM +0200, Jan Beulich wrote:
> As was pointed out by 0e2e54966af5 ("mm: fix public declaration of
> struct xen_mem_acquire_resource"), we're not currently handling structs
> correctly that have uint64_aligned_t fields. #pragma pack(4) suppresses
> the necessary alignment even if the type did properly survive (which
> it also didn't) in the process of generating the headers. Overall,
> with the above mentioned change applied, there's only a latent issue
> here afaict, i.e. no other of our interface structs is currently
> affected.
> 
> As a result it is clear that using #pragma pack(4) is not an option.
> Drop all uses from compat header generation. Make sure
> {,u}int64_aligned_t actually survives, such that explicitly aligned
> fields will remain aligned. Arrange for {,u}int64_t to be transformed
> into a type that's 64 bits wide and 4-byte aligned, by utilizing that
> in typedef-s the "aligned" attribute can be used to reduce alignment.
> Additionally, for the cases where native structures get re-used,
> enforce suitable alignment via typedef-s (which allow alignment to be
> reduced).
> 
> This use of typedef-s makes necessary changes to CHECK_*() macro
> generation: Previously get-fields.sh relied on finding struct/union
> keywords when other compound types were used. We now need to use the
> typedef-s (guaranteeing suitable alignment) now, and hence the script

Extra now before the comma I think.

> has to recognize those cases, too. (Unfortunately there are a few
> special cases to be dealt with, but this is really not much different
> from e.g. the pre-existing compat_domain_handle_t special case.)
> 
> This need to use typedef-s is certainly somewhat fragile going forward,
> as in similar future cases it is imperative to also use typedef-s, or
> else the CHECK_*() macros won't check what they're supposed to check. I
> don't currently see any means to avoid this fragility, though.
> 
> There's one change to generated code according to my observations: In
> arch_compat_vcpu_op() the runstate area "area" variable would previously
> have been put in a just 4-byte aligned stack slot (despite being 8 bytes
> in size), whereas now it gets put in an 8-byte aligned location.
> 
> There also results some curious inconsistency in struct xen_mc from
> these changes - I intend to clean this up later on. Otherwise unrelated
> code would also need adjustment right here.

Oh, so that's the reason fields in xen_mc are not all switched to use
their typedef equivalent I guess?

> --- a/xen/tools/get-fields.sh
> +++ b/xen/tools/get-fields.sh
> @@ -418,6 +418,21 @@ check_field ()
>  			"}")
>  				level=$(expr $level - 1) id=
>  				;;
> +			compat_*_t)
> +				if [ $level = 2 ]
> +				then
> +					fields=" "
> +					token="${token%_t}"
> +					token="${token#compat_}"
> +				fi
> +				;;
> +			evtchn_*_compat_t)
> +				if [ $level = 2 -a $token != evtchn_port_compat_t ]
> +				then
> +					fields=" "
> +					token="${token%_compat_t}"
> +				fi
> +				;;

Likely related to the above, but I assume we might want to add a check
here to assert no struct fields are used?

I assume this is not added here in order to prevent exploding due to
the xen_mc issues.

Thanks, Roger.

Re: [PATCH v2 1/7] x86: fix compat header generation

[Jan Beulich]

On 01.07.2020 18:10, Roger Pau Monné wrote:
> On Wed, Jul 01, 2020 at 12:25:15PM +0200, Jan Beulich wrote:
>> As was pointed out by 0e2e54966af5 ("mm: fix public declaration of
>> struct xen_mem_acquire_resource"), we're not currently handling structs
>> correctly that have uint64_aligned_t fields. #pragma pack(4) suppresses
>> the necessary alignment even if the type did properly survive (which
>> it also didn't) in the process of generating the headers. Overall,
>> with the above mentioned change applied, there's only a latent issue
>> here afaict, i.e. no other of our interface structs is currently
>> affected.
>>
>> As a result it is clear that using #pragma pack(4) is not an option.
>> Drop all uses from compat header generation. Make sure
>> {,u}int64_aligned_t actually survives, such that explicitly aligned
>> fields will remain aligned. Arrange for {,u}int64_t to be transformed
>> into a type that's 64 bits wide and 4-byte aligned, by utilizing that
>> in typedef-s the "aligned" attribute can be used to reduce alignment.
>> Additionally, for the cases where native structures get re-used,
>> enforce suitable alignment via typedef-s (which allow alignment to be
>> reduced).
>>
>> This use of typedef-s makes necessary changes to CHECK_*() macro
>> generation: Previously get-fields.sh relied on finding struct/union
>> keywords when other compound types were used. We now need to use the
>> typedef-s (guaranteeing suitable alignment) now, and hence the script
> 
> Extra now before the comma I think.
> 
>> has to recognize those cases, too. (Unfortunately there are a few
>> special cases to be dealt with, but this is really not much different
>> from e.g. the pre-existing compat_domain_handle_t special case.)
>>
>> This need to use typedef-s is certainly somewhat fragile going forward,
>> as in similar future cases it is imperative to also use typedef-s, or
>> else the CHECK_*() macros won't check what they're supposed to check. I
>> don't currently see any means to avoid this fragility, though.
>>
>> There's one change to generated code according to my observations: In
>> arch_compat_vcpu_op() the runstate area "area" variable would previously
>> have been put in a just 4-byte aligned stack slot (despite being 8 bytes
>> in size), whereas now it gets put in an 8-byte aligned location.
>>
>> There also results some curious inconsistency in struct xen_mc from
>> these changes - I intend to clean this up later on. Otherwise unrelated
>> code would also need adjustment right here.
> 
> Oh, so that's the reason fields in xen_mc are not all switched to use
> their typedef equivalent I guess?

Yes - see patches later in the series, which take care of the anomaly.

>> --- a/xen/tools/get-fields.sh
>> +++ b/xen/tools/get-fields.sh
>> @@ -418,6 +418,21 @@ check_field ()
>>  			"}")
>>  				level=$(expr $level - 1) id=
>>  				;;
>> +			compat_*_t)
>> +				if [ $level = 2 ]
>> +				then
>> +					fields=" "
>> +					token="${token%_t}"
>> +					token="${token#compat_}"
>> +				fi
>> +				;;
>> +			evtchn_*_compat_t)
>> +				if [ $level = 2 -a $token != evtchn_port_compat_t ]
>> +				then
>> +					fields=" "
>> +					token="${token%_compat_t}"
>> +				fi
>> +				;;
> 
> Likely related to the above, but I assume we might want to add a check
> here to assert no struct fields are used?

I think we could, but have you found similar assertions
elsewhere? There being any fields would, aiui, indicate a syntax
violation (or else $level can't be 2), and I'd rather leave
catching these to the compiler.

> I assume this is not added here in order to prevent exploding due to
> the xen_mc issues.

I don't think it would, as it continues handling struct/union
just fine. (We may want to drop this support, to enforce no
use of only typedef-s, but I'm not sure _that_ wouldn't
explode.)

Jan

Re: [PATCH v2 1/7] x86: fix compat header generation

[Roger Pau Monné]

On Wed, Jul 01, 2020 at 12:25:15PM +0200, Jan Beulich wrote:
> As was pointed out by 0e2e54966af5 ("mm: fix public declaration of
> struct xen_mem_acquire_resource"), we're not currently handling structs
> correctly that have uint64_aligned_t fields. #pragma pack(4) suppresses
> the necessary alignment even if the type did properly survive (which
> it also didn't) in the process of generating the headers. Overall,
> with the above mentioned change applied, there's only a latent issue
> here afaict, i.e. no other of our interface structs is currently
> affected.
> 
> As a result it is clear that using #pragma pack(4) is not an option.
> Drop all uses from compat header generation. Make sure
> {,u}int64_aligned_t actually survives, such that explicitly aligned
> fields will remain aligned. Arrange for {,u}int64_t to be transformed
> into a type that's 64 bits wide and 4-byte aligned, by utilizing that
> in typedef-s the "aligned" attribute can be used to reduce alignment.
> Additionally, for the cases where native structures get re-used,
> enforce suitable alignment via typedef-s (which allow alignment to be
> reduced).
> 
> This use of typedef-s makes necessary changes to CHECK_*() macro
> generation: Previously get-fields.sh relied on finding struct/union
> keywords when other compound types were used. We now need to use the
> typedef-s (guaranteeing suitable alignment) now, and hence the script
> has to recognize those cases, too. (Unfortunately there are a few
> special cases to be dealt with, but this is really not much different
> from e.g. the pre-existing compat_domain_handle_t special case.)
> 
> This need to use typedef-s is certainly somewhat fragile going forward,
> as in similar future cases it is imperative to also use typedef-s, or
> else the CHECK_*() macros won't check what they're supposed to check. I
> don't currently see any means to avoid this fragility, though.
> 
> There's one change to generated code according to my observations: In
> arch_compat_vcpu_op() the runstate area "area" variable would previously
> have been put in a just 4-byte aligned stack slot (despite being 8 bytes
> in size), whereas now it gets put in an 8-byte aligned location.
> 
> There also results some curious inconsistency in struct xen_mc from
> these changes - I intend to clean this up later on. Otherwise unrelated
> code would also need adjustment right here.
> 
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks.

Re: [PATCH v2 1/7] x86: fix compat header generation

[Jan Beulich]

On 15.07.2020 10:43, Roger Pau Monné wrote:
> On Wed, Jul 01, 2020 at 12:25:15PM +0200, Jan Beulich wrote:
>> As was pointed out by 0e2e54966af5 ("mm: fix public declaration of
>> struct xen_mem_acquire_resource"), we're not currently handling structs
>> correctly that have uint64_aligned_t fields. #pragma pack(4) suppresses
>> the necessary alignment even if the type did properly survive (which
>> it also didn't) in the process of generating the headers. Overall,
>> with the above mentioned change applied, there's only a latent issue
>> here afaict, i.e. no other of our interface structs is currently
>> affected.
>>
>> As a result it is clear that using #pragma pack(4) is not an option.
>> Drop all uses from compat header generation. Make sure
>> {,u}int64_aligned_t actually survives, such that explicitly aligned
>> fields will remain aligned. Arrange for {,u}int64_t to be transformed
>> into a type that's 64 bits wide and 4-byte aligned, by utilizing that
>> in typedef-s the "aligned" attribute can be used to reduce alignment.
>> Additionally, for the cases where native structures get re-used,
>> enforce suitable alignment via typedef-s (which allow alignment to be
>> reduced).
>>
>> This use of typedef-s makes necessary changes to CHECK_*() macro
>> generation: Previously get-fields.sh relied on finding struct/union
>> keywords when other compound types were used. We now need to use the
>> typedef-s (guaranteeing suitable alignment) now, and hence the script
>> has to recognize those cases, too. (Unfortunately there are a few
>> special cases to be dealt with, but this is really not much different
>> from e.g. the pre-existing compat_domain_handle_t special case.)
>>
>> This need to use typedef-s is certainly somewhat fragile going forward,
>> as in similar future cases it is imperative to also use typedef-s, or
>> else the CHECK_*() macros won't check what they're supposed to check. I
>> don't currently see any means to avoid this fragility, though.
>>
>> There's one change to generated code according to my observations: In
>> arch_compat_vcpu_op() the runstate area "area" variable would previously
>> have been put in a just 4-byte aligned stack slot (despite being 8 bytes
>> in size), whereas now it gets put in an 8-byte aligned location.
>>
>> There also results some curious inconsistency in struct xen_mc from
>> these changes - I intend to clean this up later on. Otherwise unrelated
>> code would also need adjustment right here.
>>
>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> 
> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks. I shall send out v3, as I had to fix an issue with old gcc:
There were two (identical) typedef-s for {,u}int64_compat_t, which
newer gcc tolerate, but older don't.

Jan

[PATCH v2 2/7] x86/mce: add compat struct checking for XEN_MC_inject_v2

[Jan Beulich]

84e364f2eda2 ("x86: add CMCI software injection interface") merely made
sure things would build, without any concern about things actually
working:
- despite the addition of xenctl_bitmap to xlat.lst, the resulting macro
  wasn't invoked anywhere (which would have lead to recognizing that the
  structure appeared to have no fully compatible layout, despite the use
  of a 64-bit handle),
- the interface struct itself was neither added to xlat.lst (and the
  resulting macro then invoked) nor was any manual checking of
  individual fields added.

Adjust compat header generation logic to retain XEN_GUEST_HANDLE_64(),
which is intentionally layed out to be compatible between different size
guests. Invoke the missing checking (implicitly through CHECK_mc).

No change in the resulting generated code.

Fixes: 84e364f2eda2 ("x86: add CMCI software injection interface")
Signed-off-by: Jan Beulich <jbeulich@suse.com>

--- a/xen/arch/x86/cpu/mcheck/mce.c
+++ b/xen/arch/x86/cpu/mcheck/mce.c
@@ -1312,10 +1312,12 @@ CHECK_FIELD_(struct, mc_fetch, fetch_id)
 CHECK_FIELD_(struct, mc_physcpuinfo, ncpus);
 # define CHECK_compat_mc_physcpuinfo struct mc_physcpuinfo
 
-#define CHECK_compat_mc_inject_v2   struct mc_inject_v2
+# define xen_ctl_bitmap              xenctl_bitmap
+
 CHECK_mc;
 # undef CHECK_compat_mc_fetch
 # undef CHECK_compat_mc_physcpuinfo
+# undef xen_ctl_bitmap
 
 # define xen_mc_info                 mc_info
 CHECK_mc_info;
--- a/xen/include/public/arch-x86/xen-mca.h
+++ b/xen/include/public/arch-x86/xen-mca.h
@@ -429,6 +429,7 @@ struct xen_mc_inject_v2 {
     uint32_t flags;
     xenctl_bitmap_t cpumap;
 };
+typedef struct xen_mc_inject_v2 xen_mc_inject_v2_t;
 #endif
 
 struct xen_mc {
@@ -441,7 +442,7 @@ struct xen_mc {
         xen_mc_msrinject_t         mc_msrinject;
         xen_mc_mceinject_t         mc_mceinject;
 #if defined(__XEN__) || defined(__XEN_TOOLS__)
-        struct xen_mc_inject_v2    mc_inject_v2;
+        xen_mc_inject_v2_t         mc_inject_v2;
 #endif
     } u;
 };
--- a/xen/include/xlat.lst
+++ b/xen/include/xlat.lst
@@ -25,6 +25,7 @@
 ?	mcinfo_recovery			arch-x86/xen-mca.h
 !	mc_fetch			arch-x86/xen-mca.h
 ?	mc_info				arch-x86/xen-mca.h
+?	mc_inject_v2			arch-x86/xen-mca.h
 ?	mc_mceinject			arch-x86/xen-mca.h
 ?	mc_msrinject			arch-x86/xen-mca.h
 ?	mc_notifydomain			arch-x86/xen-mca.h
--- a/xen/tools/compat-build-header.py
+++ b/xen/tools/compat-build-header.py
@@ -19,6 +19,7 @@ pats = [
  [ r"(^|[^\w])xen_?(\w*)_compat_t([^\w]|$$)", r"\1compat_\2_t\3" ],
  [ r"(^|[^\w])XEN_?", r"\1COMPAT_" ],
  [ r"(^|[^\w])Xen_?", r"\1Compat_" ],
+ [ r"(^|[^\w])COMPAT_HANDLE_64\(", r"\1XEN_GUEST_HANDLE_64(" ],
  [ r"(^|[^\w])long([^\w]|$$)", r"\1int\2" ]
 ];
 
--- a/xen/tools/compat-build-source.py
+++ b/xen/tools/compat-build-source.py
@@ -10,7 +10,7 @@ pats = [
  [ r"^\s*#\s*define\s+([A-Z_]*_GUEST_HANDLE)", r"#define HIDE_\1" ],
  [ r"^\s*#\s*define\s+([a-z_]*_guest_handle)", r"#define hide_\1" ],
  [ r"^\s*#\s*define\s+(u?int64)_aligned_t\s.*", r"typedef \1_T __attribute__((aligned(4))) \1_compat_T;" ],
- [ r"XEN_GUEST_HANDLE(_[0-9A-Fa-f]+)?", r"COMPAT_HANDLE" ],
+ [ r"XEN_GUEST_HANDLE", r"COMPAT_HANDLE" ],
 ];
 
 xlatf = open('xlat.lst', 'r')

Re: [PATCH v2 2/7] x86/mce: add compat struct checking for XEN_MC_inject_v2

[Roger Pau Monné]

On Wed, Jul 01, 2020 at 12:25:48PM +0200, Jan Beulich wrote:
> 84e364f2eda2 ("x86: add CMCI software injection interface") merely made
> sure things would build, without any concern about things actually
> working:
> - despite the addition of xenctl_bitmap to xlat.lst, the resulting macro
>   wasn't invoked anywhere (which would have lead to recognizing that the
>   structure appeared to have no fully compatible layout, despite the use
>   of a 64-bit handle),
> - the interface struct itself was neither added to xlat.lst (and the
>   resulting macro then invoked) nor was any manual checking of
>   individual fields added.
> 
> Adjust compat header generation logic to retain XEN_GUEST_HANDLE_64(),
> which is intentionally layed out to be compatible between different size
> guests. Invoke the missing checking (implicitly through CHECK_mc).
> 
> No change in the resulting generated code.
> 
> Fixes: 84e364f2eda2 ("x86: add CMCI software injection interface")
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

LGTM, just one question below.

> --- a/xen/tools/compat-build-header.py
> +++ b/xen/tools/compat-build-header.py
> @@ -19,6 +19,7 @@ pats = [
>   [ r"(^|[^\w])xen_?(\w*)_compat_t([^\w]|$$)", r"\1compat_\2_t\3" ],
>   [ r"(^|[^\w])XEN_?", r"\1COMPAT_" ],
>   [ r"(^|[^\w])Xen_?", r"\1Compat_" ],
> + [ r"(^|[^\w])COMPAT_HANDLE_64\(", r"\1XEN_GUEST_HANDLE_64(" ],

Why do you need to match with the opening parenthesis?

Is this for the #ifndef XEN_GUEST_HANDLE_64 instances? Don't they need
to also be replaced with the compat types?

Thanks, Roger.

Re: [PATCH v2 2/7] x86/mce: add compat struct checking for XEN_MC_inject_v2

[Jan Beulich]

On 14.07.2020 12:24, Roger Pau Monné wrote:
> On Wed, Jul 01, 2020 at 12:25:48PM +0200, Jan Beulich wrote:
>> --- a/xen/tools/compat-build-header.py
>> +++ b/xen/tools/compat-build-header.py
>> @@ -19,6 +19,7 @@ pats = [
>>   [ r"(^|[^\w])xen_?(\w*)_compat_t([^\w]|$$)", r"\1compat_\2_t\3" ],
>>   [ r"(^|[^\w])XEN_?", r"\1COMPAT_" ],
>>   [ r"(^|[^\w])Xen_?", r"\1Compat_" ],
>> + [ r"(^|[^\w])COMPAT_HANDLE_64\(", r"\1XEN_GUEST_HANDLE_64(" ],
> 
> Why do you need to match with the opening parenthesis?
> 
> Is this for the #ifndef XEN_GUEST_HANDLE_64 instances? Don't they need
> to also be replaced with the compat types?

Well, I wanted to be as strict as possible, i.e. along with
the matching of a non-identifier char at the beginning I
also wanted the other side to be delimited.

Jan

Re: [PATCH v2 2/7] x86/mce: add compat struct checking for XEN_MC_inject_v2

[Roger Pau Monné]

On Wed, Jul 01, 2020 at 12:25:48PM +0200, Jan Beulich wrote:
> 84e364f2eda2 ("x86: add CMCI software injection interface") merely made
> sure things would build, without any concern about things actually
> working:
> - despite the addition of xenctl_bitmap to xlat.lst, the resulting macro
>   wasn't invoked anywhere (which would have lead to recognizing that the
>   structure appeared to have no fully compatible layout, despite the use
>   of a 64-bit handle),
> - the interface struct itself was neither added to xlat.lst (and the
>   resulting macro then invoked) nor was any manual checking of
>   individual fields added.
> 
> Adjust compat header generation logic to retain XEN_GUEST_HANDLE_64(),
> which is intentionally layed out to be compatible between different size
> guests. Invoke the missing checking (implicitly through CHECK_mc).
> 
> No change in the resulting generated code.
> 
> Fixes: 84e364f2eda2 ("x86: add CMCI software injection interface")
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks.

[PATCH v2 3/7] x86/mce: bring hypercall subop compat checking in sync again

[Jan Beulich]

Use a typedef in struct xen_mc also for the two subops "manually"
translated in the handler, just for consistency. No functional
change.

Signed-off-by: Jan Beulich <jbeulich@suse.com>

--- a/xen/arch/x86/cpu/mcheck/mce.c
+++ b/xen/arch/x86/cpu/mcheck/mce.c
@@ -1307,16 +1307,16 @@ CHECK_mcinfo_common;
 
 CHECK_FIELD_(struct, mc_fetch, flags);
 CHECK_FIELD_(struct, mc_fetch, fetch_id);
-# define CHECK_compat_mc_fetch       struct mc_fetch
+# define CHECK_mc_fetch              struct mc_fetch
 
 CHECK_FIELD_(struct, mc_physcpuinfo, ncpus);
-# define CHECK_compat_mc_physcpuinfo struct mc_physcpuinfo
+# define CHECK_mc_physcpuinfo        struct mc_physcpuinfo
 
 # define xen_ctl_bitmap              xenctl_bitmap
 
 CHECK_mc;
-# undef CHECK_compat_mc_fetch
-# undef CHECK_compat_mc_physcpuinfo
+# undef CHECK_mc_fetch
+# undef CHECK_mc_physcpuinfo
 # undef xen_ctl_bitmap
 
 # define xen_mc_info                 mc_info
--- a/xen/include/public/arch-x86/xen-mca.h
+++ b/xen/include/public/arch-x86/xen-mca.h
@@ -391,6 +391,7 @@ struct xen_mc_physcpuinfo {
     /* OUT */
     XEN_GUEST_HANDLE(xen_mc_logical_cpu_t) info;
 };
+typedef struct xen_mc_physcpuinfo xen_mc_physcpuinfo_t;
 
 #define XEN_MC_msrinject    4
 #define MC_MSRINJ_MAXMSRS       8
@@ -436,9 +437,9 @@ struct xen_mc {
     uint32_t cmd;
     uint32_t interface_version; /* XEN_MCA_INTERFACE_VERSION */
     union {
-        struct xen_mc_fetch        mc_fetch;
+        xen_mc_fetch_t             mc_fetch;
         xen_mc_notifydomain_t      mc_notifydomain;
-        struct xen_mc_physcpuinfo  mc_physcpuinfo;
+        xen_mc_physcpuinfo_t       mc_physcpuinfo;
         xen_mc_msrinject_t         mc_msrinject;
         xen_mc_mceinject_t         mc_mceinject;
 #if defined(__XEN__) || defined(__XEN_TOOLS__)

Re: [PATCH v2 3/7] x86/mce: bring hypercall subop compat checking in sync again

[Roger Pau Monné]

On Wed, Jul 01, 2020 at 12:26:54PM +0200, Jan Beulich wrote:
> Use a typedef in struct xen_mc also for the two subops "manually"
> translated in the handler, just for consistency. No functional
> change.

I'm slightly puzzled by the fact that mc_fetch is marked as needs
checking while mc_physcpuinfo is marked as needs translation,
shouldn't both be marked as needing translation? (since both need to
handle a guest pointer using XEN_GUEST_HANDLE)

Thanks, Roger.

Re: [PATCH v2 3/7] x86/mce: bring hypercall subop compat checking in sync again

[Jan Beulich]

On 14.07.2020 13:19, Roger Pau Monné wrote:
> On Wed, Jul 01, 2020 at 12:26:54PM +0200, Jan Beulich wrote:
>> Use a typedef in struct xen_mc also for the two subops "manually"
>> translated in the handler, just for consistency. No functional
>> change.
> 
> I'm slightly puzzled by the fact that mc_fetch is marked as needs
> checking while mc_physcpuinfo is marked as needs translation,
> shouldn't both be marked as needing translation? (since both need to
> handle a guest pointer using XEN_GUEST_HANDLE)

I guess I'm confused - I see an exclamation mark on both respective
lines in xlat.lst.

Jan

Re: [PATCH v2 3/7] x86/mce: bring hypercall subop compat checking in sync again

[Roger Pau Monné]

On Tue, Jul 14, 2020 at 01:47:11PM +0200, Jan Beulich wrote:
> On 14.07.2020 13:19, Roger Pau Monné wrote:
> > On Wed, Jul 01, 2020 at 12:26:54PM +0200, Jan Beulich wrote:
> >> Use a typedef in struct xen_mc also for the two subops "manually"
> >> translated in the handler, just for consistency. No functional
> >> change.
> > 
> > I'm slightly puzzled by the fact that mc_fetch is marked as needs
> > checking while mc_physcpuinfo is marked as needs translation,
> > shouldn't both be marked as needing translation? (since both need to
> > handle a guest pointer using XEN_GUEST_HANDLE)
> 
> I guess I'm confused - I see an exclamation mark on both respective

No, I was the one confused, you are right that both are marked as need
translation.

Roger.

Re: [PATCH v2 3/7] x86/mce: bring hypercall subop compat checking in sync again

[Jan Beulich]

On 14.07.2020 16:31, Roger Pau Monné wrote:
> On Tue, Jul 14, 2020 at 01:47:11PM +0200, Jan Beulich wrote:
>> On 14.07.2020 13:19, Roger Pau Monné wrote:
>>> On Wed, Jul 01, 2020 at 12:26:54PM +0200, Jan Beulich wrote:
>>>> Use a typedef in struct xen_mc also for the two subops "manually"
>>>> translated in the handler, just for consistency. No functional
>>>> change.
>>>
>>> I'm slightly puzzled by the fact that mc_fetch is marked as needs
>>> checking while mc_physcpuinfo is marked as needs translation,
>>> shouldn't both be marked as needing translation? (since both need to
>>> handle a guest pointer using XEN_GUEST_HANDLE)
>>
>> I guess I'm confused - I see an exclamation mark on both respective
> 
> No, I was the one confused, you are right that both are marked as need
> translation.

And just to mention it explicitly - I think the lines could be
dropped, as they look to be there just for documentation (if at
all). The resulting XLAT_* macros don't get used anywhere.

Jan

Re: [PATCH v2 3/7] x86/mce: bring hypercall subop compat checking in sync again

[Roger Pau Monné]

On Wed, Jul 01, 2020 at 12:26:54PM +0200, Jan Beulich wrote:
> Use a typedef in struct xen_mc also for the two subops "manually"
> translated in the handler, just for consistency. No functional
> change.
> 
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks.

[PATCH v2 4/7] x86/dmop: add compat struct checking for XEN_DMOP_map_mem_type_to_ioreq_server

[Jan Beulich]

This was forgotten when the subop was added.

Also take the opportunity and move the dm_op_relocate_memory entry in
xlat.lst to its designated place.

No change in the resulting generated code.

Fixes: ca2b511d3ff4 ("x86/ioreq server: add DMOP to map guest ram with p2m_ioreq_server to an ioreq server")
Signed-off-by: Jan Beulich <jbeulich@suse.com>

--- a/xen/arch/x86/hvm/dm.c
+++ b/xen/arch/x86/hvm/dm.c
@@ -730,6 +730,7 @@ CHECK_dm_op_modified_memory;
 CHECK_dm_op_set_mem_type;
 CHECK_dm_op_inject_event;
 CHECK_dm_op_inject_msi;
+CHECK_dm_op_map_mem_type_to_ioreq_server;
 CHECK_dm_op_remote_shutdown;
 CHECK_dm_op_relocate_memory;
 CHECK_dm_op_pin_memory_cacheattr;
--- a/xen/include/xlat.lst
+++ b/xen/include/xlat.lst
@@ -67,15 +67,16 @@
 ?	grant_entry_v2			grant_table.h
 ?	gnttab_swap_grant_ref		grant_table.h
 !	dm_op_buf			hvm/dm_op.h
-?	dm_op_relocate_memory		hvm/dm_op.h
 ?	dm_op_create_ioreq_server	hvm/dm_op.h
 ?	dm_op_destroy_ioreq_server	hvm/dm_op.h
 ?	dm_op_get_ioreq_server_info	hvm/dm_op.h
 ?	dm_op_inject_event		hvm/dm_op.h
 ?	dm_op_inject_msi		hvm/dm_op.h
 ?	dm_op_ioreq_server_range	hvm/dm_op.h
+?	dm_op_map_mem_type_to_ioreq_server hvm/dm_op.h
 ?	dm_op_modified_memory		hvm/dm_op.h
 ?	dm_op_pin_memory_cacheattr	hvm/dm_op.h
+?	dm_op_relocate_memory		hvm/dm_op.h
 ?	dm_op_remote_shutdown		hvm/dm_op.h
 ?	dm_op_set_ioreq_server_state	hvm/dm_op.h
 ?	dm_op_set_isa_irq_level		hvm/dm_op.h

Re: [PATCH v2 4/7] x86/dmop: add compat struct checking for XEN_DMOP_map_mem_type_to_ioreq_server

[Roger Pau Monné]

On Wed, Jul 01, 2020 at 12:27:16PM +0200, Jan Beulich wrote:
> This was forgotten when the subop was added.
> 
> Also take the opportunity and move the dm_op_relocate_memory entry in
> xlat.lst to its designated place.
> 
> No change in the resulting generated code.
> 
> Fixes: ca2b511d3ff4 ("x86/ioreq server: add DMOP to map guest ram with p2m_ioreq_server to an ioreq server")
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks.

[PATCH v2 5/7] x86: generalize padding field handling

[Jan Beulich]

The original intention was to ignore padding fields, but the pattern
matched only ones whose names started with an underscore. Also match
fields whose names are in line with the C spec by not having a leading
underscore. (Note that the leading ^ in the sed regexps was pointless
and hence get dropped.)

This requires adjusting some vNUMA macros, to avoid triggering
"enumeration value ... not handled in switch" warnings, which - due to
-Werror - would cause the build to fail. (I have to admit that I find
these padding fields odd, when translation of the containing structure
is needed anyway.)

Signed-off-by: Jan Beulich <jbeulich@suse.com>
---
While for translation macros skipping padding fields pretty surely is a
reasonable thing to do, we may want to consider not ignoring them when
generating checking macros.

--- a/xen/common/compat/memory.c
+++ b/xen/common/compat/memory.c
@@ -354,10 +354,13 @@ int compat_memory_op(unsigned int cmd, X
                 return -EFAULT;
 
 #define XLAT_vnuma_topology_info_HNDL_vdistance_h(_d_, _s_)		\
+            case XLAT_vnuma_topology_info_vdistance_pad:                \
             guest_from_compat_handle((_d_)->vdistance.h, (_s_)->vdistance.h)
 #define XLAT_vnuma_topology_info_HNDL_vcpu_to_vnode_h(_d_, _s_)		\
+            case XLAT_vnuma_topology_info_vcpu_to_vnode_pad:            \
             guest_from_compat_handle((_d_)->vcpu_to_vnode.h, (_s_)->vcpu_to_vnode.h)
 #define XLAT_vnuma_topology_info_HNDL_vmemrange_h(_d_, _s_)		\
+            case XLAT_vnuma_topology_info_vmemrange_pad:                \
             guest_from_compat_handle((_d_)->vmemrange.h, (_s_)->vmemrange.h)
 
             XLAT_vnuma_topology_info(nat.vnuma, &cmp.vnuma);
--- a/xen/tools/get-fields.sh
+++ b/xen/tools/get-fields.sh
@@ -218,7 +218,7 @@ for line in sys.stdin.readlines():
 				fi
 				;;
 			[\,\;])
-				if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
+				if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
 				then
 					if [ $kind = union ]
 					then
@@ -347,7 +347,7 @@ build_body ()
 			fi
 			;;
 		[\,\;])
-			if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
+			if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
 			then
 				if [ -z "$array" -a -z "$array_type" ]
 				then
@@ -437,7 +437,7 @@ check_field ()
 				id=$token
 				;;
 			[\,\;])
-				if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
+				if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
 				then
 					check_field $1 $2 $3.$id "$fields"
 					test "$token" != ";" || fields= id=
@@ -491,7 +491,7 @@ build_check ()
 			test $level != 2 -o $arrlvl != 1 || id=$token
 			;;
 		[\,\;])
-			if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
+			if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
 			then
 				check_field $kind $1 $id "$fields"
 				test "$token" != ";" || fields= id=

Re: [PATCH v2 5/7] x86: generalize padding field handling

[Roger Pau Monné]

On Wed, Jul 01, 2020 at 12:27:37PM +0200, Jan Beulich wrote:
> The original intention was to ignore padding fields, but the pattern
> matched only ones whose names started with an underscore. Also match
> fields whose names are in line with the C spec by not having a leading
> underscore. (Note that the leading ^ in the sed regexps was pointless
> and hence get dropped.)
> 
> This requires adjusting some vNUMA macros, to avoid triggering
> "enumeration value ... not handled in switch" warnings, which - due to
> -Werror - would cause the build to fail. (I have to admit that I find
> these padding fields odd, when translation of the containing structure
> is needed anyway.)
> 
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

> ---
> While for translation macros skipping padding fields pretty surely is a
> reasonable thing to do, we may want to consider not ignoring them when
> generating checking macros.
> 
> --- a/xen/common/compat/memory.c
> +++ b/xen/common/compat/memory.c
> @@ -354,10 +354,13 @@ int compat_memory_op(unsigned int cmd, X
>                  return -EFAULT;
>  
>  #define XLAT_vnuma_topology_info_HNDL_vdistance_h(_d_, _s_)		\
> +            case XLAT_vnuma_topology_info_vdistance_pad:                \
>              guest_from_compat_handle((_d_)->vdistance.h, (_s_)->vdistance.h)
>  #define XLAT_vnuma_topology_info_HNDL_vcpu_to_vnode_h(_d_, _s_)		\
> +            case XLAT_vnuma_topology_info_vcpu_to_vnode_pad:            \
>              guest_from_compat_handle((_d_)->vcpu_to_vnode.h, (_s_)->vcpu_to_vnode.h)
>  #define XLAT_vnuma_topology_info_HNDL_vmemrange_h(_d_, _s_)		\
> +            case XLAT_vnuma_topology_info_vmemrange_pad:                \
>              guest_from_compat_handle((_d_)->vmemrange.h, (_s_)->vmemrange.h)

I find this quite ugly, would it be better to just handle them with a
default case in the XLAT_ macros?

AFAICT it will also set (_d_)->vmemrange.h twice?

>  
>              XLAT_vnuma_topology_info(nat.vnuma, &cmp.vnuma);
> --- a/xen/tools/get-fields.sh
> +++ b/xen/tools/get-fields.sh
> @@ -218,7 +218,7 @@ for line in sys.stdin.readlines():
>  				fi
>  				;;
>  			[\,\;])
> -				if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
> +				if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
>  				then
>  					if [ $kind = union ]
>  					then
> @@ -347,7 +347,7 @@ build_body ()
>  			fi
>  			;;
>  		[\,\;])
> -			if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
> +			if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
>  			then
>  				if [ -z "$array" -a -z "$array_type" ]
>  				then
> @@ -437,7 +437,7 @@ check_field ()
>  				id=$token
>  				;;
>  			[\,\;])
> -				if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
> +				if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
>  				then
>  					check_field $1 $2 $3.$id "$fields"
>  					test "$token" != ";" || fields= id=
> @@ -491,7 +491,7 @@ build_check ()
>  			test $level != 2 -o $arrlvl != 1 || id=$token
>  			;;
>  		[\,\;])
> -			if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
> +			if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
>  			then
>  				check_field $kind $1 $id "$fields"
>  				test "$token" != ";" || fields= id=

I have to admit I'm not overly happy with this level of repetition
(not that you introduce it here), but I would prefer to have the
regexp in a single place if possible, it's easy to miss instances
IMO.

Thanks.

Re: [PATCH v2 5/7] x86: generalize padding field handling

[Jan Beulich]

On 14.07.2020 16:29, Roger Pau Monné wrote:
> On Wed, Jul 01, 2020 at 12:27:37PM +0200, Jan Beulich wrote:
>> The original intention was to ignore padding fields, but the pattern
>> matched only ones whose names started with an underscore. Also match
>> fields whose names are in line with the C spec by not having a leading
>> underscore. (Note that the leading ^ in the sed regexps was pointless
>> and hence get dropped.)
>>
>> This requires adjusting some vNUMA macros, to avoid triggering
>> "enumeration value ... not handled in switch" warnings, which - due to
>> -Werror - would cause the build to fail. (I have to admit that I find
>> these padding fields odd, when translation of the containing structure
>> is needed anyway.)
>>
>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> 
> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks.

>> ---
>> While for translation macros skipping padding fields pretty surely is a
>> reasonable thing to do, we may want to consider not ignoring them when
>> generating checking macros.

(note this remark, towards your question at the end)

>> --- a/xen/common/compat/memory.c
>> +++ b/xen/common/compat/memory.c
>> @@ -354,10 +354,13 @@ int compat_memory_op(unsigned int cmd, X
>>                  return -EFAULT;
>>  
>>  #define XLAT_vnuma_topology_info_HNDL_vdistance_h(_d_, _s_)		\
>> +            case XLAT_vnuma_topology_info_vdistance_pad:                \
>>              guest_from_compat_handle((_d_)->vdistance.h, (_s_)->vdistance.h)
>>  #define XLAT_vnuma_topology_info_HNDL_vcpu_to_vnode_h(_d_, _s_)		\
>> +            case XLAT_vnuma_topology_info_vcpu_to_vnode_pad:            \
>>              guest_from_compat_handle((_d_)->vcpu_to_vnode.h, (_s_)->vcpu_to_vnode.h)
>>  #define XLAT_vnuma_topology_info_HNDL_vmemrange_h(_d_, _s_)		\
>> +            case XLAT_vnuma_topology_info_vmemrange_pad:                \
>>              guest_from_compat_handle((_d_)->vmemrange.h, (_s_)->vmemrange.h)
> 
> I find this quite ugly, would it be better to just handle them with a
> default case in the XLAT_ macros?

Default cases explicitly do not get added to be able to spot missing
case labels, as most compilers will warn about such when the controlling
expression is of enum type.

> AFAICT it will also set (_d_)->vmemrange.h twice?

I'm not seeing it (and if it was, I'd then also wonder why not for the
other two handles above). This is the generated macro:

#define XLAT_vnuma_topology_info(_d_, _s_) do { \
    (_d_)->domid = (_s_)->domid; \
    (_d_)->nr_vnodes = (_s_)->nr_vnodes; \
    (_d_)->nr_vcpus = (_s_)->nr_vcpus; \
    (_d_)->nr_vmemranges = (_s_)->nr_vmemranges; \
    switch (vdistance) { \
    case XLAT_vnuma_topology_info_vdistance_h: \
        XLAT_vnuma_topology_info_HNDL_vdistance_h(_d_, _s_); \
        break; \
    } \
    switch (vcpu_to_vnode) { \
    case XLAT_vnuma_topology_info_vcpu_to_vnode_h: \
        XLAT_vnuma_topology_info_HNDL_vcpu_to_vnode_h(_d_, _s_); \
        break; \
    } \
    switch (vmemrange) { \
    case XLAT_vnuma_topology_info_vmemrange_h: \
        XLAT_vnuma_topology_info_HNDL_vmemrange_h(_d_, _s_); \
        break; \
    } \
} while (0)

Am I overlooking any further aspect?

>> --- a/xen/tools/get-fields.sh
>> +++ b/xen/tools/get-fields.sh
>> @@ -218,7 +218,7 @@ for line in sys.stdin.readlines():
>>  				fi
>>  				;;
>>  			[\,\;])
>> -				if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
>> +				if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
>>  				then
>>  					if [ $kind = union ]
>>  					then
>> @@ -347,7 +347,7 @@ build_body ()
>>  			fi
>>  			;;
>>  		[\,\;])
>> -			if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
>> +			if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
>>  			then
>>  				if [ -z "$array" -a -z "$array_type" ]
>>  				then
>> @@ -437,7 +437,7 @@ check_field ()
>>  				id=$token
>>  				;;
>>  			[\,\;])
>> -				if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
>> +				if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
>>  				then
>>  					check_field $1 $2 $3.$id "$fields"
>>  					test "$token" != ";" || fields= id=
>> @@ -491,7 +491,7 @@ build_check ()
>>  			test $level != 2 -o $arrlvl != 1 || id=$token
>>  			;;
>>  		[\,\;])
>> -			if [ $level = 2 -a -n "$(echo $id | $SED 's,^_pad[[:digit:]]*,,')" ]
>> +			if [ $level = 2 -a -n "$(echo $id | $SED 's,_\?pad[[:digit:]]*,,')" ]
>>  			then
>>  				check_field $kind $1 $id "$fields"
>>  				test "$token" != ";" || fields= id=
> 
> I have to admit I'm not overly happy with this level of repetition
> (not that you introduce it here), but I would prefer to have the
> regexp in a single place if possible, it's easy to miss instances
> IMO.

I too thought so while making the changes, but besides viewing this
as an orthogonal adjustment I'm also, as per the remark further up,
unconvinced the expressions actually want to be the same between
the checking macros and the xlat ones.

Jan

Re: [PATCH v2 5/7] x86: generalize padding field handling

[Roger Pau Monné]

On Wed, Jul 15, 2020 at 08:36:10AM +0200, Jan Beulich wrote:
> On 14.07.2020 16:29, Roger Pau Monné wrote:
> > On Wed, Jul 01, 2020 at 12:27:37PM +0200, Jan Beulich wrote:
> >> The original intention was to ignore padding fields, but the pattern
> >> matched only ones whose names started with an underscore. Also match
> >> fields whose names are in line with the C spec by not having a leading
> >> underscore. (Note that the leading ^ in the sed regexps was pointless
> >> and hence get dropped.)
> >>
> >> This requires adjusting some vNUMA macros, to avoid triggering
> >> "enumeration value ... not handled in switch" warnings, which - due to
> >> -Werror - would cause the build to fail. (I have to admit that I find
> >> these padding fields odd, when translation of the containing structure
> >> is needed anyway.)
> >>
> >> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> > 
> > Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>
> 
> Thanks.
> 
> >> ---
> >> While for translation macros skipping padding fields pretty surely is a
> >> reasonable thing to do, we may want to consider not ignoring them when
> >> generating checking macros.
> 
> (note this remark, towards your question at the end)
> 
> >> --- a/xen/common/compat/memory.c
> >> +++ b/xen/common/compat/memory.c
> >> @@ -354,10 +354,13 @@ int compat_memory_op(unsigned int cmd, X
> >>                  return -EFAULT;
> >>  
> >>  #define XLAT_vnuma_topology_info_HNDL_vdistance_h(_d_, _s_)		\
> >> +            case XLAT_vnuma_topology_info_vdistance_pad:                \
> >>              guest_from_compat_handle((_d_)->vdistance.h, (_s_)->vdistance.h)
> >>  #define XLAT_vnuma_topology_info_HNDL_vcpu_to_vnode_h(_d_, _s_)		\
> >> +            case XLAT_vnuma_topology_info_vcpu_to_vnode_pad:            \
> >>              guest_from_compat_handle((_d_)->vcpu_to_vnode.h, (_s_)->vcpu_to_vnode.h)
> >>  #define XLAT_vnuma_topology_info_HNDL_vmemrange_h(_d_, _s_)		\
> >> +            case XLAT_vnuma_topology_info_vmemrange_pad:                \
> >>              guest_from_compat_handle((_d_)->vmemrange.h, (_s_)->vmemrange.h)
> > 
> > I find this quite ugly, would it be better to just handle them with a
> > default case in the XLAT_ macros?
> 
> Default cases explicitly do not get added to be able to spot missing
> case labels, as most compilers will warn about such when the controlling
> expression is of enum type.

As you say on the comment above, ignoring those for translation
macros would be better, and would avoid the ugliness of having to add
the _pad cases here.

> > AFAICT it will also set (_d_)->vmemrange.h twice?
> 
> I'm not seeing it (and if it was, I'd then also wonder why not for the
> other two handles above). This is the generated macro:
> 
> #define XLAT_vnuma_topology_info(_d_, _s_) do { \
>     (_d_)->domid = (_s_)->domid; \
>     (_d_)->nr_vnodes = (_s_)->nr_vnodes; \
>     (_d_)->nr_vcpus = (_s_)->nr_vcpus; \
>     (_d_)->nr_vmemranges = (_s_)->nr_vmemranges; \
>     switch (vdistance) { \
>     case XLAT_vnuma_topology_info_vdistance_h: \
>         XLAT_vnuma_topology_info_HNDL_vdistance_h(_d_, _s_); \
>         break; \
>     } \
>     switch (vcpu_to_vnode) { \
>     case XLAT_vnuma_topology_info_vcpu_to_vnode_h: \
>         XLAT_vnuma_topology_info_HNDL_vcpu_to_vnode_h(_d_, _s_); \
>         break; \
>     } \
>     switch (vmemrange) { \
>     case XLAT_vnuma_topology_info_vmemrange_h: \
>         XLAT_vnuma_topology_info_HNDL_vmemrange_h(_d_, _s_); \
>         break; \
>     } \
> } while (0)
> 
> Am I overlooking any further aspect?

No, vdistance, vcpu_to_vnode and vmemrange are set by the caller, so
the enums will never have the _pad value, and hence the assignation
will be done only once, you are right.

Thanks, Roger.

Re: [PATCH v2 5/7] x86: generalize padding field handling

[Jan Beulich]

On 15.07.2020 10:34, Roger Pau Monné wrote:
> On Wed, Jul 15, 2020 at 08:36:10AM +0200, Jan Beulich wrote:
>> On 14.07.2020 16:29, Roger Pau Monné wrote:
>>> On Wed, Jul 01, 2020 at 12:27:37PM +0200, Jan Beulich wrote:
>>>> --- a/xen/common/compat/memory.c
>>>> +++ b/xen/common/compat/memory.c
>>>> @@ -354,10 +354,13 @@ int compat_memory_op(unsigned int cmd, X
>>>>                  return -EFAULT;
>>>>  
>>>>  #define XLAT_vnuma_topology_info_HNDL_vdistance_h(_d_, _s_)		\
>>>> +            case XLAT_vnuma_topology_info_vdistance_pad:                \
>>>>              guest_from_compat_handle((_d_)->vdistance.h, (_s_)->vdistance.h)
>>>>  #define XLAT_vnuma_topology_info_HNDL_vcpu_to_vnode_h(_d_, _s_)		\
>>>> +            case XLAT_vnuma_topology_info_vcpu_to_vnode_pad:            \
>>>>              guest_from_compat_handle((_d_)->vcpu_to_vnode.h, (_s_)->vcpu_to_vnode.h)
>>>>  #define XLAT_vnuma_topology_info_HNDL_vmemrange_h(_d_, _s_)		\
>>>> +            case XLAT_vnuma_topology_info_vmemrange_pad:                \
>>>>              guest_from_compat_handle((_d_)->vmemrange.h, (_s_)->vmemrange.h)
>>>
>>> I find this quite ugly, would it be better to just handle them with a
>>> default case in the XLAT_ macros?
>>
>> Default cases explicitly do not get added to be able to spot missing
>> case labels, as most compilers will warn about such when the controlling
>> expression is of enum type.
> 
> As you say on the comment above, ignoring those for translation
> macros would be better, and would avoid the ugliness of having to add
> the _pad cases here.

Ah, yes, if the supposed adjustment would also suppress the generation
of respective enumerators.

Jan

[PATCH v2 6/7] flask: drop dead compat translation code

[Jan Beulich]

Translation macros aren't needed at all (or else a devicetree_label
entry would have been missing), and userlist has been removed quite some
time ago.

No functional change.

Signed-off-by: Jan Beulich <jbeulich@suse.com>

--- a/xen/include/xlat.lst
+++ b/xen/include/xlat.lst
@@ -148,14 +148,11 @@
 ?	xenoprof_init			xenoprof.h
 ?	xenoprof_passive		xenoprof.h
 ?	flask_access			xsm/flask_op.h
-!	flask_boolean			xsm/flask_op.h
 ?	flask_cache_stats		xsm/flask_op.h
 ?	flask_hash_stats		xsm/flask_op.h
-!	flask_load			xsm/flask_op.h
 ?	flask_ocontext			xsm/flask_op.h
 ?	flask_peersid			xsm/flask_op.h
 ?	flask_relabel			xsm/flask_op.h
 ?	flask_setavc_threshold		xsm/flask_op.h
 ?	flask_setenforce		xsm/flask_op.h
-!	flask_sid_context		xsm/flask_op.h
 ?	flask_transition		xsm/flask_op.h
--- a/xen/xsm/flask/flask_op.c
+++ b/xen/xsm/flask/flask_op.c
@@ -790,8 +790,6 @@ CHECK_flask_transition;
 #define xen_flask_load compat_flask_load
 #define flask_security_load compat_security_load
 
-#define xen_flask_userlist compat_flask_userlist
-
 #define xen_flask_sid_context compat_flask_sid_context
 #define flask_security_context compat_security_context
 #define flask_security_sid compat_security_sid

Re: [PATCH v2 6/7] flask: drop dead compat translation code

[Roger Pau Monné]

On Wed, Jul 01, 2020 at 12:28:07PM +0200, Jan Beulich wrote:
> Translation macros aren't needed at all (or else a devicetree_label
> entry would have been missing), and userlist has been removed quite some
> time ago.
> 
> No functional change.
> 
> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> 
> --- a/xen/include/xlat.lst
> +++ b/xen/include/xlat.lst
> @@ -148,14 +148,11 @@
>  ?	xenoprof_init			xenoprof.h
>  ?	xenoprof_passive		xenoprof.h
>  ?	flask_access			xsm/flask_op.h
> -!	flask_boolean			xsm/flask_op.h
>  ?	flask_cache_stats		xsm/flask_op.h
>  ?	flask_hash_stats		xsm/flask_op.h
> -!	flask_load			xsm/flask_op.h
>  ?	flask_ocontext			xsm/flask_op.h
>  ?	flask_peersid			xsm/flask_op.h
>  ?	flask_relabel			xsm/flask_op.h
>  ?	flask_setavc_threshold		xsm/flask_op.h
>  ?	flask_setenforce		xsm/flask_op.h
> -!	flask_sid_context		xsm/flask_op.h
>  ?	flask_transition		xsm/flask_op.h

Shouldn't those become checks then?

Sorry I realize my knowledge of all this compat stuff is very poor.

Thanks.

Re: [PATCH v2 6/7] flask: drop dead compat translation code

[Jan Beulich]

On 14.07.2020 16:58, Roger Pau Monné wrote:
> On Wed, Jul 01, 2020 at 12:28:07PM +0200, Jan Beulich wrote:
>> Translation macros aren't needed at all (or else a devicetree_label
>> entry would have been missing), and userlist has been removed quite some
>> time ago.
>>
>> No functional change.
>>
>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
>>
>> --- a/xen/include/xlat.lst
>> +++ b/xen/include/xlat.lst
>> @@ -148,14 +148,11 @@
>>  ?	xenoprof_init			xenoprof.h
>>  ?	xenoprof_passive		xenoprof.h
>>  ?	flask_access			xsm/flask_op.h
>> -!	flask_boolean			xsm/flask_op.h
>>  ?	flask_cache_stats		xsm/flask_op.h
>>  ?	flask_hash_stats		xsm/flask_op.h
>> -!	flask_load			xsm/flask_op.h
>>  ?	flask_ocontext			xsm/flask_op.h
>>  ?	flask_peersid			xsm/flask_op.h
>>  ?	flask_relabel			xsm/flask_op.h
>>  ?	flask_setavc_threshold		xsm/flask_op.h
>>  ?	flask_setenforce		xsm/flask_op.h
>> -!	flask_sid_context		xsm/flask_op.h
>>  ?	flask_transition		xsm/flask_op.h
> 
> Shouldn't those become checks then?

No, checking will never succeed for structures containing
XEN_GUEST_HANDLE(). But there's no point in generating xlat macros
when they're never used. There are two fundamentally different
strategies for handling the compat hypercalls: One is to wrap a
translation layer around the native hypercall. That's where the
xlat macros come into play. The other, used here, is to compile
the entire hypercall function a second time, arranging for the
compat structures to get used in place of the native ones. There
are no xlat macros involved here, all that's needed are correctly
translated structures. (For completeness, x86's MCA hypercall
uses yet another, quite adhoc strategy for handling, but also not
involving any xlat macro use. Hence the consideration there to
possibly drop the respective lines from the file here.)

Jan

Re: [PATCH v2 6/7] flask: drop dead compat translation code

[Roger Pau Monné]

On Wed, Jul 15, 2020 at 08:42:44AM +0200, Jan Beulich wrote:
> On 14.07.2020 16:58, Roger Pau Monné wrote:
> > On Wed, Jul 01, 2020 at 12:28:07PM +0200, Jan Beulich wrote:
> >> Translation macros aren't needed at all (or else a devicetree_label
> >> entry would have been missing), and userlist has been removed quite some
> >> time ago.
> >>
> >> No functional change.
> >>
> >> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> >>
> >> --- a/xen/include/xlat.lst
> >> +++ b/xen/include/xlat.lst
> >> @@ -148,14 +148,11 @@
> >>  ?	xenoprof_init			xenoprof.h
> >>  ?	xenoprof_passive		xenoprof.h
> >>  ?	flask_access			xsm/flask_op.h
> >> -!	flask_boolean			xsm/flask_op.h
> >>  ?	flask_cache_stats		xsm/flask_op.h
> >>  ?	flask_hash_stats		xsm/flask_op.h
> >> -!	flask_load			xsm/flask_op.h
> >>  ?	flask_ocontext			xsm/flask_op.h
> >>  ?	flask_peersid			xsm/flask_op.h
> >>  ?	flask_relabel			xsm/flask_op.h
> >>  ?	flask_setavc_threshold		xsm/flask_op.h
> >>  ?	flask_setenforce		xsm/flask_op.h
> >> -!	flask_sid_context		xsm/flask_op.h
> >>  ?	flask_transition		xsm/flask_op.h
> > 
> > Shouldn't those become checks then?
> 
> No, checking will never succeed for structures containing
> XEN_GUEST_HANDLE(). But there's no point in generating xlat macros
> when they're never used. There are two fundamentally different
> strategies for handling the compat hypercalls: One is to wrap a
> translation layer around the native hypercall. That's where the
> xlat macros come into play. The other, used here, is to compile
> the entire hypercall function a second time, arranging for the
> compat structures to get used in place of the native ones. There
> are no xlat macros involved here, all that's needed are correctly
> translated structures. (For completeness, x86's MCA hypercall
> uses yet another, quite adhoc strategy for handling, but also not
> involving any xlat macro use. Hence the consideration there to
> possibly drop the respective lines from the file here.)

Thanks, I think this explanation is helpful and I wonder whether it
would be possible to have something along this lines in a file or as a
comment somewhere, maybe at the top of xlat.lst?

Also could you add a line to the commit message noting that flask code
doesn't use any of the translation macros because it follows a
different approach to compat handling?

IMO the compat code is complicated to understand, and it also seems to
be mostly undocumented.

For the patch:

Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks.

Re: [PATCH v2 6/7] flask: drop dead compat translation code

[Jan Beulich]

On 15.07.2020 10:41, Roger Pau Monné wrote:
> On Wed, Jul 15, 2020 at 08:42:44AM +0200, Jan Beulich wrote:
>> On 14.07.2020 16:58, Roger Pau Monné wrote:
>>> On Wed, Jul 01, 2020 at 12:28:07PM +0200, Jan Beulich wrote:
>>>> Translation macros aren't needed at all (or else a devicetree_label
>>>> entry would have been missing), and userlist has been removed quite some
>>>> time ago.
>>>>
>>>> No functional change.
>>>>
>>>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
>>>>
>>>> --- a/xen/include/xlat.lst
>>>> +++ b/xen/include/xlat.lst
>>>> @@ -148,14 +148,11 @@
>>>>  ?	xenoprof_init			xenoprof.h
>>>>  ?	xenoprof_passive		xenoprof.h
>>>>  ?	flask_access			xsm/flask_op.h
>>>> -!	flask_boolean			xsm/flask_op.h
>>>>  ?	flask_cache_stats		xsm/flask_op.h
>>>>  ?	flask_hash_stats		xsm/flask_op.h
>>>> -!	flask_load			xsm/flask_op.h
>>>>  ?	flask_ocontext			xsm/flask_op.h
>>>>  ?	flask_peersid			xsm/flask_op.h
>>>>  ?	flask_relabel			xsm/flask_op.h
>>>>  ?	flask_setavc_threshold		xsm/flask_op.h
>>>>  ?	flask_setenforce		xsm/flask_op.h
>>>> -!	flask_sid_context		xsm/flask_op.h
>>>>  ?	flask_transition		xsm/flask_op.h
>>>
>>> Shouldn't those become checks then?
>>
>> No, checking will never succeed for structures containing
>> XEN_GUEST_HANDLE(). But there's no point in generating xlat macros
>> when they're never used. There are two fundamentally different
>> strategies for handling the compat hypercalls: One is to wrap a
>> translation layer around the native hypercall. That's where the
>> xlat macros come into play. The other, used here, is to compile
>> the entire hypercall function a second time, arranging for the
>> compat structures to get used in place of the native ones. There
>> are no xlat macros involved here, all that's needed are correctly
>> translated structures. (For completeness, x86's MCA hypercall
>> uses yet another, quite adhoc strategy for handling, but also not
>> involving any xlat macro use. Hence the consideration there to
>> possibly drop the respective lines from the file here.)
> 
> Thanks, I think this explanation is helpful and I wonder whether it
> would be possible to have something along this lines in a file or as a
> comment somewhere, maybe at the top of xlat.lst?

To be honest - I'm not sure: Such a comment may indeed be helpful
to have, but I don't think I can see any single good place for it
to live. For people editing xlat.lst (a file the existence of which
many aren't even aware of), this would be a good place. But how
would others have any chance of running into this comment?

> Also could you add a line to the commit message noting that flask code
> doesn't use any of the translation macros because it follows a
> different approach to compat handling?

I've made the sentence start "Translation macros aren't used (and
hence needed) at all ..." - is that enough of an adjustment?

> For the patch:
> 
> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks.

Jan

Re: [PATCH v2 6/7] flask: drop dead compat translation code

[Roger Pau Monné]

On Wed, Jul 15, 2020 at 10:52:28AM +0200, Jan Beulich wrote:
> On 15.07.2020 10:41, Roger Pau Monné wrote:
> > On Wed, Jul 15, 2020 at 08:42:44AM +0200, Jan Beulich wrote:
> >> On 14.07.2020 16:58, Roger Pau Monné wrote:
> >>> On Wed, Jul 01, 2020 at 12:28:07PM +0200, Jan Beulich wrote:
> >>>> Translation macros aren't needed at all (or else a devicetree_label
> >>>> entry would have been missing), and userlist has been removed quite some
> >>>> time ago.
> >>>>
> >>>> No functional change.
> >>>>
> >>>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> >>>>
> >>>> --- a/xen/include/xlat.lst
> >>>> +++ b/xen/include/xlat.lst
> >>>> @@ -148,14 +148,11 @@
> >>>>  ?	xenoprof_init			xenoprof.h
> >>>>  ?	xenoprof_passive		xenoprof.h
> >>>>  ?	flask_access			xsm/flask_op.h
> >>>> -!	flask_boolean			xsm/flask_op.h
> >>>>  ?	flask_cache_stats		xsm/flask_op.h
> >>>>  ?	flask_hash_stats		xsm/flask_op.h
> >>>> -!	flask_load			xsm/flask_op.h
> >>>>  ?	flask_ocontext			xsm/flask_op.h
> >>>>  ?	flask_peersid			xsm/flask_op.h
> >>>>  ?	flask_relabel			xsm/flask_op.h
> >>>>  ?	flask_setavc_threshold		xsm/flask_op.h
> >>>>  ?	flask_setenforce		xsm/flask_op.h
> >>>> -!	flask_sid_context		xsm/flask_op.h
> >>>>  ?	flask_transition		xsm/flask_op.h
> >>>
> >>> Shouldn't those become checks then?
> >>
> >> No, checking will never succeed for structures containing
> >> XEN_GUEST_HANDLE(). But there's no point in generating xlat macros
> >> when they're never used. There are two fundamentally different
> >> strategies for handling the compat hypercalls: One is to wrap a
> >> translation layer around the native hypercall. That's where the
> >> xlat macros come into play. The other, used here, is to compile
> >> the entire hypercall function a second time, arranging for the
> >> compat structures to get used in place of the native ones. There
> >> are no xlat macros involved here, all that's needed are correctly
> >> translated structures. (For completeness, x86's MCA hypercall
> >> uses yet another, quite adhoc strategy for handling, but also not
> >> involving any xlat macro use. Hence the consideration there to
> >> possibly drop the respective lines from the file here.)
> > 
> > Thanks, I think this explanation is helpful and I wonder whether it
> > would be possible to have something along this lines in a file or as a
> > comment somewhere, maybe at the top of xlat.lst?
> 
> To be honest - I'm not sure: Such a comment may indeed be helpful
> to have, but I don't think I can see any single good place for it
> to live. For people editing xlat.lst (a file the existence of which
> many aren't even aware of), this would be a good place. But how
> would others have any chance of running into this comment?

I would add it to xlat.lst rather than not adding it at all.

If we can find a better place to add it I'm all in, but as said I
would rather add it somewhere right now than just defer adding it
until the perfect placement is found.

> > Also could you add a line to the commit message noting that flask code
> > doesn't use any of the translation macros because it follows a
> > different approach to compat handling?
> 
> I've made the sentence start "Translation macros aren't used (and
> hence needed) at all ..." - is that enough of an adjustment?

Yes, that's fine.

Thanks.

[PATCH v2 7/7] x86: only generate compat headers actually needed

[Jan Beulich]

As was already the case for XSM/Flask, avoid generating compat headers
when they're not going to be needed. To address resulting build issues
- move compat/hvm/dm_op.h inclusion to the only source file needing it,
- add a little bit of #ifdef-ary.

Signed-off-by: Jan Beulich <jbeulich@suse.com>
---
Alternatively we could consistently drop conditionals (except for per-
arch cases perhaps).

--- a/xen/arch/x86/hvm/dm.c
+++ b/xen/arch/x86/hvm/dm.c
@@ -717,6 +717,8 @@ static int dm_op(const struct dmop_args
     return rc;
 }
 
+#include <compat/hvm/dm_op.h>
+
 CHECK_dm_op_create_ioreq_server;
 CHECK_dm_op_get_ioreq_server_info;
 CHECK_dm_op_ioreq_server_range;
--- a/xen/common/compat/domain.c
+++ b/xen/common/compat/domain.c
@@ -11,7 +11,6 @@ EMIT_FILE;
 #include <xen/guest_access.h>
 #include <xen/hypercall.h>
 #include <compat/vcpu.h>
-#include <compat/hvm/hvm_vcpu.h>
 
 #define xen_vcpu_set_periodic_timer vcpu_set_periodic_timer
 CHECK_vcpu_set_periodic_timer;
@@ -25,6 +24,10 @@ CHECK_SIZE_(struct, vcpu_info);
 CHECK_vcpu_register_vcpu_info;
 #undef xen_vcpu_register_vcpu_info
 
+#ifdef CONFIG_HVM
+
+#include <compat/hvm/hvm_vcpu.h>
+
 #define xen_vcpu_hvm_context vcpu_hvm_context
 #define xen_vcpu_hvm_x86_32 vcpu_hvm_x86_32
 #define xen_vcpu_hvm_x86_64 vcpu_hvm_x86_64
@@ -33,6 +36,8 @@ CHECK_vcpu_hvm_context;
 #undef xen_vcpu_hvm_x86_32
 #undef xen_vcpu_hvm_context
 
+#endif
+
 int compat_vcpu_op(int cmd, unsigned int vcpuid, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
     struct domain *d = current->domain;
@@ -49,6 +54,7 @@ int compat_vcpu_op(int cmd, unsigned int
         if ( v->vcpu_info == &dummy_vcpu_info )
             return -EINVAL;
 
+#ifdef CONFIG_HVM
         if ( is_hvm_vcpu(v) )
         {
             struct vcpu_hvm_context ctxt;
@@ -61,6 +67,7 @@ int compat_vcpu_op(int cmd, unsigned int
             domain_unlock(d);
         }
         else
+#endif
         {
             struct compat_vcpu_guest_context *ctxt;
 
--- a/xen/include/Makefile
+++ b/xen/include/Makefile
@@ -3,32 +3,34 @@ ifneq ($(CONFIG_COMPAT),)
 compat-arch-$(CONFIG_X86) := x86_32
 
 headers-y := \
-    compat/argo.h \
-    compat/callback.h \
+    compat/arch-$(compat-arch-y).h \
     compat/elfnote.h \
     compat/event_channel.h \
     compat/features.h \
-    compat/grant_table.h \
-    compat/hypfs.h \
-    compat/kexec.h \
     compat/memory.h \
     compat/nmi.h \
     compat/physdev.h \
     compat/platform.h \
+    compat/pmu.h \
     compat/sched.h \
-    compat/trace.h \
     compat/vcpu.h \
     compat/version.h \
     compat/xen.h \
-    compat/xenoprof.h
+    compat/xlat.h
 headers-$(CONFIG_X86)     += compat/arch-x86/pmu.h
 headers-$(CONFIG_X86)     += compat/arch-x86/xen-mca.h
 headers-$(CONFIG_X86)     += compat/arch-x86/xen.h
 headers-$(CONFIG_X86)     += compat/arch-x86/xen-$(compat-arch-y).h
-headers-$(CONFIG_X86)     += compat/hvm/dm_op.h
-headers-$(CONFIG_X86)     += compat/hvm/hvm_op.h
-headers-$(CONFIG_X86)     += compat/hvm/hvm_vcpu.h
-headers-y                 += compat/arch-$(compat-arch-y).h compat/pmu.h compat/xlat.h
+headers-$(CONFIG_ARGO)    += compat/argo.h
+headers-$(CONFIG_PV)      += compat/callback.h
+headers-$(CONFIG_GRANT_TABLE) += compat/grant_table.h
+headers-$(CONFIG_HVM)     += compat/hvm/dm_op.h
+headers-$(CONFIG_HVM)     += compat/hvm/hvm_op.h
+headers-$(CONFIG_HVM)     += compat/hvm/hvm_vcpu.h
+headers-$(CONFIG_HYPFS)   += compat/hypfs.h
+headers-$(CONFIG_KEXEC)   += compat/kexec.h
+headers-$(CONFIG_TRACEBUFFER) += compat/trace.h
+headers-$(CONFIG_XENOPROF) += compat/xenoprof.h
 headers-$(CONFIG_XSM_FLASK) += compat/xsm/flask_op.h
 
 cppflags-y                := -include public/xen-compat.h -DXEN_GENERATING_COMPAT_HEADERS
--- a/xen/include/xen/hypercall.h
+++ b/xen/include/xen/hypercall.h
@@ -216,8 +216,6 @@ extern long compat_argo_op(
     unsigned long arg4);
 #endif
 
-#include <compat/hvm/dm_op.h>
-
 extern int
 compat_dm_op(
     domid_t domid,

Re: [PATCH v2 7/7] x86: only generate compat headers actually needed

[Roger Pau Monné]

On Wed, Jul 01, 2020 at 12:28:27PM +0200, Jan Beulich wrote:
> As was already the case for XSM/Flask, avoid generating compat headers
> when they're not going to be needed. To address resulting build issues
> - move compat/hvm/dm_op.h inclusion to the only source file needing it,
> - add a little bit of #ifdef-ary.
> 
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

TBH I wouldn't mind generating the compat headers even when not
actually used by the build, sometimes is useful to have them for
review context without having to play with the build options.

Thanks.

Re: [PATCH v2 7/7] x86: only generate compat headers actually needed

[Jan Beulich]

On 14.07.2020 17:03, Roger Pau Monné wrote:
> On Wed, Jul 01, 2020 at 12:28:27PM +0200, Jan Beulich wrote:
>> As was already the case for XSM/Flask, avoid generating compat headers
>> when they're not going to be needed. To address resulting build issues
>> - move compat/hvm/dm_op.h inclusion to the only source file needing it,
>> - add a little bit of #ifdef-ary.
>>
>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> 
> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks.

> TBH I wouldn't mind generating the compat headers even when not
> actually used by the build, sometimes is useful to have them for
> review context without having to play with the build options.

Right, that's what the post-commit-message remark says. The main
goal is to be consistent in what we do. The primary reason for
me to have chosen this route is that the compat header generation
isn't really quick, compared to the rest of the build process. It
is so slow that commit 7e9009891688 ("include: parallelize
compat/xlat.h generation") was a significant (i.e. very
noticeable) win.

Jan

RE: [PATCH v2 0/7] x86: compat header generation and checking adjustments

[Paul Durrant]

> -----Original Message-----
> From: Jan Beulich <jbeulich@suse.com>
> Sent: 01 July 2020 11:23
> To: xen-devel@lists.xenproject.org
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>; George Dunlap <George.Dunlap@eu.citrix.com>; Ian
> Jackson <ian.jackson@citrix.com>; Julien Grall <julien@xen.org>; Wei Liu <wl@xen.org>; Stefano
> Stabellini <sstabellini@kernel.org>; Roger Pau Monné <roger.pau@citrix.com>; Paul Durrant
> <paul@xen.org>
> Subject: [PATCH v2 0/7] x86: compat header generation and checking adjustments
> 
> As was pointed out by 0e2e54966af5 ("mm: fix public declaration of
> struct xen_mem_acquire_resource"), we're not currently handling structs
> correctly that have uint64_aligned_t fields. Patch 2 demonstrates that
> there was also an issue with XEN_GUEST_HANDLE_64().
> 
> Only the 1st patch was previously sent, but the approach chosen has
> been changed altogether. All later patches are new. For 4.14 I think
> at least patch 1 should be considered.

It's now quite a large patch. Since xen_mem_acquire_resouce() has been fixed, patch #1 (as you say in the comment there) is addressing a latent issue and so I’d prefer not to take what is now quite a large patch into 4.14.

  Paul

> 
> 1: x86: fix compat header generation
> 2: x86/mce: add compat struct checking for XEN_MC_inject_v2
> 3: x86/mce: bring hypercall subop compat checking in sync again
> 4: x86/dmop: add compat struct checking for XEN_DMOP_map_mem_type_to_ioreq_server
> 5: x86: generalize padding field handling
> 6: flask: drop dead compat translation code
> 7: x86: only generate compat headers actually needed
> 
> Jan

Re: [PATCH v2 0/7] x86: compat header generation and checking adjustments

[Jan Beulich]

On 02.07.2020 09:34, Paul Durrant wrote:
>> -----Original Message-----
>> From: Jan Beulich <jbeulich@suse.com>
>> Sent: 01 July 2020 11:23
>> To: xen-devel@lists.xenproject.org
>> Cc: Andrew Cooper <andrew.cooper3@citrix.com>; George Dunlap <George.Dunlap@eu.citrix.com>; Ian
>> Jackson <ian.jackson@citrix.com>; Julien Grall <julien@xen.org>; Wei Liu <wl@xen.org>; Stefano
>> Stabellini <sstabellini@kernel.org>; Roger Pau Monné <roger.pau@citrix.com>; Paul Durrant
>> <paul@xen.org>
>> Subject: [PATCH v2 0/7] x86: compat header generation and checking adjustments
>>
>> As was pointed out by 0e2e54966af5 ("mm: fix public declaration of
>> struct xen_mem_acquire_resource"), we're not currently handling structs
>> correctly that have uint64_aligned_t fields. Patch 2 demonstrates that
>> there was also an issue with XEN_GUEST_HANDLE_64().
>>
>> Only the 1st patch was previously sent, but the approach chosen has
>> been changed altogether. All later patches are new. For 4.14 I think
>> at least patch 1 should be considered.
> 
> It's now quite a large patch.

Most parts being entirely mechanical, though. But still ...

> Since xen_mem_acquire_resouce() has been fixed, patch #1 (as you say
> in the comment there) is addressing a latent issue and so I’d prefer
> not to take what is now quite a large patch into 4.14.

... fair enough.

Jan