[PATCH v5 00/22] vITS save/restore

[PATCH v5 00/22] vITS save/restore

[Eric Auger]

This series specifies and implements an API aimed at saving and restoring
the state of the in-kernel emulated ITS device.

The ITS is programmed through registers and tables. Those latter
are allocated by the guest. Their base address is programmed in
registers or table entries before the ITS is enabled.

The ITS is free to use some of them to flush its internal caches. This
is likely to be used when entering low power state.

Therefore, for save/restore use case, it looks natural to use this
guest RAM allocated space to save the table related data. However,
currently, The ITS in-kernel emulated device does not use all of those
tables and for those it uses, it does not always sync them with its
cached data. Additional sync must happen for:
- the collection table
- the device table
- the per-device translation tables
- the LPI pending tables.

The LPI configation table and the command queues do not need extra
syncs.

Best Regards

Eric

Git: complete series available at
https://github.com/eauger/linux/tree/v4.11-rc6-its-mig-v5

* Testing:
- on Cavium ThunderX using a virtio-net-pci guest,
  virsh save/restore commands and virt-manager live migration.
  Tested with 1 and 2 stage device table.

History:
v4 -> v5:
- user API changes:
  - ITS table save/restore triggered through ITS KVM device
    KVM_DEV_ARM_VGIC_GRP_CTRL group, KVM_DEV_ARM_ITS_SAVE_TABLES,
    KVM_DEV_ARM_ITS_RESTORE_TABLES
  - RDIST pending table flush triggered through GICV3 KVM device
    KVM_DEV_ARM_VGIC_GRP_CTRL/KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES
- Introduce an ABI infrastructure, entry size report using this infra
- IIDR reports ABI chosen set by userspace if any
- pending table save moved to vgic-v3.c
- use optimisation in pending table save
- check target_addr when restore cte
- pending table sync called from restore_tables
- added KVM: arm64: vgic-its: Fix pending table sync
- simplify loopup_table and use kvm_read_guest()
- sort the device and ITE list on save
- add defines for shifts and masks, GIC_ENCODE_SZ macro

v3 -> v4:
- update the DTE format (ITT_addr 52 bit support, validity bit addition)
- Document ABI revision and implement check
- iidr save/restore (including a new patch for iidr user write access)
- changed locking: kvm lock + vcpu lock
- fix nb_eventid_bits mixup
- 2 new patches aiming at exposing next_segment() and lock_all_vcpus()
- rework errror handling of lookup_table functions

- I took into account all Andre's comments/suggestions except:
  - trigger the save/restore of pending tables from GICV3 KVM device
    instead of ITS KVM device
  - implement ITS flush/restore in KVM_DEV_ARM_VGIC_GRP_CTRL
    ITS group

See the ML replies for current justifications. In case other people
strongly disagree of course I will change the code.

v2 -> v3:
- fix restore ITS ITT_addr bit masking

v1 -> v2:
- rebased on Vijaya's v11
- all entries now are 8 byte large
- devid/eventid indexing for device table and ITT
- support 2 stage device table
- common infra to read indexed tables
- add cpu <-> le64 conversions
- itte renamed into ite
- do not care anymore about pending table 1st KB
  (not needed at the moment for coarse mapping)

RFC v1
- creation



Eric Auger (22):
  KVM: arm/arm64: Add ITS save/restore API documentation
  KVM: arm/arm64: Add GICV3 pending table save API documentation
  KVM: arm/arm64: vgic-its: rename itte into ite
  arm/arm64: vgic: turn vgic_find_mmio_region into public
  KVM: arm64: vgic-its: KVM_DEV_ARM_VGIC_GRP_ITS_REGS group
  KVM: arm/arm64: vgic: expose (un)lock_all_vcpus
  KVM: arm64: vgic-its: Implement vgic_its_has_attr_regs and
    attr_regs_access
  KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_creadr
  KVM: arm64: vgic-its: Introduce migration ABI infrastructure
  KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_iidr
  KVM: arm64: vgic-its: Interpret MAPD Size field and check related
    errors
  KVM: arm64: vgic-its: Interpret MAPD ITT_addr field
  KVM: arm64: vgic-its: Check the device id matches TYPER DEVBITS range
  KVM: arm64: vgic-its: KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
  KVM: arm64: vgic-its: vgic_its_alloc_ite/device
  KVM: arm64: vgic-its: Add infrastructure for table lookup
  KVM: arm64: vgic-its: Collection table save/restore
  KVM: arm64: vgic-its: vgic_its_check_id returns the entry's GPA
  KVM: arm64: vgic-its: ITT save and restore
  KVM: arm64: vgic-its: Device table save/restore
  KVM: arm64: vgic-its: Fix pending table sync
  KVM: arm64: vgic-v3: KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES

 Documentation/virtual/kvm/devices/arm-vgic-its.txt |   99 ++
 Documentation/virtual/kvm/devices/arm-vgic-v3.txt  |    6 +
 arch/arm/include/uapi/asm/kvm.h                    |    6 +-
 arch/arm64/include/uapi/asm/kvm.h                  |    6 +-
 include/kvm/arm_vgic.h                             |    3 +
 include/linux/irqchip/arm-gic-v3.h                 |   12 +
 virt/kvm/arm/vgic/vgic-its.c                       | 1126 +++++++++++++++++---
 virt/kvm/arm/vgic/vgic-kvm-device.c                |   24 +-
 virt/kvm/arm/vgic/vgic-mmio.c                      |   11 +-
 virt/kvm/arm/vgic/vgic-mmio.h                      |   14 +-
 virt/kvm/arm/vgic/vgic-v3.c                        |   54 +
 virt/kvm/arm/vgic/vgic.h                           |   24 +
 12 files changed, 1251 insertions(+), 134 deletions(-)

-- 
2.5.5

[PATCH v5 00/22] vITS save/restore

[Eric Auger]

This series specifies and implements an API aimed at saving and restoring
the state of the in-kernel emulated ITS device.

The ITS is programmed through registers and tables. Those latter
are allocated by the guest. Their base address is programmed in
registers or table entries before the ITS is enabled.

The ITS is free to use some of them to flush its internal caches. This
is likely to be used when entering low power state.

Therefore, for save/restore use case, it looks natural to use this
guest RAM allocated space to save the table related data. However,
currently, The ITS in-kernel emulated device does not use all of those
tables and for those it uses, it does not always sync them with its
cached data. Additional sync must happen for:
- the collection table
- the device table
- the per-device translation tables
- the LPI pending tables.

The LPI configation table and the command queues do not need extra
syncs.

Best Regards

Eric

Git: complete series available at
https://github.com/eauger/linux/tree/v4.11-rc6-its-mig-v5

* Testing:
- on Cavium ThunderX using a virtio-net-pci guest,
  virsh save/restore commands and virt-manager live migration.
  Tested with 1 and 2 stage device table.

History:
v4 -> v5:
- user API changes:
  - ITS table save/restore triggered through ITS KVM device
    KVM_DEV_ARM_VGIC_GRP_CTRL group, KVM_DEV_ARM_ITS_SAVE_TABLES,
    KVM_DEV_ARM_ITS_RESTORE_TABLES
  - RDIST pending table flush triggered through GICV3 KVM device
    KVM_DEV_ARM_VGIC_GRP_CTRL/KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES
- Introduce an ABI infrastructure, entry size report using this infra
- IIDR reports ABI chosen set by userspace if any
- pending table save moved to vgic-v3.c
- use optimisation in pending table save
- check target_addr when restore cte
- pending table sync called from restore_tables
- added KVM: arm64: vgic-its: Fix pending table sync
- simplify loopup_table and use kvm_read_guest()
- sort the device and ITE list on save
- add defines for shifts and masks, GIC_ENCODE_SZ macro

v3 -> v4:
- update the DTE format (ITT_addr 52 bit support, validity bit addition)
- Document ABI revision and implement check
- iidr save/restore (including a new patch for iidr user write access)
- changed locking: kvm lock + vcpu lock
- fix nb_eventid_bits mixup
- 2 new patches aiming at exposing next_segment() and lock_all_vcpus()
- rework errror handling of lookup_table functions

- I took into account all Andre's comments/suggestions except:
  - trigger the save/restore of pending tables from GICV3 KVM device
    instead of ITS KVM device
  - implement ITS flush/restore in KVM_DEV_ARM_VGIC_GRP_CTRL
    ITS group

See the ML replies for current justifications. In case other people
strongly disagree of course I will change the code.

v2 -> v3:
- fix restore ITS ITT_addr bit masking

v1 -> v2:
- rebased on Vijaya's v11
- all entries now are 8 byte large
- devid/eventid indexing for device table and ITT
- support 2 stage device table
- common infra to read indexed tables
- add cpu <-> le64 conversions
- itte renamed into ite
- do not care anymore about pending table 1st KB
  (not needed at the moment for coarse mapping)

RFC v1
- creation



Eric Auger (22):
  KVM: arm/arm64: Add ITS save/restore API documentation
  KVM: arm/arm64: Add GICV3 pending table save API documentation
  KVM: arm/arm64: vgic-its: rename itte into ite
  arm/arm64: vgic: turn vgic_find_mmio_region into public
  KVM: arm64: vgic-its: KVM_DEV_ARM_VGIC_GRP_ITS_REGS group
  KVM: arm/arm64: vgic: expose (un)lock_all_vcpus
  KVM: arm64: vgic-its: Implement vgic_its_has_attr_regs and
    attr_regs_access
  KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_creadr
  KVM: arm64: vgic-its: Introduce migration ABI infrastructure
  KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_iidr
  KVM: arm64: vgic-its: Interpret MAPD Size field and check related
    errors
  KVM: arm64: vgic-its: Interpret MAPD ITT_addr field
  KVM: arm64: vgic-its: Check the device id matches TYPER DEVBITS range
  KVM: arm64: vgic-its: KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
  KVM: arm64: vgic-its: vgic_its_alloc_ite/device
  KVM: arm64: vgic-its: Add infrastructure for table lookup
  KVM: arm64: vgic-its: Collection table save/restore
  KVM: arm64: vgic-its: vgic_its_check_id returns the entry's GPA
  KVM: arm64: vgic-its: ITT save and restore
  KVM: arm64: vgic-its: Device table save/restore
  KVM: arm64: vgic-its: Fix pending table sync
  KVM: arm64: vgic-v3: KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES

 Documentation/virtual/kvm/devices/arm-vgic-its.txt |   99 ++
 Documentation/virtual/kvm/devices/arm-vgic-v3.txt  |    6 +
 arch/arm/include/uapi/asm/kvm.h                    |    6 +-
 arch/arm64/include/uapi/asm/kvm.h                  |    6 +-
 include/kvm/arm_vgic.h                             |    3 +
 include/linux/irqchip/arm-gic-v3.h                 |   12 +
 virt/kvm/arm/vgic/vgic-its.c                       | 1126 +++++++++++++++++---
 virt/kvm/arm/vgic/vgic-kvm-device.c                |   24 +-
 virt/kvm/arm/vgic/vgic-mmio.c                      |   11 +-
 virt/kvm/arm/vgic/vgic-mmio.h                      |   14 +-
 virt/kvm/arm/vgic/vgic-v3.c                        |   54 +
 virt/kvm/arm/vgic/vgic.h                           |   24 +
 12 files changed, 1251 insertions(+), 134 deletions(-)

-- 
2.5.5

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Eric Auger]

Add description for how to access ITS registers and how to save/restore
ITS tables into/from memory.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- take into account Christoffer's comments
- pending table save on GICV3 side now

v3 -> v4:
- take into account Peter's comments:
  - typos
  - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
  - add a validity bit in DTE
  - document all fields in CTE and ITE
  - document ABI revision
- take into account Andre's comments:
  - document restrictions about GITS_CREADR writing and GITS_IIDR
  - document -EBUSY error if one or more VCPUS are runnning
  - document 64b registers only can be accessed with 64b access
- itt_addr field matches bits [51:8] of the itt_addr

v1 -> v2:
- DTE and ITE now are 8 bytes
- DTE and ITE now indexed by deviceid/eventid
- use ITE name instead of ITTE
- mentions ITT_addr matches bits [51:8] of the actual address
- mentions LE layout
---
 Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
 1 file changed, 99 insertions(+)

diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
index 6081a5b..b5f010d 100644
--- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
+++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
@@ -32,7 +32,106 @@ Groups:
     KVM_DEV_ARM_VGIC_CTRL_INIT
       request the initialization of the ITS, no additional parameter in
       kvm_device_attr.addr.
+
+    KVM_DEV_ARM_ITS_SAVE_TABLES
+      save the ITS table data into guest RAM, at the location provisioned
+      by the guest in corresponding registers/table entries.
+
+      The layout of the tables in guest memory defines an ABI. The entries
+      are laid out in little endian format as described in the last paragraph.
+
+    KVM_DEV_ARM_ITS_RESTORE_TABLES
+      restore the ITS tables from guest RAM to ITS internal structures.
+
+      The GICV3 must be restored before the ITS and all ITS registers but
+      the GITS_CTLR must be restored before restoring the ITS tables.
+
+      The GITS_IIDR read-only register must also be restored before
+      the table restore as the IIDR revision field encodes the ABI revision.
+
   Errors:
     -ENXIO:  ITS not properly configured as required prior to setting
              this attribute
     -ENOMEM: Memory shortage when allocating ITS internal data
+    -EINVAL: Inconsistent restored data
+    -EFAULT: Invalid guest ram access
+    -EBUSY:  One or more VCPUS are running
+
+  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
+  Attributes:
+      The attr field of kvm_device_attr encodes the offset of the
+      ITS register, relative to the ITS control frame base address
+      (ITS_base).
+
+      kvm_device_attr.addr points to a __u64 value whatever the width
+      of the addressed register (32/64 bits). 64 bit registers can only
+      be accessed with full length.
+
+      Writes to read-only registers are ignored by the kernel except for:
+      - GITS_READR. It needs to be restored otherwise commands in the queue
+        will be re-executed after restoring CWRITER. GITS_READR must be restored
+        before restoring the GITS_CTLR which is likely to enable the ITS.
+        Also it needs to be restored after GITS_CBASER since a write to
+        GITS_CBASER resets GITS_CREADR.
+      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
+        In the future we might implement direct injection of virtual LPIS.
+        This will require an upgrade of the table layout and an evolution of
+        the ABI. GITS_IIDR must be restored before the table restoration.
+
+      For other registers, getting or setting a register has the same
+      effect as reading/writing the register on real hardware.
+  Errors:
+    -ENXIO: Offset does not correspond to any supported register
+    -EFAULT: Invalid user pointer for attr->addr
+    -EINVAL: Offset is not 64-bit aligned
+    -EBUSY: one or more VCPUS are running
+
+ ITS Table ABI REV0:
+ -------------------
+
+ Revision 0 of the ABI only supports physical LPIs.
+
+ The device table and ITT are indexed by the deviceid and eventid,
+ respectively. The collection table is not indexed by collectionid:
+ CTE are written in the table in the order of collection creation. All
+ entries are 8 bytes.
+
+ Device Table Entry (DTE):
+
+ bits:     | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 |
+ values:   | V |   next    | ITT_addr |  Size   |
+
+ where;
+ - V indicates whether the entry is valid. If not, other fields
+   are not meaningful.
+ - next: equals to 0 if this entry is the last one; otherwise it
+   corresponds to the deviceid offset to the next DTE, capped by
+   2^14 -1.
+ - ITT_addr matches bits [51:8] of the ITT address (256B aligned).
+ - Size specifies the supported number of bits for the deviceid,
+   minus one
+
+ Collection Table Entry (CTE):
+
+ bits:     | 63| 62 ..  52  | 51 ... 16 | 15  ...   0 |
+ values:   | V |    RES0    |  RDBase   |    ICID     |
+
+ where:
+ - V indicates whether the entry is valid. If not, other fields are
+   not meaningful.
+ - RES0: reserved field with Should-Be-Zero-or-Preserved behavior.
+ - RDBase is the PE number (GICR_TYPER.Processor_Number semantic),
+ - ICID is the collection ID
+
+ Interrupt Translation Entry (ITE):
+
+ bits:     | 63 ... 48 | 47 ... 16 | 15 ... 0 |
+ values:   |    next   |   pINTID  |  ICID    |
+
+ where:
+ - next: equals to 0 if this entry is the last one; otherwise it corresponds
+   to the eventid offset to the next ITE capped by 2^16 -1.
+ - pINTID is the physical LPI ID; if zero, it means the entry is not valid
+   and other fields are not meaningful.
+ - ICID is the collection ID
+
-- 
2.5.5

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Eric Auger]

Add description for how to access ITS registers and how to save/restore
ITS tables into/from memory.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- take into account Christoffer's comments
- pending table save on GICV3 side now

v3 -> v4:
- take into account Peter's comments:
  - typos
  - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
  - add a validity bit in DTE
  - document all fields in CTE and ITE
  - document ABI revision
- take into account Andre's comments:
  - document restrictions about GITS_CREADR writing and GITS_IIDR
  - document -EBUSY error if one or more VCPUS are runnning
  - document 64b registers only can be accessed with 64b access
- itt_addr field matches bits [51:8] of the itt_addr

v1 -> v2:
- DTE and ITE now are 8 bytes
- DTE and ITE now indexed by deviceid/eventid
- use ITE name instead of ITTE
- mentions ITT_addr matches bits [51:8] of the actual address
- mentions LE layout
---
 Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
 1 file changed, 99 insertions(+)

diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
index 6081a5b..b5f010d 100644
--- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
+++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
@@ -32,7 +32,106 @@ Groups:
     KVM_DEV_ARM_VGIC_CTRL_INIT
       request the initialization of the ITS, no additional parameter in
       kvm_device_attr.addr.
+
+    KVM_DEV_ARM_ITS_SAVE_TABLES
+      save the ITS table data into guest RAM, at the location provisioned
+      by the guest in corresponding registers/table entries.
+
+      The layout of the tables in guest memory defines an ABI. The entries
+      are laid out in little endian format as described in the last paragraph.
+
+    KVM_DEV_ARM_ITS_RESTORE_TABLES
+      restore the ITS tables from guest RAM to ITS internal structures.
+
+      The GICV3 must be restored before the ITS and all ITS registers but
+      the GITS_CTLR must be restored before restoring the ITS tables.
+
+      The GITS_IIDR read-only register must also be restored before
+      the table restore as the IIDR revision field encodes the ABI revision.
+
   Errors:
     -ENXIO:  ITS not properly configured as required prior to setting
              this attribute
     -ENOMEM: Memory shortage when allocating ITS internal data
+    -EINVAL: Inconsistent restored data
+    -EFAULT: Invalid guest ram access
+    -EBUSY:  One or more VCPUS are running
+
+  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
+  Attributes:
+      The attr field of kvm_device_attr encodes the offset of the
+      ITS register, relative to the ITS control frame base address
+      (ITS_base).
+
+      kvm_device_attr.addr points to a __u64 value whatever the width
+      of the addressed register (32/64 bits). 64 bit registers can only
+      be accessed with full length.
+
+      Writes to read-only registers are ignored by the kernel except for:
+      - GITS_READR. It needs to be restored otherwise commands in the queue
+        will be re-executed after restoring CWRITER. GITS_READR must be restored
+        before restoring the GITS_CTLR which is likely to enable the ITS.
+        Also it needs to be restored after GITS_CBASER since a write to
+        GITS_CBASER resets GITS_CREADR.
+      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
+        In the future we might implement direct injection of virtual LPIS.
+        This will require an upgrade of the table layout and an evolution of
+        the ABI. GITS_IIDR must be restored before the table restoration.
+
+      For other registers, getting or setting a register has the same
+      effect as reading/writing the register on real hardware.
+  Errors:
+    -ENXIO: Offset does not correspond to any supported register
+    -EFAULT: Invalid user pointer for attr->addr
+    -EINVAL: Offset is not 64-bit aligned
+    -EBUSY: one or more VCPUS are running
+
+ ITS Table ABI REV0:
+ -------------------
+
+ Revision 0 of the ABI only supports physical LPIs.
+
+ The device table and ITT are indexed by the deviceid and eventid,
+ respectively. The collection table is not indexed by collectionid:
+ CTE are written in the table in the order of collection creation. All
+ entries are 8 bytes.
+
+ Device Table Entry (DTE):
+
+ bits:     | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 |
+ values:   | V |   next    | ITT_addr |  Size   |
+
+ where;
+ - V indicates whether the entry is valid. If not, other fields
+   are not meaningful.
+ - next: equals to 0 if this entry is the last one; otherwise it
+   corresponds to the deviceid offset to the next DTE, capped by
+   2^14 -1.
+ - ITT_addr matches bits [51:8] of the ITT address (256B aligned).
+ - Size specifies the supported number of bits for the deviceid,
+   minus one
+
+ Collection Table Entry (CTE):
+
+ bits:     | 63| 62 ..  52  | 51 ... 16 | 15  ...   0 |
+ values:   | V |    RES0    |  RDBase   |    ICID     |
+
+ where:
+ - V indicates whether the entry is valid. If not, other fields are
+   not meaningful.
+ - RES0: reserved field with Should-Be-Zero-or-Preserved behavior.
+ - RDBase is the PE number (GICR_TYPER.Processor_Number semantic),
+ - ICID is the collection ID
+
+ Interrupt Translation Entry (ITE):
+
+ bits:     | 63 ... 48 | 47 ... 16 | 15 ... 0 |
+ values:   |    next   |   pINTID  |  ICID    |
+
+ where:
+ - next: equals to 0 if this entry is the last one; otherwise it corresponds
+   to the eventid offset to the next ITE capped by 2^16 -1.
+ - pINTID is the physical LPI ID; if zero, it means the entry is not valid
+   and other fields are not meaningful.
+ - ICID is the collection ID
+
-- 
2.5.5

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Eric Auger]

Add description for how to save GICV3 LPI pending bit into
guest RAM pending tables.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v5: creation
---
 Documentation/virtual/kvm/devices/arm-vgic-v3.txt | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
index c1a2461..9293b45 100644
--- a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
+++ b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
@@ -167,11 +167,17 @@ Groups:
     KVM_DEV_ARM_VGIC_CTRL_INIT
       request the initialization of the VGIC, no additional parameter in
       kvm_device_attr.addr.
+    KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES
+      save all LPI pending bits into guest RAM pending tables.
+
+      The first kB of the pending table is not altered by this operation.
   Errors:
     -ENXIO: VGIC not properly configured as required prior to calling
      this attribute
     -ENODEV: no online VCPU
     -ENOMEM: memory shortage when allocating vgic internal data
+    -EFAULT: Invalid guest ram access
+    -EBUSY:  One or more VCPUS are running
 
 
   KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO
-- 
2.5.5

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Eric Auger]

Add description for how to save GICV3 LPI pending bit into
guest RAM pending tables.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v5: creation
---
 Documentation/virtual/kvm/devices/arm-vgic-v3.txt | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
index c1a2461..9293b45 100644
--- a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
+++ b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
@@ -167,11 +167,17 @@ Groups:
     KVM_DEV_ARM_VGIC_CTRL_INIT
       request the initialization of the VGIC, no additional parameter in
       kvm_device_attr.addr.
+    KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES
+      save all LPI pending bits into guest RAM pending tables.
+
+      The first kB of the pending table is not altered by this operation.
   Errors:
     -ENXIO: VGIC not properly configured as required prior to calling
      this attribute
     -ENODEV: no online VCPU
     -ENOMEM: memory shortage when allocating vgic internal data
+    -EFAULT: Invalid guest ram access
+    -EBUSY:  One or more VCPUS are running
 
 
   KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO
-- 
2.5.5

[PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Eric Auger]

The actual abbreviation for the interrupt translation table entry
is ITE. Let's rename all itte instances by ite.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>

---

v5: Add Marc's A-b
---
 virt/kvm/arm/vgic/vgic-its.c | 148 +++++++++++++++++++++----------------------
 1 file changed, 74 insertions(+), 74 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 8d1da1a..3ffcbbe 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -114,8 +114,8 @@ struct its_collection {
 #define its_is_collection_mapped(coll) ((coll) && \
 				((coll)->target_addr != COLLECTION_NOT_MAPPED))
 
-struct its_itte {
-	struct list_head itte_list;
+struct its_ite {
+	struct list_head ite_list;
 
 	struct vgic_irq *irq;
 	struct its_collection *collection;
@@ -143,27 +143,27 @@ static struct its_device *find_its_device(struct vgic_its *its, u32 device_id)
  * Device ID/Event ID pair on an ITS.
  * Must be called with the its_lock mutex held.
  */
-static struct its_itte *find_itte(struct vgic_its *its, u32 device_id,
+static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
 				  u32 event_id)
 {
 	struct its_device *device;
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 	device = find_its_device(its, device_id);
 	if (device == NULL)
 		return NULL;
 
-	list_for_each_entry(itte, &device->itt_head, itte_list)
-		if (itte->event_id == event_id)
-			return itte;
+	list_for_each_entry(ite, &device->itt_head, ite_list)
+		if (ite->event_id == event_id)
+			return ite;
 
 	return NULL;
 }
 
 /* To be used as an iterator this macro misses the enclosing parentheses */
-#define for_each_lpi_its(dev, itte, its) \
+#define for_each_lpi_its(dev, ite, its) \
 	list_for_each_entry(dev, &(its)->device_list, dev_list) \
-		list_for_each_entry(itte, &(dev)->itt_head, itte_list)
+		list_for_each_entry(ite, &(dev)->itt_head, ite_list)
 
 /*
  * We only implement 48 bits of PA at the moment, although the ITS
@@ -270,18 +270,18 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
  * Needs to be called whenever either the collection for a LPIs has
  * changed or the collection itself got retargeted.
  */
-static void update_affinity_itte(struct kvm *kvm, struct its_itte *itte)
+static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
 {
 	struct kvm_vcpu *vcpu;
 
-	if (!its_is_collection_mapped(itte->collection))
+	if (!its_is_collection_mapped(ite->collection))
 		return;
 
-	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
+	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
 
-	spin_lock(&itte->irq->irq_lock);
-	itte->irq->target_vcpu = vcpu;
-	spin_unlock(&itte->irq->irq_lock);
+	spin_lock(&ite->irq->irq_lock);
+	ite->irq->target_vcpu = vcpu;
+	spin_unlock(&ite->irq->irq_lock);
 }
 
 /*
@@ -292,13 +292,13 @@ static void update_affinity_collection(struct kvm *kvm, struct vgic_its *its,
 				       struct its_collection *coll)
 {
 	struct its_device *device;
-	struct its_itte *itte;
+	struct its_ite *ite;
 
-	for_each_lpi_its(device, itte, its) {
-		if (!itte->collection || coll != itte->collection)
+	for_each_lpi_its(device, ite, its) {
+		if (!ite->collection || coll != ite->collection)
 			continue;
 
-		update_affinity_itte(kvm, itte);
+		update_affinity_ite(kvm, ite);
 	}
 }
 
@@ -425,25 +425,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
 				u32 devid, u32 eventid)
 {
 	struct kvm_vcpu *vcpu;
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 	if (!its->enabled)
 		return -EBUSY;
 
-	itte = find_itte(its, devid, eventid);
-	if (!itte || !its_is_collection_mapped(itte->collection))
+	ite = find_ite(its, devid, eventid);
+	if (!ite || !its_is_collection_mapped(ite->collection))
 		return E_ITS_INT_UNMAPPED_INTERRUPT;
 
-	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
+	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
 	if (!vcpu)
 		return E_ITS_INT_UNMAPPED_INTERRUPT;
 
 	if (!vcpu->arch.vgic_cpu.lpis_enabled)
 		return -EBUSY;
 
-	spin_lock(&itte->irq->irq_lock);
-	itte->irq->pending_latch = true;
-	vgic_queue_irq_unlock(kvm, itte->irq);
+	spin_lock(&ite->irq->irq_lock);
+	ite->irq->pending_latch = true;
+	vgic_queue_irq_unlock(kvm, ite->irq);
 
 	return 0;
 }
@@ -511,15 +511,15 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
 }
 
 /* Requires the its_lock to be held. */
-static void its_free_itte(struct kvm *kvm, struct its_itte *itte)
+static void its_free_ite(struct kvm *kvm, struct its_ite *ite)
 {
-	list_del(&itte->itte_list);
+	list_del(&ite->ite_list);
 
 	/* This put matches the get in vgic_add_lpi. */
-	if (itte->irq)
-		vgic_put_irq(kvm, itte->irq);
+	if (ite->irq)
+		vgic_put_irq(kvm, ite->irq);
 
-	kfree(itte);
+	kfree(ite);
 }
 
 static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
@@ -544,17 +544,17 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
 {
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	u32 event_id = its_cmd_get_id(its_cmd);
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 
-	itte = find_itte(its, device_id, event_id);
-	if (itte && itte->collection) {
+	ite = find_ite(its, device_id, event_id);
+	if (ite && ite->collection) {
 		/*
 		 * Though the spec talks about removing the pending state, we
 		 * don't bother here since we clear the ITTE anyway and the
 		 * pending state is a property of the ITTE struct.
 		 */
-		its_free_itte(kvm, itte);
+		its_free_ite(kvm, ite);
 		return 0;
 	}
 
@@ -572,26 +572,26 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
 	u32 event_id = its_cmd_get_id(its_cmd);
 	u32 coll_id = its_cmd_get_collection(its_cmd);
 	struct kvm_vcpu *vcpu;
-	struct its_itte *itte;
+	struct its_ite *ite;
 	struct its_collection *collection;
 
-	itte = find_itte(its, device_id, event_id);
-	if (!itte)
+	ite = find_ite(its, device_id, event_id);
+	if (!ite)
 		return E_ITS_MOVI_UNMAPPED_INTERRUPT;
 
-	if (!its_is_collection_mapped(itte->collection))
+	if (!its_is_collection_mapped(ite->collection))
 		return E_ITS_MOVI_UNMAPPED_COLLECTION;
 
 	collection = find_collection(its, coll_id);
 	if (!its_is_collection_mapped(collection))
 		return E_ITS_MOVI_UNMAPPED_COLLECTION;
 
-	itte->collection = collection;
+	ite->collection = collection;
 	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
 
-	spin_lock(&itte->irq->irq_lock);
-	itte->irq->target_vcpu = vcpu;
-	spin_unlock(&itte->irq->irq_lock);
+	spin_lock(&ite->irq->irq_lock);
+	ite->irq->target_vcpu = vcpu;
+	spin_unlock(&ite->irq->irq_lock);
 
 	return 0;
 }
@@ -679,7 +679,7 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
 {
 	struct its_collection *collection;
 	struct its_device *device;
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 	/*
 	 * Clearing the mapping for that collection ID removes the
@@ -690,10 +690,10 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
 	if (!collection)
 		return;
 
-	for_each_lpi_its(device, itte, its)
-		if (itte->collection &&
-		    itte->collection->collection_id == coll_id)
-			itte->collection = NULL;
+	for_each_lpi_its(device, ite, its)
+		if (ite->collection &&
+		    ite->collection->collection_id == coll_id)
+			ite->collection = NULL;
 
 	list_del(&collection->coll_list);
 	kfree(collection);
@@ -709,7 +709,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	u32 event_id = its_cmd_get_id(its_cmd);
 	u32 coll_id = its_cmd_get_collection(its_cmd);
-	struct its_itte *itte;
+	struct its_ite *ite;
 	struct its_device *device;
 	struct its_collection *collection, *new_coll = NULL;
 	int lpi_nr;
@@ -728,7 +728,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 		return E_ITS_MAPTI_PHYSICALID_OOR;
 
 	/* If there is an existing mapping, behavior is UNPREDICTABLE. */
-	if (find_itte(its, device_id, event_id))
+	if (find_ite(its, device_id, event_id))
 		return 0;
 
 	collection = find_collection(its, coll_id);
@@ -739,36 +739,36 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 		new_coll = collection;
 	}
 
-	itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
-	if (!itte) {
+	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
+	if (!ite) {
 		if (new_coll)
 			vgic_its_free_collection(its, coll_id);
 		return -ENOMEM;
 	}
 
-	itte->event_id	= event_id;
-	list_add_tail(&itte->itte_list, &device->itt_head);
+	ite->event_id	= event_id;
+	list_add_tail(&ite->ite_list, &device->itt_head);
 
-	itte->collection = collection;
-	itte->lpi = lpi_nr;
+	ite->collection = collection;
+	ite->lpi = lpi_nr;
 
 	irq = vgic_add_lpi(kvm, lpi_nr);
 	if (IS_ERR(irq)) {
 		if (new_coll)
 			vgic_its_free_collection(its, coll_id);
-		its_free_itte(kvm, itte);
+		its_free_ite(kvm, ite);
 		return PTR_ERR(irq);
 	}
-	itte->irq = irq;
+	ite->irq = irq;
 
-	update_affinity_itte(kvm, itte);
+	update_affinity_ite(kvm, ite);
 
 	/*
 	 * We "cache" the configuration table entries in out struct vgic_irq's.
 	 * However we only have those structs for mapped IRQs, so we read in
 	 * the respective config data from memory here upon mapping the LPI.
 	 */
-	update_lpi_config(kvm, itte->irq, NULL);
+	update_lpi_config(kvm, ite->irq, NULL);
 
 	return 0;
 }
@@ -776,15 +776,15 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 /* Requires the its_lock to be held. */
 static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
 {
-	struct its_itte *itte, *temp;
+	struct its_ite *ite, *temp;
 
 	/*
 	 * The spec says that unmapping a device with still valid
 	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
 	 * since we cannot leave the memory unreferenced.
 	 */
-	list_for_each_entry_safe(itte, temp, &device->itt_head, itte_list)
-		its_free_itte(kvm, itte);
+	list_for_each_entry_safe(ite, temp, &device->itt_head, ite_list)
+		its_free_ite(kvm, ite);
 
 	list_del(&device->dev_list);
 	kfree(device);
@@ -883,14 +883,14 @@ static int vgic_its_cmd_handle_clear(struct kvm *kvm, struct vgic_its *its,
 {
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	u32 event_id = its_cmd_get_id(its_cmd);
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 
-	itte = find_itte(its, device_id, event_id);
-	if (!itte)
+	ite = find_ite(its, device_id, event_id);
+	if (!ite)
 		return E_ITS_CLEAR_UNMAPPED_INTERRUPT;
 
-	itte->irq->pending_latch = false;
+	ite->irq->pending_latch = false;
 
 	return 0;
 }
@@ -904,14 +904,14 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,
 {
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	u32 event_id = its_cmd_get_id(its_cmd);
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 
-	itte = find_itte(its, device_id, event_id);
-	if (!itte)
+	ite = find_ite(its, device_id, event_id);
+	if (!ite)
 		return E_ITS_INV_UNMAPPED_INTERRUPT;
 
-	return update_lpi_config(kvm, itte->irq, NULL);
+	return update_lpi_config(kvm, ite->irq, NULL);
 }
 
 /*
@@ -1435,7 +1435,7 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
 	struct kvm *kvm = kvm_dev->kvm;
 	struct vgic_its *its = kvm_dev->private;
 	struct its_device *dev;
-	struct its_itte *itte;
+	struct its_ite *ite;
 	struct list_head *dev_cur, *dev_temp;
 	struct list_head *cur, *temp;
 
@@ -1450,8 +1450,8 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
 	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
 		dev = container_of(dev_cur, struct its_device, dev_list);
 		list_for_each_safe(cur, temp, &dev->itt_head) {
-			itte = (container_of(cur, struct its_itte, itte_list));
-			its_free_itte(kvm, itte);
+			ite = (container_of(cur, struct its_ite, ite_list));
+			its_free_ite(kvm, ite);
 		}
 		list_del(dev_cur);
 		kfree(dev);
-- 
2.5.5

[PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Eric Auger]

The actual abbreviation for the interrupt translation table entry
is ITE. Let's rename all itte instances by ite.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>

---

v5: Add Marc's A-b
---
 virt/kvm/arm/vgic/vgic-its.c | 148 +++++++++++++++++++++----------------------
 1 file changed, 74 insertions(+), 74 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 8d1da1a..3ffcbbe 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -114,8 +114,8 @@ struct its_collection {
 #define its_is_collection_mapped(coll) ((coll) && \
 				((coll)->target_addr != COLLECTION_NOT_MAPPED))
 
-struct its_itte {
-	struct list_head itte_list;
+struct its_ite {
+	struct list_head ite_list;
 
 	struct vgic_irq *irq;
 	struct its_collection *collection;
@@ -143,27 +143,27 @@ static struct its_device *find_its_device(struct vgic_its *its, u32 device_id)
  * Device ID/Event ID pair on an ITS.
  * Must be called with the its_lock mutex held.
  */
-static struct its_itte *find_itte(struct vgic_its *its, u32 device_id,
+static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
 				  u32 event_id)
 {
 	struct its_device *device;
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 	device = find_its_device(its, device_id);
 	if (device == NULL)
 		return NULL;
 
-	list_for_each_entry(itte, &device->itt_head, itte_list)
-		if (itte->event_id == event_id)
-			return itte;
+	list_for_each_entry(ite, &device->itt_head, ite_list)
+		if (ite->event_id == event_id)
+			return ite;
 
 	return NULL;
 }
 
 /* To be used as an iterator this macro misses the enclosing parentheses */
-#define for_each_lpi_its(dev, itte, its) \
+#define for_each_lpi_its(dev, ite, its) \
 	list_for_each_entry(dev, &(its)->device_list, dev_list) \
-		list_for_each_entry(itte, &(dev)->itt_head, itte_list)
+		list_for_each_entry(ite, &(dev)->itt_head, ite_list)
 
 /*
  * We only implement 48 bits of PA at the moment, although the ITS
@@ -270,18 +270,18 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
  * Needs to be called whenever either the collection for a LPIs has
  * changed or the collection itself got retargeted.
  */
-static void update_affinity_itte(struct kvm *kvm, struct its_itte *itte)
+static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
 {
 	struct kvm_vcpu *vcpu;
 
-	if (!its_is_collection_mapped(itte->collection))
+	if (!its_is_collection_mapped(ite->collection))
 		return;
 
-	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
+	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
 
-	spin_lock(&itte->irq->irq_lock);
-	itte->irq->target_vcpu = vcpu;
-	spin_unlock(&itte->irq->irq_lock);
+	spin_lock(&ite->irq->irq_lock);
+	ite->irq->target_vcpu = vcpu;
+	spin_unlock(&ite->irq->irq_lock);
 }
 
 /*
@@ -292,13 +292,13 @@ static void update_affinity_collection(struct kvm *kvm, struct vgic_its *its,
 				       struct its_collection *coll)
 {
 	struct its_device *device;
-	struct its_itte *itte;
+	struct its_ite *ite;
 
-	for_each_lpi_its(device, itte, its) {
-		if (!itte->collection || coll != itte->collection)
+	for_each_lpi_its(device, ite, its) {
+		if (!ite->collection || coll != ite->collection)
 			continue;
 
-		update_affinity_itte(kvm, itte);
+		update_affinity_ite(kvm, ite);
 	}
 }
 
@@ -425,25 +425,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
 				u32 devid, u32 eventid)
 {
 	struct kvm_vcpu *vcpu;
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 	if (!its->enabled)
 		return -EBUSY;
 
-	itte = find_itte(its, devid, eventid);
-	if (!itte || !its_is_collection_mapped(itte->collection))
+	ite = find_ite(its, devid, eventid);
+	if (!ite || !its_is_collection_mapped(ite->collection))
 		return E_ITS_INT_UNMAPPED_INTERRUPT;
 
-	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
+	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
 	if (!vcpu)
 		return E_ITS_INT_UNMAPPED_INTERRUPT;
 
 	if (!vcpu->arch.vgic_cpu.lpis_enabled)
 		return -EBUSY;
 
-	spin_lock(&itte->irq->irq_lock);
-	itte->irq->pending_latch = true;
-	vgic_queue_irq_unlock(kvm, itte->irq);
+	spin_lock(&ite->irq->irq_lock);
+	ite->irq->pending_latch = true;
+	vgic_queue_irq_unlock(kvm, ite->irq);
 
 	return 0;
 }
@@ -511,15 +511,15 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
 }
 
 /* Requires the its_lock to be held. */
-static void its_free_itte(struct kvm *kvm, struct its_itte *itte)
+static void its_free_ite(struct kvm *kvm, struct its_ite *ite)
 {
-	list_del(&itte->itte_list);
+	list_del(&ite->ite_list);
 
 	/* This put matches the get in vgic_add_lpi. */
-	if (itte->irq)
-		vgic_put_irq(kvm, itte->irq);
+	if (ite->irq)
+		vgic_put_irq(kvm, ite->irq);
 
-	kfree(itte);
+	kfree(ite);
 }
 
 static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
@@ -544,17 +544,17 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
 {
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	u32 event_id = its_cmd_get_id(its_cmd);
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 
-	itte = find_itte(its, device_id, event_id);
-	if (itte && itte->collection) {
+	ite = find_ite(its, device_id, event_id);
+	if (ite && ite->collection) {
 		/*
 		 * Though the spec talks about removing the pending state, we
 		 * don't bother here since we clear the ITTE anyway and the
 		 * pending state is a property of the ITTE struct.
 		 */
-		its_free_itte(kvm, itte);
+		its_free_ite(kvm, ite);
 		return 0;
 	}
 
@@ -572,26 +572,26 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
 	u32 event_id = its_cmd_get_id(its_cmd);
 	u32 coll_id = its_cmd_get_collection(its_cmd);
 	struct kvm_vcpu *vcpu;
-	struct its_itte *itte;
+	struct its_ite *ite;
 	struct its_collection *collection;
 
-	itte = find_itte(its, device_id, event_id);
-	if (!itte)
+	ite = find_ite(its, device_id, event_id);
+	if (!ite)
 		return E_ITS_MOVI_UNMAPPED_INTERRUPT;
 
-	if (!its_is_collection_mapped(itte->collection))
+	if (!its_is_collection_mapped(ite->collection))
 		return E_ITS_MOVI_UNMAPPED_COLLECTION;
 
 	collection = find_collection(its, coll_id);
 	if (!its_is_collection_mapped(collection))
 		return E_ITS_MOVI_UNMAPPED_COLLECTION;
 
-	itte->collection = collection;
+	ite->collection = collection;
 	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
 
-	spin_lock(&itte->irq->irq_lock);
-	itte->irq->target_vcpu = vcpu;
-	spin_unlock(&itte->irq->irq_lock);
+	spin_lock(&ite->irq->irq_lock);
+	ite->irq->target_vcpu = vcpu;
+	spin_unlock(&ite->irq->irq_lock);
 
 	return 0;
 }
@@ -679,7 +679,7 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
 {
 	struct its_collection *collection;
 	struct its_device *device;
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 	/*
 	 * Clearing the mapping for that collection ID removes the
@@ -690,10 +690,10 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
 	if (!collection)
 		return;
 
-	for_each_lpi_its(device, itte, its)
-		if (itte->collection &&
-		    itte->collection->collection_id == coll_id)
-			itte->collection = NULL;
+	for_each_lpi_its(device, ite, its)
+		if (ite->collection &&
+		    ite->collection->collection_id == coll_id)
+			ite->collection = NULL;
 
 	list_del(&collection->coll_list);
 	kfree(collection);
@@ -709,7 +709,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	u32 event_id = its_cmd_get_id(its_cmd);
 	u32 coll_id = its_cmd_get_collection(its_cmd);
-	struct its_itte *itte;
+	struct its_ite *ite;
 	struct its_device *device;
 	struct its_collection *collection, *new_coll = NULL;
 	int lpi_nr;
@@ -728,7 +728,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 		return E_ITS_MAPTI_PHYSICALID_OOR;
 
 	/* If there is an existing mapping, behavior is UNPREDICTABLE. */
-	if (find_itte(its, device_id, event_id))
+	if (find_ite(its, device_id, event_id))
 		return 0;
 
 	collection = find_collection(its, coll_id);
@@ -739,36 +739,36 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 		new_coll = collection;
 	}
 
-	itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
-	if (!itte) {
+	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
+	if (!ite) {
 		if (new_coll)
 			vgic_its_free_collection(its, coll_id);
 		return -ENOMEM;
 	}
 
-	itte->event_id	= event_id;
-	list_add_tail(&itte->itte_list, &device->itt_head);
+	ite->event_id	= event_id;
+	list_add_tail(&ite->ite_list, &device->itt_head);
 
-	itte->collection = collection;
-	itte->lpi = lpi_nr;
+	ite->collection = collection;
+	ite->lpi = lpi_nr;
 
 	irq = vgic_add_lpi(kvm, lpi_nr);
 	if (IS_ERR(irq)) {
 		if (new_coll)
 			vgic_its_free_collection(its, coll_id);
-		its_free_itte(kvm, itte);
+		its_free_ite(kvm, ite);
 		return PTR_ERR(irq);
 	}
-	itte->irq = irq;
+	ite->irq = irq;
 
-	update_affinity_itte(kvm, itte);
+	update_affinity_ite(kvm, ite);
 
 	/*
 	 * We "cache" the configuration table entries in out struct vgic_irq's.
 	 * However we only have those structs for mapped IRQs, so we read in
 	 * the respective config data from memory here upon mapping the LPI.
 	 */
-	update_lpi_config(kvm, itte->irq, NULL);
+	update_lpi_config(kvm, ite->irq, NULL);
 
 	return 0;
 }
@@ -776,15 +776,15 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 /* Requires the its_lock to be held. */
 static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
 {
-	struct its_itte *itte, *temp;
+	struct its_ite *ite, *temp;
 
 	/*
 	 * The spec says that unmapping a device with still valid
 	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
 	 * since we cannot leave the memory unreferenced.
 	 */
-	list_for_each_entry_safe(itte, temp, &device->itt_head, itte_list)
-		its_free_itte(kvm, itte);
+	list_for_each_entry_safe(ite, temp, &device->itt_head, ite_list)
+		its_free_ite(kvm, ite);
 
 	list_del(&device->dev_list);
 	kfree(device);
@@ -883,14 +883,14 @@ static int vgic_its_cmd_handle_clear(struct kvm *kvm, struct vgic_its *its,
 {
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	u32 event_id = its_cmd_get_id(its_cmd);
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 
-	itte = find_itte(its, device_id, event_id);
-	if (!itte)
+	ite = find_ite(its, device_id, event_id);
+	if (!ite)
 		return E_ITS_CLEAR_UNMAPPED_INTERRUPT;
 
-	itte->irq->pending_latch = false;
+	ite->irq->pending_latch = false;
 
 	return 0;
 }
@@ -904,14 +904,14 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,
 {
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	u32 event_id = its_cmd_get_id(its_cmd);
-	struct its_itte *itte;
+	struct its_ite *ite;
 
 
-	itte = find_itte(its, device_id, event_id);
-	if (!itte)
+	ite = find_ite(its, device_id, event_id);
+	if (!ite)
 		return E_ITS_INV_UNMAPPED_INTERRUPT;
 
-	return update_lpi_config(kvm, itte->irq, NULL);
+	return update_lpi_config(kvm, ite->irq, NULL);
 }
 
 /*
@@ -1435,7 +1435,7 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
 	struct kvm *kvm = kvm_dev->kvm;
 	struct vgic_its *its = kvm_dev->private;
 	struct its_device *dev;
-	struct its_itte *itte;
+	struct its_ite *ite;
 	struct list_head *dev_cur, *dev_temp;
 	struct list_head *cur, *temp;
 
@@ -1450,8 +1450,8 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
 	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
 		dev = container_of(dev_cur, struct its_device, dev_list);
 		list_for_each_safe(cur, temp, &dev->itt_head) {
-			itte = (container_of(cur, struct its_itte, itte_list));
-			its_free_itte(kvm, itte);
+			ite = (container_of(cur, struct its_ite, ite_list));
+			its_free_ite(kvm, ite);
 		}
 		list_del(dev_cur);
 		kfree(dev);
-- 
2.5.5

[PATCH v5 04/22] arm/arm64: vgic: turn vgic_find_mmio_region into public

[Eric Auger]

We plan to use vgic_find_mmio_region in vgic-its.c so let's
turn it into a public function.

Also let's take the opportunity to rename the region parameter
into regions to emphasize this latter is an array of regions.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>

---
v4 -> v5:
- add Marc's A-b

v3 -> v4:
- rename region parameter into regions
- add Andre's R-b
---
 virt/kvm/arm/vgic/vgic-mmio.c | 11 +++++------
 virt/kvm/arm/vgic/vgic-mmio.h |  5 +++++
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-mmio.c b/virt/kvm/arm/vgic/vgic-mmio.c
index 2a5db13..1c17b2a 100644
--- a/virt/kvm/arm/vgic/vgic-mmio.c
+++ b/virt/kvm/arm/vgic/vgic-mmio.c
@@ -446,13 +446,12 @@ static int match_region(const void *key, const void *elt)
 	return 0;
 }
 
-/* Find the proper register handler entry given a certain address offset. */
-static const struct vgic_register_region *
-vgic_find_mmio_region(const struct vgic_register_region *region, int nr_regions,
-		      unsigned int offset)
+const struct vgic_register_region *
+vgic_find_mmio_region(const struct vgic_register_region *regions,
+		      int nr_regions, unsigned int offset)
 {
-	return bsearch((void *)(uintptr_t)offset, region, nr_regions,
-		       sizeof(region[0]), match_region);
+	return bsearch((void *)(uintptr_t)offset, regions, nr_regions,
+		       sizeof(regions[0]), match_region);
 }
 
 void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr)
diff --git a/virt/kvm/arm/vgic/vgic-mmio.h b/virt/kvm/arm/vgic/vgic-mmio.h
index 98bb566..6eec91b 100644
--- a/virt/kvm/arm/vgic/vgic-mmio.h
+++ b/virt/kvm/arm/vgic/vgic-mmio.h
@@ -192,4 +192,9 @@ u64 vgic_sanitise_shareability(u64 reg);
 u64 vgic_sanitise_field(u64 reg, u64 field_mask, int field_shift,
 			u64 (*sanitise_fn)(u64));
 
+/* Find the proper register handler entry given a certain address offset */
+const struct vgic_register_region *
+vgic_find_mmio_region(const struct vgic_register_region *regions,
+		      int nr_regions, unsigned int offset);
+
 #endif
-- 
2.5.5

[PATCH v5 04/22] arm/arm64: vgic: turn vgic_find_mmio_region into public

[Eric Auger]

We plan to use vgic_find_mmio_region in vgic-its.c so let's
turn it into a public function.

Also let's take the opportunity to rename the region parameter
into regions to emphasize this latter is an array of regions.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>

---
v4 -> v5:
- add Marc's A-b

v3 -> v4:
- rename region parameter into regions
- add Andre's R-b
---
 virt/kvm/arm/vgic/vgic-mmio.c | 11 +++++------
 virt/kvm/arm/vgic/vgic-mmio.h |  5 +++++
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-mmio.c b/virt/kvm/arm/vgic/vgic-mmio.c
index 2a5db13..1c17b2a 100644
--- a/virt/kvm/arm/vgic/vgic-mmio.c
+++ b/virt/kvm/arm/vgic/vgic-mmio.c
@@ -446,13 +446,12 @@ static int match_region(const void *key, const void *elt)
 	return 0;
 }
 
-/* Find the proper register handler entry given a certain address offset. */
-static const struct vgic_register_region *
-vgic_find_mmio_region(const struct vgic_register_region *region, int nr_regions,
-		      unsigned int offset)
+const struct vgic_register_region *
+vgic_find_mmio_region(const struct vgic_register_region *regions,
+		      int nr_regions, unsigned int offset)
 {
-	return bsearch((void *)(uintptr_t)offset, region, nr_regions,
-		       sizeof(region[0]), match_region);
+	return bsearch((void *)(uintptr_t)offset, regions, nr_regions,
+		       sizeof(regions[0]), match_region);
 }
 
 void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr)
diff --git a/virt/kvm/arm/vgic/vgic-mmio.h b/virt/kvm/arm/vgic/vgic-mmio.h
index 98bb566..6eec91b 100644
--- a/virt/kvm/arm/vgic/vgic-mmio.h
+++ b/virt/kvm/arm/vgic/vgic-mmio.h
@@ -192,4 +192,9 @@ u64 vgic_sanitise_shareability(u64 reg);
 u64 vgic_sanitise_field(u64 reg, u64 field_mask, int field_shift,
 			u64 (*sanitise_fn)(u64));
 
+/* Find the proper register handler entry given a certain address offset */
+const struct vgic_register_region *
+vgic_find_mmio_region(const struct vgic_register_region *regions,
+		      int nr_regions, unsigned int offset);
+
 #endif
-- 
2.5.5

[PATCH v5 05/22] KVM: arm64: vgic-its: KVM_DEV_ARM_VGIC_GRP_ITS_REGS group

[Eric Auger]

The ITS KVM device exposes a new KVM_DEV_ARM_VGIC_GRP_ITS_REGS
group which allows the userspace to save/restore ITS registers.

At this stage the get/set/has operations are not yet implemented.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>

---
v4 -> v5:
- Add Marc's A-b

v3 -> v4:
- added Andre's R-b
---
 arch/arm/include/uapi/asm/kvm.h   |  1 +
 arch/arm64/include/uapi/asm/kvm.h |  1 +
 virt/kvm/arm/vgic/vgic-its.c      | 36 +++++++++++++++++++++++++++++++++++-
 3 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index 6ebd3e6..4beb83b 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -192,6 +192,7 @@ struct kvm_arch_memory_slot {
 #define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
 #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO  7
+#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS	8
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT	10
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
 			(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index c286035..7e8dd69 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -212,6 +212,7 @@ struct kvm_arch_memory_slot {
 #define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
 #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO  7
+#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT	10
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
 			(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 3ffcbbe..f687e91 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1466,6 +1466,19 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
 	kfree(its);
 }
 
+int vgic_its_has_attr_regs(struct kvm_device *dev,
+			   struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
+
+int vgic_its_attr_regs_access(struct kvm_device *dev,
+			      struct kvm_device_attr *attr,
+			      u64 *reg, bool is_write)
+{
+	return -ENXIO;
+}
+
 static int vgic_its_has_attr(struct kvm_device *dev,
 			     struct kvm_device_attr *attr)
 {
@@ -1482,6 +1495,8 @@ static int vgic_its_has_attr(struct kvm_device *dev,
 			return 0;
 		}
 		break;
+	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS:
+		return vgic_its_has_attr_regs(dev, attr);
 	}
 	return -ENXIO;
 }
@@ -1521,6 +1536,15 @@ static int vgic_its_set_attr(struct kvm_device *dev,
 			return 0;
 		}
 		break;
+	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
+		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
+		u64 reg;
+
+		if (get_user(reg, uaddr))
+			return -EFAULT;
+
+		return vgic_its_attr_regs_access(dev, attr, &reg, true);
+	}
 	}
 	return -ENXIO;
 }
@@ -1541,10 +1565,20 @@ static int vgic_its_get_attr(struct kvm_device *dev,
 		if (copy_to_user(uaddr, &addr, sizeof(addr)))
 			return -EFAULT;
 		break;
+	}
+	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
+		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
+		u64 reg;
+		int ret;
+
+		ret = vgic_its_attr_regs_access(dev, attr, &reg, false);
+		if (ret)
+			return ret;
+		return put_user(reg, uaddr);
+	}
 	default:
 		return -ENXIO;
 	}
-	}
 
 	return 0;
 }
-- 
2.5.5

[PATCH v5 05/22] KVM: arm64: vgic-its: KVM_DEV_ARM_VGIC_GRP_ITS_REGS group

[Eric Auger]

The ITS KVM device exposes a new KVM_DEV_ARM_VGIC_GRP_ITS_REGS
group which allows the userspace to save/restore ITS registers.

At this stage the get/set/has operations are not yet implemented.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>

---
v4 -> v5:
- Add Marc's A-b

v3 -> v4:
- added Andre's R-b
---
 arch/arm/include/uapi/asm/kvm.h   |  1 +
 arch/arm64/include/uapi/asm/kvm.h |  1 +
 virt/kvm/arm/vgic/vgic-its.c      | 36 +++++++++++++++++++++++++++++++++++-
 3 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index 6ebd3e6..4beb83b 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -192,6 +192,7 @@ struct kvm_arch_memory_slot {
 #define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
 #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO  7
+#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS	8
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT	10
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
 			(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index c286035..7e8dd69 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -212,6 +212,7 @@ struct kvm_arch_memory_slot {
 #define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
 #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO  7
+#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT	10
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
 			(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 3ffcbbe..f687e91 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1466,6 +1466,19 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
 	kfree(its);
 }
 
+int vgic_its_has_attr_regs(struct kvm_device *dev,
+			   struct kvm_device_attr *attr)
+{
+	return -ENXIO;
+}
+
+int vgic_its_attr_regs_access(struct kvm_device *dev,
+			      struct kvm_device_attr *attr,
+			      u64 *reg, bool is_write)
+{
+	return -ENXIO;
+}
+
 static int vgic_its_has_attr(struct kvm_device *dev,
 			     struct kvm_device_attr *attr)
 {
@@ -1482,6 +1495,8 @@ static int vgic_its_has_attr(struct kvm_device *dev,
 			return 0;
 		}
 		break;
+	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS:
+		return vgic_its_has_attr_regs(dev, attr);
 	}
 	return -ENXIO;
 }
@@ -1521,6 +1536,15 @@ static int vgic_its_set_attr(struct kvm_device *dev,
 			return 0;
 		}
 		break;
+	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
+		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
+		u64 reg;
+
+		if (get_user(reg, uaddr))
+			return -EFAULT;
+
+		return vgic_its_attr_regs_access(dev, attr, &reg, true);
+	}
 	}
 	return -ENXIO;
 }
@@ -1541,10 +1565,20 @@ static int vgic_its_get_attr(struct kvm_device *dev,
 		if (copy_to_user(uaddr, &addr, sizeof(addr)))
 			return -EFAULT;
 		break;
+	}
+	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
+		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
+		u64 reg;
+		int ret;
+
+		ret = vgic_its_attr_regs_access(dev, attr, &reg, false);
+		if (ret)
+			return ret;
+		return put_user(reg, uaddr);
+	}
 	default:
 		return -ENXIO;
 	}
-	}
 
 	return 0;
 }
-- 
2.5.5

[PATCH v5 06/22] KVM: arm/arm64: vgic: expose (un)lock_all_vcpus

[Eric Auger]

We need to use those helpers in vgic-its.c so let's
expose them in the private vgic header.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>

---

v4 -> v5:
- Add Marc's A-b
---
 virt/kvm/arm/vgic/vgic-kvm-device.c | 4 ++--
 virt/kvm/arm/vgic/vgic.h            | 3 +++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c
index d181d2b..859bfa8 100644
--- a/virt/kvm/arm/vgic/vgic-kvm-device.c
+++ b/virt/kvm/arm/vgic/vgic-kvm-device.c
@@ -259,13 +259,13 @@ static void unlock_vcpus(struct kvm *kvm, int vcpu_lock_idx)
 	}
 }
 
-static void unlock_all_vcpus(struct kvm *kvm)
+void unlock_all_vcpus(struct kvm *kvm)
 {
 	unlock_vcpus(kvm, atomic_read(&kvm->online_vcpus) - 1);
 }
 
 /* Returns true if all vcpus were locked, false otherwise */
-static bool lock_all_vcpus(struct kvm *kvm)
+bool lock_all_vcpus(struct kvm *kvm)
 {
 	struct kvm_vcpu *tmp_vcpu;
 	int c;
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 6cf557e..b87f1c6 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -184,4 +184,7 @@ int vgic_init(struct kvm *kvm);
 int vgic_debug_init(struct kvm *kvm);
 int vgic_debug_destroy(struct kvm *kvm);
 
+bool lock_all_vcpus(struct kvm *kvm);
+void unlock_all_vcpus(struct kvm *kvm);
+
 #endif
-- 
2.5.5

[PATCH v5 06/22] KVM: arm/arm64: vgic: expose (un)lock_all_vcpus

[Eric Auger]

We need to use those helpers in vgic-its.c so let's
expose them in the private vgic header.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>

---

v4 -> v5:
- Add Marc's A-b
---
 virt/kvm/arm/vgic/vgic-kvm-device.c | 4 ++--
 virt/kvm/arm/vgic/vgic.h            | 3 +++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c
index d181d2b..859bfa8 100644
--- a/virt/kvm/arm/vgic/vgic-kvm-device.c
+++ b/virt/kvm/arm/vgic/vgic-kvm-device.c
@@ -259,13 +259,13 @@ static void unlock_vcpus(struct kvm *kvm, int vcpu_lock_idx)
 	}
 }
 
-static void unlock_all_vcpus(struct kvm *kvm)
+void unlock_all_vcpus(struct kvm *kvm)
 {
 	unlock_vcpus(kvm, atomic_read(&kvm->online_vcpus) - 1);
 }
 
 /* Returns true if all vcpus were locked, false otherwise */
-static bool lock_all_vcpus(struct kvm *kvm)
+bool lock_all_vcpus(struct kvm *kvm)
 {
 	struct kvm_vcpu *tmp_vcpu;
 	int c;
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 6cf557e..b87f1c6 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -184,4 +184,7 @@ int vgic_init(struct kvm *kvm);
 int vgic_debug_init(struct kvm *kvm);
 int vgic_debug_destroy(struct kvm *kvm);
 
+bool lock_all_vcpus(struct kvm *kvm);
+void unlock_all_vcpus(struct kvm *kvm);
+
 #endif
-- 
2.5.5

[PATCH v5 07/22] KVM: arm64: vgic-its: Implement vgic_its_has_attr_regs and attr_regs_access

[Eric Auger]

This patch implements vgic_its_has_attr_regs and vgic_its_attr_regs_access
upon the MMIO framework. VGIC ITS KVM device KVM_DEV_ARM_VGIC_GRP_ITS_REGS
group becomes functional.

At least GITS_CREADR and GITS_IIDR require to differentiate a guest write
action from a user access. As such let's introduce a new uaccess_its_write
vgic_register_region callback.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- use GITS_TYPER instead of offset 0x8
- uaccess_its_write now can return an error

v3 -> v4:
- remove changes to the REGISTER_ITS_DESC macro. This will be handled in
  subsequent patch with the introduction of a new REGISTER_ITS_DESC_UACCESS
  macro
- fix IIDR access and add a comment wrt full length access
- handle endianness
- add kvm lock and vcpus lock
---
 virt/kvm/arm/vgic/vgic-its.c  | 79 +++++++++++++++++++++++++++++++++++++++++--
 virt/kvm/arm/vgic/vgic-mmio.h |  9 +++--
 2 files changed, 84 insertions(+), 4 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index f687e91..a9a2c12 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1469,14 +1469,89 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
 int vgic_its_has_attr_regs(struct kvm_device *dev,
 			   struct kvm_device_attr *attr)
 {
-	return -ENXIO;
+	const struct vgic_register_region *region;
+	struct vgic_io_device iodev = {
+		.regions = its_registers,
+		.nr_regions = ARRAY_SIZE(its_registers),
+	};
+	gpa_t offset = attr->attr;
+
+	region = vgic_find_mmio_region(iodev.regions,
+				       iodev.nr_regions,
+				       offset);
+	if (!region)
+		return -ENXIO;
+
+	return 0;
 }
 
 int vgic_its_attr_regs_access(struct kvm_device *dev,
 			      struct kvm_device_attr *attr,
 			      u64 *reg, bool is_write)
 {
-	return -ENXIO;
+	const struct vgic_register_region *region;
+	struct vgic_io_device iodev = {
+		.regions = its_registers,
+		.nr_regions = ARRAY_SIZE(its_registers),
+	};
+	struct vgic_its *its = dev->private;
+	gpa_t addr, offset = attr->attr;
+	unsigned int len;
+	unsigned long data = 0;
+	int ret = 0;
+
+	/*
+	 * Among supported registers, only GITS_CTLR (0x0) and GITS_IIDR (0x4)
+	 * are 32 bits. Others are 64 bits.
+	 */
+	if ((offset < GITS_TYPER && offset & 0x3) ||
+	    (offset >= GITS_TYPER && offset & 0x7))
+		return -EINVAL;
+
+	mutex_lock(&dev->kvm->lock);
+
+	if (IS_VGIC_ADDR_UNDEF(its->vgic_its_base)) {
+		ret = -ENXIO;
+		goto out;
+	}
+
+	region = vgic_find_mmio_region(iodev.regions,
+				       iodev.nr_regions,
+				       offset);
+	if (!region) {
+		ret = -ENXIO;
+		goto out;
+	}
+
+	if (!lock_all_vcpus(dev->kvm)) {
+		ret = -EBUSY;
+		goto out;
+	}
+
+	addr = its->vgic_its_base + offset;
+
+	/*
+	 * Only full length register accesses are supported although
+	 * the architecture spec theoretically allows upper/lower 32
+	 * bits to be accessed independently
+	 */
+	len = region->access_flags & VGIC_ACCESS_64bit ? 8 : 4;
+
+	if (is_write) {
+		data = vgic_data_mmio_bus_to_host(reg, len);
+		if (region->uaccess_its_write)
+			ret = region->uaccess_its_write(dev->kvm, its, addr,
+							len, data);
+		else
+			region->its_write(dev->kvm, its, addr, len, data);
+	} else {
+		data = region->its_read(dev->kvm, its, addr, len);
+		vgic_data_host_to_mmio_bus(reg, len, data);
+	}
+	unlock_all_vcpus(dev->kvm);
+out:
+	mutex_unlock(&dev->kvm->lock);
+	return ret;
 }
 
 static int vgic_its_has_attr(struct kvm_device *dev,
diff --git a/virt/kvm/arm/vgic/vgic-mmio.h b/virt/kvm/arm/vgic/vgic-mmio.h
index 6eec91b..ea4171a 100644
--- a/virt/kvm/arm/vgic/vgic-mmio.h
+++ b/virt/kvm/arm/vgic/vgic-mmio.h
@@ -36,8 +36,13 @@ struct vgic_register_region {
 	};
 	unsigned long (*uaccess_read)(struct kvm_vcpu *vcpu, gpa_t addr,
 				      unsigned int len);
-	void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
-			      unsigned int len, unsigned long val);
+	union {
+		void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
+				      unsigned int len, unsigned long val);
+		int (*uaccess_its_write)(struct kvm *kvm, struct vgic_its *its,
+					 gpa_t addr, unsigned int len,
+					 unsigned long val);
+	};
 };
 
 extern struct kvm_io_device_ops kvm_io_gic_ops;
-- 
2.5.5

[PATCH v5 07/22] KVM: arm64: vgic-its: Implement vgic_its_has_attr_regs and attr_regs_access

[Eric Auger]

This patch implements vgic_its_has_attr_regs and vgic_its_attr_regs_access
upon the MMIO framework. VGIC ITS KVM device KVM_DEV_ARM_VGIC_GRP_ITS_REGS
group becomes functional.

At least GITS_CREADR and GITS_IIDR require to differentiate a guest write
action from a user access. As such let's introduce a new uaccess_its_write
vgic_register_region callback.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- use GITS_TYPER instead of offset 0x8
- uaccess_its_write now can return an error

v3 -> v4:
- remove changes to the REGISTER_ITS_DESC macro. This will be handled in
  subsequent patch with the introduction of a new REGISTER_ITS_DESC_UACCESS
  macro
- fix IIDR access and add a comment wrt full length access
- handle endianness
- add kvm lock and vcpus lock
---
 virt/kvm/arm/vgic/vgic-its.c  | 79 +++++++++++++++++++++++++++++++++++++++++--
 virt/kvm/arm/vgic/vgic-mmio.h |  9 +++--
 2 files changed, 84 insertions(+), 4 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index f687e91..a9a2c12 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1469,14 +1469,89 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
 int vgic_its_has_attr_regs(struct kvm_device *dev,
 			   struct kvm_device_attr *attr)
 {
-	return -ENXIO;
+	const struct vgic_register_region *region;
+	struct vgic_io_device iodev = {
+		.regions = its_registers,
+		.nr_regions = ARRAY_SIZE(its_registers),
+	};
+	gpa_t offset = attr->attr;
+
+	region = vgic_find_mmio_region(iodev.regions,
+				       iodev.nr_regions,
+				       offset);
+	if (!region)
+		return -ENXIO;
+
+	return 0;
 }
 
 int vgic_its_attr_regs_access(struct kvm_device *dev,
 			      struct kvm_device_attr *attr,
 			      u64 *reg, bool is_write)
 {
-	return -ENXIO;
+	const struct vgic_register_region *region;
+	struct vgic_io_device iodev = {
+		.regions = its_registers,
+		.nr_regions = ARRAY_SIZE(its_registers),
+	};
+	struct vgic_its *its = dev->private;
+	gpa_t addr, offset = attr->attr;
+	unsigned int len;
+	unsigned long data = 0;
+	int ret = 0;
+
+	/*
+	 * Among supported registers, only GITS_CTLR (0x0) and GITS_IIDR (0x4)
+	 * are 32 bits. Others are 64 bits.
+	 */
+	if ((offset < GITS_TYPER && offset & 0x3) ||
+	    (offset >= GITS_TYPER && offset & 0x7))
+		return -EINVAL;
+
+	mutex_lock(&dev->kvm->lock);
+
+	if (IS_VGIC_ADDR_UNDEF(its->vgic_its_base)) {
+		ret = -ENXIO;
+		goto out;
+	}
+
+	region = vgic_find_mmio_region(iodev.regions,
+				       iodev.nr_regions,
+				       offset);
+	if (!region) {
+		ret = -ENXIO;
+		goto out;
+	}
+
+	if (!lock_all_vcpus(dev->kvm)) {
+		ret = -EBUSY;
+		goto out;
+	}
+
+	addr = its->vgic_its_base + offset;
+
+	/*
+	 * Only full length register accesses are supported although
+	 * the architecture spec theoretically allows upper/lower 32
+	 * bits to be accessed independently
+	 */
+	len = region->access_flags & VGIC_ACCESS_64bit ? 8 : 4;
+
+	if (is_write) {
+		data = vgic_data_mmio_bus_to_host(reg, len);
+		if (region->uaccess_its_write)
+			ret = region->uaccess_its_write(dev->kvm, its, addr,
+							len, data);
+		else
+			region->its_write(dev->kvm, its, addr, len, data);
+	} else {
+		data = region->its_read(dev->kvm, its, addr, len);
+		vgic_data_host_to_mmio_bus(reg, len, data);
+	}
+	unlock_all_vcpus(dev->kvm);
+out:
+	mutex_unlock(&dev->kvm->lock);
+	return ret;
 }
 
 static int vgic_its_has_attr(struct kvm_device *dev,
diff --git a/virt/kvm/arm/vgic/vgic-mmio.h b/virt/kvm/arm/vgic/vgic-mmio.h
index 6eec91b..ea4171a 100644
--- a/virt/kvm/arm/vgic/vgic-mmio.h
+++ b/virt/kvm/arm/vgic/vgic-mmio.h
@@ -36,8 +36,13 @@ struct vgic_register_region {
 	};
 	unsigned long (*uaccess_read)(struct kvm_vcpu *vcpu, gpa_t addr,
 				      unsigned int len);
-	void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
-			      unsigned int len, unsigned long val);
+	union {
+		void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
+				      unsigned int len, unsigned long val);
+		int (*uaccess_its_write)(struct kvm *kvm, struct vgic_its *its,
+					 gpa_t addr, unsigned int len,
+					 unsigned long val);
+	};
 };
 
 extern struct kvm_io_device_ops kvm_io_gic_ops;
-- 
2.5.5

[PATCH v5 08/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_creadr

[Eric Auger]

GITS_CREADR needs to be restored so let's implement the associated
uaccess_write_its callback. The write only is allowed if the its
is disabled.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- keep Stalled bit
- vgic_mmio_uaccess_write_its_creadr can now return an error

v3 -> v4:
- REGISTER_ITS_DESC_UACCESS now introduced in this patch
- we now check the its is disabled
---
 virt/kvm/arm/vgic/vgic-its.c | 42 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 40 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index a9a2c12..79ed1c2 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1213,6 +1213,33 @@ static unsigned long vgic_mmio_read_its_creadr(struct kvm *kvm,
 	return extract_bytes(its->creadr, addr & 0x7, len);
 }
 
+static int vgic_mmio_uaccess_write_its_creadr(struct kvm *kvm,
+					      struct vgic_its *its,
+					      gpa_t addr, unsigned int len,
+					      unsigned long val)
+{
+	int ret = 0;
+	u32 reg;
+
+	mutex_lock(&its->cmd_lock);
+
+	if (its->enabled) {
+		ret = -EBUSY;
+		goto out;
+	}
+
+	reg = update_64bit_reg(its->creadr, addr & 7, len, val);
+	if (ITS_CMD_OFFSET(reg) >= ITS_CMD_BUFFER_SIZE(its->cbaser)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	its->creadr = reg;
+out:
+	mutex_unlock(&its->cmd_lock);
+	return ret;
+}
+
 #define BASER_INDEX(addr) (((addr) / sizeof(u64)) & 0x7)
 static unsigned long vgic_mmio_read_its_baser(struct kvm *kvm,
 					      struct vgic_its *its,
@@ -1317,6 +1344,16 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its,
 	.its_write = wr,					\
 }
 
+#define REGISTER_ITS_DESC_UACCESS(off, rd, wr, uwr, length, acc)\
+{								\
+	.reg_offset = off,					\
+	.len = length,						\
+	.access_flags = acc,					\
+	.its_read = rd,						\
+	.its_write = wr,					\
+	.uaccess_its_write = uwr,				\
+}
+
 static void its_mmio_write_wi(struct kvm *kvm, struct vgic_its *its,
 			      gpa_t addr, unsigned int len, unsigned long val)
 {
@@ -1339,8 +1376,9 @@ static struct vgic_register_region its_registers[] = {
 	REGISTER_ITS_DESC(GITS_CWRITER,
 		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8,
 		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
-	REGISTER_ITS_DESC(GITS_CREADR,
-		vgic_mmio_read_its_creadr, its_mmio_write_wi, 8,
+	REGISTER_ITS_DESC_UACCESS(GITS_CREADR,
+		vgic_mmio_read_its_creadr, its_mmio_write_wi,
+		vgic_mmio_uaccess_write_its_creadr, 8,
 		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
 	REGISTER_ITS_DESC(GITS_BASER,
 		vgic_mmio_read_its_baser, vgic_mmio_write_its_baser, 0x40,
-- 
2.5.5

[PATCH v5 08/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_creadr

[Eric Auger]

GITS_CREADR needs to be restored so let's implement the associated
uaccess_write_its callback. The write only is allowed if the its
is disabled.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- keep Stalled bit
- vgic_mmio_uaccess_write_its_creadr can now return an error

v3 -> v4:
- REGISTER_ITS_DESC_UACCESS now introduced in this patch
- we now check the its is disabled
---
 virt/kvm/arm/vgic/vgic-its.c | 42 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 40 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index a9a2c12..79ed1c2 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1213,6 +1213,33 @@ static unsigned long vgic_mmio_read_its_creadr(struct kvm *kvm,
 	return extract_bytes(its->creadr, addr & 0x7, len);
 }
 
+static int vgic_mmio_uaccess_write_its_creadr(struct kvm *kvm,
+					      struct vgic_its *its,
+					      gpa_t addr, unsigned int len,
+					      unsigned long val)
+{
+	int ret = 0;
+	u32 reg;
+
+	mutex_lock(&its->cmd_lock);
+
+	if (its->enabled) {
+		ret = -EBUSY;
+		goto out;
+	}
+
+	reg = update_64bit_reg(its->creadr, addr & 7, len, val);
+	if (ITS_CMD_OFFSET(reg) >= ITS_CMD_BUFFER_SIZE(its->cbaser)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	its->creadr = reg;
+out:
+	mutex_unlock(&its->cmd_lock);
+	return ret;
+}
+
 #define BASER_INDEX(addr) (((addr) / sizeof(u64)) & 0x7)
 static unsigned long vgic_mmio_read_its_baser(struct kvm *kvm,
 					      struct vgic_its *its,
@@ -1317,6 +1344,16 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its,
 	.its_write = wr,					\
 }
 
+#define REGISTER_ITS_DESC_UACCESS(off, rd, wr, uwr, length, acc)\
+{								\
+	.reg_offset = off,					\
+	.len = length,						\
+	.access_flags = acc,					\
+	.its_read = rd,						\
+	.its_write = wr,					\
+	.uaccess_its_write = uwr,				\
+}
+
 static void its_mmio_write_wi(struct kvm *kvm, struct vgic_its *its,
 			      gpa_t addr, unsigned int len, unsigned long val)
 {
@@ -1339,8 +1376,9 @@ static struct vgic_register_region its_registers[] = {
 	REGISTER_ITS_DESC(GITS_CWRITER,
 		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8,
 		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
-	REGISTER_ITS_DESC(GITS_CREADR,
-		vgic_mmio_read_its_creadr, its_mmio_write_wi, 8,
+	REGISTER_ITS_DESC_UACCESS(GITS_CREADR,
+		vgic_mmio_read_its_creadr, its_mmio_write_wi,
+		vgic_mmio_uaccess_write_its_creadr, 8,
 		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
 	REGISTER_ITS_DESC(GITS_BASER,
 		vgic_mmio_read_its_baser, vgic_mmio_write_its_baser, 0x40,
-- 
2.5.5

[PATCH v5 09/22] KVM: arm64: vgic-its: Introduce migration ABI infrastructure

[Eric Auger]

We plan to support different migration ABIs, ie. characterizing
the ITS table layout format in guest RAM. Typically a new ABI will
be needed if vLPIs get supported for nested use case.

So let's introduce an array of supported ABIs (at the moment a single
ABI is supported though). The following characteristics are foreseen
to vary with the ABI: size of table entries, save/restore operation.

By default the MAX_ABI_REV is applied on its creation. In subsequent
patches we will introduce a way for the userspace to change the ABI
in use.

The entry sizes now are set according to the ABI version and not
hardcoded anymore.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

v5: creation and squash KVM: arm64: ITS: Report the ITE size in
    GITS_TYPER
---
 include/kvm/arm_vgic.h             |  3 ++
 include/linux/irqchip/arm-gic-v3.h |  4 ++
 virt/kvm/arm/vgic/vgic-its.c       | 82 ++++++++++++++++++++++++++++++++++++--
 3 files changed, 85 insertions(+), 4 deletions(-)

diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index c0b3d99..285474a 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -162,6 +162,9 @@ struct vgic_its {
 	u32			creadr;
 	u32			cwriter;
 
+	/* migration ABI revision in use */
+	u32			abi_rev;
+
 	/* Protects the device and collection lists */
 	struct mutex		its_lock;
 	struct list_head	device_list;
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 97cbca1..9648bad 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -132,6 +132,8 @@
 #define GIC_BASER_SHAREABILITY(reg, type)				\
 	(GIC_BASER_##type << reg##_SHAREABILITY_SHIFT)
 
+#define GIC_ENCODE_SZ(n, w) (((unsigned long)(n) - 1) & GENMASK_ULL(((w) - 1), 0))
+
 #define GICR_PROPBASER_SHAREABILITY_SHIFT		(10)
 #define GICR_PROPBASER_INNER_CACHEABILITY_SHIFT		(7)
 #define GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT		(56)
@@ -232,6 +234,7 @@
 #define GITS_CTLR_QUIESCENT		(1U << 31)
 
 #define GITS_TYPER_PLPIS		(1UL << 0)
+#define GITS_TYPER_ITT_ENTRY_SIZE_SHIFT	4
 #define GITS_TYPER_IDBITS_SHIFT		8
 #define GITS_TYPER_DEVBITS_SHIFT	13
 #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
@@ -290,6 +293,7 @@
 #define GITS_BASER_TYPE(r)		(((r) >> GITS_BASER_TYPE_SHIFT) & 7)
 #define GITS_BASER_ENTRY_SIZE_SHIFT		(48)
 #define GITS_BASER_ENTRY_SIZE(r)	((((r) >> GITS_BASER_ENTRY_SIZE_SHIFT) & 0x1f) + 1)
+#define GITS_BASER_ENTRY_SIZE_MASK	GENMASK_ULL(52, 48)
 #define GITS_BASER_SHAREABILITY_SHIFT	(10)
 #define GITS_BASER_InnerShareable					\
 	GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 79ed1c2..69ecfe4 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -33,6 +33,12 @@
 #include "vgic.h"
 #include "vgic-mmio.h"
 
+/* Highest migration ABI revision supported by this code */
+#define MAX_ABI_REV 0
+
+static int vgic_its_set_abi(struct vgic_its *its, int rev);
+static const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its);
+
 /*
  * Creates a new (reference to a) struct vgic_irq for a given LPI.
  * If this LPI is already mapped on another ITS, we increase its refcount
@@ -123,6 +129,15 @@ struct its_ite {
 	u32 event_id;
 };
 
+struct vgic_its_abi {
+	int cte_esz;
+	int dte_esz;
+	int ite_esz;
+	int (*save_tables)(struct vgic_its *its);
+	int (*restore_tables)(struct vgic_its *its);
+	int (*commit)(struct vgic_its *its);
+};
+
 /*
  * Find and returns a device in the device table for an ITS.
  * Must be called with the its_lock mutex held.
@@ -364,6 +379,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
 					      struct vgic_its *its,
 					      gpa_t addr, unsigned int len)
 {
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
 	u64 reg = GITS_TYPER_PLPIS;
 
 	/*
@@ -376,6 +392,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
 	 */
 	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
 	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
+	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
 
 	return extract_bytes(reg, addr & 7, len);
 }
@@ -1268,6 +1285,7 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
 				      gpa_t addr, unsigned int len,
 				      unsigned long val)
 {
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
 	u64 entry_size, device_type;
 	u64 reg, *regptr, clearbits = 0;
 
@@ -1278,12 +1296,12 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
 	switch (BASER_INDEX(addr)) {
 	case 0:
 		regptr = &its->baser_device_table;
-		entry_size = 8;
+		entry_size = abi->dte_esz;
 		device_type = GITS_BASER_TYPE_DEVICE;
 		break;
 	case 1:
 		regptr = &its->baser_coll_table;
-		entry_size = 8;
+		entry_size = abi->cte_esz;
 		device_type = GITS_BASER_TYPE_COLLECTION;
 		clearbits = GITS_BASER_INDIRECT;
 		break;
@@ -1425,7 +1443,6 @@ static int vgic_register_its_iodev(struct kvm *kvm, struct vgic_its *its)
 	(GIC_BASER_CACHEABILITY(GITS_BASER, INNER, RaWb)		| \
 	 GIC_BASER_CACHEABILITY(GITS_BASER, OUTER, SameAsInner)		| \
 	 GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)		| \
-	 ((8ULL - 1) << GITS_BASER_ENTRY_SIZE_SHIFT)			| \
 	 GITS_BASER_PAGE_SIZE_64K)
 
 #define INITIAL_PROPBASER_VALUE						  \
@@ -1465,7 +1482,7 @@ static int vgic_its_create(struct kvm_device *dev, u32 type)
 
 	dev->private = its;
 
-	return 0;
+	return vgic_its_set_abi(its, MAX_ABI_REV);
 }
 
 static void vgic_its_destroy(struct kvm_device *kvm_dev)
@@ -1592,6 +1609,63 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
 	return ret;
 }
 
+/**
+ * vgic_its_save_tables_v0 - Save the ITS tables into guest ARM
+ * according to v0 ABI
+ */
+static int vgic_its_save_tables_v0(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+/**
+ * vgic_its_restore_tables_v0 - Restore the ITS tables from guest RAM
+ * to internal data structs according to V0 ABI
+ *
+ */
+static int vgic_its_restore_tables_v0(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+static int vgic_its_commit_v0(struct vgic_its *its)
+{
+	const struct vgic_its_abi *abi;
+
+	abi = vgic_its_get_abi(its);
+	its->baser_coll_table &= ~GITS_BASER_ENTRY_SIZE_MASK;
+	its->baser_device_table &= ~GITS_BASER_ENTRY_SIZE_MASK;
+
+	its->baser_coll_table |= (GIC_ENCODE_SZ(abi->cte_esz, 5)
+					<< GITS_BASER_ENTRY_SIZE_SHIFT);
+
+	its->baser_device_table |= (GIC_ENCODE_SZ(abi->dte_esz, 5)
+					<< GITS_BASER_ENTRY_SIZE_SHIFT);
+	return 0;
+}
+
+static const struct vgic_its_abi abi[MAX_ABI_REV + 1] = {
+	{.cte_esz = 8, .dte_esz = 8, .ite_esz = 8,
+	 .save_tables = vgic_its_save_tables_v0,
+	 .restore_tables = vgic_its_restore_tables_v0,
+	 . commit = vgic_its_commit_v0,
+	},
+};
+
+inline const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its)
+{
+	return &abi[its->abi_rev];
+}
+
+int vgic_its_set_abi(struct vgic_its *its, int rev)
+{
+	const struct vgic_its_abi *abi;
+
+	its->abi_rev = rev;
+	abi = vgic_its_get_abi(its);
+	return abi->commit(its);
+}
+
 static int vgic_its_has_attr(struct kvm_device *dev,
 			     struct kvm_device_attr *attr)
 {
-- 
2.5.5

[PATCH v5 09/22] KVM: arm64: vgic-its: Introduce migration ABI infrastructure

[Eric Auger]

We plan to support different migration ABIs, ie. characterizing
the ITS table layout format in guest RAM. Typically a new ABI will
be needed if vLPIs get supported for nested use case.

So let's introduce an array of supported ABIs (at the moment a single
ABI is supported though). The following characteristics are foreseen
to vary with the ABI: size of table entries, save/restore operation.

By default the MAX_ABI_REV is applied on its creation. In subsequent
patches we will introduce a way for the userspace to change the ABI
in use.

The entry sizes now are set according to the ABI version and not
hardcoded anymore.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

v5: creation and squash KVM: arm64: ITS: Report the ITE size in
    GITS_TYPER
---
 include/kvm/arm_vgic.h             |  3 ++
 include/linux/irqchip/arm-gic-v3.h |  4 ++
 virt/kvm/arm/vgic/vgic-its.c       | 82 ++++++++++++++++++++++++++++++++++++--
 3 files changed, 85 insertions(+), 4 deletions(-)

diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index c0b3d99..285474a 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -162,6 +162,9 @@ struct vgic_its {
 	u32			creadr;
 	u32			cwriter;
 
+	/* migration ABI revision in use */
+	u32			abi_rev;
+
 	/* Protects the device and collection lists */
 	struct mutex		its_lock;
 	struct list_head	device_list;
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 97cbca1..9648bad 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -132,6 +132,8 @@
 #define GIC_BASER_SHAREABILITY(reg, type)				\
 	(GIC_BASER_##type << reg##_SHAREABILITY_SHIFT)
 
+#define GIC_ENCODE_SZ(n, w) (((unsigned long)(n) - 1) & GENMASK_ULL(((w) - 1), 0))
+
 #define GICR_PROPBASER_SHAREABILITY_SHIFT		(10)
 #define GICR_PROPBASER_INNER_CACHEABILITY_SHIFT		(7)
 #define GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT		(56)
@@ -232,6 +234,7 @@
 #define GITS_CTLR_QUIESCENT		(1U << 31)
 
 #define GITS_TYPER_PLPIS		(1UL << 0)
+#define GITS_TYPER_ITT_ENTRY_SIZE_SHIFT	4
 #define GITS_TYPER_IDBITS_SHIFT		8
 #define GITS_TYPER_DEVBITS_SHIFT	13
 #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
@@ -290,6 +293,7 @@
 #define GITS_BASER_TYPE(r)		(((r) >> GITS_BASER_TYPE_SHIFT) & 7)
 #define GITS_BASER_ENTRY_SIZE_SHIFT		(48)
 #define GITS_BASER_ENTRY_SIZE(r)	((((r) >> GITS_BASER_ENTRY_SIZE_SHIFT) & 0x1f) + 1)
+#define GITS_BASER_ENTRY_SIZE_MASK	GENMASK_ULL(52, 48)
 #define GITS_BASER_SHAREABILITY_SHIFT	(10)
 #define GITS_BASER_InnerShareable					\
 	GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 79ed1c2..69ecfe4 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -33,6 +33,12 @@
 #include "vgic.h"
 #include "vgic-mmio.h"
 
+/* Highest migration ABI revision supported by this code */
+#define MAX_ABI_REV 0
+
+static int vgic_its_set_abi(struct vgic_its *its, int rev);
+static const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its);
+
 /*
  * Creates a new (reference to a) struct vgic_irq for a given LPI.
  * If this LPI is already mapped on another ITS, we increase its refcount
@@ -123,6 +129,15 @@ struct its_ite {
 	u32 event_id;
 };
 
+struct vgic_its_abi {
+	int cte_esz;
+	int dte_esz;
+	int ite_esz;
+	int (*save_tables)(struct vgic_its *its);
+	int (*restore_tables)(struct vgic_its *its);
+	int (*commit)(struct vgic_its *its);
+};
+
 /*
  * Find and returns a device in the device table for an ITS.
  * Must be called with the its_lock mutex held.
@@ -364,6 +379,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
 					      struct vgic_its *its,
 					      gpa_t addr, unsigned int len)
 {
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
 	u64 reg = GITS_TYPER_PLPIS;
 
 	/*
@@ -376,6 +392,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
 	 */
 	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
 	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
+	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
 
 	return extract_bytes(reg, addr & 7, len);
 }
@@ -1268,6 +1285,7 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
 				      gpa_t addr, unsigned int len,
 				      unsigned long val)
 {
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
 	u64 entry_size, device_type;
 	u64 reg, *regptr, clearbits = 0;
 
@@ -1278,12 +1296,12 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
 	switch (BASER_INDEX(addr)) {
 	case 0:
 		regptr = &its->baser_device_table;
-		entry_size = 8;
+		entry_size = abi->dte_esz;
 		device_type = GITS_BASER_TYPE_DEVICE;
 		break;
 	case 1:
 		regptr = &its->baser_coll_table;
-		entry_size = 8;
+		entry_size = abi->cte_esz;
 		device_type = GITS_BASER_TYPE_COLLECTION;
 		clearbits = GITS_BASER_INDIRECT;
 		break;
@@ -1425,7 +1443,6 @@ static int vgic_register_its_iodev(struct kvm *kvm, struct vgic_its *its)
 	(GIC_BASER_CACHEABILITY(GITS_BASER, INNER, RaWb)		| \
 	 GIC_BASER_CACHEABILITY(GITS_BASER, OUTER, SameAsInner)		| \
 	 GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)		| \
-	 ((8ULL - 1) << GITS_BASER_ENTRY_SIZE_SHIFT)			| \
 	 GITS_BASER_PAGE_SIZE_64K)
 
 #define INITIAL_PROPBASER_VALUE						  \
@@ -1465,7 +1482,7 @@ static int vgic_its_create(struct kvm_device *dev, u32 type)
 
 	dev->private = its;
 
-	return 0;
+	return vgic_its_set_abi(its, MAX_ABI_REV);
 }
 
 static void vgic_its_destroy(struct kvm_device *kvm_dev)
@@ -1592,6 +1609,63 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
 	return ret;
 }
 
+/**
+ * vgic_its_save_tables_v0 - Save the ITS tables into guest ARM
+ * according to v0 ABI
+ */
+static int vgic_its_save_tables_v0(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+/**
+ * vgic_its_restore_tables_v0 - Restore the ITS tables from guest RAM
+ * to internal data structs according to V0 ABI
+ *
+ */
+static int vgic_its_restore_tables_v0(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+static int vgic_its_commit_v0(struct vgic_its *its)
+{
+	const struct vgic_its_abi *abi;
+
+	abi = vgic_its_get_abi(its);
+	its->baser_coll_table &= ~GITS_BASER_ENTRY_SIZE_MASK;
+	its->baser_device_table &= ~GITS_BASER_ENTRY_SIZE_MASK;
+
+	its->baser_coll_table |= (GIC_ENCODE_SZ(abi->cte_esz, 5)
+					<< GITS_BASER_ENTRY_SIZE_SHIFT);
+
+	its->baser_device_table |= (GIC_ENCODE_SZ(abi->dte_esz, 5)
+					<< GITS_BASER_ENTRY_SIZE_SHIFT);
+	return 0;
+}
+
+static const struct vgic_its_abi abi[MAX_ABI_REV + 1] = {
+	{.cte_esz = 8, .dte_esz = 8, .ite_esz = 8,
+	 .save_tables = vgic_its_save_tables_v0,
+	 .restore_tables = vgic_its_restore_tables_v0,
+	 . commit = vgic_its_commit_v0,
+	},
+};
+
+inline const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its)
+{
+	return &abi[its->abi_rev];
+}
+
+int vgic_its_set_abi(struct vgic_its *its, int rev)
+{
+	const struct vgic_its_abi *abi;
+
+	its->abi_rev = rev;
+	abi = vgic_its_get_abi(its);
+	return abi->commit(its);
+}
+
 static int vgic_its_has_attr(struct kvm_device *dev,
 			     struct kvm_device_attr *attr)
 {
-- 
2.5.5

[PATCH v5 10/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_iidr

[Eric Auger]

The GITS_IIDR revision field is used to encode the migration ABI
revision. So we need to restore it to check the table layout is
readable by the destination.

By writing the IIDR, userspace thus force the ABI revision to be
used and this msut be less or equal than the max revision KVM supports.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5
- rename user_revision into abi_rev and REV into MAX_ABI_REV
- IIDR reports abi_rev set by userspace if any.
- If value set by userspace exceeds the max supported revision, an
  error is reported.
- add some defines

v4: creation
---
 include/linux/irqchip/arm-gic-v3.h |  4 ++++
 virt/kvm/arm/vgic/vgic-its.c       | 23 ++++++++++++++++++++---
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 9648bad..54c20bd 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -241,6 +241,10 @@
 #define GITS_TYPER_PTA			(1UL << 19)
 #define GITS_TYPER_HWCOLLCNT_SHIFT	24
 
+#define GITS_IIDR_REV_SHIFT		12
+#define GITS_IIDR_REV(r)		(((r) >> GITS_IIDR_REV_SHIFT) & 0xf)
+#define GITS_IIDR_PRODUCTID_SHIFT	24
+
 #define GITS_CBASER_VALID			(1ULL << 63)
 #define GITS_CBASER_SHAREABILITY_SHIFT		(10)
 #define GITS_CBASER_INNER_CACHEABILITY_SHIFT	(59)
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 69ecfe4..1b5797e 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -401,7 +401,23 @@ static unsigned long vgic_mmio_read_its_iidr(struct kvm *kvm,
 					     struct vgic_its *its,
 					     gpa_t addr, unsigned int len)
 {
-	return (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
+	return (PRODUCT_ID_KVM << GITS_IIDR_PRODUCTID_SHIFT) |
+	       (its->abi_rev << GITS_IIDR_REV_SHIFT) | IMPLEMENTER_ARM;
+}
+
+static int vgic_mmio_uaccess_write_its_iidr(struct kvm *kvm,
+					    struct vgic_its *its,
+					    gpa_t addr, unsigned int len,
+					    unsigned long val)
+{
+	u64 tmp = 0;
+
+	tmp = update_64bit_reg(tmp, addr & 3, len, val);
+	tmp = GITS_IIDR_REV(tmp);
+
+	if (tmp > MAX_ABI_REV)
+		return -EINVAL;
+	return vgic_its_set_abi(its, tmp);
 }
 
 static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,
@@ -1382,8 +1398,9 @@ static struct vgic_register_region its_registers[] = {
 	REGISTER_ITS_DESC(GITS_CTLR,
 		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4,
 		VGIC_ACCESS_32bit),
-	REGISTER_ITS_DESC(GITS_IIDR,
-		vgic_mmio_read_its_iidr, its_mmio_write_wi, 4,
+	REGISTER_ITS_DESC_UACCESS(GITS_IIDR,
+		vgic_mmio_read_its_iidr, its_mmio_write_wi,
+		vgic_mmio_uaccess_write_its_iidr, 4,
 		VGIC_ACCESS_32bit),
 	REGISTER_ITS_DESC(GITS_TYPER,
 		vgic_mmio_read_its_typer, its_mmio_write_wi, 8,
-- 
2.5.5

[PATCH v5 10/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_iidr

[Eric Auger]

The GITS_IIDR revision field is used to encode the migration ABI
revision. So we need to restore it to check the table layout is
readable by the destination.

By writing the IIDR, userspace thus force the ABI revision to be
used and this msut be less or equal than the max revision KVM supports.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5
- rename user_revision into abi_rev and REV into MAX_ABI_REV
- IIDR reports abi_rev set by userspace if any.
- If value set by userspace exceeds the max supported revision, an
  error is reported.
- add some defines

v4: creation
---
 include/linux/irqchip/arm-gic-v3.h |  4 ++++
 virt/kvm/arm/vgic/vgic-its.c       | 23 ++++++++++++++++++++---
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 9648bad..54c20bd 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -241,6 +241,10 @@
 #define GITS_TYPER_PTA			(1UL << 19)
 #define GITS_TYPER_HWCOLLCNT_SHIFT	24
 
+#define GITS_IIDR_REV_SHIFT		12
+#define GITS_IIDR_REV(r)		(((r) >> GITS_IIDR_REV_SHIFT) & 0xf)
+#define GITS_IIDR_PRODUCTID_SHIFT	24
+
 #define GITS_CBASER_VALID			(1ULL << 63)
 #define GITS_CBASER_SHAREABILITY_SHIFT		(10)
 #define GITS_CBASER_INNER_CACHEABILITY_SHIFT	(59)
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 69ecfe4..1b5797e 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -401,7 +401,23 @@ static unsigned long vgic_mmio_read_its_iidr(struct kvm *kvm,
 					     struct vgic_its *its,
 					     gpa_t addr, unsigned int len)
 {
-	return (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
+	return (PRODUCT_ID_KVM << GITS_IIDR_PRODUCTID_SHIFT) |
+	       (its->abi_rev << GITS_IIDR_REV_SHIFT) | IMPLEMENTER_ARM;
+}
+
+static int vgic_mmio_uaccess_write_its_iidr(struct kvm *kvm,
+					    struct vgic_its *its,
+					    gpa_t addr, unsigned int len,
+					    unsigned long val)
+{
+	u64 tmp = 0;
+
+	tmp = update_64bit_reg(tmp, addr & 3, len, val);
+	tmp = GITS_IIDR_REV(tmp);
+
+	if (tmp > MAX_ABI_REV)
+		return -EINVAL;
+	return vgic_its_set_abi(its, tmp);
 }
 
 static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,
@@ -1382,8 +1398,9 @@ static struct vgic_register_region its_registers[] = {
 	REGISTER_ITS_DESC(GITS_CTLR,
 		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4,
 		VGIC_ACCESS_32bit),
-	REGISTER_ITS_DESC(GITS_IIDR,
-		vgic_mmio_read_its_iidr, its_mmio_write_wi, 4,
+	REGISTER_ITS_DESC_UACCESS(GITS_IIDR,
+		vgic_mmio_read_its_iidr, its_mmio_write_wi,
+		vgic_mmio_uaccess_write_its_iidr, 4,
 		VGIC_ACCESS_32bit),
 	REGISTER_ITS_DESC(GITS_TYPER,
 		vgic_mmio_read_its_typer, its_mmio_write_wi, 8,
-- 
2.5.5

[PATCH v5 11/22] KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors

[Eric Auger]

Up to now the MAPD's ITT size field has been ignored. It encodes
the number of eventid bit minus 1. It should be used to check
the eventid when a MAPTI command is issued on a device. Let's
store the number of eventid bits in the its_device and do the
check on MAPTI. Also make sure the ITT size field does
not exceed the GITS_TYPER IDBITS field.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- its_cmd_get_size macro now returns the actual number of eventid bits
- use GIC_ENCODE_SZ macro to encode ID_bits

v3 -> v4:
- VITS_TYPER_IDBITS set to 16 to be homogeneous with VITS_ESZ definition
  and correct users
- nb_eventid_bits correspond to the actual number of eventid bits
---
 include/linux/irqchip/arm-gic-v3.h |  2 ++
 virt/kvm/arm/vgic/vgic-its.c       | 15 ++++++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 54c20bd..0c6798c 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -345,9 +345,11 @@
 #define E_ITS_INT_UNMAPPED_INTERRUPT		0x010307
 #define E_ITS_CLEAR_UNMAPPED_INTERRUPT		0x010507
 #define E_ITS_MAPD_DEVICE_OOR			0x010801
+#define E_ITS_MAPD_ITTSIZE_OOR			0x010802
 #define E_ITS_MAPC_PROCNUM_OOR			0x010902
 #define E_ITS_MAPC_COLLECTION_OOR		0x010903
 #define E_ITS_MAPTI_UNMAPPED_DEVICE		0x010a04
+#define E_ITS_MAPTI_ID_OOR			0x010a05
 #define E_ITS_MAPTI_PHYSICALID_OOR		0x010a06
 #define E_ITS_INV_UNMAPPED_INTERRUPT		0x010c07
 #define E_ITS_INVALL_UNMAPPED_COLLECTION	0x010d09
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 1b5797e..0f3c8f3 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -105,6 +105,7 @@ struct its_device {
 
 	/* the head for the list of ITTEs */
 	struct list_head itt_head;
+	u32 nb_eventid_bits;
 	u32 device_id;
 };
 
@@ -191,6 +192,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
 
 #define GIC_LPI_OFFSET 8192
 
+#define VITS_TYPER_IDBITS 16
+
 /*
  * Finds and returns a collection in the ITS collection table.
  * Must be called with the its_lock mutex held.
@@ -391,7 +394,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
 	 * DevBits low - as least for the time being.
 	 */
 	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
-	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
+	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
 	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
 
 	return extract_bytes(reg, addr & 7, len);
@@ -562,6 +565,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
 
 #define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
 #define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
+#define its_cmd_get_size(cmd)		(its_cmd_mask_field(cmd, 1,  0,  5) + 1)
 #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
 #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
 #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
@@ -752,6 +756,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 	if (!device)
 		return E_ITS_MAPTI_UNMAPPED_DEVICE;
 
+	if (event_id >= BIT_ULL(device->nb_eventid_bits))
+		return E_ITS_MAPTI_ID_OOR;
+
 	if (its_cmd_get_command(its_cmd) == GITS_CMD_MAPTI)
 		lpi_nr = its_cmd_get_physical_id(its_cmd);
 	else
@@ -832,11 +839,15 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 {
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	bool valid = its_cmd_get_validbit(its_cmd);
+	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
 	struct its_device *device;
 
 	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
 		return E_ITS_MAPD_DEVICE_OOR;
 
+	if (valid && nb_eventid_bits > VITS_TYPER_IDBITS)
+		return E_ITS_MAPD_ITTSIZE_OOR;
+
 	device = find_its_device(its, device_id);
 
 	/*
@@ -859,6 +870,8 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 		return -ENOMEM;
 
 	device->device_id = device_id;
+	device->nb_eventid_bits = nb_eventid_bits;
+
 	INIT_LIST_HEAD(&device->itt_head);
 
 	list_add_tail(&device->dev_list, &its->device_list);
-- 
2.5.5

[PATCH v5 11/22] KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors

[Eric Auger]

Up to now the MAPD's ITT size field has been ignored. It encodes
the number of eventid bit minus 1. It should be used to check
the eventid when a MAPTI command is issued on a device. Let's
store the number of eventid bits in the its_device and do the
check on MAPTI. Also make sure the ITT size field does
not exceed the GITS_TYPER IDBITS field.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- its_cmd_get_size macro now returns the actual number of eventid bits
- use GIC_ENCODE_SZ macro to encode ID_bits

v3 -> v4:
- VITS_TYPER_IDBITS set to 16 to be homogeneous with VITS_ESZ definition
  and correct users
- nb_eventid_bits correspond to the actual number of eventid bits
---
 include/linux/irqchip/arm-gic-v3.h |  2 ++
 virt/kvm/arm/vgic/vgic-its.c       | 15 ++++++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 54c20bd..0c6798c 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -345,9 +345,11 @@
 #define E_ITS_INT_UNMAPPED_INTERRUPT		0x010307
 #define E_ITS_CLEAR_UNMAPPED_INTERRUPT		0x010507
 #define E_ITS_MAPD_DEVICE_OOR			0x010801
+#define E_ITS_MAPD_ITTSIZE_OOR			0x010802
 #define E_ITS_MAPC_PROCNUM_OOR			0x010902
 #define E_ITS_MAPC_COLLECTION_OOR		0x010903
 #define E_ITS_MAPTI_UNMAPPED_DEVICE		0x010a04
+#define E_ITS_MAPTI_ID_OOR			0x010a05
 #define E_ITS_MAPTI_PHYSICALID_OOR		0x010a06
 #define E_ITS_INV_UNMAPPED_INTERRUPT		0x010c07
 #define E_ITS_INVALL_UNMAPPED_COLLECTION	0x010d09
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 1b5797e..0f3c8f3 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -105,6 +105,7 @@ struct its_device {
 
 	/* the head for the list of ITTEs */
 	struct list_head itt_head;
+	u32 nb_eventid_bits;
 	u32 device_id;
 };
 
@@ -191,6 +192,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
 
 #define GIC_LPI_OFFSET 8192
 
+#define VITS_TYPER_IDBITS 16
+
 /*
  * Finds and returns a collection in the ITS collection table.
  * Must be called with the its_lock mutex held.
@@ -391,7 +394,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
 	 * DevBits low - as least for the time being.
 	 */
 	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
-	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
+	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
 	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
 
 	return extract_bytes(reg, addr & 7, len);
@@ -562,6 +565,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
 
 #define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
 #define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
+#define its_cmd_get_size(cmd)		(its_cmd_mask_field(cmd, 1,  0,  5) + 1)
 #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
 #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
 #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
@@ -752,6 +756,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 	if (!device)
 		return E_ITS_MAPTI_UNMAPPED_DEVICE;
 
+	if (event_id >= BIT_ULL(device->nb_eventid_bits))
+		return E_ITS_MAPTI_ID_OOR;
+
 	if (its_cmd_get_command(its_cmd) == GITS_CMD_MAPTI)
 		lpi_nr = its_cmd_get_physical_id(its_cmd);
 	else
@@ -832,11 +839,15 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 {
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	bool valid = its_cmd_get_validbit(its_cmd);
+	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
 	struct its_device *device;
 
 	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
 		return E_ITS_MAPD_DEVICE_OOR;
 
+	if (valid && nb_eventid_bits > VITS_TYPER_IDBITS)
+		return E_ITS_MAPD_ITTSIZE_OOR;
+
 	device = find_its_device(its, device_id);
 
 	/*
@@ -859,6 +870,8 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 		return -ENOMEM;
 
 	device->device_id = device_id;
+	device->nb_eventid_bits = nb_eventid_bits;
+
 	INIT_LIST_HEAD(&device->itt_head);
 
 	list_add_tail(&device->dev_list, &its->device_list);
-- 
2.5.5

[PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Eric Auger]

Up to now the MAPD ITT_addr had been ignored. We will need it
for save/restore. Let's record it in the its_device struct.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- its_cmd_get_ittaddr macro now returns the actual ITT GPA

v3 -> v4:
- in vgic_its_cmd_handle_mapd, itt_addr directly is shifted
- correct ittaddr bitmask to support 48bit GPA
---
 virt/kvm/arm/vgic/vgic-its.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 0f3c8f3..757598d 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -106,6 +106,7 @@ struct its_device {
 	/* the head for the list of ITTEs */
 	struct list_head itt_head;
 	u32 nb_eventid_bits;
+	gpa_t itt_addr;
 	u32 device_id;
 };
 
@@ -569,6 +570,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
 #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
 #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
 #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
+#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 47) << 8)
 #define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
 #define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
 
@@ -840,6 +842,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	bool valid = its_cmd_get_validbit(its_cmd);
 	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
+	gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
 	struct its_device *device;
 
 	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
@@ -871,6 +874,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 
 	device->device_id = device_id;
 	device->nb_eventid_bits = nb_eventid_bits;
+	device->itt_addr = itt_addr;
 
 	INIT_LIST_HEAD(&device->itt_head);
 
-- 
2.5.5

[PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Eric Auger]

Up to now the MAPD ITT_addr had been ignored. We will need it
for save/restore. Let's record it in the its_device struct.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- its_cmd_get_ittaddr macro now returns the actual ITT GPA

v3 -> v4:
- in vgic_its_cmd_handle_mapd, itt_addr directly is shifted
- correct ittaddr bitmask to support 48bit GPA
---
 virt/kvm/arm/vgic/vgic-its.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 0f3c8f3..757598d 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -106,6 +106,7 @@ struct its_device {
 	/* the head for the list of ITTEs */
 	struct list_head itt_head;
 	u32 nb_eventid_bits;
+	gpa_t itt_addr;
 	u32 device_id;
 };
 
@@ -569,6 +570,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
 #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
 #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
 #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
+#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 47) << 8)
 #define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
 #define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
 
@@ -840,6 +842,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 	u32 device_id = its_cmd_get_deviceid(its_cmd);
 	bool valid = its_cmd_get_validbit(its_cmd);
 	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
+	gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
 	struct its_device *device;
 
 	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
@@ -871,6 +874,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 
 	device->device_id = device_id;
 	device->nb_eventid_bits = nb_eventid_bits;
+	device->itt_addr = itt_addr;
 
 	INIT_LIST_HEAD(&device->itt_head);
 
-- 
2.5.5

[PATCH v5 13/22] KVM: arm64: vgic-its: Check the device id matches TYPER DEVBITS range

[Eric Auger]

On MAPD we currently check the device id can be stored in the device table.
Let's first check it can be encoded within the range defined by TYPER
DEVBITS.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

v4 -> v5:
- use GIC_ENCODE_SZ macro

v3 -> v4:
- VITS_TYPER_DEVBITS set to 16 for homogeneity
- use BIT_ULL
---
 virt/kvm/arm/vgic/vgic-its.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 757598d..de1ed6d 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -194,6 +194,7 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
 #define GIC_LPI_OFFSET 8192
 
 #define VITS_TYPER_IDBITS 16
+#define VITS_TYPER_DEVBITS 16
 
 /*
  * Finds and returns a collection in the ITS collection table.
@@ -394,7 +395,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
 	 * To avoid memory waste in the guest, we keep the number of IDBits and
 	 * DevBits low - as least for the time being.
 	 */
-	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
+	reg |= GIC_ENCODE_SZ(VITS_TYPER_DEVBITS, 5) << GITS_TYPER_DEVBITS_SHIFT;
 	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
 	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
 
@@ -639,10 +640,10 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
  * Check whether an ID can be stored into the corresponding guest table.
  * For a direct table this is pretty easy, but gets a bit nasty for
  * indirect tables. We check whether the resulting guest physical address
- * is actually valid (covered by a memslot and guest accessbible).
+ * is actually valid (covered by a memslot and guest accessible).
  * For this we have to read the respective first level entry.
  */
-static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
+static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
 {
 	int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
 	int index;
@@ -650,6 +651,9 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
 	gfn_t gfn;
 	int esz = GITS_BASER_ENTRY_SIZE(baser);
 
+	if (id >= BIT_ULL(VITS_TYPER_DEVBITS))
+		return false;
+
 	if (!(baser & GITS_BASER_INDIRECT)) {
 		phys_addr_t addr;
 
-- 
2.5.5

[PATCH v5 13/22] KVM: arm64: vgic-its: Check the device id matches TYPER DEVBITS range

[Eric Auger]

On MAPD we currently check the device id can be stored in the device table.
Let's first check it can be encoded within the range defined by TYPER
DEVBITS.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

v4 -> v5:
- use GIC_ENCODE_SZ macro

v3 -> v4:
- VITS_TYPER_DEVBITS set to 16 for homogeneity
- use BIT_ULL
---
 virt/kvm/arm/vgic/vgic-its.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 757598d..de1ed6d 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -194,6 +194,7 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
 #define GIC_LPI_OFFSET 8192
 
 #define VITS_TYPER_IDBITS 16
+#define VITS_TYPER_DEVBITS 16
 
 /*
  * Finds and returns a collection in the ITS collection table.
@@ -394,7 +395,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
 	 * To avoid memory waste in the guest, we keep the number of IDBits and
 	 * DevBits low - as least for the time being.
 	 */
-	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
+	reg |= GIC_ENCODE_SZ(VITS_TYPER_DEVBITS, 5) << GITS_TYPER_DEVBITS_SHIFT;
 	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
 	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
 
@@ -639,10 +640,10 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
  * Check whether an ID can be stored into the corresponding guest table.
  * For a direct table this is pretty easy, but gets a bit nasty for
  * indirect tables. We check whether the resulting guest physical address
- * is actually valid (covered by a memslot and guest accessbible).
+ * is actually valid (covered by a memslot and guest accessible).
  * For this we have to read the respective first level entry.
  */
-static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
+static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
 {
 	int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
 	int index;
@@ -650,6 +651,9 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
 	gfn_t gfn;
 	int esz = GITS_BASER_ENTRY_SIZE(baser);
 
+	if (id >= BIT_ULL(VITS_TYPER_DEVBITS))
+		return false;
+
 	if (!(baser & GITS_BASER_INDIRECT)) {
 		phys_addr_t addr;
 
-- 
2.5.5

[PATCH v5 14/22] KVM: arm64: vgic-its: KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES

[Eric Auger]

Introduce new attributes in KVM_DEV_ARM_VGIC_GRP_CTRL group:
- KVM_DEV_ARM_ITS_SAVE_TABLES: saves the ITS tables into guest RAM
- KVM_DEV_ARM_ITS_RESTORE_TABLES: restores them into VGIC internal
  structures.

We hold the vcpus lock during the save and restore to make
sure no vcpu is running.

At this stage the functionality is not yet implemented. Only
the skeleton is put in place.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

v4 -> v5:
- use KVM_DEV_ARM_ITS_SAVE_TABLES and KVM_DEV_ARM_ITS_RESTORE_TABLES
- rename *flush* into *save*
- call its_sync_lpi_pending_table at the end of restore
- use abi framework

v3 -> v4:
- pass kvm struct handle to vgic_its_flush/restore_pending_tables
- take the kvm lock and vcpu locks
- ABI revision check
- check attr->attr is null

v1 -> v2:
- remove useless kvm parameter
---
 arch/arm/include/uapi/asm/kvm.h   |   4 +-
 arch/arm64/include/uapi/asm/kvm.h |   4 +-
 virt/kvm/arm/vgic/vgic-its.c      | 108 ++++++++++++++++++++++++++++++++++++--
 3 files changed, 110 insertions(+), 6 deletions(-)

diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index 4beb83b..8e6563c 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -199,7 +199,9 @@ struct kvm_arch_memory_slot {
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK 0x3ff
 #define VGIC_LEVEL_INFO_LINE_LEVEL	0
 
-#define   KVM_DEV_ARM_VGIC_CTRL_INIT    0
+#define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
+#define   KVM_DEV_ARM_ITS_SAVE_TABLES		1
+#define   KVM_DEV_ARM_ITS_RESTORE_TABLES	2
 
 /* KVM_IRQ_LINE irq field index values */
 #define KVM_ARM_IRQ_TYPE_SHIFT		24
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 7e8dd69..1e35115 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -219,7 +219,9 @@ struct kvm_arch_memory_slot {
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK	0x3ff
 #define VGIC_LEVEL_INFO_LINE_LEVEL	0
 
-#define   KVM_DEV_ARM_VGIC_CTRL_INIT	0
+#define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
+#define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
+#define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
 
 /* Device Control API on vcpu fd */
 #define KVM_ARM_VCPU_PMU_V3_CTRL	0
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index de1ed6d..55267ab 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1648,12 +1648,68 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
 }
 
 /**
+ * vgic_its_save_device_tables - Save the device table and all ITT
+ * into guest RAM
+ */
+static int vgic_its_save_device_tables(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+/**
+ * vgic_its_restore_device_tables - Restore the device table and all ITT
+ * from guest RAM to internal data structs
+ */
+static int vgic_its_restore_device_tables(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+/**
+ * vgic_its_save_collection_table - Save the collection table into
+ * guest RAM
+ */
+static int vgic_its_save_collection_table(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+/**
+ * vgic_its_restore_collection_table - reads the collection table
+ * in guest memory and restores the ITS internal state. Requires the
+ * BASER registers to be restored before.
+ */
+static int vgic_its_restore_collection_table(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+/**
  * vgic_its_save_tables_v0 - Save the ITS tables into guest ARM
  * according to v0 ABI
  */
 static int vgic_its_save_tables_v0(struct vgic_its *its)
 {
-	return -ENXIO;
+	struct kvm *kvm = its->dev->kvm;
+	int ret;
+
+	mutex_lock(&kvm->lock);
+
+	if (!lock_all_vcpus(kvm)) {
+		mutex_unlock(&kvm->lock);
+		return -EBUSY;
+	}
+
+	ret = vgic_its_save_device_tables(its);
+	if (ret)
+		goto out;
+
+	ret = vgic_its_save_collection_table(its);
+
+out:
+	unlock_all_vcpus(kvm);
+	mutex_unlock(&kvm->lock);
+	return ret;
 }
 
 /**
@@ -1663,7 +1719,41 @@ static int vgic_its_save_tables_v0(struct vgic_its *its)
  */
 static int vgic_its_restore_tables_v0(struct vgic_its *its)
 {
-	return -ENXIO;
+	struct kvm *kvm = its->dev->kvm;
+	struct kvm_vcpu *vcpu;
+	int ret, c;
+
+	mutex_lock(&kvm->lock);
+
+	if (!lock_all_vcpus(kvm)) {
+		mutex_unlock(&kvm->lock);
+		return -EBUSY;
+	}
+
+	ret = vgic_its_restore_collection_table(its);
+	if (ret)
+		goto out;
+
+	ret = vgic_its_restore_device_tables(its);
+
+	kvm_for_each_vcpu(c, vcpu, kvm) {
+		ret = its_sync_lpi_pending_table(vcpu);
+		if (ret)
+			break;
+	}
+
+out:
+	unlock_all_vcpus(kvm);
+	mutex_unlock(&kvm->lock);
+
+	if (ret)
+		return ret;
+
+	/*
+	 * On restore path, MSI injections can happen before the
+	 * first VCPU run so let's complete the GIC init here.
+	 */
+	return kvm_vgic_map_resources(its->dev->kvm);
 }
 
 static int vgic_its_commit_v0(struct vgic_its *its)
@@ -1718,6 +1808,10 @@ static int vgic_its_has_attr(struct kvm_device *dev,
 		switch (attr->attr) {
 		case KVM_DEV_ARM_VGIC_CTRL_INIT:
 			return 0;
+		case KVM_DEV_ARM_ITS_SAVE_TABLES:
+			return 0;
+		case KVM_DEV_ARM_ITS_RESTORE_TABLES:
+			return 0;
 		}
 		break;
 	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS:
@@ -1753,14 +1847,20 @@ static int vgic_its_set_attr(struct kvm_device *dev,
 
 		return 0;
 	}
-	case KVM_DEV_ARM_VGIC_GRP_CTRL:
+	case KVM_DEV_ARM_VGIC_GRP_CTRL: {
+		const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+
 		switch (attr->attr) {
 		case KVM_DEV_ARM_VGIC_CTRL_INIT:
 			its->initialized = true;
 
 			return 0;
+		case KVM_DEV_ARM_ITS_SAVE_TABLES:
+			return abi->save_tables(its);
+		case KVM_DEV_ARM_ITS_RESTORE_TABLES:
+			return abi->restore_tables(its);
 		}
-		break;
+	}
 	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
 		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
 		u64 reg;
-- 
2.5.5

[PATCH v5 14/22] KVM: arm64: vgic-its: KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES

[Eric Auger]

Introduce new attributes in KVM_DEV_ARM_VGIC_GRP_CTRL group:
- KVM_DEV_ARM_ITS_SAVE_TABLES: saves the ITS tables into guest RAM
- KVM_DEV_ARM_ITS_RESTORE_TABLES: restores them into VGIC internal
  structures.

We hold the vcpus lock during the save and restore to make
sure no vcpu is running.

At this stage the functionality is not yet implemented. Only
the skeleton is put in place.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

v4 -> v5:
- use KVM_DEV_ARM_ITS_SAVE_TABLES and KVM_DEV_ARM_ITS_RESTORE_TABLES
- rename *flush* into *save*
- call its_sync_lpi_pending_table at the end of restore
- use abi framework

v3 -> v4:
- pass kvm struct handle to vgic_its_flush/restore_pending_tables
- take the kvm lock and vcpu locks
- ABI revision check
- check attr->attr is null

v1 -> v2:
- remove useless kvm parameter
---
 arch/arm/include/uapi/asm/kvm.h   |   4 +-
 arch/arm64/include/uapi/asm/kvm.h |   4 +-
 virt/kvm/arm/vgic/vgic-its.c      | 108 ++++++++++++++++++++++++++++++++++++--
 3 files changed, 110 insertions(+), 6 deletions(-)

diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index 4beb83b..8e6563c 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -199,7 +199,9 @@ struct kvm_arch_memory_slot {
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK 0x3ff
 #define VGIC_LEVEL_INFO_LINE_LEVEL	0
 
-#define   KVM_DEV_ARM_VGIC_CTRL_INIT    0
+#define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
+#define   KVM_DEV_ARM_ITS_SAVE_TABLES		1
+#define   KVM_DEV_ARM_ITS_RESTORE_TABLES	2
 
 /* KVM_IRQ_LINE irq field index values */
 #define KVM_ARM_IRQ_TYPE_SHIFT		24
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 7e8dd69..1e35115 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -219,7 +219,9 @@ struct kvm_arch_memory_slot {
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK	0x3ff
 #define VGIC_LEVEL_INFO_LINE_LEVEL	0
 
-#define   KVM_DEV_ARM_VGIC_CTRL_INIT	0
+#define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
+#define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
+#define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
 
 /* Device Control API on vcpu fd */
 #define KVM_ARM_VCPU_PMU_V3_CTRL	0
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index de1ed6d..55267ab 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1648,12 +1648,68 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
 }
 
 /**
+ * vgic_its_save_device_tables - Save the device table and all ITT
+ * into guest RAM
+ */
+static int vgic_its_save_device_tables(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+/**
+ * vgic_its_restore_device_tables - Restore the device table and all ITT
+ * from guest RAM to internal data structs
+ */
+static int vgic_its_restore_device_tables(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+/**
+ * vgic_its_save_collection_table - Save the collection table into
+ * guest RAM
+ */
+static int vgic_its_save_collection_table(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+/**
+ * vgic_its_restore_collection_table - reads the collection table
+ * in guest memory and restores the ITS internal state. Requires the
+ * BASER registers to be restored before.
+ */
+static int vgic_its_restore_collection_table(struct vgic_its *its)
+{
+	return -ENXIO;
+}
+
+/**
  * vgic_its_save_tables_v0 - Save the ITS tables into guest ARM
  * according to v0 ABI
  */
 static int vgic_its_save_tables_v0(struct vgic_its *its)
 {
-	return -ENXIO;
+	struct kvm *kvm = its->dev->kvm;
+	int ret;
+
+	mutex_lock(&kvm->lock);
+
+	if (!lock_all_vcpus(kvm)) {
+		mutex_unlock(&kvm->lock);
+		return -EBUSY;
+	}
+
+	ret = vgic_its_save_device_tables(its);
+	if (ret)
+		goto out;
+
+	ret = vgic_its_save_collection_table(its);
+
+out:
+	unlock_all_vcpus(kvm);
+	mutex_unlock(&kvm->lock);
+	return ret;
 }
 
 /**
@@ -1663,7 +1719,41 @@ static int vgic_its_save_tables_v0(struct vgic_its *its)
  */
 static int vgic_its_restore_tables_v0(struct vgic_its *its)
 {
-	return -ENXIO;
+	struct kvm *kvm = its->dev->kvm;
+	struct kvm_vcpu *vcpu;
+	int ret, c;
+
+	mutex_lock(&kvm->lock);
+
+	if (!lock_all_vcpus(kvm)) {
+		mutex_unlock(&kvm->lock);
+		return -EBUSY;
+	}
+
+	ret = vgic_its_restore_collection_table(its);
+	if (ret)
+		goto out;
+
+	ret = vgic_its_restore_device_tables(its);
+
+	kvm_for_each_vcpu(c, vcpu, kvm) {
+		ret = its_sync_lpi_pending_table(vcpu);
+		if (ret)
+			break;
+	}
+
+out:
+	unlock_all_vcpus(kvm);
+	mutex_unlock(&kvm->lock);
+
+	if (ret)
+		return ret;
+
+	/*
+	 * On restore path, MSI injections can happen before the
+	 * first VCPU run so let's complete the GIC init here.
+	 */
+	return kvm_vgic_map_resources(its->dev->kvm);
 }
 
 static int vgic_its_commit_v0(struct vgic_its *its)
@@ -1718,6 +1808,10 @@ static int vgic_its_has_attr(struct kvm_device *dev,
 		switch (attr->attr) {
 		case KVM_DEV_ARM_VGIC_CTRL_INIT:
 			return 0;
+		case KVM_DEV_ARM_ITS_SAVE_TABLES:
+			return 0;
+		case KVM_DEV_ARM_ITS_RESTORE_TABLES:
+			return 0;
 		}
 		break;
 	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS:
@@ -1753,14 +1847,20 @@ static int vgic_its_set_attr(struct kvm_device *dev,
 
 		return 0;
 	}
-	case KVM_DEV_ARM_VGIC_GRP_CTRL:
+	case KVM_DEV_ARM_VGIC_GRP_CTRL: {
+		const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+
 		switch (attr->attr) {
 		case KVM_DEV_ARM_VGIC_CTRL_INIT:
 			its->initialized = true;
 
 			return 0;
+		case KVM_DEV_ARM_ITS_SAVE_TABLES:
+			return abi->save_tables(its);
+		case KVM_DEV_ARM_ITS_RESTORE_TABLES:
+			return abi->restore_tables(its);
 		}
-		break;
+	}
 	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
 		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
 		u64 reg;
-- 
2.5.5

[PATCH v5 15/22] KVM: arm64: vgic-its: vgic_its_alloc_ite/device

[Eric Auger]

Add two new helpers to allocate an its ite and an its device.
This will avoid duplication on restore path.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>

---

v4 -> v5:
- add Marc's A-b

v3 -> v4:
- fix allocation
- add comment about its_lock mutex hold

v1 -> v2:
- report itt_size fix and remove ITE_SIZE
- s/itte/ite/g
---
 virt/kvm/arm/vgic/vgic-its.c | 73 ++++++++++++++++++++++++++++++--------------
 1 file changed, 50 insertions(+), 23 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 55267ab..56c5123 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -742,6 +742,27 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
 	kfree(collection);
 }
 
+/* Must be called with its_lock mutex held */
+static int vgic_its_alloc_ite(struct its_device *device,
+			       struct its_ite **itep,
+			       struct its_collection *collection,
+			       u32 lpi_id, u32 event_id)
+{
+	struct its_ite *ite;
+
+	ite = kzalloc(sizeof(*ite), GFP_KERNEL);
+	if (!ite)
+		return -ENOMEM;
+
+	ite->event_id	= event_id;
+	ite->collection = collection;
+	ite->lpi = lpi_id;
+
+	list_add_tail(&ite->ite_list, &device->itt_head);
+	*itep = ite;
+	return 0;
+}
+
 /*
  * The MAPTI and MAPI commands map LPIs to ITTEs.
  * Must be called with its_lock mutex held.
@@ -755,7 +776,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 	struct its_ite *ite;
 	struct its_device *device;
 	struct its_collection *collection, *new_coll = NULL;
-	int lpi_nr;
+	int lpi_nr, ret;
 	struct vgic_irq *irq;
 
 	device = find_its_device(its, device_id);
@@ -785,19 +806,13 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 		new_coll = collection;
 	}
 
-	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
-	if (!ite) {
+	ret = vgic_its_alloc_ite(device, &ite, collection, lpi_nr, event_id);
+	if (ret) {
 		if (new_coll)
 			vgic_its_free_collection(its, coll_id);
-		return -ENOMEM;
+		return ret;
 	}
 
-	ite->event_id	= event_id;
-	list_add_tail(&ite->ite_list, &device->itt_head);
-
-	ite->collection = collection;
-	ite->lpi = lpi_nr;
-
 	irq = vgic_add_lpi(kvm, lpi_nr);
 	if (IS_ERR(irq)) {
 		if (new_coll)
@@ -836,6 +851,29 @@ static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
 	kfree(device);
 }
 
+/* Must be called with its_lock mutex held */
+static int vgic_its_alloc_device(struct vgic_its *its,
+				 struct its_device **devp,
+				 u32 device_id, gpa_t itt_addr,
+				 u8 nb_eventid_bits)
+{
+	struct its_device *device;
+
+	device = kzalloc(sizeof(*device), GFP_KERNEL);
+	if (!device)
+		return -ENOMEM;
+
+	device->device_id = device_id;
+	device->itt_addr = itt_addr;
+	device->nb_eventid_bits = nb_eventid_bits;
+	INIT_LIST_HEAD(&device->itt_head);
+
+	list_add_tail(&device->dev_list, &its->device_list);
+	*devp = device;
+
+	return 0;
+}
+
 /*
  * MAPD maps or unmaps a device ID to Interrupt Translation Tables (ITTs).
  * Must be called with the its_lock mutex held.
@@ -872,19 +910,8 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 	if (!valid)
 		return 0;
 
-	device = kzalloc(sizeof(struct its_device), GFP_KERNEL);
-	if (!device)
-		return -ENOMEM;
-
-	device->device_id = device_id;
-	device->nb_eventid_bits = nb_eventid_bits;
-	device->itt_addr = itt_addr;
-
-	INIT_LIST_HEAD(&device->itt_head);
-
-	list_add_tail(&device->dev_list, &its->device_list);
-
-	return 0;
+	return vgic_its_alloc_device(its, &device, device_id,
+				     itt_addr, nb_eventid_bits);
 }
 
 /*
-- 
2.5.5

[PATCH v5 15/22] KVM: arm64: vgic-its: vgic_its_alloc_ite/device

[Eric Auger]

Add two new helpers to allocate an its ite and an its device.
This will avoid duplication on restore path.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>

---

v4 -> v5:
- add Marc's A-b

v3 -> v4:
- fix allocation
- add comment about its_lock mutex hold

v1 -> v2:
- report itt_size fix and remove ITE_SIZE
- s/itte/ite/g
---
 virt/kvm/arm/vgic/vgic-its.c | 73 ++++++++++++++++++++++++++++++--------------
 1 file changed, 50 insertions(+), 23 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 55267ab..56c5123 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -742,6 +742,27 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
 	kfree(collection);
 }
 
+/* Must be called with its_lock mutex held */
+static int vgic_its_alloc_ite(struct its_device *device,
+			       struct its_ite **itep,
+			       struct its_collection *collection,
+			       u32 lpi_id, u32 event_id)
+{
+	struct its_ite *ite;
+
+	ite = kzalloc(sizeof(*ite), GFP_KERNEL);
+	if (!ite)
+		return -ENOMEM;
+
+	ite->event_id	= event_id;
+	ite->collection = collection;
+	ite->lpi = lpi_id;
+
+	list_add_tail(&ite->ite_list, &device->itt_head);
+	*itep = ite;
+	return 0;
+}
+
 /*
  * The MAPTI and MAPI commands map LPIs to ITTEs.
  * Must be called with its_lock mutex held.
@@ -755,7 +776,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 	struct its_ite *ite;
 	struct its_device *device;
 	struct its_collection *collection, *new_coll = NULL;
-	int lpi_nr;
+	int lpi_nr, ret;
 	struct vgic_irq *irq;
 
 	device = find_its_device(its, device_id);
@@ -785,19 +806,13 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
 		new_coll = collection;
 	}
 
-	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
-	if (!ite) {
+	ret = vgic_its_alloc_ite(device, &ite, collection, lpi_nr, event_id);
+	if (ret) {
 		if (new_coll)
 			vgic_its_free_collection(its, coll_id);
-		return -ENOMEM;
+		return ret;
 	}
 
-	ite->event_id	= event_id;
-	list_add_tail(&ite->ite_list, &device->itt_head);
-
-	ite->collection = collection;
-	ite->lpi = lpi_nr;
-
 	irq = vgic_add_lpi(kvm, lpi_nr);
 	if (IS_ERR(irq)) {
 		if (new_coll)
@@ -836,6 +851,29 @@ static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
 	kfree(device);
 }
 
+/* Must be called with its_lock mutex held */
+static int vgic_its_alloc_device(struct vgic_its *its,
+				 struct its_device **devp,
+				 u32 device_id, gpa_t itt_addr,
+				 u8 nb_eventid_bits)
+{
+	struct its_device *device;
+
+	device = kzalloc(sizeof(*device), GFP_KERNEL);
+	if (!device)
+		return -ENOMEM;
+
+	device->device_id = device_id;
+	device->itt_addr = itt_addr;
+	device->nb_eventid_bits = nb_eventid_bits;
+	INIT_LIST_HEAD(&device->itt_head);
+
+	list_add_tail(&device->dev_list, &its->device_list);
+	*devp = device;
+
+	return 0;
+}
+
 /*
  * MAPD maps or unmaps a device ID to Interrupt Translation Tables (ITTs).
  * Must be called with the its_lock mutex held.
@@ -872,19 +910,8 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 	if (!valid)
 		return 0;
 
-	device = kzalloc(sizeof(struct its_device), GFP_KERNEL);
-	if (!device)
-		return -ENOMEM;
-
-	device->device_id = device_id;
-	device->nb_eventid_bits = nb_eventid_bits;
-	device->itt_addr = itt_addr;
-
-	INIT_LIST_HEAD(&device->itt_head);
-
-	list_add_tail(&device->dev_list, &its->device_list);
-
-	return 0;
+	return vgic_its_alloc_device(its, &device, device_id,
+				     itt_addr, nb_eventid_bits);
 }
 
 /*
-- 
2.5.5

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Eric Auger]

Add a generic lookup_table() helper whose role consists in
scanning a contiguous table located in guest RAM and applying
a callback on each entry. Entries can be handled as linked lists
since the callback may return an offset to the next entry and
also tell that an entry is the last one.

Helper functions also are added to compute the device/event ID
offset to the next DTE/ITE.

compute_next_devid_offset, compute_next_eventid_offset and
lookup_table will become static in subsequent patches

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- use kvm_read_guest

v3 -> v4:
- remove static to avoid compilation warning
- correct size computation in looup_table()
- defines now encode the number of bits used for devid and eventid offsets
- use BIT() - 1 to encode the max offets
---
 virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 56c5123..c22b35d 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
 
 #define VITS_TYPER_IDBITS 16
 #define VITS_TYPER_DEVBITS 16
+#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
+#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
 
 /*
  * Finds and returns a collection in the ITS collection table.
@@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
 	return ret;
 }
 
+u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
+{
+	struct list_head *e = &dev->dev_list;
+	struct its_device *next;
+	u32 next_offset;
+
+	if (e->next == h)
+		return 0;
+	next = list_entry(e->next, struct its_device, dev_list);
+	next_offset = next->device_id - dev->device_id;
+
+	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
+}
+
+u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
+{
+	struct list_head *e = &ite->ite_list;
+	struct its_ite *next;
+	u32 next_offset;
+
+	if (e->next == h)
+		return 0;
+	next = list_entry(e->next, struct its_ite, ite_list);
+	next_offset = next->event_id - ite->event_id;
+
+	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
+}
+
+/**
+ * entry_fn_t - Callback called on a table entry restore path
+ * @its: its handle
+ * @id: id of the entry
+ * @entry: pointer to the entry
+ * @opaque: pointer to an opaque data
+ * @next_offset: minimal ID offset to the next entry. 0 if this
+ * entry is the last one, 1 if the entry is invalid, >= 1 if an
+ * entry's next_offset field was truly decoded
+ *
+ * Return: < 0 on error, 0 otherwise
+ */
+typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
+			  void *opaque, u32 *next_offset);
+
+/**
+ * lookup_table - scan a contiguous table in guest RAM and applies a function
+ * to each entry
+ *
+ * @its: its handle
+ * @base: base gpa of the table
+ * @size: size of the table in bytes
+ * @esz: entry size in bytes
+ * @start_id: first entry's ID
+ * @fn: function to apply on each entry
+ *
+ * Return: < 0 on error, 1 if last element identified, 0 otherwise
+ */
+int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
+		 int start_id, entry_fn_t fn, void *opaque)
+{
+	void *entry = kzalloc(esz, GFP_KERNEL);
+	struct kvm *kvm = its->dev->kvm;
+	unsigned long len = size;
+	u32 id = start_id;
+	gpa_t gpa = base;
+	int ret;
+
+	while (len > 0) {
+		u32 next_offset;
+		size_t byte_offset;
+
+		ret = kvm_read_guest(kvm, gpa, entry, esz);
+		if (ret)
+			goto out;
+
+		ret = fn(its, id, entry, opaque, &next_offset);
+		if (ret < 0 || (!ret && !next_offset))
+			goto out;
+
+		byte_offset = next_offset * esz;
+		id += next_offset;
+		gpa += byte_offset;
+		len -= byte_offset;
+	}
+	kfree(entry);
+	return 0;
+
+out:
+	kfree(entry);
+	return (ret < 0 ? ret : 1);
+}
+
 /**
  * vgic_its_save_device_tables - Save the device table and all ITT
  * into guest RAM
-- 
2.5.5

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Eric Auger]

Add a generic lookup_table() helper whose role consists in
scanning a contiguous table located in guest RAM and applying
a callback on each entry. Entries can be handled as linked lists
since the callback may return an offset to the next entry and
also tell that an entry is the last one.

Helper functions also are added to compute the device/event ID
offset to the next DTE/ITE.

compute_next_devid_offset, compute_next_eventid_offset and
lookup_table will become static in subsequent patches

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- use kvm_read_guest

v3 -> v4:
- remove static to avoid compilation warning
- correct size computation in looup_table()
- defines now encode the number of bits used for devid and eventid offsets
- use BIT() - 1 to encode the max offets
---
 virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 56c5123..c22b35d 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
 
 #define VITS_TYPER_IDBITS 16
 #define VITS_TYPER_DEVBITS 16
+#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
+#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
 
 /*
  * Finds and returns a collection in the ITS collection table.
@@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
 	return ret;
 }
 
+u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
+{
+	struct list_head *e = &dev->dev_list;
+	struct its_device *next;
+	u32 next_offset;
+
+	if (e->next == h)
+		return 0;
+	next = list_entry(e->next, struct its_device, dev_list);
+	next_offset = next->device_id - dev->device_id;
+
+	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
+}
+
+u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
+{
+	struct list_head *e = &ite->ite_list;
+	struct its_ite *next;
+	u32 next_offset;
+
+	if (e->next == h)
+		return 0;
+	next = list_entry(e->next, struct its_ite, ite_list);
+	next_offset = next->event_id - ite->event_id;
+
+	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
+}
+
+/**
+ * entry_fn_t - Callback called on a table entry restore path
+ * @its: its handle
+ * @id: id of the entry
+ * @entry: pointer to the entry
+ * @opaque: pointer to an opaque data
+ * @next_offset: minimal ID offset to the next entry. 0 if this
+ * entry is the last one, 1 if the entry is invalid, >= 1 if an
+ * entry's next_offset field was truly decoded
+ *
+ * Return: < 0 on error, 0 otherwise
+ */
+typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
+			  void *opaque, u32 *next_offset);
+
+/**
+ * lookup_table - scan a contiguous table in guest RAM and applies a function
+ * to each entry
+ *
+ * @its: its handle
+ * @base: base gpa of the table
+ * @size: size of the table in bytes
+ * @esz: entry size in bytes
+ * @start_id: first entry's ID
+ * @fn: function to apply on each entry
+ *
+ * Return: < 0 on error, 1 if last element identified, 0 otherwise
+ */
+int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
+		 int start_id, entry_fn_t fn, void *opaque)
+{
+	void *entry = kzalloc(esz, GFP_KERNEL);
+	struct kvm *kvm = its->dev->kvm;
+	unsigned long len = size;
+	u32 id = start_id;
+	gpa_t gpa = base;
+	int ret;
+
+	while (len > 0) {
+		u32 next_offset;
+		size_t byte_offset;
+
+		ret = kvm_read_guest(kvm, gpa, entry, esz);
+		if (ret)
+			goto out;
+
+		ret = fn(its, id, entry, opaque, &next_offset);
+		if (ret < 0 || (!ret && !next_offset))
+			goto out;
+
+		byte_offset = next_offset * esz;
+		id += next_offset;
+		gpa += byte_offset;
+		len -= byte_offset;
+	}
+	kfree(entry);
+	return 0;
+
+out:
+	kfree(entry);
+	return (ret < 0 ? ret : 1);
+}
+
 /**
  * vgic_its_save_device_tables - Save the device table and all ITT
  * into guest RAM
-- 
2.5.5

[PATCH v5 17/22] KVM: arm64: vgic-its: Collection table save/restore

[Eric Auger]

The save path copies the collection entries into guest RAM
at the GPA specified in the BASER register. This obviously
requires the BASER to be set. The last written element is a
dummy collection table entry.

We do not index by collection ID as the collection entry
can fit into 8 bytes while containing the collection ID.

On restore path we re-allocate the collection objects.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- add macros for field encoding/decoding
- use abi->cte_esz
- rename flush into save
- check the target_addr is valid

v3 -> v4:
- replaced u64 *ptr by gpa_t gpa
- check the collection does not exist before allocating it

v1 -> v2:
- reword commit message and directly use 8 as entry size
- no kvm parameter anymore
- add helper for flush/restore cte
- table size computed here
- add le64/cpu conversions
---
 virt/kvm/arm/vgic/vgic-its.c | 109 ++++++++++++++++++++++++++++++++++++++++++-
 virt/kvm/arm/vgic/vgic.h     |   9 ++++
 2 files changed, 116 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index c22b35d..484e541 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1785,13 +1785,97 @@ static int vgic_its_restore_device_tables(struct vgic_its *its)
 	return -ENXIO;
 }
 
+static int vgic_its_save_cte(struct vgic_its *its,
+			     struct its_collection *collection,
+			     gpa_t gpa, int esz)
+{
+	u64 val;
+	int ret;
+
+	val = (1ULL << KVM_ITS_CTE_VALID_SHIFT |
+	       ((u64)collection->target_addr << KVM_ITS_CTE_RDBASE_SHIFT) |
+	       collection->collection_id);
+	val = cpu_to_le64(val);
+	ret = kvm_write_guest(its->dev->kvm, gpa, &val, esz);
+	return ret;
+}
+
+static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa,
+				int esz, bool *valid)
+{
+	struct its_collection *collection;
+	struct kvm *kvm = its->dev->kvm;
+	u32 target_addr;
+	u32 coll_id;
+	u64 val;
+	int ret;
+
+	*valid = false;
+
+	ret = kvm_read_guest(kvm, gpa, &val, esz);
+	if (ret)
+		return ret;
+	val = le64_to_cpu(val);
+	*valid = val & KVM_ITS_CTE_VALID_MASK;
+
+	if (!*valid)
+		return 0;
+
+	target_addr = (u32)(val >> KVM_ITS_CTE_RDBASE_SHIFT);
+	coll_id = val & KVM_ITS_CTE_ICID_MASK;
+
+	if (target_addr >= atomic_read(&kvm->online_vcpus))
+		return -EINVAL;
+
+	collection = find_collection(its, coll_id);
+	if (collection)
+		return -EEXIST;
+	ret = vgic_its_alloc_collection(its, &collection, coll_id);
+	if (ret)
+		return ret;
+	collection->target_addr = target_addr;
+	return 0;
+}
+
 /**
  * vgic_its_save_collection_table - Save the collection table into
  * guest RAM
  */
 static int vgic_its_save_collection_table(struct vgic_its *its)
 {
-	return -ENXIO;
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	struct its_collection *collection;
+	u64 val;
+	gpa_t gpa;
+	size_t max_size, filled = 0;
+	int ret, cte_esz = abi->cte_esz;
+
+	gpa = BASER_ADDRESS(its->baser_coll_table);
+	if (!gpa)
+		return 0;
+
+	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
+
+	list_for_each_entry(collection, &its->collection_list, coll_list) {
+		if (filled == max_size)
+			return -ENOSPC;
+		ret = vgic_its_save_cte(its, collection, gpa, cte_esz);
+		if (ret)
+			return ret;
+		gpa += cte_esz;
+		filled += cte_esz;
+	}
+
+	if (filled == max_size)
+		return 0;
+
+	/*
+	 * table is not fully filled, add a last dummy element
+	 * with valid bit unset
+	 */
+	val = 0;
+	ret = kvm_write_guest(its->dev->kvm, gpa, &val, cte_esz);
+	return ret;
 }
 
 /**
@@ -1801,7 +1885,28 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
  */
 static int vgic_its_restore_collection_table(struct vgic_its *its)
 {
-	return -ENXIO;
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	size_t max_size, read = 0;
+	gpa_t gpa;
+	int ret, cte_esz = abi->cte_esz;
+
+	gpa = BASER_ADDRESS(its->baser_coll_table);
+
+	if (!gpa)
+		return 0;
+
+	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
+
+	while (read < max_size) {
+		bool valid;
+
+		ret = vgic_its_restore_cte(its, gpa, cte_esz, &valid);
+		if (!valid || ret)
+			break;
+		gpa += cte_esz;
+		read += cte_esz;
+	}
+	return ret;
 }
 
 /**
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index b87f1c6..56e57c1 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -73,6 +73,15 @@
 				      KVM_REG_ARM_VGIC_SYSREG_CRM_MASK | \
 				      KVM_REG_ARM_VGIC_SYSREG_OP2_MASK)
 
+/*
+ * As per Documentation/virtual/kvm/devices/arm-vgic-its.txt,
+ * below macros are defined for ITS table entry encoding.
+ */
+#define KVM_ITS_CTE_VALID_SHIFT		63
+#define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
+#define KVM_ITS_CTE_RDBASE_SHIFT	16
+#define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
+
 static inline bool irq_is_pending(struct vgic_irq *irq)
 {
 	if (irq->config == VGIC_CONFIG_EDGE)
-- 
2.5.5

[PATCH v5 17/22] KVM: arm64: vgic-its: Collection table save/restore

[Eric Auger]

The save path copies the collection entries into guest RAM
at the GPA specified in the BASER register. This obviously
requires the BASER to be set. The last written element is a
dummy collection table entry.

We do not index by collection ID as the collection entry
can fit into 8 bytes while containing the collection ID.

On restore path we re-allocate the collection objects.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- add macros for field encoding/decoding
- use abi->cte_esz
- rename flush into save
- check the target_addr is valid

v3 -> v4:
- replaced u64 *ptr by gpa_t gpa
- check the collection does not exist before allocating it

v1 -> v2:
- reword commit message and directly use 8 as entry size
- no kvm parameter anymore
- add helper for flush/restore cte
- table size computed here
- add le64/cpu conversions
---
 virt/kvm/arm/vgic/vgic-its.c | 109 ++++++++++++++++++++++++++++++++++++++++++-
 virt/kvm/arm/vgic/vgic.h     |   9 ++++
 2 files changed, 116 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index c22b35d..484e541 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1785,13 +1785,97 @@ static int vgic_its_restore_device_tables(struct vgic_its *its)
 	return -ENXIO;
 }
 
+static int vgic_its_save_cte(struct vgic_its *its,
+			     struct its_collection *collection,
+			     gpa_t gpa, int esz)
+{
+	u64 val;
+	int ret;
+
+	val = (1ULL << KVM_ITS_CTE_VALID_SHIFT |
+	       ((u64)collection->target_addr << KVM_ITS_CTE_RDBASE_SHIFT) |
+	       collection->collection_id);
+	val = cpu_to_le64(val);
+	ret = kvm_write_guest(its->dev->kvm, gpa, &val, esz);
+	return ret;
+}
+
+static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa,
+				int esz, bool *valid)
+{
+	struct its_collection *collection;
+	struct kvm *kvm = its->dev->kvm;
+	u32 target_addr;
+	u32 coll_id;
+	u64 val;
+	int ret;
+
+	*valid = false;
+
+	ret = kvm_read_guest(kvm, gpa, &val, esz);
+	if (ret)
+		return ret;
+	val = le64_to_cpu(val);
+	*valid = val & KVM_ITS_CTE_VALID_MASK;
+
+	if (!*valid)
+		return 0;
+
+	target_addr = (u32)(val >> KVM_ITS_CTE_RDBASE_SHIFT);
+	coll_id = val & KVM_ITS_CTE_ICID_MASK;
+
+	if (target_addr >= atomic_read(&kvm->online_vcpus))
+		return -EINVAL;
+
+	collection = find_collection(its, coll_id);
+	if (collection)
+		return -EEXIST;
+	ret = vgic_its_alloc_collection(its, &collection, coll_id);
+	if (ret)
+		return ret;
+	collection->target_addr = target_addr;
+	return 0;
+}
+
 /**
  * vgic_its_save_collection_table - Save the collection table into
  * guest RAM
  */
 static int vgic_its_save_collection_table(struct vgic_its *its)
 {
-	return -ENXIO;
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	struct its_collection *collection;
+	u64 val;
+	gpa_t gpa;
+	size_t max_size, filled = 0;
+	int ret, cte_esz = abi->cte_esz;
+
+	gpa = BASER_ADDRESS(its->baser_coll_table);
+	if (!gpa)
+		return 0;
+
+	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
+
+	list_for_each_entry(collection, &its->collection_list, coll_list) {
+		if (filled == max_size)
+			return -ENOSPC;
+		ret = vgic_its_save_cte(its, collection, gpa, cte_esz);
+		if (ret)
+			return ret;
+		gpa += cte_esz;
+		filled += cte_esz;
+	}
+
+	if (filled == max_size)
+		return 0;
+
+	/*
+	 * table is not fully filled, add a last dummy element
+	 * with valid bit unset
+	 */
+	val = 0;
+	ret = kvm_write_guest(its->dev->kvm, gpa, &val, cte_esz);
+	return ret;
 }
 
 /**
@@ -1801,7 +1885,28 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
  */
 static int vgic_its_restore_collection_table(struct vgic_its *its)
 {
-	return -ENXIO;
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	size_t max_size, read = 0;
+	gpa_t gpa;
+	int ret, cte_esz = abi->cte_esz;
+
+	gpa = BASER_ADDRESS(its->baser_coll_table);
+
+	if (!gpa)
+		return 0;
+
+	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
+
+	while (read < max_size) {
+		bool valid;
+
+		ret = vgic_its_restore_cte(its, gpa, cte_esz, &valid);
+		if (!valid || ret)
+			break;
+		gpa += cte_esz;
+		read += cte_esz;
+	}
+	return ret;
 }
 
 /**
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index b87f1c6..56e57c1 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -73,6 +73,15 @@
 				      KVM_REG_ARM_VGIC_SYSREG_CRM_MASK | \
 				      KVM_REG_ARM_VGIC_SYSREG_OP2_MASK)
 
+/*
+ * As per Documentation/virtual/kvm/devices/arm-vgic-its.txt,
+ * below macros are defined for ITS table entry encoding.
+ */
+#define KVM_ITS_CTE_VALID_SHIFT		63
+#define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
+#define KVM_ITS_CTE_RDBASE_SHIFT	16
+#define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
+
 static inline bool irq_is_pending(struct vgic_irq *irq)
 {
 	if (irq->config == VGIC_CONFIG_EDGE)
-- 
2.5.5

[PATCH v5 18/22] KVM: arm64: vgic-its: vgic_its_check_id returns the entry's GPA

[Eric Auger]

As vgic_its_check_id() computes the device/collection entry's
GPA, let's return it so that new callers can retrieve it easily.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v3 -> v4:
- check eaddr is not NULL to allow passing NULL eaddr parameter
  to vgic_its_check_id

v2: new
---
 virt/kvm/arm/vgic/vgic-its.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 484e541..35b2ca1 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -645,7 +645,8 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
  * is actually valid (covered by a memslot and guest accessible).
  * For this we have to read the respective first level entry.
  */
-static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
+static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id,
+			      gpa_t *eaddr)
 {
 	int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
 	int index;
@@ -665,6 +666,8 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
 		addr = BASER_ADDRESS(baser) + id * esz;
 		gfn = addr >> PAGE_SHIFT;
 
+		if (eaddr)
+			*eaddr = addr;
 		return kvm_is_visible_gfn(its->dev->kvm, gfn);
 	}
 
@@ -697,6 +700,8 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
 	indirect_ptr += index * esz;
 	gfn = indirect_ptr >> PAGE_SHIFT;
 
+	if (eaddr)
+		*eaddr = indirect_ptr;
 	return kvm_is_visible_gfn(its->dev->kvm, gfn);
 }
 
@@ -706,7 +711,7 @@ static int vgic_its_alloc_collection(struct vgic_its *its,
 {
 	struct its_collection *collection;
 
-	if (!vgic_its_check_id(its, its->baser_coll_table, coll_id))
+	if (!vgic_its_check_id(its, its->baser_coll_table, coll_id, NULL))
 		return E_ITS_MAPC_COLLECTION_OOR;
 
 	collection = kzalloc(sizeof(*collection), GFP_KERNEL);
@@ -889,7 +894,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 	gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
 	struct its_device *device;
 
-	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
+	if (!vgic_its_check_id(its, its->baser_device_table, device_id, NULL))
 		return E_ITS_MAPD_DEVICE_OOR;
 
 	if (valid && nb_eventid_bits > VITS_TYPER_IDBITS)
-- 
2.5.5

[PATCH v5 18/22] KVM: arm64: vgic-its: vgic_its_check_id returns the entry's GPA

[Eric Auger]

As vgic_its_check_id() computes the device/collection entry's
GPA, let's return it so that new callers can retrieve it easily.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v3 -> v4:
- check eaddr is not NULL to allow passing NULL eaddr parameter
  to vgic_its_check_id

v2: new
---
 virt/kvm/arm/vgic/vgic-its.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 484e541..35b2ca1 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -645,7 +645,8 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
  * is actually valid (covered by a memslot and guest accessible).
  * For this we have to read the respective first level entry.
  */
-static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
+static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id,
+			      gpa_t *eaddr)
 {
 	int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
 	int index;
@@ -665,6 +666,8 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
 		addr = BASER_ADDRESS(baser) + id * esz;
 		gfn = addr >> PAGE_SHIFT;
 
+		if (eaddr)
+			*eaddr = addr;
 		return kvm_is_visible_gfn(its->dev->kvm, gfn);
 	}
 
@@ -697,6 +700,8 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
 	indirect_ptr += index * esz;
 	gfn = indirect_ptr >> PAGE_SHIFT;
 
+	if (eaddr)
+		*eaddr = indirect_ptr;
 	return kvm_is_visible_gfn(its->dev->kvm, gfn);
 }
 
@@ -706,7 +711,7 @@ static int vgic_its_alloc_collection(struct vgic_its *its,
 {
 	struct its_collection *collection;
 
-	if (!vgic_its_check_id(its, its->baser_coll_table, coll_id))
+	if (!vgic_its_check_id(its, its->baser_coll_table, coll_id, NULL))
 		return E_ITS_MAPC_COLLECTION_OOR;
 
 	collection = kzalloc(sizeof(*collection), GFP_KERNEL);
@@ -889,7 +894,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
 	gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
 	struct its_device *device;
 
-	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
+	if (!vgic_its_check_id(its, its->baser_device_table, device_id, NULL))
 		return E_ITS_MAPD_DEVICE_OOR;
 
 	if (valid && nb_eventid_bits > VITS_TYPER_IDBITS)
-- 
2.5.5

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Eric Auger]

Introduce routines to save and restore device ITT and their
interrupt table entries (ITE).

The routines will be called on device table save and
restore. They will become static in subsequent patches.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- ITE are now sorted by eventid on the flush
- rename *flush* into *save*
- use macros for shits and masks
- pass ite_esz to vgic_its_save_ite

v3 -> v4:
- lookup_table and compute_next_eventid_offset become static in this
  patch
- remove static along with vgic_its_flush/restore_itt to avoid
  compilation warnings
- next field only computed with a shift (mask removed)
- handle the case where the last element has not been found

v2 -> v3:
- add return 0 in vgic_its_restore_ite (was in subsequent patch)

v2: creation
---
 virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
 virt/kvm/arm/vgic/vgic.h     |   4 ++
 2 files changed, 129 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 35b2ca1..b02fc3f 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -23,6 +23,7 @@
 #include <linux/interrupt.h>
 #include <linux/list.h>
 #include <linux/uaccess.h>
+#include <linux/list_sort.h>
 
 #include <linux/irqchip/arm-gic-v3.h>
 
@@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
 	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
 }
 
-u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
+static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
 {
 	struct list_head *e = &ite->ite_list;
 	struct its_ite *next;
@@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
  *
  * Return: < 0 on error, 1 if last element identified, 0 otherwise
  */
-int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
-		 int start_id, entry_fn_t fn, void *opaque)
+static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
+			int start_id, entry_fn_t fn, void *opaque)
 {
 	void *entry = kzalloc(esz, GFP_KERNEL);
 	struct kvm *kvm = its->dev->kvm;
@@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
 }
 
 /**
+ * vgic_its_save_ite - Save an interrupt translation entry at @gpa
+ */
+static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
+			      struct its_ite *ite, gpa_t gpa, int ite_esz)
+{
+	struct kvm *kvm = its->dev->kvm;
+	u32 next_offset;
+	u64 val;
+
+	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
+	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
+	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
+		ite->collection->collection_id;
+	val = cpu_to_le64(val);
+	return kvm_write_guest(kvm, gpa, &val, ite_esz);
+}
+
+/**
+ * vgic_its_restore_ite - restore an interrupt translation entry
+ * @event_id: id used for indexing
+ * @ptr: pointer to the ITE entry
+ * @opaque: pointer to the its_device
+ * @next: id offset to the next entry
+ */
+static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
+				void *ptr, void *opaque, u32 *next)
+{
+	struct its_device *dev = (struct its_device *)opaque;
+	struct its_collection *collection;
+	struct kvm *kvm = its->dev->kvm;
+	u64 val, *p = (u64 *)ptr;
+	struct vgic_irq *irq;
+	u32 coll_id, lpi_id;
+	struct its_ite *ite;
+	int ret;
+
+	val = *p;
+	*next = 1;
+
+	val = le64_to_cpu(val);
+
+	coll_id = val & KVM_ITS_ITE_ICID_MASK;
+	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
+
+	if (!lpi_id)
+		return 0;
+
+	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
+
+	collection = find_collection(its, coll_id);
+	if (!collection)
+		return -EINVAL;
+
+	ret = vgic_its_alloc_ite(dev, &ite, collection,
+				  lpi_id, event_id);
+	if (ret)
+		return ret;
+
+	irq = vgic_add_lpi(kvm, lpi_id);
+	if (IS_ERR(irq))
+		return PTR_ERR(irq);
+	ite->irq = irq;
+
+	/* restore the configuration of the LPI */
+	ret = update_lpi_config(kvm, irq, NULL);
+	if (ret)
+		return ret;
+
+	update_affinity_ite(kvm, ite);
+	return 0;
+}
+
+static int vgic_its_ite_cmp(void *priv, struct list_head *a,
+			    struct list_head *b)
+{
+	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
+	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
+
+	if (itea->event_id < iteb->event_id)
+		return -1;
+	else
+		return 1;
+}
+
+int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
+{
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	gpa_t base = device->itt_addr;
+	struct its_ite *ite;
+	int ret, ite_esz = abi->ite_esz;
+
+	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
+
+	list_for_each_entry(ite, &device->itt_head, ite_list) {
+		gpa_t gpa = base + ite->event_id * ite_esz;
+
+		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
+{
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	gpa_t base = dev->itt_addr;
+	int ret, ite_esz = abi->ite_esz;
+	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
+
+	ret =  lookup_table(its, base, max_size, ite_esz, 0,
+			    vgic_its_restore_ite, dev);
+
+	if (ret < 0)
+		return ret;
+
+	/* if the last element has not been found we are in trouble */
+	return ret ? 0 : -EINVAL;
+}
+
+/**
  * vgic_its_save_device_tables - Save the device table and all ITT
  * into guest RAM
  */
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 56e57c1..ce57fbd 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -81,6 +81,10 @@
 #define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
 #define KVM_ITS_CTE_RDBASE_SHIFT	16
 #define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
+#define KVM_ITS_ITE_NEXT_SHIFT		48
+#define KVM_ITS_ITE_PINTID_SHIFT	16
+#define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
+#define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
 
 static inline bool irq_is_pending(struct vgic_irq *irq)
 {
-- 
2.5.5

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Eric Auger]

Introduce routines to save and restore device ITT and their
interrupt table entries (ITE).

The routines will be called on device table save and
restore. They will become static in subsequent patches.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- ITE are now sorted by eventid on the flush
- rename *flush* into *save*
- use macros for shits and masks
- pass ite_esz to vgic_its_save_ite

v3 -> v4:
- lookup_table and compute_next_eventid_offset become static in this
  patch
- remove static along with vgic_its_flush/restore_itt to avoid
  compilation warnings
- next field only computed with a shift (mask removed)
- handle the case where the last element has not been found

v2 -> v3:
- add return 0 in vgic_its_restore_ite (was in subsequent patch)

v2: creation
---
 virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
 virt/kvm/arm/vgic/vgic.h     |   4 ++
 2 files changed, 129 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 35b2ca1..b02fc3f 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -23,6 +23,7 @@
 #include <linux/interrupt.h>
 #include <linux/list.h>
 #include <linux/uaccess.h>
+#include <linux/list_sort.h>
 
 #include <linux/irqchip/arm-gic-v3.h>
 
@@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
 	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
 }
 
-u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
+static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
 {
 	struct list_head *e = &ite->ite_list;
 	struct its_ite *next;
@@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
  *
  * Return: < 0 on error, 1 if last element identified, 0 otherwise
  */
-int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
-		 int start_id, entry_fn_t fn, void *opaque)
+static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
+			int start_id, entry_fn_t fn, void *opaque)
 {
 	void *entry = kzalloc(esz, GFP_KERNEL);
 	struct kvm *kvm = its->dev->kvm;
@@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
 }
 
 /**
+ * vgic_its_save_ite - Save an interrupt translation entry at @gpa
+ */
+static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
+			      struct its_ite *ite, gpa_t gpa, int ite_esz)
+{
+	struct kvm *kvm = its->dev->kvm;
+	u32 next_offset;
+	u64 val;
+
+	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
+	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
+	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
+		ite->collection->collection_id;
+	val = cpu_to_le64(val);
+	return kvm_write_guest(kvm, gpa, &val, ite_esz);
+}
+
+/**
+ * vgic_its_restore_ite - restore an interrupt translation entry
+ * @event_id: id used for indexing
+ * @ptr: pointer to the ITE entry
+ * @opaque: pointer to the its_device
+ * @next: id offset to the next entry
+ */
+static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
+				void *ptr, void *opaque, u32 *next)
+{
+	struct its_device *dev = (struct its_device *)opaque;
+	struct its_collection *collection;
+	struct kvm *kvm = its->dev->kvm;
+	u64 val, *p = (u64 *)ptr;
+	struct vgic_irq *irq;
+	u32 coll_id, lpi_id;
+	struct its_ite *ite;
+	int ret;
+
+	val = *p;
+	*next = 1;
+
+	val = le64_to_cpu(val);
+
+	coll_id = val & KVM_ITS_ITE_ICID_MASK;
+	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
+
+	if (!lpi_id)
+		return 0;
+
+	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
+
+	collection = find_collection(its, coll_id);
+	if (!collection)
+		return -EINVAL;
+
+	ret = vgic_its_alloc_ite(dev, &ite, collection,
+				  lpi_id, event_id);
+	if (ret)
+		return ret;
+
+	irq = vgic_add_lpi(kvm, lpi_id);
+	if (IS_ERR(irq))
+		return PTR_ERR(irq);
+	ite->irq = irq;
+
+	/* restore the configuration of the LPI */
+	ret = update_lpi_config(kvm, irq, NULL);
+	if (ret)
+		return ret;
+
+	update_affinity_ite(kvm, ite);
+	return 0;
+}
+
+static int vgic_its_ite_cmp(void *priv, struct list_head *a,
+			    struct list_head *b)
+{
+	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
+	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
+
+	if (itea->event_id < iteb->event_id)
+		return -1;
+	else
+		return 1;
+}
+
+int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
+{
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	gpa_t base = device->itt_addr;
+	struct its_ite *ite;
+	int ret, ite_esz = abi->ite_esz;
+
+	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
+
+	list_for_each_entry(ite, &device->itt_head, ite_list) {
+		gpa_t gpa = base + ite->event_id * ite_esz;
+
+		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
+{
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	gpa_t base = dev->itt_addr;
+	int ret, ite_esz = abi->ite_esz;
+	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
+
+	ret =  lookup_table(its, base, max_size, ite_esz, 0,
+			    vgic_its_restore_ite, dev);
+
+	if (ret < 0)
+		return ret;
+
+	/* if the last element has not been found we are in trouble */
+	return ret ? 0 : -EINVAL;
+}
+
+/**
  * vgic_its_save_device_tables - Save the device table and all ITT
  * into guest RAM
  */
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 56e57c1..ce57fbd 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -81,6 +81,10 @@
 #define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
 #define KVM_ITS_CTE_RDBASE_SHIFT	16
 #define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
+#define KVM_ITS_ITE_NEXT_SHIFT		48
+#define KVM_ITS_ITE_PINTID_SHIFT	16
+#define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
+#define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
 
 static inline bool irq_is_pending(struct vgic_irq *irq)
 {
-- 
2.5.5

[PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Eric Auger]

This patch saves the device table entries into guest RAM.
Both flat table and 2 stage tables are supported. DeviceId
indexing is used.

For each device listed in the device table, we also save
the translation table using the vgic_its_save/restore_itt
routines.

On restore, devices are re-allocated and their ite are
re-built.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- sort the device list by deviceid on device table save
- use defines for shifts and masks
- use abi->dte_esz
- clatify entry sizes for L1 and L2 tables

v3 -> v4:
- use the new proto for its_alloc_device
- compute_next_devid_offset, vgic_its_flush/restore_itt
  become static in this patch
- change in the DTE entry format with the introduction of the
  valid bit and next field width decrease; ittaddr encoded
  on its full range
- fix handle_l1_entry entry handling
- correct vgic_its_table_restore error handling

v2 -> v3:
- fix itt_addr bitmask in vgic_its_restore_dte
- addition of return 0 in vgic_its_restore_ite moved to
  the ITE related patch

v1 -> v2:
- use 8 byte format for DTE and ITE
- support 2 stage format
- remove kvm parameter
- ITT flush/restore moved in a separate patch
- use deviceid indexing
---
 virt/kvm/arm/vgic/vgic-its.c | 183 +++++++++++++++++++++++++++++++++++++++++--
 virt/kvm/arm/vgic/vgic.h     |   7 ++
 2 files changed, 185 insertions(+), 5 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index b02fc3f..86dfc6c 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1682,7 +1682,8 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
 	return ret;
 }
 
-u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
+static u32 compute_next_devid_offset(struct list_head *h,
+				     struct its_device *dev)
 {
 	struct list_head *e = &dev->dev_list;
 	struct its_device *next;
@@ -1858,7 +1859,7 @@ static int vgic_its_ite_cmp(void *priv, struct list_head *a,
 		return 1;
 }
 
-int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
+static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
 {
 	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
 	gpa_t base = device->itt_addr;
@@ -1877,7 +1878,7 @@ int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
 	return 0;
 }
 
-int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
+static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
 {
 	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
 	gpa_t base = dev->itt_addr;
@@ -1895,12 +1896,161 @@ int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
 }
 
 /**
+ * vgic_its_save_dte - Save a device table entry at a given GPA
+ *
+ * @its: ITS handle
+ * @dev: ITS device
+ * @ptr: GPA
+ */
+static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
+			     gpa_t ptr, int dte_esz)
+{
+	struct kvm *kvm = its->dev->kvm;
+	u64 val, itt_addr_field;
+	int ret;
+	u32 next_offset;
+
+	itt_addr_field = dev->itt_addr >> 8;
+	next_offset = compute_next_devid_offset(&its->device_list, dev);
+	val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
+	       ((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
+	       (itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
+		(dev->nb_eventid_bits - 1));
+	val = cpu_to_le64(val);
+	ret = kvm_write_guest(kvm, ptr, &val, dte_esz);
+	return ret;
+}
+
+/**
+ * vgic_its_restore_dte - restore a device table entry
+ *
+ * @its: its handle
+ * @id: device id the DTE corresponds to
+ * @ptr: kernel VA where the 8 byte DTE is located
+ * @opaque: unused
+ * @next: offset to the next valid device id
+ *
+ * Return: < 0 on error, 0 otherwise
+ */
+static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
+				void *ptr, void *opaque, u32 *next)
+{
+	struct its_device *dev;
+	gpa_t itt_addr;
+	u8 nb_eventid_bits;
+	u64 entry = *(u64 *)ptr;
+	bool valid;
+	int ret;
+
+	entry = le64_to_cpu(entry);
+
+	valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
+	nb_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
+	itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
+			>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
+	*next = 1;
+
+	if (!valid)
+		return 0;
+
+	/* dte entry is valid */
+	*next = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
+
+	ret = vgic_its_alloc_device(its, &dev, id,
+				    itt_addr, nb_eventid_bits);
+	if (ret)
+		return ret;
+	ret = vgic_its_restore_itt(its, dev);
+
+	return ret;
+}
+
+static int vgic_its_device_cmp(void *priv, struct list_head *a,
+			       struct list_head *b)
+{
+	struct its_device *deva = container_of(a, struct its_device, dev_list);
+	struct its_device *devb = container_of(b, struct its_device, dev_list);
+
+	if (deva->device_id < devb->device_id)
+		return -1;
+	else
+		return 1;
+}
+
+/**
  * vgic_its_save_device_tables - Save the device table and all ITT
  * into guest RAM
+ *
+ * L1/L2 handling is hidden by vgic_its_check_id() helper which directly
+ * returns the GPA of the device entry
  */
 static int vgic_its_save_device_tables(struct vgic_its *its)
 {
-	return -ENXIO;
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	struct its_device *dev;
+	int dte_esz = abi->dte_esz;
+	u64 baser;
+
+	baser = its->baser_device_table;
+
+	list_sort(NULL, &its->device_list, vgic_its_device_cmp);
+
+	list_for_each_entry(dev, &its->device_list, dev_list) {
+		int ret;
+		gpa_t eaddr;
+
+		if (!vgic_its_check_id(its, baser,
+				       dev->device_id, &eaddr))
+			return -EINVAL;
+
+		ret = vgic_its_save_itt(its, dev);
+		if (ret)
+			return ret;
+
+		ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+/**
+ * handle_l1_entry - callback used for L1 entries (2 stage case)
+ *
+ * @its: its handle
+ * @id: id
+ * @addr: kernel VA
+ * @opaque: unused
+ * @next_offset: offset to the next L1 entry: 0 if the last element
+ * was found, 1 otherwise
+ */
+static int handle_l1_entry(struct vgic_its *its, u32 id, void *addr,
+			   void *opaque, u32 *next_offset)
+{
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	int l2_start_id = id * (SZ_64K / GITS_LVL1_ENTRY_SIZE);
+	u64 entry = *(u64 *)addr;
+	int ret, ite_esz = abi->ite_esz;
+	gpa_t gpa;
+
+	entry = le64_to_cpu(entry);
+	*next_offset = 1;
+
+	if (!(entry & BIT_ULL(63)))
+		return 0;
+
+	gpa = entry & GENMASK_ULL(51, 16);
+
+	ret = lookup_table(its, gpa, SZ_64K, ite_esz,
+			   l2_start_id, vgic_its_restore_dte, NULL);
+
+	if (ret == 1) {
+		/* last entry was found in this L2 table */
+		*next_offset = 0;
+		ret = 0;
+	}
+
+	return ret;
 }
 
 /**
@@ -1909,7 +2059,30 @@ static int vgic_its_save_device_tables(struct vgic_its *its)
  */
 static int vgic_its_restore_device_tables(struct vgic_its *its)
 {
-	return -ENXIO;
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	u64 baser = its->baser_device_table;
+	int l1_esz, ret, l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
+	gpa_t l1_gpa;
+
+	l1_gpa = BASER_ADDRESS(baser);
+	if (!l1_gpa)
+		return 0;
+
+	if (baser & GITS_BASER_INDIRECT) {
+		l1_esz = 8;
+		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
+				   handle_l1_entry, NULL);
+	} else {
+		l1_esz = abi->dte_esz;
+		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
+				   vgic_its_restore_dte, NULL);
+	}
+
+	if (ret < 0)
+		return ret;
+
+	/* if last element was not found we have an issue here */
+	return ret ? 0 : -EINVAL;
 }
 
 static int vgic_its_save_cte(struct vgic_its *its,
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index ce57fbd..9bc52ef 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -85,6 +85,13 @@
 #define KVM_ITS_ITE_PINTID_SHIFT	16
 #define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
 #define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
+#define KVM_ITS_DTE_VALID_SHIFT		63
+#define KVM_ITS_DTE_VALID_MASK		BIT_ULL(63)
+#define KVM_ITS_DTE_NEXT_SHIFT		49
+#define KVM_ITS_DTE_NEXT_MASK		GENMASK_ULL(62, 49)
+#define KVM_ITS_DTE_ITTADDR_SHIFT	5
+#define KVM_ITS_DTE_ITTADDR_MASK	GENMASK_ULL(48, 5)
+#define KVM_ITS_DTE_SIZE_MASK		GENMASK_ULL(4, 0)
 
 static inline bool irq_is_pending(struct vgic_irq *irq)
 {
-- 
2.5.5

[PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Eric Auger]

This patch saves the device table entries into guest RAM.
Both flat table and 2 stage tables are supported. DeviceId
indexing is used.

For each device listed in the device table, we also save
the translation table using the vgic_its_save/restore_itt
routines.

On restore, devices are re-allocated and their ite are
re-built.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- sort the device list by deviceid on device table save
- use defines for shifts and masks
- use abi->dte_esz
- clatify entry sizes for L1 and L2 tables

v3 -> v4:
- use the new proto for its_alloc_device
- compute_next_devid_offset, vgic_its_flush/restore_itt
  become static in this patch
- change in the DTE entry format with the introduction of the
  valid bit and next field width decrease; ittaddr encoded
  on its full range
- fix handle_l1_entry entry handling
- correct vgic_its_table_restore error handling

v2 -> v3:
- fix itt_addr bitmask in vgic_its_restore_dte
- addition of return 0 in vgic_its_restore_ite moved to
  the ITE related patch

v1 -> v2:
- use 8 byte format for DTE and ITE
- support 2 stage format
- remove kvm parameter
- ITT flush/restore moved in a separate patch
- use deviceid indexing
---
 virt/kvm/arm/vgic/vgic-its.c | 183 +++++++++++++++++++++++++++++++++++++++++--
 virt/kvm/arm/vgic/vgic.h     |   7 ++
 2 files changed, 185 insertions(+), 5 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index b02fc3f..86dfc6c 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -1682,7 +1682,8 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
 	return ret;
 }
 
-u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
+static u32 compute_next_devid_offset(struct list_head *h,
+				     struct its_device *dev)
 {
 	struct list_head *e = &dev->dev_list;
 	struct its_device *next;
@@ -1858,7 +1859,7 @@ static int vgic_its_ite_cmp(void *priv, struct list_head *a,
 		return 1;
 }
 
-int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
+static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
 {
 	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
 	gpa_t base = device->itt_addr;
@@ -1877,7 +1878,7 @@ int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
 	return 0;
 }
 
-int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
+static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
 {
 	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
 	gpa_t base = dev->itt_addr;
@@ -1895,12 +1896,161 @@ int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
 }
 
 /**
+ * vgic_its_save_dte - Save a device table entry at a given GPA
+ *
+ * @its: ITS handle
+ * @dev: ITS device
+ * @ptr: GPA
+ */
+static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
+			     gpa_t ptr, int dte_esz)
+{
+	struct kvm *kvm = its->dev->kvm;
+	u64 val, itt_addr_field;
+	int ret;
+	u32 next_offset;
+
+	itt_addr_field = dev->itt_addr >> 8;
+	next_offset = compute_next_devid_offset(&its->device_list, dev);
+	val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
+	       ((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
+	       (itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
+		(dev->nb_eventid_bits - 1));
+	val = cpu_to_le64(val);
+	ret = kvm_write_guest(kvm, ptr, &val, dte_esz);
+	return ret;
+}
+
+/**
+ * vgic_its_restore_dte - restore a device table entry
+ *
+ * @its: its handle
+ * @id: device id the DTE corresponds to
+ * @ptr: kernel VA where the 8 byte DTE is located
+ * @opaque: unused
+ * @next: offset to the next valid device id
+ *
+ * Return: < 0 on error, 0 otherwise
+ */
+static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
+				void *ptr, void *opaque, u32 *next)
+{
+	struct its_device *dev;
+	gpa_t itt_addr;
+	u8 nb_eventid_bits;
+	u64 entry = *(u64 *)ptr;
+	bool valid;
+	int ret;
+
+	entry = le64_to_cpu(entry);
+
+	valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
+	nb_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
+	itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
+			>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
+	*next = 1;
+
+	if (!valid)
+		return 0;
+
+	/* dte entry is valid */
+	*next = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
+
+	ret = vgic_its_alloc_device(its, &dev, id,
+				    itt_addr, nb_eventid_bits);
+	if (ret)
+		return ret;
+	ret = vgic_its_restore_itt(its, dev);
+
+	return ret;
+}
+
+static int vgic_its_device_cmp(void *priv, struct list_head *a,
+			       struct list_head *b)
+{
+	struct its_device *deva = container_of(a, struct its_device, dev_list);
+	struct its_device *devb = container_of(b, struct its_device, dev_list);
+
+	if (deva->device_id < devb->device_id)
+		return -1;
+	else
+		return 1;
+}
+
+/**
  * vgic_its_save_device_tables - Save the device table and all ITT
  * into guest RAM
+ *
+ * L1/L2 handling is hidden by vgic_its_check_id() helper which directly
+ * returns the GPA of the device entry
  */
 static int vgic_its_save_device_tables(struct vgic_its *its)
 {
-	return -ENXIO;
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	struct its_device *dev;
+	int dte_esz = abi->dte_esz;
+	u64 baser;
+
+	baser = its->baser_device_table;
+
+	list_sort(NULL, &its->device_list, vgic_its_device_cmp);
+
+	list_for_each_entry(dev, &its->device_list, dev_list) {
+		int ret;
+		gpa_t eaddr;
+
+		if (!vgic_its_check_id(its, baser,
+				       dev->device_id, &eaddr))
+			return -EINVAL;
+
+		ret = vgic_its_save_itt(its, dev);
+		if (ret)
+			return ret;
+
+		ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+/**
+ * handle_l1_entry - callback used for L1 entries (2 stage case)
+ *
+ * @its: its handle
+ * @id: id
+ * @addr: kernel VA
+ * @opaque: unused
+ * @next_offset: offset to the next L1 entry: 0 if the last element
+ * was found, 1 otherwise
+ */
+static int handle_l1_entry(struct vgic_its *its, u32 id, void *addr,
+			   void *opaque, u32 *next_offset)
+{
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	int l2_start_id = id * (SZ_64K / GITS_LVL1_ENTRY_SIZE);
+	u64 entry = *(u64 *)addr;
+	int ret, ite_esz = abi->ite_esz;
+	gpa_t gpa;
+
+	entry = le64_to_cpu(entry);
+	*next_offset = 1;
+
+	if (!(entry & BIT_ULL(63)))
+		return 0;
+
+	gpa = entry & GENMASK_ULL(51, 16);
+
+	ret = lookup_table(its, gpa, SZ_64K, ite_esz,
+			   l2_start_id, vgic_its_restore_dte, NULL);
+
+	if (ret == 1) {
+		/* last entry was found in this L2 table */
+		*next_offset = 0;
+		ret = 0;
+	}
+
+	return ret;
 }
 
 /**
@@ -1909,7 +2059,30 @@ static int vgic_its_save_device_tables(struct vgic_its *its)
  */
 static int vgic_its_restore_device_tables(struct vgic_its *its)
 {
-	return -ENXIO;
+	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
+	u64 baser = its->baser_device_table;
+	int l1_esz, ret, l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
+	gpa_t l1_gpa;
+
+	l1_gpa = BASER_ADDRESS(baser);
+	if (!l1_gpa)
+		return 0;
+
+	if (baser & GITS_BASER_INDIRECT) {
+		l1_esz = 8;
+		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
+				   handle_l1_entry, NULL);
+	} else {
+		l1_esz = abi->dte_esz;
+		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
+				   vgic_its_restore_dte, NULL);
+	}
+
+	if (ret < 0)
+		return ret;
+
+	/* if last element was not found we have an issue here */
+	return ret ? 0 : -EINVAL;
 }
 
 static int vgic_its_save_cte(struct vgic_its *its,
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index ce57fbd..9bc52ef 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -85,6 +85,13 @@
 #define KVM_ITS_ITE_PINTID_SHIFT	16
 #define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
 #define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
+#define KVM_ITS_DTE_VALID_SHIFT		63
+#define KVM_ITS_DTE_VALID_MASK		BIT_ULL(63)
+#define KVM_ITS_DTE_NEXT_SHIFT		49
+#define KVM_ITS_DTE_NEXT_MASK		GENMASK_ULL(62, 49)
+#define KVM_ITS_DTE_ITTADDR_SHIFT	5
+#define KVM_ITS_DTE_ITTADDR_MASK	GENMASK_ULL(48, 5)
+#define KVM_ITS_DTE_SIZE_MASK		GENMASK_ULL(4, 0)
 
 static inline bool irq_is_pending(struct vgic_irq *irq)
 {
-- 
2.5.5

[PATCH v5 21/22] KVM: arm64: vgic-its: Fix pending table sync

[Eric Auger]

In its_sync_lpi_pending_table() we currently ignore the
target_vcpu of the LPIs. We sync the pending bit found in
the vcpu pending table even if the LPI is not targeting it.

Also in vgic_its_cmd_handle_invall() we are supposed to
read the config table data for the LPIs associated to the
collection ID. At the moment we refresh all LPI config
information.

This patch passes a vpcu to vgic_copy_lpi_list() so that
this latter returns a snapshot of the LPIs targeting this
CPU and only those.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
---
 virt/kvm/arm/vgic/vgic-its.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 86dfc6c..be848be 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -252,13 +252,13 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
 }
 
 /*
- * Create a snapshot of the current LPI list, so that we can enumerate all
- * LPIs without holding any lock.
- * Returns the array length and puts the kmalloc'ed array into intid_ptr.
+ * Create a snapshot of the current LPIs targeting @vcpu, so that we can
+ * enumerate those LPIs without holding any lock.
+ * Returns their number and puts the kmalloc'ed array into intid_ptr.
  */
-static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
+static int vgic_copy_lpi_list(struct kvm_vcpu *vcpu, u32 **intid_ptr)
 {
-	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
 	struct vgic_irq *irq;
 	u32 *intids;
 	int irq_count = dist->lpi_list_count, i = 0;
@@ -277,14 +277,14 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
 	spin_lock(&dist->lpi_list_lock);
 	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
 		/* We don't need to "get" the IRQ, as we hold the list lock. */
-		intids[i] = irq->intid;
-		if (++i == irq_count)
-			break;
+		if (irq->target_vcpu != vcpu)
+			continue;
+		intids[i++] = irq->intid;
 	}
 	spin_unlock(&dist->lpi_list_lock);
 
 	*intid_ptr = intids;
-	return irq_count;
+	return i;
 }
 
 /*
@@ -333,7 +333,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
 }
 
 /*
- * Scan the whole LPI pending table and sync the pending bit in there
+ * Sync the pending table pending bit of LPIs targeting @vcpu
  * with our own data structures. This relies on the LPI being
  * mapped before.
  */
@@ -346,7 +346,7 @@ static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
 	u32 *intids;
 	int nr_irqs, i;
 
-	nr_irqs = vgic_copy_lpi_list(vcpu->kvm, &intids);
+	nr_irqs = vgic_copy_lpi_list(vcpu, &intids);
 	if (nr_irqs < 0)
 		return nr_irqs;
 
@@ -1027,7 +1027,7 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
 
 	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
 
-	irq_count = vgic_copy_lpi_list(kvm, &intids);
+	irq_count = vgic_copy_lpi_list(vcpu, &intids);
 	if (irq_count < 0)
 		return irq_count;
 
-- 
2.5.5

[PATCH v5 21/22] KVM: arm64: vgic-its: Fix pending table sync

[Eric Auger]

In its_sync_lpi_pending_table() we currently ignore the
target_vcpu of the LPIs. We sync the pending bit found in
the vcpu pending table even if the LPI is not targeting it.

Also in vgic_its_cmd_handle_invall() we are supposed to
read the config table data for the LPIs associated to the
collection ID. At the moment we refresh all LPI config
information.

This patch passes a vpcu to vgic_copy_lpi_list() so that
this latter returns a snapshot of the LPIs targeting this
CPU and only those.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
---
 virt/kvm/arm/vgic/vgic-its.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 86dfc6c..be848be 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -252,13 +252,13 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
 }
 
 /*
- * Create a snapshot of the current LPI list, so that we can enumerate all
- * LPIs without holding any lock.
- * Returns the array length and puts the kmalloc'ed array into intid_ptr.
+ * Create a snapshot of the current LPIs targeting @vcpu, so that we can
+ * enumerate those LPIs without holding any lock.
+ * Returns their number and puts the kmalloc'ed array into intid_ptr.
  */
-static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
+static int vgic_copy_lpi_list(struct kvm_vcpu *vcpu, u32 **intid_ptr)
 {
-	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
 	struct vgic_irq *irq;
 	u32 *intids;
 	int irq_count = dist->lpi_list_count, i = 0;
@@ -277,14 +277,14 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
 	spin_lock(&dist->lpi_list_lock);
 	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
 		/* We don't need to "get" the IRQ, as we hold the list lock. */
-		intids[i] = irq->intid;
-		if (++i == irq_count)
-			break;
+		if (irq->target_vcpu != vcpu)
+			continue;
+		intids[i++] = irq->intid;
 	}
 	spin_unlock(&dist->lpi_list_lock);
 
 	*intid_ptr = intids;
-	return irq_count;
+	return i;
 }
 
 /*
@@ -333,7 +333,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
 }
 
 /*
- * Scan the whole LPI pending table and sync the pending bit in there
+ * Sync the pending table pending bit of LPIs targeting @vcpu
  * with our own data structures. This relies on the LPI being
  * mapped before.
  */
@@ -346,7 +346,7 @@ static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
 	u32 *intids;
 	int nr_irqs, i;
 
-	nr_irqs = vgic_copy_lpi_list(vcpu->kvm, &intids);
+	nr_irqs = vgic_copy_lpi_list(vcpu, &intids);
 	if (nr_irqs < 0)
 		return nr_irqs;
 
@@ -1027,7 +1027,7 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
 
 	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
 
-	irq_count = vgic_copy_lpi_list(kvm, &intids);
+	irq_count = vgic_copy_lpi_list(vcpu, &intids);
 	if (irq_count < 0)
 		return irq_count;
 
-- 
2.5.5

[PATCH v5 22/22] KVM: arm64: vgic-v3: KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES

[Eric Auger]

This patch adds a new attribute to GICV3 KVM device
KVM_DEV_ARM_VGIC_GRP_CTRL group. This Allows the userspace to
flush all GICR pending tables into guest RAM. At the moment
we do not offer any restore control as the sync is implicit.

As we need the PENDBASER_ADDRESS() in vgic-v3, let's move its
definition in the irqchip header. We restore the full length
of the field, ie [51:16]. Same for PROPBASER_ADDRESS with full
field length of [51:12].

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- move pending table save code/ctrl into VGICv3 instead of ITS
- remove useless gpa_t cast
- use the same optimization as in its_sync_lpi_pending_table

v3 -> v4:
- remove the wrong comment about locking
- pass kvm struct instead of its handle
- add comment about restore method
- remove GITR_PENDABASER.PTZ check
- continue if target_vcpu == NULL
- new locking strategy

v1 -> v2:
- do not care about the 1st KB which should be zeroed according to
  the spec.
---
 arch/arm/include/uapi/asm/kvm.h     |  1 +
 arch/arm64/include/uapi/asm/kvm.h   |  1 +
 include/linux/irqchip/arm-gic-v3.h  |  2 ++
 virt/kvm/arm/vgic/vgic-its.c        |  6 ++---
 virt/kvm/arm/vgic/vgic-kvm-device.c | 20 ++++++++++++++
 virt/kvm/arm/vgic/vgic-v3.c         | 54 +++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h            |  1 +
 7 files changed, 81 insertions(+), 4 deletions(-)

diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index 8e6563c..78fe803 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -202,6 +202,7 @@ struct kvm_arch_memory_slot {
 #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
 #define   KVM_DEV_ARM_ITS_SAVE_TABLES		1
 #define   KVM_DEV_ARM_ITS_RESTORE_TABLES	2
+#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
 
 /* KVM_IRQ_LINE irq field index values */
 #define KVM_ARM_IRQ_TYPE_SHIFT		24
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 1e35115..8a3758a 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -222,6 +222,7 @@ struct kvm_arch_memory_slot {
 #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
 #define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
 #define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
+#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
 
 /* Device Control API on vcpu fd */
 #define KVM_ARM_VCPU_PMU_V3_CTRL	0
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 0c6798c..9d3932f 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -158,6 +158,7 @@
 #define GICR_PROPBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PROPBASER, INNER, RaWaWb)
 
 #define GICR_PROPBASER_IDBITS_MASK			(0x1f)
+#define GICR_PROPBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 12))
 
 #define GICR_PENDBASER_SHAREABILITY_SHIFT		(10)
 #define GICR_PENDBASER_INNER_CACHEABILITY_SHIFT		(7)
@@ -183,6 +184,7 @@
 #define GICR_PENDBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PENDBASER, INNER, RaWaWb)
 
 #define GICR_PENDBASER_PTZ				BIT_ULL(62)
+#define GICR_PENDBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 16))
 
 /*
  * Re-Distributor registers, offsets from SGI_base
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index be848be..745c245 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -189,8 +189,6 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
  */
 #define BASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
 #define CBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
-#define PENDBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
-#define PROPBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
 
 #define GIC_LPI_OFFSET 8192
 
@@ -227,7 +225,7 @@ static struct its_collection *find_collection(struct vgic_its *its, int coll_id)
 static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
 			     struct kvm_vcpu *filter_vcpu)
 {
-	u64 propbase = PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
+	u64 propbase = GICR_PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
 	u8 prop;
 	int ret;
 
@@ -339,7 +337,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
  */
 static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
 {
-	gpa_t pendbase = PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
+	gpa_t pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
 	struct vgic_irq *irq;
 	int last_byte_offset = -1;
 	int ret = 0;
diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c
index 859bfa8..d48743c 100644
--- a/virt/kvm/arm/vgic/vgic-kvm-device.c
+++ b/virt/kvm/arm/vgic/vgic-kvm-device.c
@@ -580,6 +580,24 @@ static int vgic_v3_set_attr(struct kvm_device *dev,
 		reg = tmp32;
 		return vgic_v3_attr_regs_access(dev, attr, &reg, true);
 	}
+	case KVM_DEV_ARM_VGIC_GRP_CTRL: {
+		int ret;
+
+		switch (attr->attr) {
+		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
+			mutex_lock(&dev->kvm->lock);
+
+			if (!lock_all_vcpus(dev->kvm)) {
+				mutex_unlock(&dev->kvm->lock);
+				return -EBUSY;
+			}
+			ret = vgic_v3_save_pending_tables(dev->kvm);
+			unlock_all_vcpus(dev->kvm);
+			mutex_unlock(&dev->kvm->lock);
+			return ret;
+		}
+		break;
+	}
 	}
 	return -ENXIO;
 }
@@ -658,6 +676,8 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
 		switch (attr->attr) {
 		case KVM_DEV_ARM_VGIC_CTRL_INIT:
 			return 0;
+		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
+			return 0;
 		}
 	}
 	return -ENXIO;
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index be0f4c3..1f0977f 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -15,6 +15,7 @@
 #include <linux/irqchip/arm-gic-v3.h>
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
+//#include <linux/bitops.h>
 #include <kvm/arm_vgic.h>
 #include <asm/kvm_mmu.h>
 #include <asm/kvm_asm.h>
@@ -252,6 +253,59 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu)
 	vgic_v3->vgic_hcr = ICH_HCR_EN;
 }
 
+/**
+ * vgic_its_save_pending_tables - Save the pending tables into guest RAM
+ * kvm lock and all vcpu lock must be held
+ */
+int vgic_v3_save_pending_tables(struct kvm *kvm)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	int last_byte_offset = -1;
+	struct vgic_irq *irq;
+	int ret;
+
+	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
+		int byte_offset, bit_nr;
+		struct kvm_vcpu *vcpu;
+		gpa_t pendbase, ptr;
+		bool stored;
+		u8 val;
+
+		vcpu = irq->target_vcpu;
+		if (!vcpu)
+			continue;
+
+		pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
+
+		byte_offset = irq->intid / BITS_PER_BYTE;
+		bit_nr = irq->intid % BITS_PER_BYTE;
+		ptr = pendbase + byte_offset;
+
+		if (byte_offset != last_byte_offset) {
+			ret = kvm_read_guest(kvm, ptr, &val, 1);
+			if (ret)
+				return ret;
+			last_byte_offset = byte_offset;
+		}
+
+		stored = val & bit_nr;
+		if (stored == irq->pending_latch)
+			continue;
+
+		if (irq->pending_latch)
+			val |= 1 << bit_nr;
+		else
+			val &= ~(1 << bit_nr);
+
+		ret = kvm_write_guest(kvm, ptr, &val, 1);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+
 /* check for overlapping regions and for regions crossing the end of memory */
 static bool vgic_v3_check_base(struct kvm *kvm)
 {
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 9bc52ef..535c2fc 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -177,6 +177,7 @@ void vgic_v3_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
 void vgic_v3_enable(struct kvm_vcpu *vcpu);
 int vgic_v3_probe(const struct gic_kvm_info *info);
 int vgic_v3_map_resources(struct kvm *kvm);
+int vgic_v3_save_pending_tables(struct kvm *kvm);
 int vgic_register_redist_iodevs(struct kvm *kvm, gpa_t dist_base_address);
 
 int vgic_register_its_iodevs(struct kvm *kvm);
-- 
2.5.5

[PATCH v5 22/22] KVM: arm64: vgic-v3: KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES

[Eric Auger]

This patch adds a new attribute to GICV3 KVM device
KVM_DEV_ARM_VGIC_GRP_CTRL group. This Allows the userspace to
flush all GICR pending tables into guest RAM. At the moment
we do not offer any restore control as the sync is implicit.

As we need the PENDBASER_ADDRESS() in vgic-v3, let's move its
definition in the irqchip header. We restore the full length
of the field, ie [51:16]. Same for PROPBASER_ADDRESS with full
field length of [51:12].

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- move pending table save code/ctrl into VGICv3 instead of ITS
- remove useless gpa_t cast
- use the same optimization as in its_sync_lpi_pending_table

v3 -> v4:
- remove the wrong comment about locking
- pass kvm struct instead of its handle
- add comment about restore method
- remove GITR_PENDABASER.PTZ check
- continue if target_vcpu == NULL
- new locking strategy

v1 -> v2:
- do not care about the 1st KB which should be zeroed according to
  the spec.
---
 arch/arm/include/uapi/asm/kvm.h     |  1 +
 arch/arm64/include/uapi/asm/kvm.h   |  1 +
 include/linux/irqchip/arm-gic-v3.h  |  2 ++
 virt/kvm/arm/vgic/vgic-its.c        |  6 ++---
 virt/kvm/arm/vgic/vgic-kvm-device.c | 20 ++++++++++++++
 virt/kvm/arm/vgic/vgic-v3.c         | 54 +++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h            |  1 +
 7 files changed, 81 insertions(+), 4 deletions(-)

diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index 8e6563c..78fe803 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -202,6 +202,7 @@ struct kvm_arch_memory_slot {
 #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
 #define   KVM_DEV_ARM_ITS_SAVE_TABLES		1
 #define   KVM_DEV_ARM_ITS_RESTORE_TABLES	2
+#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
 
 /* KVM_IRQ_LINE irq field index values */
 #define KVM_ARM_IRQ_TYPE_SHIFT		24
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 1e35115..8a3758a 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -222,6 +222,7 @@ struct kvm_arch_memory_slot {
 #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
 #define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
 #define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
+#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
 
 /* Device Control API on vcpu fd */
 #define KVM_ARM_VCPU_PMU_V3_CTRL	0
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 0c6798c..9d3932f 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -158,6 +158,7 @@
 #define GICR_PROPBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PROPBASER, INNER, RaWaWb)
 
 #define GICR_PROPBASER_IDBITS_MASK			(0x1f)
+#define GICR_PROPBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 12))
 
 #define GICR_PENDBASER_SHAREABILITY_SHIFT		(10)
 #define GICR_PENDBASER_INNER_CACHEABILITY_SHIFT		(7)
@@ -183,6 +184,7 @@
 #define GICR_PENDBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PENDBASER, INNER, RaWaWb)
 
 #define GICR_PENDBASER_PTZ				BIT_ULL(62)
+#define GICR_PENDBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 16))
 
 /*
  * Re-Distributor registers, offsets from SGI_base
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index be848be..745c245 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -189,8 +189,6 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
  */
 #define BASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
 #define CBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
-#define PENDBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
-#define PROPBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
 
 #define GIC_LPI_OFFSET 8192
 
@@ -227,7 +225,7 @@ static struct its_collection *find_collection(struct vgic_its *its, int coll_id)
 static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
 			     struct kvm_vcpu *filter_vcpu)
 {
-	u64 propbase = PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
+	u64 propbase = GICR_PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
 	u8 prop;
 	int ret;
 
@@ -339,7 +337,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
  */
 static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
 {
-	gpa_t pendbase = PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
+	gpa_t pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
 	struct vgic_irq *irq;
 	int last_byte_offset = -1;
 	int ret = 0;
diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c
index 859bfa8..d48743c 100644
--- a/virt/kvm/arm/vgic/vgic-kvm-device.c
+++ b/virt/kvm/arm/vgic/vgic-kvm-device.c
@@ -580,6 +580,24 @@ static int vgic_v3_set_attr(struct kvm_device *dev,
 		reg = tmp32;
 		return vgic_v3_attr_regs_access(dev, attr, &reg, true);
 	}
+	case KVM_DEV_ARM_VGIC_GRP_CTRL: {
+		int ret;
+
+		switch (attr->attr) {
+		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
+			mutex_lock(&dev->kvm->lock);
+
+			if (!lock_all_vcpus(dev->kvm)) {
+				mutex_unlock(&dev->kvm->lock);
+				return -EBUSY;
+			}
+			ret = vgic_v3_save_pending_tables(dev->kvm);
+			unlock_all_vcpus(dev->kvm);
+			mutex_unlock(&dev->kvm->lock);
+			return ret;
+		}
+		break;
+	}
 	}
 	return -ENXIO;
 }
@@ -658,6 +676,8 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
 		switch (attr->attr) {
 		case KVM_DEV_ARM_VGIC_CTRL_INIT:
 			return 0;
+		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
+			return 0;
 		}
 	}
 	return -ENXIO;
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index be0f4c3..1f0977f 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -15,6 +15,7 @@
 #include <linux/irqchip/arm-gic-v3.h>
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
+//#include <linux/bitops.h>
 #include <kvm/arm_vgic.h>
 #include <asm/kvm_mmu.h>
 #include <asm/kvm_asm.h>
@@ -252,6 +253,59 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu)
 	vgic_v3->vgic_hcr = ICH_HCR_EN;
 }
 
+/**
+ * vgic_its_save_pending_tables - Save the pending tables into guest RAM
+ * kvm lock and all vcpu lock must be held
+ */
+int vgic_v3_save_pending_tables(struct kvm *kvm)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	int last_byte_offset = -1;
+	struct vgic_irq *irq;
+	int ret;
+
+	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
+		int byte_offset, bit_nr;
+		struct kvm_vcpu *vcpu;
+		gpa_t pendbase, ptr;
+		bool stored;
+		u8 val;
+
+		vcpu = irq->target_vcpu;
+		if (!vcpu)
+			continue;
+
+		pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
+
+		byte_offset = irq->intid / BITS_PER_BYTE;
+		bit_nr = irq->intid % BITS_PER_BYTE;
+		ptr = pendbase + byte_offset;
+
+		if (byte_offset != last_byte_offset) {
+			ret = kvm_read_guest(kvm, ptr, &val, 1);
+			if (ret)
+				return ret;
+			last_byte_offset = byte_offset;
+		}
+
+		stored = val & bit_nr;
+		if (stored == irq->pending_latch)
+			continue;
+
+		if (irq->pending_latch)
+			val |= 1 << bit_nr;
+		else
+			val &= ~(1 << bit_nr);
+
+		ret = kvm_write_guest(kvm, ptr, &val, 1);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+
 /* check for overlapping regions and for regions crossing the end of memory */
 static bool vgic_v3_check_base(struct kvm *kvm)
 {
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 9bc52ef..535c2fc 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -177,6 +177,7 @@ void vgic_v3_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
 void vgic_v3_enable(struct kvm_vcpu *vcpu);
 int vgic_v3_probe(const struct gic_kvm_info *info);
 int vgic_v3_map_resources(struct kvm *kvm);
+int vgic_v3_save_pending_tables(struct kvm *kvm);
 int vgic_register_redist_iodevs(struct kvm *kvm, gpa_t dist_base_address);
 
 int vgic_register_its_iodevs(struct kvm *kvm);
-- 
2.5.5

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Peter Maydell]

On 14 April 2017 at 11:15, Eric Auger <eric.auger@redhat.com> wrote:
> Add description for how to access ITS registers and how to save/restore
> ITS tables into/from memory.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>

Acked-by: Peter Maydell <peter.maydell@linaro.org>

thanks
-- PMM

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Peter Maydell]

On 14 April 2017 at 11:15, Eric Auger <eric.auger@redhat.com> wrote:
> Add description for how to access ITS registers and how to save/restore
> ITS tables into/from memory.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>

Acked-by: Peter Maydell <peter.maydell@linaro.org>

thanks
-- PMM

Re: [PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Peter Maydell]

On 14 April 2017 at 11:15, Eric Auger <eric.auger@redhat.com> wrote:
> Add description for how to save GICV3 LPI pending bit into
> guest RAM pending tables.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>
> ---
> v5: creation
> ---
>  Documentation/virtual/kvm/devices/arm-vgic-v3.txt | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
> index c1a2461..9293b45 100644
> --- a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
> +++ b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
> @@ -167,11 +167,17 @@ Groups:
>      KVM_DEV_ARM_VGIC_CTRL_INIT
>        request the initialization of the VGIC, no additional parameter in
>        kvm_device_attr.addr.
> +    KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES
> +      save all LPI pending bits into guest RAM pending tables.
> +
> +      The first kB of the pending table is not altered by this operation.
>    Errors:
>      -ENXIO: VGIC not properly configured as required prior to calling
>       this attribute
>      -ENODEV: no online VCPU
>      -ENOMEM: memory shortage when allocating vgic internal data
> +    -EFAULT: Invalid guest ram access
> +    -EBUSY:  One or more VCPUS are running
>
>
>    KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO
> --
> 2.5.5

When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
etc at invalid memory, presumably?) How does the QEMU migration code
handle this case? Failing migration because the guest has done something
silly doesn't seem too palatable, but trying to avoid that could be
more effort than an obscure corner case really merits.

thanks
-- PMM

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Peter Maydell]

On 14 April 2017 at 11:15, Eric Auger <eric.auger@redhat.com> wrote:
> Add description for how to save GICV3 LPI pending bit into
> guest RAM pending tables.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>
> ---
> v5: creation
> ---
>  Documentation/virtual/kvm/devices/arm-vgic-v3.txt | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
> index c1a2461..9293b45 100644
> --- a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
> +++ b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
> @@ -167,11 +167,17 @@ Groups:
>      KVM_DEV_ARM_VGIC_CTRL_INIT
>        request the initialization of the VGIC, no additional parameter in
>        kvm_device_attr.addr.
> +    KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES
> +      save all LPI pending bits into guest RAM pending tables.
> +
> +      The first kB of the pending table is not altered by this operation.
>    Errors:
>      -ENXIO: VGIC not properly configured as required prior to calling
>       this attribute
>      -ENODEV: no online VCPU
>      -ENOMEM: memory shortage when allocating vgic internal data
> +    -EFAULT: Invalid guest ram access
> +    -EBUSY:  One or more VCPUS are running
>
>
>    KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO
> --
> 2.5.5

When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
etc at invalid memory, presumably?) How does the QEMU migration code
handle this case? Failing migration because the guest has done something
silly doesn't seem too palatable, but trying to avoid that could be
more effort than an obscure corner case really merits.

thanks
-- PMM

Re: [PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Auger Eric]

Hi Peter,

On 25/04/2017 12:43, Peter Maydell wrote:
> On 14 April 2017 at 11:15, Eric Auger <eric.auger@redhat.com> wrote:
>> Add description for how to save GICV3 LPI pending bit into
>> guest RAM pending tables.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v5: creation
>> ---
>>  Documentation/virtual/kvm/devices/arm-vgic-v3.txt | 6 ++++++
>>  1 file changed, 6 insertions(+)
>>
>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
>> index c1a2461..9293b45 100644
>> --- a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
>> @@ -167,11 +167,17 @@ Groups:
>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>        request the initialization of the VGIC, no additional parameter in
>>        kvm_device_attr.addr.
>> +    KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES
>> +      save all LPI pending bits into guest RAM pending tables.
>> +
>> +      The first kB of the pending table is not altered by this operation.
>>    Errors:
>>      -ENXIO: VGIC not properly configured as required prior to calling
>>       this attribute
>>      -ENODEV: no online VCPU
>>      -ENOMEM: memory shortage when allocating vgic internal data
>> +    -EFAULT: Invalid guest ram access
>> +    -EBUSY:  One or more VCPUS are running
>>
>>
>>    KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO
>> --
>> 2.5.5
> 
> When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
> etc at invalid memory, presumably?)
Yes that's correct, when GICR_PENDBASER contains a bad GPA.
 How does the QEMU migration code
> handle this case? Failing migration because the guest has done something
> silly doesn't seem too palatable, but trying to avoid that could be
> more effort than an obscure corner case really merits.
The kvm_device_access will cause an abort() as for other errors returned
by kvm_device_ioctl().

Thanks

Eric
> 
> thanks
> -- PMM
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Auger Eric]

Hi Peter,

On 25/04/2017 12:43, Peter Maydell wrote:
> On 14 April 2017 at 11:15, Eric Auger <eric.auger@redhat.com> wrote:
>> Add description for how to save GICV3 LPI pending bit into
>> guest RAM pending tables.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v5: creation
>> ---
>>  Documentation/virtual/kvm/devices/arm-vgic-v3.txt | 6 ++++++
>>  1 file changed, 6 insertions(+)
>>
>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
>> index c1a2461..9293b45 100644
>> --- a/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-v3.txt
>> @@ -167,11 +167,17 @@ Groups:
>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>        request the initialization of the VGIC, no additional parameter in
>>        kvm_device_attr.addr.
>> +    KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES
>> +      save all LPI pending bits into guest RAM pending tables.
>> +
>> +      The first kB of the pending table is not altered by this operation.
>>    Errors:
>>      -ENXIO: VGIC not properly configured as required prior to calling
>>       this attribute
>>      -ENODEV: no online VCPU
>>      -ENOMEM: memory shortage when allocating vgic internal data
>> +    -EFAULT: Invalid guest ram access
>> +    -EBUSY:  One or more VCPUS are running
>>
>>
>>    KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO
>> --
>> 2.5.5
> 
> When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
> etc at invalid memory, presumably?)
Yes that's correct, when GICR_PENDBASER contains a bad GPA.
 How does the QEMU migration code
> handle this case? Failing migration because the guest has done something
> silly doesn't seem too palatable, but trying to avoid that could be
> more effort than an obscure corner case really merits.
The kvm_device_access will cause an abort() as for other errors returned
by kvm_device_ioctl().

Thanks

Eric
> 
> thanks
> -- PMM
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Peter Maydell]

On 26 April 2017 at 09:26, Auger Eric <eric.auger@redhat.com> wrote:
> On 25/04/2017 12:43, Peter Maydell wrote:
>> When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
>> etc at invalid memory, presumably?)
>
> Yes that's correct, when GICR_PENDBASER contains a bad GPA.
>
>>  How does the QEMU migration code
>> handle this case? Failing migration because the guest has done something
>> silly doesn't seem too palatable, but trying to avoid that could be
>> more effort than an obscure corner case really merits.
>
> The kvm_device_access will cause an abort() as for other errors returned
> by kvm_device_ioctl().

That's pretty nasty. Guests shouldn't be able to provoke QEMU
into abort()ing, ideally. We don't necessarily have to produce
a successful migration, but we should at least fail it cleanly.

thanks
-- PMM

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Peter Maydell]

On 26 April 2017 at 09:26, Auger Eric <eric.auger@redhat.com> wrote:
> On 25/04/2017 12:43, Peter Maydell wrote:
>> When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
>> etc at invalid memory, presumably?)
>
> Yes that's correct, when GICR_PENDBASER contains a bad GPA.
>
>>  How does the QEMU migration code
>> handle this case? Failing migration because the guest has done something
>> silly doesn't seem too palatable, but trying to avoid that could be
>> more effort than an obscure corner case really merits.
>
> The kvm_device_access will cause an abort() as for other errors returned
> by kvm_device_ioctl().

That's pretty nasty. Guests shouldn't be able to provoke QEMU
into abort()ing, ideally. We don't necessarily have to produce
a successful migration, but we should at least fail it cleanly.

thanks
-- PMM

Re: [PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Dr. David Alan Gilbert]

* Peter Maydell (peter.maydell@linaro.org) wrote:
> On 26 April 2017 at 09:26, Auger Eric <eric.auger@redhat.com> wrote:
> > On 25/04/2017 12:43, Peter Maydell wrote:
> >> When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
> >> etc at invalid memory, presumably?)
> >
> > Yes that's correct, when GICR_PENDBASER contains a bad GPA.
> >
> >>  How does the QEMU migration code
> >> handle this case? Failing migration because the guest has done something
> >> silly doesn't seem too palatable, but trying to avoid that could be
> >> more effort than an obscure corner case really merits.
> >
> > The kvm_device_access will cause an abort() as for other errors returned
> > by kvm_device_ioctl().
> 
> That's pretty nasty. Guests shouldn't be able to provoke QEMU
> into abort()ing, ideally. We don't necessarily have to produce
> a successful migration, but we should at least fail it cleanly.

Yes, no abort()'s during migration due to guest behaviour.
They always end up coming back around to being filed as migration
bugs and people worry why they've got cores.

Ideally log a message into stderr to say that the guest state
is inconsistent so that when someone comes to debug it then they
can see it's obvious.

Dave

> thanks
> -- PMM
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Dr. David Alan Gilbert]

* Peter Maydell (peter.maydell at linaro.org) wrote:
> On 26 April 2017 at 09:26, Auger Eric <eric.auger@redhat.com> wrote:
> > On 25/04/2017 12:43, Peter Maydell wrote:
> >> When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
> >> etc at invalid memory, presumably?)
> >
> > Yes that's correct, when GICR_PENDBASER contains a bad GPA.
> >
> >>  How does the QEMU migration code
> >> handle this case? Failing migration because the guest has done something
> >> silly doesn't seem too palatable, but trying to avoid that could be
> >> more effort than an obscure corner case really merits.
> >
> > The kvm_device_access will cause an abort() as for other errors returned
> > by kvm_device_ioctl().
> 
> That's pretty nasty. Guests shouldn't be able to provoke QEMU
> into abort()ing, ideally. We don't necessarily have to produce
> a successful migration, but we should at least fail it cleanly.

Yes, no abort()'s during migration due to guest behaviour.
They always end up coming back around to being filed as migration
bugs and people worry why they've got cores.

Ideally log a message into stderr to say that the guest state
is inconsistent so that when someone comes to debug it then they
can see it's obvious.

Dave

> thanks
> -- PMM
--
Dr. David Alan Gilbert / dgilbert at redhat.com / Manchester, UK

Re: [PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Auger Eric]

Hi Peter, Dave,

On 26/04/2017 10:48, Dr. David Alan Gilbert wrote:
> * Peter Maydell (peter.maydell@linaro.org) wrote:
>> On 26 April 2017 at 09:26, Auger Eric <eric.auger@redhat.com> wrote:
>>> On 25/04/2017 12:43, Peter Maydell wrote:
>>>> When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
>>>> etc at invalid memory, presumably?)
>>>
>>> Yes that's correct, when GICR_PENDBASER contains a bad GPA.
>>>
>>>>  How does the QEMU migration code
>>>> handle this case? Failing migration because the guest has done something
>>>> silly doesn't seem too palatable, but trying to avoid that could be
>>>> more effort than an obscure corner case really merits.
>>>
>>> The kvm_device_access will cause an abort() as for other errors returned
>>> by kvm_device_ioctl().
>>
>> That's pretty nasty. Guests shouldn't be able to provoke QEMU
>> into abort()ing, ideally. We don't necessarily have to produce
>> a successful migration, but we should at least fail it cleanly.
> 
> Yes, no abort()'s during migration due to guest behaviour.
> They always end up coming back around to being filed as migration
> bugs and people worry why they've got cores.
> 
> Ideally log a message into stderr to say that the guest state
> is inconsistent so that when someone comes to debug it then they
> can see it's obvious.

OK I agree. I will respin the QEMU part accordingly and in that
situation I won't abort and will print a message.

Thanks

Eric
> 
> Dave
> 
>> thanks
>> -- PMM
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Auger Eric]

Hi Peter, Dave,

On 26/04/2017 10:48, Dr. David Alan Gilbert wrote:
> * Peter Maydell (peter.maydell at linaro.org) wrote:
>> On 26 April 2017 at 09:26, Auger Eric <eric.auger@redhat.com> wrote:
>>> On 25/04/2017 12:43, Peter Maydell wrote:
>>>> When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
>>>> etc at invalid memory, presumably?)
>>>
>>> Yes that's correct, when GICR_PENDBASER contains a bad GPA.
>>>
>>>>  How does the QEMU migration code
>>>> handle this case? Failing migration because the guest has done something
>>>> silly doesn't seem too palatable, but trying to avoid that could be
>>>> more effort than an obscure corner case really merits.
>>>
>>> The kvm_device_access will cause an abort() as for other errors returned
>>> by kvm_device_ioctl().
>>
>> That's pretty nasty. Guests shouldn't be able to provoke QEMU
>> into abort()ing, ideally. We don't necessarily have to produce
>> a successful migration, but we should at least fail it cleanly.
> 
> Yes, no abort()'s during migration due to guest behaviour.
> They always end up coming back around to being filed as migration
> bugs and people worry why they've got cores.
> 
> Ideally log a message into stderr to say that the guest state
> is inconsistent so that when someone comes to debug it then they
> can see it's obvious.

OK I agree. I will respin the QEMU part accordingly and in that
situation I won't abort and will print a message.

Thanks

Eric
> 
> Dave
> 
>> thanks
>> -- PMM
> --
> Dr. David Alan Gilbert / dgilbert at redhat.com / Manchester, UK
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> The actual abbreviation for the interrupt translation table entry
> is ITE. Let's rename all itte instances by ite.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> The actual abbreviation for the interrupt translation table entry
> is ITE. Let's rename all itte instances by ite.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 04/22] arm/arm64: vgic: turn vgic_find_mmio_region into public

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> We plan to use vgic_find_mmio_region in vgic-its.c so let's
> turn it into a public function.
>
> Also let's take the opportunity to rename the region parameter
> into regions to emphasize this latter is an array of regions.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Andre Przywara <andre.przywara@arm.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 04/22] arm/arm64: vgic: turn vgic_find_mmio_region into public

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> We plan to use vgic_find_mmio_region in vgic-its.c so let's
> turn it into a public function.
>
> Also let's take the opportunity to rename the region parameter
> into regions to emphasize this latter is an array of regions.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Andre Przywara <andre.przywara@arm.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 05/22] KVM: arm64: vgic-its: KVM_DEV_ARM_VGIC_GRP_ITS_REGS group

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> The ITS KVM device exposes a new KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> group which allows the userspace to save/restore ITS registers.
>
> At this stage the get/set/has operations are not yet implemented.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Andre Przywara <andre.przywara@arm.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 05/22] KVM: arm64: vgic-its: KVM_DEV_ARM_VGIC_GRP_ITS_REGS group

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> The ITS KVM device exposes a new KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> group which allows the userspace to save/restore ITS registers.
>
> At this stage the get/set/has operations are not yet implemented.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Andre Przywara <andre.przywara@arm.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 06/22] KVM: arm/arm64: vgic: expose (un)lock_all_vcpus

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> We need to use those helpers in vgic-its.c so let's
> expose them in the private vgic header.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 06/22] KVM: arm/arm64: vgic: expose (un)lock_all_vcpus

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> We need to use those helpers in vgic-its.c so let's
> expose them in the private vgic header.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 07/22] KVM: arm64: vgic-its: Implement vgic_its_has_attr_regs and attr_regs_access

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> This patch implements vgic_its_has_attr_regs and vgic_its_attr_regs_access
> upon the MMIO framework. VGIC ITS KVM device KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> group becomes functional.
>
> At least GITS_CREADR and GITS_IIDR require to differentiate a guest write
> action from a user access. As such let's introduce a new uaccess_its_write
> vgic_register_region callback.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 07/22] KVM: arm64: vgic-its: Implement vgic_its_has_attr_regs and attr_regs_access

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> This patch implements vgic_its_has_attr_regs and vgic_its_attr_regs_access
> upon the MMIO framework. VGIC ITS KVM device KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> group becomes functional.
>
> At least GITS_CREADR and GITS_IIDR require to differentiate a guest write
> action from a user access. As such let's introduce a new uaccess_its_write
> vgic_register_region callback.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 08/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_creadr

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> GITS_CREADR needs to be restored so let's implement the associated
> uaccess_write_its callback. The write only is allowed if the its
> is disabled.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 08/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_creadr

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> GITS_CREADR needs to be restored so let's implement the associated
> uaccess_write_its callback. The write only is allowed if the its
> is disabled.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 09/22] KVM: arm64: vgic-its: Introduce migration ABI infrastructure

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> We plan to support different migration ABIs, ie. characterizing
> the ITS table layout format in guest RAM. Typically a new ABI will
> be needed if vLPIs get supported for nested use case.
>
> So let's introduce an array of supported ABIs (at the moment a single
> ABI is supported though). The following characteristics are foreseen
> to vary with the ABI: size of table entries, save/restore operation.
>
> By default the MAX_ABI_REV is applied on its creation. In subsequent
> patches we will introduce a way for the userspace to change the ABI
> in use.
>
> The entry sizes now are set according to the ABI version and not
> hardcoded anymore.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 09/22] KVM: arm64: vgic-its: Introduce migration ABI infrastructure

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> We plan to support different migration ABIs, ie. characterizing
> the ITS table layout format in guest RAM. Typically a new ABI will
> be needed if vLPIs get supported for nested use case.
>
> So let's introduce an array of supported ABIs (at the moment a single
> ABI is supported though). The following characteristics are foreseen
> to vary with the ABI: size of table entries, save/restore operation.
>
> By default the MAX_ABI_REV is applied on its creation. In subsequent
> patches we will introduce a way for the userspace to change the ABI
> in use.
>
> The entry sizes now are set according to the ABI version and not
> hardcoded anymore.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>

Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 10/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_iidr

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> The GITS_IIDR revision field is used to encode the migration ABI
> revision. So we need to restore it to check the table layout is
> readable by the destination.
>
> By writing the IIDR, userspace thus force the ABI revision to be
> used and this msut be less or equal than the max revision KVM supports.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 10/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_iidr

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> The GITS_IIDR revision field is used to encode the migration ABI
> revision. So we need to restore it to check the table layout is
> readable by the destination.
>
> By writing the IIDR, userspace thus force the ABI revision to be
> used and this msut be less or equal than the max revision KVM supports.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 11/22] KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Up to now the MAPD's ITT size field has been ignored. It encodes
> the number of eventid bit minus 1. It should be used to check
> the eventid when a MAPTI command is issued on a device. Let's
> store the number of eventid bits in the its_device and do the
> check on MAPTI. Also make sure the ITT size field does
> not exceed the GITS_TYPER IDBITS field.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 11/22] KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Up to now the MAPD's ITT size field has been ignored. It encodes
> the number of eventid bit minus 1. It should be used to check
> the eventid when a MAPTI command is issued on a device. Let's
> store the number of eventid bits in the its_device and do the
> check on MAPTI. Also make sure the ITT size field does
> not exceed the GITS_TYPER IDBITS field.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Up to now the MAPD ITT_addr had been ignored. We will need it
> for save/restore. Let's record it in the its_device struct.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Up to now the MAPD ITT_addr had been ignored. We will need it
> for save/restore. Let's record it in the its_device struct.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 13/22] KVM: arm64: vgic-its: Check the device id matches TYPER DEVBITS range

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> On MAPD we currently check the device id can be stored in the device table.
> Let's first check it can be encoded within the range defined by TYPER
> DEVBITS.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 13/22] KVM: arm64: vgic-its: Check the device id matches TYPER DEVBITS range

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> On MAPD we currently check the device id can be stored in the device table.
> Let's first check it can be encoded within the range defined by TYPER
> DEVBITS.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 14/22] KVM: arm64: vgic-its: KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Introduce new attributes in KVM_DEV_ARM_VGIC_GRP_CTRL group:
> - KVM_DEV_ARM_ITS_SAVE_TABLES: saves the ITS tables into guest RAM
> - KVM_DEV_ARM_ITS_RESTORE_TABLES: restores them into VGIC internal
>   structures.
>
> We hold the vcpus lock during the save and restore to make
> sure no vcpu is running.
>
> At this stage the functionality is not yet implemented. Only
> the skeleton is put in place.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 14/22] KVM: arm64: vgic-its: KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Introduce new attributes in KVM_DEV_ARM_VGIC_GRP_CTRL group:
> - KVM_DEV_ARM_ITS_SAVE_TABLES: saves the ITS tables into guest RAM
> - KVM_DEV_ARM_ITS_RESTORE_TABLES: restores them into VGIC internal
>   structures.
>
> We hold the vcpus lock during the save and restore to make
> sure no vcpu is running.
>
> At this stage the functionality is not yet implemented. Only
> the skeleton is put in place.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 15/22] KVM: arm64: vgic-its: vgic_its_alloc_ite/device

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Add two new helpers to allocate an its ite and an its device.
> This will avoid duplication on restore path.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 15/22] KVM: arm64: vgic-its: vgic_its_alloc_ite/device

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Add two new helpers to allocate an its ite and an its device.
> This will avoid duplication on restore path.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Add a generic lookup_table() helper whose role consists in
> scanning a contiguous table located in guest RAM and applying
> a callback on each entry. Entries can be handled as linked lists
> since the callback may return an offset to the next entry and
> also tell that an entry is the last one.
>
> Helper functions also are added to compute the device/event ID
> offset to the next DTE/ITE.
>
> compute_next_devid_offset, compute_next_eventid_offset and
> lookup_table will become static in subsequent patches
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Add a generic lookup_table() helper whose role consists in
> scanning a contiguous table located in guest RAM and applying
> a callback on each entry. Entries can be handled as linked lists
> since the callback may return an offset to the next entry and
> also tell that an entry is the last one.
>
> Helper functions also are added to compute the device/event ID
> offset to the next DTE/ITE.
>
> compute_next_devid_offset, compute_next_eventid_offset and
> lookup_table will become static in subsequent patches
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 17/22] KVM: arm64: vgic-its: Collection table save/restore

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> The save path copies the collection entries into guest RAM
> at the GPA specified in the BASER register. This obviously
> requires the BASER to be set. The last written element is a
> dummy collection table entry.
>
> We do not index by collection ID as the collection entry
> can fit into 8 bytes while containing the collection ID.
>
> On restore path we re-allocate the collection objects.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 17/22] KVM: arm64: vgic-its: Collection table save/restore

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> The save path copies the collection entries into guest RAM
> at the GPA specified in the BASER register. This obviously
> requires the BASER to be set. The last written element is a
> dummy collection table entry.
>
> We do not index by collection ID as the collection entry
> can fit into 8 bytes while containing the collection ID.
>
> On restore path we re-allocate the collection objects.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 18/22] KVM: arm64: vgic-its: vgic_its_check_id returns the entry's GPA

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> As vgic_its_check_id() computes the device/collection entry's
> GPA, let's return it so that new callers can retrieve it easily.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 18/22] KVM: arm64: vgic-its: vgic_its_check_id returns the entry's GPA

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> As vgic_its_check_id() computes the device/collection entry's
> GPA, let's return it so that new callers can retrieve it easily.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Introduce routines to save and restore device ITT and their
> interrupt table entries (ITE).
>
> The routines will be called on device table save and
> restore. They will become static in subsequent patches.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> Introduce routines to save and restore device ITT and their
> interrupt table entries (ITE).
>
> The routines will be called on device table save and
> restore. They will become static in subsequent patches.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> This patch saves the device table entries into guest RAM.
> Both flat table and 2 stage tables are supported. DeviceId
> indexing is used.
>
> For each device listed in the device table, we also save
> the translation table using the vgic_its_save/restore_itt
> routines.
>
> On restore, devices are re-allocated and their ite are
> re-built.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> This patch saves the device table entries into guest RAM.
> Both flat table and 2 stage tables are supported. DeviceId
> indexing is used.
>
> For each device listed in the device table, we also save
> the translation table using the vgic_its_save/restore_itt
> routines.
>
> On restore, devices are re-allocated and their ite are
> re-built.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 21/22] KVM: arm64: vgic-its: Fix pending table sync

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> In its_sync_lpi_pending_table() we currently ignore the
> target_vcpu of the LPIs. We sync the pending bit found in
> the vcpu pending table even if the LPI is not targeting it.
>
> Also in vgic_its_cmd_handle_invall() we are supposed to
> read the config table data for the LPIs associated to the
> collection ID. At the moment we refresh all LPI config
> information.
>
> This patch passes a vpcu to vgic_copy_lpi_list() so that
> this latter returns a snapshot of the LPIs targeting this
> CPU and only those.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 21/22] KVM: arm64: vgic-its: Fix pending table sync

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> In its_sync_lpi_pending_table() we currently ignore the
> target_vcpu of the LPIs. We sync the pending bit found in
> the vcpu pending table even if the LPI is not targeting it.
>
> Also in vgic_its_cmd_handle_invall() we are supposed to
> read the config table data for the LPIs associated to the
> collection ID. At the moment we refresh all LPI config
> information.
>
> This patch passes a vpcu to vgic_copy_lpi_list() so that
> this latter returns a snapshot of the LPIs targeting this
> CPU and only those.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 22/22] KVM: arm64: vgic-v3: KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> This patch adds a new attribute to GICV3 KVM device
> KVM_DEV_ARM_VGIC_GRP_CTRL group. This Allows the userspace to
> flush all GICR pending tables into guest RAM. At the moment
> we do not offer any restore control as the sync is implicit.
>
> As we need the PENDBASER_ADDRESS() in vgic-v3, let's move its
> definition in the irqchip header. We restore the full length
> of the field, ie [51:16]. Same for PROPBASER_ADDRESS with full
> field length of [51:12].
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 22/22] KVM: arm64: vgic-v3: KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES

[Prakash B]

On Fri, Apr 14, 2017 at 3:45 PM, Eric Auger <eric.auger@redhat.com> wrote:
> This patch adds a new attribute to GICV3 KVM device
> KVM_DEV_ARM_VGIC_GRP_CTRL group. This Allows the userspace to
> flush all GICR pending tables into guest RAM. At the moment
> we do not offer any restore control as the sync is implicit.
>
> As we need the PENDBASER_ADDRESS() in vgic-v3, let's move its
> definition in the irqchip header. We restore the full length
> of the field, ie [51:16]. Same for PROPBASER_ADDRESS with full
> field length of [51:12].
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
Tested-by: Prakash, Brahmajyosyula <Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 00/22] vITS save/restore

[Prakash B]

Verified on Cavium ThunderX platform
Continousily migrated Guest more than 500 times b/w different  hosts.
Please feel free to add  Tested-by: Prakash, Brahmajyosyula
<Brahmajyosyula.Prakash@cavium.com>

[PATCH v5 00/22] vITS save/restore

[Prakash B]

Verified on Cavium ThunderX platform
Continousily migrated Guest more than 500 times b/w different  hosts.
Please feel free to add  Tested-by: Prakash, Brahmajyosyula
<Brahmajyosyula.Prakash@cavium.com>

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> Add description for how to access ITS registers and how to save/restore
> ITS tables into/from memory.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - take into account Christoffer's comments
> - pending table save on GICV3 side now
> 
> v3 -> v4:
> - take into account Peter's comments:
>   - typos
>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>   - add a validity bit in DTE
>   - document all fields in CTE and ITE
>   - document ABI revision
> - take into account Andre's comments:
>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>   - document -EBUSY error if one or more VCPUS are runnning
>   - document 64b registers only can be accessed with 64b access
> - itt_addr field matches bits [51:8] of the itt_addr
> 
> v1 -> v2:
> - DTE and ITE now are 8 bytes
> - DTE and ITE now indexed by deviceid/eventid
> - use ITE name instead of ITTE
> - mentions ITT_addr matches bits [51:8] of the actual address
> - mentions LE layout
> ---
>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>  1 file changed, 99 insertions(+)
> 
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> index 6081a5b..b5f010d 100644
> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> @@ -32,7 +32,106 @@ Groups:
>      KVM_DEV_ARM_VGIC_CTRL_INIT
>        request the initialization of the ITS, no additional parameter in
>        kvm_device_attr.addr.
> +
> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> +      save the ITS table data into guest RAM, at the location provisioned
> +      by the guest in corresponding registers/table entries.
> +
> +      The layout of the tables in guest memory defines an ABI. The entries
> +      are laid out in little endian format as described in the last paragraph.
> +
> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> +      restore the ITS tables from guest RAM to ITS internal structures.
> +
> +      The GICV3 must be restored before the ITS and all ITS registers but
> +      the GITS_CTLR must be restored before restoring the ITS tables.
> +
> +      The GITS_IIDR read-only register must also be restored before
> +      the table restore as the IIDR revision field encodes the ABI revision.
> +

what is the expected sequence of operations.  For example, to restore
the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?

Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
and restore GITS_CTLR (which enables the ITS)?

>    Errors:
>      -ENXIO:  ITS not properly configured as required prior to setting
>               this attribute
>      -ENOMEM: Memory shortage when allocating ITS internal data
> +    -EINVAL: Inconsistent restored data
> +    -EFAULT: Invalid guest ram access
> +    -EBUSY:  One or more VCPUS are running
> +
> +  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> +  Attributes:
> +      The attr field of kvm_device_attr encodes the offset of the
> +      ITS register, relative to the ITS control frame base address
> +      (ITS_base).
> +
> +      kvm_device_attr.addr points to a __u64 value whatever the width
> +      of the addressed register (32/64 bits). 64 bit registers can only
> +      be accessed with full length.
> +
> +      Writes to read-only registers are ignored by the kernel except for:
> +      - GITS_READR. It needs to be restored otherwise commands in the queue
> +        will be re-executed after restoring CWRITER. GITS_READR must be restored
> +        before restoring the GITS_CTLR which is likely to enable the ITS.
> +        Also it needs to be restored after GITS_CBASER since a write to
> +        GITS_CBASER resets GITS_CREADR.
> +      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
> +        In the future we might implement direct injection of virtual LPIS.
> +        This will require an upgrade of the table layout and an evolution of
> +        the ABI. GITS_IIDR must be restored before the table restoration.
> +
> +      For other registers, getting or setting a register has the same
> +      effect as reading/writing the register on real hardware.
> +  Errors:
> +    -ENXIO: Offset does not correspond to any supported register
> +    -EFAULT: Invalid user pointer for attr->addr
> +    -EINVAL: Offset is not 64-bit aligned
> +    -EBUSY: one or more VCPUS are running


It may be helpful to state the ordering requirements somewhere:

Restoring the ITS:
------------------
Restoring the ITS requires certain things to happen in order.
Specifically:
 1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
 2. Restore GITS_IIDR
 3. Restore GITS_CBASER
 4. Restore GITS_READR
 5. Restore remainin registers except GITS_CTLR
 6. Make sure all guest memory is restored
 7. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)


> +
> + ITS Table ABI REV0:
> + -------------------
> +
> + Revision 0 of the ABI only supports physical LPIs.
> +
> + The device table and ITT are indexed by the deviceid and eventid,
> + respectively. The collection table is not indexed by collectionid:
> + CTE are written in the table in the order of collection creation. All
> + entries are 8 bytes.
> +
> + Device Table Entry (DTE):
> +
> + bits:     | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 |
> + values:   | V |   next    | ITT_addr |  Size   |
> +
> + where;
> + - V indicates whether the entry is valid. If not, other fields
> +   are not meaningful.
> + - next: equals to 0 if this entry is the last one; otherwise it
> +   corresponds to the deviceid offset to the next DTE, capped by
> +   2^14 -1.
> + - ITT_addr matches bits [51:8] of the ITT address (256B aligned).

I assume the B here is bytes.  Where does this requirement come from?

> + - Size specifies the supported number of bits for the deviceid,
> +   minus one

deviceid or eventid?

> +
> + Collection Table Entry (CTE):
> +
> + bits:     | 63| 62 ..  52  | 51 ... 16 | 15  ...   0 |
> + values:   | V |    RES0    |  RDBase   |    ICID     |
> +
> + where:
> + - V indicates whether the entry is valid. If not, other fields are
> +   not meaningful.
> + - RES0: reserved field with Should-Be-Zero-or-Preserved behavior.
> + - RDBase is the PE number (GICR_TYPER.Processor_Number semantic),
> + - ICID is the collection ID
> +
> + Interrupt Translation Entry (ITE):
> +
> + bits:     | 63 ... 48 | 47 ... 16 | 15 ... 0 |
> + values:   |    next   |   pINTID  |  ICID    |
> +
> + where:
> + - next: equals to 0 if this entry is the last one; otherwise it corresponds
> +   to the eventid offset to the next ITE capped by 2^16 -1.
> + - pINTID is the physical LPI ID; if zero, it means the entry is not valid
> +   and other fields are not meaningful.
> + - ICID is the collection ID
> +
> -- 
> 2.5.5
> 

Besides the minor suggestions above:

Reviewed-by: Christoffer Dall <cdall@linaro.org>

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> Add description for how to access ITS registers and how to save/restore
> ITS tables into/from memory.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - take into account Christoffer's comments
> - pending table save on GICV3 side now
> 
> v3 -> v4:
> - take into account Peter's comments:
>   - typos
>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>   - add a validity bit in DTE
>   - document all fields in CTE and ITE
>   - document ABI revision
> - take into account Andre's comments:
>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>   - document -EBUSY error if one or more VCPUS are runnning
>   - document 64b registers only can be accessed with 64b access
> - itt_addr field matches bits [51:8] of the itt_addr
> 
> v1 -> v2:
> - DTE and ITE now are 8 bytes
> - DTE and ITE now indexed by deviceid/eventid
> - use ITE name instead of ITTE
> - mentions ITT_addr matches bits [51:8] of the actual address
> - mentions LE layout
> ---
>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>  1 file changed, 99 insertions(+)
> 
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> index 6081a5b..b5f010d 100644
> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> @@ -32,7 +32,106 @@ Groups:
>      KVM_DEV_ARM_VGIC_CTRL_INIT
>        request the initialization of the ITS, no additional parameter in
>        kvm_device_attr.addr.
> +
> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> +      save the ITS table data into guest RAM, at the location provisioned
> +      by the guest in corresponding registers/table entries.
> +
> +      The layout of the tables in guest memory defines an ABI. The entries
> +      are laid out in little endian format as described in the last paragraph.
> +
> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> +      restore the ITS tables from guest RAM to ITS internal structures.
> +
> +      The GICV3 must be restored before the ITS and all ITS registers but
> +      the GITS_CTLR must be restored before restoring the ITS tables.
> +
> +      The GITS_IIDR read-only register must also be restored before
> +      the table restore as the IIDR revision field encodes the ABI revision.
> +

what is the expected sequence of operations.  For example, to restore
the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?

Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
and restore GITS_CTLR (which enables the ITS)?

>    Errors:
>      -ENXIO:  ITS not properly configured as required prior to setting
>               this attribute
>      -ENOMEM: Memory shortage when allocating ITS internal data
> +    -EINVAL: Inconsistent restored data
> +    -EFAULT: Invalid guest ram access
> +    -EBUSY:  One or more VCPUS are running
> +
> +  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> +  Attributes:
> +      The attr field of kvm_device_attr encodes the offset of the
> +      ITS register, relative to the ITS control frame base address
> +      (ITS_base).
> +
> +      kvm_device_attr.addr points to a __u64 value whatever the width
> +      of the addressed register (32/64 bits). 64 bit registers can only
> +      be accessed with full length.
> +
> +      Writes to read-only registers are ignored by the kernel except for:
> +      - GITS_READR. It needs to be restored otherwise commands in the queue
> +        will be re-executed after restoring CWRITER. GITS_READR must be restored
> +        before restoring the GITS_CTLR which is likely to enable the ITS.
> +        Also it needs to be restored after GITS_CBASER since a write to
> +        GITS_CBASER resets GITS_CREADR.
> +      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
> +        In the future we might implement direct injection of virtual LPIS.
> +        This will require an upgrade of the table layout and an evolution of
> +        the ABI. GITS_IIDR must be restored before the table restoration.
> +
> +      For other registers, getting or setting a register has the same
> +      effect as reading/writing the register on real hardware.
> +  Errors:
> +    -ENXIO: Offset does not correspond to any supported register
> +    -EFAULT: Invalid user pointer for attr->addr
> +    -EINVAL: Offset is not 64-bit aligned
> +    -EBUSY: one or more VCPUS are running


It may be helpful to state the ordering requirements somewhere:

Restoring the ITS:
------------------
Restoring the ITS requires certain things to happen in order.
Specifically:
 1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
 2. Restore GITS_IIDR
 3. Restore GITS_CBASER
 4. Restore GITS_READR
 5. Restore remainin registers except GITS_CTLR
 6. Make sure all guest memory is restored
 7. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)


> +
> + ITS Table ABI REV0:
> + -------------------
> +
> + Revision 0 of the ABI only supports physical LPIs.
> +
> + The device table and ITT are indexed by the deviceid and eventid,
> + respectively. The collection table is not indexed by collectionid:
> + CTE are written in the table in the order of collection creation. All
> + entries are 8 bytes.
> +
> + Device Table Entry (DTE):
> +
> + bits:     | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 |
> + values:   | V |   next    | ITT_addr |  Size   |
> +
> + where;
> + - V indicates whether the entry is valid. If not, other fields
> +   are not meaningful.
> + - next: equals to 0 if this entry is the last one; otherwise it
> +   corresponds to the deviceid offset to the next DTE, capped by
> +   2^14 -1.
> + - ITT_addr matches bits [51:8] of the ITT address (256B aligned).

I assume the B here is bytes.  Where does this requirement come from?

> + - Size specifies the supported number of bits for the deviceid,
> +   minus one

deviceid or eventid?

> +
> + Collection Table Entry (CTE):
> +
> + bits:     | 63| 62 ..  52  | 51 ... 16 | 15  ...   0 |
> + values:   | V |    RES0    |  RDBase   |    ICID     |
> +
> + where:
> + - V indicates whether the entry is valid. If not, other fields are
> +   not meaningful.
> + - RES0: reserved field with Should-Be-Zero-or-Preserved behavior.
> + - RDBase is the PE number (GICR_TYPER.Processor_Number semantic),
> + - ICID is the collection ID
> +
> + Interrupt Translation Entry (ITE):
> +
> + bits:     | 63 ... 48 | 47 ... 16 | 15 ... 0 |
> + values:   |    next   |   pINTID  |  ICID    |
> +
> + where:
> + - next: equals to 0 if this entry is the last one; otherwise it corresponds
> +   to the eventid offset to the next ITE capped by 2^16 -1.
> + - pINTID is the physical LPI ID; if zero, it means the entry is not valid
> +   and other fields are not meaningful.
> + - ICID is the collection ID
> +
> -- 
> 2.5.5
> 

Besides the minor suggestions above:

Reviewed-by: Christoffer Dall <cdall@linaro.org>

Re: [PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Christoffer Dall]

On Wed, Apr 26, 2017 at 11:57:16AM +0200, Auger Eric wrote:
> Hi Peter, Dave,
> 
> On 26/04/2017 10:48, Dr. David Alan Gilbert wrote:
> > * Peter Maydell (peter.maydell@linaro.org) wrote:
> >> On 26 April 2017 at 09:26, Auger Eric <eric.auger@redhat.com> wrote:
> >>> On 25/04/2017 12:43, Peter Maydell wrote:
> >>>> When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
> >>>> etc at invalid memory, presumably?)
> >>>
> >>> Yes that's correct, when GICR_PENDBASER contains a bad GPA.
> >>>
> >>>>  How does the QEMU migration code
> >>>> handle this case? Failing migration because the guest has done something
> >>>> silly doesn't seem too palatable, but trying to avoid that could be
> >>>> more effort than an obscure corner case really merits.
> >>>
> >>> The kvm_device_access will cause an abort() as for other errors returned
> >>> by kvm_device_ioctl().
> >>
> >> That's pretty nasty. Guests shouldn't be able to provoke QEMU
> >> into abort()ing, ideally. We don't necessarily have to produce
> >> a successful migration, but we should at least fail it cleanly.
> > 
> > Yes, no abort()'s during migration due to guest behaviour.
> > They always end up coming back around to being filed as migration
> > bugs and people worry why they've got cores.
> > 
> > Ideally log a message into stderr to say that the guest state
> > is inconsistent so that when someone comes to debug it then they
> > can see it's obvious.
> 
> OK I agree. I will respin the QEMU part accordingly and in that
> situation I won't abort and will print a message.
> 
Alternatively we should mark a pending error notification to the guest
in KVM, so that when the guest boots it gets something like an SError
instead, given that presumably the guest wrote the weird value.  Except
of course if the problem is caused by QEMU fudging with the register
value for the PENDBASER.

Just a thought.

Thanks,
-Christoffer

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Christoffer Dall]

On Wed, Apr 26, 2017 at 11:57:16AM +0200, Auger Eric wrote:
> Hi Peter, Dave,
> 
> On 26/04/2017 10:48, Dr. David Alan Gilbert wrote:
> > * Peter Maydell (peter.maydell at linaro.org) wrote:
> >> On 26 April 2017 at 09:26, Auger Eric <eric.auger@redhat.com> wrote:
> >>> On 25/04/2017 12:43, Peter Maydell wrote:
> >>>> When does the -EFAULT return happen? (if the guest points GITS_BASER<n>
> >>>> etc at invalid memory, presumably?)
> >>>
> >>> Yes that's correct, when GICR_PENDBASER contains a bad GPA.
> >>>
> >>>>  How does the QEMU migration code
> >>>> handle this case? Failing migration because the guest has done something
> >>>> silly doesn't seem too palatable, but trying to avoid that could be
> >>>> more effort than an obscure corner case really merits.
> >>>
> >>> The kvm_device_access will cause an abort() as for other errors returned
> >>> by kvm_device_ioctl().
> >>
> >> That's pretty nasty. Guests shouldn't be able to provoke QEMU
> >> into abort()ing, ideally. We don't necessarily have to produce
> >> a successful migration, but we should at least fail it cleanly.
> > 
> > Yes, no abort()'s during migration due to guest behaviour.
> > They always end up coming back around to being filed as migration
> > bugs and people worry why they've got cores.
> > 
> > Ideally log a message into stderr to say that the guest state
> > is inconsistent so that when someone comes to debug it then they
> > can see it's obvious.
> 
> OK I agree. I will respin the QEMU part accordingly and in that
> situation I won't abort and will print a message.
> 
Alternatively we should mark a pending error notification to the guest
in KVM, so that when the guest boots it gets something like an SError
instead, given that presumably the guest wrote the weird value.  Except
of course if the problem is caused by QEMU fudging with the register
value for the PENDBASER.

Just a thought.

Thanks,
-Christoffer

Re: [PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Peter Maydell]

On 26 April 2017 at 14:00, Christoffer Dall <cdall@linaro.org> wrote:
> Alternatively we should mark a pending error notification to the guest
> in KVM, so that when the guest boots it gets something like an SError
> instead, given that presumably the guest wrote the weird value.  Except
> of course if the problem is caused by QEMU fudging with the register
> value for the PENDBASER.

If we have scope for complaining at the guest we should do it at
the point where the guest sets PENDBASER in the first place...

thanks
-- PMM

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Peter Maydell]

On 26 April 2017 at 14:00, Christoffer Dall <cdall@linaro.org> wrote:
> Alternatively we should mark a pending error notification to the guest
> in KVM, so that when the guest boots it gets something like an SError
> instead, given that presumably the guest wrote the weird value.  Except
> of course if the problem is caused by QEMU fudging with the register
> value for the PENDBASER.

If we have scope for complaining at the guest we should do it at
the point where the guest sets PENDBASER in the first place...

thanks
-- PMM

Re: [PATCH v5 00/22] vITS save/restore

[Christoffer Dall]

Hi Prakash,

On Wed, Apr 26, 2017 at 05:08:07PM +0530, Prakash B wrote:
> Verified on Cavium ThunderX platform
> Continousily migrated Guest more than 500 times b/w different  hosts.
> Please feel free to add  Tested-by: Prakash, Brahmajyosyula
> <Brahmajyosyula.Prakash@cavium.com>

I appreciate you looking at these patches and testing them (reviews
would have been even more helpful), but in the future you don't need to
reply to each individual patch that you've tested it, but can just give
your tested-by as a single reply to the cover letter.

Thanks,
-Christoffer

[PATCH v5 00/22] vITS save/restore

[Christoffer Dall]

Hi Prakash,

On Wed, Apr 26, 2017 at 05:08:07PM +0530, Prakash B wrote:
> Verified on Cavium ThunderX platform
> Continousily migrated Guest more than 500 times b/w different  hosts.
> Please feel free to add  Tested-by: Prakash, Brahmajyosyula
> <Brahmajyosyula.Prakash@cavium.com>

I appreciate you looking at these patches and testing them (reviews
would have been even more helpful), but in the future you don't need to
reply to each individual patch that you've tested it, but can just give
your tested-by as a single reply to the cover letter.

Thanks,
-Christoffer

Re: [PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Christoffer Dall]

On Wed, Apr 26, 2017 at 02:01:55PM +0100, Peter Maydell wrote:
> On 26 April 2017 at 14:00, Christoffer Dall <cdall@linaro.org> wrote:
> > Alternatively we should mark a pending error notification to the guest
> > in KVM, so that when the guest boots it gets something like an SError
> > instead, given that presumably the guest wrote the weird value.  Except
> > of course if the problem is caused by QEMU fudging with the register
> > value for the PENDBASER.
> 
> If we have scope for complaining at the guest we should do it at
> the point where the guest sets PENDBASER in the first place...
> 

Is that what the hardware would have done?

Also, userspace could restore a bogus value in the PENDBASER (even
though the guest wrote something sane), so maybe we should just keep
this as is and handle it nicely in QEMU?

Thanks,
-Christoffer

> thanks
> -- PMM

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Christoffer Dall]

On Wed, Apr 26, 2017 at 02:01:55PM +0100, Peter Maydell wrote:
> On 26 April 2017 at 14:00, Christoffer Dall <cdall@linaro.org> wrote:
> > Alternatively we should mark a pending error notification to the guest
> > in KVM, so that when the guest boots it gets something like an SError
> > instead, given that presumably the guest wrote the weird value.  Except
> > of course if the problem is caused by QEMU fudging with the register
> > value for the PENDBASER.
> 
> If we have scope for complaining at the guest we should do it at
> the point where the guest sets PENDBASER in the first place...
> 

Is that what the hardware would have done?

Also, userspace could restore a bogus value in the PENDBASER (even
though the guest wrote something sane), so maybe we should just keep
this as is and handle it nicely in QEMU?

Thanks,
-Christoffer

> thanks
> -- PMM

Re: [PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Peter Maydell]

On 26 April 2017 at 14:14, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Wed, Apr 26, 2017 at 02:01:55PM +0100, Peter Maydell wrote:
>> On 26 April 2017 at 14:00, Christoffer Dall <cdall@linaro.org> wrote:
>> > Alternatively we should mark a pending error notification to the guest
>> > in KVM, so that when the guest boots it gets something like an SError
>> > instead, given that presumably the guest wrote the weird value.  Except
>> > of course if the problem is caused by QEMU fudging with the register
>> > value for the PENDBASER.
>>
>> If we have scope for complaining at the guest we should do it at
>> the point where the guest sets PENDBASER in the first place...
>>
>
> Is that what the hardware would have done?

I think it's UNPREDICTABLE to enable the GIC with a bogus PENDBASER,
but I can't find the bit in the spec that actually says that.
I don't know what hardware actually does, but I imagine it will
only notice that it's been handed bogus memory at the point where
it tries to use it.

> Also, userspace could restore a bogus value in the PENDBASER (even
> though the guest wrote something sane), so maybe we should just keep
> this as is and handle it nicely in QEMU?

Yeah, I don't have a strong objection to doing it that way round.

thanks
-- PMM

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Peter Maydell]

On 26 April 2017 at 14:14, Christoffer Dall <christoffer.dall@linaro.org> wrote:
> On Wed, Apr 26, 2017 at 02:01:55PM +0100, Peter Maydell wrote:
>> On 26 April 2017 at 14:00, Christoffer Dall <cdall@linaro.org> wrote:
>> > Alternatively we should mark a pending error notification to the guest
>> > in KVM, so that when the guest boots it gets something like an SError
>> > instead, given that presumably the guest wrote the weird value.  Except
>> > of course if the problem is caused by QEMU fudging with the register
>> > value for the PENDBASER.
>>
>> If we have scope for complaining at the guest we should do it at
>> the point where the guest sets PENDBASER in the first place...
>>
>
> Is that what the hardware would have done?

I think it's UNPREDICTABLE to enable the GIC with a bogus PENDBASER,
but I can't find the bit in the spec that actually says that.
I don't know what hardware actually does, but I imagine it will
only notice that it's been handed bogus memory at the point where
it tries to use it.

> Also, userspace could restore a bogus value in the PENDBASER (even
> though the guest wrote something sane), so maybe we should just keep
> this as is and handle it nicely in QEMU?

Yeah, I don't have a strong objection to doing it that way round.

thanks
-- PMM

Re: [PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Auger Eric]

Hi Peter, Christoffer,

On 26/04/2017 15:26, Peter Maydell wrote:
> On 26 April 2017 at 14:14, Christoffer Dall <christoffer.dall@linaro.org> wrote:
>> On Wed, Apr 26, 2017 at 02:01:55PM +0100, Peter Maydell wrote:
>>> On 26 April 2017 at 14:00, Christoffer Dall <cdall@linaro.org> wrote:
>>>> Alternatively we should mark a pending error notification to the guest
>>>> in KVM, so that when the guest boots it gets something like an SError
>>>> instead, given that presumably the guest wrote the weird value.  Except
>>>> of course if the problem is caused by QEMU fudging with the register
>>>> value for the PENDBASER.
>>>
>>> If we have scope for complaining at the guest we should do it at
>>> the point where the guest sets PENDBASER in the first place...
>>>
>>
>> Is that what the hardware would have done?
> 
> I think it's UNPREDICTABLE to enable the GIC with a bogus PENDBASER,
> but I can't find the bit in the spec that actually says that.
> I don't know what hardware actually does, but I imagine it will
> only notice that it's been handed bogus memory at the point where
> it tries to use it.
> 
>> Also, userspace could restore a bogus value in the PENDBASER (even
>> though the guest wrote something sane), so maybe we should just keep
>> this as is and handle it nicely in QEMU?
> 
> Yeah, I don't have a strong objection to doing it that way round.

OK. I will only update the QEMU code then.

For info, without talking about save/restore, the GICR_PENDBASER is
sync'ed on LPI enable. if the vITS gets an error on kvm_read_guest, we
currently abort the sync without reporting any error.

GICR_PROPBASER is read on cmd execution (MAPI, INV, INVALL). No error is
reported at the moment. My understanding is our implementation chose the
3d alternative of GICV3 arch spec (6.3.2), ie. "the data that generated
the error or errors is treated as having a legal value",  increment the
read cursor and currently we don't report any system error to the guest.

Thanks

Eric
> 
> thanks
> -- PMM
> 

[PATCH v5 02/22] KVM: arm/arm64: Add GICV3 pending table save API documentation

[Auger Eric]

Hi Peter, Christoffer,

On 26/04/2017 15:26, Peter Maydell wrote:
> On 26 April 2017 at 14:14, Christoffer Dall <christoffer.dall@linaro.org> wrote:
>> On Wed, Apr 26, 2017 at 02:01:55PM +0100, Peter Maydell wrote:
>>> On 26 April 2017 at 14:00, Christoffer Dall <cdall@linaro.org> wrote:
>>>> Alternatively we should mark a pending error notification to the guest
>>>> in KVM, so that when the guest boots it gets something like an SError
>>>> instead, given that presumably the guest wrote the weird value.  Except
>>>> of course if the problem is caused by QEMU fudging with the register
>>>> value for the PENDBASER.
>>>
>>> If we have scope for complaining at the guest we should do it at
>>> the point where the guest sets PENDBASER in the first place...
>>>
>>
>> Is that what the hardware would have done?
> 
> I think it's UNPREDICTABLE to enable the GIC with a bogus PENDBASER,
> but I can't find the bit in the spec that actually says that.
> I don't know what hardware actually does, but I imagine it will
> only notice that it's been handed bogus memory at the point where
> it tries to use it.
> 
>> Also, userspace could restore a bogus value in the PENDBASER (even
>> though the guest wrote something sane), so maybe we should just keep
>> this as is and handle it nicely in QEMU?
> 
> Yeah, I don't have a strong objection to doing it that way round.

OK. I will only update the QEMU code then.

For info, without talking about save/restore, the GICR_PENDBASER is
sync'ed on LPI enable. if the vITS gets an error on kvm_read_guest, we
currently abort the sync without reporting any error.

GICR_PROPBASER is read on cmd execution (MAPI, INV, INVALL). No error is
reported at the moment. My understanding is our implementation chose the
3d alternative of GICV3 arch spec (6.3.2), ie. "the data that generated
the error or errors is treated as having a legal value",  increment the
read cursor and currently we don't report any system error to the guest.

Thanks

Eric
> 
> thanks
> -- PMM
> 

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Christoffer,

On 26/04/2017 14:31, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>> Add description for how to access ITS registers and how to save/restore
>> ITS tables into/from memory.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - take into account Christoffer's comments
>> - pending table save on GICV3 side now
>>
>> v3 -> v4:
>> - take into account Peter's comments:
>>   - typos
>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>   - add a validity bit in DTE
>>   - document all fields in CTE and ITE
>>   - document ABI revision
>> - take into account Andre's comments:
>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>   - document -EBUSY error if one or more VCPUS are runnning
>>   - document 64b registers only can be accessed with 64b access
>> - itt_addr field matches bits [51:8] of the itt_addr
>>
>> v1 -> v2:
>> - DTE and ITE now are 8 bytes
>> - DTE and ITE now indexed by deviceid/eventid
>> - use ITE name instead of ITTE
>> - mentions ITT_addr matches bits [51:8] of the actual address
>> - mentions LE layout
>> ---
>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>  1 file changed, 99 insertions(+)
>>
>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>> index 6081a5b..b5f010d 100644
>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>> @@ -32,7 +32,106 @@ Groups:
>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>        request the initialization of the ITS, no additional parameter in
>>        kvm_device_attr.addr.
>> +
>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>> +      save the ITS table data into guest RAM, at the location provisioned
>> +      by the guest in corresponding registers/table entries.
>> +
>> +      The layout of the tables in guest memory defines an ABI. The entries
>> +      are laid out in little endian format as described in the last paragraph.
>> +
>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>> +      restore the ITS tables from guest RAM to ITS internal structures.
>> +
>> +      The GICV3 must be restored before the ITS and all ITS registers but
>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>> +
>> +      The GITS_IIDR read-only register must also be restored before
>> +      the table restore as the IIDR revision field encodes the ABI revision.
>> +
> 
> what is the expected sequence of operations.  For example, to restore
> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
except GITS_CTLR, then table restore, then GITS_CTLR
> 
> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> and restore GITS_CTLR (which enables the ITS)?

Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
that the pending table is read. But the whole pending table is not read
as we only iterate on registered LPIs. So the ITT must have been
restored previously.

I became aware that the pending table sync is done twice, once in the
pending table restore,  and once in the GITS_CTLR restore. So if we
leave this order specification, I should be able to remove the sync on
table restore. This was the original reason why GITS_CTLR restore has
been done at the very end.
> 
>>    Errors:
>>      -ENXIO:  ITS not properly configured as required prior to setting
>>               this attribute
>>      -ENOMEM: Memory shortage when allocating ITS internal data
>> +    -EINVAL: Inconsistent restored data
>> +    -EFAULT: Invalid guest ram access
>> +    -EBUSY:  One or more VCPUS are running
>> +
>> +  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
>> +  Attributes:
>> +      The attr field of kvm_device_attr encodes the offset of the
>> +      ITS register, relative to the ITS control frame base address
>> +      (ITS_base).
>> +
>> +      kvm_device_attr.addr points to a __u64 value whatever the width
>> +      of the addressed register (32/64 bits). 64 bit registers can only
>> +      be accessed with full length.
>> +
>> +      Writes to read-only registers are ignored by the kernel except for:
>> +      - GITS_READR. It needs to be restored otherwise commands in the queue
>> +        will be re-executed after restoring CWRITER. GITS_READR must be restored
>> +        before restoring the GITS_CTLR which is likely to enable the ITS.
>> +        Also it needs to be restored after GITS_CBASER since a write to
>> +        GITS_CBASER resets GITS_CREADR.
>> +      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
>> +        In the future we might implement direct injection of virtual LPIS.
>> +        This will require an upgrade of the table layout and an evolution of
>> +        the ABI. GITS_IIDR must be restored before the table restoration.
>> +
>> +      For other registers, getting or setting a register has the same
>> +      effect as reading/writing the register on real hardware.
>> +  Errors:
>> +    -ENXIO: Offset does not correspond to any supported register
>> +    -EFAULT: Invalid user pointer for attr->addr
>> +    -EINVAL: Offset is not 64-bit aligned
>> +    -EBUSY: one or more VCPUS are running
> 
> 
> It may be helpful to state the ordering requirements somewhere:
> 
> Restoring the ITS:
> ------------------
> Restoring the ITS requires certain things to happen in order.
> Specifically:
>  1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>  2. Restore GITS_IIDR
>  3. Restore GITS_CBASER
>  4. Restore GITS_READR
>  5. Restore remainin registers except GITS_CTLR
>  6. Make sure all guest memory is restored
>  7. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)

OK I will try to fit that description somewhere.

> 
> 
>> +
>> + ITS Table ABI REV0:
>> + -------------------
>> +
>> + Revision 0 of the ABI only supports physical LPIs.
>> +
>> + The device table and ITT are indexed by the deviceid and eventid,
>> + respectively. The collection table is not indexed by collectionid:
>> + CTE are written in the table in the order of collection creation. All
>> + entries are 8 bytes.
>> +
>> + Device Table Entry (DTE):
>> +
>> + bits:     | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 |
>> + values:   | V |   next    | ITT_addr |  Size   |
>> +
>> + where;
>> + - V indicates whether the entry is valid. If not, other fields
>> +   are not meaningful.
>> + - next: equals to 0 if this entry is the last one; otherwise it
>> +   corresponds to the deviceid offset to the next DTE, capped by
>> +   2^14 -1.
>> + - ITT_addr matches bits [51:8] of the ITT address (256B aligned).
> 
> I assume the B here is bytes.  Where does this requirement come from?
Yes this is 256 byte aligned. I guess this is to save memory as the ITT
is devised to encode a small range of eventids.
> 
>> + - Size specifies the supported number of bits for the deviceid,
>> +   minus one
> 
> deviceid or eventid?
oups eventid thanks.
> 
>> +
>> + Collection Table Entry (CTE):
>> +
>> + bits:     | 63| 62 ..  52  | 51 ... 16 | 15  ...   0 |
>> + values:   | V |    RES0    |  RDBase   |    ICID     |
>> +
>> + where:
>> + - V indicates whether the entry is valid. If not, other fields are
>> +   not meaningful.
>> + - RES0: reserved field with Should-Be-Zero-or-Preserved behavior.
>> + - RDBase is the PE number (GICR_TYPER.Processor_Number semantic),
>> + - ICID is the collection ID
>> +
>> + Interrupt Translation Entry (ITE):
>> +
>> + bits:     | 63 ... 48 | 47 ... 16 | 15 ... 0 |
>> + values:   |    next   |   pINTID  |  ICID    |
>> +
>> + where:
>> + - next: equals to 0 if this entry is the last one; otherwise it corresponds
>> +   to the eventid offset to the next ITE capped by 2^16 -1.
>> + - pINTID is the physical LPI ID; if zero, it means the entry is not valid
>> +   and other fields are not meaningful.
>> + - ICID is the collection ID
>> +
>> -- 
>> 2.5.5
>>
> 
> Besides the minor suggestions above:
> 
> Reviewed-by: Christoffer Dall <cdall@linaro.org>
Thanks

Eric
> 

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Christoffer,

On 26/04/2017 14:31, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>> Add description for how to access ITS registers and how to save/restore
>> ITS tables into/from memory.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - take into account Christoffer's comments
>> - pending table save on GICV3 side now
>>
>> v3 -> v4:
>> - take into account Peter's comments:
>>   - typos
>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>   - add a validity bit in DTE
>>   - document all fields in CTE and ITE
>>   - document ABI revision
>> - take into account Andre's comments:
>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>   - document -EBUSY error if one or more VCPUS are runnning
>>   - document 64b registers only can be accessed with 64b access
>> - itt_addr field matches bits [51:8] of the itt_addr
>>
>> v1 -> v2:
>> - DTE and ITE now are 8 bytes
>> - DTE and ITE now indexed by deviceid/eventid
>> - use ITE name instead of ITTE
>> - mentions ITT_addr matches bits [51:8] of the actual address
>> - mentions LE layout
>> ---
>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>  1 file changed, 99 insertions(+)
>>
>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>> index 6081a5b..b5f010d 100644
>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>> @@ -32,7 +32,106 @@ Groups:
>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>        request the initialization of the ITS, no additional parameter in
>>        kvm_device_attr.addr.
>> +
>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>> +      save the ITS table data into guest RAM, at the location provisioned
>> +      by the guest in corresponding registers/table entries.
>> +
>> +      The layout of the tables in guest memory defines an ABI. The entries
>> +      are laid out in little endian format as described in the last paragraph.
>> +
>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>> +      restore the ITS tables from guest RAM to ITS internal structures.
>> +
>> +      The GICV3 must be restored before the ITS and all ITS registers but
>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>> +
>> +      The GITS_IIDR read-only register must also be restored before
>> +      the table restore as the IIDR revision field encodes the ABI revision.
>> +
> 
> what is the expected sequence of operations.  For example, to restore
> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
except GITS_CTLR, then table restore, then GITS_CTLR
> 
> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> and restore GITS_CTLR (which enables the ITS)?

Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
that the pending table is read. But the whole pending table is not read
as we only iterate on registered LPIs. So the ITT must have been
restored previously.

I became aware that the pending table sync is done twice, once in the
pending table restore,  and once in the GITS_CTLR restore. So if we
leave this order specification, I should be able to remove the sync on
table restore. This was the original reason why GITS_CTLR restore has
been done at the very end.
> 
>>    Errors:
>>      -ENXIO:  ITS not properly configured as required prior to setting
>>               this attribute
>>      -ENOMEM: Memory shortage when allocating ITS internal data
>> +    -EINVAL: Inconsistent restored data
>> +    -EFAULT: Invalid guest ram access
>> +    -EBUSY:  One or more VCPUS are running
>> +
>> +  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
>> +  Attributes:
>> +      The attr field of kvm_device_attr encodes the offset of the
>> +      ITS register, relative to the ITS control frame base address
>> +      (ITS_base).
>> +
>> +      kvm_device_attr.addr points to a __u64 value whatever the width
>> +      of the addressed register (32/64 bits). 64 bit registers can only
>> +      be accessed with full length.
>> +
>> +      Writes to read-only registers are ignored by the kernel except for:
>> +      - GITS_READR. It needs to be restored otherwise commands in the queue
>> +        will be re-executed after restoring CWRITER. GITS_READR must be restored
>> +        before restoring the GITS_CTLR which is likely to enable the ITS.
>> +        Also it needs to be restored after GITS_CBASER since a write to
>> +        GITS_CBASER resets GITS_CREADR.
>> +      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
>> +        In the future we might implement direct injection of virtual LPIS.
>> +        This will require an upgrade of the table layout and an evolution of
>> +        the ABI. GITS_IIDR must be restored before the table restoration.
>> +
>> +      For other registers, getting or setting a register has the same
>> +      effect as reading/writing the register on real hardware.
>> +  Errors:
>> +    -ENXIO: Offset does not correspond to any supported register
>> +    -EFAULT: Invalid user pointer for attr->addr
>> +    -EINVAL: Offset is not 64-bit aligned
>> +    -EBUSY: one or more VCPUS are running
> 
> 
> It may be helpful to state the ordering requirements somewhere:
> 
> Restoring the ITS:
> ------------------
> Restoring the ITS requires certain things to happen in order.
> Specifically:
>  1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>  2. Restore GITS_IIDR
>  3. Restore GITS_CBASER
>  4. Restore GITS_READR
>  5. Restore remainin registers except GITS_CTLR
>  6. Make sure all guest memory is restored
>  7. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)

OK I will try to fit that description somewhere.

> 
> 
>> +
>> + ITS Table ABI REV0:
>> + -------------------
>> +
>> + Revision 0 of the ABI only supports physical LPIs.
>> +
>> + The device table and ITT are indexed by the deviceid and eventid,
>> + respectively. The collection table is not indexed by collectionid:
>> + CTE are written in the table in the order of collection creation. All
>> + entries are 8 bytes.
>> +
>> + Device Table Entry (DTE):
>> +
>> + bits:     | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 |
>> + values:   | V |   next    | ITT_addr |  Size   |
>> +
>> + where;
>> + - V indicates whether the entry is valid. If not, other fields
>> +   are not meaningful.
>> + - next: equals to 0 if this entry is the last one; otherwise it
>> +   corresponds to the deviceid offset to the next DTE, capped by
>> +   2^14 -1.
>> + - ITT_addr matches bits [51:8] of the ITT address (256B aligned).
> 
> I assume the B here is bytes.  Where does this requirement come from?
Yes this is 256 byte aligned. I guess this is to save memory as the ITT
is devised to encode a small range of eventids.
> 
>> + - Size specifies the supported number of bits for the deviceid,
>> +   minus one
> 
> deviceid or eventid?
oups eventid thanks.
> 
>> +
>> + Collection Table Entry (CTE):
>> +
>> + bits:     | 63| 62 ..  52  | 51 ... 16 | 15  ...   0 |
>> + values:   | V |    RES0    |  RDBase   |    ICID     |
>> +
>> + where:
>> + - V indicates whether the entry is valid. If not, other fields are
>> +   not meaningful.
>> + - RES0: reserved field with Should-Be-Zero-or-Preserved behavior.
>> + - RDBase is the PE number (GICR_TYPER.Processor_Number semantic),
>> + - ICID is the collection ID
>> +
>> + Interrupt Translation Entry (ITE):
>> +
>> + bits:     | 63 ... 48 | 47 ... 16 | 15 ... 0 |
>> + values:   |    next   |   pINTID  |  ICID    |
>> +
>> + where:
>> + - next: equals to 0 if this entry is the last one; otherwise it corresponds
>> +   to the eventid offset to the next ITE capped by 2^16 -1.
>> + - pINTID is the physical LPI ID; if zero, it means the entry is not valid
>> +   and other fields are not meaningful.
>> + - ICID is the collection ID
>> +
>> -- 
>> 2.5.5
>>
> 
> Besides the minor suggestions above:
> 
> Reviewed-by: Christoffer Dall <cdall@linaro.org>
Thanks

Eric
> 

Re: [PATCH v5 00/22] vITS save/restore

[Auger Eric]

Hi Brahmajyosyula,

On 26/04/2017 13:38, Prakash B wrote:
> Verified on Cavium ThunderX platform
> Continousily migrated Guest more than 500 times b/w different  hosts.
> Please feel free to add  Tested-by: Prakash, Brahmajyosyula
> <Brahmajyosyula.Prakash@cavium.com>

I really appreciate. Many thanks!

Best Regards

Eric

* note you are not obliged to bother to send individual T-b on each
patch. You can just send T-b on the cover letter and if requested
exclude patches or features you could not test.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 00/22] vITS save/restore

[Auger Eric]

Hi Brahmajyosyula,

On 26/04/2017 13:38, Prakash B wrote:
> Verified on Cavium ThunderX platform
> Continousily migrated Guest more than 500 times b/w different  hosts.
> Please feel free to add  Tested-by: Prakash, Brahmajyosyula
> <Brahmajyosyula.Prakash@cavium.com>

I really appreciate. Many thanks!

Best Regards

Eric

* note you are not obliged to bother to send individual T-b on each
patch. You can just send T-b on the cover letter and if requested
exclude patches or features you could not test.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 26/04/2017 14:31, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >> Add description for how to access ITS registers and how to save/restore
> >> ITS tables into/from memory.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - take into account Christoffer's comments
> >> - pending table save on GICV3 side now
> >>
> >> v3 -> v4:
> >> - take into account Peter's comments:
> >>   - typos
> >>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>   - add a validity bit in DTE
> >>   - document all fields in CTE and ITE
> >>   - document ABI revision
> >> - take into account Andre's comments:
> >>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>   - document -EBUSY error if one or more VCPUS are runnning
> >>   - document 64b registers only can be accessed with 64b access
> >> - itt_addr field matches bits [51:8] of the itt_addr
> >>
> >> v1 -> v2:
> >> - DTE and ITE now are 8 bytes
> >> - DTE and ITE now indexed by deviceid/eventid
> >> - use ITE name instead of ITTE
> >> - mentions ITT_addr matches bits [51:8] of the actual address
> >> - mentions LE layout
> >> ---
> >>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>  1 file changed, 99 insertions(+)
> >>
> >> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >> index 6081a5b..b5f010d 100644
> >> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >> @@ -32,7 +32,106 @@ Groups:
> >>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>        request the initialization of the ITS, no additional parameter in
> >>        kvm_device_attr.addr.
> >> +
> >> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >> +      save the ITS table data into guest RAM, at the location provisioned
> >> +      by the guest in corresponding registers/table entries.
> >> +
> >> +      The layout of the tables in guest memory defines an ABI. The entries
> >> +      are laid out in little endian format as described in the last paragraph.
> >> +
> >> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >> +      restore the ITS tables from guest RAM to ITS internal structures.
> >> +
> >> +      The GICV3 must be restored before the ITS and all ITS registers but
> >> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >> +
> >> +      The GITS_IIDR read-only register must also be restored before
> >> +      the table restore as the IIDR revision field encodes the ABI revision.
> >> +
> > 
> > what is the expected sequence of operations.  For example, to restore
> > the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> > the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> except GITS_CTLR, then table restore, then GITS_CTLR
> > 
> > Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> > and restore GITS_CTLR (which enables the ITS)?
> 
> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> that the pending table is read. But the whole pending table is not read
> as we only iterate on registered LPIs. So the ITT must have been
> restored previously.
> 
> I became aware that the pending table sync is done twice, once in the
> pending table restore,  and once in the GITS_CTLR restore. So if we
> leave this order specification, I should be able to remove the sync on
> table restore. This was the original reason why GITS_CTLR restore has
> been done at the very end.

I'm sorry, I'm a bit confused.  Do we not need
KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?

> > 
> >>    Errors:
> >>      -ENXIO:  ITS not properly configured as required prior to setting
> >>               this attribute
> >>      -ENOMEM: Memory shortage when allocating ITS internal data
> >> +    -EINVAL: Inconsistent restored data
> >> +    -EFAULT: Invalid guest ram access
> >> +    -EBUSY:  One or more VCPUS are running
> >> +
> >> +  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> >> +  Attributes:
> >> +      The attr field of kvm_device_attr encodes the offset of the
> >> +      ITS register, relative to the ITS control frame base address
> >> +      (ITS_base).
> >> +
> >> +      kvm_device_attr.addr points to a __u64 value whatever the width
> >> +      of the addressed register (32/64 bits). 64 bit registers can only
> >> +      be accessed with full length.
> >> +
> >> +      Writes to read-only registers are ignored by the kernel except for:
> >> +      - GITS_READR. It needs to be restored otherwise commands in the queue
> >> +        will be re-executed after restoring CWRITER. GITS_READR must be restored
> >> +        before restoring the GITS_CTLR which is likely to enable the ITS.
> >> +        Also it needs to be restored after GITS_CBASER since a write to
> >> +        GITS_CBASER resets GITS_CREADR.
> >> +      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
> >> +        In the future we might implement direct injection of virtual LPIS.
> >> +        This will require an upgrade of the table layout and an evolution of
> >> +        the ABI. GITS_IIDR must be restored before the table restoration.
> >> +
> >> +      For other registers, getting or setting a register has the same
> >> +      effect as reading/writing the register on real hardware.
> >> +  Errors:
> >> +    -ENXIO: Offset does not correspond to any supported register
> >> +    -EFAULT: Invalid user pointer for attr->addr
> >> +    -EINVAL: Offset is not 64-bit aligned
> >> +    -EBUSY: one or more VCPUS are running
> > 
> > 
> > It may be helpful to state the ordering requirements somewhere:
> > 
> > Restoring the ITS:
> > ------------------
> > Restoring the ITS requires certain things to happen in order.
> > Specifically:
> >  1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >  2. Restore GITS_IIDR
> >  3. Restore GITS_CBASER
> >  4. Restore GITS_READR
> >  5. Restore remainin registers except GITS_CTLR
> >  6. Make sure all guest memory is restored
> >  7. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> 
> OK I will try to fit that description somewhere.
> 

Thanks,
-Christoffer

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 26/04/2017 14:31, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >> Add description for how to access ITS registers and how to save/restore
> >> ITS tables into/from memory.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - take into account Christoffer's comments
> >> - pending table save on GICV3 side now
> >>
> >> v3 -> v4:
> >> - take into account Peter's comments:
> >>   - typos
> >>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>   - add a validity bit in DTE
> >>   - document all fields in CTE and ITE
> >>   - document ABI revision
> >> - take into account Andre's comments:
> >>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>   - document -EBUSY error if one or more VCPUS are runnning
> >>   - document 64b registers only can be accessed with 64b access
> >> - itt_addr field matches bits [51:8] of the itt_addr
> >>
> >> v1 -> v2:
> >> - DTE and ITE now are 8 bytes
> >> - DTE and ITE now indexed by deviceid/eventid
> >> - use ITE name instead of ITTE
> >> - mentions ITT_addr matches bits [51:8] of the actual address
> >> - mentions LE layout
> >> ---
> >>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>  1 file changed, 99 insertions(+)
> >>
> >> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >> index 6081a5b..b5f010d 100644
> >> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >> @@ -32,7 +32,106 @@ Groups:
> >>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>        request the initialization of the ITS, no additional parameter in
> >>        kvm_device_attr.addr.
> >> +
> >> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >> +      save the ITS table data into guest RAM, at the location provisioned
> >> +      by the guest in corresponding registers/table entries.
> >> +
> >> +      The layout of the tables in guest memory defines an ABI. The entries
> >> +      are laid out in little endian format as described in the last paragraph.
> >> +
> >> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >> +      restore the ITS tables from guest RAM to ITS internal structures.
> >> +
> >> +      The GICV3 must be restored before the ITS and all ITS registers but
> >> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >> +
> >> +      The GITS_IIDR read-only register must also be restored before
> >> +      the table restore as the IIDR revision field encodes the ABI revision.
> >> +
> > 
> > what is the expected sequence of operations.  For example, to restore
> > the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> > the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> except GITS_CTLR, then table restore, then GITS_CTLR
> > 
> > Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> > and restore GITS_CTLR (which enables the ITS)?
> 
> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> that the pending table is read. But the whole pending table is not read
> as we only iterate on registered LPIs. So the ITT must have been
> restored previously.
> 
> I became aware that the pending table sync is done twice, once in the
> pending table restore,  and once in the GITS_CTLR restore. So if we
> leave this order specification, I should be able to remove the sync on
> table restore. This was the original reason why GITS_CTLR restore has
> been done at the very end.

I'm sorry, I'm a bit confused.  Do we not need
KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?

> > 
> >>    Errors:
> >>      -ENXIO:  ITS not properly configured as required prior to setting
> >>               this attribute
> >>      -ENOMEM: Memory shortage when allocating ITS internal data
> >> +    -EINVAL: Inconsistent restored data
> >> +    -EFAULT: Invalid guest ram access
> >> +    -EBUSY:  One or more VCPUS are running
> >> +
> >> +  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> >> +  Attributes:
> >> +      The attr field of kvm_device_attr encodes the offset of the
> >> +      ITS register, relative to the ITS control frame base address
> >> +      (ITS_base).
> >> +
> >> +      kvm_device_attr.addr points to a __u64 value whatever the width
> >> +      of the addressed register (32/64 bits). 64 bit registers can only
> >> +      be accessed with full length.
> >> +
> >> +      Writes to read-only registers are ignored by the kernel except for:
> >> +      - GITS_READR. It needs to be restored otherwise commands in the queue
> >> +        will be re-executed after restoring CWRITER. GITS_READR must be restored
> >> +        before restoring the GITS_CTLR which is likely to enable the ITS.
> >> +        Also it needs to be restored after GITS_CBASER since a write to
> >> +        GITS_CBASER resets GITS_CREADR.
> >> +      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
> >> +        In the future we might implement direct injection of virtual LPIS.
> >> +        This will require an upgrade of the table layout and an evolution of
> >> +        the ABI. GITS_IIDR must be restored before the table restoration.
> >> +
> >> +      For other registers, getting or setting a register has the same
> >> +      effect as reading/writing the register on real hardware.
> >> +  Errors:
> >> +    -ENXIO: Offset does not correspond to any supported register
> >> +    -EFAULT: Invalid user pointer for attr->addr
> >> +    -EINVAL: Offset is not 64-bit aligned
> >> +    -EBUSY: one or more VCPUS are running
> > 
> > 
> > It may be helpful to state the ordering requirements somewhere:
> > 
> > Restoring the ITS:
> > ------------------
> > Restoring the ITS requires certain things to happen in order.
> > Specifically:
> >  1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >  2. Restore GITS_IIDR
> >  3. Restore GITS_CBASER
> >  4. Restore GITS_READR
> >  5. Restore remainin registers except GITS_CTLR
> >  6. Make sure all guest memory is restored
> >  7. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> 
> OK I will try to fit that description somewhere.
> 

Thanks,
-Christoffer

Re: [PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:15PM +0200, Eric Auger wrote:
> The actual abbreviation for the interrupt translation table entry
> is ITE. Let's rename all itte instances by ite.

Is there really any confusion or problems with using itte?  This is a
lot of churn...


Thanks,
-Christoffer

> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
> 
> ---
> 
> v5: Add Marc's A-b
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 148 +++++++++++++++++++++----------------------
>  1 file changed, 74 insertions(+), 74 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 8d1da1a..3ffcbbe 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -114,8 +114,8 @@ struct its_collection {
>  #define its_is_collection_mapped(coll) ((coll) && \
>  				((coll)->target_addr != COLLECTION_NOT_MAPPED))
>  
> -struct its_itte {
> -	struct list_head itte_list;
> +struct its_ite {
> +	struct list_head ite_list;
>  
>  	struct vgic_irq *irq;
>  	struct its_collection *collection;
> @@ -143,27 +143,27 @@ static struct its_device *find_its_device(struct vgic_its *its, u32 device_id)
>   * Device ID/Event ID pair on an ITS.
>   * Must be called with the its_lock mutex held.
>   */
> -static struct its_itte *find_itte(struct vgic_its *its, u32 device_id,
> +static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  				  u32 event_id)
>  {
>  	struct its_device *device;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  	device = find_its_device(its, device_id);
>  	if (device == NULL)
>  		return NULL;
>  
> -	list_for_each_entry(itte, &device->itt_head, itte_list)
> -		if (itte->event_id == event_id)
> -			return itte;
> +	list_for_each_entry(ite, &device->itt_head, ite_list)
> +		if (ite->event_id == event_id)
> +			return ite;
>  
>  	return NULL;
>  }
>  
>  /* To be used as an iterator this macro misses the enclosing parentheses */
> -#define for_each_lpi_its(dev, itte, its) \
> +#define for_each_lpi_its(dev, ite, its) \
>  	list_for_each_entry(dev, &(its)->device_list, dev_list) \
> -		list_for_each_entry(itte, &(dev)->itt_head, itte_list)
> +		list_for_each_entry(ite, &(dev)->itt_head, ite_list)
>  
>  /*
>   * We only implement 48 bits of PA at the moment, although the ITS
> @@ -270,18 +270,18 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>   * Needs to be called whenever either the collection for a LPIs has
>   * changed or the collection itself got retargeted.
>   */
> -static void update_affinity_itte(struct kvm *kvm, struct its_itte *itte)
> +static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
>  {
>  	struct kvm_vcpu *vcpu;
>  
> -	if (!its_is_collection_mapped(itte->collection))
> +	if (!its_is_collection_mapped(ite->collection))
>  		return;
>  
> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>  
> -	spin_lock(&itte->irq->irq_lock);
> -	itte->irq->target_vcpu = vcpu;
> -	spin_unlock(&itte->irq->irq_lock);
> +	spin_lock(&ite->irq->irq_lock);
> +	ite->irq->target_vcpu = vcpu;
> +	spin_unlock(&ite->irq->irq_lock);
>  }
>  
>  /*
> @@ -292,13 +292,13 @@ static void update_affinity_collection(struct kvm *kvm, struct vgic_its *its,
>  				       struct its_collection *coll)
>  {
>  	struct its_device *device;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
> -	for_each_lpi_its(device, itte, its) {
> -		if (!itte->collection || coll != itte->collection)
> +	for_each_lpi_its(device, ite, its) {
> +		if (!ite->collection || coll != ite->collection)
>  			continue;
>  
> -		update_affinity_itte(kvm, itte);
> +		update_affinity_ite(kvm, ite);
>  	}
>  }
>  
> @@ -425,25 +425,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
>  				u32 devid, u32 eventid)
>  {
>  	struct kvm_vcpu *vcpu;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  	if (!its->enabled)
>  		return -EBUSY;
>  
> -	itte = find_itte(its, devid, eventid);
> -	if (!itte || !its_is_collection_mapped(itte->collection))
> +	ite = find_ite(its, devid, eventid);
> +	if (!ite || !its_is_collection_mapped(ite->collection))
>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>  
> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>  	if (!vcpu)
>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>  
>  	if (!vcpu->arch.vgic_cpu.lpis_enabled)
>  		return -EBUSY;
>  
> -	spin_lock(&itte->irq->irq_lock);
> -	itte->irq->pending_latch = true;
> -	vgic_queue_irq_unlock(kvm, itte->irq);
> +	spin_lock(&ite->irq->irq_lock);
> +	ite->irq->pending_latch = true;
> +	vgic_queue_irq_unlock(kvm, ite->irq);
>  
>  	return 0;
>  }
> @@ -511,15 +511,15 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
>  }
>  
>  /* Requires the its_lock to be held. */
> -static void its_free_itte(struct kvm *kvm, struct its_itte *itte)
> +static void its_free_ite(struct kvm *kvm, struct its_ite *ite)
>  {
> -	list_del(&itte->itte_list);
> +	list_del(&ite->ite_list);
>  
>  	/* This put matches the get in vgic_add_lpi. */
> -	if (itte->irq)
> -		vgic_put_irq(kvm, itte->irq);
> +	if (ite->irq)
> +		vgic_put_irq(kvm, ite->irq);
>  
> -	kfree(itte);
> +	kfree(ite);
>  }
>  
>  static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
> @@ -544,17 +544,17 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
>  {
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	u32 event_id = its_cmd_get_id(its_cmd);
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  
> -	itte = find_itte(its, device_id, event_id);
> -	if (itte && itte->collection) {
> +	ite = find_ite(its, device_id, event_id);
> +	if (ite && ite->collection) {
>  		/*
>  		 * Though the spec talks about removing the pending state, we
>  		 * don't bother here since we clear the ITTE anyway and the
>  		 * pending state is a property of the ITTE struct.
>  		 */
> -		its_free_itte(kvm, itte);
> +		its_free_ite(kvm, ite);
>  		return 0;
>  	}
>  
> @@ -572,26 +572,26 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>  	u32 event_id = its_cmd_get_id(its_cmd);
>  	u32 coll_id = its_cmd_get_collection(its_cmd);
>  	struct kvm_vcpu *vcpu;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  	struct its_collection *collection;
>  
> -	itte = find_itte(its, device_id, event_id);
> -	if (!itte)
> +	ite = find_ite(its, device_id, event_id);
> +	if (!ite)
>  		return E_ITS_MOVI_UNMAPPED_INTERRUPT;
>  
> -	if (!its_is_collection_mapped(itte->collection))
> +	if (!its_is_collection_mapped(ite->collection))
>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>  
>  	collection = find_collection(its, coll_id);
>  	if (!its_is_collection_mapped(collection))
>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>  
> -	itte->collection = collection;
> +	ite->collection = collection;
>  	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>  
> -	spin_lock(&itte->irq->irq_lock);
> -	itte->irq->target_vcpu = vcpu;
> -	spin_unlock(&itte->irq->irq_lock);
> +	spin_lock(&ite->irq->irq_lock);
> +	ite->irq->target_vcpu = vcpu;
> +	spin_unlock(&ite->irq->irq_lock);
>  
>  	return 0;
>  }
> @@ -679,7 +679,7 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>  {
>  	struct its_collection *collection;
>  	struct its_device *device;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  	/*
>  	 * Clearing the mapping for that collection ID removes the
> @@ -690,10 +690,10 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>  	if (!collection)
>  		return;
>  
> -	for_each_lpi_its(device, itte, its)
> -		if (itte->collection &&
> -		    itte->collection->collection_id == coll_id)
> -			itte->collection = NULL;
> +	for_each_lpi_its(device, ite, its)
> +		if (ite->collection &&
> +		    ite->collection->collection_id == coll_id)
> +			ite->collection = NULL;
>  
>  	list_del(&collection->coll_list);
>  	kfree(collection);
> @@ -709,7 +709,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	u32 event_id = its_cmd_get_id(its_cmd);
>  	u32 coll_id = its_cmd_get_collection(its_cmd);
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  	struct its_device *device;
>  	struct its_collection *collection, *new_coll = NULL;
>  	int lpi_nr;
> @@ -728,7 +728,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  		return E_ITS_MAPTI_PHYSICALID_OOR;
>  
>  	/* If there is an existing mapping, behavior is UNPREDICTABLE. */
> -	if (find_itte(its, device_id, event_id))
> +	if (find_ite(its, device_id, event_id))
>  		return 0;
>  
>  	collection = find_collection(its, coll_id);
> @@ -739,36 +739,36 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  		new_coll = collection;
>  	}
>  
> -	itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
> -	if (!itte) {
> +	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
> +	if (!ite) {
>  		if (new_coll)
>  			vgic_its_free_collection(its, coll_id);
>  		return -ENOMEM;
>  	}
>  
> -	itte->event_id	= event_id;
> -	list_add_tail(&itte->itte_list, &device->itt_head);
> +	ite->event_id	= event_id;
> +	list_add_tail(&ite->ite_list, &device->itt_head);
>  
> -	itte->collection = collection;
> -	itte->lpi = lpi_nr;
> +	ite->collection = collection;
> +	ite->lpi = lpi_nr;
>  
>  	irq = vgic_add_lpi(kvm, lpi_nr);
>  	if (IS_ERR(irq)) {
>  		if (new_coll)
>  			vgic_its_free_collection(its, coll_id);
> -		its_free_itte(kvm, itte);
> +		its_free_ite(kvm, ite);
>  		return PTR_ERR(irq);
>  	}
> -	itte->irq = irq;
> +	ite->irq = irq;
>  
> -	update_affinity_itte(kvm, itte);
> +	update_affinity_ite(kvm, ite);
>  
>  	/*
>  	 * We "cache" the configuration table entries in out struct vgic_irq's.
>  	 * However we only have those structs for mapped IRQs, so we read in
>  	 * the respective config data from memory here upon mapping the LPI.
>  	 */
> -	update_lpi_config(kvm, itte->irq, NULL);
> +	update_lpi_config(kvm, ite->irq, NULL);
>  
>  	return 0;
>  }
> @@ -776,15 +776,15 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  /* Requires the its_lock to be held. */
>  static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
>  {
> -	struct its_itte *itte, *temp;
> +	struct its_ite *ite, *temp;
>  
>  	/*
>  	 * The spec says that unmapping a device with still valid
>  	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
>  	 * since we cannot leave the memory unreferenced.
>  	 */
> -	list_for_each_entry_safe(itte, temp, &device->itt_head, itte_list)
> -		its_free_itte(kvm, itte);
> +	list_for_each_entry_safe(ite, temp, &device->itt_head, ite_list)
> +		its_free_ite(kvm, ite);
>  
>  	list_del(&device->dev_list);
>  	kfree(device);
> @@ -883,14 +883,14 @@ static int vgic_its_cmd_handle_clear(struct kvm *kvm, struct vgic_its *its,
>  {
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	u32 event_id = its_cmd_get_id(its_cmd);
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  
> -	itte = find_itte(its, device_id, event_id);
> -	if (!itte)
> +	ite = find_ite(its, device_id, event_id);
> +	if (!ite)
>  		return E_ITS_CLEAR_UNMAPPED_INTERRUPT;
>  
> -	itte->irq->pending_latch = false;
> +	ite->irq->pending_latch = false;
>  
>  	return 0;
>  }
> @@ -904,14 +904,14 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,
>  {
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	u32 event_id = its_cmd_get_id(its_cmd);
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  
> -	itte = find_itte(its, device_id, event_id);
> -	if (!itte)
> +	ite = find_ite(its, device_id, event_id);
> +	if (!ite)
>  		return E_ITS_INV_UNMAPPED_INTERRUPT;
>  
> -	return update_lpi_config(kvm, itte->irq, NULL);
> +	return update_lpi_config(kvm, ite->irq, NULL);
>  }
>  
>  /*
> @@ -1435,7 +1435,7 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>  	struct kvm *kvm = kvm_dev->kvm;
>  	struct vgic_its *its = kvm_dev->private;
>  	struct its_device *dev;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  	struct list_head *dev_cur, *dev_temp;
>  	struct list_head *cur, *temp;
>  
> @@ -1450,8 +1450,8 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>  	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
>  		dev = container_of(dev_cur, struct its_device, dev_list);
>  		list_for_each_safe(cur, temp, &dev->itt_head) {
> -			itte = (container_of(cur, struct its_itte, itte_list));
> -			its_free_itte(kvm, itte);
> +			ite = (container_of(cur, struct its_ite, ite_list));
> +			its_free_ite(kvm, ite);
>  		}
>  		list_del(dev_cur);
>  		kfree(dev);
> -- 
> 2.5.5
> 

[PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:15PM +0200, Eric Auger wrote:
> The actual abbreviation for the interrupt translation table entry
> is ITE. Let's rename all itte instances by ite.

Is there really any confusion or problems with using itte?  This is a
lot of churn...


Thanks,
-Christoffer

> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
> 
> ---
> 
> v5: Add Marc's A-b
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 148 +++++++++++++++++++++----------------------
>  1 file changed, 74 insertions(+), 74 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 8d1da1a..3ffcbbe 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -114,8 +114,8 @@ struct its_collection {
>  #define its_is_collection_mapped(coll) ((coll) && \
>  				((coll)->target_addr != COLLECTION_NOT_MAPPED))
>  
> -struct its_itte {
> -	struct list_head itte_list;
> +struct its_ite {
> +	struct list_head ite_list;
>  
>  	struct vgic_irq *irq;
>  	struct its_collection *collection;
> @@ -143,27 +143,27 @@ static struct its_device *find_its_device(struct vgic_its *its, u32 device_id)
>   * Device ID/Event ID pair on an ITS.
>   * Must be called with the its_lock mutex held.
>   */
> -static struct its_itte *find_itte(struct vgic_its *its, u32 device_id,
> +static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  				  u32 event_id)
>  {
>  	struct its_device *device;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  	device = find_its_device(its, device_id);
>  	if (device == NULL)
>  		return NULL;
>  
> -	list_for_each_entry(itte, &device->itt_head, itte_list)
> -		if (itte->event_id == event_id)
> -			return itte;
> +	list_for_each_entry(ite, &device->itt_head, ite_list)
> +		if (ite->event_id == event_id)
> +			return ite;
>  
>  	return NULL;
>  }
>  
>  /* To be used as an iterator this macro misses the enclosing parentheses */
> -#define for_each_lpi_its(dev, itte, its) \
> +#define for_each_lpi_its(dev, ite, its) \
>  	list_for_each_entry(dev, &(its)->device_list, dev_list) \
> -		list_for_each_entry(itte, &(dev)->itt_head, itte_list)
> +		list_for_each_entry(ite, &(dev)->itt_head, ite_list)
>  
>  /*
>   * We only implement 48 bits of PA at the moment, although the ITS
> @@ -270,18 +270,18 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>   * Needs to be called whenever either the collection for a LPIs has
>   * changed or the collection itself got retargeted.
>   */
> -static void update_affinity_itte(struct kvm *kvm, struct its_itte *itte)
> +static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
>  {
>  	struct kvm_vcpu *vcpu;
>  
> -	if (!its_is_collection_mapped(itte->collection))
> +	if (!its_is_collection_mapped(ite->collection))
>  		return;
>  
> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>  
> -	spin_lock(&itte->irq->irq_lock);
> -	itte->irq->target_vcpu = vcpu;
> -	spin_unlock(&itte->irq->irq_lock);
> +	spin_lock(&ite->irq->irq_lock);
> +	ite->irq->target_vcpu = vcpu;
> +	spin_unlock(&ite->irq->irq_lock);
>  }
>  
>  /*
> @@ -292,13 +292,13 @@ static void update_affinity_collection(struct kvm *kvm, struct vgic_its *its,
>  				       struct its_collection *coll)
>  {
>  	struct its_device *device;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
> -	for_each_lpi_its(device, itte, its) {
> -		if (!itte->collection || coll != itte->collection)
> +	for_each_lpi_its(device, ite, its) {
> +		if (!ite->collection || coll != ite->collection)
>  			continue;
>  
> -		update_affinity_itte(kvm, itte);
> +		update_affinity_ite(kvm, ite);
>  	}
>  }
>  
> @@ -425,25 +425,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
>  				u32 devid, u32 eventid)
>  {
>  	struct kvm_vcpu *vcpu;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  	if (!its->enabled)
>  		return -EBUSY;
>  
> -	itte = find_itte(its, devid, eventid);
> -	if (!itte || !its_is_collection_mapped(itte->collection))
> +	ite = find_ite(its, devid, eventid);
> +	if (!ite || !its_is_collection_mapped(ite->collection))
>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>  
> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>  	if (!vcpu)
>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>  
>  	if (!vcpu->arch.vgic_cpu.lpis_enabled)
>  		return -EBUSY;
>  
> -	spin_lock(&itte->irq->irq_lock);
> -	itte->irq->pending_latch = true;
> -	vgic_queue_irq_unlock(kvm, itte->irq);
> +	spin_lock(&ite->irq->irq_lock);
> +	ite->irq->pending_latch = true;
> +	vgic_queue_irq_unlock(kvm, ite->irq);
>  
>  	return 0;
>  }
> @@ -511,15 +511,15 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
>  }
>  
>  /* Requires the its_lock to be held. */
> -static void its_free_itte(struct kvm *kvm, struct its_itte *itte)
> +static void its_free_ite(struct kvm *kvm, struct its_ite *ite)
>  {
> -	list_del(&itte->itte_list);
> +	list_del(&ite->ite_list);
>  
>  	/* This put matches the get in vgic_add_lpi. */
> -	if (itte->irq)
> -		vgic_put_irq(kvm, itte->irq);
> +	if (ite->irq)
> +		vgic_put_irq(kvm, ite->irq);
>  
> -	kfree(itte);
> +	kfree(ite);
>  }
>  
>  static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
> @@ -544,17 +544,17 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
>  {
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	u32 event_id = its_cmd_get_id(its_cmd);
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  
> -	itte = find_itte(its, device_id, event_id);
> -	if (itte && itte->collection) {
> +	ite = find_ite(its, device_id, event_id);
> +	if (ite && ite->collection) {
>  		/*
>  		 * Though the spec talks about removing the pending state, we
>  		 * don't bother here since we clear the ITTE anyway and the
>  		 * pending state is a property of the ITTE struct.
>  		 */
> -		its_free_itte(kvm, itte);
> +		its_free_ite(kvm, ite);
>  		return 0;
>  	}
>  
> @@ -572,26 +572,26 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>  	u32 event_id = its_cmd_get_id(its_cmd);
>  	u32 coll_id = its_cmd_get_collection(its_cmd);
>  	struct kvm_vcpu *vcpu;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  	struct its_collection *collection;
>  
> -	itte = find_itte(its, device_id, event_id);
> -	if (!itte)
> +	ite = find_ite(its, device_id, event_id);
> +	if (!ite)
>  		return E_ITS_MOVI_UNMAPPED_INTERRUPT;
>  
> -	if (!its_is_collection_mapped(itte->collection))
> +	if (!its_is_collection_mapped(ite->collection))
>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>  
>  	collection = find_collection(its, coll_id);
>  	if (!its_is_collection_mapped(collection))
>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>  
> -	itte->collection = collection;
> +	ite->collection = collection;
>  	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>  
> -	spin_lock(&itte->irq->irq_lock);
> -	itte->irq->target_vcpu = vcpu;
> -	spin_unlock(&itte->irq->irq_lock);
> +	spin_lock(&ite->irq->irq_lock);
> +	ite->irq->target_vcpu = vcpu;
> +	spin_unlock(&ite->irq->irq_lock);
>  
>  	return 0;
>  }
> @@ -679,7 +679,7 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>  {
>  	struct its_collection *collection;
>  	struct its_device *device;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  	/*
>  	 * Clearing the mapping for that collection ID removes the
> @@ -690,10 +690,10 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>  	if (!collection)
>  		return;
>  
> -	for_each_lpi_its(device, itte, its)
> -		if (itte->collection &&
> -		    itte->collection->collection_id == coll_id)
> -			itte->collection = NULL;
> +	for_each_lpi_its(device, ite, its)
> +		if (ite->collection &&
> +		    ite->collection->collection_id == coll_id)
> +			ite->collection = NULL;
>  
>  	list_del(&collection->coll_list);
>  	kfree(collection);
> @@ -709,7 +709,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	u32 event_id = its_cmd_get_id(its_cmd);
>  	u32 coll_id = its_cmd_get_collection(its_cmd);
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  	struct its_device *device;
>  	struct its_collection *collection, *new_coll = NULL;
>  	int lpi_nr;
> @@ -728,7 +728,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  		return E_ITS_MAPTI_PHYSICALID_OOR;
>  
>  	/* If there is an existing mapping, behavior is UNPREDICTABLE. */
> -	if (find_itte(its, device_id, event_id))
> +	if (find_ite(its, device_id, event_id))
>  		return 0;
>  
>  	collection = find_collection(its, coll_id);
> @@ -739,36 +739,36 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  		new_coll = collection;
>  	}
>  
> -	itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
> -	if (!itte) {
> +	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
> +	if (!ite) {
>  		if (new_coll)
>  			vgic_its_free_collection(its, coll_id);
>  		return -ENOMEM;
>  	}
>  
> -	itte->event_id	= event_id;
> -	list_add_tail(&itte->itte_list, &device->itt_head);
> +	ite->event_id	= event_id;
> +	list_add_tail(&ite->ite_list, &device->itt_head);
>  
> -	itte->collection = collection;
> -	itte->lpi = lpi_nr;
> +	ite->collection = collection;
> +	ite->lpi = lpi_nr;
>  
>  	irq = vgic_add_lpi(kvm, lpi_nr);
>  	if (IS_ERR(irq)) {
>  		if (new_coll)
>  			vgic_its_free_collection(its, coll_id);
> -		its_free_itte(kvm, itte);
> +		its_free_ite(kvm, ite);
>  		return PTR_ERR(irq);
>  	}
> -	itte->irq = irq;
> +	ite->irq = irq;
>  
> -	update_affinity_itte(kvm, itte);
> +	update_affinity_ite(kvm, ite);
>  
>  	/*
>  	 * We "cache" the configuration table entries in out struct vgic_irq's.
>  	 * However we only have those structs for mapped IRQs, so we read in
>  	 * the respective config data from memory here upon mapping the LPI.
>  	 */
> -	update_lpi_config(kvm, itte->irq, NULL);
> +	update_lpi_config(kvm, ite->irq, NULL);
>  
>  	return 0;
>  }
> @@ -776,15 +776,15 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  /* Requires the its_lock to be held. */
>  static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
>  {
> -	struct its_itte *itte, *temp;
> +	struct its_ite *ite, *temp;
>  
>  	/*
>  	 * The spec says that unmapping a device with still valid
>  	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
>  	 * since we cannot leave the memory unreferenced.
>  	 */
> -	list_for_each_entry_safe(itte, temp, &device->itt_head, itte_list)
> -		its_free_itte(kvm, itte);
> +	list_for_each_entry_safe(ite, temp, &device->itt_head, ite_list)
> +		its_free_ite(kvm, ite);
>  
>  	list_del(&device->dev_list);
>  	kfree(device);
> @@ -883,14 +883,14 @@ static int vgic_its_cmd_handle_clear(struct kvm *kvm, struct vgic_its *its,
>  {
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	u32 event_id = its_cmd_get_id(its_cmd);
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  
> -	itte = find_itte(its, device_id, event_id);
> -	if (!itte)
> +	ite = find_ite(its, device_id, event_id);
> +	if (!ite)
>  		return E_ITS_CLEAR_UNMAPPED_INTERRUPT;
>  
> -	itte->irq->pending_latch = false;
> +	ite->irq->pending_latch = false;
>  
>  	return 0;
>  }
> @@ -904,14 +904,14 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,
>  {
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	u32 event_id = its_cmd_get_id(its_cmd);
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  
>  
> -	itte = find_itte(its, device_id, event_id);
> -	if (!itte)
> +	ite = find_ite(its, device_id, event_id);
> +	if (!ite)
>  		return E_ITS_INV_UNMAPPED_INTERRUPT;
>  
> -	return update_lpi_config(kvm, itte->irq, NULL);
> +	return update_lpi_config(kvm, ite->irq, NULL);
>  }
>  
>  /*
> @@ -1435,7 +1435,7 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>  	struct kvm *kvm = kvm_dev->kvm;
>  	struct vgic_its *its = kvm_dev->private;
>  	struct its_device *dev;
> -	struct its_itte *itte;
> +	struct its_ite *ite;
>  	struct list_head *dev_cur, *dev_temp;
>  	struct list_head *cur, *temp;
>  
> @@ -1450,8 +1450,8 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>  	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
>  		dev = container_of(dev_cur, struct its_device, dev_list);
>  		list_for_each_safe(cur, temp, &dev->itt_head) {
> -			itte = (container_of(cur, struct its_itte, itte_list));
> -			its_free_itte(kvm, itte);
> +			ite = (container_of(cur, struct its_ite, ite_list));
> +			its_free_ite(kvm, ite);
>  		}
>  		list_del(dev_cur);
>  		kfree(dev);
> -- 
> 2.5.5
> 

Re: [PATCH v5 04/22] arm/arm64: vgic: turn vgic_find_mmio_region into public

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:16PM +0200, Eric Auger wrote:
> We plan to use vgic_find_mmio_region in vgic-its.c so let's
> turn it into a public function.
> 
> Also let's take the opportunity to rename the region parameter
> into regions to emphasize this latter is an array of regions.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Andre Przywara <andre.przywara@arm.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
> 
> ---
> v4 -> v5:
> - add Marc's A-b
> 
> v3 -> v4:
> - rename region parameter into regions
> - add Andre's R-b
> ---
>  virt/kvm/arm/vgic/vgic-mmio.c | 11 +++++------
>  virt/kvm/arm/vgic/vgic-mmio.h |  5 +++++
>  2 files changed, 10 insertions(+), 6 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-mmio.c b/virt/kvm/arm/vgic/vgic-mmio.c
> index 2a5db13..1c17b2a 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio.c
> +++ b/virt/kvm/arm/vgic/vgic-mmio.c
> @@ -446,13 +446,12 @@ static int match_region(const void *key, const void *elt)
>  	return 0;
>  }
>  
> -/* Find the proper register handler entry given a certain address offset. */
> -static const struct vgic_register_region *
> -vgic_find_mmio_region(const struct vgic_register_region *region, int nr_regions,
> -		      unsigned int offset)
> +const struct vgic_register_region *
> +vgic_find_mmio_region(const struct vgic_register_region *regions,
> +		      int nr_regions, unsigned int offset)
>  {
> -	return bsearch((void *)(uintptr_t)offset, region, nr_regions,
> -		       sizeof(region[0]), match_region);
> +	return bsearch((void *)(uintptr_t)offset, regions, nr_regions,
> +		       sizeof(regions[0]), match_region);
>  }
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr)
> diff --git a/virt/kvm/arm/vgic/vgic-mmio.h b/virt/kvm/arm/vgic/vgic-mmio.h
> index 98bb566..6eec91b 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio.h
> +++ b/virt/kvm/arm/vgic/vgic-mmio.h
> @@ -192,4 +192,9 @@ u64 vgic_sanitise_shareability(u64 reg);
>  u64 vgic_sanitise_field(u64 reg, u64 field_mask, int field_shift,
>  			u64 (*sanitise_fn)(u64));
>  
> +/* Find the proper register handler entry given a certain address offset */


nit: we don't usually comment the functions in the prototype in the
header files but keep comments with the function implementation.

But I don't care deeply about this.

Acked-by: Christoffer Dall <cdall@linaro.org>

> +const struct vgic_register_region *
> +vgic_find_mmio_region(const struct vgic_register_region *regions,
> +		      int nr_regions, unsigned int offset);
> +
>  #endif
> -- 
> 2.5.5
> 

[PATCH v5 04/22] arm/arm64: vgic: turn vgic_find_mmio_region into public

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:16PM +0200, Eric Auger wrote:
> We plan to use vgic_find_mmio_region in vgic-its.c so let's
> turn it into a public function.
> 
> Also let's take the opportunity to rename the region parameter
> into regions to emphasize this latter is an array of regions.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Andre Przywara <andre.przywara@arm.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
> 
> ---
> v4 -> v5:
> - add Marc's A-b
> 
> v3 -> v4:
> - rename region parameter into regions
> - add Andre's R-b
> ---
>  virt/kvm/arm/vgic/vgic-mmio.c | 11 +++++------
>  virt/kvm/arm/vgic/vgic-mmio.h |  5 +++++
>  2 files changed, 10 insertions(+), 6 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-mmio.c b/virt/kvm/arm/vgic/vgic-mmio.c
> index 2a5db13..1c17b2a 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio.c
> +++ b/virt/kvm/arm/vgic/vgic-mmio.c
> @@ -446,13 +446,12 @@ static int match_region(const void *key, const void *elt)
>  	return 0;
>  }
>  
> -/* Find the proper register handler entry given a certain address offset. */
> -static const struct vgic_register_region *
> -vgic_find_mmio_region(const struct vgic_register_region *region, int nr_regions,
> -		      unsigned int offset)
> +const struct vgic_register_region *
> +vgic_find_mmio_region(const struct vgic_register_region *regions,
> +		      int nr_regions, unsigned int offset)
>  {
> -	return bsearch((void *)(uintptr_t)offset, region, nr_regions,
> -		       sizeof(region[0]), match_region);
> +	return bsearch((void *)(uintptr_t)offset, regions, nr_regions,
> +		       sizeof(regions[0]), match_region);
>  }
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr)
> diff --git a/virt/kvm/arm/vgic/vgic-mmio.h b/virt/kvm/arm/vgic/vgic-mmio.h
> index 98bb566..6eec91b 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio.h
> +++ b/virt/kvm/arm/vgic/vgic-mmio.h
> @@ -192,4 +192,9 @@ u64 vgic_sanitise_shareability(u64 reg);
>  u64 vgic_sanitise_field(u64 reg, u64 field_mask, int field_shift,
>  			u64 (*sanitise_fn)(u64));
>  
> +/* Find the proper register handler entry given a certain address offset */


nit: we don't usually comment the functions in the prototype in the
header files but keep comments with the function implementation.

But I don't care deeply about this.

Acked-by: Christoffer Dall <cdall@linaro.org>

> +const struct vgic_register_region *
> +vgic_find_mmio_region(const struct vgic_register_region *regions,
> +		      int nr_regions, unsigned int offset);
> +
>  #endif
> -- 
> 2.5.5
> 

Re: [PATCH v5 05/22] KVM: arm64: vgic-its: KVM_DEV_ARM_VGIC_GRP_ITS_REGS group

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:17PM +0200, Eric Auger wrote:
> The ITS KVM device exposes a new KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> group which allows the userspace to save/restore ITS registers.
> 
> At this stage the get/set/has operations are not yet implemented.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Andre Przywara <andre.przywara@arm.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>

Reviewed-by: Christoffer Dall <cdall@linaro.org>

> 
> ---
> v4 -> v5:
> - Add Marc's A-b
> 
> v3 -> v4:
> - added Andre's R-b
> ---
>  arch/arm/include/uapi/asm/kvm.h   |  1 +
>  arch/arm64/include/uapi/asm/kvm.h |  1 +
>  virt/kvm/arm/vgic/vgic-its.c      | 36 +++++++++++++++++++++++++++++++++++-
>  3 files changed, 37 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
> index 6ebd3e6..4beb83b 100644
> --- a/arch/arm/include/uapi/asm/kvm.h
> +++ b/arch/arm/include/uapi/asm/kvm.h
> @@ -192,6 +192,7 @@ struct kvm_arch_memory_slot {
>  #define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
>  #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
>  #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO  7
> +#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS	8
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT	10
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
>  			(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index c286035..7e8dd69 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -212,6 +212,7 @@ struct kvm_arch_memory_slot {
>  #define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
>  #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
>  #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO  7
> +#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT	10
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
>  			(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 3ffcbbe..f687e91 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1466,6 +1466,19 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>  	kfree(its);
>  }
>  
> +int vgic_its_has_attr_regs(struct kvm_device *dev,
> +			   struct kvm_device_attr *attr)
> +{
> +	return -ENXIO;
> +}
> +
> +int vgic_its_attr_regs_access(struct kvm_device *dev,
> +			      struct kvm_device_attr *attr,
> +			      u64 *reg, bool is_write)
> +{
> +	return -ENXIO;
> +}
> +
>  static int vgic_its_has_attr(struct kvm_device *dev,
>  			     struct kvm_device_attr *attr)
>  {
> @@ -1482,6 +1495,8 @@ static int vgic_its_has_attr(struct kvm_device *dev,
>  			return 0;
>  		}
>  		break;
> +	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS:
> +		return vgic_its_has_attr_regs(dev, attr);
>  	}
>  	return -ENXIO;
>  }
> @@ -1521,6 +1536,15 @@ static int vgic_its_set_attr(struct kvm_device *dev,
>  			return 0;
>  		}
>  		break;
> +	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
> +		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
> +		u64 reg;
> +
> +		if (get_user(reg, uaddr))
> +			return -EFAULT;
> +
> +		return vgic_its_attr_regs_access(dev, attr, &reg, true);
> +	}
>  	}
>  	return -ENXIO;
>  }
> @@ -1541,10 +1565,20 @@ static int vgic_its_get_attr(struct kvm_device *dev,
>  		if (copy_to_user(uaddr, &addr, sizeof(addr)))
>  			return -EFAULT;
>  		break;
> +	}
> +	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
> +		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
> +		u64 reg;
> +		int ret;
> +
> +		ret = vgic_its_attr_regs_access(dev, attr, &reg, false);
> +		if (ret)
> +			return ret;
> +		return put_user(reg, uaddr);
> +	}
>  	default:
>  		return -ENXIO;
>  	}
> -	}
>  
>  	return 0;
>  }
> -- 
> 2.5.5
> 

[PATCH v5 05/22] KVM: arm64: vgic-its: KVM_DEV_ARM_VGIC_GRP_ITS_REGS group

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:17PM +0200, Eric Auger wrote:
> The ITS KVM device exposes a new KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> group which allows the userspace to save/restore ITS registers.
> 
> At this stage the get/set/has operations are not yet implemented.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Reviewed-by: Andre Przywara <andre.przywara@arm.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>

Reviewed-by: Christoffer Dall <cdall@linaro.org>

> 
> ---
> v4 -> v5:
> - Add Marc's A-b
> 
> v3 -> v4:
> - added Andre's R-b
> ---
>  arch/arm/include/uapi/asm/kvm.h   |  1 +
>  arch/arm64/include/uapi/asm/kvm.h |  1 +
>  virt/kvm/arm/vgic/vgic-its.c      | 36 +++++++++++++++++++++++++++++++++++-
>  3 files changed, 37 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
> index 6ebd3e6..4beb83b 100644
> --- a/arch/arm/include/uapi/asm/kvm.h
> +++ b/arch/arm/include/uapi/asm/kvm.h
> @@ -192,6 +192,7 @@ struct kvm_arch_memory_slot {
>  #define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
>  #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
>  #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO  7
> +#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS	8
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT	10
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
>  			(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index c286035..7e8dd69 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -212,6 +212,7 @@ struct kvm_arch_memory_slot {
>  #define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
>  #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
>  #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO  7
> +#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT	10
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
>  			(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 3ffcbbe..f687e91 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1466,6 +1466,19 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>  	kfree(its);
>  }
>  
> +int vgic_its_has_attr_regs(struct kvm_device *dev,
> +			   struct kvm_device_attr *attr)
> +{
> +	return -ENXIO;
> +}
> +
> +int vgic_its_attr_regs_access(struct kvm_device *dev,
> +			      struct kvm_device_attr *attr,
> +			      u64 *reg, bool is_write)
> +{
> +	return -ENXIO;
> +}
> +
>  static int vgic_its_has_attr(struct kvm_device *dev,
>  			     struct kvm_device_attr *attr)
>  {
> @@ -1482,6 +1495,8 @@ static int vgic_its_has_attr(struct kvm_device *dev,
>  			return 0;
>  		}
>  		break;
> +	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS:
> +		return vgic_its_has_attr_regs(dev, attr);
>  	}
>  	return -ENXIO;
>  }
> @@ -1521,6 +1536,15 @@ static int vgic_its_set_attr(struct kvm_device *dev,
>  			return 0;
>  		}
>  		break;
> +	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
> +		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
> +		u64 reg;
> +
> +		if (get_user(reg, uaddr))
> +			return -EFAULT;
> +
> +		return vgic_its_attr_regs_access(dev, attr, &reg, true);
> +	}
>  	}
>  	return -ENXIO;
>  }
> @@ -1541,10 +1565,20 @@ static int vgic_its_get_attr(struct kvm_device *dev,
>  		if (copy_to_user(uaddr, &addr, sizeof(addr)))
>  			return -EFAULT;
>  		break;
> +	}
> +	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
> +		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
> +		u64 reg;
> +		int ret;
> +
> +		ret = vgic_its_attr_regs_access(dev, attr, &reg, false);
> +		if (ret)
> +			return ret;
> +		return put_user(reg, uaddr);
> +	}
>  	default:
>  		return -ENXIO;
>  	}
> -	}
>  
>  	return 0;
>  }
> -- 
> 2.5.5
> 

Re: [PATCH v5 06/22] KVM: arm/arm64: vgic: expose (un)lock_all_vcpus

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:18PM +0200, Eric Auger wrote:
> We need to use those helpers in vgic-its.c so let's
> expose them in the private vgic header.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>

Acked-by: Christoffer Dall <cdall@linaro.org>

> 
> ---
> 
> v4 -> v5:
> - Add Marc's A-b
> ---
>  virt/kvm/arm/vgic/vgic-kvm-device.c | 4 ++--
>  virt/kvm/arm/vgic/vgic.h            | 3 +++
>  2 files changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c
> index d181d2b..859bfa8 100644
> --- a/virt/kvm/arm/vgic/vgic-kvm-device.c
> +++ b/virt/kvm/arm/vgic/vgic-kvm-device.c
> @@ -259,13 +259,13 @@ static void unlock_vcpus(struct kvm *kvm, int vcpu_lock_idx)
>  	}
>  }
>  
> -static void unlock_all_vcpus(struct kvm *kvm)
> +void unlock_all_vcpus(struct kvm *kvm)
>  {
>  	unlock_vcpus(kvm, atomic_read(&kvm->online_vcpus) - 1);
>  }
>  
>  /* Returns true if all vcpus were locked, false otherwise */
> -static bool lock_all_vcpus(struct kvm *kvm)
> +bool lock_all_vcpus(struct kvm *kvm)
>  {
>  	struct kvm_vcpu *tmp_vcpu;
>  	int c;
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 6cf557e..b87f1c6 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -184,4 +184,7 @@ int vgic_init(struct kvm *kvm);
>  int vgic_debug_init(struct kvm *kvm);
>  int vgic_debug_destroy(struct kvm *kvm);
>  
> +bool lock_all_vcpus(struct kvm *kvm);
> +void unlock_all_vcpus(struct kvm *kvm);
> +
>  #endif
> -- 
> 2.5.5
> 

[PATCH v5 06/22] KVM: arm/arm64: vgic: expose (un)lock_all_vcpus

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:18PM +0200, Eric Auger wrote:
> We need to use those helpers in vgic-its.c so let's
> expose them in the private vgic header.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>

Acked-by: Christoffer Dall <cdall@linaro.org>

> 
> ---
> 
> v4 -> v5:
> - Add Marc's A-b
> ---
>  virt/kvm/arm/vgic/vgic-kvm-device.c | 4 ++--
>  virt/kvm/arm/vgic/vgic.h            | 3 +++
>  2 files changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c
> index d181d2b..859bfa8 100644
> --- a/virt/kvm/arm/vgic/vgic-kvm-device.c
> +++ b/virt/kvm/arm/vgic/vgic-kvm-device.c
> @@ -259,13 +259,13 @@ static void unlock_vcpus(struct kvm *kvm, int vcpu_lock_idx)
>  	}
>  }
>  
> -static void unlock_all_vcpus(struct kvm *kvm)
> +void unlock_all_vcpus(struct kvm *kvm)
>  {
>  	unlock_vcpus(kvm, atomic_read(&kvm->online_vcpus) - 1);
>  }
>  
>  /* Returns true if all vcpus were locked, false otherwise */
> -static bool lock_all_vcpus(struct kvm *kvm)
> +bool lock_all_vcpus(struct kvm *kvm)
>  {
>  	struct kvm_vcpu *tmp_vcpu;
>  	int c;
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 6cf557e..b87f1c6 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -184,4 +184,7 @@ int vgic_init(struct kvm *kvm);
>  int vgic_debug_init(struct kvm *kvm);
>  int vgic_debug_destroy(struct kvm *kvm);
>  
> +bool lock_all_vcpus(struct kvm *kvm);
> +void unlock_all_vcpus(struct kvm *kvm);
> +
>  #endif
> -- 
> 2.5.5
> 

Re: [PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Andre Przywara]

Hi,

On 27/04/17 10:05, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:15PM +0200, Eric Auger wrote:
>> The actual abbreviation for the interrupt translation table entry
>> is ITE. Let's rename all itte instances by ite.
> 
> Is there really any confusion or problems with using itte?  This is a
> lot of churn...

I tend to agree (just didn't dare to mention this before).
I see that the spec speaks of "ITE", but the spelled out term hints more
at ITTE (because it's a "translation table").
Besides three letters tend to be more ambiguous than a four letter
identifier.

Would adding a comment to the structure definition help?

But speaking of churn I am not sure how much more work dropping this
patch now creates on Eric's side ...

Cheers,
Andre.

>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
>>
>> ---
>>
>> v5: Add Marc's A-b
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 148 +++++++++++++++++++++----------------------
>>  1 file changed, 74 insertions(+), 74 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 8d1da1a..3ffcbbe 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -114,8 +114,8 @@ struct its_collection {
>>  #define its_is_collection_mapped(coll) ((coll) && \
>>  				((coll)->target_addr != COLLECTION_NOT_MAPPED))
>>  
>> -struct its_itte {
>> -	struct list_head itte_list;
>> +struct its_ite {
>> +	struct list_head ite_list;
>>  
>>  	struct vgic_irq *irq;
>>  	struct its_collection *collection;
>> @@ -143,27 +143,27 @@ static struct its_device *find_its_device(struct vgic_its *its, u32 device_id)
>>   * Device ID/Event ID pair on an ITS.
>>   * Must be called with the its_lock mutex held.
>>   */
>> -static struct its_itte *find_itte(struct vgic_its *its, u32 device_id,
>> +static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  				  u32 event_id)
>>  {
>>  	struct its_device *device;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  	device = find_its_device(its, device_id);
>>  	if (device == NULL)
>>  		return NULL;
>>  
>> -	list_for_each_entry(itte, &device->itt_head, itte_list)
>> -		if (itte->event_id == event_id)
>> -			return itte;
>> +	list_for_each_entry(ite, &device->itt_head, ite_list)
>> +		if (ite->event_id == event_id)
>> +			return ite;
>>  
>>  	return NULL;
>>  }
>>  
>>  /* To be used as an iterator this macro misses the enclosing parentheses */
>> -#define for_each_lpi_its(dev, itte, its) \
>> +#define for_each_lpi_its(dev, ite, its) \
>>  	list_for_each_entry(dev, &(its)->device_list, dev_list) \
>> -		list_for_each_entry(itte, &(dev)->itt_head, itte_list)
>> +		list_for_each_entry(ite, &(dev)->itt_head, ite_list)
>>  
>>  /*
>>   * We only implement 48 bits of PA at the moment, although the ITS
>> @@ -270,18 +270,18 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>>   * Needs to be called whenever either the collection for a LPIs has
>>   * changed or the collection itself got retargeted.
>>   */
>> -static void update_affinity_itte(struct kvm *kvm, struct its_itte *itte)
>> +static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
>>  {
>>  	struct kvm_vcpu *vcpu;
>>  
>> -	if (!its_is_collection_mapped(itte->collection))
>> +	if (!its_is_collection_mapped(ite->collection))
>>  		return;
>>  
>> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
>> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>>  
>> -	spin_lock(&itte->irq->irq_lock);
>> -	itte->irq->target_vcpu = vcpu;
>> -	spin_unlock(&itte->irq->irq_lock);
>> +	spin_lock(&ite->irq->irq_lock);
>> +	ite->irq->target_vcpu = vcpu;
>> +	spin_unlock(&ite->irq->irq_lock);
>>  }
>>  
>>  /*
>> @@ -292,13 +292,13 @@ static void update_affinity_collection(struct kvm *kvm, struct vgic_its *its,
>>  				       struct its_collection *coll)
>>  {
>>  	struct its_device *device;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>> -	for_each_lpi_its(device, itte, its) {
>> -		if (!itte->collection || coll != itte->collection)
>> +	for_each_lpi_its(device, ite, its) {
>> +		if (!ite->collection || coll != ite->collection)
>>  			continue;
>>  
>> -		update_affinity_itte(kvm, itte);
>> +		update_affinity_ite(kvm, ite);
>>  	}
>>  }
>>  
>> @@ -425,25 +425,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
>>  				u32 devid, u32 eventid)
>>  {
>>  	struct kvm_vcpu *vcpu;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  	if (!its->enabled)
>>  		return -EBUSY;
>>  
>> -	itte = find_itte(its, devid, eventid);
>> -	if (!itte || !its_is_collection_mapped(itte->collection))
>> +	ite = find_ite(its, devid, eventid);
>> +	if (!ite || !its_is_collection_mapped(ite->collection))
>>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>>  
>> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
>> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>>  	if (!vcpu)
>>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>>  
>>  	if (!vcpu->arch.vgic_cpu.lpis_enabled)
>>  		return -EBUSY;
>>  
>> -	spin_lock(&itte->irq->irq_lock);
>> -	itte->irq->pending_latch = true;
>> -	vgic_queue_irq_unlock(kvm, itte->irq);
>> +	spin_lock(&ite->irq->irq_lock);
>> +	ite->irq->pending_latch = true;
>> +	vgic_queue_irq_unlock(kvm, ite->irq);
>>  
>>  	return 0;
>>  }
>> @@ -511,15 +511,15 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
>>  }
>>  
>>  /* Requires the its_lock to be held. */
>> -static void its_free_itte(struct kvm *kvm, struct its_itte *itte)
>> +static void its_free_ite(struct kvm *kvm, struct its_ite *ite)
>>  {
>> -	list_del(&itte->itte_list);
>> +	list_del(&ite->ite_list);
>>  
>>  	/* This put matches the get in vgic_add_lpi. */
>> -	if (itte->irq)
>> -		vgic_put_irq(kvm, itte->irq);
>> +	if (ite->irq)
>> +		vgic_put_irq(kvm, ite->irq);
>>  
>> -	kfree(itte);
>> +	kfree(ite);
>>  }
>>  
>>  static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>> @@ -544,17 +544,17 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
>>  {
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	u32 event_id = its_cmd_get_id(its_cmd);
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  
>> -	itte = find_itte(its, device_id, event_id);
>> -	if (itte && itte->collection) {
>> +	ite = find_ite(its, device_id, event_id);
>> +	if (ite && ite->collection) {
>>  		/*
>>  		 * Though the spec talks about removing the pending state, we
>>  		 * don't bother here since we clear the ITTE anyway and the
>>  		 * pending state is a property of the ITTE struct.
>>  		 */
>> -		its_free_itte(kvm, itte);
>> +		its_free_ite(kvm, ite);
>>  		return 0;
>>  	}
>>  
>> @@ -572,26 +572,26 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>  	u32 coll_id = its_cmd_get_collection(its_cmd);
>>  	struct kvm_vcpu *vcpu;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  	struct its_collection *collection;
>>  
>> -	itte = find_itte(its, device_id, event_id);
>> -	if (!itte)
>> +	ite = find_ite(its, device_id, event_id);
>> +	if (!ite)
>>  		return E_ITS_MOVI_UNMAPPED_INTERRUPT;
>>  
>> -	if (!its_is_collection_mapped(itte->collection))
>> +	if (!its_is_collection_mapped(ite->collection))
>>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>>  
>>  	collection = find_collection(its, coll_id);
>>  	if (!its_is_collection_mapped(collection))
>>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>>  
>> -	itte->collection = collection;
>> +	ite->collection = collection;
>>  	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>>  
>> -	spin_lock(&itte->irq->irq_lock);
>> -	itte->irq->target_vcpu = vcpu;
>> -	spin_unlock(&itte->irq->irq_lock);
>> +	spin_lock(&ite->irq->irq_lock);
>> +	ite->irq->target_vcpu = vcpu;
>> +	spin_unlock(&ite->irq->irq_lock);
>>  
>>  	return 0;
>>  }
>> @@ -679,7 +679,7 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>>  {
>>  	struct its_collection *collection;
>>  	struct its_device *device;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  	/*
>>  	 * Clearing the mapping for that collection ID removes the
>> @@ -690,10 +690,10 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>>  	if (!collection)
>>  		return;
>>  
>> -	for_each_lpi_its(device, itte, its)
>> -		if (itte->collection &&
>> -		    itte->collection->collection_id == coll_id)
>> -			itte->collection = NULL;
>> +	for_each_lpi_its(device, ite, its)
>> +		if (ite->collection &&
>> +		    ite->collection->collection_id == coll_id)
>> +			ite->collection = NULL;
>>  
>>  	list_del(&collection->coll_list);
>>  	kfree(collection);
>> @@ -709,7 +709,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>  	u32 coll_id = its_cmd_get_collection(its_cmd);
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  	struct its_device *device;
>>  	struct its_collection *collection, *new_coll = NULL;
>>  	int lpi_nr;
>> @@ -728,7 +728,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>  		return E_ITS_MAPTI_PHYSICALID_OOR;
>>  
>>  	/* If there is an existing mapping, behavior is UNPREDICTABLE. */
>> -	if (find_itte(its, device_id, event_id))
>> +	if (find_ite(its, device_id, event_id))
>>  		return 0;
>>  
>>  	collection = find_collection(its, coll_id);
>> @@ -739,36 +739,36 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>  		new_coll = collection;
>>  	}
>>  
>> -	itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
>> -	if (!itte) {
>> +	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
>> +	if (!ite) {
>>  		if (new_coll)
>>  			vgic_its_free_collection(its, coll_id);
>>  		return -ENOMEM;
>>  	}
>>  
>> -	itte->event_id	= event_id;
>> -	list_add_tail(&itte->itte_list, &device->itt_head);
>> +	ite->event_id	= event_id;
>> +	list_add_tail(&ite->ite_list, &device->itt_head);
>>  
>> -	itte->collection = collection;
>> -	itte->lpi = lpi_nr;
>> +	ite->collection = collection;
>> +	ite->lpi = lpi_nr;
>>  
>>  	irq = vgic_add_lpi(kvm, lpi_nr);
>>  	if (IS_ERR(irq)) {
>>  		if (new_coll)
>>  			vgic_its_free_collection(its, coll_id);
>> -		its_free_itte(kvm, itte);
>> +		its_free_ite(kvm, ite);
>>  		return PTR_ERR(irq);
>>  	}
>> -	itte->irq = irq;
>> +	ite->irq = irq;
>>  
>> -	update_affinity_itte(kvm, itte);
>> +	update_affinity_ite(kvm, ite);
>>  
>>  	/*
>>  	 * We "cache" the configuration table entries in out struct vgic_irq's.
>>  	 * However we only have those structs for mapped IRQs, so we read in
>>  	 * the respective config data from memory here upon mapping the LPI.
>>  	 */
>> -	update_lpi_config(kvm, itte->irq, NULL);
>> +	update_lpi_config(kvm, ite->irq, NULL);
>>  
>>  	return 0;
>>  }
>> @@ -776,15 +776,15 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>  /* Requires the its_lock to be held. */
>>  static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
>>  {
>> -	struct its_itte *itte, *temp;
>> +	struct its_ite *ite, *temp;
>>  
>>  	/*
>>  	 * The spec says that unmapping a device with still valid
>>  	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
>>  	 * since we cannot leave the memory unreferenced.
>>  	 */
>> -	list_for_each_entry_safe(itte, temp, &device->itt_head, itte_list)
>> -		its_free_itte(kvm, itte);
>> +	list_for_each_entry_safe(ite, temp, &device->itt_head, ite_list)
>> +		its_free_ite(kvm, ite);
>>  
>>  	list_del(&device->dev_list);
>>  	kfree(device);
>> @@ -883,14 +883,14 @@ static int vgic_its_cmd_handle_clear(struct kvm *kvm, struct vgic_its *its,
>>  {
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	u32 event_id = its_cmd_get_id(its_cmd);
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  
>> -	itte = find_itte(its, device_id, event_id);
>> -	if (!itte)
>> +	ite = find_ite(its, device_id, event_id);
>> +	if (!ite)
>>  		return E_ITS_CLEAR_UNMAPPED_INTERRUPT;
>>  
>> -	itte->irq->pending_latch = false;
>> +	ite->irq->pending_latch = false;
>>  
>>  	return 0;
>>  }
>> @@ -904,14 +904,14 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,
>>  {
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	u32 event_id = its_cmd_get_id(its_cmd);
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  
>> -	itte = find_itte(its, device_id, event_id);
>> -	if (!itte)
>> +	ite = find_ite(its, device_id, event_id);
>> +	if (!ite)
>>  		return E_ITS_INV_UNMAPPED_INTERRUPT;
>>  
>> -	return update_lpi_config(kvm, itte->irq, NULL);
>> +	return update_lpi_config(kvm, ite->irq, NULL);
>>  }
>>  
>>  /*
>> @@ -1435,7 +1435,7 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>>  	struct kvm *kvm = kvm_dev->kvm;
>>  	struct vgic_its *its = kvm_dev->private;
>>  	struct its_device *dev;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  	struct list_head *dev_cur, *dev_temp;
>>  	struct list_head *cur, *temp;
>>  
>> @@ -1450,8 +1450,8 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>>  	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
>>  		dev = container_of(dev_cur, struct its_device, dev_list);
>>  		list_for_each_safe(cur, temp, &dev->itt_head) {
>> -			itte = (container_of(cur, struct its_itte, itte_list));
>> -			its_free_itte(kvm, itte);
>> +			ite = (container_of(cur, struct its_ite, ite_list));
>> +			its_free_ite(kvm, ite);
>>  		}
>>  		list_del(dev_cur);
>>  		kfree(dev);
>> -- 
>> 2.5.5
>>

[PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Andre Przywara]

Hi,

On 27/04/17 10:05, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:15PM +0200, Eric Auger wrote:
>> The actual abbreviation for the interrupt translation table entry
>> is ITE. Let's rename all itte instances by ite.
> 
> Is there really any confusion or problems with using itte?  This is a
> lot of churn...

I tend to agree (just didn't dare to mention this before).
I see that the spec speaks of "ITE", but the spelled out term hints more
at ITTE (because it's a "translation table").
Besides three letters tend to be more ambiguous than a four letter
identifier.

Would adding a comment to the structure definition help?

But speaking of churn I am not sure how much more work dropping this
patch now creates on Eric's side ...

Cheers,
Andre.

>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
>>
>> ---
>>
>> v5: Add Marc's A-b
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 148 +++++++++++++++++++++----------------------
>>  1 file changed, 74 insertions(+), 74 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 8d1da1a..3ffcbbe 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -114,8 +114,8 @@ struct its_collection {
>>  #define its_is_collection_mapped(coll) ((coll) && \
>>  				((coll)->target_addr != COLLECTION_NOT_MAPPED))
>>  
>> -struct its_itte {
>> -	struct list_head itte_list;
>> +struct its_ite {
>> +	struct list_head ite_list;
>>  
>>  	struct vgic_irq *irq;
>>  	struct its_collection *collection;
>> @@ -143,27 +143,27 @@ static struct its_device *find_its_device(struct vgic_its *its, u32 device_id)
>>   * Device ID/Event ID pair on an ITS.
>>   * Must be called with the its_lock mutex held.
>>   */
>> -static struct its_itte *find_itte(struct vgic_its *its, u32 device_id,
>> +static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  				  u32 event_id)
>>  {
>>  	struct its_device *device;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  	device = find_its_device(its, device_id);
>>  	if (device == NULL)
>>  		return NULL;
>>  
>> -	list_for_each_entry(itte, &device->itt_head, itte_list)
>> -		if (itte->event_id == event_id)
>> -			return itte;
>> +	list_for_each_entry(ite, &device->itt_head, ite_list)
>> +		if (ite->event_id == event_id)
>> +			return ite;
>>  
>>  	return NULL;
>>  }
>>  
>>  /* To be used as an iterator this macro misses the enclosing parentheses */
>> -#define for_each_lpi_its(dev, itte, its) \
>> +#define for_each_lpi_its(dev, ite, its) \
>>  	list_for_each_entry(dev, &(its)->device_list, dev_list) \
>> -		list_for_each_entry(itte, &(dev)->itt_head, itte_list)
>> +		list_for_each_entry(ite, &(dev)->itt_head, ite_list)
>>  
>>  /*
>>   * We only implement 48 bits of PA at the moment, although the ITS
>> @@ -270,18 +270,18 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>>   * Needs to be called whenever either the collection for a LPIs has
>>   * changed or the collection itself got retargeted.
>>   */
>> -static void update_affinity_itte(struct kvm *kvm, struct its_itte *itte)
>> +static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
>>  {
>>  	struct kvm_vcpu *vcpu;
>>  
>> -	if (!its_is_collection_mapped(itte->collection))
>> +	if (!its_is_collection_mapped(ite->collection))
>>  		return;
>>  
>> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
>> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>>  
>> -	spin_lock(&itte->irq->irq_lock);
>> -	itte->irq->target_vcpu = vcpu;
>> -	spin_unlock(&itte->irq->irq_lock);
>> +	spin_lock(&ite->irq->irq_lock);
>> +	ite->irq->target_vcpu = vcpu;
>> +	spin_unlock(&ite->irq->irq_lock);
>>  }
>>  
>>  /*
>> @@ -292,13 +292,13 @@ static void update_affinity_collection(struct kvm *kvm, struct vgic_its *its,
>>  				       struct its_collection *coll)
>>  {
>>  	struct its_device *device;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>> -	for_each_lpi_its(device, itte, its) {
>> -		if (!itte->collection || coll != itte->collection)
>> +	for_each_lpi_its(device, ite, its) {
>> +		if (!ite->collection || coll != ite->collection)
>>  			continue;
>>  
>> -		update_affinity_itte(kvm, itte);
>> +		update_affinity_ite(kvm, ite);
>>  	}
>>  }
>>  
>> @@ -425,25 +425,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
>>  				u32 devid, u32 eventid)
>>  {
>>  	struct kvm_vcpu *vcpu;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  	if (!its->enabled)
>>  		return -EBUSY;
>>  
>> -	itte = find_itte(its, devid, eventid);
>> -	if (!itte || !its_is_collection_mapped(itte->collection))
>> +	ite = find_ite(its, devid, eventid);
>> +	if (!ite || !its_is_collection_mapped(ite->collection))
>>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>>  
>> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
>> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>>  	if (!vcpu)
>>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>>  
>>  	if (!vcpu->arch.vgic_cpu.lpis_enabled)
>>  		return -EBUSY;
>>  
>> -	spin_lock(&itte->irq->irq_lock);
>> -	itte->irq->pending_latch = true;
>> -	vgic_queue_irq_unlock(kvm, itte->irq);
>> +	spin_lock(&ite->irq->irq_lock);
>> +	ite->irq->pending_latch = true;
>> +	vgic_queue_irq_unlock(kvm, ite->irq);
>>  
>>  	return 0;
>>  }
>> @@ -511,15 +511,15 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
>>  }
>>  
>>  /* Requires the its_lock to be held. */
>> -static void its_free_itte(struct kvm *kvm, struct its_itte *itte)
>> +static void its_free_ite(struct kvm *kvm, struct its_ite *ite)
>>  {
>> -	list_del(&itte->itte_list);
>> +	list_del(&ite->ite_list);
>>  
>>  	/* This put matches the get in vgic_add_lpi. */
>> -	if (itte->irq)
>> -		vgic_put_irq(kvm, itte->irq);
>> +	if (ite->irq)
>> +		vgic_put_irq(kvm, ite->irq);
>>  
>> -	kfree(itte);
>> +	kfree(ite);
>>  }
>>  
>>  static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>> @@ -544,17 +544,17 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
>>  {
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	u32 event_id = its_cmd_get_id(its_cmd);
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  
>> -	itte = find_itte(its, device_id, event_id);
>> -	if (itte && itte->collection) {
>> +	ite = find_ite(its, device_id, event_id);
>> +	if (ite && ite->collection) {
>>  		/*
>>  		 * Though the spec talks about removing the pending state, we
>>  		 * don't bother here since we clear the ITTE anyway and the
>>  		 * pending state is a property of the ITTE struct.
>>  		 */
>> -		its_free_itte(kvm, itte);
>> +		its_free_ite(kvm, ite);
>>  		return 0;
>>  	}
>>  
>> @@ -572,26 +572,26 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>  	u32 coll_id = its_cmd_get_collection(its_cmd);
>>  	struct kvm_vcpu *vcpu;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  	struct its_collection *collection;
>>  
>> -	itte = find_itte(its, device_id, event_id);
>> -	if (!itte)
>> +	ite = find_ite(its, device_id, event_id);
>> +	if (!ite)
>>  		return E_ITS_MOVI_UNMAPPED_INTERRUPT;
>>  
>> -	if (!its_is_collection_mapped(itte->collection))
>> +	if (!its_is_collection_mapped(ite->collection))
>>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>>  
>>  	collection = find_collection(its, coll_id);
>>  	if (!its_is_collection_mapped(collection))
>>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>>  
>> -	itte->collection = collection;
>> +	ite->collection = collection;
>>  	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>>  
>> -	spin_lock(&itte->irq->irq_lock);
>> -	itte->irq->target_vcpu = vcpu;
>> -	spin_unlock(&itte->irq->irq_lock);
>> +	spin_lock(&ite->irq->irq_lock);
>> +	ite->irq->target_vcpu = vcpu;
>> +	spin_unlock(&ite->irq->irq_lock);
>>  
>>  	return 0;
>>  }
>> @@ -679,7 +679,7 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>>  {
>>  	struct its_collection *collection;
>>  	struct its_device *device;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  	/*
>>  	 * Clearing the mapping for that collection ID removes the
>> @@ -690,10 +690,10 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>>  	if (!collection)
>>  		return;
>>  
>> -	for_each_lpi_its(device, itte, its)
>> -		if (itte->collection &&
>> -		    itte->collection->collection_id == coll_id)
>> -			itte->collection = NULL;
>> +	for_each_lpi_its(device, ite, its)
>> +		if (ite->collection &&
>> +		    ite->collection->collection_id == coll_id)
>> +			ite->collection = NULL;
>>  
>>  	list_del(&collection->coll_list);
>>  	kfree(collection);
>> @@ -709,7 +709,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>  	u32 coll_id = its_cmd_get_collection(its_cmd);
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  	struct its_device *device;
>>  	struct its_collection *collection, *new_coll = NULL;
>>  	int lpi_nr;
>> @@ -728,7 +728,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>  		return E_ITS_MAPTI_PHYSICALID_OOR;
>>  
>>  	/* If there is an existing mapping, behavior is UNPREDICTABLE. */
>> -	if (find_itte(its, device_id, event_id))
>> +	if (find_ite(its, device_id, event_id))
>>  		return 0;
>>  
>>  	collection = find_collection(its, coll_id);
>> @@ -739,36 +739,36 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>  		new_coll = collection;
>>  	}
>>  
>> -	itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
>> -	if (!itte) {
>> +	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
>> +	if (!ite) {
>>  		if (new_coll)
>>  			vgic_its_free_collection(its, coll_id);
>>  		return -ENOMEM;
>>  	}
>>  
>> -	itte->event_id	= event_id;
>> -	list_add_tail(&itte->itte_list, &device->itt_head);
>> +	ite->event_id	= event_id;
>> +	list_add_tail(&ite->ite_list, &device->itt_head);
>>  
>> -	itte->collection = collection;
>> -	itte->lpi = lpi_nr;
>> +	ite->collection = collection;
>> +	ite->lpi = lpi_nr;
>>  
>>  	irq = vgic_add_lpi(kvm, lpi_nr);
>>  	if (IS_ERR(irq)) {
>>  		if (new_coll)
>>  			vgic_its_free_collection(its, coll_id);
>> -		its_free_itte(kvm, itte);
>> +		its_free_ite(kvm, ite);
>>  		return PTR_ERR(irq);
>>  	}
>> -	itte->irq = irq;
>> +	ite->irq = irq;
>>  
>> -	update_affinity_itte(kvm, itte);
>> +	update_affinity_ite(kvm, ite);
>>  
>>  	/*
>>  	 * We "cache" the configuration table entries in out struct vgic_irq's.
>>  	 * However we only have those structs for mapped IRQs, so we read in
>>  	 * the respective config data from memory here upon mapping the LPI.
>>  	 */
>> -	update_lpi_config(kvm, itte->irq, NULL);
>> +	update_lpi_config(kvm, ite->irq, NULL);
>>  
>>  	return 0;
>>  }
>> @@ -776,15 +776,15 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>  /* Requires the its_lock to be held. */
>>  static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
>>  {
>> -	struct its_itte *itte, *temp;
>> +	struct its_ite *ite, *temp;
>>  
>>  	/*
>>  	 * The spec says that unmapping a device with still valid
>>  	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
>>  	 * since we cannot leave the memory unreferenced.
>>  	 */
>> -	list_for_each_entry_safe(itte, temp, &device->itt_head, itte_list)
>> -		its_free_itte(kvm, itte);
>> +	list_for_each_entry_safe(ite, temp, &device->itt_head, ite_list)
>> +		its_free_ite(kvm, ite);
>>  
>>  	list_del(&device->dev_list);
>>  	kfree(device);
>> @@ -883,14 +883,14 @@ static int vgic_its_cmd_handle_clear(struct kvm *kvm, struct vgic_its *its,
>>  {
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	u32 event_id = its_cmd_get_id(its_cmd);
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  
>> -	itte = find_itte(its, device_id, event_id);
>> -	if (!itte)
>> +	ite = find_ite(its, device_id, event_id);
>> +	if (!ite)
>>  		return E_ITS_CLEAR_UNMAPPED_INTERRUPT;
>>  
>> -	itte->irq->pending_latch = false;
>> +	ite->irq->pending_latch = false;
>>  
>>  	return 0;
>>  }
>> @@ -904,14 +904,14 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,
>>  {
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	u32 event_id = its_cmd_get_id(its_cmd);
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  
>>  
>> -	itte = find_itte(its, device_id, event_id);
>> -	if (!itte)
>> +	ite = find_ite(its, device_id, event_id);
>> +	if (!ite)
>>  		return E_ITS_INV_UNMAPPED_INTERRUPT;
>>  
>> -	return update_lpi_config(kvm, itte->irq, NULL);
>> +	return update_lpi_config(kvm, ite->irq, NULL);
>>  }
>>  
>>  /*
>> @@ -1435,7 +1435,7 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>>  	struct kvm *kvm = kvm_dev->kvm;
>>  	struct vgic_its *its = kvm_dev->private;
>>  	struct its_device *dev;
>> -	struct its_itte *itte;
>> +	struct its_ite *ite;
>>  	struct list_head *dev_cur, *dev_temp;
>>  	struct list_head *cur, *temp;
>>  
>> @@ -1450,8 +1450,8 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>>  	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
>>  		dev = container_of(dev_cur, struct its_device, dev_list);
>>  		list_for_each_safe(cur, temp, &dev->itt_head) {
>> -			itte = (container_of(cur, struct its_itte, itte_list));
>> -			its_free_itte(kvm, itte);
>> +			ite = (container_of(cur, struct its_ite, ite_list));
>> +			its_free_ite(kvm, ite);
>>  		}
>>  		list_del(dev_cur);
>>  		kfree(dev);
>> -- 
>> 2.5.5
>>

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Christoffer,

On 27/04/2017 10:57, Christoffer Dall wrote:
> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>> Add description for how to access ITS registers and how to save/restore
>>>> ITS tables into/from memory.
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>> v4 -> v5:
>>>> - take into account Christoffer's comments
>>>> - pending table save on GICV3 side now
>>>>
>>>> v3 -> v4:
>>>> - take into account Peter's comments:
>>>>   - typos
>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>   - add a validity bit in DTE
>>>>   - document all fields in CTE and ITE
>>>>   - document ABI revision
>>>> - take into account Andre's comments:
>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>   - document 64b registers only can be accessed with 64b access
>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>
>>>> v1 -> v2:
>>>> - DTE and ITE now are 8 bytes
>>>> - DTE and ITE now indexed by deviceid/eventid
>>>> - use ITE name instead of ITTE
>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>> - mentions LE layout
>>>> ---
>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>  1 file changed, 99 insertions(+)
>>>>
>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>> index 6081a5b..b5f010d 100644
>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>> @@ -32,7 +32,106 @@ Groups:
>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>        request the initialization of the ITS, no additional parameter in
>>>>        kvm_device_attr.addr.
>>>> +
>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>> +      by the guest in corresponding registers/table entries.
>>>> +
>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>> +      are laid out in little endian format as described in the last paragraph.
>>>> +
>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>> +
>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>> +
>>>> +      The GITS_IIDR read-only register must also be restored before
>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>> +
>>>
>>> what is the expected sequence of operations.  For example, to restore
>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>
>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>> and restore GITS_CTLR (which enables the ITS)?
>>
>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>> that the pending table is read. But the whole pending table is not read
>> as we only iterate on registered LPIs. So the ITT must have been
>> restored previously.
>>
>> I became aware that the pending table sync is done twice, once in the
>> pending table restore,  and once in the GITS_CTLR restore. So if we
>> leave this order specification, I should be able to remove the sync on
>> table restore. This was the original reason why GITS_CTLR restore has
>> been done at the very end.
> 
> I'm sorry, I'm a bit confused.  Do we not need
> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?

Yes you do. I was talking about the RDIST pending table sync. The save
is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
which is not requested I think since GITS_CTLR restore does it already.

KVM_DEV_ARM_ITS_RESTORE_TABLES restores all the ITS tables (device,
collection, ITT)

Hope it clarifies.

Thanks

Eric

> 
>>>
>>>>    Errors:
>>>>      -ENXIO:  ITS not properly configured as required prior to setting
>>>>               this attribute
>>>>      -ENOMEM: Memory shortage when allocating ITS internal data
>>>> +    -EINVAL: Inconsistent restored data
>>>> +    -EFAULT: Invalid guest ram access
>>>> +    -EBUSY:  One or more VCPUS are running
>>>> +
>>>> +  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
>>>> +  Attributes:
>>>> +      The attr field of kvm_device_attr encodes the offset of the
>>>> +      ITS register, relative to the ITS control frame base address
>>>> +      (ITS_base).
>>>> +
>>>> +      kvm_device_attr.addr points to a __u64 value whatever the width
>>>> +      of the addressed register (32/64 bits). 64 bit registers can only
>>>> +      be accessed with full length.
>>>> +
>>>> +      Writes to read-only registers are ignored by the kernel except for:
>>>> +      - GITS_READR. It needs to be restored otherwise commands in the queue
>>>> +        will be re-executed after restoring CWRITER. GITS_READR must be restored
>>>> +        before restoring the GITS_CTLR which is likely to enable the ITS.
>>>> +        Also it needs to be restored after GITS_CBASER since a write to
>>>> +        GITS_CBASER resets GITS_CREADR.
>>>> +      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
>>>> +        In the future we might implement direct injection of virtual LPIS.
>>>> +        This will require an upgrade of the table layout and an evolution of
>>>> +        the ABI. GITS_IIDR must be restored before the table restoration.
>>>> +
>>>> +      For other registers, getting or setting a register has the same
>>>> +      effect as reading/writing the register on real hardware.
>>>> +  Errors:
>>>> +    -ENXIO: Offset does not correspond to any supported register
>>>> +    -EFAULT: Invalid user pointer for attr->addr
>>>> +    -EINVAL: Offset is not 64-bit aligned
>>>> +    -EBUSY: one or more VCPUS are running
>>>
>>>
>>> It may be helpful to state the ordering requirements somewhere:
>>>
>>> Restoring the ITS:
>>> ------------------
>>> Restoring the ITS requires certain things to happen in order.
>>> Specifically:
>>>  1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>>  2. Restore GITS_IIDR
>>>  3. Restore GITS_CBASER
>>>  4. Restore GITS_READR
>>>  5. Restore remainin registers except GITS_CTLR
>>>  6. Make sure all guest memory is restored
>>>  7. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>
>> OK I will try to fit that description somewhere.
>>
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Christoffer,

On 27/04/2017 10:57, Christoffer Dall wrote:
> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>> Add description for how to access ITS registers and how to save/restore
>>>> ITS tables into/from memory.
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>> v4 -> v5:
>>>> - take into account Christoffer's comments
>>>> - pending table save on GICV3 side now
>>>>
>>>> v3 -> v4:
>>>> - take into account Peter's comments:
>>>>   - typos
>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>   - add a validity bit in DTE
>>>>   - document all fields in CTE and ITE
>>>>   - document ABI revision
>>>> - take into account Andre's comments:
>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>   - document 64b registers only can be accessed with 64b access
>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>
>>>> v1 -> v2:
>>>> - DTE and ITE now are 8 bytes
>>>> - DTE and ITE now indexed by deviceid/eventid
>>>> - use ITE name instead of ITTE
>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>> - mentions LE layout
>>>> ---
>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>  1 file changed, 99 insertions(+)
>>>>
>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>> index 6081a5b..b5f010d 100644
>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>> @@ -32,7 +32,106 @@ Groups:
>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>        request the initialization of the ITS, no additional parameter in
>>>>        kvm_device_attr.addr.
>>>> +
>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>> +      by the guest in corresponding registers/table entries.
>>>> +
>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>> +      are laid out in little endian format as described in the last paragraph.
>>>> +
>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>> +
>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>> +
>>>> +      The GITS_IIDR read-only register must also be restored before
>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>> +
>>>
>>> what is the expected sequence of operations.  For example, to restore
>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>
>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>> and restore GITS_CTLR (which enables the ITS)?
>>
>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>> that the pending table is read. But the whole pending table is not read
>> as we only iterate on registered LPIs. So the ITT must have been
>> restored previously.
>>
>> I became aware that the pending table sync is done twice, once in the
>> pending table restore,  and once in the GITS_CTLR restore. So if we
>> leave this order specification, I should be able to remove the sync on
>> table restore. This was the original reason why GITS_CTLR restore has
>> been done at the very end.
> 
> I'm sorry, I'm a bit confused.  Do we not need
> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?

Yes you do. I was talking about the RDIST pending table sync. The save
is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
which is not requested I think since GITS_CTLR restore does it already.

KVM_DEV_ARM_ITS_RESTORE_TABLES restores all the ITS tables (device,
collection, ITT)

Hope it clarifies.

Thanks

Eric

> 
>>>
>>>>    Errors:
>>>>      -ENXIO:  ITS not properly configured as required prior to setting
>>>>               this attribute
>>>>      -ENOMEM: Memory shortage when allocating ITS internal data
>>>> +    -EINVAL: Inconsistent restored data
>>>> +    -EFAULT: Invalid guest ram access
>>>> +    -EBUSY:  One or more VCPUS are running
>>>> +
>>>> +  KVM_DEV_ARM_VGIC_GRP_ITS_REGS
>>>> +  Attributes:
>>>> +      The attr field of kvm_device_attr encodes the offset of the
>>>> +      ITS register, relative to the ITS control frame base address
>>>> +      (ITS_base).
>>>> +
>>>> +      kvm_device_attr.addr points to a __u64 value whatever the width
>>>> +      of the addressed register (32/64 bits). 64 bit registers can only
>>>> +      be accessed with full length.
>>>> +
>>>> +      Writes to read-only registers are ignored by the kernel except for:
>>>> +      - GITS_READR. It needs to be restored otherwise commands in the queue
>>>> +        will be re-executed after restoring CWRITER. GITS_READR must be restored
>>>> +        before restoring the GITS_CTLR which is likely to enable the ITS.
>>>> +        Also it needs to be restored after GITS_CBASER since a write to
>>>> +        GITS_CBASER resets GITS_CREADR.
>>>> +      - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
>>>> +        In the future we might implement direct injection of virtual LPIS.
>>>> +        This will require an upgrade of the table layout and an evolution of
>>>> +        the ABI. GITS_IIDR must be restored before the table restoration.
>>>> +
>>>> +      For other registers, getting or setting a register has the same
>>>> +      effect as reading/writing the register on real hardware.
>>>> +  Errors:
>>>> +    -ENXIO: Offset does not correspond to any supported register
>>>> +    -EFAULT: Invalid user pointer for attr->addr
>>>> +    -EINVAL: Offset is not 64-bit aligned
>>>> +    -EBUSY: one or more VCPUS are running
>>>
>>>
>>> It may be helpful to state the ordering requirements somewhere:
>>>
>>> Restoring the ITS:
>>> ------------------
>>> Restoring the ITS requires certain things to happen in order.
>>> Specifically:
>>>  1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>>  2. Restore GITS_IIDR
>>>  3. Restore GITS_CBASER
>>>  4. Restore GITS_READR
>>>  5. Restore remainin registers except GITS_CTLR
>>>  6. Make sure all guest memory is restored
>>>  7. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>
>> OK I will try to fit that description somewhere.
>>
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Auger Eric]

Hi Christoffer, Andre,

On 27/04/2017 11:20, Andre Przywara wrote:
> Hi,
> 
> On 27/04/17 10:05, Christoffer Dall wrote:
>> On Fri, Apr 14, 2017 at 12:15:15PM +0200, Eric Auger wrote:
>>> The actual abbreviation for the interrupt translation table entry
>>> is ITE. Let's rename all itte instances by ite.
>>
>> Is there really any confusion or problems with using itte?  This is a
>> lot of churn...
> 
> I tend to agree (just didn't dare to mention this before).
> I see that the spec speaks of "ITE", but the spelled out term hints more
> at ITTE (because it's a "translation table").
> Besides three letters tend to be more ambiguous than a four letter
> identifier.
> 
> Would adding a comment to the structure definition help?
> 
> But speaking of churn I am not sure how much more work dropping this
> patch now creates on Eric's side ...

I addressed one comment from Marc saying that ITTE was a wrong name and
he had a patch ready to rename them.

see https://patchwork.kernel.org/patch/9513491/

Renaming back would largely impact the other patches I am afraid

Thanks

Eric
> 
> Cheers,
> Andre.
> 
>>>
>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
>>>
>>> ---
>>>
>>> v5: Add Marc's A-b
>>> ---
>>>  virt/kvm/arm/vgic/vgic-its.c | 148 +++++++++++++++++++++----------------------
>>>  1 file changed, 74 insertions(+), 74 deletions(-)
>>>
>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>> index 8d1da1a..3ffcbbe 100644
>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>> @@ -114,8 +114,8 @@ struct its_collection {
>>>  #define its_is_collection_mapped(coll) ((coll) && \
>>>  				((coll)->target_addr != COLLECTION_NOT_MAPPED))
>>>  
>>> -struct its_itte {
>>> -	struct list_head itte_list;
>>> +struct its_ite {
>>> +	struct list_head ite_list;
>>>  
>>>  	struct vgic_irq *irq;
>>>  	struct its_collection *collection;
>>> @@ -143,27 +143,27 @@ static struct its_device *find_its_device(struct vgic_its *its, u32 device_id)
>>>   * Device ID/Event ID pair on an ITS.
>>>   * Must be called with the its_lock mutex held.
>>>   */
>>> -static struct its_itte *find_itte(struct vgic_its *its, u32 device_id,
>>> +static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>>  				  u32 event_id)
>>>  {
>>>  	struct its_device *device;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  	device = find_its_device(its, device_id);
>>>  	if (device == NULL)
>>>  		return NULL;
>>>  
>>> -	list_for_each_entry(itte, &device->itt_head, itte_list)
>>> -		if (itte->event_id == event_id)
>>> -			return itte;
>>> +	list_for_each_entry(ite, &device->itt_head, ite_list)
>>> +		if (ite->event_id == event_id)
>>> +			return ite;
>>>  
>>>  	return NULL;
>>>  }
>>>  
>>>  /* To be used as an iterator this macro misses the enclosing parentheses */
>>> -#define for_each_lpi_its(dev, itte, its) \
>>> +#define for_each_lpi_its(dev, ite, its) \
>>>  	list_for_each_entry(dev, &(its)->device_list, dev_list) \
>>> -		list_for_each_entry(itte, &(dev)->itt_head, itte_list)
>>> +		list_for_each_entry(ite, &(dev)->itt_head, ite_list)
>>>  
>>>  /*
>>>   * We only implement 48 bits of PA at the moment, although the ITS
>>> @@ -270,18 +270,18 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>>>   * Needs to be called whenever either the collection for a LPIs has
>>>   * changed or the collection itself got retargeted.
>>>   */
>>> -static void update_affinity_itte(struct kvm *kvm, struct its_itte *itte)
>>> +static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
>>>  {
>>>  	struct kvm_vcpu *vcpu;
>>>  
>>> -	if (!its_is_collection_mapped(itte->collection))
>>> +	if (!its_is_collection_mapped(ite->collection))
>>>  		return;
>>>  
>>> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
>>> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>>>  
>>> -	spin_lock(&itte->irq->irq_lock);
>>> -	itte->irq->target_vcpu = vcpu;
>>> -	spin_unlock(&itte->irq->irq_lock);
>>> +	spin_lock(&ite->irq->irq_lock);
>>> +	ite->irq->target_vcpu = vcpu;
>>> +	spin_unlock(&ite->irq->irq_lock);
>>>  }
>>>  
>>>  /*
>>> @@ -292,13 +292,13 @@ static void update_affinity_collection(struct kvm *kvm, struct vgic_its *its,
>>>  				       struct its_collection *coll)
>>>  {
>>>  	struct its_device *device;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>> -	for_each_lpi_its(device, itte, its) {
>>> -		if (!itte->collection || coll != itte->collection)
>>> +	for_each_lpi_its(device, ite, its) {
>>> +		if (!ite->collection || coll != ite->collection)
>>>  			continue;
>>>  
>>> -		update_affinity_itte(kvm, itte);
>>> +		update_affinity_ite(kvm, ite);
>>>  	}
>>>  }
>>>  
>>> @@ -425,25 +425,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
>>>  				u32 devid, u32 eventid)
>>>  {
>>>  	struct kvm_vcpu *vcpu;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  	if (!its->enabled)
>>>  		return -EBUSY;
>>>  
>>> -	itte = find_itte(its, devid, eventid);
>>> -	if (!itte || !its_is_collection_mapped(itte->collection))
>>> +	ite = find_ite(its, devid, eventid);
>>> +	if (!ite || !its_is_collection_mapped(ite->collection))
>>>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>>>  
>>> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
>>> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>>>  	if (!vcpu)
>>>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>>>  
>>>  	if (!vcpu->arch.vgic_cpu.lpis_enabled)
>>>  		return -EBUSY;
>>>  
>>> -	spin_lock(&itte->irq->irq_lock);
>>> -	itte->irq->pending_latch = true;
>>> -	vgic_queue_irq_unlock(kvm, itte->irq);
>>> +	spin_lock(&ite->irq->irq_lock);
>>> +	ite->irq->pending_latch = true;
>>> +	vgic_queue_irq_unlock(kvm, ite->irq);
>>>  
>>>  	return 0;
>>>  }
>>> @@ -511,15 +511,15 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
>>>  }
>>>  
>>>  /* Requires the its_lock to be held. */
>>> -static void its_free_itte(struct kvm *kvm, struct its_itte *itte)
>>> +static void its_free_ite(struct kvm *kvm, struct its_ite *ite)
>>>  {
>>> -	list_del(&itte->itte_list);
>>> +	list_del(&ite->ite_list);
>>>  
>>>  	/* This put matches the get in vgic_add_lpi. */
>>> -	if (itte->irq)
>>> -		vgic_put_irq(kvm, itte->irq);
>>> +	if (ite->irq)
>>> +		vgic_put_irq(kvm, ite->irq);
>>>  
>>> -	kfree(itte);
>>> +	kfree(ite);
>>>  }
>>>  
>>>  static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>>> @@ -544,17 +544,17 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
>>>  {
>>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  
>>> -	itte = find_itte(its, device_id, event_id);
>>> -	if (itte && itte->collection) {
>>> +	ite = find_ite(its, device_id, event_id);
>>> +	if (ite && ite->collection) {
>>>  		/*
>>>  		 * Though the spec talks about removing the pending state, we
>>>  		 * don't bother here since we clear the ITTE anyway and the
>>>  		 * pending state is a property of the ITTE struct.
>>>  		 */
>>> -		its_free_itte(kvm, itte);
>>> +		its_free_ite(kvm, ite);
>>>  		return 0;
>>>  	}
>>>  
>>> @@ -572,26 +572,26 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>>  	u32 coll_id = its_cmd_get_collection(its_cmd);
>>>  	struct kvm_vcpu *vcpu;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  	struct its_collection *collection;
>>>  
>>> -	itte = find_itte(its, device_id, event_id);
>>> -	if (!itte)
>>> +	ite = find_ite(its, device_id, event_id);
>>> +	if (!ite)
>>>  		return E_ITS_MOVI_UNMAPPED_INTERRUPT;
>>>  
>>> -	if (!its_is_collection_mapped(itte->collection))
>>> +	if (!its_is_collection_mapped(ite->collection))
>>>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>>>  
>>>  	collection = find_collection(its, coll_id);
>>>  	if (!its_is_collection_mapped(collection))
>>>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>>>  
>>> -	itte->collection = collection;
>>> +	ite->collection = collection;
>>>  	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>>>  
>>> -	spin_lock(&itte->irq->irq_lock);
>>> -	itte->irq->target_vcpu = vcpu;
>>> -	spin_unlock(&itte->irq->irq_lock);
>>> +	spin_lock(&ite->irq->irq_lock);
>>> +	ite->irq->target_vcpu = vcpu;
>>> +	spin_unlock(&ite->irq->irq_lock);
>>>  
>>>  	return 0;
>>>  }
>>> @@ -679,7 +679,7 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>>>  {
>>>  	struct its_collection *collection;
>>>  	struct its_device *device;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  	/*
>>>  	 * Clearing the mapping for that collection ID removes the
>>> @@ -690,10 +690,10 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>>>  	if (!collection)
>>>  		return;
>>>  
>>> -	for_each_lpi_its(device, itte, its)
>>> -		if (itte->collection &&
>>> -		    itte->collection->collection_id == coll_id)
>>> -			itte->collection = NULL;
>>> +	for_each_lpi_its(device, ite, its)
>>> +		if (ite->collection &&
>>> +		    ite->collection->collection_id == coll_id)
>>> +			ite->collection = NULL;
>>>  
>>>  	list_del(&collection->coll_list);
>>>  	kfree(collection);
>>> @@ -709,7 +709,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>>  	u32 coll_id = its_cmd_get_collection(its_cmd);
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  	struct its_device *device;
>>>  	struct its_collection *collection, *new_coll = NULL;
>>>  	int lpi_nr;
>>> @@ -728,7 +728,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>>  		return E_ITS_MAPTI_PHYSICALID_OOR;
>>>  
>>>  	/* If there is an existing mapping, behavior is UNPREDICTABLE. */
>>> -	if (find_itte(its, device_id, event_id))
>>> +	if (find_ite(its, device_id, event_id))
>>>  		return 0;
>>>  
>>>  	collection = find_collection(its, coll_id);
>>> @@ -739,36 +739,36 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>>  		new_coll = collection;
>>>  	}
>>>  
>>> -	itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
>>> -	if (!itte) {
>>> +	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
>>> +	if (!ite) {
>>>  		if (new_coll)
>>>  			vgic_its_free_collection(its, coll_id);
>>>  		return -ENOMEM;
>>>  	}
>>>  
>>> -	itte->event_id	= event_id;
>>> -	list_add_tail(&itte->itte_list, &device->itt_head);
>>> +	ite->event_id	= event_id;
>>> +	list_add_tail(&ite->ite_list, &device->itt_head);
>>>  
>>> -	itte->collection = collection;
>>> -	itte->lpi = lpi_nr;
>>> +	ite->collection = collection;
>>> +	ite->lpi = lpi_nr;
>>>  
>>>  	irq = vgic_add_lpi(kvm, lpi_nr);
>>>  	if (IS_ERR(irq)) {
>>>  		if (new_coll)
>>>  			vgic_its_free_collection(its, coll_id);
>>> -		its_free_itte(kvm, itte);
>>> +		its_free_ite(kvm, ite);
>>>  		return PTR_ERR(irq);
>>>  	}
>>> -	itte->irq = irq;
>>> +	ite->irq = irq;
>>>  
>>> -	update_affinity_itte(kvm, itte);
>>> +	update_affinity_ite(kvm, ite);
>>>  
>>>  	/*
>>>  	 * We "cache" the configuration table entries in out struct vgic_irq's.
>>>  	 * However we only have those structs for mapped IRQs, so we read in
>>>  	 * the respective config data from memory here upon mapping the LPI.
>>>  	 */
>>> -	update_lpi_config(kvm, itte->irq, NULL);
>>> +	update_lpi_config(kvm, ite->irq, NULL);
>>>  
>>>  	return 0;
>>>  }
>>> @@ -776,15 +776,15 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>>  /* Requires the its_lock to be held. */
>>>  static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
>>>  {
>>> -	struct its_itte *itte, *temp;
>>> +	struct its_ite *ite, *temp;
>>>  
>>>  	/*
>>>  	 * The spec says that unmapping a device with still valid
>>>  	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
>>>  	 * since we cannot leave the memory unreferenced.
>>>  	 */
>>> -	list_for_each_entry_safe(itte, temp, &device->itt_head, itte_list)
>>> -		its_free_itte(kvm, itte);
>>> +	list_for_each_entry_safe(ite, temp, &device->itt_head, ite_list)
>>> +		its_free_ite(kvm, ite);
>>>  
>>>  	list_del(&device->dev_list);
>>>  	kfree(device);
>>> @@ -883,14 +883,14 @@ static int vgic_its_cmd_handle_clear(struct kvm *kvm, struct vgic_its *its,
>>>  {
>>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  
>>> -	itte = find_itte(its, device_id, event_id);
>>> -	if (!itte)
>>> +	ite = find_ite(its, device_id, event_id);
>>> +	if (!ite)
>>>  		return E_ITS_CLEAR_UNMAPPED_INTERRUPT;
>>>  
>>> -	itte->irq->pending_latch = false;
>>> +	ite->irq->pending_latch = false;
>>>  
>>>  	return 0;
>>>  }
>>> @@ -904,14 +904,14 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,
>>>  {
>>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  
>>> -	itte = find_itte(its, device_id, event_id);
>>> -	if (!itte)
>>> +	ite = find_ite(its, device_id, event_id);
>>> +	if (!ite)
>>>  		return E_ITS_INV_UNMAPPED_INTERRUPT;
>>>  
>>> -	return update_lpi_config(kvm, itte->irq, NULL);
>>> +	return update_lpi_config(kvm, ite->irq, NULL);
>>>  }
>>>  
>>>  /*
>>> @@ -1435,7 +1435,7 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>>>  	struct kvm *kvm = kvm_dev->kvm;
>>>  	struct vgic_its *its = kvm_dev->private;
>>>  	struct its_device *dev;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  	struct list_head *dev_cur, *dev_temp;
>>>  	struct list_head *cur, *temp;
>>>  
>>> @@ -1450,8 +1450,8 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>>>  	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
>>>  		dev = container_of(dev_cur, struct its_device, dev_list);
>>>  		list_for_each_safe(cur, temp, &dev->itt_head) {
>>> -			itte = (container_of(cur, struct its_itte, itte_list));
>>> -			its_free_itte(kvm, itte);
>>> +			ite = (container_of(cur, struct its_ite, ite_list));
>>> +			its_free_ite(kvm, ite);
>>>  		}
>>>  		list_del(dev_cur);
>>>  		kfree(dev);
>>> -- 
>>> 2.5.5
>>>

[PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Auger Eric]

Hi Christoffer, Andre,

On 27/04/2017 11:20, Andre Przywara wrote:
> Hi,
> 
> On 27/04/17 10:05, Christoffer Dall wrote:
>> On Fri, Apr 14, 2017 at 12:15:15PM +0200, Eric Auger wrote:
>>> The actual abbreviation for the interrupt translation table entry
>>> is ITE. Let's rename all itte instances by ite.
>>
>> Is there really any confusion or problems with using itte?  This is a
>> lot of churn...
> 
> I tend to agree (just didn't dare to mention this before).
> I see that the spec speaks of "ITE", but the spelled out term hints more
> at ITTE (because it's a "translation table").
> Besides three letters tend to be more ambiguous than a four letter
> identifier.
> 
> Would adding a comment to the structure definition help?
> 
> But speaking of churn I am not sure how much more work dropping this
> patch now creates on Eric's side ...

I addressed one comment from Marc saying that ITTE was a wrong name and
he had a patch ready to rename them.

see https://patchwork.kernel.org/patch/9513491/

Renaming back would largely impact the other patches I am afraid

Thanks

Eric
> 
> Cheers,
> Andre.
> 
>>>
>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
>>>
>>> ---
>>>
>>> v5: Add Marc's A-b
>>> ---
>>>  virt/kvm/arm/vgic/vgic-its.c | 148 +++++++++++++++++++++----------------------
>>>  1 file changed, 74 insertions(+), 74 deletions(-)
>>>
>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>> index 8d1da1a..3ffcbbe 100644
>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>> @@ -114,8 +114,8 @@ struct its_collection {
>>>  #define its_is_collection_mapped(coll) ((coll) && \
>>>  				((coll)->target_addr != COLLECTION_NOT_MAPPED))
>>>  
>>> -struct its_itte {
>>> -	struct list_head itte_list;
>>> +struct its_ite {
>>> +	struct list_head ite_list;
>>>  
>>>  	struct vgic_irq *irq;
>>>  	struct its_collection *collection;
>>> @@ -143,27 +143,27 @@ static struct its_device *find_its_device(struct vgic_its *its, u32 device_id)
>>>   * Device ID/Event ID pair on an ITS.
>>>   * Must be called with the its_lock mutex held.
>>>   */
>>> -static struct its_itte *find_itte(struct vgic_its *its, u32 device_id,
>>> +static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>>  				  u32 event_id)
>>>  {
>>>  	struct its_device *device;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  	device = find_its_device(its, device_id);
>>>  	if (device == NULL)
>>>  		return NULL;
>>>  
>>> -	list_for_each_entry(itte, &device->itt_head, itte_list)
>>> -		if (itte->event_id == event_id)
>>> -			return itte;
>>> +	list_for_each_entry(ite, &device->itt_head, ite_list)
>>> +		if (ite->event_id == event_id)
>>> +			return ite;
>>>  
>>>  	return NULL;
>>>  }
>>>  
>>>  /* To be used as an iterator this macro misses the enclosing parentheses */
>>> -#define for_each_lpi_its(dev, itte, its) \
>>> +#define for_each_lpi_its(dev, ite, its) \
>>>  	list_for_each_entry(dev, &(its)->device_list, dev_list) \
>>> -		list_for_each_entry(itte, &(dev)->itt_head, itte_list)
>>> +		list_for_each_entry(ite, &(dev)->itt_head, ite_list)
>>>  
>>>  /*
>>>   * We only implement 48 bits of PA at the moment, although the ITS
>>> @@ -270,18 +270,18 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>>>   * Needs to be called whenever either the collection for a LPIs has
>>>   * changed or the collection itself got retargeted.
>>>   */
>>> -static void update_affinity_itte(struct kvm *kvm, struct its_itte *itte)
>>> +static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
>>>  {
>>>  	struct kvm_vcpu *vcpu;
>>>  
>>> -	if (!its_is_collection_mapped(itte->collection))
>>> +	if (!its_is_collection_mapped(ite->collection))
>>>  		return;
>>>  
>>> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
>>> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>>>  
>>> -	spin_lock(&itte->irq->irq_lock);
>>> -	itte->irq->target_vcpu = vcpu;
>>> -	spin_unlock(&itte->irq->irq_lock);
>>> +	spin_lock(&ite->irq->irq_lock);
>>> +	ite->irq->target_vcpu = vcpu;
>>> +	spin_unlock(&ite->irq->irq_lock);
>>>  }
>>>  
>>>  /*
>>> @@ -292,13 +292,13 @@ static void update_affinity_collection(struct kvm *kvm, struct vgic_its *its,
>>>  				       struct its_collection *coll)
>>>  {
>>>  	struct its_device *device;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>> -	for_each_lpi_its(device, itte, its) {
>>> -		if (!itte->collection || coll != itte->collection)
>>> +	for_each_lpi_its(device, ite, its) {
>>> +		if (!ite->collection || coll != ite->collection)
>>>  			continue;
>>>  
>>> -		update_affinity_itte(kvm, itte);
>>> +		update_affinity_ite(kvm, ite);
>>>  	}
>>>  }
>>>  
>>> @@ -425,25 +425,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
>>>  				u32 devid, u32 eventid)
>>>  {
>>>  	struct kvm_vcpu *vcpu;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  	if (!its->enabled)
>>>  		return -EBUSY;
>>>  
>>> -	itte = find_itte(its, devid, eventid);
>>> -	if (!itte || !its_is_collection_mapped(itte->collection))
>>> +	ite = find_ite(its, devid, eventid);
>>> +	if (!ite || !its_is_collection_mapped(ite->collection))
>>>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>>>  
>>> -	vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
>>> +	vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
>>>  	if (!vcpu)
>>>  		return E_ITS_INT_UNMAPPED_INTERRUPT;
>>>  
>>>  	if (!vcpu->arch.vgic_cpu.lpis_enabled)
>>>  		return -EBUSY;
>>>  
>>> -	spin_lock(&itte->irq->irq_lock);
>>> -	itte->irq->pending_latch = true;
>>> -	vgic_queue_irq_unlock(kvm, itte->irq);
>>> +	spin_lock(&ite->irq->irq_lock);
>>> +	ite->irq->pending_latch = true;
>>> +	vgic_queue_irq_unlock(kvm, ite->irq);
>>>  
>>>  	return 0;
>>>  }
>>> @@ -511,15 +511,15 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
>>>  }
>>>  
>>>  /* Requires the its_lock to be held. */
>>> -static void its_free_itte(struct kvm *kvm, struct its_itte *itte)
>>> +static void its_free_ite(struct kvm *kvm, struct its_ite *ite)
>>>  {
>>> -	list_del(&itte->itte_list);
>>> +	list_del(&ite->ite_list);
>>>  
>>>  	/* This put matches the get in vgic_add_lpi. */
>>> -	if (itte->irq)
>>> -		vgic_put_irq(kvm, itte->irq);
>>> +	if (ite->irq)
>>> +		vgic_put_irq(kvm, ite->irq);
>>>  
>>> -	kfree(itte);
>>> +	kfree(ite);
>>>  }
>>>  
>>>  static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>>> @@ -544,17 +544,17 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
>>>  {
>>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  
>>> -	itte = find_itte(its, device_id, event_id);
>>> -	if (itte && itte->collection) {
>>> +	ite = find_ite(its, device_id, event_id);
>>> +	if (ite && ite->collection) {
>>>  		/*
>>>  		 * Though the spec talks about removing the pending state, we
>>>  		 * don't bother here since we clear the ITTE anyway and the
>>>  		 * pending state is a property of the ITTE struct.
>>>  		 */
>>> -		its_free_itte(kvm, itte);
>>> +		its_free_ite(kvm, ite);
>>>  		return 0;
>>>  	}
>>>  
>>> @@ -572,26 +572,26 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>>  	u32 coll_id = its_cmd_get_collection(its_cmd);
>>>  	struct kvm_vcpu *vcpu;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  	struct its_collection *collection;
>>>  
>>> -	itte = find_itte(its, device_id, event_id);
>>> -	if (!itte)
>>> +	ite = find_ite(its, device_id, event_id);
>>> +	if (!ite)
>>>  		return E_ITS_MOVI_UNMAPPED_INTERRUPT;
>>>  
>>> -	if (!its_is_collection_mapped(itte->collection))
>>> +	if (!its_is_collection_mapped(ite->collection))
>>>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>>>  
>>>  	collection = find_collection(its, coll_id);
>>>  	if (!its_is_collection_mapped(collection))
>>>  		return E_ITS_MOVI_UNMAPPED_COLLECTION;
>>>  
>>> -	itte->collection = collection;
>>> +	ite->collection = collection;
>>>  	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>>>  
>>> -	spin_lock(&itte->irq->irq_lock);
>>> -	itte->irq->target_vcpu = vcpu;
>>> -	spin_unlock(&itte->irq->irq_lock);
>>> +	spin_lock(&ite->irq->irq_lock);
>>> +	ite->irq->target_vcpu = vcpu;
>>> +	spin_unlock(&ite->irq->irq_lock);
>>>  
>>>  	return 0;
>>>  }
>>> @@ -679,7 +679,7 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>>>  {
>>>  	struct its_collection *collection;
>>>  	struct its_device *device;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  	/*
>>>  	 * Clearing the mapping for that collection ID removes the
>>> @@ -690,10 +690,10 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>>>  	if (!collection)
>>>  		return;
>>>  
>>> -	for_each_lpi_its(device, itte, its)
>>> -		if (itte->collection &&
>>> -		    itte->collection->collection_id == coll_id)
>>> -			itte->collection = NULL;
>>> +	for_each_lpi_its(device, ite, its)
>>> +		if (ite->collection &&
>>> +		    ite->collection->collection_id == coll_id)
>>> +			ite->collection = NULL;
>>>  
>>>  	list_del(&collection->coll_list);
>>>  	kfree(collection);
>>> @@ -709,7 +709,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>>  	u32 coll_id = its_cmd_get_collection(its_cmd);
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  	struct its_device *device;
>>>  	struct its_collection *collection, *new_coll = NULL;
>>>  	int lpi_nr;
>>> @@ -728,7 +728,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>>  		return E_ITS_MAPTI_PHYSICALID_OOR;
>>>  
>>>  	/* If there is an existing mapping, behavior is UNPREDICTABLE. */
>>> -	if (find_itte(its, device_id, event_id))
>>> +	if (find_ite(its, device_id, event_id))
>>>  		return 0;
>>>  
>>>  	collection = find_collection(its, coll_id);
>>> @@ -739,36 +739,36 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>>  		new_coll = collection;
>>>  	}
>>>  
>>> -	itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
>>> -	if (!itte) {
>>> +	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
>>> +	if (!ite) {
>>>  		if (new_coll)
>>>  			vgic_its_free_collection(its, coll_id);
>>>  		return -ENOMEM;
>>>  	}
>>>  
>>> -	itte->event_id	= event_id;
>>> -	list_add_tail(&itte->itte_list, &device->itt_head);
>>> +	ite->event_id	= event_id;
>>> +	list_add_tail(&ite->ite_list, &device->itt_head);
>>>  
>>> -	itte->collection = collection;
>>> -	itte->lpi = lpi_nr;
>>> +	ite->collection = collection;
>>> +	ite->lpi = lpi_nr;
>>>  
>>>  	irq = vgic_add_lpi(kvm, lpi_nr);
>>>  	if (IS_ERR(irq)) {
>>>  		if (new_coll)
>>>  			vgic_its_free_collection(its, coll_id);
>>> -		its_free_itte(kvm, itte);
>>> +		its_free_ite(kvm, ite);
>>>  		return PTR_ERR(irq);
>>>  	}
>>> -	itte->irq = irq;
>>> +	ite->irq = irq;
>>>  
>>> -	update_affinity_itte(kvm, itte);
>>> +	update_affinity_ite(kvm, ite);
>>>  
>>>  	/*
>>>  	 * We "cache" the configuration table entries in out struct vgic_irq's.
>>>  	 * However we only have those structs for mapped IRQs, so we read in
>>>  	 * the respective config data from memory here upon mapping the LPI.
>>>  	 */
>>> -	update_lpi_config(kvm, itte->irq, NULL);
>>> +	update_lpi_config(kvm, ite->irq, NULL);
>>>  
>>>  	return 0;
>>>  }
>>> @@ -776,15 +776,15 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>>  /* Requires the its_lock to be held. */
>>>  static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
>>>  {
>>> -	struct its_itte *itte, *temp;
>>> +	struct its_ite *ite, *temp;
>>>  
>>>  	/*
>>>  	 * The spec says that unmapping a device with still valid
>>>  	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
>>>  	 * since we cannot leave the memory unreferenced.
>>>  	 */
>>> -	list_for_each_entry_safe(itte, temp, &device->itt_head, itte_list)
>>> -		its_free_itte(kvm, itte);
>>> +	list_for_each_entry_safe(ite, temp, &device->itt_head, ite_list)
>>> +		its_free_ite(kvm, ite);
>>>  
>>>  	list_del(&device->dev_list);
>>>  	kfree(device);
>>> @@ -883,14 +883,14 @@ static int vgic_its_cmd_handle_clear(struct kvm *kvm, struct vgic_its *its,
>>>  {
>>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  
>>> -	itte = find_itte(its, device_id, event_id);
>>> -	if (!itte)
>>> +	ite = find_ite(its, device_id, event_id);
>>> +	if (!ite)
>>>  		return E_ITS_CLEAR_UNMAPPED_INTERRUPT;
>>>  
>>> -	itte->irq->pending_latch = false;
>>> +	ite->irq->pending_latch = false;
>>>  
>>>  	return 0;
>>>  }
>>> @@ -904,14 +904,14 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,
>>>  {
>>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>>  	u32 event_id = its_cmd_get_id(its_cmd);
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  
>>>  
>>> -	itte = find_itte(its, device_id, event_id);
>>> -	if (!itte)
>>> +	ite = find_ite(its, device_id, event_id);
>>> +	if (!ite)
>>>  		return E_ITS_INV_UNMAPPED_INTERRUPT;
>>>  
>>> -	return update_lpi_config(kvm, itte->irq, NULL);
>>> +	return update_lpi_config(kvm, ite->irq, NULL);
>>>  }
>>>  
>>>  /*
>>> @@ -1435,7 +1435,7 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>>>  	struct kvm *kvm = kvm_dev->kvm;
>>>  	struct vgic_its *its = kvm_dev->private;
>>>  	struct its_device *dev;
>>> -	struct its_itte *itte;
>>> +	struct its_ite *ite;
>>>  	struct list_head *dev_cur, *dev_temp;
>>>  	struct list_head *cur, *temp;
>>>  
>>> @@ -1450,8 +1450,8 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>>>  	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
>>>  		dev = container_of(dev_cur, struct its_device, dev_list);
>>>  		list_for_each_safe(cur, temp, &dev->itt_head) {
>>> -			itte = (container_of(cur, struct its_itte, itte_list));
>>> -			its_free_itte(kvm, itte);
>>> +			ite = (container_of(cur, struct its_ite, ite_list));
>>> +			its_free_ite(kvm, ite);
>>>  		}
>>>  		list_del(dev_cur);
>>>  		kfree(dev);
>>> -- 
>>> 2.5.5
>>>

Re: [PATCH v5 07/22] KVM: arm64: vgic-its: Implement vgic_its_has_attr_regs and attr_regs_access

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:19PM +0200, Eric Auger wrote:
> This patch implements vgic_its_has_attr_regs and vgic_its_attr_regs_access
> upon the MMIO framework. VGIC ITS KVM device KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> group becomes functional.
> 
> At least GITS_CREADR and GITS_IIDR require to differentiate a guest write
> action from a user access. As such let's introduce a new uaccess_its_write
> vgic_register_region callback.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - use GITS_TYPER instead of offset 0x8
> - uaccess_its_write now can return an error
> 
> v3 -> v4:
> - remove changes to the REGISTER_ITS_DESC macro. This will be handled in
>   subsequent patch with the introduction of a new REGISTER_ITS_DESC_UACCESS
>   macro
> - fix IIDR access and add a comment wrt full length access
> - handle endianness
> - add kvm lock and vcpus lock
> ---
>  virt/kvm/arm/vgic/vgic-its.c  | 79 +++++++++++++++++++++++++++++++++++++++++--
>  virt/kvm/arm/vgic/vgic-mmio.h |  9 +++--
>  2 files changed, 84 insertions(+), 4 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index f687e91..a9a2c12 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1469,14 +1469,89 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>  int vgic_its_has_attr_regs(struct kvm_device *dev,
>  			   struct kvm_device_attr *attr)
>  {
> -	return -ENXIO;
> +	const struct vgic_register_region *region;
> +	struct vgic_io_device iodev = {
> +		.regions = its_registers,
> +		.nr_regions = ARRAY_SIZE(its_registers),
> +	};
> +	gpa_t offset = attr->attr;
> +
> +	region = vgic_find_mmio_region(iodev.regions,
> +				       iodev.nr_regions,
> +				       offset);

why do you need to define the iodev here?  Can't you just pass
its_registers and ARRAY_SIZE(its_registers) directly?

> +	if (!region)
> +		return -ENXIO;
> +
> +	return 0;
>  }
>  
>  int vgic_its_attr_regs_access(struct kvm_device *dev,
>  			      struct kvm_device_attr *attr,
>  			      u64 *reg, bool is_write)
>  {
> -	return -ENXIO;
> +	const struct vgic_register_region *region;
> +	struct vgic_io_device iodev = {
> +		.regions = its_registers,
> +		.nr_regions = ARRAY_SIZE(its_registers),
> +	};
> +	struct vgic_its *its = dev->private;
> +	gpa_t addr, offset = attr->attr;
> +	unsigned int len;
> +	unsigned long data = 0;
> +	int ret = 0;

nit: you could structure this whole declaration block more nicely by
having a separate line for the declaration and initialization of offset,
and by moving the iodev declaration/initialization to the end.

It might also be cleaner to do any non-zero initialization in a separate
block following the declarations.

> +
> +	/*
> +	 * Among supported registers, only GITS_CTLR (0x0) and GITS_IIDR (0x4)
> +	 * are 32 bits. Others are 64 bits.
> +	 */
> +	if ((offset < GITS_TYPER && offset & 0x3) ||
> +	    (offset >= GITS_TYPER && offset & 0x7))
> +		return -EINVAL;
> +
> +	mutex_lock(&dev->kvm->lock);
> +
> +	if (IS_VGIC_ADDR_UNDEF(its->vgic_its_base)) {
> +		ret = -ENXIO;
> +		goto out;
> +	}
> +
> +	region = vgic_find_mmio_region(iodev.regions,
> +				       iodev.nr_regions,
> +				       offset);
> +	if (!region) {
> +		ret = -ENXIO;
> +		goto out;
> +	}
> +
> +	if (!lock_all_vcpus(dev->kvm)) {
> +		ret = -EBUSY;
> +		goto out;
> +	}
> +
> +	addr = its->vgic_its_base + offset;
> +
> +	/*
> +	 * Only full length register accesses are supported although
> +	 * the architecture spec theoretically allows upper/lower 32

does the spec allow 32-bit accesses, or only theoretically ?  :)

> +	 * bits to be accessed independently
> +	 */

In any case, the comment is a bit confusing, because it seems to imply
that we only support 64-bit accesses, but we do set the length below to
4 or 8.

Did you mean:

	/* 
	 * Althought the spec supports upper/lower 32-bit accesses to
	 * 64-bit ITS registers, the userspace ABI requires 64-bit
	 * accesses to all 64-bit wide registers.  We therefore only 32-bit
	 * accesses to the GITS_CTLR, GITS_IIDR registers.
	 */

Also, I don't understand how this works with the ID registers?  For
example, if userspace wants to read GITS_PIDR1 does it have to read
GITS_PIDR0 as a 64-bit register and split it afterwards?  (that doesn't
work with this implementation)

> +	len = region->access_flags & VGIC_ACCESS_64bit ? 8 : 4;
> +
> +	if (is_write) {
> +		data = vgic_data_mmio_bus_to_host(reg, len);

I don't think we need this anymore; we no longer share the guest
trapping MMIO path with uaccesses.

> +		if (region->uaccess_its_write)
> +			ret = region->uaccess_its_write(dev->kvm, its, addr,
> +							len, data);
> +		else
> +			region->its_write(dev->kvm, its, addr, len, data);
> +	} else {
> +		data = region->its_read(dev->kvm, its, addr, len);
> +		vgic_data_host_to_mmio_bus(reg, len, data);

same here.

> +	}
> +	unlock_all_vcpus(dev->kvm);
> +out:
> +	mutex_unlock(&dev->kvm->lock);
> +	return ret;
>  }
>  
>  static int vgic_its_has_attr(struct kvm_device *dev,
> diff --git a/virt/kvm/arm/vgic/vgic-mmio.h b/virt/kvm/arm/vgic/vgic-mmio.h
> index 6eec91b..ea4171a 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio.h
> +++ b/virt/kvm/arm/vgic/vgic-mmio.h
> @@ -36,8 +36,13 @@ struct vgic_register_region {
>  	};
>  	unsigned long (*uaccess_read)(struct kvm_vcpu *vcpu, gpa_t addr,
>  				      unsigned int len);
> -	void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
> -			      unsigned int len, unsigned long val);
> +	union {
> +		void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
> +				      unsigned int len, unsigned long val);
> +		int (*uaccess_its_write)(struct kvm *kvm, struct vgic_its *its,
> +					 gpa_t addr, unsigned int len,
> +					 unsigned long val);
> +	};
>  };
>  
>  extern struct kvm_io_device_ops kvm_io_gic_ops;
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

[PATCH v5 07/22] KVM: arm64: vgic-its: Implement vgic_its_has_attr_regs and attr_regs_access

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:19PM +0200, Eric Auger wrote:
> This patch implements vgic_its_has_attr_regs and vgic_its_attr_regs_access
> upon the MMIO framework. VGIC ITS KVM device KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> group becomes functional.
> 
> At least GITS_CREADR and GITS_IIDR require to differentiate a guest write
> action from a user access. As such let's introduce a new uaccess_its_write
> vgic_register_region callback.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - use GITS_TYPER instead of offset 0x8
> - uaccess_its_write now can return an error
> 
> v3 -> v4:
> - remove changes to the REGISTER_ITS_DESC macro. This will be handled in
>   subsequent patch with the introduction of a new REGISTER_ITS_DESC_UACCESS
>   macro
> - fix IIDR access and add a comment wrt full length access
> - handle endianness
> - add kvm lock and vcpus lock
> ---
>  virt/kvm/arm/vgic/vgic-its.c  | 79 +++++++++++++++++++++++++++++++++++++++++--
>  virt/kvm/arm/vgic/vgic-mmio.h |  9 +++--
>  2 files changed, 84 insertions(+), 4 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index f687e91..a9a2c12 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1469,14 +1469,89 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>  int vgic_its_has_attr_regs(struct kvm_device *dev,
>  			   struct kvm_device_attr *attr)
>  {
> -	return -ENXIO;
> +	const struct vgic_register_region *region;
> +	struct vgic_io_device iodev = {
> +		.regions = its_registers,
> +		.nr_regions = ARRAY_SIZE(its_registers),
> +	};
> +	gpa_t offset = attr->attr;
> +
> +	region = vgic_find_mmio_region(iodev.regions,
> +				       iodev.nr_regions,
> +				       offset);

why do you need to define the iodev here?  Can't you just pass
its_registers and ARRAY_SIZE(its_registers) directly?

> +	if (!region)
> +		return -ENXIO;
> +
> +	return 0;
>  }
>  
>  int vgic_its_attr_regs_access(struct kvm_device *dev,
>  			      struct kvm_device_attr *attr,
>  			      u64 *reg, bool is_write)
>  {
> -	return -ENXIO;
> +	const struct vgic_register_region *region;
> +	struct vgic_io_device iodev = {
> +		.regions = its_registers,
> +		.nr_regions = ARRAY_SIZE(its_registers),
> +	};
> +	struct vgic_its *its = dev->private;
> +	gpa_t addr, offset = attr->attr;
> +	unsigned int len;
> +	unsigned long data = 0;
> +	int ret = 0;

nit: you could structure this whole declaration block more nicely by
having a separate line for the declaration and initialization of offset,
and by moving the iodev declaration/initialization to the end.

It might also be cleaner to do any non-zero initialization in a separate
block following the declarations.

> +
> +	/*
> +	 * Among supported registers, only GITS_CTLR (0x0) and GITS_IIDR (0x4)
> +	 * are 32 bits. Others are 64 bits.
> +	 */
> +	if ((offset < GITS_TYPER && offset & 0x3) ||
> +	    (offset >= GITS_TYPER && offset & 0x7))
> +		return -EINVAL;
> +
> +	mutex_lock(&dev->kvm->lock);
> +
> +	if (IS_VGIC_ADDR_UNDEF(its->vgic_its_base)) {
> +		ret = -ENXIO;
> +		goto out;
> +	}
> +
> +	region = vgic_find_mmio_region(iodev.regions,
> +				       iodev.nr_regions,
> +				       offset);
> +	if (!region) {
> +		ret = -ENXIO;
> +		goto out;
> +	}
> +
> +	if (!lock_all_vcpus(dev->kvm)) {
> +		ret = -EBUSY;
> +		goto out;
> +	}
> +
> +	addr = its->vgic_its_base + offset;
> +
> +	/*
> +	 * Only full length register accesses are supported although
> +	 * the architecture spec theoretically allows upper/lower 32

does the spec allow 32-bit accesses, or only theoretically ?  :)

> +	 * bits to be accessed independently
> +	 */

In any case, the comment is a bit confusing, because it seems to imply
that we only support 64-bit accesses, but we do set the length below to
4 or 8.

Did you mean:

	/* 
	 * Althought the spec supports upper/lower 32-bit accesses to
	 * 64-bit ITS registers, the userspace ABI requires 64-bit
	 * accesses to all 64-bit wide registers.  We therefore only 32-bit
	 * accesses to the GITS_CTLR, GITS_IIDR registers.
	 */

Also, I don't understand how this works with the ID registers?  For
example, if userspace wants to read GITS_PIDR1 does it have to read
GITS_PIDR0 as a 64-bit register and split it afterwards?  (that doesn't
work with this implementation)

> +	len = region->access_flags & VGIC_ACCESS_64bit ? 8 : 4;
> +
> +	if (is_write) {
> +		data = vgic_data_mmio_bus_to_host(reg, len);

I don't think we need this anymore; we no longer share the guest
trapping MMIO path with uaccesses.

> +		if (region->uaccess_its_write)
> +			ret = region->uaccess_its_write(dev->kvm, its, addr,
> +							len, data);
> +		else
> +			region->its_write(dev->kvm, its, addr, len, data);
> +	} else {
> +		data = region->its_read(dev->kvm, its, addr, len);
> +		vgic_data_host_to_mmio_bus(reg, len, data);

same here.

> +	}
> +	unlock_all_vcpus(dev->kvm);
> +out:
> +	mutex_unlock(&dev->kvm->lock);
> +	return ret;
>  }
>  
>  static int vgic_its_has_attr(struct kvm_device *dev,
> diff --git a/virt/kvm/arm/vgic/vgic-mmio.h b/virt/kvm/arm/vgic/vgic-mmio.h
> index 6eec91b..ea4171a 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio.h
> +++ b/virt/kvm/arm/vgic/vgic-mmio.h
> @@ -36,8 +36,13 @@ struct vgic_register_region {
>  	};
>  	unsigned long (*uaccess_read)(struct kvm_vcpu *vcpu, gpa_t addr,
>  				      unsigned int len);
> -	void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
> -			      unsigned int len, unsigned long val);
> +	union {
> +		void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
> +				      unsigned int len, unsigned long val);
> +		int (*uaccess_its_write)(struct kvm *kvm, struct vgic_its *its,
> +					 gpa_t addr, unsigned int len,
> +					 unsigned long val);
> +	};
>  };
>  
>  extern struct kvm_io_device_ops kvm_io_gic_ops;
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 27/04/2017 10:57, Christoffer Dall wrote:
> > On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >> Hi Christoffer,
> >>
> >> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>> Add description for how to access ITS registers and how to save/restore
> >>>> ITS tables into/from memory.
> >>>>
> >>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>
> >>>> ---
> >>>> v4 -> v5:
> >>>> - take into account Christoffer's comments
> >>>> - pending table save on GICV3 side now
> >>>>
> >>>> v3 -> v4:
> >>>> - take into account Peter's comments:
> >>>>   - typos
> >>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>   - add a validity bit in DTE
> >>>>   - document all fields in CTE and ITE
> >>>>   - document ABI revision
> >>>> - take into account Andre's comments:
> >>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>   - document 64b registers only can be accessed with 64b access
> >>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>
> >>>> v1 -> v2:
> >>>> - DTE and ITE now are 8 bytes
> >>>> - DTE and ITE now indexed by deviceid/eventid
> >>>> - use ITE name instead of ITTE
> >>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>> - mentions LE layout
> >>>> ---
> >>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>  1 file changed, 99 insertions(+)
> >>>>
> >>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>> index 6081a5b..b5f010d 100644
> >>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>> @@ -32,7 +32,106 @@ Groups:
> >>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>        request the initialization of the ITS, no additional parameter in
> >>>>        kvm_device_attr.addr.
> >>>> +
> >>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>> +      by the guest in corresponding registers/table entries.
> >>>> +
> >>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>> +      are laid out in little endian format as described in the last paragraph.
> >>>> +
> >>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>> +
> >>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>> +
> >>>> +      The GITS_IIDR read-only register must also be restored before
> >>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>> +
> >>>
> >>> what is the expected sequence of operations.  For example, to restore
> >>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>
> >>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>> and restore GITS_CTLR (which enables the ITS)?
> >>
> >> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >> that the pending table is read. But the whole pending table is not read
> >> as we only iterate on registered LPIs. So the ITT must have been
> >> restored previously.
> >>
> >> I became aware that the pending table sync is done twice, once in the
> >> pending table restore,  and once in the GITS_CTLR restore. So if we
> >> leave this order specification, I should be able to remove the sync on
> >> table restore. This was the original reason why GITS_CTLR restore has
> >> been done at the very end.
> > 
> > I'm sorry, I'm a bit confused.  Do we not need
> > KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> 
> Yes you do. I was talking about the RDIST pending table sync. The save
> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> which is not requested I think since GITS_CTLR restore does it already.

Shouldn't restoring the pending tables happen when restoring some
redeistributor state and not anything related to the ITS?

> 
> KVM_DEV_ARM_ITS_RESTORE_TABLES restores all the ITS tables (device,
> collection, ITT)

Why do you need this if you anyway need to restore the CTLR as the last
thing?  Just to make it absolutely clear when it happens, or is there
something which has to happen between the CTLR and the RESTORE?

Thanks,
-Christoffer

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 27/04/2017 10:57, Christoffer Dall wrote:
> > On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >> Hi Christoffer,
> >>
> >> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>> Add description for how to access ITS registers and how to save/restore
> >>>> ITS tables into/from memory.
> >>>>
> >>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>
> >>>> ---
> >>>> v4 -> v5:
> >>>> - take into account Christoffer's comments
> >>>> - pending table save on GICV3 side now
> >>>>
> >>>> v3 -> v4:
> >>>> - take into account Peter's comments:
> >>>>   - typos
> >>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>   - add a validity bit in DTE
> >>>>   - document all fields in CTE and ITE
> >>>>   - document ABI revision
> >>>> - take into account Andre's comments:
> >>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>   - document 64b registers only can be accessed with 64b access
> >>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>
> >>>> v1 -> v2:
> >>>> - DTE and ITE now are 8 bytes
> >>>> - DTE and ITE now indexed by deviceid/eventid
> >>>> - use ITE name instead of ITTE
> >>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>> - mentions LE layout
> >>>> ---
> >>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>  1 file changed, 99 insertions(+)
> >>>>
> >>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>> index 6081a5b..b5f010d 100644
> >>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>> @@ -32,7 +32,106 @@ Groups:
> >>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>        request the initialization of the ITS, no additional parameter in
> >>>>        kvm_device_attr.addr.
> >>>> +
> >>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>> +      by the guest in corresponding registers/table entries.
> >>>> +
> >>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>> +      are laid out in little endian format as described in the last paragraph.
> >>>> +
> >>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>> +
> >>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>> +
> >>>> +      The GITS_IIDR read-only register must also be restored before
> >>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>> +
> >>>
> >>> what is the expected sequence of operations.  For example, to restore
> >>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>
> >>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>> and restore GITS_CTLR (which enables the ITS)?
> >>
> >> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >> that the pending table is read. But the whole pending table is not read
> >> as we only iterate on registered LPIs. So the ITT must have been
> >> restored previously.
> >>
> >> I became aware that the pending table sync is done twice, once in the
> >> pending table restore,  and once in the GITS_CTLR restore. So if we
> >> leave this order specification, I should be able to remove the sync on
> >> table restore. This was the original reason why GITS_CTLR restore has
> >> been done at the very end.
> > 
> > I'm sorry, I'm a bit confused.  Do we not need
> > KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> 
> Yes you do. I was talking about the RDIST pending table sync. The save
> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> which is not requested I think since GITS_CTLR restore does it already.

Shouldn't restoring the pending tables happen when restoring some
redeistributor state and not anything related to the ITS?

> 
> KVM_DEV_ARM_ITS_RESTORE_TABLES restores all the ITS tables (device,
> collection, ITT)

Why do you need this if you anyway need to restore the CTLR as the last
thing?  Just to make it absolutely clear when it happens, or is there
something which has to happen between the CTLR and the RESTORE?

Thanks,
-Christoffer

Re: [PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Christoffer Dall]

On Thu, Apr 27, 2017 at 11:40:16AM +0200, Auger Eric wrote:
> Hi Christoffer, Andre,
> 
> On 27/04/2017 11:20, Andre Przywara wrote:
> > Hi,
> > 
> > On 27/04/17 10:05, Christoffer Dall wrote:
> >> On Fri, Apr 14, 2017 at 12:15:15PM +0200, Eric Auger wrote:
> >>> The actual abbreviation for the interrupt translation table entry
> >>> is ITE. Let's rename all itte instances by ite.
> >>
> >> Is there really any confusion or problems with using itte?  This is a
> >> lot of churn...
> > 
> > I tend to agree (just didn't dare to mention this before).
> > I see that the spec speaks of "ITE", but the spelled out term hints more
> > at ITTE (because it's a "translation table").
> > Besides three letters tend to be more ambiguous than a four letter
> > identifier.
> > 
> > Would adding a comment to the structure definition help?
> > 
> > But speaking of churn I am not sure how much more work dropping this
> > patch now creates on Eric's side ...
> 
> I addressed one comment from Marc saying that ITTE was a wrong name and
> he had a patch ready to rename them.
> 
> see https://patchwork.kernel.org/patch/9513491/
> 
> Renaming back would largely impact the other patches I am afraid
> 
Ok, it's fine, Marc likes the change, and Andre is happy that you are
now responsible for all ITS bugs, and I'm happy if you're all happy.

Let's keep this change :)

-Christoffer

[PATCH v5 03/22] KVM: arm/arm64: vgic-its: rename itte into ite

[Christoffer Dall]

On Thu, Apr 27, 2017 at 11:40:16AM +0200, Auger Eric wrote:
> Hi Christoffer, Andre,
> 
> On 27/04/2017 11:20, Andre Przywara wrote:
> > Hi,
> > 
> > On 27/04/17 10:05, Christoffer Dall wrote:
> >> On Fri, Apr 14, 2017 at 12:15:15PM +0200, Eric Auger wrote:
> >>> The actual abbreviation for the interrupt translation table entry
> >>> is ITE. Let's rename all itte instances by ite.
> >>
> >> Is there really any confusion or problems with using itte?  This is a
> >> lot of churn...
> > 
> > I tend to agree (just didn't dare to mention this before).
> > I see that the spec speaks of "ITE", but the spelled out term hints more
> > at ITTE (because it's a "translation table").
> > Besides three letters tend to be more ambiguous than a four letter
> > identifier.
> > 
> > Would adding a comment to the structure definition help?
> > 
> > But speaking of churn I am not sure how much more work dropping this
> > patch now creates on Eric's side ...
> 
> I addressed one comment from Marc saying that ITTE was a wrong name and
> he had a patch ready to rename them.
> 
> see https://patchwork.kernel.org/patch/9513491/
> 
> Renaming back would largely impact the other patches I am afraid
> 
Ok, it's fine, Marc likes the change, and Andre is happy that you are
now responsible for all ITS bugs, and I'm happy if you're all happy.

Let's keep this change :)

-Christoffer

Re: [PATCH v5 08/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_creadr

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:20PM +0200, Eric Auger wrote:
> GITS_CREADR needs to be restored so let's implement the associated
> uaccess_write_its callback. The write only is allowed if the its
> is disabled.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - keep Stalled bit
> - vgic_mmio_uaccess_write_its_creadr can now return an error
> 
> v3 -> v4:
> - REGISTER_ITS_DESC_UACCESS now introduced in this patch
> - we now check the its is disabled
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 42 ++++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 40 insertions(+), 2 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index a9a2c12..79ed1c2 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1213,6 +1213,33 @@ static unsigned long vgic_mmio_read_its_creadr(struct kvm *kvm,
>  	return extract_bytes(its->creadr, addr & 0x7, len);
>  }
>  
> +static int vgic_mmio_uaccess_write_its_creadr(struct kvm *kvm,
> +					      struct vgic_its *its,
> +					      gpa_t addr, unsigned int len,
> +					      unsigned long val)
> +{
> +	int ret = 0;
> +	u32 reg;
> +
> +	mutex_lock(&its->cmd_lock);
> +
> +	if (its->enabled) {
> +		ret = -EBUSY;
> +		goto out;
> +	}
> +
> +	reg = update_64bit_reg(its->creadr, addr & 7, len, val);

you theoretically don't need this, since you prevent 32-bit accesses to
this register, but I guess it doesn't hurt...

> +	if (ITS_CMD_OFFSET(reg) >= ITS_CMD_BUFFER_SIZE(its->cbaser)) {
> +		ret = -EINVAL;
> +		goto out;
> +	}

can the creadr value be unaligned to the command size?  I don't think
you check that anywhere here?

Thanks,
-Christoffer

> +
> +	its->creadr = reg;
> +out:
> +	mutex_unlock(&its->cmd_lock);
> +	return ret;
> +}
> +
>  #define BASER_INDEX(addr) (((addr) / sizeof(u64)) & 0x7)
>  static unsigned long vgic_mmio_read_its_baser(struct kvm *kvm,
>  					      struct vgic_its *its,
> @@ -1317,6 +1344,16 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its,
>  	.its_write = wr,					\
>  }
>  
> +#define REGISTER_ITS_DESC_UACCESS(off, rd, wr, uwr, length, acc)\
> +{								\
> +	.reg_offset = off,					\
> +	.len = length,						\
> +	.access_flags = acc,					\
> +	.its_read = rd,						\
> +	.its_write = wr,					\
> +	.uaccess_its_write = uwr,				\
> +}
> +
>  static void its_mmio_write_wi(struct kvm *kvm, struct vgic_its *its,
>  			      gpa_t addr, unsigned int len, unsigned long val)
>  {
> @@ -1339,8 +1376,9 @@ static struct vgic_register_region its_registers[] = {
>  	REGISTER_ITS_DESC(GITS_CWRITER,
>  		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8,
>  		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
> -	REGISTER_ITS_DESC(GITS_CREADR,
> -		vgic_mmio_read_its_creadr, its_mmio_write_wi, 8,
> +	REGISTER_ITS_DESC_UACCESS(GITS_CREADR,
> +		vgic_mmio_read_its_creadr, its_mmio_write_wi,
> +		vgic_mmio_uaccess_write_its_creadr, 8,
>  		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
>  	REGISTER_ITS_DESC(GITS_BASER,
>  		vgic_mmio_read_its_baser, vgic_mmio_write_its_baser, 0x40,
> -- 
> 2.5.5
> 

[PATCH v5 08/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_creadr

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:20PM +0200, Eric Auger wrote:
> GITS_CREADR needs to be restored so let's implement the associated
> uaccess_write_its callback. The write only is allowed if the its
> is disabled.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - keep Stalled bit
> - vgic_mmio_uaccess_write_its_creadr can now return an error
> 
> v3 -> v4:
> - REGISTER_ITS_DESC_UACCESS now introduced in this patch
> - we now check the its is disabled
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 42 ++++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 40 insertions(+), 2 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index a9a2c12..79ed1c2 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1213,6 +1213,33 @@ static unsigned long vgic_mmio_read_its_creadr(struct kvm *kvm,
>  	return extract_bytes(its->creadr, addr & 0x7, len);
>  }
>  
> +static int vgic_mmio_uaccess_write_its_creadr(struct kvm *kvm,
> +					      struct vgic_its *its,
> +					      gpa_t addr, unsigned int len,
> +					      unsigned long val)
> +{
> +	int ret = 0;
> +	u32 reg;
> +
> +	mutex_lock(&its->cmd_lock);
> +
> +	if (its->enabled) {
> +		ret = -EBUSY;
> +		goto out;
> +	}
> +
> +	reg = update_64bit_reg(its->creadr, addr & 7, len, val);

you theoretically don't need this, since you prevent 32-bit accesses to
this register, but I guess it doesn't hurt...

> +	if (ITS_CMD_OFFSET(reg) >= ITS_CMD_BUFFER_SIZE(its->cbaser)) {
> +		ret = -EINVAL;
> +		goto out;
> +	}

can the creadr value be unaligned to the command size?  I don't think
you check that anywhere here?

Thanks,
-Christoffer

> +
> +	its->creadr = reg;
> +out:
> +	mutex_unlock(&its->cmd_lock);
> +	return ret;
> +}
> +
>  #define BASER_INDEX(addr) (((addr) / sizeof(u64)) & 0x7)
>  static unsigned long vgic_mmio_read_its_baser(struct kvm *kvm,
>  					      struct vgic_its *its,
> @@ -1317,6 +1344,16 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its,
>  	.its_write = wr,					\
>  }
>  
> +#define REGISTER_ITS_DESC_UACCESS(off, rd, wr, uwr, length, acc)\
> +{								\
> +	.reg_offset = off,					\
> +	.len = length,						\
> +	.access_flags = acc,					\
> +	.its_read = rd,						\
> +	.its_write = wr,					\
> +	.uaccess_its_write = uwr,				\
> +}
> +
>  static void its_mmio_write_wi(struct kvm *kvm, struct vgic_its *its,
>  			      gpa_t addr, unsigned int len, unsigned long val)
>  {
> @@ -1339,8 +1376,9 @@ static struct vgic_register_region its_registers[] = {
>  	REGISTER_ITS_DESC(GITS_CWRITER,
>  		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8,
>  		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
> -	REGISTER_ITS_DESC(GITS_CREADR,
> -		vgic_mmio_read_its_creadr, its_mmio_write_wi, 8,
> +	REGISTER_ITS_DESC_UACCESS(GITS_CREADR,
> +		vgic_mmio_read_its_creadr, its_mmio_write_wi,
> +		vgic_mmio_uaccess_write_its_creadr, 8,
>  		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
>  	REGISTER_ITS_DESC(GITS_BASER,
>  		vgic_mmio_read_its_baser, vgic_mmio_write_its_baser, 0x40,
> -- 
> 2.5.5
> 

Re: [PATCH v5 07/22] KVM: arm64: vgic-its: Implement vgic_its_has_attr_regs and attr_regs_access

[Auger Eric]

Hi Christoffer,

On 27/04/2017 13:00, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:19PM +0200, Eric Auger wrote:
>> This patch implements vgic_its_has_attr_regs and vgic_its_attr_regs_access
>> upon the MMIO framework. VGIC ITS KVM device KVM_DEV_ARM_VGIC_GRP_ITS_REGS
>> group becomes functional.
>>
>> At least GITS_CREADR and GITS_IIDR require to differentiate a guest write
>> action from a user access. As such let's introduce a new uaccess_its_write
>> vgic_register_region callback.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - use GITS_TYPER instead of offset 0x8
>> - uaccess_its_write now can return an error
>>
>> v3 -> v4:
>> - remove changes to the REGISTER_ITS_DESC macro. This will be handled in
>>   subsequent patch with the introduction of a new REGISTER_ITS_DESC_UACCESS
>>   macro
>> - fix IIDR access and add a comment wrt full length access
>> - handle endianness
>> - add kvm lock and vcpus lock
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c  | 79 +++++++++++++++++++++++++++++++++++++++++--
>>  virt/kvm/arm/vgic/vgic-mmio.h |  9 +++--
>>  2 files changed, 84 insertions(+), 4 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index f687e91..a9a2c12 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -1469,14 +1469,89 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>>  int vgic_its_has_attr_regs(struct kvm_device *dev,
>>  			   struct kvm_device_attr *attr)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_register_region *region;
>> +	struct vgic_io_device iodev = {
>> +		.regions = its_registers,
>> +		.nr_regions = ARRAY_SIZE(its_registers),
>> +	};
>> +	gpa_t offset = attr->attr;
>> +
>> +	region = vgic_find_mmio_region(iodev.regions,
>> +				       iodev.nr_regions,
>> +				       offset);
> 
> why do you need to define the iodev here?  Can't you just pass
> its_registers and ARRAY_SIZE(its_registers) directly?
Yes makes sense.
> 
>> +	if (!region)
>> +		return -ENXIO;
>> +
>> +	return 0;
>>  }
>>  
>>  int vgic_its_attr_regs_access(struct kvm_device *dev,
>>  			      struct kvm_device_attr *attr,
>>  			      u64 *reg, bool is_write)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_register_region *region;
>> +	struct vgic_io_device iodev = {
>> +		.regions = its_registers,
>> +		.nr_regions = ARRAY_SIZE(its_registers),
>> +	};
>> +	struct vgic_its *its = dev->private;
>> +	gpa_t addr, offset = attr->attr;
>> +	unsigned int len;
>> +	unsigned long data = 0;
>> +	int ret = 0;
> 
> nit: you could structure this whole declaration block more nicely by
> having a separate line for the declaration and initialization of offset,
> and by moving the iodev declaration/initialization to the end.
> 
> It might also be cleaner to do any non-zero initialization in a separate
> block following the declarations.

OK
> 
>> +
>> +	/*
>> +	 * Among supported registers, only GITS_CTLR (0x0) and GITS_IIDR (0x4)
>> +	 * are 32 bits. Others are 64 bits.
>> +	 */
>> +	if ((offset < GITS_TYPER && offset & 0x3) ||
>> +	    (offset >= GITS_TYPER && offset & 0x7))
>> +		return -EINVAL;
>> +
>> +	mutex_lock(&dev->kvm->lock);
>> +
>> +	if (IS_VGIC_ADDR_UNDEF(its->vgic_its_base)) {
>> +		ret = -ENXIO;
>> +		goto out;
>> +	}
>> +
>> +	region = vgic_find_mmio_region(iodev.regions,
>> +				       iodev.nr_regions,
>> +				       offset);
>> +	if (!region) {
>> +		ret = -ENXIO;
>> +		goto out;
>> +	}
>> +
>> +	if (!lock_all_vcpus(dev->kvm)) {
>> +		ret = -EBUSY;
>> +		goto out;
>> +	}
>> +
>> +	addr = its->vgic_its_base + offset;
>> +
>> +	/*
>> +	 * Only full length register accesses are supported although
>> +	 * the architecture spec theoretically allows upper/lower 32
> 
> does the spec allow 32-bit accesses, or only theoretically ?  :)
Yes the spec allows 32-bit access.
> 
>> +	 * bits to be accessed independently
>> +	 */
> 
> In any case, the comment is a bit confusing, because it seems to imply
> that we only support 64-bit accesses, but we do set the length below to
> 4 or 8.
> 
> Did you mean:
> 
> 	/* 
> 	 * Althought the spec supports upper/lower 32-bit accesses to
> 	 * 64-bit ITS registers, the userspace ABI requires 64-bit
> 	 * accesses to all 64-bit wide registers.  We therefore only 32-bit
> 	 * accesses to the GITS_CTLR, GITS_IIDR registers.
> 	 */
Yes this is what I meant. I will rephrase as you suggest.
> 
> Also, I don't understand how this works with the ID registers?  For
> example, if userspace wants to read GITS_PIDR1 does it have to read
> GITS_PIDR0 as a 64-bit register and split it afterwards?  (that doesn't
> work with this implementation)
> 
Hum the IDREGS were not described in the archi spec 8.19 (ITS register
descriptions) and I "forgot" them. they are 32 bit and 32 bit aligned
access should be allowed. Thanks for spotting this.
>> +	len = region->access_flags & VGIC_ACCESS_64bit ? 8 : 4;
>> +
>> +	if (is_write) {
>> +		data = vgic_data_mmio_bus_to_host(reg, len);
> 
> I don't think we need this anymore; we no longer share the guest
> trapping MMIO path with uaccesses.
agreed
> 
>> +		if (region->uaccess_its_write)
>> +			ret = region->uaccess_its_write(dev->kvm, its, addr,
>> +							len, data);
>> +		else
>> +			region->its_write(dev->kvm, its, addr, len, data);
>> +	} else {
>> +		data = region->its_read(dev->kvm, its, addr, len);
>> +		vgic_data_host_to_mmio_bus(reg, len, data);
> 
> same here.
OK

Thanks

Eric
> 
>> +	}
>> +	unlock_all_vcpus(dev->kvm);
>> +out:
>> +	mutex_unlock(&dev->kvm->lock);
>> +	return ret;
>>  }
>>  
>>  static int vgic_its_has_attr(struct kvm_device *dev,
>> diff --git a/virt/kvm/arm/vgic/vgic-mmio.h b/virt/kvm/arm/vgic/vgic-mmio.h
>> index 6eec91b..ea4171a 100644
>> --- a/virt/kvm/arm/vgic/vgic-mmio.h
>> +++ b/virt/kvm/arm/vgic/vgic-mmio.h
>> @@ -36,8 +36,13 @@ struct vgic_register_region {
>>  	};
>>  	unsigned long (*uaccess_read)(struct kvm_vcpu *vcpu, gpa_t addr,
>>  				      unsigned int len);
>> -	void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
>> -			      unsigned int len, unsigned long val);
>> +	union {
>> +		void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
>> +				      unsigned int len, unsigned long val);
>> +		int (*uaccess_its_write)(struct kvm *kvm, struct vgic_its *its,
>> +					 gpa_t addr, unsigned int len,
>> +					 unsigned long val);
>> +	};
>>  };
>>  
>>  extern struct kvm_io_device_ops kvm_io_gic_ops;
>> -- 
>> 2.5.5
>>
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 07/22] KVM: arm64: vgic-its: Implement vgic_its_has_attr_regs and attr_regs_access

[Auger Eric]

Hi Christoffer,

On 27/04/2017 13:00, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:19PM +0200, Eric Auger wrote:
>> This patch implements vgic_its_has_attr_regs and vgic_its_attr_regs_access
>> upon the MMIO framework. VGIC ITS KVM device KVM_DEV_ARM_VGIC_GRP_ITS_REGS
>> group becomes functional.
>>
>> At least GITS_CREADR and GITS_IIDR require to differentiate a guest write
>> action from a user access. As such let's introduce a new uaccess_its_write
>> vgic_register_region callback.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - use GITS_TYPER instead of offset 0x8
>> - uaccess_its_write now can return an error
>>
>> v3 -> v4:
>> - remove changes to the REGISTER_ITS_DESC macro. This will be handled in
>>   subsequent patch with the introduction of a new REGISTER_ITS_DESC_UACCESS
>>   macro
>> - fix IIDR access and add a comment wrt full length access
>> - handle endianness
>> - add kvm lock and vcpus lock
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c  | 79 +++++++++++++++++++++++++++++++++++++++++--
>>  virt/kvm/arm/vgic/vgic-mmio.h |  9 +++--
>>  2 files changed, 84 insertions(+), 4 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index f687e91..a9a2c12 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -1469,14 +1469,89 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
>>  int vgic_its_has_attr_regs(struct kvm_device *dev,
>>  			   struct kvm_device_attr *attr)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_register_region *region;
>> +	struct vgic_io_device iodev = {
>> +		.regions = its_registers,
>> +		.nr_regions = ARRAY_SIZE(its_registers),
>> +	};
>> +	gpa_t offset = attr->attr;
>> +
>> +	region = vgic_find_mmio_region(iodev.regions,
>> +				       iodev.nr_regions,
>> +				       offset);
> 
> why do you need to define the iodev here?  Can't you just pass
> its_registers and ARRAY_SIZE(its_registers) directly?
Yes makes sense.
> 
>> +	if (!region)
>> +		return -ENXIO;
>> +
>> +	return 0;
>>  }
>>  
>>  int vgic_its_attr_regs_access(struct kvm_device *dev,
>>  			      struct kvm_device_attr *attr,
>>  			      u64 *reg, bool is_write)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_register_region *region;
>> +	struct vgic_io_device iodev = {
>> +		.regions = its_registers,
>> +		.nr_regions = ARRAY_SIZE(its_registers),
>> +	};
>> +	struct vgic_its *its = dev->private;
>> +	gpa_t addr, offset = attr->attr;
>> +	unsigned int len;
>> +	unsigned long data = 0;
>> +	int ret = 0;
> 
> nit: you could structure this whole declaration block more nicely by
> having a separate line for the declaration and initialization of offset,
> and by moving the iodev declaration/initialization to the end.
> 
> It might also be cleaner to do any non-zero initialization in a separate
> block following the declarations.

OK
> 
>> +
>> +	/*
>> +	 * Among supported registers, only GITS_CTLR (0x0) and GITS_IIDR (0x4)
>> +	 * are 32 bits. Others are 64 bits.
>> +	 */
>> +	if ((offset < GITS_TYPER && offset & 0x3) ||
>> +	    (offset >= GITS_TYPER && offset & 0x7))
>> +		return -EINVAL;
>> +
>> +	mutex_lock(&dev->kvm->lock);
>> +
>> +	if (IS_VGIC_ADDR_UNDEF(its->vgic_its_base)) {
>> +		ret = -ENXIO;
>> +		goto out;
>> +	}
>> +
>> +	region = vgic_find_mmio_region(iodev.regions,
>> +				       iodev.nr_regions,
>> +				       offset);
>> +	if (!region) {
>> +		ret = -ENXIO;
>> +		goto out;
>> +	}
>> +
>> +	if (!lock_all_vcpus(dev->kvm)) {
>> +		ret = -EBUSY;
>> +		goto out;
>> +	}
>> +
>> +	addr = its->vgic_its_base + offset;
>> +
>> +	/*
>> +	 * Only full length register accesses are supported although
>> +	 * the architecture spec theoretically allows upper/lower 32
> 
> does the spec allow 32-bit accesses, or only theoretically ?  :)
Yes the spec allows 32-bit access.
> 
>> +	 * bits to be accessed independently
>> +	 */
> 
> In any case, the comment is a bit confusing, because it seems to imply
> that we only support 64-bit accesses, but we do set the length below to
> 4 or 8.
> 
> Did you mean:
> 
> 	/* 
> 	 * Althought the spec supports upper/lower 32-bit accesses to
> 	 * 64-bit ITS registers, the userspace ABI requires 64-bit
> 	 * accesses to all 64-bit wide registers.  We therefore only 32-bit
> 	 * accesses to the GITS_CTLR, GITS_IIDR registers.
> 	 */
Yes this is what I meant. I will rephrase as you suggest.
> 
> Also, I don't understand how this works with the ID registers?  For
> example, if userspace wants to read GITS_PIDR1 does it have to read
> GITS_PIDR0 as a 64-bit register and split it afterwards?  (that doesn't
> work with this implementation)
> 
Hum the IDREGS were not described in the archi spec 8.19 (ITS register
descriptions) and I "forgot" them. they are 32 bit and 32 bit aligned
access should be allowed. Thanks for spotting this.
>> +	len = region->access_flags & VGIC_ACCESS_64bit ? 8 : 4;
>> +
>> +	if (is_write) {
>> +		data = vgic_data_mmio_bus_to_host(reg, len);
> 
> I don't think we need this anymore; we no longer share the guest
> trapping MMIO path with uaccesses.
agreed
> 
>> +		if (region->uaccess_its_write)
>> +			ret = region->uaccess_its_write(dev->kvm, its, addr,
>> +							len, data);
>> +		else
>> +			region->its_write(dev->kvm, its, addr, len, data);
>> +	} else {
>> +		data = region->its_read(dev->kvm, its, addr, len);
>> +		vgic_data_host_to_mmio_bus(reg, len, data);
> 
> same here.
OK

Thanks

Eric
> 
>> +	}
>> +	unlock_all_vcpus(dev->kvm);
>> +out:
>> +	mutex_unlock(&dev->kvm->lock);
>> +	return ret;
>>  }
>>  
>>  static int vgic_its_has_attr(struct kvm_device *dev,
>> diff --git a/virt/kvm/arm/vgic/vgic-mmio.h b/virt/kvm/arm/vgic/vgic-mmio.h
>> index 6eec91b..ea4171a 100644
>> --- a/virt/kvm/arm/vgic/vgic-mmio.h
>> +++ b/virt/kvm/arm/vgic/vgic-mmio.h
>> @@ -36,8 +36,13 @@ struct vgic_register_region {
>>  	};
>>  	unsigned long (*uaccess_read)(struct kvm_vcpu *vcpu, gpa_t addr,
>>  				      unsigned int len);
>> -	void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
>> -			      unsigned int len, unsigned long val);
>> +	union {
>> +		void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
>> +				      unsigned int len, unsigned long val);
>> +		int (*uaccess_its_write)(struct kvm *kvm, struct vgic_its *its,
>> +					 gpa_t addr, unsigned int len,
>> +					 unsigned long val);
>> +	};
>>  };
>>  
>>  extern struct kvm_io_device_ops kvm_io_gic_ops;
>> -- 
>> 2.5.5
>>
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Christoffer,

On 27/04/2017 13:02, Christoffer Dall wrote:
> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>> Hi Christoffer,
>>>>
>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>> ITS tables into/from memory.
>>>>>>
>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>
>>>>>> ---
>>>>>> v4 -> v5:
>>>>>> - take into account Christoffer's comments
>>>>>> - pending table save on GICV3 side now
>>>>>>
>>>>>> v3 -> v4:
>>>>>> - take into account Peter's comments:
>>>>>>   - typos
>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>   - add a validity bit in DTE
>>>>>>   - document all fields in CTE and ITE
>>>>>>   - document ABI revision
>>>>>> - take into account Andre's comments:
>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>
>>>>>> v1 -> v2:
>>>>>> - DTE and ITE now are 8 bytes
>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>> - use ITE name instead of ITTE
>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>> - mentions LE layout
>>>>>> ---
>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>  1 file changed, 99 insertions(+)
>>>>>>
>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>> index 6081a5b..b5f010d 100644
>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>        kvm_device_attr.addr.
>>>>>> +
>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>> +
>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>> +
>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>> +
>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>> +
>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>> +
>>>>>
>>>>> what is the expected sequence of operations.  For example, to restore
>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>
>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>
>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>> that the pending table is read. But the whole pending table is not read
>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>> restored previously.
>>>>
>>>> I became aware that the pending table sync is done twice, once in the
>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>> leave this order specification, I should be able to remove the sync on
>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>> been done at the very end.
>>>
>>> I'm sorry, I'm a bit confused.  Do we not need
>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>
>> Yes you do. I was talking about the RDIST pending table sync. The save
>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>> which is not requested I think since GITS_CTLR restore does it already.
> 
> Shouldn't restoring the pending tables happen when restoring some
> redeistributor state and not anything related to the ITS?

Marc wrote:
"
I don't think you necessarily need a coarse map. When restoring the ITS
tables, you can always read the pending bit when creating the LPI
structure (it has been written to RAM at save time). Note that we
already do something like this in vgic_enable_lpis().
"

This is currently what is implemented I think. the pending tables are
currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
also on on ITS table restore

The problematic is: Either you know in advance which LPI INTIDare used
or you need to parse the whole pending table (possibly using the 1st kB
as coarse mapping).

If you don't know the LPI INTIDs in advance it is only possible to
restore the pending bit of pending LPIs. At that time you would
re-allocate those pending LPI (vgic_add_lpi) and when you restore the
ITS ITT you would do the same for those which were not pending. Looks
really heavy to me: coarse mapping + dual vgic_add_lpi path.

Otherwise we would need to add another dependency between RDIST pending
table restore and ITS table restore but this looks even more weird, no?


> 
>>
>> KVM_DEV_ARM_ITS_RESTORE_TABLES restores all the ITS tables (device,
>> collection, ITT)
> 
> Why do you need this if you anyway need to restore the CTLR as the last
> thing?  Just to make it absolutely clear when it happens, or is there
> something which has to happen between the CTLR and the RESTORE?

Not sure I get this one.

by doing the lookup of Device table and ITT,
KVM_DEV_ARM_ITS_RESTORE_TABLES enumerates all the LPI INTIDs that exist,
re-allocate them

Then when GITS_CTLR is set, you can use those INTIDs to parse the exact
pending status in the pending tables. In case
KVM_DEV_ARM_ITS_RESTORE_TABLES were not called you would be "blind" and
you would need to parse the whole pending table.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Christoffer,

On 27/04/2017 13:02, Christoffer Dall wrote:
> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>> Hi Christoffer,
>>>>
>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>> ITS tables into/from memory.
>>>>>>
>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>
>>>>>> ---
>>>>>> v4 -> v5:
>>>>>> - take into account Christoffer's comments
>>>>>> - pending table save on GICV3 side now
>>>>>>
>>>>>> v3 -> v4:
>>>>>> - take into account Peter's comments:
>>>>>>   - typos
>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>   - add a validity bit in DTE
>>>>>>   - document all fields in CTE and ITE
>>>>>>   - document ABI revision
>>>>>> - take into account Andre's comments:
>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>
>>>>>> v1 -> v2:
>>>>>> - DTE and ITE now are 8 bytes
>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>> - use ITE name instead of ITTE
>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>> - mentions LE layout
>>>>>> ---
>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>  1 file changed, 99 insertions(+)
>>>>>>
>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>> index 6081a5b..b5f010d 100644
>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>        kvm_device_attr.addr.
>>>>>> +
>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>> +
>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>> +
>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>> +
>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>> +
>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>> +
>>>>>
>>>>> what is the expected sequence of operations.  For example, to restore
>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>
>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>
>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>> that the pending table is read. But the whole pending table is not read
>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>> restored previously.
>>>>
>>>> I became aware that the pending table sync is done twice, once in the
>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>> leave this order specification, I should be able to remove the sync on
>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>> been done at the very end.
>>>
>>> I'm sorry, I'm a bit confused.  Do we not need
>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>
>> Yes you do. I was talking about the RDIST pending table sync. The save
>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>> which is not requested I think since GITS_CTLR restore does it already.
> 
> Shouldn't restoring the pending tables happen when restoring some
> redeistributor state and not anything related to the ITS?

Marc wrote:
"
I don't think you necessarily need a coarse map. When restoring the ITS
tables, you can always read the pending bit when creating the LPI
structure (it has been written to RAM at save time). Note that we
already do something like this in vgic_enable_lpis().
"

This is currently what is implemented I think. the pending tables are
currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
also on on ITS table restore

The problematic is: Either you know in advance which LPI INTIDare used
or you need to parse the whole pending table (possibly using the 1st kB
as coarse mapping).

If you don't know the LPI INTIDs in advance it is only possible to
restore the pending bit of pending LPIs. At that time you would
re-allocate those pending LPI (vgic_add_lpi) and when you restore the
ITS ITT you would do the same for those which were not pending. Looks
really heavy to me: coarse mapping + dual vgic_add_lpi path.

Otherwise we would need to add another dependency between RDIST pending
table restore and ITS table restore but this looks even more weird, no?


> 
>>
>> KVM_DEV_ARM_ITS_RESTORE_TABLES restores all the ITS tables (device,
>> collection, ITT)
> 
> Why do you need this if you anyway need to restore the CTLR as the last
> thing?  Just to make it absolutely clear when it happens, or is there
> something which has to happen between the CTLR and the RESTORE?

Not sure I get this one.

by doing the lookup of Device table and ITT,
KVM_DEV_ARM_ITS_RESTORE_TABLES enumerates all the LPI INTIDs that exist,
re-allocate them

Then when GITS_CTLR is set, you can use those INTIDs to parse the exact
pending status in the pending tables. In case
KVM_DEV_ARM_ITS_RESTORE_TABLES were not called you would be "blind" and
you would need to parse the whole pending table.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 08/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_creadr

[Auger Eric]

Hi,

On 27/04/2017 13:27, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:20PM +0200, Eric Auger wrote:
>> GITS_CREADR needs to be restored so let's implement the associated
>> uaccess_write_its callback. The write only is allowed if the its
>> is disabled.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - keep Stalled bit
>> - vgic_mmio_uaccess_write_its_creadr can now return an error
>>
>> v3 -> v4:
>> - REGISTER_ITS_DESC_UACCESS now introduced in this patch
>> - we now check the its is disabled
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 42 ++++++++++++++++++++++++++++++++++++++++--
>>  1 file changed, 40 insertions(+), 2 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index a9a2c12..79ed1c2 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -1213,6 +1213,33 @@ static unsigned long vgic_mmio_read_its_creadr(struct kvm *kvm,
>>  	return extract_bytes(its->creadr, addr & 0x7, len);
>>  }
>>  
>> +static int vgic_mmio_uaccess_write_its_creadr(struct kvm *kvm,
>> +					      struct vgic_its *its,
>> +					      gpa_t addr, unsigned int len,
>> +					      unsigned long val)
>> +{
>> +	int ret = 0;
>> +	u32 reg;
>> +
>> +	mutex_lock(&its->cmd_lock);
>> +
>> +	if (its->enabled) {
>> +		ret = -EBUSY;
>> +		goto out;
>> +	}
>> +
>> +	reg = update_64bit_reg(its->creadr, addr & 7, len, val);
> 
> you theoretically don't need this, since you prevent 32-bit accesses to
> this register, but I guess it doesn't hurt...
OK
> 
>> +	if (ITS_CMD_OFFSET(reg) >= ITS_CMD_BUFFER_SIZE(its->cbaser)) {
>> +		ret = -EINVAL;
>> +		goto out;
>> +	}
> 
> can the creadr value be unaligned to the command size?  I don't think
> you check that anywhere here?
makes sense. I will add this check.

Thanks!

Eric
> 
> Thanks,
> -Christoffer
> 
>> +
>> +	its->creadr = reg;
>> +out:
>> +	mutex_unlock(&its->cmd_lock);
>> +	return ret;
>> +}
>> +
>>  #define BASER_INDEX(addr) (((addr) / sizeof(u64)) & 0x7)
>>  static unsigned long vgic_mmio_read_its_baser(struct kvm *kvm,
>>  					      struct vgic_its *its,
>> @@ -1317,6 +1344,16 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its,
>>  	.its_write = wr,					\
>>  }
>>  
>> +#define REGISTER_ITS_DESC_UACCESS(off, rd, wr, uwr, length, acc)\
>> +{								\
>> +	.reg_offset = off,					\
>> +	.len = length,						\
>> +	.access_flags = acc,					\
>> +	.its_read = rd,						\
>> +	.its_write = wr,					\
>> +	.uaccess_its_write = uwr,				\
>> +}
>> +
>>  static void its_mmio_write_wi(struct kvm *kvm, struct vgic_its *its,
>>  			      gpa_t addr, unsigned int len, unsigned long val)
>>  {
>> @@ -1339,8 +1376,9 @@ static struct vgic_register_region its_registers[] = {
>>  	REGISTER_ITS_DESC(GITS_CWRITER,
>>  		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8,
>>  		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
>> -	REGISTER_ITS_DESC(GITS_CREADR,
>> -		vgic_mmio_read_its_creadr, its_mmio_write_wi, 8,
>> +	REGISTER_ITS_DESC_UACCESS(GITS_CREADR,
>> +		vgic_mmio_read_its_creadr, its_mmio_write_wi,
>> +		vgic_mmio_uaccess_write_its_creadr, 8,
>>  		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
>>  	REGISTER_ITS_DESC(GITS_BASER,
>>  		vgic_mmio_read_its_baser, vgic_mmio_write_its_baser, 0x40,
>> -- 
>> 2.5.5
>>

[PATCH v5 08/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_creadr

[Auger Eric]

Hi,

On 27/04/2017 13:27, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:20PM +0200, Eric Auger wrote:
>> GITS_CREADR needs to be restored so let's implement the associated
>> uaccess_write_its callback. The write only is allowed if the its
>> is disabled.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - keep Stalled bit
>> - vgic_mmio_uaccess_write_its_creadr can now return an error
>>
>> v3 -> v4:
>> - REGISTER_ITS_DESC_UACCESS now introduced in this patch
>> - we now check the its is disabled
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 42 ++++++++++++++++++++++++++++++++++++++++--
>>  1 file changed, 40 insertions(+), 2 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index a9a2c12..79ed1c2 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -1213,6 +1213,33 @@ static unsigned long vgic_mmio_read_its_creadr(struct kvm *kvm,
>>  	return extract_bytes(its->creadr, addr & 0x7, len);
>>  }
>>  
>> +static int vgic_mmio_uaccess_write_its_creadr(struct kvm *kvm,
>> +					      struct vgic_its *its,
>> +					      gpa_t addr, unsigned int len,
>> +					      unsigned long val)
>> +{
>> +	int ret = 0;
>> +	u32 reg;
>> +
>> +	mutex_lock(&its->cmd_lock);
>> +
>> +	if (its->enabled) {
>> +		ret = -EBUSY;
>> +		goto out;
>> +	}
>> +
>> +	reg = update_64bit_reg(its->creadr, addr & 7, len, val);
> 
> you theoretically don't need this, since you prevent 32-bit accesses to
> this register, but I guess it doesn't hurt...
OK
> 
>> +	if (ITS_CMD_OFFSET(reg) >= ITS_CMD_BUFFER_SIZE(its->cbaser)) {
>> +		ret = -EINVAL;
>> +		goto out;
>> +	}
> 
> can the creadr value be unaligned to the command size?  I don't think
> you check that anywhere here?
makes sense. I will add this check.

Thanks!

Eric
> 
> Thanks,
> -Christoffer
> 
>> +
>> +	its->creadr = reg;
>> +out:
>> +	mutex_unlock(&its->cmd_lock);
>> +	return ret;
>> +}
>> +
>>  #define BASER_INDEX(addr) (((addr) / sizeof(u64)) & 0x7)
>>  static unsigned long vgic_mmio_read_its_baser(struct kvm *kvm,
>>  					      struct vgic_its *its,
>> @@ -1317,6 +1344,16 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its,
>>  	.its_write = wr,					\
>>  }
>>  
>> +#define REGISTER_ITS_DESC_UACCESS(off, rd, wr, uwr, length, acc)\
>> +{								\
>> +	.reg_offset = off,					\
>> +	.len = length,						\
>> +	.access_flags = acc,					\
>> +	.its_read = rd,						\
>> +	.its_write = wr,					\
>> +	.uaccess_its_write = uwr,				\
>> +}
>> +
>>  static void its_mmio_write_wi(struct kvm *kvm, struct vgic_its *its,
>>  			      gpa_t addr, unsigned int len, unsigned long val)
>>  {
>> @@ -1339,8 +1376,9 @@ static struct vgic_register_region its_registers[] = {
>>  	REGISTER_ITS_DESC(GITS_CWRITER,
>>  		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8,
>>  		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
>> -	REGISTER_ITS_DESC(GITS_CREADR,
>> -		vgic_mmio_read_its_creadr, its_mmio_write_wi, 8,
>> +	REGISTER_ITS_DESC_UACCESS(GITS_CREADR,
>> +		vgic_mmio_read_its_creadr, its_mmio_write_wi,
>> +		vgic_mmio_uaccess_write_its_creadr, 8,
>>  		VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
>>  	REGISTER_ITS_DESC(GITS_BASER,
>>  		vgic_mmio_read_its_baser, vgic_mmio_write_its_baser, 0x40,
>> -- 
>> 2.5.5
>>

Re: [PATCH v5 09/22] KVM: arm64: vgic-its: Introduce migration ABI infrastructure

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:21PM +0200, Eric Auger wrote:
> We plan to support different migration ABIs, ie. characterizing
> the ITS table layout format in guest RAM. Typically a new ABI will

nit: s/Typically/For example,

> be needed if vLPIs get supported for nested use case.
> 
> So let's introduce an array of supported ABIs (at the moment a single
> ABI is supported though). The following characteristics are foreseen
> to vary with the ABI: size of table entries, save/restore operation.
> 
> By default the MAX_ABI_REV is applied on its creation. In subsequent
> patches we will introduce a way for the userspace to change the ABI
> in use.
> 
> The entry sizes now are set according to the ABI version and not
> hardcoded anymore.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> 
> v5: creation and squash KVM: arm64: ITS: Report the ITE size in
>     GITS_TYPER
> ---
>  include/kvm/arm_vgic.h             |  3 ++
>  include/linux/irqchip/arm-gic-v3.h |  4 ++
>  virt/kvm/arm/vgic/vgic-its.c       | 82 ++++++++++++++++++++++++++++++++++++--
>  3 files changed, 85 insertions(+), 4 deletions(-)
> 
> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
> index c0b3d99..285474a 100644
> --- a/include/kvm/arm_vgic.h
> +++ b/include/kvm/arm_vgic.h
> @@ -162,6 +162,9 @@ struct vgic_its {
>  	u32			creadr;
>  	u32			cwriter;
>  
> +	/* migration ABI revision in use */
> +	u32			abi_rev;
> +
>  	/* Protects the device and collection lists */
>  	struct mutex		its_lock;
>  	struct list_head	device_list;
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index 97cbca1..9648bad 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -132,6 +132,8 @@
>  #define GIC_BASER_SHAREABILITY(reg, type)				\
>  	(GIC_BASER_##type << reg##_SHAREABILITY_SHIFT)
>  
> +#define GIC_ENCODE_SZ(n, w) (((unsigned long)(n) - 1) & GENMASK_ULL(((w) - 1), 0))
> +

This is a very generic name.  Could you add a comment about what the
idea is?

>  #define GICR_PROPBASER_SHAREABILITY_SHIFT		(10)
>  #define GICR_PROPBASER_INNER_CACHEABILITY_SHIFT		(7)
>  #define GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT		(56)
> @@ -232,6 +234,7 @@
>  #define GITS_CTLR_QUIESCENT		(1U << 31)
>  
>  #define GITS_TYPER_PLPIS		(1UL << 0)
> +#define GITS_TYPER_ITT_ENTRY_SIZE_SHIFT	4
>  #define GITS_TYPER_IDBITS_SHIFT		8
>  #define GITS_TYPER_DEVBITS_SHIFT	13
>  #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
> @@ -290,6 +293,7 @@
>  #define GITS_BASER_TYPE(r)		(((r) >> GITS_BASER_TYPE_SHIFT) & 7)
>  #define GITS_BASER_ENTRY_SIZE_SHIFT		(48)
>  #define GITS_BASER_ENTRY_SIZE(r)	((((r) >> GITS_BASER_ENTRY_SIZE_SHIFT) & 0x1f) + 1)
> +#define GITS_BASER_ENTRY_SIZE_MASK	GENMASK_ULL(52, 48)
>  #define GITS_BASER_SHAREABILITY_SHIFT	(10)
>  #define GITS_BASER_InnerShareable					\
>  	GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 79ed1c2..69ecfe4 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -33,6 +33,12 @@
>  #include "vgic.h"
>  #include "vgic-mmio.h"
>  
> +/* Highest migration ABI revision supported by this code */
> +#define MAX_ABI_REV 0
> +
> +static int vgic_its_set_abi(struct vgic_its *its, int rev);
> +static const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its);
> +
>  /*
>   * Creates a new (reference to a) struct vgic_irq for a given LPI.
>   * If this LPI is already mapped on another ITS, we increase its refcount
> @@ -123,6 +129,15 @@ struct its_ite {
>  	u32 event_id;
>  };
>  
> +struct vgic_its_abi {
> +	int cte_esz;
> +	int dte_esz;
> +	int ite_esz;
> +	int (*save_tables)(struct vgic_its *its);
> +	int (*restore_tables)(struct vgic_its *its);
> +	int (*commit)(struct vgic_its *its);

Can you document what these functions do (especially the commit function
seems to be internal and not be related to the ABI as such).

> +};
> +
>  /*
>   * Find and returns a device in the device table for an ITS.
>   * Must be called with the its_lock mutex held.
> @@ -364,6 +379,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>  					      struct vgic_its *its,
>  					      gpa_t addr, unsigned int len)
>  {
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>  	u64 reg = GITS_TYPER_PLPIS;
>  
>  	/*
> @@ -376,6 +392,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>  	 */
>  	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>  	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
> +	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
>  
>  	return extract_bytes(reg, addr & 7, len);
>  }
> @@ -1268,6 +1285,7 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
>  				      gpa_t addr, unsigned int len,
>  				      unsigned long val)
>  {
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>  	u64 entry_size, device_type;
>  	u64 reg, *regptr, clearbits = 0;
>  
> @@ -1278,12 +1296,12 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
>  	switch (BASER_INDEX(addr)) {
>  	case 0:
>  		regptr = &its->baser_device_table;
> -		entry_size = 8;
> +		entry_size = abi->dte_esz;
>  		device_type = GITS_BASER_TYPE_DEVICE;
>  		break;
>  	case 1:
>  		regptr = &its->baser_coll_table;
> -		entry_size = 8;
> +		entry_size = abi->cte_esz;
>  		device_type = GITS_BASER_TYPE_COLLECTION;
>  		clearbits = GITS_BASER_INDIRECT;
>  		break;
> @@ -1425,7 +1443,6 @@ static int vgic_register_its_iodev(struct kvm *kvm, struct vgic_its *its)
>  	(GIC_BASER_CACHEABILITY(GITS_BASER, INNER, RaWb)		| \
>  	 GIC_BASER_CACHEABILITY(GITS_BASER, OUTER, SameAsInner)		| \
>  	 GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)		| \
> -	 ((8ULL - 1) << GITS_BASER_ENTRY_SIZE_SHIFT)			| \
>  	 GITS_BASER_PAGE_SIZE_64K)
>  
>  #define INITIAL_PROPBASER_VALUE						  \
> @@ -1465,7 +1482,7 @@ static int vgic_its_create(struct kvm_device *dev, u32 type)
>  
>  	dev->private = its;
>  
> -	return 0;
> +	return vgic_its_set_abi(its, MAX_ABI_REV);
>  }
>  
>  static void vgic_its_destroy(struct kvm_device *kvm_dev)
> @@ -1592,6 +1609,63 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> +/**
> + * vgic_its_save_tables_v0 - Save the ITS tables into guest ARM
> + * according to v0 ABI
> + */
> +static int vgic_its_save_tables_v0(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +/**
> + * vgic_its_restore_tables_v0 - Restore the ITS tables from guest RAM
> + * to internal data structs according to V0 ABI
> + *
> + */
> +static int vgic_its_restore_tables_v0(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +static int vgic_its_commit_v0(struct vgic_its *its)
> +{
> +	const struct vgic_its_abi *abi;
> +
> +	abi = vgic_its_get_abi(its);
> +	its->baser_coll_table &= ~GITS_BASER_ENTRY_SIZE_MASK;
> +	its->baser_device_table &= ~GITS_BASER_ENTRY_SIZE_MASK;
> +
> +	its->baser_coll_table |= (GIC_ENCODE_SZ(abi->cte_esz, 5)
> +					<< GITS_BASER_ENTRY_SIZE_SHIFT);
> +
> +	its->baser_device_table |= (GIC_ENCODE_SZ(abi->dte_esz, 5)
> +					<< GITS_BASER_ENTRY_SIZE_SHIFT);
> +	return 0;
> +}
> +
> +static const struct vgic_its_abi abi[MAX_ABI_REV + 1] = {

nit: can we name this its_table_abi_versions or something like that
which is slightly more descriptive?

> +	{.cte_esz = 8, .dte_esz = 8, .ite_esz = 8,
> +	 .save_tables = vgic_its_save_tables_v0,
> +	 .restore_tables = vgic_its_restore_tables_v0,
> +	 . commit = vgic_its_commit_v0,

stray whitespace

> +	},
> +};
> +
> +inline const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its)
> +{
> +	return &abi[its->abi_rev];
> +}
> +
> +int vgic_its_set_abi(struct vgic_its *its, int rev)
> +{
> +	const struct vgic_its_abi *abi;
> +
> +	its->abi_rev = rev;
> +	abi = vgic_its_get_abi(its);
> +	return abi->commit(its);
> +}
> +
>  static int vgic_its_has_attr(struct kvm_device *dev,
>  			     struct kvm_device_attr *attr)
>  {
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

[PATCH v5 09/22] KVM: arm64: vgic-its: Introduce migration ABI infrastructure

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:21PM +0200, Eric Auger wrote:
> We plan to support different migration ABIs, ie. characterizing
> the ITS table layout format in guest RAM. Typically a new ABI will

nit: s/Typically/For example,

> be needed if vLPIs get supported for nested use case.
> 
> So let's introduce an array of supported ABIs (at the moment a single
> ABI is supported though). The following characteristics are foreseen
> to vary with the ABI: size of table entries, save/restore operation.
> 
> By default the MAX_ABI_REV is applied on its creation. In subsequent
> patches we will introduce a way for the userspace to change the ABI
> in use.
> 
> The entry sizes now are set according to the ABI version and not
> hardcoded anymore.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> 
> v5: creation and squash KVM: arm64: ITS: Report the ITE size in
>     GITS_TYPER
> ---
>  include/kvm/arm_vgic.h             |  3 ++
>  include/linux/irqchip/arm-gic-v3.h |  4 ++
>  virt/kvm/arm/vgic/vgic-its.c       | 82 ++++++++++++++++++++++++++++++++++++--
>  3 files changed, 85 insertions(+), 4 deletions(-)
> 
> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
> index c0b3d99..285474a 100644
> --- a/include/kvm/arm_vgic.h
> +++ b/include/kvm/arm_vgic.h
> @@ -162,6 +162,9 @@ struct vgic_its {
>  	u32			creadr;
>  	u32			cwriter;
>  
> +	/* migration ABI revision in use */
> +	u32			abi_rev;
> +
>  	/* Protects the device and collection lists */
>  	struct mutex		its_lock;
>  	struct list_head	device_list;
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index 97cbca1..9648bad 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -132,6 +132,8 @@
>  #define GIC_BASER_SHAREABILITY(reg, type)				\
>  	(GIC_BASER_##type << reg##_SHAREABILITY_SHIFT)
>  
> +#define GIC_ENCODE_SZ(n, w) (((unsigned long)(n) - 1) & GENMASK_ULL(((w) - 1), 0))
> +

This is a very generic name.  Could you add a comment about what the
idea is?

>  #define GICR_PROPBASER_SHAREABILITY_SHIFT		(10)
>  #define GICR_PROPBASER_INNER_CACHEABILITY_SHIFT		(7)
>  #define GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT		(56)
> @@ -232,6 +234,7 @@
>  #define GITS_CTLR_QUIESCENT		(1U << 31)
>  
>  #define GITS_TYPER_PLPIS		(1UL << 0)
> +#define GITS_TYPER_ITT_ENTRY_SIZE_SHIFT	4
>  #define GITS_TYPER_IDBITS_SHIFT		8
>  #define GITS_TYPER_DEVBITS_SHIFT	13
>  #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
> @@ -290,6 +293,7 @@
>  #define GITS_BASER_TYPE(r)		(((r) >> GITS_BASER_TYPE_SHIFT) & 7)
>  #define GITS_BASER_ENTRY_SIZE_SHIFT		(48)
>  #define GITS_BASER_ENTRY_SIZE(r)	((((r) >> GITS_BASER_ENTRY_SIZE_SHIFT) & 0x1f) + 1)
> +#define GITS_BASER_ENTRY_SIZE_MASK	GENMASK_ULL(52, 48)
>  #define GITS_BASER_SHAREABILITY_SHIFT	(10)
>  #define GITS_BASER_InnerShareable					\
>  	GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 79ed1c2..69ecfe4 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -33,6 +33,12 @@
>  #include "vgic.h"
>  #include "vgic-mmio.h"
>  
> +/* Highest migration ABI revision supported by this code */
> +#define MAX_ABI_REV 0
> +
> +static int vgic_its_set_abi(struct vgic_its *its, int rev);
> +static const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its);
> +
>  /*
>   * Creates a new (reference to a) struct vgic_irq for a given LPI.
>   * If this LPI is already mapped on another ITS, we increase its refcount
> @@ -123,6 +129,15 @@ struct its_ite {
>  	u32 event_id;
>  };
>  
> +struct vgic_its_abi {
> +	int cte_esz;
> +	int dte_esz;
> +	int ite_esz;
> +	int (*save_tables)(struct vgic_its *its);
> +	int (*restore_tables)(struct vgic_its *its);
> +	int (*commit)(struct vgic_its *its);

Can you document what these functions do (especially the commit function
seems to be internal and not be related to the ABI as such).

> +};
> +
>  /*
>   * Find and returns a device in the device table for an ITS.
>   * Must be called with the its_lock mutex held.
> @@ -364,6 +379,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>  					      struct vgic_its *its,
>  					      gpa_t addr, unsigned int len)
>  {
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>  	u64 reg = GITS_TYPER_PLPIS;
>  
>  	/*
> @@ -376,6 +392,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>  	 */
>  	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>  	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
> +	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
>  
>  	return extract_bytes(reg, addr & 7, len);
>  }
> @@ -1268,6 +1285,7 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
>  				      gpa_t addr, unsigned int len,
>  				      unsigned long val)
>  {
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>  	u64 entry_size, device_type;
>  	u64 reg, *regptr, clearbits = 0;
>  
> @@ -1278,12 +1296,12 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
>  	switch (BASER_INDEX(addr)) {
>  	case 0:
>  		regptr = &its->baser_device_table;
> -		entry_size = 8;
> +		entry_size = abi->dte_esz;
>  		device_type = GITS_BASER_TYPE_DEVICE;
>  		break;
>  	case 1:
>  		regptr = &its->baser_coll_table;
> -		entry_size = 8;
> +		entry_size = abi->cte_esz;
>  		device_type = GITS_BASER_TYPE_COLLECTION;
>  		clearbits = GITS_BASER_INDIRECT;
>  		break;
> @@ -1425,7 +1443,6 @@ static int vgic_register_its_iodev(struct kvm *kvm, struct vgic_its *its)
>  	(GIC_BASER_CACHEABILITY(GITS_BASER, INNER, RaWb)		| \
>  	 GIC_BASER_CACHEABILITY(GITS_BASER, OUTER, SameAsInner)		| \
>  	 GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)		| \
> -	 ((8ULL - 1) << GITS_BASER_ENTRY_SIZE_SHIFT)			| \
>  	 GITS_BASER_PAGE_SIZE_64K)
>  
>  #define INITIAL_PROPBASER_VALUE						  \
> @@ -1465,7 +1482,7 @@ static int vgic_its_create(struct kvm_device *dev, u32 type)
>  
>  	dev->private = its;
>  
> -	return 0;
> +	return vgic_its_set_abi(its, MAX_ABI_REV);
>  }
>  
>  static void vgic_its_destroy(struct kvm_device *kvm_dev)
> @@ -1592,6 +1609,63 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> +/**
> + * vgic_its_save_tables_v0 - Save the ITS tables into guest ARM
> + * according to v0 ABI
> + */
> +static int vgic_its_save_tables_v0(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +/**
> + * vgic_its_restore_tables_v0 - Restore the ITS tables from guest RAM
> + * to internal data structs according to V0 ABI
> + *
> + */
> +static int vgic_its_restore_tables_v0(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +static int vgic_its_commit_v0(struct vgic_its *its)
> +{
> +	const struct vgic_its_abi *abi;
> +
> +	abi = vgic_its_get_abi(its);
> +	its->baser_coll_table &= ~GITS_BASER_ENTRY_SIZE_MASK;
> +	its->baser_device_table &= ~GITS_BASER_ENTRY_SIZE_MASK;
> +
> +	its->baser_coll_table |= (GIC_ENCODE_SZ(abi->cte_esz, 5)
> +					<< GITS_BASER_ENTRY_SIZE_SHIFT);
> +
> +	its->baser_device_table |= (GIC_ENCODE_SZ(abi->dte_esz, 5)
> +					<< GITS_BASER_ENTRY_SIZE_SHIFT);
> +	return 0;
> +}
> +
> +static const struct vgic_its_abi abi[MAX_ABI_REV + 1] = {

nit: can we name this its_table_abi_versions or something like that
which is slightly more descriptive?

> +	{.cte_esz = 8, .dte_esz = 8, .ite_esz = 8,
> +	 .save_tables = vgic_its_save_tables_v0,
> +	 .restore_tables = vgic_its_restore_tables_v0,
> +	 . commit = vgic_its_commit_v0,

stray whitespace

> +	},
> +};
> +
> +inline const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its)
> +{
> +	return &abi[its->abi_rev];
> +}
> +
> +int vgic_its_set_abi(struct vgic_its *its, int rev)
> +{
> +	const struct vgic_its_abi *abi;
> +
> +	its->abi_rev = rev;
> +	abi = vgic_its_get_abi(its);
> +	return abi->commit(its);
> +}
> +
>  static int vgic_its_has_attr(struct kvm_device *dev,
>  			     struct kvm_device_attr *attr)
>  {
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

Hi Eric,

On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
> On 27/04/2017 13:02, Christoffer Dall wrote:
> > On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> >> On 27/04/2017 10:57, Christoffer Dall wrote:
> >>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >>>> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>>>> Add description for how to access ITS registers and how to save/restore
> >>>>>> ITS tables into/from memory.
> >>>>>>
> >>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>
> >>>>>> ---
> >>>>>> v4 -> v5:
> >>>>>> - take into account Christoffer's comments
> >>>>>> - pending table save on GICV3 side now
> >>>>>>
> >>>>>> v3 -> v4:
> >>>>>> - take into account Peter's comments:
> >>>>>>   - typos
> >>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>>>   - add a validity bit in DTE
> >>>>>>   - document all fields in CTE and ITE
> >>>>>>   - document ABI revision
> >>>>>> - take into account Andre's comments:
> >>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>>>   - document 64b registers only can be accessed with 64b access
> >>>>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>>>
> >>>>>> v1 -> v2:
> >>>>>> - DTE and ITE now are 8 bytes
> >>>>>> - DTE and ITE now indexed by deviceid/eventid
> >>>>>> - use ITE name instead of ITTE
> >>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>>>> - mentions LE layout
> >>>>>> ---
> >>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>>>  1 file changed, 99 insertions(+)
> >>>>>>
> >>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>> index 6081a5b..b5f010d 100644
> >>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>> @@ -32,7 +32,106 @@ Groups:
> >>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>>>        request the initialization of the ITS, no additional parameter in
> >>>>>>        kvm_device_attr.addr.
> >>>>>> +
> >>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>>>> +      by the guest in corresponding registers/table entries.
> >>>>>> +
> >>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>>>> +      are laid out in little endian format as described in the last paragraph.
> >>>>>> +
> >>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>>>> +
> >>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>>>> +
> >>>>>> +      The GITS_IIDR read-only register must also be restored before
> >>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>>>> +
> >>>>>
> >>>>> what is the expected sequence of operations.  For example, to restore
> >>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >>>> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>>>
> >>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>> and restore GITS_CTLR (which enables the ITS)?
> >>>>
> >>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >>>> that the pending table is read. But the whole pending table is not read
> >>>> as we only iterate on registered LPIs. So the ITT must have been
> >>>> restored previously.
> >>>>
> >>>> I became aware that the pending table sync is done twice, once in the
> >>>> pending table restore,  and once in the GITS_CTLR restore. So if we
> >>>> leave this order specification, I should be able to remove the sync on
> >>>> table restore. This was the original reason why GITS_CTLR restore has
> >>>> been done at the very end.
> >>>
> >>> I'm sorry, I'm a bit confused.  Do we not need
> >>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> >>
> >> Yes you do. I was talking about the RDIST pending table sync. The save
> >> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> >> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> >> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> >> which is not requested I think since GITS_CTLR restore does it already.
> > 
> > Shouldn't restoring the pending tables happen when restoring some
> > redeistributor state and not anything related to the ITS?
> 
> Marc wrote:
> "
> I don't think you necessarily need a coarse map. When restoring the ITS
> tables, you can always read the pending bit when creating the LPI
> structure (it has been written to RAM at save time). Note that we
> already do something like this in vgic_enable_lpis().
> "
> 
> This is currently what is implemented I think. the pending tables are
> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
> also on on ITS table restore
> 
> The problematic is: Either you know in advance which LPI INTIDare used
> or you need to parse the whole pending table (possibly using the 1st kB
> as coarse mapping).
> 
> If you don't know the LPI INTIDs in advance it is only possible to
> restore the pending bit of pending LPIs. At that time you would
> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
> ITS ITT you would do the same for those which were not pending. Looks
> really heavy to me: coarse mapping + dual vgic_add_lpi path.
> 
> Otherwise we would need to add another dependency between RDIST pending
> table restore and ITS table restore but this looks even more weird, no?
> 
> 
So I just sat down with Andre and Marc and we tried to work through this
and came up with the best scheme.  I apologize in advance for the
one-way nature of this e-mail, and I am of course open to discussing the
following proposal again if you do not agree.

What I think this document should say, is that the following ordering
must be followed when restoring the GIC and the ITS:

  First, restore all guest memory

  Second, restore ALL redistributors

  Third, restore the ITS, in the following order:
    1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
    2. Restore GITS_CBASER
    3. Restore all other GITS_ registers, except GITS_CTLR!
    4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
    5. Restore GITS_CTLR

The rationale is that we really want the redistributor and the ITS
restore to be independent and follow the architecture.  This means that
our ABI for the redistributor should still work without restoring an ITS
(if we ever decide to support LPIs for KVM without the ITS).

In terms of our current implementation this means that vgic_add_lpi()
should ask the redistributor what the state of the LPI is (priority,
enabled, pending).  I suggest you do the pending check by adding a
function called something like vgic_v3_lpi_is_pending() which scans the
bit in memory, clears the memory bit, and returns the value.  Clearing
the pending bit in memory when moving it to the struct irq is nice,
because you then don't have to clear out the entire pending table later
and we don't keep 'consumed' data lying around.  This change should be
implemented in its_sync_lpi_pending_table() as well, but note that you
need never call that function in the normal restore path using this
design.

I hope this makes sense.

Thanks,
-Christoffer

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

Hi Eric,

On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
> On 27/04/2017 13:02, Christoffer Dall wrote:
> > On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> >> On 27/04/2017 10:57, Christoffer Dall wrote:
> >>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >>>> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>>>> Add description for how to access ITS registers and how to save/restore
> >>>>>> ITS tables into/from memory.
> >>>>>>
> >>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>
> >>>>>> ---
> >>>>>> v4 -> v5:
> >>>>>> - take into account Christoffer's comments
> >>>>>> - pending table save on GICV3 side now
> >>>>>>
> >>>>>> v3 -> v4:
> >>>>>> - take into account Peter's comments:
> >>>>>>   - typos
> >>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>>>   - add a validity bit in DTE
> >>>>>>   - document all fields in CTE and ITE
> >>>>>>   - document ABI revision
> >>>>>> - take into account Andre's comments:
> >>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>>>   - document 64b registers only can be accessed with 64b access
> >>>>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>>>
> >>>>>> v1 -> v2:
> >>>>>> - DTE and ITE now are 8 bytes
> >>>>>> - DTE and ITE now indexed by deviceid/eventid
> >>>>>> - use ITE name instead of ITTE
> >>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>>>> - mentions LE layout
> >>>>>> ---
> >>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>>>  1 file changed, 99 insertions(+)
> >>>>>>
> >>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>> index 6081a5b..b5f010d 100644
> >>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>> @@ -32,7 +32,106 @@ Groups:
> >>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>>>        request the initialization of the ITS, no additional parameter in
> >>>>>>        kvm_device_attr.addr.
> >>>>>> +
> >>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>>>> +      by the guest in corresponding registers/table entries.
> >>>>>> +
> >>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>>>> +      are laid out in little endian format as described in the last paragraph.
> >>>>>> +
> >>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>>>> +
> >>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>>>> +
> >>>>>> +      The GITS_IIDR read-only register must also be restored before
> >>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>>>> +
> >>>>>
> >>>>> what is the expected sequence of operations.  For example, to restore
> >>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >>>> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>>>
> >>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>> and restore GITS_CTLR (which enables the ITS)?
> >>>>
> >>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >>>> that the pending table is read. But the whole pending table is not read
> >>>> as we only iterate on registered LPIs. So the ITT must have been
> >>>> restored previously.
> >>>>
> >>>> I became aware that the pending table sync is done twice, once in the
> >>>> pending table restore,  and once in the GITS_CTLR restore. So if we
> >>>> leave this order specification, I should be able to remove the sync on
> >>>> table restore. This was the original reason why GITS_CTLR restore has
> >>>> been done at the very end.
> >>>
> >>> I'm sorry, I'm a bit confused.  Do we not need
> >>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> >>
> >> Yes you do. I was talking about the RDIST pending table sync. The save
> >> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> >> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> >> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> >> which is not requested I think since GITS_CTLR restore does it already.
> > 
> > Shouldn't restoring the pending tables happen when restoring some
> > redeistributor state and not anything related to the ITS?
> 
> Marc wrote:
> "
> I don't think you necessarily need a coarse map. When restoring the ITS
> tables, you can always read the pending bit when creating the LPI
> structure (it has been written to RAM at save time). Note that we
> already do something like this in vgic_enable_lpis().
> "
> 
> This is currently what is implemented I think. the pending tables are
> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
> also on on ITS table restore
> 
> The problematic is: Either you know in advance which LPI INTIDare used
> or you need to parse the whole pending table (possibly using the 1st kB
> as coarse mapping).
> 
> If you don't know the LPI INTIDs in advance it is only possible to
> restore the pending bit of pending LPIs. At that time you would
> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
> ITS ITT you would do the same for those which were not pending. Looks
> really heavy to me: coarse mapping + dual vgic_add_lpi path.
> 
> Otherwise we would need to add another dependency between RDIST pending
> table restore and ITS table restore but this looks even more weird, no?
> 
> 
So I just sat down with Andre and Marc and we tried to work through this
and came up with the best scheme.  I apologize in advance for the
one-way nature of this e-mail, and I am of course open to discussing the
following proposal again if you do not agree.

What I think this document should say, is that the following ordering
must be followed when restoring the GIC and the ITS:

  First, restore all guest memory

  Second, restore ALL redistributors

  Third, restore the ITS, in the following order:
    1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
    2. Restore GITS_CBASER
    3. Restore all other GITS_ registers, except GITS_CTLR!
    4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
    5. Restore GITS_CTLR

The rationale is that we really want the redistributor and the ITS
restore to be independent and follow the architecture.  This means that
our ABI for the redistributor should still work without restoring an ITS
(if we ever decide to support LPIs for KVM without the ITS).

In terms of our current implementation this means that vgic_add_lpi()
should ask the redistributor what the state of the LPI is (priority,
enabled, pending).  I suggest you do the pending check by adding a
function called something like vgic_v3_lpi_is_pending() which scans the
bit in memory, clears the memory bit, and returns the value.  Clearing
the pending bit in memory when moving it to the struct irq is nice,
because you then don't have to clear out the entire pending table later
and we don't keep 'consumed' data lying around.  This change should be
implemented in its_sync_lpi_pending_table() as well, but note that you
need never call that function in the normal restore path using this
design.

I hope this makes sense.

Thanks,
-Christoffer

Re: [PATCH v5 10/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_iidr

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:22PM +0200, Eric Auger wrote:
> The GITS_IIDR revision field is used to encode the migration ABI
> revision. So we need to restore it to check the table layout is
> readable by the destination.
> 
> By writing the IIDR, userspace thus force the ABI revision to be

forces

> used and this msut be less or equal than the max revision KVM supports.

must

less than or equal to

> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5
> - rename user_revision into abi_rev and REV into MAX_ABI_REV
> - IIDR reports abi_rev set by userspace if any.
> - If value set by userspace exceeds the max supported revision, an
>   error is reported.
> - add some defines
> 
> v4: creation
> ---
>  include/linux/irqchip/arm-gic-v3.h |  4 ++++
>  virt/kvm/arm/vgic/vgic-its.c       | 23 ++++++++++++++++++++---
>  2 files changed, 24 insertions(+), 3 deletions(-)
> 
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index 9648bad..54c20bd 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -241,6 +241,10 @@
>  #define GITS_TYPER_PTA			(1UL << 19)
>  #define GITS_TYPER_HWCOLLCNT_SHIFT	24
>  
> +#define GITS_IIDR_REV_SHIFT		12
> +#define GITS_IIDR_REV(r)		(((r) >> GITS_IIDR_REV_SHIFT) & 0xf)
> +#define GITS_IIDR_PRODUCTID_SHIFT	24
> +
>  #define GITS_CBASER_VALID			(1ULL << 63)
>  #define GITS_CBASER_SHAREABILITY_SHIFT		(10)
>  #define GITS_CBASER_INNER_CACHEABILITY_SHIFT	(59)
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 69ecfe4..1b5797e 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -401,7 +401,23 @@ static unsigned long vgic_mmio_read_its_iidr(struct kvm *kvm,
>  					     struct vgic_its *its,
>  					     gpa_t addr, unsigned int len)
>  {
> -	return (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
> +	return (PRODUCT_ID_KVM << GITS_IIDR_PRODUCTID_SHIFT) |
> +	       (its->abi_rev << GITS_IIDR_REV_SHIFT) | IMPLEMENTER_ARM;
> +}
> +
> +static int vgic_mmio_uaccess_write_its_iidr(struct kvm *kvm,
> +					    struct vgic_its *its,
> +					    gpa_t addr, unsigned int len,
> +					    unsigned long val)
> +{
> +	u64 tmp = 0;
> +
> +	tmp = update_64bit_reg(tmp, addr & 3, len, val);

but the IIDR is not a 64bit register?

> +	tmp = GITS_IIDR_REV(tmp);

You should be able to call rev = GITS_IIDR_REV(val) and be done.

> +
> +	if (tmp > MAX_ABI_REV)
> +		return -EINVAL;
> +	return vgic_its_set_abi(its, tmp);
>  }
>  
>  static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,
> @@ -1382,8 +1398,9 @@ static struct vgic_register_region its_registers[] = {
>  	REGISTER_ITS_DESC(GITS_CTLR,
>  		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4,
>  		VGIC_ACCESS_32bit),
> -	REGISTER_ITS_DESC(GITS_IIDR,
> -		vgic_mmio_read_its_iidr, its_mmio_write_wi, 4,
> +	REGISTER_ITS_DESC_UACCESS(GITS_IIDR,
> +		vgic_mmio_read_its_iidr, its_mmio_write_wi,
> +		vgic_mmio_uaccess_write_its_iidr, 4,
>  		VGIC_ACCESS_32bit),
>  	REGISTER_ITS_DESC(GITS_TYPER,
>  		vgic_mmio_read_its_typer, its_mmio_write_wi, 8,
> -- 
> 2.5.5
> 

Besides these smaller details, this looks ok to me.

Thanks,
-Christoffer

[PATCH v5 10/22] KVM: arm64: vgic-its: Implement vgic_mmio_uaccess_write_its_iidr

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:22PM +0200, Eric Auger wrote:
> The GITS_IIDR revision field is used to encode the migration ABI
> revision. So we need to restore it to check the table layout is
> readable by the destination.
> 
> By writing the IIDR, userspace thus force the ABI revision to be

forces

> used and this msut be less or equal than the max revision KVM supports.

must

less than or equal to

> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5
> - rename user_revision into abi_rev and REV into MAX_ABI_REV
> - IIDR reports abi_rev set by userspace if any.
> - If value set by userspace exceeds the max supported revision, an
>   error is reported.
> - add some defines
> 
> v4: creation
> ---
>  include/linux/irqchip/arm-gic-v3.h |  4 ++++
>  virt/kvm/arm/vgic/vgic-its.c       | 23 ++++++++++++++++++++---
>  2 files changed, 24 insertions(+), 3 deletions(-)
> 
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index 9648bad..54c20bd 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -241,6 +241,10 @@
>  #define GITS_TYPER_PTA			(1UL << 19)
>  #define GITS_TYPER_HWCOLLCNT_SHIFT	24
>  
> +#define GITS_IIDR_REV_SHIFT		12
> +#define GITS_IIDR_REV(r)		(((r) >> GITS_IIDR_REV_SHIFT) & 0xf)
> +#define GITS_IIDR_PRODUCTID_SHIFT	24
> +
>  #define GITS_CBASER_VALID			(1ULL << 63)
>  #define GITS_CBASER_SHAREABILITY_SHIFT		(10)
>  #define GITS_CBASER_INNER_CACHEABILITY_SHIFT	(59)
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 69ecfe4..1b5797e 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -401,7 +401,23 @@ static unsigned long vgic_mmio_read_its_iidr(struct kvm *kvm,
>  					     struct vgic_its *its,
>  					     gpa_t addr, unsigned int len)
>  {
> -	return (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
> +	return (PRODUCT_ID_KVM << GITS_IIDR_PRODUCTID_SHIFT) |
> +	       (its->abi_rev << GITS_IIDR_REV_SHIFT) | IMPLEMENTER_ARM;
> +}
> +
> +static int vgic_mmio_uaccess_write_its_iidr(struct kvm *kvm,
> +					    struct vgic_its *its,
> +					    gpa_t addr, unsigned int len,
> +					    unsigned long val)
> +{
> +	u64 tmp = 0;
> +
> +	tmp = update_64bit_reg(tmp, addr & 3, len, val);

but the IIDR is not a 64bit register?

> +	tmp = GITS_IIDR_REV(tmp);

You should be able to call rev = GITS_IIDR_REV(val) and be done.

> +
> +	if (tmp > MAX_ABI_REV)
> +		return -EINVAL;
> +	return vgic_its_set_abi(its, tmp);
>  }
>  
>  static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,
> @@ -1382,8 +1398,9 @@ static struct vgic_register_region its_registers[] = {
>  	REGISTER_ITS_DESC(GITS_CTLR,
>  		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4,
>  		VGIC_ACCESS_32bit),
> -	REGISTER_ITS_DESC(GITS_IIDR,
> -		vgic_mmio_read_its_iidr, its_mmio_write_wi, 4,
> +	REGISTER_ITS_DESC_UACCESS(GITS_IIDR,
> +		vgic_mmio_read_its_iidr, its_mmio_write_wi,
> +		vgic_mmio_uaccess_write_its_iidr, 4,
>  		VGIC_ACCESS_32bit),
>  	REGISTER_ITS_DESC(GITS_TYPER,
>  		vgic_mmio_read_its_typer, its_mmio_write_wi, 8,
> -- 
> 2.5.5
> 

Besides these smaller details, this looks ok to me.

Thanks,
-Christoffer

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

On 27/04/2017 16:45, Christoffer Dall wrote:
> Hi Eric,
> 
> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>> ITS tables into/from memory.
>>>>>>>>
>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> v4 -> v5:
>>>>>>>> - take into account Christoffer's comments
>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>
>>>>>>>> v3 -> v4:
>>>>>>>> - take into account Peter's comments:
>>>>>>>>   - typos
>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>   - add a validity bit in DTE
>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>   - document ABI revision
>>>>>>>> - take into account Andre's comments:
>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>
>>>>>>>> v1 -> v2:
>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>> - use ITE name instead of ITTE
>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>> - mentions LE layout
>>>>>>>> ---
>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>        kvm_device_attr.addr.
>>>>>>>> +
>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>> +
>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>> +
>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>> +
>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>> +
>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>> +
>>>>>>>
>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>
>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>
>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>> restored previously.
>>>>>>
>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>> been done at the very end.
>>>>>
>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>
>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>
>>> Shouldn't restoring the pending tables happen when restoring some
>>> redeistributor state and not anything related to the ITS?
>>
>> Marc wrote:
>> "
>> I don't think you necessarily need a coarse map. When restoring the ITS
>> tables, you can always read the pending bit when creating the LPI
>> structure (it has been written to RAM at save time). Note that we
>> already do something like this in vgic_enable_lpis().
>> "
>>
>> This is currently what is implemented I think. the pending tables are
>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>> also on on ITS table restore
>>
>> The problematic is: Either you know in advance which LPI INTIDare used
>> or you need to parse the whole pending table (possibly using the 1st kB
>> as coarse mapping).
>>
>> If you don't know the LPI INTIDs in advance it is only possible to
>> restore the pending bit of pending LPIs. At that time you would
>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>> ITS ITT you would do the same for those which were not pending. Looks
>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>
>> Otherwise we would need to add another dependency between RDIST pending
>> table restore and ITS table restore but this looks even more weird, no?
>>
>>
> So I just sat down with Andre and Marc and we tried to work through this
> and came up with the best scheme.  I apologize in advance for the
> one-way nature of this e-mail, and I am of course open to discussing the
> following proposal again if you do not agree.
> 
> What I think this document should say, is that the following ordering
> must be followed when restoring the GIC and the ITS:
> 
>   First, restore all guest memory
> 
>   Second, restore ALL redistributors
> 
>   Third, restore the ITS, in the following order:
>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>     2. Restore GITS_CBASER
>     3. Restore all other GITS_ registers, except GITS_CTLR!
>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>     5. Restore GITS_CTLR
> 
> The rationale is that we really want the redistributor and the ITS
> restore to be independent and follow the architecture.  This means that
> our ABI for the redistributor should still work without restoring an ITS
> (if we ever decide to support LPIs for KVM without the ITS).

OK. Note I already mentioned that GICv3 must be restored before the ITS.
To me this comprised the RDIST.

I understand the above description of the ordering comes in addition to
the existing text, right? in other words I keep the GITS_READR,
GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
section.

> 
> In terms of our current implementation this means that vgic_add_lpi()
> should ask the redistributor what the state of the LPI is (priority,
> enabled, pending).
this practically means I move update_lpi_config call from
vgic_its_restore_ite to vgic_add_lpi(). OK

However for getting the LPI pending state I must know which RDIST the
LPI is attached to. This is not known at LPI allocation time. Do I
misunderstand something?

Thanks

Eric

  I suggest you do the pending check by adding a
> function called something like vgic_v3_lpi_is_pending() which scans the
> bit in memory, clears the memory bit, and returns the value.  Clearing
> the pending bit in memory when moving it to the struct irq is nice,
> because you then don't have to clear out the entire pending table later
> and we don't keep 'consumed' data lying around.  This change should be
> implemented in its_sync_lpi_pending_table() as well, but note that you
> need never call that function in the normal restore path using this
> design.
> 
> I hope this makes sense.
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

On 27/04/2017 16:45, Christoffer Dall wrote:
> Hi Eric,
> 
> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>> ITS tables into/from memory.
>>>>>>>>
>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> v4 -> v5:
>>>>>>>> - take into account Christoffer's comments
>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>
>>>>>>>> v3 -> v4:
>>>>>>>> - take into account Peter's comments:
>>>>>>>>   - typos
>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>   - add a validity bit in DTE
>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>   - document ABI revision
>>>>>>>> - take into account Andre's comments:
>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>
>>>>>>>> v1 -> v2:
>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>> - use ITE name instead of ITTE
>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>> - mentions LE layout
>>>>>>>> ---
>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>        kvm_device_attr.addr.
>>>>>>>> +
>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>> +
>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>> +
>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>> +
>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>> +
>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>> +
>>>>>>>
>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>
>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>
>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>> restored previously.
>>>>>>
>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>> been done at the very end.
>>>>>
>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>
>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>
>>> Shouldn't restoring the pending tables happen when restoring some
>>> redeistributor state and not anything related to the ITS?
>>
>> Marc wrote:
>> "
>> I don't think you necessarily need a coarse map. When restoring the ITS
>> tables, you can always read the pending bit when creating the LPI
>> structure (it has been written to RAM at save time). Note that we
>> already do something like this in vgic_enable_lpis().
>> "
>>
>> This is currently what is implemented I think. the pending tables are
>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>> also on on ITS table restore
>>
>> The problematic is: Either you know in advance which LPI INTIDare used
>> or you need to parse the whole pending table (possibly using the 1st kB
>> as coarse mapping).
>>
>> If you don't know the LPI INTIDs in advance it is only possible to
>> restore the pending bit of pending LPIs. At that time you would
>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>> ITS ITT you would do the same for those which were not pending. Looks
>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>
>> Otherwise we would need to add another dependency between RDIST pending
>> table restore and ITS table restore but this looks even more weird, no?
>>
>>
> So I just sat down with Andre and Marc and we tried to work through this
> and came up with the best scheme.  I apologize in advance for the
> one-way nature of this e-mail, and I am of course open to discussing the
> following proposal again if you do not agree.
> 
> What I think this document should say, is that the following ordering
> must be followed when restoring the GIC and the ITS:
> 
>   First, restore all guest memory
> 
>   Second, restore ALL redistributors
> 
>   Third, restore the ITS, in the following order:
>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>     2. Restore GITS_CBASER
>     3. Restore all other GITS_ registers, except GITS_CTLR!
>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>     5. Restore GITS_CTLR
> 
> The rationale is that we really want the redistributor and the ITS
> restore to be independent and follow the architecture.  This means that
> our ABI for the redistributor should still work without restoring an ITS
> (if we ever decide to support LPIs for KVM without the ITS).

OK. Note I already mentioned that GICv3 must be restored before the ITS.
To me this comprised the RDIST.

I understand the above description of the ordering comes in addition to
the existing text, right? in other words I keep the GITS_READR,
GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
section.

> 
> In terms of our current implementation this means that vgic_add_lpi()
> should ask the redistributor what the state of the LPI is (priority,
> enabled, pending).
this practically means I move update_lpi_config call from
vgic_its_restore_ite to vgic_add_lpi(). OK

However for getting the LPI pending state I must know which RDIST the
LPI is attached to. This is not known at LPI allocation time. Do I
misunderstand something?

Thanks

Eric

  I suggest you do the pending check by adding a
> function called something like vgic_v3_lpi_is_pending() which scans the
> bit in memory, clears the memory bit, and returns the value.  Clearing
> the pending bit in memory when moving it to the struct irq is nice,
> because you then don't have to clear out the entire pending table later
> and we don't keep 'consumed' data lying around.  This change should be
> implemented in its_sync_lpi_pending_table() as well, but note that you
> need never call that function in the normal restore path using this
> design.
> 
> I hope this makes sense.
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Marc Zyngier]

On 27/04/17 16:29, Auger Eric wrote:
> 
> 
> On 27/04/2017 16:45, Christoffer Dall wrote:
>> Hi Eric,
>>
>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> v4 -> v5:
>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>
>>>>>>>>> v3 -> v4:
>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>   - typos
>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>   - document ABI revision
>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>
>>>>>>>>> v1 -> v2:
>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>> - mentions LE layout
>>>>>>>>> ---
>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>> +
>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>> +
>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>> +
>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>> +
>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>> +
>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>> +
>>>>>>>>
>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>
>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>
>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>> restored previously.
>>>>>>>
>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>> been done at the very end.
>>>>>>
>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>
>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>
>>>> Shouldn't restoring the pending tables happen when restoring some
>>>> redeistributor state and not anything related to the ITS?
>>>
>>> Marc wrote:
>>> "
>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>> tables, you can always read the pending bit when creating the LPI
>>> structure (it has been written to RAM at save time). Note that we
>>> already do something like this in vgic_enable_lpis().
>>> "
>>>
>>> This is currently what is implemented I think. the pending tables are
>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>> also on on ITS table restore
>>>
>>> The problematic is: Either you know in advance which LPI INTIDare used
>>> or you need to parse the whole pending table (possibly using the 1st kB
>>> as coarse mapping).
>>>
>>> If you don't know the LPI INTIDs in advance it is only possible to
>>> restore the pending bit of pending LPIs. At that time you would
>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>> ITS ITT you would do the same for those which were not pending. Looks
>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>
>>> Otherwise we would need to add another dependency between RDIST pending
>>> table restore and ITS table restore but this looks even more weird, no?
>>>
>>>
>> So I just sat down with Andre and Marc and we tried to work through this
>> and came up with the best scheme.  I apologize in advance for the
>> one-way nature of this e-mail, and I am of course open to discussing the
>> following proposal again if you do not agree.
>>
>> What I think this document should say, is that the following ordering
>> must be followed when restoring the GIC and the ITS:
>>
>>   First, restore all guest memory
>>
>>   Second, restore ALL redistributors
>>
>>   Third, restore the ITS, in the following order:
>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>     2. Restore GITS_CBASER
>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>     5. Restore GITS_CTLR
>>
>> The rationale is that we really want the redistributor and the ITS
>> restore to be independent and follow the architecture.  This means that
>> our ABI for the redistributor should still work without restoring an ITS
>> (if we ever decide to support LPIs for KVM without the ITS).
> 
> OK. Note I already mentioned that GICv3 must be restored before the ITS.
> To me this comprised the RDIST.
> 
> I understand the above description of the ordering comes in addition to
> the existing text, right? in other words I keep the GITS_READR,
> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
> section.
> 
>>
>> In terms of our current implementation this means that vgic_add_lpi()
>> should ask the redistributor what the state of the LPI is (priority,
>> enabled, pending).
> this practically means I move update_lpi_config call from
> vgic_its_restore_ite to vgic_add_lpi(). OK
> 
> However for getting the LPI pending state I must know which RDIST the
> LPI is attached to. This is not known at LPI allocation time. Do I
> misunderstand something?

Once you have rebuilt the ITS data structures and allocated the IRQ
structures, you should have a target_cpu field pointing to the right
vcpu. From there, you can surely find the corresponding redistributor
and the pending table.

BTW, we should document the fact that vcpus must have been created
before reloading the ITS (that's not completely obvious).

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Marc Zyngier]

On 27/04/17 16:29, Auger Eric wrote:
> 
> 
> On 27/04/2017 16:45, Christoffer Dall wrote:
>> Hi Eric,
>>
>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> v4 -> v5:
>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>
>>>>>>>>> v3 -> v4:
>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>   - typos
>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>   - document ABI revision
>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>
>>>>>>>>> v1 -> v2:
>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>> - mentions LE layout
>>>>>>>>> ---
>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>> +
>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>> +
>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>> +
>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>> +
>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>> +
>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>> +
>>>>>>>>
>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>
>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>
>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>> restored previously.
>>>>>>>
>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>> been done at the very end.
>>>>>>
>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>
>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>
>>>> Shouldn't restoring the pending tables happen when restoring some
>>>> redeistributor state and not anything related to the ITS?
>>>
>>> Marc wrote:
>>> "
>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>> tables, you can always read the pending bit when creating the LPI
>>> structure (it has been written to RAM at save time). Note that we
>>> already do something like this in vgic_enable_lpis().
>>> "
>>>
>>> This is currently what is implemented I think. the pending tables are
>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>> also on on ITS table restore
>>>
>>> The problematic is: Either you know in advance which LPI INTIDare used
>>> or you need to parse the whole pending table (possibly using the 1st kB
>>> as coarse mapping).
>>>
>>> If you don't know the LPI INTIDs in advance it is only possible to
>>> restore the pending bit of pending LPIs. At that time you would
>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>> ITS ITT you would do the same for those which were not pending. Looks
>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>
>>> Otherwise we would need to add another dependency between RDIST pending
>>> table restore and ITS table restore but this looks even more weird, no?
>>>
>>>
>> So I just sat down with Andre and Marc and we tried to work through this
>> and came up with the best scheme.  I apologize in advance for the
>> one-way nature of this e-mail, and I am of course open to discussing the
>> following proposal again if you do not agree.
>>
>> What I think this document should say, is that the following ordering
>> must be followed when restoring the GIC and the ITS:
>>
>>   First, restore all guest memory
>>
>>   Second, restore ALL redistributors
>>
>>   Third, restore the ITS, in the following order:
>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>     2. Restore GITS_CBASER
>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>     5. Restore GITS_CTLR
>>
>> The rationale is that we really want the redistributor and the ITS
>> restore to be independent and follow the architecture.  This means that
>> our ABI for the redistributor should still work without restoring an ITS
>> (if we ever decide to support LPIs for KVM without the ITS).
> 
> OK. Note I already mentioned that GICv3 must be restored before the ITS.
> To me this comprised the RDIST.
> 
> I understand the above description of the ordering comes in addition to
> the existing text, right? in other words I keep the GITS_READR,
> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
> section.
> 
>>
>> In terms of our current implementation this means that vgic_add_lpi()
>> should ask the redistributor what the state of the LPI is (priority,
>> enabled, pending).
> this practically means I move update_lpi_config call from
> vgic_its_restore_ite to vgic_add_lpi(). OK
> 
> However for getting the LPI pending state I must know which RDIST the
> LPI is attached to. This is not known at LPI allocation time. Do I
> misunderstand something?

Once you have rebuilt the ITS data structures and allocated the IRQ
structures, you should have a target_cpu field pointing to the right
vcpu. From there, you can surely find the corresponding redistributor
and the pending table.

BTW, we should document the fact that vcpus must have been created
before reloading the ITS (that's not completely obvious).

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH v5 11/22] KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:23PM +0200, Eric Auger wrote:
> Up to now the MAPD's ITT size field has been ignored. It encodes
> the number of eventid bit minus 1. It should be used to check
> the eventid when a MAPTI command is issued on a device. Let's
> store the number of eventid bits in the its_device and do the
> check on MAPTI. Also make sure the ITT size field does
> not exceed the GITS_TYPER IDBITS field.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - its_cmd_get_size macro now returns the actual number of eventid bits
> - use GIC_ENCODE_SZ macro to encode ID_bits
> 
> v3 -> v4:
> - VITS_TYPER_IDBITS set to 16 to be homogeneous with VITS_ESZ definition
>   and correct users
> - nb_eventid_bits correspond to the actual number of eventid bits
> ---
>  include/linux/irqchip/arm-gic-v3.h |  2 ++
>  virt/kvm/arm/vgic/vgic-its.c       | 15 ++++++++++++++-
>  2 files changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index 54c20bd..0c6798c 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -345,9 +345,11 @@
>  #define E_ITS_INT_UNMAPPED_INTERRUPT		0x010307
>  #define E_ITS_CLEAR_UNMAPPED_INTERRUPT		0x010507
>  #define E_ITS_MAPD_DEVICE_OOR			0x010801
> +#define E_ITS_MAPD_ITTSIZE_OOR			0x010802
>  #define E_ITS_MAPC_PROCNUM_OOR			0x010902
>  #define E_ITS_MAPC_COLLECTION_OOR		0x010903
>  #define E_ITS_MAPTI_UNMAPPED_DEVICE		0x010a04
> +#define E_ITS_MAPTI_ID_OOR			0x010a05
>  #define E_ITS_MAPTI_PHYSICALID_OOR		0x010a06
>  #define E_ITS_INV_UNMAPPED_INTERRUPT		0x010c07
>  #define E_ITS_INVALL_UNMAPPED_COLLECTION	0x010d09
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 1b5797e..0f3c8f3 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -105,6 +105,7 @@ struct its_device {
>  
>  	/* the head for the list of ITTEs */
>  	struct list_head itt_head;
> +	u32 nb_eventid_bits;

what does nb_ mean?

>  	u32 device_id;
>  };
>  
> @@ -191,6 +192,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  
>  #define GIC_LPI_OFFSET 8192
>  
> +#define VITS_TYPER_IDBITS 16
> +
>  /*
>   * Finds and returns a collection in the ITS collection table.
>   * Must be called with the its_lock mutex held.
> @@ -391,7 +394,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>  	 * DevBits low - as least for the time being.
>  	 */
>  	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
> -	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
> +	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
>  	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
>  
>  	return extract_bytes(reg, addr & 7, len);
> @@ -562,6 +565,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>  
>  #define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
>  #define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
> +#define its_cmd_get_size(cmd)		(its_cmd_mask_field(cmd, 1,  0,  5) + 1)
>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
> @@ -752,6 +756,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  	if (!device)
>  		return E_ITS_MAPTI_UNMAPPED_DEVICE;
>  
> +	if (event_id >= BIT_ULL(device->nb_eventid_bits))
> +		return E_ITS_MAPTI_ID_OOR;
> +
>  	if (its_cmd_get_command(its_cmd) == GITS_CMD_MAPTI)
>  		lpi_nr = its_cmd_get_physical_id(its_cmd);
>  	else
> @@ -832,11 +839,15 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>  {
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	bool valid = its_cmd_get_validbit(its_cmd);
> +	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
>  	struct its_device *device;
>  
>  	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
>  		return E_ITS_MAPD_DEVICE_OOR;
>  
> +	if (valid && nb_eventid_bits > VITS_TYPER_IDBITS)
> +		return E_ITS_MAPD_ITTSIZE_OOR;
> +
>  	device = find_its_device(its, device_id);
>  
>  	/*
> @@ -859,6 +870,8 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>  		return -ENOMEM;
>  
>  	device->device_id = device_id;
> +	device->nb_eventid_bits = nb_eventid_bits;
> +
>  	INIT_LIST_HEAD(&device->itt_head);
>  
>  	list_add_tail(&device->dev_list, &its->device_list);
> -- 
> 2.5.5
> 

Otherwise:

Reviewed-by: Christoffer Dall <cdall@linaro.org>

[PATCH v5 11/22] KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:23PM +0200, Eric Auger wrote:
> Up to now the MAPD's ITT size field has been ignored. It encodes
> the number of eventid bit minus 1. It should be used to check
> the eventid when a MAPTI command is issued on a device. Let's
> store the number of eventid bits in the its_device and do the
> check on MAPTI. Also make sure the ITT size field does
> not exceed the GITS_TYPER IDBITS field.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - its_cmd_get_size macro now returns the actual number of eventid bits
> - use GIC_ENCODE_SZ macro to encode ID_bits
> 
> v3 -> v4:
> - VITS_TYPER_IDBITS set to 16 to be homogeneous with VITS_ESZ definition
>   and correct users
> - nb_eventid_bits correspond to the actual number of eventid bits
> ---
>  include/linux/irqchip/arm-gic-v3.h |  2 ++
>  virt/kvm/arm/vgic/vgic-its.c       | 15 ++++++++++++++-
>  2 files changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index 54c20bd..0c6798c 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -345,9 +345,11 @@
>  #define E_ITS_INT_UNMAPPED_INTERRUPT		0x010307
>  #define E_ITS_CLEAR_UNMAPPED_INTERRUPT		0x010507
>  #define E_ITS_MAPD_DEVICE_OOR			0x010801
> +#define E_ITS_MAPD_ITTSIZE_OOR			0x010802
>  #define E_ITS_MAPC_PROCNUM_OOR			0x010902
>  #define E_ITS_MAPC_COLLECTION_OOR		0x010903
>  #define E_ITS_MAPTI_UNMAPPED_DEVICE		0x010a04
> +#define E_ITS_MAPTI_ID_OOR			0x010a05
>  #define E_ITS_MAPTI_PHYSICALID_OOR		0x010a06
>  #define E_ITS_INV_UNMAPPED_INTERRUPT		0x010c07
>  #define E_ITS_INVALL_UNMAPPED_COLLECTION	0x010d09
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 1b5797e..0f3c8f3 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -105,6 +105,7 @@ struct its_device {
>  
>  	/* the head for the list of ITTEs */
>  	struct list_head itt_head;
> +	u32 nb_eventid_bits;

what does nb_ mean?

>  	u32 device_id;
>  };
>  
> @@ -191,6 +192,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  
>  #define GIC_LPI_OFFSET 8192
>  
> +#define VITS_TYPER_IDBITS 16
> +
>  /*
>   * Finds and returns a collection in the ITS collection table.
>   * Must be called with the its_lock mutex held.
> @@ -391,7 +394,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>  	 * DevBits low - as least for the time being.
>  	 */
>  	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
> -	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
> +	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
>  	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
>  
>  	return extract_bytes(reg, addr & 7, len);
> @@ -562,6 +565,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>  
>  #define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
>  #define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
> +#define its_cmd_get_size(cmd)		(its_cmd_mask_field(cmd, 1,  0,  5) + 1)
>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
> @@ -752,6 +756,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  	if (!device)
>  		return E_ITS_MAPTI_UNMAPPED_DEVICE;
>  
> +	if (event_id >= BIT_ULL(device->nb_eventid_bits))
> +		return E_ITS_MAPTI_ID_OOR;
> +
>  	if (its_cmd_get_command(its_cmd) == GITS_CMD_MAPTI)
>  		lpi_nr = its_cmd_get_physical_id(its_cmd);
>  	else
> @@ -832,11 +839,15 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>  {
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	bool valid = its_cmd_get_validbit(its_cmd);
> +	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
>  	struct its_device *device;
>  
>  	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
>  		return E_ITS_MAPD_DEVICE_OOR;
>  
> +	if (valid && nb_eventid_bits > VITS_TYPER_IDBITS)
> +		return E_ITS_MAPD_ITTSIZE_OOR;
> +
>  	device = find_its_device(its, device_id);
>  
>  	/*
> @@ -859,6 +870,8 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>  		return -ENOMEM;
>  
>  	device->device_id = device_id;
> +	device->nb_eventid_bits = nb_eventid_bits;
> +
>  	INIT_LIST_HEAD(&device->itt_head);
>  
>  	list_add_tail(&device->dev_list, &its->device_list);
> -- 
> 2.5.5
> 

Otherwise:

Reviewed-by: Christoffer Dall <cdall@linaro.org>

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Thu, Apr 27, 2017 at 05:29:35PM +0200, Auger Eric wrote:
> 
> 
> On 27/04/2017 16:45, Christoffer Dall wrote:
> > Hi Eric,
> > 
> > On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
> >> On 27/04/2017 13:02, Christoffer Dall wrote:
> >>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> >>>> On 27/04/2017 10:57, Christoffer Dall wrote:
> >>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>>>>>> Add description for how to access ITS registers and how to save/restore
> >>>>>>>> ITS tables into/from memory.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> v4 -> v5:
> >>>>>>>> - take into account Christoffer's comments
> >>>>>>>> - pending table save on GICV3 side now
> >>>>>>>>
> >>>>>>>> v3 -> v4:
> >>>>>>>> - take into account Peter's comments:
> >>>>>>>>   - typos
> >>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>>>>>   - add a validity bit in DTE
> >>>>>>>>   - document all fields in CTE and ITE
> >>>>>>>>   - document ABI revision
> >>>>>>>> - take into account Andre's comments:
> >>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>>>>>   - document 64b registers only can be accessed with 64b access
> >>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>>>>>
> >>>>>>>> v1 -> v2:
> >>>>>>>> - DTE and ITE now are 8 bytes
> >>>>>>>> - DTE and ITE now indexed by deviceid/eventid
> >>>>>>>> - use ITE name instead of ITTE
> >>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>>>>>> - mentions LE layout
> >>>>>>>> ---
> >>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>>>>>  1 file changed, 99 insertions(+)
> >>>>>>>>
> >>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> index 6081a5b..b5f010d 100644
> >>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> @@ -32,7 +32,106 @@ Groups:
> >>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>>>>>        request the initialization of the ITS, no additional parameter in
> >>>>>>>>        kvm_device_attr.addr.
> >>>>>>>> +
> >>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>>>>>> +      by the guest in corresponding registers/table entries.
> >>>>>>>> +
> >>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>>>>>> +      are laid out in little endian format as described in the last paragraph.
> >>>>>>>> +
> >>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>>>>>> +
> >>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>>>>>> +
> >>>>>>>> +      The GITS_IIDR read-only register must also be restored before
> >>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>>>>>> +
> >>>>>>>
> >>>>>>> what is the expected sequence of operations.  For example, to restore
> >>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>>>>>
> >>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>> and restore GITS_CTLR (which enables the ITS)?
> >>>>>>
> >>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >>>>>> that the pending table is read. But the whole pending table is not read
> >>>>>> as we only iterate on registered LPIs. So the ITT must have been
> >>>>>> restored previously.
> >>>>>>
> >>>>>> I became aware that the pending table sync is done twice, once in the
> >>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
> >>>>>> leave this order specification, I should be able to remove the sync on
> >>>>>> table restore. This was the original reason why GITS_CTLR restore has
> >>>>>> been done at the very end.
> >>>>>
> >>>>> I'm sorry, I'm a bit confused.  Do we not need
> >>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> >>>>
> >>>> Yes you do. I was talking about the RDIST pending table sync. The save
> >>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> >>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> >>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>> which is not requested I think since GITS_CTLR restore does it already.
> >>>
> >>> Shouldn't restoring the pending tables happen when restoring some
> >>> redeistributor state and not anything related to the ITS?
> >>
> >> Marc wrote:
> >> "
> >> I don't think you necessarily need a coarse map. When restoring the ITS
> >> tables, you can always read the pending bit when creating the LPI
> >> structure (it has been written to RAM at save time). Note that we
> >> already do something like this in vgic_enable_lpis().
> >> "
> >>
> >> This is currently what is implemented I think. the pending tables are
> >> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
> >> also on on ITS table restore
> >>
> >> The problematic is: Either you know in advance which LPI INTIDare used
> >> or you need to parse the whole pending table (possibly using the 1st kB
> >> as coarse mapping).
> >>
> >> If you don't know the LPI INTIDs in advance it is only possible to
> >> restore the pending bit of pending LPIs. At that time you would
> >> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
> >> ITS ITT you would do the same for those which were not pending. Looks
> >> really heavy to me: coarse mapping + dual vgic_add_lpi path.
> >>
> >> Otherwise we would need to add another dependency between RDIST pending
> >> table restore and ITS table restore but this looks even more weird, no?
> >>
> >>
> > So I just sat down with Andre and Marc and we tried to work through this
> > and came up with the best scheme.  I apologize in advance for the
> > one-way nature of this e-mail, and I am of course open to discussing the
> > following proposal again if you do not agree.
> > 
> > What I think this document should say, is that the following ordering
> > must be followed when restoring the GIC and the ITS:
> > 
> >   First, restore all guest memory
> > 
> >   Second, restore ALL redistributors
> > 
> >   Third, restore the ITS, in the following order:
> >     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >     2. Restore GITS_CBASER
> >     3. Restore all other GITS_ registers, except GITS_CTLR!
> >     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> >     5. Restore GITS_CTLR
> > 
> > The rationale is that we really want the redistributor and the ITS
> > restore to be independent and follow the architecture.  This means that
> > our ABI for the redistributor should still work without restoring an ITS
> > (if we ever decide to support LPIs for KVM without the ITS).
> 
> OK. Note I already mentioned that GICv3 must be restored before the ITS.
> To me this comprised the RDIST.

Possibly, but I think it's good to write out the whole thing so we
clearly understand the flow.  That could better be achieved by
correcting my proposed text above to say something like "Second, restore
ALL redistributors to ensure the pending and configuration tables can be
read."

> 
> I understand the above description of the ordering comes in addition to
> the existing text, right? 

Yes

> in other words I keep the GITS_READR,
> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
> section.
> 

Yes.  But you don't need to do any reading of the pending table on any
of the restore operations.

Btw., I think it's GITS_CREADR, not GITS_READR.

> > 
> > In terms of our current implementation this means that vgic_add_lpi()
> > should ask the redistributor what the state of the LPI is (priority,
> > enabled, pending).
> this practically means I move update_lpi_config call from
> vgic_its_restore_ite to vgic_add_lpi(). OK

Pretty much, yes.

Thanks,
-Christoffer

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Thu, Apr 27, 2017 at 05:29:35PM +0200, Auger Eric wrote:
> 
> 
> On 27/04/2017 16:45, Christoffer Dall wrote:
> > Hi Eric,
> > 
> > On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
> >> On 27/04/2017 13:02, Christoffer Dall wrote:
> >>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> >>>> On 27/04/2017 10:57, Christoffer Dall wrote:
> >>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>>>>>> Add description for how to access ITS registers and how to save/restore
> >>>>>>>> ITS tables into/from memory.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> v4 -> v5:
> >>>>>>>> - take into account Christoffer's comments
> >>>>>>>> - pending table save on GICV3 side now
> >>>>>>>>
> >>>>>>>> v3 -> v4:
> >>>>>>>> - take into account Peter's comments:
> >>>>>>>>   - typos
> >>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>>>>>   - add a validity bit in DTE
> >>>>>>>>   - document all fields in CTE and ITE
> >>>>>>>>   - document ABI revision
> >>>>>>>> - take into account Andre's comments:
> >>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>>>>>   - document 64b registers only can be accessed with 64b access
> >>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>>>>>
> >>>>>>>> v1 -> v2:
> >>>>>>>> - DTE and ITE now are 8 bytes
> >>>>>>>> - DTE and ITE now indexed by deviceid/eventid
> >>>>>>>> - use ITE name instead of ITTE
> >>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>>>>>> - mentions LE layout
> >>>>>>>> ---
> >>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>>>>>  1 file changed, 99 insertions(+)
> >>>>>>>>
> >>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> index 6081a5b..b5f010d 100644
> >>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> @@ -32,7 +32,106 @@ Groups:
> >>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>>>>>        request the initialization of the ITS, no additional parameter in
> >>>>>>>>        kvm_device_attr.addr.
> >>>>>>>> +
> >>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>>>>>> +      by the guest in corresponding registers/table entries.
> >>>>>>>> +
> >>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>>>>>> +      are laid out in little endian format as described in the last paragraph.
> >>>>>>>> +
> >>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>>>>>> +
> >>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>>>>>> +
> >>>>>>>> +      The GITS_IIDR read-only register must also be restored before
> >>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>>>>>> +
> >>>>>>>
> >>>>>>> what is the expected sequence of operations.  For example, to restore
> >>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>>>>>
> >>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>> and restore GITS_CTLR (which enables the ITS)?
> >>>>>>
> >>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >>>>>> that the pending table is read. But the whole pending table is not read
> >>>>>> as we only iterate on registered LPIs. So the ITT must have been
> >>>>>> restored previously.
> >>>>>>
> >>>>>> I became aware that the pending table sync is done twice, once in the
> >>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
> >>>>>> leave this order specification, I should be able to remove the sync on
> >>>>>> table restore. This was the original reason why GITS_CTLR restore has
> >>>>>> been done at the very end.
> >>>>>
> >>>>> I'm sorry, I'm a bit confused.  Do we not need
> >>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> >>>>
> >>>> Yes you do. I was talking about the RDIST pending table sync. The save
> >>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> >>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> >>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>> which is not requested I think since GITS_CTLR restore does it already.
> >>>
> >>> Shouldn't restoring the pending tables happen when restoring some
> >>> redeistributor state and not anything related to the ITS?
> >>
> >> Marc wrote:
> >> "
> >> I don't think you necessarily need a coarse map. When restoring the ITS
> >> tables, you can always read the pending bit when creating the LPI
> >> structure (it has been written to RAM at save time). Note that we
> >> already do something like this in vgic_enable_lpis().
> >> "
> >>
> >> This is currently what is implemented I think. the pending tables are
> >> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
> >> also on on ITS table restore
> >>
> >> The problematic is: Either you know in advance which LPI INTIDare used
> >> or you need to parse the whole pending table (possibly using the 1st kB
> >> as coarse mapping).
> >>
> >> If you don't know the LPI INTIDs in advance it is only possible to
> >> restore the pending bit of pending LPIs. At that time you would
> >> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
> >> ITS ITT you would do the same for those which were not pending. Looks
> >> really heavy to me: coarse mapping + dual vgic_add_lpi path.
> >>
> >> Otherwise we would need to add another dependency between RDIST pending
> >> table restore and ITS table restore but this looks even more weird, no?
> >>
> >>
> > So I just sat down with Andre and Marc and we tried to work through this
> > and came up with the best scheme.  I apologize in advance for the
> > one-way nature of this e-mail, and I am of course open to discussing the
> > following proposal again if you do not agree.
> > 
> > What I think this document should say, is that the following ordering
> > must be followed when restoring the GIC and the ITS:
> > 
> >   First, restore all guest memory
> > 
> >   Second, restore ALL redistributors
> > 
> >   Third, restore the ITS, in the following order:
> >     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >     2. Restore GITS_CBASER
> >     3. Restore all other GITS_ registers, except GITS_CTLR!
> >     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> >     5. Restore GITS_CTLR
> > 
> > The rationale is that we really want the redistributor and the ITS
> > restore to be independent and follow the architecture.  This means that
> > our ABI for the redistributor should still work without restoring an ITS
> > (if we ever decide to support LPIs for KVM without the ITS).
> 
> OK. Note I already mentioned that GICv3 must be restored before the ITS.
> To me this comprised the RDIST.

Possibly, but I think it's good to write out the whole thing so we
clearly understand the flow.  That could better be achieved by
correcting my proposed text above to say something like "Second, restore
ALL redistributors to ensure the pending and configuration tables can be
read."

> 
> I understand the above description of the ordering comes in addition to
> the existing text, right? 

Yes

> in other words I keep the GITS_READR,
> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
> section.
> 

Yes.  But you don't need to do any reading of the pending table on any
of the restore operations.

Btw., I think it's GITS_CREADR, not GITS_READR.

> > 
> > In terms of our current implementation this means that vgic_add_lpi()
> > should ask the redistributor what the state of the LPI is (priority,
> > enabled, pending).
> this practically means I move update_lpi_config call from
> vgic_its_restore_ite to vgic_add_lpi(). OK

Pretty much, yes.

Thanks,
-Christoffer

Re: [PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:24PM +0200, Eric Auger wrote:
> Up to now the MAPD ITT_addr had been ignored. We will need it
> for save/restore. Let's record it in the its_device struct.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - its_cmd_get_ittaddr macro now returns the actual ITT GPA
> 
> v3 -> v4:
> - in vgic_its_cmd_handle_mapd, itt_addr directly is shifted
> - correct ittaddr bitmask to support 48bit GPA
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 0f3c8f3..757598d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -106,6 +106,7 @@ struct its_device {
>  	/* the head for the list of ITTEs */
>  	struct list_head itt_head;
>  	u32 nb_eventid_bits;
> +	gpa_t itt_addr;
>  	u32 device_id;
>  };
>  
> @@ -569,6 +570,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
> +#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 47) << 8)

Why 47?  The ITT_addr field is bits [51:8] unless I'm looking at
something wrong?

>  #define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
>  #define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
>  
> @@ -840,6 +842,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	bool valid = its_cmd_get_validbit(its_cmd);
>  	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
> +	gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
>  	struct its_device *device;
>  
>  	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
> @@ -871,6 +874,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>  
>  	device->device_id = device_id;
>  	device->nb_eventid_bits = nb_eventid_bits;
> +	device->itt_addr = itt_addr;
>  
>  	INIT_LIST_HEAD(&device->itt_head);
>  
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

[PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:24PM +0200, Eric Auger wrote:
> Up to now the MAPD ITT_addr had been ignored. We will need it
> for save/restore. Let's record it in the its_device struct.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - its_cmd_get_ittaddr macro now returns the actual ITT GPA
> 
> v3 -> v4:
> - in vgic_its_cmd_handle_mapd, itt_addr directly is shifted
> - correct ittaddr bitmask to support 48bit GPA
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 0f3c8f3..757598d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -106,6 +106,7 @@ struct its_device {
>  	/* the head for the list of ITTEs */
>  	struct list_head itt_head;
>  	u32 nb_eventid_bits;
> +	gpa_t itt_addr;
>  	u32 device_id;
>  };
>  
> @@ -569,6 +570,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
> +#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 47) << 8)

Why 47?  The ITT_addr field is bits [51:8] unless I'm looking at
something wrong?

>  #define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
>  #define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
>  
> @@ -840,6 +842,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>  	bool valid = its_cmd_get_validbit(its_cmd);
>  	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
> +	gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
>  	struct its_device *device;
>  
>  	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
> @@ -871,6 +874,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>  
>  	device->device_id = device_id;
>  	device->nb_eventid_bits = nb_eventid_bits;
> +	device->itt_addr = itt_addr;
>  
>  	INIT_LIST_HEAD(&device->itt_head);
>  
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

Re: [PATCH v5 13/22] KVM: arm64: vgic-its: Check the device id matches TYPER DEVBITS range

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:25PM +0200, Eric Auger wrote:
> On MAPD we currently check the device id can be stored in the device table.
> Let's first check it can be encoded within the range defined by TYPER
> DEVBITS.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> 
> v4 -> v5:
> - use GIC_ENCODE_SZ macro
> 
> v3 -> v4:
> - VITS_TYPER_DEVBITS set to 16 for homogeneity
> - use BIT_ULL
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 757598d..de1ed6d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -194,6 +194,7 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  #define GIC_LPI_OFFSET 8192
>  
>  #define VITS_TYPER_IDBITS 16
> +#define VITS_TYPER_DEVBITS 16
>  
>  /*
>   * Finds and returns a collection in the ITS collection table.
> @@ -394,7 +395,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>  	 * To avoid memory waste in the guest, we keep the number of IDBits and
>  	 * DevBits low - as least for the time being.
>  	 */
> -	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
> +	reg |= GIC_ENCODE_SZ(VITS_TYPER_DEVBITS, 5) << GITS_TYPER_DEVBITS_SHIFT;
>  	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
>  	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
>  
> @@ -639,10 +640,10 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>   * Check whether an ID can be stored into the corresponding guest table.
>   * For a direct table this is pretty easy, but gets a bit nasty for
>   * indirect tables. We check whether the resulting guest physical address
> - * is actually valid (covered by a memslot and guest accessbible).
> + * is actually valid (covered by a memslot and guest accessible).
>   * For this we have to read the respective first level entry.
>   */
> -static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
> +static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
>  {
>  	int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
>  	int index;
> @@ -650,6 +651,9 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
>  	gfn_t gfn;
>  	int esz = GITS_BASER_ENTRY_SIZE(baser);
>  
> +	if (id >= BIT_ULL(VITS_TYPER_DEVBITS))
> +		return false;
> +

Isn't vgic_its_check_id called with both a device id and a collection
id?  How can this then be a valid check?

>  	if (!(baser & GITS_BASER_INDIRECT)) {
>  		phys_addr_t addr;
>  
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

[PATCH v5 13/22] KVM: arm64: vgic-its: Check the device id matches TYPER DEVBITS range

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:25PM +0200, Eric Auger wrote:
> On MAPD we currently check the device id can be stored in the device table.
> Let's first check it can be encoded within the range defined by TYPER
> DEVBITS.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> 
> v4 -> v5:
> - use GIC_ENCODE_SZ macro
> 
> v3 -> v4:
> - VITS_TYPER_DEVBITS set to 16 for homogeneity
> - use BIT_ULL
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 757598d..de1ed6d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -194,6 +194,7 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  #define GIC_LPI_OFFSET 8192
>  
>  #define VITS_TYPER_IDBITS 16
> +#define VITS_TYPER_DEVBITS 16
>  
>  /*
>   * Finds and returns a collection in the ITS collection table.
> @@ -394,7 +395,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>  	 * To avoid memory waste in the guest, we keep the number of IDBits and
>  	 * DevBits low - as least for the time being.
>  	 */
> -	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
> +	reg |= GIC_ENCODE_SZ(VITS_TYPER_DEVBITS, 5) << GITS_TYPER_DEVBITS_SHIFT;
>  	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
>  	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
>  
> @@ -639,10 +640,10 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>   * Check whether an ID can be stored into the corresponding guest table.
>   * For a direct table this is pretty easy, but gets a bit nasty for
>   * indirect tables. We check whether the resulting guest physical address
> - * is actually valid (covered by a memslot and guest accessbible).
> + * is actually valid (covered by a memslot and guest accessible).
>   * For this we have to read the respective first level entry.
>   */
> -static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
> +static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
>  {
>  	int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
>  	int index;
> @@ -650,6 +651,9 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
>  	gfn_t gfn;
>  	int esz = GITS_BASER_ENTRY_SIZE(baser);
>  
> +	if (id >= BIT_ULL(VITS_TYPER_DEVBITS))
> +		return false;
> +

Isn't vgic_its_check_id called with both a device id and a collection
id?  How can this then be a valid check?

>  	if (!(baser & GITS_BASER_INDIRECT)) {
>  		phys_addr_t addr;
>  
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Marc, Christoffer,

On 27/04/2017 18:23, Marc Zyngier wrote:
> On 27/04/17 16:29, Auger Eric wrote:
>>
>>
>> On 27/04/2017 16:45, Christoffer Dall wrote:
>>> Hi Eric,
>>>
>>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> v4 -> v5:
>>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>>
>>>>>>>>>> v3 -> v4:
>>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>>   - typos
>>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>>   - document ABI revision
>>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>>
>>>>>>>>>> v1 -> v2:
>>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>>> - mentions LE layout
>>>>>>>>>> ---
>>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>>> +
>>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>>> +
>>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>>> +
>>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>>
>>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>>
>>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>>> restored previously.
>>>>>>>>
>>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>>> been done at the very end.
>>>>>>>
>>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>>
>>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>>
>>>>> Shouldn't restoring the pending tables happen when restoring some
>>>>> redeistributor state and not anything related to the ITS?
>>>>
>>>> Marc wrote:
>>>> "
>>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>>> tables, you can always read the pending bit when creating the LPI
>>>> structure (it has been written to RAM at save time). Note that we
>>>> already do something like this in vgic_enable_lpis().
>>>> "
>>>>
>>>> This is currently what is implemented I think. the pending tables are
>>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>>> also on on ITS table restore
>>>>
>>>> The problematic is: Either you know in advance which LPI INTIDare used
>>>> or you need to parse the whole pending table (possibly using the 1st kB
>>>> as coarse mapping).
>>>>
>>>> If you don't know the LPI INTIDs in advance it is only possible to
>>>> restore the pending bit of pending LPIs. At that time you would
>>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>>> ITS ITT you would do the same for those which were not pending. Looks
>>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>>
>>>> Otherwise we would need to add another dependency between RDIST pending
>>>> table restore and ITS table restore but this looks even more weird, no?
>>>>
>>>>
>>> So I just sat down with Andre and Marc and we tried to work through this
>>> and came up with the best scheme.  I apologize in advance for the
>>> one-way nature of this e-mail, and I am of course open to discussing the
>>> following proposal again if you do not agree.
>>>
>>> What I think this document should say, is that the following ordering
>>> must be followed when restoring the GIC and the ITS:
>>>
>>>   First, restore all guest memory
>>>
>>>   Second, restore ALL redistributors
>>>
>>>   Third, restore the ITS, in the following order:
>>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>>     2. Restore GITS_CBASER
>>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>>     5. Restore GITS_CTLR
>>>
>>> The rationale is that we really want the redistributor and the ITS
>>> restore to be independent and follow the architecture.  This means that
>>> our ABI for the redistributor should still work without restoring an ITS
>>> (if we ever decide to support LPIs for KVM without the ITS).
>>
>> OK. Note I already mentioned that GICv3 must be restored before the ITS.
>> To me this comprised the RDIST.
>>
>> I understand the above description of the ordering comes in addition to
>> the existing text, right? in other words I keep the GITS_READR,
>> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
>> section.
>>
>>>
>>> In terms of our current implementation this means that vgic_add_lpi()
>>> should ask the redistributor what the state of the LPI is (priority,
>>> enabled, pending).
>> this practically means I move update_lpi_config call from
>> vgic_its_restore_ite to vgic_add_lpi(). OK
>>
>> However for getting the LPI pending state I must know which RDIST the
>> LPI is attached to. This is not known at LPI allocation time. Do I
>> misunderstand something?
> 
> Once you have rebuilt the ITS data structures and allocated the IRQ
> structures, you should have a target_cpu field pointing to the right
> vcpu. From there, you can surely find the corresponding redistributor
> and the pending table.
Yes that's understood but Christoffer said "vgic_add_lpi() should ask
the redistributor what the state of the LPI is (priority,enabled,
pending)." Fetching the properties is fine.

vgic_add_lpi() is called before update_affinity_ite() which uses
ite->irq and sets the target_vcpu.

Well at least this requires some function reshape. I will investigate
though.

Thanks

Eric
> 
> BTW, we should document the fact that vcpus must have been created
> before reloading the ITS (that's not completely obvious).
> 
> Thanks,
> 
> 	M.
> 

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Marc, Christoffer,

On 27/04/2017 18:23, Marc Zyngier wrote:
> On 27/04/17 16:29, Auger Eric wrote:
>>
>>
>> On 27/04/2017 16:45, Christoffer Dall wrote:
>>> Hi Eric,
>>>
>>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> v4 -> v5:
>>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>>
>>>>>>>>>> v3 -> v4:
>>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>>   - typos
>>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>>   - document ABI revision
>>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>>
>>>>>>>>>> v1 -> v2:
>>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>>> - mentions LE layout
>>>>>>>>>> ---
>>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>>> +
>>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>>> +
>>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>>> +
>>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>>
>>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>>
>>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>>> restored previously.
>>>>>>>>
>>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>>> been done at the very end.
>>>>>>>
>>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>>
>>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>>
>>>>> Shouldn't restoring the pending tables happen when restoring some
>>>>> redeistributor state and not anything related to the ITS?
>>>>
>>>> Marc wrote:
>>>> "
>>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>>> tables, you can always read the pending bit when creating the LPI
>>>> structure (it has been written to RAM at save time). Note that we
>>>> already do something like this in vgic_enable_lpis().
>>>> "
>>>>
>>>> This is currently what is implemented I think. the pending tables are
>>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>>> also on on ITS table restore
>>>>
>>>> The problematic is: Either you know in advance which LPI INTIDare used
>>>> or you need to parse the whole pending table (possibly using the 1st kB
>>>> as coarse mapping).
>>>>
>>>> If you don't know the LPI INTIDs in advance it is only possible to
>>>> restore the pending bit of pending LPIs. At that time you would
>>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>>> ITS ITT you would do the same for those which were not pending. Looks
>>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>>
>>>> Otherwise we would need to add another dependency between RDIST pending
>>>> table restore and ITS table restore but this looks even more weird, no?
>>>>
>>>>
>>> So I just sat down with Andre and Marc and we tried to work through this
>>> and came up with the best scheme.  I apologize in advance for the
>>> one-way nature of this e-mail, and I am of course open to discussing the
>>> following proposal again if you do not agree.
>>>
>>> What I think this document should say, is that the following ordering
>>> must be followed when restoring the GIC and the ITS:
>>>
>>>   First, restore all guest memory
>>>
>>>   Second, restore ALL redistributors
>>>
>>>   Third, restore the ITS, in the following order:
>>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>>     2. Restore GITS_CBASER
>>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>>     5. Restore GITS_CTLR
>>>
>>> The rationale is that we really want the redistributor and the ITS
>>> restore to be independent and follow the architecture.  This means that
>>> our ABI for the redistributor should still work without restoring an ITS
>>> (if we ever decide to support LPIs for KVM without the ITS).
>>
>> OK. Note I already mentioned that GICv3 must be restored before the ITS.
>> To me this comprised the RDIST.
>>
>> I understand the above description of the ordering comes in addition to
>> the existing text, right? in other words I keep the GITS_READR,
>> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
>> section.
>>
>>>
>>> In terms of our current implementation this means that vgic_add_lpi()
>>> should ask the redistributor what the state of the LPI is (priority,
>>> enabled, pending).
>> this practically means I move update_lpi_config call from
>> vgic_its_restore_ite to vgic_add_lpi(). OK
>>
>> However for getting the LPI pending state I must know which RDIST the
>> LPI is attached to. This is not known at LPI allocation time. Do I
>> misunderstand something?
> 
> Once you have rebuilt the ITS data structures and allocated the IRQ
> structures, you should have a target_cpu field pointing to the right
> vcpu. From there, you can surely find the corresponding redistributor
> and the pending table.
Yes that's understood but Christoffer said "vgic_add_lpi() should ask
the redistributor what the state of the LPI is (priority,enabled,
pending)." Fetching the properties is fine.

vgic_add_lpi() is called before update_affinity_ite() which uses
ite->irq and sets the target_vcpu.

Well at least this requires some function reshape. I will investigate
though.

Thanks

Eric
> 
> BTW, we should document the fact that vcpus must have been created
> before reloading the ITS (that's not completely obvious).
> 
> Thanks,
> 
> 	M.
> 

Re: [PATCH v5 11/22] KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors

[Auger Eric]

Hi,
On 27/04/2017 18:25, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:23PM +0200, Eric Auger wrote:
>> Up to now the MAPD's ITT size field has been ignored. It encodes
>> the number of eventid bit minus 1. It should be used to check
>> the eventid when a MAPTI command is issued on a device. Let's
>> store the number of eventid bits in the its_device and do the
>> check on MAPTI. Also make sure the ITT size field does
>> not exceed the GITS_TYPER IDBITS field.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - its_cmd_get_size macro now returns the actual number of eventid bits
>> - use GIC_ENCODE_SZ macro to encode ID_bits
>>
>> v3 -> v4:
>> - VITS_TYPER_IDBITS set to 16 to be homogeneous with VITS_ESZ definition
>>   and correct users
>> - nb_eventid_bits correspond to the actual number of eventid bits
>> ---
>>  include/linux/irqchip/arm-gic-v3.h |  2 ++
>>  virt/kvm/arm/vgic/vgic-its.c       | 15 ++++++++++++++-
>>  2 files changed, 16 insertions(+), 1 deletion(-)
>>
>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>> index 54c20bd..0c6798c 100644
>> --- a/include/linux/irqchip/arm-gic-v3.h
>> +++ b/include/linux/irqchip/arm-gic-v3.h
>> @@ -345,9 +345,11 @@
>>  #define E_ITS_INT_UNMAPPED_INTERRUPT		0x010307
>>  #define E_ITS_CLEAR_UNMAPPED_INTERRUPT		0x010507
>>  #define E_ITS_MAPD_DEVICE_OOR			0x010801
>> +#define E_ITS_MAPD_ITTSIZE_OOR			0x010802
>>  #define E_ITS_MAPC_PROCNUM_OOR			0x010902
>>  #define E_ITS_MAPC_COLLECTION_OOR		0x010903
>>  #define E_ITS_MAPTI_UNMAPPED_DEVICE		0x010a04
>> +#define E_ITS_MAPTI_ID_OOR			0x010a05
>>  #define E_ITS_MAPTI_PHYSICALID_OOR		0x010a06
>>  #define E_ITS_INV_UNMAPPED_INTERRUPT		0x010c07
>>  #define E_ITS_INVALL_UNMAPPED_COLLECTION	0x010d09
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 1b5797e..0f3c8f3 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -105,6 +105,7 @@ struct its_device {
>>  
>>  	/* the head for the list of ITTEs */
>>  	struct list_head itt_head;
>> +	u32 nb_eventid_bits;
> 
> what does nb_ mean?
number ;-)

Eric
> 
>>  	u32 device_id;
>>  };
>>  
>> @@ -191,6 +192,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  
>>  #define GIC_LPI_OFFSET 8192
>>  
>> +#define VITS_TYPER_IDBITS 16
>> +
>>  /*
>>   * Finds and returns a collection in the ITS collection table.
>>   * Must be called with the its_lock mutex held.
>> @@ -391,7 +394,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>>  	 * DevBits low - as least for the time being.
>>  	 */
>>  	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>> -	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>> +	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
>>  	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
>>  
>>  	return extract_bytes(reg, addr & 7, len);
>> @@ -562,6 +565,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>>  
>>  #define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
>>  #define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
>> +#define its_cmd_get_size(cmd)		(its_cmd_mask_field(cmd, 1,  0,  5) + 1)
>>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
>> @@ -752,6 +756,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>  	if (!device)
>>  		return E_ITS_MAPTI_UNMAPPED_DEVICE;
>>  
>> +	if (event_id >= BIT_ULL(device->nb_eventid_bits))
>> +		return E_ITS_MAPTI_ID_OOR;
>> +
>>  	if (its_cmd_get_command(its_cmd) == GITS_CMD_MAPTI)
>>  		lpi_nr = its_cmd_get_physical_id(its_cmd);
>>  	else
>> @@ -832,11 +839,15 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>>  {
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	bool valid = its_cmd_get_validbit(its_cmd);
>> +	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
>>  	struct its_device *device;
>>  
>>  	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
>>  		return E_ITS_MAPD_DEVICE_OOR;
>>  
>> +	if (valid && nb_eventid_bits > VITS_TYPER_IDBITS)
>> +		return E_ITS_MAPD_ITTSIZE_OOR;
>> +
>>  	device = find_its_device(its, device_id);
>>  
>>  	/*
>> @@ -859,6 +870,8 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>>  		return -ENOMEM;
>>  
>>  	device->device_id = device_id;
>> +	device->nb_eventid_bits = nb_eventid_bits;
>> +
>>  	INIT_LIST_HEAD(&device->itt_head);
>>  
>>  	list_add_tail(&device->dev_list, &its->device_list);
>> -- 
>> 2.5.5
>>
> 
> Otherwise:
> 
> Reviewed-by: Christoffer Dall <cdall@linaro.org>
> 

[PATCH v5 11/22] KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors

[Auger Eric]

Hi,
On 27/04/2017 18:25, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:23PM +0200, Eric Auger wrote:
>> Up to now the MAPD's ITT size field has been ignored. It encodes
>> the number of eventid bit minus 1. It should be used to check
>> the eventid when a MAPTI command is issued on a device. Let's
>> store the number of eventid bits in the its_device and do the
>> check on MAPTI. Also make sure the ITT size field does
>> not exceed the GITS_TYPER IDBITS field.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - its_cmd_get_size macro now returns the actual number of eventid bits
>> - use GIC_ENCODE_SZ macro to encode ID_bits
>>
>> v3 -> v4:
>> - VITS_TYPER_IDBITS set to 16 to be homogeneous with VITS_ESZ definition
>>   and correct users
>> - nb_eventid_bits correspond to the actual number of eventid bits
>> ---
>>  include/linux/irqchip/arm-gic-v3.h |  2 ++
>>  virt/kvm/arm/vgic/vgic-its.c       | 15 ++++++++++++++-
>>  2 files changed, 16 insertions(+), 1 deletion(-)
>>
>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>> index 54c20bd..0c6798c 100644
>> --- a/include/linux/irqchip/arm-gic-v3.h
>> +++ b/include/linux/irqchip/arm-gic-v3.h
>> @@ -345,9 +345,11 @@
>>  #define E_ITS_INT_UNMAPPED_INTERRUPT		0x010307
>>  #define E_ITS_CLEAR_UNMAPPED_INTERRUPT		0x010507
>>  #define E_ITS_MAPD_DEVICE_OOR			0x010801
>> +#define E_ITS_MAPD_ITTSIZE_OOR			0x010802
>>  #define E_ITS_MAPC_PROCNUM_OOR			0x010902
>>  #define E_ITS_MAPC_COLLECTION_OOR		0x010903
>>  #define E_ITS_MAPTI_UNMAPPED_DEVICE		0x010a04
>> +#define E_ITS_MAPTI_ID_OOR			0x010a05
>>  #define E_ITS_MAPTI_PHYSICALID_OOR		0x010a06
>>  #define E_ITS_INV_UNMAPPED_INTERRUPT		0x010c07
>>  #define E_ITS_INVALL_UNMAPPED_COLLECTION	0x010d09
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 1b5797e..0f3c8f3 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -105,6 +105,7 @@ struct its_device {
>>  
>>  	/* the head for the list of ITTEs */
>>  	struct list_head itt_head;
>> +	u32 nb_eventid_bits;
> 
> what does nb_ mean?
number ;-)

Eric
> 
>>  	u32 device_id;
>>  };
>>  
>> @@ -191,6 +192,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  
>>  #define GIC_LPI_OFFSET 8192
>>  
>> +#define VITS_TYPER_IDBITS 16
>> +
>>  /*
>>   * Finds and returns a collection in the ITS collection table.
>>   * Must be called with the its_lock mutex held.
>> @@ -391,7 +394,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>>  	 * DevBits low - as least for the time being.
>>  	 */
>>  	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>> -	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>> +	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
>>  	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
>>  
>>  	return extract_bytes(reg, addr & 7, len);
>> @@ -562,6 +565,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>>  
>>  #define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
>>  #define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
>> +#define its_cmd_get_size(cmd)		(its_cmd_mask_field(cmd, 1,  0,  5) + 1)
>>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
>> @@ -752,6 +756,9 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>>  	if (!device)
>>  		return E_ITS_MAPTI_UNMAPPED_DEVICE;
>>  
>> +	if (event_id >= BIT_ULL(device->nb_eventid_bits))
>> +		return E_ITS_MAPTI_ID_OOR;
>> +
>>  	if (its_cmd_get_command(its_cmd) == GITS_CMD_MAPTI)
>>  		lpi_nr = its_cmd_get_physical_id(its_cmd);
>>  	else
>> @@ -832,11 +839,15 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>>  {
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	bool valid = its_cmd_get_validbit(its_cmd);
>> +	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
>>  	struct its_device *device;
>>  
>>  	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
>>  		return E_ITS_MAPD_DEVICE_OOR;
>>  
>> +	if (valid && nb_eventid_bits > VITS_TYPER_IDBITS)
>> +		return E_ITS_MAPD_ITTSIZE_OOR;
>> +
>>  	device = find_its_device(its, device_id);
>>  
>>  	/*
>> @@ -859,6 +870,8 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>>  		return -ENOMEM;
>>  
>>  	device->device_id = device_id;
>> +	device->nb_eventid_bits = nb_eventid_bits;
>> +
>>  	INIT_LIST_HEAD(&device->itt_head);
>>  
>>  	list_add_tail(&device->dev_list, &its->device_list);
>> -- 
>> 2.5.5
>>
> 
> Otherwise:
> 
> Reviewed-by: Christoffer Dall <cdall@linaro.org>
> 

Re: [PATCH v5 14/22] KVM: arm64: vgic-its: KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:26PM +0200, Eric Auger wrote:
> Introduce new attributes in KVM_DEV_ARM_VGIC_GRP_CTRL group:
> - KVM_DEV_ARM_ITS_SAVE_TABLES: saves the ITS tables into guest RAM
> - KVM_DEV_ARM_ITS_RESTORE_TABLES: restores them into VGIC internal
>   structures.
> 
> We hold the vcpus lock during the save and restore to make
> sure no vcpu is running.
> 
> At this stage the functionality is not yet implemented. Only
> the skeleton is put in place.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> 
> v4 -> v5:
> - use KVM_DEV_ARM_ITS_SAVE_TABLES and KVM_DEV_ARM_ITS_RESTORE_TABLES
> - rename *flush* into *save*
> - call its_sync_lpi_pending_table at the end of restore
> - use abi framework
> 
> v3 -> v4:
> - pass kvm struct handle to vgic_its_flush/restore_pending_tables
> - take the kvm lock and vcpu locks
> - ABI revision check
> - check attr->attr is null
> 
> v1 -> v2:
> - remove useless kvm parameter
> ---
>  arch/arm/include/uapi/asm/kvm.h   |   4 +-
>  arch/arm64/include/uapi/asm/kvm.h |   4 +-
>  virt/kvm/arm/vgic/vgic-its.c      | 108 ++++++++++++++++++++++++++++++++++++--
>  3 files changed, 110 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
> index 4beb83b..8e6563c 100644
> --- a/arch/arm/include/uapi/asm/kvm.h
> +++ b/arch/arm/include/uapi/asm/kvm.h
> @@ -199,7 +199,9 @@ struct kvm_arch_memory_slot {
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK 0x3ff
>  #define VGIC_LEVEL_INFO_LINE_LEVEL	0
>  
> -#define   KVM_DEV_ARM_VGIC_CTRL_INIT    0
> +#define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
> +#define   KVM_DEV_ARM_ITS_SAVE_TABLES		1
> +#define   KVM_DEV_ARM_ITS_RESTORE_TABLES	2
>  
>  /* KVM_IRQ_LINE irq field index values */
>  #define KVM_ARM_IRQ_TYPE_SHIFT		24
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index 7e8dd69..1e35115 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -219,7 +219,9 @@ struct kvm_arch_memory_slot {
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK	0x3ff
>  #define VGIC_LEVEL_INFO_LINE_LEVEL	0
>  
> -#define   KVM_DEV_ARM_VGIC_CTRL_INIT	0
> +#define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
> +#define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
> +#define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
>  
>  /* Device Control API on vcpu fd */
>  #define KVM_ARM_VCPU_PMU_V3_CTRL	0
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index de1ed6d..55267ab 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1648,12 +1648,68 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  }
>  
>  /**
> + * vgic_its_save_device_tables - Save the device table and all ITT
> + * into guest RAM
> + */
> +static int vgic_its_save_device_tables(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +/**
> + * vgic_its_restore_device_tables - Restore the device table and all ITT
> + * from guest RAM to internal data structs
> + */
> +static int vgic_its_restore_device_tables(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +/**
> + * vgic_its_save_collection_table - Save the collection table into
> + * guest RAM
> + */
> +static int vgic_its_save_collection_table(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +/**
> + * vgic_its_restore_collection_table - reads the collection table
> + * in guest memory and restores the ITS internal state. Requires the
> + * BASER registers to be restored before.
> + */
> +static int vgic_its_restore_collection_table(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +/**
>   * vgic_its_save_tables_v0 - Save the ITS tables into guest ARM
>   * according to v0 ABI
>   */
>  static int vgic_its_save_tables_v0(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	struct kvm *kvm = its->dev->kvm;
> +	int ret;
> +
> +	mutex_lock(&kvm->lock);
> +
> +	if (!lock_all_vcpus(kvm)) {
> +		mutex_unlock(&kvm->lock);
> +		return -EBUSY;
> +	}
> +
> +	ret = vgic_its_save_device_tables(its);
> +	if (ret)
> +		goto out;
> +
> +	ret = vgic_its_save_collection_table(its);
> +
> +out:
> +	unlock_all_vcpus(kvm);
> +	mutex_unlock(&kvm->lock);
> +	return ret;
>  }
>  
>  /**
> @@ -1663,7 +1719,41 @@ static int vgic_its_save_tables_v0(struct vgic_its *its)
>   */
>  static int vgic_its_restore_tables_v0(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	struct kvm *kvm = its->dev->kvm;
> +	struct kvm_vcpu *vcpu;
> +	int ret, c;
> +
> +	mutex_lock(&kvm->lock);
> +
> +	if (!lock_all_vcpus(kvm)) {
> +		mutex_unlock(&kvm->lock);
> +		return -EBUSY;
> +	}
> +
> +	ret = vgic_its_restore_collection_table(its);
> +	if (ret)
> +		goto out;
> +
> +	ret = vgic_its_restore_device_tables(its);
> +
> +	kvm_for_each_vcpu(c, vcpu, kvm) {
> +		ret = its_sync_lpi_pending_table(vcpu);
> +		if (ret)
> +			break;
> +	}
> +
> +out:
> +	unlock_all_vcpus(kvm);
> +	mutex_unlock(&kvm->lock);
> +
> +	if (ret)
> +		return ret;
> +
> +	/*
> +	 * On restore path, MSI injections can happen before the
> +	 * first VCPU run so let's complete the GIC init here.
> +	 */
> +	return kvm_vgic_map_resources(its->dev->kvm);

wait, this is supposed to run when everything else is done and just
before running a VCPU, but now we run this way earlier?  Is that safe.
It feels incredibly dangerous.

Otherwise I'll wait with looking at this patch in more detail until you
respin based on not doing the its_sync_lpi_pending_table() here anymore
as a result of the ABI discussion.

Thanks,
-Christoffer

>  }
>  
>  static int vgic_its_commit_v0(struct vgic_its *its)
> @@ -1718,6 +1808,10 @@ static int vgic_its_has_attr(struct kvm_device *dev,
>  		switch (attr->attr) {
>  		case KVM_DEV_ARM_VGIC_CTRL_INIT:
>  			return 0;
> +		case KVM_DEV_ARM_ITS_SAVE_TABLES:
> +			return 0;
> +		case KVM_DEV_ARM_ITS_RESTORE_TABLES:
> +			return 0;
>  		}
>  		break;
>  	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS:
> @@ -1753,14 +1847,20 @@ static int vgic_its_set_attr(struct kvm_device *dev,
>  
>  		return 0;
>  	}
> -	case KVM_DEV_ARM_VGIC_GRP_CTRL:
> +	case KVM_DEV_ARM_VGIC_GRP_CTRL: {
> +		const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +
>  		switch (attr->attr) {
>  		case KVM_DEV_ARM_VGIC_CTRL_INIT:
>  			its->initialized = true;
>  
>  			return 0;
> +		case KVM_DEV_ARM_ITS_SAVE_TABLES:
> +			return abi->save_tables(its);
> +		case KVM_DEV_ARM_ITS_RESTORE_TABLES:
> +			return abi->restore_tables(its);
>  		}
> -		break;
> +	}
>  	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
>  		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
>  		u64 reg;
> -- 
> 2.5.5
> 

[PATCH v5 14/22] KVM: arm64: vgic-its: KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:26PM +0200, Eric Auger wrote:
> Introduce new attributes in KVM_DEV_ARM_VGIC_GRP_CTRL group:
> - KVM_DEV_ARM_ITS_SAVE_TABLES: saves the ITS tables into guest RAM
> - KVM_DEV_ARM_ITS_RESTORE_TABLES: restores them into VGIC internal
>   structures.
> 
> We hold the vcpus lock during the save and restore to make
> sure no vcpu is running.
> 
> At this stage the functionality is not yet implemented. Only
> the skeleton is put in place.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> 
> v4 -> v5:
> - use KVM_DEV_ARM_ITS_SAVE_TABLES and KVM_DEV_ARM_ITS_RESTORE_TABLES
> - rename *flush* into *save*
> - call its_sync_lpi_pending_table at the end of restore
> - use abi framework
> 
> v3 -> v4:
> - pass kvm struct handle to vgic_its_flush/restore_pending_tables
> - take the kvm lock and vcpu locks
> - ABI revision check
> - check attr->attr is null
> 
> v1 -> v2:
> - remove useless kvm parameter
> ---
>  arch/arm/include/uapi/asm/kvm.h   |   4 +-
>  arch/arm64/include/uapi/asm/kvm.h |   4 +-
>  virt/kvm/arm/vgic/vgic-its.c      | 108 ++++++++++++++++++++++++++++++++++++--
>  3 files changed, 110 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
> index 4beb83b..8e6563c 100644
> --- a/arch/arm/include/uapi/asm/kvm.h
> +++ b/arch/arm/include/uapi/asm/kvm.h
> @@ -199,7 +199,9 @@ struct kvm_arch_memory_slot {
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK 0x3ff
>  #define VGIC_LEVEL_INFO_LINE_LEVEL	0
>  
> -#define   KVM_DEV_ARM_VGIC_CTRL_INIT    0
> +#define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
> +#define   KVM_DEV_ARM_ITS_SAVE_TABLES		1
> +#define   KVM_DEV_ARM_ITS_RESTORE_TABLES	2
>  
>  /* KVM_IRQ_LINE irq field index values */
>  #define KVM_ARM_IRQ_TYPE_SHIFT		24
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index 7e8dd69..1e35115 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -219,7 +219,9 @@ struct kvm_arch_memory_slot {
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK	0x3ff
>  #define VGIC_LEVEL_INFO_LINE_LEVEL	0
>  
> -#define   KVM_DEV_ARM_VGIC_CTRL_INIT	0
> +#define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
> +#define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
> +#define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
>  
>  /* Device Control API on vcpu fd */
>  #define KVM_ARM_VCPU_PMU_V3_CTRL	0
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index de1ed6d..55267ab 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1648,12 +1648,68 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  }
>  
>  /**
> + * vgic_its_save_device_tables - Save the device table and all ITT
> + * into guest RAM
> + */
> +static int vgic_its_save_device_tables(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +/**
> + * vgic_its_restore_device_tables - Restore the device table and all ITT
> + * from guest RAM to internal data structs
> + */
> +static int vgic_its_restore_device_tables(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +/**
> + * vgic_its_save_collection_table - Save the collection table into
> + * guest RAM
> + */
> +static int vgic_its_save_collection_table(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +/**
> + * vgic_its_restore_collection_table - reads the collection table
> + * in guest memory and restores the ITS internal state. Requires the
> + * BASER registers to be restored before.
> + */
> +static int vgic_its_restore_collection_table(struct vgic_its *its)
> +{
> +	return -ENXIO;
> +}
> +
> +/**
>   * vgic_its_save_tables_v0 - Save the ITS tables into guest ARM
>   * according to v0 ABI
>   */
>  static int vgic_its_save_tables_v0(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	struct kvm *kvm = its->dev->kvm;
> +	int ret;
> +
> +	mutex_lock(&kvm->lock);
> +
> +	if (!lock_all_vcpus(kvm)) {
> +		mutex_unlock(&kvm->lock);
> +		return -EBUSY;
> +	}
> +
> +	ret = vgic_its_save_device_tables(its);
> +	if (ret)
> +		goto out;
> +
> +	ret = vgic_its_save_collection_table(its);
> +
> +out:
> +	unlock_all_vcpus(kvm);
> +	mutex_unlock(&kvm->lock);
> +	return ret;
>  }
>  
>  /**
> @@ -1663,7 +1719,41 @@ static int vgic_its_save_tables_v0(struct vgic_its *its)
>   */
>  static int vgic_its_restore_tables_v0(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	struct kvm *kvm = its->dev->kvm;
> +	struct kvm_vcpu *vcpu;
> +	int ret, c;
> +
> +	mutex_lock(&kvm->lock);
> +
> +	if (!lock_all_vcpus(kvm)) {
> +		mutex_unlock(&kvm->lock);
> +		return -EBUSY;
> +	}
> +
> +	ret = vgic_its_restore_collection_table(its);
> +	if (ret)
> +		goto out;
> +
> +	ret = vgic_its_restore_device_tables(its);
> +
> +	kvm_for_each_vcpu(c, vcpu, kvm) {
> +		ret = its_sync_lpi_pending_table(vcpu);
> +		if (ret)
> +			break;
> +	}
> +
> +out:
> +	unlock_all_vcpus(kvm);
> +	mutex_unlock(&kvm->lock);
> +
> +	if (ret)
> +		return ret;
> +
> +	/*
> +	 * On restore path, MSI injections can happen before the
> +	 * first VCPU run so let's complete the GIC init here.
> +	 */
> +	return kvm_vgic_map_resources(its->dev->kvm);

wait, this is supposed to run when everything else is done and just
before running a VCPU, but now we run this way earlier?  Is that safe.
It feels incredibly dangerous.

Otherwise I'll wait with looking at this patch in more detail until you
respin based on not doing the its_sync_lpi_pending_table() here anymore
as a result of the ABI discussion.

Thanks,
-Christoffer

>  }
>  
>  static int vgic_its_commit_v0(struct vgic_its *its)
> @@ -1718,6 +1808,10 @@ static int vgic_its_has_attr(struct kvm_device *dev,
>  		switch (attr->attr) {
>  		case KVM_DEV_ARM_VGIC_CTRL_INIT:
>  			return 0;
> +		case KVM_DEV_ARM_ITS_SAVE_TABLES:
> +			return 0;
> +		case KVM_DEV_ARM_ITS_RESTORE_TABLES:
> +			return 0;
>  		}
>  		break;
>  	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS:
> @@ -1753,14 +1847,20 @@ static int vgic_its_set_attr(struct kvm_device *dev,
>  
>  		return 0;
>  	}
> -	case KVM_DEV_ARM_VGIC_GRP_CTRL:
> +	case KVM_DEV_ARM_VGIC_GRP_CTRL: {
> +		const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +
>  		switch (attr->attr) {
>  		case KVM_DEV_ARM_VGIC_CTRL_INIT:
>  			its->initialized = true;
>  
>  			return 0;
> +		case KVM_DEV_ARM_ITS_SAVE_TABLES:
> +			return abi->save_tables(its);
> +		case KVM_DEV_ARM_ITS_RESTORE_TABLES:
> +			return abi->restore_tables(its);
>  		}
> -		break;
> +	}
>  	case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
>  		u64 __user *uaddr = (u64 __user *)(long)attr->addr;
>  		u64 reg;
> -- 
> 2.5.5
> 

Re: [PATCH v5 13/22] KVM: arm64: vgic-its: Check the device id matches TYPER DEVBITS range

[Auger Eric]

Hi,

On 27/04/2017 18:48, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:25PM +0200, Eric Auger wrote:
>> On MAPD we currently check the device id can be stored in the device table.
>> Let's first check it can be encoded within the range defined by TYPER
>> DEVBITS.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>>
>> v4 -> v5:
>> - use GIC_ENCODE_SZ macro
>>
>> v3 -> v4:
>> - VITS_TYPER_DEVBITS set to 16 for homogeneity
>> - use BIT_ULL
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 10 +++++++---
>>  1 file changed, 7 insertions(+), 3 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 757598d..de1ed6d 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -194,6 +194,7 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  #define GIC_LPI_OFFSET 8192
>>  
>>  #define VITS_TYPER_IDBITS 16
>> +#define VITS_TYPER_DEVBITS 16
>>  
>>  /*
>>   * Finds and returns a collection in the ITS collection table.
>> @@ -394,7 +395,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>>  	 * To avoid memory waste in the guest, we keep the number of IDBits and
>>  	 * DevBits low - as least for the time being.
>>  	 */
>> -	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>> +	reg |= GIC_ENCODE_SZ(VITS_TYPER_DEVBITS, 5) << GITS_TYPER_DEVBITS_SHIFT;
>>  	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
>>  	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
>>  
>> @@ -639,10 +640,10 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>>   * Check whether an ID can be stored into the corresponding guest table.
>>   * For a direct table this is pretty easy, but gets a bit nasty for
>>   * indirect tables. We check whether the resulting guest physical address
>> - * is actually valid (covered by a memslot and guest accessbible).
>> + * is actually valid (covered by a memslot and guest accessible).
>>   * For this we have to read the respective first level entry.
>>   */
>> -static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
>> +static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
>>  {
>>  	int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
>>  	int index;
>> @@ -650,6 +651,9 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
>>  	gfn_t gfn;
>>  	int esz = GITS_BASER_ENTRY_SIZE(baser);
>>  
>> +	if (id >= BIT_ULL(VITS_TYPER_DEVBITS))
>> +		return false;
>> +
> 
> Isn't vgic_its_check_id called with both a device id and a collection
> id?  How can this then be a valid check?

Hum yes that's correct. In practice the test is correct for collection
ID too since our virtual implementation supports collections in memory
(GITS_TYPER.CIL ==0, spec 8.19.8) and a 16-bit collection ID is
supported. But this is by chance and this really deserves some proper
differentiation. Thank you for spotting that one too!

Thanks

Eric
> 
>>  	if (!(baser & GITS_BASER_INDIRECT)) {
>>  		phys_addr_t addr;
>>  
>> -- 
>> 2.5.5
>>
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 13/22] KVM: arm64: vgic-its: Check the device id matches TYPER DEVBITS range

[Auger Eric]

Hi,

On 27/04/2017 18:48, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:25PM +0200, Eric Auger wrote:
>> On MAPD we currently check the device id can be stored in the device table.
>> Let's first check it can be encoded within the range defined by TYPER
>> DEVBITS.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>>
>> v4 -> v5:
>> - use GIC_ENCODE_SZ macro
>>
>> v3 -> v4:
>> - VITS_TYPER_DEVBITS set to 16 for homogeneity
>> - use BIT_ULL
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 10 +++++++---
>>  1 file changed, 7 insertions(+), 3 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 757598d..de1ed6d 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -194,6 +194,7 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  #define GIC_LPI_OFFSET 8192
>>  
>>  #define VITS_TYPER_IDBITS 16
>> +#define VITS_TYPER_DEVBITS 16
>>  
>>  /*
>>   * Finds and returns a collection in the ITS collection table.
>> @@ -394,7 +395,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
>>  	 * To avoid memory waste in the guest, we keep the number of IDBits and
>>  	 * DevBits low - as least for the time being.
>>  	 */
>> -	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>> +	reg |= GIC_ENCODE_SZ(VITS_TYPER_DEVBITS, 5) << GITS_TYPER_DEVBITS_SHIFT;
>>  	reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
>>  	reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
>>  
>> @@ -639,10 +640,10 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>>   * Check whether an ID can be stored into the corresponding guest table.
>>   * For a direct table this is pretty easy, but gets a bit nasty for
>>   * indirect tables. We check whether the resulting guest physical address
>> - * is actually valid (covered by a memslot and guest accessbible).
>> + * is actually valid (covered by a memslot and guest accessible).
>>   * For this we have to read the respective first level entry.
>>   */
>> -static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
>> +static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
>>  {
>>  	int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
>>  	int index;
>> @@ -650,6 +651,9 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
>>  	gfn_t gfn;
>>  	int esz = GITS_BASER_ENTRY_SIZE(baser);
>>  
>> +	if (id >= BIT_ULL(VITS_TYPER_DEVBITS))
>> +		return false;
>> +
> 
> Isn't vgic_its_check_id called with both a device id and a collection
> id?  How can this then be a valid check?

Hum yes that's correct. In practice the test is correct for collection
ID too since our virtual implementation supports collections in memory
(GITS_TYPER.CIL ==0, spec 8.19.8) and a 16-bit collection ID is
supported. But this is by chance and this really deserves some proper
differentiation. Thank you for spotting that one too!

Thanks

Eric
> 
>>  	if (!(baser & GITS_BASER_INDIRECT)) {
>>  		phys_addr_t addr;
>>  
>> -- 
>> 2.5.5
>>
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Thu, Apr 27, 2017 at 07:14:29PM +0200, Auger Eric wrote:
> Hi Marc, Christoffer,
> 
> On 27/04/2017 18:23, Marc Zyngier wrote:
> > On 27/04/17 16:29, Auger Eric wrote:
> >>
> >>
> >> On 27/04/2017 16:45, Christoffer Dall wrote:
> >>> Hi Eric,
> >>>
> >>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
> >>>> On 27/04/2017 13:02, Christoffer Dall wrote:
> >>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> >>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
> >>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>>>>>>>> Add description for how to access ITS registers and how to save/restore
> >>>>>>>>>> ITS tables into/from memory.
> >>>>>>>>>>
> >>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> v4 -> v5:
> >>>>>>>>>> - take into account Christoffer's comments
> >>>>>>>>>> - pending table save on GICV3 side now
> >>>>>>>>>>
> >>>>>>>>>> v3 -> v4:
> >>>>>>>>>> - take into account Peter's comments:
> >>>>>>>>>>   - typos
> >>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>>>>>>>   - add a validity bit in DTE
> >>>>>>>>>>   - document all fields in CTE and ITE
> >>>>>>>>>>   - document ABI revision
> >>>>>>>>>> - take into account Andre's comments:
> >>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>>>>>>>   - document 64b registers only can be accessed with 64b access
> >>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>>>>>>>
> >>>>>>>>>> v1 -> v2:
> >>>>>>>>>> - DTE and ITE now are 8 bytes
> >>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
> >>>>>>>>>> - use ITE name instead of ITTE
> >>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>>>>>>>> - mentions LE layout
> >>>>>>>>>> ---
> >>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>>>>>>>  1 file changed, 99 insertions(+)
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> index 6081a5b..b5f010d 100644
> >>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
> >>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>>>>>>>        request the initialization of the ITS, no additional parameter in
> >>>>>>>>>>        kvm_device_attr.addr.
> >>>>>>>>>> +
> >>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>>>>>>>> +      by the guest in corresponding registers/table entries.
> >>>>>>>>>> +
> >>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
> >>>>>>>>>> +
> >>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>>>>>>>> +
> >>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>>>>>>>> +
> >>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
> >>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>>>>>>>> +
> >>>>>>>>>
> >>>>>>>>> what is the expected sequence of operations.  For example, to restore
> >>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>>>>>>>
> >>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
> >>>>>>>>
> >>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >>>>>>>> that the pending table is read. But the whole pending table is not read
> >>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
> >>>>>>>> restored previously.
> >>>>>>>>
> >>>>>>>> I became aware that the pending table sync is done twice, once in the
> >>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
> >>>>>>>> leave this order specification, I should be able to remove the sync on
> >>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
> >>>>>>>> been done at the very end.
> >>>>>>>
> >>>>>>> I'm sorry, I'm a bit confused.  Do we not need
> >>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> >>>>>>
> >>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
> >>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> >>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> >>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>> which is not requested I think since GITS_CTLR restore does it already.
> >>>>>
> >>>>> Shouldn't restoring the pending tables happen when restoring some
> >>>>> redeistributor state and not anything related to the ITS?
> >>>>
> >>>> Marc wrote:
> >>>> "
> >>>> I don't think you necessarily need a coarse map. When restoring the ITS
> >>>> tables, you can always read the pending bit when creating the LPI
> >>>> structure (it has been written to RAM at save time). Note that we
> >>>> already do something like this in vgic_enable_lpis().
> >>>> "
> >>>>
> >>>> This is currently what is implemented I think. the pending tables are
> >>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
> >>>> also on on ITS table restore
> >>>>
> >>>> The problematic is: Either you know in advance which LPI INTIDare used
> >>>> or you need to parse the whole pending table (possibly using the 1st kB
> >>>> as coarse mapping).
> >>>>
> >>>> If you don't know the LPI INTIDs in advance it is only possible to
> >>>> restore the pending bit of pending LPIs. At that time you would
> >>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
> >>>> ITS ITT you would do the same for those which were not pending. Looks
> >>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
> >>>>
> >>>> Otherwise we would need to add another dependency between RDIST pending
> >>>> table restore and ITS table restore but this looks even more weird, no?
> >>>>
> >>>>
> >>> So I just sat down with Andre and Marc and we tried to work through this
> >>> and came up with the best scheme.  I apologize in advance for the
> >>> one-way nature of this e-mail, and I am of course open to discussing the
> >>> following proposal again if you do not agree.
> >>>
> >>> What I think this document should say, is that the following ordering
> >>> must be followed when restoring the GIC and the ITS:
> >>>
> >>>   First, restore all guest memory
> >>>
> >>>   Second, restore ALL redistributors
> >>>
> >>>   Third, restore the ITS, in the following order:
> >>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >>>     2. Restore GITS_CBASER
> >>>     3. Restore all other GITS_ registers, except GITS_CTLR!
> >>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> >>>     5. Restore GITS_CTLR
> >>>
> >>> The rationale is that we really want the redistributor and the ITS
> >>> restore to be independent and follow the architecture.  This means that
> >>> our ABI for the redistributor should still work without restoring an ITS
> >>> (if we ever decide to support LPIs for KVM without the ITS).
> >>
> >> OK. Note I already mentioned that GICv3 must be restored before the ITS.
> >> To me this comprised the RDIST.
> >>
> >> I understand the above description of the ordering comes in addition to
> >> the existing text, right? in other words I keep the GITS_READR,
> >> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
> >> section.
> >>
> >>>
> >>> In terms of our current implementation this means that vgic_add_lpi()
> >>> should ask the redistributor what the state of the LPI is (priority,
> >>> enabled, pending).
> >> this practically means I move update_lpi_config call from
> >> vgic_its_restore_ite to vgic_add_lpi(). OK
> >>
> >> However for getting the LPI pending state I must know which RDIST the
> >> LPI is attached to. This is not known at LPI allocation time. Do I
> >> misunderstand something?
> > 
> > Once you have rebuilt the ITS data structures and allocated the IRQ
> > structures, you should have a target_cpu field pointing to the right
> > vcpu. From there, you can surely find the corresponding redistributor
> > and the pending table.
> Yes that's understood but Christoffer said "vgic_add_lpi() should ask
> the redistributor what the state of the LPI is (priority,enabled,
> pending)." Fetching the properties is fine.
> 
> vgic_add_lpi() is called before update_affinity_ite() which uses
> ite->irq and sets the target_vcpu.
> 
> Well at least this requires some function reshape. I will investigate
> though.

Thanks.  If it looks impossible, let me know, and I can help having a
look at the code.

-Christoffer

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Thu, Apr 27, 2017 at 07:14:29PM +0200, Auger Eric wrote:
> Hi Marc, Christoffer,
> 
> On 27/04/2017 18:23, Marc Zyngier wrote:
> > On 27/04/17 16:29, Auger Eric wrote:
> >>
> >>
> >> On 27/04/2017 16:45, Christoffer Dall wrote:
> >>> Hi Eric,
> >>>
> >>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
> >>>> On 27/04/2017 13:02, Christoffer Dall wrote:
> >>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> >>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
> >>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>>>>>>>> Add description for how to access ITS registers and how to save/restore
> >>>>>>>>>> ITS tables into/from memory.
> >>>>>>>>>>
> >>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> v4 -> v5:
> >>>>>>>>>> - take into account Christoffer's comments
> >>>>>>>>>> - pending table save on GICV3 side now
> >>>>>>>>>>
> >>>>>>>>>> v3 -> v4:
> >>>>>>>>>> - take into account Peter's comments:
> >>>>>>>>>>   - typos
> >>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>>>>>>>   - add a validity bit in DTE
> >>>>>>>>>>   - document all fields in CTE and ITE
> >>>>>>>>>>   - document ABI revision
> >>>>>>>>>> - take into account Andre's comments:
> >>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>>>>>>>   - document 64b registers only can be accessed with 64b access
> >>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>>>>>>>
> >>>>>>>>>> v1 -> v2:
> >>>>>>>>>> - DTE and ITE now are 8 bytes
> >>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
> >>>>>>>>>> - use ITE name instead of ITTE
> >>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>>>>>>>> - mentions LE layout
> >>>>>>>>>> ---
> >>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>>>>>>>  1 file changed, 99 insertions(+)
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> index 6081a5b..b5f010d 100644
> >>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
> >>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>>>>>>>        request the initialization of the ITS, no additional parameter in
> >>>>>>>>>>        kvm_device_attr.addr.
> >>>>>>>>>> +
> >>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>>>>>>>> +      by the guest in corresponding registers/table entries.
> >>>>>>>>>> +
> >>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
> >>>>>>>>>> +
> >>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>>>>>>>> +
> >>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>>>>>>>> +
> >>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
> >>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>>>>>>>> +
> >>>>>>>>>
> >>>>>>>>> what is the expected sequence of operations.  For example, to restore
> >>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>>>>>>>
> >>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
> >>>>>>>>
> >>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >>>>>>>> that the pending table is read. But the whole pending table is not read
> >>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
> >>>>>>>> restored previously.
> >>>>>>>>
> >>>>>>>> I became aware that the pending table sync is done twice, once in the
> >>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
> >>>>>>>> leave this order specification, I should be able to remove the sync on
> >>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
> >>>>>>>> been done at the very end.
> >>>>>>>
> >>>>>>> I'm sorry, I'm a bit confused.  Do we not need
> >>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> >>>>>>
> >>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
> >>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> >>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> >>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>> which is not requested I think since GITS_CTLR restore does it already.
> >>>>>
> >>>>> Shouldn't restoring the pending tables happen when restoring some
> >>>>> redeistributor state and not anything related to the ITS?
> >>>>
> >>>> Marc wrote:
> >>>> "
> >>>> I don't think you necessarily need a coarse map. When restoring the ITS
> >>>> tables, you can always read the pending bit when creating the LPI
> >>>> structure (it has been written to RAM at save time). Note that we
> >>>> already do something like this in vgic_enable_lpis().
> >>>> "
> >>>>
> >>>> This is currently what is implemented I think. the pending tables are
> >>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
> >>>> also on on ITS table restore
> >>>>
> >>>> The problematic is: Either you know in advance which LPI INTIDare used
> >>>> or you need to parse the whole pending table (possibly using the 1st kB
> >>>> as coarse mapping).
> >>>>
> >>>> If you don't know the LPI INTIDs in advance it is only possible to
> >>>> restore the pending bit of pending LPIs. At that time you would
> >>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
> >>>> ITS ITT you would do the same for those which were not pending. Looks
> >>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
> >>>>
> >>>> Otherwise we would need to add another dependency between RDIST pending
> >>>> table restore and ITS table restore but this looks even more weird, no?
> >>>>
> >>>>
> >>> So I just sat down with Andre and Marc and we tried to work through this
> >>> and came up with the best scheme.  I apologize in advance for the
> >>> one-way nature of this e-mail, and I am of course open to discussing the
> >>> following proposal again if you do not agree.
> >>>
> >>> What I think this document should say, is that the following ordering
> >>> must be followed when restoring the GIC and the ITS:
> >>>
> >>>   First, restore all guest memory
> >>>
> >>>   Second, restore ALL redistributors
> >>>
> >>>   Third, restore the ITS, in the following order:
> >>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >>>     2. Restore GITS_CBASER
> >>>     3. Restore all other GITS_ registers, except GITS_CTLR!
> >>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> >>>     5. Restore GITS_CTLR
> >>>
> >>> The rationale is that we really want the redistributor and the ITS
> >>> restore to be independent and follow the architecture.  This means that
> >>> our ABI for the redistributor should still work without restoring an ITS
> >>> (if we ever decide to support LPIs for KVM without the ITS).
> >>
> >> OK. Note I already mentioned that GICv3 must be restored before the ITS.
> >> To me this comprised the RDIST.
> >>
> >> I understand the above description of the ordering comes in addition to
> >> the existing text, right? in other words I keep the GITS_READR,
> >> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
> >> section.
> >>
> >>>
> >>> In terms of our current implementation this means that vgic_add_lpi()
> >>> should ask the redistributor what the state of the LPI is (priority,
> >>> enabled, pending).
> >> this practically means I move update_lpi_config call from
> >> vgic_its_restore_ite to vgic_add_lpi(). OK
> >>
> >> However for getting the LPI pending state I must know which RDIST the
> >> LPI is attached to. This is not known at LPI allocation time. Do I
> >> misunderstand something?
> > 
> > Once you have rebuilt the ITS data structures and allocated the IRQ
> > structures, you should have a target_cpu field pointing to the right
> > vcpu. From there, you can surely find the corresponding redistributor
> > and the pending table.
> Yes that's understood but Christoffer said "vgic_add_lpi() should ask
> the redistributor what the state of the LPI is (priority,enabled,
> pending)." Fetching the properties is fine.
> 
> vgic_add_lpi() is called before update_affinity_ite() which uses
> ite->irq and sets the target_vcpu.
> 
> Well at least this requires some function reshape. I will investigate
> though.

Thanks.  If it looks impossible, let me know, and I can help having a
look at the code.

-Christoffer

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Christoffer,

On 27/04/2017 18:38, Christoffer Dall wrote:
> On Thu, Apr 27, 2017 at 05:29:35PM +0200, Auger Eric wrote:
>>
>>
>> On 27/04/2017 16:45, Christoffer Dall wrote:
>>> Hi Eric,
>>>
>>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> v4 -> v5:
>>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>>
>>>>>>>>>> v3 -> v4:
>>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>>   - typos
>>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>>   - document ABI revision
>>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>>
>>>>>>>>>> v1 -> v2:
>>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>>> - mentions LE layout
>>>>>>>>>> ---
>>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>>> +
>>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>>> +
>>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>>> +
>>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>>
>>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>>
>>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>>> restored previously.
>>>>>>>>
>>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>>> been done at the very end.
>>>>>>>
>>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>>
>>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>>
>>>>> Shouldn't restoring the pending tables happen when restoring some
>>>>> redeistributor state and not anything related to the ITS?
>>>>
>>>> Marc wrote:
>>>> "
>>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>>> tables, you can always read the pending bit when creating the LPI
>>>> structure (it has been written to RAM at save time). Note that we
>>>> already do something like this in vgic_enable_lpis().
>>>> "
>>>>
>>>> This is currently what is implemented I think. the pending tables are
>>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>>> also on on ITS table restore
>>>>
>>>> The problematic is: Either you know in advance which LPI INTIDare used
>>>> or you need to parse the whole pending table (possibly using the 1st kB
>>>> as coarse mapping).
>>>>
>>>> If you don't know the LPI INTIDs in advance it is only possible to
>>>> restore the pending bit of pending LPIs. At that time you would
>>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>>> ITS ITT you would do the same for those which were not pending. Looks
>>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>>
>>>> Otherwise we would need to add another dependency between RDIST pending
>>>> table restore and ITS table restore but this looks even more weird, no?
>>>>
>>>>
>>> So I just sat down with Andre and Marc and we tried to work through this
>>> and came up with the best scheme.  I apologize in advance for the
>>> one-way nature of this e-mail, and I am of course open to discussing the
>>> following proposal again if you do not agree.
>>>
>>> What I think this document should say, is that the following ordering
>>> must be followed when restoring the GIC and the ITS:
>>>
>>>   First, restore all guest memory
>>>
>>>   Second, restore ALL redistributors
>>>
>>>   Third, restore the ITS, in the following order:
>>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>>     2. Restore GITS_CBASER
>>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>>     5. Restore GITS_CTLR
>>>
>>> The rationale is that we really want the redistributor and the ITS
>>> restore to be independent and follow the architecture.  This means that
>>> our ABI for the redistributor should still work without restoring an ITS
>>> (if we ever decide to support LPIs for KVM without the ITS).
>>
>> OK. Note I already mentioned that GICv3 must be restored before the ITS.
>> To me this comprised the RDIST.
> 
> Possibly, but I think it's good to write out the whole thing so we
> clearly understand the flow.  That could better be achieved by
> correcting my proposed text above to say something like "Second, restore
> ALL redistributors to ensure the pending and configuration tables can be
> read."
> 
>>
>> I understand the above description of the ordering comes in addition to
>> the existing text, right? 
> 
> Yes
> 
>> in other words I keep the GITS_READR,
>> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
>> section.
>>
> 
> Yes.  But you don't need to do any reading of the pending table on any
> of the restore operations.
well you told me to do it on vgic_add_lpi(). This is obviously called on
ITS table restore. /me confused. Obviously this is implicit and should
not be documented. Is that what you meant? btw this is not documented
atm I think.
> 
> Btw., I think it's GITS_CREADR, not GITS_READR.
right.

Thanks

Eric
> 
>>>
>>> In terms of our current implementation this means that vgic_add_lpi()
>>> should ask the redistributor what the state of the LPI is (priority,
>>> enabled, pending).
>> this practically means I move update_lpi_config call from
>> vgic_its_restore_ite to vgic_add_lpi(). OK
> 
> Pretty much, yes.
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Christoffer,

On 27/04/2017 18:38, Christoffer Dall wrote:
> On Thu, Apr 27, 2017 at 05:29:35PM +0200, Auger Eric wrote:
>>
>>
>> On 27/04/2017 16:45, Christoffer Dall wrote:
>>> Hi Eric,
>>>
>>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> v4 -> v5:
>>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>>
>>>>>>>>>> v3 -> v4:
>>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>>   - typos
>>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>>   - document ABI revision
>>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>>
>>>>>>>>>> v1 -> v2:
>>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>>> - mentions LE layout
>>>>>>>>>> ---
>>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>>> +
>>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>>> +
>>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>>> +
>>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>>
>>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>>
>>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>>> restored previously.
>>>>>>>>
>>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>>> been done at the very end.
>>>>>>>
>>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>>
>>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>>
>>>>> Shouldn't restoring the pending tables happen when restoring some
>>>>> redeistributor state and not anything related to the ITS?
>>>>
>>>> Marc wrote:
>>>> "
>>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>>> tables, you can always read the pending bit when creating the LPI
>>>> structure (it has been written to RAM at save time). Note that we
>>>> already do something like this in vgic_enable_lpis().
>>>> "
>>>>
>>>> This is currently what is implemented I think. the pending tables are
>>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>>> also on on ITS table restore
>>>>
>>>> The problematic is: Either you know in advance which LPI INTIDare used
>>>> or you need to parse the whole pending table (possibly using the 1st kB
>>>> as coarse mapping).
>>>>
>>>> If you don't know the LPI INTIDs in advance it is only possible to
>>>> restore the pending bit of pending LPIs. At that time you would
>>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>>> ITS ITT you would do the same for those which were not pending. Looks
>>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>>
>>>> Otherwise we would need to add another dependency between RDIST pending
>>>> table restore and ITS table restore but this looks even more weird, no?
>>>>
>>>>
>>> So I just sat down with Andre and Marc and we tried to work through this
>>> and came up with the best scheme.  I apologize in advance for the
>>> one-way nature of this e-mail, and I am of course open to discussing the
>>> following proposal again if you do not agree.
>>>
>>> What I think this document should say, is that the following ordering
>>> must be followed when restoring the GIC and the ITS:
>>>
>>>   First, restore all guest memory
>>>
>>>   Second, restore ALL redistributors
>>>
>>>   Third, restore the ITS, in the following order:
>>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>>     2. Restore GITS_CBASER
>>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>>     5. Restore GITS_CTLR
>>>
>>> The rationale is that we really want the redistributor and the ITS
>>> restore to be independent and follow the architecture.  This means that
>>> our ABI for the redistributor should still work without restoring an ITS
>>> (if we ever decide to support LPIs for KVM without the ITS).
>>
>> OK. Note I already mentioned that GICv3 must be restored before the ITS.
>> To me this comprised the RDIST.
> 
> Possibly, but I think it's good to write out the whole thing so we
> clearly understand the flow.  That could better be achieved by
> correcting my proposed text above to say something like "Second, restore
> ALL redistributors to ensure the pending and configuration tables can be
> read."
> 
>>
>> I understand the above description of the ordering comes in addition to
>> the existing text, right? 
> 
> Yes
> 
>> in other words I keep the GITS_READR,
>> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
>> section.
>>
> 
> Yes.  But you don't need to do any reading of the pending table on any
> of the restore operations.
well you told me to do it on vgic_add_lpi(). This is obviously called on
ITS table restore. /me confused. Obviously this is implicit and should
not be documented. Is that what you meant? btw this is not documented
atm I think.
> 
> Btw., I think it's GITS_CREADR, not GITS_READR.
right.

Thanks

Eric
> 
>>>
>>> In terms of our current implementation this means that vgic_add_lpi()
>>> should ask the redistributor what the state of the LPI is (priority,
>>> enabled, pending).
>> this practically means I move update_lpi_config call from
>> vgic_its_restore_ite to vgic_add_lpi(). OK
> 
> Pretty much, yes.
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 11/22] KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors

[Christoffer Dall]

On Thu, Apr 27, 2017 at 07:15:27PM +0200, Auger Eric wrote:
> Hi,
> On 27/04/2017 18:25, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:23PM +0200, Eric Auger wrote:
> >> Up to now the MAPD's ITT size field has been ignored. It encodes
> >> the number of eventid bit minus 1. It should be used to check
> >> the eventid when a MAPTI command is issued on a device. Let's
> >> store the number of eventid bits in the its_device and do the
> >> check on MAPTI. Also make sure the ITT size field does
> >> not exceed the GITS_TYPER IDBITS field.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - its_cmd_get_size macro now returns the actual number of eventid bits
> >> - use GIC_ENCODE_SZ macro to encode ID_bits
> >>
> >> v3 -> v4:
> >> - VITS_TYPER_IDBITS set to 16 to be homogeneous with VITS_ESZ definition
> >>   and correct users
> >> - nb_eventid_bits correspond to the actual number of eventid bits
> >> ---
> >>  include/linux/irqchip/arm-gic-v3.h |  2 ++
> >>  virt/kvm/arm/vgic/vgic-its.c       | 15 ++++++++++++++-
> >>  2 files changed, 16 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> >> index 54c20bd..0c6798c 100644
> >> --- a/include/linux/irqchip/arm-gic-v3.h
> >> +++ b/include/linux/irqchip/arm-gic-v3.h
> >> @@ -345,9 +345,11 @@
> >>  #define E_ITS_INT_UNMAPPED_INTERRUPT		0x010307
> >>  #define E_ITS_CLEAR_UNMAPPED_INTERRUPT		0x010507
> >>  #define E_ITS_MAPD_DEVICE_OOR			0x010801
> >> +#define E_ITS_MAPD_ITTSIZE_OOR			0x010802
> >>  #define E_ITS_MAPC_PROCNUM_OOR			0x010902
> >>  #define E_ITS_MAPC_COLLECTION_OOR		0x010903
> >>  #define E_ITS_MAPTI_UNMAPPED_DEVICE		0x010a04
> >> +#define E_ITS_MAPTI_ID_OOR			0x010a05
> >>  #define E_ITS_MAPTI_PHYSICALID_OOR		0x010a06
> >>  #define E_ITS_INV_UNMAPPED_INTERRUPT		0x010c07
> >>  #define E_ITS_INVALL_UNMAPPED_COLLECTION	0x010d09
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 1b5797e..0f3c8f3 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -105,6 +105,7 @@ struct its_device {
> >>  
> >>  	/* the head for the list of ITTEs */
> >>  	struct list_head itt_head;
> >> +	u32 nb_eventid_bits;
> > 
> > what does nb_ mean?
> number ;-)
> 

Ah, I think the plural form of bits here implied that, but if you
insist, then can you call it num_eventid_bits?

I was like, "notifier block", "nota bene", I'm lost...

Thanks,
-Christoffer

[PATCH v5 11/22] KVM: arm64: vgic-its: Interpret MAPD Size field and check related errors

[Christoffer Dall]

On Thu, Apr 27, 2017 at 07:15:27PM +0200, Auger Eric wrote:
> Hi,
> On 27/04/2017 18:25, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:23PM +0200, Eric Auger wrote:
> >> Up to now the MAPD's ITT size field has been ignored. It encodes
> >> the number of eventid bit minus 1. It should be used to check
> >> the eventid when a MAPTI command is issued on a device. Let's
> >> store the number of eventid bits in the its_device and do the
> >> check on MAPTI. Also make sure the ITT size field does
> >> not exceed the GITS_TYPER IDBITS field.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - its_cmd_get_size macro now returns the actual number of eventid bits
> >> - use GIC_ENCODE_SZ macro to encode ID_bits
> >>
> >> v3 -> v4:
> >> - VITS_TYPER_IDBITS set to 16 to be homogeneous with VITS_ESZ definition
> >>   and correct users
> >> - nb_eventid_bits correspond to the actual number of eventid bits
> >> ---
> >>  include/linux/irqchip/arm-gic-v3.h |  2 ++
> >>  virt/kvm/arm/vgic/vgic-its.c       | 15 ++++++++++++++-
> >>  2 files changed, 16 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> >> index 54c20bd..0c6798c 100644
> >> --- a/include/linux/irqchip/arm-gic-v3.h
> >> +++ b/include/linux/irqchip/arm-gic-v3.h
> >> @@ -345,9 +345,11 @@
> >>  #define E_ITS_INT_UNMAPPED_INTERRUPT		0x010307
> >>  #define E_ITS_CLEAR_UNMAPPED_INTERRUPT		0x010507
> >>  #define E_ITS_MAPD_DEVICE_OOR			0x010801
> >> +#define E_ITS_MAPD_ITTSIZE_OOR			0x010802
> >>  #define E_ITS_MAPC_PROCNUM_OOR			0x010902
> >>  #define E_ITS_MAPC_COLLECTION_OOR		0x010903
> >>  #define E_ITS_MAPTI_UNMAPPED_DEVICE		0x010a04
> >> +#define E_ITS_MAPTI_ID_OOR			0x010a05
> >>  #define E_ITS_MAPTI_PHYSICALID_OOR		0x010a06
> >>  #define E_ITS_INV_UNMAPPED_INTERRUPT		0x010c07
> >>  #define E_ITS_INVALL_UNMAPPED_COLLECTION	0x010d09
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 1b5797e..0f3c8f3 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -105,6 +105,7 @@ struct its_device {
> >>  
> >>  	/* the head for the list of ITTEs */
> >>  	struct list_head itt_head;
> >> +	u32 nb_eventid_bits;
> > 
> > what does nb_ mean?
> number ;-)
> 

Ah, I think the plural form of bits here implied that, but if you
insist, then can you call it num_eventid_bits?

I was like, "notifier block", "nota bene", I'm lost...

Thanks,
-Christoffer

Re: [PATCH v5 15/22] KVM: arm64: vgic-its: vgic_its_alloc_ite/device

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:27PM +0200, Eric Auger wrote:
> Add two new helpers to allocate an its ite and an its device.
> This will avoid duplication on restore path.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
> 
> ---
> 
> v4 -> v5:
> - add Marc's A-b
> 
> v3 -> v4:
> - fix allocation
> - add comment about its_lock mutex hold
> 
> v1 -> v2:
> - report itt_size fix and remove ITE_SIZE
> - s/itte/ite/g
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 73 ++++++++++++++++++++++++++++++--------------
>  1 file changed, 50 insertions(+), 23 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 55267ab..56c5123 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -742,6 +742,27 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>  	kfree(collection);
>  }
>  
> +/* Must be called with its_lock mutex held */
> +static int vgic_its_alloc_ite(struct its_device *device,
> +			       struct its_ite **itep,
> +			       struct its_collection *collection,
> +			       u32 lpi_id, u32 event_id)
> +{
> +	struct its_ite *ite;
> +
> +	ite = kzalloc(sizeof(*ite), GFP_KERNEL);
> +	if (!ite)
> +		return -ENOMEM;
> +
> +	ite->event_id	= event_id;
> +	ite->collection = collection;
> +	ite->lpi = lpi_id;
> +
> +	list_add_tail(&ite->ite_list, &device->itt_head);
> +	*itep = ite;
> +	return 0;
> +}

This static function should just return a 'struct its_ite *itep' and on
error you can return ERR_PTR(-ENOMEM).

> +
>  /*
>   * The MAPTI and MAPI commands map LPIs to ITTEs.
>   * Must be called with its_lock mutex held.
> @@ -755,7 +776,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  	struct its_ite *ite;
>  	struct its_device *device;
>  	struct its_collection *collection, *new_coll = NULL;
> -	int lpi_nr;
> +	int lpi_nr, ret;
>  	struct vgic_irq *irq;
>  
>  	device = find_its_device(its, device_id);
> @@ -785,19 +806,13 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  		new_coll = collection;
>  	}
>  
> -	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
> -	if (!ite) {
> +	ret = vgic_its_alloc_ite(device, &ite, collection, lpi_nr, event_id);
> +	if (ret) {
>  		if (new_coll)
>  			vgic_its_free_collection(its, coll_id);
> -		return -ENOMEM;
> +		return ret;
>  	}
>  
> -	ite->event_id	= event_id;
> -	list_add_tail(&ite->ite_list, &device->itt_head);
> -
> -	ite->collection = collection;
> -	ite->lpi = lpi_nr;
> -
>  	irq = vgic_add_lpi(kvm, lpi_nr);
>  	if (IS_ERR(irq)) {
>  		if (new_coll)
> @@ -836,6 +851,29 @@ static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
>  	kfree(device);
>  }
>  
> +/* Must be called with its_lock mutex held */
> +static int vgic_its_alloc_device(struct vgic_its *its,
> +				 struct its_device **devp,
> +				 u32 device_id, gpa_t itt_addr,
> +				 u8 nb_eventid_bits)
> +{
> +	struct its_device *device;
> +
> +	device = kzalloc(sizeof(*device), GFP_KERNEL);
> +	if (!device)
> +		return -ENOMEM;
> +
> +	device->device_id = device_id;
> +	device->itt_addr = itt_addr;
> +	device->nb_eventid_bits = nb_eventid_bits;
> +	INIT_LIST_HEAD(&device->itt_head);
> +
> +	list_add_tail(&device->dev_list, &its->device_list);
> +	*devp = device;
> +
> +	return 0;
> +}

also here, return ERR_PTR

> +
>  /*
>   * MAPD maps or unmaps a device ID to Interrupt Translation Tables (ITTs).
>   * Must be called with the its_lock mutex held.
> @@ -872,19 +910,8 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>  	if (!valid)
>  		return 0;
>  
> -	device = kzalloc(sizeof(struct its_device), GFP_KERNEL);
> -	if (!device)
> -		return -ENOMEM;
> -
> -	device->device_id = device_id;
> -	device->nb_eventid_bits = nb_eventid_bits;
> -	device->itt_addr = itt_addr;
> -
> -	INIT_LIST_HEAD(&device->itt_head);
> -
> -	list_add_tail(&device->dev_list, &its->device_list);
> -
> -	return 0;
> +	return vgic_its_alloc_device(its, &device, device_id,
> +				     itt_addr, nb_eventid_bits);
>  }
>  
>  /*
> -- 
> 2.5.5
> 
Thanks,
-Christoffer

[PATCH v5 15/22] KVM: arm64: vgic-its: vgic_its_alloc_ite/device

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:27PM +0200, Eric Auger wrote:
> Add two new helpers to allocate an its ite and an its device.
> This will avoid duplication on restore path.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Marc Zyngier <marc.zyngier@arm.com>
> 
> ---
> 
> v4 -> v5:
> - add Marc's A-b
> 
> v3 -> v4:
> - fix allocation
> - add comment about its_lock mutex hold
> 
> v1 -> v2:
> - report itt_size fix and remove ITE_SIZE
> - s/itte/ite/g
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 73 ++++++++++++++++++++++++++++++--------------
>  1 file changed, 50 insertions(+), 23 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 55267ab..56c5123 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -742,6 +742,27 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
>  	kfree(collection);
>  }
>  
> +/* Must be called with its_lock mutex held */
> +static int vgic_its_alloc_ite(struct its_device *device,
> +			       struct its_ite **itep,
> +			       struct its_collection *collection,
> +			       u32 lpi_id, u32 event_id)
> +{
> +	struct its_ite *ite;
> +
> +	ite = kzalloc(sizeof(*ite), GFP_KERNEL);
> +	if (!ite)
> +		return -ENOMEM;
> +
> +	ite->event_id	= event_id;
> +	ite->collection = collection;
> +	ite->lpi = lpi_id;
> +
> +	list_add_tail(&ite->ite_list, &device->itt_head);
> +	*itep = ite;
> +	return 0;
> +}

This static function should just return a 'struct its_ite *itep' and on
error you can return ERR_PTR(-ENOMEM).

> +
>  /*
>   * The MAPTI and MAPI commands map LPIs to ITTEs.
>   * Must be called with its_lock mutex held.
> @@ -755,7 +776,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  	struct its_ite *ite;
>  	struct its_device *device;
>  	struct its_collection *collection, *new_coll = NULL;
> -	int lpi_nr;
> +	int lpi_nr, ret;
>  	struct vgic_irq *irq;
>  
>  	device = find_its_device(its, device_id);
> @@ -785,19 +806,13 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
>  		new_coll = collection;
>  	}
>  
> -	ite = kzalloc(sizeof(struct its_ite), GFP_KERNEL);
> -	if (!ite) {
> +	ret = vgic_its_alloc_ite(device, &ite, collection, lpi_nr, event_id);
> +	if (ret) {
>  		if (new_coll)
>  			vgic_its_free_collection(its, coll_id);
> -		return -ENOMEM;
> +		return ret;
>  	}
>  
> -	ite->event_id	= event_id;
> -	list_add_tail(&ite->ite_list, &device->itt_head);
> -
> -	ite->collection = collection;
> -	ite->lpi = lpi_nr;
> -
>  	irq = vgic_add_lpi(kvm, lpi_nr);
>  	if (IS_ERR(irq)) {
>  		if (new_coll)
> @@ -836,6 +851,29 @@ static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
>  	kfree(device);
>  }
>  
> +/* Must be called with its_lock mutex held */
> +static int vgic_its_alloc_device(struct vgic_its *its,
> +				 struct its_device **devp,
> +				 u32 device_id, gpa_t itt_addr,
> +				 u8 nb_eventid_bits)
> +{
> +	struct its_device *device;
> +
> +	device = kzalloc(sizeof(*device), GFP_KERNEL);
> +	if (!device)
> +		return -ENOMEM;
> +
> +	device->device_id = device_id;
> +	device->itt_addr = itt_addr;
> +	device->nb_eventid_bits = nb_eventid_bits;
> +	INIT_LIST_HEAD(&device->itt_head);
> +
> +	list_add_tail(&device->dev_list, &its->device_list);
> +	*devp = device;
> +
> +	return 0;
> +}

also here, return ERR_PTR

> +
>  /*
>   * MAPD maps or unmaps a device ID to Interrupt Translation Tables (ITTs).
>   * Must be called with the its_lock mutex held.
> @@ -872,19 +910,8 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>  	if (!valid)
>  		return 0;
>  
> -	device = kzalloc(sizeof(struct its_device), GFP_KERNEL);
> -	if (!device)
> -		return -ENOMEM;
> -
> -	device->device_id = device_id;
> -	device->nb_eventid_bits = nb_eventid_bits;
> -	device->itt_addr = itt_addr;
> -
> -	INIT_LIST_HEAD(&device->itt_head);
> -
> -	list_add_tail(&device->dev_list, &its->device_list);
> -
> -	return 0;
> +	return vgic_its_alloc_device(its, &device, device_id,
> +				     itt_addr, nb_eventid_bits);
>  }
>  
>  /*
> -- 
> 2.5.5
> 
Thanks,
-Christoffer

Re: [PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Auger Eric]

Hi,

On 27/04/2017 18:43, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:24PM +0200, Eric Auger wrote:
>> Up to now the MAPD ITT_addr had been ignored. We will need it
>> for save/restore. Let's record it in the its_device struct.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - its_cmd_get_ittaddr macro now returns the actual ITT GPA
>>
>> v3 -> v4:
>> - in vgic_its_cmd_handle_mapd, itt_addr directly is shifted
>> - correct ittaddr bitmask to support 48bit GPA
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 0f3c8f3..757598d 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -106,6 +106,7 @@ struct its_device {
>>  	/* the head for the list of ITTEs */
>>  	struct list_head itt_head;
>>  	u32 nb_eventid_bits;
>> +	gpa_t itt_addr;
>>  	u32 device_id;
>>  };
>>  
>> @@ -569,6 +570,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
>> +#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 47) << 8)
> 
> Why 47?  The ITT_addr field is bits [51:8] unless I'm looking at
> something wrong?

Initially I limited to 48 bits since I found in the code:

/*
 * We only implement 48 bits of PA at the moment, although the ITS
 * supports more. Let's be restrictive here.
 */
#define BASER_ADDRESS(x)        ((x) & GENMASK_ULL(47, 16))
#define CBASER_ADDRESS(x)       ((x) & GENMASK_ULL(47, 12))

But practically as I encode the ITT addr field on 52 bits now in the DTE
there is no reason anymore.

Thanks

Eric
> 
>>  #define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
>>  #define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
>>  
>> @@ -840,6 +842,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	bool valid = its_cmd_get_validbit(its_cmd);
>>  	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
>> +	gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
>>  	struct its_device *device;
>>  
>>  	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
>> @@ -871,6 +874,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>>  
>>  	device->device_id = device_id;
>>  	device->nb_eventid_bits = nb_eventid_bits;
>> +	device->itt_addr = itt_addr;
>>  
>>  	INIT_LIST_HEAD(&device->itt_head);
>>  
>> -- 
>> 2.5.5
>>
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Auger Eric]

Hi,

On 27/04/2017 18:43, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:24PM +0200, Eric Auger wrote:
>> Up to now the MAPD ITT_addr had been ignored. We will need it
>> for save/restore. Let's record it in the its_device struct.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - its_cmd_get_ittaddr macro now returns the actual ITT GPA
>>
>> v3 -> v4:
>> - in vgic_its_cmd_handle_mapd, itt_addr directly is shifted
>> - correct ittaddr bitmask to support 48bit GPA
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 0f3c8f3..757598d 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -106,6 +106,7 @@ struct its_device {
>>  	/* the head for the list of ITTEs */
>>  	struct list_head itt_head;
>>  	u32 nb_eventid_bits;
>> +	gpa_t itt_addr;
>>  	u32 device_id;
>>  };
>>  
>> @@ -569,6 +570,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
>> +#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 47) << 8)
> 
> Why 47?  The ITT_addr field is bits [51:8] unless I'm looking at
> something wrong?

Initially I limited to 48 bits since I found in the code:

/*
 * We only implement 48 bits of PA at the moment, although the ITS
 * supports more. Let's be restrictive here.
 */
#define BASER_ADDRESS(x)        ((x) & GENMASK_ULL(47, 16))
#define CBASER_ADDRESS(x)       ((x) & GENMASK_ULL(47, 12))

But practically as I encode the ITT addr field on 52 bits now in the DTE
there is no reason anymore.

Thanks

Eric
> 
>>  #define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
>>  #define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
>>  
>> @@ -840,6 +842,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>>  	u32 device_id = its_cmd_get_deviceid(its_cmd);
>>  	bool valid = its_cmd_get_validbit(its_cmd);
>>  	u8 nb_eventid_bits = its_cmd_get_size(its_cmd);
>> +	gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
>>  	struct its_device *device;
>>  
>>  	if (!vgic_its_check_id(its, its->baser_device_table, device_id))
>> @@ -871,6 +874,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>>  
>>  	device->device_id = device_id;
>>  	device->nb_eventid_bits = nb_eventid_bits;
>> +	device->itt_addr = itt_addr;
>>  
>>  	INIT_LIST_HEAD(&device->itt_head);
>>  
>> -- 
>> 2.5.5
>>
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Thu, Apr 27, 2017 at 07:27:22PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 27/04/2017 18:38, Christoffer Dall wrote:
> > On Thu, Apr 27, 2017 at 05:29:35PM +0200, Auger Eric wrote:
> >>
> >>
> >> On 27/04/2017 16:45, Christoffer Dall wrote:
> >>> Hi Eric,
> >>>
> >>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
> >>>> On 27/04/2017 13:02, Christoffer Dall wrote:
> >>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> >>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
> >>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>>>>>>>> Add description for how to access ITS registers and how to save/restore
> >>>>>>>>>> ITS tables into/from memory.
> >>>>>>>>>>
> >>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> v4 -> v5:
> >>>>>>>>>> - take into account Christoffer's comments
> >>>>>>>>>> - pending table save on GICV3 side now
> >>>>>>>>>>
> >>>>>>>>>> v3 -> v4:
> >>>>>>>>>> - take into account Peter's comments:
> >>>>>>>>>>   - typos
> >>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>>>>>>>   - add a validity bit in DTE
> >>>>>>>>>>   - document all fields in CTE and ITE
> >>>>>>>>>>   - document ABI revision
> >>>>>>>>>> - take into account Andre's comments:
> >>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>>>>>>>   - document 64b registers only can be accessed with 64b access
> >>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>>>>>>>
> >>>>>>>>>> v1 -> v2:
> >>>>>>>>>> - DTE and ITE now are 8 bytes
> >>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
> >>>>>>>>>> - use ITE name instead of ITTE
> >>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>>>>>>>> - mentions LE layout
> >>>>>>>>>> ---
> >>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>>>>>>>  1 file changed, 99 insertions(+)
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> index 6081a5b..b5f010d 100644
> >>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
> >>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>>>>>>>        request the initialization of the ITS, no additional parameter in
> >>>>>>>>>>        kvm_device_attr.addr.
> >>>>>>>>>> +
> >>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>>>>>>>> +      by the guest in corresponding registers/table entries.
> >>>>>>>>>> +
> >>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
> >>>>>>>>>> +
> >>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>>>>>>>> +
> >>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>>>>>>>> +
> >>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
> >>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>>>>>>>> +
> >>>>>>>>>
> >>>>>>>>> what is the expected sequence of operations.  For example, to restore
> >>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>>>>>>>
> >>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
> >>>>>>>>
> >>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >>>>>>>> that the pending table is read. But the whole pending table is not read
> >>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
> >>>>>>>> restored previously.
> >>>>>>>>
> >>>>>>>> I became aware that the pending table sync is done twice, once in the
> >>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
> >>>>>>>> leave this order specification, I should be able to remove the sync on
> >>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
> >>>>>>>> been done at the very end.
> >>>>>>>
> >>>>>>> I'm sorry, I'm a bit confused.  Do we not need
> >>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> >>>>>>
> >>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
> >>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> >>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> >>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>> which is not requested I think since GITS_CTLR restore does it already.
> >>>>>
> >>>>> Shouldn't restoring the pending tables happen when restoring some
> >>>>> redeistributor state and not anything related to the ITS?
> >>>>
> >>>> Marc wrote:
> >>>> "
> >>>> I don't think you necessarily need a coarse map. When restoring the ITS
> >>>> tables, you can always read the pending bit when creating the LPI
> >>>> structure (it has been written to RAM at save time). Note that we
> >>>> already do something like this in vgic_enable_lpis().
> >>>> "
> >>>>
> >>>> This is currently what is implemented I think. the pending tables are
> >>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
> >>>> also on on ITS table restore
> >>>>
> >>>> The problematic is: Either you know in advance which LPI INTIDare used
> >>>> or you need to parse the whole pending table (possibly using the 1st kB
> >>>> as coarse mapping).
> >>>>
> >>>> If you don't know the LPI INTIDs in advance it is only possible to
> >>>> restore the pending bit of pending LPIs. At that time you would
> >>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
> >>>> ITS ITT you would do the same for those which were not pending. Looks
> >>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
> >>>>
> >>>> Otherwise we would need to add another dependency between RDIST pending
> >>>> table restore and ITS table restore but this looks even more weird, no?
> >>>>
> >>>>
> >>> So I just sat down with Andre and Marc and we tried to work through this
> >>> and came up with the best scheme.  I apologize in advance for the
> >>> one-way nature of this e-mail, and I am of course open to discussing the
> >>> following proposal again if you do not agree.
> >>>
> >>> What I think this document should say, is that the following ordering
> >>> must be followed when restoring the GIC and the ITS:
> >>>
> >>>   First, restore all guest memory
> >>>
> >>>   Second, restore ALL redistributors
> >>>
> >>>   Third, restore the ITS, in the following order:
> >>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >>>     2. Restore GITS_CBASER
> >>>     3. Restore all other GITS_ registers, except GITS_CTLR!
> >>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> >>>     5. Restore GITS_CTLR
> >>>
> >>> The rationale is that we really want the redistributor and the ITS
> >>> restore to be independent and follow the architecture.  This means that
> >>> our ABI for the redistributor should still work without restoring an ITS
> >>> (if we ever decide to support LPIs for KVM without the ITS).
> >>
> >> OK. Note I already mentioned that GICv3 must be restored before the ITS.
> >> To me this comprised the RDIST.
> > 
> > Possibly, but I think it's good to write out the whole thing so we
> > clearly understand the flow.  That could better be achieved by
> > correcting my proposed text above to say something like "Second, restore
> > ALL redistributors to ensure the pending and configuration tables can be
> > read."
> > 
> >>
> >> I understand the above description of the ordering comes in addition to
> >> the existing text, right? 
> > 
> > Yes
> > 
> >> in other words I keep the GITS_READR,
> >> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
> >> section.
> >>
> > 
> > Yes.  But you don't need to do any reading of the pending table on any
> > of the restore operations.
> well you told me to do it on vgic_add_lpi(). This is obviously called on
> ITS table restore. /me confused. 

Sorry, I meant you do not need to scan the entire table independently
from restoring other state that requires building the data structures.

> Obviously this is implicit and should
> not be documented. Is that what you meant? btw this is not documented
> atm I think.

What I care about is that the ABI is clear and represents what the
architecture does.  So in terms of documentation in the ABI, we don't
need to mention anything about when this is done, but we also do not
need to specify any interaction between the pending tables and the ITS,
beyond that the redestributors and memory must be restored before the
ITS.

Hope this clarifies.

Thanks,
-Christoffer

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Thu, Apr 27, 2017 at 07:27:22PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 27/04/2017 18:38, Christoffer Dall wrote:
> > On Thu, Apr 27, 2017 at 05:29:35PM +0200, Auger Eric wrote:
> >>
> >>
> >> On 27/04/2017 16:45, Christoffer Dall wrote:
> >>> Hi Eric,
> >>>
> >>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
> >>>> On 27/04/2017 13:02, Christoffer Dall wrote:
> >>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> >>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
> >>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>>>>>>>> Add description for how to access ITS registers and how to save/restore
> >>>>>>>>>> ITS tables into/from memory.
> >>>>>>>>>>
> >>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> v4 -> v5:
> >>>>>>>>>> - take into account Christoffer's comments
> >>>>>>>>>> - pending table save on GICV3 side now
> >>>>>>>>>>
> >>>>>>>>>> v3 -> v4:
> >>>>>>>>>> - take into account Peter's comments:
> >>>>>>>>>>   - typos
> >>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>>>>>>>   - add a validity bit in DTE
> >>>>>>>>>>   - document all fields in CTE and ITE
> >>>>>>>>>>   - document ABI revision
> >>>>>>>>>> - take into account Andre's comments:
> >>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>>>>>>>   - document 64b registers only can be accessed with 64b access
> >>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>>>>>>>
> >>>>>>>>>> v1 -> v2:
> >>>>>>>>>> - DTE and ITE now are 8 bytes
> >>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
> >>>>>>>>>> - use ITE name instead of ITTE
> >>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>>>>>>>> - mentions LE layout
> >>>>>>>>>> ---
> >>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>>>>>>>  1 file changed, 99 insertions(+)
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> index 6081a5b..b5f010d 100644
> >>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
> >>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>>>>>>>        request the initialization of the ITS, no additional parameter in
> >>>>>>>>>>        kvm_device_attr.addr.
> >>>>>>>>>> +
> >>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>>>>>>>> +      by the guest in corresponding registers/table entries.
> >>>>>>>>>> +
> >>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
> >>>>>>>>>> +
> >>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>>>>>>>> +
> >>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>>>>>>>> +
> >>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
> >>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>>>>>>>> +
> >>>>>>>>>
> >>>>>>>>> what is the expected sequence of operations.  For example, to restore
> >>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>>>>>>>
> >>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
> >>>>>>>>
> >>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >>>>>>>> that the pending table is read. But the whole pending table is not read
> >>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
> >>>>>>>> restored previously.
> >>>>>>>>
> >>>>>>>> I became aware that the pending table sync is done twice, once in the
> >>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
> >>>>>>>> leave this order specification, I should be able to remove the sync on
> >>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
> >>>>>>>> been done at the very end.
> >>>>>>>
> >>>>>>> I'm sorry, I'm a bit confused.  Do we not need
> >>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> >>>>>>
> >>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
> >>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> >>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> >>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>> which is not requested I think since GITS_CTLR restore does it already.
> >>>>>
> >>>>> Shouldn't restoring the pending tables happen when restoring some
> >>>>> redeistributor state and not anything related to the ITS?
> >>>>
> >>>> Marc wrote:
> >>>> "
> >>>> I don't think you necessarily need a coarse map. When restoring the ITS
> >>>> tables, you can always read the pending bit when creating the LPI
> >>>> structure (it has been written to RAM at save time). Note that we
> >>>> already do something like this in vgic_enable_lpis().
> >>>> "
> >>>>
> >>>> This is currently what is implemented I think. the pending tables are
> >>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
> >>>> also on on ITS table restore
> >>>>
> >>>> The problematic is: Either you know in advance which LPI INTIDare used
> >>>> or you need to parse the whole pending table (possibly using the 1st kB
> >>>> as coarse mapping).
> >>>>
> >>>> If you don't know the LPI INTIDs in advance it is only possible to
> >>>> restore the pending bit of pending LPIs. At that time you would
> >>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
> >>>> ITS ITT you would do the same for those which were not pending. Looks
> >>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
> >>>>
> >>>> Otherwise we would need to add another dependency between RDIST pending
> >>>> table restore and ITS table restore but this looks even more weird, no?
> >>>>
> >>>>
> >>> So I just sat down with Andre and Marc and we tried to work through this
> >>> and came up with the best scheme.  I apologize in advance for the
> >>> one-way nature of this e-mail, and I am of course open to discussing the
> >>> following proposal again if you do not agree.
> >>>
> >>> What I think this document should say, is that the following ordering
> >>> must be followed when restoring the GIC and the ITS:
> >>>
> >>>   First, restore all guest memory
> >>>
> >>>   Second, restore ALL redistributors
> >>>
> >>>   Third, restore the ITS, in the following order:
> >>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >>>     2. Restore GITS_CBASER
> >>>     3. Restore all other GITS_ registers, except GITS_CTLR!
> >>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> >>>     5. Restore GITS_CTLR
> >>>
> >>> The rationale is that we really want the redistributor and the ITS
> >>> restore to be independent and follow the architecture.  This means that
> >>> our ABI for the redistributor should still work without restoring an ITS
> >>> (if we ever decide to support LPIs for KVM without the ITS).
> >>
> >> OK. Note I already mentioned that GICv3 must be restored before the ITS.
> >> To me this comprised the RDIST.
> > 
> > Possibly, but I think it's good to write out the whole thing so we
> > clearly understand the flow.  That could better be achieved by
> > correcting my proposed text above to say something like "Second, restore
> > ALL redistributors to ensure the pending and configuration tables can be
> > read."
> > 
> >>
> >> I understand the above description of the ordering comes in addition to
> >> the existing text, right? 
> > 
> > Yes
> > 
> >> in other words I keep the GITS_READR,
> >> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
> >> section.
> >>
> > 
> > Yes.  But you don't need to do any reading of the pending table on any
> > of the restore operations.
> well you told me to do it on vgic_add_lpi(). This is obviously called on
> ITS table restore. /me confused. 

Sorry, I meant you do not need to scan the entire table independently
from restoring other state that requires building the data structures.

> Obviously this is implicit and should
> not be documented. Is that what you meant? btw this is not documented
> atm I think.

What I care about is that the ABI is clear and represents what the
architecture does.  So in terms of documentation in the ABI, we don't
need to mention anything about when this is done, but we also do not
need to specify any interaction between the pending tables and the ITS,
beyond that the redestributors and memory must be restored before the
ITS.

Hope this clarifies.

Thanks,
-Christoffer

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> Add a generic lookup_table() helper whose role consists in
> scanning a contiguous table located in guest RAM and applying
> a callback on each entry. Entries can be handled as linked lists
> since the callback may return an offset to the next entry and
> also tell that an entry is the last one.
> 
> Helper functions also are added to compute the device/event ID
> offset to the next DTE/ITE.
> 
> compute_next_devid_offset, compute_next_eventid_offset and
> lookup_table will become static in subsequent patches
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - use kvm_read_guest
> 
> v3 -> v4:
> - remove static to avoid compilation warning
> - correct size computation in looup_table()
> - defines now encode the number of bits used for devid and eventid offsets
> - use BIT() - 1 to encode the max offets
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 93 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 56c5123..c22b35d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  
>  #define VITS_TYPER_IDBITS 16
>  #define VITS_TYPER_DEVBITS 16
> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>  
>  /*
>   * Finds and returns a collection in the ITS collection table.
> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> +{
> +	struct list_head *e = &dev->dev_list;
> +	struct its_device *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_device, dev_list);
> +	next_offset = next->device_id - dev->device_id;
> +
> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> +}
> +
> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> +{
> +	struct list_head *e = &ite->ite_list;
> +	struct its_ite *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_ite, ite_list);
> +	next_offset = next->event_id - ite->event_id;
> +
> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> +}
> +
> +/**
> + * entry_fn_t - Callback called on a table entry restore path
> + * @its: its handle
> + * @id: id of the entry
> + * @entry: pointer to the entry
> + * @opaque: pointer to an opaque data
> + * @next_offset: minimal ID offset to the next entry. 0 if this
> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> + * entry's next_offset field was truly decoded
> + *
> + * Return: < 0 on error, 0 otherwise
> + */
> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> +			  void *opaque, u32 *next_offset);
> +
> +/**
> + * lookup_table - scan a contiguous table in guest RAM and applies a function
> + * to each entry
> + *
> + * @its: its handle
> + * @base: base gpa of the table
> + * @size: size of the table in bytes
> + * @esz: entry size in bytes
> + * @start_id: first entry's ID

I'm a little confused.  Why is this not 0?

> + * @fn: function to apply on each entry
> + *
> + * Return: < 0 on error, 1 if last element identified, 0 otherwise

How can you scan the entire table and not find the last element?

> + */
> +int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> +		 int start_id, entry_fn_t fn, void *opaque)

Is this really a lookup?  I think this should be called
'scan_its_table'.

> +{
> +	void *entry = kzalloc(esz, GFP_KERNEL);
> +	struct kvm *kvm = its->dev->kvm;
> +	unsigned long len = size;
> +	u32 id = start_id;

why are ids u32 and not just an unsigned long?

> +	gpa_t gpa = base;
> +	int ret;
> +
> +	while (len > 0) {
> +		u32 next_offset;
> +		size_t byte_offset;
> +
> +		ret = kvm_read_guest(kvm, gpa, entry, esz);
> +		if (ret)
> +			goto out;
> +
> +		ret = fn(its, id, entry, opaque, &next_offset);
> +		if (ret < 0 || (!ret && !next_offset))
> +			goto out;
> +
> +		byte_offset = next_offset * esz;
> +		id += next_offset;
> +		gpa += byte_offset;
> +		len -= byte_offset;
> +	}
> +	kfree(entry);
> +	return 0;
> +

here you can just set 'ret = 0;'

> +out:
> +	kfree(entry);
> +	return (ret < 0 ? ret : 1);

and then here you can do 'return (ret <= 0 ? ret : 1)'
but I actually think it's nicer to just write:

	if (ret <= 0)
		return ret;
	return 1;

> +}
> +
>  /**
>   * vgic_its_save_device_tables - Save the device table and all ITT
>   * into guest RAM
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> Add a generic lookup_table() helper whose role consists in
> scanning a contiguous table located in guest RAM and applying
> a callback on each entry. Entries can be handled as linked lists
> since the callback may return an offset to the next entry and
> also tell that an entry is the last one.
> 
> Helper functions also are added to compute the device/event ID
> offset to the next DTE/ITE.
> 
> compute_next_devid_offset, compute_next_eventid_offset and
> lookup_table will become static in subsequent patches
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - use kvm_read_guest
> 
> v3 -> v4:
> - remove static to avoid compilation warning
> - correct size computation in looup_table()
> - defines now encode the number of bits used for devid and eventid offsets
> - use BIT() - 1 to encode the max offets
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 93 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 56c5123..c22b35d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  
>  #define VITS_TYPER_IDBITS 16
>  #define VITS_TYPER_DEVBITS 16
> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>  
>  /*
>   * Finds and returns a collection in the ITS collection table.
> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> +{
> +	struct list_head *e = &dev->dev_list;
> +	struct its_device *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_device, dev_list);
> +	next_offset = next->device_id - dev->device_id;
> +
> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> +}
> +
> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> +{
> +	struct list_head *e = &ite->ite_list;
> +	struct its_ite *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_ite, ite_list);
> +	next_offset = next->event_id - ite->event_id;
> +
> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> +}
> +
> +/**
> + * entry_fn_t - Callback called on a table entry restore path
> + * @its: its handle
> + * @id: id of the entry
> + * @entry: pointer to the entry
> + * @opaque: pointer to an opaque data
> + * @next_offset: minimal ID offset to the next entry. 0 if this
> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> + * entry's next_offset field was truly decoded
> + *
> + * Return: < 0 on error, 0 otherwise
> + */
> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> +			  void *opaque, u32 *next_offset);
> +
> +/**
> + * lookup_table - scan a contiguous table in guest RAM and applies a function
> + * to each entry
> + *
> + * @its: its handle
> + * @base: base gpa of the table
> + * @size: size of the table in bytes
> + * @esz: entry size in bytes
> + * @start_id: first entry's ID

I'm a little confused.  Why is this not 0?

> + * @fn: function to apply on each entry
> + *
> + * Return: < 0 on error, 1 if last element identified, 0 otherwise

How can you scan the entire table and not find the last element?

> + */
> +int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> +		 int start_id, entry_fn_t fn, void *opaque)

Is this really a lookup?  I think this should be called
'scan_its_table'.

> +{
> +	void *entry = kzalloc(esz, GFP_KERNEL);
> +	struct kvm *kvm = its->dev->kvm;
> +	unsigned long len = size;
> +	u32 id = start_id;

why are ids u32 and not just an unsigned long?

> +	gpa_t gpa = base;
> +	int ret;
> +
> +	while (len > 0) {
> +		u32 next_offset;
> +		size_t byte_offset;
> +
> +		ret = kvm_read_guest(kvm, gpa, entry, esz);
> +		if (ret)
> +			goto out;
> +
> +		ret = fn(its, id, entry, opaque, &next_offset);
> +		if (ret < 0 || (!ret && !next_offset))
> +			goto out;
> +
> +		byte_offset = next_offset * esz;
> +		id += next_offset;
> +		gpa += byte_offset;
> +		len -= byte_offset;
> +	}
> +	kfree(entry);
> +	return 0;
> +

here you can just set 'ret = 0;'

> +out:
> +	kfree(entry);
> +	return (ret < 0 ? ret : 1);

and then here you can do 'return (ret <= 0 ? ret : 1)'
but I actually think it's nicer to just write:

	if (ret <= 0)
		return ret;
	return 1;

> +}
> +
>  /**
>   * vgic_its_save_device_tables - Save the device table and all ITT
>   * into guest RAM
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

Re: [PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Christoffer Dall]

On Thu, Apr 27, 2017 at 07:44:25PM +0200, Auger Eric wrote:
> Hi,
> 
> On 27/04/2017 18:43, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:24PM +0200, Eric Auger wrote:
> >> Up to now the MAPD ITT_addr had been ignored. We will need it
> >> for save/restore. Let's record it in the its_device struct.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - its_cmd_get_ittaddr macro now returns the actual ITT GPA
> >>
> >> v3 -> v4:
> >> - in vgic_its_cmd_handle_mapd, itt_addr directly is shifted
> >> - correct ittaddr bitmask to support 48bit GPA
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 4 ++++
> >>  1 file changed, 4 insertions(+)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 0f3c8f3..757598d 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -106,6 +106,7 @@ struct its_device {
> >>  	/* the head for the list of ITTEs */
> >>  	struct list_head itt_head;
> >>  	u32 nb_eventid_bits;
> >> +	gpa_t itt_addr;
> >>  	u32 device_id;
> >>  };
> >>  
> >> @@ -569,6 +570,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
> >>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
> >>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
> >>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
> >> +#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 47) << 8)
> > 
> > Why 47?  The ITT_addr field is bits [51:8] unless I'm looking at
> > something wrong?
> 
> Initially I limited to 48 bits since I found in the code:
> 
> /*
>  * We only implement 48 bits of PA at the moment, although the ITS
>  * supports more. Let's be restrictive here.
>  */
> #define BASER_ADDRESS(x)        ((x) & GENMASK_ULL(47, 16))
> #define CBASER_ADDRESS(x)       ((x) & GENMASK_ULL(47, 12))
> 
> But practically as I encode the ITT addr field on 52 bits now in the DTE
> there is no reason anymore.

Well, regardless, shouldn't you be bassing 44 to its_cmd_mask_field(),
because you're shifting left 8 bits afterwards?

Thanks,
-Christoffer

[PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Christoffer Dall]

On Thu, Apr 27, 2017 at 07:44:25PM +0200, Auger Eric wrote:
> Hi,
> 
> On 27/04/2017 18:43, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:24PM +0200, Eric Auger wrote:
> >> Up to now the MAPD ITT_addr had been ignored. We will need it
> >> for save/restore. Let's record it in the its_device struct.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - its_cmd_get_ittaddr macro now returns the actual ITT GPA
> >>
> >> v3 -> v4:
> >> - in vgic_its_cmd_handle_mapd, itt_addr directly is shifted
> >> - correct ittaddr bitmask to support 48bit GPA
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 4 ++++
> >>  1 file changed, 4 insertions(+)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 0f3c8f3..757598d 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -106,6 +106,7 @@ struct its_device {
> >>  	/* the head for the list of ITTEs */
> >>  	struct list_head itt_head;
> >>  	u32 nb_eventid_bits;
> >> +	gpa_t itt_addr;
> >>  	u32 device_id;
> >>  };
> >>  
> >> @@ -569,6 +570,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
> >>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
> >>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
> >>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
> >> +#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 47) << 8)
> > 
> > Why 47?  The ITT_addr field is bits [51:8] unless I'm looking at
> > something wrong?
> 
> Initially I limited to 48 bits since I found in the code:
> 
> /*
>  * We only implement 48 bits of PA at the moment, although the ITS
>  * supports more. Let's be restrictive here.
>  */
> #define BASER_ADDRESS(x)        ((x) & GENMASK_ULL(47, 16))
> #define CBASER_ADDRESS(x)       ((x) & GENMASK_ULL(47, 12))
> 
> But practically as I encode the ITT addr field on 52 bits now in the DTE
> there is no reason anymore.

Well, regardless, shouldn't you be bassing 44 to its_cmd_mask_field(),
because you're shifting left 8 bits afterwards?

Thanks,
-Christoffer

Re: [PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Auger Eric]

Hi,

On 27/04/2017 20:09, Christoffer Dall wrote:
> On Thu, Apr 27, 2017 at 07:44:25PM +0200, Auger Eric wrote:
>> Hi,
>>
>> On 27/04/2017 18:43, Christoffer Dall wrote:
>>> On Fri, Apr 14, 2017 at 12:15:24PM +0200, Eric Auger wrote:
>>>> Up to now the MAPD ITT_addr had been ignored. We will need it
>>>> for save/restore. Let's record it in the its_device struct.
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>> v4 -> v5:
>>>> - its_cmd_get_ittaddr macro now returns the actual ITT GPA
>>>>
>>>> v3 -> v4:
>>>> - in vgic_its_cmd_handle_mapd, itt_addr directly is shifted
>>>> - correct ittaddr bitmask to support 48bit GPA
>>>> ---
>>>>  virt/kvm/arm/vgic/vgic-its.c | 4 ++++
>>>>  1 file changed, 4 insertions(+)
>>>>
>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>> index 0f3c8f3..757598d 100644
>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>> @@ -106,6 +106,7 @@ struct its_device {
>>>>  	/* the head for the list of ITTEs */
>>>>  	struct list_head itt_head;
>>>>  	u32 nb_eventid_bits;
>>>> +	gpa_t itt_addr;
>>>>  	u32 device_id;
>>>>  };
>>>>  
>>>> @@ -569,6 +570,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>>>>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>>>>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>>>>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
>>>> +#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 47) << 8)
>>>
>>> Why 47?  The ITT_addr field is bits [51:8] unless I'm looking at
>>> something wrong?
>>
>> Initially I limited to 48 bits since I found in the code:
>>
>> /*
>>  * We only implement 48 bits of PA at the moment, although the ITS
>>  * supports more. Let's be restrictive here.
>>  */
>> #define BASER_ADDRESS(x)        ((x) & GENMASK_ULL(47, 16))
>> #define CBASER_ADDRESS(x)       ((x) & GENMASK_ULL(47, 12))
>>
>> But practically as I encode the ITT addr field on 52 bits now in the DTE
>> there is no reason anymore.
> 
> Well, regardless, shouldn't you be bassing 44 to its_cmd_mask_field(),
> because you're shifting left 8 bits afterwards?

Hum yes that's correct. Andre already warned me against that mistake :-(

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 12/22] KVM: arm64: vgic-its: Interpret MAPD ITT_addr field

[Auger Eric]

Hi,

On 27/04/2017 20:09, Christoffer Dall wrote:
> On Thu, Apr 27, 2017 at 07:44:25PM +0200, Auger Eric wrote:
>> Hi,
>>
>> On 27/04/2017 18:43, Christoffer Dall wrote:
>>> On Fri, Apr 14, 2017 at 12:15:24PM +0200, Eric Auger wrote:
>>>> Up to now the MAPD ITT_addr had been ignored. We will need it
>>>> for save/restore. Let's record it in the its_device struct.
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>> v4 -> v5:
>>>> - its_cmd_get_ittaddr macro now returns the actual ITT GPA
>>>>
>>>> v3 -> v4:
>>>> - in vgic_its_cmd_handle_mapd, itt_addr directly is shifted
>>>> - correct ittaddr bitmask to support 48bit GPA
>>>> ---
>>>>  virt/kvm/arm/vgic/vgic-its.c | 4 ++++
>>>>  1 file changed, 4 insertions(+)
>>>>
>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>> index 0f3c8f3..757598d 100644
>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>> @@ -106,6 +106,7 @@ struct its_device {
>>>>  	/* the head for the list of ITTEs */
>>>>  	struct list_head itt_head;
>>>>  	u32 nb_eventid_bits;
>>>> +	gpa_t itt_addr;
>>>>  	u32 device_id;
>>>>  };
>>>>  
>>>> @@ -569,6 +570,7 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>>>>  #define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>>>>  #define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>>>>  #define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
>>>> +#define its_cmd_get_ittaddr(cmd)	(its_cmd_mask_field(cmd, 2,  8, 47) << 8)
>>>
>>> Why 47?  The ITT_addr field is bits [51:8] unless I'm looking at
>>> something wrong?
>>
>> Initially I limited to 48 bits since I found in the code:
>>
>> /*
>>  * We only implement 48 bits of PA at the moment, although the ITS
>>  * supports more. Let's be restrictive here.
>>  */
>> #define BASER_ADDRESS(x)        ((x) & GENMASK_ULL(47, 16))
>> #define CBASER_ADDRESS(x)       ((x) & GENMASK_ULL(47, 12))
>>
>> But practically as I encode the ITT addr field on 52 bits now in the DTE
>> there is no reason anymore.
> 
> Well, regardless, shouldn't you be bassing 44 to its_cmd_mask_field(),
> because you're shifting left 8 bits afterwards?

Hum yes that's correct. Andre already warned me against that mistake :-(

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Auger Eric]

Hi,

On 27/04/2017 20:06, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
>> Add a generic lookup_table() helper whose role consists in
>> scanning a contiguous table located in guest RAM and applying
>> a callback on each entry. Entries can be handled as linked lists
>> since the callback may return an offset to the next entry and
>> also tell that an entry is the last one.
>>
>> Helper functions also are added to compute the device/event ID
>> offset to the next DTE/ITE.
>>
>> compute_next_devid_offset, compute_next_eventid_offset and
>> lookup_table will become static in subsequent patches
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - use kvm_read_guest
>>
>> v3 -> v4:
>> - remove static to avoid compilation warning
>> - correct size computation in looup_table()
>> - defines now encode the number of bits used for devid and eventid offsets
>> - use BIT() - 1 to encode the max offets
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 93 insertions(+)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 56c5123..c22b35d 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  
>>  #define VITS_TYPER_IDBITS 16
>>  #define VITS_TYPER_DEVBITS 16
>> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
>> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>>  
>>  /*
>>   * Finds and returns a collection in the ITS collection table.
>> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>  	return ret;
>>  }
>>  
>> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>> +{
>> +	struct list_head *e = &dev->dev_list;
>> +	struct its_device *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_device, dev_list);
>> +	next_offset = next->device_id - dev->device_id;
>> +
>> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>> +}
>> +
>> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>> +{
>> +	struct list_head *e = &ite->ite_list;
>> +	struct its_ite *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_ite, ite_list);
>> +	next_offset = next->event_id - ite->event_id;
>> +
>> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
>> +}
>> +
>> +/**
>> + * entry_fn_t - Callback called on a table entry restore path
>> + * @its: its handle
>> + * @id: id of the entry
>> + * @entry: pointer to the entry
>> + * @opaque: pointer to an opaque data
>> + * @next_offset: minimal ID offset to the next entry. 0 if this
>> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
>> + * entry's next_offset field was truly decoded
>> + *
>> + * Return: < 0 on error, 0 otherwise
>> + */
>> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>> +			  void *opaque, u32 *next_offset);
>> +
>> +/**
>> + * lookup_table - scan a contiguous table in guest RAM and applies a function
>> + * to each entry
>> + *
>> + * @its: its handle
>> + * @base: base gpa of the table
>> + * @size: size of the table in bytes
>> + * @esz: entry size in bytes
>> + * @start_id: first entry's ID
> 
> I'm a little confused.  Why is this not 0?
Because this is meant to be called on a second-level table as well
(device table). In that case the start ID != 0.
> 
>> + * @fn: function to apply on each entry
>> + *
>> + * Return: < 0 on error, 1 if last element identified, 0 otherwise
> 
> How can you scan the entire table and not find the last element?
This happens on 2d level tables. The last element might be in another 2d
level table.
> 
>> + */
>> +int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>> +		 int start_id, entry_fn_t fn, void *opaque)
> 
> Is this really a lookup?  I think this should be called
> 'scan_its_table'.
ok no strong opinion here.
> 
>> +{
>> +	void *entry = kzalloc(esz, GFP_KERNEL);
>> +	struct kvm *kvm = its->dev->kvm;
>> +	unsigned long len = size;
>> +	u32 id = start_id;
> 
> why are ids u32 and not just an unsigned long?
will change that.
> 
>> +	gpa_t gpa = base;
>> +	int ret;
>> +
>> +	while (len > 0) {
>> +		u32 next_offset;
>> +		size_t byte_offset;
>> +
>> +		ret = kvm_read_guest(kvm, gpa, entry, esz);
>> +		if (ret)
>> +			goto out;
>> +
>> +		ret = fn(its, id, entry, opaque, &next_offset);
>> +		if (ret < 0 || (!ret && !next_offset))
>> +			goto out;
>> +
>> +		byte_offset = next_offset * esz;
>> +		id += next_offset;
>> +		gpa += byte_offset;
>> +		len -= byte_offset;
>> +	}
>> +	kfree(entry);
>> +	return 0;
>> +
> 
> here you can just set 'ret = 0;'
> 
>> +out:
>> +	kfree(entry);
>> +	return (ret < 0 ? ret : 1);
> 
> and then here you can do 'return (ret <= 0 ? ret : 1)'
> but I actually think it's nicer to just write:
> 
> 	if (ret <= 0)
> 		return ret;
> 	return 1;
OK

thanks

Eric
> 
>> +}
>> +
>>  /**
>>   * vgic_its_save_device_tables - Save the device table and all ITT
>>   * into guest RAM
>> -- 
>> 2.5.5
>>
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Auger Eric]

Hi,

On 27/04/2017 20:06, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
>> Add a generic lookup_table() helper whose role consists in
>> scanning a contiguous table located in guest RAM and applying
>> a callback on each entry. Entries can be handled as linked lists
>> since the callback may return an offset to the next entry and
>> also tell that an entry is the last one.
>>
>> Helper functions also are added to compute the device/event ID
>> offset to the next DTE/ITE.
>>
>> compute_next_devid_offset, compute_next_eventid_offset and
>> lookup_table will become static in subsequent patches
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - use kvm_read_guest
>>
>> v3 -> v4:
>> - remove static to avoid compilation warning
>> - correct size computation in looup_table()
>> - defines now encode the number of bits used for devid and eventid offsets
>> - use BIT() - 1 to encode the max offets
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 93 insertions(+)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 56c5123..c22b35d 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  
>>  #define VITS_TYPER_IDBITS 16
>>  #define VITS_TYPER_DEVBITS 16
>> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
>> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>>  
>>  /*
>>   * Finds and returns a collection in the ITS collection table.
>> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>  	return ret;
>>  }
>>  
>> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>> +{
>> +	struct list_head *e = &dev->dev_list;
>> +	struct its_device *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_device, dev_list);
>> +	next_offset = next->device_id - dev->device_id;
>> +
>> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>> +}
>> +
>> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>> +{
>> +	struct list_head *e = &ite->ite_list;
>> +	struct its_ite *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_ite, ite_list);
>> +	next_offset = next->event_id - ite->event_id;
>> +
>> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
>> +}
>> +
>> +/**
>> + * entry_fn_t - Callback called on a table entry restore path
>> + * @its: its handle
>> + * @id: id of the entry
>> + * @entry: pointer to the entry
>> + * @opaque: pointer to an opaque data
>> + * @next_offset: minimal ID offset to the next entry. 0 if this
>> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
>> + * entry's next_offset field was truly decoded
>> + *
>> + * Return: < 0 on error, 0 otherwise
>> + */
>> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>> +			  void *opaque, u32 *next_offset);
>> +
>> +/**
>> + * lookup_table - scan a contiguous table in guest RAM and applies a function
>> + * to each entry
>> + *
>> + * @its: its handle
>> + * @base: base gpa of the table
>> + * @size: size of the table in bytes
>> + * @esz: entry size in bytes
>> + * @start_id: first entry's ID
> 
> I'm a little confused.  Why is this not 0?
Because this is meant to be called on a second-level table as well
(device table). In that case the start ID != 0.
> 
>> + * @fn: function to apply on each entry
>> + *
>> + * Return: < 0 on error, 1 if last element identified, 0 otherwise
> 
> How can you scan the entire table and not find the last element?
This happens on 2d level tables. The last element might be in another 2d
level table.
> 
>> + */
>> +int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>> +		 int start_id, entry_fn_t fn, void *opaque)
> 
> Is this really a lookup?  I think this should be called
> 'scan_its_table'.
ok no strong opinion here.
> 
>> +{
>> +	void *entry = kzalloc(esz, GFP_KERNEL);
>> +	struct kvm *kvm = its->dev->kvm;
>> +	unsigned long len = size;
>> +	u32 id = start_id;
> 
> why are ids u32 and not just an unsigned long?
will change that.
> 
>> +	gpa_t gpa = base;
>> +	int ret;
>> +
>> +	while (len > 0) {
>> +		u32 next_offset;
>> +		size_t byte_offset;
>> +
>> +		ret = kvm_read_guest(kvm, gpa, entry, esz);
>> +		if (ret)
>> +			goto out;
>> +
>> +		ret = fn(its, id, entry, opaque, &next_offset);
>> +		if (ret < 0 || (!ret && !next_offset))
>> +			goto out;
>> +
>> +		byte_offset = next_offset * esz;
>> +		id += next_offset;
>> +		gpa += byte_offset;
>> +		len -= byte_offset;
>> +	}
>> +	kfree(entry);
>> +	return 0;
>> +
> 
> here you can just set 'ret = 0;'
> 
>> +out:
>> +	kfree(entry);
>> +	return (ret < 0 ? ret : 1);
> 
> and then here you can do 'return (ret <= 0 ? ret : 1)'
> but I actually think it's nicer to just write:
> 
> 	if (ret <= 0)
> 		return ret;
> 	return 1;
OK

thanks

Eric
> 
>> +}
>> +
>>  /**
>>   * vgic_its_save_device_tables - Save the device table and all ITT
>>   * into guest RAM
>> -- 
>> 2.5.5
>>
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

On 27/04/2017 19:54, Christoffer Dall wrote:
> On Thu, Apr 27, 2017 at 07:27:22PM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 27/04/2017 18:38, Christoffer Dall wrote:
>>> On Thu, Apr 27, 2017 at 05:29:35PM +0200, Auger Eric wrote:
>>>>
>>>>
>>>> On 27/04/2017 16:45, Christoffer Dall wrote:
>>>>> Hi Eric,
>>>>>
>>>>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>>>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>>>>
>>>>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> v4 -> v5:
>>>>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>>>>
>>>>>>>>>>>> v3 -> v4:
>>>>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>>>>   - typos
>>>>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>>>>   - document ABI revision
>>>>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>>>>
>>>>>>>>>>>> v1 -> v2:
>>>>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>>>>> - mentions LE layout
>>>>>>>>>>>> ---
>>>>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>>>>> +
>>>>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>>>>> +
>>>>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>>>>> +
>>>>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>>>>> +
>>>>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>>>>> +
>>>>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>>>>> +
>>>>>>>>>>>
>>>>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>>>>
>>>>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>>>>
>>>>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>>>>> restored previously.
>>>>>>>>>>
>>>>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>>>>> been done at the very end.
>>>>>>>>>
>>>>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>>>>
>>>>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>>>>
>>>>>>> Shouldn't restoring the pending tables happen when restoring some
>>>>>>> redeistributor state and not anything related to the ITS?
>>>>>>
>>>>>> Marc wrote:
>>>>>> "
>>>>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>>>>> tables, you can always read the pending bit when creating the LPI
>>>>>> structure (it has been written to RAM at save time). Note that we
>>>>>> already do something like this in vgic_enable_lpis().
>>>>>> "
>>>>>>
>>>>>> This is currently what is implemented I think. the pending tables are
>>>>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>>>>> also on on ITS table restore
>>>>>>
>>>>>> The problematic is: Either you know in advance which LPI INTIDare used
>>>>>> or you need to parse the whole pending table (possibly using the 1st kB
>>>>>> as coarse mapping).
>>>>>>
>>>>>> If you don't know the LPI INTIDs in advance it is only possible to
>>>>>> restore the pending bit of pending LPIs. At that time you would
>>>>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>>>>> ITS ITT you would do the same for those which were not pending. Looks
>>>>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>>>>
>>>>>> Otherwise we would need to add another dependency between RDIST pending
>>>>>> table restore and ITS table restore but this looks even more weird, no?
>>>>>>
>>>>>>
>>>>> So I just sat down with Andre and Marc and we tried to work through this
>>>>> and came up with the best scheme.  I apologize in advance for the
>>>>> one-way nature of this e-mail, and I am of course open to discussing the
>>>>> following proposal again if you do not agree.
>>>>>
>>>>> What I think this document should say, is that the following ordering
>>>>> must be followed when restoring the GIC and the ITS:
>>>>>
>>>>>   First, restore all guest memory
>>>>>
>>>>>   Second, restore ALL redistributors
>>>>>
>>>>>   Third, restore the ITS, in the following order:
>>>>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>>>>     2. Restore GITS_CBASER
>>>>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>>>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>>>>     5. Restore GITS_CTLR
>>>>>
>>>>> The rationale is that we really want the redistributor and the ITS
>>>>> restore to be independent and follow the architecture.  This means that
>>>>> our ABI for the redistributor should still work without restoring an ITS
>>>>> (if we ever decide to support LPIs for KVM without the ITS).
>>>>
>>>> OK. Note I already mentioned that GICv3 must be restored before the ITS.
>>>> To me this comprised the RDIST.
>>>
>>> Possibly, but I think it's good to write out the whole thing so we
>>> clearly understand the flow.  That could better be achieved by
>>> correcting my proposed text above to say something like "Second, restore
>>> ALL redistributors to ensure the pending and configuration tables can be
>>> read."
>>>
>>>>
>>>> I understand the above description of the ordering comes in addition to
>>>> the existing text, right? 
>>>
>>> Yes
>>>
>>>> in other words I keep the GITS_READR,
>>>> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
>>>> section.
>>>>
>>>
>>> Yes.  But you don't need to do any reading of the pending table on any
>>> of the restore operations.
>> well you told me to do it on vgic_add_lpi(). This is obviously called on
>> ITS table restore. /me confused. 
> 
> Sorry, I meant you do not need to scan the entire table independently
> from restoring other state that requires building the data structures.
> 
>> Obviously this is implicit and should
>> not be documented. Is that what you meant? btw this is not documented
>> atm I think.
> 
> What I care about is that the ABI is clear and represents what the
> architecture does.  So in terms of documentation in the ABI, we don't
> need to mention anything about when this is done, but we also do not
> need to specify any interaction between the pending tables and the ITS,
> beyond that the redestributors and memory must be restored before the
> ITS.
> 
> Hope this clarifies.

yes it does

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

On 27/04/2017 19:54, Christoffer Dall wrote:
> On Thu, Apr 27, 2017 at 07:27:22PM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 27/04/2017 18:38, Christoffer Dall wrote:
>>> On Thu, Apr 27, 2017 at 05:29:35PM +0200, Auger Eric wrote:
>>>>
>>>>
>>>> On 27/04/2017 16:45, Christoffer Dall wrote:
>>>>> Hi Eric,
>>>>>
>>>>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>>>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>>>>
>>>>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> v4 -> v5:
>>>>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>>>>
>>>>>>>>>>>> v3 -> v4:
>>>>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>>>>   - typos
>>>>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>>>>   - document ABI revision
>>>>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>>>>
>>>>>>>>>>>> v1 -> v2:
>>>>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>>>>> - mentions LE layout
>>>>>>>>>>>> ---
>>>>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>>>>> +
>>>>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>>>>> +
>>>>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>>>>> +
>>>>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>>>>> +
>>>>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>>>>> +
>>>>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>>>>> +
>>>>>>>>>>>
>>>>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>>>>
>>>>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>>>>
>>>>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>>>>> restored previously.
>>>>>>>>>>
>>>>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>>>>> been done at the very end.
>>>>>>>>>
>>>>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>>>>
>>>>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>>>>
>>>>>>> Shouldn't restoring the pending tables happen when restoring some
>>>>>>> redeistributor state and not anything related to the ITS?
>>>>>>
>>>>>> Marc wrote:
>>>>>> "
>>>>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>>>>> tables, you can always read the pending bit when creating the LPI
>>>>>> structure (it has been written to RAM at save time). Note that we
>>>>>> already do something like this in vgic_enable_lpis().
>>>>>> "
>>>>>>
>>>>>> This is currently what is implemented I think. the pending tables are
>>>>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>>>>> also on on ITS table restore
>>>>>>
>>>>>> The problematic is: Either you know in advance which LPI INTIDare used
>>>>>> or you need to parse the whole pending table (possibly using the 1st kB
>>>>>> as coarse mapping).
>>>>>>
>>>>>> If you don't know the LPI INTIDs in advance it is only possible to
>>>>>> restore the pending bit of pending LPIs. At that time you would
>>>>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>>>>> ITS ITT you would do the same for those which were not pending. Looks
>>>>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>>>>
>>>>>> Otherwise we would need to add another dependency between RDIST pending
>>>>>> table restore and ITS table restore but this looks even more weird, no?
>>>>>>
>>>>>>
>>>>> So I just sat down with Andre and Marc and we tried to work through this
>>>>> and came up with the best scheme.  I apologize in advance for the
>>>>> one-way nature of this e-mail, and I am of course open to discussing the
>>>>> following proposal again if you do not agree.
>>>>>
>>>>> What I think this document should say, is that the following ordering
>>>>> must be followed when restoring the GIC and the ITS:
>>>>>
>>>>>   First, restore all guest memory
>>>>>
>>>>>   Second, restore ALL redistributors
>>>>>
>>>>>   Third, restore the ITS, in the following order:
>>>>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>>>>     2. Restore GITS_CBASER
>>>>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>>>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>>>>     5. Restore GITS_CTLR
>>>>>
>>>>> The rationale is that we really want the redistributor and the ITS
>>>>> restore to be independent and follow the architecture.  This means that
>>>>> our ABI for the redistributor should still work without restoring an ITS
>>>>> (if we ever decide to support LPIs for KVM without the ITS).
>>>>
>>>> OK. Note I already mentioned that GICv3 must be restored before the ITS.
>>>> To me this comprised the RDIST.
>>>
>>> Possibly, but I think it's good to write out the whole thing so we
>>> clearly understand the flow.  That could better be achieved by
>>> correcting my proposed text above to say something like "Second, restore
>>> ALL redistributors to ensure the pending and configuration tables can be
>>> read."
>>>
>>>>
>>>> I understand the above description of the ordering comes in addition to
>>>> the existing text, right? 
>>>
>>> Yes
>>>
>>>> in other words I keep the GITS_READR,
>>>> GITS_IIDR specific text as well as KVM_DEV_ARM_ITS_SAVE/RESTORE_TABLES
>>>> section.
>>>>
>>>
>>> Yes.  But you don't need to do any reading of the pending table on any
>>> of the restore operations.
>> well you told me to do it on vgic_add_lpi(). This is obviously called on
>> ITS table restore. /me confused. 
> 
> Sorry, I meant you do not need to scan the entire table independently
> from restoring other state that requires building the data structures.
> 
>> Obviously this is implicit and should
>> not be documented. Is that what you meant? btw this is not documented
>> atm I think.
> 
> What I care about is that the ABI is clear and represents what the
> architecture does.  So in terms of documentation in the ABI, we don't
> need to mention anything about when this is done, but we also do not
> need to specify any interaction between the pending tables and the ITS,
> beyond that the redestributors and memory must be restored before the
> ITS.
> 
> Hope this clarifies.

yes it does

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Thu, Apr 27, 2017 at 09:24:57PM +0200, Auger Eric wrote:
> Hi,
> 
> On 27/04/2017 20:06, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> >> Add a generic lookup_table() helper whose role consists in
> >> scanning a contiguous table located in guest RAM and applying
> >> a callback on each entry. Entries can be handled as linked lists
> >> since the callback may return an offset to the next entry and
> >> also tell that an entry is the last one.
> >>
> >> Helper functions also are added to compute the device/event ID
> >> offset to the next DTE/ITE.
> >>
> >> compute_next_devid_offset, compute_next_eventid_offset and
> >> lookup_table will become static in subsequent patches
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - use kvm_read_guest
> >>
> >> v3 -> v4:
> >> - remove static to avoid compilation warning
> >> - correct size computation in looup_table()
> >> - defines now encode the number of bits used for devid and eventid offsets
> >> - use BIT() - 1 to encode the max offets
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 93 insertions(+)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 56c5123..c22b35d 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
> >>  
> >>  #define VITS_TYPER_IDBITS 16
> >>  #define VITS_TYPER_DEVBITS 16
> >> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> >> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
> >>  
> >>  /*
> >>   * Finds and returns a collection in the ITS collection table.
> >> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
> >>  	return ret;
> >>  }
> >>  
> >> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >> +{
> >> +	struct list_head *e = &dev->dev_list;
> >> +	struct its_device *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_device, dev_list);
> >> +	next_offset = next->device_id - dev->device_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >> +}
> >> +
> >> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >> +{
> >> +	struct list_head *e = &ite->ite_list;
> >> +	struct its_ite *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_ite, ite_list);
> >> +	next_offset = next->event_id - ite->event_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> >> +}
> >> +
> >> +/**
> >> + * entry_fn_t - Callback called on a table entry restore path
> >> + * @its: its handle
> >> + * @id: id of the entry
> >> + * @entry: pointer to the entry
> >> + * @opaque: pointer to an opaque data
> >> + * @next_offset: minimal ID offset to the next entry. 0 if this
> >> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> >> + * entry's next_offset field was truly decoded
> >> + *
> >> + * Return: < 0 on error, 0 otherwise
> >> + */
> >> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> >> +			  void *opaque, u32 *next_offset);
> >> +
> >> +/**
> >> + * lookup_table - scan a contiguous table in guest RAM and applies a function
> >> + * to each entry
> >> + *
> >> + * @its: its handle
> >> + * @base: base gpa of the table
> >> + * @size: size of the table in bytes
> >> + * @esz: entry size in bytes
> >> + * @start_id: first entry's ID
> > 
> > I'm a little confused.  Why is this not 0?
> Because this is meant to be called on a second-level table as well
> (device table). In that case the start ID != 0.

Ah, ok, makes sense.  You could use the comment to explain that:
  @start_id: The ID of the first entry in the table
             (non-zero for second-level tables)

> > 
> >> + * @fn: function to apply on each entry
> >> + *
> >> + * Return: < 0 on error, 1 if last element identified, 0 otherwise
> > 
> > How can you scan the entire table and not find the last element?
> This happens on 2d level tables. The last element might be in another 2d
> level table.

Ah, I think you should explain this so that the intended use is clear.

Thanks,
-Christoffer

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Thu, Apr 27, 2017 at 09:24:57PM +0200, Auger Eric wrote:
> Hi,
> 
> On 27/04/2017 20:06, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> >> Add a generic lookup_table() helper whose role consists in
> >> scanning a contiguous table located in guest RAM and applying
> >> a callback on each entry. Entries can be handled as linked lists
> >> since the callback may return an offset to the next entry and
> >> also tell that an entry is the last one.
> >>
> >> Helper functions also are added to compute the device/event ID
> >> offset to the next DTE/ITE.
> >>
> >> compute_next_devid_offset, compute_next_eventid_offset and
> >> lookup_table will become static in subsequent patches
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - use kvm_read_guest
> >>
> >> v3 -> v4:
> >> - remove static to avoid compilation warning
> >> - correct size computation in looup_table()
> >> - defines now encode the number of bits used for devid and eventid offsets
> >> - use BIT() - 1 to encode the max offets
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 93 insertions(+)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 56c5123..c22b35d 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
> >>  
> >>  #define VITS_TYPER_IDBITS 16
> >>  #define VITS_TYPER_DEVBITS 16
> >> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> >> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
> >>  
> >>  /*
> >>   * Finds and returns a collection in the ITS collection table.
> >> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
> >>  	return ret;
> >>  }
> >>  
> >> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >> +{
> >> +	struct list_head *e = &dev->dev_list;
> >> +	struct its_device *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_device, dev_list);
> >> +	next_offset = next->device_id - dev->device_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >> +}
> >> +
> >> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >> +{
> >> +	struct list_head *e = &ite->ite_list;
> >> +	struct its_ite *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_ite, ite_list);
> >> +	next_offset = next->event_id - ite->event_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> >> +}
> >> +
> >> +/**
> >> + * entry_fn_t - Callback called on a table entry restore path
> >> + * @its: its handle
> >> + * @id: id of the entry
> >> + * @entry: pointer to the entry
> >> + * @opaque: pointer to an opaque data
> >> + * @next_offset: minimal ID offset to the next entry. 0 if this
> >> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> >> + * entry's next_offset field was truly decoded
> >> + *
> >> + * Return: < 0 on error, 0 otherwise
> >> + */
> >> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> >> +			  void *opaque, u32 *next_offset);
> >> +
> >> +/**
> >> + * lookup_table - scan a contiguous table in guest RAM and applies a function
> >> + * to each entry
> >> + *
> >> + * @its: its handle
> >> + * @base: base gpa of the table
> >> + * @size: size of the table in bytes
> >> + * @esz: entry size in bytes
> >> + * @start_id: first entry's ID
> > 
> > I'm a little confused.  Why is this not 0?
> Because this is meant to be called on a second-level table as well
> (device table). In that case the start ID != 0.

Ah, ok, makes sense.  You could use the comment to explain that:
  @start_id: The ID of the first entry in the table
             (non-zero for second-level tables)

> > 
> >> + * @fn: function to apply on each entry
> >> + *
> >> + * Return: < 0 on error, 1 if last element identified, 0 otherwise
> > 
> > How can you scan the entire table and not find the last element?
> This happens on 2d level tables. The last element might be in another 2d
> level table.

Ah, I think you should explain this so that the intended use is clear.

Thanks,
-Christoffer

Re: [PATCH v5 17/22] KVM: arm64: vgic-its: Collection table save/restore

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:29PM +0200, Eric Auger wrote:
> The save path copies the collection entries into guest RAM
> at the GPA specified in the BASER register. This obviously
> requires the BASER to be set. The last written element is a
> dummy collection table entry.
> 
> We do not index by collection ID as the collection entry
> can fit into 8 bytes while containing the collection ID.
> 
> On restore path we re-allocate the collection objects.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - add macros for field encoding/decoding
> - use abi->cte_esz
> - rename flush into save
> - check the target_addr is valid
> 
> v3 -> v4:
> - replaced u64 *ptr by gpa_t gpa
> - check the collection does not exist before allocating it
> 
> v1 -> v2:
> - reword commit message and directly use 8 as entry size
> - no kvm parameter anymore
> - add helper for flush/restore cte
> - table size computed here
> - add le64/cpu conversions
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 109 ++++++++++++++++++++++++++++++++++++++++++-
>  virt/kvm/arm/vgic/vgic.h     |   9 ++++
>  2 files changed, 116 insertions(+), 2 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index c22b35d..484e541 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1785,13 +1785,97 @@ static int vgic_its_restore_device_tables(struct vgic_its *its)
>  	return -ENXIO;
>  }
>  
> +static int vgic_its_save_cte(struct vgic_its *its,
> +			     struct its_collection *collection,
> +			     gpa_t gpa, int esz)
> +{
> +	u64 val;
> +	int ret;
> +
> +	val = (1ULL << KVM_ITS_CTE_VALID_SHIFT |
> +	       ((u64)collection->target_addr << KVM_ITS_CTE_RDBASE_SHIFT) |
> +	       collection->collection_id);
> +	val = cpu_to_le64(val);
> +	ret = kvm_write_guest(its->dev->kvm, gpa, &val, esz);
> +	return ret;
> +}
> +
> +static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa,
> +				int esz, bool *valid)
> +{
> +	struct its_collection *collection;
> +	struct kvm *kvm = its->dev->kvm;
> +	u32 target_addr;
> +	u32 coll_id;
> +	u64 val;
> +	int ret;
> +
> +	*valid = false;

I don't see why you need this.

> +
> +	ret = kvm_read_guest(kvm, gpa, &val, esz);

hmm, we better not have an esz larger than sizeof(u64) here then.

> +	if (ret)
> +		return ret;
> +	val = le64_to_cpu(val);
> +	*valid = val & KVM_ITS_CTE_VALID_MASK;
> +
> +	if (!*valid)
> +		return 0;

I think this would be nicer with just a bool on the stack and then set
*valid = in the end of the function.  Alternatively, you could just have
the return value return > 0 when there are more entries.

> +
> +	target_addr = (u32)(val >> KVM_ITS_CTE_RDBASE_SHIFT);
> +	coll_id = val & KVM_ITS_CTE_ICID_MASK;
> +
> +	if (target_addr >= atomic_read(&kvm->online_vcpus))
> +		return -EINVAL;
> +
> +	collection = find_collection(its, coll_id);
> +	if (collection)
> +		return -EEXIST;
> +	ret = vgic_its_alloc_collection(its, &collection, coll_id);
> +	if (ret)
> +		return ret;
> +	collection->target_addr = target_addr;
> +	return 0;
> +}
> +
>  /**
>   * vgic_its_save_collection_table - Save the collection table into
>   * guest RAM
>   */
>  static int vgic_its_save_collection_table(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	struct its_collection *collection;
> +	u64 val;
> +	gpa_t gpa;
> +	size_t max_size, filled = 0;
> +	int ret, cte_esz = abi->cte_esz;
> +
> +	gpa = BASER_ADDRESS(its->baser_coll_table);
> +	if (!gpa)
> +		return 0;
> +
> +	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
> +
> +	list_for_each_entry(collection, &its->collection_list, coll_list) {
> +		if (filled == max_size)
> +			return -ENOSPC;
> +		ret = vgic_its_save_cte(its, collection, gpa, cte_esz);
> +		if (ret)
> +			return ret;
> +		gpa += cte_esz;
> +		filled += cte_esz;
> +	}
> +
> +	if (filled == max_size)
> +		return 0;
> +
> +	/*
> +	 * table is not fully filled, add a last dummy element
> +	 * with valid bit unset
> +	 */
> +	val = 0;
> +	ret = kvm_write_guest(its->dev->kvm, gpa, &val, cte_esz);
> +	return ret;
>  }
>  
>  /**
> @@ -1801,7 +1885,28 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
>   */
>  static int vgic_its_restore_collection_table(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	size_t max_size, read = 0;
> +	gpa_t gpa;
> +	int ret, cte_esz = abi->cte_esz;
> +
> +	gpa = BASER_ADDRESS(its->baser_coll_table);
> +
> +	if (!gpa)
> +		return 0;

I don't understand this check.

Don't you want to do do
	if (!(its->baser_coll_table) & GITS_BASER_VALID)
		return 0;

Or what am I missing?

Thanks,
-Christoffer

> +
> +	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
> +
> +	while (read < max_size) {
> +		bool valid;
> +
> +		ret = vgic_its_restore_cte(its, gpa, cte_esz, &valid);
> +		if (!valid || ret)
> +			break;
> +		gpa += cte_esz;
> +		read += cte_esz;
> +	}
> +	return ret;
>  }
>  
>  /**
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index b87f1c6..56e57c1 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -73,6 +73,15 @@
>  				      KVM_REG_ARM_VGIC_SYSREG_CRM_MASK | \
>  				      KVM_REG_ARM_VGIC_SYSREG_OP2_MASK)
>  
> +/*
> + * As per Documentation/virtual/kvm/devices/arm-vgic-its.txt,
> + * below macros are defined for ITS table entry encoding.
> + */
> +#define KVM_ITS_CTE_VALID_SHIFT		63
> +#define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
> +#define KVM_ITS_CTE_RDBASE_SHIFT	16
> +#define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
> +
>  static inline bool irq_is_pending(struct vgic_irq *irq)
>  {
>  	if (irq->config == VGIC_CONFIG_EDGE)
> -- 
> 2.5.5
> 

[PATCH v5 17/22] KVM: arm64: vgic-its: Collection table save/restore

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:29PM +0200, Eric Auger wrote:
> The save path copies the collection entries into guest RAM
> at the GPA specified in the BASER register. This obviously
> requires the BASER to be set. The last written element is a
> dummy collection table entry.
> 
> We do not index by collection ID as the collection entry
> can fit into 8 bytes while containing the collection ID.
> 
> On restore path we re-allocate the collection objects.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - add macros for field encoding/decoding
> - use abi->cte_esz
> - rename flush into save
> - check the target_addr is valid
> 
> v3 -> v4:
> - replaced u64 *ptr by gpa_t gpa
> - check the collection does not exist before allocating it
> 
> v1 -> v2:
> - reword commit message and directly use 8 as entry size
> - no kvm parameter anymore
> - add helper for flush/restore cte
> - table size computed here
> - add le64/cpu conversions
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 109 ++++++++++++++++++++++++++++++++++++++++++-
>  virt/kvm/arm/vgic/vgic.h     |   9 ++++
>  2 files changed, 116 insertions(+), 2 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index c22b35d..484e541 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1785,13 +1785,97 @@ static int vgic_its_restore_device_tables(struct vgic_its *its)
>  	return -ENXIO;
>  }
>  
> +static int vgic_its_save_cte(struct vgic_its *its,
> +			     struct its_collection *collection,
> +			     gpa_t gpa, int esz)
> +{
> +	u64 val;
> +	int ret;
> +
> +	val = (1ULL << KVM_ITS_CTE_VALID_SHIFT |
> +	       ((u64)collection->target_addr << KVM_ITS_CTE_RDBASE_SHIFT) |
> +	       collection->collection_id);
> +	val = cpu_to_le64(val);
> +	ret = kvm_write_guest(its->dev->kvm, gpa, &val, esz);
> +	return ret;
> +}
> +
> +static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa,
> +				int esz, bool *valid)
> +{
> +	struct its_collection *collection;
> +	struct kvm *kvm = its->dev->kvm;
> +	u32 target_addr;
> +	u32 coll_id;
> +	u64 val;
> +	int ret;
> +
> +	*valid = false;

I don't see why you need this.

> +
> +	ret = kvm_read_guest(kvm, gpa, &val, esz);

hmm, we better not have an esz larger than sizeof(u64) here then.

> +	if (ret)
> +		return ret;
> +	val = le64_to_cpu(val);
> +	*valid = val & KVM_ITS_CTE_VALID_MASK;
> +
> +	if (!*valid)
> +		return 0;

I think this would be nicer with just a bool on the stack and then set
*valid = in the end of the function.  Alternatively, you could just have
the return value return > 0 when there are more entries.

> +
> +	target_addr = (u32)(val >> KVM_ITS_CTE_RDBASE_SHIFT);
> +	coll_id = val & KVM_ITS_CTE_ICID_MASK;
> +
> +	if (target_addr >= atomic_read(&kvm->online_vcpus))
> +		return -EINVAL;
> +
> +	collection = find_collection(its, coll_id);
> +	if (collection)
> +		return -EEXIST;
> +	ret = vgic_its_alloc_collection(its, &collection, coll_id);
> +	if (ret)
> +		return ret;
> +	collection->target_addr = target_addr;
> +	return 0;
> +}
> +
>  /**
>   * vgic_its_save_collection_table - Save the collection table into
>   * guest RAM
>   */
>  static int vgic_its_save_collection_table(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	struct its_collection *collection;
> +	u64 val;
> +	gpa_t gpa;
> +	size_t max_size, filled = 0;
> +	int ret, cte_esz = abi->cte_esz;
> +
> +	gpa = BASER_ADDRESS(its->baser_coll_table);
> +	if (!gpa)
> +		return 0;
> +
> +	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
> +
> +	list_for_each_entry(collection, &its->collection_list, coll_list) {
> +		if (filled == max_size)
> +			return -ENOSPC;
> +		ret = vgic_its_save_cte(its, collection, gpa, cte_esz);
> +		if (ret)
> +			return ret;
> +		gpa += cte_esz;
> +		filled += cte_esz;
> +	}
> +
> +	if (filled == max_size)
> +		return 0;
> +
> +	/*
> +	 * table is not fully filled, add a last dummy element
> +	 * with valid bit unset
> +	 */
> +	val = 0;
> +	ret = kvm_write_guest(its->dev->kvm, gpa, &val, cte_esz);
> +	return ret;
>  }
>  
>  /**
> @@ -1801,7 +1885,28 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
>   */
>  static int vgic_its_restore_collection_table(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	size_t max_size, read = 0;
> +	gpa_t gpa;
> +	int ret, cte_esz = abi->cte_esz;
> +
> +	gpa = BASER_ADDRESS(its->baser_coll_table);
> +
> +	if (!gpa)
> +		return 0;

I don't understand this check.

Don't you want to do do
	if (!(its->baser_coll_table) & GITS_BASER_VALID)
		return 0;

Or what am I missing?

Thanks,
-Christoffer

> +
> +	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
> +
> +	while (read < max_size) {
> +		bool valid;
> +
> +		ret = vgic_its_restore_cte(its, gpa, cte_esz, &valid);
> +		if (!valid || ret)
> +			break;
> +		gpa += cte_esz;
> +		read += cte_esz;
> +	}
> +	return ret;
>  }
>  
>  /**
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index b87f1c6..56e57c1 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -73,6 +73,15 @@
>  				      KVM_REG_ARM_VGIC_SYSREG_CRM_MASK | \
>  				      KVM_REG_ARM_VGIC_SYSREG_OP2_MASK)
>  
> +/*
> + * As per Documentation/virtual/kvm/devices/arm-vgic-its.txt,
> + * below macros are defined for ITS table entry encoding.
> + */
> +#define KVM_ITS_CTE_VALID_SHIFT		63
> +#define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
> +#define KVM_ITS_CTE_RDBASE_SHIFT	16
> +#define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
> +
>  static inline bool irq_is_pending(struct vgic_irq *irq)
>  {
>  	if (irq->config == VGIC_CONFIG_EDGE)
> -- 
> 2.5.5
> 

Re: [PATCH v5 17/22] KVM: arm64: vgic-its: Collection table save/restore

[Auger Eric]

Hi Christoffer,

On 28/04/2017 12:44, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:29PM +0200, Eric Auger wrote:
>> The save path copies the collection entries into guest RAM
>> at the GPA specified in the BASER register. This obviously
>> requires the BASER to be set. The last written element is a
>> dummy collection table entry.
>>
>> We do not index by collection ID as the collection entry
>> can fit into 8 bytes while containing the collection ID.
>>
>> On restore path we re-allocate the collection objects.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - add macros for field encoding/decoding
>> - use abi->cte_esz
>> - rename flush into save
>> - check the target_addr is valid
>>
>> v3 -> v4:
>> - replaced u64 *ptr by gpa_t gpa
>> - check the collection does not exist before allocating it
>>
>> v1 -> v2:
>> - reword commit message and directly use 8 as entry size
>> - no kvm parameter anymore
>> - add helper for flush/restore cte
>> - table size computed here
>> - add le64/cpu conversions
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 109 ++++++++++++++++++++++++++++++++++++++++++-
>>  virt/kvm/arm/vgic/vgic.h     |   9 ++++
>>  2 files changed, 116 insertions(+), 2 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index c22b35d..484e541 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -1785,13 +1785,97 @@ static int vgic_its_restore_device_tables(struct vgic_its *its)
>>  	return -ENXIO;
>>  }
>>  
>> +static int vgic_its_save_cte(struct vgic_its *its,
>> +			     struct its_collection *collection,
>> +			     gpa_t gpa, int esz)
>> +{
>> +	u64 val;
>> +	int ret;
>> +
>> +	val = (1ULL << KVM_ITS_CTE_VALID_SHIFT |
>> +	       ((u64)collection->target_addr << KVM_ITS_CTE_RDBASE_SHIFT) |
>> +	       collection->collection_id);
>> +	val = cpu_to_le64(val);
>> +	ret = kvm_write_guest(its->dev->kvm, gpa, &val, esz);
>> +	return ret;
>> +}
>> +
>> +static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa,
>> +				int esz, bool *valid)
>> +{
>> +	struct its_collection *collection;
>> +	struct kvm *kvm = its->dev->kvm;
>> +	u32 target_addr;
>> +	u32 coll_id;
>> +	u64 val;
>> +	int ret;
>> +
>> +	*valid = false;
> 
> I don't see why you need this.
I initialized it here in case kvm_read_guest() fails
> 
>> +
>> +	ret = kvm_read_guest(kvm, gpa, &val, esz);
> 
> hmm, we better not have an esz larger than sizeof(u64) here then.
Yes this could be part if the ABI ops but is that worth the effort now?
I can add a comment somewhere to mention that trap.
> 
>> +	if (ret)
>> +		return ret;
>> +	val = le64_to_cpu(val);
>> +	*valid = val & KVM_ITS_CTE_VALID_MASK;
>> +
>> +	if (!*valid)
>> +		return 0;
> 
> I think this would be nicer with just a bool on the stack and then set
> *valid = in the end of the function.
OK
  Alternatively, you could just have
> the return value return > 0 when there are more entries.
> 
>> +
>> +	target_addr = (u32)(val >> KVM_ITS_CTE_RDBASE_SHIFT);
>> +	coll_id = val & KVM_ITS_CTE_ICID_MASK;
>> +
>> +	if (target_addr >= atomic_read(&kvm->online_vcpus))
>> +		return -EINVAL;
>> +
>> +	collection = find_collection(its, coll_id);
>> +	if (collection)
>> +		return -EEXIST;
>> +	ret = vgic_its_alloc_collection(its, &collection, coll_id);
>> +	if (ret)
>> +		return ret;
>> +	collection->target_addr = target_addr;
>> +	return 0;
>> +}
>> +
>>  /**
>>   * vgic_its_save_collection_table - Save the collection table into
>>   * guest RAM
>>   */
>>  static int vgic_its_save_collection_table(struct vgic_its *its)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	struct its_collection *collection;
>> +	u64 val;
>> +	gpa_t gpa;
>> +	size_t max_size, filled = 0;
>> +	int ret, cte_esz = abi->cte_esz;
>> +
>> +	gpa = BASER_ADDRESS(its->baser_coll_table);
>> +	if (!gpa)
>> +		return 0;
>> +
>> +	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
>> +
>> +	list_for_each_entry(collection, &its->collection_list, coll_list) {
>> +		if (filled == max_size)
>> +			return -ENOSPC;
>> +		ret = vgic_its_save_cte(its, collection, gpa, cte_esz);
>> +		if (ret)
>> +			return ret;
>> +		gpa += cte_esz;
>> +		filled += cte_esz;
>> +	}
>> +
>> +	if (filled == max_size)
>> +		return 0;
>> +
>> +	/*
>> +	 * table is not fully filled, add a last dummy element
>> +	 * with valid bit unset
>> +	 */
>> +	val = 0;
>> +	ret = kvm_write_guest(its->dev->kvm, gpa, &val, cte_esz);
>> +	return ret;
>>  }
>>  
>>  /**
>> @@ -1801,7 +1885,28 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
>>   */
>>  static int vgic_its_restore_collection_table(struct vgic_its *its)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	size_t max_size, read = 0;
>> +	gpa_t gpa;
>> +	int ret, cte_esz = abi->cte_esz;
>> +
>> +	gpa = BASER_ADDRESS(its->baser_coll_table);
>> +
>> +	if (!gpa)
>> +		return 0;
> 
> I don't understand this check.
> 
> Don't you want to do do
> 	if (!(its->baser_coll_table) & GITS_BASER_VALID)
> 		return 0;
> 
> Or what am I missing?
Yes that's correct I will fix that.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
>> +
>> +	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
>> +
>> +	while (read < max_size) {
>> +		bool valid;
>> +
>> +		ret = vgic_its_restore_cte(its, gpa, cte_esz, &valid);
>> +		if (!valid || ret)
>> +			break;
>> +		gpa += cte_esz;
>> +		read += cte_esz;
>> +	}
>> +	return ret;
>>  }
>>  
>>  /**
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index b87f1c6..56e57c1 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -73,6 +73,15 @@
>>  				      KVM_REG_ARM_VGIC_SYSREG_CRM_MASK | \
>>  				      KVM_REG_ARM_VGIC_SYSREG_OP2_MASK)
>>  
>> +/*
>> + * As per Documentation/virtual/kvm/devices/arm-vgic-its.txt,
>> + * below macros are defined for ITS table entry encoding.
>> + */
>> +#define KVM_ITS_CTE_VALID_SHIFT		63
>> +#define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
>> +#define KVM_ITS_CTE_RDBASE_SHIFT	16
>> +#define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
>> +
>>  static inline bool irq_is_pending(struct vgic_irq *irq)
>>  {
>>  	if (irq->config == VGIC_CONFIG_EDGE)
>> -- 
>> 2.5.5
>>
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 17/22] KVM: arm64: vgic-its: Collection table save/restore

[Auger Eric]

Hi Christoffer,

On 28/04/2017 12:44, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:29PM +0200, Eric Auger wrote:
>> The save path copies the collection entries into guest RAM
>> at the GPA specified in the BASER register. This obviously
>> requires the BASER to be set. The last written element is a
>> dummy collection table entry.
>>
>> We do not index by collection ID as the collection entry
>> can fit into 8 bytes while containing the collection ID.
>>
>> On restore path we re-allocate the collection objects.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - add macros for field encoding/decoding
>> - use abi->cte_esz
>> - rename flush into save
>> - check the target_addr is valid
>>
>> v3 -> v4:
>> - replaced u64 *ptr by gpa_t gpa
>> - check the collection does not exist before allocating it
>>
>> v1 -> v2:
>> - reword commit message and directly use 8 as entry size
>> - no kvm parameter anymore
>> - add helper for flush/restore cte
>> - table size computed here
>> - add le64/cpu conversions
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 109 ++++++++++++++++++++++++++++++++++++++++++-
>>  virt/kvm/arm/vgic/vgic.h     |   9 ++++
>>  2 files changed, 116 insertions(+), 2 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index c22b35d..484e541 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -1785,13 +1785,97 @@ static int vgic_its_restore_device_tables(struct vgic_its *its)
>>  	return -ENXIO;
>>  }
>>  
>> +static int vgic_its_save_cte(struct vgic_its *its,
>> +			     struct its_collection *collection,
>> +			     gpa_t gpa, int esz)
>> +{
>> +	u64 val;
>> +	int ret;
>> +
>> +	val = (1ULL << KVM_ITS_CTE_VALID_SHIFT |
>> +	       ((u64)collection->target_addr << KVM_ITS_CTE_RDBASE_SHIFT) |
>> +	       collection->collection_id);
>> +	val = cpu_to_le64(val);
>> +	ret = kvm_write_guest(its->dev->kvm, gpa, &val, esz);
>> +	return ret;
>> +}
>> +
>> +static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa,
>> +				int esz, bool *valid)
>> +{
>> +	struct its_collection *collection;
>> +	struct kvm *kvm = its->dev->kvm;
>> +	u32 target_addr;
>> +	u32 coll_id;
>> +	u64 val;
>> +	int ret;
>> +
>> +	*valid = false;
> 
> I don't see why you need this.
I initialized it here in case kvm_read_guest() fails
> 
>> +
>> +	ret = kvm_read_guest(kvm, gpa, &val, esz);
> 
> hmm, we better not have an esz larger than sizeof(u64) here then.
Yes this could be part if the ABI ops but is that worth the effort now?
I can add a comment somewhere to mention that trap.
> 
>> +	if (ret)
>> +		return ret;
>> +	val = le64_to_cpu(val);
>> +	*valid = val & KVM_ITS_CTE_VALID_MASK;
>> +
>> +	if (!*valid)
>> +		return 0;
> 
> I think this would be nicer with just a bool on the stack and then set
> *valid = in the end of the function.
OK
  Alternatively, you could just have
> the return value return > 0 when there are more entries.
> 
>> +
>> +	target_addr = (u32)(val >> KVM_ITS_CTE_RDBASE_SHIFT);
>> +	coll_id = val & KVM_ITS_CTE_ICID_MASK;
>> +
>> +	if (target_addr >= atomic_read(&kvm->online_vcpus))
>> +		return -EINVAL;
>> +
>> +	collection = find_collection(its, coll_id);
>> +	if (collection)
>> +		return -EEXIST;
>> +	ret = vgic_its_alloc_collection(its, &collection, coll_id);
>> +	if (ret)
>> +		return ret;
>> +	collection->target_addr = target_addr;
>> +	return 0;
>> +}
>> +
>>  /**
>>   * vgic_its_save_collection_table - Save the collection table into
>>   * guest RAM
>>   */
>>  static int vgic_its_save_collection_table(struct vgic_its *its)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	struct its_collection *collection;
>> +	u64 val;
>> +	gpa_t gpa;
>> +	size_t max_size, filled = 0;
>> +	int ret, cte_esz = abi->cte_esz;
>> +
>> +	gpa = BASER_ADDRESS(its->baser_coll_table);
>> +	if (!gpa)
>> +		return 0;
>> +
>> +	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
>> +
>> +	list_for_each_entry(collection, &its->collection_list, coll_list) {
>> +		if (filled == max_size)
>> +			return -ENOSPC;
>> +		ret = vgic_its_save_cte(its, collection, gpa, cte_esz);
>> +		if (ret)
>> +			return ret;
>> +		gpa += cte_esz;
>> +		filled += cte_esz;
>> +	}
>> +
>> +	if (filled == max_size)
>> +		return 0;
>> +
>> +	/*
>> +	 * table is not fully filled, add a last dummy element
>> +	 * with valid bit unset
>> +	 */
>> +	val = 0;
>> +	ret = kvm_write_guest(its->dev->kvm, gpa, &val, cte_esz);
>> +	return ret;
>>  }
>>  
>>  /**
>> @@ -1801,7 +1885,28 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
>>   */
>>  static int vgic_its_restore_collection_table(struct vgic_its *its)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	size_t max_size, read = 0;
>> +	gpa_t gpa;
>> +	int ret, cte_esz = abi->cte_esz;
>> +
>> +	gpa = BASER_ADDRESS(its->baser_coll_table);
>> +
>> +	if (!gpa)
>> +		return 0;
> 
> I don't understand this check.
> 
> Don't you want to do do
> 	if (!(its->baser_coll_table) & GITS_BASER_VALID)
> 		return 0;
> 
> Or what am I missing?
Yes that's correct I will fix that.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
>> +
>> +	max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
>> +
>> +	while (read < max_size) {
>> +		bool valid;
>> +
>> +		ret = vgic_its_restore_cte(its, gpa, cte_esz, &valid);
>> +		if (!valid || ret)
>> +			break;
>> +		gpa += cte_esz;
>> +		read += cte_esz;
>> +	}
>> +	return ret;
>>  }
>>  
>>  /**
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index b87f1c6..56e57c1 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -73,6 +73,15 @@
>>  				      KVM_REG_ARM_VGIC_SYSREG_CRM_MASK | \
>>  				      KVM_REG_ARM_VGIC_SYSREG_OP2_MASK)
>>  
>> +/*
>> + * As per Documentation/virtual/kvm/devices/arm-vgic-its.txt,
>> + * below macros are defined for ITS table entry encoding.
>> + */
>> +#define KVM_ITS_CTE_VALID_SHIFT		63
>> +#define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
>> +#define KVM_ITS_CTE_RDBASE_SHIFT	16
>> +#define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
>> +
>>  static inline bool irq_is_pending(struct vgic_irq *irq)
>>  {
>>  	if (irq->config == VGIC_CONFIG_EDGE)
>> -- 
>> 2.5.5
>>
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 17/22] KVM: arm64: vgic-its: Collection table save/restore

[Christoffer Dall]

On Fri, Apr 28, 2017 at 01:05:15PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 28/04/2017 12:44, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:29PM +0200, Eric Auger wrote:
> >> The save path copies the collection entries into guest RAM
> >> at the GPA specified in the BASER register. This obviously
> >> requires the BASER to be set. The last written element is a
> >> dummy collection table entry.
> >>
> >> We do not index by collection ID as the collection entry
> >> can fit into 8 bytes while containing the collection ID.
> >>
> >> On restore path we re-allocate the collection objects.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - add macros for field encoding/decoding
> >> - use abi->cte_esz
> >> - rename flush into save
> >> - check the target_addr is valid
> >>
> >> v3 -> v4:
> >> - replaced u64 *ptr by gpa_t gpa
> >> - check the collection does not exist before allocating it
> >>
> >> v1 -> v2:
> >> - reword commit message and directly use 8 as entry size
> >> - no kvm parameter anymore
> >> - add helper for flush/restore cte
> >> - table size computed here
> >> - add le64/cpu conversions
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 109 ++++++++++++++++++++++++++++++++++++++++++-
> >>  virt/kvm/arm/vgic/vgic.h     |   9 ++++
> >>  2 files changed, 116 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index c22b35d..484e541 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -1785,13 +1785,97 @@ static int vgic_its_restore_device_tables(struct vgic_its *its)
> >>  	return -ENXIO;
> >>  }
> >>  
> >> +static int vgic_its_save_cte(struct vgic_its *its,
> >> +			     struct its_collection *collection,
> >> +			     gpa_t gpa, int esz)
> >> +{
> >> +	u64 val;
> >> +	int ret;
> >> +
> >> +	val = (1ULL << KVM_ITS_CTE_VALID_SHIFT |
> >> +	       ((u64)collection->target_addr << KVM_ITS_CTE_RDBASE_SHIFT) |
> >> +	       collection->collection_id);
> >> +	val = cpu_to_le64(val);
> >> +	ret = kvm_write_guest(its->dev->kvm, gpa, &val, esz);
> >> +	return ret;
> >> +}
> >> +
> >> +static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa,
> >> +				int esz, bool *valid)
> >> +{
> >> +	struct its_collection *collection;
> >> +	struct kvm *kvm = its->dev->kvm;
> >> +	u32 target_addr;
> >> +	u32 coll_id;
> >> +	u64 val;
> >> +	int ret;
> >> +
> >> +	*valid = false;
> > 
> > I don't see why you need this.
> I initialized it here in case kvm_read_guest() fails

If the caller looks at the valid setting when it receives an error code,
that's really weird.

> > 
> >> +
> >> +	ret = kvm_read_guest(kvm, gpa, &val, esz);
> > 
> > hmm, we better not have an esz larger than sizeof(u64) here then.
> Yes this could be part if the ABI ops but is that worth the effort now?
> I can add a comment somewhere to mention that trap.

I wasn't really sure what we should do here, but I think we should add
BUG_ON(esz > sizeof(val)) when going over this again.

Thanks,
-Christoffer

[PATCH v5 17/22] KVM: arm64: vgic-its: Collection table save/restore

[Christoffer Dall]

On Fri, Apr 28, 2017 at 01:05:15PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 28/04/2017 12:44, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:29PM +0200, Eric Auger wrote:
> >> The save path copies the collection entries into guest RAM
> >> at the GPA specified in the BASER register. This obviously
> >> requires the BASER to be set. The last written element is a
> >> dummy collection table entry.
> >>
> >> We do not index by collection ID as the collection entry
> >> can fit into 8 bytes while containing the collection ID.
> >>
> >> On restore path we re-allocate the collection objects.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - add macros for field encoding/decoding
> >> - use abi->cte_esz
> >> - rename flush into save
> >> - check the target_addr is valid
> >>
> >> v3 -> v4:
> >> - replaced u64 *ptr by gpa_t gpa
> >> - check the collection does not exist before allocating it
> >>
> >> v1 -> v2:
> >> - reword commit message and directly use 8 as entry size
> >> - no kvm parameter anymore
> >> - add helper for flush/restore cte
> >> - table size computed here
> >> - add le64/cpu conversions
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 109 ++++++++++++++++++++++++++++++++++++++++++-
> >>  virt/kvm/arm/vgic/vgic.h     |   9 ++++
> >>  2 files changed, 116 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index c22b35d..484e541 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -1785,13 +1785,97 @@ static int vgic_its_restore_device_tables(struct vgic_its *its)
> >>  	return -ENXIO;
> >>  }
> >>  
> >> +static int vgic_its_save_cte(struct vgic_its *its,
> >> +			     struct its_collection *collection,
> >> +			     gpa_t gpa, int esz)
> >> +{
> >> +	u64 val;
> >> +	int ret;
> >> +
> >> +	val = (1ULL << KVM_ITS_CTE_VALID_SHIFT |
> >> +	       ((u64)collection->target_addr << KVM_ITS_CTE_RDBASE_SHIFT) |
> >> +	       collection->collection_id);
> >> +	val = cpu_to_le64(val);
> >> +	ret = kvm_write_guest(its->dev->kvm, gpa, &val, esz);
> >> +	return ret;
> >> +}
> >> +
> >> +static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa,
> >> +				int esz, bool *valid)
> >> +{
> >> +	struct its_collection *collection;
> >> +	struct kvm *kvm = its->dev->kvm;
> >> +	u32 target_addr;
> >> +	u32 coll_id;
> >> +	u64 val;
> >> +	int ret;
> >> +
> >> +	*valid = false;
> > 
> > I don't see why you need this.
> I initialized it here in case kvm_read_guest() fails

If the caller looks at the valid setting when it receives an error code,
that's really weird.

> > 
> >> +
> >> +	ret = kvm_read_guest(kvm, gpa, &val, esz);
> > 
> > hmm, we better not have an esz larger than sizeof(u64) here then.
> Yes this could be part if the ABI ops but is that worth the effort now?
> I can add a comment somewhere to mention that trap.

I wasn't really sure what we should do here, but I think we should add
BUG_ON(esz > sizeof(val)) when going over this again.

Thanks,
-Christoffer

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> Add a generic lookup_table() helper whose role consists in
> scanning a contiguous table located in guest RAM and applying
> a callback on each entry. Entries can be handled as linked lists
> since the callback may return an offset to the next entry and
> also tell that an entry is the last one.
> 
> Helper functions also are added to compute the device/event ID
> offset to the next DTE/ITE.
> 
> compute_next_devid_offset, compute_next_eventid_offset and
> lookup_table will become static in subsequent patches
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - use kvm_read_guest
> 
> v3 -> v4:
> - remove static to avoid compilation warning
> - correct size computation in looup_table()
> - defines now encode the number of bits used for devid and eventid offsets
> - use BIT() - 1 to encode the max offets
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 93 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 56c5123..c22b35d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  
>  #define VITS_TYPER_IDBITS 16
>  #define VITS_TYPER_DEVBITS 16
> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>  
>  /*
>   * Finds and returns a collection in the ITS collection table.
> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> +{
> +	struct list_head *e = &dev->dev_list;
> +	struct its_device *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_device, dev_list);
> +	next_offset = next->device_id - dev->device_id;
> +
> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> +}
> +
> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> +{
> +	struct list_head *e = &ite->ite_list;
> +	struct its_ite *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_ite, ite_list);
> +	next_offset = next->event_id - ite->event_id;
> +
> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> +}
> +
> +/**
> + * entry_fn_t - Callback called on a table entry restore path
> + * @its: its handle
> + * @id: id of the entry
> + * @entry: pointer to the entry
> + * @opaque: pointer to an opaque data
> + * @next_offset: minimal ID offset to the next entry. 0 if this
> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> + * entry's next_offset field was truly decoded
> + *
> + * Return: < 0 on error, 0 otherwise
> + */
> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> +			  void *opaque, u32 *next_offset);

Just noticed.  All the table entries are 64-bit long at this point,
right?  So why not make entry a u64 * instead?  Could we end up with
some endianness mess with using void pointers the way it is now?

Thanks,
-Christoffer

> +
> +/**
> + * lookup_table - scan a contiguous table in guest RAM and applies a function
> + * to each entry
> + *
> + * @its: its handle
> + * @base: base gpa of the table
> + * @size: size of the table in bytes
> + * @esz: entry size in bytes
> + * @start_id: first entry's ID
> + * @fn: function to apply on each entry
> + *
> + * Return: < 0 on error, 1 if last element identified, 0 otherwise
> + */
> +int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> +		 int start_id, entry_fn_t fn, void *opaque)
> +{
> +	void *entry = kzalloc(esz, GFP_KERNEL);
> +	struct kvm *kvm = its->dev->kvm;
> +	unsigned long len = size;
> +	u32 id = start_id;
> +	gpa_t gpa = base;
> +	int ret;
> +
> +	while (len > 0) {
> +		u32 next_offset;
> +		size_t byte_offset;
> +
> +		ret = kvm_read_guest(kvm, gpa, entry, esz);
> +		if (ret)
> +			goto out;
> +
> +		ret = fn(its, id, entry, opaque, &next_offset);
> +		if (ret < 0 || (!ret && !next_offset))
> +			goto out;
> +
> +		byte_offset = next_offset * esz;
> +		id += next_offset;
> +		gpa += byte_offset;
> +		len -= byte_offset;
> +	}
> +	kfree(entry);
> +	return 0;
> +
> +out:
> +	kfree(entry);
> +	return (ret < 0 ? ret : 1);
> +}
> +
>  /**
>   * vgic_its_save_device_tables - Save the device table and all ITT
>   * into guest RAM
> -- 
> 2.5.5
> 

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> Add a generic lookup_table() helper whose role consists in
> scanning a contiguous table located in guest RAM and applying
> a callback on each entry. Entries can be handled as linked lists
> since the callback may return an offset to the next entry and
> also tell that an entry is the last one.
> 
> Helper functions also are added to compute the device/event ID
> offset to the next DTE/ITE.
> 
> compute_next_devid_offset, compute_next_eventid_offset and
> lookup_table will become static in subsequent patches
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - use kvm_read_guest
> 
> v3 -> v4:
> - remove static to avoid compilation warning
> - correct size computation in looup_table()
> - defines now encode the number of bits used for devid and eventid offsets
> - use BIT() - 1 to encode the max offets
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 93 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 56c5123..c22b35d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  
>  #define VITS_TYPER_IDBITS 16
>  #define VITS_TYPER_DEVBITS 16
> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>  
>  /*
>   * Finds and returns a collection in the ITS collection table.
> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> +{
> +	struct list_head *e = &dev->dev_list;
> +	struct its_device *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_device, dev_list);
> +	next_offset = next->device_id - dev->device_id;
> +
> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> +}
> +
> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> +{
> +	struct list_head *e = &ite->ite_list;
> +	struct its_ite *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_ite, ite_list);
> +	next_offset = next->event_id - ite->event_id;
> +
> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> +}
> +
> +/**
> + * entry_fn_t - Callback called on a table entry restore path
> + * @its: its handle
> + * @id: id of the entry
> + * @entry: pointer to the entry
> + * @opaque: pointer to an opaque data
> + * @next_offset: minimal ID offset to the next entry. 0 if this
> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> + * entry's next_offset field was truly decoded
> + *
> + * Return: < 0 on error, 0 otherwise
> + */
> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> +			  void *opaque, u32 *next_offset);

Just noticed.  All the table entries are 64-bit long at this point,
right?  So why not make entry a u64 * instead?  Could we end up with
some endianness mess with using void pointers the way it is now?

Thanks,
-Christoffer

> +
> +/**
> + * lookup_table - scan a contiguous table in guest RAM and applies a function
> + * to each entry
> + *
> + * @its: its handle
> + * @base: base gpa of the table
> + * @size: size of the table in bytes
> + * @esz: entry size in bytes
> + * @start_id: first entry's ID
> + * @fn: function to apply on each entry
> + *
> + * Return: < 0 on error, 1 if last element identified, 0 otherwise
> + */
> +int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> +		 int start_id, entry_fn_t fn, void *opaque)
> +{
> +	void *entry = kzalloc(esz, GFP_KERNEL);
> +	struct kvm *kvm = its->dev->kvm;
> +	unsigned long len = size;
> +	u32 id = start_id;
> +	gpa_t gpa = base;
> +	int ret;
> +
> +	while (len > 0) {
> +		u32 next_offset;
> +		size_t byte_offset;
> +
> +		ret = kvm_read_guest(kvm, gpa, entry, esz);
> +		if (ret)
> +			goto out;
> +
> +		ret = fn(its, id, entry, opaque, &next_offset);
> +		if (ret < 0 || (!ret && !next_offset))
> +			goto out;
> +
> +		byte_offset = next_offset * esz;
> +		id += next_offset;
> +		gpa += byte_offset;
> +		len -= byte_offset;
> +	}
> +	kfree(entry);
> +	return 0;
> +
> +out:
> +	kfree(entry);
> +	return (ret < 0 ? ret : 1);
> +}
> +
>  /**
>   * vgic_its_save_device_tables - Save the device table and all ITT
>   * into guest RAM
> -- 
> 2.5.5
> 

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> Add a generic lookup_table() helper whose role consists in
> scanning a contiguous table located in guest RAM and applying
> a callback on each entry. Entries can be handled as linked lists
> since the callback may return an offset to the next entry and
> also tell that an entry is the last one.
> 
> Helper functions also are added to compute the device/event ID
> offset to the next DTE/ITE.
> 
> compute_next_devid_offset, compute_next_eventid_offset and
> lookup_table will become static in subsequent patches
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - use kvm_read_guest
> 
> v3 -> v4:
> - remove static to avoid compilation warning
> - correct size computation in looup_table()
> - defines now encode the number of bits used for devid and eventid offsets
> - use BIT() - 1 to encode the max offets
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 93 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 56c5123..c22b35d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  
>  #define VITS_TYPER_IDBITS 16
>  #define VITS_TYPER_DEVBITS 16
> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>  
>  /*
>   * Finds and returns a collection in the ITS collection table.
> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> +{
> +	struct list_head *e = &dev->dev_list;
> +	struct its_device *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_device, dev_list);
> +	next_offset = next->device_id - dev->device_id;
> +
> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> +}
> +
> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> +{
> +	struct list_head *e = &ite->ite_list;
> +	struct its_ite *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_ite, ite_list);
> +	next_offset = next->event_id - ite->event_id;
> +
> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> +}
> +
> +/**
> + * entry_fn_t - Callback called on a table entry restore path
> + * @its: its handle
> + * @id: id of the entry
> + * @entry: pointer to the entry
> + * @opaque: pointer to an opaque data
> + * @next_offset: minimal ID offset to the next entry. 0 if this
> + * entry is the last one, 1 if the entry is invalid, >= 1 if an

eh, also, did you mean -1 if the entry is invalid?

Thanks,
-Christoffer

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> Add a generic lookup_table() helper whose role consists in
> scanning a contiguous table located in guest RAM and applying
> a callback on each entry. Entries can be handled as linked lists
> since the callback may return an offset to the next entry and
> also tell that an entry is the last one.
> 
> Helper functions also are added to compute the device/event ID
> offset to the next DTE/ITE.
> 
> compute_next_devid_offset, compute_next_eventid_offset and
> lookup_table will become static in subsequent patches
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - use kvm_read_guest
> 
> v3 -> v4:
> - remove static to avoid compilation warning
> - correct size computation in looup_table()
> - defines now encode the number of bits used for devid and eventid offsets
> - use BIT() - 1 to encode the max offets
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 93 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 56c5123..c22b35d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  
>  #define VITS_TYPER_IDBITS 16
>  #define VITS_TYPER_DEVBITS 16
> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>  
>  /*
>   * Finds and returns a collection in the ITS collection table.
> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> +{
> +	struct list_head *e = &dev->dev_list;
> +	struct its_device *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_device, dev_list);
> +	next_offset = next->device_id - dev->device_id;
> +
> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> +}
> +
> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> +{
> +	struct list_head *e = &ite->ite_list;
> +	struct its_ite *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_ite, ite_list);
> +	next_offset = next->event_id - ite->event_id;
> +
> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> +}
> +
> +/**
> + * entry_fn_t - Callback called on a table entry restore path
> + * @its: its handle
> + * @id: id of the entry
> + * @entry: pointer to the entry
> + * @opaque: pointer to an opaque data
> + * @next_offset: minimal ID offset to the next entry. 0 if this
> + * entry is the last one, 1 if the entry is invalid, >= 1 if an

eh, also, did you mean -1 if the entry is invalid?

Thanks,
-Christoffer

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> Add a generic lookup_table() helper whose role consists in
> scanning a contiguous table located in guest RAM and applying
> a callback on each entry. Entries can be handled as linked lists
> since the callback may return an offset to the next entry and
> also tell that an entry is the last one.
> 
> Helper functions also are added to compute the device/event ID
> offset to the next DTE/ITE.
> 
> compute_next_devid_offset, compute_next_eventid_offset and
> lookup_table will become static in subsequent patches
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - use kvm_read_guest
> 
> v3 -> v4:
> - remove static to avoid compilation warning
> - correct size computation in looup_table()
> - defines now encode the number of bits used for devid and eventid offsets
> - use BIT() - 1 to encode the max offets
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 93 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 56c5123..c22b35d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  
>  #define VITS_TYPER_IDBITS 16
>  #define VITS_TYPER_DEVBITS 16
> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>  
>  /*
>   * Finds and returns a collection in the ITS collection table.
> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> +{
> +	struct list_head *e = &dev->dev_list;
> +	struct its_device *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_device, dev_list);
> +	next_offset = next->device_id - dev->device_id;
> +
> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> +}
> +
> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> +{
> +	struct list_head *e = &ite->ite_list;
> +	struct its_ite *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_ite, ite_list);
> +	next_offset = next->event_id - ite->event_id;
> +
> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);

nit: in both of these functions you can avoid the extra e variable and
maybe make this slightly more readable by using these two calls instead:

list_is_last(&ite->ite_list, h);
next = list_next_entry(ite, ite_list);

Thanks,
-Christoffer

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> Add a generic lookup_table() helper whose role consists in
> scanning a contiguous table located in guest RAM and applying
> a callback on each entry. Entries can be handled as linked lists
> since the callback may return an offset to the next entry and
> also tell that an entry is the last one.
> 
> Helper functions also are added to compute the device/event ID
> offset to the next DTE/ITE.
> 
> compute_next_devid_offset, compute_next_eventid_offset and
> lookup_table will become static in subsequent patches
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - use kvm_read_guest
> 
> v3 -> v4:
> - remove static to avoid compilation warning
> - correct size computation in looup_table()
> - defines now encode the number of bits used for devid and eventid offsets
> - use BIT() - 1 to encode the max offets
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 93 insertions(+)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 56c5123..c22b35d 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>  
>  #define VITS_TYPER_IDBITS 16
>  #define VITS_TYPER_DEVBITS 16
> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>  
>  /*
>   * Finds and returns a collection in the ITS collection table.
> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> +{
> +	struct list_head *e = &dev->dev_list;
> +	struct its_device *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_device, dev_list);
> +	next_offset = next->device_id - dev->device_id;
> +
> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> +}
> +
> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> +{
> +	struct list_head *e = &ite->ite_list;
> +	struct its_ite *next;
> +	u32 next_offset;
> +
> +	if (e->next == h)
> +		return 0;
> +	next = list_entry(e->next, struct its_ite, ite_list);
> +	next_offset = next->event_id - ite->event_id;
> +
> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);

nit: in both of these functions you can avoid the extra e variable and
maybe make this slightly more readable by using these two calls instead:

list_is_last(&ite->ite_list, h);
next = list_next_entry(ite, ite_list);

Thanks,
-Christoffer

Re: [PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
> Introduce routines to save and restore device ITT and their
> interrupt table entries (ITE).
> 
> The routines will be called on device table save and
> restore. They will become static in subsequent patches.

Why this bottom-up approach?  Couldn't you start by having the patch
that restores the device table and define the static functions that
return an error there, and then fill them in with subsequent patches
(liek this one)?

That would have the added benefit of being able to tell how things are
designed to be called.

> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - ITE are now sorted by eventid on the flush
> - rename *flush* into *save*
> - use macros for shits and masks
> - pass ite_esz to vgic_its_save_ite
> 
> v3 -> v4:
> - lookup_table and compute_next_eventid_offset become static in this
>   patch
> - remove static along with vgic_its_flush/restore_itt to avoid
>   compilation warnings
> - next field only computed with a shift (mask removed)
> - handle the case where the last element has not been found
> 
> v2 -> v3:
> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
> 
> v2: creation
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
>  virt/kvm/arm/vgic/vgic.h     |   4 ++
>  2 files changed, 129 insertions(+), 3 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 35b2ca1..b02fc3f 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -23,6 +23,7 @@
>  #include <linux/interrupt.h>
>  #include <linux/list.h>
>  #include <linux/uaccess.h>
> +#include <linux/list_sort.h>
>  
>  #include <linux/irqchip/arm-gic-v3.h>
>  
> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>  }
>  
> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>  {
>  	struct list_head *e = &ite->ite_list;
>  	struct its_ite *next;
> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>   *
>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
>   */
> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> -		 int start_id, entry_fn_t fn, void *opaque)
> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> +			int start_id, entry_fn_t fn, void *opaque)
>  {
>  	void *entry = kzalloc(esz, GFP_KERNEL);
>  	struct kvm *kvm = its->dev->kvm;
> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>  }
>  
>  /**
> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
> + */
> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
> +{
> +	struct kvm *kvm = its->dev->kvm;
> +	u32 next_offset;
> +	u64 val;
> +
> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
> +		ite->collection->collection_id;
> +	val = cpu_to_le64(val);
> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
> +}
> +
> +/**
> + * vgic_its_restore_ite - restore an interrupt translation entry
> + * @event_id: id used for indexing
> + * @ptr: pointer to the ITE entry
> + * @opaque: pointer to the its_device
> + * @next: id offset to the next entry
> + */
> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
> +				void *ptr, void *opaque, u32 *next)
> +{
> +	struct its_device *dev = (struct its_device *)opaque;
> +	struct its_collection *collection;
> +	struct kvm *kvm = its->dev->kvm;
> +	u64 val, *p = (u64 *)ptr;

nit: initializations on separate line (and possible do that just above
assigning val).

> +	struct vgic_irq *irq;
> +	u32 coll_id, lpi_id;
> +	struct its_ite *ite;
> +	int ret;
> +
> +	val = *p;
> +	*next = 1;
> +
> +	val = le64_to_cpu(val);
> +
> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
> +
> +	if (!lpi_id)
> +		return 0;

are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
the ID is valid?

(looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
and PPIs here)

> +
> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;

Don't we need to validate this somehow since it will presumably be used
to forward a pointer somehow by the caller?

> +
> +	collection = find_collection(its, coll_id);
> +	if (!collection)
> +		return -EINVAL;
> +
> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
> +				  lpi_id, event_id);
> +	if (ret)
> +		return ret;
> +
> +	irq = vgic_add_lpi(kvm, lpi_id);
> +	if (IS_ERR(irq))
> +		return PTR_ERR(irq);
> +	ite->irq = irq;
> +
> +	/* restore the configuration of the LPI */
> +	ret = update_lpi_config(kvm, irq, NULL);
> +	if (ret)
> +		return ret;
> +
> +	update_affinity_ite(kvm, ite);
> +	return 0;
> +}
> +
> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
> +			    struct list_head *b)
> +{
> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
> +
> +	if (itea->event_id < iteb->event_id)
> +		return -1;
> +	else
> +		return 1;
> +}
> +
> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> +{
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	gpa_t base = device->itt_addr;
> +	struct its_ite *ite;
> +	int ret, ite_esz = abi->ite_esz;

nit: initializations on separate line

> +
> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
> +
> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
> +		gpa_t gpa = base + ite->event_id * ite_esz;
> +
> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
> +		if (ret)
> +			return ret;
> +	}
> +	return 0;
> +}
> +
> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> +{
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	gpa_t base = dev->itt_addr;
> +	int ret, ite_esz = abi->ite_esz;
> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;

nit: initializations on separate line

> +
> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
> +			    vgic_its_restore_ite, dev);

nit: extra white space

> +
> +	if (ret < 0)
> +		return ret;
> +
> +	/* if the last element has not been found we are in trouble */
> +	return ret ? 0 : -EINVAL;

hmm, these are values potentially created by the guest in guest RAM,
right?  So do we really abort migration and return an error to userspace
in this case?

Also, this comment doesn't really tell me what this situation is and how
we handle things...

> +}
> +
> +/**
>   * vgic_its_save_device_tables - Save the device table and all ITT
>   * into guest RAM
>   */
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 56e57c1..ce57fbd 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -81,6 +81,10 @@
>  #define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
>  #define KVM_ITS_CTE_RDBASE_SHIFT	16
>  #define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
> +#define KVM_ITS_ITE_NEXT_SHIFT		48
> +#define KVM_ITS_ITE_PINTID_SHIFT	16
> +#define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
> +#define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
>  
>  static inline bool irq_is_pending(struct vgic_irq *irq)
>  {
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
> Introduce routines to save and restore device ITT and their
> interrupt table entries (ITE).
> 
> The routines will be called on device table save and
> restore. They will become static in subsequent patches.

Why this bottom-up approach?  Couldn't you start by having the patch
that restores the device table and define the static functions that
return an error there, and then fill them in with subsequent patches
(liek this one)?

That would have the added benefit of being able to tell how things are
designed to be called.

> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - ITE are now sorted by eventid on the flush
> - rename *flush* into *save*
> - use macros for shits and masks
> - pass ite_esz to vgic_its_save_ite
> 
> v3 -> v4:
> - lookup_table and compute_next_eventid_offset become static in this
>   patch
> - remove static along with vgic_its_flush/restore_itt to avoid
>   compilation warnings
> - next field only computed with a shift (mask removed)
> - handle the case where the last element has not been found
> 
> v2 -> v3:
> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
> 
> v2: creation
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
>  virt/kvm/arm/vgic/vgic.h     |   4 ++
>  2 files changed, 129 insertions(+), 3 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 35b2ca1..b02fc3f 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -23,6 +23,7 @@
>  #include <linux/interrupt.h>
>  #include <linux/list.h>
>  #include <linux/uaccess.h>
> +#include <linux/list_sort.h>
>  
>  #include <linux/irqchip/arm-gic-v3.h>
>  
> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>  }
>  
> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>  {
>  	struct list_head *e = &ite->ite_list;
>  	struct its_ite *next;
> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>   *
>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
>   */
> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> -		 int start_id, entry_fn_t fn, void *opaque)
> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> +			int start_id, entry_fn_t fn, void *opaque)
>  {
>  	void *entry = kzalloc(esz, GFP_KERNEL);
>  	struct kvm *kvm = its->dev->kvm;
> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>  }
>  
>  /**
> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
> + */
> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
> +{
> +	struct kvm *kvm = its->dev->kvm;
> +	u32 next_offset;
> +	u64 val;
> +
> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
> +		ite->collection->collection_id;
> +	val = cpu_to_le64(val);
> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
> +}
> +
> +/**
> + * vgic_its_restore_ite - restore an interrupt translation entry
> + * @event_id: id used for indexing
> + * @ptr: pointer to the ITE entry
> + * @opaque: pointer to the its_device
> + * @next: id offset to the next entry
> + */
> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
> +				void *ptr, void *opaque, u32 *next)
> +{
> +	struct its_device *dev = (struct its_device *)opaque;
> +	struct its_collection *collection;
> +	struct kvm *kvm = its->dev->kvm;
> +	u64 val, *p = (u64 *)ptr;

nit: initializations on separate line (and possible do that just above
assigning val).

> +	struct vgic_irq *irq;
> +	u32 coll_id, lpi_id;
> +	struct its_ite *ite;
> +	int ret;
> +
> +	val = *p;
> +	*next = 1;
> +
> +	val = le64_to_cpu(val);
> +
> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
> +
> +	if (!lpi_id)
> +		return 0;

are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
the ID is valid?

(looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
and PPIs here)

> +
> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;

Don't we need to validate this somehow since it will presumably be used
to forward a pointer somehow by the caller?

> +
> +	collection = find_collection(its, coll_id);
> +	if (!collection)
> +		return -EINVAL;
> +
> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
> +				  lpi_id, event_id);
> +	if (ret)
> +		return ret;
> +
> +	irq = vgic_add_lpi(kvm, lpi_id);
> +	if (IS_ERR(irq))
> +		return PTR_ERR(irq);
> +	ite->irq = irq;
> +
> +	/* restore the configuration of the LPI */
> +	ret = update_lpi_config(kvm, irq, NULL);
> +	if (ret)
> +		return ret;
> +
> +	update_affinity_ite(kvm, ite);
> +	return 0;
> +}
> +
> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
> +			    struct list_head *b)
> +{
> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
> +
> +	if (itea->event_id < iteb->event_id)
> +		return -1;
> +	else
> +		return 1;
> +}
> +
> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> +{
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	gpa_t base = device->itt_addr;
> +	struct its_ite *ite;
> +	int ret, ite_esz = abi->ite_esz;

nit: initializations on separate line

> +
> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
> +
> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
> +		gpa_t gpa = base + ite->event_id * ite_esz;
> +
> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
> +		if (ret)
> +			return ret;
> +	}
> +	return 0;
> +}
> +
> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> +{
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	gpa_t base = dev->itt_addr;
> +	int ret, ite_esz = abi->ite_esz;
> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;

nit: initializations on separate line

> +
> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
> +			    vgic_its_restore_ite, dev);

nit: extra white space

> +
> +	if (ret < 0)
> +		return ret;
> +
> +	/* if the last element has not been found we are in trouble */
> +	return ret ? 0 : -EINVAL;

hmm, these are values potentially created by the guest in guest RAM,
right?  So do we really abort migration and return an error to userspace
in this case?

Also, this comment doesn't really tell me what this situation is and how
we handle things...

> +}
> +
> +/**
>   * vgic_its_save_device_tables - Save the device table and all ITT
>   * into guest RAM
>   */
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 56e57c1..ce57fbd 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -81,6 +81,10 @@
>  #define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
>  #define KVM_ITS_CTE_RDBASE_SHIFT	16
>  #define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
> +#define KVM_ITS_ITE_NEXT_SHIFT		48
> +#define KVM_ITS_ITE_PINTID_SHIFT	16
> +#define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
> +#define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
>  
>  static inline bool irq_is_pending(struct vgic_irq *irq)
>  {
> -- 
> 2.5.5
> 

Thanks,
-Christoffer

Re: [PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:32PM +0200, Eric Auger wrote:
> This patch saves the device table entries into guest RAM.
> Both flat table and 2 stage tables are supported. DeviceId
> indexing is used.
> 
> For each device listed in the device table, we also save
> the translation table using the vgic_its_save/restore_itt
> routines.
> 
> On restore, devices are re-allocated and their ite are
> re-built.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - sort the device list by deviceid on device table save
> - use defines for shifts and masks
> - use abi->dte_esz
> - clatify entry sizes for L1 and L2 tables
> 
> v3 -> v4:
> - use the new proto for its_alloc_device
> - compute_next_devid_offset, vgic_its_flush/restore_itt
>   become static in this patch
> - change in the DTE entry format with the introduction of the
>   valid bit and next field width decrease; ittaddr encoded
>   on its full range
> - fix handle_l1_entry entry handling
> - correct vgic_its_table_restore error handling
> 
> v2 -> v3:
> - fix itt_addr bitmask in vgic_its_restore_dte
> - addition of return 0 in vgic_its_restore_ite moved to
>   the ITE related patch
> 
> v1 -> v2:
> - use 8 byte format for DTE and ITE
> - support 2 stage format
> - remove kvm parameter
> - ITT flush/restore moved in a separate patch
> - use deviceid indexing
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 183 +++++++++++++++++++++++++++++++++++++++++--
>  virt/kvm/arm/vgic/vgic.h     |   7 ++
>  2 files changed, 185 insertions(+), 5 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index b02fc3f..86dfc6c 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1682,7 +1682,8 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> -u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> +static u32 compute_next_devid_offset(struct list_head *h,
> +				     struct its_device *dev)
>  {
>  	struct list_head *e = &dev->dev_list;
>  	struct its_device *next;
> @@ -1858,7 +1859,7 @@ static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>  		return 1;
>  }
>  
> -int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> +static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>  {
>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>  	gpa_t base = device->itt_addr;
> @@ -1877,7 +1878,7 @@ int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>  	return 0;
>  }
>  
> -int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> +static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>  {
>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>  	gpa_t base = dev->itt_addr;
> @@ -1895,12 +1896,161 @@ int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>  }
>  
>  /**
> + * vgic_its_save_dte - Save a device table entry at a given GPA
> + *
> + * @its: ITS handle
> + * @dev: ITS device
> + * @ptr: GPA
> + */
> +static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
> +			     gpa_t ptr, int dte_esz)
> +{
> +	struct kvm *kvm = its->dev->kvm;
> +	u64 val, itt_addr_field;
> +	int ret;
> +	u32 next_offset;
> +
> +	itt_addr_field = dev->itt_addr >> 8;
> +	next_offset = compute_next_devid_offset(&its->device_list, dev);
> +	val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
> +	       ((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |

I think this implies that the next field in your ABI points to the next
offset, regardless of whether or not this is in a a level 2 or lavel 1
table.  See more comments on this below (I reviewed this patch from the
bottom up).

I have a feeling this wasn't tested with 2 level device tables.  Could
that be true?

> +	       (itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
> +		(dev->nb_eventid_bits - 1));
> +	val = cpu_to_le64(val);
> +	ret = kvm_write_guest(kvm, ptr, &val, dte_esz);
> +	return ret;
> +}
> +
> +/**
> + * vgic_its_restore_dte - restore a device table entry
> + *
> + * @its: its handle
> + * @id: device id the DTE corresponds to
> + * @ptr: kernel VA where the 8 byte DTE is located
> + * @opaque: unused
> + * @next: offset to the next valid device id
> + *
> + * Return: < 0 on error, 0 otherwise
> + */
> +static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
> +				void *ptr, void *opaque, u32 *next)
> +{
> +	struct its_device *dev;
> +	gpa_t itt_addr;
> +	u8 nb_eventid_bits;
> +	u64 entry = *(u64 *)ptr;
> +	bool valid;
> +	int ret;
> +
> +	entry = le64_to_cpu(entry);
> +
> +	valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
> +	nb_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
> +	itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
> +			>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
> +	*next = 1;
> +
> +	if (!valid)
> +		return 0;
> +
> +	/* dte entry is valid */
> +	*next = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
> +
> +	ret = vgic_its_alloc_device(its, &dev, id,
> +				    itt_addr, nb_eventid_bits);
> +	if (ret)
> +		return ret;
> +	ret = vgic_its_restore_itt(its, dev);
> +
> +	return ret;
> +}
> +
> +static int vgic_its_device_cmp(void *priv, struct list_head *a,
> +			       struct list_head *b)
> +{
> +	struct its_device *deva = container_of(a, struct its_device, dev_list);
> +	struct its_device *devb = container_of(b, struct its_device, dev_list);
> +
> +	if (deva->device_id < devb->device_id)
> +		return -1;
> +	else
> +		return 1;
> +}
> +
> +/**
>   * vgic_its_save_device_tables - Save the device table and all ITT
>   * into guest RAM
> + *
> + * L1/L2 handling is hidden by vgic_its_check_id() helper which directly
> + * returns the GPA of the device entry
>   */
>  static int vgic_its_save_device_tables(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	struct its_device *dev;
> +	int dte_esz = abi->dte_esz;
> +	u64 baser;
> +
> +	baser = its->baser_device_table;
> +
> +	list_sort(NULL, &its->device_list, vgic_its_device_cmp);
> +
> +	list_for_each_entry(dev, &its->device_list, dev_list) {
> +		int ret;
> +		gpa_t eaddr;
> +
> +		if (!vgic_its_check_id(its, baser,
> +				       dev->device_id, &eaddr))
> +			return -EINVAL;
> +
> +		ret = vgic_its_save_itt(its, dev);
> +		if (ret)
> +			return ret;
> +
> +		ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
> +		if (ret)
> +			return ret;
> +	}
> +	return 0;
> +}
> +
> +/**
> + * handle_l1_entry - callback used for L1 entries (2 stage case)
> + *
> + * @its: its handle
> + * @id: id

IIUC, this is actually the index of the entry in the L1 table.  I think
this should be clarified.

> + * @addr: kernel VA
> + * @opaque: unused
> + * @next_offset: offset to the next L1 entry: 0 if the last element
> + * was found, 1 otherwise
> + */
> +static int handle_l1_entry(struct vgic_its *its, u32 id, void *addr,
> +			   void *opaque, u32 *next_offset)

nit: shouldn't this be called handle_l1_device_table_entry ?

> +{
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	int l2_start_id = id * (SZ_64K / GITS_LVL1_ENTRY_SIZE);

Hmmm, is this not actually supposed to be (SZ_64K / abi->dte_esz) ?

> +	u64 entry = *(u64 *)addr;
> +	int ret, ite_esz = abi->ite_esz;

Should this be ite_esz or dte_esz?

> +	gpa_t gpa;

nit: put declarations with initialization on separate lines.

> +
> +	entry = le64_to_cpu(entry);
> +	*next_offset = 1;

I think you could attach a comment here saying that level 1 tables have
to be scanned entirely.

But this also reminds me.  Does that mean that the next field in the DTE
in your documented ABI format points to the next DTE within that level-2
table, or does it point across to different level-2 tables?  I think
this needs to be clarified in the ABI unless I'm missing something.

> +
> +	if (!(entry & BIT_ULL(63)))
> +		return 0;
> +
> +	gpa = entry & GENMASK_ULL(51, 16);

Can you define the bit fields for the level-1 entries as well please?

> +
> +	ret = lookup_table(its, gpa, SZ_64K, ite_esz,
> +			   l2_start_id, vgic_its_restore_dte, NULL);
> +
> +	if (ret == 1) {
> +		/* last entry was found in this L2 table */

maybe you should define these return codes for you table scan function,
and you wouldn't have to have a separate comment and it would be
generally easier to follow the code.

> +		*next_offset = 0;
> +		ret = 0;
> +	}
> +
> +	return ret;
>  }
>  
>  /**
> @@ -1909,7 +2059,30 @@ static int vgic_its_save_device_tables(struct vgic_its *its)
>   */
>  static int vgic_its_restore_device_tables(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	u64 baser = its->baser_device_table;
> +	int l1_esz, ret, l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;

nit: put this initialization on a separate line.

> +	gpa_t l1_gpa;
> +
> +	l1_gpa = BASER_ADDRESS(baser);
> +	if (!l1_gpa)
> +		return 0;

I think you meant to check the valid bit here.

> +
> +	if (baser & GITS_BASER_INDIRECT) {
> +		l1_esz = 8;
> +		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
> +				   handle_l1_entry, NULL);
> +	} else {
> +		l1_esz = abi->dte_esz;
> +		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
> +				   vgic_its_restore_dte, NULL);
> +	}
> +
> +	if (ret < 0)
> +		return ret;
> +
> +	/* if last element was not found we have an issue here */

same comment as other patch

> +	return ret ? 0 : -EINVAL;
>  }
>  
>  static int vgic_its_save_cte(struct vgic_its *its,
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index ce57fbd..9bc52ef 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -85,6 +85,13 @@
>  #define KVM_ITS_ITE_PINTID_SHIFT	16
>  #define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
>  #define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
> +#define KVM_ITS_DTE_VALID_SHIFT		63
> +#define KVM_ITS_DTE_VALID_MASK		BIT_ULL(63)
> +#define KVM_ITS_DTE_NEXT_SHIFT		49
> +#define KVM_ITS_DTE_NEXT_MASK		GENMASK_ULL(62, 49)
> +#define KVM_ITS_DTE_ITTADDR_SHIFT	5
> +#define KVM_ITS_DTE_ITTADDR_MASK	GENMASK_ULL(48, 5)
> +#define KVM_ITS_DTE_SIZE_MASK		GENMASK_ULL(4, 0)
>  
>  static inline bool irq_is_pending(struct vgic_irq *irq)
>  {
> -- 
> 2.5.5
> 
Thanks,
-Christoffer

[PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:32PM +0200, Eric Auger wrote:
> This patch saves the device table entries into guest RAM.
> Both flat table and 2 stage tables are supported. DeviceId
> indexing is used.
> 
> For each device listed in the device table, we also save
> the translation table using the vgic_its_save/restore_itt
> routines.
> 
> On restore, devices are re-allocated and their ite are
> re-built.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - sort the device list by deviceid on device table save
> - use defines for shifts and masks
> - use abi->dte_esz
> - clatify entry sizes for L1 and L2 tables
> 
> v3 -> v4:
> - use the new proto for its_alloc_device
> - compute_next_devid_offset, vgic_its_flush/restore_itt
>   become static in this patch
> - change in the DTE entry format with the introduction of the
>   valid bit and next field width decrease; ittaddr encoded
>   on its full range
> - fix handle_l1_entry entry handling
> - correct vgic_its_table_restore error handling
> 
> v2 -> v3:
> - fix itt_addr bitmask in vgic_its_restore_dte
> - addition of return 0 in vgic_its_restore_ite moved to
>   the ITE related patch
> 
> v1 -> v2:
> - use 8 byte format for DTE and ITE
> - support 2 stage format
> - remove kvm parameter
> - ITT flush/restore moved in a separate patch
> - use deviceid indexing
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 183 +++++++++++++++++++++++++++++++++++++++++--
>  virt/kvm/arm/vgic/vgic.h     |   7 ++
>  2 files changed, 185 insertions(+), 5 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index b02fc3f..86dfc6c 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -1682,7 +1682,8 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>  	return ret;
>  }
>  
> -u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> +static u32 compute_next_devid_offset(struct list_head *h,
> +				     struct its_device *dev)
>  {
>  	struct list_head *e = &dev->dev_list;
>  	struct its_device *next;
> @@ -1858,7 +1859,7 @@ static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>  		return 1;
>  }
>  
> -int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> +static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>  {
>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>  	gpa_t base = device->itt_addr;
> @@ -1877,7 +1878,7 @@ int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>  	return 0;
>  }
>  
> -int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> +static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>  {
>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>  	gpa_t base = dev->itt_addr;
> @@ -1895,12 +1896,161 @@ int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>  }
>  
>  /**
> + * vgic_its_save_dte - Save a device table entry at a given GPA
> + *
> + * @its: ITS handle
> + * @dev: ITS device
> + * @ptr: GPA
> + */
> +static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
> +			     gpa_t ptr, int dte_esz)
> +{
> +	struct kvm *kvm = its->dev->kvm;
> +	u64 val, itt_addr_field;
> +	int ret;
> +	u32 next_offset;
> +
> +	itt_addr_field = dev->itt_addr >> 8;
> +	next_offset = compute_next_devid_offset(&its->device_list, dev);
> +	val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
> +	       ((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |

I think this implies that the next field in your ABI points to the next
offset, regardless of whether or not this is in a a level 2 or lavel 1
table.  See more comments on this below (I reviewed this patch from the
bottom up).

I have a feeling this wasn't tested with 2 level device tables.  Could
that be true?

> +	       (itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
> +		(dev->nb_eventid_bits - 1));
> +	val = cpu_to_le64(val);
> +	ret = kvm_write_guest(kvm, ptr, &val, dte_esz);
> +	return ret;
> +}
> +
> +/**
> + * vgic_its_restore_dte - restore a device table entry
> + *
> + * @its: its handle
> + * @id: device id the DTE corresponds to
> + * @ptr: kernel VA where the 8 byte DTE is located
> + * @opaque: unused
> + * @next: offset to the next valid device id
> + *
> + * Return: < 0 on error, 0 otherwise
> + */
> +static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
> +				void *ptr, void *opaque, u32 *next)
> +{
> +	struct its_device *dev;
> +	gpa_t itt_addr;
> +	u8 nb_eventid_bits;
> +	u64 entry = *(u64 *)ptr;
> +	bool valid;
> +	int ret;
> +
> +	entry = le64_to_cpu(entry);
> +
> +	valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
> +	nb_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
> +	itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
> +			>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
> +	*next = 1;
> +
> +	if (!valid)
> +		return 0;
> +
> +	/* dte entry is valid */
> +	*next = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
> +
> +	ret = vgic_its_alloc_device(its, &dev, id,
> +				    itt_addr, nb_eventid_bits);
> +	if (ret)
> +		return ret;
> +	ret = vgic_its_restore_itt(its, dev);
> +
> +	return ret;
> +}
> +
> +static int vgic_its_device_cmp(void *priv, struct list_head *a,
> +			       struct list_head *b)
> +{
> +	struct its_device *deva = container_of(a, struct its_device, dev_list);
> +	struct its_device *devb = container_of(b, struct its_device, dev_list);
> +
> +	if (deva->device_id < devb->device_id)
> +		return -1;
> +	else
> +		return 1;
> +}
> +
> +/**
>   * vgic_its_save_device_tables - Save the device table and all ITT
>   * into guest RAM
> + *
> + * L1/L2 handling is hidden by vgic_its_check_id() helper which directly
> + * returns the GPA of the device entry
>   */
>  static int vgic_its_save_device_tables(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	struct its_device *dev;
> +	int dte_esz = abi->dte_esz;
> +	u64 baser;
> +
> +	baser = its->baser_device_table;
> +
> +	list_sort(NULL, &its->device_list, vgic_its_device_cmp);
> +
> +	list_for_each_entry(dev, &its->device_list, dev_list) {
> +		int ret;
> +		gpa_t eaddr;
> +
> +		if (!vgic_its_check_id(its, baser,
> +				       dev->device_id, &eaddr))
> +			return -EINVAL;
> +
> +		ret = vgic_its_save_itt(its, dev);
> +		if (ret)
> +			return ret;
> +
> +		ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
> +		if (ret)
> +			return ret;
> +	}
> +	return 0;
> +}
> +
> +/**
> + * handle_l1_entry - callback used for L1 entries (2 stage case)
> + *
> + * @its: its handle
> + * @id: id

IIUC, this is actually the index of the entry in the L1 table.  I think
this should be clarified.

> + * @addr: kernel VA
> + * @opaque: unused
> + * @next_offset: offset to the next L1 entry: 0 if the last element
> + * was found, 1 otherwise
> + */
> +static int handle_l1_entry(struct vgic_its *its, u32 id, void *addr,
> +			   void *opaque, u32 *next_offset)

nit: shouldn't this be called handle_l1_device_table_entry ?

> +{
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	int l2_start_id = id * (SZ_64K / GITS_LVL1_ENTRY_SIZE);

Hmmm, is this not actually supposed to be (SZ_64K / abi->dte_esz) ?

> +	u64 entry = *(u64 *)addr;
> +	int ret, ite_esz = abi->ite_esz;

Should this be ite_esz or dte_esz?

> +	gpa_t gpa;

nit: put declarations with initialization on separate lines.

> +
> +	entry = le64_to_cpu(entry);
> +	*next_offset = 1;

I think you could attach a comment here saying that level 1 tables have
to be scanned entirely.

But this also reminds me.  Does that mean that the next field in the DTE
in your documented ABI format points to the next DTE within that level-2
table, or does it point across to different level-2 tables?  I think
this needs to be clarified in the ABI unless I'm missing something.

> +
> +	if (!(entry & BIT_ULL(63)))
> +		return 0;
> +
> +	gpa = entry & GENMASK_ULL(51, 16);

Can you define the bit fields for the level-1 entries as well please?

> +
> +	ret = lookup_table(its, gpa, SZ_64K, ite_esz,
> +			   l2_start_id, vgic_its_restore_dte, NULL);
> +
> +	if (ret == 1) {
> +		/* last entry was found in this L2 table */

maybe you should define these return codes for you table scan function,
and you wouldn't have to have a separate comment and it would be
generally easier to follow the code.

> +		*next_offset = 0;
> +		ret = 0;
> +	}
> +
> +	return ret;
>  }
>  
>  /**
> @@ -1909,7 +2059,30 @@ static int vgic_its_save_device_tables(struct vgic_its *its)
>   */
>  static int vgic_its_restore_device_tables(struct vgic_its *its)
>  {
> -	return -ENXIO;
> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> +	u64 baser = its->baser_device_table;
> +	int l1_esz, ret, l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;

nit: put this initialization on a separate line.

> +	gpa_t l1_gpa;
> +
> +	l1_gpa = BASER_ADDRESS(baser);
> +	if (!l1_gpa)
> +		return 0;

I think you meant to check the valid bit here.

> +
> +	if (baser & GITS_BASER_INDIRECT) {
> +		l1_esz = 8;
> +		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
> +				   handle_l1_entry, NULL);
> +	} else {
> +		l1_esz = abi->dte_esz;
> +		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
> +				   vgic_its_restore_dte, NULL);
> +	}
> +
> +	if (ret < 0)
> +		return ret;
> +
> +	/* if last element was not found we have an issue here */

same comment as other patch

> +	return ret ? 0 : -EINVAL;
>  }
>  
>  static int vgic_its_save_cte(struct vgic_its *its,
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index ce57fbd..9bc52ef 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -85,6 +85,13 @@
>  #define KVM_ITS_ITE_PINTID_SHIFT	16
>  #define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
>  #define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
> +#define KVM_ITS_DTE_VALID_SHIFT		63
> +#define KVM_ITS_DTE_VALID_MASK		BIT_ULL(63)
> +#define KVM_ITS_DTE_NEXT_SHIFT		49
> +#define KVM_ITS_DTE_NEXT_MASK		GENMASK_ULL(62, 49)
> +#define KVM_ITS_DTE_ITTADDR_SHIFT	5
> +#define KVM_ITS_DTE_ITTADDR_MASK	GENMASK_ULL(48, 5)
> +#define KVM_ITS_DTE_SIZE_MASK		GENMASK_ULL(4, 0)
>  
>  static inline bool irq_is_pending(struct vgic_irq *irq)
>  {
> -- 
> 2.5.5
> 
Thanks,
-Christoffer

Re: [PATCH v5 21/22] KVM: arm64: vgic-its: Fix pending table sync

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:33PM +0200, Eric Auger wrote:
> In its_sync_lpi_pending_table() we currently ignore the
> target_vcpu of the LPIs. We sync the pending bit found in
> the vcpu pending table even if the LPI is not targeting it.
> 
> Also in vgic_its_cmd_handle_invall() we are supposed to
> read the config table data for the LPIs associated to the
> collection ID. At the moment we refresh all LPI config
> information.
> 
> This patch passes a vpcu to vgic_copy_lpi_list() so that
> this latter returns a snapshot of the LPIs targeting this
> CPU and only those.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 24 ++++++++++++------------
>  1 file changed, 12 insertions(+), 12 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 86dfc6c..be848be 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -252,13 +252,13 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
>  }
>  
>  /*
> - * Create a snapshot of the current LPI list, so that we can enumerate all
> - * LPIs without holding any lock.
> - * Returns the array length and puts the kmalloc'ed array into intid_ptr.
> + * Create a snapshot of the current LPIs targeting @vcpu, so that we can
> + * enumerate those LPIs without holding any lock.
> + * Returns their number and puts the kmalloc'ed array into intid_ptr.
>   */
> -static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
> +static int vgic_copy_lpi_list(struct kvm_vcpu *vcpu, u32 **intid_ptr)
>  {
> -	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>  	struct vgic_irq *irq;
>  	u32 *intids;
>  	int irq_count = dist->lpi_list_count, i = 0;
> @@ -277,14 +277,14 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>  	spin_lock(&dist->lpi_list_lock);
>  	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
>  		/* We don't need to "get" the IRQ, as we hold the list lock. */
> -		intids[i] = irq->intid;
> -		if (++i == irq_count)
> -			break;
> +		if (irq->target_vcpu != vcpu)
> +			continue;
> +		intids[i++] = irq->intid;

were we checking the ++i == irq_count condition for no good reason
before since we can just drop it now?

>  	}
>  	spin_unlock(&dist->lpi_list_lock);
>  
>  	*intid_ptr = intids;
> -	return irq_count;
> +	return i;
>  }
>  
>  /*
> @@ -333,7 +333,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
>  }
>  
>  /*
> - * Scan the whole LPI pending table and sync the pending bit in there
> + * Sync the pending table pending bit of LPIs targeting @vcpu
>   * with our own data structures. This relies on the LPI being
>   * mapped before.
>   */
> @@ -346,7 +346,7 @@ static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
>  	u32 *intids;
>  	int nr_irqs, i;
>  
> -	nr_irqs = vgic_copy_lpi_list(vcpu->kvm, &intids);
> +	nr_irqs = vgic_copy_lpi_list(vcpu, &intids);
>  	if (nr_irqs < 0)
>  		return nr_irqs;
>  
> @@ -1027,7 +1027,7 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
>  
>  	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>  
> -	irq_count = vgic_copy_lpi_list(kvm, &intids);
> +	irq_count = vgic_copy_lpi_list(vcpu, &intids);
>  	if (irq_count < 0)
>  		return irq_count;
>  
> -- 
> 2.5.5
> 

Assuming that it's ok to remove the irq_count check above, the rest of
this patch looks good to me.

Thanks,
-Christoffer

[PATCH v5 21/22] KVM: arm64: vgic-its: Fix pending table sync

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:33PM +0200, Eric Auger wrote:
> In its_sync_lpi_pending_table() we currently ignore the
> target_vcpu of the LPIs. We sync the pending bit found in
> the vcpu pending table even if the LPI is not targeting it.
> 
> Also in vgic_its_cmd_handle_invall() we are supposed to
> read the config table data for the LPIs associated to the
> collection ID. At the moment we refresh all LPI config
> information.
> 
> This patch passes a vpcu to vgic_copy_lpi_list() so that
> this latter returns a snapshot of the LPIs targeting this
> CPU and only those.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 24 ++++++++++++------------
>  1 file changed, 12 insertions(+), 12 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 86dfc6c..be848be 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -252,13 +252,13 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
>  }
>  
>  /*
> - * Create a snapshot of the current LPI list, so that we can enumerate all
> - * LPIs without holding any lock.
> - * Returns the array length and puts the kmalloc'ed array into intid_ptr.
> + * Create a snapshot of the current LPIs targeting @vcpu, so that we can
> + * enumerate those LPIs without holding any lock.
> + * Returns their number and puts the kmalloc'ed array into intid_ptr.
>   */
> -static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
> +static int vgic_copy_lpi_list(struct kvm_vcpu *vcpu, u32 **intid_ptr)
>  {
> -	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>  	struct vgic_irq *irq;
>  	u32 *intids;
>  	int irq_count = dist->lpi_list_count, i = 0;
> @@ -277,14 +277,14 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>  	spin_lock(&dist->lpi_list_lock);
>  	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
>  		/* We don't need to "get" the IRQ, as we hold the list lock. */
> -		intids[i] = irq->intid;
> -		if (++i == irq_count)
> -			break;
> +		if (irq->target_vcpu != vcpu)
> +			continue;
> +		intids[i++] = irq->intid;

were we checking the ++i == irq_count condition for no good reason
before since we can just drop it now?

>  	}
>  	spin_unlock(&dist->lpi_list_lock);
>  
>  	*intid_ptr = intids;
> -	return irq_count;
> +	return i;
>  }
>  
>  /*
> @@ -333,7 +333,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
>  }
>  
>  /*
> - * Scan the whole LPI pending table and sync the pending bit in there
> + * Sync the pending table pending bit of LPIs targeting @vcpu
>   * with our own data structures. This relies on the LPI being
>   * mapped before.
>   */
> @@ -346,7 +346,7 @@ static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
>  	u32 *intids;
>  	int nr_irqs, i;
>  
> -	nr_irqs = vgic_copy_lpi_list(vcpu->kvm, &intids);
> +	nr_irqs = vgic_copy_lpi_list(vcpu, &intids);
>  	if (nr_irqs < 0)
>  		return nr_irqs;
>  
> @@ -1027,7 +1027,7 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
>  
>  	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>  
> -	irq_count = vgic_copy_lpi_list(kvm, &intids);
> +	irq_count = vgic_copy_lpi_list(vcpu, &intids);
>  	if (irq_count < 0)
>  		return irq_count;
>  
> -- 
> 2.5.5
> 

Assuming that it's ok to remove the irq_count check above, the rest of
this patch looks good to me.

Thanks,
-Christoffer

Re: [PATCH v5 22/22] KVM: arm64: vgic-v3: KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:34PM +0200, Eric Auger wrote:
> This patch adds a new attribute to GICV3 KVM device
> KVM_DEV_ARM_VGIC_GRP_CTRL group. This Allows the userspace to

nit: allows (lowercase)
nit: s/the userspace/userspace/

> flush all GICR pending tables into guest RAM. At the moment
> we do not offer any restore control as the sync is implicit.

I would probably remove the last sentence here.

> 
> As we need the PENDBASER_ADDRESS() in vgic-v3, let's move its
> definition in the irqchip header. We restore the full length
> of the field, ie [51:16]. Same for PROPBASER_ADDRESS with full
> field length of [51:12].
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - move pending table save code/ctrl into VGICv3 instead of ITS
> - remove useless gpa_t cast
> - use the same optimization as in its_sync_lpi_pending_table
> 
> v3 -> v4:
> - remove the wrong comment about locking
> - pass kvm struct instead of its handle
> - add comment about restore method
> - remove GITR_PENDABASER.PTZ check
> - continue if target_vcpu == NULL
> - new locking strategy
> 
> v1 -> v2:
> - do not care about the 1st KB which should be zeroed according to
>   the spec.
> ---
>  arch/arm/include/uapi/asm/kvm.h     |  1 +
>  arch/arm64/include/uapi/asm/kvm.h   |  1 +
>  include/linux/irqchip/arm-gic-v3.h  |  2 ++
>  virt/kvm/arm/vgic/vgic-its.c        |  6 ++---
>  virt/kvm/arm/vgic/vgic-kvm-device.c | 20 ++++++++++++++
>  virt/kvm/arm/vgic/vgic-v3.c         | 54 +++++++++++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h            |  1 +
>  7 files changed, 81 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
> index 8e6563c..78fe803 100644
> --- a/arch/arm/include/uapi/asm/kvm.h
> +++ b/arch/arm/include/uapi/asm/kvm.h
> @@ -202,6 +202,7 @@ struct kvm_arch_memory_slot {
>  #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
>  #define   KVM_DEV_ARM_ITS_SAVE_TABLES		1
>  #define   KVM_DEV_ARM_ITS_RESTORE_TABLES	2
> +#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
>  
>  /* KVM_IRQ_LINE irq field index values */
>  #define KVM_ARM_IRQ_TYPE_SHIFT		24
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index 1e35115..8a3758a 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -222,6 +222,7 @@ struct kvm_arch_memory_slot {
>  #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
>  #define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
>  #define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
> +#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
>  
>  /* Device Control API on vcpu fd */
>  #define KVM_ARM_VCPU_PMU_V3_CTRL	0
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index 0c6798c..9d3932f 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -158,6 +158,7 @@
>  #define GICR_PROPBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PROPBASER, INNER, RaWaWb)
>  
>  #define GICR_PROPBASER_IDBITS_MASK			(0x1f)
> +#define GICR_PROPBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 12))
>  
>  #define GICR_PENDBASER_SHAREABILITY_SHIFT		(10)
>  #define GICR_PENDBASER_INNER_CACHEABILITY_SHIFT		(7)
> @@ -183,6 +184,7 @@
>  #define GICR_PENDBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PENDBASER, INNER, RaWaWb)
>  
>  #define GICR_PENDBASER_PTZ				BIT_ULL(62)
> +#define GICR_PENDBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 16))
>  
>  /*
>   * Re-Distributor registers, offsets from SGI_base
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index be848be..745c245 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -189,8 +189,6 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>   */
>  #define BASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
>  #define CBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
> -#define PENDBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
> -#define PROPBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
>  
>  #define GIC_LPI_OFFSET 8192
>  
> @@ -227,7 +225,7 @@ static struct its_collection *find_collection(struct vgic_its *its, int coll_id)
>  static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
>  			     struct kvm_vcpu *filter_vcpu)
>  {
> -	u64 propbase = PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
> +	u64 propbase = GICR_PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
>  	u8 prop;
>  	int ret;
>  
> @@ -339,7 +337,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
>   */
>  static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
>  {
> -	gpa_t pendbase = PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
> +	gpa_t pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
>  	struct vgic_irq *irq;
>  	int last_byte_offset = -1;
>  	int ret = 0;
> diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c
> index 859bfa8..d48743c 100644
> --- a/virt/kvm/arm/vgic/vgic-kvm-device.c
> +++ b/virt/kvm/arm/vgic/vgic-kvm-device.c
> @@ -580,6 +580,24 @@ static int vgic_v3_set_attr(struct kvm_device *dev,
>  		reg = tmp32;
>  		return vgic_v3_attr_regs_access(dev, attr, &reg, true);
>  	}
> +	case KVM_DEV_ARM_VGIC_GRP_CTRL: {
> +		int ret;
> +
> +		switch (attr->attr) {
> +		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
> +			mutex_lock(&dev->kvm->lock);
> +
> +			if (!lock_all_vcpus(dev->kvm)) {
> +				mutex_unlock(&dev->kvm->lock);
> +				return -EBUSY;
> +			}
> +			ret = vgic_v3_save_pending_tables(dev->kvm);
> +			unlock_all_vcpus(dev->kvm);
> +			mutex_unlock(&dev->kvm->lock);
> +			return ret;
> +		}
> +		break;
> +	}
>  	}
>  	return -ENXIO;
>  }
> @@ -658,6 +676,8 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
>  		switch (attr->attr) {
>  		case KVM_DEV_ARM_VGIC_CTRL_INIT:
>  			return 0;
> +		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
> +			return 0;
>  		}
>  	}
>  	return -ENXIO;
> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
> index be0f4c3..1f0977f 100644
> --- a/virt/kvm/arm/vgic/vgic-v3.c
> +++ b/virt/kvm/arm/vgic/vgic-v3.c
> @@ -15,6 +15,7 @@
>  #include <linux/irqchip/arm-gic-v3.h>
>  #include <linux/kvm.h>
>  #include <linux/kvm_host.h>
> +//#include <linux/bitops.h>
>  #include <kvm/arm_vgic.h>
>  #include <asm/kvm_mmu.h>
>  #include <asm/kvm_asm.h>
> @@ -252,6 +253,59 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu)
>  	vgic_v3->vgic_hcr = ICH_HCR_EN;
>  }
>  
> +/**
> + * vgic_its_save_pending_tables - Save the pending tables into guest RAM
> + * kvm lock and all vcpu lock must be held
> + */
> +int vgic_v3_save_pending_tables(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	int last_byte_offset = -1;
> +	struct vgic_irq *irq;
> +	int ret;
> +
> +	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
> +		int byte_offset, bit_nr;
> +		struct kvm_vcpu *vcpu;
> +		gpa_t pendbase, ptr;
> +		bool stored;
> +		u8 val;
> +
> +		vcpu = irq->target_vcpu;
> +		if (!vcpu)
> +			continue;
> +
> +		pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
> +
> +		byte_offset = irq->intid / BITS_PER_BYTE;
> +		bit_nr = irq->intid % BITS_PER_BYTE;
> +		ptr = pendbase + byte_offset;
> +
> +		if (byte_offset != last_byte_offset) {
> +			ret = kvm_read_guest(kvm, ptr, &val, 1);
> +			if (ret)
> +				return ret;
> +			last_byte_offset = byte_offset;
> +		}
> +
> +		stored = val & bit_nr;

didn't you mean 'stored = val & (1 << bit_nr)' ?

> +		if (stored == irq->pending_latch)
> +			continue;
> +
> +		if (irq->pending_latch)
> +			val |= 1 << bit_nr;
> +		else
> +			val &= ~(1 << bit_nr);
> +
> +		ret = kvm_write_guest(kvm, ptr, &val, 1);
> +		if (ret)
> +			return ret;
> +	}

This loop could probably be written a bit more efficiently and
simplicity by reading the memory one word at a time (and remembering to
do le64_to_cpu) and then doing something like:

		if (irq->pending_latch)
			old = __test_and_set_bit(bit_nr, &val);
		else
			old = __test_and_clear_bit(bit_nr, &val);

		if (old != val) {
			tmp = cpu_to_le64(val);
			ret = kvm_write_guest(kvm, ptr, &tmp,
					      sizeof(unsigned long));
			if (ret)
				retur ret;
		}

Further, you could also detect when the word_offset changes and write
back the entire word with all its changes then, but you'd also have to
check at the end of the loop then.  Not sure that's worth the
optimization or easier to read than what you heave.  It's up to you.

Thanks,
-Christoffer

> +
> +	return 0;
> +}
> +
> +
>  /* check for overlapping regions and for regions crossing the end of memory */
>  static bool vgic_v3_check_base(struct kvm *kvm)
>  {
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 9bc52ef..535c2fc 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -177,6 +177,7 @@ void vgic_v3_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>  void vgic_v3_enable(struct kvm_vcpu *vcpu);
>  int vgic_v3_probe(const struct gic_kvm_info *info);
>  int vgic_v3_map_resources(struct kvm *kvm);
> +int vgic_v3_save_pending_tables(struct kvm *kvm);
>  int vgic_register_redist_iodevs(struct kvm *kvm, gpa_t dist_base_address);
>  
>  int vgic_register_its_iodevs(struct kvm *kvm);
> -- 
> 2.5.5
> 

[PATCH v5 22/22] KVM: arm64: vgic-v3: KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:34PM +0200, Eric Auger wrote:
> This patch adds a new attribute to GICV3 KVM device
> KVM_DEV_ARM_VGIC_GRP_CTRL group. This Allows the userspace to

nit: allows (lowercase)
nit: s/the userspace/userspace/

> flush all GICR pending tables into guest RAM. At the moment
> we do not offer any restore control as the sync is implicit.

I would probably remove the last sentence here.

> 
> As we need the PENDBASER_ADDRESS() in vgic-v3, let's move its
> definition in the irqchip header. We restore the full length
> of the field, ie [51:16]. Same for PROPBASER_ADDRESS with full
> field length of [51:12].
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v4 -> v5:
> - move pending table save code/ctrl into VGICv3 instead of ITS
> - remove useless gpa_t cast
> - use the same optimization as in its_sync_lpi_pending_table
> 
> v3 -> v4:
> - remove the wrong comment about locking
> - pass kvm struct instead of its handle
> - add comment about restore method
> - remove GITR_PENDABASER.PTZ check
> - continue if target_vcpu == NULL
> - new locking strategy
> 
> v1 -> v2:
> - do not care about the 1st KB which should be zeroed according to
>   the spec.
> ---
>  arch/arm/include/uapi/asm/kvm.h     |  1 +
>  arch/arm64/include/uapi/asm/kvm.h   |  1 +
>  include/linux/irqchip/arm-gic-v3.h  |  2 ++
>  virt/kvm/arm/vgic/vgic-its.c        |  6 ++---
>  virt/kvm/arm/vgic/vgic-kvm-device.c | 20 ++++++++++++++
>  virt/kvm/arm/vgic/vgic-v3.c         | 54 +++++++++++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h            |  1 +
>  7 files changed, 81 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
> index 8e6563c..78fe803 100644
> --- a/arch/arm/include/uapi/asm/kvm.h
> +++ b/arch/arm/include/uapi/asm/kvm.h
> @@ -202,6 +202,7 @@ struct kvm_arch_memory_slot {
>  #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
>  #define   KVM_DEV_ARM_ITS_SAVE_TABLES		1
>  #define   KVM_DEV_ARM_ITS_RESTORE_TABLES	2
> +#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
>  
>  /* KVM_IRQ_LINE irq field index values */
>  #define KVM_ARM_IRQ_TYPE_SHIFT		24
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index 1e35115..8a3758a 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -222,6 +222,7 @@ struct kvm_arch_memory_slot {
>  #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
>  #define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
>  #define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
> +#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
>  
>  /* Device Control API on vcpu fd */
>  #define KVM_ARM_VCPU_PMU_V3_CTRL	0
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index 0c6798c..9d3932f 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -158,6 +158,7 @@
>  #define GICR_PROPBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PROPBASER, INNER, RaWaWb)
>  
>  #define GICR_PROPBASER_IDBITS_MASK			(0x1f)
> +#define GICR_PROPBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 12))
>  
>  #define GICR_PENDBASER_SHAREABILITY_SHIFT		(10)
>  #define GICR_PENDBASER_INNER_CACHEABILITY_SHIFT		(7)
> @@ -183,6 +184,7 @@
>  #define GICR_PENDBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PENDBASER, INNER, RaWaWb)
>  
>  #define GICR_PENDBASER_PTZ				BIT_ULL(62)
> +#define GICR_PENDBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 16))
>  
>  /*
>   * Re-Distributor registers, offsets from SGI_base
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index be848be..745c245 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -189,8 +189,6 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>   */
>  #define BASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
>  #define CBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
> -#define PENDBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
> -#define PROPBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
>  
>  #define GIC_LPI_OFFSET 8192
>  
> @@ -227,7 +225,7 @@ static struct its_collection *find_collection(struct vgic_its *its, int coll_id)
>  static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
>  			     struct kvm_vcpu *filter_vcpu)
>  {
> -	u64 propbase = PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
> +	u64 propbase = GICR_PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
>  	u8 prop;
>  	int ret;
>  
> @@ -339,7 +337,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
>   */
>  static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
>  {
> -	gpa_t pendbase = PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
> +	gpa_t pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
>  	struct vgic_irq *irq;
>  	int last_byte_offset = -1;
>  	int ret = 0;
> diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c
> index 859bfa8..d48743c 100644
> --- a/virt/kvm/arm/vgic/vgic-kvm-device.c
> +++ b/virt/kvm/arm/vgic/vgic-kvm-device.c
> @@ -580,6 +580,24 @@ static int vgic_v3_set_attr(struct kvm_device *dev,
>  		reg = tmp32;
>  		return vgic_v3_attr_regs_access(dev, attr, &reg, true);
>  	}
> +	case KVM_DEV_ARM_VGIC_GRP_CTRL: {
> +		int ret;
> +
> +		switch (attr->attr) {
> +		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
> +			mutex_lock(&dev->kvm->lock);
> +
> +			if (!lock_all_vcpus(dev->kvm)) {
> +				mutex_unlock(&dev->kvm->lock);
> +				return -EBUSY;
> +			}
> +			ret = vgic_v3_save_pending_tables(dev->kvm);
> +			unlock_all_vcpus(dev->kvm);
> +			mutex_unlock(&dev->kvm->lock);
> +			return ret;
> +		}
> +		break;
> +	}
>  	}
>  	return -ENXIO;
>  }
> @@ -658,6 +676,8 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
>  		switch (attr->attr) {
>  		case KVM_DEV_ARM_VGIC_CTRL_INIT:
>  			return 0;
> +		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
> +			return 0;
>  		}
>  	}
>  	return -ENXIO;
> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
> index be0f4c3..1f0977f 100644
> --- a/virt/kvm/arm/vgic/vgic-v3.c
> +++ b/virt/kvm/arm/vgic/vgic-v3.c
> @@ -15,6 +15,7 @@
>  #include <linux/irqchip/arm-gic-v3.h>
>  #include <linux/kvm.h>
>  #include <linux/kvm_host.h>
> +//#include <linux/bitops.h>
>  #include <kvm/arm_vgic.h>
>  #include <asm/kvm_mmu.h>
>  #include <asm/kvm_asm.h>
> @@ -252,6 +253,59 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu)
>  	vgic_v3->vgic_hcr = ICH_HCR_EN;
>  }
>  
> +/**
> + * vgic_its_save_pending_tables - Save the pending tables into guest RAM
> + * kvm lock and all vcpu lock must be held
> + */
> +int vgic_v3_save_pending_tables(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	int last_byte_offset = -1;
> +	struct vgic_irq *irq;
> +	int ret;
> +
> +	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
> +		int byte_offset, bit_nr;
> +		struct kvm_vcpu *vcpu;
> +		gpa_t pendbase, ptr;
> +		bool stored;
> +		u8 val;
> +
> +		vcpu = irq->target_vcpu;
> +		if (!vcpu)
> +			continue;
> +
> +		pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
> +
> +		byte_offset = irq->intid / BITS_PER_BYTE;
> +		bit_nr = irq->intid % BITS_PER_BYTE;
> +		ptr = pendbase + byte_offset;
> +
> +		if (byte_offset != last_byte_offset) {
> +			ret = kvm_read_guest(kvm, ptr, &val, 1);
> +			if (ret)
> +				return ret;
> +			last_byte_offset = byte_offset;
> +		}
> +
> +		stored = val & bit_nr;

didn't you mean 'stored = val & (1 << bit_nr)' ?

> +		if (stored == irq->pending_latch)
> +			continue;
> +
> +		if (irq->pending_latch)
> +			val |= 1 << bit_nr;
> +		else
> +			val &= ~(1 << bit_nr);
> +
> +		ret = kvm_write_guest(kvm, ptr, &val, 1);
> +		if (ret)
> +			return ret;
> +	}

This loop could probably be written a bit more efficiently and
simplicity by reading the memory one word at a time (and remembering to
do le64_to_cpu) and then doing something like:

		if (irq->pending_latch)
			old = __test_and_set_bit(bit_nr, &val);
		else
			old = __test_and_clear_bit(bit_nr, &val);

		if (old != val) {
			tmp = cpu_to_le64(val);
			ret = kvm_write_guest(kvm, ptr, &tmp,
					      sizeof(unsigned long));
			if (ret)
				retur ret;
		}

Further, you could also detect when the word_offset changes and write
back the entire word with all its changes then, but you'd also have to
check at the end of the loop then.  Not sure that's worth the
optimization or easier to read than what you heave.  It's up to you.

Thanks,
-Christoffer

> +
> +	return 0;
> +}
> +
> +
>  /* check for overlapping regions and for regions crossing the end of memory */
>  static bool vgic_v3_check_base(struct kvm *kvm)
>  {
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 9bc52ef..535c2fc 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -177,6 +177,7 @@ void vgic_v3_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>  void vgic_v3_enable(struct kvm_vcpu *vcpu);
>  int vgic_v3_probe(const struct gic_kvm_info *info);
>  int vgic_v3_map_resources(struct kvm *kvm);
> +int vgic_v3_save_pending_tables(struct kvm *kvm);
>  int vgic_register_redist_iodevs(struct kvm *kvm, gpa_t dist_base_address);
>  
>  int vgic_register_its_iodevs(struct kvm *kvm);
> -- 
> 2.5.5
> 

Re: [PATCH v5 18/22] KVM: arm64: vgic-its: vgic_its_check_id returns the entry's GPA

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:30PM +0200, Eric Auger wrote:
> As vgic_its_check_id() computes the device/collection entry's
> GPA, let's return it so that new callers can retrieve it easily.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>

Acked-by: Christoffer Dall <cdall@linaro.org>

> ---
> v3 -> v4:
> - check eaddr is not NULL to allow passing NULL eaddr parameter
>   to vgic_its_check_id
>
> v2: new
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 484e541..35b2ca1 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -645,7 +645,8 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>   * is actually valid (covered by a memslot and guest accessible).
>   * For this we have to read the respective first level entry.
>   */
> -static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
> +static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id,
> +      gpa_t *eaddr)
>  {
>   int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
>   int index;
> @@ -665,6 +666,8 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
>   addr = BASER_ADDRESS(baser) + id * esz;
>   gfn = addr >> PAGE_SHIFT;
>
> + if (eaddr)
> + *eaddr = addr;
>   return kvm_is_visible_gfn(its->dev->kvm, gfn);
>   }
>
> @@ -697,6 +700,8 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
>   indirect_ptr += index * esz;
>   gfn = indirect_ptr >> PAGE_SHIFT;
>
> + if (eaddr)
> + *eaddr = indirect_ptr;
>   return kvm_is_visible_gfn(its->dev->kvm, gfn);
>  }
>
> @@ -706,7 +711,7 @@ static int vgic_its_alloc_collection(struct vgic_its *its,
>  {
>   struct its_collection *collection;
>
> - if (!vgic_its_check_id(its, its->baser_coll_table, coll_id))
> + if (!vgic_its_check_id(its, its->baser_coll_table, coll_id, NULL))
>   return E_ITS_MAPC_COLLECTION_OOR;
>
>   collection = kzalloc(sizeof(*collection), GFP_KERNEL);
> @@ -889,7 +894,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>   gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
>   struct its_device *device;
>
> - if (!vgic_its_check_id(its, its->baser_device_table, device_id))
> + if (!vgic_its_check_id(its, its->baser_device_table, device_id, NULL))
>   return E_ITS_MAPD_DEVICE_OOR;
>
>   if (valid && nb_eventid_bits > VITS_TYPER_IDBITS)
> --
> 2.5.5
>

[PATCH v5 18/22] KVM: arm64: vgic-its: vgic_its_check_id returns the entry's GPA

[Christoffer Dall]

On Fri, Apr 14, 2017 at 12:15:30PM +0200, Eric Auger wrote:
> As vgic_its_check_id() computes the device/collection entry's
> GPA, let's return it so that new callers can retrieve it easily.
>
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>

Acked-by: Christoffer Dall <cdall@linaro.org>

> ---
> v3 -> v4:
> - check eaddr is not NULL to allow passing NULL eaddr parameter
>   to vgic_its_check_id
>
> v2: new
> ---
>  virt/kvm/arm/vgic/vgic-its.c | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> index 484e541..35b2ca1 100644
> --- a/virt/kvm/arm/vgic/vgic-its.c
> +++ b/virt/kvm/arm/vgic/vgic-its.c
> @@ -645,7 +645,8 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
>   * is actually valid (covered by a memslot and guest accessible).
>   * For this we have to read the respective first level entry.
>   */
> -static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
> +static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id,
> +      gpa_t *eaddr)
>  {
>   int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
>   int index;
> @@ -665,6 +666,8 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
>   addr = BASER_ADDRESS(baser) + id * esz;
>   gfn = addr >> PAGE_SHIFT;
>
> + if (eaddr)
> + *eaddr = addr;
>   return kvm_is_visible_gfn(its->dev->kvm, gfn);
>   }
>
> @@ -697,6 +700,8 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id)
>   indirect_ptr += index * esz;
>   gfn = indirect_ptr >> PAGE_SHIFT;
>
> + if (eaddr)
> + *eaddr = indirect_ptr;
>   return kvm_is_visible_gfn(its->dev->kvm, gfn);
>  }
>
> @@ -706,7 +711,7 @@ static int vgic_its_alloc_collection(struct vgic_its *its,
>  {
>   struct its_collection *collection;
>
> - if (!vgic_its_check_id(its, its->baser_coll_table, coll_id))
> + if (!vgic_its_check_id(its, its->baser_coll_table, coll_id, NULL))
>   return E_ITS_MAPC_COLLECTION_OOR;
>
>   collection = kzalloc(sizeof(*collection), GFP_KERNEL);
> @@ -889,7 +894,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
>   gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
>   struct its_device *device;
>
> - if (!vgic_its_check_id(its, its->baser_device_table, device_id))
> + if (!vgic_its_check_id(its, its->baser_device_table, device_id, NULL))
>   return E_ITS_MAPD_DEVICE_OOR;
>
>   if (valid && nb_eventid_bits > VITS_TYPER_IDBITS)
> --
> 2.5.5
>

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Auger Eric]

Hi Christoffer,

On 30/04/2017 21:35, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
>> Add a generic lookup_table() helper whose role consists in
>> scanning a contiguous table located in guest RAM and applying
>> a callback on each entry. Entries can be handled as linked lists
>> since the callback may return an offset to the next entry and
>> also tell that an entry is the last one.
>>
>> Helper functions also are added to compute the device/event ID
>> offset to the next DTE/ITE.
>>
>> compute_next_devid_offset, compute_next_eventid_offset and
>> lookup_table will become static in subsequent patches
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - use kvm_read_guest
>>
>> v3 -> v4:
>> - remove static to avoid compilation warning
>> - correct size computation in looup_table()
>> - defines now encode the number of bits used for devid and eventid offsets
>> - use BIT() - 1 to encode the max offets
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 93 insertions(+)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 56c5123..c22b35d 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  
>>  #define VITS_TYPER_IDBITS 16
>>  #define VITS_TYPER_DEVBITS 16
>> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
>> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>>  
>>  /*
>>   * Finds and returns a collection in the ITS collection table.
>> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>  	return ret;
>>  }
>>  
>> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>> +{
>> +	struct list_head *e = &dev->dev_list;
>> +	struct its_device *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_device, dev_list);
>> +	next_offset = next->device_id - dev->device_id;
>> +
>> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>> +}
>> +
>> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>> +{
>> +	struct list_head *e = &ite->ite_list;
>> +	struct its_ite *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_ite, ite_list);
>> +	next_offset = next->event_id - ite->event_id;
>> +
>> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
>> +}
>> +
>> +/**
>> + * entry_fn_t - Callback called on a table entry restore path
>> + * @its: its handle
>> + * @id: id of the entry
>> + * @entry: pointer to the entry
>> + * @opaque: pointer to an opaque data
>> + * @next_offset: minimal ID offset to the next entry. 0 if this
>> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> 
> eh, also, did you mean -1 if the entry is invalid?
no in case the entry is invalid, we tell the caller that it must inspect
the next entry, hence the next_offset = +1.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Auger Eric]

Hi Christoffer,

On 30/04/2017 21:35, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
>> Add a generic lookup_table() helper whose role consists in
>> scanning a contiguous table located in guest RAM and applying
>> a callback on each entry. Entries can be handled as linked lists
>> since the callback may return an offset to the next entry and
>> also tell that an entry is the last one.
>>
>> Helper functions also are added to compute the device/event ID
>> offset to the next DTE/ITE.
>>
>> compute_next_devid_offset, compute_next_eventid_offset and
>> lookup_table will become static in subsequent patches
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - use kvm_read_guest
>>
>> v3 -> v4:
>> - remove static to avoid compilation warning
>> - correct size computation in looup_table()
>> - defines now encode the number of bits used for devid and eventid offsets
>> - use BIT() - 1 to encode the max offets
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 93 insertions(+)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 56c5123..c22b35d 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  
>>  #define VITS_TYPER_IDBITS 16
>>  #define VITS_TYPER_DEVBITS 16
>> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
>> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>>  
>>  /*
>>   * Finds and returns a collection in the ITS collection table.
>> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>  	return ret;
>>  }
>>  
>> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>> +{
>> +	struct list_head *e = &dev->dev_list;
>> +	struct its_device *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_device, dev_list);
>> +	next_offset = next->device_id - dev->device_id;
>> +
>> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>> +}
>> +
>> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>> +{
>> +	struct list_head *e = &ite->ite_list;
>> +	struct its_ite *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_ite, ite_list);
>> +	next_offset = next->event_id - ite->event_id;
>> +
>> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
>> +}
>> +
>> +/**
>> + * entry_fn_t - Callback called on a table entry restore path
>> + * @its: its handle
>> + * @id: id of the entry
>> + * @entry: pointer to the entry
>> + * @opaque: pointer to an opaque data
>> + * @next_offset: minimal ID offset to the next entry. 0 if this
>> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> 
> eh, also, did you mean -1 if the entry is invalid?
no in case the entry is invalid, we tell the caller that it must inspect
the next entry, hence the next_offset = +1.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Wed, May 03, 2017 at 08:53:36AM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 30/04/2017 21:35, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> >> Add a generic lookup_table() helper whose role consists in
> >> scanning a contiguous table located in guest RAM and applying
> >> a callback on each entry. Entries can be handled as linked lists
> >> since the callback may return an offset to the next entry and
> >> also tell that an entry is the last one.
> >>
> >> Helper functions also are added to compute the device/event ID
> >> offset to the next DTE/ITE.
> >>
> >> compute_next_devid_offset, compute_next_eventid_offset and
> >> lookup_table will become static in subsequent patches
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - use kvm_read_guest
> >>
> >> v3 -> v4:
> >> - remove static to avoid compilation warning
> >> - correct size computation in looup_table()
> >> - defines now encode the number of bits used for devid and eventid offsets
> >> - use BIT() - 1 to encode the max offets
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 93 insertions(+)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 56c5123..c22b35d 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
> >>  
> >>  #define VITS_TYPER_IDBITS 16
> >>  #define VITS_TYPER_DEVBITS 16
> >> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> >> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
> >>  
> >>  /*
> >>   * Finds and returns a collection in the ITS collection table.
> >> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
> >>  	return ret;
> >>  }
> >>  
> >> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >> +{
> >> +	struct list_head *e = &dev->dev_list;
> >> +	struct its_device *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_device, dev_list);
> >> +	next_offset = next->device_id - dev->device_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >> +}
> >> +
> >> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >> +{
> >> +	struct list_head *e = &ite->ite_list;
> >> +	struct its_ite *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_ite, ite_list);
> >> +	next_offset = next->event_id - ite->event_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> >> +}
> >> +
> >> +/**
> >> + * entry_fn_t - Callback called on a table entry restore path
> >> + * @its: its handle
> >> + * @id: id of the entry
> >> + * @entry: pointer to the entry
> >> + * @opaque: pointer to an opaque data
> >> + * @next_offset: minimal ID offset to the next entry. 0 if this
> >> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> > 
> > eh, also, did you mean -1 if the entry is invalid?
> no in case the entry is invalid, we tell the caller that it must inspect
> the next entry, hence the next_offset = +1.

hmm, but you say aftterwards that >= 1 if an entry's next_offset field
was truly decoded, so this is confusing.  Perhaps it would make more
sense to get rid of the parameter entirely and change the return value
to say:

  Return: < 0 on error, 0 if the entry was the last one, and > 0 to
          indicate the offset to the next entry that must be processed
	  when scanning a table.

  Note that we return 1 for an invalid entry, because scanning a table
  in this case simply requires us looking at the next entry, but we may
  return >= to 1 if we found a valid entry and decoded the next field in
  the entry.

What do you think?

Thanks,
-Christoffer

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Wed, May 03, 2017 at 08:53:36AM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 30/04/2017 21:35, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> >> Add a generic lookup_table() helper whose role consists in
> >> scanning a contiguous table located in guest RAM and applying
> >> a callback on each entry. Entries can be handled as linked lists
> >> since the callback may return an offset to the next entry and
> >> also tell that an entry is the last one.
> >>
> >> Helper functions also are added to compute the device/event ID
> >> offset to the next DTE/ITE.
> >>
> >> compute_next_devid_offset, compute_next_eventid_offset and
> >> lookup_table will become static in subsequent patches
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - use kvm_read_guest
> >>
> >> v3 -> v4:
> >> - remove static to avoid compilation warning
> >> - correct size computation in looup_table()
> >> - defines now encode the number of bits used for devid and eventid offsets
> >> - use BIT() - 1 to encode the max offets
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 93 insertions(+)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 56c5123..c22b35d 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
> >>  
> >>  #define VITS_TYPER_IDBITS 16
> >>  #define VITS_TYPER_DEVBITS 16
> >> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> >> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
> >>  
> >>  /*
> >>   * Finds and returns a collection in the ITS collection table.
> >> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
> >>  	return ret;
> >>  }
> >>  
> >> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >> +{
> >> +	struct list_head *e = &dev->dev_list;
> >> +	struct its_device *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_device, dev_list);
> >> +	next_offset = next->device_id - dev->device_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >> +}
> >> +
> >> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >> +{
> >> +	struct list_head *e = &ite->ite_list;
> >> +	struct its_ite *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_ite, ite_list);
> >> +	next_offset = next->event_id - ite->event_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> >> +}
> >> +
> >> +/**
> >> + * entry_fn_t - Callback called on a table entry restore path
> >> + * @its: its handle
> >> + * @id: id of the entry
> >> + * @entry: pointer to the entry
> >> + * @opaque: pointer to an opaque data
> >> + * @next_offset: minimal ID offset to the next entry. 0 if this
> >> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> > 
> > eh, also, did you mean -1 if the entry is invalid?
> no in case the entry is invalid, we tell the caller that it must inspect
> the next entry, hence the next_offset = +1.

hmm, but you say aftterwards that >= 1 if an entry's next_offset field
was truly decoded, so this is confusing.  Perhaps it would make more
sense to get rid of the parameter entirely and change the return value
to say:

  Return: < 0 on error, 0 if the entry was the last one, and > 0 to
          indicate the offset to the next entry that must be processed
	  when scanning a table.

  Note that we return 1 for an invalid entry, because scanning a table
  in this case simply requires us looking at the next entry, but we may
  return >= to 1 if we found a valid entry and decoded the next field in
  the entry.

What do you think?

Thanks,
-Christoffer

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Auger Eric]

Hi Christoffer,

On 03/05/2017 10:01, Christoffer Dall wrote:
> On Wed, May 03, 2017 at 08:53:36AM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 30/04/2017 21:35, Christoffer Dall wrote:
>>> On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
>>>> Add a generic lookup_table() helper whose role consists in
>>>> scanning a contiguous table located in guest RAM and applying
>>>> a callback on each entry. Entries can be handled as linked lists
>>>> since the callback may return an offset to the next entry and
>>>> also tell that an entry is the last one.
>>>>
>>>> Helper functions also are added to compute the device/event ID
>>>> offset to the next DTE/ITE.
>>>>
>>>> compute_next_devid_offset, compute_next_eventid_offset and
>>>> lookup_table will become static in subsequent patches
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>> v4 -> v5:
>>>> - use kvm_read_guest
>>>>
>>>> v3 -> v4:
>>>> - remove static to avoid compilation warning
>>>> - correct size computation in looup_table()
>>>> - defines now encode the number of bits used for devid and eventid offsets
>>>> - use BIT() - 1 to encode the max offets
>>>> ---
>>>>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>>>>  1 file changed, 93 insertions(+)
>>>>
>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>> index 56c5123..c22b35d 100644
>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>>>  
>>>>  #define VITS_TYPER_IDBITS 16
>>>>  #define VITS_TYPER_DEVBITS 16
>>>> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
>>>> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>>>>  
>>>>  /*
>>>>   * Finds and returns a collection in the ITS collection table.
>>>> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>>>  	return ret;
>>>>  }
>>>>  
>>>> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>>> +{
>>>> +	struct list_head *e = &dev->dev_list;
>>>> +	struct its_device *next;
>>>> +	u32 next_offset;
>>>> +
>>>> +	if (e->next == h)
>>>> +		return 0;
>>>> +	next = list_entry(e->next, struct its_device, dev_list);
>>>> +	next_offset = next->device_id - dev->device_id;
>>>> +
>>>> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>>>> +}
>>>> +
>>>> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>> +{
>>>> +	struct list_head *e = &ite->ite_list;
>>>> +	struct its_ite *next;
>>>> +	u32 next_offset;
>>>> +
>>>> +	if (e->next == h)
>>>> +		return 0;
>>>> +	next = list_entry(e->next, struct its_ite, ite_list);
>>>> +	next_offset = next->event_id - ite->event_id;
>>>> +
>>>> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
>>>> +}
>>>> +
>>>> +/**
>>>> + * entry_fn_t - Callback called on a table entry restore path
>>>> + * @its: its handle
>>>> + * @id: id of the entry
>>>> + * @entry: pointer to the entry
>>>> + * @opaque: pointer to an opaque data
>>>> + * @next_offset: minimal ID offset to the next entry. 0 if this
>>>> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
>>>
>>> eh, also, did you mean -1 if the entry is invalid?
>> no in case the entry is invalid, we tell the caller that it must inspect
>> the next entry, hence the next_offset = +1.
> jjjjj
> hmm, but you say aftterwards that >= 1 if an entry's next_offset field
> was truly decoded, so this is confusing.  Perhaps it would make more
> sense to get rid of the parameter entirely and change the return value
> to say:
> 
>   Return: < 0 on error, 0 if the entry was the last one, and > 0 to
>           indicate the offset to the next entry that must be processed
> 	  when scanning a table.
> 
>   Note that we return 1 for an invalid entry, because scanning a table
>   in this case simply requires us looking at the next entry, but we may
>   return >= to 1 if we found a valid entry and decoded the next field in
>   the entry.
> 
> What do you think?
Yes I'm going to experiment that change.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Auger Eric]

Hi Christoffer,

On 03/05/2017 10:01, Christoffer Dall wrote:
> On Wed, May 03, 2017 at 08:53:36AM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 30/04/2017 21:35, Christoffer Dall wrote:
>>> On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
>>>> Add a generic lookup_table() helper whose role consists in
>>>> scanning a contiguous table located in guest RAM and applying
>>>> a callback on each entry. Entries can be handled as linked lists
>>>> since the callback may return an offset to the next entry and
>>>> also tell that an entry is the last one.
>>>>
>>>> Helper functions also are added to compute the device/event ID
>>>> offset to the next DTE/ITE.
>>>>
>>>> compute_next_devid_offset, compute_next_eventid_offset and
>>>> lookup_table will become static in subsequent patches
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>> v4 -> v5:
>>>> - use kvm_read_guest
>>>>
>>>> v3 -> v4:
>>>> - remove static to avoid compilation warning
>>>> - correct size computation in looup_table()
>>>> - defines now encode the number of bits used for devid and eventid offsets
>>>> - use BIT() - 1 to encode the max offets
>>>> ---
>>>>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>>>>  1 file changed, 93 insertions(+)
>>>>
>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>> index 56c5123..c22b35d 100644
>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>>>  
>>>>  #define VITS_TYPER_IDBITS 16
>>>>  #define VITS_TYPER_DEVBITS 16
>>>> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
>>>> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>>>>  
>>>>  /*
>>>>   * Finds and returns a collection in the ITS collection table.
>>>> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>>>  	return ret;
>>>>  }
>>>>  
>>>> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>>> +{
>>>> +	struct list_head *e = &dev->dev_list;
>>>> +	struct its_device *next;
>>>> +	u32 next_offset;
>>>> +
>>>> +	if (e->next == h)
>>>> +		return 0;
>>>> +	next = list_entry(e->next, struct its_device, dev_list);
>>>> +	next_offset = next->device_id - dev->device_id;
>>>> +
>>>> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>>>> +}
>>>> +
>>>> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>> +{
>>>> +	struct list_head *e = &ite->ite_list;
>>>> +	struct its_ite *next;
>>>> +	u32 next_offset;
>>>> +
>>>> +	if (e->next == h)
>>>> +		return 0;
>>>> +	next = list_entry(e->next, struct its_ite, ite_list);
>>>> +	next_offset = next->event_id - ite->event_id;
>>>> +
>>>> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
>>>> +}
>>>> +
>>>> +/**
>>>> + * entry_fn_t - Callback called on a table entry restore path
>>>> + * @its: its handle
>>>> + * @id: id of the entry
>>>> + * @entry: pointer to the entry
>>>> + * @opaque: pointer to an opaque data
>>>> + * @next_offset: minimal ID offset to the next entry. 0 if this
>>>> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
>>>
>>> eh, also, did you mean -1 if the entry is invalid?
>> no in case the entry is invalid, we tell the caller that it must inspect
>> the next entry, hence the next_offset = +1.
> jjjjj
> hmm, but you say aftterwards that >= 1 if an entry's next_offset field
> was truly decoded, so this is confusing.  Perhaps it would make more
> sense to get rid of the parameter entirely and change the return value
> to say:
> 
>   Return: < 0 on error, 0 if the entry was the last one, and > 0 to
>           indicate the offset to the next entry that must be processed
> 	  when scanning a table.
> 
>   Note that we return 1 for an invalid entry, because scanning a table
>   in this case simply requires us looking at the next entry, but we may
>   return >= to 1 if we found a valid entry and decoded the next field in
>   the entry.
> 
> What do you think?
Yes I'm going to experiment that change.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Auger Eric]

Hi Christoffer,

On 30/04/2017 21:33, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
>> Add a generic lookup_table() helper whose role consists in
>> scanning a contiguous table located in guest RAM and applying
>> a callback on each entry. Entries can be handled as linked lists
>> since the callback may return an offset to the next entry and
>> also tell that an entry is the last one.
>>
>> Helper functions also are added to compute the device/event ID
>> offset to the next DTE/ITE.
>>
>> compute_next_devid_offset, compute_next_eventid_offset and
>> lookup_table will become static in subsequent patches
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - use kvm_read_guest
>>
>> v3 -> v4:
>> - remove static to avoid compilation warning
>> - correct size computation in looup_table()
>> - defines now encode the number of bits used for devid and eventid offsets
>> - use BIT() - 1 to encode the max offets
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 93 insertions(+)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 56c5123..c22b35d 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  
>>  #define VITS_TYPER_IDBITS 16
>>  #define VITS_TYPER_DEVBITS 16
>> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
>> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>>  
>>  /*
>>   * Finds and returns a collection in the ITS collection table.
>> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>  	return ret;
>>  }
>>  
>> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>> +{
>> +	struct list_head *e = &dev->dev_list;
>> +	struct its_device *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_device, dev_list);
>> +	next_offset = next->device_id - dev->device_id;
>> +
>> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>> +}
>> +
>> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>> +{
>> +	struct list_head *e = &ite->ite_list;
>> +	struct its_ite *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_ite, ite_list);
>> +	next_offset = next->event_id - ite->event_id;
>> +
>> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
>> +}
>> +
>> +/**
>> + * entry_fn_t - Callback called on a table entry restore path
>> + * @its: its handle
>> + * @id: id of the entry
>> + * @entry: pointer to the entry
>> + * @opaque: pointer to an opaque data
>> + * @next_offset: minimal ID offset to the next entry. 0 if this
>> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
>> + * entry's next_offset field was truly decoded
>> + *
>> + * Return: < 0 on error, 0 otherwise
>> + */
>> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>> +			  void *opaque, u32 *next_offset);
> 
> Just noticed.  All the table entries are 64-bit long at this point,
> right?  So why not make entry a u64 * instead?  Could we end up with
> some endianness mess with using void pointers the way it is now?
the size of the entry is ABI dependent while this infrastructure is
generic. In each of such function we use

u64 entry = *(u64 *)addr;
and we do a le64_to_cpu(entry).

Do you see something wrong? Otherwise I would be tempted to leave as is.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
>> +
>> +/**
>> + * lookup_table - scan a contiguous table in guest RAM and applies a function
>> + * to each entry
>> + *
>> + * @its: its handle
>> + * @base: base gpa of the table
>> + * @size: size of the table in bytes
>> + * @esz: entry size in bytes
>> + * @start_id: first entry's ID
>> + * @fn: function to apply on each entry
>> + *
>> + * Return: < 0 on error, 1 if last element identified, 0 otherwise
>> + */
>> +int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>> +		 int start_id, entry_fn_t fn, void *opaque)
>> +{
>> +	void *entry = kzalloc(esz, GFP_KERNEL);
>> +	struct kvm *kvm = its->dev->kvm;
>> +	unsigned long len = size;
>> +	u32 id = start_id;
>> +	gpa_t gpa = base;
>> +	int ret;
>> +
>> +	while (len > 0) {
>> +		u32 next_offset;
>> +		size_t byte_offset;
>> +
>> +		ret = kvm_read_guest(kvm, gpa, entry, esz);
>> +		if (ret)
>> +			goto out;
>> +
>> +		ret = fn(its, id, entry, opaque, &next_offset);
>> +		if (ret < 0 || (!ret && !next_offset))
>> +			goto out;
>> +
>> +		byte_offset = next_offset * esz;
>> +		id += next_offset;
>> +		gpa += byte_offset;
>> +		len -= byte_offset;
>> +	}
>> +	kfree(entry);
>> +	return 0;
>> +
>> +out:
>> +	kfree(entry);
>> +	return (ret < 0 ? ret : 1);
>> +}
>> +
>>  /**
>>   * vgic_its_save_device_tables - Save the device table and all ITT
>>   * into guest RAM
>> -- 
>> 2.5.5
>>

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Auger Eric]

Hi Christoffer,

On 30/04/2017 21:33, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
>> Add a generic lookup_table() helper whose role consists in
>> scanning a contiguous table located in guest RAM and applying
>> a callback on each entry. Entries can be handled as linked lists
>> since the callback may return an offset to the next entry and
>> also tell that an entry is the last one.
>>
>> Helper functions also are added to compute the device/event ID
>> offset to the next DTE/ITE.
>>
>> compute_next_devid_offset, compute_next_eventid_offset and
>> lookup_table will become static in subsequent patches
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - use kvm_read_guest
>>
>> v3 -> v4:
>> - remove static to avoid compilation warning
>> - correct size computation in looup_table()
>> - defines now encode the number of bits used for devid and eventid offsets
>> - use BIT() - 1 to encode the max offets
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 93 insertions(+)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 56c5123..c22b35d 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>  
>>  #define VITS_TYPER_IDBITS 16
>>  #define VITS_TYPER_DEVBITS 16
>> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
>> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
>>  
>>  /*
>>   * Finds and returns a collection in the ITS collection table.
>> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>  	return ret;
>>  }
>>  
>> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>> +{
>> +	struct list_head *e = &dev->dev_list;
>> +	struct its_device *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_device, dev_list);
>> +	next_offset = next->device_id - dev->device_id;
>> +
>> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>> +}
>> +
>> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>> +{
>> +	struct list_head *e = &ite->ite_list;
>> +	struct its_ite *next;
>> +	u32 next_offset;
>> +
>> +	if (e->next == h)
>> +		return 0;
>> +	next = list_entry(e->next, struct its_ite, ite_list);
>> +	next_offset = next->event_id - ite->event_id;
>> +
>> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
>> +}
>> +
>> +/**
>> + * entry_fn_t - Callback called on a table entry restore path
>> + * @its: its handle
>> + * @id: id of the entry
>> + * @entry: pointer to the entry
>> + * @opaque: pointer to an opaque data
>> + * @next_offset: minimal ID offset to the next entry. 0 if this
>> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
>> + * entry's next_offset field was truly decoded
>> + *
>> + * Return: < 0 on error, 0 otherwise
>> + */
>> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>> +			  void *opaque, u32 *next_offset);
> 
> Just noticed.  All the table entries are 64-bit long at this point,
> right?  So why not make entry a u64 * instead?  Could we end up with
> some endianness mess with using void pointers the way it is now?
the size of the entry is ABI dependent while this infrastructure is
generic. In each of such function we use

u64 entry = *(u64 *)addr;
and we do a le64_to_cpu(entry).

Do you see something wrong? Otherwise I would be tempted to leave as is.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
>> +
>> +/**
>> + * lookup_table - scan a contiguous table in guest RAM and applies a function
>> + * to each entry
>> + *
>> + * @its: its handle
>> + * @base: base gpa of the table
>> + * @size: size of the table in bytes
>> + * @esz: entry size in bytes
>> + * @start_id: first entry's ID
>> + * @fn: function to apply on each entry
>> + *
>> + * Return: < 0 on error, 1 if last element identified, 0 otherwise
>> + */
>> +int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>> +		 int start_id, entry_fn_t fn, void *opaque)
>> +{
>> +	void *entry = kzalloc(esz, GFP_KERNEL);
>> +	struct kvm *kvm = its->dev->kvm;
>> +	unsigned long len = size;
>> +	u32 id = start_id;
>> +	gpa_t gpa = base;
>> +	int ret;
>> +
>> +	while (len > 0) {
>> +		u32 next_offset;
>> +		size_t byte_offset;
>> +
>> +		ret = kvm_read_guest(kvm, gpa, entry, esz);
>> +		if (ret)
>> +			goto out;
>> +
>> +		ret = fn(its, id, entry, opaque, &next_offset);
>> +		if (ret < 0 || (!ret && !next_offset))
>> +			goto out;
>> +
>> +		byte_offset = next_offset * esz;
>> +		id += next_offset;
>> +		gpa += byte_offset;
>> +		len -= byte_offset;
>> +	}
>> +	kfree(entry);
>> +	return 0;
>> +
>> +out:
>> +	kfree(entry);
>> +	return (ret < 0 ? ret : 1);
>> +}
>> +
>>  /**
>>   * vgic_its_save_device_tables - Save the device table and all ITT
>>   * into guest RAM
>> -- 
>> 2.5.5
>>

Re: [PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Auger Eric]

Hi Christoffer,
On 30/04/2017 22:55, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:32PM +0200, Eric Auger wrote:
>> This patch saves the device table entries into guest RAM.
>> Both flat table and 2 stage tables are supported. DeviceId
>> indexing is used.
>>
>> For each device listed in the device table, we also save
>> the translation table using the vgic_its_save/restore_itt
>> routines.
>>
>> On restore, devices are re-allocated and their ite are
>> re-built.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - sort the device list by deviceid on device table save
>> - use defines for shifts and masks
>> - use abi->dte_esz
>> - clatify entry sizes for L1 and L2 tables
>>
>> v3 -> v4:
>> - use the new proto for its_alloc_device
>> - compute_next_devid_offset, vgic_its_flush/restore_itt
>>   become static in this patch
>> - change in the DTE entry format with the introduction of the
>>   valid bit and next field width decrease; ittaddr encoded
>>   on its full range
>> - fix handle_l1_entry entry handling
>> - correct vgic_its_table_restore error handling
>>
>> v2 -> v3:
>> - fix itt_addr bitmask in vgic_its_restore_dte
>> - addition of return 0 in vgic_its_restore_ite moved to
>>   the ITE related patch
>>
>> v1 -> v2:
>> - use 8 byte format for DTE and ITE
>> - support 2 stage format
>> - remove kvm parameter
>> - ITT flush/restore moved in a separate patch
>> - use deviceid indexing
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 183 +++++++++++++++++++++++++++++++++++++++++--
>>  virt/kvm/arm/vgic/vgic.h     |   7 ++
>>  2 files changed, 185 insertions(+), 5 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index b02fc3f..86dfc6c 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -1682,7 +1682,8 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>  	return ret;
>>  }
>>  
>> -u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>> +static u32 compute_next_devid_offset(struct list_head *h,
>> +				     struct its_device *dev)
>>  {
>>  	struct list_head *e = &dev->dev_list;
>>  	struct its_device *next;
>> @@ -1858,7 +1859,7 @@ static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>>  		return 1;
>>  }
>>  
>> -int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>> +static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>  {
>>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>  	gpa_t base = device->itt_addr;
>> @@ -1877,7 +1878,7 @@ int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>  	return 0;
>>  }
>>  
>> -int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>> +static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>  {
>>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>  	gpa_t base = dev->itt_addr;
>> @@ -1895,12 +1896,161 @@ int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>  }
>>  
>>  /**
>> + * vgic_its_save_dte - Save a device table entry at a given GPA
>> + *
>> + * @its: ITS handle
>> + * @dev: ITS device
>> + * @ptr: GPA
>> + */
>> +static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
>> +			     gpa_t ptr, int dte_esz)
>> +{
>> +	struct kvm *kvm = its->dev->kvm;
>> +	u64 val, itt_addr_field;
>> +	int ret;
>> +	u32 next_offset;
>> +
>> +	itt_addr_field = dev->itt_addr >> 8;
>> +	next_offset = compute_next_devid_offset(&its->device_list, dev);
>> +	val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
>> +	       ((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
> 
> I think this implies that the next field in your ABI points to the next
> offset, regardless of whether or not this is in a a level 2 or lavel 1
> table.  See more comments on this below (I reviewed this patch from the
> bottom up).
Not sure I understand your comment.

Doc says:
 - next: equals to 0 if this entry is the last one; otherwise it
   corresponds to the deviceid offset to the next DTE, capped by
   2^14 -1.

This is independent on 1 or 2 levels as we sort the devices by
deviceid's and compute the delta between those id.
> 
> I have a feeling this wasn't tested with 2 level device tables.  Could
> that be true?
No this was tested with 1 & and 2 levels (I hacked the guest to force 2
levels). 1 test hole I have though is all my dte's currently belong to
the same 2d level page, ie. my deviceid are not scattered enough.
> 
>> +	       (itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
>> +		(dev->nb_eventid_bits - 1));
>> +	val = cpu_to_le64(val);
>> +	ret = kvm_write_guest(kvm, ptr, &val, dte_esz);
>> +	return ret;
>> +}
>> +
>> +/**
>> + * vgic_its_restore_dte - restore a device table entry
>> + *
>> + * @its: its handle
>> + * @id: device id the DTE corresponds to
>> + * @ptr: kernel VA where the 8 byte DTE is located
>> + * @opaque: unused
>> + * @next: offset to the next valid device id
>> + *
>> + * Return: < 0 on error, 0 otherwise
>> + */
>> +static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
>> +				void *ptr, void *opaque, u32 *next)
>> +{
>> +	struct its_device *dev;
>> +	gpa_t itt_addr;
>> +	u8 nb_eventid_bits;
>> +	u64 entry = *(u64 *)ptr;
>> +	bool valid;
>> +	int ret;
>> +
>> +	entry = le64_to_cpu(entry);
>> +
>> +	valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
>> +	nb_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
>> +	itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
>> +			>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
>> +	*next = 1;
>> +
>> +	if (!valid)
>> +		return 0;
>> +
>> +	/* dte entry is valid */
>> +	*next = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
>> +
>> +	ret = vgic_its_alloc_device(its, &dev, id,
>> +				    itt_addr, nb_eventid_bits);
>> +	if (ret)
>> +		return ret;
>> +	ret = vgic_its_restore_itt(its, dev);
>> +
>> +	return ret;
>> +}
>> +
>> +static int vgic_its_device_cmp(void *priv, struct list_head *a,
>> +			       struct list_head *b)
>> +{
>> +	struct its_device *deva = container_of(a, struct its_device, dev_list);
>> +	struct its_device *devb = container_of(b, struct its_device, dev_list);
>> +
>> +	if (deva->device_id < devb->device_id)
>> +		return -1;
>> +	else
>> +		return 1;
>> +}
>> +
>> +/**
>>   * vgic_its_save_device_tables - Save the device table and all ITT
>>   * into guest RAM
>> + *
>> + * L1/L2 handling is hidden by vgic_its_check_id() helper which directly
>> + * returns the GPA of the device entry
>>   */
>>  static int vgic_its_save_device_tables(struct vgic_its *its)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	struct its_device *dev;
>> +	int dte_esz = abi->dte_esz;
>> +	u64 baser;
>> +
>> +	baser = its->baser_device_table;
>> +
>> +	list_sort(NULL, &its->device_list, vgic_its_device_cmp);
>> +
>> +	list_for_each_entry(dev, &its->device_list, dev_list) {
>> +		int ret;
>> +		gpa_t eaddr;
>> +
>> +		if (!vgic_its_check_id(its, baser,
>> +				       dev->device_id, &eaddr))
>> +			return -EINVAL;
>> +
>> +		ret = vgic_its_save_itt(its, dev);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
>> +		if (ret)
>> +			return ret;
>> +	}
>> +	return 0;
>> +}
>> +
>> +/**
>> + * handle_l1_entry - callback used for L1 entries (2 stage case)
>> + *
>> + * @its: its handle
>> + * @id: id
> 
> IIUC, this is actually the index of the entry in the L1 table.  I think
> this should be clarified.
yep
> 
>> + * @addr: kernel VA
>> + * @opaque: unused
>> + * @next_offset: offset to the next L1 entry: 0 if the last element
>> + * was found, 1 otherwise
>> + */
>> +static int handle_l1_entry(struct vgic_its *its, u32 id, void *addr,
>> +			   void *opaque, u32 *next_offset)
> 
> nit: shouldn't this be called handle_l1_device_table_entry ?
renamed into handle_l1_dte
> 
>> +{
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	int l2_start_id = id * (SZ_64K / GITS_LVL1_ENTRY_SIZE);
> 
> Hmmm, is this not actually supposed to be (SZ_64K / abi->dte_esz) ?
no because 1st level entries have a fixed size of GITS_LVL1_ENTRY_SIZE bytes
> 
>> +	u64 entry = *(u64 *)addr;
>> +	int ret, ite_esz = abi->ite_esz;
> 
> Should this be ite_esz or dte_esz?

you're correct. dte_esz should be used.
> 
>> +	gpa_t gpa;
> 
> nit: put declarations with initialization on separate lines.
OK
> 
>> +
>> +	entry = le64_to_cpu(entry);
>> +	*next_offset = 1;
> 
> I think you could attach a comment here saying that level 1 tables have
> to be scanned entirely.
added. note we exit as soon as the last element is found when scanning
l2 tables.
> 
> But this also reminds me.  Does that mean that the next field in the DTE
> in your documented ABI format points to the next DTE within that level-2
> table, or does it point across to different level-2 tables?  I think
> this needs to be clarified in the ABI unless I'm missing something.
see above comment on next_index semantic. In the doc I talk about
deviceid offset and not of table index.

> 
>> +
>> +	if (!(entry & BIT_ULL(63)))
>> +		return 0;
>> +
>> +	gpa = entry & GENMASK_ULL(51, 16);
> 
> Can you define the bit fields for the level-1 entries as well please?
yep
> 
>> +
>> +	ret = lookup_table(its, gpa, SZ_64K, ite_esz,
>> +			   l2_start_id, vgic_its_restore_dte, NULL);
>> +
>> +	if (ret == 1) {
>> +		/* last entry was found in this L2 table */
> 
> maybe you should define these return codes for you table scan function,
> and you wouldn't have to have a separate comment and it would be
> generally easier to follow the code.
I revisited the error values according to your suggestion and this looks
simpler to me now.
> 
>> +		*next_offset = 0;
>> +		ret = 0;
>> +	}
>> +
>> +	return ret;
>>  }
>>  
>>  /**
>> @@ -1909,7 +2059,30 @@ static int vgic_its_save_device_tables(struct vgic_its *its)
>>   */
>>  static int vgic_its_restore_device_tables(struct vgic_its *its)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	u64 baser = its->baser_device_table;
>> +	int l1_esz, ret, l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
> 
> nit: put this initialization on a separate line.
done
> 
>> +	gpa_t l1_gpa;
>> +
>> +	l1_gpa = BASER_ADDRESS(baser);
>> +	if (!l1_gpa)
>> +		return 0;
> 
> I think you meant to check the valid bit here.
yes
> 
>> +
>> +	if (baser & GITS_BASER_INDIRECT) {
>> +		l1_esz = 8;
>> +		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
>> +				   handle_l1_entry, NULL);
>> +	} else {
>> +		l1_esz = abi->dte_esz;
>> +		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
>> +				   vgic_its_restore_dte, NULL);
>> +	}
>> +
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* if last element was not found we have an issue here */
> 
> same comment as other patch
> 
>> +	return ret ? 0 : -EINVAL;
revisited

Thanks

Eric
>>  }
>>  
>>  static int vgic_its_save_cte(struct vgic_its *its,
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index ce57fbd..9bc52ef 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -85,6 +85,13 @@
>>  #define KVM_ITS_ITE_PINTID_SHIFT	16
>>  #define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
>>  #define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
>> +#define KVM_ITS_DTE_VALID_SHIFT		63
>> +#define KVM_ITS_DTE_VALID_MASK		BIT_ULL(63)
>> +#define KVM_ITS_DTE_NEXT_SHIFT		49
>> +#define KVM_ITS_DTE_NEXT_MASK		GENMASK_ULL(62, 49)
>> +#define KVM_ITS_DTE_ITTADDR_SHIFT	5
>> +#define KVM_ITS_DTE_ITTADDR_MASK	GENMASK_ULL(48, 5)
>> +#define KVM_ITS_DTE_SIZE_MASK		GENMASK_ULL(4, 0)
>>  
>>  static inline bool irq_is_pending(struct vgic_irq *irq)
>>  {
>> -- 
>> 2.5.5
>>
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Auger Eric]

Hi Christoffer,
On 30/04/2017 22:55, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:32PM +0200, Eric Auger wrote:
>> This patch saves the device table entries into guest RAM.
>> Both flat table and 2 stage tables are supported. DeviceId
>> indexing is used.
>>
>> For each device listed in the device table, we also save
>> the translation table using the vgic_its_save/restore_itt
>> routines.
>>
>> On restore, devices are re-allocated and their ite are
>> re-built.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - sort the device list by deviceid on device table save
>> - use defines for shifts and masks
>> - use abi->dte_esz
>> - clatify entry sizes for L1 and L2 tables
>>
>> v3 -> v4:
>> - use the new proto for its_alloc_device
>> - compute_next_devid_offset, vgic_its_flush/restore_itt
>>   become static in this patch
>> - change in the DTE entry format with the introduction of the
>>   valid bit and next field width decrease; ittaddr encoded
>>   on its full range
>> - fix handle_l1_entry entry handling
>> - correct vgic_its_table_restore error handling
>>
>> v2 -> v3:
>> - fix itt_addr bitmask in vgic_its_restore_dte
>> - addition of return 0 in vgic_its_restore_ite moved to
>>   the ITE related patch
>>
>> v1 -> v2:
>> - use 8 byte format for DTE and ITE
>> - support 2 stage format
>> - remove kvm parameter
>> - ITT flush/restore moved in a separate patch
>> - use deviceid indexing
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 183 +++++++++++++++++++++++++++++++++++++++++--
>>  virt/kvm/arm/vgic/vgic.h     |   7 ++
>>  2 files changed, 185 insertions(+), 5 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index b02fc3f..86dfc6c 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -1682,7 +1682,8 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>  	return ret;
>>  }
>>  
>> -u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>> +static u32 compute_next_devid_offset(struct list_head *h,
>> +				     struct its_device *dev)
>>  {
>>  	struct list_head *e = &dev->dev_list;
>>  	struct its_device *next;
>> @@ -1858,7 +1859,7 @@ static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>>  		return 1;
>>  }
>>  
>> -int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>> +static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>  {
>>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>  	gpa_t base = device->itt_addr;
>> @@ -1877,7 +1878,7 @@ int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>  	return 0;
>>  }
>>  
>> -int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>> +static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>  {
>>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>  	gpa_t base = dev->itt_addr;
>> @@ -1895,12 +1896,161 @@ int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>  }
>>  
>>  /**
>> + * vgic_its_save_dte - Save a device table entry at a given GPA
>> + *
>> + * @its: ITS handle
>> + * @dev: ITS device
>> + * @ptr: GPA
>> + */
>> +static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
>> +			     gpa_t ptr, int dte_esz)
>> +{
>> +	struct kvm *kvm = its->dev->kvm;
>> +	u64 val, itt_addr_field;
>> +	int ret;
>> +	u32 next_offset;
>> +
>> +	itt_addr_field = dev->itt_addr >> 8;
>> +	next_offset = compute_next_devid_offset(&its->device_list, dev);
>> +	val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
>> +	       ((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
> 
> I think this implies that the next field in your ABI points to the next
> offset, regardless of whether or not this is in a a level 2 or lavel 1
> table.  See more comments on this below (I reviewed this patch from the
> bottom up).
Not sure I understand your comment.

Doc says:
 - next: equals to 0 if this entry is the last one; otherwise it
   corresponds to the deviceid offset to the next DTE, capped by
   2^14 -1.

This is independent on 1 or 2 levels as we sort the devices by
deviceid's and compute the delta between those id.
> 
> I have a feeling this wasn't tested with 2 level device tables.  Could
> that be true?
No this was tested with 1 & and 2 levels (I hacked the guest to force 2
levels). 1 test hole I have though is all my dte's currently belong to
the same 2d level page, ie. my deviceid are not scattered enough.
> 
>> +	       (itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
>> +		(dev->nb_eventid_bits - 1));
>> +	val = cpu_to_le64(val);
>> +	ret = kvm_write_guest(kvm, ptr, &val, dte_esz);
>> +	return ret;
>> +}
>> +
>> +/**
>> + * vgic_its_restore_dte - restore a device table entry
>> + *
>> + * @its: its handle
>> + * @id: device id the DTE corresponds to
>> + * @ptr: kernel VA where the 8 byte DTE is located
>> + * @opaque: unused
>> + * @next: offset to the next valid device id
>> + *
>> + * Return: < 0 on error, 0 otherwise
>> + */
>> +static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
>> +				void *ptr, void *opaque, u32 *next)
>> +{
>> +	struct its_device *dev;
>> +	gpa_t itt_addr;
>> +	u8 nb_eventid_bits;
>> +	u64 entry = *(u64 *)ptr;
>> +	bool valid;
>> +	int ret;
>> +
>> +	entry = le64_to_cpu(entry);
>> +
>> +	valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
>> +	nb_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
>> +	itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
>> +			>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
>> +	*next = 1;
>> +
>> +	if (!valid)
>> +		return 0;
>> +
>> +	/* dte entry is valid */
>> +	*next = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
>> +
>> +	ret = vgic_its_alloc_device(its, &dev, id,
>> +				    itt_addr, nb_eventid_bits);
>> +	if (ret)
>> +		return ret;
>> +	ret = vgic_its_restore_itt(its, dev);
>> +
>> +	return ret;
>> +}
>> +
>> +static int vgic_its_device_cmp(void *priv, struct list_head *a,
>> +			       struct list_head *b)
>> +{
>> +	struct its_device *deva = container_of(a, struct its_device, dev_list);
>> +	struct its_device *devb = container_of(b, struct its_device, dev_list);
>> +
>> +	if (deva->device_id < devb->device_id)
>> +		return -1;
>> +	else
>> +		return 1;
>> +}
>> +
>> +/**
>>   * vgic_its_save_device_tables - Save the device table and all ITT
>>   * into guest RAM
>> + *
>> + * L1/L2 handling is hidden by vgic_its_check_id() helper which directly
>> + * returns the GPA of the device entry
>>   */
>>  static int vgic_its_save_device_tables(struct vgic_its *its)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	struct its_device *dev;
>> +	int dte_esz = abi->dte_esz;
>> +	u64 baser;
>> +
>> +	baser = its->baser_device_table;
>> +
>> +	list_sort(NULL, &its->device_list, vgic_its_device_cmp);
>> +
>> +	list_for_each_entry(dev, &its->device_list, dev_list) {
>> +		int ret;
>> +		gpa_t eaddr;
>> +
>> +		if (!vgic_its_check_id(its, baser,
>> +				       dev->device_id, &eaddr))
>> +			return -EINVAL;
>> +
>> +		ret = vgic_its_save_itt(its, dev);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
>> +		if (ret)
>> +			return ret;
>> +	}
>> +	return 0;
>> +}
>> +
>> +/**
>> + * handle_l1_entry - callback used for L1 entries (2 stage case)
>> + *
>> + * @its: its handle
>> + * @id: id
> 
> IIUC, this is actually the index of the entry in the L1 table.  I think
> this should be clarified.
yep
> 
>> + * @addr: kernel VA
>> + * @opaque: unused
>> + * @next_offset: offset to the next L1 entry: 0 if the last element
>> + * was found, 1 otherwise
>> + */
>> +static int handle_l1_entry(struct vgic_its *its, u32 id, void *addr,
>> +			   void *opaque, u32 *next_offset)
> 
> nit: shouldn't this be called handle_l1_device_table_entry ?
renamed into handle_l1_dte
> 
>> +{
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	int l2_start_id = id * (SZ_64K / GITS_LVL1_ENTRY_SIZE);
> 
> Hmmm, is this not actually supposed to be (SZ_64K / abi->dte_esz) ?
no because 1st level entries have a fixed size of GITS_LVL1_ENTRY_SIZE bytes
> 
>> +	u64 entry = *(u64 *)addr;
>> +	int ret, ite_esz = abi->ite_esz;
> 
> Should this be ite_esz or dte_esz?

you're correct. dte_esz should be used.
> 
>> +	gpa_t gpa;
> 
> nit: put declarations with initialization on separate lines.
OK
> 
>> +
>> +	entry = le64_to_cpu(entry);
>> +	*next_offset = 1;
> 
> I think you could attach a comment here saying that level 1 tables have
> to be scanned entirely.
added. note we exit as soon as the last element is found when scanning
l2 tables.
> 
> But this also reminds me.  Does that mean that the next field in the DTE
> in your documented ABI format points to the next DTE within that level-2
> table, or does it point across to different level-2 tables?  I think
> this needs to be clarified in the ABI unless I'm missing something.
see above comment on next_index semantic. In the doc I talk about
deviceid offset and not of table index.

> 
>> +
>> +	if (!(entry & BIT_ULL(63)))
>> +		return 0;
>> +
>> +	gpa = entry & GENMASK_ULL(51, 16);
> 
> Can you define the bit fields for the level-1 entries as well please?
yep
> 
>> +
>> +	ret = lookup_table(its, gpa, SZ_64K, ite_esz,
>> +			   l2_start_id, vgic_its_restore_dte, NULL);
>> +
>> +	if (ret == 1) {
>> +		/* last entry was found in this L2 table */
> 
> maybe you should define these return codes for you table scan function,
> and you wouldn't have to have a separate comment and it would be
> generally easier to follow the code.
I revisited the error values according to your suggestion and this looks
simpler to me now.
> 
>> +		*next_offset = 0;
>> +		ret = 0;
>> +	}
>> +
>> +	return ret;
>>  }
>>  
>>  /**
>> @@ -1909,7 +2059,30 @@ static int vgic_its_save_device_tables(struct vgic_its *its)
>>   */
>>  static int vgic_its_restore_device_tables(struct vgic_its *its)
>>  {
>> -	return -ENXIO;
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	u64 baser = its->baser_device_table;
>> +	int l1_esz, ret, l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
> 
> nit: put this initialization on a separate line.
done
> 
>> +	gpa_t l1_gpa;
>> +
>> +	l1_gpa = BASER_ADDRESS(baser);
>> +	if (!l1_gpa)
>> +		return 0;
> 
> I think you meant to check the valid bit here.
yes
> 
>> +
>> +	if (baser & GITS_BASER_INDIRECT) {
>> +		l1_esz = 8;
>> +		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
>> +				   handle_l1_entry, NULL);
>> +	} else {
>> +		l1_esz = abi->dte_esz;
>> +		ret = lookup_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
>> +				   vgic_its_restore_dte, NULL);
>> +	}
>> +
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* if last element was not found we have an issue here */
> 
> same comment as other patch
> 
>> +	return ret ? 0 : -EINVAL;
revisited

Thanks

Eric
>>  }
>>  
>>  static int vgic_its_save_cte(struct vgic_its *its,
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index ce57fbd..9bc52ef 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -85,6 +85,13 @@
>>  #define KVM_ITS_ITE_PINTID_SHIFT	16
>>  #define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
>>  #define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
>> +#define KVM_ITS_DTE_VALID_SHIFT		63
>> +#define KVM_ITS_DTE_VALID_MASK		BIT_ULL(63)
>> +#define KVM_ITS_DTE_NEXT_SHIFT		49
>> +#define KVM_ITS_DTE_NEXT_MASK		GENMASK_ULL(62, 49)
>> +#define KVM_ITS_DTE_ITTADDR_SHIFT	5
>> +#define KVM_ITS_DTE_ITTADDR_MASK	GENMASK_ULL(48, 5)
>> +#define KVM_ITS_DTE_SIZE_MASK		GENMASK_ULL(4, 0)
>>  
>>  static inline bool irq_is_pending(struct vgic_irq *irq)
>>  {
>> -- 
>> 2.5.5
>>
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Wed, May 03, 2017 at 03:40:34PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 30/04/2017 21:33, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> >> Add a generic lookup_table() helper whose role consists in
> >> scanning a contiguous table located in guest RAM and applying
> >> a callback on each entry. Entries can be handled as linked lists
> >> since the callback may return an offset to the next entry and
> >> also tell that an entry is the last one.
> >>
> >> Helper functions also are added to compute the device/event ID
> >> offset to the next DTE/ITE.
> >>
> >> compute_next_devid_offset, compute_next_eventid_offset and
> >> lookup_table will become static in subsequent patches
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - use kvm_read_guest
> >>
> >> v3 -> v4:
> >> - remove static to avoid compilation warning
> >> - correct size computation in looup_table()
> >> - defines now encode the number of bits used for devid and eventid offsets
> >> - use BIT() - 1 to encode the max offets
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 93 insertions(+)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 56c5123..c22b35d 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
> >>  
> >>  #define VITS_TYPER_IDBITS 16
> >>  #define VITS_TYPER_DEVBITS 16
> >> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> >> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
> >>  
> >>  /*
> >>   * Finds and returns a collection in the ITS collection table.
> >> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
> >>  	return ret;
> >>  }
> >>  
> >> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >> +{
> >> +	struct list_head *e = &dev->dev_list;
> >> +	struct its_device *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_device, dev_list);
> >> +	next_offset = next->device_id - dev->device_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >> +}
> >> +
> >> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >> +{
> >> +	struct list_head *e = &ite->ite_list;
> >> +	struct its_ite *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_ite, ite_list);
> >> +	next_offset = next->event_id - ite->event_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> >> +}
> >> +
> >> +/**
> >> + * entry_fn_t - Callback called on a table entry restore path
> >> + * @its: its handle
> >> + * @id: id of the entry
> >> + * @entry: pointer to the entry
> >> + * @opaque: pointer to an opaque data
> >> + * @next_offset: minimal ID offset to the next entry. 0 if this
> >> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> >> + * entry's next_offset field was truly decoded
> >> + *
> >> + * Return: < 0 on error, 0 otherwise
> >> + */
> >> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> >> +			  void *opaque, u32 *next_offset);
> > 
> > Just noticed.  All the table entries are 64-bit long at this point,
> > right?  So why not make entry a u64 * instead?  Could we end up with
> > some endianness mess with using void pointers the way it is now?
> the size of the entry is ABI dependent while this infrastructure is
> generic. 

Yes, for a single version of the ABI where all the entries are 64-bit.


> In each of such function we use
> 
> u64 entry = *(u64 *)addr;
> and we do a le64_to_cpu(entry).
> 
> Do you see something wrong? Otherwise I would be tempted to leave as is.
> 

I don't think there's anything wrong with the current version, and
you're right, this always points to an ITS data structure which is LE,
so there shouldn't be a problem.  I always just quiver when I see void
pointers cast to a type in the caller and callee.

Just leave it for now.

Thanks,
-Christoffer

[PATCH v5 16/22] KVM: arm64: vgic-its: Add infrastructure for table lookup

[Christoffer Dall]

On Wed, May 03, 2017 at 03:40:34PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 30/04/2017 21:33, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:28PM +0200, Eric Auger wrote:
> >> Add a generic lookup_table() helper whose role consists in
> >> scanning a contiguous table located in guest RAM and applying
> >> a callback on each entry. Entries can be handled as linked lists
> >> since the callback may return an offset to the next entry and
> >> also tell that an entry is the last one.
> >>
> >> Helper functions also are added to compute the device/event ID
> >> offset to the next DTE/ITE.
> >>
> >> compute_next_devid_offset, compute_next_eventid_offset and
> >> lookup_table will become static in subsequent patches
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - use kvm_read_guest
> >>
> >> v3 -> v4:
> >> - remove static to avoid compilation warning
> >> - correct size computation in looup_table()
> >> - defines now encode the number of bits used for devid and eventid offsets
> >> - use BIT() - 1 to encode the max offets
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 93 ++++++++++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 93 insertions(+)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 56c5123..c22b35d 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -195,6 +195,8 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
> >>  
> >>  #define VITS_TYPER_IDBITS 16
> >>  #define VITS_TYPER_DEVBITS 16
> >> +#define VITS_DTE_MAX_DEVID_OFFSET	(BIT(14) - 1)
> >> +#define VITS_ITE_MAX_EVENTID_OFFSET	(BIT(16) - 1)
> >>  
> >>  /*
> >>   * Finds and returns a collection in the ITS collection table.
> >> @@ -1674,6 +1676,97 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
> >>  	return ret;
> >>  }
> >>  
> >> +u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >> +{
> >> +	struct list_head *e = &dev->dev_list;
> >> +	struct its_device *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_device, dev_list);
> >> +	next_offset = next->device_id - dev->device_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >> +}
> >> +
> >> +u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >> +{
> >> +	struct list_head *e = &ite->ite_list;
> >> +	struct its_ite *next;
> >> +	u32 next_offset;
> >> +
> >> +	if (e->next == h)
> >> +		return 0;
> >> +	next = list_entry(e->next, struct its_ite, ite_list);
> >> +	next_offset = next->event_id - ite->event_id;
> >> +
> >> +	return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
> >> +}
> >> +
> >> +/**
> >> + * entry_fn_t - Callback called on a table entry restore path
> >> + * @its: its handle
> >> + * @id: id of the entry
> >> + * @entry: pointer to the entry
> >> + * @opaque: pointer to an opaque data
> >> + * @next_offset: minimal ID offset to the next entry. 0 if this
> >> + * entry is the last one, 1 if the entry is invalid, >= 1 if an
> >> + * entry's next_offset field was truly decoded
> >> + *
> >> + * Return: < 0 on error, 0 otherwise
> >> + */
> >> +typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> >> +			  void *opaque, u32 *next_offset);
> > 
> > Just noticed.  All the table entries are 64-bit long at this point,
> > right?  So why not make entry a u64 * instead?  Could we end up with
> > some endianness mess with using void pointers the way it is now?
> the size of the entry is ABI dependent while this infrastructure is
> generic. 

Yes, for a single version of the ABI where all the entries are 64-bit.


> In each of such function we use
> 
> u64 entry = *(u64 *)addr;
> and we do a le64_to_cpu(entry).
> 
> Do you see something wrong? Otherwise I would be tempted to leave as is.
> 

I don't think there's anything wrong with the current version, and
you're right, this always points to an ITS data structure which is LE,
so there shouldn't be a problem.  I always just quiver when I see void
pointers cast to a type in the caller and callee.

Just leave it for now.

Thanks,
-Christoffer

Re: [PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Christoffer Dall]

On Wed, May 03, 2017 at 04:07:45PM +0200, Auger Eric wrote:
> Hi Christoffer,
> On 30/04/2017 22:55, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:32PM +0200, Eric Auger wrote:
> >> This patch saves the device table entries into guest RAM.
> >> Both flat table and 2 stage tables are supported. DeviceId
> >> indexing is used.
> >>
> >> For each device listed in the device table, we also save
> >> the translation table using the vgic_its_save/restore_itt
> >> routines.
> >>
> >> On restore, devices are re-allocated and their ite are
> >> re-built.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - sort the device list by deviceid on device table save
> >> - use defines for shifts and masks
> >> - use abi->dte_esz
> >> - clatify entry sizes for L1 and L2 tables
> >>
> >> v3 -> v4:
> >> - use the new proto for its_alloc_device
> >> - compute_next_devid_offset, vgic_its_flush/restore_itt
> >>   become static in this patch
> >> - change in the DTE entry format with the introduction of the
> >>   valid bit and next field width decrease; ittaddr encoded
> >>   on its full range
> >> - fix handle_l1_entry entry handling
> >> - correct vgic_its_table_restore error handling
> >>
> >> v2 -> v3:
> >> - fix itt_addr bitmask in vgic_its_restore_dte
> >> - addition of return 0 in vgic_its_restore_ite moved to
> >>   the ITE related patch
> >>
> >> v1 -> v2:
> >> - use 8 byte format for DTE and ITE
> >> - support 2 stage format
> >> - remove kvm parameter
> >> - ITT flush/restore moved in a separate patch
> >> - use deviceid indexing
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 183 +++++++++++++++++++++++++++++++++++++++++--
> >>  virt/kvm/arm/vgic/vgic.h     |   7 ++
> >>  2 files changed, 185 insertions(+), 5 deletions(-)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index b02fc3f..86dfc6c 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -1682,7 +1682,8 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
> >>  	return ret;
> >>  }
> >>  
> >> -u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >> +static u32 compute_next_devid_offset(struct list_head *h,
> >> +				     struct its_device *dev)
> >>  {
> >>  	struct list_head *e = &dev->dev_list;
> >>  	struct its_device *next;
> >> @@ -1858,7 +1859,7 @@ static int vgic_its_ite_cmp(void *priv, struct list_head *a,
> >>  		return 1;
> >>  }
> >>  
> >> -int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >> +static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >>  {
> >>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>  	gpa_t base = device->itt_addr;
> >> @@ -1877,7 +1878,7 @@ int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >>  	return 0;
> >>  }
> >>  
> >> -int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >> +static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >>  {
> >>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>  	gpa_t base = dev->itt_addr;
> >> @@ -1895,12 +1896,161 @@ int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >>  }
> >>  
> >>  /**
> >> + * vgic_its_save_dte - Save a device table entry at a given GPA
> >> + *
> >> + * @its: ITS handle
> >> + * @dev: ITS device
> >> + * @ptr: GPA
> >> + */
> >> +static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
> >> +			     gpa_t ptr, int dte_esz)
> >> +{
> >> +	struct kvm *kvm = its->dev->kvm;
> >> +	u64 val, itt_addr_field;
> >> +	int ret;
> >> +	u32 next_offset;
> >> +
> >> +	itt_addr_field = dev->itt_addr >> 8;
> >> +	next_offset = compute_next_devid_offset(&its->device_list, dev);
> >> +	val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
> >> +	       ((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
> > 
> > I think this implies that the next field in your ABI points to the next
> > offset, regardless of whether or not this is in a a level 2 or lavel 1
> > table.  See more comments on this below (I reviewed this patch from the
> > bottom up).
> Not sure I understand your comment.
> 
> Doc says:
>  - next: equals to 0 if this entry is the last one; otherwise it
>    corresponds to the deviceid offset to the next DTE, capped by
>    2^14 -1.
> 
> This is independent on 1 or 2 levels as we sort the devices by
> deviceid's and compute the delta between those id.

see below.

> > 
> > I have a feeling this wasn't tested with 2 level device tables.  Could
> > that be true?
> No this was tested with 1 & and 2 levels (I hacked the guest to force 2
> levels). 1 test hole I have though is all my dte's currently belong to
> the same 2d level page, ie. my deviceid are not scattered enough.
> > 
> >> +	       (itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
> >> +		(dev->nb_eventid_bits - 1));
> >> +	val = cpu_to_le64(val);
> >> +	ret = kvm_write_guest(kvm, ptr, &val, dte_esz);
> >> +	return ret;
> >> +}
> >> +
> >> +/**
> >> + * vgic_its_restore_dte - restore a device table entry
> >> + *
> >> + * @its: its handle
> >> + * @id: device id the DTE corresponds to
> >> + * @ptr: kernel VA where the 8 byte DTE is located
> >> + * @opaque: unused
> >> + * @next: offset to the next valid device id
> >> + *
> >> + * Return: < 0 on error, 0 otherwise
> >> + */
> >> +static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
> >> +				void *ptr, void *opaque, u32 *next)
> >> +{
> >> +	struct its_device *dev;
> >> +	gpa_t itt_addr;
> >> +	u8 nb_eventid_bits;
> >> +	u64 entry = *(u64 *)ptr;
> >> +	bool valid;
> >> +	int ret;
> >> +
> >> +	entry = le64_to_cpu(entry);
> >> +
> >> +	valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
> >> +	nb_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
> >> +	itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
> >> +			>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
> >> +	*next = 1;
> >> +
> >> +	if (!valid)
> >> +		return 0;
> >> +
> >> +	/* dte entry is valid */
> >> +	*next = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
> >> +
> >> +	ret = vgic_its_alloc_device(its, &dev, id,
> >> +				    itt_addr, nb_eventid_bits);
> >> +	if (ret)
> >> +		return ret;
> >> +	ret = vgic_its_restore_itt(its, dev);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static int vgic_its_device_cmp(void *priv, struct list_head *a,
> >> +			       struct list_head *b)
> >> +{
> >> +	struct its_device *deva = container_of(a, struct its_device, dev_list);
> >> +	struct its_device *devb = container_of(b, struct its_device, dev_list);
> >> +
> >> +	if (deva->device_id < devb->device_id)
> >> +		return -1;
> >> +	else
> >> +		return 1;
> >> +}
> >> +
> >> +/**
> >>   * vgic_its_save_device_tables - Save the device table and all ITT
> >>   * into guest RAM
> >> + *
> >> + * L1/L2 handling is hidden by vgic_its_check_id() helper which directly
> >> + * returns the GPA of the device entry
> >>   */
> >>  static int vgic_its_save_device_tables(struct vgic_its *its)
> >>  {
> >> -	return -ENXIO;
> >> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >> +	struct its_device *dev;
> >> +	int dte_esz = abi->dte_esz;
> >> +	u64 baser;
> >> +
> >> +	baser = its->baser_device_table;
> >> +
> >> +	list_sort(NULL, &its->device_list, vgic_its_device_cmp);
> >> +
> >> +	list_for_each_entry(dev, &its->device_list, dev_list) {
> >> +		int ret;
> >> +		gpa_t eaddr;
> >> +
> >> +		if (!vgic_its_check_id(its, baser,
> >> +				       dev->device_id, &eaddr))
> >> +			return -EINVAL;
> >> +
> >> +		ret = vgic_its_save_itt(its, dev);
> >> +		if (ret)
> >> +			return ret;
> >> +
> >> +		ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
> >> +		if (ret)
> >> +			return ret;
> >> +	}
> >> +	return 0;
> >> +}
> >> +
> >> +/**
> >> + * handle_l1_entry - callback used for L1 entries (2 stage case)
> >> + *
> >> + * @its: its handle
> >> + * @id: id
> > 
> > IIUC, this is actually the index of the entry in the L1 table.  I think
> > this should be clarified.
> yep
> > 
> >> + * @addr: kernel VA
> >> + * @opaque: unused
> >> + * @next_offset: offset to the next L1 entry: 0 if the last element
> >> + * was found, 1 otherwise
> >> + */
> >> +static int handle_l1_entry(struct vgic_its *its, u32 id, void *addr,
> >> +			   void *opaque, u32 *next_offset)
> > 
> > nit: shouldn't this be called handle_l1_device_table_entry ?
> renamed into handle_l1_dte
> > 
> >> +{
> >> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >> +	int l2_start_id = id * (SZ_64K / GITS_LVL1_ENTRY_SIZE);
> > 
> > Hmmm, is this not actually supposed to be (SZ_64K / abi->dte_esz) ?
> no because 1st level entries have a fixed size of GITS_LVL1_ENTRY_SIZE bytes

yes, but the ID you calculate is a result of how many IDs each 64K 2nd
level table can hold, which depends on the size of each entry in the 2nd
level table, right?  Or am I misunderstanding how this works completely.

> > 
> >> +	u64 entry = *(u64 *)addr;
> >> +	int ret, ite_esz = abi->ite_esz;
> > 
> > Should this be ite_esz or dte_esz?
> 
> you're correct. dte_esz should be used.
> > 
> >> +	gpa_t gpa;
> > 
> > nit: put declarations with initialization on separate lines.
> OK
> > 
> >> +
> >> +	entry = le64_to_cpu(entry);
> >> +	*next_offset = 1;
> > 
> > I think you could attach a comment here saying that level 1 tables have
> > to be scanned entirely.
> added. note we exit as soon as the last element is found when scanning
> l2 tables.
> > 
> > But this also reminds me.  Does that mean that the next field in the DTE
> > in your documented ABI format points to the next DTE within that level-2
> > table, or does it point across to different level-2 tables?  I think
> > this needs to be clarified in the ABI unless I'm missing something.
> see above comment on next_index semantic. In the doc I talk about
> deviceid offset and not of table index.
> 

ok, I see, I was misled by the definition of lookup_table saying that it
returns 1 if the last element is identified, which is only true when you
actually find an element that is valid and where the next field is zero.
I understood it to mean if it found the last item in the table it was
scanning.  So it is implied that lookup table can be called in two
levels and the return value indicates if the element was the last from
the point of view of the highest level, not in the context the last
instance was called.

Note that it's further confusing that the handler function has the
return value the other way around, where 0 means it's the last element.
Perhaps you could make this much more readable by introducing a define
for the return values.

Thanks,
-Christoffer

[PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Christoffer Dall]

On Wed, May 03, 2017 at 04:07:45PM +0200, Auger Eric wrote:
> Hi Christoffer,
> On 30/04/2017 22:55, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:32PM +0200, Eric Auger wrote:
> >> This patch saves the device table entries into guest RAM.
> >> Both flat table and 2 stage tables are supported. DeviceId
> >> indexing is used.
> >>
> >> For each device listed in the device table, we also save
> >> the translation table using the vgic_its_save/restore_itt
> >> routines.
> >>
> >> On restore, devices are re-allocated and their ite are
> >> re-built.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - sort the device list by deviceid on device table save
> >> - use defines for shifts and masks
> >> - use abi->dte_esz
> >> - clatify entry sizes for L1 and L2 tables
> >>
> >> v3 -> v4:
> >> - use the new proto for its_alloc_device
> >> - compute_next_devid_offset, vgic_its_flush/restore_itt
> >>   become static in this patch
> >> - change in the DTE entry format with the introduction of the
> >>   valid bit and next field width decrease; ittaddr encoded
> >>   on its full range
> >> - fix handle_l1_entry entry handling
> >> - correct vgic_its_table_restore error handling
> >>
> >> v2 -> v3:
> >> - fix itt_addr bitmask in vgic_its_restore_dte
> >> - addition of return 0 in vgic_its_restore_ite moved to
> >>   the ITE related patch
> >>
> >> v1 -> v2:
> >> - use 8 byte format for DTE and ITE
> >> - support 2 stage format
> >> - remove kvm parameter
> >> - ITT flush/restore moved in a separate patch
> >> - use deviceid indexing
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 183 +++++++++++++++++++++++++++++++++++++++++--
> >>  virt/kvm/arm/vgic/vgic.h     |   7 ++
> >>  2 files changed, 185 insertions(+), 5 deletions(-)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index b02fc3f..86dfc6c 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -1682,7 +1682,8 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
> >>  	return ret;
> >>  }
> >>  
> >> -u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >> +static u32 compute_next_devid_offset(struct list_head *h,
> >> +				     struct its_device *dev)
> >>  {
> >>  	struct list_head *e = &dev->dev_list;
> >>  	struct its_device *next;
> >> @@ -1858,7 +1859,7 @@ static int vgic_its_ite_cmp(void *priv, struct list_head *a,
> >>  		return 1;
> >>  }
> >>  
> >> -int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >> +static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >>  {
> >>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>  	gpa_t base = device->itt_addr;
> >> @@ -1877,7 +1878,7 @@ int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >>  	return 0;
> >>  }
> >>  
> >> -int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >> +static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >>  {
> >>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>  	gpa_t base = dev->itt_addr;
> >> @@ -1895,12 +1896,161 @@ int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >>  }
> >>  
> >>  /**
> >> + * vgic_its_save_dte - Save a device table entry at a given GPA
> >> + *
> >> + * @its: ITS handle
> >> + * @dev: ITS device
> >> + * @ptr: GPA
> >> + */
> >> +static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
> >> +			     gpa_t ptr, int dte_esz)
> >> +{
> >> +	struct kvm *kvm = its->dev->kvm;
> >> +	u64 val, itt_addr_field;
> >> +	int ret;
> >> +	u32 next_offset;
> >> +
> >> +	itt_addr_field = dev->itt_addr >> 8;
> >> +	next_offset = compute_next_devid_offset(&its->device_list, dev);
> >> +	val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
> >> +	       ((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
> > 
> > I think this implies that the next field in your ABI points to the next
> > offset, regardless of whether or not this is in a a level 2 or lavel 1
> > table.  See more comments on this below (I reviewed this patch from the
> > bottom up).
> Not sure I understand your comment.
> 
> Doc says:
>  - next: equals to 0 if this entry is the last one; otherwise it
>    corresponds to the deviceid offset to the next DTE, capped by
>    2^14 -1.
> 
> This is independent on 1 or 2 levels as we sort the devices by
> deviceid's and compute the delta between those id.

see below.

> > 
> > I have a feeling this wasn't tested with 2 level device tables.  Could
> > that be true?
> No this was tested with 1 & and 2 levels (I hacked the guest to force 2
> levels). 1 test hole I have though is all my dte's currently belong to
> the same 2d level page, ie. my deviceid are not scattered enough.
> > 
> >> +	       (itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
> >> +		(dev->nb_eventid_bits - 1));
> >> +	val = cpu_to_le64(val);
> >> +	ret = kvm_write_guest(kvm, ptr, &val, dte_esz);
> >> +	return ret;
> >> +}
> >> +
> >> +/**
> >> + * vgic_its_restore_dte - restore a device table entry
> >> + *
> >> + * @its: its handle
> >> + * @id: device id the DTE corresponds to
> >> + * @ptr: kernel VA where the 8 byte DTE is located
> >> + * @opaque: unused
> >> + * @next: offset to the next valid device id
> >> + *
> >> + * Return: < 0 on error, 0 otherwise
> >> + */
> >> +static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
> >> +				void *ptr, void *opaque, u32 *next)
> >> +{
> >> +	struct its_device *dev;
> >> +	gpa_t itt_addr;
> >> +	u8 nb_eventid_bits;
> >> +	u64 entry = *(u64 *)ptr;
> >> +	bool valid;
> >> +	int ret;
> >> +
> >> +	entry = le64_to_cpu(entry);
> >> +
> >> +	valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
> >> +	nb_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
> >> +	itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
> >> +			>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
> >> +	*next = 1;
> >> +
> >> +	if (!valid)
> >> +		return 0;
> >> +
> >> +	/* dte entry is valid */
> >> +	*next = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
> >> +
> >> +	ret = vgic_its_alloc_device(its, &dev, id,
> >> +				    itt_addr, nb_eventid_bits);
> >> +	if (ret)
> >> +		return ret;
> >> +	ret = vgic_its_restore_itt(its, dev);
> >> +
> >> +	return ret;
> >> +}
> >> +
> >> +static int vgic_its_device_cmp(void *priv, struct list_head *a,
> >> +			       struct list_head *b)
> >> +{
> >> +	struct its_device *deva = container_of(a, struct its_device, dev_list);
> >> +	struct its_device *devb = container_of(b, struct its_device, dev_list);
> >> +
> >> +	if (deva->device_id < devb->device_id)
> >> +		return -1;
> >> +	else
> >> +		return 1;
> >> +}
> >> +
> >> +/**
> >>   * vgic_its_save_device_tables - Save the device table and all ITT
> >>   * into guest RAM
> >> + *
> >> + * L1/L2 handling is hidden by vgic_its_check_id() helper which directly
> >> + * returns the GPA of the device entry
> >>   */
> >>  static int vgic_its_save_device_tables(struct vgic_its *its)
> >>  {
> >> -	return -ENXIO;
> >> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >> +	struct its_device *dev;
> >> +	int dte_esz = abi->dte_esz;
> >> +	u64 baser;
> >> +
> >> +	baser = its->baser_device_table;
> >> +
> >> +	list_sort(NULL, &its->device_list, vgic_its_device_cmp);
> >> +
> >> +	list_for_each_entry(dev, &its->device_list, dev_list) {
> >> +		int ret;
> >> +		gpa_t eaddr;
> >> +
> >> +		if (!vgic_its_check_id(its, baser,
> >> +				       dev->device_id, &eaddr))
> >> +			return -EINVAL;
> >> +
> >> +		ret = vgic_its_save_itt(its, dev);
> >> +		if (ret)
> >> +			return ret;
> >> +
> >> +		ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
> >> +		if (ret)
> >> +			return ret;
> >> +	}
> >> +	return 0;
> >> +}
> >> +
> >> +/**
> >> + * handle_l1_entry - callback used for L1 entries (2 stage case)
> >> + *
> >> + * @its: its handle
> >> + * @id: id
> > 
> > IIUC, this is actually the index of the entry in the L1 table.  I think
> > this should be clarified.
> yep
> > 
> >> + * @addr: kernel VA
> >> + * @opaque: unused
> >> + * @next_offset: offset to the next L1 entry: 0 if the last element
> >> + * was found, 1 otherwise
> >> + */
> >> +static int handle_l1_entry(struct vgic_its *its, u32 id, void *addr,
> >> +			   void *opaque, u32 *next_offset)
> > 
> > nit: shouldn't this be called handle_l1_device_table_entry ?
> renamed into handle_l1_dte
> > 
> >> +{
> >> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >> +	int l2_start_id = id * (SZ_64K / GITS_LVL1_ENTRY_SIZE);
> > 
> > Hmmm, is this not actually supposed to be (SZ_64K / abi->dte_esz) ?
> no because 1st level entries have a fixed size of GITS_LVL1_ENTRY_SIZE bytes

yes, but the ID you calculate is a result of how many IDs each 64K 2nd
level table can hold, which depends on the size of each entry in the 2nd
level table, right?  Or am I misunderstanding how this works completely.

> > 
> >> +	u64 entry = *(u64 *)addr;
> >> +	int ret, ite_esz = abi->ite_esz;
> > 
> > Should this be ite_esz or dte_esz?
> 
> you're correct. dte_esz should be used.
> > 
> >> +	gpa_t gpa;
> > 
> > nit: put declarations with initialization on separate lines.
> OK
> > 
> >> +
> >> +	entry = le64_to_cpu(entry);
> >> +	*next_offset = 1;
> > 
> > I think you could attach a comment here saying that level 1 tables have
> > to be scanned entirely.
> added. note we exit as soon as the last element is found when scanning
> l2 tables.
> > 
> > But this also reminds me.  Does that mean that the next field in the DTE
> > in your documented ABI format points to the next DTE within that level-2
> > table, or does it point across to different level-2 tables?  I think
> > this needs to be clarified in the ABI unless I'm missing something.
> see above comment on next_index semantic. In the doc I talk about
> deviceid offset and not of table index.
> 

ok, I see, I was misled by the definition of lookup_table saying that it
returns 1 if the last element is identified, which is only true when you
actually find an element that is valid and where the next field is zero.
I understood it to mean if it found the last item in the table it was
scanning.  So it is implied that lookup table can be called in two
levels and the return value indicates if the element was the last from
the point of view of the highest level, not in the context the last
instance was called.

Note that it's further confusing that the handler function has the
return value the other way around, where 0 means it's the last element.
Perhaps you could make this much more readable by introducing a define
for the return values.

Thanks,
-Christoffer

Re: [PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Auger Eric]

Hi Christoffer,

On 30/04/2017 22:14, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
>> Introduce routines to save and restore device ITT and their
>> interrupt table entries (ITE).
>>
>> The routines will be called on device table save and
>> restore. They will become static in subsequent patches.
> 
> Why this bottom-up approach?  Couldn't you start by having the patch
> that restores the device table and define the static functions that
> return an error there
done
, and then fill them in with subsequent patches
> (liek this one)?
> 
> That would have the added benefit of being able to tell how things are
> designed to be called.
> 
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - ITE are now sorted by eventid on the flush
>> - rename *flush* into *save*
>> - use macros for shits and masks
>> - pass ite_esz to vgic_its_save_ite
>>
>> v3 -> v4:
>> - lookup_table and compute_next_eventid_offset become static in this
>>   patch
>> - remove static along with vgic_its_flush/restore_itt to avoid
>>   compilation warnings
>> - next field only computed with a shift (mask removed)
>> - handle the case where the last element has not been found
>>
>> v2 -> v3:
>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
>>
>> v2: creation
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
>>  2 files changed, 129 insertions(+), 3 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 35b2ca1..b02fc3f 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -23,6 +23,7 @@
>>  #include <linux/interrupt.h>
>>  #include <linux/list.h>
>>  #include <linux/uaccess.h>
>> +#include <linux/list_sort.h>
>>  
>>  #include <linux/irqchip/arm-gic-v3.h>
>>  
>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>>  }
>>  
>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>  {
>>  	struct list_head *e = &ite->ite_list;
>>  	struct its_ite *next;
>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>>   *
>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
>>   */
>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>> -		 int start_id, entry_fn_t fn, void *opaque)
>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>> +			int start_id, entry_fn_t fn, void *opaque)
>>  {
>>  	void *entry = kzalloc(esz, GFP_KERNEL);
>>  	struct kvm *kvm = its->dev->kvm;
>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>  }
>>  
>>  /**
>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
>> + */
>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
>> +{
>> +	struct kvm *kvm = its->dev->kvm;
>> +	u32 next_offset;
>> +	u64 val;
>> +
>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
>> +		ite->collection->collection_id;
>> +	val = cpu_to_le64(val);
>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
>> +}
>> +
>> +/**
>> + * vgic_its_restore_ite - restore an interrupt translation entry
>> + * @event_id: id used for indexing
>> + * @ptr: pointer to the ITE entry
>> + * @opaque: pointer to the its_device
>> + * @next: id offset to the next entry
>> + */
>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
>> +				void *ptr, void *opaque, u32 *next)
>> +{
>> +	struct its_device *dev = (struct its_device *)opaque;
>> +	struct its_collection *collection;
>> +	struct kvm *kvm = its->dev->kvm;
>> +	u64 val, *p = (u64 *)ptr;
> 
> nit: initializations on separate line (and possible do that just above
> assigning val).
done
> 
>> +	struct vgic_irq *irq;
>> +	u32 coll_id, lpi_id;
>> +	struct its_ite *ite;
>> +	int ret;
>> +
>> +	val = *p;
>> +	*next = 1;
>> +
>> +	val = le64_to_cpu(val);
>> +
>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
>> +
>> +	if (!lpi_id)
>> +		return 0;
> 
> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
> the ID is valid?
no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
GIC_MIN_LPI cause an -EINVAL error
> 
> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
> and PPIs here)

> 
>> +
>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
> 
> Don't we need to validate this somehow since it will presumably be used
> to forward a pointer somehow by the caller?
checked against max number of eventids supported by the device
> 
>> +
>> +	collection = find_collection(its, coll_id);
>> +	if (!collection)
>> +		return -EINVAL;
>> +
>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
>> +				  lpi_id, event_id);
>> +	if (ret)
>> +		return ret;
>> +
>> +	irq = vgic_add_lpi(kvm, lpi_id);
>> +	if (IS_ERR(irq))
>> +		return PTR_ERR(irq);
>> +	ite->irq = irq;
>> +
>> +	/* restore the configuration of the LPI */
>> +	ret = update_lpi_config(kvm, irq, NULL);
>> +	if (ret)
>> +		return ret;
>> +
>> +	update_affinity_ite(kvm, ite);
>> +	return 0;
>> +}
>> +
>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>> +			    struct list_head *b)
>> +{
>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
>> +
>> +	if (itea->event_id < iteb->event_id)
>> +		return -1;
>> +	else
>> +		return 1;
>> +}
>> +
>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>> +{
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	gpa_t base = device->itt_addr;
>> +	struct its_ite *ite;
>> +	int ret, ite_esz = abi->ite_esz;
> 
> nit: initializations on separate line
OK
> 
>> +
>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
>> +
>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
>> +		gpa_t gpa = base + ite->event_id * ite_esz;
>> +
>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
>> +		if (ret)
>> +			return ret;
>> +	}
>> +	return 0;
>> +}
>> +
>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>> +{
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	gpa_t base = dev->itt_addr;
>> +	int ret, ite_esz = abi->ite_esz;
>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
> 
> nit: initializations on separate line
OK
> 
>> +
>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
>> +			    vgic_its_restore_ite, dev);
> 
> nit: extra white space
> 
>> +
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* if the last element has not been found we are in trouble */
>> +	return ret ? 0 : -EINVAL;
> 
> hmm, these are values potentially created by the guest in guest RAM,
> right?  So do we really abort migration and return an error to userspace
> in this case?
So we discussed with Peter/dave we shouldn't abort() in qemu in case of
such error. The restore table IOCTL will return an error. Up to qemu to
print the error. Destination guest will not be functional though.

Thanks

Eric
> 
> Also, this comment doesn't really tell me what this situation is and how
> we handle things...
> 
>> +}
>> +
>> +/**
>>   * vgic_its_save_device_tables - Save the device table and all ITT
>>   * into guest RAM
>>   */
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index 56e57c1..ce57fbd 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -81,6 +81,10 @@
>>  #define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
>>  #define KVM_ITS_CTE_RDBASE_SHIFT	16
>>  #define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
>> +#define KVM_ITS_ITE_NEXT_SHIFT		48
>> +#define KVM_ITS_ITE_PINTID_SHIFT	16
>> +#define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
>> +#define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
>>  
>>  static inline bool irq_is_pending(struct vgic_irq *irq)
>>  {
>> -- 
>> 2.5.5
>>
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Auger Eric]

Hi Christoffer,

On 30/04/2017 22:14, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
>> Introduce routines to save and restore device ITT and their
>> interrupt table entries (ITE).
>>
>> The routines will be called on device table save and
>> restore. They will become static in subsequent patches.
> 
> Why this bottom-up approach?  Couldn't you start by having the patch
> that restores the device table and define the static functions that
> return an error there
done
, and then fill them in with subsequent patches
> (liek this one)?
> 
> That would have the added benefit of being able to tell how things are
> designed to be called.
> 
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - ITE are now sorted by eventid on the flush
>> - rename *flush* into *save*
>> - use macros for shits and masks
>> - pass ite_esz to vgic_its_save_ite
>>
>> v3 -> v4:
>> - lookup_table and compute_next_eventid_offset become static in this
>>   patch
>> - remove static along with vgic_its_flush/restore_itt to avoid
>>   compilation warnings
>> - next field only computed with a shift (mask removed)
>> - handle the case where the last element has not been found
>>
>> v2 -> v3:
>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
>>
>> v2: creation
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
>>  2 files changed, 129 insertions(+), 3 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 35b2ca1..b02fc3f 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -23,6 +23,7 @@
>>  #include <linux/interrupt.h>
>>  #include <linux/list.h>
>>  #include <linux/uaccess.h>
>> +#include <linux/list_sort.h>
>>  
>>  #include <linux/irqchip/arm-gic-v3.h>
>>  
>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>>  }
>>  
>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>  {
>>  	struct list_head *e = &ite->ite_list;
>>  	struct its_ite *next;
>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>>   *
>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
>>   */
>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>> -		 int start_id, entry_fn_t fn, void *opaque)
>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>> +			int start_id, entry_fn_t fn, void *opaque)
>>  {
>>  	void *entry = kzalloc(esz, GFP_KERNEL);
>>  	struct kvm *kvm = its->dev->kvm;
>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>  }
>>  
>>  /**
>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
>> + */
>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
>> +{
>> +	struct kvm *kvm = its->dev->kvm;
>> +	u32 next_offset;
>> +	u64 val;
>> +
>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
>> +		ite->collection->collection_id;
>> +	val = cpu_to_le64(val);
>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
>> +}
>> +
>> +/**
>> + * vgic_its_restore_ite - restore an interrupt translation entry
>> + * @event_id: id used for indexing
>> + * @ptr: pointer to the ITE entry
>> + * @opaque: pointer to the its_device
>> + * @next: id offset to the next entry
>> + */
>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
>> +				void *ptr, void *opaque, u32 *next)
>> +{
>> +	struct its_device *dev = (struct its_device *)opaque;
>> +	struct its_collection *collection;
>> +	struct kvm *kvm = its->dev->kvm;
>> +	u64 val, *p = (u64 *)ptr;
> 
> nit: initializations on separate line (and possible do that just above
> assigning val).
done
> 
>> +	struct vgic_irq *irq;
>> +	u32 coll_id, lpi_id;
>> +	struct its_ite *ite;
>> +	int ret;
>> +
>> +	val = *p;
>> +	*next = 1;
>> +
>> +	val = le64_to_cpu(val);
>> +
>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
>> +
>> +	if (!lpi_id)
>> +		return 0;
> 
> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
> the ID is valid?
no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
GIC_MIN_LPI cause an -EINVAL error
> 
> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
> and PPIs here)

> 
>> +
>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
> 
> Don't we need to validate this somehow since it will presumably be used
> to forward a pointer somehow by the caller?
checked against max number of eventids supported by the device
> 
>> +
>> +	collection = find_collection(its, coll_id);
>> +	if (!collection)
>> +		return -EINVAL;
>> +
>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
>> +				  lpi_id, event_id);
>> +	if (ret)
>> +		return ret;
>> +
>> +	irq = vgic_add_lpi(kvm, lpi_id);
>> +	if (IS_ERR(irq))
>> +		return PTR_ERR(irq);
>> +	ite->irq = irq;
>> +
>> +	/* restore the configuration of the LPI */
>> +	ret = update_lpi_config(kvm, irq, NULL);
>> +	if (ret)
>> +		return ret;
>> +
>> +	update_affinity_ite(kvm, ite);
>> +	return 0;
>> +}
>> +
>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>> +			    struct list_head *b)
>> +{
>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
>> +
>> +	if (itea->event_id < iteb->event_id)
>> +		return -1;
>> +	else
>> +		return 1;
>> +}
>> +
>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>> +{
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	gpa_t base = device->itt_addr;
>> +	struct its_ite *ite;
>> +	int ret, ite_esz = abi->ite_esz;
> 
> nit: initializations on separate line
OK
> 
>> +
>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
>> +
>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
>> +		gpa_t gpa = base + ite->event_id * ite_esz;
>> +
>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
>> +		if (ret)
>> +			return ret;
>> +	}
>> +	return 0;
>> +}
>> +
>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>> +{
>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>> +	gpa_t base = dev->itt_addr;
>> +	int ret, ite_esz = abi->ite_esz;
>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
> 
> nit: initializations on separate line
OK
> 
>> +
>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
>> +			    vgic_its_restore_ite, dev);
> 
> nit: extra white space
> 
>> +
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* if the last element has not been found we are in trouble */
>> +	return ret ? 0 : -EINVAL;
> 
> hmm, these are values potentially created by the guest in guest RAM,
> right?  So do we really abort migration and return an error to userspace
> in this case?
So we discussed with Peter/dave we shouldn't abort() in qemu in case of
such error. The restore table IOCTL will return an error. Up to qemu to
print the error. Destination guest will not be functional though.

Thanks

Eric
> 
> Also, this comment doesn't really tell me what this situation is and how
> we handle things...
> 
>> +}
>> +
>> +/**
>>   * vgic_its_save_device_tables - Save the device table and all ITT
>>   * into guest RAM
>>   */
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index 56e57c1..ce57fbd 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -81,6 +81,10 @@
>>  #define KVM_ITS_CTE_VALID_MASK		BIT_ULL(63)
>>  #define KVM_ITS_CTE_RDBASE_SHIFT	16
>>  #define KVM_ITS_CTE_ICID_MASK		GENMASK_ULL(15, 0)
>> +#define KVM_ITS_ITE_NEXT_SHIFT		48
>> +#define KVM_ITS_ITE_PINTID_SHIFT	16
>> +#define KVM_ITS_ITE_PINTID_MASK		GENMASK_ULL(47, 16)
>> +#define KVM_ITS_ITE_ICID_MASK		GENMASK_ULL(15, 0)
>>  
>>  static inline bool irq_is_pending(struct vgic_irq *irq)
>>  {
>> -- 
>> 2.5.5
>>
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Christoffer Dall]

On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 30/04/2017 22:14, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
> >> Introduce routines to save and restore device ITT and their
> >> interrupt table entries (ITE).
> >>
> >> The routines will be called on device table save and
> >> restore. They will become static in subsequent patches.
> > 
> > Why this bottom-up approach?  Couldn't you start by having the patch
> > that restores the device table and define the static functions that
> > return an error there
> done
> , and then fill them in with subsequent patches
> > (liek this one)?
> > 
> > That would have the added benefit of being able to tell how things are
> > designed to be called.
> > 
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - ITE are now sorted by eventid on the flush
> >> - rename *flush* into *save*
> >> - use macros for shits and masks
> >> - pass ite_esz to vgic_its_save_ite
> >>
> >> v3 -> v4:
> >> - lookup_table and compute_next_eventid_offset become static in this
> >>   patch
> >> - remove static along with vgic_its_flush/restore_itt to avoid
> >>   compilation warnings
> >> - next field only computed with a shift (mask removed)
> >> - handle the case where the last element has not been found
> >>
> >> v2 -> v3:
> >> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
> >>
> >> v2: creation
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
> >>  virt/kvm/arm/vgic/vgic.h     |   4 ++
> >>  2 files changed, 129 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 35b2ca1..b02fc3f 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -23,6 +23,7 @@
> >>  #include <linux/interrupt.h>
> >>  #include <linux/list.h>
> >>  #include <linux/uaccess.h>
> >> +#include <linux/list_sort.h>
> >>  
> >>  #include <linux/irqchip/arm-gic-v3.h>
> >>  
> >> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >>  }
> >>  
> >> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >>  {
> >>  	struct list_head *e = &ite->ite_list;
> >>  	struct its_ite *next;
> >> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> >>   *
> >>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
> >>   */
> >> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >> -		 int start_id, entry_fn_t fn, void *opaque)
> >> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >> +			int start_id, entry_fn_t fn, void *opaque)
> >>  {
> >>  	void *entry = kzalloc(esz, GFP_KERNEL);
> >>  	struct kvm *kvm = its->dev->kvm;
> >> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>  }
> >>  
> >>  /**
> >> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
> >> + */
> >> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
> >> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
> >> +{
> >> +	struct kvm *kvm = its->dev->kvm;
> >> +	u32 next_offset;
> >> +	u64 val;
> >> +
> >> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
> >> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
> >> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
> >> +		ite->collection->collection_id;
> >> +	val = cpu_to_le64(val);
> >> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
> >> +}
> >> +
> >> +/**
> >> + * vgic_its_restore_ite - restore an interrupt translation entry
> >> + * @event_id: id used for indexing
> >> + * @ptr: pointer to the ITE entry
> >> + * @opaque: pointer to the its_device
> >> + * @next: id offset to the next entry
> >> + */
> >> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
> >> +				void *ptr, void *opaque, u32 *next)
> >> +{
> >> +	struct its_device *dev = (struct its_device *)opaque;
> >> +	struct its_collection *collection;
> >> +	struct kvm *kvm = its->dev->kvm;
> >> +	u64 val, *p = (u64 *)ptr;
> > 
> > nit: initializations on separate line (and possible do that just above
> > assigning val).
> done
> > 
> >> +	struct vgic_irq *irq;
> >> +	u32 coll_id, lpi_id;
> >> +	struct its_ite *ite;
> >> +	int ret;
> >> +
> >> +	val = *p;
> >> +	*next = 1;
> >> +
> >> +	val = le64_to_cpu(val);
> >> +
> >> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
> >> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
> >> +
> >> +	if (!lpi_id)
> >> +		return 0;
> > 
> > are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
> > the ID is valid?
> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
> GIC_MIN_LPI cause an -EINVAL error
> > 
> > (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
> > and PPIs here)
> 
> > 
> >> +
> >> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
> > 
> > Don't we need to validate this somehow since it will presumably be used
> > to forward a pointer somehow by the caller?
> checked against max number of eventids supported by the device
> > 
> >> +
> >> +	collection = find_collection(its, coll_id);
> >> +	if (!collection)
> >> +		return -EINVAL;
> >> +
> >> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
> >> +				  lpi_id, event_id);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	irq = vgic_add_lpi(kvm, lpi_id);
> >> +	if (IS_ERR(irq))
> >> +		return PTR_ERR(irq);
> >> +	ite->irq = irq;
> >> +
> >> +	/* restore the configuration of the LPI */
> >> +	ret = update_lpi_config(kvm, irq, NULL);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	update_affinity_ite(kvm, ite);
> >> +	return 0;
> >> +}
> >> +
> >> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
> >> +			    struct list_head *b)
> >> +{
> >> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
> >> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
> >> +
> >> +	if (itea->event_id < iteb->event_id)
> >> +		return -1;
> >> +	else
> >> +		return 1;
> >> +}
> >> +
> >> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >> +{
> >> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >> +	gpa_t base = device->itt_addr;
> >> +	struct its_ite *ite;
> >> +	int ret, ite_esz = abi->ite_esz;
> > 
> > nit: initializations on separate line
> OK
> > 
> >> +
> >> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
> >> +
> >> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
> >> +		gpa_t gpa = base + ite->event_id * ite_esz;
> >> +
> >> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
> >> +		if (ret)
> >> +			return ret;
> >> +	}
> >> +	return 0;
> >> +}
> >> +
> >> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >> +{
> >> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >> +	gpa_t base = dev->itt_addr;
> >> +	int ret, ite_esz = abi->ite_esz;
> >> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
> > 
> > nit: initializations on separate line
> OK
> > 
> >> +
> >> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
> >> +			    vgic_its_restore_ite, dev);
> > 
> > nit: extra white space
> > 
> >> +
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	/* if the last element has not been found we are in trouble */
> >> +	return ret ? 0 : -EINVAL;
> > 
> > hmm, these are values potentially created by the guest in guest RAM,
> > right?  So do we really abort migration and return an error to userspace
> > in this case?
> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
> such error. The restore table IOCTL will return an error. Up to qemu to
> print the error. Destination guest will not be functional though.
> 

ok, I'm just wondering if userspace can make a qualified decision based
on this error code.  EINVAL typically means that userspace provided
something incorrect, which I suppose in a sense is true, but this should
be the only case where we return EINVAL here.  Userspace must be able to
tell the cases apart where the guest programmed bogus into memory before
migration started, in which case we should ignore-and-resume, and where
QEMU errornously provide some bogus value where the machine state
becomes unreliable and must be powered down.

Thanks,
-Christoffer

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Christoffer Dall]

On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 30/04/2017 22:14, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
> >> Introduce routines to save and restore device ITT and their
> >> interrupt table entries (ITE).
> >>
> >> The routines will be called on device table save and
> >> restore. They will become static in subsequent patches.
> > 
> > Why this bottom-up approach?  Couldn't you start by having the patch
> > that restores the device table and define the static functions that
> > return an error there
> done
> , and then fill them in with subsequent patches
> > (liek this one)?
> > 
> > That would have the added benefit of being able to tell how things are
> > designed to be called.
> > 
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v4 -> v5:
> >> - ITE are now sorted by eventid on the flush
> >> - rename *flush* into *save*
> >> - use macros for shits and masks
> >> - pass ite_esz to vgic_its_save_ite
> >>
> >> v3 -> v4:
> >> - lookup_table and compute_next_eventid_offset become static in this
> >>   patch
> >> - remove static along with vgic_its_flush/restore_itt to avoid
> >>   compilation warnings
> >> - next field only computed with a shift (mask removed)
> >> - handle the case where the last element has not been found
> >>
> >> v2 -> v3:
> >> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
> >>
> >> v2: creation
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
> >>  virt/kvm/arm/vgic/vgic.h     |   4 ++
> >>  2 files changed, 129 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 35b2ca1..b02fc3f 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -23,6 +23,7 @@
> >>  #include <linux/interrupt.h>
> >>  #include <linux/list.h>
> >>  #include <linux/uaccess.h>
> >> +#include <linux/list_sort.h>
> >>  
> >>  #include <linux/irqchip/arm-gic-v3.h>
> >>  
> >> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >>  }
> >>  
> >> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >>  {
> >>  	struct list_head *e = &ite->ite_list;
> >>  	struct its_ite *next;
> >> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> >>   *
> >>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
> >>   */
> >> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >> -		 int start_id, entry_fn_t fn, void *opaque)
> >> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >> +			int start_id, entry_fn_t fn, void *opaque)
> >>  {
> >>  	void *entry = kzalloc(esz, GFP_KERNEL);
> >>  	struct kvm *kvm = its->dev->kvm;
> >> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>  }
> >>  
> >>  /**
> >> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
> >> + */
> >> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
> >> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
> >> +{
> >> +	struct kvm *kvm = its->dev->kvm;
> >> +	u32 next_offset;
> >> +	u64 val;
> >> +
> >> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
> >> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
> >> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
> >> +		ite->collection->collection_id;
> >> +	val = cpu_to_le64(val);
> >> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
> >> +}
> >> +
> >> +/**
> >> + * vgic_its_restore_ite - restore an interrupt translation entry
> >> + * @event_id: id used for indexing
> >> + * @ptr: pointer to the ITE entry
> >> + * @opaque: pointer to the its_device
> >> + * @next: id offset to the next entry
> >> + */
> >> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
> >> +				void *ptr, void *opaque, u32 *next)
> >> +{
> >> +	struct its_device *dev = (struct its_device *)opaque;
> >> +	struct its_collection *collection;
> >> +	struct kvm *kvm = its->dev->kvm;
> >> +	u64 val, *p = (u64 *)ptr;
> > 
> > nit: initializations on separate line (and possible do that just above
> > assigning val).
> done
> > 
> >> +	struct vgic_irq *irq;
> >> +	u32 coll_id, lpi_id;
> >> +	struct its_ite *ite;
> >> +	int ret;
> >> +
> >> +	val = *p;
> >> +	*next = 1;
> >> +
> >> +	val = le64_to_cpu(val);
> >> +
> >> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
> >> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
> >> +
> >> +	if (!lpi_id)
> >> +		return 0;
> > 
> > are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
> > the ID is valid?
> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
> GIC_MIN_LPI cause an -EINVAL error
> > 
> > (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
> > and PPIs here)
> 
> > 
> >> +
> >> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
> > 
> > Don't we need to validate this somehow since it will presumably be used
> > to forward a pointer somehow by the caller?
> checked against max number of eventids supported by the device
> > 
> >> +
> >> +	collection = find_collection(its, coll_id);
> >> +	if (!collection)
> >> +		return -EINVAL;
> >> +
> >> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
> >> +				  lpi_id, event_id);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	irq = vgic_add_lpi(kvm, lpi_id);
> >> +	if (IS_ERR(irq))
> >> +		return PTR_ERR(irq);
> >> +	ite->irq = irq;
> >> +
> >> +	/* restore the configuration of the LPI */
> >> +	ret = update_lpi_config(kvm, irq, NULL);
> >> +	if (ret)
> >> +		return ret;
> >> +
> >> +	update_affinity_ite(kvm, ite);
> >> +	return 0;
> >> +}
> >> +
> >> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
> >> +			    struct list_head *b)
> >> +{
> >> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
> >> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
> >> +
> >> +	if (itea->event_id < iteb->event_id)
> >> +		return -1;
> >> +	else
> >> +		return 1;
> >> +}
> >> +
> >> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >> +{
> >> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >> +	gpa_t base = device->itt_addr;
> >> +	struct its_ite *ite;
> >> +	int ret, ite_esz = abi->ite_esz;
> > 
> > nit: initializations on separate line
> OK
> > 
> >> +
> >> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
> >> +
> >> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
> >> +		gpa_t gpa = base + ite->event_id * ite_esz;
> >> +
> >> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
> >> +		if (ret)
> >> +			return ret;
> >> +	}
> >> +	return 0;
> >> +}
> >> +
> >> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >> +{
> >> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >> +	gpa_t base = dev->itt_addr;
> >> +	int ret, ite_esz = abi->ite_esz;
> >> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
> > 
> > nit: initializations on separate line
> OK
> > 
> >> +
> >> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
> >> +			    vgic_its_restore_ite, dev);
> > 
> > nit: extra white space
> > 
> >> +
> >> +	if (ret < 0)
> >> +		return ret;
> >> +
> >> +	/* if the last element has not been found we are in trouble */
> >> +	return ret ? 0 : -EINVAL;
> > 
> > hmm, these are values potentially created by the guest in guest RAM,
> > right?  So do we really abort migration and return an error to userspace
> > in this case?
> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
> such error. The restore table IOCTL will return an error. Up to qemu to
> print the error. Destination guest will not be functional though.
> 

ok, I'm just wondering if userspace can make a qualified decision based
on this error code.  EINVAL typically means that userspace provided
something incorrect, which I suppose in a sense is true, but this should
be the only case where we return EINVAL here.  Userspace must be able to
tell the cases apart where the guest programmed bogus into memory before
migration started, in which case we should ignore-and-resume, and where
QEMU errornously provide some bogus value where the machine state
becomes unreliable and must be powered down.

Thanks,
-Christoffer

Re: [PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Auger Eric]

Hi Christoffer,

On 03/05/2017 17:29, Christoffer Dall wrote:
> On Wed, May 03, 2017 at 04:07:45PM +0200, Auger Eric wrote:
>> Hi Christoffer,
>> On 30/04/2017 22:55, Christoffer Dall wrote:
>>> On Fri, Apr 14, 2017 at 12:15:32PM +0200, Eric Auger wrote:
>>>> This patch saves the device table entries into guest RAM.
>>>> Both flat table and 2 stage tables are supported. DeviceId
>>>> indexing is used.
>>>>
>>>> For each device listed in the device table, we also save
>>>> the translation table using the vgic_its_save/restore_itt
>>>> routines.
>>>>
>>>> On restore, devices are re-allocated and their ite are
>>>> re-built.
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>> v4 -> v5:
>>>> - sort the device list by deviceid on device table save
>>>> - use defines for shifts and masks
>>>> - use abi->dte_esz
>>>> - clatify entry sizes for L1 and L2 tables
>>>>
>>>> v3 -> v4:
>>>> - use the new proto for its_alloc_device
>>>> - compute_next_devid_offset, vgic_its_flush/restore_itt
>>>>   become static in this patch
>>>> - change in the DTE entry format with the introduction of the
>>>>   valid bit and next field width decrease; ittaddr encoded
>>>>   on its full range
>>>> - fix handle_l1_entry entry handling
>>>> - correct vgic_its_table_restore error handling
>>>>
>>>> v2 -> v3:
>>>> - fix itt_addr bitmask in vgic_its_restore_dte
>>>> - addition of return 0 in vgic_its_restore_ite moved to
>>>>   the ITE related patch
>>>>
>>>> v1 -> v2:
>>>> - use 8 byte format for DTE and ITE
>>>> - support 2 stage format
>>>> - remove kvm parameter
>>>> - ITT flush/restore moved in a separate patch
>>>> - use deviceid indexing
>>>> ---
>>>>  virt/kvm/arm/vgic/vgic-its.c | 183 +++++++++++++++++++++++++++++++++++++++++--
>>>>  virt/kvm/arm/vgic/vgic.h     |   7 ++
>>>>  2 files changed, 185 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>> index b02fc3f..86dfc6c 100644
>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>> @@ -1682,7 +1682,8 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>>>  	return ret;
>>>>  }
>>>>  
>>>> -u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>>> +static u32 compute_next_devid_offset(struct list_head *h,
>>>> +				     struct its_device *dev)
>>>>  {
>>>>  	struct list_head *e = &dev->dev_list;
>>>>  	struct its_device *next;
>>>> @@ -1858,7 +1859,7 @@ static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>>>>  		return 1;
>>>>  }
>>>>  
>>>> -int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>> +static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>>  {
>>>>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>  	gpa_t base = device->itt_addr;
>>>> @@ -1877,7 +1878,7 @@ int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>>  	return 0;
>>>>  }
>>>>  
>>>> -int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>> +static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>>  {
>>>>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>  	gpa_t base = dev->itt_addr;
>>>> @@ -1895,12 +1896,161 @@ int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>>  }
>>>>  
>>>>  /**
>>>> + * vgic_its_save_dte - Save a device table entry at a given GPA
>>>> + *
>>>> + * @its: ITS handle
>>>> + * @dev: ITS device
>>>> + * @ptr: GPA
>>>> + */
>>>> +static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
>>>> +			     gpa_t ptr, int dte_esz)
>>>> +{
>>>> +	struct kvm *kvm = its->dev->kvm;
>>>> +	u64 val, itt_addr_field;
>>>> +	int ret;
>>>> +	u32 next_offset;
>>>> +
>>>> +	itt_addr_field = dev->itt_addr >> 8;
>>>> +	next_offset = compute_next_devid_offset(&its->device_list, dev);
>>>> +	val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
>>>> +	       ((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
>>>
>>> I think this implies that the next field in your ABI points to the next
>>> offset, regardless of whether or not this is in a a level 2 or lavel 1
>>> table.  See more comments on this below (I reviewed this patch from the
>>> bottom up).
>> Not sure I understand your comment.
>>
>> Doc says:
>>  - next: equals to 0 if this entry is the last one; otherwise it
>>    corresponds to the deviceid offset to the next DTE, capped by
>>    2^14 -1.
>>
>> This is independent on 1 or 2 levels as we sort the devices by
>> deviceid's and compute the delta between those id.
> 
> see below.
> 
>>>
>>> I have a feeling this wasn't tested with 2 level device tables.  Could
>>> that be true?
>> No this was tested with 1 & and 2 levels (I hacked the guest to force 2
>> levels). 1 test hole I have though is all my dte's currently belong to
>> the same 2d level page, ie. my deviceid are not scattered enough.
>>>
>>>> +	       (itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
>>>> +		(dev->nb_eventid_bits - 1));
>>>> +	val = cpu_to_le64(val);
>>>> +	ret = kvm_write_guest(kvm, ptr, &val, dte_esz);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +/**
>>>> + * vgic_its_restore_dte - restore a device table entry
>>>> + *
>>>> + * @its: its handle
>>>> + * @id: device id the DTE corresponds to
>>>> + * @ptr: kernel VA where the 8 byte DTE is located
>>>> + * @opaque: unused
>>>> + * @next: offset to the next valid device id
>>>> + *
>>>> + * Return: < 0 on error, 0 otherwise
>>>> + */
>>>> +static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
>>>> +				void *ptr, void *opaque, u32 *next)
>>>> +{
>>>> +	struct its_device *dev;
>>>> +	gpa_t itt_addr;
>>>> +	u8 nb_eventid_bits;
>>>> +	u64 entry = *(u64 *)ptr;
>>>> +	bool valid;
>>>> +	int ret;
>>>> +
>>>> +	entry = le64_to_cpu(entry);
>>>> +
>>>> +	valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
>>>> +	nb_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
>>>> +	itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
>>>> +			>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
>>>> +	*next = 1;
>>>> +
>>>> +	if (!valid)
>>>> +		return 0;
>>>> +
>>>> +	/* dte entry is valid */
>>>> +	*next = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
>>>> +
>>>> +	ret = vgic_its_alloc_device(its, &dev, id,
>>>> +				    itt_addr, nb_eventid_bits);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +	ret = vgic_its_restore_itt(its, dev);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int vgic_its_device_cmp(void *priv, struct list_head *a,
>>>> +			       struct list_head *b)
>>>> +{
>>>> +	struct its_device *deva = container_of(a, struct its_device, dev_list);
>>>> +	struct its_device *devb = container_of(b, struct its_device, dev_list);
>>>> +
>>>> +	if (deva->device_id < devb->device_id)
>>>> +		return -1;
>>>> +	else
>>>> +		return 1;
>>>> +}
>>>> +
>>>> +/**
>>>>   * vgic_its_save_device_tables - Save the device table and all ITT
>>>>   * into guest RAM
>>>> + *
>>>> + * L1/L2 handling is hidden by vgic_its_check_id() helper which directly
>>>> + * returns the GPA of the device entry
>>>>   */
>>>>  static int vgic_its_save_device_tables(struct vgic_its *its)
>>>>  {
>>>> -	return -ENXIO;
>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>> +	struct its_device *dev;
>>>> +	int dte_esz = abi->dte_esz;
>>>> +	u64 baser;
>>>> +
>>>> +	baser = its->baser_device_table;
>>>> +
>>>> +	list_sort(NULL, &its->device_list, vgic_its_device_cmp);
>>>> +
>>>> +	list_for_each_entry(dev, &its->device_list, dev_list) {
>>>> +		int ret;
>>>> +		gpa_t eaddr;
>>>> +
>>>> +		if (!vgic_its_check_id(its, baser,
>>>> +				       dev->device_id, &eaddr))
>>>> +			return -EINVAL;
>>>> +
>>>> +		ret = vgic_its_save_itt(its, dev);
>>>> +		if (ret)
>>>> +			return ret;
>>>> +
>>>> +		ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
>>>> +		if (ret)
>>>> +			return ret;
>>>> +	}
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +/**
>>>> + * handle_l1_entry - callback used for L1 entries (2 stage case)
>>>> + *
>>>> + * @its: its handle
>>>> + * @id: id
>>>
>>> IIUC, this is actually the index of the entry in the L1 table.  I think
>>> this should be clarified.
>> yep
>>>
>>>> + * @addr: kernel VA
>>>> + * @opaque: unused
>>>> + * @next_offset: offset to the next L1 entry: 0 if the last element
>>>> + * was found, 1 otherwise
>>>> + */
>>>> +static int handle_l1_entry(struct vgic_its *its, u32 id, void *addr,
>>>> +			   void *opaque, u32 *next_offset)
>>>
>>> nit: shouldn't this be called handle_l1_device_table_entry ?
>> renamed into handle_l1_dte
>>>
>>>> +{
>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>> +	int l2_start_id = id * (SZ_64K / GITS_LVL1_ENTRY_SIZE);
>>>
>>> Hmmm, is this not actually supposed to be (SZ_64K / abi->dte_esz) ?
>> no because 1st level entries have a fixed size of GITS_LVL1_ENTRY_SIZE bytes
> 
> yes, but the ID you calculate is a result of how many IDs each 64K 2nd
> level table can hold, which depends on the size of each entry in the 2nd
> level table, right?  Or am I misunderstanding how this works completely.
Hum damn you're fully right. Thank you for insisting.
GITS_LVL1_ENTRY_SIZE must be passed instead as l1_esz in lookup_table()

Eric
> 
>>>
>>>> +	u64 entry = *(u64 *)addr;
>>>> +	int ret, ite_esz = abi->ite_esz;
>>>
>>> Should this be ite_esz or dte_esz?
>>
>> you're correct. dte_esz should be used.
>>>
>>>> +	gpa_t gpa;
>>>
>>> nit: put declarations with initialization on separate lines.
>> OK
>>>
>>>> +
>>>> +	entry = le64_to_cpu(entry);
>>>> +	*next_offset = 1;
>>>
>>> I think you could attach a comment here saying that level 1 tables have
>>> to be scanned entirely.
>> added. note we exit as soon as the last element is found when scanning
>> l2 tables.
>>>
>>> But this also reminds me.  Does that mean that the next field in the DTE
>>> in your documented ABI format points to the next DTE within that level-2
>>> table, or does it point across to different level-2 tables?  I think
>>> this needs to be clarified in the ABI unless I'm missing something.
>> see above comment on next_index semantic. In the doc I talk about
>> deviceid offset and not of table index.
>>
> 
> ok, I see, I was misled by the definition of lookup_table saying that it
> returns 1 if the last element is identified, which is only true when you
> actually find an element that is valid and where the next field is zero.
> I understood it to mean if it found the last item in the table it was
> scanning.  So it is implied that lookup table can be called in two
> levels and the return value indicates if the element was the last from
> the point of view of the highest level, not in the context the last
> instance was called.
> 
> Note that it's further confusing that the handler function has the
> return value the other way around, where 0 means it's the last element.
> Perhaps you could make this much more readable by introducing a define
> for the return values.
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 20/22] KVM: arm64: vgic-its: Device table save/restore

[Auger Eric]

Hi Christoffer,

On 03/05/2017 17:29, Christoffer Dall wrote:
> On Wed, May 03, 2017 at 04:07:45PM +0200, Auger Eric wrote:
>> Hi Christoffer,
>> On 30/04/2017 22:55, Christoffer Dall wrote:
>>> On Fri, Apr 14, 2017 at 12:15:32PM +0200, Eric Auger wrote:
>>>> This patch saves the device table entries into guest RAM.
>>>> Both flat table and 2 stage tables are supported. DeviceId
>>>> indexing is used.
>>>>
>>>> For each device listed in the device table, we also save
>>>> the translation table using the vgic_its_save/restore_itt
>>>> routines.
>>>>
>>>> On restore, devices are re-allocated and their ite are
>>>> re-built.
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>> v4 -> v5:
>>>> - sort the device list by deviceid on device table save
>>>> - use defines for shifts and masks
>>>> - use abi->dte_esz
>>>> - clatify entry sizes for L1 and L2 tables
>>>>
>>>> v3 -> v4:
>>>> - use the new proto for its_alloc_device
>>>> - compute_next_devid_offset, vgic_its_flush/restore_itt
>>>>   become static in this patch
>>>> - change in the DTE entry format with the introduction of the
>>>>   valid bit and next field width decrease; ittaddr encoded
>>>>   on its full range
>>>> - fix handle_l1_entry entry handling
>>>> - correct vgic_its_table_restore error handling
>>>>
>>>> v2 -> v3:
>>>> - fix itt_addr bitmask in vgic_its_restore_dte
>>>> - addition of return 0 in vgic_its_restore_ite moved to
>>>>   the ITE related patch
>>>>
>>>> v1 -> v2:
>>>> - use 8 byte format for DTE and ITE
>>>> - support 2 stage format
>>>> - remove kvm parameter
>>>> - ITT flush/restore moved in a separate patch
>>>> - use deviceid indexing
>>>> ---
>>>>  virt/kvm/arm/vgic/vgic-its.c | 183 +++++++++++++++++++++++++++++++++++++++++--
>>>>  virt/kvm/arm/vgic/vgic.h     |   7 ++
>>>>  2 files changed, 185 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>> index b02fc3f..86dfc6c 100644
>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>> @@ -1682,7 +1682,8 @@ int vgic_its_attr_regs_access(struct kvm_device *dev,
>>>>  	return ret;
>>>>  }
>>>>  
>>>> -u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>>> +static u32 compute_next_devid_offset(struct list_head *h,
>>>> +				     struct its_device *dev)
>>>>  {
>>>>  	struct list_head *e = &dev->dev_list;
>>>>  	struct its_device *next;
>>>> @@ -1858,7 +1859,7 @@ static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>>>>  		return 1;
>>>>  }
>>>>  
>>>> -int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>> +static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>>  {
>>>>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>  	gpa_t base = device->itt_addr;
>>>> @@ -1877,7 +1878,7 @@ int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>>  	return 0;
>>>>  }
>>>>  
>>>> -int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>> +static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>>  {
>>>>  	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>  	gpa_t base = dev->itt_addr;
>>>> @@ -1895,12 +1896,161 @@ int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>>  }
>>>>  
>>>>  /**
>>>> + * vgic_its_save_dte - Save a device table entry at a given GPA
>>>> + *
>>>> + * @its: ITS handle
>>>> + * @dev: ITS device
>>>> + * @ptr: GPA
>>>> + */
>>>> +static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
>>>> +			     gpa_t ptr, int dte_esz)
>>>> +{
>>>> +	struct kvm *kvm = its->dev->kvm;
>>>> +	u64 val, itt_addr_field;
>>>> +	int ret;
>>>> +	u32 next_offset;
>>>> +
>>>> +	itt_addr_field = dev->itt_addr >> 8;
>>>> +	next_offset = compute_next_devid_offset(&its->device_list, dev);
>>>> +	val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
>>>> +	       ((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
>>>
>>> I think this implies that the next field in your ABI points to the next
>>> offset, regardless of whether or not this is in a a level 2 or lavel 1
>>> table.  See more comments on this below (I reviewed this patch from the
>>> bottom up).
>> Not sure I understand your comment.
>>
>> Doc says:
>>  - next: equals to 0 if this entry is the last one; otherwise it
>>    corresponds to the deviceid offset to the next DTE, capped by
>>    2^14 -1.
>>
>> This is independent on 1 or 2 levels as we sort the devices by
>> deviceid's and compute the delta between those id.
> 
> see below.
> 
>>>
>>> I have a feeling this wasn't tested with 2 level device tables.  Could
>>> that be true?
>> No this was tested with 1 & and 2 levels (I hacked the guest to force 2
>> levels). 1 test hole I have though is all my dte's currently belong to
>> the same 2d level page, ie. my deviceid are not scattered enough.
>>>
>>>> +	       (itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
>>>> +		(dev->nb_eventid_bits - 1));
>>>> +	val = cpu_to_le64(val);
>>>> +	ret = kvm_write_guest(kvm, ptr, &val, dte_esz);
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +/**
>>>> + * vgic_its_restore_dte - restore a device table entry
>>>> + *
>>>> + * @its: its handle
>>>> + * @id: device id the DTE corresponds to
>>>> + * @ptr: kernel VA where the 8 byte DTE is located
>>>> + * @opaque: unused
>>>> + * @next: offset to the next valid device id
>>>> + *
>>>> + * Return: < 0 on error, 0 otherwise
>>>> + */
>>>> +static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
>>>> +				void *ptr, void *opaque, u32 *next)
>>>> +{
>>>> +	struct its_device *dev;
>>>> +	gpa_t itt_addr;
>>>> +	u8 nb_eventid_bits;
>>>> +	u64 entry = *(u64 *)ptr;
>>>> +	bool valid;
>>>> +	int ret;
>>>> +
>>>> +	entry = le64_to_cpu(entry);
>>>> +
>>>> +	valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
>>>> +	nb_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
>>>> +	itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
>>>> +			>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
>>>> +	*next = 1;
>>>> +
>>>> +	if (!valid)
>>>> +		return 0;
>>>> +
>>>> +	/* dte entry is valid */
>>>> +	*next = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
>>>> +
>>>> +	ret = vgic_its_alloc_device(its, &dev, id,
>>>> +				    itt_addr, nb_eventid_bits);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +	ret = vgic_its_restore_itt(its, dev);
>>>> +
>>>> +	return ret;
>>>> +}
>>>> +
>>>> +static int vgic_its_device_cmp(void *priv, struct list_head *a,
>>>> +			       struct list_head *b)
>>>> +{
>>>> +	struct its_device *deva = container_of(a, struct its_device, dev_list);
>>>> +	struct its_device *devb = container_of(b, struct its_device, dev_list);
>>>> +
>>>> +	if (deva->device_id < devb->device_id)
>>>> +		return -1;
>>>> +	else
>>>> +		return 1;
>>>> +}
>>>> +
>>>> +/**
>>>>   * vgic_its_save_device_tables - Save the device table and all ITT
>>>>   * into guest RAM
>>>> + *
>>>> + * L1/L2 handling is hidden by vgic_its_check_id() helper which directly
>>>> + * returns the GPA of the device entry
>>>>   */
>>>>  static int vgic_its_save_device_tables(struct vgic_its *its)
>>>>  {
>>>> -	return -ENXIO;
>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>> +	struct its_device *dev;
>>>> +	int dte_esz = abi->dte_esz;
>>>> +	u64 baser;
>>>> +
>>>> +	baser = its->baser_device_table;
>>>> +
>>>> +	list_sort(NULL, &its->device_list, vgic_its_device_cmp);
>>>> +
>>>> +	list_for_each_entry(dev, &its->device_list, dev_list) {
>>>> +		int ret;
>>>> +		gpa_t eaddr;
>>>> +
>>>> +		if (!vgic_its_check_id(its, baser,
>>>> +				       dev->device_id, &eaddr))
>>>> +			return -EINVAL;
>>>> +
>>>> +		ret = vgic_its_save_itt(its, dev);
>>>> +		if (ret)
>>>> +			return ret;
>>>> +
>>>> +		ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
>>>> +		if (ret)
>>>> +			return ret;
>>>> +	}
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +/**
>>>> + * handle_l1_entry - callback used for L1 entries (2 stage case)
>>>> + *
>>>> + * @its: its handle
>>>> + * @id: id
>>>
>>> IIUC, this is actually the index of the entry in the L1 table.  I think
>>> this should be clarified.
>> yep
>>>
>>>> + * @addr: kernel VA
>>>> + * @opaque: unused
>>>> + * @next_offset: offset to the next L1 entry: 0 if the last element
>>>> + * was found, 1 otherwise
>>>> + */
>>>> +static int handle_l1_entry(struct vgic_its *its, u32 id, void *addr,
>>>> +			   void *opaque, u32 *next_offset)
>>>
>>> nit: shouldn't this be called handle_l1_device_table_entry ?
>> renamed into handle_l1_dte
>>>
>>>> +{
>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>> +	int l2_start_id = id * (SZ_64K / GITS_LVL1_ENTRY_SIZE);
>>>
>>> Hmmm, is this not actually supposed to be (SZ_64K / abi->dte_esz) ?
>> no because 1st level entries have a fixed size of GITS_LVL1_ENTRY_SIZE bytes
> 
> yes, but the ID you calculate is a result of how many IDs each 64K 2nd
> level table can hold, which depends on the size of each entry in the 2nd
> level table, right?  Or am I misunderstanding how this works completely.
Hum damn you're fully right. Thank you for insisting.
GITS_LVL1_ENTRY_SIZE must be passed instead as l1_esz in lookup_table()

Eric
> 
>>>
>>>> +	u64 entry = *(u64 *)addr;
>>>> +	int ret, ite_esz = abi->ite_esz;
>>>
>>> Should this be ite_esz or dte_esz?
>>
>> you're correct. dte_esz should be used.
>>>
>>>> +	gpa_t gpa;
>>>
>>> nit: put declarations with initialization on separate lines.
>> OK
>>>
>>>> +
>>>> +	entry = le64_to_cpu(entry);
>>>> +	*next_offset = 1;
>>>
>>> I think you could attach a comment here saying that level 1 tables have
>>> to be scanned entirely.
>> added. note we exit as soon as the last element is found when scanning
>> l2 tables.
>>>
>>> But this also reminds me.  Does that mean that the next field in the DTE
>>> in your documented ABI format points to the next DTE within that level-2
>>> table, or does it point across to different level-2 tables?  I think
>>> this needs to be clarified in the ABI unless I'm missing something.
>> see above comment on next_index semantic. In the doc I talk about
>> deviceid offset and not of table index.
>>
> 
> ok, I see, I was misled by the definition of lookup_table saying that it
> returns 1 if the last element is identified, which is only true when you
> actually find an element that is valid and where the next field is zero.
> I understood it to mean if it found the last item in the table it was
> scanning.  So it is implied that lookup table can be called in two
> levels and the return value indicates if the element was the last from
> the point of view of the highest level, not in the context the last
> instance was called.
> 
> Note that it's further confusing that the handler function has the
> return value the other way around, where 0 means it's the last element.
> Perhaps you could make this much more readable by introducing a define
> for the return values.
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Auger Eric]

Hi Christoffer,

On 03/05/2017 18:37, Christoffer Dall wrote:
> On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 30/04/2017 22:14, Christoffer Dall wrote:
>>> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
>>>> Introduce routines to save and restore device ITT and their
>>>> interrupt table entries (ITE).
>>>>
>>>> The routines will be called on device table save and
>>>> restore. They will become static in subsequent patches.
>>>
>>> Why this bottom-up approach?  Couldn't you start by having the patch
>>> that restores the device table and define the static functions that
>>> return an error there
>> done
>> , and then fill them in with subsequent patches
>>> (liek this one)?
>>>
>>> That would have the added benefit of being able to tell how things are
>>> designed to be called.
>>>
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>> v4 -> v5:
>>>> - ITE are now sorted by eventid on the flush
>>>> - rename *flush* into *save*
>>>> - use macros for shits and masks
>>>> - pass ite_esz to vgic_its_save_ite
>>>>
>>>> v3 -> v4:
>>>> - lookup_table and compute_next_eventid_offset become static in this
>>>>   patch
>>>> - remove static along with vgic_its_flush/restore_itt to avoid
>>>>   compilation warnings
>>>> - next field only computed with a shift (mask removed)
>>>> - handle the case where the last element has not been found
>>>>
>>>> v2 -> v3:
>>>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
>>>>
>>>> v2: creation
>>>> ---
>>>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
>>>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
>>>>  2 files changed, 129 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>> index 35b2ca1..b02fc3f 100644
>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>> @@ -23,6 +23,7 @@
>>>>  #include <linux/interrupt.h>
>>>>  #include <linux/list.h>
>>>>  #include <linux/uaccess.h>
>>>> +#include <linux/list_sort.h>
>>>>  
>>>>  #include <linux/irqchip/arm-gic-v3.h>
>>>>  
>>>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>>>>  }
>>>>  
>>>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>>  {
>>>>  	struct list_head *e = &ite->ite_list;
>>>>  	struct its_ite *next;
>>>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>>>>   *
>>>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
>>>>   */
>>>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>> -		 int start_id, entry_fn_t fn, void *opaque)
>>>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>> +			int start_id, entry_fn_t fn, void *opaque)
>>>>  {
>>>>  	void *entry = kzalloc(esz, GFP_KERNEL);
>>>>  	struct kvm *kvm = its->dev->kvm;
>>>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>  }
>>>>  
>>>>  /**
>>>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
>>>> + */
>>>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
>>>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
>>>> +{
>>>> +	struct kvm *kvm = its->dev->kvm;
>>>> +	u32 next_offset;
>>>> +	u64 val;
>>>> +
>>>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
>>>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
>>>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
>>>> +		ite->collection->collection_id;
>>>> +	val = cpu_to_le64(val);
>>>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
>>>> +}
>>>> +
>>>> +/**
>>>> + * vgic_its_restore_ite - restore an interrupt translation entry
>>>> + * @event_id: id used for indexing
>>>> + * @ptr: pointer to the ITE entry
>>>> + * @opaque: pointer to the its_device
>>>> + * @next: id offset to the next entry
>>>> + */
>>>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
>>>> +				void *ptr, void *opaque, u32 *next)
>>>> +{
>>>> +	struct its_device *dev = (struct its_device *)opaque;
>>>> +	struct its_collection *collection;
>>>> +	struct kvm *kvm = its->dev->kvm;
>>>> +	u64 val, *p = (u64 *)ptr;
>>>
>>> nit: initializations on separate line (and possible do that just above
>>> assigning val).
>> done
>>>
>>>> +	struct vgic_irq *irq;
>>>> +	u32 coll_id, lpi_id;
>>>> +	struct its_ite *ite;
>>>> +	int ret;
>>>> +
>>>> +	val = *p;
>>>> +	*next = 1;
>>>> +
>>>> +	val = le64_to_cpu(val);
>>>> +
>>>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
>>>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
>>>> +
>>>> +	if (!lpi_id)
>>>> +		return 0;
>>>
>>> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
>>> the ID is valid?
>> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
>> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
>> GIC_MIN_LPI cause an -EINVAL error
>>>
>>> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
>>> and PPIs here)
>>
>>>
>>>> +
>>>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
>>>
>>> Don't we need to validate this somehow since it will presumably be used
>>> to forward a pointer somehow by the caller?
>> checked against max number of eventids supported by the device
>>>
>>>> +
>>>> +	collection = find_collection(its, coll_id);
>>>> +	if (!collection)
>>>> +		return -EINVAL;
>>>> +
>>>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
>>>> +				  lpi_id, event_id);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	irq = vgic_add_lpi(kvm, lpi_id);
>>>> +	if (IS_ERR(irq))
>>>> +		return PTR_ERR(irq);
>>>> +	ite->irq = irq;
>>>> +
>>>> +	/* restore the configuration of the LPI */
>>>> +	ret = update_lpi_config(kvm, irq, NULL);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	update_affinity_ite(kvm, ite);
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>>>> +			    struct list_head *b)
>>>> +{
>>>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
>>>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
>>>> +
>>>> +	if (itea->event_id < iteb->event_id)
>>>> +		return -1;
>>>> +	else
>>>> +		return 1;
>>>> +}
>>>> +
>>>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>> +{
>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>> +	gpa_t base = device->itt_addr;
>>>> +	struct its_ite *ite;
>>>> +	int ret, ite_esz = abi->ite_esz;
>>>
>>> nit: initializations on separate line
>> OK
>>>
>>>> +
>>>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
>>>> +
>>>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
>>>> +		gpa_t gpa = base + ite->event_id * ite_esz;
>>>> +
>>>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
>>>> +		if (ret)
>>>> +			return ret;
>>>> +	}
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>> +{
>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>> +	gpa_t base = dev->itt_addr;
>>>> +	int ret, ite_esz = abi->ite_esz;
>>>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
>>>
>>> nit: initializations on separate line
>> OK
>>>
>>>> +
>>>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
>>>> +			    vgic_its_restore_ite, dev);
>>>
>>> nit: extra white space
>>>
>>>> +
>>>> +	if (ret < 0)
>>>> +		return ret;
>>>> +
>>>> +	/* if the last element has not been found we are in trouble */
>>>> +	return ret ? 0 : -EINVAL;
>>>
>>> hmm, these are values potentially created by the guest in guest RAM,
>>> right?  So do we really abort migration and return an error to userspace
>>> in this case?
>> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
>> such error. The restore table IOCTL will return an error. Up to qemu to
>> print the error. Destination guest will not be functional though.
>>
> 
> ok, I'm just wondering if userspace can make a qualified decision based
> on this error code.  EINVAL typically means that userspace provided
> something incorrect, which I suppose in a sense is true, but this should
> be the only case where we return EINVAL here.
  Userspace must be able to
> tell the cases apart where the guest programmed bogus into memory before
> migration started, in which case we should ignore-and-resume, and where
> QEMU errornously provide some bogus value where the machine state
> becomes unreliable and must be powered down.
guest does not feed much besides few registers the ITS table restore
depends on. In case we want a more subtle error management at userspace
level all the error codes need to be revisited I am afraid. My plan was
to be more rough at the beginning and ignore & resume if ITS table
restore fails.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Auger Eric]

Hi Christoffer,

On 03/05/2017 18:37, Christoffer Dall wrote:
> On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 30/04/2017 22:14, Christoffer Dall wrote:
>>> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
>>>> Introduce routines to save and restore device ITT and their
>>>> interrupt table entries (ITE).
>>>>
>>>> The routines will be called on device table save and
>>>> restore. They will become static in subsequent patches.
>>>
>>> Why this bottom-up approach?  Couldn't you start by having the patch
>>> that restores the device table and define the static functions that
>>> return an error there
>> done
>> , and then fill them in with subsequent patches
>>> (liek this one)?
>>>
>>> That would have the added benefit of being able to tell how things are
>>> designed to be called.
>>>
>>>>
>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>
>>>> ---
>>>> v4 -> v5:
>>>> - ITE are now sorted by eventid on the flush
>>>> - rename *flush* into *save*
>>>> - use macros for shits and masks
>>>> - pass ite_esz to vgic_its_save_ite
>>>>
>>>> v3 -> v4:
>>>> - lookup_table and compute_next_eventid_offset become static in this
>>>>   patch
>>>> - remove static along with vgic_its_flush/restore_itt to avoid
>>>>   compilation warnings
>>>> - next field only computed with a shift (mask removed)
>>>> - handle the case where the last element has not been found
>>>>
>>>> v2 -> v3:
>>>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
>>>>
>>>> v2: creation
>>>> ---
>>>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
>>>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
>>>>  2 files changed, 129 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>> index 35b2ca1..b02fc3f 100644
>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>> @@ -23,6 +23,7 @@
>>>>  #include <linux/interrupt.h>
>>>>  #include <linux/list.h>
>>>>  #include <linux/uaccess.h>
>>>> +#include <linux/list_sort.h>
>>>>  
>>>>  #include <linux/irqchip/arm-gic-v3.h>
>>>>  
>>>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>>>>  }
>>>>  
>>>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>>  {
>>>>  	struct list_head *e = &ite->ite_list;
>>>>  	struct its_ite *next;
>>>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>>>>   *
>>>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
>>>>   */
>>>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>> -		 int start_id, entry_fn_t fn, void *opaque)
>>>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>> +			int start_id, entry_fn_t fn, void *opaque)
>>>>  {
>>>>  	void *entry = kzalloc(esz, GFP_KERNEL);
>>>>  	struct kvm *kvm = its->dev->kvm;
>>>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>  }
>>>>  
>>>>  /**
>>>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
>>>> + */
>>>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
>>>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
>>>> +{
>>>> +	struct kvm *kvm = its->dev->kvm;
>>>> +	u32 next_offset;
>>>> +	u64 val;
>>>> +
>>>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
>>>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
>>>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
>>>> +		ite->collection->collection_id;
>>>> +	val = cpu_to_le64(val);
>>>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
>>>> +}
>>>> +
>>>> +/**
>>>> + * vgic_its_restore_ite - restore an interrupt translation entry
>>>> + * @event_id: id used for indexing
>>>> + * @ptr: pointer to the ITE entry
>>>> + * @opaque: pointer to the its_device
>>>> + * @next: id offset to the next entry
>>>> + */
>>>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
>>>> +				void *ptr, void *opaque, u32 *next)
>>>> +{
>>>> +	struct its_device *dev = (struct its_device *)opaque;
>>>> +	struct its_collection *collection;
>>>> +	struct kvm *kvm = its->dev->kvm;
>>>> +	u64 val, *p = (u64 *)ptr;
>>>
>>> nit: initializations on separate line (and possible do that just above
>>> assigning val).
>> done
>>>
>>>> +	struct vgic_irq *irq;
>>>> +	u32 coll_id, lpi_id;
>>>> +	struct its_ite *ite;
>>>> +	int ret;
>>>> +
>>>> +	val = *p;
>>>> +	*next = 1;
>>>> +
>>>> +	val = le64_to_cpu(val);
>>>> +
>>>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
>>>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
>>>> +
>>>> +	if (!lpi_id)
>>>> +		return 0;
>>>
>>> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
>>> the ID is valid?
>> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
>> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
>> GIC_MIN_LPI cause an -EINVAL error
>>>
>>> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
>>> and PPIs here)
>>
>>>
>>>> +
>>>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
>>>
>>> Don't we need to validate this somehow since it will presumably be used
>>> to forward a pointer somehow by the caller?
>> checked against max number of eventids supported by the device
>>>
>>>> +
>>>> +	collection = find_collection(its, coll_id);
>>>> +	if (!collection)
>>>> +		return -EINVAL;
>>>> +
>>>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
>>>> +				  lpi_id, event_id);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	irq = vgic_add_lpi(kvm, lpi_id);
>>>> +	if (IS_ERR(irq))
>>>> +		return PTR_ERR(irq);
>>>> +	ite->irq = irq;
>>>> +
>>>> +	/* restore the configuration of the LPI */
>>>> +	ret = update_lpi_config(kvm, irq, NULL);
>>>> +	if (ret)
>>>> +		return ret;
>>>> +
>>>> +	update_affinity_ite(kvm, ite);
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>>>> +			    struct list_head *b)
>>>> +{
>>>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
>>>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
>>>> +
>>>> +	if (itea->event_id < iteb->event_id)
>>>> +		return -1;
>>>> +	else
>>>> +		return 1;
>>>> +}
>>>> +
>>>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>> +{
>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>> +	gpa_t base = device->itt_addr;
>>>> +	struct its_ite *ite;
>>>> +	int ret, ite_esz = abi->ite_esz;
>>>
>>> nit: initializations on separate line
>> OK
>>>
>>>> +
>>>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
>>>> +
>>>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
>>>> +		gpa_t gpa = base + ite->event_id * ite_esz;
>>>> +
>>>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
>>>> +		if (ret)
>>>> +			return ret;
>>>> +	}
>>>> +	return 0;
>>>> +}
>>>> +
>>>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>> +{
>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>> +	gpa_t base = dev->itt_addr;
>>>> +	int ret, ite_esz = abi->ite_esz;
>>>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
>>>
>>> nit: initializations on separate line
>> OK
>>>
>>>> +
>>>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
>>>> +			    vgic_its_restore_ite, dev);
>>>
>>> nit: extra white space
>>>
>>>> +
>>>> +	if (ret < 0)
>>>> +		return ret;
>>>> +
>>>> +	/* if the last element has not been found we are in trouble */
>>>> +	return ret ? 0 : -EINVAL;
>>>
>>> hmm, these are values potentially created by the guest in guest RAM,
>>> right?  So do we really abort migration and return an error to userspace
>>> in this case?
>> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
>> such error. The restore table IOCTL will return an error. Up to qemu to
>> print the error. Destination guest will not be functional though.
>>
> 
> ok, I'm just wondering if userspace can make a qualified decision based
> on this error code.  EINVAL typically means that userspace provided
> something incorrect, which I suppose in a sense is true, but this should
> be the only case where we return EINVAL here.
  Userspace must be able to
> tell the cases apart where the guest programmed bogus into memory before
> migration started, in which case we should ignore-and-resume, and where
> QEMU errornously provide some bogus value where the machine state
> becomes unreliable and must be powered down.
guest does not feed much besides few registers the ITS table restore
depends on. In case we want a more subtle error management at userspace
level all the error codes need to be revisited I am afraid. My plan was
to be more rough at the beginning and ignore & resume if ITS table
restore fails.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 21/22] KVM: arm64: vgic-its: Fix pending table sync

[Auger Eric]

Hi,

On 30/04/2017 23:10, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:33PM +0200, Eric Auger wrote:
>> In its_sync_lpi_pending_table() we currently ignore the
>> target_vcpu of the LPIs. We sync the pending bit found in
>> the vcpu pending table even if the LPI is not targeting it.
>>
>> Also in vgic_its_cmd_handle_invall() we are supposed to
>> read the config table data for the LPIs associated to the
>> collection ID. At the moment we refresh all LPI config
>> information.
>>
>> This patch passes a vpcu to vgic_copy_lpi_list() so that
>> this latter returns a snapshot of the LPIs targeting this
>> CPU and only those.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 24 ++++++++++++------------
>>  1 file changed, 12 insertions(+), 12 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 86dfc6c..be848be 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -252,13 +252,13 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
>>  }
>>  
>>  /*
>> - * Create a snapshot of the current LPI list, so that we can enumerate all
>> - * LPIs without holding any lock.
>> - * Returns the array length and puts the kmalloc'ed array into intid_ptr.
>> + * Create a snapshot of the current LPIs targeting @vcpu, so that we can
>> + * enumerate those LPIs without holding any lock.
>> + * Returns their number and puts the kmalloc'ed array into intid_ptr.
>>   */
>> -static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>> +static int vgic_copy_lpi_list(struct kvm_vcpu *vcpu, u32 **intid_ptr)
>>  {
>> -	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>>  	struct vgic_irq *irq;
>>  	u32 *intids;
>>  	int irq_count = dist->lpi_list_count, i = 0;
>> @@ -277,14 +277,14 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>>  	spin_lock(&dist->lpi_list_lock);
>>  	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
>>  		/* We don't need to "get" the IRQ, as we hold the list lock. */
>> -		intids[i] = irq->intid;
>> -		if (++i == irq_count)
>> -			break;
>> +		if (irq->target_vcpu != vcpu)
>> +			continue;
>> +		intids[i++] = irq->intid;
> 
> were we checking the ++i == irq_count condition for no good reason
> before since we can just drop it now?
I didn't get why we had that check.

Thanks

Eric
> 
>>  	}
>>  	spin_unlock(&dist->lpi_list_lock);
>>  
>>  	*intid_ptr = intids;
>> -	return irq_count;
>> +	return i;
>>  }
>>  
>>  /*
>> @@ -333,7 +333,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
>>  }
>>  
>>  /*
>> - * Scan the whole LPI pending table and sync the pending bit in there
>> + * Sync the pending table pending bit of LPIs targeting @vcpu
>>   * with our own data structures. This relies on the LPI being
>>   * mapped before.
>>   */
>> @@ -346,7 +346,7 @@ static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
>>  	u32 *intids;
>>  	int nr_irqs, i;
>>  
>> -	nr_irqs = vgic_copy_lpi_list(vcpu->kvm, &intids);
>> +	nr_irqs = vgic_copy_lpi_list(vcpu, &intids);
>>  	if (nr_irqs < 0)
>>  		return nr_irqs;
>>  
>> @@ -1027,7 +1027,7 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
>>  
>>  	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>>  
>> -	irq_count = vgic_copy_lpi_list(kvm, &intids);
>> +	irq_count = vgic_copy_lpi_list(vcpu, &intids);
>>  	if (irq_count < 0)
>>  		return irq_count;
>>  
>> -- 
>> 2.5.5
>>
> 
> Assuming that it's ok to remove the irq_count check above, the rest of
> this patch looks good to me.
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 21/22] KVM: arm64: vgic-its: Fix pending table sync

[Auger Eric]

Hi,

On 30/04/2017 23:10, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:33PM +0200, Eric Auger wrote:
>> In its_sync_lpi_pending_table() we currently ignore the
>> target_vcpu of the LPIs. We sync the pending bit found in
>> the vcpu pending table even if the LPI is not targeting it.
>>
>> Also in vgic_its_cmd_handle_invall() we are supposed to
>> read the config table data for the LPIs associated to the
>> collection ID. At the moment we refresh all LPI config
>> information.
>>
>> This patch passes a vpcu to vgic_copy_lpi_list() so that
>> this latter returns a snapshot of the LPIs targeting this
>> CPU and only those.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>> ---
>>  virt/kvm/arm/vgic/vgic-its.c | 24 ++++++++++++------------
>>  1 file changed, 12 insertions(+), 12 deletions(-)
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index 86dfc6c..be848be 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -252,13 +252,13 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
>>  }
>>  
>>  /*
>> - * Create a snapshot of the current LPI list, so that we can enumerate all
>> - * LPIs without holding any lock.
>> - * Returns the array length and puts the kmalloc'ed array into intid_ptr.
>> + * Create a snapshot of the current LPIs targeting @vcpu, so that we can
>> + * enumerate those LPIs without holding any lock.
>> + * Returns their number and puts the kmalloc'ed array into intid_ptr.
>>   */
>> -static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>> +static int vgic_copy_lpi_list(struct kvm_vcpu *vcpu, u32 **intid_ptr)
>>  {
>> -	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>>  	struct vgic_irq *irq;
>>  	u32 *intids;
>>  	int irq_count = dist->lpi_list_count, i = 0;
>> @@ -277,14 +277,14 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
>>  	spin_lock(&dist->lpi_list_lock);
>>  	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
>>  		/* We don't need to "get" the IRQ, as we hold the list lock. */
>> -		intids[i] = irq->intid;
>> -		if (++i == irq_count)
>> -			break;
>> +		if (irq->target_vcpu != vcpu)
>> +			continue;
>> +		intids[i++] = irq->intid;
> 
> were we checking the ++i == irq_count condition for no good reason
> before since we can just drop it now?
I didn't get why we had that check.

Thanks

Eric
> 
>>  	}
>>  	spin_unlock(&dist->lpi_list_lock);
>>  
>>  	*intid_ptr = intids;
>> -	return irq_count;
>> +	return i;
>>  }
>>  
>>  /*
>> @@ -333,7 +333,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
>>  }
>>  
>>  /*
>> - * Scan the whole LPI pending table and sync the pending bit in there
>> + * Sync the pending table pending bit of LPIs targeting @vcpu
>>   * with our own data structures. This relies on the LPI being
>>   * mapped before.
>>   */
>> @@ -346,7 +346,7 @@ static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
>>  	u32 *intids;
>>  	int nr_irqs, i;
>>  
>> -	nr_irqs = vgic_copy_lpi_list(vcpu->kvm, &intids);
>> +	nr_irqs = vgic_copy_lpi_list(vcpu, &intids);
>>  	if (nr_irqs < 0)
>>  		return nr_irqs;
>>  
>> @@ -1027,7 +1027,7 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
>>  
>>  	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>>  
>> -	irq_count = vgic_copy_lpi_list(kvm, &intids);
>> +	irq_count = vgic_copy_lpi_list(vcpu, &intids);
>>  	if (irq_count < 0)
>>  		return irq_count;
>>  
>> -- 
>> 2.5.5
>>
> 
> Assuming that it's ok to remove the irq_count check above, the rest of
> this patch looks good to me.
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 22/22] KVM: arm64: vgic-v3: KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES

[Auger Eric]

Hi Christoffer,

On 30/04/2017 23:32, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:34PM +0200, Eric Auger wrote:
>> This patch adds a new attribute to GICV3 KVM device
>> KVM_DEV_ARM_VGIC_GRP_CTRL group. This Allows the userspace to
> 
> nit: allows (lowercase)
> nit: s/the userspace/userspace/
> 
>> flush all GICR pending tables into guest RAM. At the moment
>> we do not offer any restore control as the sync is implicit.
> 
> I would probably remove the last sentence here.
> 
>>
>> As we need the PENDBASER_ADDRESS() in vgic-v3, let's move its
>> definition in the irqchip header. We restore the full length
>> of the field, ie [51:16]. Same for PROPBASER_ADDRESS with full
>> field length of [51:12].
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - move pending table save code/ctrl into VGICv3 instead of ITS
>> - remove useless gpa_t cast
>> - use the same optimization as in its_sync_lpi_pending_table
>>
>> v3 -> v4:
>> - remove the wrong comment about locking
>> - pass kvm struct instead of its handle
>> - add comment about restore method
>> - remove GITR_PENDABASER.PTZ check
>> - continue if target_vcpu == NULL
>> - new locking strategy
>>
>> v1 -> v2:
>> - do not care about the 1st KB which should be zeroed according to
>>   the spec.
>> ---
>>  arch/arm/include/uapi/asm/kvm.h     |  1 +
>>  arch/arm64/include/uapi/asm/kvm.h   |  1 +
>>  include/linux/irqchip/arm-gic-v3.h  |  2 ++
>>  virt/kvm/arm/vgic/vgic-its.c        |  6 ++---
>>  virt/kvm/arm/vgic/vgic-kvm-device.c | 20 ++++++++++++++
>>  virt/kvm/arm/vgic/vgic-v3.c         | 54 +++++++++++++++++++++++++++++++++++++
>>  virt/kvm/arm/vgic/vgic.h            |  1 +
>>  7 files changed, 81 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
>> index 8e6563c..78fe803 100644
>> --- a/arch/arm/include/uapi/asm/kvm.h
>> +++ b/arch/arm/include/uapi/asm/kvm.h
>> @@ -202,6 +202,7 @@ struct kvm_arch_memory_slot {
>>  #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
>>  #define   KVM_DEV_ARM_ITS_SAVE_TABLES		1
>>  #define   KVM_DEV_ARM_ITS_RESTORE_TABLES	2
>> +#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
>>  
>>  /* KVM_IRQ_LINE irq field index values */
>>  #define KVM_ARM_IRQ_TYPE_SHIFT		24
>> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>> index 1e35115..8a3758a 100644
>> --- a/arch/arm64/include/uapi/asm/kvm.h
>> +++ b/arch/arm64/include/uapi/asm/kvm.h
>> @@ -222,6 +222,7 @@ struct kvm_arch_memory_slot {
>>  #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
>>  #define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
>>  #define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
>> +#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
>>  
>>  /* Device Control API on vcpu fd */
>>  #define KVM_ARM_VCPU_PMU_V3_CTRL	0
>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>> index 0c6798c..9d3932f 100644
>> --- a/include/linux/irqchip/arm-gic-v3.h
>> +++ b/include/linux/irqchip/arm-gic-v3.h
>> @@ -158,6 +158,7 @@
>>  #define GICR_PROPBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PROPBASER, INNER, RaWaWb)
>>  
>>  #define GICR_PROPBASER_IDBITS_MASK			(0x1f)
>> +#define GICR_PROPBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 12))
>>  
>>  #define GICR_PENDBASER_SHAREABILITY_SHIFT		(10)
>>  #define GICR_PENDBASER_INNER_CACHEABILITY_SHIFT		(7)
>> @@ -183,6 +184,7 @@
>>  #define GICR_PENDBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PENDBASER, INNER, RaWaWb)
>>  
>>  #define GICR_PENDBASER_PTZ				BIT_ULL(62)
>> +#define GICR_PENDBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 16))
>>  
>>  /*
>>   * Re-Distributor registers, offsets from SGI_base
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index be848be..745c245 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -189,8 +189,6 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>   */
>>  #define BASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
>>  #define CBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
>> -#define PENDBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
>> -#define PROPBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
>>  
>>  #define GIC_LPI_OFFSET 8192
>>  
>> @@ -227,7 +225,7 @@ static struct its_collection *find_collection(struct vgic_its *its, int coll_id)
>>  static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
>>  			     struct kvm_vcpu *filter_vcpu)
>>  {
>> -	u64 propbase = PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
>> +	u64 propbase = GICR_PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
>>  	u8 prop;
>>  	int ret;
>>  
>> @@ -339,7 +337,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
>>   */
>>  static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
>>  {
>> -	gpa_t pendbase = PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
>> +	gpa_t pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
>>  	struct vgic_irq *irq;
>>  	int last_byte_offset = -1;
>>  	int ret = 0;
>> diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c
>> index 859bfa8..d48743c 100644
>> --- a/virt/kvm/arm/vgic/vgic-kvm-device.c
>> +++ b/virt/kvm/arm/vgic/vgic-kvm-device.c
>> @@ -580,6 +580,24 @@ static int vgic_v3_set_attr(struct kvm_device *dev,
>>  		reg = tmp32;
>>  		return vgic_v3_attr_regs_access(dev, attr, &reg, true);
>>  	}
>> +	case KVM_DEV_ARM_VGIC_GRP_CTRL: {
>> +		int ret;
>> +
>> +		switch (attr->attr) {
>> +		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
>> +			mutex_lock(&dev->kvm->lock);
>> +
>> +			if (!lock_all_vcpus(dev->kvm)) {
>> +				mutex_unlock(&dev->kvm->lock);
>> +				return -EBUSY;
>> +			}
>> +			ret = vgic_v3_save_pending_tables(dev->kvm);
>> +			unlock_all_vcpus(dev->kvm);
>> +			mutex_unlock(&dev->kvm->lock);
>> +			return ret;
>> +		}
>> +		break;
>> +	}
>>  	}
>>  	return -ENXIO;
>>  }
>> @@ -658,6 +676,8 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
>>  		switch (attr->attr) {
>>  		case KVM_DEV_ARM_VGIC_CTRL_INIT:
>>  			return 0;
>> +		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
>> +			return 0;
>>  		}
>>  	}
>>  	return -ENXIO;
>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
>> index be0f4c3..1f0977f 100644
>> --- a/virt/kvm/arm/vgic/vgic-v3.c
>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
>> @@ -15,6 +15,7 @@
>>  #include <linux/irqchip/arm-gic-v3.h>
>>  #include <linux/kvm.h>
>>  #include <linux/kvm_host.h>
>> +//#include <linux/bitops.h>
>>  #include <kvm/arm_vgic.h>
>>  #include <asm/kvm_mmu.h>
>>  #include <asm/kvm_asm.h>
>> @@ -252,6 +253,59 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu)
>>  	vgic_v3->vgic_hcr = ICH_HCR_EN;
>>  }
>>  
>> +/**
>> + * vgic_its_save_pending_tables - Save the pending tables into guest RAM
>> + * kvm lock and all vcpu lock must be held
>> + */
>> +int vgic_v3_save_pending_tables(struct kvm *kvm)
>> +{
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	int last_byte_offset = -1;
>> +	struct vgic_irq *irq;
>> +	int ret;
>> +
>> +	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
>> +		int byte_offset, bit_nr;
>> +		struct kvm_vcpu *vcpu;
>> +		gpa_t pendbase, ptr;
>> +		bool stored;
>> +		u8 val;
>> +
>> +		vcpu = irq->target_vcpu;
>> +		if (!vcpu)
>> +			continue;
>> +
>> +		pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
>> +
>> +		byte_offset = irq->intid / BITS_PER_BYTE;
>> +		bit_nr = irq->intid % BITS_PER_BYTE;
>> +		ptr = pendbase + byte_offset;
>> +
>> +		if (byte_offset != last_byte_offset) {
>> +			ret = kvm_read_guest(kvm, ptr, &val, 1);
>> +			if (ret)
>> +				return ret;
>> +			last_byte_offset = byte_offset;
>> +		}
>> +
>> +		stored = val & bit_nr;
> 
> didn't you mean 'stored = val & (1 << bit_nr)' ?
yes thanks!
> 
>> +		if (stored == irq->pending_latch)
>> +			continue;
>> +
>> +		if (irq->pending_latch)
>> +			val |= 1 << bit_nr;
>> +		else
>> +			val &= ~(1 << bit_nr);
>> +
>> +		ret = kvm_write_guest(kvm, ptr, &val, 1);
>> +		if (ret)
>> +			return ret;
>> +	}
> 
> This loop could probably be written a bit more efficiently and
> simplicity by reading the memory one word at a time (and remembering to
> do le64_to_cpu) and then doing something like:
> 
> 		if (irq->pending_latch)
> 			old = __test_and_set_bit(bit_nr, &val);
> 		else
> 			old = __test_and_clear_bit(bit_nr, &val);
> 
> 		if (old != val) {
> 			tmp = cpu_to_le64(val);
> 			ret = kvm_write_guest(kvm, ptr, &tmp,
> 					      sizeof(unsigned long));
> 			if (ret)
> 				retur ret;
> 		}
> 
> Further, you could also detect when the word_offset changes and write
> back the entire word with all its changes then, but you'd also have to
> check at the end of the loop then.  Not sure that's worth the
> optimization or easier to read than what you heave.  It's up to you.
I eventually chose to leave the code as it is today. One of the
rationale is it is pretty similar to the its_sync_lpi_pending_table
function in vgic-its.c.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
>> +
>> +	return 0;
>> +}
>> +
>> +
>>  /* check for overlapping regions and for regions crossing the end of memory */
>>  static bool vgic_v3_check_base(struct kvm *kvm)
>>  {
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index 9bc52ef..535c2fc 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -177,6 +177,7 @@ void vgic_v3_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>>  void vgic_v3_enable(struct kvm_vcpu *vcpu);
>>  int vgic_v3_probe(const struct gic_kvm_info *info);
>>  int vgic_v3_map_resources(struct kvm *kvm);
>> +int vgic_v3_save_pending_tables(struct kvm *kvm);
>>  int vgic_register_redist_iodevs(struct kvm *kvm, gpa_t dist_base_address);
>>  
>>  int vgic_register_its_iodevs(struct kvm *kvm);
>> -- 
>> 2.5.5
>>

[PATCH v5 22/22] KVM: arm64: vgic-v3: KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES

[Auger Eric]

Hi Christoffer,

On 30/04/2017 23:32, Christoffer Dall wrote:
> On Fri, Apr 14, 2017 at 12:15:34PM +0200, Eric Auger wrote:
>> This patch adds a new attribute to GICV3 KVM device
>> KVM_DEV_ARM_VGIC_GRP_CTRL group. This Allows the userspace to
> 
> nit: allows (lowercase)
> nit: s/the userspace/userspace/
> 
>> flush all GICR pending tables into guest RAM. At the moment
>> we do not offer any restore control as the sync is implicit.
> 
> I would probably remove the last sentence here.
> 
>>
>> As we need the PENDBASER_ADDRESS() in vgic-v3, let's move its
>> definition in the irqchip header. We restore the full length
>> of the field, ie [51:16]. Same for PROPBASER_ADDRESS with full
>> field length of [51:12].
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v4 -> v5:
>> - move pending table save code/ctrl into VGICv3 instead of ITS
>> - remove useless gpa_t cast
>> - use the same optimization as in its_sync_lpi_pending_table
>>
>> v3 -> v4:
>> - remove the wrong comment about locking
>> - pass kvm struct instead of its handle
>> - add comment about restore method
>> - remove GITR_PENDABASER.PTZ check
>> - continue if target_vcpu == NULL
>> - new locking strategy
>>
>> v1 -> v2:
>> - do not care about the 1st KB which should be zeroed according to
>>   the spec.
>> ---
>>  arch/arm/include/uapi/asm/kvm.h     |  1 +
>>  arch/arm64/include/uapi/asm/kvm.h   |  1 +
>>  include/linux/irqchip/arm-gic-v3.h  |  2 ++
>>  virt/kvm/arm/vgic/vgic-its.c        |  6 ++---
>>  virt/kvm/arm/vgic/vgic-kvm-device.c | 20 ++++++++++++++
>>  virt/kvm/arm/vgic/vgic-v3.c         | 54 +++++++++++++++++++++++++++++++++++++
>>  virt/kvm/arm/vgic/vgic.h            |  1 +
>>  7 files changed, 81 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
>> index 8e6563c..78fe803 100644
>> --- a/arch/arm/include/uapi/asm/kvm.h
>> +++ b/arch/arm/include/uapi/asm/kvm.h
>> @@ -202,6 +202,7 @@ struct kvm_arch_memory_slot {
>>  #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
>>  #define   KVM_DEV_ARM_ITS_SAVE_TABLES		1
>>  #define   KVM_DEV_ARM_ITS_RESTORE_TABLES	2
>> +#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
>>  
>>  /* KVM_IRQ_LINE irq field index values */
>>  #define KVM_ARM_IRQ_TYPE_SHIFT		24
>> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>> index 1e35115..8a3758a 100644
>> --- a/arch/arm64/include/uapi/asm/kvm.h
>> +++ b/arch/arm64/include/uapi/asm/kvm.h
>> @@ -222,6 +222,7 @@ struct kvm_arch_memory_slot {
>>  #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
>>  #define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
>>  #define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
>> +#define   KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES	3
>>  
>>  /* Device Control API on vcpu fd */
>>  #define KVM_ARM_VCPU_PMU_V3_CTRL	0
>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>> index 0c6798c..9d3932f 100644
>> --- a/include/linux/irqchip/arm-gic-v3.h
>> +++ b/include/linux/irqchip/arm-gic-v3.h
>> @@ -158,6 +158,7 @@
>>  #define GICR_PROPBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PROPBASER, INNER, RaWaWb)
>>  
>>  #define GICR_PROPBASER_IDBITS_MASK			(0x1f)
>> +#define GICR_PROPBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 12))
>>  
>>  #define GICR_PENDBASER_SHAREABILITY_SHIFT		(10)
>>  #define GICR_PENDBASER_INNER_CACHEABILITY_SHIFT		(7)
>> @@ -183,6 +184,7 @@
>>  #define GICR_PENDBASER_RaWaWb	GIC_BASER_CACHEABILITY(GICR_PENDBASER, INNER, RaWaWb)
>>  
>>  #define GICR_PENDBASER_PTZ				BIT_ULL(62)
>> +#define GICR_PENDBASER_ADDRESS(x)    ((x) & GENMASK_ULL(51, 16))
>>  
>>  /*
>>   * Re-Distributor registers, offsets from SGI_base
>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>> index be848be..745c245 100644
>> --- a/virt/kvm/arm/vgic/vgic-its.c
>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>> @@ -189,8 +189,6 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
>>   */
>>  #define BASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
>>  #define CBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
>> -#define PENDBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 16))
>> -#define PROPBASER_ADDRESS(x)	((x) & GENMASK_ULL(47, 12))
>>  
>>  #define GIC_LPI_OFFSET 8192
>>  
>> @@ -227,7 +225,7 @@ static struct its_collection *find_collection(struct vgic_its *its, int coll_id)
>>  static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
>>  			     struct kvm_vcpu *filter_vcpu)
>>  {
>> -	u64 propbase = PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
>> +	u64 propbase = GICR_PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
>>  	u8 prop;
>>  	int ret;
>>  
>> @@ -339,7 +337,7 @@ static u32 max_lpis_propbaser(u64 propbaser)
>>   */
>>  static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
>>  {
>> -	gpa_t pendbase = PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
>> +	gpa_t pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
>>  	struct vgic_irq *irq;
>>  	int last_byte_offset = -1;
>>  	int ret = 0;
>> diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c
>> index 859bfa8..d48743c 100644
>> --- a/virt/kvm/arm/vgic/vgic-kvm-device.c
>> +++ b/virt/kvm/arm/vgic/vgic-kvm-device.c
>> @@ -580,6 +580,24 @@ static int vgic_v3_set_attr(struct kvm_device *dev,
>>  		reg = tmp32;
>>  		return vgic_v3_attr_regs_access(dev, attr, &reg, true);
>>  	}
>> +	case KVM_DEV_ARM_VGIC_GRP_CTRL: {
>> +		int ret;
>> +
>> +		switch (attr->attr) {
>> +		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
>> +			mutex_lock(&dev->kvm->lock);
>> +
>> +			if (!lock_all_vcpus(dev->kvm)) {
>> +				mutex_unlock(&dev->kvm->lock);
>> +				return -EBUSY;
>> +			}
>> +			ret = vgic_v3_save_pending_tables(dev->kvm);
>> +			unlock_all_vcpus(dev->kvm);
>> +			mutex_unlock(&dev->kvm->lock);
>> +			return ret;
>> +		}
>> +		break;
>> +	}
>>  	}
>>  	return -ENXIO;
>>  }
>> @@ -658,6 +676,8 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
>>  		switch (attr->attr) {
>>  		case KVM_DEV_ARM_VGIC_CTRL_INIT:
>>  			return 0;
>> +		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
>> +			return 0;
>>  		}
>>  	}
>>  	return -ENXIO;
>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
>> index be0f4c3..1f0977f 100644
>> --- a/virt/kvm/arm/vgic/vgic-v3.c
>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
>> @@ -15,6 +15,7 @@
>>  #include <linux/irqchip/arm-gic-v3.h>
>>  #include <linux/kvm.h>
>>  #include <linux/kvm_host.h>
>> +//#include <linux/bitops.h>
>>  #include <kvm/arm_vgic.h>
>>  #include <asm/kvm_mmu.h>
>>  #include <asm/kvm_asm.h>
>> @@ -252,6 +253,59 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu)
>>  	vgic_v3->vgic_hcr = ICH_HCR_EN;
>>  }
>>  
>> +/**
>> + * vgic_its_save_pending_tables - Save the pending tables into guest RAM
>> + * kvm lock and all vcpu lock must be held
>> + */
>> +int vgic_v3_save_pending_tables(struct kvm *kvm)
>> +{
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	int last_byte_offset = -1;
>> +	struct vgic_irq *irq;
>> +	int ret;
>> +
>> +	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
>> +		int byte_offset, bit_nr;
>> +		struct kvm_vcpu *vcpu;
>> +		gpa_t pendbase, ptr;
>> +		bool stored;
>> +		u8 val;
>> +
>> +		vcpu = irq->target_vcpu;
>> +		if (!vcpu)
>> +			continue;
>> +
>> +		pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
>> +
>> +		byte_offset = irq->intid / BITS_PER_BYTE;
>> +		bit_nr = irq->intid % BITS_PER_BYTE;
>> +		ptr = pendbase + byte_offset;
>> +
>> +		if (byte_offset != last_byte_offset) {
>> +			ret = kvm_read_guest(kvm, ptr, &val, 1);
>> +			if (ret)
>> +				return ret;
>> +			last_byte_offset = byte_offset;
>> +		}
>> +
>> +		stored = val & bit_nr;
> 
> didn't you mean 'stored = val & (1 << bit_nr)' ?
yes thanks!
> 
>> +		if (stored == irq->pending_latch)
>> +			continue;
>> +
>> +		if (irq->pending_latch)
>> +			val |= 1 << bit_nr;
>> +		else
>> +			val &= ~(1 << bit_nr);
>> +
>> +		ret = kvm_write_guest(kvm, ptr, &val, 1);
>> +		if (ret)
>> +			return ret;
>> +	}
> 
> This loop could probably be written a bit more efficiently and
> simplicity by reading the memory one word at a time (and remembering to
> do le64_to_cpu) and then doing something like:
> 
> 		if (irq->pending_latch)
> 			old = __test_and_set_bit(bit_nr, &val);
> 		else
> 			old = __test_and_clear_bit(bit_nr, &val);
> 
> 		if (old != val) {
> 			tmp = cpu_to_le64(val);
> 			ret = kvm_write_guest(kvm, ptr, &tmp,
> 					      sizeof(unsigned long));
> 			if (ret)
> 				retur ret;
> 		}
> 
> Further, you could also detect when the word_offset changes and write
> back the entire word with all its changes then, but you'd also have to
> check at the end of the loop then.  Not sure that's worth the
> optimization or easier to read than what you heave.  It's up to you.
I eventually chose to leave the code as it is today. One of the
rationale is it is pretty similar to the its_sync_lpi_pending_table
function in vgic-its.c.

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
>> +
>> +	return 0;
>> +}
>> +
>> +
>>  /* check for overlapping regions and for regions crossing the end of memory */
>>  static bool vgic_v3_check_base(struct kvm *kvm)
>>  {
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index 9bc52ef..535c2fc 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -177,6 +177,7 @@ void vgic_v3_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>>  void vgic_v3_enable(struct kvm_vcpu *vcpu);
>>  int vgic_v3_probe(const struct gic_kvm_info *info);
>>  int vgic_v3_map_resources(struct kvm *kvm);
>> +int vgic_v3_save_pending_tables(struct kvm *kvm);
>>  int vgic_register_redist_iodevs(struct kvm *kvm, gpa_t dist_base_address);
>>  
>>  int vgic_register_its_iodevs(struct kvm *kvm);
>> -- 
>> 2.5.5
>>

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Christoffer,

On 27/04/2017 16:45, Christoffer Dall wrote:
> Hi Eric,
> 
> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>> ITS tables into/from memory.
>>>>>>>>
>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> v4 -> v5:
>>>>>>>> - take into account Christoffer's comments
>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>
>>>>>>>> v3 -> v4:
>>>>>>>> - take into account Peter's comments:
>>>>>>>>   - typos
>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>   - add a validity bit in DTE
>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>   - document ABI revision
>>>>>>>> - take into account Andre's comments:
>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>
>>>>>>>> v1 -> v2:
>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>> - use ITE name instead of ITTE
>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>> - mentions LE layout
>>>>>>>> ---
>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>        kvm_device_attr.addr.
>>>>>>>> +
>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>> +
>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>> +
>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>> +
>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>> +
>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>> +
>>>>>>>
>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>
>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>
>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>> restored previously.
>>>>>>
>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>> been done at the very end.
>>>>>
>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>
>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>
>>> Shouldn't restoring the pending tables happen when restoring some
>>> redeistributor state and not anything related to the ITS?
>>
>> Marc wrote:
>> "
>> I don't think you necessarily need a coarse map. When restoring the ITS
>> tables, you can always read the pending bit when creating the LPI
>> structure (it has been written to RAM at save time). Note that we
>> already do something like this in vgic_enable_lpis().
>> "
>>
>> This is currently what is implemented I think. the pending tables are
>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>> also on on ITS table restore
>>
>> The problematic is: Either you know in advance which LPI INTIDare used
>> or you need to parse the whole pending table (possibly using the 1st kB
>> as coarse mapping).
>>
>> If you don't know the LPI INTIDs in advance it is only possible to
>> restore the pending bit of pending LPIs. At that time you would
>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>> ITS ITT you would do the same for those which were not pending. Looks
>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>
>> Otherwise we would need to add another dependency between RDIST pending
>> table restore and ITS table restore but this looks even more weird, no?
>>
>>
> So I just sat down with Andre and Marc and we tried to work through this
> and came up with the best scheme.  I apologize in advance for the
> one-way nature of this e-mail, and I am of course open to discussing the
> following proposal again if you do not agree.
> 
> What I think this document should say, is that the following ordering
> must be followed when restoring the GIC and the ITS:
> 
>   First, restore all guest memory
> 
>   Second, restore ALL redistributors
> 
>   Third, restore the ITS, in the following order:
>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>     2. Restore GITS_CBASER
>     3. Restore all other GITS_ registers, except GITS_CTLR!
>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>     5. Restore GITS_CTLR
> 
> The rationale is that we really want the redistributor and the ITS
> restore to be independent and follow the architecture.  This means that
> our ABI for the redistributor should still work without restoring an ITS
> (if we ever decide to support LPIs for KVM without the ITS).
> 
> In terms of our current implementation this means that vgic_add_lpi()
> should ask the redistributor what the state of the LPI is (priority,
> enabled, pending).  I suggest you do the pending check by adding a
> function called something like vgic_v3_lpi_is_pending() which scans the
> bit in memory, clears the memory bit, and returns the value.  Clearing
> the pending bit in memory when moving it to the struct irq is nice,
> because you then don't have to clear out the entire pending table later
> and we don't keep 'consumed' data lying around.  This change should be
> implemented in its_sync_lpi_pending_table() as well, but note that you
> need never call that function in the normal restore path using this
> design.
> 
> I hope this makes sense.

I am dubious about the above changes at the moment.
its_sync_lpi_pending_table() gets called on GITS_CTLR setting which is
documented to be the last step of the restoration. I wonder why the
above changes cannot be part of another series later on.

Consuming the RAM bit status means we record it in irq->pending_latch so
I guess we should have the irq->pending_latch setting in the same
function as the one that retrieves the bit status in guest RAM. So I
would rename vgic_v3_lpi_is_pending into something like
int vgic_v3_sync_lpi_pending_status(struct kvm *kvm, u32 intid)
Since this covers a single LPI, the removes the byte access optimization
found in its_sync_lpi_pending_table

Also if I understand it correctly this means the sync will be done on
both add_lpi and GITS_CTLR setting

What do you think?

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Christoffer,

On 27/04/2017 16:45, Christoffer Dall wrote:
> Hi Eric,
> 
> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>> ITS tables into/from memory.
>>>>>>>>
>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> v4 -> v5:
>>>>>>>> - take into account Christoffer's comments
>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>
>>>>>>>> v3 -> v4:
>>>>>>>> - take into account Peter's comments:
>>>>>>>>   - typos
>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>   - add a validity bit in DTE
>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>   - document ABI revision
>>>>>>>> - take into account Andre's comments:
>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>
>>>>>>>> v1 -> v2:
>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>> - use ITE name instead of ITTE
>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>> - mentions LE layout
>>>>>>>> ---
>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>        kvm_device_attr.addr.
>>>>>>>> +
>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>> +
>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>> +
>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>> +
>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>> +
>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>> +
>>>>>>>
>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>
>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>
>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>> restored previously.
>>>>>>
>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>> been done at the very end.
>>>>>
>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>
>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>
>>> Shouldn't restoring the pending tables happen when restoring some
>>> redeistributor state and not anything related to the ITS?
>>
>> Marc wrote:
>> "
>> I don't think you necessarily need a coarse map. When restoring the ITS
>> tables, you can always read the pending bit when creating the LPI
>> structure (it has been written to RAM at save time). Note that we
>> already do something like this in vgic_enable_lpis().
>> "
>>
>> This is currently what is implemented I think. the pending tables are
>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>> also on on ITS table restore
>>
>> The problematic is: Either you know in advance which LPI INTIDare used
>> or you need to parse the whole pending table (possibly using the 1st kB
>> as coarse mapping).
>>
>> If you don't know the LPI INTIDs in advance it is only possible to
>> restore the pending bit of pending LPIs. At that time you would
>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>> ITS ITT you would do the same for those which were not pending. Looks
>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>
>> Otherwise we would need to add another dependency between RDIST pending
>> table restore and ITS table restore but this looks even more weird, no?
>>
>>
> So I just sat down with Andre and Marc and we tried to work through this
> and came up with the best scheme.  I apologize in advance for the
> one-way nature of this e-mail, and I am of course open to discussing the
> following proposal again if you do not agree.
> 
> What I think this document should say, is that the following ordering
> must be followed when restoring the GIC and the ITS:
> 
>   First, restore all guest memory
> 
>   Second, restore ALL redistributors
> 
>   Third, restore the ITS, in the following order:
>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>     2. Restore GITS_CBASER
>     3. Restore all other GITS_ registers, except GITS_CTLR!
>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>     5. Restore GITS_CTLR
> 
> The rationale is that we really want the redistributor and the ITS
> restore to be independent and follow the architecture.  This means that
> our ABI for the redistributor should still work without restoring an ITS
> (if we ever decide to support LPIs for KVM without the ITS).
> 
> In terms of our current implementation this means that vgic_add_lpi()
> should ask the redistributor what the state of the LPI is (priority,
> enabled, pending).  I suggest you do the pending check by adding a
> function called something like vgic_v3_lpi_is_pending() which scans the
> bit in memory, clears the memory bit, and returns the value.  Clearing
> the pending bit in memory when moving it to the struct irq is nice,
> because you then don't have to clear out the entire pending table later
> and we don't keep 'consumed' data lying around.  This change should be
> implemented in its_sync_lpi_pending_table() as well, but note that you
> need never call that function in the normal restore path using this
> design.
> 
> I hope this makes sense.

I am dubious about the above changes at the moment.
its_sync_lpi_pending_table() gets called on GITS_CTLR setting which is
documented to be the last step of the restoration. I wonder why the
above changes cannot be part of another series later on.

Consuming the RAM bit status means we record it in irq->pending_latch so
I guess we should have the irq->pending_latch setting in the same
function as the one that retrieves the bit status in guest RAM. So I
would rename vgic_v3_lpi_is_pending into something like
int vgic_v3_sync_lpi_pending_status(struct kvm *kvm, u32 intid)
Since this covers a single LPI, the removes the byte access optimization
found in its_sync_lpi_pending_table

Also if I understand it correctly this means the sync will be done on
both add_lpi and GITS_CTLR setting

What do you think?

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Christoffer Dall]

On Wed, May 03, 2017 at 11:55:34PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 03/05/2017 18:37, Christoffer Dall wrote:
> > On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
> >> Hi Christoffer,
> >>
> >> On 30/04/2017 22:14, Christoffer Dall wrote:
> >>> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
> >>>> Introduce routines to save and restore device ITT and their
> >>>> interrupt table entries (ITE).
> >>>>
> >>>> The routines will be called on device table save and
> >>>> restore. They will become static in subsequent patches.
> >>>
> >>> Why this bottom-up approach?  Couldn't you start by having the patch
> >>> that restores the device table and define the static functions that
> >>> return an error there
> >> done
> >> , and then fill them in with subsequent patches
> >>> (liek this one)?
> >>>
> >>> That would have the added benefit of being able to tell how things are
> >>> designed to be called.
> >>>
> >>>>
> >>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>
> >>>> ---
> >>>> v4 -> v5:
> >>>> - ITE are now sorted by eventid on the flush
> >>>> - rename *flush* into *save*
> >>>> - use macros for shits and masks
> >>>> - pass ite_esz to vgic_its_save_ite
> >>>>
> >>>> v3 -> v4:
> >>>> - lookup_table and compute_next_eventid_offset become static in this
> >>>>   patch
> >>>> - remove static along with vgic_its_flush/restore_itt to avoid
> >>>>   compilation warnings
> >>>> - next field only computed with a shift (mask removed)
> >>>> - handle the case where the last element has not been found
> >>>>
> >>>> v2 -> v3:
> >>>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
> >>>>
> >>>> v2: creation
> >>>> ---
> >>>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
> >>>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
> >>>>  2 files changed, 129 insertions(+), 3 deletions(-)
> >>>>
> >>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >>>> index 35b2ca1..b02fc3f 100644
> >>>> --- a/virt/kvm/arm/vgic/vgic-its.c
> >>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >>>> @@ -23,6 +23,7 @@
> >>>>  #include <linux/interrupt.h>
> >>>>  #include <linux/list.h>
> >>>>  #include <linux/uaccess.h>
> >>>> +#include <linux/list_sort.h>
> >>>>  
> >>>>  #include <linux/irqchip/arm-gic-v3.h>
> >>>>  
> >>>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >>>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >>>>  }
> >>>>  
> >>>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >>>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >>>>  {
> >>>>  	struct list_head *e = &ite->ite_list;
> >>>>  	struct its_ite *next;
> >>>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> >>>>   *
> >>>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
> >>>>   */
> >>>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>> -		 int start_id, entry_fn_t fn, void *opaque)
> >>>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>> +			int start_id, entry_fn_t fn, void *opaque)
> >>>>  {
> >>>>  	void *entry = kzalloc(esz, GFP_KERNEL);
> >>>>  	struct kvm *kvm = its->dev->kvm;
> >>>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>>  }
> >>>>  
> >>>>  /**
> >>>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
> >>>> + */
> >>>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
> >>>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
> >>>> +{
> >>>> +	struct kvm *kvm = its->dev->kvm;
> >>>> +	u32 next_offset;
> >>>> +	u64 val;
> >>>> +
> >>>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
> >>>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
> >>>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
> >>>> +		ite->collection->collection_id;
> >>>> +	val = cpu_to_le64(val);
> >>>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
> >>>> +}
> >>>> +
> >>>> +/**
> >>>> + * vgic_its_restore_ite - restore an interrupt translation entry
> >>>> + * @event_id: id used for indexing
> >>>> + * @ptr: pointer to the ITE entry
> >>>> + * @opaque: pointer to the its_device
> >>>> + * @next: id offset to the next entry
> >>>> + */
> >>>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
> >>>> +				void *ptr, void *opaque, u32 *next)
> >>>> +{
> >>>> +	struct its_device *dev = (struct its_device *)opaque;
> >>>> +	struct its_collection *collection;
> >>>> +	struct kvm *kvm = its->dev->kvm;
> >>>> +	u64 val, *p = (u64 *)ptr;
> >>>
> >>> nit: initializations on separate line (and possible do that just above
> >>> assigning val).
> >> done
> >>>
> >>>> +	struct vgic_irq *irq;
> >>>> +	u32 coll_id, lpi_id;
> >>>> +	struct its_ite *ite;
> >>>> +	int ret;
> >>>> +
> >>>> +	val = *p;
> >>>> +	*next = 1;
> >>>> +
> >>>> +	val = le64_to_cpu(val);
> >>>> +
> >>>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
> >>>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
> >>>> +
> >>>> +	if (!lpi_id)
> >>>> +		return 0;
> >>>
> >>> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
> >>> the ID is valid?
> >> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
> >> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
> >> GIC_MIN_LPI cause an -EINVAL error
> >>>
> >>> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
> >>> and PPIs here)
> >>
> >>>
> >>>> +
> >>>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
> >>>
> >>> Don't we need to validate this somehow since it will presumably be used
> >>> to forward a pointer somehow by the caller?
> >> checked against max number of eventids supported by the device
> >>>
> >>>> +
> >>>> +	collection = find_collection(its, coll_id);
> >>>> +	if (!collection)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
> >>>> +				  lpi_id, event_id);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	irq = vgic_add_lpi(kvm, lpi_id);
> >>>> +	if (IS_ERR(irq))
> >>>> +		return PTR_ERR(irq);
> >>>> +	ite->irq = irq;
> >>>> +
> >>>> +	/* restore the configuration of the LPI */
> >>>> +	ret = update_lpi_config(kvm, irq, NULL);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	update_affinity_ite(kvm, ite);
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
> >>>> +			    struct list_head *b)
> >>>> +{
> >>>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
> >>>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
> >>>> +
> >>>> +	if (itea->event_id < iteb->event_id)
> >>>> +		return -1;
> >>>> +	else
> >>>> +		return 1;
> >>>> +}
> >>>> +
> >>>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >>>> +{
> >>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>>> +	gpa_t base = device->itt_addr;
> >>>> +	struct its_ite *ite;
> >>>> +	int ret, ite_esz = abi->ite_esz;
> >>>
> >>> nit: initializations on separate line
> >> OK
> >>>
> >>>> +
> >>>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
> >>>> +
> >>>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
> >>>> +		gpa_t gpa = base + ite->event_id * ite_esz;
> >>>> +
> >>>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
> >>>> +		if (ret)
> >>>> +			return ret;
> >>>> +	}
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >>>> +{
> >>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>>> +	gpa_t base = dev->itt_addr;
> >>>> +	int ret, ite_esz = abi->ite_esz;
> >>>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
> >>>
> >>> nit: initializations on separate line
> >> OK
> >>>
> >>>> +
> >>>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
> >>>> +			    vgic_its_restore_ite, dev);
> >>>
> >>> nit: extra white space
> >>>
> >>>> +
> >>>> +	if (ret < 0)
> >>>> +		return ret;
> >>>> +
> >>>> +	/* if the last element has not been found we are in trouble */
> >>>> +	return ret ? 0 : -EINVAL;
> >>>
> >>> hmm, these are values potentially created by the guest in guest RAM,
> >>> right?  So do we really abort migration and return an error to userspace
> >>> in this case?
> >> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
> >> such error. The restore table IOCTL will return an error. Up to qemu to
> >> print the error. Destination guest will not be functional though.
> >>
> > 
> > ok, I'm just wondering if userspace can make a qualified decision based
> > on this error code.  EINVAL typically means that userspace provided
> > something incorrect, which I suppose in a sense is true, but this should
> > be the only case where we return EINVAL here.
>   Userspace must be able to
> > tell the cases apart where the guest programmed bogus into memory before
> > migration started, in which case we should ignore-and-resume, and where
> > QEMU errornously provide some bogus value where the machine state
> > becomes unreliable and must be powered down.
> guest does not feed much besides few registers the ITS table restore
> depends on. In case we want a more subtle error management at userspace
> level all the error codes need to be revisited I am afraid. My plan was
> to be more rough at the beginning and ignore & resume if ITS table
> restore fails.
> 

Do we require that the VM is quiesced the entire time between saving the
ITS state to memory and copying all memory over the wire and capturing
all register state?  If so, then an error to restore would be because of
userspace doing something wrong and handling that accordingly is fine.

However, if there is any situation where the guest can by accident
write some incorrect value into RAM where the ITS data structures happen
to be, and the VM is migrated afterwards with the potential result of
just killing the VM, then that's unacceptable, because it's a gross
deviation from how the hardware works, and the migration should be
transparent to the VM.

Thanks,
-Christoffer

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Christoffer Dall]

On Wed, May 03, 2017 at 11:55:34PM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 03/05/2017 18:37, Christoffer Dall wrote:
> > On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
> >> Hi Christoffer,
> >>
> >> On 30/04/2017 22:14, Christoffer Dall wrote:
> >>> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
> >>>> Introduce routines to save and restore device ITT and their
> >>>> interrupt table entries (ITE).
> >>>>
> >>>> The routines will be called on device table save and
> >>>> restore. They will become static in subsequent patches.
> >>>
> >>> Why this bottom-up approach?  Couldn't you start by having the patch
> >>> that restores the device table and define the static functions that
> >>> return an error there
> >> done
> >> , and then fill them in with subsequent patches
> >>> (liek this one)?
> >>>
> >>> That would have the added benefit of being able to tell how things are
> >>> designed to be called.
> >>>
> >>>>
> >>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>
> >>>> ---
> >>>> v4 -> v5:
> >>>> - ITE are now sorted by eventid on the flush
> >>>> - rename *flush* into *save*
> >>>> - use macros for shits and masks
> >>>> - pass ite_esz to vgic_its_save_ite
> >>>>
> >>>> v3 -> v4:
> >>>> - lookup_table and compute_next_eventid_offset become static in this
> >>>>   patch
> >>>> - remove static along with vgic_its_flush/restore_itt to avoid
> >>>>   compilation warnings
> >>>> - next field only computed with a shift (mask removed)
> >>>> - handle the case where the last element has not been found
> >>>>
> >>>> v2 -> v3:
> >>>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
> >>>>
> >>>> v2: creation
> >>>> ---
> >>>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
> >>>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
> >>>>  2 files changed, 129 insertions(+), 3 deletions(-)
> >>>>
> >>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >>>> index 35b2ca1..b02fc3f 100644
> >>>> --- a/virt/kvm/arm/vgic/vgic-its.c
> >>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >>>> @@ -23,6 +23,7 @@
> >>>>  #include <linux/interrupt.h>
> >>>>  #include <linux/list.h>
> >>>>  #include <linux/uaccess.h>
> >>>> +#include <linux/list_sort.h>
> >>>>  
> >>>>  #include <linux/irqchip/arm-gic-v3.h>
> >>>>  
> >>>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >>>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >>>>  }
> >>>>  
> >>>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >>>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >>>>  {
> >>>>  	struct list_head *e = &ite->ite_list;
> >>>>  	struct its_ite *next;
> >>>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> >>>>   *
> >>>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
> >>>>   */
> >>>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>> -		 int start_id, entry_fn_t fn, void *opaque)
> >>>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>> +			int start_id, entry_fn_t fn, void *opaque)
> >>>>  {
> >>>>  	void *entry = kzalloc(esz, GFP_KERNEL);
> >>>>  	struct kvm *kvm = its->dev->kvm;
> >>>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>>  }
> >>>>  
> >>>>  /**
> >>>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
> >>>> + */
> >>>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
> >>>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
> >>>> +{
> >>>> +	struct kvm *kvm = its->dev->kvm;
> >>>> +	u32 next_offset;
> >>>> +	u64 val;
> >>>> +
> >>>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
> >>>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
> >>>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
> >>>> +		ite->collection->collection_id;
> >>>> +	val = cpu_to_le64(val);
> >>>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
> >>>> +}
> >>>> +
> >>>> +/**
> >>>> + * vgic_its_restore_ite - restore an interrupt translation entry
> >>>> + * @event_id: id used for indexing
> >>>> + * @ptr: pointer to the ITE entry
> >>>> + * @opaque: pointer to the its_device
> >>>> + * @next: id offset to the next entry
> >>>> + */
> >>>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
> >>>> +				void *ptr, void *opaque, u32 *next)
> >>>> +{
> >>>> +	struct its_device *dev = (struct its_device *)opaque;
> >>>> +	struct its_collection *collection;
> >>>> +	struct kvm *kvm = its->dev->kvm;
> >>>> +	u64 val, *p = (u64 *)ptr;
> >>>
> >>> nit: initializations on separate line (and possible do that just above
> >>> assigning val).
> >> done
> >>>
> >>>> +	struct vgic_irq *irq;
> >>>> +	u32 coll_id, lpi_id;
> >>>> +	struct its_ite *ite;
> >>>> +	int ret;
> >>>> +
> >>>> +	val = *p;
> >>>> +	*next = 1;
> >>>> +
> >>>> +	val = le64_to_cpu(val);
> >>>> +
> >>>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
> >>>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
> >>>> +
> >>>> +	if (!lpi_id)
> >>>> +		return 0;
> >>>
> >>> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
> >>> the ID is valid?
> >> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
> >> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
> >> GIC_MIN_LPI cause an -EINVAL error
> >>>
> >>> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
> >>> and PPIs here)
> >>
> >>>
> >>>> +
> >>>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
> >>>
> >>> Don't we need to validate this somehow since it will presumably be used
> >>> to forward a pointer somehow by the caller?
> >> checked against max number of eventids supported by the device
> >>>
> >>>> +
> >>>> +	collection = find_collection(its, coll_id);
> >>>> +	if (!collection)
> >>>> +		return -EINVAL;
> >>>> +
> >>>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
> >>>> +				  lpi_id, event_id);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	irq = vgic_add_lpi(kvm, lpi_id);
> >>>> +	if (IS_ERR(irq))
> >>>> +		return PTR_ERR(irq);
> >>>> +	ite->irq = irq;
> >>>> +
> >>>> +	/* restore the configuration of the LPI */
> >>>> +	ret = update_lpi_config(kvm, irq, NULL);
> >>>> +	if (ret)
> >>>> +		return ret;
> >>>> +
> >>>> +	update_affinity_ite(kvm, ite);
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
> >>>> +			    struct list_head *b)
> >>>> +{
> >>>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
> >>>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
> >>>> +
> >>>> +	if (itea->event_id < iteb->event_id)
> >>>> +		return -1;
> >>>> +	else
> >>>> +		return 1;
> >>>> +}
> >>>> +
> >>>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >>>> +{
> >>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>>> +	gpa_t base = device->itt_addr;
> >>>> +	struct its_ite *ite;
> >>>> +	int ret, ite_esz = abi->ite_esz;
> >>>
> >>> nit: initializations on separate line
> >> OK
> >>>
> >>>> +
> >>>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
> >>>> +
> >>>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
> >>>> +		gpa_t gpa = base + ite->event_id * ite_esz;
> >>>> +
> >>>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
> >>>> +		if (ret)
> >>>> +			return ret;
> >>>> +	}
> >>>> +	return 0;
> >>>> +}
> >>>> +
> >>>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >>>> +{
> >>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>>> +	gpa_t base = dev->itt_addr;
> >>>> +	int ret, ite_esz = abi->ite_esz;
> >>>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
> >>>
> >>> nit: initializations on separate line
> >> OK
> >>>
> >>>> +
> >>>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
> >>>> +			    vgic_its_restore_ite, dev);
> >>>
> >>> nit: extra white space
> >>>
> >>>> +
> >>>> +	if (ret < 0)
> >>>> +		return ret;
> >>>> +
> >>>> +	/* if the last element has not been found we are in trouble */
> >>>> +	return ret ? 0 : -EINVAL;
> >>>
> >>> hmm, these are values potentially created by the guest in guest RAM,
> >>> right?  So do we really abort migration and return an error to userspace
> >>> in this case?
> >> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
> >> such error. The restore table IOCTL will return an error. Up to qemu to
> >> print the error. Destination guest will not be functional though.
> >>
> > 
> > ok, I'm just wondering if userspace can make a qualified decision based
> > on this error code.  EINVAL typically means that userspace provided
> > something incorrect, which I suppose in a sense is true, but this should
> > be the only case where we return EINVAL here.
>   Userspace must be able to
> > tell the cases apart where the guest programmed bogus into memory before
> > migration started, in which case we should ignore-and-resume, and where
> > QEMU errornously provide some bogus value where the machine state
> > becomes unreliable and must be powered down.
> guest does not feed much besides few registers the ITS table restore
> depends on. In case we want a more subtle error management at userspace
> level all the error codes need to be revisited I am afraid. My plan was
> to be more rough at the beginning and ignore & resume if ITS table
> restore fails.
> 

Do we require that the VM is quiesced the entire time between saving the
ITS state to memory and copying all memory over the wire and capturing
all register state?  If so, then an error to restore would be because of
userspace doing something wrong and handling that accordingly is fine.

However, if there is any situation where the guest can by accident
write some incorrect value into RAM where the ITS data structures happen
to be, and the VM is migrated afterwards with the potential result of
just killing the VM, then that's unacceptable, because it's a gross
deviation from how the hardware works, and the migration should be
transparent to the VM.

Thanks,
-Christoffer

Re: [PATCH v5 21/22] KVM: arm64: vgic-its: Fix pending table sync

[Christoffer Dall]

On Thu, May 04, 2017 at 12:20:13AM +0200, Auger Eric wrote:
> Hi,
> 
> On 30/04/2017 23:10, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:33PM +0200, Eric Auger wrote:
> >> In its_sync_lpi_pending_table() we currently ignore the
> >> target_vcpu of the LPIs. We sync the pending bit found in
> >> the vcpu pending table even if the LPI is not targeting it.
> >>
> >> Also in vgic_its_cmd_handle_invall() we are supposed to
> >> read the config table data for the LPIs associated to the
> >> collection ID. At the moment we refresh all LPI config
> >> information.
> >>
> >> This patch passes a vpcu to vgic_copy_lpi_list() so that
> >> this latter returns a snapshot of the LPIs targeting this
> >> CPU and only those.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 24 ++++++++++++------------
> >>  1 file changed, 12 insertions(+), 12 deletions(-)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 86dfc6c..be848be 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -252,13 +252,13 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
> >>  }
> >>  
> >>  /*
> >> - * Create a snapshot of the current LPI list, so that we can enumerate all
> >> - * LPIs without holding any lock.
> >> - * Returns the array length and puts the kmalloc'ed array into intid_ptr.
> >> + * Create a snapshot of the current LPIs targeting @vcpu, so that we can
> >> + * enumerate those LPIs without holding any lock.
> >> + * Returns their number and puts the kmalloc'ed array into intid_ptr.
> >>   */
> >> -static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
> >> +static int vgic_copy_lpi_list(struct kvm_vcpu *vcpu, u32 **intid_ptr)
> >>  {
> >> -	struct vgic_dist *dist = &kvm->arch.vgic;
> >> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> >>  	struct vgic_irq *irq;
> >>  	u32 *intids;
> >>  	int irq_count = dist->lpi_list_count, i = 0;
> >> @@ -277,14 +277,14 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
> >>  	spin_lock(&dist->lpi_list_lock);
> >>  	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
> >>  		/* We don't need to "get" the IRQ, as we hold the list lock. */
> >> -		intids[i] = irq->intid;
> >> -		if (++i == irq_count)
> >> -			break;
> >> +		if (irq->target_vcpu != vcpu)
> >> +			continue;
> >> +		intids[i++] = irq->intid;
> > 
> > were we checking the ++i == irq_count condition for no good reason
> > before since we can just drop it now?
> I didn't get why we had that check.
> 

ok, Andre is cc'ed so unless he complains let's just get rid of it.

Thanks,
-Christoffer

[PATCH v5 21/22] KVM: arm64: vgic-its: Fix pending table sync

[Christoffer Dall]

On Thu, May 04, 2017 at 12:20:13AM +0200, Auger Eric wrote:
> Hi,
> 
> On 30/04/2017 23:10, Christoffer Dall wrote:
> > On Fri, Apr 14, 2017 at 12:15:33PM +0200, Eric Auger wrote:
> >> In its_sync_lpi_pending_table() we currently ignore the
> >> target_vcpu of the LPIs. We sync the pending bit found in
> >> the vcpu pending table even if the LPI is not targeting it.
> >>
> >> Also in vgic_its_cmd_handle_invall() we are supposed to
> >> read the config table data for the LPIs associated to the
> >> collection ID. At the moment we refresh all LPI config
> >> information.
> >>
> >> This patch passes a vpcu to vgic_copy_lpi_list() so that
> >> this latter returns a snapshot of the LPIs targeting this
> >> CPU and only those.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >> ---
> >>  virt/kvm/arm/vgic/vgic-its.c | 24 ++++++++++++------------
> >>  1 file changed, 12 insertions(+), 12 deletions(-)
> >>
> >> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >> index 86dfc6c..be848be 100644
> >> --- a/virt/kvm/arm/vgic/vgic-its.c
> >> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >> @@ -252,13 +252,13 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
> >>  }
> >>  
> >>  /*
> >> - * Create a snapshot of the current LPI list, so that we can enumerate all
> >> - * LPIs without holding any lock.
> >> - * Returns the array length and puts the kmalloc'ed array into intid_ptr.
> >> + * Create a snapshot of the current LPIs targeting @vcpu, so that we can
> >> + * enumerate those LPIs without holding any lock.
> >> + * Returns their number and puts the kmalloc'ed array into intid_ptr.
> >>   */
> >> -static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
> >> +static int vgic_copy_lpi_list(struct kvm_vcpu *vcpu, u32 **intid_ptr)
> >>  {
> >> -	struct vgic_dist *dist = &kvm->arch.vgic;
> >> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> >>  	struct vgic_irq *irq;
> >>  	u32 *intids;
> >>  	int irq_count = dist->lpi_list_count, i = 0;
> >> @@ -277,14 +277,14 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
> >>  	spin_lock(&dist->lpi_list_lock);
> >>  	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
> >>  		/* We don't need to "get" the IRQ, as we hold the list lock. */
> >> -		intids[i] = irq->intid;
> >> -		if (++i == irq_count)
> >> -			break;
> >> +		if (irq->target_vcpu != vcpu)
> >> +			continue;
> >> +		intids[i++] = irq->intid;
> > 
> > were we checking the ++i == irq_count condition for no good reason
> > before since we can just drop it now?
> I didn't get why we had that check.
> 

ok, Andre is cc'ed so unless he complains let's just get rid of it.

Thanks,
-Christoffer

Re: [PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Auger Eric]

Hi Christoffer,

On 04/05/2017 09:31, Christoffer Dall wrote:
> On Wed, May 03, 2017 at 11:55:34PM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 03/05/2017 18:37, Christoffer Dall wrote:
>>> On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
>>>> Hi Christoffer,
>>>>
>>>> On 30/04/2017 22:14, Christoffer Dall wrote:
>>>>> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
>>>>>> Introduce routines to save and restore device ITT and their
>>>>>> interrupt table entries (ITE).
>>>>>>
>>>>>> The routines will be called on device table save and
>>>>>> restore. They will become static in subsequent patches.
>>>>>
>>>>> Why this bottom-up approach?  Couldn't you start by having the patch
>>>>> that restores the device table and define the static functions that
>>>>> return an error there
>>>> done
>>>> , and then fill them in with subsequent patches
>>>>> (liek this one)?
>>>>>
>>>>> That would have the added benefit of being able to tell how things are
>>>>> designed to be called.
>>>>>
>>>>>>
>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>
>>>>>> ---
>>>>>> v4 -> v5:
>>>>>> - ITE are now sorted by eventid on the flush
>>>>>> - rename *flush* into *save*
>>>>>> - use macros for shits and masks
>>>>>> - pass ite_esz to vgic_its_save_ite
>>>>>>
>>>>>> v3 -> v4:
>>>>>> - lookup_table and compute_next_eventid_offset become static in this
>>>>>>   patch
>>>>>> - remove static along with vgic_its_flush/restore_itt to avoid
>>>>>>   compilation warnings
>>>>>> - next field only computed with a shift (mask removed)
>>>>>> - handle the case where the last element has not been found
>>>>>>
>>>>>> v2 -> v3:
>>>>>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
>>>>>>
>>>>>> v2: creation
>>>>>> ---
>>>>>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
>>>>>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
>>>>>>  2 files changed, 129 insertions(+), 3 deletions(-)
>>>>>>
>>>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>>>> index 35b2ca1..b02fc3f 100644
>>>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>>>> @@ -23,6 +23,7 @@
>>>>>>  #include <linux/interrupt.h>
>>>>>>  #include <linux/list.h>
>>>>>>  #include <linux/uaccess.h>
>>>>>> +#include <linux/list_sort.h>
>>>>>>  
>>>>>>  #include <linux/irqchip/arm-gic-v3.h>
>>>>>>  
>>>>>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>>>>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>>>>>>  }
>>>>>>  
>>>>>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>>>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>>>>  {
>>>>>>  	struct list_head *e = &ite->ite_list;
>>>>>>  	struct its_ite *next;
>>>>>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>>>>>>   *
>>>>>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
>>>>>>   */
>>>>>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>> -		 int start_id, entry_fn_t fn, void *opaque)
>>>>>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>> +			int start_id, entry_fn_t fn, void *opaque)
>>>>>>  {
>>>>>>  	void *entry = kzalloc(esz, GFP_KERNEL);
>>>>>>  	struct kvm *kvm = its->dev->kvm;
>>>>>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>>  }
>>>>>>  
>>>>>>  /**
>>>>>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
>>>>>> + */
>>>>>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
>>>>>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
>>>>>> +{
>>>>>> +	struct kvm *kvm = its->dev->kvm;
>>>>>> +	u32 next_offset;
>>>>>> +	u64 val;
>>>>>> +
>>>>>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
>>>>>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
>>>>>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
>>>>>> +		ite->collection->collection_id;
>>>>>> +	val = cpu_to_le64(val);
>>>>>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
>>>>>> +}
>>>>>> +
>>>>>> +/**
>>>>>> + * vgic_its_restore_ite - restore an interrupt translation entry
>>>>>> + * @event_id: id used for indexing
>>>>>> + * @ptr: pointer to the ITE entry
>>>>>> + * @opaque: pointer to the its_device
>>>>>> + * @next: id offset to the next entry
>>>>>> + */
>>>>>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
>>>>>> +				void *ptr, void *opaque, u32 *next)
>>>>>> +{
>>>>>> +	struct its_device *dev = (struct its_device *)opaque;
>>>>>> +	struct its_collection *collection;
>>>>>> +	struct kvm *kvm = its->dev->kvm;
>>>>>> +	u64 val, *p = (u64 *)ptr;
>>>>>
>>>>> nit: initializations on separate line (and possible do that just above
>>>>> assigning val).
>>>> done
>>>>>
>>>>>> +	struct vgic_irq *irq;
>>>>>> +	u32 coll_id, lpi_id;
>>>>>> +	struct its_ite *ite;
>>>>>> +	int ret;
>>>>>> +
>>>>>> +	val = *p;
>>>>>> +	*next = 1;
>>>>>> +
>>>>>> +	val = le64_to_cpu(val);
>>>>>> +
>>>>>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
>>>>>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
>>>>>> +
>>>>>> +	if (!lpi_id)
>>>>>> +		return 0;
>>>>>
>>>>> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
>>>>> the ID is valid?
>>>> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
>>>> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
>>>> GIC_MIN_LPI cause an -EINVAL error
>>>>>
>>>>> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
>>>>> and PPIs here)
>>>>
>>>>>
>>>>>> +
>>>>>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
>>>>>
>>>>> Don't we need to validate this somehow since it will presumably be used
>>>>> to forward a pointer somehow by the caller?
>>>> checked against max number of eventids supported by the device
>>>>>
>>>>>> +
>>>>>> +	collection = find_collection(its, coll_id);
>>>>>> +	if (!collection)
>>>>>> +		return -EINVAL;
>>>>>> +
>>>>>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
>>>>>> +				  lpi_id, event_id);
>>>>>> +	if (ret)
>>>>>> +		return ret;
>>>>>> +
>>>>>> +	irq = vgic_add_lpi(kvm, lpi_id);
>>>>>> +	if (IS_ERR(irq))
>>>>>> +		return PTR_ERR(irq);
>>>>>> +	ite->irq = irq;
>>>>>> +
>>>>>> +	/* restore the configuration of the LPI */
>>>>>> +	ret = update_lpi_config(kvm, irq, NULL);
>>>>>> +	if (ret)
>>>>>> +		return ret;
>>>>>> +
>>>>>> +	update_affinity_ite(kvm, ite);
>>>>>> +	return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>>>>>> +			    struct list_head *b)
>>>>>> +{
>>>>>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
>>>>>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
>>>>>> +
>>>>>> +	if (itea->event_id < iteb->event_id)
>>>>>> +		return -1;
>>>>>> +	else
>>>>>> +		return 1;
>>>>>> +}
>>>>>> +
>>>>>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>>>> +{
>>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>>> +	gpa_t base = device->itt_addr;
>>>>>> +	struct its_ite *ite;
>>>>>> +	int ret, ite_esz = abi->ite_esz;
>>>>>
>>>>> nit: initializations on separate line
>>>> OK
>>>>>
>>>>>> +
>>>>>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
>>>>>> +
>>>>>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
>>>>>> +		gpa_t gpa = base + ite->event_id * ite_esz;
>>>>>> +
>>>>>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
>>>>>> +		if (ret)
>>>>>> +			return ret;
>>>>>> +	}
>>>>>> +	return 0;
>>>>>> +}
>>>>>> +
>>>>>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>>>> +{
>>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>>> +	gpa_t base = dev->itt_addr;
>>>>>> +	int ret, ite_esz = abi->ite_esz;
>>>>>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
>>>>>
>>>>> nit: initializations on separate line
>>>> OK
>>>>>
>>>>>> +
>>>>>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
>>>>>> +			    vgic_its_restore_ite, dev);
>>>>>
>>>>> nit: extra white space
>>>>>
>>>>>> +
>>>>>> +	if (ret < 0)
>>>>>> +		return ret;
>>>>>> +
>>>>>> +	/* if the last element has not been found we are in trouble */
>>>>>> +	return ret ? 0 : -EINVAL;
>>>>>
>>>>> hmm, these are values potentially created by the guest in guest RAM,
>>>>> right?  So do we really abort migration and return an error to userspace
>>>>> in this case?
>>>> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
>>>> such error. The restore table IOCTL will return an error. Up to qemu to
>>>> print the error. Destination guest will not be functional though.
>>>>
>>>
>>> ok, I'm just wondering if userspace can make a qualified decision based
>>> on this error code.  EINVAL typically means that userspace provided
>>> something incorrect, which I suppose in a sense is true, but this should
>>> be the only case where we return EINVAL here.
>>   Userspace must be able to
>>> tell the cases apart where the guest programmed bogus into memory before
>>> migration started, in which case we should ignore-and-resume, and where
>>> QEMU errornously provide some bogus value where the machine state
>>> becomes unreliable and must be powered down.
>> guest does not feed much besides few registers the ITS table restore
>> depends on. In case we want a more subtle error management at userspace
>> level all the error codes need to be revisited I am afraid. My plan was
>> to be more rough at the beginning and ignore & resume if ITS table
>> restore fails.
>>
> 
> Do we require that the VM is quiesced the entire time between saving the
> ITS state to memory and copying all memory over the wire and capturing
> all register state?  If so, then an error to restore would be because of
> userspace doing something wrong and handling that accordingly is fine.

yes the ITS table save into RAM starts when we have a guarantee that all
the VCPUS are stopped (we take all locks). The restore happens before
the VM gets resumed. At least this is the QEMU integration as of today.

Thanks

Eric
> 
> However, if there is any situation where the guest can by accident
> write some incorrect value into RAM where the ITS data structures happen
> to be, and the VM is migrated afterwards with the potential result of
> just killing the VM, then that's unacceptable, because it's a gross
> deviation from how the hardware works, and the migration should be
> transparent to the VM.
> 
> Thanks,
> -Christoffer
> 

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Auger Eric]

Hi Christoffer,

On 04/05/2017 09:31, Christoffer Dall wrote:
> On Wed, May 03, 2017 at 11:55:34PM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 03/05/2017 18:37, Christoffer Dall wrote:
>>> On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
>>>> Hi Christoffer,
>>>>
>>>> On 30/04/2017 22:14, Christoffer Dall wrote:
>>>>> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
>>>>>> Introduce routines to save and restore device ITT and their
>>>>>> interrupt table entries (ITE).
>>>>>>
>>>>>> The routines will be called on device table save and
>>>>>> restore. They will become static in subsequent patches.
>>>>>
>>>>> Why this bottom-up approach?  Couldn't you start by having the patch
>>>>> that restores the device table and define the static functions that
>>>>> return an error there
>>>> done
>>>> , and then fill them in with subsequent patches
>>>>> (liek this one)?
>>>>>
>>>>> That would have the added benefit of being able to tell how things are
>>>>> designed to be called.
>>>>>
>>>>>>
>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>
>>>>>> ---
>>>>>> v4 -> v5:
>>>>>> - ITE are now sorted by eventid on the flush
>>>>>> - rename *flush* into *save*
>>>>>> - use macros for shits and masks
>>>>>> - pass ite_esz to vgic_its_save_ite
>>>>>>
>>>>>> v3 -> v4:
>>>>>> - lookup_table and compute_next_eventid_offset become static in this
>>>>>>   patch
>>>>>> - remove static along with vgic_its_flush/restore_itt to avoid
>>>>>>   compilation warnings
>>>>>> - next field only computed with a shift (mask removed)
>>>>>> - handle the case where the last element has not been found
>>>>>>
>>>>>> v2 -> v3:
>>>>>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
>>>>>>
>>>>>> v2: creation
>>>>>> ---
>>>>>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
>>>>>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
>>>>>>  2 files changed, 129 insertions(+), 3 deletions(-)
>>>>>>
>>>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>>>> index 35b2ca1..b02fc3f 100644
>>>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>>>> @@ -23,6 +23,7 @@
>>>>>>  #include <linux/interrupt.h>
>>>>>>  #include <linux/list.h>
>>>>>>  #include <linux/uaccess.h>
>>>>>> +#include <linux/list_sort.h>
>>>>>>  
>>>>>>  #include <linux/irqchip/arm-gic-v3.h>
>>>>>>  
>>>>>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>>>>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>>>>>>  }
>>>>>>  
>>>>>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>>>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>>>>  {
>>>>>>  	struct list_head *e = &ite->ite_list;
>>>>>>  	struct its_ite *next;
>>>>>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>>>>>>   *
>>>>>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
>>>>>>   */
>>>>>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>> -		 int start_id, entry_fn_t fn, void *opaque)
>>>>>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>> +			int start_id, entry_fn_t fn, void *opaque)
>>>>>>  {
>>>>>>  	void *entry = kzalloc(esz, GFP_KERNEL);
>>>>>>  	struct kvm *kvm = its->dev->kvm;
>>>>>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>>  }
>>>>>>  
>>>>>>  /**
>>>>>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
>>>>>> + */
>>>>>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
>>>>>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
>>>>>> +{
>>>>>> +	struct kvm *kvm = its->dev->kvm;
>>>>>> +	u32 next_offset;
>>>>>> +	u64 val;
>>>>>> +
>>>>>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
>>>>>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
>>>>>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
>>>>>> +		ite->collection->collection_id;
>>>>>> +	val = cpu_to_le64(val);
>>>>>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
>>>>>> +}
>>>>>> +
>>>>>> +/**
>>>>>> + * vgic_its_restore_ite - restore an interrupt translation entry
>>>>>> + * @event_id: id used for indexing
>>>>>> + * @ptr: pointer to the ITE entry
>>>>>> + * @opaque: pointer to the its_device
>>>>>> + * @next: id offset to the next entry
>>>>>> + */
>>>>>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
>>>>>> +				void *ptr, void *opaque, u32 *next)
>>>>>> +{
>>>>>> +	struct its_device *dev = (struct its_device *)opaque;
>>>>>> +	struct its_collection *collection;
>>>>>> +	struct kvm *kvm = its->dev->kvm;
>>>>>> +	u64 val, *p = (u64 *)ptr;
>>>>>
>>>>> nit: initializations on separate line (and possible do that just above
>>>>> assigning val).
>>>> done
>>>>>
>>>>>> +	struct vgic_irq *irq;
>>>>>> +	u32 coll_id, lpi_id;
>>>>>> +	struct its_ite *ite;
>>>>>> +	int ret;
>>>>>> +
>>>>>> +	val = *p;
>>>>>> +	*next = 1;
>>>>>> +
>>>>>> +	val = le64_to_cpu(val);
>>>>>> +
>>>>>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
>>>>>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
>>>>>> +
>>>>>> +	if (!lpi_id)
>>>>>> +		return 0;
>>>>>
>>>>> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
>>>>> the ID is valid?
>>>> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
>>>> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
>>>> GIC_MIN_LPI cause an -EINVAL error
>>>>>
>>>>> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
>>>>> and PPIs here)
>>>>
>>>>>
>>>>>> +
>>>>>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
>>>>>
>>>>> Don't we need to validate this somehow since it will presumably be used
>>>>> to forward a pointer somehow by the caller?
>>>> checked against max number of eventids supported by the device
>>>>>
>>>>>> +
>>>>>> +	collection = find_collection(its, coll_id);
>>>>>> +	if (!collection)
>>>>>> +		return -EINVAL;
>>>>>> +
>>>>>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
>>>>>> +				  lpi_id, event_id);
>>>>>> +	if (ret)
>>>>>> +		return ret;
>>>>>> +
>>>>>> +	irq = vgic_add_lpi(kvm, lpi_id);
>>>>>> +	if (IS_ERR(irq))
>>>>>> +		return PTR_ERR(irq);
>>>>>> +	ite->irq = irq;
>>>>>> +
>>>>>> +	/* restore the configuration of the LPI */
>>>>>> +	ret = update_lpi_config(kvm, irq, NULL);
>>>>>> +	if (ret)
>>>>>> +		return ret;
>>>>>> +
>>>>>> +	update_affinity_ite(kvm, ite);
>>>>>> +	return 0;
>>>>>> +}
>>>>>> +
>>>>>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>>>>>> +			    struct list_head *b)
>>>>>> +{
>>>>>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
>>>>>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
>>>>>> +
>>>>>> +	if (itea->event_id < iteb->event_id)
>>>>>> +		return -1;
>>>>>> +	else
>>>>>> +		return 1;
>>>>>> +}
>>>>>> +
>>>>>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>>>> +{
>>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>>> +	gpa_t base = device->itt_addr;
>>>>>> +	struct its_ite *ite;
>>>>>> +	int ret, ite_esz = abi->ite_esz;
>>>>>
>>>>> nit: initializations on separate line
>>>> OK
>>>>>
>>>>>> +
>>>>>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
>>>>>> +
>>>>>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
>>>>>> +		gpa_t gpa = base + ite->event_id * ite_esz;
>>>>>> +
>>>>>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
>>>>>> +		if (ret)
>>>>>> +			return ret;
>>>>>> +	}
>>>>>> +	return 0;
>>>>>> +}
>>>>>> +
>>>>>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>>>> +{
>>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>>> +	gpa_t base = dev->itt_addr;
>>>>>> +	int ret, ite_esz = abi->ite_esz;
>>>>>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
>>>>>
>>>>> nit: initializations on separate line
>>>> OK
>>>>>
>>>>>> +
>>>>>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
>>>>>> +			    vgic_its_restore_ite, dev);
>>>>>
>>>>> nit: extra white space
>>>>>
>>>>>> +
>>>>>> +	if (ret < 0)
>>>>>> +		return ret;
>>>>>> +
>>>>>> +	/* if the last element has not been found we are in trouble */
>>>>>> +	return ret ? 0 : -EINVAL;
>>>>>
>>>>> hmm, these are values potentially created by the guest in guest RAM,
>>>>> right?  So do we really abort migration and return an error to userspace
>>>>> in this case?
>>>> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
>>>> such error. The restore table IOCTL will return an error. Up to qemu to
>>>> print the error. Destination guest will not be functional though.
>>>>
>>>
>>> ok, I'm just wondering if userspace can make a qualified decision based
>>> on this error code.  EINVAL typically means that userspace provided
>>> something incorrect, which I suppose in a sense is true, but this should
>>> be the only case where we return EINVAL here.
>>   Userspace must be able to
>>> tell the cases apart where the guest programmed bogus into memory before
>>> migration started, in which case we should ignore-and-resume, and where
>>> QEMU errornously provide some bogus value where the machine state
>>> becomes unreliable and must be powered down.
>> guest does not feed much besides few registers the ITS table restore
>> depends on. In case we want a more subtle error management at userspace
>> level all the error codes need to be revisited I am afraid. My plan was
>> to be more rough at the beginning and ignore & resume if ITS table
>> restore fails.
>>
> 
> Do we require that the VM is quiesced the entire time between saving the
> ITS state to memory and copying all memory over the wire and capturing
> all register state?  If so, then an error to restore would be because of
> userspace doing something wrong and handling that accordingly is fine.

yes the ITS table save into RAM starts when we have a guarantee that all
the VCPUS are stopped (we take all locks). The restore happens before
the VM gets resumed. At least this is the QEMU integration as of today.

Thanks

Eric
> 
> However, if there is any situation where the guest can by accident
> write some incorrect value into RAM where the ITS data structures happen
> to be, and the VM is migrated afterwards with the potential result of
> just killing the VM, then that's unacceptable, because it's a gross
> deviation from how the hardware works, and the migration should be
> transparent to the VM.
> 
> Thanks,
> -Christoffer
> 

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Marc Zyngier]

On 04/05/17 08:00, Auger Eric wrote:
> Hi Christoffer,
> 
> On 27/04/2017 16:45, Christoffer Dall wrote:
>> Hi Eric,
>>
>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> v4 -> v5:
>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>
>>>>>>>>> v3 -> v4:
>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>   - typos
>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>   - document ABI revision
>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>
>>>>>>>>> v1 -> v2:
>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>> - mentions LE layout
>>>>>>>>> ---
>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>> +
>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>> +
>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>> +
>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>> +
>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>> +
>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>> +
>>>>>>>>
>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>
>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>
>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>> restored previously.
>>>>>>>
>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>> been done at the very end.
>>>>>>
>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>
>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>
>>>> Shouldn't restoring the pending tables happen when restoring some
>>>> redeistributor state and not anything related to the ITS?
>>>
>>> Marc wrote:
>>> "
>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>> tables, you can always read the pending bit when creating the LPI
>>> structure (it has been written to RAM at save time). Note that we
>>> already do something like this in vgic_enable_lpis().
>>> "
>>>
>>> This is currently what is implemented I think. the pending tables are
>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>> also on on ITS table restore
>>>
>>> The problematic is: Either you know in advance which LPI INTIDare used
>>> or you need to parse the whole pending table (possibly using the 1st kB
>>> as coarse mapping).
>>>
>>> If you don't know the LPI INTIDs in advance it is only possible to
>>> restore the pending bit of pending LPIs. At that time you would
>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>> ITS ITT you would do the same for those which were not pending. Looks
>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>
>>> Otherwise we would need to add another dependency between RDIST pending
>>> table restore and ITS table restore but this looks even more weird, no?
>>>
>>>
>> So I just sat down with Andre and Marc and we tried to work through this
>> and came up with the best scheme.  I apologize in advance for the
>> one-way nature of this e-mail, and I am of course open to discussing the
>> following proposal again if you do not agree.
>>
>> What I think this document should say, is that the following ordering
>> must be followed when restoring the GIC and the ITS:
>>
>>   First, restore all guest memory
>>
>>   Second, restore ALL redistributors
>>
>>   Third, restore the ITS, in the following order:
>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>     2. Restore GITS_CBASER
>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>     5. Restore GITS_CTLR
>>
>> The rationale is that we really want the redistributor and the ITS
>> restore to be independent and follow the architecture.  This means that
>> our ABI for the redistributor should still work without restoring an ITS
>> (if we ever decide to support LPIs for KVM without the ITS).
>>
>> In terms of our current implementation this means that vgic_add_lpi()
>> should ask the redistributor what the state of the LPI is (priority,
>> enabled, pending).  I suggest you do the pending check by adding a
>> function called something like vgic_v3_lpi_is_pending() which scans the
>> bit in memory, clears the memory bit, and returns the value.  Clearing
>> the pending bit in memory when moving it to the struct irq is nice,
>> because you then don't have to clear out the entire pending table later
>> and we don't keep 'consumed' data lying around.  This change should be
>> implemented in its_sync_lpi_pending_table() as well, but note that you
>> need never call that function in the normal restore path using this
>> design.
>>
>> I hope this makes sense.
> 
> I am dubious about the above changes at the moment.
> its_sync_lpi_pending_table() gets called on GITS_CTLR setting which is
> documented to be the last step of the restoration. I wonder why the
> above changes cannot be part of another series later on.

I think that's one of the issues. See below.

> Consuming the RAM bit status means we record it in irq->pending_latch so
> I guess we should have the irq->pending_latch setting in the same
> function as the one that retrieves the bit status in guest RAM. So I
> would rename vgic_v3_lpi_is_pending into something like
> int vgic_v3_sync_lpi_pending_status(struct kvm *kvm, u32 intid)
> Since this covers a single LPI, the removes the byte access optimization
> found in its_sync_lpi_pending_table

Well, never mind the optimization. How many LPIs are we restoring in a
typical VM? 10? 1000? That's just one byte access per LPI. Of course,
I'd rather have fewer guest memory accesses, but a restore is an
incredibly rare event, so I'm not too bothered about the extra usec! ;-)

> 
> Also if I understand it correctly this means the sync will be done on
> both add_lpi and GITS_CTLR setting

Why GITS_CTLR? The Enable bit only controls the effect of
GITS_TRANSLATER... I believe that vgic_add_lpi() is the only point where
we should snapshot the pending state.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Marc Zyngier]

On 04/05/17 08:00, Auger Eric wrote:
> Hi Christoffer,
> 
> On 27/04/2017 16:45, Christoffer Dall wrote:
>> Hi Eric,
>>
>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> v4 -> v5:
>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>
>>>>>>>>> v3 -> v4:
>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>   - typos
>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>   - document ABI revision
>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>
>>>>>>>>> v1 -> v2:
>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>> - mentions LE layout
>>>>>>>>> ---
>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>> +
>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>> +
>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>> +
>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>> +
>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>> +
>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>> +
>>>>>>>>
>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>
>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>
>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>> restored previously.
>>>>>>>
>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>> been done at the very end.
>>>>>>
>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>
>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>
>>>> Shouldn't restoring the pending tables happen when restoring some
>>>> redeistributor state and not anything related to the ITS?
>>>
>>> Marc wrote:
>>> "
>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>> tables, you can always read the pending bit when creating the LPI
>>> structure (it has been written to RAM at save time). Note that we
>>> already do something like this in vgic_enable_lpis().
>>> "
>>>
>>> This is currently what is implemented I think. the pending tables are
>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>> also on on ITS table restore
>>>
>>> The problematic is: Either you know in advance which LPI INTIDare used
>>> or you need to parse the whole pending table (possibly using the 1st kB
>>> as coarse mapping).
>>>
>>> If you don't know the LPI INTIDs in advance it is only possible to
>>> restore the pending bit of pending LPIs. At that time you would
>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>> ITS ITT you would do the same for those which were not pending. Looks
>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>
>>> Otherwise we would need to add another dependency between RDIST pending
>>> table restore and ITS table restore but this looks even more weird, no?
>>>
>>>
>> So I just sat down with Andre and Marc and we tried to work through this
>> and came up with the best scheme.  I apologize in advance for the
>> one-way nature of this e-mail, and I am of course open to discussing the
>> following proposal again if you do not agree.
>>
>> What I think this document should say, is that the following ordering
>> must be followed when restoring the GIC and the ITS:
>>
>>   First, restore all guest memory
>>
>>   Second, restore ALL redistributors
>>
>>   Third, restore the ITS, in the following order:
>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>     2. Restore GITS_CBASER
>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>     5. Restore GITS_CTLR
>>
>> The rationale is that we really want the redistributor and the ITS
>> restore to be independent and follow the architecture.  This means that
>> our ABI for the redistributor should still work without restoring an ITS
>> (if we ever decide to support LPIs for KVM without the ITS).
>>
>> In terms of our current implementation this means that vgic_add_lpi()
>> should ask the redistributor what the state of the LPI is (priority,
>> enabled, pending).  I suggest you do the pending check by adding a
>> function called something like vgic_v3_lpi_is_pending() which scans the
>> bit in memory, clears the memory bit, and returns the value.  Clearing
>> the pending bit in memory when moving it to the struct irq is nice,
>> because you then don't have to clear out the entire pending table later
>> and we don't keep 'consumed' data lying around.  This change should be
>> implemented in its_sync_lpi_pending_table() as well, but note that you
>> need never call that function in the normal restore path using this
>> design.
>>
>> I hope this makes sense.
> 
> I am dubious about the above changes at the moment.
> its_sync_lpi_pending_table() gets called on GITS_CTLR setting which is
> documented to be the last step of the restoration. I wonder why the
> above changes cannot be part of another series later on.

I think that's one of the issues. See below.

> Consuming the RAM bit status means we record it in irq->pending_latch so
> I guess we should have the irq->pending_latch setting in the same
> function as the one that retrieves the bit status in guest RAM. So I
> would rename vgic_v3_lpi_is_pending into something like
> int vgic_v3_sync_lpi_pending_status(struct kvm *kvm, u32 intid)
> Since this covers a single LPI, the removes the byte access optimization
> found in its_sync_lpi_pending_table

Well, never mind the optimization. How many LPIs are we restoring in a
typical VM? 10? 1000? That's just one byte access per LPI. Of course,
I'd rather have fewer guest memory accesses, but a restore is an
incredibly rare event, so I'm not too bothered about the extra usec! ;-)

> 
> Also if I understand it correctly this means the sync will be done on
> both add_lpi and GITS_CTLR setting

Why GITS_CTLR? The Enable bit only controls the effect of
GITS_TRANSLATER... I believe that vgic_add_lpi() is the only point where
we should snapshot the pending state.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Thu, May 04, 2017 at 09:00:29AM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 27/04/2017 16:45, Christoffer Dall wrote:
> > Hi Eric,
> > 
> > On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
> >> On 27/04/2017 13:02, Christoffer Dall wrote:
> >>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> >>>> On 27/04/2017 10:57, Christoffer Dall wrote:
> >>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>>>>>> Add description for how to access ITS registers and how to save/restore
> >>>>>>>> ITS tables into/from memory.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> v4 -> v5:
> >>>>>>>> - take into account Christoffer's comments
> >>>>>>>> - pending table save on GICV3 side now
> >>>>>>>>
> >>>>>>>> v3 -> v4:
> >>>>>>>> - take into account Peter's comments:
> >>>>>>>>   - typos
> >>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>>>>>   - add a validity bit in DTE
> >>>>>>>>   - document all fields in CTE and ITE
> >>>>>>>>   - document ABI revision
> >>>>>>>> - take into account Andre's comments:
> >>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>>>>>   - document 64b registers only can be accessed with 64b access
> >>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>>>>>
> >>>>>>>> v1 -> v2:
> >>>>>>>> - DTE and ITE now are 8 bytes
> >>>>>>>> - DTE and ITE now indexed by deviceid/eventid
> >>>>>>>> - use ITE name instead of ITTE
> >>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>>>>>> - mentions LE layout
> >>>>>>>> ---
> >>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>>>>>  1 file changed, 99 insertions(+)
> >>>>>>>>
> >>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> index 6081a5b..b5f010d 100644
> >>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> @@ -32,7 +32,106 @@ Groups:
> >>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>>>>>        request the initialization of the ITS, no additional parameter in
> >>>>>>>>        kvm_device_attr.addr.
> >>>>>>>> +
> >>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>>>>>> +      by the guest in corresponding registers/table entries.
> >>>>>>>> +
> >>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>>>>>> +      are laid out in little endian format as described in the last paragraph.
> >>>>>>>> +
> >>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>>>>>> +
> >>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>>>>>> +
> >>>>>>>> +      The GITS_IIDR read-only register must also be restored before
> >>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>>>>>> +
> >>>>>>>
> >>>>>>> what is the expected sequence of operations.  For example, to restore
> >>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>>>>>
> >>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>> and restore GITS_CTLR (which enables the ITS)?
> >>>>>>
> >>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >>>>>> that the pending table is read. But the whole pending table is not read
> >>>>>> as we only iterate on registered LPIs. So the ITT must have been
> >>>>>> restored previously.
> >>>>>>
> >>>>>> I became aware that the pending table sync is done twice, once in the
> >>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
> >>>>>> leave this order specification, I should be able to remove the sync on
> >>>>>> table restore. This was the original reason why GITS_CTLR restore has
> >>>>>> been done at the very end.
> >>>>>
> >>>>> I'm sorry, I'm a bit confused.  Do we not need
> >>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> >>>>
> >>>> Yes you do. I was talking about the RDIST pending table sync. The save
> >>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> >>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> >>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>> which is not requested I think since GITS_CTLR restore does it already.
> >>>
> >>> Shouldn't restoring the pending tables happen when restoring some
> >>> redeistributor state and not anything related to the ITS?
> >>
> >> Marc wrote:
> >> "
> >> I don't think you necessarily need a coarse map. When restoring the ITS
> >> tables, you can always read the pending bit when creating the LPI
> >> structure (it has been written to RAM at save time). Note that we
> >> already do something like this in vgic_enable_lpis().
> >> "
> >>
> >> This is currently what is implemented I think. the pending tables are
> >> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
> >> also on on ITS table restore
> >>
> >> The problematic is: Either you know in advance which LPI INTIDare used
> >> or you need to parse the whole pending table (possibly using the 1st kB
> >> as coarse mapping).
> >>
> >> If you don't know the LPI INTIDs in advance it is only possible to
> >> restore the pending bit of pending LPIs. At that time you would
> >> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
> >> ITS ITT you would do the same for those which were not pending. Looks
> >> really heavy to me: coarse mapping + dual vgic_add_lpi path.
> >>
> >> Otherwise we would need to add another dependency between RDIST pending
> >> table restore and ITS table restore but this looks even more weird, no?
> >>
> >>
> > So I just sat down with Andre and Marc and we tried to work through this
> > and came up with the best scheme.  I apologize in advance for the
> > one-way nature of this e-mail, and I am of course open to discussing the
> > following proposal again if you do not agree.
> > 
> > What I think this document should say, is that the following ordering
> > must be followed when restoring the GIC and the ITS:
> > 
> >   First, restore all guest memory
> > 
> >   Second, restore ALL redistributors
> > 
> >   Third, restore the ITS, in the following order:
> >     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >     2. Restore GITS_CBASER
> >     3. Restore all other GITS_ registers, except GITS_CTLR!
> >     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> >     5. Restore GITS_CTLR
> > 
> > The rationale is that we really want the redistributor and the ITS
> > restore to be independent and follow the architecture.  This means that
> > our ABI for the redistributor should still work without restoring an ITS
> > (if we ever decide to support LPIs for KVM without the ITS).
> > 
> > In terms of our current implementation this means that vgic_add_lpi()
> > should ask the redistributor what the state of the LPI is (priority,
> > enabled, pending).  I suggest you do the pending check by adding a
> > function called something like vgic_v3_lpi_is_pending() which scans the
> > bit in memory, clears the memory bit, and returns the value.  Clearing
> > the pending bit in memory when moving it to the struct irq is nice,
> > because you then don't have to clear out the entire pending table later
> > and we don't keep 'consumed' data lying around.  This change should be
> > implemented in its_sync_lpi_pending_table() as well, but note that you
> > need never call that function in the normal restore path using this
> > design.
> > 
> > I hope this makes sense.
> 
> I am dubious about the above changes at the moment.
> its_sync_lpi_pending_table() gets called on GITS_CTLR setting which is
> documented to be the last step of the restoration. I wonder why the
> above changes cannot be part of another series later on.

I suppose they could be.  I just really dislike introducing wording in
the ABI which implies that state that belongs to the redistributor gets
restored when you restore a certain ITS register; this gives the wrong
impression of how the hardware works and it is going to bite us when we
introduce LPI support without the ITS.

Implementation wise, I don't care strongly if you wish to scan the table
multiple times or just lookup the pending bit when you have to.  I find
it much cleaner to emulate what the hardware would be doing which is the
ITS asking the redistributor for the pending state, and the
redistributor is in charge of maintaining consistency between memory an
cached state.

I don't think it's weird to introduce a dependency between the
redistributor and the ITS in the restore sequence, because the ITS is
inherently built to be able to interface with the redistributor, and I
don't see any way around it, given you need to know the address of the
pending table anyway.

> 
> Consuming the RAM bit status means we record it in irq->pending_latch so
> I guess we should have the irq->pending_latch setting in the same
> function as the one that retrieves the bit status in guest RAM.

Yes.

> So I
> would rename vgic_v3_lpi_is_pending into something like
> int vgic_v3_sync_lpi_pending_status(struct kvm *kvm, u32 intid)

no objection

> Since this covers a single LPI, the removes the byte access optimization
> found in its_sync_lpi_pending_table

sure

> 
> Also if I understand it correctly this means the sync will be done on
> both add_lpi and GITS_CTLR setting
> 
No, it would only be done when needing the information for a specific
LPI; there should be no real need to scan the entire table until we have
support for LPIs without the ITS in which case the table must be scanned
when restoring an enabled redistributor.

Thanks,
-Christoffer

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Christoffer Dall]

On Thu, May 04, 2017 at 09:00:29AM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 27/04/2017 16:45, Christoffer Dall wrote:
> > Hi Eric,
> > 
> > On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
> >> On 27/04/2017 13:02, Christoffer Dall wrote:
> >>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
> >>>> On 27/04/2017 10:57, Christoffer Dall wrote:
> >>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
> >>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
> >>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
> >>>>>>>> Add description for how to access ITS registers and how to save/restore
> >>>>>>>> ITS tables into/from memory.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> v4 -> v5:
> >>>>>>>> - take into account Christoffer's comments
> >>>>>>>> - pending table save on GICV3 side now
> >>>>>>>>
> >>>>>>>> v3 -> v4:
> >>>>>>>> - take into account Peter's comments:
> >>>>>>>>   - typos
> >>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >>>>>>>>   - add a validity bit in DTE
> >>>>>>>>   - document all fields in CTE and ITE
> >>>>>>>>   - document ABI revision
> >>>>>>>> - take into account Andre's comments:
> >>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
> >>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
> >>>>>>>>   - document 64b registers only can be accessed with 64b access
> >>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
> >>>>>>>>
> >>>>>>>> v1 -> v2:
> >>>>>>>> - DTE and ITE now are 8 bytes
> >>>>>>>> - DTE and ITE now indexed by deviceid/eventid
> >>>>>>>> - use ITE name instead of ITTE
> >>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
> >>>>>>>> - mentions LE layout
> >>>>>>>> ---
> >>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
> >>>>>>>>  1 file changed, 99 insertions(+)
> >>>>>>>>
> >>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> index 6081a5b..b5f010d 100644
> >>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >>>>>>>> @@ -32,7 +32,106 @@ Groups:
> >>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
> >>>>>>>>        request the initialization of the ITS, no additional parameter in
> >>>>>>>>        kvm_device_attr.addr.
> >>>>>>>> +
> >>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
> >>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
> >>>>>>>> +      by the guest in corresponding registers/table entries.
> >>>>>>>> +
> >>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
> >>>>>>>> +      are laid out in little endian format as described in the last paragraph.
> >>>>>>>> +
> >>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
> >>>>>>>> +
> >>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
> >>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
> >>>>>>>> +
> >>>>>>>> +      The GITS_IIDR read-only register must also be restored before
> >>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
> >>>>>>>> +
> >>>>>>>
> >>>>>>> what is the expected sequence of operations.  For example, to restore
> >>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
> >>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
> >>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
> >>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
> >>>>>>>
> >>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>>>>> and restore GITS_CTLR (which enables the ITS)?
> >>>>>>
> >>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
> >>>>>> that the pending table is read. But the whole pending table is not read
> >>>>>> as we only iterate on registered LPIs. So the ITT must have been
> >>>>>> restored previously.
> >>>>>>
> >>>>>> I became aware that the pending table sync is done twice, once in the
> >>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
> >>>>>> leave this order specification, I should be able to remove the sync on
> >>>>>> table restore. This was the original reason why GITS_CTLR restore has
> >>>>>> been done at the very end.
> >>>>>
> >>>>> I'm sorry, I'm a bit confused.  Do we not need
> >>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
> >>>>
> >>>> Yes you do. I was talking about the RDIST pending table sync. The save
> >>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
> >>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
> >>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
> >>>> which is not requested I think since GITS_CTLR restore does it already.
> >>>
> >>> Shouldn't restoring the pending tables happen when restoring some
> >>> redeistributor state and not anything related to the ITS?
> >>
> >> Marc wrote:
> >> "
> >> I don't think you necessarily need a coarse map. When restoring the ITS
> >> tables, you can always read the pending bit when creating the LPI
> >> structure (it has been written to RAM at save time). Note that we
> >> already do something like this in vgic_enable_lpis().
> >> "
> >>
> >> This is currently what is implemented I think. the pending tables are
> >> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
> >> also on on ITS table restore
> >>
> >> The problematic is: Either you know in advance which LPI INTIDare used
> >> or you need to parse the whole pending table (possibly using the 1st kB
> >> as coarse mapping).
> >>
> >> If you don't know the LPI INTIDs in advance it is only possible to
> >> restore the pending bit of pending LPIs. At that time you would
> >> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
> >> ITS ITT you would do the same for those which were not pending. Looks
> >> really heavy to me: coarse mapping + dual vgic_add_lpi path.
> >>
> >> Otherwise we would need to add another dependency between RDIST pending
> >> table restore and ITS table restore but this looks even more weird, no?
> >>
> >>
> > So I just sat down with Andre and Marc and we tried to work through this
> > and came up with the best scheme.  I apologize in advance for the
> > one-way nature of this e-mail, and I am of course open to discussing the
> > following proposal again if you do not agree.
> > 
> > What I think this document should say, is that the following ordering
> > must be followed when restoring the GIC and the ITS:
> > 
> >   First, restore all guest memory
> > 
> >   Second, restore ALL redistributors
> > 
> >   Third, restore the ITS, in the following order:
> >     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >     2. Restore GITS_CBASER
> >     3. Restore all other GITS_ registers, except GITS_CTLR!
> >     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> >     5. Restore GITS_CTLR
> > 
> > The rationale is that we really want the redistributor and the ITS
> > restore to be independent and follow the architecture.  This means that
> > our ABI for the redistributor should still work without restoring an ITS
> > (if we ever decide to support LPIs for KVM without the ITS).
> > 
> > In terms of our current implementation this means that vgic_add_lpi()
> > should ask the redistributor what the state of the LPI is (priority,
> > enabled, pending).  I suggest you do the pending check by adding a
> > function called something like vgic_v3_lpi_is_pending() which scans the
> > bit in memory, clears the memory bit, and returns the value.  Clearing
> > the pending bit in memory when moving it to the struct irq is nice,
> > because you then don't have to clear out the entire pending table later
> > and we don't keep 'consumed' data lying around.  This change should be
> > implemented in its_sync_lpi_pending_table() as well, but note that you
> > need never call that function in the normal restore path using this
> > design.
> > 
> > I hope this makes sense.
> 
> I am dubious about the above changes at the moment.
> its_sync_lpi_pending_table() gets called on GITS_CTLR setting which is
> documented to be the last step of the restoration. I wonder why the
> above changes cannot be part of another series later on.

I suppose they could be.  I just really dislike introducing wording in
the ABI which implies that state that belongs to the redistributor gets
restored when you restore a certain ITS register; this gives the wrong
impression of how the hardware works and it is going to bite us when we
introduce LPI support without the ITS.

Implementation wise, I don't care strongly if you wish to scan the table
multiple times or just lookup the pending bit when you have to.  I find
it much cleaner to emulate what the hardware would be doing which is the
ITS asking the redistributor for the pending state, and the
redistributor is in charge of maintaining consistency between memory an
cached state.

I don't think it's weird to introduce a dependency between the
redistributor and the ITS in the restore sequence, because the ITS is
inherently built to be able to interface with the redistributor, and I
don't see any way around it, given you need to know the address of the
pending table anyway.

> 
> Consuming the RAM bit status means we record it in irq->pending_latch so
> I guess we should have the irq->pending_latch setting in the same
> function as the one that retrieves the bit status in guest RAM.

Yes.

> So I
> would rename vgic_v3_lpi_is_pending into something like
> int vgic_v3_sync_lpi_pending_status(struct kvm *kvm, u32 intid)

no objection

> Since this covers a single LPI, the removes the byte access optimization
> found in its_sync_lpi_pending_table

sure

> 
> Also if I understand it correctly this means the sync will be done on
> both add_lpi and GITS_CTLR setting
> 
No, it would only be done when needing the information for a specific
LPI; there should be no real need to scan the entire table until we have
support for LPIs without the ITS in which case the table must be scanned
when restoring an enabled redistributor.

Thanks,
-Christoffer

Re: [PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Marc,
On 04/05/2017 09:40, Marc Zyngier wrote:
> On 04/05/17 08:00, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 27/04/2017 16:45, Christoffer Dall wrote:
>>> Hi Eric,
>>>
>>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> v4 -> v5:
>>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>>
>>>>>>>>>> v3 -> v4:
>>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>>   - typos
>>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>>   - document ABI revision
>>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>>
>>>>>>>>>> v1 -> v2:
>>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>>> - mentions LE layout
>>>>>>>>>> ---
>>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>>> +
>>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>>> +
>>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>>> +
>>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>>
>>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>>
>>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>>> restored previously.
>>>>>>>>
>>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>>> been done at the very end.
>>>>>>>
>>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>>
>>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>>
>>>>> Shouldn't restoring the pending tables happen when restoring some
>>>>> redeistributor state and not anything related to the ITS?
>>>>
>>>> Marc wrote:
>>>> "
>>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>>> tables, you can always read the pending bit when creating the LPI
>>>> structure (it has been written to RAM at save time). Note that we
>>>> already do something like this in vgic_enable_lpis().
>>>> "
>>>>
>>>> This is currently what is implemented I think. the pending tables are
>>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>>> also on on ITS table restore
>>>>
>>>> The problematic is: Either you know in advance which LPI INTIDare used
>>>> or you need to parse the whole pending table (possibly using the 1st kB
>>>> as coarse mapping).
>>>>
>>>> If you don't know the LPI INTIDs in advance it is only possible to
>>>> restore the pending bit of pending LPIs. At that time you would
>>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>>> ITS ITT you would do the same for those which were not pending. Looks
>>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>>
>>>> Otherwise we would need to add another dependency between RDIST pending
>>>> table restore and ITS table restore but this looks even more weird, no?
>>>>
>>>>
>>> So I just sat down with Andre and Marc and we tried to work through this
>>> and came up with the best scheme.  I apologize in advance for the
>>> one-way nature of this e-mail, and I am of course open to discussing the
>>> following proposal again if you do not agree.
>>>
>>> What I think this document should say, is that the following ordering
>>> must be followed when restoring the GIC and the ITS:
>>>
>>>   First, restore all guest memory
>>>
>>>   Second, restore ALL redistributors
>>>
>>>   Third, restore the ITS, in the following order:
>>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>>     2. Restore GITS_CBASER
>>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>>     5. Restore GITS_CTLR
>>>
>>> The rationale is that we really want the redistributor and the ITS
>>> restore to be independent and follow the architecture.  This means that
>>> our ABI for the redistributor should still work without restoring an ITS
>>> (if we ever decide to support LPIs for KVM without the ITS).
>>>
>>> In terms of our current implementation this means that vgic_add_lpi()
>>> should ask the redistributor what the state of the LPI is (priority,
>>> enabled, pending).  I suggest you do the pending check by adding a
>>> function called something like vgic_v3_lpi_is_pending() which scans the
>>> bit in memory, clears the memory bit, and returns the value.  Clearing
>>> the pending bit in memory when moving it to the struct irq is nice,
>>> because you then don't have to clear out the entire pending table later
>>> and we don't keep 'consumed' data lying around.  This change should be
>>> implemented in its_sync_lpi_pending_table() as well, but note that you
>>> need never call that function in the normal restore path using this
>>> design.
>>>
>>> I hope this makes sense.
>>
>> I am dubious about the above changes at the moment.
>> its_sync_lpi_pending_table() gets called on GITS_CTLR setting which is
>> documented to be the last step of the restoration. I wonder why the
>> above changes cannot be part of another series later on.
> 
> I think that's one of the issues. See below.
> 
>> Consuming the RAM bit status means we record it in irq->pending_latch so
>> I guess we should have the irq->pending_latch setting in the same
>> function as the one that retrieves the bit status in guest RAM. So I
>> would rename vgic_v3_lpi_is_pending into something like
>> int vgic_v3_sync_lpi_pending_status(struct kvm *kvm, u32 intid)
>> Since this covers a single LPI, the removes the byte access optimization
>> found in its_sync_lpi_pending_table
> 
> Well, never mind the optimization. How many LPIs are we restoring in a
> typical VM? 10? 1000? That's just one byte access per LPI. Of course,
> I'd rather have fewer guest memory accesses, but a restore is an
> incredibly rare event, so I'm not too bothered about the extra usec! ;-)
> 
>>
>> Also if I understand it correctly this means the sync will be done on
>> both add_lpi and GITS_CTLR setting
> 
> Why GITS_CTLR? The Enable bit only controls the effect of
> GITS_TRANSLATER...

Hum sorry I mixed up. the sync is currently done on GIC*R*_CTLR
vgic_mmio_write_v3r_ctlr/vgic_enable_lpis/its_sync_lpi_pending_table

As the redistributors are restored *before* the ITS this sync is void as
no LPI exist at that time.

That's why I did the sync (again) on ITS table restore. Sorry for the
noise.

OK let's go with the sync in vgic_add_lpi() ...

Thanks

Eric

 I believe that vgic_add_lpi() is the only point where
> we should snapshot the pending state.
> 
> Thanks,
> 
> 	M.
> 

[PATCH v5 01/22] KVM: arm/arm64: Add ITS save/restore API documentation

[Auger Eric]

Hi Marc,
On 04/05/2017 09:40, Marc Zyngier wrote:
> On 04/05/17 08:00, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 27/04/2017 16:45, Christoffer Dall wrote:
>>> Hi Eric,
>>>
>>> On Thu, Apr 27, 2017 at 02:51:00PM +0200, Auger Eric wrote:
>>>> On 27/04/2017 13:02, Christoffer Dall wrote:
>>>>> On Thu, Apr 27, 2017 at 11:33:39AM +0200, Auger Eric wrote:
>>>>>> On 27/04/2017 10:57, Christoffer Dall wrote:
>>>>>>> On Wed, Apr 26, 2017 at 05:48:32PM +0200, Auger Eric wrote:
>>>>>>>> On 26/04/2017 14:31, Christoffer Dall wrote:
>>>>>>>>> On Fri, Apr 14, 2017 at 12:15:13PM +0200, Eric Auger wrote:
>>>>>>>>>> Add description for how to access ITS registers and how to save/restore
>>>>>>>>>> ITS tables into/from memory.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> v4 -> v5:
>>>>>>>>>> - take into account Christoffer's comments
>>>>>>>>>> - pending table save on GICV3 side now
>>>>>>>>>>
>>>>>>>>>> v3 -> v4:
>>>>>>>>>> - take into account Peter's comments:
>>>>>>>>>>   - typos
>>>>>>>>>>   - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
>>>>>>>>>>   - add a validity bit in DTE
>>>>>>>>>>   - document all fields in CTE and ITE
>>>>>>>>>>   - document ABI revision
>>>>>>>>>> - take into account Andre's comments:
>>>>>>>>>>   - document restrictions about GITS_CREADR writing and GITS_IIDR
>>>>>>>>>>   - document -EBUSY error if one or more VCPUS are runnning
>>>>>>>>>>   - document 64b registers only can be accessed with 64b access
>>>>>>>>>> - itt_addr field matches bits [51:8] of the itt_addr
>>>>>>>>>>
>>>>>>>>>> v1 -> v2:
>>>>>>>>>> - DTE and ITE now are 8 bytes
>>>>>>>>>> - DTE and ITE now indexed by deviceid/eventid
>>>>>>>>>> - use ITE name instead of ITTE
>>>>>>>>>> - mentions ITT_addr matches bits [51:8] of the actual address
>>>>>>>>>> - mentions LE layout
>>>>>>>>>> ---
>>>>>>>>>>  Documentation/virtual/kvm/devices/arm-vgic-its.txt | 99 ++++++++++++++++++++++
>>>>>>>>>>  1 file changed, 99 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> index 6081a5b..b5f010d 100644
>>>>>>>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
>>>>>>>>>> @@ -32,7 +32,106 @@ Groups:
>>>>>>>>>>      KVM_DEV_ARM_VGIC_CTRL_INIT
>>>>>>>>>>        request the initialization of the ITS, no additional parameter in
>>>>>>>>>>        kvm_device_attr.addr.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_SAVE_TABLES
>>>>>>>>>> +      save the ITS table data into guest RAM, at the location provisioned
>>>>>>>>>> +      by the guest in corresponding registers/table entries.
>>>>>>>>>> +
>>>>>>>>>> +      The layout of the tables in guest memory defines an ABI. The entries
>>>>>>>>>> +      are laid out in little endian format as described in the last paragraph.
>>>>>>>>>> +
>>>>>>>>>> +    KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>>> +      restore the ITS tables from guest RAM to ITS internal structures.
>>>>>>>>>> +
>>>>>>>>>> +      The GICV3 must be restored before the ITS and all ITS registers but
>>>>>>>>>> +      the GITS_CTLR must be restored before restoring the ITS tables.
>>>>>>>>>> +
>>>>>>>>>> +      The GITS_IIDR read-only register must also be restored before
>>>>>>>>>> +      the table restore as the IIDR revision field encodes the ABI revision.
>>>>>>>>>> +
>>>>>>>>>
>>>>>>>>> what is the expected sequence of operations.  For example, to restore
>>>>>>>>> the ITS, do I call KVM_DEV_ARM_VGIC_CTRL_INIT first, then restore all
>>>>>>>>> the memory and registers, and finally call KVM_DEV_ARM_ITS_RESTORE_TABLES?
>>>>>>>> Yes KVM_DEV_ARM_VGIC_CTRL_INIT comes first, then restore all registers
>>>>>>>> except GITS_CTLR, then table restore, then GITS_CTLR
>>>>>>>>>
>>>>>>>>> Is there any interaction between when you call KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>>>>> and restore GITS_CTLR (which enables the ITS)?
>>>>>>>>
>>>>>>>> Yep, when GITS_CTLR is set, LPIs may be enabled and this on that event
>>>>>>>> that the pending table is read. But the whole pending table is not read
>>>>>>>> as we only iterate on registered LPIs. So the ITT must have been
>>>>>>>> restored previously.
>>>>>>>>
>>>>>>>> I became aware that the pending table sync is done twice, once in the
>>>>>>>> pending table restore,  and once in the GITS_CTLR restore. So if we
>>>>>>>> leave this order specification, I should be able to remove the sync on
>>>>>>>> table restore. This was the original reason why GITS_CTLR restore has
>>>>>>>> been done at the very end.
>>>>>>>
>>>>>>> I'm sorry, I'm a bit confused.  Do we not need
>>>>>>> KVM_DEV_ARM_ITS_RESTORE_TABLES at all then?
>>>>>>
>>>>>> Yes you do. I was talking about the RDIST pending table sync. The save
>>>>>> is explicit using GICV3 device KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES.
>>>>>> However the sync is implicit on GITS_CTLR restore if LPIs are enabled.
>>>>>> and today I do it also on ITS device KVM_DEV_ARM_ITS_RESTORE_TABLES
>>>>>> which is not requested I think since GITS_CTLR restore does it already.
>>>>>
>>>>> Shouldn't restoring the pending tables happen when restoring some
>>>>> redeistributor state and not anything related to the ITS?
>>>>
>>>> Marc wrote:
>>>> "
>>>> I don't think you necessarily need a coarse map. When restoring the ITS
>>>> tables, you can always read the pending bit when creating the LPI
>>>> structure (it has been written to RAM at save time). Note that we
>>>> already do something like this in vgic_enable_lpis().
>>>> "
>>>>
>>>> This is currently what is implemented I think. the pending tables are
>>>> currently sync'ed on GITS_CTLR set (if LPI are enabled) + erroneously
>>>> also on on ITS table restore
>>>>
>>>> The problematic is: Either you know in advance which LPI INTIDare used
>>>> or you need to parse the whole pending table (possibly using the 1st kB
>>>> as coarse mapping).
>>>>
>>>> If you don't know the LPI INTIDs in advance it is only possible to
>>>> restore the pending bit of pending LPIs. At that time you would
>>>> re-allocate those pending LPI (vgic_add_lpi) and when you restore the
>>>> ITS ITT you would do the same for those which were not pending. Looks
>>>> really heavy to me: coarse mapping + dual vgic_add_lpi path.
>>>>
>>>> Otherwise we would need to add another dependency between RDIST pending
>>>> table restore and ITS table restore but this looks even more weird, no?
>>>>
>>>>
>>> So I just sat down with Andre and Marc and we tried to work through this
>>> and came up with the best scheme.  I apologize in advance for the
>>> one-way nature of this e-mail, and I am of course open to discussing the
>>> following proposal again if you do not agree.
>>>
>>> What I think this document should say, is that the following ordering
>>> must be followed when restoring the GIC and the ITS:
>>>
>>>   First, restore all guest memory
>>>
>>>   Second, restore ALL redistributors
>>>
>>>   Third, restore the ITS, in the following order:
>>>     1. Initialize the ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
>>>     2. Restore GITS_CBASER
>>>     3. Restore all other GITS_ registers, except GITS_CTLR!
>>>     4. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
>>>     5. Restore GITS_CTLR
>>>
>>> The rationale is that we really want the redistributor and the ITS
>>> restore to be independent and follow the architecture.  This means that
>>> our ABI for the redistributor should still work without restoring an ITS
>>> (if we ever decide to support LPIs for KVM without the ITS).
>>>
>>> In terms of our current implementation this means that vgic_add_lpi()
>>> should ask the redistributor what the state of the LPI is (priority,
>>> enabled, pending).  I suggest you do the pending check by adding a
>>> function called something like vgic_v3_lpi_is_pending() which scans the
>>> bit in memory, clears the memory bit, and returns the value.  Clearing
>>> the pending bit in memory when moving it to the struct irq is nice,
>>> because you then don't have to clear out the entire pending table later
>>> and we don't keep 'consumed' data lying around.  This change should be
>>> implemented in its_sync_lpi_pending_table() as well, but note that you
>>> need never call that function in the normal restore path using this
>>> design.
>>>
>>> I hope this makes sense.
>>
>> I am dubious about the above changes at the moment.
>> its_sync_lpi_pending_table() gets called on GITS_CTLR setting which is
>> documented to be the last step of the restoration. I wonder why the
>> above changes cannot be part of another series later on.
> 
> I think that's one of the issues. See below.
> 
>> Consuming the RAM bit status means we record it in irq->pending_latch so
>> I guess we should have the irq->pending_latch setting in the same
>> function as the one that retrieves the bit status in guest RAM. So I
>> would rename vgic_v3_lpi_is_pending into something like
>> int vgic_v3_sync_lpi_pending_status(struct kvm *kvm, u32 intid)
>> Since this covers a single LPI, the removes the byte access optimization
>> found in its_sync_lpi_pending_table
> 
> Well, never mind the optimization. How many LPIs are we restoring in a
> typical VM? 10? 1000? That's just one byte access per LPI. Of course,
> I'd rather have fewer guest memory accesses, but a restore is an
> incredibly rare event, so I'm not too bothered about the extra usec! ;-)
> 
>>
>> Also if I understand it correctly this means the sync will be done on
>> both add_lpi and GITS_CTLR setting
> 
> Why GITS_CTLR? The Enable bit only controls the effect of
> GITS_TRANSLATER...

Hum sorry I mixed up. the sync is currently done on GIC*R*_CTLR
vgic_mmio_write_v3r_ctlr/vgic_enable_lpis/its_sync_lpi_pending_table

As the redistributors are restored *before* the ITS this sync is void as
no LPI exist at that time.

That's why I did the sync (again) on ITS table restore. Sorry for the
noise.

OK let's go with the sync in vgic_add_lpi() ...

Thanks

Eric

 I believe that vgic_add_lpi() is the only point where
> we should snapshot the pending state.
> 
> Thanks,
> 
> 	M.
> 

Re: [PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Christoffer Dall]

On Thu, May 04, 2017 at 09:40:35AM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 04/05/2017 09:31, Christoffer Dall wrote:
> > On Wed, May 03, 2017 at 11:55:34PM +0200, Auger Eric wrote:
> >> Hi Christoffer,
> >>
> >> On 03/05/2017 18:37, Christoffer Dall wrote:
> >>> On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
> >>>> Hi Christoffer,
> >>>>
> >>>> On 30/04/2017 22:14, Christoffer Dall wrote:
> >>>>> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
> >>>>>> Introduce routines to save and restore device ITT and their
> >>>>>> interrupt table entries (ITE).
> >>>>>>
> >>>>>> The routines will be called on device table save and
> >>>>>> restore. They will become static in subsequent patches.
> >>>>>
> >>>>> Why this bottom-up approach?  Couldn't you start by having the patch
> >>>>> that restores the device table and define the static functions that
> >>>>> return an error there
> >>>> done
> >>>> , and then fill them in with subsequent patches
> >>>>> (liek this one)?
> >>>>>
> >>>>> That would have the added benefit of being able to tell how things are
> >>>>> designed to be called.
> >>>>>
> >>>>>>
> >>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>
> >>>>>> ---
> >>>>>> v4 -> v5:
> >>>>>> - ITE are now sorted by eventid on the flush
> >>>>>> - rename *flush* into *save*
> >>>>>> - use macros for shits and masks
> >>>>>> - pass ite_esz to vgic_its_save_ite
> >>>>>>
> >>>>>> v3 -> v4:
> >>>>>> - lookup_table and compute_next_eventid_offset become static in this
> >>>>>>   patch
> >>>>>> - remove static along with vgic_its_flush/restore_itt to avoid
> >>>>>>   compilation warnings
> >>>>>> - next field only computed with a shift (mask removed)
> >>>>>> - handle the case where the last element has not been found
> >>>>>>
> >>>>>> v2 -> v3:
> >>>>>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
> >>>>>>
> >>>>>> v2: creation
> >>>>>> ---
> >>>>>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
> >>>>>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
> >>>>>>  2 files changed, 129 insertions(+), 3 deletions(-)
> >>>>>>
> >>>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >>>>>> index 35b2ca1..b02fc3f 100644
> >>>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
> >>>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >>>>>> @@ -23,6 +23,7 @@
> >>>>>>  #include <linux/interrupt.h>
> >>>>>>  #include <linux/list.h>
> >>>>>>  #include <linux/uaccess.h>
> >>>>>> +#include <linux/list_sort.h>
> >>>>>>  
> >>>>>>  #include <linux/irqchip/arm-gic-v3.h>
> >>>>>>  
> >>>>>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >>>>>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >>>>>>  }
> >>>>>>  
> >>>>>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >>>>>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >>>>>>  {
> >>>>>>  	struct list_head *e = &ite->ite_list;
> >>>>>>  	struct its_ite *next;
> >>>>>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> >>>>>>   *
> >>>>>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
> >>>>>>   */
> >>>>>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>>>> -		 int start_id, entry_fn_t fn, void *opaque)
> >>>>>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>>>> +			int start_id, entry_fn_t fn, void *opaque)
> >>>>>>  {
> >>>>>>  	void *entry = kzalloc(esz, GFP_KERNEL);
> >>>>>>  	struct kvm *kvm = its->dev->kvm;
> >>>>>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>>>>  }
> >>>>>>  
> >>>>>>  /**
> >>>>>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
> >>>>>> + */
> >>>>>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
> >>>>>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
> >>>>>> +{
> >>>>>> +	struct kvm *kvm = its->dev->kvm;
> >>>>>> +	u32 next_offset;
> >>>>>> +	u64 val;
> >>>>>> +
> >>>>>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
> >>>>>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
> >>>>>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
> >>>>>> +		ite->collection->collection_id;
> >>>>>> +	val = cpu_to_le64(val);
> >>>>>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
> >>>>>> +}
> >>>>>> +
> >>>>>> +/**
> >>>>>> + * vgic_its_restore_ite - restore an interrupt translation entry
> >>>>>> + * @event_id: id used for indexing
> >>>>>> + * @ptr: pointer to the ITE entry
> >>>>>> + * @opaque: pointer to the its_device
> >>>>>> + * @next: id offset to the next entry
> >>>>>> + */
> >>>>>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
> >>>>>> +				void *ptr, void *opaque, u32 *next)
> >>>>>> +{
> >>>>>> +	struct its_device *dev = (struct its_device *)opaque;
> >>>>>> +	struct its_collection *collection;
> >>>>>> +	struct kvm *kvm = its->dev->kvm;
> >>>>>> +	u64 val, *p = (u64 *)ptr;
> >>>>>
> >>>>> nit: initializations on separate line (and possible do that just above
> >>>>> assigning val).
> >>>> done
> >>>>>
> >>>>>> +	struct vgic_irq *irq;
> >>>>>> +	u32 coll_id, lpi_id;
> >>>>>> +	struct its_ite *ite;
> >>>>>> +	int ret;
> >>>>>> +
> >>>>>> +	val = *p;
> >>>>>> +	*next = 1;
> >>>>>> +
> >>>>>> +	val = le64_to_cpu(val);
> >>>>>> +
> >>>>>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
> >>>>>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
> >>>>>> +
> >>>>>> +	if (!lpi_id)
> >>>>>> +		return 0;
> >>>>>
> >>>>> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
> >>>>> the ID is valid?
> >>>> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
> >>>> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
> >>>> GIC_MIN_LPI cause an -EINVAL error
> >>>>>
> >>>>> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
> >>>>> and PPIs here)
> >>>>
> >>>>>
> >>>>>> +
> >>>>>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
> >>>>>
> >>>>> Don't we need to validate this somehow since it will presumably be used
> >>>>> to forward a pointer somehow by the caller?
> >>>> checked against max number of eventids supported by the device
> >>>>>
> >>>>>> +
> >>>>>> +	collection = find_collection(its, coll_id);
> >>>>>> +	if (!collection)
> >>>>>> +		return -EINVAL;
> >>>>>> +
> >>>>>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
> >>>>>> +				  lpi_id, event_id);
> >>>>>> +	if (ret)
> >>>>>> +		return ret;
> >>>>>> +
> >>>>>> +	irq = vgic_add_lpi(kvm, lpi_id);
> >>>>>> +	if (IS_ERR(irq))
> >>>>>> +		return PTR_ERR(irq);
> >>>>>> +	ite->irq = irq;
> >>>>>> +
> >>>>>> +	/* restore the configuration of the LPI */
> >>>>>> +	ret = update_lpi_config(kvm, irq, NULL);
> >>>>>> +	if (ret)
> >>>>>> +		return ret;
> >>>>>> +
> >>>>>> +	update_affinity_ite(kvm, ite);
> >>>>>> +	return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
> >>>>>> +			    struct list_head *b)
> >>>>>> +{
> >>>>>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
> >>>>>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
> >>>>>> +
> >>>>>> +	if (itea->event_id < iteb->event_id)
> >>>>>> +		return -1;
> >>>>>> +	else
> >>>>>> +		return 1;
> >>>>>> +}
> >>>>>> +
> >>>>>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >>>>>> +{
> >>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>>>>> +	gpa_t base = device->itt_addr;
> >>>>>> +	struct its_ite *ite;
> >>>>>> +	int ret, ite_esz = abi->ite_esz;
> >>>>>
> >>>>> nit: initializations on separate line
> >>>> OK
> >>>>>
> >>>>>> +
> >>>>>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
> >>>>>> +
> >>>>>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
> >>>>>> +		gpa_t gpa = base + ite->event_id * ite_esz;
> >>>>>> +
> >>>>>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
> >>>>>> +		if (ret)
> >>>>>> +			return ret;
> >>>>>> +	}
> >>>>>> +	return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >>>>>> +{
> >>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>>>>> +	gpa_t base = dev->itt_addr;
> >>>>>> +	int ret, ite_esz = abi->ite_esz;
> >>>>>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
> >>>>>
> >>>>> nit: initializations on separate line
> >>>> OK
> >>>>>
> >>>>>> +
> >>>>>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
> >>>>>> +			    vgic_its_restore_ite, dev);
> >>>>>
> >>>>> nit: extra white space
> >>>>>
> >>>>>> +
> >>>>>> +	if (ret < 0)
> >>>>>> +		return ret;
> >>>>>> +
> >>>>>> +	/* if the last element has not been found we are in trouble */
> >>>>>> +	return ret ? 0 : -EINVAL;
> >>>>>
> >>>>> hmm, these are values potentially created by the guest in guest RAM,
> >>>>> right?  So do we really abort migration and return an error to userspace
> >>>>> in this case?
> >>>> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
> >>>> such error. The restore table IOCTL will return an error. Up to qemu to
> >>>> print the error. Destination guest will not be functional though.
> >>>>
> >>>
> >>> ok, I'm just wondering if userspace can make a qualified decision based
> >>> on this error code.  EINVAL typically means that userspace provided
> >>> something incorrect, which I suppose in a sense is true, but this should
> >>> be the only case where we return EINVAL here.
> >>   Userspace must be able to
> >>> tell the cases apart where the guest programmed bogus into memory before
> >>> migration started, in which case we should ignore-and-resume, and where
> >>> QEMU errornously provide some bogus value where the machine state
> >>> becomes unreliable and must be powered down.
> >> guest does not feed much besides few registers the ITS table restore
> >> depends on. In case we want a more subtle error management at userspace
> >> level all the error codes need to be revisited I am afraid. My plan was
> >> to be more rough at the beginning and ignore & resume if ITS table
> >> restore fails.
> >>
> > 
> > Do we require that the VM is quiesced the entire time between saving the
> > ITS state to memory and copying all memory over the wire and capturing
> > all register state?  If so, then an error to restore would be because of
> > userspace doing something wrong and handling that accordingly is fine.
> 
> yes the ITS table save into RAM starts when we have a guarantee that all
> the VCPUS are stopped (we take all locks). 

The important bit is whether or not userspace is allowed to start any
VCPUs again before copying over all RAM etc.  I suppose not.

> The restore happens before
> the VM gets resumed. At least this is the QEMU integration as of today.
> 

Does our ABI mandate this behavior (document it somewhere) ?

Thanks,
-Christoffer

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Christoffer Dall]

On Thu, May 04, 2017 at 09:40:35AM +0200, Auger Eric wrote:
> Hi Christoffer,
> 
> On 04/05/2017 09:31, Christoffer Dall wrote:
> > On Wed, May 03, 2017 at 11:55:34PM +0200, Auger Eric wrote:
> >> Hi Christoffer,
> >>
> >> On 03/05/2017 18:37, Christoffer Dall wrote:
> >>> On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
> >>>> Hi Christoffer,
> >>>>
> >>>> On 30/04/2017 22:14, Christoffer Dall wrote:
> >>>>> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
> >>>>>> Introduce routines to save and restore device ITT and their
> >>>>>> interrupt table entries (ITE).
> >>>>>>
> >>>>>> The routines will be called on device table save and
> >>>>>> restore. They will become static in subsequent patches.
> >>>>>
> >>>>> Why this bottom-up approach?  Couldn't you start by having the patch
> >>>>> that restores the device table and define the static functions that
> >>>>> return an error there
> >>>> done
> >>>> , and then fill them in with subsequent patches
> >>>>> (liek this one)?
> >>>>>
> >>>>> That would have the added benefit of being able to tell how things are
> >>>>> designed to be called.
> >>>>>
> >>>>>>
> >>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>>>>>
> >>>>>> ---
> >>>>>> v4 -> v5:
> >>>>>> - ITE are now sorted by eventid on the flush
> >>>>>> - rename *flush* into *save*
> >>>>>> - use macros for shits and masks
> >>>>>> - pass ite_esz to vgic_its_save_ite
> >>>>>>
> >>>>>> v3 -> v4:
> >>>>>> - lookup_table and compute_next_eventid_offset become static in this
> >>>>>>   patch
> >>>>>> - remove static along with vgic_its_flush/restore_itt to avoid
> >>>>>>   compilation warnings
> >>>>>> - next field only computed with a shift (mask removed)
> >>>>>> - handle the case where the last element has not been found
> >>>>>>
> >>>>>> v2 -> v3:
> >>>>>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
> >>>>>>
> >>>>>> v2: creation
> >>>>>> ---
> >>>>>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
> >>>>>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
> >>>>>>  2 files changed, 129 insertions(+), 3 deletions(-)
> >>>>>>
> >>>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
> >>>>>> index 35b2ca1..b02fc3f 100644
> >>>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
> >>>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
> >>>>>> @@ -23,6 +23,7 @@
> >>>>>>  #include <linux/interrupt.h>
> >>>>>>  #include <linux/list.h>
> >>>>>>  #include <linux/uaccess.h>
> >>>>>> +#include <linux/list_sort.h>
> >>>>>>  
> >>>>>>  #include <linux/irqchip/arm-gic-v3.h>
> >>>>>>  
> >>>>>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
> >>>>>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
> >>>>>>  }
> >>>>>>  
> >>>>>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >>>>>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
> >>>>>>  {
> >>>>>>  	struct list_head *e = &ite->ite_list;
> >>>>>>  	struct its_ite *next;
> >>>>>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
> >>>>>>   *
> >>>>>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
> >>>>>>   */
> >>>>>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>>>> -		 int start_id, entry_fn_t fn, void *opaque)
> >>>>>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>>>> +			int start_id, entry_fn_t fn, void *opaque)
> >>>>>>  {
> >>>>>>  	void *entry = kzalloc(esz, GFP_KERNEL);
> >>>>>>  	struct kvm *kvm = its->dev->kvm;
> >>>>>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
> >>>>>>  }
> >>>>>>  
> >>>>>>  /**
> >>>>>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
> >>>>>> + */
> >>>>>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
> >>>>>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
> >>>>>> +{
> >>>>>> +	struct kvm *kvm = its->dev->kvm;
> >>>>>> +	u32 next_offset;
> >>>>>> +	u64 val;
> >>>>>> +
> >>>>>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
> >>>>>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
> >>>>>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
> >>>>>> +		ite->collection->collection_id;
> >>>>>> +	val = cpu_to_le64(val);
> >>>>>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
> >>>>>> +}
> >>>>>> +
> >>>>>> +/**
> >>>>>> + * vgic_its_restore_ite - restore an interrupt translation entry
> >>>>>> + * @event_id: id used for indexing
> >>>>>> + * @ptr: pointer to the ITE entry
> >>>>>> + * @opaque: pointer to the its_device
> >>>>>> + * @next: id offset to the next entry
> >>>>>> + */
> >>>>>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
> >>>>>> +				void *ptr, void *opaque, u32 *next)
> >>>>>> +{
> >>>>>> +	struct its_device *dev = (struct its_device *)opaque;
> >>>>>> +	struct its_collection *collection;
> >>>>>> +	struct kvm *kvm = its->dev->kvm;
> >>>>>> +	u64 val, *p = (u64 *)ptr;
> >>>>>
> >>>>> nit: initializations on separate line (and possible do that just above
> >>>>> assigning val).
> >>>> done
> >>>>>
> >>>>>> +	struct vgic_irq *irq;
> >>>>>> +	u32 coll_id, lpi_id;
> >>>>>> +	struct its_ite *ite;
> >>>>>> +	int ret;
> >>>>>> +
> >>>>>> +	val = *p;
> >>>>>> +	*next = 1;
> >>>>>> +
> >>>>>> +	val = le64_to_cpu(val);
> >>>>>> +
> >>>>>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
> >>>>>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
> >>>>>> +
> >>>>>> +	if (!lpi_id)
> >>>>>> +		return 0;
> >>>>>
> >>>>> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
> >>>>> the ID is valid?
> >>>> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
> >>>> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
> >>>> GIC_MIN_LPI cause an -EINVAL error
> >>>>>
> >>>>> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
> >>>>> and PPIs here)
> >>>>
> >>>>>
> >>>>>> +
> >>>>>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
> >>>>>
> >>>>> Don't we need to validate this somehow since it will presumably be used
> >>>>> to forward a pointer somehow by the caller?
> >>>> checked against max number of eventids supported by the device
> >>>>>
> >>>>>> +
> >>>>>> +	collection = find_collection(its, coll_id);
> >>>>>> +	if (!collection)
> >>>>>> +		return -EINVAL;
> >>>>>> +
> >>>>>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
> >>>>>> +				  lpi_id, event_id);
> >>>>>> +	if (ret)
> >>>>>> +		return ret;
> >>>>>> +
> >>>>>> +	irq = vgic_add_lpi(kvm, lpi_id);
> >>>>>> +	if (IS_ERR(irq))
> >>>>>> +		return PTR_ERR(irq);
> >>>>>> +	ite->irq = irq;
> >>>>>> +
> >>>>>> +	/* restore the configuration of the LPI */
> >>>>>> +	ret = update_lpi_config(kvm, irq, NULL);
> >>>>>> +	if (ret)
> >>>>>> +		return ret;
> >>>>>> +
> >>>>>> +	update_affinity_ite(kvm, ite);
> >>>>>> +	return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
> >>>>>> +			    struct list_head *b)
> >>>>>> +{
> >>>>>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
> >>>>>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
> >>>>>> +
> >>>>>> +	if (itea->event_id < iteb->event_id)
> >>>>>> +		return -1;
> >>>>>> +	else
> >>>>>> +		return 1;
> >>>>>> +}
> >>>>>> +
> >>>>>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
> >>>>>> +{
> >>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>>>>> +	gpa_t base = device->itt_addr;
> >>>>>> +	struct its_ite *ite;
> >>>>>> +	int ret, ite_esz = abi->ite_esz;
> >>>>>
> >>>>> nit: initializations on separate line
> >>>> OK
> >>>>>
> >>>>>> +
> >>>>>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
> >>>>>> +
> >>>>>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
> >>>>>> +		gpa_t gpa = base + ite->event_id * ite_esz;
> >>>>>> +
> >>>>>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
> >>>>>> +		if (ret)
> >>>>>> +			return ret;
> >>>>>> +	}
> >>>>>> +	return 0;
> >>>>>> +}
> >>>>>> +
> >>>>>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
> >>>>>> +{
> >>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
> >>>>>> +	gpa_t base = dev->itt_addr;
> >>>>>> +	int ret, ite_esz = abi->ite_esz;
> >>>>>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
> >>>>>
> >>>>> nit: initializations on separate line
> >>>> OK
> >>>>>
> >>>>>> +
> >>>>>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
> >>>>>> +			    vgic_its_restore_ite, dev);
> >>>>>
> >>>>> nit: extra white space
> >>>>>
> >>>>>> +
> >>>>>> +	if (ret < 0)
> >>>>>> +		return ret;
> >>>>>> +
> >>>>>> +	/* if the last element has not been found we are in trouble */
> >>>>>> +	return ret ? 0 : -EINVAL;
> >>>>>
> >>>>> hmm, these are values potentially created by the guest in guest RAM,
> >>>>> right?  So do we really abort migration and return an error to userspace
> >>>>> in this case?
> >>>> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
> >>>> such error. The restore table IOCTL will return an error. Up to qemu to
> >>>> print the error. Destination guest will not be functional though.
> >>>>
> >>>
> >>> ok, I'm just wondering if userspace can make a qualified decision based
> >>> on this error code.  EINVAL typically means that userspace provided
> >>> something incorrect, which I suppose in a sense is true, but this should
> >>> be the only case where we return EINVAL here.
> >>   Userspace must be able to
> >>> tell the cases apart where the guest programmed bogus into memory before
> >>> migration started, in which case we should ignore-and-resume, and where
> >>> QEMU errornously provide some bogus value where the machine state
> >>> becomes unreliable and must be powered down.
> >> guest does not feed much besides few registers the ITS table restore
> >> depends on. In case we want a more subtle error management at userspace
> >> level all the error codes need to be revisited I am afraid. My plan was
> >> to be more rough at the beginning and ignore & resume if ITS table
> >> restore fails.
> >>
> > 
> > Do we require that the VM is quiesced the entire time between saving the
> > ITS state to memory and copying all memory over the wire and capturing
> > all register state?  If so, then an error to restore would be because of
> > userspace doing something wrong and handling that accordingly is fine.
> 
> yes the ITS table save into RAM starts when we have a guarantee that all
> the VCPUS are stopped (we take all locks). 

The important bit is whether or not userspace is allowed to start any
VCPUs again before copying over all RAM etc.  I suppose not.

> The restore happens before
> the VM gets resumed. At least this is the QEMU integration as of today.
> 

Does our ABI mandate this behavior (document it somewhere) ?

Thanks,
-Christoffer

Re: [PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Auger Eric]

Hi,
On 04/05/2017 10:23, Christoffer Dall wrote:
> On Thu, May 04, 2017 at 09:40:35AM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 04/05/2017 09:31, Christoffer Dall wrote:
>>> On Wed, May 03, 2017 at 11:55:34PM +0200, Auger Eric wrote:
>>>> Hi Christoffer,
>>>>
>>>> On 03/05/2017 18:37, Christoffer Dall wrote:
>>>>> On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
>>>>>> Hi Christoffer,
>>>>>>
>>>>>> On 30/04/2017 22:14, Christoffer Dall wrote:
>>>>>>> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
>>>>>>>> Introduce routines to save and restore device ITT and their
>>>>>>>> interrupt table entries (ITE).
>>>>>>>>
>>>>>>>> The routines will be called on device table save and
>>>>>>>> restore. They will become static in subsequent patches.
>>>>>>>
>>>>>>> Why this bottom-up approach?  Couldn't you start by having the patch
>>>>>>> that restores the device table and define the static functions that
>>>>>>> return an error there
>>>>>> done
>>>>>> , and then fill them in with subsequent patches
>>>>>>> (liek this one)?
>>>>>>>
>>>>>>> That would have the added benefit of being able to tell how things are
>>>>>>> designed to be called.
>>>>>>>
>>>>>>>>
>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> v4 -> v5:
>>>>>>>> - ITE are now sorted by eventid on the flush
>>>>>>>> - rename *flush* into *save*
>>>>>>>> - use macros for shits and masks
>>>>>>>> - pass ite_esz to vgic_its_save_ite
>>>>>>>>
>>>>>>>> v3 -> v4:
>>>>>>>> - lookup_table and compute_next_eventid_offset become static in this
>>>>>>>>   patch
>>>>>>>> - remove static along with vgic_its_flush/restore_itt to avoid
>>>>>>>>   compilation warnings
>>>>>>>> - next field only computed with a shift (mask removed)
>>>>>>>> - handle the case where the last element has not been found
>>>>>>>>
>>>>>>>> v2 -> v3:
>>>>>>>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
>>>>>>>>
>>>>>>>> v2: creation
>>>>>>>> ---
>>>>>>>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
>>>>>>>>  2 files changed, 129 insertions(+), 3 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>>>>>> index 35b2ca1..b02fc3f 100644
>>>>>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>>>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>>>>>> @@ -23,6 +23,7 @@
>>>>>>>>  #include <linux/interrupt.h>
>>>>>>>>  #include <linux/list.h>
>>>>>>>>  #include <linux/uaccess.h>
>>>>>>>> +#include <linux/list_sort.h>
>>>>>>>>  
>>>>>>>>  #include <linux/irqchip/arm-gic-v3.h>
>>>>>>>>  
>>>>>>>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>>>>>>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>>>>>>>>  }
>>>>>>>>  
>>>>>>>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>>>>>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>>>>>>  {
>>>>>>>>  	struct list_head *e = &ite->ite_list;
>>>>>>>>  	struct its_ite *next;
>>>>>>>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>>>>>>>>   *
>>>>>>>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
>>>>>>>>   */
>>>>>>>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>>>> -		 int start_id, entry_fn_t fn, void *opaque)
>>>>>>>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>>>> +			int start_id, entry_fn_t fn, void *opaque)
>>>>>>>>  {
>>>>>>>>  	void *entry = kzalloc(esz, GFP_KERNEL);
>>>>>>>>  	struct kvm *kvm = its->dev->kvm;
>>>>>>>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>>>>  }
>>>>>>>>  
>>>>>>>>  /**
>>>>>>>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
>>>>>>>> + */
>>>>>>>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
>>>>>>>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
>>>>>>>> +{
>>>>>>>> +	struct kvm *kvm = its->dev->kvm;
>>>>>>>> +	u32 next_offset;
>>>>>>>> +	u64 val;
>>>>>>>> +
>>>>>>>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
>>>>>>>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
>>>>>>>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
>>>>>>>> +		ite->collection->collection_id;
>>>>>>>> +	val = cpu_to_le64(val);
>>>>>>>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +/**
>>>>>>>> + * vgic_its_restore_ite - restore an interrupt translation entry
>>>>>>>> + * @event_id: id used for indexing
>>>>>>>> + * @ptr: pointer to the ITE entry
>>>>>>>> + * @opaque: pointer to the its_device
>>>>>>>> + * @next: id offset to the next entry
>>>>>>>> + */
>>>>>>>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
>>>>>>>> +				void *ptr, void *opaque, u32 *next)
>>>>>>>> +{
>>>>>>>> +	struct its_device *dev = (struct its_device *)opaque;
>>>>>>>> +	struct its_collection *collection;
>>>>>>>> +	struct kvm *kvm = its->dev->kvm;
>>>>>>>> +	u64 val, *p = (u64 *)ptr;
>>>>>>>
>>>>>>> nit: initializations on separate line (and possible do that just above
>>>>>>> assigning val).
>>>>>> done
>>>>>>>
>>>>>>>> +	struct vgic_irq *irq;
>>>>>>>> +	u32 coll_id, lpi_id;
>>>>>>>> +	struct its_ite *ite;
>>>>>>>> +	int ret;
>>>>>>>> +
>>>>>>>> +	val = *p;
>>>>>>>> +	*next = 1;
>>>>>>>> +
>>>>>>>> +	val = le64_to_cpu(val);
>>>>>>>> +
>>>>>>>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
>>>>>>>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
>>>>>>>> +
>>>>>>>> +	if (!lpi_id)
>>>>>>>> +		return 0;
>>>>>>>
>>>>>>> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
>>>>>>> the ID is valid?
>>>>>> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
>>>>>> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
>>>>>> GIC_MIN_LPI cause an -EINVAL error
>>>>>>>
>>>>>>> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
>>>>>>> and PPIs here)
>>>>>>
>>>>>>>
>>>>>>>> +
>>>>>>>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
>>>>>>>
>>>>>>> Don't we need to validate this somehow since it will presumably be used
>>>>>>> to forward a pointer somehow by the caller?
>>>>>> checked against max number of eventids supported by the device
>>>>>>>
>>>>>>>> +
>>>>>>>> +	collection = find_collection(its, coll_id);
>>>>>>>> +	if (!collection)
>>>>>>>> +		return -EINVAL;
>>>>>>>> +
>>>>>>>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
>>>>>>>> +				  lpi_id, event_id);
>>>>>>>> +	if (ret)
>>>>>>>> +		return ret;
>>>>>>>> +
>>>>>>>> +	irq = vgic_add_lpi(kvm, lpi_id);
>>>>>>>> +	if (IS_ERR(irq))
>>>>>>>> +		return PTR_ERR(irq);
>>>>>>>> +	ite->irq = irq;
>>>>>>>> +
>>>>>>>> +	/* restore the configuration of the LPI */
>>>>>>>> +	ret = update_lpi_config(kvm, irq, NULL);
>>>>>>>> +	if (ret)
>>>>>>>> +		return ret;
>>>>>>>> +
>>>>>>>> +	update_affinity_ite(kvm, ite);
>>>>>>>> +	return 0;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>>>>>>>> +			    struct list_head *b)
>>>>>>>> +{
>>>>>>>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
>>>>>>>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
>>>>>>>> +
>>>>>>>> +	if (itea->event_id < iteb->event_id)
>>>>>>>> +		return -1;
>>>>>>>> +	else
>>>>>>>> +		return 1;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>>>>>> +{
>>>>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>>>>> +	gpa_t base = device->itt_addr;
>>>>>>>> +	struct its_ite *ite;
>>>>>>>> +	int ret, ite_esz = abi->ite_esz;
>>>>>>>
>>>>>>> nit: initializations on separate line
>>>>>> OK
>>>>>>>
>>>>>>>> +
>>>>>>>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
>>>>>>>> +
>>>>>>>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
>>>>>>>> +		gpa_t gpa = base + ite->event_id * ite_esz;
>>>>>>>> +
>>>>>>>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
>>>>>>>> +		if (ret)
>>>>>>>> +			return ret;
>>>>>>>> +	}
>>>>>>>> +	return 0;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>>>>>> +{
>>>>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>>>>> +	gpa_t base = dev->itt_addr;
>>>>>>>> +	int ret, ite_esz = abi->ite_esz;
>>>>>>>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
>>>>>>>
>>>>>>> nit: initializations on separate line
>>>>>> OK
>>>>>>>
>>>>>>>> +
>>>>>>>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
>>>>>>>> +			    vgic_its_restore_ite, dev);
>>>>>>>
>>>>>>> nit: extra white space
>>>>>>>
>>>>>>>> +
>>>>>>>> +	if (ret < 0)
>>>>>>>> +		return ret;
>>>>>>>> +
>>>>>>>> +	/* if the last element has not been found we are in trouble */
>>>>>>>> +	return ret ? 0 : -EINVAL;
>>>>>>>
>>>>>>> hmm, these are values potentially created by the guest in guest RAM,
>>>>>>> right?  So do we really abort migration and return an error to userspace
>>>>>>> in this case?
>>>>>> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
>>>>>> such error. The restore table IOCTL will return an error. Up to qemu to
>>>>>> print the error. Destination guest will not be functional though.
>>>>>>
>>>>>
>>>>> ok, I'm just wondering if userspace can make a qualified decision based
>>>>> on this error code.  EINVAL typically means that userspace provided
>>>>> something incorrect, which I suppose in a sense is true, but this should
>>>>> be the only case where we return EINVAL here.
>>>>   Userspace must be able to
>>>>> tell the cases apart where the guest programmed bogus into memory before
>>>>> migration started, in which case we should ignore-and-resume, and where
>>>>> QEMU errornously provide some bogus value where the machine state
>>>>> becomes unreliable and must be powered down.
>>>> guest does not feed much besides few registers the ITS table restore
>>>> depends on. In case we want a more subtle error management at userspace
>>>> level all the error codes need to be revisited I am afraid. My plan was
>>>> to be more rough at the beginning and ignore & resume if ITS table
>>>> restore fails.
>>>>
>>>
>>> Do we require that the VM is quiesced the entire time between saving the
>>> ITS state to memory and copying all memory over the wire and capturing
>>> all register state?  If so, then an error to restore would be because of
>>> userspace doing something wrong and handling that accordingly is fine.
>>
>> yes the ITS table save into RAM starts when we have a guarantee that all
>> the VCPUS are stopped (we take all locks). 
> 
> The important bit is whether or not userspace is allowed to start any
> VCPUs again before copying over all RAM etc.  I suppose not.
no it is not meant to happen.
> 
>> The restore happens before
>> the VM gets resumed. At least this is the QEMU integration as of today.
>>
> 
> Does our ABI mandate this behavior (document it somewhere) ?
I will add this

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

[PATCH v5 19/22] KVM: arm64: vgic-its: ITT save and restore

[Auger Eric]

Hi,
On 04/05/2017 10:23, Christoffer Dall wrote:
> On Thu, May 04, 2017 at 09:40:35AM +0200, Auger Eric wrote:
>> Hi Christoffer,
>>
>> On 04/05/2017 09:31, Christoffer Dall wrote:
>>> On Wed, May 03, 2017 at 11:55:34PM +0200, Auger Eric wrote:
>>>> Hi Christoffer,
>>>>
>>>> On 03/05/2017 18:37, Christoffer Dall wrote:
>>>>> On Wed, May 03, 2017 at 06:08:58PM +0200, Auger Eric wrote:
>>>>>> Hi Christoffer,
>>>>>>
>>>>>> On 30/04/2017 22:14, Christoffer Dall wrote:
>>>>>>> On Fri, Apr 14, 2017 at 12:15:31PM +0200, Eric Auger wrote:
>>>>>>>> Introduce routines to save and restore device ITT and their
>>>>>>>> interrupt table entries (ITE).
>>>>>>>>
>>>>>>>> The routines will be called on device table save and
>>>>>>>> restore. They will become static in subsequent patches.
>>>>>>>
>>>>>>> Why this bottom-up approach?  Couldn't you start by having the patch
>>>>>>> that restores the device table and define the static functions that
>>>>>>> return an error there
>>>>>> done
>>>>>> , and then fill them in with subsequent patches
>>>>>>> (liek this one)?
>>>>>>>
>>>>>>> That would have the added benefit of being able to tell how things are
>>>>>>> designed to be called.
>>>>>>>
>>>>>>>>
>>>>>>>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> v4 -> v5:
>>>>>>>> - ITE are now sorted by eventid on the flush
>>>>>>>> - rename *flush* into *save*
>>>>>>>> - use macros for shits and masks
>>>>>>>> - pass ite_esz to vgic_its_save_ite
>>>>>>>>
>>>>>>>> v3 -> v4:
>>>>>>>> - lookup_table and compute_next_eventid_offset become static in this
>>>>>>>>   patch
>>>>>>>> - remove static along with vgic_its_flush/restore_itt to avoid
>>>>>>>>   compilation warnings
>>>>>>>> - next field only computed with a shift (mask removed)
>>>>>>>> - handle the case where the last element has not been found
>>>>>>>>
>>>>>>>> v2 -> v3:
>>>>>>>> - add return 0 in vgic_its_restore_ite (was in subsequent patch)
>>>>>>>>
>>>>>>>> v2: creation
>>>>>>>> ---
>>>>>>>>  virt/kvm/arm/vgic/vgic-its.c | 128 ++++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>  virt/kvm/arm/vgic/vgic.h     |   4 ++
>>>>>>>>  2 files changed, 129 insertions(+), 3 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
>>>>>>>> index 35b2ca1..b02fc3f 100644
>>>>>>>> --- a/virt/kvm/arm/vgic/vgic-its.c
>>>>>>>> +++ b/virt/kvm/arm/vgic/vgic-its.c
>>>>>>>> @@ -23,6 +23,7 @@
>>>>>>>>  #include <linux/interrupt.h>
>>>>>>>>  #include <linux/list.h>
>>>>>>>>  #include <linux/uaccess.h>
>>>>>>>> +#include <linux/list_sort.h>
>>>>>>>>  
>>>>>>>>  #include <linux/irqchip/arm-gic-v3.h>
>>>>>>>>  
>>>>>>>> @@ -1695,7 +1696,7 @@ u32 compute_next_devid_offset(struct list_head *h, struct its_device *dev)
>>>>>>>>  	return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
>>>>>>>>  }
>>>>>>>>  
>>>>>>>> -u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>>>>>> +static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
>>>>>>>>  {
>>>>>>>>  	struct list_head *e = &ite->ite_list;
>>>>>>>>  	struct its_ite *next;
>>>>>>>> @@ -1737,8 +1738,8 @@ typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
>>>>>>>>   *
>>>>>>>>   * Return: < 0 on error, 1 if last element identified, 0 otherwise
>>>>>>>>   */
>>>>>>>> -int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>>>> -		 int start_id, entry_fn_t fn, void *opaque)
>>>>>>>> +static int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>>>> +			int start_id, entry_fn_t fn, void *opaque)
>>>>>>>>  {
>>>>>>>>  	void *entry = kzalloc(esz, GFP_KERNEL);
>>>>>>>>  	struct kvm *kvm = its->dev->kvm;
>>>>>>>> @@ -1773,6 +1774,127 @@ int lookup_table(struct vgic_its *its, gpa_t base, int size, int esz,
>>>>>>>>  }
>>>>>>>>  
>>>>>>>>  /**
>>>>>>>> + * vgic_its_save_ite - Save an interrupt translation entry at @gpa
>>>>>>>> + */
>>>>>>>> +static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
>>>>>>>> +			      struct its_ite *ite, gpa_t gpa, int ite_esz)
>>>>>>>> +{
>>>>>>>> +	struct kvm *kvm = its->dev->kvm;
>>>>>>>> +	u32 next_offset;
>>>>>>>> +	u64 val;
>>>>>>>> +
>>>>>>>> +	next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
>>>>>>>> +	val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
>>>>>>>> +	       ((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
>>>>>>>> +		ite->collection->collection_id;
>>>>>>>> +	val = cpu_to_le64(val);
>>>>>>>> +	return kvm_write_guest(kvm, gpa, &val, ite_esz);
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +/**
>>>>>>>> + * vgic_its_restore_ite - restore an interrupt translation entry
>>>>>>>> + * @event_id: id used for indexing
>>>>>>>> + * @ptr: pointer to the ITE entry
>>>>>>>> + * @opaque: pointer to the its_device
>>>>>>>> + * @next: id offset to the next entry
>>>>>>>> + */
>>>>>>>> +static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
>>>>>>>> +				void *ptr, void *opaque, u32 *next)
>>>>>>>> +{
>>>>>>>> +	struct its_device *dev = (struct its_device *)opaque;
>>>>>>>> +	struct its_collection *collection;
>>>>>>>> +	struct kvm *kvm = its->dev->kvm;
>>>>>>>> +	u64 val, *p = (u64 *)ptr;
>>>>>>>
>>>>>>> nit: initializations on separate line (and possible do that just above
>>>>>>> assigning val).
>>>>>> done
>>>>>>>
>>>>>>>> +	struct vgic_irq *irq;
>>>>>>>> +	u32 coll_id, lpi_id;
>>>>>>>> +	struct its_ite *ite;
>>>>>>>> +	int ret;
>>>>>>>> +
>>>>>>>> +	val = *p;
>>>>>>>> +	*next = 1;
>>>>>>>> +
>>>>>>>> +	val = le64_to_cpu(val);
>>>>>>>> +
>>>>>>>> +	coll_id = val & KVM_ITS_ITE_ICID_MASK;
>>>>>>>> +	lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
>>>>>>>> +
>>>>>>>> +	if (!lpi_id)
>>>>>>>> +		return 0;
>>>>>>>
>>>>>>> are all non-zero LPI IDs valid?  Don't we have a wrapper that tests if
>>>>>>> the ID is valid?
>>>>>> no, lpi_id must be >= GIC_MIN_LPI=8192; added that check.
>>>>>> ABI Doc says lpi_id==0 is interpreted as invalid. Other values <
>>>>>> GIC_MIN_LPI cause an -EINVAL error
>>>>>>>
>>>>>>> (looks like it's possible to add LPIs with the INTID range of SPIs, SGIs
>>>>>>> and PPIs here)
>>>>>>
>>>>>>>
>>>>>>>> +
>>>>>>>> +	*next = val >> KVM_ITS_ITE_NEXT_SHIFT;
>>>>>>>
>>>>>>> Don't we need to validate this somehow since it will presumably be used
>>>>>>> to forward a pointer somehow by the caller?
>>>>>> checked against max number of eventids supported by the device
>>>>>>>
>>>>>>>> +
>>>>>>>> +	collection = find_collection(its, coll_id);
>>>>>>>> +	if (!collection)
>>>>>>>> +		return -EINVAL;
>>>>>>>> +
>>>>>>>> +	ret = vgic_its_alloc_ite(dev, &ite, collection,
>>>>>>>> +				  lpi_id, event_id);
>>>>>>>> +	if (ret)
>>>>>>>> +		return ret;
>>>>>>>> +
>>>>>>>> +	irq = vgic_add_lpi(kvm, lpi_id);
>>>>>>>> +	if (IS_ERR(irq))
>>>>>>>> +		return PTR_ERR(irq);
>>>>>>>> +	ite->irq = irq;
>>>>>>>> +
>>>>>>>> +	/* restore the configuration of the LPI */
>>>>>>>> +	ret = update_lpi_config(kvm, irq, NULL);
>>>>>>>> +	if (ret)
>>>>>>>> +		return ret;
>>>>>>>> +
>>>>>>>> +	update_affinity_ite(kvm, ite);
>>>>>>>> +	return 0;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +static int vgic_its_ite_cmp(void *priv, struct list_head *a,
>>>>>>>> +			    struct list_head *b)
>>>>>>>> +{
>>>>>>>> +	struct its_ite *itea = container_of(a, struct its_ite, ite_list);
>>>>>>>> +	struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
>>>>>>>> +
>>>>>>>> +	if (itea->event_id < iteb->event_id)
>>>>>>>> +		return -1;
>>>>>>>> +	else
>>>>>>>> +		return 1;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
>>>>>>>> +{
>>>>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>>>>> +	gpa_t base = device->itt_addr;
>>>>>>>> +	struct its_ite *ite;
>>>>>>>> +	int ret, ite_esz = abi->ite_esz;
>>>>>>>
>>>>>>> nit: initializations on separate line
>>>>>> OK
>>>>>>>
>>>>>>>> +
>>>>>>>> +	list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
>>>>>>>> +
>>>>>>>> +	list_for_each_entry(ite, &device->itt_head, ite_list) {
>>>>>>>> +		gpa_t gpa = base + ite->event_id * ite_esz;
>>>>>>>> +
>>>>>>>> +		ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
>>>>>>>> +		if (ret)
>>>>>>>> +			return ret;
>>>>>>>> +	}
>>>>>>>> +	return 0;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
>>>>>>>> +{
>>>>>>>> +	const struct vgic_its_abi *abi = vgic_its_get_abi(its);
>>>>>>>> +	gpa_t base = dev->itt_addr;
>>>>>>>> +	int ret, ite_esz = abi->ite_esz;
>>>>>>>> +	size_t max_size = BIT_ULL(dev->nb_eventid_bits) * ite_esz;
>>>>>>>
>>>>>>> nit: initializations on separate line
>>>>>> OK
>>>>>>>
>>>>>>>> +
>>>>>>>> +	ret =  lookup_table(its, base, max_size, ite_esz, 0,
>>>>>>>> +			    vgic_its_restore_ite, dev);
>>>>>>>
>>>>>>> nit: extra white space
>>>>>>>
>>>>>>>> +
>>>>>>>> +	if (ret < 0)
>>>>>>>> +		return ret;
>>>>>>>> +
>>>>>>>> +	/* if the last element has not been found we are in trouble */
>>>>>>>> +	return ret ? 0 : -EINVAL;
>>>>>>>
>>>>>>> hmm, these are values potentially created by the guest in guest RAM,
>>>>>>> right?  So do we really abort migration and return an error to userspace
>>>>>>> in this case?
>>>>>> So we discussed with Peter/dave we shouldn't abort() in qemu in case of
>>>>>> such error. The restore table IOCTL will return an error. Up to qemu to
>>>>>> print the error. Destination guest will not be functional though.
>>>>>>
>>>>>
>>>>> ok, I'm just wondering if userspace can make a qualified decision based
>>>>> on this error code.  EINVAL typically means that userspace provided
>>>>> something incorrect, which I suppose in a sense is true, but this should
>>>>> be the only case where we return EINVAL here.
>>>>   Userspace must be able to
>>>>> tell the cases apart where the guest programmed bogus into memory before
>>>>> migration started, in which case we should ignore-and-resume, and where
>>>>> QEMU errornously provide some bogus value where the machine state
>>>>> becomes unreliable and must be powered down.
>>>> guest does not feed much besides few registers the ITS table restore
>>>> depends on. In case we want a more subtle error management at userspace
>>>> level all the error codes need to be revisited I am afraid. My plan was
>>>> to be more rough at the beginning and ignore & resume if ITS table
>>>> restore fails.
>>>>
>>>
>>> Do we require that the VM is quiesced the entire time between saving the
>>> ITS state to memory and copying all memory over the wire and capturing
>>> all register state?  If so, then an error to restore would be because of
>>> userspace doing something wrong and handling that accordingly is fine.
>>
>> yes the ITS table save into RAM starts when we have a guarantee that all
>> the VCPUS are stopped (we take all locks). 
> 
> The important bit is whether or not userspace is allowed to start any
> VCPUs again before copying over all RAM etc.  I suppose not.
no it is not meant to happen.
> 
>> The restore happens before
>> the VM gets resumed. At least this is the QEMU integration as of today.
>>
> 
> Does our ABI mandate this behavior (document it somewhere) ?
I will add this

Thanks

Eric
> 
> Thanks,
> -Christoffer
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>