[kvm-unit-tests PATCH 0/2] arm/arm64: Add IPI/vtimer latency

[kvm-unit-tests PATCH 0/2] arm/arm64: Add IPI/vtimer latency

[Jingyi Wang]

With the development of arm gic architecture, we think it will be useful
to add some simple performance test in kut to measure the cost of
interrupts. X86 arch has implemented similar test.

Jingyi Wang (2):
  arm/arm64: gic: Add IPI latency test
  arm/arm64: Add vtimer latency test

 arm/gic.c   | 27 +++++++++++++++++++++++++++
 arm/timer.c | 11 +++++++++++
 2 files changed, 38 insertions(+)

-- 
2.19.1

[kvm-unit-tests PATCH 1/2] arm/arm64: gic: Add IPI latency test

[Jingyi Wang]

This patch add a test to measure the latency of IPI injection.

Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arm/gic.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/arm/gic.c b/arm/gic.c
index fcf4c1f..f5e830e 100644
--- a/arm/gic.c
+++ b/arm/gic.c
@@ -27,6 +27,9 @@ struct gic {
 	struct {
 		void (*send_self)(void);
 		void (*send_broadcast)(void);
+		u64 ipi_inject_time;
+		u64 ipi_receive_time[NR_CPUS];
+		int nr_records;
 	} ipi;
 };
 
@@ -142,6 +145,7 @@ static void check_irqnr(u32 irqnr)
 
 static void ipi_handler(struct pt_regs *regs __unused)
 {
+	gic->ipi.ipi_receive_time[gic->ipi.nr_records++] = get_cntvct();
 	u32 irqstat = gic_read_iar();
 	u32 irqnr = gic_iar_irqnr(irqstat);
 
@@ -187,9 +191,16 @@ static void ipi_test_self(void)
 	stats_reset();
 	cpumask_clear(&mask);
 	cpumask_set_cpu(smp_processor_id(), &mask);
+
+	gic->ipi.nr_records = 0;
+	gic->ipi.ipi_inject_time = get_cntvct();
 	gic->ipi.send_self();
+
 	check_acked("IPI: self", &mask);
 	report_prefix_pop();
+
+	report_info("The latency of ipi_test_self: %ld cycles",
+			gic->ipi.ipi_receive_time[0] - gic->ipi.ipi_inject_time);
 }
 
 static void ipi_test_smp(void)
@@ -202,17 +213,33 @@ static void ipi_test_smp(void)
 	cpumask_copy(&mask, &cpu_present_mask);
 	for (i = smp_processor_id() & 1; i < nr_cpus; i += 2)
 		cpumask_clear_cpu(i, &mask);
+
+	gic->ipi.nr_records = 0;
+	gic->ipi.ipi_inject_time = get_cntvct();
 	gic_ipi_send_mask(IPI_IRQ, &mask);
+
 	check_acked("IPI: directed", &mask);
 	report_prefix_pop();
 
+	for (i = 0; i < gic->ipi.nr_records; i++)
+		report_info("The latency of ipi_test_smp(directed): %ld cycles",
+			gic->ipi.ipi_receive_time[i] - gic->ipi.ipi_inject_time);
+
 	report_prefix_push("broadcast");
 	stats_reset();
 	cpumask_copy(&mask, &cpu_present_mask);
 	cpumask_clear_cpu(smp_processor_id(), &mask);
+
+	gic->ipi.nr_records = 0;
+	gic->ipi.ipi_inject_time = get_cntvct();
 	gic->ipi.send_broadcast();
+
 	check_acked("IPI: broadcast", &mask);
 	report_prefix_pop();
+
+	for (i = 0; i < gic->ipi.nr_records; i++)
+		report_info("The latency of ipi_test_smp(broadcast): %ld cycles",
+			gic->ipi.ipi_receive_time[i] - gic->ipi.ipi_inject_time);
 }
 
 static void ipi_enable(void)
-- 
2.19.1

[kvm-unit-tests PATCH 2/2] arm/arm64: Add vtimer latency test

[Jingyi Wang]

This patch add a test to measure the precise vtimer firing time.

Signed-off-by: Jingyi Wang <wangjingyi11@huawei.com>
---
 arm/timer.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arm/timer.c b/arm/timer.c
index f390e8e..1d5b3dc 100644
--- a/arm/timer.c
+++ b/arm/timer.c
@@ -16,6 +16,9 @@
 #define ARCH_TIMER_CTL_IMASK   (1 << 1)
 #define ARCH_TIMER_CTL_ISTATUS (1 << 2)
 
+static u64 inject_time;
+static u64 receive_time;
+
 static void *gic_ispendr;
 static void *gic_isenabler;
 static void *gic_icenabler;
@@ -149,6 +152,8 @@ static void irq_handler(struct pt_regs *regs)
 	u32 irqstat = gic_read_iar();
 	u32 irqnr = gic_iar_irqnr(irqstat);
 
+	receive_time = get_cntvct();
+
 	if (irqnr != GICC_INT_SPURIOUS)
 		gic_write_eoir(irqstat);
 
@@ -163,6 +168,11 @@ static void irq_handler(struct pt_regs *regs)
 
 	info->write_ctl(ARCH_TIMER_CTL_IMASK | ARCH_TIMER_CTL_ENABLE);
 	info->irq_received = true;
+
+	if (inject_time != 0)
+		report_info("vtimer latency: %ld cycles/10ms\n",
+			receive_time - inject_time);
+	inject_time = 0;
 }
 
 static bool gic_timer_pending(struct timer_info *info)
@@ -179,6 +189,7 @@ static bool test_cval_10msec(struct timer_info *info)
 
 	/* Program timer to fire in 10 ms */
 	before_timer = info->read_counter();
+	inject_time = get_cntvct();
 	info->write_cval(before_timer + time_10ms);
 	info->write_ctl(ARCH_TIMER_CTL_ENABLE);
 	isb();
-- 
2.19.1

Re: [kvm-unit-tests PATCH 0/2] arm/arm64: Add IPI/vtimer latency

[Andrew Jones]

On Wed, Apr 01, 2020 at 06:08:10PM +0800, Jingyi Wang wrote:
> With the development of arm gic architecture, we think it will be useful
> to add some simple performance test in kut to measure the cost of
> interrupts. X86 arch has implemented similar test.
> 
> Jingyi Wang (2):
>   arm/arm64: gic: Add IPI latency test
>   arm/arm64: Add vtimer latency test
> 
>  arm/gic.c   | 27 +++++++++++++++++++++++++++
>  arm/timer.c | 11 +++++++++++
>  2 files changed, 38 insertions(+)
> 
> -- 
> 2.19.1
> 
>

Hi Jingyi,

We already have an IPI latency test in arm/micro-bench.c I'd prefer that
one be used, if possible, rather than conflating the gic functional tests
with latency tests. Can you take a look at it and see if it satisfies
your needs, extending it if necessary?

Thanks,
drew

Re: [kvm-unit-tests PATCH 0/2] arm/arm64: Add IPI/vtimer latency

[Zenghui Yu]

Hi Drew, Jingyi,

On 2020/4/1 20:24, Andrew Jones wrote:
> On Wed, Apr 01, 2020 at 06:08:10PM +0800, Jingyi Wang wrote:
>> With the development of arm gic architecture, we think it will be useful
>> to add some simple performance test in kut to measure the cost of
>> interrupts. X86 arch has implemented similar test.
>>
>> Jingyi Wang (2):
>>    arm/arm64: gic: Add IPI latency test
>>    arm/arm64: Add vtimer latency test
>>
>>   arm/gic.c   | 27 +++++++++++++++++++++++++++
>>   arm/timer.c | 11 +++++++++++
>>   2 files changed, 38 insertions(+)
>>
>> -- 
>> 2.19.1
>>
>>
> 
> Hi Jingyi,
> 
> We already have an IPI latency test in arm/micro-bench.c I'd prefer that
> one be used, if possible, rather than conflating the gic functional tests
> with latency tests. Can you take a look at it and see if it satisfies
> your needs, extending it if necessary?

I think it'd be good to have these interrupt latency measurements in
kvm-unit-tests, and we can take the following interrupt types into
account:

- IPI
   As Drew pointed out, we already have one in the micro-bench group.
   But what I'm actually interested in is the latency of the new GICv4.1
   vSGIs (which will be directly injected through ITS).  To measure it,
   we should first make KUT be GICv4.1-awareness, see [1] for details.
   (This way, we can even have a look at the interrupt latency in HW
   level. Is it possible to have this in kvm-unit-tests, Drew?)

- PPI
   Like what has been done in patch #2, you can choose the vtimer
   interrupt as an example.

- LPI
   I think we can easily build a LPI latency test based on Eric's "ITS
   tests" series [2], which should be upstreamed soon.

- if you want to add more...

What do you think? I'd like to see a V2 of this series :-).


[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0b04758b002bde9434053be2fff8064ac3d9d8bb
[2] 
https://lore.kernel.org/kvm/20200320092428.20880-1-eric.auger@redhat.com/


Thanks,
Zenghui

Re: [kvm-unit-tests PATCH 0/2] arm/arm64: Add IPI/vtimer latency

[Jingyi Wang]

Hi Drew, Zenghui,

On 4/2/2020 7:52 PM, Zenghui Yu wrote:
> Hi Drew, Jingyi,
> 
> On 2020/4/1 20:24, Andrew Jones wrote:
>> On Wed, Apr 01, 2020 at 06:08:10PM +0800, Jingyi Wang wrote:
>>> With the development of arm gic architecture, we think it will be useful
>>> to add some simple performance test in kut to measure the cost of
>>> interrupts. X86 arch has implemented similar test.
>>>
>>> Jingyi Wang (2):
>>>    arm/arm64: gic: Add IPI latency test
>>>    arm/arm64: Add vtimer latency test
>>>
>>>   arm/gic.c   | 27 +++++++++++++++++++++++++++
>>>   arm/timer.c | 11 +++++++++++
>>>   2 files changed, 38 insertions(+)
>>>
>>> -- 
>>> 2.19.1
>>>
>>>
>>
>> Hi Jingyi,
>>
>> We already have an IPI latency test in arm/micro-bench.c I'd prefer that
>> one be used, if possible, rather than conflating the gic functional tests
>> with latency tests. Can you take a look at it and see if it satisfies
>> your needs, extending it if necessary?
> 
> I think it'd be good to have these interrupt latency measurements in
> kvm-unit-tests, and we can take the following interrupt types into
> account:
> 
> - IPI
>    As Drew pointed out, we already have one in the micro-bench group.
>    But what I'm actually interested in is the latency of the new GICv4.1
>    vSGIs (which will be directly injected through ITS).  To measure it,
>    we should first make KUT be GICv4.1-awareness, see [1] for details.
>    (This way, we can even have a look at the interrupt latency in HW
>    level. Is it possible to have this in kvm-unit-tests, Drew?)
> 
> - PPI
>    Like what has been done in patch #2, you can choose the vtimer
>    interrupt as an example.
> 
> - LPI
>    I think we can easily build a LPI latency test based on Eric's "ITS
>    tests" series [2], which should be upstreamed soon.
> 
> - if you want to add more...
> 
> What do you think? I'd like to see a V2 of this series :-).
> 
> 
> [1] 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0b04758b002bde9434053be2fff8064ac3d9d8bb 
> 
> [2] 
> https://lore.kernel.org/kvm/20200320092428.20880-1-eric.auger@redhat.com/
> 
> 
> Thanks,
> Zenghui
> 
> 
> .

As Drew mentioned, I am thinking about adding other gic functional tests
in arm/micro-bench.c. Thanks for your suggestion, I think it's of great
help.

Thanks,
Jingyi

Re: [kvm-unit-tests PATCH 0/2] arm/arm64: Add IPI/vtimer latency

[Andrew Jones]

On Thu, Apr 02, 2020 at 07:52:43PM +0800, Zenghui Yu wrote:
>   But what I'm actually interested in is the latency of the new GICv4.1
>   vSGIs (which will be directly injected through ITS).  To measure it,
>   we should first make KUT be GICv4.1-awareness, see [1] for details.
>   (This way, we can even have a look at the interrupt latency in HW
>   level. Is it possible to have this in kvm-unit-tests, Drew?)

I would certainly welcome gicv4 support in kvm-unit-tests. Let's get
Eric's ITS series merged first, but then patches welcome :-)

Thanks,
drew