* [PATCH v4 0/2] bugfix and optimization about CMD_SYNC
@ 2018-08-19 7:51 Zhen Lei
2018-08-19 7:51 ` [PATCH v4 1/2] iommu/arm-smmu-v3: fix unexpected CMD_SYNC timeout Zhen Lei
2018-08-19 7:51 ` [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible Zhen Lei
0 siblings, 2 replies; 5+ messages in thread
From: Zhen Lei @ 2018-08-19 7:51 UTC (permalink / raw)
To: Robin Murphy, Will Deacon, Joerg Roedel, linux-arm-kernel, iommu,
linux-kernel
Cc: Zhen Lei, LinuxArm, Hanjun Guo, Libin, John Garry
v3->v4:
1. create a new function arm_smmu_cmdq_build_sync_msi_cmd, it's only used to
build CMD_SYNC for CS=SIG_IRQ mode.
2. In order to observe the optimization effect, I conducted 5 tests for each
case. Although the test result is volatility, but we can still get which case
is good or bad.
Test command: fio -numjobs=8 -rw=randread -runtime=30 ... -bs=4k
Test Result: IOPS
Case 1: (without these patches)
675480
672055
665275
648610
661146
Case 2: (only apply the variant of patch 1, move arm_smmu_cmdq_build_cmd into lock)
688714
697355
632951
700540
678459
Case 3: (only apply patch 1)
721582
729226
689574
679710
727770
Case 4: (apply both patch 1 and patch 2)
734077
742868
738194
682544
740586
v2 -> v3:
Although I have no data to show how many performance will be impacted
because of arm_smmu_cmdq_build_cmd is protected by spinlock. But it's
clear that the performance is bound to drop, a memset operation and
a complicate switch..case in the function arm_smmu_cmdq_build_cmd.
v1 -> v2:
1. move the call to arm_smmu_cmdq_build_cmd into the critical section,
and keep itself unchange.
2. Although patch2 can make sure no two CMD_SYNCs will be adjacent,
but patch1 is still needed, see below:
cpu0 cpu1 cpu2
msidata=0
msidata=1
insert cmd1
insert a TLBI command
insert cmd0
smmu execute cmd1
smmu execute TLBI
smmu execute cmd0
poll timeout, because msidata=1 is overridden by
cmd0, that means VAL=0, sync_idx=1.
Zhen Lei (2):
iommu/arm-smmu-v3: fix unexpected CMD_SYNC timeout
iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible
drivers/iommu/arm-smmu-v3.c | 44 ++++++++++++++++++++++++++++++++------------
1 file changed, 32 insertions(+), 12 deletions(-)
--
1.8.3
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v4 1/2] iommu/arm-smmu-v3: fix unexpected CMD_SYNC timeout
2018-08-19 7:51 [PATCH v4 0/2] bugfix and optimization about CMD_SYNC Zhen Lei
@ 2018-08-19 7:51 ` Zhen Lei
2018-08-19 7:51 ` [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible Zhen Lei
1 sibling, 0 replies; 5+ messages in thread
From: Zhen Lei @ 2018-08-19 7:51 UTC (permalink / raw)
To: Robin Murphy, Will Deacon, Joerg Roedel, linux-arm-kernel, iommu,
linux-kernel
Cc: Zhen Lei, LinuxArm, Hanjun Guo, Libin, John Garry
The condition "(int)(VAL - sync_idx) >= 0" to break loop in function
__arm_smmu_sync_poll_msi requires that sync_idx must be increased
monotonously according to the sequence of the CMDs in the cmdq.
But ".msidata = atomic_inc_return_relaxed(&smmu->sync_nr)" is not protected
by spinlock, so the following scenarios may appear:
cpu0 cpu1
msidata=0
msidata=1
insert cmd1
insert cmd0
smmu execute cmd1
smmu execute cmd0
poll timeout, because msidata=1 is overridden by
cmd0, that means VAL=0, sync_idx=1.
This is not a functional problem, just make the caller wait for a long
time until TIMEOUT. It's rare to happen, because any other CMD_SYNCs
during the waiting period will break it.
Split the building of CMD_SYNC for SIG_IRQ mode into a single function,
to ensure better performance and good coding style.
Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
drivers/iommu/arm-smmu-v3.c | 26 +++++++++++++++-----------
1 file changed, 15 insertions(+), 11 deletions(-)
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 1d64710..ac6d6df 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -566,7 +566,7 @@ struct arm_smmu_device {
int gerr_irq;
int combined_irq;
- atomic_t sync_nr;
+ u32 sync_nr;
unsigned long ias; /* IPA */
unsigned long oas; /* PA */
@@ -775,6 +775,17 @@ static int queue_remove_raw(struct arm_smmu_queue *q, u64 *ent)
return 0;
}
+static inline
+void arm_smmu_cmdq_build_sync_msi_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
+{
+ cmd[0] = FIELD_PREP(CMDQ_0_OP, ent->opcode);
+ cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ);
+ cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH);
+ cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB);
+ cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIDATA, ent->sync.msidata);
+ cmd[1] = ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK;
+}
+
/* High-level queue accessors */
static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
{
@@ -830,14 +841,9 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
cmd[1] |= FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp);
break;
case CMDQ_OP_CMD_SYNC:
- if (ent->sync.msiaddr)
- cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ);
- else
- cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV);
+ cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV);
cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH);
cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB);
- cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIDATA, ent->sync.msidata);
- cmd[1] |= ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK;
break;
default:
return -ENOENT;
@@ -947,14 +953,13 @@ static int __arm_smmu_cmdq_issue_sync_msi(struct arm_smmu_device *smmu)
struct arm_smmu_cmdq_ent ent = {
.opcode = CMDQ_OP_CMD_SYNC,
.sync = {
- .msidata = atomic_inc_return_relaxed(&smmu->sync_nr),
.msiaddr = virt_to_phys(&smmu->sync_count),
},
};
- arm_smmu_cmdq_build_cmd(cmd, &ent);
-
spin_lock_irqsave(&smmu->cmdq.lock, flags);
+ ent.sync.msidata = ++smmu->sync_nr;
+ arm_smmu_cmdq_build_sync_msi_cmd(cmd, &ent);
arm_smmu_cmdq_insert_cmd(smmu, cmd);
spin_unlock_irqrestore(&smmu->cmdq.lock, flags);
@@ -2179,7 +2184,6 @@ static int arm_smmu_init_structures(struct arm_smmu_device *smmu)
{
int ret;
- atomic_set(&smmu->sync_nr, 0);
ret = arm_smmu_init_queues(smmu);
if (ret)
return ret;
--
1.8.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible
2018-08-19 7:51 [PATCH v4 0/2] bugfix and optimization about CMD_SYNC Zhen Lei
2018-08-19 7:51 ` [PATCH v4 1/2] iommu/arm-smmu-v3: fix unexpected CMD_SYNC timeout Zhen Lei
@ 2018-08-19 7:51 ` Zhen Lei
2018-08-30 11:18 ` John Garry
1 sibling, 1 reply; 5+ messages in thread
From: Zhen Lei @ 2018-08-19 7:51 UTC (permalink / raw)
To: Robin Murphy, Will Deacon, Joerg Roedel, linux-arm-kernel, iommu,
linux-kernel
Cc: Zhen Lei, LinuxArm, Hanjun Guo, Libin, John Garry
More than two CMD_SYNCs maybe adjacent in the command queue, and the first
one has done what others want to do. Drop the redundant CMD_SYNCs can
improve IO performance especially under the pressure scene.
I did the statistics in my test environment, the number of CMD_SYNCs can
be reduced about 1/3. See below:
CMD_SYNCs reduced: 19542181
CMD_SYNCs total: 58098548 (include reduced)
CMDs total: 116197099 (TLBI:SYNC about 1:1)
Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
drivers/iommu/arm-smmu-v3.c | 22 +++++++++++++++++++---
1 file changed, 19 insertions(+), 3 deletions(-)
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index ac6d6df..f3a56e1 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -567,6 +567,7 @@ struct arm_smmu_device {
int gerr_irq;
int combined_irq;
u32 sync_nr;
+ u8 prev_cmd_opcode;
unsigned long ias; /* IPA */
unsigned long oas; /* PA */
@@ -786,6 +787,11 @@ void arm_smmu_cmdq_build_sync_msi_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
cmd[1] = ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK;
}
+static inline u8 arm_smmu_cmd_opcode_get(u64 *cmd)
+{
+ return cmd[0] & CMDQ_0_OP;
+}
+
/* High-level queue accessors */
static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
{
@@ -906,6 +912,8 @@ static void arm_smmu_cmdq_insert_cmd(struct arm_smmu_device *smmu, u64 *cmd)
struct arm_smmu_queue *q = &smmu->cmdq.q;
bool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV);
+ smmu->prev_cmd_opcode = arm_smmu_cmd_opcode_get(cmd);
+
while (queue_insert_raw(q, cmd) == -ENOSPC) {
if (queue_poll_cons(q, false, wfe))
dev_err_ratelimited(smmu->dev, "CMDQ timeout\n");
@@ -958,9 +966,17 @@ static int __arm_smmu_cmdq_issue_sync_msi(struct arm_smmu_device *smmu)
};
spin_lock_irqsave(&smmu->cmdq.lock, flags);
- ent.sync.msidata = ++smmu->sync_nr;
- arm_smmu_cmdq_build_sync_msi_cmd(cmd, &ent);
- arm_smmu_cmdq_insert_cmd(smmu, cmd);
+ if (smmu->prev_cmd_opcode == CMDQ_OP_CMD_SYNC) {
+ /*
+ * Previous command is CMD_SYNC also, there is no need to add
+ * one more. Just poll it.
+ */
+ ent.sync.msidata = smmu->sync_nr;
+ } else {
+ ent.sync.msidata = ++smmu->sync_nr;
+ arm_smmu_cmdq_build_sync_msi_cmd(cmd, &ent);
+ arm_smmu_cmdq_insert_cmd(smmu, cmd);
+ }
spin_unlock_irqrestore(&smmu->cmdq.lock, flags);
return __arm_smmu_sync_poll_msi(smmu, ent.sync.msidata);
--
1.8.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible
2018-08-19 7:51 ` [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible Zhen Lei
@ 2018-08-30 11:18 ` John Garry
2018-09-05 1:15 ` Leizhen (ThunderTown)
0 siblings, 1 reply; 5+ messages in thread
From: John Garry @ 2018-08-30 11:18 UTC (permalink / raw)
To: Zhen Lei, Robin Murphy, Will Deacon, Joerg Roedel,
linux-arm-kernel, iommu, linux-kernel
Cc: LinuxArm, Hanjun Guo, Libin
On 19/08/2018 08:51, Zhen Lei wrote:
> More than two CMD_SYNCs maybe adjacent in the command queue, and the first
> one has done what others want to do. Drop the redundant CMD_SYNCs can
> improve IO performance especially under the pressure scene.
>
> I did the statistics in my test environment, the number of CMD_SYNCs can
> be reduced about 1/3. See below:
> CMD_SYNCs reduced: 19542181
> CMD_SYNCs total: 58098548 (include reduced)
> CMDs total: 116197099 (TLBI:SYNC about 1:1)
>
> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
> ---
> drivers/iommu/arm-smmu-v3.c | 22 +++++++++++++++++++---
> 1 file changed, 19 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
> index ac6d6df..f3a56e1 100644
> --- a/drivers/iommu/arm-smmu-v3.c
> +++ b/drivers/iommu/arm-smmu-v3.c
> @@ -567,6 +567,7 @@ struct arm_smmu_device {
> int gerr_irq;
> int combined_irq;
> u32 sync_nr;
> + u8 prev_cmd_opcode;
>
> unsigned long ias; /* IPA */
> unsigned long oas; /* PA */
> @@ -786,6 +787,11 @@ void arm_smmu_cmdq_build_sync_msi_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
> cmd[1] = ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK;
> }
>
> +static inline u8 arm_smmu_cmd_opcode_get(u64 *cmd)
> +{
> + return cmd[0] & CMDQ_0_OP;
> +}
> +
> /* High-level queue accessors */
> static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
> {
> @@ -906,6 +912,8 @@ static void arm_smmu_cmdq_insert_cmd(struct arm_smmu_device *smmu, u64 *cmd)
> struct arm_smmu_queue *q = &smmu->cmdq.q;
> bool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV);
>
> + smmu->prev_cmd_opcode = arm_smmu_cmd_opcode_get(cmd);
> +
> while (queue_insert_raw(q, cmd) == -ENOSPC) {
> if (queue_poll_cons(q, false, wfe))
> dev_err_ratelimited(smmu->dev, "CMDQ timeout\n");
> @@ -958,9 +966,17 @@ static int __arm_smmu_cmdq_issue_sync_msi(struct arm_smmu_device *smmu)
> };
>
> spin_lock_irqsave(&smmu->cmdq.lock, flags);
> - ent.sync.msidata = ++smmu->sync_nr;
> - arm_smmu_cmdq_build_sync_msi_cmd(cmd, &ent);
> - arm_smmu_cmdq_insert_cmd(smmu, cmd);
> + if (smmu->prev_cmd_opcode == CMDQ_OP_CMD_SYNC) {
> + /*
> + * Previous command is CMD_SYNC also, there is no need to add
> + * one more. Just poll it.
> + */
> + ent.sync.msidata = smmu->sync_nr;
> + } else {
> + ent.sync.msidata = ++smmu->sync_nr;
> + arm_smmu_cmdq_build_sync_msi_cmd(cmd, &ent);
> + arm_smmu_cmdq_insert_cmd(smmu, cmd);
> + }
> spin_unlock_irqrestore(&smmu->cmdq.lock, flags);
I find something like this adds support for combining CMD_SYNC commands
for regular polling mode:
@@ -569,6 +569,7 @@ struct arm_smmu_device {
int combined_irq;
u32 sync_nr;
u8 prev_cmd_opcode;
+ int prev_cmd_sync_res;
unsigned long ias; /* IPA */
unsigned long oas; /* PA */
@@ -985,17 +986,33 @@ static int __arm_smmu_cmdq_issue_sync_msi(struct
arm_smmu_device *smmu)
static int __arm_smmu_cmdq_issue_sync(struct arm_smmu_device *smmu)
{
- u64 cmd[CMDQ_ENT_DWORDS];
+ static u64 cmd[CMDQ_ENT_DWORDS] = {
+ _FIELD_PREP(CMDQ_0_OP, CMDQ_OP_CMD_SYNC) |
+ _FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV) |
+ _FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH) |
+ _FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB)
+ };
unsigned long flags;
bool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV);
- struct arm_smmu_cmdq_ent ent = { .opcode = CMDQ_OP_CMD_SYNC };
- int ret;
+ int ret = 0;
- arm_smmu_cmdq_build_cmd(cmd, &ent);
spin_lock_irqsave(&smmu->cmdq.lock, flags);
- arm_smmu_cmdq_insert_cmd(smmu, cmd);
- ret = queue_poll_cons(&smmu->cmdq.q, true, wfe);
+ if (smmu->prev_cmd_opcode != CMDQ_OP_CMD_SYNC ||
+ smmu->prev_cmd_sync_res != 0) {
+ arm_smmu_cmdq_insert_cmd(smmu, cmd);
+ smmu->prev_cmd_sync_res = ret =
+ queue_poll_cons(&smmu->cmdq.q, true, wfe);
+ }
I tested iperf on a 1G network link and was seeing 6-10% CMD_SYNC
commands combined. I would really need to test this on a faster
connection to see any throughout difference.
From the above figures, I think leizhen was seeing 25% combine rate, right?
As for this code, it could be neatened...
Cheers,
John
>
> return __arm_smmu_sync_poll_msi(smmu, ent.sync.msidata);
> --
> 1.8.3
>
>
>
> .
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible
2018-08-30 11:18 ` John Garry
@ 2018-09-05 1:15 ` Leizhen (ThunderTown)
0 siblings, 0 replies; 5+ messages in thread
From: Leizhen (ThunderTown) @ 2018-09-05 1:15 UTC (permalink / raw)
To: John Garry, Robin Murphy, Will Deacon, Joerg Roedel,
linux-arm-kernel, iommu, linux-kernel
Cc: LinuxArm, Hanjun Guo, Libin
On 2018/8/30 19:18, John Garry wrote:
> On 19/08/2018 08:51, Zhen Lei wrote:
>> spin_unlock_irqrestore(&smmu->cmdq.lock, flags);
>
> I find something like this adds support for combining CMD_SYNC commands for regular polling mode:
>
> @@ -569,6 +569,7 @@ struct arm_smmu_device {
> int combined_irq;
> u32 sync_nr;
> u8 prev_cmd_opcode;
> + int prev_cmd_sync_res;
>
> unsigned long ias; /* IPA */
> unsigned long oas; /* PA */
> @@ -985,17 +986,33 @@ static int __arm_smmu_cmdq_issue_sync_msi(struct arm_smmu_device *smmu)
>
> static int __arm_smmu_cmdq_issue_sync(struct arm_smmu_device *smmu)
> {
> - u64 cmd[CMDQ_ENT_DWORDS];
> + static u64 cmd[CMDQ_ENT_DWORDS] = {
> + _FIELD_PREP(CMDQ_0_OP, CMDQ_OP_CMD_SYNC) |
> + _FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV) |
> + _FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH) |
> + _FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB)
> + };
> unsigned long flags;
> bool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV);
> - struct arm_smmu_cmdq_ent ent = { .opcode = CMDQ_OP_CMD_SYNC };
> - int ret;
> + int ret = 0;
>
> - arm_smmu_cmdq_build_cmd(cmd, &ent);
>
> spin_lock_irqsave(&smmu->cmdq.lock, flags);
> - arm_smmu_cmdq_insert_cmd(smmu, cmd);
> - ret = queue_poll_cons(&smmu->cmdq.q, true, wfe);
> + if (smmu->prev_cmd_opcode != CMDQ_OP_CMD_SYNC ||
> + smmu->prev_cmd_sync_res != 0) {
> + arm_smmu_cmdq_insert_cmd(smmu, cmd);
> + smmu->prev_cmd_sync_res = ret =
> + queue_poll_cons(&smmu->cmdq.q, true, wfe);
> + }
>
> I tested iperf on a 1G network link and was seeing 6-10% CMD_SYNC commands combined. I would really need to test this on a faster connection to see any throughout difference.
>
> From the above figures, I think leizhen was seeing 25% combine rate, right?
Yes. In my test case, the size of unmap are almost one page, that means 1 TLBI follows 1 SYNC,
so the probability that two CMD_SYNCs next to each other will be greater.
>
> As for this code, it could be neatened...
>
> Cheers,
> John
>
>>
>> return __arm_smmu_sync_poll_msi(smmu, ent.sync.msidata);
>> --
>> 1.8.3
>>
>>
>>
>> .
>>
>
>
>
> .
>
--
Thanks!
BestRegards
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-09-05 1:15 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-08-19 7:51 [PATCH v4 0/2] bugfix and optimization about CMD_SYNC Zhen Lei
2018-08-19 7:51 ` [PATCH v4 1/2] iommu/arm-smmu-v3: fix unexpected CMD_SYNC timeout Zhen Lei
2018-08-19 7:51 ` [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible Zhen Lei
2018-08-30 11:18 ` John Garry
2018-09-05 1:15 ` Leizhen (ThunderTown)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).