* [PATCH v4 0/2] bugfix and optimization about CMD_SYNC @ 2018-08-19 7:51 Zhen Lei 2018-08-19 7:51 ` [PATCH v4 1/2] iommu/arm-smmu-v3: fix unexpected CMD_SYNC timeout Zhen Lei 2018-08-19 7:51 ` [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible Zhen Lei 0 siblings, 2 replies; 5+ messages in thread From: Zhen Lei @ 2018-08-19 7:51 UTC (permalink / raw) To: Robin Murphy, Will Deacon, Joerg Roedel, linux-arm-kernel, iommu, linux-kernel Cc: Zhen Lei, LinuxArm, Hanjun Guo, Libin, John Garry v3->v4: 1. create a new function arm_smmu_cmdq_build_sync_msi_cmd, it's only used to build CMD_SYNC for CS=SIG_IRQ mode. 2. In order to observe the optimization effect, I conducted 5 tests for each case. Although the test result is volatility, but we can still get which case is good or bad. Test command: fio -numjobs=8 -rw=randread -runtime=30 ... -bs=4k Test Result: IOPS Case 1: (without these patches) 675480 672055 665275 648610 661146 Case 2: (only apply the variant of patch 1, move arm_smmu_cmdq_build_cmd into lock) 688714 697355 632951 700540 678459 Case 3: (only apply patch 1) 721582 729226 689574 679710 727770 Case 4: (apply both patch 1 and patch 2) 734077 742868 738194 682544 740586 v2 -> v3: Although I have no data to show how many performance will be impacted because of arm_smmu_cmdq_build_cmd is protected by spinlock. But it's clear that the performance is bound to drop, a memset operation and a complicate switch..case in the function arm_smmu_cmdq_build_cmd. v1 -> v2: 1. move the call to arm_smmu_cmdq_build_cmd into the critical section, and keep itself unchange. 2. Although patch2 can make sure no two CMD_SYNCs will be adjacent, but patch1 is still needed, see below: cpu0 cpu1 cpu2 msidata=0 msidata=1 insert cmd1 insert a TLBI command insert cmd0 smmu execute cmd1 smmu execute TLBI smmu execute cmd0 poll timeout, because msidata=1 is overridden by cmd0, that means VAL=0, sync_idx=1. Zhen Lei (2): iommu/arm-smmu-v3: fix unexpected CMD_SYNC timeout iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible drivers/iommu/arm-smmu-v3.c | 44 ++++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-) -- 1.8.3 ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v4 1/2] iommu/arm-smmu-v3: fix unexpected CMD_SYNC timeout 2018-08-19 7:51 [PATCH v4 0/2] bugfix and optimization about CMD_SYNC Zhen Lei @ 2018-08-19 7:51 ` Zhen Lei 2018-08-19 7:51 ` [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible Zhen Lei 1 sibling, 0 replies; 5+ messages in thread From: Zhen Lei @ 2018-08-19 7:51 UTC (permalink / raw) To: Robin Murphy, Will Deacon, Joerg Roedel, linux-arm-kernel, iommu, linux-kernel Cc: Zhen Lei, LinuxArm, Hanjun Guo, Libin, John Garry The condition "(int)(VAL - sync_idx) >= 0" to break loop in function __arm_smmu_sync_poll_msi requires that sync_idx must be increased monotonously according to the sequence of the CMDs in the cmdq. But ".msidata = atomic_inc_return_relaxed(&smmu->sync_nr)" is not protected by spinlock, so the following scenarios may appear: cpu0 cpu1 msidata=0 msidata=1 insert cmd1 insert cmd0 smmu execute cmd1 smmu execute cmd0 poll timeout, because msidata=1 is overridden by cmd0, that means VAL=0, sync_idx=1. This is not a functional problem, just make the caller wait for a long time until TIMEOUT. It's rare to happen, because any other CMD_SYNCs during the waiting period will break it. Split the building of CMD_SYNC for SIG_IRQ mode into a single function, to ensure better performance and good coding style. Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> --- drivers/iommu/arm-smmu-v3.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 1d64710..ac6d6df 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -566,7 +566,7 @@ struct arm_smmu_device { int gerr_irq; int combined_irq; - atomic_t sync_nr; + u32 sync_nr; unsigned long ias; /* IPA */ unsigned long oas; /* PA */ @@ -775,6 +775,17 @@ static int queue_remove_raw(struct arm_smmu_queue *q, u64 *ent) return 0; } +static inline +void arm_smmu_cmdq_build_sync_msi_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) +{ + cmd[0] = FIELD_PREP(CMDQ_0_OP, ent->opcode); + cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ); + cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH); + cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB); + cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIDATA, ent->sync.msidata); + cmd[1] = ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK; +} + /* High-level queue accessors */ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) { @@ -830,14 +841,9 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) cmd[1] |= FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); break; case CMDQ_OP_CMD_SYNC: - if (ent->sync.msiaddr) - cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ); - else - cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); + cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH); cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB); - cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIDATA, ent->sync.msidata); - cmd[1] |= ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK; break; default: return -ENOENT; @@ -947,14 +953,13 @@ static int __arm_smmu_cmdq_issue_sync_msi(struct arm_smmu_device *smmu) struct arm_smmu_cmdq_ent ent = { .opcode = CMDQ_OP_CMD_SYNC, .sync = { - .msidata = atomic_inc_return_relaxed(&smmu->sync_nr), .msiaddr = virt_to_phys(&smmu->sync_count), }, }; - arm_smmu_cmdq_build_cmd(cmd, &ent); - spin_lock_irqsave(&smmu->cmdq.lock, flags); + ent.sync.msidata = ++smmu->sync_nr; + arm_smmu_cmdq_build_sync_msi_cmd(cmd, &ent); arm_smmu_cmdq_insert_cmd(smmu, cmd); spin_unlock_irqrestore(&smmu->cmdq.lock, flags); @@ -2179,7 +2184,6 @@ static int arm_smmu_init_structures(struct arm_smmu_device *smmu) { int ret; - atomic_set(&smmu->sync_nr, 0); ret = arm_smmu_init_queues(smmu); if (ret) return ret; -- 1.8.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible 2018-08-19 7:51 [PATCH v4 0/2] bugfix and optimization about CMD_SYNC Zhen Lei 2018-08-19 7:51 ` [PATCH v4 1/2] iommu/arm-smmu-v3: fix unexpected CMD_SYNC timeout Zhen Lei @ 2018-08-19 7:51 ` Zhen Lei 2018-08-30 11:18 ` John Garry 1 sibling, 1 reply; 5+ messages in thread From: Zhen Lei @ 2018-08-19 7:51 UTC (permalink / raw) To: Robin Murphy, Will Deacon, Joerg Roedel, linux-arm-kernel, iommu, linux-kernel Cc: Zhen Lei, LinuxArm, Hanjun Guo, Libin, John Garry More than two CMD_SYNCs maybe adjacent in the command queue, and the first one has done what others want to do. Drop the redundant CMD_SYNCs can improve IO performance especially under the pressure scene. I did the statistics in my test environment, the number of CMD_SYNCs can be reduced about 1/3. See below: CMD_SYNCs reduced: 19542181 CMD_SYNCs total: 58098548 (include reduced) CMDs total: 116197099 (TLBI:SYNC about 1:1) Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> --- drivers/iommu/arm-smmu-v3.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index ac6d6df..f3a56e1 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -567,6 +567,7 @@ struct arm_smmu_device { int gerr_irq; int combined_irq; u32 sync_nr; + u8 prev_cmd_opcode; unsigned long ias; /* IPA */ unsigned long oas; /* PA */ @@ -786,6 +787,11 @@ void arm_smmu_cmdq_build_sync_msi_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) cmd[1] = ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK; } +static inline u8 arm_smmu_cmd_opcode_get(u64 *cmd) +{ + return cmd[0] & CMDQ_0_OP; +} + /* High-level queue accessors */ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) { @@ -906,6 +912,8 @@ static void arm_smmu_cmdq_insert_cmd(struct arm_smmu_device *smmu, u64 *cmd) struct arm_smmu_queue *q = &smmu->cmdq.q; bool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV); + smmu->prev_cmd_opcode = arm_smmu_cmd_opcode_get(cmd); + while (queue_insert_raw(q, cmd) == -ENOSPC) { if (queue_poll_cons(q, false, wfe)) dev_err_ratelimited(smmu->dev, "CMDQ timeout\n"); @@ -958,9 +966,17 @@ static int __arm_smmu_cmdq_issue_sync_msi(struct arm_smmu_device *smmu) }; spin_lock_irqsave(&smmu->cmdq.lock, flags); - ent.sync.msidata = ++smmu->sync_nr; - arm_smmu_cmdq_build_sync_msi_cmd(cmd, &ent); - arm_smmu_cmdq_insert_cmd(smmu, cmd); + if (smmu->prev_cmd_opcode == CMDQ_OP_CMD_SYNC) { + /* + * Previous command is CMD_SYNC also, there is no need to add + * one more. Just poll it. + */ + ent.sync.msidata = smmu->sync_nr; + } else { + ent.sync.msidata = ++smmu->sync_nr; + arm_smmu_cmdq_build_sync_msi_cmd(cmd, &ent); + arm_smmu_cmdq_insert_cmd(smmu, cmd); + } spin_unlock_irqrestore(&smmu->cmdq.lock, flags); return __arm_smmu_sync_poll_msi(smmu, ent.sync.msidata); -- 1.8.3 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible 2018-08-19 7:51 ` [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible Zhen Lei @ 2018-08-30 11:18 ` John Garry 2018-09-05 1:15 ` Leizhen (ThunderTown) 0 siblings, 1 reply; 5+ messages in thread From: John Garry @ 2018-08-30 11:18 UTC (permalink / raw) To: Zhen Lei, Robin Murphy, Will Deacon, Joerg Roedel, linux-arm-kernel, iommu, linux-kernel Cc: LinuxArm, Hanjun Guo, Libin On 19/08/2018 08:51, Zhen Lei wrote: > More than two CMD_SYNCs maybe adjacent in the command queue, and the first > one has done what others want to do. Drop the redundant CMD_SYNCs can > improve IO performance especially under the pressure scene. > > I did the statistics in my test environment, the number of CMD_SYNCs can > be reduced about 1/3. See below: > CMD_SYNCs reduced: 19542181 > CMD_SYNCs total: 58098548 (include reduced) > CMDs total: 116197099 (TLBI:SYNC about 1:1) > > Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> > --- > drivers/iommu/arm-smmu-v3.c | 22 +++++++++++++++++++--- > 1 file changed, 19 insertions(+), 3 deletions(-) > > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c > index ac6d6df..f3a56e1 100644 > --- a/drivers/iommu/arm-smmu-v3.c > +++ b/drivers/iommu/arm-smmu-v3.c > @@ -567,6 +567,7 @@ struct arm_smmu_device { > int gerr_irq; > int combined_irq; > u32 sync_nr; > + u8 prev_cmd_opcode; > > unsigned long ias; /* IPA */ > unsigned long oas; /* PA */ > @@ -786,6 +787,11 @@ void arm_smmu_cmdq_build_sync_msi_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) > cmd[1] = ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK; > } > > +static inline u8 arm_smmu_cmd_opcode_get(u64 *cmd) > +{ > + return cmd[0] & CMDQ_0_OP; > +} > + > /* High-level queue accessors */ > static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) > { > @@ -906,6 +912,8 @@ static void arm_smmu_cmdq_insert_cmd(struct arm_smmu_device *smmu, u64 *cmd) > struct arm_smmu_queue *q = &smmu->cmdq.q; > bool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV); > > + smmu->prev_cmd_opcode = arm_smmu_cmd_opcode_get(cmd); > + > while (queue_insert_raw(q, cmd) == -ENOSPC) { > if (queue_poll_cons(q, false, wfe)) > dev_err_ratelimited(smmu->dev, "CMDQ timeout\n"); > @@ -958,9 +966,17 @@ static int __arm_smmu_cmdq_issue_sync_msi(struct arm_smmu_device *smmu) > }; > > spin_lock_irqsave(&smmu->cmdq.lock, flags); > - ent.sync.msidata = ++smmu->sync_nr; > - arm_smmu_cmdq_build_sync_msi_cmd(cmd, &ent); > - arm_smmu_cmdq_insert_cmd(smmu, cmd); > + if (smmu->prev_cmd_opcode == CMDQ_OP_CMD_SYNC) { > + /* > + * Previous command is CMD_SYNC also, there is no need to add > + * one more. Just poll it. > + */ > + ent.sync.msidata = smmu->sync_nr; > + } else { > + ent.sync.msidata = ++smmu->sync_nr; > + arm_smmu_cmdq_build_sync_msi_cmd(cmd, &ent); > + arm_smmu_cmdq_insert_cmd(smmu, cmd); > + } > spin_unlock_irqrestore(&smmu->cmdq.lock, flags); I find something like this adds support for combining CMD_SYNC commands for regular polling mode: @@ -569,6 +569,7 @@ struct arm_smmu_device { int combined_irq; u32 sync_nr; u8 prev_cmd_opcode; + int prev_cmd_sync_res; unsigned long ias; /* IPA */ unsigned long oas; /* PA */ @@ -985,17 +986,33 @@ static int __arm_smmu_cmdq_issue_sync_msi(struct arm_smmu_device *smmu) static int __arm_smmu_cmdq_issue_sync(struct arm_smmu_device *smmu) { - u64 cmd[CMDQ_ENT_DWORDS]; + static u64 cmd[CMDQ_ENT_DWORDS] = { + _FIELD_PREP(CMDQ_0_OP, CMDQ_OP_CMD_SYNC) | + _FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV) | + _FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH) | + _FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB) + }; unsigned long flags; bool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV); - struct arm_smmu_cmdq_ent ent = { .opcode = CMDQ_OP_CMD_SYNC }; - int ret; + int ret = 0; - arm_smmu_cmdq_build_cmd(cmd, &ent); spin_lock_irqsave(&smmu->cmdq.lock, flags); - arm_smmu_cmdq_insert_cmd(smmu, cmd); - ret = queue_poll_cons(&smmu->cmdq.q, true, wfe); + if (smmu->prev_cmd_opcode != CMDQ_OP_CMD_SYNC || + smmu->prev_cmd_sync_res != 0) { + arm_smmu_cmdq_insert_cmd(smmu, cmd); + smmu->prev_cmd_sync_res = ret = + queue_poll_cons(&smmu->cmdq.q, true, wfe); + } I tested iperf on a 1G network link and was seeing 6-10% CMD_SYNC commands combined. I would really need to test this on a faster connection to see any throughout difference. From the above figures, I think leizhen was seeing 25% combine rate, right? As for this code, it could be neatened... Cheers, John > > return __arm_smmu_sync_poll_msi(smmu, ent.sync.msidata); > -- > 1.8.3 > > > > . > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible 2018-08-30 11:18 ` John Garry @ 2018-09-05 1:15 ` Leizhen (ThunderTown) 0 siblings, 0 replies; 5+ messages in thread From: Leizhen (ThunderTown) @ 2018-09-05 1:15 UTC (permalink / raw) To: John Garry, Robin Murphy, Will Deacon, Joerg Roedel, linux-arm-kernel, iommu, linux-kernel Cc: LinuxArm, Hanjun Guo, Libin On 2018/8/30 19:18, John Garry wrote: > On 19/08/2018 08:51, Zhen Lei wrote: >> spin_unlock_irqrestore(&smmu->cmdq.lock, flags); > > I find something like this adds support for combining CMD_SYNC commands for regular polling mode: > > @@ -569,6 +569,7 @@ struct arm_smmu_device { > int combined_irq; > u32 sync_nr; > u8 prev_cmd_opcode; > + int prev_cmd_sync_res; > > unsigned long ias; /* IPA */ > unsigned long oas; /* PA */ > @@ -985,17 +986,33 @@ static int __arm_smmu_cmdq_issue_sync_msi(struct arm_smmu_device *smmu) > > static int __arm_smmu_cmdq_issue_sync(struct arm_smmu_device *smmu) > { > - u64 cmd[CMDQ_ENT_DWORDS]; > + static u64 cmd[CMDQ_ENT_DWORDS] = { > + _FIELD_PREP(CMDQ_0_OP, CMDQ_OP_CMD_SYNC) | > + _FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV) | > + _FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH) | > + _FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB) > + }; > unsigned long flags; > bool wfe = !!(smmu->features & ARM_SMMU_FEAT_SEV); > - struct arm_smmu_cmdq_ent ent = { .opcode = CMDQ_OP_CMD_SYNC }; > - int ret; > + int ret = 0; > > - arm_smmu_cmdq_build_cmd(cmd, &ent); > > spin_lock_irqsave(&smmu->cmdq.lock, flags); > - arm_smmu_cmdq_insert_cmd(smmu, cmd); > - ret = queue_poll_cons(&smmu->cmdq.q, true, wfe); > + if (smmu->prev_cmd_opcode != CMDQ_OP_CMD_SYNC || > + smmu->prev_cmd_sync_res != 0) { > + arm_smmu_cmdq_insert_cmd(smmu, cmd); > + smmu->prev_cmd_sync_res = ret = > + queue_poll_cons(&smmu->cmdq.q, true, wfe); > + } > > I tested iperf on a 1G network link and was seeing 6-10% CMD_SYNC commands combined. I would really need to test this on a faster connection to see any throughout difference. > > From the above figures, I think leizhen was seeing 25% combine rate, right? Yes. In my test case, the size of unmap are almost one page, that means 1 TLBI follows 1 SYNC, so the probability that two CMD_SYNCs next to each other will be greater. > > As for this code, it could be neatened... > > Cheers, > John > >> >> return __arm_smmu_sync_poll_msi(smmu, ent.sync.msidata); >> -- >> 1.8.3 >> >> >> >> . >> > > > > . > -- Thanks! BestRegards ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-09-05 1:15 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-08-19 7:51 [PATCH v4 0/2] bugfix and optimization about CMD_SYNC Zhen Lei 2018-08-19 7:51 ` [PATCH v4 1/2] iommu/arm-smmu-v3: fix unexpected CMD_SYNC timeout Zhen Lei 2018-08-19 7:51 ` [PATCH v4 2/2] iommu/arm-smmu-v3: avoid redundant CMD_SYNCs if possible Zhen Lei 2018-08-30 11:18 ` John Garry 2018-09-05 1:15 ` Leizhen (ThunderTown)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).