From: "zhangfei.gao@foxmail.com" <zhangfei.gao@foxmail.com>
To: Fenghua Yu <fenghua.yu@intel.com>
Cc: Dave Hansen <dave.hansen@intel.com>,
Joerg Roedel <joro@8bytes.org>,
jean-philippe <jean-philippe@linaro.org>,
Ravi V Shankar <ravi.v.shankar@intel.com>,
Tony Luck <tony.luck@intel.com>, Ashok Raj <ashok.raj@intel.com>,
Peter Zijlstra <peterz@infradead.org>,
Dave Hansen <dave.hansen@linux.intel.com>, x86 <x86@kernel.org>,
linux-kernel <linux-kernel@vger.kernel.org>,
iommu <iommu@lists.linux-foundation.org>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
Andy Lutomirski <luto@kernel.org>,
Josh Poimboeuf <jpoimboe@redhat.com>,
Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH v4 05/11] iommu/sva: Assign a PASID to mm on PASID allocation and free it on mm exit
Date: Thu, 14 Apr 2022 18:08:09 +0800 [thread overview]
Message-ID: <tencent_8B6D7835F62688B4CD069C0EFC41B308B407@qq.com> (raw)
In-Reply-To: <tencent_A9458C6CEBAADD361DA765356477B00E920A@qq.com>
On 2022/4/12 下午11:35, zhangfei.gao@foxmail.com wrote:
> Hi, Fenghua
>
> On 2022/4/12 下午9:41, Fenghua Yu wrote:
>> Hi, Zhangfei,
>>
>> On Tue, Apr 12, 2022 at 03:04:09PM +0800, zhangfei.gao@foxmail.com
>> wrote:
>>>
>>> On 2022/4/11 下午10:52, Dave Hansen wrote:
>>>> On 4/11/22 07:44, zhangfei.gao@foxmail.com wrote:
>>>>> On 2022/4/11 下午10:36, Dave Hansen wrote:
>>>>>> On 4/11/22 07:20, zhangfei.gao@foxmail.com wrote:
>>>>>>>> Is there nothing before this call trace? Usually there will be
>>>>>>>> at least
>>>>>>>> some warning text.
>>>>>>> I added dump_stack() in ioasid_free.
>>>>>> Hold on a sec, though...
>>>>>>
>>>>>> What's the *problem* here? Did something break or are you just
>>>>>> saying
>>>>>> that something looks weird to _you_?
>>>>> After this, nginx is not working at all, and hardware reports error.
>>>>> Suppose the the master use the ioasid for init, but got freed.
>>>>>
>>>>> hardware reports:
>>>>> [ 152.731869] hisi_sec2 0000:76:00.0: qm_acc_do_task_timeout
>>>>> [error status=0x20] found
>>>>> [ 152.739657] hisi_sec2 0000:76:00.0: qm_acc_wb_not_ready_timeout
>>>>> [error status=0x40] found
>>>>> [ 152.747877] hisi_sec2 0000:76:00.0: sec_fsm_hbeat_rint [error
>>>>> status=0x20] found
>>>>> [ 152.755340] hisi_sec2 0000:76:00.0: Controller resetting...
>>>>> [ 152.762044] hisi_sec2 0000:76:00.0: QM mailbox operation timeout!
>>>>> [ 152.768198] hisi_sec2 0000:76:00.0: Failed to dump sqc!
>>>>> [ 152.773490] hisi_sec2 0000:76:00.0: Failed to drain out data
>>>>> for stopping!
>>>>> [ 152.781426] hisi_sec2 0000:76:00.0: QM mailbox is busy to start!
>>>>> [ 152.787468] hisi_sec2 0000:76:00.0: Failed to dump sqc!
>>>>> [ 152.792753] hisi_sec2 0000:76:00.0: Failed to drain out data
>>>>> for stopping!
>>>>> [ 152.800685] hisi_sec2 0000:76:00.0: QM mailbox is busy to start!
>>>>> [ 152.806730] hisi_sec2 0000:76:00.0: Failed to dump sqc!
>>>>> [ 152.812017] hisi_sec2 0000:76:00.0: Failed to drain out data
>>>>> for stopping!
>>>>> [ 152.819946] hisi_sec2 0000:76:00.0: QM mailbox is busy to start!
>>>>> [ 152.825992] hisi_sec2 0000:76:00.0: Failed to dump sqc!
>>>> That would have been awfully handy information to have in an
>>>> initial bug report. :)
>>>> Is there a chance you could dump out that ioasid alloc *and* free
>>>> information in ioasid_alloc/free()? This could be some kind of
>>>> problem with the allocator, or with copying the ioasid at fork.
>>> The issue is nginx master process init resource, start daemon
>>> process, then
>>> master process quit and free ioasid.
>>> The daemon nginx process is not the original master process.
>>>
>>> master process: init resource
>>> driver -> iommu_sva_bind_device -> ioasid_alloc
>> Which code in the master process/daemon calls
>> driver->iommu_sva_unbind_device?
> Our calling sequence is nginx -> openssl -> openssl engine -> kernel
> driver
> The calling entrence should be ngx_ssl_init : OPENSSL_config(NULL);
>
> nginx:
> src/event/ngx_event_openssl.c
> ngx_ssl_init
> if (OPENSSL_init_ssl(OPENSSL_INIT_LOAD_CONFIG, NULL) == 0)
>
> I add some print.
>
> /usr/local/nginx$ sudo sbin/nginx
> ngx_ssl_init pid=2361
> bind_fn
> ngx_openssl_create_conf pid=2361
> hisi sec init Kunpeng920!
> ngx_ssl_create pid=2361
> ngx_ssl_certificates pid=2361
> ngx_ssl_certificate pid=2361
> uadk_e_wd_digest_init
> hisi sec init Kunpeng920!
> ngx_ssl_ciphers pid=2361
> ngx_daemon pid=2361 fork daemon
> master pid=2361 will exit // here
> master process is exit
> fork return 0 pid=2364 // here
> daemon process started
> ngx_daemon fork ngx_pid=2364, ngx_parent=2361
> $ ps -aux | grep nginx
> root 2364 0.0 0.0 31324 15380 ? Ssl 15:21 0:00
> nginx: master process sbin/nginx
> nobody 2366 0.0 0.0 32304 16448 ? Sl 15:21 0:00
> nginx: worker process
> linaro 2371 0.0 0.0 7696 2048 pts/0 S+ 15:22 0:00
> grep --color=auto nginx
>
> nginx
> src/os/unix/ngx_daemon.c
> ngx_daemon(ngx_log_t *log)
> {
> int fd;
>
> switch (fork()) {
> case -1:
> ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "fork() failed");
> return NGX_ERROR;
>
> case 0:
> // here fork daemon process
> break;
>
> default:
> // master process directly exit, and release mm as well as ioasid
> exit(0);
> }
>
> // only daemon process
> ngx_parent = ngx_pid;
> ngx_pid = ngx_getpid();
>>> Any suggestion?
>> ioasid is per process or per mm. A daemon process shouldn't share the
>> same
>> ioasid with any other process with even its parent process. Its
>> parent gets
>> an ioasid and frees it on exit. The ioasid is gone and shouldn't be used
>> by its child process.
>>
>> Each daemon process should call driver -> iommu_sva_bind_device ->
>> ioasid_alloc
>> to get its own ioasid/PASID. On daemon quit, the ioasid is freed.
>>
>> That means nqnix needs to be changed.
>
> Agree with Dave, I think user space should not be broken.
>
> Thanks
Any plan about this regression?
Currently I need this patch to workaround the issue.
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 22ddd05bbdcd..2d74ac53d11c 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -4,6 +4,7 @@
*/
#include <linux/mm.h>
+#include <linux/sched/mm.h>
#include <linux/mmu_context.h>
#include <linux/mmu_notifier.h>
#include <linux/slab.h>
@@ -363,6 +364,7 @@ arm_smmu_sva_bind(struct device *dev, struct
mm_struct *mm, void *drvdata)
mutex_lock(&sva_lock);
handle = __arm_smmu_sva_bind(dev, mm);
+ mmget(mm);
mutex_unlock(&sva_lock);
return handle;
}
@@ -377,6 +379,7 @@ void arm_smmu_sva_unbind(struct iommu_sva *handle)
arm_smmu_mmu_notifier_put(bond->smmu_mn);
kfree(bond);
}
+ mmput(bond->mm);
mutex_unlock(&sva_lock);
}
Thanks
next prev parent reply other threads:[~2022-04-14 10:08 UTC|newest]
Thread overview: 113+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-07 23:02 [PATCH v4 00/11] Re-enable ENQCMD and PASID MSR Fenghua Yu
2022-02-07 23:02 ` [PATCH v4 01/11] iommu/sva: Rename CONFIG_IOMMU_SVA_LIB to CONFIG_IOMMU_SVA Fenghua Yu
2022-02-08 2:39 ` Lu Baolu
2022-02-15 10:54 ` [tip: x86/pasid] " tip-bot2 for Fenghua Yu
2022-02-07 23:02 ` [PATCH v4 02/11] mm: Change CONFIG option for mm->pasid field Fenghua Yu
2022-02-08 2:40 ` Lu Baolu
2022-02-15 10:54 ` [tip: x86/pasid] " tip-bot2 for Fenghua Yu
2022-02-07 23:02 ` [PATCH v4 03/11] iommu/ioasid: Introduce a helper to check for valid PASIDs Fenghua Yu
2022-02-08 2:40 ` Lu Baolu
2022-02-15 10:54 ` [tip: x86/pasid] " tip-bot2 for Fenghua Yu
2022-02-07 23:02 ` [PATCH v4 04/11] kernel/fork: Initialize mm's PASID Fenghua Yu
2022-02-14 17:23 ` Thomas Gleixner
2022-02-15 10:54 ` [tip: x86/pasid] " tip-bot2 for Fenghua Yu
2022-02-07 23:02 ` [PATCH v4 05/11] iommu/sva: Assign a PASID to mm on PASID allocation and free it on mm exit Fenghua Yu
2022-02-08 2:41 ` Lu Baolu
2022-02-08 15:01 ` Fenghua Yu
2022-02-10 3:16 ` Jacob Pan
2022-02-10 16:27 ` Fenghua Yu
2022-02-10 17:24 ` Luck, Tony
2022-02-10 18:31 ` Fenghua Yu
2022-02-10 23:52 ` Fenghua Yu
2022-02-10 18:49 ` Jacob Pan
2022-02-10 23:15 ` Fenghua Yu
2022-02-11 22:00 ` Dave Hansen
2022-02-14 17:24 ` Thomas Gleixner
2022-02-15 9:55 ` Joerg Roedel
[not found] ` <tencent_F6830A1196DB4C6A904D7C691F0D961D1108@qq.com>
2022-04-11 14:10 ` Dave Hansen
2022-04-11 14:20 ` zhangfei.gao
2022-04-11 14:36 ` Dave Hansen
2022-04-11 14:44 ` zhangfei.gao
2022-04-11 14:52 ` Dave Hansen
2022-04-11 15:13 ` zhangfei.gao
2022-04-12 7:04 ` zhangfei.gao
2022-04-12 13:41 ` Fenghua Yu
2022-04-12 14:39 ` Dave Hansen
2022-04-15 9:59 ` Fenghua Yu
2022-04-12 15:35 ` zhangfei.gao
2022-04-14 10:08 ` zhangfei.gao [this message]
2022-04-15 9:51 ` Fenghua Yu
[not found] ` <99bcb9f5-4776-9c40-a776-cdecfa9e1010@foxmail.com>
2022-04-15 10:14 ` zhangfei.gao
2022-04-15 10:50 ` Fenghua Yu
2022-04-15 11:52 ` zhangfei.gao
2022-04-15 12:37 ` Fenghua Yu
2022-04-16 1:30 ` zhangfei.gao
2022-04-15 19:07 ` Fenghua Yu
2022-04-15 21:00 ` Jacob Pan
2022-04-16 1:43 ` zhangfei.gao
2022-04-18 18:14 ` Jacob Pan
2022-04-19 1:02 ` zhangfei.gao
2022-04-18 6:34 ` Tian, Kevin
2022-04-18 18:11 ` Jacob Pan
2022-04-20 16:45 ` Jean-Philippe Brucker
2022-04-21 6:47 ` zhangfei.gao
2022-04-22 9:03 ` zhangfei.gao
2022-04-22 10:11 ` Jean-Philippe Brucker
2022-04-22 13:15 ` zhangfei.gao
2022-04-22 15:50 ` Jean-Philippe Brucker
2022-04-23 11:13 ` zhangfei.gao
2022-04-24 2:58 ` Zhangfei Gao
2022-04-24 9:52 ` Zhangfei Gao
2022-04-25 13:53 ` Jean-Philippe Brucker
2022-04-25 14:18 ` Dave Hansen
2022-04-25 14:26 ` Jean-Philippe Brucker
2022-04-25 15:34 ` Jacob Pan
2022-04-25 16:13 ` Jean-Philippe Brucker
2022-04-25 22:32 ` Jacob Pan
2022-04-26 4:20 ` Fenghua Yu
2022-04-26 5:04 ` Zhangfei Gao
2022-04-28 0:54 ` Fenghua Yu
2022-04-28 8:43 ` Jean-Philippe Brucker
2022-04-28 15:09 ` Dave Hansen
2022-04-28 15:28 ` Fenghua Yu
2022-04-28 15:42 ` Dave Hansen
2022-04-28 16:01 ` Jean-Philippe Brucker
2022-04-28 16:35 ` Dave Hansen
2022-04-26 4:28 ` Zhangfei Gao
2022-04-26 4:36 ` Fenghua Yu
2022-04-26 5:19 ` Zhangfei Gao
2022-04-25 15:55 ` Dave Hansen
2022-04-25 16:40 ` Jean-Philippe Brucker
2022-04-26 15:27 ` Dave Hansen
2022-04-26 16:48 ` Jean-Philippe Brucker
2022-04-26 23:31 ` Dave Hansen
2022-04-28 8:39 ` Jean-Philippe Brucker
2022-04-29 7:53 ` Baolu Lu
2022-04-29 13:51 ` Fenghua Yu
2022-04-29 14:34 ` Jean-Philippe Brucker
2022-04-29 22:19 ` Fenghua Yu
2022-04-30 7:33 ` Baolu Lu
2022-05-03 7:49 ` Jean-Philippe Brucker
2022-05-06 5:36 ` Baolu Lu
2022-04-12 14:36 ` Dave Hansen
2022-04-12 15:10 ` Jean-Philippe Brucker
2022-04-12 15:35 ` Dave Hansen
2022-04-13 11:14 ` Lu Baolu
2022-04-25 2:57 ` zhangfei.gao
2022-02-15 10:54 ` [tip: x86/pasid] " tip-bot2 for Fenghua Yu
2022-02-07 23:02 ` [PATCH v4 06/11] x86/fpu: Clear PASID when copying fpstate Fenghua Yu
2022-02-15 10:54 ` [tip: x86/pasid] " tip-bot2 for Fenghua Yu
2022-02-07 23:02 ` [PATCH v4 07/11] sched: Define and initialize a flag to identify valid PASID in the task Fenghua Yu
2022-02-15 10:54 ` [tip: x86/pasid] " tip-bot2 for Peter Zijlstra
2022-02-07 23:02 ` [PATCH v4 08/11] x86/traps: Demand-populate PASID MSR via #GP Fenghua Yu
2022-02-15 10:54 ` [tip: x86/pasid] " tip-bot2 for Fenghua Yu
2022-02-07 23:02 ` [PATCH v4 09/11] x86/cpufeatures: Re-enable ENQCMD Fenghua Yu
2022-02-15 10:54 ` [tip: x86/pasid] " tip-bot2 for Fenghua Yu
2022-02-07 23:02 ` [PATCH v4 10/11] tools/objtool: Check for use of the ENQCMD instruction in the kernel Fenghua Yu
2022-02-15 10:54 ` [tip: x86/pasid] " tip-bot2 for Fenghua Yu
2022-03-09 7:55 ` [tip: x86/core] " tip-bot2 for Fenghua Yu
2022-03-15 10:44 ` tip-bot2 for Fenghua Yu
2022-02-07 23:02 ` [PATCH v4 11/11] docs: x86: Change documentation for SVA (Shared Virtual Addressing) Fenghua Yu
2022-02-14 17:25 ` Thomas Gleixner
2022-02-15 10:54 ` [tip: x86/pasid] Documentation/x86: Update " tip-bot2 for Fenghua Yu
2022-02-11 20:04 ` [PATCH v4 00/11] Re-enable ENQCMD and PASID MSR Fenghua Yu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=tencent_8B6D7835F62688B4CD069C0EFC41B308B407@qq.com \
--to=zhangfei.gao@foxmail.com \
--cc=ashok.raj@intel.com \
--cc=bp@alien8.de \
--cc=dave.hansen@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=fenghua.yu@intel.com \
--cc=iommu@lists.linux-foundation.org \
--cc=jean-philippe@linaro.org \
--cc=joro@8bytes.org \
--cc=jpoimboe@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=ravi.v.shankar@intel.com \
--cc=tglx@linutronix.de \
--cc=tony.luck@intel.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).