From: "Paul E. McKenney" <paulmck@kernel.org>
To: "代子为 (Ziwei Dai)" <Ziwei.Dai@unisoc.com>
Cc: "Joel Fernandes" <joel@joelfernandes.org>,
"urezki@gmail.com" <urezki@gmail.com>,
"frederic@kernel.org" <frederic@kernel.org>,
"quic_neeraju@quicinc.com" <quic_neeraju@quicinc.com>,
"josh@joshtriplett.org" <josh@joshtriplett.org>,
"rostedt@goodmis.org" <rostedt@goodmis.org>,
"mathieu.desnoyers@efficios.com" <mathieu.desnoyers@efficios.com>,
"jiangshanlai@gmail.com" <jiangshanlai@gmail.com>,
"rcu@vger.kernel.org" <rcu@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"王双 (Shuang Wang)" <shuang.wang@unisoc.com>,
"辛依凡 (Yifan Xin)" <Yifan.Xin@unisoc.com>,
"王科 (Ke Wang)" <Ke.Wang@unisoc.com>,
"闫学文 (Xuewen Yan)" <Xuewen.Yan@unisoc.com>,
"牛志国 (Zhiguo Niu)" <Zhiguo.Niu@unisoc.com>,
"黄朝阳 (Zhaoyang Huang)" <zhaoyang.huang@unisoc.com>
Subject: Re: 答复: [PATCH V2] rcu: Make sure new krcp free business is handled after the wanted rcu grace period.
Date: Wed, 5 Apr 2023 20:46:15 -0700 [thread overview]
Message-ID: <6ff4eb09-13df-403f-ba4a-c5abb0f3fa8f@paulmck-laptop> (raw)
In-Reply-To: <94c28b46617e46d1804a397a54f9fd8d@BJMBX01.spreadtrum.com>
On Thu, Apr 06, 2023 at 01:38:09AM +0000, 代子为 (Ziwei Dai) wrote:
>
>
> > -----邮件原件-----
> > 发件人: Paul E. McKenney <paulmck@kernel.org>
> > 发送时间: 2023年4月6日 2:46
> > 收件人: Joel Fernandes <joel@joelfernandes.org>
> > 抄送: 代子为 (Ziwei Dai) <Ziwei.Dai@unisoc.com>; urezki@gmail.com; frederic@kernel.org; quic_neeraju@quicinc.com;
> > josh@joshtriplett.org; rostedt@goodmis.org; mathieu.desnoyers@efficios.com; jiangshanlai@gmail.com; rcu@vger.kernel.org;
> > linux-kernel@vger.kernel.org; 王双 (Shuang Wang) <shuang.wang@unisoc.com>; 辛依凡 (Yifan Xin) <Yifan.Xin@unisoc.com>; 王科
> > (Ke Wang) <Ke.Wang@unisoc.com>; 闫学文 (Xuewen Yan) <Xuewen.Yan@unisoc.com>; 牛志国 (Zhiguo Niu)
> > <Zhiguo.Niu@unisoc.com>; 黄朝阳 (Zhaoyang Huang) <zhaoyang.huang@unisoc.com>
> > 主题: Re: [PATCH V2] rcu: Make sure new krcp free business is handled after the wanted rcu grace period.
> >
> >
> > 注意: 这封邮件来自于外部。除非你确定邮件内容安全,否则不要点击任何链接和附件。
> > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the
> > sender and know the content is safe.
> >
> >
> >
> > On Wed, Apr 05, 2023 at 02:12:02PM -0400, Joel Fernandes wrote:
> > > On Wed, Apr 5, 2023 at 1:39 PM Joel Fernandes <joel@joelfernandes.org> wrote:
> > > >
> > > > On Fri, Mar 31, 2023 at 8:43 AM Ziwei Dai <ziwei.dai@unisoc.com> wrote:
> > > > >
> > > > > In kfree_rcu_monitor(), new free business at krcp is attached to
> > > > > any free channel at krwp. kfree_rcu_monitor() is responsible to
> > > > > make sure new free business is handled after the rcu grace period.
> > > > > But if there is any none-free channel at krwp already, that means
> > > > > there is an on-going rcu work, which will cause the
> > > > > kvfree_call_rcu()-triggered free business is done before the wanted rcu grace period ends.
> > > > >
> > > > > This commit ignore krwp which has non-free channel at
> > > > > kfree_rcu_monitor(), to fix the issue that kvfree_call_rcu() loses effectiveness.
> > > > >
> > > > > Below is the css_set obj "from_cset" use-after-free case caused by
> > > > > kvfree_call_rcu() losing effectiveness.
> > > > > CPU 0 calls rcu_read_lock(), then use "from_cset", then hard irq
> > > > > comes, the task is schedule out.
> > > > > CPU 1 calls kfree_rcu(cset, rcu_head), willing to free "from_cset" after new gp.
> > > > > But "from_cset" is freed right after current gp end. "from_cset" is reallocated.
> > > > > CPU 0 's task arrives back, references "from_cset"'s member, which causes crash.
> > > > >
> > > > > CPU 0 CPU 1
> > > > > count_memcg_event_mm()
> > > > > |rcu_read_lock() <---
> > > > > |mem_cgroup_from_task()
> > > > > |// css_set_ptr is the "from_cset" mentioned on CPU 1
> > > > > |css_set_ptr = rcu_dereference((task)->cgroups) |// Hard irq
> > > > > comes, current task is scheduled out.
> > > > >
> > > > > cgroup_attach_task()
> > > > > |cgroup_migrate()
> > > > > |cgroup_migrate_execute()
> > > > > |css_set_move_task(task, from_cset, to_cset, true)
> > > > > |cgroup_move_task(task, to_cset)
> > > > > |rcu_assign_pointer(.., to_cset)
> > > > > |...
> > > > > |cgroup_migrate_finish()
> > > > > |put_css_set_locked(from_cset)
> > > > > |from_cset->refcount return 0
> > > > > |kfree_rcu(cset, rcu_head) // means to free from_cset after new gp
> > > > > |add_ptr_to_bulk_krc_lock()
> > > > >
> > > > > |schedule_delayed_work(&krcp->monitor_work, ..)
> > > > >
> > > > > kfree_rcu_monitor()
> > > > > |krcp->bulk_head[0]'s work attached to krwp->bulk_head_free[]
> > > > > |queue_rcu_work(system_wq, &krwp->rcu_work)
> > > > > |if rwork->rcu.work is not in WORK_STRUCT_PENDING_BIT state,
> > > > > |call_rcu(&rwork->rcu,
> > > > > rcu_work_rcufn) <--- request a new gp
> > > > >
> > > > > // There is a perious call_rcu(.., rcu_work_rcufn)
> > > > > // gp end, rcu_work_rcufn() is called.
> > > > > rcu_work_rcufn()
> > > > > |__queue_work(..,
> > > > > rwork->wq, &rwork->work);
> > > > >
> > > > > |kfree_rcu_work()
> > > > > |krwp->bulk_head_free[0] bulk is freed before new gp end!!!
> > > > > |The "from_cset" is freed before new gp end.
> > > > >
> > > > > // the task is scheduled in after many ms.
> > > > > |css_set_ptr->subsys[(subsys_id) <--- Caused kernel crash, because css_set_ptr is freed.
> > > > >
> > > > > v2: Use helper function instead of inserted code block at kfree_rcu_monitor().
> > > > >
> > > > > Fixes: c014efeef76a ("rcu: Add multiple in-flight batches of
> > > > > kfree_rcu() work")
> > > > > Signed-off-by: Ziwei Dai <ziwei.dai@unisoc.com>
> > > >
> > > > Please update the fixes tag to:
> > > > 5f3c8d620447 ("rcu/tree: Maintain separate array for vmalloc ptrs")
> > >
> > > Vlad pointed out in another thread that the fix is actually to 34c881745549.
> > >
> > > So just to be sure, it could be updated to:
> > > Fixes: 34c881745549 ("rcu: Support kfree_bulk() interface in
> > > kfree_rcu()")
> > > Fixes: 5f3c8d620447 ("rcu/tree: Maintain separate array for vmalloc
> > > ptrs")
> >
> > Ziwei Dai, does this change in Fixes look good to you?
> >
> > If so, I will update the commit log in this commit that I am planning to submit into v6.3. It is strictly speaking not a v6.3 regression,
> > but it is starting to show up in the wild and the patch is contained enough to be considered an urgent fix.
> >
> > Thanx, Paul
>
> Hi Paul, it looks good to me and thanks!
Thank you, and I will fix on my next rebase.
Thanx, Paul
next prev parent reply other threads:[~2023-04-06 3:46 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-31 12:42 [PATCH V2] rcu: Make sure new krcp free business is handled after the wanted rcu grace period Ziwei Dai
2023-03-31 15:01 ` Uladzislau Rezki
2023-04-03 7:28 ` Uladzislau Rezki
2023-04-03 14:01 ` Mukesh Ojha
2023-04-03 22:58 ` Paul E. McKenney
2023-04-04 2:49 ` 答复: " 代子为 (Ziwei Dai)
2023-04-04 3:23 ` Paul E. McKenney
2023-04-05 17:39 ` Joel Fernandes
2023-04-05 18:12 ` Joel Fernandes
2023-04-05 18:46 ` Paul E. McKenney
2023-04-06 1:38 ` 答复: " 代子为 (Ziwei Dai)
2023-04-06 3:46 ` Paul E. McKenney [this message]
2023-04-06 4:44 ` Uladzislau Rezki
2023-04-06 14:13 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6ff4eb09-13df-403f-ba4a-c5abb0f3fa8f@paulmck-laptop \
--to=paulmck@kernel.org \
--cc=Ke.Wang@unisoc.com \
--cc=Xuewen.Yan@unisoc.com \
--cc=Yifan.Xin@unisoc.com \
--cc=Zhiguo.Niu@unisoc.com \
--cc=Ziwei.Dai@unisoc.com \
--cc=frederic@kernel.org \
--cc=jiangshanlai@gmail.com \
--cc=joel@joelfernandes.org \
--cc=josh@joshtriplett.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=quic_neeraju@quicinc.com \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=shuang.wang@unisoc.com \
--cc=urezki@gmail.com \
--cc=zhaoyang.huang@unisoc.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).