From: Petr Mladek <pmladek@suse.com>
To: Qian Cai <cai@lca.pw>
Cc: Michal Hocko <mhocko@kernel.org>,
sergey.senozhatsky.work@gmail.com, rostedt@goodmis.org,
peterz@infradead.org, linux-mm@kvack.org,
john.ogness@linutronix.de, akpm@linux-foundation.org,
david@redhat.com, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] mm/page_isolation: fix a deadlock with printk()
Date: Mon, 7 Oct 2019 16:10:59 +0200 [thread overview]
Message-ID: <20191007141059.friotqx2ymwvlo3j@pathway.suse.cz> (raw)
In-Reply-To: <1570453622.5576.288.camel@lca.pw>
On Mon 2019-10-07 09:07:02, Qian Cai wrote:
> On Mon, 2019-10-07 at 14:43 +0200, Michal Hocko wrote:
> > On Mon 07-10-19 08:11:44, Qian Cai wrote:
> > > On Mon, 2019-10-07 at 13:37 +0200, Michal Hocko wrote:
> > > > On Mon 07-10-19 07:04:00, Qian Cai wrote:
> > > > >
> > > > >
> > > > > > On Oct 7, 2019, at 4:07 AM, Michal Hocko <mhocko@kernel.org> wrote:
> > > > > >
> > > > > > I do not think that removing the printk is the right long term solution.
> > > > > > While I do agree that removing the debugging printk __offline_isolated_pages
> > > > > > does make sense because it is essentially of a very limited use, this
> > > > > > doesn't really solve the underlying problem. There are likely other
> > > > > > printks from zone->lock. It would be much more saner to actually
> > > > > > disallow consoles to allocate any memory while printk is called from an
> > > > > > atomic context.
> > > > >
> > > > > No, there is only a handful of places called printk() from
> > > > > zone->lock. It is normal that the callers will quietly process
> > > > > “struct zone” modification in a short section with zone->lock
> > > > > held.
> > > >
> > > > It is extremely error prone to have any zone->lock vs. printk
> > > > dependency. I do not want to play an endless whack a mole.
> > > >
> > > > > No, it is not about “allocate any memory while printk is called from an
> > > > > atomic context”. It is opposite lock chain from different processors which has the same effect. For example,
> > > > >
> > > > > CPU0: CPU1: CPU2:
> > > > > console_owner
> > > > > sclp_lock
> > > > > sclp_lock zone_lock
> > > > > zone_lock
> > > > > console_owner
> > > >
> > > > Why would sclp_lock ever take a zone->lock (apart from an allocation).
> > > > So really if sclp_lock is a lock that might be taken from many contexts
> > > > and generate very subtle lock dependencies then it should better be
> > > > really careful what it is calling into.
> > > >
> > > > In other words you are trying to fix a wrong end of the problem. Fix the
> > > > console to not allocate or depend on MM by other means.
> > >
> > > It looks there are way too many places that could generate those indirect lock
> > > chains that are hard to eliminate them all. Here is anther example, where it
> > > has,
> >
> > Yeah and I strongly suspect they are consoles which are broken and need
> > to be fixed rathert than the problem papered over.
> >
> > I do realize how tempting it is to remove all printks from the
> > zone->lock but do realize that as soon as the allocator starts using any
> > other locks then we are back to square one and the problem is there
> > again. We would have to drop _all_ printks from any locked section in
> > the allocator and I do not think this is viable.
> >
> > Really, the only way forward is to make these consoles be more careful
> > of external dependencies.
>
> Even with the new printk() Petr proposed. There is no guarantee it will fix it
> properly. It looks like just reduce the chance of this kind of deadlocks as it
> may or may not call wake_up_klogd() in vprintk_emit() depends on timing.
The chain below is wrong:
> zone->lock
> printk_deferred()
> vprintk_emit()
> wake_up_klogd()
wake_up_klogd() calls irq_work_queue(). It queues the work for
an interrupt handler and triggers the interrupt.
> wake_up_klogd_work_func()
> console_unlock()
The work is done in the interrupt context. The interrupt could
never be handled under zone->lock.
So, printk_deferred() would help. But I do not think that it is
really needed. I am going to answer the original mail with
all the full lockdep report.
Best Regards,
Petr
next prev parent reply other threads:[~2019-10-07 14:11 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-04 22:26 [PATCH v2] mm/page_isolation: fix a deadlock with printk() Qian Cai
2019-10-07 8:07 ` Michal Hocko
2019-10-07 9:05 ` Petr Mladek
2019-10-07 11:33 ` Michal Hocko
2019-10-07 12:34 ` Qian Cai
[not found] ` <FB72D947-A0F9-43E7-80D9-D7ACE33849C7@lca.pw>
2019-10-07 11:37 ` Michal Hocko
2019-10-07 12:11 ` Qian Cai
2019-10-07 12:43 ` Michal Hocko
2019-10-07 13:07 ` Qian Cai
2019-10-07 14:10 ` Petr Mladek [this message]
2019-10-07 14:30 ` Petr Mladek
2019-10-07 14:49 ` Michal Hocko
2019-10-08 7:43 ` Petr Mladek
2019-10-08 8:27 ` Michal Hocko
2019-10-08 12:56 ` Christian Borntraeger
2019-10-08 16:08 ` Qian Cai
2019-10-08 18:35 ` Michal Hocko
2019-10-08 19:06 ` Qian Cai
2019-10-08 19:17 ` Michal Hocko
2019-10-08 19:35 ` Qian Cai
2019-10-09 11:49 ` Petr Mladek
2019-10-09 13:06 ` Qian Cai
2019-10-09 13:27 ` Michal Hocko
2019-10-09 13:43 ` Qian Cai
2019-10-09 13:51 ` Michal Hocko
2019-10-09 14:19 ` Qian Cai
2019-10-09 14:34 ` Michal Hocko
2019-10-09 15:08 ` Qian Cai
2019-10-09 16:23 ` Michal Hocko
2019-10-10 9:01 ` Qian Cai
2019-10-10 10:59 ` Michal Hocko
2019-10-10 13:11 ` Qian Cai
2019-10-10 14:18 ` Michal Hocko
2019-10-10 14:47 ` Qian Cai
2019-10-10 17:30 ` Michal Hocko
2019-10-10 17:48 ` Qian Cai
2019-10-10 18:06 ` Michal Hocko
2019-10-10 18:59 ` David Hildenbrand
2019-10-09 14:24 ` Petr Mladek
2019-10-09 14:46 ` Qian Cai
2019-10-10 7:57 ` Petr Mladek
2019-10-09 11:39 ` Petr Mladek
2019-10-09 13:56 ` Peter Oberparleiter
2019-10-09 14:26 ` Michal Hocko
2019-10-10 5:12 ` Sergey Senozhatsky
2019-10-10 7:40 ` Michal Hocko
2019-10-10 8:16 ` Sergey Senozhatsky
2019-10-10 8:37 ` Michal Hocko
2019-10-10 8:21 ` Petr Mladek
2019-10-10 8:39 ` Sergey Senozhatsky
2019-10-10 11:11 ` Petr Mladek
2019-10-09 15:25 ` Qian Cai
2019-10-07 14:59 ` Qian Cai
2019-10-07 15:12 ` Michal Hocko
2019-10-07 15:33 ` Qian Cai
2019-10-08 8:15 ` Petr Mladek
2019-10-08 9:32 ` Qian Cai
2019-10-08 13:13 ` Steven Rostedt
2019-10-08 13:23 ` Qian Cai
2019-10-08 13:33 ` Steven Rostedt
2019-10-08 13:42 ` Petr Mladek
2019-10-08 13:48 ` Michal Hocko
2019-10-08 14:03 ` Qian Cai
2019-10-08 14:08 ` Michal Hocko
2019-10-08 8:40 ` Michal Hocko
2019-10-08 10:04 ` Qian Cai
2019-10-08 10:39 ` Michal Hocko
2019-10-08 12:00 ` Qian Cai
2019-10-08 12:39 ` Michal Hocko
2019-10-08 13:06 ` Qian Cai
2019-10-08 13:37 ` Michal Hocko
2019-10-08 13:08 ` Petr Mladek
2019-10-08 13:33 ` Qian Cai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191007141059.friotqx2ymwvlo3j@pathway.suse.cz \
--to=pmladek@suse.com \
--cc=akpm@linux-foundation.org \
--cc=cai@lca.pw \
--cc=david@redhat.com \
--cc=john.ogness@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=sergey.senozhatsky.work@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).