kvm.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Maxim Levitsky <mlevitsk@redhat.com>
To: Jiri Slaby <jirislaby@kernel.org>,
	Pedro Falcato <pedro.falcato@gmail.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>,
	kvm@vger.kernel.org, Andrew Morton <akpm@linux-foundation.org>,
	mm <linux-mm@kvack.org>,
	yuzhao@google.com, Michal Hocko <MHocko@suse.com>,
	Vlastimil Babka <vbabka@suse.cz>,
	shy828301@gmail.com
Subject: Re: Stalls in qemu with host running 6.1 (everything stuck at mmap_read_lock())
Date: Thu, 12 Jan 2023 12:31:29 +0200	[thread overview]
Message-ID: <6d989ca2330748ed682c81fc5f43e054a70e70a8.camel@redhat.com> (raw)
In-Reply-To: <7aa90802-d25c-baa3-9c03-2502ad3c708a@kernel.org>

On Thu, 2023-01-12 at 07:07 +0100, Jiri Slaby wrote:
> Hi,
> 
> On 12. 01. 23, 1:37, Pedro Falcato wrote:
> > I just want to chime in and say that I've also hit this regression
> > right as I (Arch) updated to 6.1 a few weeks ago.
> > This completely ruined my qemu workflow such that I had to fallback to
> > using an LTS kernel.
> > 
> > Some data I've gathered:
> > 1) It seems to not happen right after booting - I'm unsure if this is
> > due to memory pressure or less CPU load or any other factor
> 
> +1 as I wrote.
> 
> > 2) It seems to intensify after swapping a fair amount? At least this
> > has been my experience.
> 
> I have no swap.
> 
> > 3) The largest slowdown seems to be when qemu is booting the guest,
> > possibly during heavy memory allocation - problems range from "takes
> > tens of seconds to boot" to "qemu is completely blocked and needs a
> > SIGKILL spam".
> 
> +1
> 
> > 4) While traditional process monitoring tools break (likely due to
> > mmap_lock getting hogged), I can (empirically, using /bin/free) tell
> > that the system seems to be swapping in/out quite a fair bit
> 
> Yes, htop/top/ps and such are stuck at the read of /proc/<pid>/cmdline 
> as I wrote (waiting for the mmap lock).
> 
> > My 4) is particularly confusing to me as I had originally blamed the
> > problem on the MGLRU changes, while you don't seem to be swapping at
> > all.
> > Could this be related to the maple tree patches? Should we CC both the
> > MGLRU folks and the maple folks?
> > 
> > I have little insight into what the kernel's state actually is apart
> > from this - perf seems to break, and I have no kernel debugger as this
> > is my live personal machine :/
> > I would love it if someone hinted to possible things I/we could try in
> > order to track this down. Is this not git-bisectable at all?
> 
> I have rebooted to a fresh kernel which 1) have lockdep enabled, and 2) 
> I have debuginfo for. So next time this happens, I can print held locks 
> and dump a kcore (kdump is set up).
> 
> regards,

It is also possible that I noticed something like that on 6.1:

For me it happens when my system (also no swap, 96G out which 48 are permanetly reserved as 1G hugepages,
and this happens with VMs which don't use this hugepage reserve) 
is somewhat low on memory and qemu tries to lock all memory (I use -overcommit mem-lock=on)

Like it usually happens when I start 32 GB VM while having lot of stuff open in background, but
still not nearly close to 16GB.
As a workaround I lowered the reserved area to 32G.

I also see indication that things like htop or even opening a new shell hang quite hard.

What almost instantly helps is 'echo 3 | sudo tee /proc/sys/vm/drop_caches'
e.g that makes the VM start booting, and unlocks everything.

Best regards,
	Maxim Levitsky


  reply	other threads:[~2023-01-12 10:39 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-01-11  8:00 Stalls in qemu with host running 6.1 (everything stuck at mmap_read_lock()) Jiri Slaby
2023-01-11  8:23 ` Vlastimil Babka
2023-01-11  8:31 ` Michal Hocko
2023-01-11  9:21 ` Linux kernel regression tracking (#adding)
2023-01-19 14:58   ` Vlastimil Babka
2023-01-11 22:50 ` Paolo Bonzini
2023-01-12  0:37 ` Pedro Falcato
2023-01-12  6:07   ` Jiri Slaby
2023-01-12 10:31     ` Maxim Levitsky [this message]
2023-01-12 10:35       ` Vlastimil Babka
2023-01-12 15:43     ` Jiri Slaby
2023-01-12 16:57       ` Jiri Slaby
2023-01-13 17:14         ` Liam Howlett
2023-01-12  8:23   ` Yu Zhao
2023-01-13 17:27     ` Liam Howlett

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6d989ca2330748ed682c81fc5f43e054a70e70a8.camel@redhat.com \
    --to=mlevitsk@redhat.com \
    --cc=MHocko@suse.com \
    --cc=akpm@linux-foundation.org \
    --cc=jirislaby@kernel.org \
    --cc=kvm@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=pbonzini@redhat.com \
    --cc=pedro.falcato@gmail.com \
    --cc=shy828301@gmail.com \
    --cc=vbabka@suse.cz \
    --cc=yuzhao@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).