kvm.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jiri Slaby <jirislaby@kernel.org>
To: Pedro Falcato <pedro.falcato@gmail.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>,
	kvm@vger.kernel.org, Andrew Morton <akpm@linux-foundation.org>,
	mm <linux-mm@kvack.org>,
	yuzhao@google.com, Michal Hocko <MHocko@suse.com>,
	Vlastimil Babka <vbabka@suse.cz>,
	shy828301@gmail.com
Subject: Re: Stalls in qemu with host running 6.1 (everything stuck at mmap_read_lock())
Date: Thu, 12 Jan 2023 07:07:26 +0100	[thread overview]
Message-ID: <7aa90802-d25c-baa3-9c03-2502ad3c708a@kernel.org> (raw)
In-Reply-To: <CAKbZUD0Tqct_G9OcO8ocdH1J_YvLSEod-ofr97hsyoHgcvBwuw@mail.gmail.com>

Hi,

On 12. 01. 23, 1:37, Pedro Falcato wrote:
> I just want to chime in and say that I've also hit this regression
> right as I (Arch) updated to 6.1 a few weeks ago.
> This completely ruined my qemu workflow such that I had to fallback to
> using an LTS kernel.
> 
> Some data I've gathered:
> 1) It seems to not happen right after booting - I'm unsure if this is
> due to memory pressure or less CPU load or any other factor

+1 as I wrote.

> 2) It seems to intensify after swapping a fair amount? At least this
> has been my experience.

I have no swap.

> 3) The largest slowdown seems to be when qemu is booting the guest,
> possibly during heavy memory allocation - problems range from "takes
> tens of seconds to boot" to "qemu is completely blocked and needs a
> SIGKILL spam".

+1

> 4) While traditional process monitoring tools break (likely due to
> mmap_lock getting hogged), I can (empirically, using /bin/free) tell
> that the system seems to be swapping in/out quite a fair bit

Yes, htop/top/ps and such are stuck at the read of /proc/<pid>/cmdline 
as I wrote (waiting for the mmap lock).

> My 4) is particularly confusing to me as I had originally blamed the
> problem on the MGLRU changes, while you don't seem to be swapping at
> all.
> Could this be related to the maple tree patches? Should we CC both the
> MGLRU folks and the maple folks?
> 
> I have little insight into what the kernel's state actually is apart
> from this - perf seems to break, and I have no kernel debugger as this
> is my live personal machine :/
> I would love it if someone hinted to possible things I/we could try in
> order to track this down. Is this not git-bisectable at all?

I have rebooted to a fresh kernel which 1) have lockdep enabled, and 2) 
I have debuginfo for. So next time this happens, I can print held locks 
and dump a kcore (kdump is set up).

regards,
-- 
js
suse labs


  reply	other threads:[~2023-01-12  6:07 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-01-11  8:00 Stalls in qemu with host running 6.1 (everything stuck at mmap_read_lock()) Jiri Slaby
2023-01-11  8:23 ` Vlastimil Babka
2023-01-11  8:31 ` Michal Hocko
2023-01-11  9:21 ` Linux kernel regression tracking (#adding)
2023-01-19 14:58   ` Vlastimil Babka
2023-01-11 22:50 ` Paolo Bonzini
2023-01-12  0:37 ` Pedro Falcato
2023-01-12  6:07   ` Jiri Slaby [this message]
2023-01-12 10:31     ` Maxim Levitsky
2023-01-12 10:35       ` Vlastimil Babka
2023-01-12 15:43     ` Jiri Slaby
2023-01-12 16:57       ` Jiri Slaby
2023-01-13 17:14         ` Liam Howlett
2023-01-12  8:23   ` Yu Zhao
2023-01-13 17:27     ` Liam Howlett

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=7aa90802-d25c-baa3-9c03-2502ad3c708a@kernel.org \
    --to=jirislaby@kernel.org \
    --cc=MHocko@suse.com \
    --cc=akpm@linux-foundation.org \
    --cc=kvm@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=pbonzini@redhat.com \
    --cc=pedro.falcato@gmail.com \
    --cc=shy828301@gmail.com \
    --cc=vbabka@suse.cz \
    --cc=yuzhao@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).