kvm.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Yu Zhao <yuzhao@google.com>
To: Pedro Falcato <pedro.falcato@gmail.com>,
	Liam Howlett <liam.howlett@oracle.com>
Cc: Jiri Slaby <jirislaby@kernel.org>,
	Paolo Bonzini <pbonzini@redhat.com>,
	kvm@vger.kernel.org, Andrew Morton <akpm@linux-foundation.org>,
	mm <linux-mm@kvack.org>, Michal Hocko <MHocko@suse.com>,
	Vlastimil Babka <vbabka@suse.cz>,
	shy828301@gmail.com
Subject: Re: Stalls in qemu with host running 6.1 (everything stuck at mmap_read_lock())
Date: Thu, 12 Jan 2023 01:23:17 -0700	[thread overview]
Message-ID: <CAOUHufapeXp_EFC2L5TFhLcVD95Wap2Zir8zHOEsANLV5CLdXQ@mail.gmail.com> (raw)
In-Reply-To: <CAKbZUD0Tqct_G9OcO8ocdH1J_YvLSEod-ofr97hsyoHgcvBwuw@mail.gmail.com>

On Wed, Jan 11, 2023 at 5:37 PM Pedro Falcato <pedro.falcato@gmail.com> wrote:
>
> On Wed, Jan 11, 2023 at 8:00 AM Jiri Slaby <jirislaby@kernel.org> wrote:
> >
> > Hi,
> >
> > after I updated the host from 6.0 to 6.1 (being at 6.1.4 ATM), my qemu
> > VMs started stalling (and the host at the same point too). It doesn't
> > happen right after boot, maybe a suspend-resume cycle is needed (or
> > longer uptime, or a couple of qemu VM starts, or ...). But when it
> > happens, it happens all the time till the next reboot.
> >
> > Older guest's kernels/distros are affected as well as Win10.
> >
> > In guests, I see for example stalls in memset_orig or
> > smp_call_function_many_cond -- traces below.
> >
> > qemu-kvm-7.1.0-13.34.x86_64 from openSUSE.
> >
> > It's quite interesting that:
> >    $ cat /proc/<PID_OF_QEMU>/cmdline
> > is stuck at read:
> >
> > openat(AT_FDCWD, "/proc/12239/cmdline", O_RDONLY) = 3
> > newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
> > fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
> > mmap(NULL, 139264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> > 0) = 0x7f22f0487000
> > read(3, ^C^C^C^\^C
> >
> > too. So I dumped blocked tasks (sysrq-w) on _host_ (see below) and
> > everything seems to stall on mmap_read_lock() or
> > mmap_write_lock_killable(). I don't see the hog (the one actually
> > _having_ and sitting on the (presumably write) lock) in the dump though.
> > I will perhaps boot a LOCKDEP-enabled kernel, so that I can do sysrq-d
> > next time and see the holder.
> >
> >
> > There should be enough free memory (note caches at 8G):
> >                 total        used        free      shared  buff/cache
> > available
> > Mem:            15Gi        10Gi       400Mi       2,5Gi       8,0Gi
> >    5,0Gi
> > Swap:             0B          0B          0B
> >
> >
> > I rmmoded kvm-intel now, so:
> >    qemu-kvm: failed to initialize kvm: No such file or directory
> >    qemu-kvm: falling back to tcg
> > and it behaves the same (more or less expected).
> >
> > Is this known? Any idea how to debug this? Or maybe someone (I CCed a
> > couple of guys who Acked mmap_*_lock() shuffling patches in 6.1) has a
> > clue? Bisection is hard as it reproduces only under certain unknown
> > circumstances.
>
> Hi,
>
> I just want to chime in and say that I've also hit this regression
> right as I (Arch) updated to 6.1 a few weeks ago.
> This completely ruined my qemu workflow such that I had to fallback to
> using an LTS kernel.
>
> Some data I've gathered:
> 1) It seems to not happen right after booting - I'm unsure if this is
> due to memory pressure or less CPU load or any other factor
> 2) It seems to intensify after swapping a fair amount? At least this
> has been my experience.
> 3) The largest slowdown seems to be when qemu is booting the guest,
> possibly during heavy memory allocation - problems range from "takes
> tens of seconds to boot" to "qemu is completely blocked and needs a
> SIGKILL spam".
> 4) While traditional process monitoring tools break (likely due to
> mmap_lock getting hogged), I can (empirically, using /bin/free) tell
> that the system seems to be swapping in/out quite a fair bit
>
> My 4) is particularly confusing to me as I had originally blamed the
> problem on the MGLRU changes, while you don't seem to be swapping at
> all.
> Could this be related to the maple tree patches? Should we CC both the
> MGLRU folks and the maple folks?

I don't think it's MGLRU because the way it uses mmap_lock is very
simple. Also you could prevent MGLRU from taking mmap_lock by echo 3
>/sys/kernel/mm/lru_gen/enabled, or you could disable MGLRU entirely
by echoing 0 to the same file, or even at build time, to rule it out.
(I assume you
turned on MGLRU in the first place.)

Adding Liam. He can speak for the maple tree.

  parent reply	other threads:[~2023-01-12  8:24 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-01-11  8:00 Stalls in qemu with host running 6.1 (everything stuck at mmap_read_lock()) Jiri Slaby
2023-01-11  8:23 ` Vlastimil Babka
2023-01-11  8:31 ` Michal Hocko
2023-01-11  9:21 ` Linux kernel regression tracking (#adding)
2023-01-19 14:58   ` Vlastimil Babka
2023-01-11 22:50 ` Paolo Bonzini
2023-01-12  0:37 ` Pedro Falcato
2023-01-12  6:07   ` Jiri Slaby
2023-01-12 10:31     ` Maxim Levitsky
2023-01-12 10:35       ` Vlastimil Babka
2023-01-12 15:43     ` Jiri Slaby
2023-01-12 16:57       ` Jiri Slaby
2023-01-13 17:14         ` Liam Howlett
2023-01-12  8:23   ` Yu Zhao [this message]
2023-01-13 17:27     ` Liam Howlett

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAOUHufapeXp_EFC2L5TFhLcVD95Wap2Zir8zHOEsANLV5CLdXQ@mail.gmail.com \
    --to=yuzhao@google.com \
    --cc=MHocko@suse.com \
    --cc=akpm@linux-foundation.org \
    --cc=jirislaby@kernel.org \
    --cc=kvm@vger.kernel.org \
    --cc=liam.howlett@oracle.com \
    --cc=linux-mm@kvack.org \
    --cc=pbonzini@redhat.com \
    --cc=pedro.falcato@gmail.com \
    --cc=shy828301@gmail.com \
    --cc=vbabka@suse.cz \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).