From: David Hildenbrand <david@redhat.com>
To: "Daniel P. Berrangé" <berrange@redhat.com>
Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>,
Eduardo Habkost <ehabkost@redhat.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
qemu-devel@nongnu.org,
"Dr . David Alan Gilbert" <dgilbert@redhat.com>,
Paolo Bonzini <pbonzini@redhat.com>,
Igor Mammedov <imammedo@redhat.com>,
Marek Kedzierski <mkedzier@redhat.com>
Subject: Re: [PATCH v1 1/3] util/oslib-posix: Support MADV_POPULATE_WRITE for os_mem_prealloc()
Date: Tue, 20 Jul 2021 16:34:04 +0200 [thread overview]
Message-ID: <f8ab8d7c-200f-471b-5881-b0c42b3f3939@redhat.com> (raw)
In-Reply-To: <YPbY45FmTYNVPKCs@redhat.com>
>> memset_thread_failed = false;
>> threads_created_flag = false;
>> memset_num_threads = get_memset_num_threads(smp_cpus);
>> @@ -534,7 +558,7 @@ static bool touch_all_pages(char *area, size_t hpagesize, size_t numpages,
>> memset_thread[i].numpages = numpages_per_thread + (i < leftover);
>> memset_thread[i].hpagesize = hpagesize;
>> qemu_thread_create(&memset_thread[i].pgthread, "touch_pages",
>> - do_touch_pages, &memset_thread[i],
>> + touch_fn, &memset_thread[i],
>> QEMU_THREAD_JOINABLE);
>> addr += memset_thread[i].numpages * hpagesize;
>> }
>
> Do you have an indication of what the speed differential is for the
> old read/write dance vs the kernel madvise. We needed to use threads
> previously because the read/write dance is pretty terribly slow.
The kernel patch has some performance numbers:
https://lkml.kernel.org/r/20210712083917.16361-1-david@redhat.com
For example (compressed),
**************************************************
4096 MiB MAP_PRIVATE:
**************************************************
Anon 4 KiB : Read/Write : 1054.041 ms
Anon 4 KiB : POPULATE_WRITE : 572.582 ms
Memfd 4 KiB : Read/Write : 1106.561 ms
Memfd 4 KiB : POPULATE_WRITE : 805.881 ms
Memfd 2 MiB : Read/Write : 357.606 ms
Memfd 2 MiB : POPULATE_WRITE : 356.937 ms
tmpfs : Read/Write : 1105.954 ms
tmpfs : POPULATE_WRITE : 822.826 ms
file : Read/Write : 1107.439 ms
file : POPULATE_WRITE : 857.622 ms
hugetlbfs : Read/Write : 356.127 ms
hugetlbfs : POPULATE_WRITE : 355.138 ms
4096 MiB MAP_SHARED:
**************************************************
Anon 4 KiB : Read/Write : 1060.350 m
Anon 4 KiB : POPULATE_WRITE : 782.885 ms
Anon 2 MiB : Read/Write : 357.992 ms
Anon 2 MiB : POPULATE_WRITE : 357.808 ms
Memfd 4 KiB : Read/Write : 1100.391 ms
Memfd 4 KiB : POPULATE_WRITE : 804.394 ms
Memfd 2 MiB : Read/Write : 358.250 ms
Memfd 2 MiB : POPULATE_WRITE : 357.334 ms
tmpfs : Read/Write : 1107.567 ms
tmpfs : POPULATE_WRITE : 810.094 ms
file : Read/Write : 1289.509 ms
file : POPULATE_WRITE : 1106.816 ms
hugetlbfs : Read/Write : 357.120 ms
hugetlbfs : POPULATE_WRITE : 356.693 ms
For huge pages, it barely makes a difference with smallish VMs. In the
other cases, it speeds it up, but not as extreme as that it would allow
for dropping multi-threading.
The original MADV_POPULATE from 2016
https://lore.kernel.org/patchwork/patch/389581/ mentiones that it
especially helps speed up multi-threaded pre-faulting, due to reduced
mmap_lock contention. I did not do any multi-threading benchmarks, though.
[...]
>
> Initialized with random garbage from the stack
>
>> +
>> + /*
>> + * Sense on every invocation, as MADV_POPULATE_WRITE cannot be used for
>> + * some special mappings, such as mapping /dev/mem.
>> + */
>> + if (madv_populate_write_possible(area, hpagesize)) {
>> + use_madv_populate_write = true;
>> + }
>
> but this implicitly assumes it was initialized to false.
Indeed, thanks for catching that!
--
Thanks,
David / dhildenb
next prev parent reply other threads:[~2021-07-20 14:35 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-14 11:23 [PATCH v1 0/3] util/oslib-posix: Support MADV_POPULATE_WRITE for os_mem_prealloc() David Hildenbrand
2021-07-14 11:23 ` [PATCH v1 1/3] " David Hildenbrand
2021-07-20 14:08 ` Daniel P. Berrangé
2021-07-20 14:34 ` David Hildenbrand [this message]
2021-07-14 11:23 ` [PATCH v1 2/3] util/oslib-posix: Introduce and use MemsetContext for touch_all_pages() David Hildenbrand
2021-07-20 14:27 ` Daniel P. Berrangé
2021-07-14 11:23 ` [PATCH v1 3/3] util/oslib-posix: Support concurrent os_mem_prealloc() invocation David Hildenbrand
2021-07-20 14:22 ` Daniel P. Berrangé
2021-07-20 14:27 ` David Hildenbrand
2021-07-20 14:31 ` Daniel P. Berrangé
2021-07-20 14:35 ` David Hildenbrand
2021-07-20 13:55 ` [PATCH v1 0/3] util/oslib-posix: Support MADV_POPULATE_WRITE for os_mem_prealloc() Pankaj Gupta
2021-07-20 13:58 ` Pankaj Gupta
2021-07-20 14:45 ` Daniel P. Berrangé
2021-07-21 8:23 ` David Hildenbrand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f8ab8d7c-200f-471b-5881-b0c42b3f3939@redhat.com \
--to=david@redhat.com \
--cc=berrange@redhat.com \
--cc=dgilbert@redhat.com \
--cc=ehabkost@redhat.com \
--cc=imammedo@redhat.com \
--cc=mkedzier@redhat.com \
--cc=mst@redhat.com \
--cc=pankaj.gupta.linux@gmail.com \
--cc=pbonzini@redhat.com \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.