[Qemu-devel] qemu core file size

[Qemu-devel] qemu core file size

[Wanpeng Li]

Hi all,

qemu core dump, max_core="unlimited", dump_guest_core=0, kill -11 pid
to generate a qemu core file, the Rss of qemu itself is ~40MB, the
core file is almost ~40MB in centos 6.x, but ~400MB in cents 7.x, any
idea?

Regards,
Wanpeng Li

Re: [Qemu-devel] qemu core file size

[Paolo Bonzini]

On 06/11/2017 10:11, Wanpeng Li wrote:
> Hi all,
> 
> qemu core dump, max_core="unlimited", dump_guest_core=0, kill -11 pid
> to generate a qemu core file, the Rss of qemu itself is ~40MB, the
> core file is almost ~40MB in centos 6.x, but ~400MB in cents 7.x, any
> idea?

My suspicion is the memory API, which isn't used in CentOS 6.x.  QEMU
2.11 should fix that.

Paolo

Re: [Qemu-devel] qemu core file size

[Wanpeng Li]

2017-11-06 17:41 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
> On 06/11/2017 10:11, Wanpeng Li wrote:
>> Hi all,
>>
>> qemu core dump, max_core="unlimited", dump_guest_core=0, kill -11 pid
>> to generate a qemu core file, the Rss of qemu itself is ~40MB, the
>> core file is almost ~40MB in centos 6.x, but ~400MB in cents 7.x, any
>> idea?
>
> My suspicion is the memory API, which isn't used in CentOS 6.x.  QEMU
> 2.11 should fix that.

Could you point out the patchset for the fix?

Regards,
Wanpeng Li

Re: [Qemu-devel] qemu core file size

[Paolo Bonzini]

On 06/11/2017 11:01, Wanpeng Li wrote:
> 2017-11-06 17:41 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
>> On 06/11/2017 10:11, Wanpeng Li wrote:
>>> Hi all,
>>>
>>> qemu core dump, max_core="unlimited", dump_guest_core=0, kill -11 pid
>>> to generate a qemu core file, the Rss of qemu itself is ~40MB, the
>>> core file is almost ~40MB in centos 6.x, but ~400MB in cents 7.x, any
>>> idea?
>>
>> My suspicion is the memory API, which isn't used in CentOS 6.x.  QEMU
>> 2.11 should fix that.
> 
> Could you point out the patchset for the fix?

Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
092aa2fc65b7a35121616aad8f39d47b8f921618.

Thanks,

Paolo

> 
> Regards,
> Wanpeng Li
> 

Re: [Qemu-devel] qemu core file size

[Fam Zheng]

On Mon, 11/06 11:26, Paolo Bonzini wrote:
> On 06/11/2017 11:01, Wanpeng Li wrote:
> > 2017-11-06 17:41 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
> >> On 06/11/2017 10:11, Wanpeng Li wrote:
> >>> Hi all,
> >>>
> >>> qemu core dump, max_core="unlimited", dump_guest_core=0, kill -11 pid
> >>> to generate a qemu core file, the Rss of qemu itself is ~40MB, the
> >>> core file is almost ~40MB in centos 6.x, but ~400MB in cents 7.x, any
> >>> idea?
> >>
> >> My suspicion is the memory API, which isn't used in CentOS 6.x.  QEMU
> >> 2.11 should fix that.
> > 
> > Could you point out the patchset for the fix?
> 
> Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
> 092aa2fc65b7a35121616aad8f39d47b8f921618.

Not sure how these relate to the core size, but I've tested upstream
(ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c) and with dump-guest-core=off the core
file is 363M, still significantly larger than rss (~73M).

What is bloating the core file?

Fam

Re: [Qemu-devel] qemu core file size

[Paolo Bonzini]

On 06/11/2017 12:59, Fam Zheng wrote:
>>> Could you point out the patchset for the fix?
>> Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
>> 092aa2fc65b7a35121616aad8f39d47b8f921618.
> Not sure how these relate to the core size, but I've tested upstream
> (ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c) and with dump-guest-core=off the core
> file is 363M, still significantly larger than rss (~73M).
> 
> What is bloating the core file?

My guess would have been fragmented heap.  The core file, unlike the
RSS, includes all the mmaped memory (e.g. from shared libraries) that
has never been used.

For example, all the Ceph/Gluster/PulseAudio/SPICE/whatever libraries
are included in the core file but likely are not in the RSS.

Paolo

Re: [Qemu-devel] qemu core file size

[Wanpeng Li]

2017-11-06 20:02 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
> On 06/11/2017 12:59, Fam Zheng wrote:
>>>> Could you point out the patchset for the fix?
>>> Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
>>> 092aa2fc65b7a35121616aad8f39d47b8f921618.
>> Not sure how these relate to the core size, but I've tested upstream
>> (ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c) and with dump-guest-core=off the core
>> file is 363M, still significantly larger than rss (~73M).
>>
>> What is bloating the core file?
>
> My guess would have been fragmented heap.  The core file, unlike the
> RSS, includes all the mmaped memory (e.g. from shared libraries) that
> has never been used.
>
> For example, all the Ceph/Gluster/PulseAudio/SPICE/whatever libraries
> are included in the core file but likely are not in the RSS.

Do you mean not use Memory API will avoid the fragmented heap?

Regards,
Wanpeng Li

Re: [Qemu-devel] qemu core file size

[Paolo Bonzini]

On 06/11/2017 13:18, Wanpeng Li wrote:
> 2017-11-06 20:02 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
>> On 06/11/2017 12:59, Fam Zheng wrote:
>>>>> Could you point out the patchset for the fix?
>>>> Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
>>>> 092aa2fc65b7a35121616aad8f39d47b8f921618.
>>> Not sure how these relate to the core size, but I've tested upstream
>>> (ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c) and with dump-guest-core=off the core
>>> file is 363M, still significantly larger than rss (~73M).
>>>
>>> What is bloating the core file?
>>
>> My guess would have been fragmented heap.  The core file, unlike the
>> RSS, includes all the mmaped memory (e.g. from shared libraries) that
>> has never been used.
>>
>> For example, all the Ceph/Gluster/PulseAudio/SPICE/whatever libraries
>> are included in the core file but likely are not in the RSS.
> 
> Do you mean not use Memory API will avoid the fragmented heap?

The high memory usage from the memory API causes excessive
fragmentation.  Alexey's work should help reducing memory usage and thus
the fragmentation.

Paolo

Re: [Qemu-devel] qemu core file size

[Wanpeng Li]

2017-11-06 22:08 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
> On 06/11/2017 13:18, Wanpeng Li wrote:
>> 2017-11-06 20:02 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
>>> On 06/11/2017 12:59, Fam Zheng wrote:
>>>>>> Could you point out the patchset for the fix?
>>>>> Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
>>>>> 092aa2fc65b7a35121616aad8f39d47b8f921618.
>>>> Not sure how these relate to the core size, but I've tested upstream
>>>> (ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c) and with dump-guest-core=off the core
>>>> file is 363M, still significantly larger than rss (~73M).
>>>>
>>>> What is bloating the core file?
>>>
>>> My guess would have been fragmented heap.  The core file, unlike the
>>> RSS, includes all the mmaped memory (e.g. from shared libraries) that
>>> has never been used.
>>>
>>> For example, all the Ceph/Gluster/PulseAudio/SPICE/whatever libraries
>>> are included in the core file but likely are not in the RSS.
>>
>> Do you mean not use Memory API will avoid the fragmented heap?
>
> The high memory usage from the memory API causes excessive
> fragmentation.  Alexey's work should help reducing memory usage and thus
> the fragmentation.

But as Fam mentioned, the latest qemu still has issue.

Regards,
Wanpeng Li

Re: [Qemu-devel] qemu core file size

[Alexey Kardashevskiy]

On 07/11/17 01:08, Paolo Bonzini wrote:
> On 06/11/2017 13:18, Wanpeng Li wrote:
>> 2017-11-06 20:02 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
>>> On 06/11/2017 12:59, Fam Zheng wrote:
>>>>>> Could you point out the patchset for the fix?
>>>>> Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
>>>>> 092aa2fc65b7a35121616aad8f39d47b8f921618.
>>>> Not sure how these relate to the core size, but I've tested upstream
>>>> (ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c) and with dump-guest-core=off the core
>>>> file is 363M, still significantly larger than rss (~73M).
>>>>
>>>> What is bloating the core file?
>>>
>>> My guess would have been fragmented heap.  The core file, unlike the
>>> RSS, includes all the mmaped memory (e.g. from shared libraries) that
>>> has never been used.
>>>
>>> For example, all the Ceph/Gluster/PulseAudio/SPICE/whatever libraries
>>> are included in the core file but likely are not in the RSS.
>>
>> Do you mean not use Memory API will avoid the fragmented heap?
> 
> The high memory usage from the memory API causes excessive
> fragmentation.  Alexey's work should help reducing memory usage and thus
> the fragmentation.

Since centos6 does not have this issue and centos7 does, I'd suggest
MALLOC_ARENA_MAX&co
https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html

glibc changed some defaults between rhel/centos 6 and 7 if I recall
correctly, and these arenas now create big anon mappings per thread, these
are not really touched (so they do not use resident memory) but may appear
in these dumps, dunno.

btw do you configure the machine to make "kill -11" produce qemu core dumps?


-- 
Alexey

Re: [Qemu-devel] qemu core file size

[Wanpeng Li]

Hi Alexey,
2017-11-07 13:46 GMT+08:00 Alexey Kardashevskiy <aik@ozlabs.ru>:
> On 07/11/17 01:08, Paolo Bonzini wrote:
>> On 06/11/2017 13:18, Wanpeng Li wrote:
>>> 2017-11-06 20:02 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
>>>> On 06/11/2017 12:59, Fam Zheng wrote:
>>>>>>> Could you point out the patchset for the fix?
>>>>>> Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
>>>>>> 092aa2fc65b7a35121616aad8f39d47b8f921618.
>>>>> Not sure how these relate to the core size, but I've tested upstream
>>>>> (ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c) and with dump-guest-core=off the core
>>>>> file is 363M, still significantly larger than rss (~73M).
>>>>>
>>>>> What is bloating the core file?
>>>>
>>>> My guess would have been fragmented heap.  The core file, unlike the
>>>> RSS, includes all the mmaped memory (e.g. from shared libraries) that
>>>> has never been used.
>>>>
>>>> For example, all the Ceph/Gluster/PulseAudio/SPICE/whatever libraries
>>>> are included in the core file but likely are not in the RSS.
>>>
>>> Do you mean not use Memory API will avoid the fragmented heap?
>>
>> The high memory usage from the memory API causes excessive
>> fragmentation.  Alexey's work should help reducing memory usage and thus
>> the fragmentation.
>
> Since centos6 does not have this issue and centos7 does, I'd suggest
> MALLOC_ARENA_MAX&co
> https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html

Thanks for the inputs. What's the recommend glibc.malloc.arena_max
value from you? :)

>
> glibc changed some defaults between rhel/centos 6 and 7 if I recall
> correctly, and these arenas now create big anon mappings per thread, these
> are not really touched (so they do not use resident memory) but may appear
> in these dumps, dunno.
>
> btw do you configure the machine to make "kill -11" produce qemu core dumps?

Yeah, some tuning in /etc/libvirt/qemu.conf, max_core="unlimited",
dump_guest_core=0

Regards,
Wanpeng Li

Re: [Qemu-devel] qemu core file size

[Alexey Kardashevskiy]

On 07/11/17 17:02, Wanpeng Li wrote:
> Hi Alexey,
> 2017-11-07 13:46 GMT+08:00 Alexey Kardashevskiy <aik@ozlabs.ru>:
>> On 07/11/17 01:08, Paolo Bonzini wrote:
>>> On 06/11/2017 13:18, Wanpeng Li wrote:
>>>> 2017-11-06 20:02 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
>>>>> On 06/11/2017 12:59, Fam Zheng wrote:
>>>>>>>> Could you point out the patchset for the fix?
>>>>>>> Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
>>>>>>> 092aa2fc65b7a35121616aad8f39d47b8f921618.
>>>>>> Not sure how these relate to the core size, but I've tested upstream
>>>>>> (ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c) and with dump-guest-core=off the core
>>>>>> file is 363M, still significantly larger than rss (~73M).
>>>>>>
>>>>>> What is bloating the core file?
>>>>>
>>>>> My guess would have been fragmented heap.  The core file, unlike the
>>>>> RSS, includes all the mmaped memory (e.g. from shared libraries) that
>>>>> has never been used.
>>>>>
>>>>> For example, all the Ceph/Gluster/PulseAudio/SPICE/whatever libraries
>>>>> are included in the core file but likely are not in the RSS.
>>>>
>>>> Do you mean not use Memory API will avoid the fragmented heap?
>>>
>>> The high memory usage from the memory API causes excessive
>>> fragmentation.  Alexey's work should help reducing memory usage and thus
>>> the fragmentation.
>>
>> Since centos6 does not have this issue and centos7 does, I'd suggest
>> MALLOC_ARENA_MAX&co
>> https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html
> 
> Thanks for the inputs. What's the recommend glibc.malloc.arena_max
> value from you? :)

I have no idea, you have to try different values as you are the one who is
unhappy about the sizes :) I do not normally mess with this as things
should just work with the defaults whatever they are.


> 
>>
>> glibc changed some defaults between rhel/centos 6 and 7 if I recall
>> correctly, and these arenas now create big anon mappings per thread, these
>> are not really touched (so they do not use resident memory) but may appear
>> in these dumps, dunno.
>>
>> btw do you configure the machine to make "kill -11" produce qemu core dumps?
> 
> Yeah, some tuning in /etc/libvirt/qemu.conf, max_core="unlimited",
> dump_guest_core=0

dump_guest_core=0 should mean "no dumps", no?
Do you know how to enable this without libvirt?


-- 
Alexey

Re: [Qemu-devel] qemu core file size

[Wanpeng Li]

2017-11-07 14:12 GMT+08:00 Alexey Kardashevskiy <aik@ozlabs.ru>:
> On 07/11/17 17:02, Wanpeng Li wrote:
>> Hi Alexey,
>> 2017-11-07 13:46 GMT+08:00 Alexey Kardashevskiy <aik@ozlabs.ru>:
>>> On 07/11/17 01:08, Paolo Bonzini wrote:
>>>> On 06/11/2017 13:18, Wanpeng Li wrote:
>>>>> 2017-11-06 20:02 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
>>>>>> On 06/11/2017 12:59, Fam Zheng wrote:
>>>>>>>>> Could you point out the patchset for the fix?
>>>>>>>> Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
>>>>>>>> 092aa2fc65b7a35121616aad8f39d47b8f921618.
>>>>>>> Not sure how these relate to the core size, but I've tested upstream
>>>>>>> (ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c) and with dump-guest-core=off the core
>>>>>>> file is 363M, still significantly larger than rss (~73M).
>>>>>>>
>>>>>>> What is bloating the core file?
>>>>>>
>>>>>> My guess would have been fragmented heap.  The core file, unlike the
>>>>>> RSS, includes all the mmaped memory (e.g. from shared libraries) that
>>>>>> has never been used.
>>>>>>
>>>>>> For example, all the Ceph/Gluster/PulseAudio/SPICE/whatever libraries
>>>>>> are included in the core file but likely are not in the RSS.
>>>>>
>>>>> Do you mean not use Memory API will avoid the fragmented heap?
>>>>
>>>> The high memory usage from the memory API causes excessive
>>>> fragmentation.  Alexey's work should help reducing memory usage and thus
>>>> the fragmentation.
>>>
>>> Since centos6 does not have this issue and centos7 does, I'd suggest
>>> MALLOC_ARENA_MAX&co
>>> https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html
>>
>> Thanks for the inputs. What's the recommend glibc.malloc.arena_max
>> value from you? :)
>
> I have no idea, you have to try different values as you are the one who is
> unhappy about the sizes :) I do not normally mess with this as things
> should just work with the defaults whatever they are.
>
>
>>
>>>
>>> glibc changed some defaults between rhel/centos 6 and 7 if I recall
>>> correctly, and these arenas now create big anon mappings per thread, these
>>> are not really touched (so they do not use resident memory) but may appear
>>> in these dumps, dunno.
>>>
>>> btw do you configure the machine to make "kill -11" produce qemu core dumps?
>>
>> Yeah, some tuning in /etc/libvirt/qemu.conf, max_core="unlimited",
>> dump_guest_core=0
>
> dump_guest_core=0 should mean "no dumps", no?
> Do you know how to enable this without libvirt?

Please add dump_guest_core=off in your qemu command-line,
"dump_guest_core" mean not dump guest memory and just dump qemu
itself.

Regards,
Wanpeng Li

Re: [Qemu-devel] qemu core file size

[Alexey Kardashevskiy]

On 07/11/17 17:16, Wanpeng Li wrote:
> 2017-11-07 14:12 GMT+08:00 Alexey Kardashevskiy <aik@ozlabs.ru>:
>> On 07/11/17 17:02, Wanpeng Li wrote:
>>> Hi Alexey,
>>> 2017-11-07 13:46 GMT+08:00 Alexey Kardashevskiy <aik@ozlabs.ru>:
>>>> On 07/11/17 01:08, Paolo Bonzini wrote:
>>>>> On 06/11/2017 13:18, Wanpeng Li wrote:
>>>>>> 2017-11-06 20:02 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
>>>>>>> On 06/11/2017 12:59, Fam Zheng wrote:
>>>>>>>>>> Could you point out the patchset for the fix?
>>>>>>>>> Between 447b0d0b9ee8a0ac216c3186e0f3c427a1001f0c and
>>>>>>>>> 092aa2fc65b7a35121616aad8f39d47b8f921618.
>>>>>>>> Not sure how these relate to the core size, but I've tested upstream
>>>>>>>> (ec7a8bf0b8f7dc7288fe8745464ee8217528cc6c) and with dump-guest-core=off the core
>>>>>>>> file is 363M, still significantly larger than rss (~73M).
>>>>>>>>
>>>>>>>> What is bloating the core file?
>>>>>>>
>>>>>>> My guess would have been fragmented heap.  The core file, unlike the
>>>>>>> RSS, includes all the mmaped memory (e.g. from shared libraries) that
>>>>>>> has never been used.
>>>>>>>
>>>>>>> For example, all the Ceph/Gluster/PulseAudio/SPICE/whatever libraries
>>>>>>> are included in the core file but likely are not in the RSS.
>>>>>>
>>>>>> Do you mean not use Memory API will avoid the fragmented heap?
>>>>>
>>>>> The high memory usage from the memory API causes excessive
>>>>> fragmentation.  Alexey's work should help reducing memory usage and thus
>>>>> the fragmentation.
>>>>
>>>> Since centos6 does not have this issue and centos7 does, I'd suggest
>>>> MALLOC_ARENA_MAX&co
>>>> https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html
>>>
>>> Thanks for the inputs. What's the recommend glibc.malloc.arena_max
>>> value from you? :)
>>
>> I have no idea, you have to try different values as you are the one who is
>> unhappy about the sizes :) I do not normally mess with this as things
>> should just work with the defaults whatever they are.


So I tried:
- QEMU with "-smp 8" and "(gdb) generate-core-file 1"  creates 820MB file;
- QEMU with "-smp 256" creates 11GB file,
- QEMU with "-smp 256" with MALLOC_ARENA_MAX=1 creates 2.5GB file,
- QEMU with "-smp 128" with MALLOC_ARENA_MAX=1 creates 1.3GB file;
- QEMU with "-smp 1" creates 290MB file;
- QEMU with "-smp 1" with MALLOC_ARENA_MAX=1 creates 222MB file.


It is "qemu-system-ppc64 -nodefaults -m 2G -enable-kvm -S" (i.e. stopped)
pretty much.


-- 
Alexey