* Restart domU failure (memory allocation issue)
@ 2014-03-13 18:33 Oleksandr Tyshchenko
2014-03-13 19:48 ` Julien Grall
0 siblings, 1 reply; 11+ messages in thread
From: Oleksandr Tyshchenko @ 2014-03-13 18:33 UTC (permalink / raw)
To: xen-devel; +Cc: Andrii Anisov
Hello, all.
I am working with DRA7 (OMAP5) platform and I am using XEN 4.4 and
Kernel 3.8 for both domains.
As we still have 1:1 mapping for domU we need to allocate memory (1024
MB in our case) for it at one chunk.
The issue is that I can't create domain again after destroying it (I
am destroying domain by a "xl destroy" command) because we no longer
have contiguous memory region and as result allocation failed.
It seems, we have memory leak. I see that not all memory is freed
after domain destroy.
I see that some pages (which were allocated for kernel segment) don't
returned to the heap.
Some debug info:
1. before creating domain:
(XEN) Physical memory information:
(XEN) Xen heap: 196264kB free
(XEN) heap[20]: 1193908kB free
(XEN) Dom heap: 1193908kB free
(XEN) heap[node=0][zone=0] -> 49066 pages
(XEN) heap[node=0][zone=1] -> 0 pages
(XEN) heap[node=0][zone=2] -> 0 pages
(XEN) heap[node=0][zone=3] -> 0 pages
(XEN) heap[node=0][zone=4] -> 0 pages
(XEN) heap[node=0][zone=5] -> 0 pages
(XEN) heap[node=0][zone=6] -> 0 pages
(XEN) heap[node=0][zone=7] -> 0 pages
(XEN) heap[node=0][zone=8] -> 0 pages
(XEN) heap[node=0][zone=9] -> 0 pages
(XEN) heap[node=0][zone=10] -> 0 pages
(XEN) heap[node=0][zone=11] -> 0 pages
(XEN) heap[node=0][zone=12] -> 0 pages
(XEN) heap[node=0][zone=13] -> 0 pages
(XEN) heap[node=0][zone=14] -> 0 pages
(XEN) heap[node=0][zone=15] -> 0 pages
(XEN) heap[node=0][zone=16] -> 0 pages
(XEN) heap[node=0][zone=17] -> 0 pages
(XEN) heap[node=0][zone=18] -> 0 pages
(XEN) heap[node=0][zone=19] -> 0 pages
(XEN) heap[node=0][zone=20] -> 298477 pages
(XEN) heap[node=0][zone=21] -> 0 pages
(XEN) heap[node=0][zone=22] -> 0 pages
(XEN) heap[node=0][zone=23] -> 0 pages
(XEN) heap[node=0][zone=24] -> 0 pages
(XEN) heap[node=0][zone=25] -> 0 pages
(XEN) heap[node=0][zone=26] -> 0 pages
(XEN) heap[node=0][zone=27] -> 0 pages
(XEN) heap[node=0][zone=28] -> 0 pages
2. create domain:
/ # xl -vvv create /xen/images/DomULinux_nodisk.cfg
Parsing config from /xen/images/DomULinux_nodisk.cfg
libxl: debug: libxl_create.c:1340:do_domain_create: ao 0x33108:
create: how=(nil) callback=(nil) poller=0x32840
libxl: verbose:
libxl_create.c:134:libxl__domain_build_info_setdefault: qemu-xen is
unavailable, use qemu-xen-traditional instead: No such file or
directory
libxl: debug: libxl_create.c:797:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no
bootloader configured, using user supplied kernel
libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch
w=0x33320: deregister unregistered
libxl: debug: libxl_numa.c:475:libxl__get_numa_candidate: New best
NUMA placement candidate found: nr_nodes=1, nr_cpus=2, nr_vcpus=4,
free_memkb=1357
libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA placement
candidate with 1 nodes, 2 cpus and 1357 KB free selected
domainbuilder: detail: xc_dom_allocate: cmdline="", features="(null)"
libxl: debug: libxl_dom.c:361:libxl__build_pv: pv kernel mapped 0 path
/xen/images/DomULinux.img
domainbuilder: detail: xc_dom_kernel_file: filename="/xen/images/DomULinux.img"
domainbuilder: detail: xc_dom_malloc_filemap : 5616 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.4, caps xen-3.0-armv7l
domainbuilder: detail: xc_dom_rambase_init: RAM starts at 80000
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux uImage (ARM) loader ...
domainbuilder: detail: xc_dom_probe_uimage_kernel: kernel is not a uImage
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM64)
loader ...
domainbuilder: detail: xc_dom_probe_zimage64_kernel: kernel is not an
arm64 Image
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM32)
loader ...
domainbuilder: detail: xc_dom_probe_zimage32_kernel: found an appended DTB
domainbuilder: detail: loader probe OK
domainbuilder: detail: xc_dom_parse_zimage32_kernel: called
domainbuilder: detail: xc_dom_parse_zimage32_kernel: xen-3.0-armv7l:
0x80008000 -> 0x805842df
libxl: debug: libxl_arm.c:511:libxl__arch_domain_configure: Skip
constructing DTB (DTB is already prepared).
// allocate contiguous 1 GB memory for domU: pages from 0x80000 to 0xC0000
domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000
pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x40000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: set_mode: guest xen-3.0-armv7l, address size 32
domainbuilder: detail: xc_dom_malloc : 2048 kB
domainbuilder: detail: Allocating 2^18 pages at one chunk
total_pages:0x00040000 i:0
// our print
(XEN) >>>>> alloc_heap_pages [735] page: 0x80000 order: 18
// map memory for kernel segment: pages from 0x80008 to 0x80585
// As I understand these pages from domU memory are mapped to dom0 address space
domainbuilder: detail: xc_dom_alloc_segment: kernel :
0x80008000 -> 0x80585000 (pfn 0x80008 + 0x57d pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x80008+0x57d at 0xb606b000
domainbuilder: detail: xc_dom_load_zimage_kernel: called
domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg
0x80008000-0x80585000
domainbuilder: detail: xc_dom_load_zimage_kernel: copy 5751519 bytes
from blob 0xb67e9000 to dst 0xb606b000
domainbuilder: detail: alloc_magic_pages: called
domainbuilder: detail: count_pgtables_arm: called
domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x80585000
domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-armv7l <= matches
domainbuilder: detail: setup_pgtables_arm: called
domainbuilder: detail: clear_page: pfn 0xc0000, mfn 0xc0000
domainbuilder: detail: clear_page: pfn 0xc0001, mfn 0xc0001
domainbuilder: detail: start_info_arm: called
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail: allocated
domainbuilder: detail: malloc : 2082 kB
domainbuilder: detail: anon mmap : 0 bytes
domainbuilder: detail: mapped
domainbuilder: detail: file mmap : 5616 kB
domainbuilder: detail: domU mmap : 5620 kB
domainbuilder: detail: vcpu_arm32: called
domainbuilder: detail: Initial state CPSR 0x1d3 PC 0x80008000
// unmap memory for kernel segment
// before call munmap()
domainbuilder: detail: !!!!! xc_dom_unmap_one [604] ptr: 0xb606b000 count: 0x57d
// before call xc_domain_cacheflush()
domainbuilder: detail: !!!!! xc_dom_unmap_one [611] page: 0x80008 count: 0x57d
domainbuilder: detail: !!!!! xc_dom_unmap_all [619]
domainbuilder: detail: launch_vm: called, ctxt=0xb6d6a004
domainbuilder: detail: xc_dom_release: called
...
...
...
libxl: debug: libxl_event.c:1761:libxl__ao_progress_report: ao
0x33108: progress report: ignored
libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x33108: complete, rc=0
libxl: debug: libxl_create.c:1354:do_domain_create: ao 0x33108:
inprogress: poller=0x32840, flags=ic
libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x33108: destroy
xc: debug: hypercall buffer: total allocations:237 total releases:237
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:230 misses:4 toobig:3
3. after creating domain:
(XEN) Physical memory information:
(XEN) Xen heap: 196140kB free
(XEN) heap[20]: 143208kB free
(XEN) Dom heap: 143208kB free
(XEN) heap[node=0][zone=0] -> 49035 pages
(XEN) heap[node=0][zone=1] -> 0 pages
(XEN) heap[node=0][zone=2] -> 0 pages
(XEN) heap[node=0][zone=3] -> 0 pages
(XEN) heap[node=0][zone=4] -> 0 pages
(XEN) heap[node=0][zone=5] -> 0 pages
(XEN) heap[node=0][zone=6] -> 0 pages
(XEN) heap[node=0][zone=7] -> 0 pages
(XEN) heap[node=0][zone=8] -> 0 pages
(XEN) heap[node=0][zone=9] -> 0 pages
(XEN) heap[node=0][zone=10] -> 0 pages
(XEN) heap[node=0][zone=11] -> 0 pages
(XEN) heap[node=0][zone=12] -> 0 pages
(XEN) heap[node=0][zone=13] -> 0 pages
(XEN) heap[node=0][zone=14] -> 0 pages
(XEN) heap[node=0][zone=15] -> 0 pages
(XEN) heap[node=0][zone=16] -> 0 pages
(XEN) heap[node=0][zone=17] -> 0 pages
(XEN) heap[node=0][zone=18] -> 0 pages
(XEN) heap[node=0][zone=19] -> 0 pages
(XEN) heap[node=0][zone=20] -> 35802 pages
(XEN) heap[node=0][zone=21] -> 0 pages
(XEN) heap[node=0][zone=22] -> 0 pages
(XEN) heap[node=0][zone=23] -> 0 pages
(XEN) heap[node=0][zone=24] -> 0 pages
(XEN) heap[node=0][zone=25] -> 0 pages
(XEN) heap[node=0][zone=26] -> 0 pages
(XEN) heap[node=0][zone=27] -> 0 pages
(XEN) heap[node=0][zone=28] -> 0 pages
/ # xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 128 2 r----- 3.2
win 1 1024 2 -b---- 0.8
4. destroy domain:
libxl: debug: libxl.c:1250:libxl_domain_destroy: ao 0x32850: create:
how=(nil) callback=(nil) poller=0x326d0
// our prints
(XEN) <<<<< free_heap_pages [831] page: 0x80000 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80001 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80002 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80003 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80004 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80005 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80006 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80007 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80585 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80586 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80587 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80588 order: 0
(XEN) <<<<< free_heap_pages [831] page: 0x80589 order: 0
...
...
...
We can see that pages which were using for kernel segment (from
0x80008 to 0x80585) are absent here.
5. after destroying domain:
(XEN) Physical memory information:
(XEN) Xen heap: 196140kB free
(XEN) heap[20]: 1186164kB free
(XEN) Dom heap: 1186164kB free
(XEN) heap[node=0][zone=0] -> 49035 pages
(XEN) heap[node=0][zone=1] -> 0 pages
(XEN) heap[node=0][zone=2] -> 0 pages
(XEN) heap[node=0][zone=3] -> 0 pages
(XEN) heap[node=0][zone=4] -> 0 pages
(XEN) heap[node=0][zone=5] -> 0 pages
(XEN) heap[node=0][zone=6] -> 0 pages
(XEN) heap[node=0][zone=7] -> 0 pages
(XEN) heap[node=0][zone=8] -> 0 pages
(XEN) heap[node=0][zone=9] -> 0 pages
(XEN) heap[node=0][zone=10] -> 0 pages
(XEN) heap[node=0][zone=11] -> 0 pages
(XEN) heap[node=0][zone=12] -> 0 pages
(XEN) heap[node=0][zone=13] -> 0 pages
(XEN) heap[node=0][zone=14] -> 0 pages
(XEN) heap[node=0][zone=15] -> 0 pages
(XEN) heap[node=0][zone=16] -> 0 pages
(XEN) heap[node=0][zone=17] -> 0 pages
(XEN) heap[node=0][zone=18] -> 0 pages
(XEN) heap[node=0][zone=19] -> 0 pages
(XEN) heap[node=0][zone=20] -> 296541 pages
(XEN) heap[node=0][zone=21] -> 0 pages
(XEN) heap[node=0][zone=22] -> 0 pages
(XEN) heap[node=0][zone=23] -> 0 pages
(XEN) heap[node=0][zone=24] -> 0 pages
(XEN) heap[node=0][zone=25] -> 0 pages
(XEN) heap[node=0][zone=26] -> 0 pages
(XEN) heap[node=0][zone=27] -> 0 pages
(XEN) heap[node=0][zone=28] -> 0 pages
/ # xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 128 2 r----- 4.1
(null) 1 5 2 --p--d 0.9
6. Create domain again:
/ # xl -vvv create /xen/images/DomULinux_nodisk.cfg
Parsing config from /xen/images/DomULinux_nodisk.cfg
libxl: debug: libxl_create.c:1340:do_domain_create: ao 0x33108:
create: how=(nil) callback=(nil) poller=0x32840
libxl: verbose:
libxl_create.c:134:libxl__domain_build_info_setdefault: qemu-xen is
unavailable, use qemu-xen-traditional instead: No such file or
directory
libxl: debug: libxl_create.c:797:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no
bootloader configured, using user supplied kernel
libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch
w=0x33320: deregister unregistered
libxl: debug: libxl_numa.c:475:libxl__get_numa_candidate: New best
NUMA placement candidate found: nr_nodes=1, nr_cpus=2, nr_vcpus=6,
free_memkb=1349
libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA placement
candidate with 1 nodes, 2 cpus and 1349 KB free selected
domainbuilder: detail: xc_dom_allocate: cmdline="", features="(null)"
libxl: debug: libxl_dom.c:361:libxl__build_pv: pv kernel mapped 0 path
/xen/images/DomULinux.img
domainbuilder: detail: xc_dom_kernel_file: filename="/xen/images/DomULinux.img"
domainbuilder: detail: xc_dom_malloc_filemap : 5616 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.4, caps xen-3.0-armv7l
domainbuilder: detail: xc_dom_rambase_init: RAM starts at 80000
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux uImage (ARM) loader ...
domainbuilder: detail: xc_dom_probe_uimage_kernel: kernel is not a uImage
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM64)
loader ...
domainbuilder: detail: xc_dom_probe_zimage64_kernel: kernel is not an
arm64 Image
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM32)
loader ...
domainbuilder: detail: xc_dom_probe_zimage32_kernel: found an appended DTB
domainbuilder: detail: loader probe OK
domainbuilder: detail: xc_dom_parse_zimage32_kernel: called
domainbuilder: detail: xc_dom_parse_zimage32_kernel: xen-3.0-armv7l:
0x80008000 -> 0x805842df
libxl: debug: libxl_arm.c:511:libxl__arch_domain_configure: Skip
constructing DTB (DTB is already prepared).
domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000
pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x40000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: set_mode: guest xen-3.0-armv7l, address size 32
domainbuilder: detail: xc_dom_malloc : 2048 kB
domainbuilder: detail: Allocating 2^18 pages at one chunk
total_pages:0x00040000 i:0
// our print: we no longer have contiguous memory region (
(XEN) >>>>> alloc_heap_pages [674] page: NULL order: 18
(XEN) memory.c:160:d0 Could not allocate order=18 extent: id=2
memflags=0 (0 of 1)
xc: detail: Failed allocation for dom 2: 1 extents of order 18
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment: kernel :
0x80008000 -> 0x80585000 (pfn 0x80008 + 0x57d pages)
(XEN) dom2 IPA 0x0000000080008000
(XEN) P2M @ 02bddb00 mfn:0xdeed8
(XEN) 1ST[0x2] = 0x0000000000000000
[ 51.851562] Failed to map pfn to mfn rc:0:-22 pfn:cc995 mfn:80008
(XEN) dom2 IPA 0x0000000080009000
(XEN) P2M @ 02bddb00 mfn:0xdeed8
(XEN) 1ST[0x2] = 0x0000000000000000
[ 51.867187] Failed to map pfn to mfn rc:0:-22 pfn:cc994 mfn:80009
(XEN) dom2 IPA 0x000000008000a000
(XEN) P2M @ 02bddb00 mfn:0xdeed8
(XEN) 1ST[0x2] = 0x0000000000000000
[ 51.882812] Failed to map pfn to mfn rc:0:-22 pfn:cc993 mfn:8000a
(XEN) dom2 IPA 0x000000008000b000
(XEN) P2M @ 02bddb00 mfn:0xdeed8
(XEN) 1ST[0x2] = 0x0000000000000000
...
...
...
xc: error: panic: xc_dom_boot.c:191: xc_dom_boot_domU_map: failed to
mmap domU pages 0x80008+0x57d [mmap, errno=14 (Bad address)]: Internal
error
libxl: error: libxl_dom.c:427:libxl__build_pv: xc_dom_build_image
failed: Bad address
domainbuilder: detail: xc_dom_release: called
libxl: error: libxl_create.c:1022:domcreate_rebuild_done: cannot
(re-)build domain: -3
libxl: debug: libxl_event.c:1591:libxl__ao_complete: ao 0x33108: complete, rc=-3
libxl: debug: libxl_create.c:1354:do_domain_create: ao 0x33108:
inprogress: poller=0x32840, flags=ic
libxl: debug: libxl_event.c:1563:libxl__ao__destroy: ao 0x33108: destroy
xc: debug: hypercall buffer: total allocations:92 total releases:92
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:85 misses:4 toobig:3
Please, could anyone explain me what is wrong? Can I perform any steps
manually to free this memory region?
Yes, we have to drop 1:1 mapping, but from my point of view this
behavior looks odd (what happened with these pages after domain
create/destroy?).
Thank you.
--
Oleksandr Tyshchenko | Embedded Developer
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Restart domU failure (memory allocation issue)
2014-03-13 18:33 Restart domU failure (memory allocation issue) Oleksandr Tyshchenko
@ 2014-03-13 19:48 ` Julien Grall
2014-03-14 11:10 ` Andrii Anisov
0 siblings, 1 reply; 11+ messages in thread
From: Julien Grall @ 2014-03-13 19:48 UTC (permalink / raw)
To: Oleksandr Tyshchenko, xen-devel
Cc: Tim Deegan, Ian Campbell, Andrii Anisov, Stefano Stabellini
(Adding ARM people in CC)
On 13/03/14 18:33, Oleksandr Tyshchenko wrote:
> Hello, all.
Hello Oleksandr,
> I am working with DRA7 (OMAP5) platform and I am using XEN 4.4 and
> Kernel 3.8 for both domains.
> As we still have 1:1 mapping for domU we need to allocate memory (1024
> MB in our case) for it at one chunk.
>
> The issue is that I can't create domain again after destroying it (I
> am destroying domain by a "xl destroy" command) because we no longer
> have contiguous memory region and as result allocation failed.
>
> It seems, we have memory leak. I see that not all memory is freed
> after domain destroy.
> I see that some pages (which were allocated for kernel segment) don't
> returned to the heap.
I gave a try on Xen 4.4 with normal guest (e.g without the 1:1 mapping)
and I don't find any memory leaks. How do you create the 1:1 mapping for
the guest?
> / # xl list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 128 2 r----- 4.1
> (null) 1 5 2 --p--d 0.9
It seems that your domain is not completely destroyed. It may be because
some pages still belongs to the domain.
Xen will free a page only when the reference count is 0, do you have
keep some mapping in dom0? (for instance in libxl).
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Restart domU failure (memory allocation issue)
2014-03-13 19:48 ` Julien Grall
@ 2014-03-14 11:10 ` Andrii Anisov
2014-03-14 11:14 ` Andrii Anisov
0 siblings, 1 reply; 11+ messages in thread
From: Andrii Anisov @ 2014-03-14 11:10 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr Tyshchenko, Stefano Stabellini, Tim Deegan,
Ian Campbell, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 2526 bytes --]
Julien,
I gave a try on Xen 4.4 with normal guest (e.g without the 1:1 mapping) and
> I don't find any memory leaks. How do you create the 1:1 mapping for the
> guest?
>
The way we get 1:1 mapping is pretty tricky.
We have a patch to allocate DomU space as a one multipage section with the
order enough for requested size. While we place all the XEN stuff after the
first gigabyte of RAM, we do have the only 1GB extent at address 0x80000000
for our Android.
We do work to get rid of this dirty hack. But still have a lot of things to
do with GPU and IPU.
>From the investigations done by Oleksandr it seems to me that some mappings
done for so called kernel segment by xc_dom_alloc_segment() are left in
dom0 space.
Andrii Anisov | Software Engineer
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664 M +380505738852 S andriyanisov
www.globallogic.com
<http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt
On Thu, Mar 13, 2014 at 9:48 PM, Julien Grall <julien.grall@linaro.org>wrote:
> (Adding ARM people in CC)
>
> On 13/03/14 18:33, Oleksandr Tyshchenko wrote:
>
>> Hello, all.
>>
>
> Hello Oleksandr,
>
>
> I am working with DRA7 (OMAP5) platform and I am using XEN 4.4 and
>> Kernel 3.8 for both domains.
>> As we still have 1:1 mapping for domU we need to allocate memory (1024
>> MB in our case) for it at one chunk.
>>
>> The issue is that I can't create domain again after destroying it (I
>> am destroying domain by a "xl destroy" command) because we no longer
>> have contiguous memory region and as result allocation failed.
>>
>> It seems, we have memory leak. I see that not all memory is freed
>> after domain destroy.
>> I see that some pages (which were allocated for kernel segment) don't
>> returned to the heap.
>>
>
> I gave a try on Xen 4.4 with normal guest (e.g without the 1:1 mapping)
> and I don't find any memory leaks. How do you create the 1:1 mapping for
> the guest?
>
>
> / # xl list
>> Name ID Mem VCPUs State
>> Time(s)
>> Domain-0 0 128 2 r-----
>> 4.1
>> (null) 1 5 2 --p--d
>> 0.9
>>
>
> It seems that your domain is not completely destroyed. It may be because
> some pages still belongs to the domain.
>
> Xen will free a page only when the reference count is 0, do you have keep
> some mapping in dom0? (for instance in libxl).
>
> Regards,
>
> --
> Julien Grall
>
[-- Attachment #1.2: Type: text/html, Size: 6202 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Restart domU failure (memory allocation issue)
2014-03-14 11:10 ` Andrii Anisov
@ 2014-03-14 11:14 ` Andrii Anisov
2014-03-14 12:15 ` Julien Grall
0 siblings, 1 reply; 11+ messages in thread
From: Andrii Anisov @ 2014-03-14 11:14 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr Tyshchenko, Stefano Stabellini, Tim Deegan,
Ian Campbell, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 548 bytes --]
>
> From the investigations done by Oleksandr it seems to me that some
> mappings done for so called kernel segment by xc_dom_alloc_segment() are
> left in dom0 space.
>
Actually I do not see how pages from domU space (i.e. that kernel segment)
are unmapped from dom0 after domain creation.
Andrii Anisov | Software Engineer
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664 M +380505738852 S andriyanisov
www.globallogic.com
<http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt
[-- Attachment #1.2: Type: text/html, Size: 2812 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Restart domU failure (memory allocation issue)
2014-03-14 11:14 ` Andrii Anisov
@ 2014-03-14 12:15 ` Julien Grall
2014-03-14 12:45 ` Andrii Anisov
0 siblings, 1 reply; 11+ messages in thread
From: Julien Grall @ 2014-03-14 12:15 UTC (permalink / raw)
To: Andrii Anisov
Cc: Oleksandr Tyshchenko, Stefano Stabellini, Tim Deegan,
Ian Campbell, xen-devel
Hello Andrii,
On 03/14/2014 11:14 AM, Andrii Anisov wrote:
> From the investigations done by Oleksandr it seems to me that some
> mappings done for so called kernel segment by xc_dom_alloc_segment()
> are left in dom0 space.
>
> Actually I do not see how pages from domU space (i.e. that kernel
> segment) are unmapped from dom0 after domain creation.
It seems that unmap will be done at the end of xc_dom_boot_image (see
xc_dom_unmap_all).
The latter function will go through the list of all mappings and close
it one by one.
You can add a print in xc_dom_unmap_one (tools/libxc/xc_dom_core.c) and
check if you have the same number of print with xc_dom_pfn_to_ptr_retcount.
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Restart domU failure (memory allocation issue)
2014-03-14 12:15 ` Julien Grall
@ 2014-03-14 12:45 ` Andrii Anisov
2014-03-14 14:00 ` Julien Grall
0 siblings, 1 reply; 11+ messages in thread
From: Andrii Anisov @ 2014-03-14 12:45 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr Tyshchenko, Stefano Stabellini, Tim Deegan,
Ian Campbell, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 784 bytes --]
>
> It seems that unmap will be done at the end of xc_dom_boot_image (see
> xc_dom_unmap_all).
> The latter function will go through the list of all mappings and close
> it one by one.
> You can add a print in xc_dom_unmap_one (tools/libxc/xc_dom_core.c) and
> check if you have the same number of print with xc_dom_pfn_to_ptr_retcount.
As I see xc_dom_unmap_all ends with munmap(), what actually unmaps pages
from process virtual memory space. But I have doubts if it is aware about
cross domain mapping and cares about it.
Andrii Anisov | Software Engineer
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664 M +380505738852 S andriyanisov
www.globallogic.com
<http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt
[-- Attachment #1.2: Type: text/html, Size: 3046 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Restart domU failure (memory allocation issue)
2014-03-14 12:45 ` Andrii Anisov
@ 2014-03-14 14:00 ` Julien Grall
2014-03-14 14:05 ` Andrii Anisov
2014-03-14 14:08 ` Oleksandr Tyshchenko
0 siblings, 2 replies; 11+ messages in thread
From: Julien Grall @ 2014-03-14 14:00 UTC (permalink / raw)
To: Andrii Anisov
Cc: Oleksandr Tyshchenko, Stefano Stabellini, Tim Deegan,
Ian Campbell, xen-devel
On 03/14/2014 12:45 PM, Andrii Anisov wrote:
> It seems that unmap will be done at the end of xc_dom_boot_image (see
> xc_dom_unmap_all).
> The latter function will go through the list of all mappings and close
> it one by one.
> You can add a print in xc_dom_unmap_one (tools/libxc/xc_dom_core.c) and
> check if you have the same number of print with
> xc_dom_pfn_to_ptr_retcount.
>
>
> As I see xc_dom_unmap_all ends with munmap(), what actually unmaps pages
> from process virtual memory space. But I have doubts if it is aware
> about cross domain mapping and cares about it.
This is done in kernel space. When you map the pages, privcmd will set
specific callbacks that will be use when munmap will be called (see
privcmd_close in drivers/xen/privcmd.c).
The privcmd_close function will call the hypercall to remove the foreign
page from the p2m.
--
Julien Grall
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Restart domU failure (memory allocation issue)
2014-03-14 14:00 ` Julien Grall
@ 2014-03-14 14:05 ` Andrii Anisov
2014-03-14 14:17 ` Julien Grall
2014-03-14 14:08 ` Oleksandr Tyshchenko
1 sibling, 1 reply; 11+ messages in thread
From: Andrii Anisov @ 2014-03-14 14:05 UTC (permalink / raw)
To: Julien Grall
Cc: Oleksandr Tyshchenko, Stefano Stabellini, Tim Deegan,
Ian Campbell, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 609 bytes --]
>
> This is done in kernel space. When you map the pages, privcmd will set
> specific callbacks that will be use when munmap will be called (see
> privcmd_close in drivers/xen/privcmd.c).
> The privcmd_close function will call the hypercall to remove the foreign
> page from the p2m.
We will check if it works on our site (but it looks it does not).
Andrii Anisov | Software Engineer
GlobalLogic
Kyiv, 03038, Protasov Business Park, M.Grinchenka, 2/1
P +38.044.492.9695x3664 M +380505738852 S andriyanisov
www.globallogic.com
<http://www.globallogic.com/>
http://www.globallogic.com/email_disclaimer.txt
[-- Attachment #1.2: Type: text/html, Size: 2852 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Restart domU failure (memory allocation issue)
2014-03-14 14:00 ` Julien Grall
2014-03-14 14:05 ` Andrii Anisov
@ 2014-03-14 14:08 ` Oleksandr Tyshchenko
1 sibling, 0 replies; 11+ messages in thread
From: Oleksandr Tyshchenko @ 2014-03-14 14:08 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, Tim Deegan, Ian Campbell, Andrii Anisov, xen-devel
Hello, Julien.
>
> This is done in kernel space. When you map the pages, privcmd will set
> specific callbacks that will be use when munmap will be called (see
> privcmd_close in drivers/xen/privcmd.c).
>
> The privcmd_close function will call the hypercall to remove the foreign
> page from the p2m.
>
> --
> Julien Grall
It is clear. I will check.
Thank you.
--
Oleksandr Tyshchenko | Embedded Developer
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Restart domU failure (memory allocation issue)
2014-03-14 14:05 ` Andrii Anisov
@ 2014-03-14 14:17 ` Julien Grall
2014-03-14 14:37 ` Oleksandr Tyshchenko
0 siblings, 1 reply; 11+ messages in thread
From: Julien Grall @ 2014-03-14 14:17 UTC (permalink / raw)
To: Andrii Anisov
Cc: Oleksandr Tyshchenko, Stefano Stabellini, Tim Deegan,
Ian Campbell, xen-devel
On 03/14/2014 02:05 PM, Andrii Anisov wrote:
> This is done in kernel space. When you map the pages, privcmd will set
> specific callbacks that will be use when munmap will be called (see
> privcmd_close in drivers/xen/privcmd.c).
> The privcmd_close function will call the hypercall to remove the foreign
> page from the p2m.
>
>
> We will check if it works on our site (but it looks it does not).
I think I've found why ... the privcmd_close is buggy on 3.8. You might
need to backport this commit:
commit 9eff37a8713939f218ab8bf0dc93f1d67af7b8b4
Author: Dan Carpenter <dan.carpenter@oracle.com>
Date: Mon Nov 5 09:42:17 2012 +0300
xen/privcmd: fix condition in privcmd_close()
The parenthesis are in the wrong place so the original code is
equivalent to:
if (!xen_feature(XENFEAT_writable_descriptor_tables)) { ...
Which obviously was not intended.
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Regards,
--
Julien Grall
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Restart domU failure (memory allocation issue)
2014-03-14 14:17 ` Julien Grall
@ 2014-03-14 14:37 ` Oleksandr Tyshchenko
0 siblings, 0 replies; 11+ messages in thread
From: Oleksandr Tyshchenko @ 2014-03-14 14:37 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, Tim Deegan, Ian Campbell, Andrii Anisov, xen-devel
> I think I've found why ... the privcmd_close is buggy on 3.8. You might
> need to backport this commit:
>
> commit 9eff37a8713939f218ab8bf0dc93f1d67af7b8b4
> Author: Dan Carpenter <dan.carpenter@oracle.com>
> Date: Mon Nov 5 09:42:17 2012 +0300
>
> xen/privcmd: fix condition in privcmd_close()
>
> The parenthesis are in the wrong place so the original code is
> equivalent to:
>
> if (!xen_feature(XENFEAT_writable_descriptor_tables)) { ...
>
> Which obviously was not intended.
>
> Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>
> Regards,
>
> --
> Julien Grall
It works! Thank you!
--
Oleksandr Tyshchenko | Embedded Developer
GlobalLogic
www.globallogic.com
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2014-03-14 14:37 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-03-13 18:33 Restart domU failure (memory allocation issue) Oleksandr Tyshchenko
2014-03-13 19:48 ` Julien Grall
2014-03-14 11:10 ` Andrii Anisov
2014-03-14 11:14 ` Andrii Anisov
2014-03-14 12:15 ` Julien Grall
2014-03-14 12:45 ` Andrii Anisov
2014-03-14 14:00 ` Julien Grall
2014-03-14 14:05 ` Andrii Anisov
2014-03-14 14:17 ` Julien Grall
2014-03-14 14:37 ` Oleksandr Tyshchenko
2014-03-14 14:08 ` Oleksandr Tyshchenko
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.