* [libvirt] libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose).
@ 2015-04-08 14:47 Konrad Rzeszutek Wilk
2015-04-08 15:45 ` Andrew Cooper
2015-04-14 17:31 ` [libvirt] [PATCH 0/2] " Ian Jackson
0 siblings, 2 replies; 13+ messages in thread
From: Konrad Rzeszutek Wilk @ 2015-04-08 14:47 UTC (permalink / raw)
To: jfehlig, xen-devel, ian.jackson, andrew.cooper3; +Cc: libvir-list
Hey Jim, Andrew, and Ian,
This is libvirt v1.2.14 + three patches:
c82a59b libxl: drop virDomainObj lock when destroying a domain
a1c9d30 libxl: acquire a job when destroying a domain
5bd5406 libxl: Move job acquisition in libxlDomainStart to callers
For fun I've set up an guest with PCI passthrough and tried to save it
(HAHAH) with an disastrous result (xc_save_helper was stuck). Probably
due to outstanding DMA operations wreaking havoc.
I will shortly send an patch to disallow 'save' with hosted devices
but this scenario exposed an interesting livelock that can happen
without passthrough if things go wrong.
I have not dug entirely in the code so my observation and theory
are most likely incorrect. CC-ing Ian and Andrew.
The full dump is below, the end result is that libvirtd is locked.
Doing 'libvirt list' or any other commands ends with such command
hanged.
libvirtd itself locks to be deadlocked on CTX_LOCK. On one thread
it has received an event (libxl_osevent_occurred_fd) and tries to
take the CTX_LOCK, while another thread (number 10) is stuck
in eventloop_iteration which holds the CTX_LOCK.
It looks like thread #10 is blocking in libxl_read_exactly waiting
for 'libxl-save-helper'. Said application (see below) has dispatched
an message through helper_getreply and is blocking on __read_nocancel.
I presume the message it has dispatched is the same message
the 'libxl_osevent_occurred_fd' is trying to get.
In other words, one thread has taken the lock, and is blocked
waiting for 'libxl-save-helper'. While 'libxl-save-helper' is blocked
waiting for its message to be replied to - which another libvirtd
thread tries to do but it cannot as the lock has been already taken.
Since I did not dig completely in the code I am probably very
wrong about this. Would appreciate thoughts whether this is possible
and where I've gotten it wrong.
Sorry about the control characters and such. I was on a serial
console and capturing this.
thread apply bt full
(gdb) bt
#0 0x00007fd3c50d5f1d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fd3c50d0906 in pthread_mutex_lock () from /lib64/libpthread.so.0
#2 0x00007fd3c5b849b9 in libxl__ctx_lock (ctx=0x7fd38c027ed0) at libxl_internal.h:3259
#3 0x00007fd3c5b87c8e in libxl_osevent_occurred_fd (ctx=0x7fd38c027ed0, for_libxl=0x7fd3ac004230, fd=23, events_ign=0,
revents_ign=1) at libxl_event.c:1235
#4 0x00007fd3c6825dc3 in virEventPollDispatchHandles (fds=<optimized out>, nfds=<optimized out>) at util/vireventpoll.c:508
#5 virEventPollRunOnce () at util/vireventpoll.c:657
#6 0x00007fd3c6824651 in virEventRunDefaultImpl () at util/virevent.c:308
#7 0x00007fd3c72aee2d in virNetServerRun (srv=0x7fd3c8bfab40) at rpc/virnetserver.c:1139
#8 0x00007fd3c727add5 in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1491
(gdb) thread apply all bt full
Thread 12 (Thread 0x7fd3bc1b6700 (LWP 1263)):
#0 0x00007fd3c50d3590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c685c0e6 in virCondWait (c=c@entry=0x7fd3c8bfad98, m=m@entry=0x7fd3c8bfad70) at util/virthread.c:154
ret = <optimized out>
#2 0x00007fd3c685c99b in virThreadPoolWorker (opaque=opaque@entry=0x7fd3c8bfa7a0) at util/virthreadpool.c:105
data = 0x0
pool = 0x7fd3c8bfad30
cond = 0x7fd3c8bfad98
priority = false
job = 0x0
#3 0x00007fd3c685be88 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
args = 0x0
local = {func = 0x7fd3c685c7b0 <virThreadPoolWorker>, funcName = 0x7fd3c72c0bb1 "virNetServerHandleJob", worker = true,
opaque = 0x7fd3c8bfa7a0}
#4 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 11 (Thread 0x7fd3bb9b5700 (LWP 1264)):
#0 0x00007fd3c50d3590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c685c0e6 in virCondWait (c=c@entry=0x7fd3c8bfad98, m=m@entry=0x7fd3c8bfad70) at util/virthread.c:154
ret = <optimized out>
#2 0x00007fd3c685c99b in virThreadPoolWorker (opaque=opaque@entry=0x7fd3c8bfa960) at util/virthreadpool.c:105
data = 0x0
pool = 0x7fd3c8bfad30
cond = 0x7fd3c8bfad98
priority = false
job = 0x0
#3 0x00007fd3c685be88 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
args = 0x0
local = {func = 0x7fd3c685c7b0 <virThreadPoolWorker>, funcName = 0x7fd3c72c0bb1 "virNetServerHandleJob", worker = true,
opaque = 0x7fd3c8bfa960}
#4 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 10 (Thread 0x7fd3bb1b4700 (LWP 1265)):
---Type <return> to continue, or q <return> to quit---
#0 0x00007fd3c50d61cd in read () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c5b78df2 in libxl_read_exactly (ctx=0x7fd38c027ed0, fd=31, data=0x7fd3bb1b37ce, sz=2,
source=0x7fd3b4001520 "domain 5 save/restore helper stdout pipe", what=0x7fd3c5bb7cca "ipc msg header") at libxl_utils.c:430
got = 8313484836034408292
__func__ = "libxl_read_exactly"
#2 0x00007fd3c5b80fce in helper_stdout_readable (egc=0x7fd3bb1b3930, ev=0x7fd3b4002868, fd=31, events=3, revents=1)
at libxl_save_callout.c:279
shs = 0x7fd3b40027f0
ao = 0x7fd3b4000e50
gc = 0x7fd3b4000e60
rc = 32723
errnoval = 32723
__func__ = "helper_stdout_readable"
msglen = 62
msg = <error reading variable msg (Insufficient memory in host GDB for object of size 8026576060204213623 bytes, maximum allowed 2305843009213693951 bytes.)>
#3 0x00007fd3c5b87928 in afterpoll_internal (egc=0x7fd3bb1b3930, poller=0x7fd3ac000e30, nfds=4, fds=0x7fd3ac0016f0, now=...)
at libxl_event.c:1178
revents = 1
gc = 0x7fd3bb1b3930
efd = 0x7fd3b4002868
__func__ = "afterpoll_internal"
__PRETTY_FUNCTION__ = "afterpoll_internal"
#4 0x00007fd3c5b89070 in eventloop_iteration (egc=0x7fd3bb1b3930, poller=0x7fd3ac000e30) at libxl_event.c:1638
gc = 0x7fd3bb1b3930
rc = 0
nfds = 4
now = {tv_sec = 1428428961, tv_usec = 930003}
timeout = -1
__func__ = "eventloop_iteration"
#5 0x00007fd3c5b89bb6 in libxl__ao_inprogress (ao=0x7fd3b4000e50, file=0x7fd3c5baca3b "libxl.c", line=982,
func=0x7fd3c5baf300 <__func__.17587> "libxl_domain_suspend") at libxl_event.c:1889
egc = {gc = {alloc_maxsize = 0, alloc_ptrs = 0x0, owner = 0x7fd38c027ed0}, occurred_for_callback = {tqh_first = 0x0,
tqh_last = 0x7fd3bb1b3948}, aos_for_callback = {tqh_first = 0x0, tqh_last = 0x7fd3bb1b3958}, aops_for_callback = {
tqh_first = 0x0, tqh_last = 0x7fd3bb1b3968}}
gc = 0x7fd3b4000e60
rc = 0
__PRETTY_FUNCTION__ = "libxl__ao_inprogress"
__func__ = "libxl__ao_inprogress"
#6 0x00007fd3c5b46178 in libxl_domain_suspend (ctx=0x7fd38c027ed0, domid=5, fd=27, flags=0, ao_how=0x0) at libxl.c:982
ao__ctx = 0x7fd38c027ed0
---Type <return> to continue, or q <return> to quit---
ao__rc = 0
ao = 0x7fd3b4000e50
__func__ = "libxl_domain_suspend"
egc = {{gc = {alloc_maxsize = 0, alloc_ptrs = 0x0, owner = 0x7fd38c027ed0}, occurred_for_callback = {tqh_first = 0x0,
tqh_last = 0x7fd3bb1b3a08}, aos_for_callback = {tqh_first = 0x0, tqh_last = 0x7fd3bb1b3a18}, aops_for_callback = {
tqh_first = 0x0, tqh_last = 0x7fd3bb1b3a28}}}
gc = 0x7fd3b4000e60
rc = 0
type = LIBXL_DOMAIN_TYPE_HVM
dss = 0x7fd3b40025f0
__PRETTY_FUNCTION__ = "libxl_domain_suspend"
#7 0x00007fd399cce9a8 in libxlDoDomainSave (driver=driver@entry=0x7fd38c0a3af0, vm=vm@entry=0x7fd38c00cb80,
to=to@entry=0x7fd3b4000980 "/root/save") at libxl/libxl_driver.c:1584
cfg = 0x7fd38c014050
hdr = {magic = "libvirt-xml\n \000 \r", version = 1, xmlLen = 1555, unused = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}}
event = 0x0
xml = 0x7fd3b40019d0 "<domain type='xen' id='5'>\n <name>OL6</name>\n <uuid>8c46a459-f9b8-42bb-94aa-c469adda5a97</uuid>\n <memory unit='KiB'>2097152</memory>\n <currentMemory unit='KiB'>2097152</currentMemory>\n <vcpu plac"...
xml_len = 1555
fd = 27
ret = -1
__FUNCTION__ = "libxlDoDomainSave"
#8 0x00007fd399ccefc2 in libxlDomainSaveFlags (dom=0x7fd3b4001410, to=0x7fd3b4000980 "/root/save", dxml=<optimized out>,
flags=<optimized out>) at libxl/libxl_driver.c:1653
driver = 0x7fd38c0a3af0
vm = 0x7fd38c00cb80
ret = -1
remove_dom = false
__FUNCTION__ = "libxlDomainSaveFlags"
#9 0x00007fd3c68e95a4 in virDomainSave (domain=domain@entry=0x7fd3b4001410, to=0x7fd3b4001390 "/root/save")
at libvirt-domain.c:841
ret = <optimized out>
absolute_to = 0x7fd3b4000980 "/root/save"
conn = 0x7fd3b4001240
__func__ = "virDomainSave"
__FUNCTION__ = "virDomainSave"
#10 0x00007fd3c729be8f in remoteDispatchDomainSave (server=0x7fd3c8bfab40, msg=0x7fd3c8bfded0, args=0x7fd3b4001450,
rerr=0x7fd3bb1b3ca0, client=<optimized out>) at remote_dispatch.h:8071
dom = 0x7fd3b4001410
priv = <optimized out>
#11 remoteDispatchDomainSaveHelper (server=0x7fd3c8bfab40, client=<optimized out>, msg=0x7fd3c8bfded0, rerr=0x7fd3bb1b3ca0,
args=0x7fd3b4001450, ret=0x7fd3b4000e30) at remote_dispatch.h:8047
---Type <return> to continue, or q <return> to quit---
rv = -1275063216
__func__ = "remoteDispatchDomainSaveHelper"
#12 0x00007fd3c6948789 in virNetServerProgramDispatchCall (msg=0x7fd3c8bfded0, client=0x7fd3c8c02330, server=0x7fd3c8bfab40,
prog=0x7fd3c8bf4d10) at rpc/virnetserverprogram.c:437
ret = 0x7fd3b4000e30 ""
rv = -1
i = <optimized out>
identity = 0x7fd3ac0009a0
arg = 0x7fd3b4001450 "\300\r"
dispatcher = 0x7fd3c74d8c90 <remoteProcs+2640>
rerr = {code = 0, domain = 0, message = 0x0, level = 0, dom = 0x0, str1 = 0x0, str2 = 0x0, str3 = 0x0, int1 = 0,
int2 = 0, net = 0x0}
#13 virNetServerProgramDispatch (prog=0x7fd3c8bf4d10, server=server@entry=0x7fd3c8bfab40, client=0x7fd3c8c02330,
msg=0x7fd3c8bfded0) at rpc/virnetserverprogram.c:307
rerr = {code = 0, domain = 0, message = 0x0, level = 0, dom = 0x0, str1 = 0x0, str2 = 0x0, str3 = 0x0, int1 = 0,
int2 = 0, net = 0x0}
__func__ = "virNetServerProgramDispatch"
__FUNCTION__ = "virNetServerProgramDispatch"
#14 0x00007fd3c72ad9f8 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>,
srv=0x7fd3c8bfab40) at rpc/virnetserver.c:172
ret = -1
#15 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7fd3c8bfab40) at rpc/virnetserver.c:193
srv = 0x7fd3c8bfab40
job = 0x7fd3c8c0e9b0
__func__ = "virNetServerHandleJob"
#16 0x00007fd3c685c8fe in virThreadPoolWorker (opaque=opaque@entry=0x7fd3c8bfa650) at util/virthreadpool.c:145
data = 0x0
pool = 0x7fd3c8bfad30
cond = 0x7fd3c8bfad98
priority = false
job = 0x7fd3c8bf51b0
#17 0x00007fd3c685be88 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
args = 0x0
local = {func = 0x7fd3c685c7b0 <virThreadPoolWorker>, funcName = 0x7fd3c72c0bb1 "virNetServerHandleJob", worker = true,
opaque = 0x7fd3c8bfa650}
#18 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#19 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 9 (Thread 0x7fd3ba9b3700 (LWP 1266)):
#0 0x00007fd3c50d3590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#1 0x00007fd3c685c0e6 in virCondWait (c=c@entry=0x7fd3c8bfad98, m=m@entry=0x7fd3c8bfad70) at util/virthread.c:154
ret = <optimized out>
#2 0x00007fd3c685c99b in virThreadPoolWorker (opaque=opaque@entry=0x7fd3c8bfa5e0) at util/virthreadpool.c:105
data = 0x0
pool = 0x7fd3c8bfad30
cond = 0x7fd3c8bfad98
priority = false
job = 0x0
#3 0x00007fd3c685be88 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
args = 0x0
local = {func = 0x7fd3c685c7b0 <virThreadPoolWorker>, funcName = 0x7fd3c72c0bb1 "virNetServerHandleJob", worker = true,
opaque = 0x7fd3c8bfa5e0}
#4 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 8 (Thread 0x7fd3ba1b2700 (LWP 1267)):
#0 0x00007fd3c50d3590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c685c0e6 in virCondWait (c=c@entry=0x7fd3c8bfad98, m=m@entry=0x7fd3c8bfad70) at util/virthread.c:154
ret = <optimized out>
#2 0x00007fd3c685c99b in virThreadPoolWorker (opaque=opaque@entry=0x7fd3c8c048a0) at util/virthreadpool.c:105
data = 0x0
pool = 0x7fd3c8bfad30
cond = 0x7fd3c8bfad98
priority = false
job = 0x0
#3 0x00007fd3c685be88 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
args = 0x0
local = {func = 0x7fd3c685c7b0 <virThreadPoolWorker>, funcName = 0x7fd3c72c0bb1 "virNetServerHandleJob", worker = true,
opaque = 0x7fd3c8c048a0}
#4 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 7 (Thread 0x7fd3b99b1700 (LWP 1268)):
#0 0x00007fd3c50d3590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c685c0e6 in virCondWait (c=c@entry=0x7fd3c8bfae30, m=m@entry=0x7fd3c8bfad70) at util/virthread.c:154
---Type <return> to continue, or q <return> to quit---
ret = <optimized out>
#2 0x00007fd3c685c99b in virThreadPoolWorker (opaque=opaque@entry=0x7fd3c8bfa960) at util/virthreadpool.c:105
data = 0x0
pool = 0x7fd3c8bfad30
cond = 0x7fd3c8bfae30
priority = true
job = 0x0
#3 0x00007fd3c685be88 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
args = 0x0
local = {func = 0x7fd3c685c7b0 <virThreadPoolWorker>, funcName = 0x7fd3c72c0bb1 "virNetServerHandleJob", worker = true,
opaque = 0x7fd3c8bfa960}
#4 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 6 (Thread 0x7fd3b91b0700 (LWP 1269)):
#0 0x00007fd3c50d3590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c685c0e6 in virCondWait (c=c@entry=0x7fd3c8bfae30, m=m@entry=0x7fd3c8bfad70) at util/virthread.c:154
ret = <optimized out>
#2 0x00007fd3c685c99b in virThreadPoolWorker (opaque=opaque@entry=0x7fd3c8bfa5e0) at util/virthreadpool.c:105
data = 0x0
pool = 0x7fd3c8bfad30
cond = 0x7fd3c8bfae30
priority = true
job = 0x0
#3 0x00007fd3c685be88 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
args = 0x0
local = {func = 0x7fd3c685c7b0 <virThreadPoolWorker>, funcName = 0x7fd3c72c0bb1 "virNetServerHandleJob", worker = true,
opaque = 0x7fd3c8bfa5e0}
#4 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 5 (Thread 0x7fd3b89af700 (LWP 1270)):
#0 0x00007fd3c50d3590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c685c0e6 in virCondWait (c=c@entry=0x7fd3c8bfae30, m=m@entry=0x7fd3c8bfad70) at util/virthread.c:154
ret = <optimized out>
#2 0x00007fd3c685c99b in virThreadPoolWorker (opaque=opaque@entry=0x7fd3c8c048a0) at util/virthreadpool.c:105
---Type <return> to continue, or q <return> to quit---
data = 0x0
pool = 0x7fd3c8bfad30
cond = 0x7fd3c8bfae30
priority = true
job = 0x0
#3 0x00007fd3c685be88 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
args = 0x0
local = {func = 0x7fd3c685c7b0 <virThreadPoolWorker>, funcName = 0x7fd3c72c0bb1 "virNetServerHandleJob", worker = true,
opaque = 0x7fd3c8c048a0}
#4 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 4 (Thread 0x7fd39bfff700 (LWP 1271)):
#0 0x00007fd3c50d3590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c685c0e6 in virCondWait (c=c@entry=0x7fd3c8bfae30, m=m@entry=0x7fd3c8bfad70) at util/virthread.c:154
ret = <optimized out>
#2 0x00007fd3c685c99b in virThreadPoolWorker (opaque=opaque@entry=0x7fd3c8bfa7a0) at util/virthreadpool.c:105
data = 0x0
pool = 0x7fd3c8bfad30
cond = 0x7fd3c8bfae30
priority = true
job = 0x0
#3 0x00007fd3c685be88 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
args = 0x0
local = {func = 0x7fd3c685c7b0 <virThreadPoolWorker>, funcName = 0x7fd3c72c0bb1 "virNetServerHandleJob", worker = true,
opaque = 0x7fd3c8bfa7a0}
#4 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 3 (Thread 0x7fd39b7fe700 (LWP 1272)):
#0 0x00007fd3c50d3590 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c685c0e6 in virCondWait (c=c@entry=0x7fd3c8bfae30, m=m@entry=0x7fd3c8bfad70) at util/virthread.c:154
ret = <optimized out>
#2 0x00007fd3c685c99b in virThreadPoolWorker (opaque=opaque@entry=0x7fd3c8bfa5e0) at util/virthreadpool.c:105
data = 0x0
pool = 0x7fd3c8bfad30
---Type <return> to continue, or q <return> to quit---
cond = 0x7fd3c8bfae30
priority = true
job = 0x0
#3 0x00007fd3c685be88 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
args = 0x0
local = {func = 0x7fd3c685c7b0 <virThreadPoolWorker>, funcName = 0x7fd3c72c0bb1 "virNetServerHandleJob", worker = true,
opaque = 0x7fd3c8bfa5e0}
#4 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 2 (Thread 0x7fd3c7208700 (LWP 1862)):
#0 0x00007fd3c50d61cd in read () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c2f61c70 in read_all (fd=16, data=0x7fd38c02a450, len=16, nonblocking=0) at xs.c:378
done = 0
#2 0x00007fd3c2f636fc in read_message (h=0x7fd38c028150, nonblocking=0) at xs.c:1150
__cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {0, -6517440478529039152, 140547370732032, 0, 140547555624704,
140547555625408, -6517440478602439472, -6517432384286599984}, __mask_was_saved = 0}}, __pad = {0x7fd3c7207df0,
0x0, 0x0, 0x0}}
__cancel_routine = 0x7fd3c4d8d440 <free>
__cancel_arg = 0x7fd38c02a440
__not_first_call = 0
msg = 0x7fd38c02a440
body = 0x0
saved_errno = 0
ret = -1
#3 0x00007fd3c2f63a4c in read_thread (arg=0x7fd38c028150) at xs.c:1222
h = 0x7fd38c028150
fd = 0
#4 0x00007fd3c50ce52a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fd3c4e0a22d in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 1 (Thread 0x7fd3c7209880 (LWP 1262)):
#0 0x00007fd3c50d5f1d in __lll_lock_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fd3c50d0906 in pthread_mutex_lock () from /lib64/libpthread.so.0
No symbol table info available.
#2 0x00007fd3c5b849b9 in libxl__ctx_lock (ctx=0x7fd38c027ed0) at libxl_internal.h:3259
---Type <return> to continue, or q <return> to quit---
r = 32767
__PRETTY_FUNCTION__ = "libxl__ctx_lock"
#3 0x00007fd3c5b87c8e in libxl_osevent_occurred_fd (ctx=0x7fd38c027ed0, for_libxl=0x7fd3ac004230, fd=23, events_ign=0,
revents_ign=1) at libxl_event.c:1235
egc = {{gc = {alloc_maxsize = 0, alloc_ptrs = 0x0, owner = 0x7fd38c027ed0}, occurred_for_callback = {tqh_first = 0x0,
tqh_last = 0x7fff5b0bf878}, aos_for_callback = {tqh_first = 0x0, tqh_last = 0x7fff5b0bf888}, aops_for_callback = {
tqh_first = 0x0, tqh_last = 0x7fff5b0bf898}}}
gc = 0x7fff5b0bf860
__PRETTY_FUNCTION__ = "libxl_osevent_occurred_fd"
ev = 0x7fd3c6976f38
check = {fd = 11, events = 0, revents = 0}
__func__ = "libxl_osevent_occurred_fd"
#4 0x00007fd3c6825dc3 in virEventPollDispatchHandles (fds=<optimized out>, nfds=<optimized out>) at util/vireventpoll.c:508
cb = 0x7fd399cc70d0 <libxlFDEventCallback>
watch = 66
opaque = <optimized out>
hEvents = 1
i = 6
n = 6
#5 virEventPollRunOnce () at util/vireventpoll.c:657
fds = 0x7fd3c8c0ee10
ret = <optimized out>
timeout = <optimized out>
nfds = <optimized out>
__func__ = "virEventPollRunOnce"
__FUNCTION__ = "virEventPollRunOnce"
#6 0x00007fd3c6824651 in virEventRunDefaultImpl () at util/virevent.c:308
__func__ = "virEventRunDefaultImpl"
#7 0x00007fd3c72aee2d in virNetServerRun (srv=0x7fd3c8bfab40) at rpc/virnetserver.c:1139
timerid = -1
timerActive = false
i = <optimized out>
__FUNCTION__ = "virNetServerRun"
__func__ = "virNetServerRun"
#8 0x00007fd3c727add5 in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1491
srv = 0x7fd3c8bfab40
remote_config_file = 0x7fd3c8bf32a0 "/etc/libvirt/libvirtd.conf"
statuswrite = -1
ret = 1
pid_file_fd = 4
pid_file = 0x7fd3c8bfa880 "/var/run/libvirtd.pid"
sock_file = 0x7fd3c8c04aa0 "/var/run/libvirt/libvirt-sock"
---Type <return> to continue, or q <return> to quit---
sock_file_ro = 0x7fd3c8c04a30 "/var/run/libvirt/libvirt-sock-ro"
timeout = -1
verbose = 0
godaemon = 100
ipsock = 0
config = 0x7fd3c8bf6840
privileged = <optimized out>
implicit_conf = <optimized out>
run_dir = 0x7fd3c8bfa810 "/var/run/libvirt"
old_umask = <optimized out>
opts = {{name = 0x7fd3c72b1afe "verbose", has_arg = 0, flag = 0x7fff5b0bf9e8, val = 118}, {
name = 0x7fd3c72b1b06 "daemon", has_arg = 0, flag = 0x7fff5b0bf9ec, val = 100}, {name = 0x7fd3c72b1b0d "listen",
has_arg = 0, flag = 0x7fff5b0bf9f0, val = 108}, {name = 0x7fd3c72b1c15 "config", has_arg = 1, flag = 0x0, val = 102},
{name = 0x7fd3c72b1b75 "timeout", has_arg = 1, flag = 0x0, val = 116}, {name = 0x7fd3c72b1b14 "pid-file", has_arg = 1,
flag = 0x0, val = 112}, {name = 0x7fd3c72b1b1d "version", has_arg = 0, flag = 0x0, val = 86}, {
name = 0x7fd3c72b1b25 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
__func__ = "main"
(gdb)
(gdb) quit
A debugging session is active.
Inferior 1 [process 1262] will be detached.
Quit anyway? (y or n) y
Detaching from program: /usr/sbin/libvirtd, process 1262
^[kroot@nic54:/root/libvirt/src^[\[root@nic54 src]# ps -eff|grep virsh
root 3333 2603 0 13:46 pts/3 00:00:00 ^[[01;31m^[[Kvirsh^[[m^[[K console OL6
root 3412 2856 0 13:49 pts/4 00:00:00 ^[[01;31m^[[Kvirsh^[[m^[[K list
root 3466 1830 0 13:56 pts/0 00:00:00 grep --color=auto ^[[01;31m^[[Kvirsh^[[m^[[K
^[kroot@nic54:/root/libvirt/src^[\[root@nic54 src]# gdb virsh 3412
GNU gdb (GDB) Fedora 7.8.2-38.fc21
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from virsh...done.
Attaching to program: /usr/bin/virsh, process 3412
Reading symbols from /lib64/libvirt.so.0...done.
Loaded symbols for /lib64/libvirt.so.0
Reading symbols from /lib64/libvirt-lxc.so.0...done.
Loaded symbols for /lib64/libvirt-lxc.so.0
Reading symbols from /lib64/libvirt-qemu.so.0...done.
Loaded symbols for /lib64/libvirt-qemu.so.0
Reading symbols from /lib64/libyajl.so.2...Reading symbols from /lib64/libyajl.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libyajl.so.2
Reading symbols from /lib64/libdevmapper.so.1.02...Reading symbols from /lib64/libdevmapper.so.1.02...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libdevmapper.so.1.02
Reading symbols from /lib64/libsystemd.so.0...Reading symbols from /lib64/libsystemd.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsystemd.so.0
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib/libxenlight.so.4.5...done.
Loaded symbols for /lib/libxenlight.so.4.5
Reading symbols from /lib/libxenctrl.so.4.5...done.
Loaded symbols for /lib/libxenctrl.so.4.5
Reading symbols from /lib/libxlutil.so.4.3...done.
Loaded symbols for /lib/libxlutil.so.4.3
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libxml2.so.2...Reading symbols from /lib64/libxml2.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libxml2.so.2
Reading symbols from /lib64/libncurses.so.5...Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libtinfo.so.5...Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 3413]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libsepol.so.1...Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libudev.so.1...Reading symbols from /lib64/libudev.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libudev.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/liblzma.so.5...Reading symbols from /lib64/liblzma.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblzma.so.5
Reading symbols from /lib64/libgcrypt.so.20...Reading symbols from /lib64/libgcrypt.so.20...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcrypt.so.20
Reading symbols from /lib64/libgpg-error.so.0...Reading symbols from /lib64/libgpg-error.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgpg-error.so.0
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libdw.so.1...Reading symbols from /lib64/libdw.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libdw.so.1
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib/libxenguest.so.4.5...done.
Loaded symbols for /lib/libxenguest.so.4.5
Reading symbols from /lib/libxenstore.so.3.0...done.
Loaded symbols for /lib/libxenstore.so.3.0
Reading symbols from /lib/libblktapctl.so.1.0...done.
Loaded symbols for /lib/libblktapctl.so.1.0
Reading symbols from /lib64/libuuid.so.1...Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/libz.so.1...Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libelf.so.1...Reading symbols from /lib64/libelf.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libelf.so.1
Reading symbols from /lib64/libbz2.so.1...Reading symbols from /lib64/libbz2.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libbz2.so.1
0x00007f08a9cdcc8d in poll () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-14.fc21.x86_64 device-mapper-libs-1.02.93-3.fc21.x86_64 elfutils-libelf-0.161-2.fc21.x86_64 elfutils-libs-0.161-2.fc21.x86_64 glibc-2.20-8.fc21.x86_64 libgcc-4.9.2-6.fc21.x86_64 libgcrypt-1.6.1-7.fc21.x86_64 libgpg-error-1.17-2.fc21.x86_64 libselinux-2.3-5.fc21.x86_64 libsepol-2.3-4.fc21.x86_64 libuuid-2.25.2-2.fc21.x86_64 libxml2-2.9.1-6.fc21.x86_64 ncurses-libs-5.9-16.20140323.fc21.x86_64 pcre-8.35-8.fc21.x86_64 systemd-libs-216-21.fc21.x86_64 xz-libs-5.1.2-14alpha.fc21.x86_64 yajl-2.1.0-3.fc21.x86_64 zlib-1.2.8-7.fc21.x86_64
(gdb) b\b^[[Kthreads
Undefined command: "threads". Try "help".
(gdb) thread
[Current thread is 1 (Thread 0x7f08ac538880 (LWP 3412))]
(gdb) bt full
#0 0x00007f08a9cdcc8d in poll () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f08ac078b6f in poll (__timeout=-1, __nfds=2, __fds=0x7fffc5a59e20) at /usr/include/bits/poll2.h:46
No locals.
#2 virNetClientIOEventLoop (client=client@entry=0x7f08aced6fa0, thiscall=thiscall@entry=0x7f08aced72f0)
at rpc/virnetclient.c:1513
ignore = 0 '\000'
oldmask = {__val = {0, 140736509353752, 140736509353552, 8926616876308275712, 140736509353568, 8926616876308275712,
139675237709568, 139675221565240, 139675223111672, 140736509353600, 206158430256, 140736509353832, 140736509353632,
8926616876308275712, 139675237798912, 139675223112640}}
timeout = -1
msg = 0x0
blockedsigs = {__val = {134287360, 0 <repeats 15 times>}}
fds = {{fd = 5, events = 1, revents = 0}, {fd = 6, events = 1, revents = 0}}
ret = <optimized out>
__FUNCTION__ = "virNetClientIOEventLoop"
__func__ = "virNetClientIOEventLoop"
#3 0x00007f08ac079316 in virNetClientIO (thiscall=0x7f08aced72f0, client=0x7f08aced6fa0) at rpc/virnetclient.c:1784
rv = -1
#4 virNetClientSendInternal (client=client@entry=0x7f08aced6fa0, msg=msg@entry=0x7f08aced7700,
expectReply=expectReply@entry=true, nonBlock=nonBlock@entry=false) at rpc/virnetclient.c:1956
call = 0x7f08aced72f0
ret = -1
__func__ = "virNetClientSendInternal"
__FUNCTION__ = "virNetClientSendInternal"
#5 0x00007f08ac07a293 in virNetClientSendWithReply (client=client@entry=0x7f08aced6fa0, msg=msg@entry=0x7f08aced7700)
at rpc/virnetclient.c:1984
ret = <optimized out>
#6 0x00007f08ac07aa62 in virNetClientProgramCall (prog=prog@entry=0x7f08aced71f0, client=client@entry=0x7f08aced6fa0,
serial=serial@entry=0, proc=proc@entry=66, noutfds=noutfds@entry=0, outfds=outfds@entry=0x0, ninfds=0x0, infds=0x0,
args_filter=0x7f08a9d1a880 <xdr_void>, args=0x0, ret_filter=0x7f08ac072b50 <xdr_remote_auth_list_ret>, ret=0x7fffc5a5a240)
at rpc/virnetclientprogram.c:329
msg = 0x7f08aced7700
i = <optimized out>
__FUNCTION__ = "virNetClientProgramCall"
#7 0x00007f08ac054b54 in callFull (priv=priv@entry=0x7f08aced7460, flags=flags@entry=0, fdin=fdin@entry=0x0,
fdinlen=fdinlen@entry=0, fdout=fdout@entry=0x0, fdoutlen=fdoutlen@entry=0x0, proc_nr=66,
args_filter=0x7f08a9d1a880 <xdr_void>, args=0x0, ret_filter=0x7f08ac072b50 <xdr_remote_auth_list_ret>, ret=0x7fffc5a5a240 "",
conn=0x7f08acedaa90) at remote/remote_driver.c:6600
rv = <optimized out>
prog = 0x7f08aced71f0
counter = 0
---Type <return> to continue, or q <return> to quit---
client = 0x7f08aced6fa0
#8 0x00007f08ac06d20e in call (conn=0x7f08acedaa90, ret=0x7fffc5a5a240 "", ret_filter=<optimized out>, args=0x0,
args_filter=<optimized out>, proc_nr=66, flags=0, priv=0x7f08aced7460) at remote/remote_driver.c:6622
No locals.
#9 remoteAuthenticate (conn=0x7f08acedaa90, auth=0x7f08ac36d400 <virConnectAuthDefault>, authtype=0x0, priv=0x7f08aced7460)
at remote/remote_driver.c:3885
ret = {types = {types_len = 0, types_val = 0x0}}
err = <optimized out>
type = 0
#10 doRemoteOpen (conn=conn@entry=0x7f08acedaa90, priv=priv@entry=0x7f08aced7460,
auth=auth@entry=0x7f08ac36d400 <virConnectAuthDefault>, flags=flags@entry=0) at remote/remote_driver.c:1002
transport_str = 0x0
transport = <optimized out>
daemonPath = 0x0
__FUNCTION__ = "doRemoteOpen"
name = 0x7f08aced7500 ""
command = 0x0
sockname = 0x7f08aced6270 "/var/run/libvirt/libvirt-sock"
netcat = 0x0
port = 0x0
authtype = 0x0
username = 0x0
sanity = true
verify = true
tty = true
pkipath = 0x0
keyfile = 0x0
sshauth = 0x0
knownHostsVerify = 0x0
knownHosts = 0x0
retcode = -2
i = <optimized out>
__func__ = "doRemoteOpen"
__PRETTY_FUNCTION__ = "doRemoteOpen"
#11 0x00007f08ac06e57e in remoteConnectOpen (conn=0x7f08acedaa90, auth=0x7f08ac36d400 <virConnectAuthDefault>,
flags=<optimized out>) at remote/remote_driver.c:1195
priv = <optimized out>
ret = <optimized out>
rflags = 0
autostart = 0x0
__func__ = "remoteConnectOpen"
#12 0x00007f08ac01d65d in do_open (name=name@entry=0x0, auth=auth@entry=0x7f08ac36d400 <virConnectAuthDefault>,
---Type <return> to continue, or q <return> to quit---
flags=flags@entry=0) at libvirt.c:1189
i = 0
res = <optimized out>
ret = 0x7f08acedaa90
conf = 0x7f08acedaa60
__FUNCTION__ = "do_open"
__func__ = "do_open"
#13 0x00007f08ac01ebf0 in virConnectOpenAuth (name=0x0, auth=0x7f08ac36d400 <virConnectAuthDefault>, flags=0) at libvirt.c:1341
ret = 0x0
__func__ = "virConnectOpenAuth"
#14 0x00007f08ac5ba66c in vshConnect (ctl=0x7fffc5a5a510, uri=<optimized out>, readonly=<optimized out>) at virsh.c:355
c = 0x0
interval = 5
count = 6
keepalive_forced = false
#15 0x00007f08ac5bd773 in vshReconnect (ctl=0x7fffc5a5a510) at virsh.c:401
connected = false
#16 0x00007f08ac5bdc08 in vshCommandRun (ctl=0x7fffc5a5a510, cmd=0x7f08acedd320) at virsh.c:1951
before = {tv_sec = 139675222862792, tv_usec = 1}
after = {tv_sec = 4294967295, tv_usec = 139675228146928}
enable_timing = false
ret = true
#17 0x00007f08ac5b88f7 in main (argc=<optimized out>, argv=0x7fffc5a5aca8) at virsh.c:3738
_ctl = {name = 0x0, conn = 0x0, cmd = 0x7f08acedd320, cmdstr = 0x0, imode = false, quiet = false, debug = 4,
timing = false, readonly = false, logfile = 0x0, log_fd = -1, historydir = 0x0, historyfile = 0x0, useGetInfo = false,
useSnapshotOld = false, blockJobNoBytes = false, eventLoop = {thread = 139675038205696}, lock = {lock = {__data = {
__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 512, __spins = 0, __elision = 0, __list = {
__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 17 times>, "\002", '\000' <repeats 21 times>,
__align = 0}}, eventLoopStarted = true, quit = false, eventPipe = {-1, -1}, eventTimerId = 1,
escapeChar = 0x7f08ac5eee65 "^]", keepalive_interval = -1, keepalive_count = -1, termattr = {c_iflag = 1280,
c_oflag = 5, c_cflag = 191, c_lflag = 35387, c_line = 0 '\000',
c_cc = "\003\034\177\025\004\000\001\000\021\023\032\000\022\017\027\026", '\000' <repeats 15 times>, c_ispeed = 15,
c_ospeed = 15}, istty = true}
ctl = 0x7fffc5a5a510
defaultConn = <optimized out>
ret = true
(gdb) quit
A debugging session is active.
Inferior 1 [process 3412] will be detached.
Quit anyway? (y or n) y
Detaching from program: /usr/bin/virsh, process 3412
^[kroot@nic54:/root/libvirt/src^[\[root@nic54 src]# ps -eff|Grep \b\b\b\b\b\b\b\b\b\b\b\b\b^[[Kps -eff|Grep \b^[[K\b^[[K\b^[[K\b^[[K\b^[[Kgrep xc
root 3257 1 1 13:46 ? 00:00:07 /usr/lib/xen/bin/qemu-system-i386 -xen-domid 5 -chardev socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-5,server,nowait -no-shutdown -mon chardev=libxl-cmd,mode=control -nodefaults -name OL6 -vnc 0.0.0.0:0 -display none -serial pty -device cirrus-vga,vgamem_mb=8 -boot order=c -usb -usbdevice tablet -smp 4,ma^[[01;31m^[[Kxc^[[m^[[Kpus=4 -net none -incoming fd:26 -machine xenfv -m 2040 -drive file=/dev/G/OL6,if=ide,index=0,media=disk,format=raw,cache=writeback
root 3473 1830 0 13:57 pts/0 00:00:00 grep --color=auto ^[[01;31m^[[Kxc^[[m^[[K
^[kroot@nic54:/root/libvirt/src^[\[root@nic54 src]# cd ../
^[kroot@nic54:/root/libvirt^[\[root@nic54 libvirt]# l\b^[[Kps -eff|grep xl
root 3257 1 1 13:46 ? 00:00:07 /usr/lib/xen/bin/qemu-system-i386 -xen-domid 5 -chardev socket,id=lib^[[01;31m^[[Kxl^[[m^[[K-cmd,path=/var/run/xen/qmp-lib^[[01;31m^[[Kxl^[[m^[[K-5,server,nowait -no-shutdown -mon chardev=lib^[[01;31m^[[Kxl^[[m^[[K-cmd,mode=control -nodefaults -name OL6 -vnc 0.0.0.0:0 -display none -serial pty -device cirrus-vga,vgamem_mb=8 -boot order=c -usb -usbdevice tablet -smp 4,maxcpus=4 -net none -incoming fd:26 -machine xenfv -m 2040 -drive file=/dev/G/OL6,if=ide,index=0,media=disk,format=raw,cache=writeback
root 3411 1262 0 13:49 ? 00:00:00 /usr/lib/xen/bin/lib^[[01;31m^[[Kxl^[[m^[[K-save-helper --save-domain 27 5 0 0 4 1 28 49 72
root 3475 1830 0 13:57 pts/0 00:00:00 grep --color=auto ^[[01;31m^[[Kxl^[[m^[[K
^[kroot@nic54:/root/libvirt^[\[root@nic54 libvirt]# cd
^[kroot@nic54:/root^[\[root@nic54 ~]# cd xen
^[kroot@nic54:/root/xen^[\[root@nic54 xen]# ls
^[[0m^[[01;32mautogen.sh^[[0m ^[[01;32mconfig.guess^[[0m ^[[01;32mconfig.status^[[0m configure.ac ^[[01;34mdist^[[0m INSTALL MAINTAINERS README ^[[01;34mtools^[[0m ^[[01;34mxen^[[0m
CODING_STYLE config.log ^[[01;32mconfig.sub^[[0m COPYING ^[[01;34mdocs^[[0m install.sh Makefile ^[[01;34mscripts^[[0m ^[[01;34munmodified_drivers^[[0m
^[[01;34mconfig^[[0m Config.mk ^[[01;32mconfigure^[[0m CREDITS ^[[01;34mextras^[[0m ^[[01;34mm4^[[0m ^[[01;34mmisc^[[0m ^[[01;34mstubdom^[[0m ^[[01;32mversion.sh^[[0m
^[kroot@nic54:/root/xen^[\[root@nic54 xen]# cd tools/
^[kroot@nic54:/root/xen/tools^[\[root@nic54 tools]# ls
^[[0m^[[01;34mblktap2^[[0m ^[[01;32mconfigure^[[0m ^[[01;34mexamples^[[0m ^[[01;34mlibfsimage^[[0m ^[[01;34mmemshr^[[0m ^[[01;36mqemu-xen-dir^[[0m ^[[01;34mtests^[[0m ^[[01;34mxenpmd^[[0m
config.h configure.ac ^[[01;34mfirmware^[[0m ^[[01;34mlibvchan^[[0m ^[[01;34mmisc^[[0m ^[[01;34mqemu-xen-dir-remote^[[0m ^[[01;34mxcutils^[[0m ^[[01;34mxenstat^[[0m
config.h.in ^[[01;34mconsole^[[0m ^[[01;34mflask^[[0m ^[[01;34mlibxc^[[0m ^[[01;34mocaml^[[0m ^[[01;36mqemu-xen-traditional-dir^[[0m ^[[01;34mxenbackendd^[[0m ^[[01;34mxenstore^[[0m
config.log ^[[01;32mcross-install^[[0m ^[[01;34mhotplug^[[0m ^[[01;34mlibxl^[[0m ^[[01;34mpygrub^[[0m ^[[01;34mqemu-xen-traditional-dir-remote^[[0m ^[[01;34mxenmon^[[0m ^[[01;34mxentrace^[[0m
^[[01;32mconfig.status^[[0m ^[[01;34mdebugger^[[0m ^[[01;34minclude^[[0m Makefile ^[[01;34mpython^[[0m Rules.mk ^[[01;34mxenpaging^[[0m
^[kroot@nic54:/root/xen/tools^[\[root@nic54 tools]# find ./ -\b\b\b\b\b\b\b\b\b^[[Kcd libxl
^[kroot@nic54:/root/xen/tools/libxl^[\[root@nic54 libxl]# gdb libxl-save-helper 3411
GNU gdb (GDB) Fedora 7.8.2-38.fc21
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from libxl-save-helper...done.
Attaching to program: /root/xen/tools/libxl/libxl-save-helper, process 3411
Reading symbols from /lib/libxenctrl.so.4.5...done.
Loaded symbols for /lib/libxenctrl.so.4.5
Reading symbols from /lib/libxenguest.so.4.5...done.
Loaded symbols for /lib/libxenguest.so.4.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libz.so.1...Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007f31876761b0 in __read_nocancel () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.20-8.fc21.x86_64 zlib-1.2.8-7.fc21.x86_64
(gdb) b\b^[[Kthread
[Current thread is 1 (Thread 0x7f3187ef7700 (LWP 3411))]
(gdb) bt full
#0 0x00007f31876761b0 in __read_nocancel () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x0000000000400ef3 in read_exactly (fd=0, buf=0x7fff06a86078, len=4) at libxl_save_helper.c:105
r = 4198631
__PRETTY_FUNCTION__ = "read_exactly"
#2 0x000000000040110b in helper_getreply (user=0x0) at libxl_save_helper.c:153
v = 39437520
r = 131072
#3 0x00000000004020c2 in helper_stub_suspend (user=0x0) at _libxl_save_msgs_helper.c:128
buf = 0x259c4d0 ""
len = 2
allocd = 2
__PRETTY_FUNCTION__ = "helper_stub_suspend"
r = 111698000
#4 0x00007f3187893c3c in suspend_and_state (suspend=0x401fec <helper_stub_suspend>, data=0x0, xch=0x259c010, io_fd=27, dom=5,
info=0x7fff06a8a9c0) at xc_domain_save.c:364
No locals.
#5 0x00007f3187895fe8 in xc_domain_save (xch=0x259c010, io_fd=27, dom=5, max_iters=29, max_factor=3, flags=4,
callbacks=0x604160 <helper_save_callbacks>, hvm=1) at xc_domain_save.c:987
info = {domid = 5, ssidref = 0, dying = 0, crashed = 0, shutdown = 0, paused = 0, blocked = 1, running = 0, hvm = 1,
debugged = 0, pvh = 0, shutdown_reason = 255, nr_pages = 524298, nr_outstanding_pages = 0, nr_shared_pages = 0,
nr_paged_pages = 0, shared_info_frame = 7640, cpu_time = 32660372500, max_memkb = 2098176, nr_online_vcpus = 4,
max_vcpu_id = 3, handle = "\214F\244Y\371\270B\273\224\252\304i\255\332Z\227", cpupool = 0}
domctl = {cmd = 2280706376, interface_version = 32561, domain = 0, u = {createdomain = {ssidref = 2280625784,
handle = "1\177\000\000\370Gχ1\177\000\000\004\000\000", flags = 0}, getdomaininfo = {domain = 38520,
flags = 32561, tot_pages = 139850708633592, max_pages = 4, outstanding_pages = 139850710750840, shr_pages = 1,
paged_pages = 0, shared_info_frame = 1, cpu_time = 139850710831432, nr_online_vcpus = 279491920, max_vcpu_id = 0,
ssidref = 4196058, handle = "\000\000\000\000\377\377\377\377\000\000\000\000\000\000\000", cpupool = 0},
getmemlist = {max_pfns = 139850710750840, start_pfn = 139850708633592, buffer = {p = 0x4},
num_pfns = 139850710750840}, getpageframeinfo = {gmfn = 139850710750840, type = 2278508536}, getpageframeinfo2 = {
num = 139850710750840, array = {p = 0x7f3187cf47f8 <_dl_lookup_symbol_x+312>}}, getpageframeinfo3 = {
num = 139850710750840, array = {p = 0x7f3187cf47f8 <_dl_lookup_symbol_x+312>}}, nodeaffinity = {nodemap = {
bitmap = {p = 0x7f3187ef9678 "\210\a@"}, nr_bits = 2278508536}}, vcpuaffinity = {vcpu = 2280625784,
flags = 32561, cpumap_hard = {bitmap = {
p = 0x7f3187cf47f8 <_dl_lookup_symbol_x+312> "H\203\304\060\203", <incomplete sequence \370>}, nr_bits = 4},
cpumap_soft = {bitmap = {p = 0x7f3187ef9678 "\210\a@"}, nr_bits = 1}}, shadow_op = {op = 2280625784, mode = 32561,
mb = 2278508536, dirty_bitmap = {p = 0x4 <error: Cannot access memory at address 0x4>}, pages = 139850710750840,
stats = {fault_count = 1, dirty_count = 0}}, max_mem = {max_memkb = 139850710750840}, vcpucontext = {
vcpu = 2280625784, ctxt = {p = 0x7f3187cf47f8 <_dl_lookup_symbol_x+312>}}, getvcpuinfo = {vcpu = 2280625784,
online = 49 '1', blocked = 127 '\177', running = 0 '\000', cpu_time = 139850708633592, cpu = 4}, max_vcpus = {
max = 2280625784}, scheduler_op = {sched_id = 2280625784, cmd = 32561, u = {sedf = {period = 139850708633592,
slice = 4, latency = 139850710750840, extratime = 1, weight = 0}, credit = {weight = 18424, cap = 34767},
---Type <return> to continue, or q <return> to quit---
credit2 = {weight = 18424}, rtds = {period = 2278508536, budget = 32561}}}, setdomainhandle = {
handle = "x\226\357\207\061\177\000\000\370Gχ1\177\000"}, setdebugging = {enable = 120 'x'}, irq_permission = {
pirq = 120 'x', allow_access = 150 '\226'}, iomem_permission = {first_mfn = 139850710750840,
nr_mfns = 139850708633592, allow_access = 4 '\004'}, ioport_permission = {first_port = 2280625784,
nr_ports = 32561, allow_access = 248 '\370'}, hypercall_init = {gmfn = 139850710750840}, settimeoffset = {
time_offset_seconds = 139850710750840}, disable_migrate = {disable = 2280625784}, tsc_info = {out_info = {
p = 0x7f3187ef9678}, info = {tsc_mode = 2278508536, gtsc_khz = 32561, incarnation = 4, pad = 0,
elapsed_nsec = 139850710750840}}, hvmcontext = {size = 2280625784, buffer = {
p = 0x7f3187cf47f8 <_dl_lookup_symbol_x+312> "H\203\304\060\203", <incomplete sequence \370>}},
hvmcontext_partial = {type = 2280625784, instance = 32561, buffer = {
p = 0x7f3187cf47f8 <_dl_lookup_symbol_x+312> "H\203\304\060\203", <incomplete sequence \370>}}, address_size = {
size = 2280625784}, sendtrigger = {trigger = 2280625784, vcpu = 32561}, get_device_group = {
machine_sbdf = 2280625784, max_sdevs = 32561, num_sdevs = 2278508536, sdev_array = {p = 0x4}}, assign_device = {
machine_sbdf = 2280625784}, bind_pt_irq = {machine_irq = 2280625784, irq_type = 32561, hvm_domid = 2278508536, u = {
isa = {isa_irq = 4 '\004'}, pci = {bus = 4 '\004', device = 0 '\000', intx = 0 '\000'}, msi = {gvec = 4 '\004',
gflags = 0, gtable = 139850710750840}}}, memory_mapping = {first_gfn = 139850710750840,
first_mfn = 139850708633592, nr_mfns = 4, add_mapping = 2280625784, padding = 32561}, ioport_mapping = {
first_gport = 2280625784, first_mport = 32561, nr_ports = 2278508536, add_mapping = 32561}, pin_mem_cacheattr = {
start = 139850710750840, end = 139850708633592, type = 4}, ext_vcpucontext = {vcpu = 2280625784, size = 32561,
syscall32_callback_eip = 139850708633592, sysenter_callback_eip = 4, syscall32_callback_cs = 38520,
sysenter_callback_cs = 34799, syscall32_disables_events = 49 '1', sysenter_disables_events = 127 '\177', {
mcg_cap = 1, vmce = {caps = 1, mci_ctl2_bank0 = 0, mci_ctl2_bank1 = 1}}}, set_target = {target = 38520},
subscribe = {port = 2280625784}, debug_op = {op = 2280625784, vcpu = 32561}, mem_event_op = {op = 2280625784,
mode = 32561, port = 2278508536}, mem_sharing_op = {op = 120 'x', u = {enable = 150 '\226'}}, cpuid = {input = {
2280625784, 32561}, eax = 2278508536, ebx = 32561, ecx = 4, edx = 0}, vcpuextstate = {vcpu = 2280625784,
xfeature_mask = 139850708633592, size = 4, buffer = {p = 0x7f3187ef9678}}, vcpu_msrs = {vcpu = 2280625784,
msr_count = 32561, msrs = {p = 0x7f3187cf47f8 <_dl_lookup_symbol_x+312>}}, access_required = {
access_required = 120 'x'}, audit_p2m = {orphans = 139850710750840, m2p_bad = 139850708633592, p2m_bad = 4},
set_virq_handler = {virq = 2280625784}, set_max_evtchn = {max_port = 2280625784}, gdbsx_guest_memio = {
pgd3val = 139850710750840, gva = 139850708633592, uva = 4, len = 2280625784, gwr = 49 '1', remain = 1},
set_broken_page_p2m = {pfn = 139850710750840}, cacheflush = {start_pfn = 139850710750840, nr_pfns = 139850708633592},
gdbsx_pauseunp_vcpu = {vcpu = 2280625784}, gdbsx_domstatus = {paused = 120 'x', vcpu_id = 32561,
vcpu_ev = 2278508536}, vnuma = {nr_vnodes = 2280625784, nr_vmemranges = 32561, nr_vcpus = 2278508536, pad = 32561,
vdistance = {p = 0x4}, vcpu_to_vnode = {p = 0x7f3187ef9678}, vnode_to_pnode = {p = 0x1}, vmemrange = {p = 0x0}},
psr_cmt_op = {cmd = 2280625784, data = 32561},
pad = "x\226\357\207\061\177\000\000\370Gχ1\177\000\000\004\000\000\000\000\000\000\000x\226\357\207\061\177\000\000\001", '\000' <repeats 15 times>, "\001\000\000\000\000\000\000\000H\321\360\207\061\177\000\000P\265\250\020\000\000\000\000\332\006@\000\000\000\000\000\377\377\377\377", '\000' <repeats 12 times>, "\240\324\360\207\061\177\000\000੨\006\377\177\000\000\377\377\377\377\001\000\000\000Щ\250\006\377\177\000"}}
rc = 0
frc = 0
i = 32767
---Type <return> to continue, or q <return> to quit---
j = 0
last_iter = 0
iter = 0
live = 0
debug = 0
superpages = 1
race = 0
sent_last_iter = 111717528
skip_this_iter = 0
sent_this_iter = 0
tmem_saved = 0
shared_info_frame = 7640
ctxt = {x64 = {fpu_ctxt = {x = '\000' <repeats 511 times>}, flags = 0, user_regs = {r15 = 0, r14 = 0, r13 = 0, r12 = 0, {
rbp = 0, ebp = 0}, {rbx = 0, ebx = 0}, r11 = 0, r10 = 0, r9 = 0, r8 = 0, {rax = 0, eax = 0}, {rcx = 0, ecx = 0}, {
rdx = 0, edx = 0}, {rsi = 0, esi = 0}, {rdi = 0, edi = 0}, error_code = 0, entry_vector = 0, {rip = 0, eip = 0},
cs = 0, _pad0 = {0}, saved_upcall_mask = 0 '\000', _pad1 = "\000\000", {rflags = 0, eflags = 0}, {rsp = 0,
esp = 0}, ss = 0, _pad2 = {0, 0, 0}, es = 0, _pad3 = {0, 0, 0}, ds = 0, _pad4 = {0, 0, 0}, fs = 0, _pad5 = {0, 0,
0}, gs = 0, _pad6 = {0, 0, 0}}, trap_ctxt = {{vector = 0 '\000', flags = 0 '\000', cs = 0,
address = 0} <repeats 157 times>, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850708618830}, {
vector = 0 '\000', flags = 0 '\000', cs = 0, address = 12288}, {vector = 104 'h', flags = 33 '!', cs = 0,
address = 8552}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850708618830}, {vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 86016}, {vector = 108 'l', flags = 70 'F', cs = 1, address = 83564}, {
vector = 0 '\000', flags = 0 '\000', cs = 0, address = 5}, {vector = 0 '\000', flags = 64 '@', cs = 33,
address = 2187264}, {vector = 0 '\000', flags = 82 'R', cs = 33, address = 2183688}, {vector = 0 '\000',
flags = 64 '@', cs = 1, address = 3}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0}, {
vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0}, {vector = 0 '\000', flags = 0 '\000', cs = 0,
address = 0}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0}, {vector = 0 '\000', flags = 0 '\000',
cs = 0, address = 0}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0}, {vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 139850708688590}, {vector = 0 '\000', flags = 0 '\000', cs = 0,
address = 139850710757048}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850708638876}, {
vector = 108 'l', flags = 46 '.', cs = 27, address = 139850710757048}, {vector = 17 '\021', flags = 0 '\000',
cs = 0, address = 139850708660020}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850693546923}, {
vector = 0 '\000', flags = 144 '\220', cs = 34799, address = 139850708625717}, {vector = 96 '`',
flags = 164 '\244', cs = 1704, address = 6573502260}, {vector = 0 '\000', flags = 0 '\000', cs = 0,
address = 139850701779419}, {vector = 24 '\030', flags = 160 '\240', cs = 34799, address = 139850708625717}, {
vector = 144 '\220', flags = 164 '\244', cs = 1704, address = 4294967296}, {vector = 0 '\000', flags = 0 '\000',
cs = 0, address = 139850695725576}, {vector = 8 '\b', flags = 82 'R', cs = 34570, address = 140733305103152}, {
vector = 208 '\320', flags = 161 '\241', cs = 1704, address = 2183688}, {vector = 4 '\004', flags = 0 '\000',
cs = 0, address = 6}, {vector = 1 '\001', flags = 0 '\000', cs = 0, address = 139850710757048}, {vector = 55 '7',
flags = 96 '`', cs = 34696, address = 139850710831392}, {vector = 0 '\000', flags = 253 '\375', cs = 0,
address = 2100235}, {vector = 1 '\001', flags = 0 '\000', cs = 0, address = 33261}, {vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 0}, {vector = 136 '\210', flags = 95 '_', cs = 1, address = 4096}, {
---Type <return> to continue, or q <return> to quit---
vector = 176 '\260', flags = 0 '\000', cs = 0, address = 1428421374}, {vector = 128 '\200', flags = 139 '\213',
cs = 8758, address = 1408431017}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 1415029273}, {
vector = 145 '\221', flags = 17 '\021', cs = 54, address = 0}, {vector = 0 '\000', flags = 0 '\000', cs = 0,
address = 0}, {vector = 1 '\001', flags = 0 '\000', cs = 0, address = 139850708660020}, {vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 139850703986771}, {vector = 48 '0', flags = 171 '\253', cs = 34800,
address = 139850708625717}, {vector = 48 '0', flags = 171 '\253', cs = 34800, address = 6573468531}, {
vector = 0 '\000', flags = 0 '\000', cs = 0, address = 140733305103600}, {vector = 0 '\000', flags = 0 '\000',
cs = 0, address = 139850708630484}, {vector = 88 'X', flags = 166 '\246', cs = 34800, address = 139850710754536},
{vector = 180 '\264', flags = 206 '\316', cs = 34602, address = 1298}, {vector = 48 '0', flags = 221 '\335',
cs = 34602, address = 139850708630484}, {vector = 18 '\022', flags = 5 '\005', cs = 0,
address = 139850710753304}, {vector = 216 '\330', flags = 124 '|', cs = 34662, address = 196}, {
vector = 16 '\020', flags = 127 '\177', cs = 34662, address = 139850708632351}, {vector = 196 '\304',
flags = 0 '\000', cs = 0, address = 139850701774504}, {vector = 248 '\370', flags = 162 '\242', cs = 34799,
address = 139850710754120}, {vector = 152 '\230', flags = 165 '\245', cs = 1704, address = 140733305103764}, {
vector = 159 '\237', flags = 20 '\024', cs = 34732, address = 139850701774504}, {vector = 0 '\000',
flags = 206 '\316', cs = 34731, address = 139850701771120}, {vector = 152 '\230', flags = 165 '\245', cs = 1704,
address = 258797660}, {vector = 193 '\301', flags = 179 '\263', cs = 61, address = 28}, {vector = 112 'p',
flags = 166 '\246', cs = 1704, address = 140733305103968}, {vector = 248 '\370', flags = 162 '\242', cs = 34799,
address = 139850710750728}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 8}, {vector = 88 'X',
flags = 166 '\246', cs = 34800, address = 0}, {vector = 56 '8', flags = 151 '\227', cs = 34799,
address = 139850710821296}, {vector = 0 '\000', flags = 167 '\247', cs = 1704, address = 140733305104168}, {
vector = 88 'X', flags = 166 '\246', cs = 34800, address = 0}, {vector = 80 'P', flags = 151 '\227', cs = 34799,
address = 139850708633592}, {vector = 3 '\003', flags = 0 '\000', cs = 0, address = 139850710751056}, {
vector = 1 '\001', flags = 0 '\000', cs = 0, address = 0}, {vector = 1 '\001', flags = 0 '\000', cs = 0,
address = 139850710820440}, {vector = 56 '8', flags = 151 '\227', cs = 34799, address = 140733305103920}, {
vector = 174 '\256', flags = 96 '`', cs = 65415, address = 0}, {vector = 176 '\260', flags = 169 '\251',
cs = 34800, address = 140733305103984}, {vector = 224 '\340', flags = 86 'V', cs = 34603,
address = 140733305103968}, {vector = 92 '\\', flags = 240 '\360', cs = 3948, address = 139850706326687}, {
vector = 255 '\377', flags = 255 '\377', cs = 65535, address = 0}, {vector = 112 'p', flags = 145 '\221',
cs = 34662, address = 139850710753304}, {vector = 16 '\020', flags = 29 '\035', cs = 34662,
address = 140733305104064}, {vector = 65 'A', flags = 89 'Y', cs = 63831, address = 140733305104063}, {
vector = 255 '\377', flags = 255 '\377', cs = 65535, address = 139850708586528}, {vector = 64 '@',
flags = 12 '\f', cs = 64, address = 140733305106160}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0},
{vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850708586696}, {vector = 64 '@', flags = 12 '\f',
cs = 64, address = 140733305106160}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0}, {
vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850708652908}, {vector = 1 '\001', flags = 0 '\000',
cs = 0, address = 0}, {vector = 67 'C', flags = 0 '\000', cs = 0, address = 139850701771120}, {vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 140733305104400}, {vector = 64 '@', flags = 12 '\f', cs = 64,
address = 139850708681520}, {vector = 97 'a', flags = 88 'X', cs = 63831, address = 140733305104287}, {
vector = 16 '\020', flags = 192 '\300', cs = 601, address = 139850698074918}, {vector = 152 '\230',
flags = 196 '\304', cs = 601, address = 0}, {vector = 16 '\020', flags = 168 '\250', cs = 1704, address = 16}, {
vector = 224 '\340', flags = 167 '\247', cs = 1704, address = 140733305104288}, {vector = 16 '\020',
---Type <return> to continue, or q <return> to quit---
flags = 192 '\300', cs = 601, address = 1}, {vector = 1 '\001', flags = 0 '\000', cs = 0, address = 39436288}},
ldt_base = 139850701741336, ldt_ents = 139850710820440, gdt_frames = {0, 139850706447238, 140733305106160, 39436304,
0, 17179869185, 4, 0, 140733305105696, 139850706396927, 88, 0, 6308096, 6308096, 0, 6308096}, gdt_ents = 0,
kernel_ss = 6308096, kernel_sp = 0, ctrlreg = {139637976727552, 139850710755272, 139850710755352, 140733305104648,
140733305104644, 4196203, 139850697911728, 4195048}, debugreg = {139850697911440, 140733305104648, 2090266759,
32660418, 7, 140733305104864, 140733305104848, 139850710755272}, event_callback_eip = 139850710750728,
failsafe_callback_eip = 139848430125056, syscall_callback_eip = 8, vm_assist = 2280625808, fs_base = 0,
gs_base_kernel = 3, gs_base_user = 139850710832288}, x32 = {fpu_ctxt = {x = '\000' <repeats 511 times>}, flags = 0,
user_regs = {ebx = 0, ecx = 0, edx = 0, esi = 0, edi = 0, ebp = 0, eax = 0, error_code = 0, entry_vector = 0,
eip = 0, cs = 0, saved_upcall_mask = 0 '\000', _pad0 = 0 '\000', eflags = 0, esp = 0, ss = 0, _pad1 = 0, es = 0,
_pad2 = 0, ds = 0, _pad3 = 0, fs = 0, _pad4 = 0, gs = 0, _pad5 = 0}, trap_ctxt = {{vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 0} <repeats 256 times>}, ldt_base = 0, ldt_ents = 0, gdt_frames = {
0 <repeats 16 times>}, gdt_ents = 0, kernel_ss = 0, kernel_sp = 0, ctrlreg = {0, 0, 0, 0, 0, 0, 0, 0}, debugreg = {
0, 0, 0, 0, 0, 0, 0, 0}, event_callback_cs = 0, event_callback_eip = 0, failsafe_callback_cs = 0,
failsafe_callback_eip = 0, vm_assist = 0}, c = {fpu_ctxt = {x = '\000' <repeats 511 times>}, flags = 0, user_regs = {
r15 = 0, r14 = 0, r13 = 0, r12 = 0, {rbp = 0, ebp = 0, _ebp = 0}, {rbx = 0, ebx = 0, _ebx = 0}, r11 = 0, r10 = 0,
r9 = 0, r8 = 0, {rax = 0, eax = 0, _eax = 0}, {rcx = 0, ecx = 0, _ecx = 0}, {rdx = 0, edx = 0, _edx = 0}, {rsi = 0,
esi = 0, _esi = 0}, {rdi = 0, edi = 0, _edi = 0}, error_code = 0, entry_vector = 0, {rip = 0, eip = 0, _eip = 0},
cs = 0, _pad0 = {0}, saved_upcall_mask = 0 '\000', _pad1 = "\000\000", {rflags = 0, eflags = 0, _eflags = 0}, {
rsp = 0, esp = 0, _esp = 0}, ss = 0, _pad2 = {0, 0, 0}, es = 0, _pad3 = {0, 0, 0}, ds = 0, _pad4 = {0, 0, 0},
fs = 0, _pad5 = {0, 0, 0}, gs = 0, _pad6 = {0, 0, 0}}, trap_ctxt = {{vector = 0 '\000', flags = 0 '\000', cs = 0,
address = 0} <repeats 157 times>, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850708618830}, {
vector = 0 '\000', flags = 0 '\000', cs = 0, address = 12288}, {vector = 104 'h', flags = 33 '!', cs = 0,
address = 8552}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850708618830}, {vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 86016}, {vector = 108 'l', flags = 70 'F', cs = 1, address = 83564}, {
vector = 0 '\000', flags = 0 '\000', cs = 0, address = 5}, {vector = 0 '\000', flags = 64 '@', cs = 33,
address = 2187264}, {vector = 0 '\000', flags = 82 'R', cs = 33, address = 2183688}, {vector = 0 '\000',
flags = 64 '@', cs = 1, address = 3}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0}, {
vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0}, {vector = 0 '\000', flags = 0 '\000', cs = 0,
address = 0}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0}, {vector = 0 '\000', flags = 0 '\000',
cs = 0, address = 0}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0}, {vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 139850708688590}, {vector = 0 '\000', flags = 0 '\000', cs = 0,
address = 139850710757048}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850708638876}, {
vector = 108 'l', flags = 46 '.', cs = 27, address = 139850710757048}, {vector = 17 '\021', flags = 0 '\000',
cs = 0, address = 139850708660020}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850693546923}, {
vector = 0 '\000', flags = 144 '\220', cs = 34799, address = 139850708625717}, {vector = 96 '`',
flags = 164 '\244', cs = 1704, address = 6573502260}, {vector = 0 '\000', flags = 0 '\000', cs = 0,
address = 139850701779419}, {vector = 24 '\030', flags = 160 '\240', cs = 34799, address = 139850708625717}, {
vector = 144 '\220', flags = 164 '\244', cs = 1704, address = 4294967296}, {vector = 0 '\000', flags = 0 '\000',
cs = 0, address = 139850695725576}, {vector = 8 '\b', flags = 82 'R', cs = 34570, address = 140733305103152}, {
vector = 208 '\320', flags = 161 '\241', cs = 1704, address = 2183688}, {vector = 4 '\004', flags = 0 '\000',
cs = 0, address = 6}, {vector = 1 '\001', flags = 0 '\000', cs = 0, address = 139850710757048}, {vector = 55 '7',
---Type <return> to continue, or q <return> to quit---
flags = 96 '`', cs = 34696, address = 139850710831392}, {vector = 0 '\000', flags = 253 '\375', cs = 0,
address = 2100235}, {vector = 1 '\001', flags = 0 '\000', cs = 0, address = 33261}, {vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 0}, {vector = 136 '\210', flags = 95 '_', cs = 1, address = 4096}, {
vector = 176 '\260', flags = 0 '\000', cs = 0, address = 1428421374}, {vector = 128 '\200', flags = 139 '\213',
cs = 8758, address = 1408431017}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 1415029273}, {
vector = 145 '\221', flags = 17 '\021', cs = 54, address = 0}, {vector = 0 '\000', flags = 0 '\000', cs = 0,
address = 0}, {vector = 1 '\001', flags = 0 '\000', cs = 0, address = 139850708660020}, {vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 139850703986771}, {vector = 48 '0', flags = 171 '\253', cs = 34800,
address = 139850708625717}, {vector = 48 '0', flags = 171 '\253', cs = 34800, address = 6573468531}, {
vector = 0 '\000', flags = 0 '\000', cs = 0, address = 140733305103600}, {vector = 0 '\000', flags = 0 '\000',
cs = 0, address = 139850708630484}, {vector = 88 'X', flags = 166 '\246', cs = 34800, address = 139850710754536},
{vector = 180 '\264', flags = 206 '\316', cs = 34602, address = 1298}, {vector = 48 '0', flags = 221 '\335',
cs = 34602, address = 139850708630484}, {vector = 18 '\022', flags = 5 '\005', cs = 0,
address = 139850710753304}, {vector = 216 '\330', flags = 124 '|', cs = 34662, address = 196}, {
vector = 16 '\020', flags = 127 '\177', cs = 34662, address = 139850708632351}, {vector = 196 '\304',
flags = 0 '\000', cs = 0, address = 139850701774504}, {vector = 248 '\370', flags = 162 '\242', cs = 34799,
address = 139850710754120}, {vector = 152 '\230', flags = 165 '\245', cs = 1704, address = 140733305103764}, {
vector = 159 '\237', flags = 20 '\024', cs = 34732, address = 139850701774504}, {vector = 0 '\000',
flags = 206 '\316', cs = 34731, address = 139850701771120}, {vector = 152 '\230', flags = 165 '\245', cs = 1704,
address = 258797660}, {vector = 193 '\301', flags = 179 '\263', cs = 61, address = 28}, {vector = 112 'p',
flags = 166 '\246', cs = 1704, address = 140733305103968}, {vector = 248 '\370', flags = 162 '\242', cs = 34799,
address = 139850710750728}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 8}, {vector = 88 'X',
flags = 166 '\246', cs = 34800, address = 0}, {vector = 56 '8', flags = 151 '\227', cs = 34799,
address = 139850710821296}, {vector = 0 '\000', flags = 167 '\247', cs = 1704, address = 140733305104168}, {
vector = 88 'X', flags = 166 '\246', cs = 34800, address = 0}, {vector = 80 'P', flags = 151 '\227', cs = 34799,
address = 139850708633592}, {vector = 3 '\003', flags = 0 '\000', cs = 0, address = 139850710751056}, {
vector = 1 '\001', flags = 0 '\000', cs = 0, address = 0}, {vector = 1 '\001', flags = 0 '\000', cs = 0,
address = 139850710820440}, {vector = 56 '8', flags = 151 '\227', cs = 34799, address = 140733305103920}, {
vector = 174 '\256', flags = 96 '`', cs = 65415, address = 0}, {vector = 176 '\260', flags = 169 '\251',
cs = 34800, address = 140733305103984}, {vector = 224 '\340', flags = 86 'V', cs = 34603,
address = 140733305103968}, {vector = 92 '\\', flags = 240 '\360', cs = 3948, address = 139850706326687}, {
vector = 255 '\377', flags = 255 '\377', cs = 65535, address = 0}, {vector = 112 'p', flags = 145 '\221',
cs = 34662, address = 139850710753304}, {vector = 16 '\020', flags = 29 '\035', cs = 34662,
address = 140733305104064}, {vector = 65 'A', flags = 89 'Y', cs = 63831, address = 140733305104063}, {
vector = 255 '\377', flags = 255 '\377', cs = 65535, address = 139850708586528}, {vector = 64 '@',
flags = 12 '\f', cs = 64, address = 140733305106160}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0},
{vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850708586696}, {vector = 64 '@', flags = 12 '\f',
cs = 64, address = 140733305106160}, {vector = 0 '\000', flags = 0 '\000', cs = 0, address = 0}, {
vector = 0 '\000', flags = 0 '\000', cs = 0, address = 139850708652908}, {vector = 1 '\001', flags = 0 '\000',
cs = 0, address = 0}, {vector = 67 'C', flags = 0 '\000', cs = 0, address = 139850701771120}, {vector = 0 '\000',
flags = 0 '\000', cs = 0, address = 140733305104400}, {vector = 64 '@', flags = 12 '\f', cs = 64,
address = 139850708681520}, {vector = 97 'a', flags = 88 'X', cs = 63831, address = 140733305104287}, {
---Type <return> to continue, or q <return> to quit---
vector = 16 '\020', flags = 192 '\300', cs = 601, address = 139850698074918}, {vector = 152 '\230',
flags = 196 '\304', cs = 601, address = 0}, {vector = 16 '\020', flags = 168 '\250', cs = 1704, address = 16}, {
vector = 224 '\340', flags = 167 '\247', cs = 1704, address = 140733305104288}, {vector = 16 '\020',
flags = 192 '\300', cs = 601, address = 1}, {vector = 1 '\001', flags = 0 '\000', cs = 0, address = 39436288}},
ldt_base = 139850701741336, ldt_ents = 139850710820440, gdt_frames = {0, 139850706447238, 140733305106160, 39436304,
0, 17179869185, 4, 0, 140733305105696, 139850706396927, 88, 0, 6308096, 6308096, 0, 6308096}, gdt_ents = 0,
kernel_ss = 6308096, kernel_sp = 0, ctrlreg = {139637976727552, 139850710755272, 139850710755352, 140733305104648,
140733305104644, 4196203, 139850697911728, 4195048}, debugreg = {139850697911440, 140733305104648, 2090266759,
32660418, 7, 140733305104864, 140733305104848, 139850710755272}, event_callback_eip = 139850710750728,
failsafe_callback_eip = 139848430125056, syscall_callback_eip = 8, vm_assist = 2280625808, fs_base = 0,
gs_base_kernel = 3, gs_base_user = 139850710832288}}
pfn_type = 0x0
pfn_batch = 0x0
pfn_err = 0x0
page = '\000' <repeats 4095 times>
live_shinfo = 0x0
region_base = 0x0
buffer = 0x0
xc__hypercall_buffer_buffer = {hbuf = 0x0, param_shadow = 0x0, dir = 0, ubuf = 0xffffffffffffffff, sz = 0}
to_skip = 0x0
xc__hypercall_buffer_to_skip = {hbuf = 0x0, param_shadow = 0x0, dir = 0, ubuf = 0xffffffffffffffff, sz = 0}
to_send = 0x0
xc__hypercall_buffer_to_send = {hbuf = 0x0, param_shadow = 0x0, dir = 0, ubuf = 0xffffffffffffffff, sz = 0}
to_fix = 0x0
time_stats = {wall = {tv_sec = 0, tv_usec = 0}, d0_cpu = 0, d1_cpu = 0}
shadow_stats = {fault_count = 0, dirty_count = 0}
needed_to_fix = 0
total_sent = 0
vcpumap = {1, 0 <repeats 63 times>}
hvm_buf_size = 0
hvm_buf = 0x0
magic_pfns = {0, 0, 0}
mfn = 140733305105560
ob_pagebuf = {buf = 0x7f3185e8f010, size = 16777216, pos = 0, write_count = 0}
ob_tailbuf = {buf = 0x0, size = 0, pos = 0, write_count = 0}
ob = 0x0
_ctx = {hvirt_start = 18446603336221196288, pt_levels = 4, max_mfn = 8650240, live_p2m = 0x0, live_m2p = 0x0,
m2p_mfn0 = 0, dinfo = {guest_width = 8, p2m_size = 1044481}}
ctx = 0x7fff06a88190
dinfo = 0x7fff06a881c0
compress_ctx = 0x0
compressing = 0
---Type <return> to continue, or q <return> to quit---
completed = 0
__func__ = "xc_domain_save"
__PRETTY_FUNCTION__ = "xc_domain_save"
#6 0x00000000004016c0 in main (argc=11, argv=0x7fff06a8af50) at libxl_save_helper.c:231
io_fd = 27
max_factor = 0
flags = 4
hvm = 1
dom = 5
max_iters = 0
cbflags = 72
r = 0
mode = 0x7fff06a8cf3c "--save-domain"
__PRETTY_FUNCTION__ = "main"
(gdb)
(gdb)
(gdb)
(gdb) bt
#0 0x00007f31876761b0 in __read_nocancel () from /lib64/libpthread.so.0
#1 0x0000000000400ef3 in read_exactly (fd=0, buf=0x7fff06a86078, len=4) at libxl_save_helper.c:105
#2 0x000000000040110b in helper_getreply (user=0x0) at libxl_save_helper.c:153
#3 0x00000000004020c2 in helper_stub_suspend (user=0x0) at _libxl_save_msgs_helper.c:128
#4 0x00007f3187893c3c in suspend_and_state (suspend=0x401fec <helper_stub_suspend>, data=0x0, xch=0x259c010, io_fd=27, dom=5,
info=0x7fff06a8a9c0) at xc_domain_save.c:364
#5 0x00007f3187895fe8 in xc_domain_save (xch=0x259c010, io_fd=27, dom=5, max_iters=29, max_factor=3, flags=4,
callbacks=0x604160 <helper_save_callbacks>, hvm=1) at xc_domain_save.c:987
#6 0x00000000004016c0 in main (argc=11, argv=0x7fff06a8af50) at libxl_save_helper.c:231
(gdb) quit
A debugging session is active.
Inferior 1 [process 3411] will be detached.
^[kroot@nic54:/usr/lib/xen/bin^[\[root@nic54 bin]# ./xenctx --help\b\b\b\b\b\b\b^[[K -f -D 5\b\b\b\b\b\b^[[4@-a -C^[[C^[[C^[[C^[[C^[[C\b\b\b\b\b\b\b\b\b\b\b\bh^[[K\bf -a -C -D 5\b\b\b\b\b\b\b\b^[[1P\b^[[1P^[[1P\b\b^[[1P\b^[[1P^[[1@f^[[1@ ^[[C^[[C^[[C^[[C^[[C^[[C\b\b^[[1P 5\b\b\b^[[1P 5\b\b\b\b\b\b\b^[[1P\b^[[1P
vcpu0:
rip: ffffffff810013a8
flags: 00000046 z p
rsp: ffff88007a8bdd50
rax: 0000000000000000 rcx: 0000000000000000 rdx: 0000000000000000
rbx: ffff88007a54bdb8 rsi: deadbeefdeadf00d rdi: deadbeefdeadf00d
rbp: ffff88007a8bdd78 r8: 0000000000000000 r9: 0000000000000000
r10: 0000000000000001 r11: 0000000000000010 r12: 0000000000000000
r13: 0000000000000003 r14: ffff88007a54bd4c r15: ffff88007a54bd01
cs: 0010 ss: 0018 ds: 0000 es: 0000
fs: 0000 @ 0000000000000000
gs: 0000 @ ffff88007be00000/0000000000000000
Code (instr addr ffffffff810013a8)
cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc
Stack:
ffffffff81332553 ffff880000000002 0000000000013fc0 ffff88007a54bd28
0000000000000296 ffff88007a8bddb8 ffffffff810d8dcb ffff88007a8ba200
ffff88007a54bc98 ffff88007a8ba200 ffff88007be0eea0 ffffffff810d8d20
ffff88007a54bd28 ffff88007a8bdeb8 ffffffff810d8a50 000000007a8bde18
ffff88007a8bdfd8 ffff88007a8ba200 ffff88007be0eea8 0000000000000000
Call Trace:
[<ffffffff810013a8>] <--
ffff88007a8bdd50: [<ffffffff81332553>]
ffff88007a8bdd80: [<ffffffff810d8dcb>]
ffff88007a8bdda8: [<ffffffff810d8d20>]
ffff88007a8bddc0: [<ffffffff810d8a50>]
ffff88007a8bde08: [<ffffffff819032a0>]
ffff88007a8bde20: [<ffffffff8159161d>]
ffff88007a8bdea8: [<ffffffff810d8960>]
ffff88007a8bdec0: [<ffffffff81082a6e>]
ffff88007a8bdf28: [<ffffffff810829a0>]
ffff88007a8bdf50: [<ffffffff8159b66c>]
ffff88007a8bdf80: [<ffffffff810829a0>]
ffff88007a8bdfd0: [<ffffffffffffffff>]
vcpu1:
rip: ffffffff810d8da3
flags: 00000097 s nz a p c
rsp: ffff88007abc3d88
rax: 0000000000000003 rcx: dead000000200200 rdx: 0000000000000000
rbx: ffff88007a54bd28 rsi: ffff88007abc00c0 rdi: ffff88007a54bd28
rbp: ffff88007abc3db8 r8: ffff88007be2eec0 r9: 000000444d5210c0
r10: 0000000000000000 r11: 0000000000000010 r12: 0000000000000296
r13: 0000000000000003 r14: ffff88007a54bd4c r15: ffff88007a54bd00
cs: 0010 ss: 0018 ds: 0000 es: 0000
fs: 0000 @ 0000000000000000
gs: 0000 @ ffff88007be20000/0000000000000000
Code (instr addr ffffffff810d8da3)
10 83 c0 01 89 4b 24 89 43 20 41 83 fd 04 74 32 44 89 e8 f3 90 <44> 8b 6b 20 41 39 c5 74 ec 41 83
Stack:
ffff88007abc00c0 ffff88007a54bc98 ffff88007abc00c0 ffff88007be2eea0
ffffffff810d8d20 ffff88007a54bd28 ffff88007abc3eb8 ffffffff810d8a50
000000017abc3e18 ffff88007abc3fd8 ffff88007abc00c0 ffff88007be2eea8
0000000000000000 0000000000000000 ffff88007be33fc0 0000000000000000
ffffffff819032a0 0000000000000000 ffff88007abc3ea8 ffffffff8159161d
Call Trace:
[<ffffffff810d8da3>] <--
ffff88007abc3da8: [<ffffffff810d8d20>]
ffff88007abc3dc0: [<ffffffff810d8a50>]
ffff88007abc3e08: [<ffffffff819032a0>]
ffff88007abc3e20: [<ffffffff8159161d>]
ffff88007abc3ea8: [<ffffffff810d8960>]
ffff88007abc3ec0: [<ffffffff81082a6e>]
ffff88007abc3f28: [<ffffffff810829a0>]
ffff88007abc3f50: [<ffffffff8159b66c>]
ffff88007abc3f80: [<ffffffff810829a0>]
ffff88007abc3fd0: [<ffffffffffffffff>]
vcpu2:
rip: ffffffff810d8da1
flags: 00000097 s nz a p c
rsp: ffff88007a401d88
rax: 0000000000000003 rcx: 0000000000000004 rdx: 0000000000000000
rbx: ffff88007a54bd28 rsi: ffff88007abfe200 rdi: ffff88007a54bd28
rbp: ffff88007a401db8 r8: ffff88007be4eec0 r9: 000000444d5210c0
r10: 0000000000000000 r11: 0000000000000010 r12: 0000000000000296
r13: 0000000000000003 r14: ffff88007a54bd4c r15: ffff88007a54bd00
cs: 0010 ss: 0018 ds: 0000 es: 0000
fs: 0000 @ 0000000000000000
gs: 0000 @ ffff88007be40000/0000000000000000
Code (instr addr ffffffff810d8da1)
8b 4b 10 83 c0 01 89 4b 24 89 43 20 41 83 fd 04 74 32 44 89 e8 <f3> 90 44 8b 6b 20 41 39 c5 74 ec
Stack:
ffff88007abfe200 ffff88007a54bc98 ffff88007abfe200 ffff88007be4eea0
ffffffff810d8d20 ffff88007a54bd28 ffff88007a401eb8 ffffffff810d8a50
000000027a401e18 ffff88007a401fd8 ffff88007abfe200 ffff88007be4eea8
0000000000000000 0000000000000000 ffff88007be53fc0 0000000000000000
ffffffff819032a0 0000000000000000 ffff88007a401ea8 ffffffff8159161d
Call Trace:
[<ffffffff810d8da1>] <--
ffff88007a401da8: [<ffffffff810d8d20>]
ffff88007a401dc0: [<ffffffff810d8a50>]
ffff88007a401e08: [<ffffffff819032a0>]
ffff88007a401e20: [<ffffffff8159161d>]
ffff88007a401ea8: [<ffffffff810d8960>]
ffff88007a401ec0: [<ffffffff81082a6e>]
ffff88007a401f28: [<ffffffff810829a0>]
ffff88007a401f50: [<ffffffff8159b66c>]
ffff88007a401f80: [<ffffffff810829a0>]
ffff88007a401fd0: [<ffffffffffffffff>]
vcpu3:
rip: ffffffff810d8da3
flags: 00000097 s nz a p c
rsp: ffff88007a45dd88
rax: 0000000000000003 rcx: dead000000200200 rdx: 0000000000000000
rbx: ffff88007a54bd28 rsi: ffff88007a45a340 rdi: ffff88007a54bd28
rbp: ffff88007a45ddb8 r8: ffff88007be6eec0 r9: 0000000000000001
r10: 0000000000000001 r11: 0000000000000000 r12: 0000000000000296
r13: 0000000000000003 r14: ffff88007a54bd4c r15: ffff88007a54bd00
cs: 0010 ss: 0018 ds: 0000 es: 0000
fs: 0000 @ 0000000000000000
gs: 0000 @ ffff88007be60000/0000000000000000
Code (instr addr ffffffff810d8da3)
10 83 c0 01 89 4b 24 89 43 20 41 83 fd 04 74 32 44 89 e8 f3 90 <44> 8b 6b 20 41 39 c5 74 ec 41 83
Stack:
ffff88007a45a340 ffff88007a54bc98 ffff88007a45a340 ffff88007be6eea0
ffffffff810d8d20 ffff88007a54bd28 ffff88007a45deb8 ffffffff810d8a50
000000037a45de18 ffff88007a45dfd8 ffff88007a45a340 ffff88007be6eea8
0000000000000000 0000000000000000 ffff88007be73fc0 0000000000000000
ffffffff819032a0 0000000000000000 ffff88007a45dea8 ffffffff8159161d
Call Trace:
[<ffffffff810d8da3>] <--
ffff88007a45dda8: [<ffffffff810d8d20>]
ffff88007a45ddc0: [<ffffffff810d8a50>]
ffff88007a45de08: [<ffffffff819032a0>]
ffff88007a45de20: [<ffffffff8159161d>]
ffff88007a45dea8: [<ffffffff810d8960>]
ffff88007a45dec0: [<ffffffff81082a6e>]
ffff88007a45df28: [<ffffffff810829a0>]
ffff88007a45df50: [<ffffffff8159b66c>]
ffff88007a45df80: [<ffffffff810829a0>]
ffff88007a45dfd0: [<ffffffffffffffff>]
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [libvirt] libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose).
2015-04-08 14:47 [libvirt] libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose) Konrad Rzeszutek Wilk
@ 2015-04-08 15:45 ` Andrew Cooper
2015-04-10 15:44 ` Konrad Rzeszutek Wilk
2015-04-14 17:31 ` [libvirt] [PATCH 0/2] " Ian Jackson
1 sibling, 1 reply; 13+ messages in thread
From: Andrew Cooper @ 2015-04-08 15:45 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, jfehlig, xen-devel, ian.jackson; +Cc: libvir-list
On 08/04/15 15:47, Konrad Rzeszutek Wilk wrote:
> Hey Jim, Andrew, and Ian,
>
> This is libvirt v1.2.14 + three patches:
> c82a59b libxl: drop virDomainObj lock when destroying a domain
> a1c9d30 libxl: acquire a job when destroying a domain
> 5bd5406 libxl: Move job acquisition in libxlDomainStart to callers
>
> For fun I've set up an guest with PCI passthrough and tried to save it
> (HAHAH) with an disastrous result (xc_save_helper was stuck). Probably
> due to outstanding DMA operations wreaking havoc.
Outstanding DMA wont make any difference. It isn't (and can't) be
reflected in the logdirty bitmap, so libxc simply wont know about it.
xc_save_helper is blocked because it has called back into the libxl with
the suspend_and_state() callback.
i.e. libxc has requested that libxl pause the domain, and that request
is still outstanding.
The vcpu trace from the very bottom shows that the guest has not yet
paused itself. 1 vcpu is blocked in the hypervisor while the other look
to be in some spinlock code.
~Andrew
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [libvirt] libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose).
2015-04-08 15:45 ` Andrew Cooper
@ 2015-04-10 15:44 ` Konrad Rzeszutek Wilk
2015-04-10 16:05 ` Andrew Cooper
0 siblings, 1 reply; 13+ messages in thread
From: Konrad Rzeszutek Wilk @ 2015-04-10 15:44 UTC (permalink / raw)
To: Andrew Cooper; +Cc: libvir-list, xen-devel, ian.jackson
On Wed, Apr 08, 2015 at 04:45:24PM +0100, Andrew Cooper wrote:
> On 08/04/15 15:47, Konrad Rzeszutek Wilk wrote:
> > Hey Jim, Andrew, and Ian,
> >
> > This is libvirt v1.2.14 + three patches:
> > c82a59b libxl: drop virDomainObj lock when destroying a domain
> > a1c9d30 libxl: acquire a job when destroying a domain
> > 5bd5406 libxl: Move job acquisition in libxlDomainStart to callers
> >
> > For fun I've set up an guest with PCI passthrough and tried to save it
> > (HAHAH) with an disastrous result (xc_save_helper was stuck). Probably
> > due to outstanding DMA operations wreaking havoc.
>
> Outstanding DMA wont make any difference. It isn't (and can't) be
> reflected in the logdirty bitmap, so libxc simply wont know about it.
>
> xc_save_helper is blocked because it has called back into the libxl with
> the suspend_and_state() callback.
>
> i.e. libxc has requested that libxl pause the domain, and that request
> is still outstanding.
>
>
> The vcpu trace from the very bottom shows that the guest has not yet
> paused itself. 1 vcpu is blocked in the hypervisor while the other look
> to be in some spinlock code.
Except the guest is in '---ss- ' so it _should_ be paused by now.
>
> ~Andrew
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose).
2015-04-10 15:44 ` Konrad Rzeszutek Wilk
@ 2015-04-10 16:05 ` Andrew Cooper
0 siblings, 0 replies; 13+ messages in thread
From: Andrew Cooper @ 2015-04-10 16:05 UTC (permalink / raw)
To: xen-devel
On 10/04/15 16:44, Konrad Rzeszutek Wilk wrote:
> On Wed, Apr 08, 2015 at 04:45:24PM +0100, Andrew Cooper wrote:
>> On 08/04/15 15:47, Konrad Rzeszutek Wilk wrote:
>>> Hey Jim, Andrew, and Ian,
>>>
>>> This is libvirt v1.2.14 + three patches:
>>> c82a59b libxl: drop virDomainObj lock when destroying a domain
>>> a1c9d30 libxl: acquire a job when destroying a domain
>>> 5bd5406 libxl: Move job acquisition in libxlDomainStart to callers
>>>
>>> For fun I've set up an guest with PCI passthrough and tried to save it
>>> (HAHAH) with an disastrous result (xc_save_helper was stuck). Probably
>>> due to outstanding DMA operations wreaking havoc.
>> Outstanding DMA wont make any difference. It isn't (and can't) be
>> reflected in the logdirty bitmap, so libxc simply wont know about it.
>>
>> xc_save_helper is blocked because it has called back into the libxl with
>> the suspend_and_state() callback.
>>
>> i.e. libxc has requested that libxl pause the domain, and that request
>> is still outstanding.
>>
>>
>> The vcpu trace from the very bottom shows that the guest has not yet
>> paused itself. 1 vcpu is blocked in the hypervisor while the other look
>> to be in some spinlock code.
> Except the guest is in '---ss- ' so it _should_ be paused by now.
You cannot trust this line for an HVM guest. It simply means that the
toolstack has performed the remote_shutdown hypercall, not that the
guest has finally stopped.
~Andrew
^ permalink raw reply [flat|nested] 13+ messages in thread
* [libvirt] [PATCH 0/2] Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose).
2015-04-08 14:47 [libvirt] libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose) Konrad Rzeszutek Wilk
2015-04-08 15:45 ` Andrew Cooper
@ 2015-04-14 17:31 ` Ian Jackson
2015-04-14 17:31 ` [PATCH 1/2] libxl: fd events: Document spurious callbacks, break out libxl__ev_fd_recheck Ian Jackson
2015-04-16 16:44 ` [libvirt] [PATCH 0/2] Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose) Jim Fehlig
1 sibling, 2 replies; 13+ messages in thread
From: Ian Jackson @ 2015-04-14 17:31 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk; +Cc: libvir-list, andrew.cooper3, xen-devel
Konrad Rzeszutek Wilk writes ("libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose)."):
> It looks like thread #10 is blocking in libxl_read_exactly waiting
> for 'libxl-save-helper'. Said application (see below) has dispatched
> an message through helper_getreply and is blocking on __read_nocancel.
This is not supposed to block.
helper_stdout_readable assumes that the fd is actually readable.
However, for complicated reasons it can happen in a multithreaded
program that the fd was _reviously_ readable and is now no longer.
This was not clearly documented in the internal API documentation.
I have produced what I think are two patches that will fix this. I
have compiled them but I haven't tested them. Konrad, are you able to
check whether they fix your bug ?
If they do they are candidates for backporting.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 1/2] libxl: fd events: Document spurious callbacks, break out libxl__ev_fd_recheck
2015-04-14 17:31 ` [libvirt] [PATCH 0/2] " Ian Jackson
@ 2015-04-14 17:31 ` Ian Jackson
2015-04-14 17:31 ` [PATCH 2/2] libxl: save helper: Recheck fd events Ian Jackson
2015-04-16 13:01 ` [PATCH 1/2] libxl: fd events: Document spurious callbacks, break out libxl__ev_fd_recheck Ian Campbell
2015-04-16 16:44 ` [libvirt] [PATCH 0/2] Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose) Jim Fehlig
1 sibling, 2 replies; 13+ messages in thread
From: Ian Jackson @ 2015-04-14 17:31 UTC (permalink / raw)
To: xen-devel; +Cc: jfehlig, Ian Jackson
No functional change, other than to debug and error message output.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
tools/libxl/libxl_event.c | 39 +++++++++++++++++++++++----------------
tools/libxl/libxl_internal.h | 8 ++++++++
2 files changed, 31 insertions(+), 16 deletions(-)
diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 595da2b..9ede135 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -236,6 +236,25 @@ void libxl__ev_fd_deregister(libxl__gc *gc, libxl__ev_fd *ev)
CTX_UNLOCK;
}
+short libxl__ev_fd_recheck(libxl__egc *egc, libxl__ev_fd *ev) {
+ struct pollfd recheck;
+ int r;
+
+ recheck.fd = ev->fd;
+ recheck.events = ev->events;
+ recheck.revents = 0;
+ r = poll(&recheck, 1, 0);
+ DBG("ev_fd=%p recheck fd=%d r=%d revents=%#x", ev, ev->fd,
+ r, recheck.revents);
+ if (r < 0) {
+ LIBXL__EVENT_DISASTER(egc, "unexpected failure rechecking fd",
+ errno, 0);
+ return 0;
+ }
+ assert(!!r == !!recheck.revents);
+ return recheck.revents;
+}
+
/*
* timeouts
*/
@@ -661,9 +680,8 @@ static void evtchn_fd_callback(libxl__egc *egc, libxl__ev_fd *ev,
{
EGC_GC;
libxl__ev_evtchn *evev;
- int r, rc;
+ int rc;
evtchn_port_or_error_t port;
- struct pollfd recheck;
rc = evtchn_revents_check(egc, revents);
if (rc) return;
@@ -674,21 +692,10 @@ static void evtchn_fd_callback(libxl__egc *egc, libxl__ev_fd *ev,
* held continuously since someone noticed the fd. Normally
* this wouldn't be a problem but evtchn devices don't always
* honour O_NONBLOCK (see xenctrl.h). */
-
- recheck.fd = fd;
- recheck.events = POLLIN;
- recheck.revents = 0;
- r = poll(&recheck, 1, 0);
- DBG("ev_evtchn recheck r=%d revents=%#x", r, recheck.revents);
- if (r < 0) {
- LIBXL__EVENT_DISASTER(egc,
- "unexpected failure polling event channel fd for recheck",
- errno, 0);
- return;
- }
- if (r == 0)
+ revents = libxl__ev_fd_recheck(egc,ev);
+ if (!revents)
break;
- rc = evtchn_revents_check(egc, recheck.revents);
+ rc = evtchn_revents_check(egc, revents);
if (rc) return;
/* OK, that's that workaround done. We can actually check for
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 9c22309..d3a5fba 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -175,6 +175,9 @@ typedef void libxl__ev_fd_callback(libxl__egc *egc, libxl__ev_fd *ev,
* even if only POLLIN was set in events. (POLLNVAL is a fatal
* error and will cause libxl event machinery to fail an assertion.)
*
+ * Note that spurious callbacks are possible. If this is a problem,
+ * use libxl__ev_fd_recheck;
+ *
* It is not permitted to listen for the same or overlapping events
* on the same fd using multiple different libxl__ev_fd's.
*/
@@ -788,6 +791,11 @@ static inline void libxl__ev_fd_init(libxl__ev_fd *efd)
static inline int libxl__ev_fd_isregistered(const libxl__ev_fd *efd)
{ return efd->fd >= 0; }
+/* Calls poll() again - useful to check whether this was a spurious
+ * wakeup. Cannot fail. Returns currently-true revents. */
+short libxl__ev_fd_recheck(libxl__egc *egc, libxl__ev_fd *ev);
+
+
_hidden int libxl__ev_time_register_rel(libxl__gc*, libxl__ev_time *ev_out,
libxl__ev_time_callback*,
int milliseconds /* as for poll(2) */);
--
1.7.10.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 2/2] libxl: save helper: Recheck fd events
2015-04-14 17:31 ` [PATCH 1/2] libxl: fd events: Document spurious callbacks, break out libxl__ev_fd_recheck Ian Jackson
@ 2015-04-14 17:31 ` Ian Jackson
2015-04-16 13:02 ` Ian Campbell
2015-04-16 13:01 ` [PATCH 1/2] libxl: fd events: Document spurious callbacks, break out libxl__ev_fd_recheck Ian Campbell
1 sibling, 1 reply; 13+ messages in thread
From: Ian Jackson @ 2015-04-14 17:31 UTC (permalink / raw)
To: xen-devel; +Cc: jfehlig, Ian Jackson
The save helper message reader does operates with the fd in blocking
mode. So spurious wakeups could cause it to block, unless it takes
precautions.
Reported-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
tools/libxl/libxl_save_callout.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 40b25e4..0f392ef 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -265,6 +265,8 @@ static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
STATE_AO_GC(shs->ao);
int rc, errnoval;
+ revents = libxl__ev_fd_recheck(egc, ev);
+
if (revents & (POLLERR|POLLPRI)) {
LOG(ERROR, "%s signaled POLLERR|POLLPRI (%#x)",
shs->stdout_what, revents);
--
1.7.10.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 1/2] libxl: fd events: Document spurious callbacks, break out libxl__ev_fd_recheck
2015-04-14 17:31 ` [PATCH 1/2] libxl: fd events: Document spurious callbacks, break out libxl__ev_fd_recheck Ian Jackson
2015-04-14 17:31 ` [PATCH 2/2] libxl: save helper: Recheck fd events Ian Jackson
@ 2015-04-16 13:01 ` Ian Campbell
1 sibling, 0 replies; 13+ messages in thread
From: Ian Campbell @ 2015-04-16 13:01 UTC (permalink / raw)
To: Ian Jackson; +Cc: jfehlig, xen-devel
On Tue, 2015-04-14 at 18:31 +0100, Ian Jackson wrote:
> No functional change, other than to debug and error message output.
>
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/2] libxl: save helper: Recheck fd events
2015-04-14 17:31 ` [PATCH 2/2] libxl: save helper: Recheck fd events Ian Jackson
@ 2015-04-16 13:02 ` Ian Campbell
2015-04-16 14:23 ` Ian Jackson
0 siblings, 1 reply; 13+ messages in thread
From: Ian Campbell @ 2015-04-16 13:02 UTC (permalink / raw)
To: Ian Jackson; +Cc: jfehlig, xen-devel
On Tue, 2015-04-14 at 18:31 +0100, Ian Jackson wrote:
> The save helper message reader does operates with the fd in blocking
s/does //?
> mode. So spurious wakeups could cause it to block, unless it takes
> precautions.
>
> Reported-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
I think that makes both of them, but based on 0/2 I think we are
awaiting a Tested-by from Konrad (who wasn't CC-d I've done so now).
> ---
> tools/libxl/libxl_save_callout.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
> index 40b25e4..0f392ef 100644
> --- a/tools/libxl/libxl_save_callout.c
> +++ b/tools/libxl/libxl_save_callout.c
> @@ -265,6 +265,8 @@ static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
> STATE_AO_GC(shs->ao);
> int rc, errnoval;
>
> + revents = libxl__ev_fd_recheck(egc, ev);
> +
> if (revents & (POLLERR|POLLPRI)) {
> LOG(ERROR, "%s signaled POLLERR|POLLPRI (%#x)",
> shs->stdout_what, revents);
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/2] libxl: save helper: Recheck fd events
2015-04-16 13:02 ` Ian Campbell
@ 2015-04-16 14:23 ` Ian Jackson
0 siblings, 0 replies; 13+ messages in thread
From: Ian Jackson @ 2015-04-16 14:23 UTC (permalink / raw)
To: Ian Campbell; +Cc: jfehlig, xen-devel
Ian Campbell writes ("Re: [Xen-devel] [PATCH 2/2] libxl: save helper: Recheck fd events"):
> On Tue, 2015-04-14 at 18:31 +0100, Ian Jackson wrote:
> > The save helper message reader does operates with the fd in blocking
>
> s/does //?
Oops, fixed.
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
>
> I think that makes both of them, but based on 0/2 I think we are
> awaiting a Tested-by from Konrad (who wasn't CC-d I've done so now).
Thanks. Konrad was CC'd on the 0/2, but you weren't, so I have just
bounced it to you.
Having slept on this I wonder if it might be better to always do this
recheck in the event core before making the fd callback. The current
semantics are a bit of a gotcha.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [libvirt] [PATCH 0/2] Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose).
2015-04-14 17:31 ` [libvirt] [PATCH 0/2] " Ian Jackson
2015-04-14 17:31 ` [PATCH 1/2] libxl: fd events: Document spurious callbacks, break out libxl__ev_fd_recheck Ian Jackson
@ 2015-04-16 16:44 ` Jim Fehlig
2015-04-16 17:18 ` Ian Jackson
1 sibling, 1 reply; 13+ messages in thread
From: Jim Fehlig @ 2015-04-16 16:44 UTC (permalink / raw)
To: Ian Jackson, Konrad Rzeszutek Wilk
Cc: libvir-list, andrew.cooper3, xen-devel, Ian Campbell
On 04/14/2015 11:31 AM, Ian Jackson wrote:
> Konrad Rzeszutek Wilk writes ("libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose)."):
>> It looks like thread #10 is blocking in libxl_read_exactly waiting
>> for 'libxl-save-helper'. Said application (see below) has dispatched
>> an message through helper_getreply and is blocking on __read_nocancel.
> This is not supposed to block.
>
> helper_stdout_readable assumes that the fd is actually readable.
> However, for complicated reasons it can happen in a multithreaded
> program that the fd was _reviously_ readable and is now no longer.
>
> This was not clearly documented in the internal API documentation.
>
> I have produced what I think are two patches that will fix this. I
> have compiled them but I haven't tested them. Konrad, are you able to
> check whether they fix your bug ?
I too saw this bug just before Konrad's report, but the patches don't seem to
help. Running a script that continually saves and restores domains will
eventually lock libvirtd with essentially the same traces reported by Konrad
Thread 4 (Thread 0x7fffee3a0700 (LWP 39068)):
#0 0x00007ffff3a9aa9d in read () from /lib64/libpthread.so.0
#1 0x00007ffff4540ea0 in libxl_read_exactly (ctx=0x7fffe00445e0, fd=37,
data=0x7fffee39f36e,
sz=2, source=0x7fffc80010c0 "domain 6 save/restore helper stdout pipe",
what=0x7ffff458112a "ipc msg header") at libxl_utils.c:430
#2 0x00007ffff454913a in helper_stdout_readable (egc=0x7fffee39f540,
ev=0x7fffc8002038, fd=37,
events=3, revents=1) at libxl_save_callout.c:281
#3 0x00007ffff454fafb in afterpoll_internal (egc=0x7fffee39f540,
poller=0x7fffe0000a00, nfds=4,
fds=0x7fffe0000930, now=...) at libxl_event.c:1185
#4 0x00007ffff455127a in eventloop_iteration (egc=0x7fffee39f540,
poller=0x7fffe0000a00)
at libxl_event.c:1645
#5 0x00007ffff4551df1 in libxl__ao_inprogress (ao=0x7fffc8001060,
file=0x7ffff4575e1b "libxl.c",
line=982, func=0x7ffff4578750 <__func__.17561> "libxl_domain_suspend") at
libxl_event.c:1896
#6 0x00007ffff450e051 in libxl_domain_suspend (ctx=0x7fffe00445e0, domid=6,
fd=29, flags=0,
ao_how=0x0) at libxl.c:982
#7 0x00007fffe8774636 in libxlDoDomainSave (driver=0x7fffe011f1c0,
vm=0x7fffe004f950,
to=0x7fffc8000990 "/tmp/sles12gm-pv.img") at libxl/libxl_driver.c:1584
#8 0x00007fffe8774a35 in libxlDomainSaveFlags (dom=0x7fffc8000de0,
to=0x7fffc8000990 "/tmp/sles12gm-pv.img", dxml=0x0, flags=0) at
libxl/libxl_driver.c:1653
#9 0x00007fffe8774b11 in libxlDomainSave (dom=0x7fffc8000de0,
to=0x7fffc8000990 "/tmp/sles12gm-pv.img") at libxl/libxl_driver.c:1678
#10 0x00007ffff751db15 in virDomainSave (domain=0x7fffc8000de0,
to=0x7fffc80009d0 "/tmp/sles12gm-pv.img") at libvirt-domain.c:839
...
Thread 1 (Thread 0x7ffff7fc18c0 (LWP 39059)):
#0 0x00007ffff3a9a7bc in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007ffff3a964a4 in _L_lock_952 () from /lib64/libpthread.so.0
#2 0x00007ffff3a96306 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007ffff454caf6 in libxl__ctx_lock (ctx=0x7fffe00445e0) at
libxl_internal.h:3268
#4 0x00007ffff454fe98 in libxl_osevent_occurred_fd (ctx=0x7fffe00445e0,
for_libxl=0x7fffe004f210, fd=32, events_ign=0, revents_ign=1) at
libxl_event.c:1242
#5 0x00007fffe8770573 in libxlFDEventCallback (watch=24, fd=32, vir_events=1,
fd_info=0x555555896c60) at libxl/libxl_driver.c:123
#6 0x00007ffff73f71bc in virEventPollDispatchHandles (nfds=14, fds=0x555555897fa0)
at util/vireventpoll.c:508
#7 0x00007ffff73f79f9 in virEventPollRunOnce () at util/vireventpoll.c:657
#8 0x00007ffff73f58fa in virEventRunDefaultImpl () at util/virevent.c:308
#9 0x00005555555c2131 in virNetServerRun (srv=0x555555889980) at
rpc/virnetserver.c:1139
#10 0x000055555556cf88 in main (argc=2, argv=0x7fffffffe378) at libvirtd.c:1489
Regards,
Jim
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [libvirt] [PATCH 0/2] Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose).
2015-04-16 16:44 ` [libvirt] [PATCH 0/2] Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose) Jim Fehlig
@ 2015-04-16 17:18 ` Ian Jackson
2015-04-17 8:58 ` Ian Campbell
0 siblings, 1 reply; 13+ messages in thread
From: Ian Jackson @ 2015-04-16 17:18 UTC (permalink / raw)
To: Jim Fehlig
Cc: libvir-list, andrew.cooper3, xen-devel, Ian Campbell,
Konrad Rzeszutek Wilk
Jim Fehlig writes ("Re: [PATCH 0/2] Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose)."):
> On 04/14/2015 11:31 AM, Ian Jackson wrote:
> > I have produced what I think are two patches that will fix this. I
> > have compiled them but I haven't tested them. Konrad, are you able to
> > check whether they fix your bug ?
>
> I too saw this bug just before Konrad's report, but the patches don't seem to
> help. Running a script that continually saves and restores domains will
> eventually lock libvirtd with essentially the same traces reported by Konrad
I'm a total idiot. I do the recheck but I don't pay any attention to
the result.
I will send an updated approach which does this centrally.
Ian.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [libvirt] [PATCH 0/2] Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose).
2015-04-16 17:18 ` Ian Jackson
@ 2015-04-17 8:58 ` Ian Campbell
0 siblings, 0 replies; 13+ messages in thread
From: Ian Campbell @ 2015-04-17 8:58 UTC (permalink / raw)
To: Ian Jackson; +Cc: libvir-list, andrew.cooper3, xen-devel, Konrad Rzeszutek Wilk
On Thu, 2015-04-16 at 18:18 +0100, Ian Jackson wrote:
> Jim Fehlig writes ("Re: [PATCH 0/2] Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose)."):
> > On 04/14/2015 11:31 AM, Ian Jackson wrote:
> > > I have produced what I think are two patches that will fix this. I
> > > have compiled them but I haven't tested them. Konrad, are you able to
> > > check whether they fix your bug ?
> >
> > I too saw this bug just before Konrad's report, but the patches don't seem to
> > help. Running a script that continually saves and restores domains will
> > eventually lock libvirtd with essentially the same traces reported by Konrad
>
> I'm a total idiot. I do the recheck but I don't pay any attention to
> the result.
Your second patch was updating revents which was used in the next if,
what have I missed?
> I will send an updated approach which does this centrally.
Ack, that's probably best anyhow.
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2015-04-17 8:58 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-04-08 14:47 [libvirt] libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose) Konrad Rzeszutek Wilk
2015-04-08 15:45 ` Andrew Cooper
2015-04-10 15:44 ` Konrad Rzeszutek Wilk
2015-04-10 16:05 ` Andrew Cooper
2015-04-14 17:31 ` [libvirt] [PATCH 0/2] " Ian Jackson
2015-04-14 17:31 ` [PATCH 1/2] libxl: fd events: Document spurious callbacks, break out libxl__ev_fd_recheck Ian Jackson
2015-04-14 17:31 ` [PATCH 2/2] libxl: save helper: Recheck fd events Ian Jackson
2015-04-16 13:02 ` Ian Campbell
2015-04-16 14:23 ` Ian Jackson
2015-04-16 13:01 ` [PATCH 1/2] libxl: fd events: Document spurious callbacks, break out libxl__ev_fd_recheck Ian Campbell
2015-04-16 16:44 ` [libvirt] [PATCH 0/2] Re: libvirtd live-locking on CTX_LOCK when doing 'virsh <domid> save /tmp/blah' with guest corrupting memory (on purpose) Jim Fehlig
2015-04-16 17:18 ` Ian Jackson
2015-04-17 8:58 ` Ian Campbell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.