xend segfaults when starting

xend segfaults when starting

[Christoph Egger]

Hi!

xend causes python to segfault on startup.
The changeset in error is: 21904:6a0dd2c29999

Core was generated by `python2.5'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f7ffca981de in _malloc_prefork () from /usr/lib/libc.so.12
(gdb) bt
#0  0x00007f7ffca981de in _malloc_prefork () from /usr/lib/libc.so.12
#1  0x00007f7ffca98473 in free () from /usr/lib/libc.so.12
#2  0x00007f7ffac0db52 in _xc_clean_hcall_buf (m=<value optimized out>)
    at xc_private.c:226
#3  0x00007f7ffac0e955 in xc_interface_close (xch=0x7f7ff9f47000)
    at xc_private.c:240
#4  0x00007f7ffb0054f9 in PyXc_dealloc (self=0x7f7ffd114378)
    at xen/lowlevel/xc/xc.c:2967
#5  0x00007f7ffd9a0923 in PyEval_EvalFrameEx ()
   from /usr/pkg/lib/libpython2.5.so.1.0
#6  0x00007f7ffd9a401d in PyEval_EvalFrameEx ()
   from /usr/pkg/lib/libpython2.5.so.1.0
#7  0x00007f7ffd9a498f in PyEval_EvalCodeEx () 
from /usr/pkg/lib/libpython2.5.so.1.0
#8  0x00007f7ffd9a3cb9 in PyEval_EvalFrameEx ()
   from /usr/pkg/lib/libpython2.5.so.1.0
#9  0x00007f7ffd9a401d in PyEval_EvalFrameEx ()
   from /usr/pkg/lib/libpython2.5.so.1.0
#10 0x00007f7ffd9a498f in PyEval_EvalCodeEx () 
from /usr/pkg/lib/libpython2.5.so.1.0
#11 0x00007f7ffd9a49e0 in PyEval_EvalCode () 
from /usr/pkg/lib/libpython2.5.so.1.0
#12 0x00007f7ffd9bbf0b in run_mod () from /usr/pkg/lib/libpython2.5.so.1.0
#13 0x00007f7ffd9bbfbb in PyRun_FileExFlags () 
from /usr/pkg/lib/libpython2.5.so.1.0
#14 0x00007f7ffd9bd30c in PyRun_SimpleFileExFlags ()
   from /usr/pkg/lib/libpython2.5.so.1.0
#15 0x00007f7ffd9c5816 in Py_Main () from /usr/pkg/lib/libpython2.5.so.1.0
#16 0x0000000000400884 in ___start ()
#17 0x0000000000000003 in ?? ()
#18 0x00007f7ffffffdc8 in ?? ()
#19 0x00007f7ffffffddf in ?? ()
#20 0x00007f7ffffffdfa in ?? ()
#21 0x0000000000000000 in ?? ()


Christoph


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: xend segfaults when starting

[Gianni Tedesco]

On Wed, 2010-08-04 at 14:12 +0100, Christoph Egger wrote:
> Hi!
> 
> xend causes python to segfault on startup.
> The changeset in error is: 21904:6a0dd2c29999

There is a fix for this in:

[PATCH 1 of 6] xl: PCI code cleanups

However the python wrapper ought not segfault because of an unexpected
error code in either case

Stefano, please apply that, this has cropped up several times for people
now, typically as a spurious error message in xl.

Re: xend segfaults when starting

[Christoph Egger]

On Wednesday 04 August 2010 15:27:02 Gianni Tedesco wrote:
> On Wed, 2010-08-04 at 14:12 +0100, Christoph Egger wrote:
> > Hi!
> >
> > xend causes python to segfault on startup.
> > The changeset in error is: 21904:6a0dd2c29999
>
> There is a fix for this in:
>
> [PATCH 1 of 6] xl: PCI code cleanups
>
> However the python wrapper ought not segfault because of an unexpected
> error code in either case
>
> Stefano, please apply that, this has cropped up several times for people
> now, typically as a spurious error message in xl.

How does this affect the python wrapper code ?

Christoph

-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: xend segfaults when starting

[Stefano Stabellini]

On Wed, 4 Aug 2010, Christoph Egger wrote:
> On Wednesday 04 August 2010 15:27:02 Gianni Tedesco wrote:
> > On Wed, 2010-08-04 at 14:12 +0100, Christoph Egger wrote:
> > > Hi!
> > >
> > > xend causes python to segfault on startup.
> > > The changeset in error is: 21904:6a0dd2c29999
> >
> > There is a fix for this in:
> >
> > [PATCH 1 of 6] xl: PCI code cleanups
> >
> > However the python wrapper ought not segfault because of an unexpected
> > error code in either case
> >
> > Stefano, please apply that, this has cropped up several times for people
> > now, typically as a spurious error message in xl.
> 
> How does this affect the python wrapper code ?
> 

It doesn't, in fact:

changeset:   21907:6a0dd2c29999
parent:      21904:9f49667fec71
user:        Ian Campbell <ian.campbell@citrix.com>
date:        Fri Jul 30 16:20:48 2010 +0100
summary:     libxc: free thread specific hypercall buffer on xc_interface_close

I am going to revert this and leave it to Ian to fix it properly
(currently on vacation).

Re: xend segfaults when starting

[Gianni Tedesco]

On Wed, 2010-08-04 at 14:27 +0100, Gianni Tedesco wrote:
> On Wed, 2010-08-04 at 14:12 +0100, Christoph Egger wrote:
> > Hi!
> > 
> > xend causes python to segfault on startup.
> > The changeset in error is: 21904:6a0dd2c29999
> 
> There is a fix for this in:
> 
> [PATCH 1 of 6] xl: PCI code cleanups

My mistake, ignore the spam, local changes in what I thought was a
pristine tree. This is unrelated to the afore-mentioned bug.

Of course xend etc. is not using xl yet, so I am living in my own
fantasy :)

Re: xend segfaults when starting

[Ian Campbell]

On Wed, 2010-08-04 at 15:55 +0100, Stefano Stabellini wrote:
> > > On Wed, 2010-08-04 at 14:12 +0100, Christoph Egger wrote:
> > > > Hi!
> > > >
> > > > xend causes python to segfault on startup.
> > > > The changeset in error is: 21904:6a0dd2c29999
> > >

> It doesn't, in fact:
> 
> changeset:   21907:6a0dd2c29999
> parent:      21904:9f49667fec71
> user:        Ian Campbell <ian.campbell@citrix.com>
> date:        Fri Jul 30 16:20:48 2010 +0100
> summary:     libxc: free thread specific hypercall buffer on
> xc_interface_close
> 
> I am going to revert this and leave it to Ian to fix it properly
> (currently on vacation). 

I'm currently looking at this but I'm not seeing this issue, xend starts
up fine and I can start a (PV) VM.

When you said "segfault on startup" did you mean of xend or of a domain?
(I think the former).

Can you give me a little more information about your environment please?
Is it NetBSD by any chance?

Please could you reapply this changeset add some tracing to
hcall_buf_prep and _xc_clean_hcall_buf to print out the hcall_buff and
hcall_buff->buf as they are allocated and freed. The line numbers
indicate that the free(hcall_buf->buf) is faulting. We've just called
unlock_pages on the same address but since we seem to deliberately throw
away any errors from munlock (see "safe_munlock") that doesn't really
tell us much about its validity.

Perhaps this whole area needs looking at with an eye to NetBSD
portability?

Ian.

Re: xend segfaults when starting

[Stefano Stabellini]

On Wed, 11 Aug 2010, Ian Campbell wrote:
> On Wed, 2010-08-04 at 15:55 +0100, Stefano Stabellini wrote:
> > > > On Wed, 2010-08-04 at 14:12 +0100, Christoph Egger wrote:
> > > > > Hi!
> > > > >
> > > > > xend causes python to segfault on startup.
> > > > > The changeset in error is: 21904:6a0dd2c29999
> > > >
> 
> > It doesn't, in fact:
> > 
> > changeset:   21907:6a0dd2c29999
> > parent:      21904:9f49667fec71
> > user:        Ian Campbell <ian.campbell@citrix.com>
> > date:        Fri Jul 30 16:20:48 2010 +0100
> > summary:     libxc: free thread specific hypercall buffer on
> > xc_interface_close
> > 
> > I am going to revert this and leave it to Ian to fix it properly
> > (currently on vacation). 
> 
> I'm currently looking at this but I'm not seeing this issue, xend starts
> up fine and I can start a (PV) VM.
> 
> When you said "segfault on startup" did you mean of xend or of a domain?
> (I think the former).
> 
> Can you give me a little more information about your environment please?
> Is it NetBSD by any chance?
> 
> Please could you reapply this changeset add some tracing to
> hcall_buf_prep and _xc_clean_hcall_buf to print out the hcall_buff and
> hcall_buff->buf as they are allocated and freed. The line numbers
> indicate that the free(hcall_buf->buf) is faulting. We've just called
> unlock_pages on the same address but since we seem to deliberately throw
> away any errors from munlock (see "safe_munlock") that doesn't really
> tell us much about its validity.
> 
> Perhaps this whole area needs looking at with an eye to NetBSD
> portability?
> 
 
I gave few days to Christoph to reply, I'll reapply the patch for now
but if Christoph can come up with a good explanation of the problem I'll
revert it again or fix the bug.

Re: xend segfaults when starting

[Christoph Egger]

On Friday 13 August 2010 13:58:53 Stefano Stabellini wrote:
> On Wed, 11 Aug 2010, Ian Campbell wrote:
> > On Wed, 2010-08-04 at 15:55 +0100, Stefano Stabellini wrote:
> > > > > On Wed, 2010-08-04 at 14:12 +0100, Christoph Egger wrote:
> > > > > > Hi!
> > > > > >
> > > > > > xend causes python to segfault on startup.
> > > > > > The changeset in error is: 21904:6a0dd2c29999
> > >
> > > It doesn't, in fact:
> > >
> > > changeset:   21907:6a0dd2c29999
> > > parent:      21904:9f49667fec71
> > > user:        Ian Campbell <ian.campbell@citrix.com>
> > > date:        Fri Jul 30 16:20:48 2010 +0100
> > > summary:     libxc: free thread specific hypercall buffer on
> > > xc_interface_close
> > >
> > > I am going to revert this and leave it to Ian to fix it properly
> > > (currently on vacation).
> >
> > I'm currently looking at this but I'm not seeing this issue, xend starts
> > up fine and I can start a (PV) VM.
> >
> > When you said "segfault on startup" did you mean of xend or of a domain?
> > (I think the former).
> >
> > Can you give me a little more information about your environment please?
> > Is it NetBSD by any chance?

Yes, it is NetBSD -current.

> > Please could you reapply this changeset add some tracing to
> > hcall_buf_prep and _xc_clean_hcall_buf to print out the hcall_buff and
> > hcall_buff->buf as they are allocated and freed. The line numbers
> > indicate that the free(hcall_buf->buf) is faulting. We've just called
> > unlock_pages on the same address but since we seem to deliberately throw
> > away any errors from munlock (see "safe_munlock") that doesn't really
> > tell us much about its validity.

Will do when I get some time.

> > Perhaps this whole area needs looking at with an eye to NetBSD
> > portability?
>
> I gave few days to Christoph to reply, I'll reapply the patch for now
> but if Christoph can come up with a good explanation of the problem I'll
> revert it again or fix the bug.

I haven't got the opportunity to further analysis. I am pretty busy with
nested virtualization. Sorry.

If you apply it, I'll revert it in my local tree to keep it in working state.
(I already have several local patches to do so, i.e. I have the 
blktap/noblktap changes for libxl in my tree, I need to ping Ian Jackson for 
this again)

Christoph


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: xend segfaults when starting

[Christoph Egger]

On Wednesday 11 August 2010 15:11:54 Ian Campbell wrote:
> On Wed, 2010-08-04 at 15:55 +0100, Stefano Stabellini wrote:
> > > > On Wed, 2010-08-04 at 14:12 +0100, Christoph Egger wrote:
> > > > > Hi!
> > > > >
> > > > > xend causes python to segfault on startup.
> > > > > The changeset in error is: 21904:6a0dd2c29999
> >
> > It doesn't, in fact:
> >
> > changeset:   21907:6a0dd2c29999
> > parent:      21904:9f49667fec71
> > user:        Ian Campbell <ian.campbell@citrix.com>
> > date:        Fri Jul 30 16:20:48 2010 +0100
> > summary:     libxc: free thread specific hypercall buffer on
> > xc_interface_close
> >
> > I am going to revert this and leave it to Ian to fix it properly
> > (currently on vacation).
>
> I'm currently looking at this but I'm not seeing this issue, xend starts
> up fine and I can start a (PV) VM.
>
> When you said "segfault on startup" did you mean of xend or of a domain?
> (I think the former).
>
> Can you give me a little more information about your environment please?
> Is it NetBSD by any chance?
>
> Please could you reapply this changeset add some tracing to
> hcall_buf_prep and _xc_clean_hcall_buf to print out the hcall_buff and
> hcall_buff->buf as they are allocated and freed. The line numbers
> indicate that the free(hcall_buf->buf) is faulting. We've just called
> unlock_pages on the same address but since we seem to deliberately throw
> away any errors from munlock (see "safe_munlock") that doesn't really
> tell us much about its validity.

I tracked down where the error happens. In safe_munlock(),
the munlock() fails.

The trace is:

xc_interface_close -> _xc_clean_hcall_buf -> unlock_pages -> safe_munlock -> 
munlock

hcall_buf->buf has the address 0x7f7ffdfe7040

In unlock_pages, the address and length passed to munlock() is:

 laddr 0x7f7ffdfe7000, llen 0x2000

The reason why munlock() fails is that mlock() hasn't been called before.
The hcall_buf_prep() is not called at all before the first call to 
_xc_clean_hcall_buf().

Christoph


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: xend segfaults when starting

[Ian Campbell]

[-- Attachment #1: Type: text/plain, Size: 3065 bytes --]

Thanks for the analysis. I'm a bit confused though.

On Wed, 2010-08-18 at 11:44 +0100, Christoph Egger wrote:

> I tracked down where the error happens. In safe_munlock(),
> the munlock() fails.
> 
> The trace is:
> 
> xc_interface_close -> _xc_clean_hcall_buf -> unlock_pages -> safe_munlock -> 
> munlock
> 
> hcall_buf->buf has the address 0x7f7ffdfe7040

Mustn't this be page aligned, due to 
        hcall_buf->buf = xc_memalign(PAGE_SIZE, PAGE_SIZE);
?

This appears to turn into valloc on NetBSD which (at least according to
the Linux manpages) returns a page aligned result.

> In unlock_pages, the address and length passed to munlock() is:
> 
>  laddr 0x7f7ffdfe7000, llen 0x2000

> The reason why munlock() fails is that mlock() hasn't been called before.
> The hcall_buf_prep() is not called at all before the first call to 
> _xc_clean_hcall_buf().

If hcall_buf_prep() has never been called then
"pthread_getspecific(hcall_buf_pkey)" should return NULL and
_xc_clean_hcall_buf will never be called from xc_clean_hcall_buf.
_xc_clean_hcall_buf also ignores NULL values itself.

However you say that hcall_buf_pkey is not NULL, but rather contains a
valid hcall_buf containing 0x7f7ffdfe7040. The only call to
"pthread_setspecific(hcall_buf_pkey, ...)" with a non-NULL value is in
hcall_buf_prep(), so it must have been called at some point.

Please can you confirm if _xc_init_hcall_buf() is ever called and what
the behaviour of "pthread_getspecific(hcall_buf_pkey)" is if
_xc_init_hcall_buf() has never been called. I think it is supposed to
return NULL in this case and we certainly rely on that.

pthread_getspecific(hcall_buf_pkey) is supposed to return NULL on error,
however hcall_buf_pkey is uninitialised until _xc_init_hcall_buf,
perhaps on NetBSD the uninitialised value somehow looks valid? It's not
clear what the correct value to initialise a pthread_key_t to in order
for it to appear invalid until it is properly setup is, but I suppose we
should be initialising it before use. Please can you try this patch:

diff -r 7e4d798e8726 tools/libxc/xc_private.c
--- a/tools/libxc/xc_private.c	Wed Aug 18 11:29:35 2010 +0100
+++ b/tools/libxc/xc_private.c	Wed Aug 18 13:12:41 2010 +0100
@@ -232,17 +236,19 @@ static void _xc_clean_hcall_buf(void *m)
     pthread_setspecific(hcall_buf_pkey, NULL);
 }
 
+static void _xc_init_hcall_buf(void)
+{
+    pthread_key_create(&hcall_buf_pkey, _xc_clean_hcall_buf);
+}
+
 static void xc_clean_hcall_buf(void)
 {
     void *hcall_buf = pthread_getspecific(hcall_buf_pkey);
 
+    pthread_once(&hcall_buf_pkey_once, _xc_init_hcall_buf);
+
     if (hcall_buf)
         _xc_clean_hcall_buf(hcall_buf);
-}
-
-static void _xc_init_hcall_buf(void)
-{
-    pthread_key_create(&hcall_buf_pkey, _xc_clean_hcall_buf);
 }
 
 int hcall_buf_prep(void **addr, size_t len)


If that doesn't work perhaps you can reduce the issue to a simple test
case like the attached? (which doesn't reproduce the issue for me on
Linux) If you can do that then please run it with the attached libxc
patch and post the output.

Ian.

[-- Attachment #2: libxc-mlock-debug.patch --]
[-- Type: text/x-patch, Size: 2555 bytes --]

diff -r 7e4d798e8726 tools/libxc/xc_private.c
--- a/tools/libxc/xc_private.c	Wed Aug 18 11:29:35 2010 +0100
+++ b/tools/libxc/xc_private.c	Wed Aug 18 13:07:15 2010 +0100
@@ -195,6 +195,7 @@ int lock_pages(void *addr, size_t len)
       void *laddr = (void *)((unsigned long)addr & PAGE_MASK);
       size_t llen = (len + ((unsigned long)addr - (unsigned long)laddr) +
                      PAGE_SIZE - 1) & PAGE_MASK;
+      printf("%s(%p,%zd) -> mlock(%p,%zd)\n", __func__, addr, len, laddr, llen);
       e = mlock(laddr, llen);
       return e;
 }
@@ -204,6 +205,7 @@ void unlock_pages(void *addr, size_t len
     void *laddr = (void *)((unsigned long)addr & PAGE_MASK);
     size_t llen = (len + ((unsigned long)addr - (unsigned long)laddr) +
                    PAGE_SIZE - 1) & PAGE_MASK;
+    printf("%s(%p,%zd) -> safe_munlock(%p,%zd)\n", __func__, addr, len, laddr, llen);
     safe_munlock(laddr, llen);
 }
 
@@ -217,6 +219,8 @@ static void _xc_clean_hcall_buf(void *m)
 static void _xc_clean_hcall_buf(void *m)
 {
     struct hcall_buf *hcall_buf = m;
+
+    printf("%s(%p) -> %p\n", __func__, m, hcall_buf ? hcall_buf->buf : NULL);
 
     if ( hcall_buf )
     {
@@ -236,6 +240,8 @@ static void xc_clean_hcall_buf(void)
 {
     void *hcall_buf = pthread_getspecific(hcall_buf_pkey);
 
+    printf("%s(%p)\n", __func__, hcall_buf);
+
     if (hcall_buf)
         _xc_clean_hcall_buf(hcall_buf);
 }
@@ -249,6 +255,7 @@ int hcall_buf_prep(void **addr, size_t l
 {
     struct hcall_buf *hcall_buf;
 
+    printf("%s(%p,%zd)\n", __func__, *addr, len);
     pthread_once(&hcall_buf_pkey_once, _xc_init_hcall_buf);
 
     hcall_buf = pthread_getspecific(hcall_buf_pkey);
@@ -257,12 +264,14 @@ int hcall_buf_prep(void **addr, size_t l
         hcall_buf = calloc(1, sizeof(*hcall_buf));
         if ( !hcall_buf )
             goto out;
+        printf("%s hcall buf at %p\n", __func__, hcall_buf);
         pthread_setspecific(hcall_buf_pkey, hcall_buf);
     }
 
     if ( !hcall_buf->buf )
     {
         hcall_buf->buf = xc_memalign(PAGE_SIZE, PAGE_SIZE);
+        printf("%s hcall_buf->buf at %p\n", __func__, hcall_buf->buf);
         if ( !hcall_buf->buf || lock_pages(hcall_buf->buf, PAGE_SIZE) )
         {
             free(hcall_buf->buf);
@@ -286,6 +295,8 @@ void hcall_buf_release(void **addr, size
 void hcall_buf_release(void **addr, size_t len)
 {
     struct hcall_buf *hcall_buf = pthread_getspecific(hcall_buf_pkey);
+
+    printf("%s(%p,%zd)\n", __func__, *addr, len);
 
     if ( hcall_buf && (hcall_buf->buf == *addr) )
     {

[-- Attachment #3: mlock.c --]
[-- Type: text/x-csrc, Size: 748 bytes --]

// gcc -o mlock -Itools/include -Itools/libxc/ -Ltools/libxc/ -lxenctrl mlock.c
#include <stdlib.h>

#include <xenctrl.h>

extern int hcall_buf_prep(void **addr, size_t len);
extern void hcall_buf_release(void **addr, size_t len);

int main(void)
{
	xc_interface *xc = xc_interface_open(NULL, NULL, 0);
	int ret;

	void *mybuf = malloc(0x3000);
	void *mybuf2 = mybuf + 0x40;

	printf("xc interface is %p\n", xc);
	printf("before prep buf is %p / %p\n", mybuf, mybuf2);
	hcall_buf_prep(&mybuf2, 0x200);
	printf("after prep buf is %p / %p\n", mybuf, mybuf2);

	hcall_buf_release(&mybuf2, 0x200);
	printf("after release buf is %p / %p\n", mybuf, mybuf2);

	ret = xc_interface_close(xc);

	printf("xc interface close returned %d\n", ret);
	return 0;
}

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: xend segfaults when starting

[Ian Campbell]

On Wed, 2010-08-18 at 13:14 +0100, Ian Campbell wrote:
> Please can you try this patch:

Ooops, that's not quite right (it still calls pthread_getspecific before
_xc_init_hcall_buf). Please can you try this replacement for the
original patch instead:

diff -r ddbd38da0739 -r d31bfd188fd0 tools/libxc/xc_private.c
--- a/tools/libxc/xc_private.c	Wed Aug 18 13:14:57 2010 +0100
+++ b/tools/libxc/xc_private.c	Wed Aug 18 13:17:48 2010 +0100
@@ -57,6 +57,8 @@ xc_interface *xc_interface_open(xentooll
     return 0;
 }
 
+static void xc_clean_hcall_buf(void);
+
 int xc_interface_close(xc_interface *xch)
 {
     int rc = 0;
@@ -68,6 +70,9 @@ int xc_interface_close(xc_interface *xch
         rc = xc_interface_close_core(xch, xch->fd);
         if (rc) PERROR("Could not close hypervisor interface");
     }
+
+    xc_clean_hcall_buf();
+
     free(xch);
     return rc;
 }
@@ -180,6 +185,8 @@ int hcall_buf_prep(void **addr, size_t l
 int hcall_buf_prep(void **addr, size_t len) { return 0; }
 void hcall_buf_release(void **addr, size_t len) { }
 
+static void xc_clean_hcall_buf(void) { }
+
 #else /* !__sun__ */
 
 int lock_pages(void *addr, size_t len)
@@ -228,6 +235,13 @@ static void _xc_init_hcall_buf(void)
 static void _xc_init_hcall_buf(void)
 {
     pthread_key_create(&hcall_buf_pkey, _xc_clean_hcall_buf);
+}
+
+static void xc_clean_hcall_buf(void)
+{
+    pthread_once(&hcall_buf_pkey_once, _xc_init_hcall_buf);
+
+    _xc_clean_hcall_buf(pthread_getspecific(hcall_buf_pkey));
 }
 
 int hcall_buf_prep(void **addr, size_t len)

Re: xend segfaults when starting

[Christoph Egger]

On Wednesday 18 August 2010 14:14:19 Ian Campbell wrote:
> Thanks for the analysis. I'm a bit confused though.
>
> On Wed, 2010-08-18 at 11:44 +0100, Christoph Egger wrote:
> > I tracked down where the error happens. In safe_munlock(),
> > the munlock() fails.
> >
> > The trace is:
> >
> > xc_interface_close -> _xc_clean_hcall_buf -> unlock_pages -> safe_munlock
> > -> munlock
> >
> > hcall_buf->buf has the address 0x7f7ffdfe7040
>
> Mustn't this be page aligned, due to
>         hcall_buf->buf = xc_memalign(PAGE_SIZE, PAGE_SIZE);
> ?
>
> This appears to turn into valloc on NetBSD which (at least according to
> the Linux manpages) returns a page aligned result.

Yes, correct.

> > In unlock_pages, the address and length passed to munlock() is:
> >
> >  laddr 0x7f7ffdfe7000, llen 0x2000
> >
> > The reason why munlock() fails is that mlock() hasn't been called before.
> > The hcall_buf_prep() is not called at all before the first call to
> > _xc_clean_hcall_buf().
>
> If hcall_buf_prep() has never been called then
> "pthread_getspecific(hcall_buf_pkey)" should return NULL and
> _xc_clean_hcall_buf will never be called from xc_clean_hcall_buf.
> _xc_clean_hcall_buf also ignores NULL values itself.

Who calls hcall_buf_prep() in your case ?

Only hypercalls call hcall_buf_prep().
What if no hypercalls are not called during xend startup ?

If you call xc_clean_hcall_buf() from xc_interface_close()
then you should also call hcall_buf_prep() from xc_interface_open().

> However you say that hcall_buf_pkey is not NULL, but rather contains a
> valid hcall_buf containing 0x7f7ffdfe7040.

hcall_buf itself has the address 0x7f7ffdfe7000.

hcall_buf->buf has the address 0x7f7ffdfe7040.

> The only call to "pthread_setspecific(hcall_buf_pkey, ...)" with a non-NULL
> value is in hcall_buf_prep(), so it must have been called at some point.

In that case, I am puzzled why I don't get the trace.
Something really fishy is going on.

> Please can you confirm if _xc_init_hcall_buf() is ever called and what
> the behaviour of "pthread_getspecific(hcall_buf_pkey)" is if
> _xc_init_hcall_buf() has never been called. I think it is supposed to
> return NULL in this case and we certainly rely on that.

_xc_init_hcall_buf() is not called.  pthread_getspecific() should return NULL
but doesn't.

I am starting to ask myself "How did libxc ever work?". It feels like we are
hunting down a long-term hidden bug.

> pthread_getspecific(hcall_buf_pkey) is supposed to return NULL on error,
> however hcall_buf_pkey is uninitialised until _xc_init_hcall_buf,
> perhaps on NetBSD the uninitialised value somehow looks valid? It's not
> clear what the correct value to initialise a pthread_key_t to in order
> for it to appear invalid until it is properly setup is, but I suppose we
> should be initialising it before use. Please can you try this patch:

I tried the replacement patch from the other mail.
With it, xend does not crash, hcall_buf is NULL,
pthread_getspecific() returns NULL,
and I am not able to start a guest with 'xm'

Xend has probably crashed!  Invalid or missing HTTP status code.

>
> If that doesn't work perhaps you can reduce the issue to a simple test
> case like the attached? (which doesn't reproduce the issue for me on
> Linux) If you can do that then please run it with the attached libxc
> patch and post the output.

xc_interface is 0x7f7ffdb03800
before prep buf is 0x7f7ffdb0b000 / 0x7f7ffdb0b040
after prep buf is 0x7f7ffdb0b000 / 0x7f7ffdb20000
after release buf is 0x7f7ffdb0b000 / 0x7f7ffdb0b040
xc interface close returned 0

No crash. Is this the expected output ?

Christoph

-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: xend segfaults when starting

[Ian Campbell]

On Wed, 2010-08-18 at 15:02 +0100, Christoph Egger wrote:
> On Wednesday 18 August 2010 14:14:19 Ian Campbell wrote:

> > > In unlock_pages, the address and length passed to munlock() is:
> > >
> > >  laddr 0x7f7ffdfe7000, llen 0x2000
> > >
> > > The reason why munlock() fails is that mlock() hasn't been called before.
> > > The hcall_buf_prep() is not called at all before the first call to
> > > _xc_clean_hcall_buf().
> >
> > If hcall_buf_prep() has never been called then
> > "pthread_getspecific(hcall_buf_pkey)" should return NULL and
> > _xc_clean_hcall_buf will never be called from xc_clean_hcall_buf.
> > _xc_clean_hcall_buf also ignores NULL values itself.
> 
> Who calls hcall_buf_prep() in your case ?
> 
> Only hypercalls call hcall_buf_prep().
> What if no hypercalls are not called during xend startup ?

Then I would have expected pthread_getspecific(hcall_buf_pkey) to return
NULL (because _xc_init_hcall_buf was never called) and therefore for
xc_clean_hcall_buf to not doing any unlocking.

However I think my expectation was wrong. If _xc_init_hcall_buf is never
called then hcall_buf_pkey is undefined but not necessarily invalid --
and it seems to be the case on your system that it turns out to be valid
(perhaps pthread_key_t is valid on NetBSD and invalid on Linux or
something like that) and therefore we try an unlock some random address.

My updated patch ensured that hcall_buf_pkey is always initialised
before use.

> If you call xc_clean_hcall_buf() from xc_interface_close()
> then you should also call hcall_buf_prep() from xc_interface_open().
> 
> > However you say that hcall_buf_pkey is not NULL, but rather contains a
> > valid hcall_buf containing 0x7f7ffdfe7040.
> 
> hcall_buf itself has the address 0x7f7ffdfe7000.
> 
> hcall_buf->buf has the address 0x7f7ffdfe7040.

That's very odd -- hcall_buf->buf is allocated with xc_memalign and
therefore should be page aligned. Are you sure the addresses aren't the
other way round?

> > The only call to "pthread_setspecific(hcall_buf_pkey, ...)" with a non-NULL
> > value is in hcall_buf_prep(), so it must have been called at some point.
> 
> In that case, I am puzzled why I don't get the trace.
> Something really fishy is going on.
> 
> > Please can you confirm if _xc_init_hcall_buf() is ever called and what
> > the behaviour of "pthread_getspecific(hcall_buf_pkey)" is if
> > _xc_init_hcall_buf() has never been called. I think it is supposed to
> > return NULL in this case and we certainly rely on that.
> 
> _xc_init_hcall_buf() is not called.  pthread_getspecific() should return NULL
> but doesn't.
> 
> I am starting to ask myself "How did libxc ever work?". It feels like we are
> hunting down a long-term hidden bug.

Previously _xc_clean_hcall_buf would be called IFF hcall_buf_prep had
been called. My patch changed this to also be called on close (even if
hcall_buf_prep was never called) and could therefore access an
uninitialised hcall_buf_pkey.

I am reasonably confident that before my patch libxc was OK.

> > pthread_getspecific(hcall_buf_pkey) is supposed to return NULL on error,
> > however hcall_buf_pkey is uninitialised until _xc_init_hcall_buf,
> > perhaps on NetBSD the uninitialised value somehow looks valid? It's not
> > clear what the correct value to initialise a pthread_key_t to in order
> > for it to appear invalid until it is properly setup is, but I suppose we
> > should be initialising it before use. Please can you try this patch:
> 
> I tried the replacement patch from the other mail.
> With it, xend does not crash, hcall_buf is NULL,
> pthread_getspecific() returns NULL,

OK, I think that suggests that my updated patch does the right thing
here.

> and I am not able to start a guest with 'xm'
> 
> Xend has probably crashed!  Invalid or missing HTTP status code.

There was another HTTP (XML/RPC) related mail on the list this morning
-- is this related to that? Are you sure it is related to the libxc
patch?

(did you by any chance update to python2.7 recently?)

> > If that doesn't work perhaps you can reduce the issue to a simple test
> > case like the attached? (which doesn't reproduce the issue for me on
> > Linux) If you can do that then please run it with the attached libxc
> > patch and post the output.
> 
> xc_interface is 0x7f7ffdb03800
> before prep buf is 0x7f7ffdb0b000 / 0x7f7ffdb0b040
> after prep buf is 0x7f7ffdb0b000 / 0x7f7ffdb20000
> after release buf is 0x7f7ffdb0b000 / 0x7f7ffdb0b040
> xc interface close returned 0
> 
> No crash. Is this the expected output ?

It looks correct but didn't reproduce the crash so is of limited
utility.

Ian.

Re: xend segfaults when starting

[Christoph Egger]

On Wednesday 18 August 2010 16:59:30 Ian Campbell wrote:
> On Wed, 2010-08-18 at 15:02 +0100, Christoph Egger wrote:
> > On Wednesday 18 August 2010 14:14:19 Ian Campbell wrote:
> > > > In unlock_pages, the address and length passed to munlock() is:
> > > >
> > > >  laddr 0x7f7ffdfe7000, llen 0x2000
> > > >
> > > > The reason why munlock() fails is that mlock() hasn't been called
> > > > before. The hcall_buf_prep() is not called at all before the first
> > > > call to _xc_clean_hcall_buf().
> > >
> > > If hcall_buf_prep() has never been called then
> > > "pthread_getspecific(hcall_buf_pkey)" should return NULL and
> > > _xc_clean_hcall_buf will never be called from xc_clean_hcall_buf.
> > > _xc_clean_hcall_buf also ignores NULL values itself.
> >
> > Who calls hcall_buf_prep() in your case ?
> >
> > Only hypercalls call hcall_buf_prep().
> > What if no hypercalls are not called during xend startup ?
>
> Then I would have expected pthread_getspecific(hcall_buf_pkey) to return
> NULL (because _xc_init_hcall_buf was never called) and therefore for
> xc_clean_hcall_buf to not doing any unlocking.
>
> However I think my expectation was wrong. If _xc_init_hcall_buf is never
> called then hcall_buf_pkey is undefined but not necessarily invalid --
> and it seems to be the case on your system that it turns out to be valid
> (perhaps pthread_key_t is valid on NetBSD and invalid on Linux or
> something like that) and therefore we try an unlock some random address.

To make it even more mysterious, the "random" address is always the same
even across machine reboots.

>
> My updated patch ensured that hcall_buf_pkey is always initialised
> before use.

Yes, but we also need to figure out why hcall_buf_prep is never called.
Who calls hcall_buf_prep() on your machine ?
Can you provide a call trace when hcall_buf_prep() is called the first time, 
please ?

> > If you call xc_clean_hcall_buf() from xc_interface_close()
> > then you should also call hcall_buf_prep() from xc_interface_open().
> >
> > > However you say that hcall_buf_pkey is not NULL, but rather contains a
> > > valid hcall_buf containing 0x7f7ffdfe7040.
> >
> > hcall_buf itself has the address 0x7f7ffdfe7000.
> >
> > hcall_buf->buf has the address 0x7f7ffdfe7040.
>
> That's very odd -- hcall_buf->buf is allocated with xc_memalign and
> therefore should be page aligned. Are you sure the addresses aren't the
> other way round?

Yes, I am.

>
> > > The only call to "pthread_setspecific(hcall_buf_pkey, ...)" with a
> > > non-NULL value is in hcall_buf_prep(), so it must have been called at
> > > some point.
> >
> > In that case, I am puzzled why I don't get the trace.
> > Something really fishy is going on.
> >
> > > Please can you confirm if _xc_init_hcall_buf() is ever called and what
> > > the behaviour of "pthread_getspecific(hcall_buf_pkey)" is if
> > > _xc_init_hcall_buf() has never been called. I think it is supposed to
> > > return NULL in this case and we certainly rely on that.
> >
> > _xc_init_hcall_buf() is not called.  pthread_getspecific() should return
> > NULL but doesn't.
> >
> > I am starting to ask myself "How did libxc ever work?". It feels like we
> > are hunting down a long-term hidden bug.
>
> Previously _xc_clean_hcall_buf would be called IFF hcall_buf_prep had
> been called. My patch changed this to also be called on close (even if
> hcall_buf_prep was never called) and could therefore access an
> uninitialised hcall_buf_pkey.

Calling _xc_clean_hcall_buf() unconditionally and hcall_buf_prep()
conditionally sounds to me like calling free() unconditionally
and malloc() conditionally.

I will give calling hcall_buf_prep() from xc_interface_open() a try with your
patch tomorrow.

> I am reasonably confident that before my patch libxc was OK.

And is ok again after it has been backed out. :)

> > > pthread_getspecific(hcall_buf_pkey) is supposed to return NULL on
> > > error, however hcall_buf_pkey is uninitialised until
> > > _xc_init_hcall_buf, perhaps on NetBSD the uninitialised value somehow
> > > looks valid? It's not clear what the correct value to initialise a
> > > pthread_key_t to in order for it to appear invalid until it is properly
> > > setup is, but I suppose we should be initialising it before use. Please
> > > can you try this patch:
> >
> > I tried the replacement patch from the other mail.
> > With it, xend does not crash, hcall_buf is NULL,
> > pthread_getspecific() returns NULL,
>
> OK, I think that suggests that my updated patch does the right thing
> here.

Is it possible that xend can call xc_interface_close() during startup
and hcall_buf_prep() later when xend comes in interaction with xm ?

> > and I am not able to start a guest with 'xm'
> >
> > Xend has probably crashed!  Invalid or missing HTTP status code.
>
> There was another HTTP (XML/RPC) related mail on the list this morning

I saw this mail. No, I don't think it is related to this.

> -- is this related to that? Are you sure it is related to the libxc
> patch?

Yes.

> (did you by any chance update to python2.7 recently?)

No, I am on python 2.5.

> > > If that doesn't work perhaps you can reduce the issue to a simple test
> > > case like the attached? (which doesn't reproduce the issue for me on
> > > Linux) If you can do that then please run it with the attached libxc
> > > patch and post the output.
> >
> > xc_interface is 0x7f7ffdb03800
> > before prep buf is 0x7f7ffdb0b000 / 0x7f7ffdb0b040
> > after prep buf is 0x7f7ffdb0b000 / 0x7f7ffdb20000
> > after release buf is 0x7f7ffdb0b000 / 0x7f7ffdb0b040
> > xc interface close returned 0
> >
> > No crash. Is this the expected output ?
>
> It looks correct but didn't reproduce the crash so is of limited
> utility.
>
> Ian.

Christoph


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: xend segfaults when starting

[Ian Campbell]

On Wed, 2010-08-18 at 17:02 +0100, Christoph Egger wrote:
> 
> > > hcall_buf itself has the address 0x7f7ffdfe7000.
> > >
> > > hcall_buf->buf has the address 0x7f7ffdfe7040.
> >
> > That's very odd -- hcall_buf->buf is allocated with xc_memalign and
> > therefore should be page aligned. Are you sure the addresses aren't
> the
> > other way round?
> 
> Yes, I am. 

Then NetBSD has a bug where it apparently returns non-page aligned
allocations from valloc.

> Calling _xc_clean_hcall_buf() unconditionally and hcall_buf_prep()
> conditionally sounds to me like calling free() unconditionally
> and malloc() conditionally.

Not quite because (in the updated patch) _xc_clean_hcall_buf()
effectively contains a check to see if hcall_buf_prep() has been called.
IOW it is OK for the same reason free(NULL) is OK.

> Is it possible that xend can call xc_interface_close() during startup
> and hcall_buf_prep() later when xend comes in interaction with xm ?

Yes, but this is safe.

Ian.