Problem about dump-core

Problem about dump-core

[Wen Congyang]

Hi, everyone:

The command 'xl dump-core' will fail after migration. The guest is HVM guest(without pv driver).
I use the newest staging branch to test. Both source and dest dom0 use the same kernel.

I use gdb to run 'xl dump-core' on dest dom0::
# gdb --args xl dump-core 1 vmcore
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/xl...done.
(gdb) b main
Breakpoint 1 at 0x406ad8: file xl.c, line 298.
(gdb) b xc_core.c:482
No source file named xc_core.c.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb) r
Starting program: /usr/sbin/xl dump-core 1 vmcore
[Thread debugging using libthread_db enabled]

Breakpoint 1, main (argc=4, argv=0x7fffffffe3d8) at xl.c:298
298	    void *config_data = 0;
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 glibc-2.12-1.80.el6.x86_64 libuuid-2.17.2-12.7.el6.x86_64 yajl-1.0.7-3.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) b xc_core.c:482
Breakpoint 2 at 0x7ffff794559d: file xc_core.c, line 482.
(gdb) c
Continuing.

Breakpoint 2, xc_domain_dumpcore_via_callback (xch=0x6262d0, domid=1, args=0x7fffffffe140, dump_rtn=0x7ffff79450c0 <local_file_dump>) at xc_core.c:482
482	    live_shinfo = xc_map_foreign_range(xch, domid, PAGE_SIZE,
(gdb) p live_shinfo 
$1 = (shared_info_any_t *) 0x0
(gdb) n
484	    if ( !live_shinfo && !info.hvm )
(gdb) p live_shinfo 
$2 = (shared_info_any_t *) 0x7ffff7ffb000
(gdb) p *live_shinfo 
Cannot access memory at address 0x7ffff7ffb000                  <==================== We cannot access live_shinfo
(gdb) b 763
Breakpoint 3 at 0x7ffff7946588: file xc_core.c, line 763.
(gdb) c
Continuing.

Breakpoint 3, xc_domain_dumpcore_via_callback (xch=0x6262d0, domid=1, args=0x7fffffffe140, dump_rtn=0x7ffff79450c0 <local_file_dump>) at xc_core.c:763
763	        sts = dump_rtn(xch, args, (char*)live_shinfo, PAGE_SIZE);
(gdb) s
local_file_dump (xch=0x6262d0, args=0x7fffffffe140, buffer=0x7ffff7ffb000 <Address 0x7ffff7ffb000 out of bounds>, length=4096) at xc_core.c:931
931	    if ( write_exact(da->fd, buffer, length) == -1 )
(gdb) s
write_exact (fd=14, data=0x7ffff7ffb000, size=4096) at xc_private.c:848
848	    while ( offset < size )
(gdb) n
850	        len = write(fd, (const char *)data + offset, size - offset);      <=============== We write live_shinfo to the core file, and fail
(gdb) p data
$3 = (const void *) 0x7ffff7ffb000
(gdb) p *data
Attempt to dereference a generic pointer.
(gdb) n
851	        if ( (len == -1) && (errno == EINTR) )
(gdb) p len
$4 = -1
(gdb) p errno
$5 = 14
(gdb) 

I try it on source dom0:
gdb --args xl dump-core 1 vmcore 
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/xl...done.
(gdb) b main
Breakpoint 1 at 0x406ad8: file xl.c, line 298.
(gdb) r
Starting program: /usr/sbin/xl dump-core 1 vmcore
[Thread debugging using libthread_db enabled]

Breakpoint 1, main (argc=4, argv=0x7fffffffe438) at xl.c:298
298	    void *config_data = 0;
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 glibc-2.12-1.80.el6.x86_64 libuuid-2.17.2-12.7.el6.x86_64 yajl-1.0.7-3.el6.x86_64 zlib-1.2.3-27.el6.x86_64
(gdb) b xc_core.c:482
Breakpoint 2 at 0x7ffff794459d: file xc_core.c, line 482.
(gdb) c
Continuing.

Breakpoint 2, xc_domain_dumpcore_via_callback (xch=0x6262d0, domid=1, args=0x7fffffffe1a0, dump_rtn=0x7ffff79440c0 <local_file_dump>) at xc_core.c:482
482	    live_shinfo = xc_map_foreign_range(xch, domid, PAGE_SIZE,
(gdb) p live_shinfo 
$1 = (shared_info_any_t *) 0x0
(gdb) n
484	    if ( !live_shinfo && !info.hvm )
(gdb) p live_shinfo 
$2 = (shared_info_any_t *) 0x7ffff7ffb000
(gdb) p *live_shinfo 
Cannot access memory at address 0x7ffff7ffb000            <======== We also cannot access live_shinfo
(gdb) b 763
Breakpoint 4 at 0x7ffff7945588: file xc_core.c, line 763.
(gdb) c
Continuing.

Breakpoint 4, xc_domain_dumpcore_via_callback (xch=0x6262d0, domid=1, args=0x7fffffffe1a0, dump_rtn=0x7ffff79440c0 <local_file_dump>) at xc_core.c:763
763	        sts = dump_rtn(xch, args, (char*)live_shinfo, PAGE_SIZE);
(gdb) s
local_file_dump (xch=0x6262d0, args=0x7fffffffe1a0, buffer=0x7ffff7ffb000 <Address 0x7ffff7ffb000 out of bounds>, length=4096) at xc_core.c:931
931	    if ( write_exact(da->fd, buffer, length) == -1 )
(gdb) p buffer
$3 = 0x7ffff7ffb000 <Address 0x7ffff7ffb000 out of bounds>
(gdb) p *buffer
Cannot access memory at address 0x7ffff7ffb000
(gdb) n
937	    if ( length >= (DUMP_INCREMENT * PAGE_SIZE) )            <===== But we can write live_shinfo to corefile. Why???

Re: Problem about dump-core

[Wen Congyang]

On 09/16/2014 10:35 AM, Wen Congyang wrote:
> Hi, everyone:
> 
> The command 'xl dump-core' will fail after migration. The guest is HVM guest(without pv driver).
> I use the newest staging branch to test. Both source and dest dom0 use the same kernel.

The kernel version is 3.2, and it only supports IOCTL_PRIVCMD_MMAPBATCH.

After more investigation, the reason is that the mfn is ~0UL, and xc_map_foreign_range()
doesn't return NULL on dest host.

This patch can fix this problem:

From: Wen Congyang <wency@cn.fujitsu.com>
Date: Tue, 16 Sep 2014 14:56:03 +0800
Subject: [PATCH] check if mfn is valid before checking if PRIVCMD_MMAPBATCH_MFN_ERROR is set

If mfn is invalid, ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH, ..) also returns 0,
and we set mfn to mfn | PRIVCMD_MMAPBATCH_MFN_ERROR. But if mfn is ~0UL,
pfn[i] ^ arr[i] returns 0, and we cannot find this error. So we should
check if mfn is valid first before testing pfn[i] ^ arr[i].

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 tools/libxc/xc_linux_osdep.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tools/libxc/xc_linux_osdep.c b/tools/libxc/xc_linux_osdep.c
index a19e4b6..baa36e6 100644
--- a/tools/libxc/xc_linux_osdep.c
+++ b/tools/libxc/xc_linux_osdep.c
@@ -333,6 +333,13 @@ static void *linux_privcmd_map_foreign_bulk(xc_interface *xch, xc_osdep_handle h
 
         for ( i = 0; i < num; ++i )
         {
+            if ( arr[i] & PRIVCMD_MMAPBATCH_MFN_ERROR )
+            {
+                /* Invalid mfn, and pfn[i] may be equal to arr[i] */
+                err[i] = -EINVAL;
+                continue;
+            }
+
             switch ( pfn[i] ^ arr[i] )
             {
             case 0:
-- 
1.9.3

> 
> I use gdb to run 'xl dump-core' on dest dom0::
> # gdb --args xl dump-core 1 vmcore
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/xl...done.
> (gdb) b main
> Breakpoint 1 at 0x406ad8: file xl.c, line 298.
> (gdb) b xc_core.c:482
> No source file named xc_core.c.
> Make breakpoint pending on future shared library load? (y or [n]) n
> (gdb) r
> Starting program: /usr/sbin/xl dump-core 1 vmcore
> [Thread debugging using libthread_db enabled]
> 
> Breakpoint 1, main (argc=4, argv=0x7fffffffe3d8) at xl.c:298
> 298	    void *config_data = 0;
> Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 glibc-2.12-1.80.el6.x86_64 libuuid-2.17.2-12.7.el6.x86_64 yajl-1.0.7-3.el6.x86_64 zlib-1.2.3-27.el6.x86_64
> (gdb) b xc_core.c:482
> Breakpoint 2 at 0x7ffff794559d: file xc_core.c, line 482.
> (gdb) c
> Continuing.
> 
> Breakpoint 2, xc_domain_dumpcore_via_callback (xch=0x6262d0, domid=1, args=0x7fffffffe140, dump_rtn=0x7ffff79450c0 <local_file_dump>) at xc_core.c:482
> 482	    live_shinfo = xc_map_foreign_range(xch, domid, PAGE_SIZE,
> (gdb) p live_shinfo 
> $1 = (shared_info_any_t *) 0x0
> (gdb) n
> 484	    if ( !live_shinfo && !info.hvm )
> (gdb) p live_shinfo 
> $2 = (shared_info_any_t *) 0x7ffff7ffb000
> (gdb) p *live_shinfo 
> Cannot access memory at address 0x7ffff7ffb000                  <==================== We cannot access live_shinfo
> (gdb) b 763
> Breakpoint 3 at 0x7ffff7946588: file xc_core.c, line 763.
> (gdb) c
> Continuing.
> 
> Breakpoint 3, xc_domain_dumpcore_via_callback (xch=0x6262d0, domid=1, args=0x7fffffffe140, dump_rtn=0x7ffff79450c0 <local_file_dump>) at xc_core.c:763
> 763	        sts = dump_rtn(xch, args, (char*)live_shinfo, PAGE_SIZE);
> (gdb) s
> local_file_dump (xch=0x6262d0, args=0x7fffffffe140, buffer=0x7ffff7ffb000 <Address 0x7ffff7ffb000 out of bounds>, length=4096) at xc_core.c:931
> 931	    if ( write_exact(da->fd, buffer, length) == -1 )
> (gdb) s
> write_exact (fd=14, data=0x7ffff7ffb000, size=4096) at xc_private.c:848
> 848	    while ( offset < size )
> (gdb) n
> 850	        len = write(fd, (const char *)data + offset, size - offset);      <=============== We write live_shinfo to the core file, and fail
> (gdb) p data
> $3 = (const void *) 0x7ffff7ffb000
> (gdb) p *data
> Attempt to dereference a generic pointer.
> (gdb) n
> 851	        if ( (len == -1) && (errno == EINTR) )
> (gdb) p len
> $4 = -1
> (gdb) p errno
> $5 = 14
> (gdb) 
> 
> I try it on source dom0:
> gdb --args xl dump-core 1 vmcore 
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/xl...done.
> (gdb) b main
> Breakpoint 1 at 0x406ad8: file xl.c, line 298.
> (gdb) r
> Starting program: /usr/sbin/xl dump-core 1 vmcore
> [Thread debugging using libthread_db enabled]
> 
> Breakpoint 1, main (argc=4, argv=0x7fffffffe438) at xl.c:298
> 298	    void *config_data = 0;
> Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 glibc-2.12-1.80.el6.x86_64 libuuid-2.17.2-12.7.el6.x86_64 yajl-1.0.7-3.el6.x86_64 zlib-1.2.3-27.el6.x86_64
> (gdb) b xc_core.c:482
> Breakpoint 2 at 0x7ffff794459d: file xc_core.c, line 482.
> (gdb) c
> Continuing.
> 
> Breakpoint 2, xc_domain_dumpcore_via_callback (xch=0x6262d0, domid=1, args=0x7fffffffe1a0, dump_rtn=0x7ffff79440c0 <local_file_dump>) at xc_core.c:482
> 482	    live_shinfo = xc_map_foreign_range(xch, domid, PAGE_SIZE,
> (gdb) p live_shinfo 
> $1 = (shared_info_any_t *) 0x0
> (gdb) n
> 484	    if ( !live_shinfo && !info.hvm )
> (gdb) p live_shinfo 
> $2 = (shared_info_any_t *) 0x7ffff7ffb000
> (gdb) p *live_shinfo 
> Cannot access memory at address 0x7ffff7ffb000            <======== We also cannot access live_shinfo
> (gdb) b 763
> Breakpoint 4 at 0x7ffff7945588: file xc_core.c, line 763.
> (gdb) c
> Continuing.
> 
> Breakpoint 4, xc_domain_dumpcore_via_callback (xch=0x6262d0, domid=1, args=0x7fffffffe1a0, dump_rtn=0x7ffff79440c0 <local_file_dump>) at xc_core.c:763
> 763	        sts = dump_rtn(xch, args, (char*)live_shinfo, PAGE_SIZE);
> (gdb) s
> local_file_dump (xch=0x6262d0, args=0x7fffffffe1a0, buffer=0x7ffff7ffb000 <Address 0x7ffff7ffb000 out of bounds>, length=4096) at xc_core.c:931
> 931	    if ( write_exact(da->fd, buffer, length) == -1 )
> (gdb) p buffer
> $3 = 0x7ffff7ffb000 <Address 0x7ffff7ffb000 out of bounds>
> (gdb) p *buffer
> Cannot access memory at address 0x7ffff7ffb000
> (gdb) n
> 937	    if ( length >= (DUMP_INCREMENT * PAGE_SIZE) )            <===== But we can write live_shinfo to corefile. Why???
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> .
> 

Re: Problem about dump-core

[Ian Campbell]

On Tue, 2014-09-16 at 15:23 +0800, Wen Congyang wrote:
> On 09/16/2014 10:35 AM, Wen Congyang wrote:
> > Hi, everyone:
> > 
> > The command 'xl dump-core' will fail after migration. The guest is HVM guest(without pv driver).
> > I use the newest staging branch to test. Both source and dest dom0 use the same kernel.
> 
> The kernel version is 3.2, and it only supports IOCTL_PRIVCMD_MMAPBATCH.
> 
> After more investigation, the reason is that the mfn is ~0UL, and xc_map_foreign_range()
> doesn't return NULL on dest host.
> 
> This patch can fix this problem:
> 
> From: Wen Congyang <wency@cn.fujitsu.com>
> Date: Tue, 16 Sep 2014 14:56:03 +0800
> Subject: [PATCH] check if mfn is valid before checking if PRIVCMD_MMAPBATCH_MFN_ERROR is set
> 
> If mfn is invalid, ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH, ..) also returns 0,
> and we set mfn to mfn | PRIVCMD_MMAPBATCH_MFN_ERROR. But if mfn is ~0UL,
> pfn[i] ^ arr[i] returns 0, and we cannot find this error. So we should
> check if mfn is valid first before testing pfn[i] ^ arr[i].

I don't think we can reliably distinguish between a large MFN and an
error with this approach, can we?

I suspect it would be better to perform a check before making the
hypercall that the 4 (?) error bits are not set in the input.

Ian.

Re: Problem about dump-core

[Wen Congyang]

On 09/17/2014 12:38 AM, Ian Campbell wrote:
> On Tue, 2014-09-16 at 15:23 +0800, Wen Congyang wrote:
>> On 09/16/2014 10:35 AM, Wen Congyang wrote:
>>> Hi, everyone:
>>>
>>> The command 'xl dump-core' will fail after migration. The guest is HVM guest(without pv driver).
>>> I use the newest staging branch to test. Both source and dest dom0 use the same kernel.
>>
>> The kernel version is 3.2, and it only supports IOCTL_PRIVCMD_MMAPBATCH.
>>
>> After more investigation, the reason is that the mfn is ~0UL, and xc_map_foreign_range()
>> doesn't return NULL on dest host.
>>
>> This patch can fix this problem:
>>
>> From: Wen Congyang <wency@cn.fujitsu.com>
>> Date: Tue, 16 Sep 2014 14:56:03 +0800
>> Subject: [PATCH] check if mfn is valid before checking if PRIVCMD_MMAPBATCH_MFN_ERROR is set
>>
>> If mfn is invalid, ioctl(fd, IOCTL_PRIVCMD_MMAPBATCH, ..) also returns 0,
>> and we set mfn to mfn | PRIVCMD_MMAPBATCH_MFN_ERROR. But if mfn is ~0UL,
>> pfn[i] ^ arr[i] returns 0, and we cannot find this error. So we should
>> check if mfn is valid first before testing pfn[i] ^ arr[i].
> 
> I don't think we can reliably distinguish between a large MFN and an
> error with this approach, can we?
> 
> I suspect it would be better to perform a check before making the
> hypercall that the 4 (?) error bits are not set in the input.

Yes, we can check it more earlier. If the 4 error bits are set, is it a valid
mfn?

Thanks
Wen Congyang

> 
> Ian.
> 
> .
> 

Re: Problem about dump-core

[Ian Campbell]

On Wed, 2014-09-17 at 08:50 +0800, Wen Congyang wrote:

> Yes, we can check it more earlier. If the 4 error bits are set, is it a valid
> mfn?

It doesn't really matter, since valid or not you can't pass such an mfn
to this interface because of the error handling behaviour.

Ian.

Re: Problem about dump-core

[Wen Congyang]

On 09/17/2014 09:11 AM, Ian Campbell wrote:
> On Wed, 2014-09-17 at 08:50 +0800, Wen Congyang wrote:
> 
>> Yes, we can check it more earlier. If the 4 error bits are set, is it a valid
>> mfn?
> 
> It doesn't really matter, since valid or not you can't pass such an mfn
> to this interface because of the error handling behaviour.

OK. I will update this patch. I think that if such mfn is valid, the user only
should use IOCTL_PRIVCMD_MMAPBATCH_V2 to map it.

Thanks
Wen Congyang

> 
> Ian.
> 
> .
> 

Re: Problem about dump-core

[Wen Congyang]

On 09/17/2014 09:11 AM, Ian Campbell wrote:
> On Wed, 2014-09-17 at 08:50 +0800, Wen Congyang wrote:
> 
>> Yes, we can check it more earlier. If the 4 error bits are set, is it a valid
>> mfn?
> 
> It doesn't really matter, since valid or not you can't pass such an mfn
> to this interface because of the error handling behaviour.

No, the user may pass more than one mfn, and part of them are ~0UL:
==============================================
    for ( i = 0; i < j; i++ )
    {
        unsigned long pfn, pagetype;
        pfn      = pagebuf->pfn_types[i + curbatch] & ~XEN_DOMCTL_PFINFO_LTAB_MASK;
        pagetype = pagebuf->pfn_types[i + curbatch] &  XEN_DOMCTL_PFINFO_LTAB_MASK;

        if ( pagetype != XEN_DOMCTL_PFINFO_XTAB
             && ctx->p2m[pfn] == (INVALID_P2M_ENTRY-1) )
        {
            /* We just allocated a new mfn above; update p2m */
            ctx->p2m[pfn] = ctx->p2m_batch[nr_mfns++]; 
            ctx->nr_pfns++; 
        }

        /* setup region_mfn[] for batch map, if necessary.
         * For HVM guests, this interface takes PFNs, not MFNs */
        if ( pagetype == XEN_DOMCTL_PFINFO_XTAB
             || pagetype == XEN_DOMCTL_PFINFO_XALLOC )
            region_mfn[i] = ~0UL; /* map will fail but we don't care */
        else
            region_mfn[i] = ctx->hvm ? pfn : ctx->p2m[pfn];
    }

    /* Map relevant mfns */
    pfn_err = calloc(j, sizeof(*pfn_err));
    if ( pfn_err == NULL )
    {
        PERROR("allocation for pfn_err failed");
        return -1;
    }
    region_base = xc_map_foreign_bulk(
        xch, dom, PROT_WRITE, region_mfn, pfn_err, j);
==============================================
The above codes are the function apply_batch(). In this case, we cannot
return NULL.

I think we should check the mfn, and if it is large mfn, and some error bits
are set, pass ~0UL to ioctl(), and mark this mfn is mapped failed in err[].

Thanks
Wen Congyang

> 
> Ian.
> 
> .
>