Test report: Migration from 4.1 to 4.2 works

Test report: Migration from 4.1 to 4.2 works

[Ian Jackson]

Migration 4.1 xend -> 4.2 xend
  OK

Migration 4.2 -> 4.1 (xend or xl)
  xend: Fails, guest ends up destroyed
  xl: Fails, xl tries to resume at sender but guest gets BUG (see below)
      This is probably a guest bug?

Migration 4.1 xend -> 4.2 xl
  Needs to be done with xl
  Stop xend on source, which leaves domain running and manipulable by xl
  xl migrate -C /etc/xen/debian.guest.osstest.cfg domain potato-beetle
  Works.

However, xl fails on config files which are missing the final
newline.  This should be fixed for 4.2.

Ian.

  xc: error: Max batch size exceeded (-18). Giving up.: Internal error
  xc: error: Error when reading batch (90 = Message too long): Internal error
  libxl: error: libxl_dom.c:313:libxl__domain_restore_common restoring domain: Resource temporarily unavailable
  cannot (re-)build domain: -3
  libxl: error: libxl.c:711:libxl_domain_destroy non-existant domain 6
  migration target: Domain creation failed (code -3).
  libxl: error: libxl_utils.c:363:libxl_read_exactly: file/stream truncated reading ready message from migration receiver stream
  libxl: info: libxl_exec.c:118:libxl_report_child_exitstatus: migration target process [15654] exited with error status 3
  Migration failed, resuming at sender.

[   37.151396] Setting capacity to 8388608
[   37.151988] Setting capacity to 8388608
[   37.172710] Setting capacity to 2048000
[   90.507105] ------------[ cut here ]------------
[   90.507105] kernel BUG at drivers/xen/events.c:1344!
[   90.507105] invalid opcode: 0000 [#1] SMP 
[   90.507105] last sysfs file: /sys/devices/virtual/net/lo/operstate
[   90.507105] Modules linked in: nbd [last unloaded: scsi_wait_scan]
[   90.507105] 
[   90.507105] Pid: 1299, comm: kstop/0 Not tainted (2.6.32.57 #1) 
[   90.507105] EIP: 0061:[<c121b9e4>] EFLAGS: 00010082 CPU: 0
[   90.507105] EIP is at xen_irq_resume+0xe3/0x2b6
[   90.507105] EAX: ffffffef EBX: 00000000 ECX: deadbeef EDX: c4c8df24
[   90.507105] ESI: 000001ff EDI: 00001ff0 EBP: c4c8df3c ESP: c4c8deec
[   90.507105]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[   90.507105] Process kstop/0 (pid: 1299, ti=c4c8c000 task=db980000 task.ti=c4c8c000)
[   90.507105] Stack:
[   90.507105]  c102ce92 c4c8df14 c1752288 c1752228 c1770004 00000000 c4c8df24 c102c270
[   90.507105] <0> c1771004 c4de1720 c1770004 c102c267 c1464b35 c166fed0 00000000 00000000
[   90.507105] <0> deadbeef deadbeef 00000003 c6000c14 c4c8df5c c121d011 00000000 dfc63f5c
[   90.507105] Call Trace:
[   90.507105]  [<c102ce92>] ? __xen_spin_lock+0xcb/0xdf
[   90.507105]  [<c102c270>] ? check_events+0x8/0xc
[   90.507105]  [<c102c267>] ? xen_restore_fl_direct_end+0x0/0x1
[   90.507105]  [<c1464b35>] ? _spin_unlock_irqrestore+0x40/0x43
[   90.507105]  [<c121d011>] ? xen_suspend+0x8c/0xa6
[   90.507105]  [<c1097f4f>] ? stop_cpu+0x7d/0xc9
[   90.507105]  [<c1073582>] ? worker_thread+0x15c/0x1f4
[   90.507105]  [<c1097ed2>] ? stop_cpu+0x0/0xc9
[   90.507105]  [<c107664d>] ? autoremove_wake_function+0x0/0x2f
[   90.507105]  [<c1073426>] ? worker_thread+0x0/0x1f4
[   90.507105]  [<c1076333>] ? kthread+0x5f/0x64
[   90.507105]  [<c10762d4>] ? kthread+0x0/0x64
[   90.507105]  [<c102f4d7>] ? kernel_thread_helper+0x7/0x10
[   90.507105] Code: 0f 0b eb fe 0f b7 40 08 3b 45 c4 74 04 0f 0b eb fe 8b 45 c4 8d 55 e8 89 5d ec 89 45 e8 b8 01 00 00 00 e8 69 f9 ff ff 85 c0 74 04 <0f> 0b eb fe 8b 55 f0 89 55 c0 8b 15 60 a0 7f c1 8b 4d c0 89 34 
[   90.507105] EIP: [<c121b9e4>] xen_irq_resume+0xe3/0x2b6 SS:ESP 0069:c4c8deec
[   90.507105] ---[ end trace c48e0191332db3e4 ]---
[   90.507105] ------------[ cut here ]------------
[   90.507105] WARNING: at kernel/time/timekeeping.c:260 ktime_get+0x21/0xce()
[   90.507105] Modules linked in: nbd [last unloaded: scsi_wait_scan]
[   90.507105] Pid: 0, comm: swapper Tainted: G      D    2.6.32.57 #1
[   90.507105] Call Trace:
[   90.507105]  [<c1061200>] warn_slowpath_common+0x65/0x7c
[   90.507105]  [<c107de3f>] ? ktime_get+0x21/0xce
[   90.507105]  [<c1061224>] warn_slowpath_null+0xd/0x10
[   90.507105]  [<c107de3f>] ktime_get+0x21/0xce
[   90.507105]  [<c14637fa>] ? schedule+0x82d/0x87a
[   90.507105]  [<c108226c>] tick_nohz_stop_sched_tick+0x76/0x387
[   90.507105]  [<c1082633>] ? T.504+0x1d/0x25
[   90.507105]  [<c10827c2>] ? tick_nohz_restart_sched_tick+0x187/0x18f
[   90.507105]  [<c102bb75>] ? xen_safe_halt+0x12/0x1f
[   90.507105]  [<c102dc1b>] cpu_idle+0x27/0x70
[   90.507105]  [<c1449ca1>] rest_init+0x5d/0x5f
[   90.507105]  [<c16dd85f>] start_kernel+0x315/0x31a
[   90.507105]  [<c16dd0a8>] i386_start_kernel+0x97/0x9e
[   90.507105]  [<c16e0cae>] xen_start_kernel+0x557/0x55f
[   90.507105] ---[ end trace c48e0191332db3e5 ]---

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Campbell]

On Fri, 2012-08-31 at 12:04 +0100, Ian Jackson wrote:
> Migration 4.1 xend -> 4.2 xend
>   OK
> 
> Migration 4.2 -> 4.1 (xend or xl)

We don't support this, so I'm not too concerned about the failures.

>   xend: Fails, guest ends up destroyed
>   xl: Fails, xl tries to resume at sender but guest gets BUG (see below)
>       This is probably a guest bug?
> 
> Migration 4.1 xend -> 4.2 xl
>   Needs to be done with xl
>   Stop xend on source, which leaves domain running and manipulable by xl
>   xl migrate -C /etc/xen/debian.guest.osstest.cfg domain potato-beetle
>   Works.

This is great. We should write this down somewhere. I guess
http://wiki.xen.org/wiki/XL#Upgrading_from_xend ?

> However, xl fails on config files which are missing the final
> newline.  This should be fixed for 4.2.

Since that I suppose involves the parser are you going to do that?

Did you also try xl 4.1 -> 4.2? 

Ian.

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Campbell]

On Fri, 2012-08-31 at 12:09 +0100, Ian Campbell wrote:

Should have said in here somewhere: Thank for doing this.

Re: Test report: Migration from 4.1 to 4.2 works

[Jan Beulich]

>>> On 31.08.12 at 13:04, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Migration 4.1 xend -> 4.2 xl
>   Needs to be done with xl
>   Stop xend on source, which leaves domain running and manipulable by xl
>   xl migrate -C /etc/xen/debian.guest.osstest.cfg domain potato-beetle
>   Works.

Is that really an acceptable approach? What if you have multiple
VMs running, and want to migrate just part of them? All the other
would remain unmanageable at least for the duration of the
migration(s). (And I also wonder if 4.1's xl is complete/stable
enough to recommend such an approach as a general mechanism.)

Jan

Re: Test report: Migration from 4.1 to 4.2 works

[Mathias Gaunard]

On 31/08/2012 13:04, Ian Jackson wrote:

> Migration 4.1 xend -> 4.2 xl
>    Needs to be done with xl
>    Stop xend on source, which leaves domain running and manipulable by xl
>    xl migrate -C /etc/xen/debian.guest.osstest.cfg domain potato-beetle
>    Works.

Does it work out of the box if you just choose to keep using xend and 
not move to xl?

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Campbell]

On Fri, 2012-08-31 at 12:56 +0100, Mathias Gaunard wrote:
> On 31/08/2012 13:04, Ian Jackson wrote:
> 
> > Migration 4.1 xend -> 4.2 xl
> >    Needs to be done with xl
> >    Stop xend on source, which leaves domain running and manipulable by xl
> >    xl migrate -C /etc/xen/debian.guest.osstest.cfg domain potato-beetle
> >    Works.
> 
> Does it work out of the box if you just choose to keep using xend and 
> not move to xl?

That was the very first line of Ian's report.

Ian.

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Campbell]

On Fri, 2012-08-31 at 12:30 +0100, Jan Beulich wrote:
> >>> On 31.08.12 at 13:04, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> > Migration 4.1 xend -> 4.2 xl
> >   Needs to be done with xl
> >   Stop xend on source, which leaves domain running and manipulable by xl
> >   xl migrate -C /etc/xen/debian.guest.osstest.cfg domain potato-beetle
> >   Works.
> 
> Is that really an acceptable approach? What if you have multiple
> VMs running, and want to migrate just part of them? All the other
> would remain unmanageable at least for the duration of the
> migration(s). (And I also wonder if 4.1's xl is complete/stable
> enough to recommend such an approach as a general mechanism.)

The alternative I suppose would be to:
      * start xend on new host (running 4.2)
      * migrate the domains you want over to the new system
      * stop xend on the new host
      * xl migrate localhost for each domain.

Ian.

Re: Test report: Migration from 4.1 to 4.2 works

[Jan Beulich]

>>> On 31.08.12 at 15:01, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2012-08-31 at 12:30 +0100, Jan Beulich wrote:
>> >>> On 31.08.12 at 13:04, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
>> > Migration 4.1 xend -> 4.2 xl
>> >   Needs to be done with xl
>> >   Stop xend on source, which leaves domain running and manipulable by xl
>> >   xl migrate -C /etc/xen/debian.guest.osstest.cfg domain potato-beetle
>> >   Works.
>> 
>> Is that really an acceptable approach? What if you have multiple
>> VMs running, and want to migrate just part of them? All the other
>> would remain unmanageable at least for the duration of the
>> migration(s). (And I also wonder if 4.1's xl is complete/stable
>> enough to recommend such an approach as a general mechanism.)
> 
> The alternative I suppose would be to:
>       * start xend on new host (running 4.2)
>       * migrate the domains you want over to the new system
>       * stop xend on the new host
>       * xl migrate localhost for each domain.

So why is it that xend and xl can't talk to each other for a migration?

Jan

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Campbell]

On Fri, 2012-08-31 at 14:11 +0100, Jan Beulich wrote:
> >>> On 31.08.12 at 15:01, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Fri, 2012-08-31 at 12:30 +0100, Jan Beulich wrote:
> >> >>> On 31.08.12 at 13:04, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> >> > Migration 4.1 xend -> 4.2 xl
> >> >   Needs to be done with xl
> >> >   Stop xend on source, which leaves domain running and manipulable by xl
> >> >   xl migrate -C /etc/xen/debian.guest.osstest.cfg domain potato-beetle
> >> >   Works.
> >> 
> >> Is that really an acceptable approach? What if you have multiple
> >> VMs running, and want to migrate just part of them? All the other
> >> would remain unmanageable at least for the duration of the
> >> migration(s). (And I also wonder if 4.1's xl is complete/stable
> >> enough to recommend such an approach as a general mechanism.)
> > 
> > The alternative I suppose would be to:
> >       * start xend on new host (running 4.2)
> >       * migrate the domains you want over to the new system
> >       * stop xend on the new host
> >       * xl migrate localhost for each domain.
> 
> So why is it that xend and xl can't talk to each other for a migration?

The wire protocol is different, also xend doesn't know how to start the
xl receive process on the other end.

I suppose we could implement xl migrate-receive-from-xend which the user
could manually run on the target. Bit late for 4.2 though.

Ian.

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Jackson]

Ian Campbell writes ("Re: Test report: Migration from 4.1 to 4.2 works"):
> On Fri, 2012-08-31 at 12:04 +0100, Ian Jackson wrote:
> > However, xl fails on config files which are missing the final
> > newline.  This should be fixed for 4.2.
> 
> Since that I suppose involves the parser are you going to do that?

Below.

> Did you also try xl 4.1 -> 4.2? 

Yes.  Although that's just the same as 4.1+xend -> 4.2+xl really.  But
I did try migrating a domain created with xl as well as one created
with xm.


From: Ian Jackson <ian.jackson@eu.citrix.com>
Subject: [PATCH] libxl: Tolerate xl config files missing trailing newline

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

---
 tools/libxl/libxlu_cfg_y.c |  154 +++++++++++++++++++++++---------------------
 tools/libxl/libxlu_cfg_y.y |   12 ++-
 2 files changed, 88 insertions(+), 78 deletions(-)

diff --git a/tools/libxl/libxlu_cfg_y.c b/tools/libxl/libxlu_cfg_y.c
index 5214386..218933e 100644
--- a/tools/libxl/libxlu_cfg_y.c
+++ b/tools/libxl/libxlu_cfg_y.c
@@ -373,18 +373,18 @@ union yyalloc
 #endif
 
 /* YYFINAL -- State number of the termination state.  */
-#define YYFINAL  2
+#define YYFINAL  3
 /* YYLAST -- Last index in YYTABLE.  */
-#define YYLAST   23
+#define YYLAST   24
 
 /* YYNTOKENS -- Number of terminals.  */
 #define YYNTOKENS  12
 /* YYNNTS -- Number of nonterminals.  */
-#define YYNNTS  9
+#define YYNNTS  11
 /* YYNRULES -- Number of rules.  */
-#define YYNRULES  19
+#define YYNRULES  22
 /* YYNRULES -- Number of states.  */
-#define YYNSTATES  28
+#define YYNSTATES  30
 
 /* YYTRANSLATE(YYLEX) -- Bison symbol number corresponding to YYLEX.  */
 #define YYUNDEFTOK  2
@@ -430,26 +430,28 @@ static const yytype_uint8 yytranslate[] =
    YYRHS.  */
 static const yytype_uint8 yyprhs[] =
 {
-       0,     0,     3,     4,     7,    12,    14,    17,    19,    21,
-      23,    28,    30,    32,    33,    35,    39,    42,    48,    49
+       0,     0,     3,     5,     8,     9,    12,    15,    17,    20,
+      24,    26,    28,    30,    35,    37,    39,    40,    42,    46,
+      49,    55,    56
 };
 
 /* YYRHS -- A `-1'-separated list of the rules' RHS.  */
 static const yytype_int8 yyrhs[] =
 {
-      13,     0,    -1,    -1,    13,    14,    -1,     3,     7,    16,
-      15,    -1,    15,    -1,     1,     6,    -1,     6,    -1,     8,
-      -1,    17,    -1,     9,    20,    18,    10,    -1,     4,    -1,
-       5,    -1,    -1,    19,    -1,    19,    11,    20,    -1,    17,
-      20,    -1,    19,    11,    20,    17,    20,    -1,    -1,    20,
-       6,    -1
+      13,     0,    -1,    14,    -1,    14,    16,    -1,    -1,    14,
+      15,    -1,    16,    17,    -1,    17,    -1,     1,     6,    -1,
+       3,     7,    18,    -1,     6,    -1,     8,    -1,    19,    -1,
+       9,    22,    20,    10,    -1,     4,    -1,     5,    -1,    -1,
+      21,    -1,    21,    11,    22,    -1,    19,    22,    -1,    21,
+      11,    22,    19,    22,    -1,    -1,    22,     6,    -1
 };
 
 /* YYRLINE[YYN] -- source line where rule number YYN was defined.  */
 static const yytype_uint8 yyrline[] =
 {
-       0,    47,    47,    48,    50,    52,    53,    55,    56,    58,
-      59,    61,    62,    64,    65,    66,    68,    69,    71,    73
+       0,    47,    47,    48,    50,    51,    53,    54,    55,    57,
+      59,    60,    62,    63,    65,    66,    68,    69,    70,    72,
+      73,    75,    77
 };
 #endif
 
@@ -459,8 +461,8 @@ static const yytype_uint8 yyrline[] =
 static const char *const yytname[] =
 {
   "$end", "error", "$undefined", "IDENT", "STRING", "NUMBER", "NEWLINE",
-  "'='", "';'", "'['", "']'", "','", "$accept", "file", "assignment",
-  "endstmt", "value", "atom", "valuelist", "values", "nlok", 0
+  "'='", "';'", "'['", "']'", "','", "$accept", "file", "stmts", "stmt",
+  "assignment", "endstmt", "value", "atom", "valuelist", "values", "nlok", 0
 };
 #endif
 
@@ -477,15 +479,17 @@ static const yytype_uint16 yytoknum[] =
 /* YYR1[YYN] -- Symbol number of symbol that rule YYN derives.  */
 static const yytype_uint8 yyr1[] =
 {
-       0,    12,    13,    13,    14,    14,    14,    15,    15,    16,
-      16,    17,    17,    18,    18,    18,    19,    19,    20,    20
+       0,    12,    13,    13,    14,    14,    15,    15,    15,    16,
+      17,    17,    18,    18,    19,    19,    20,    20,    20,    21,
+      21,    22,    22
 };
 
 /* YYR2[YYN] -- Number of symbols composing right hand side of rule YYN.  */
 static const yytype_uint8 yyr2[] =
 {
-       0,     2,     0,     2,     4,     1,     2,     1,     1,     1,
-       4,     1,     1,     0,     1,     3,     2,     5,     0,     2
+       0,     2,     1,     2,     0,     2,     2,     1,     2,     3,
+       1,     1,     1,     4,     1,     1,     0,     1,     3,     2,
+       5,     0,     2
 };
 
 /* YYDEFACT[STATE-NAME] -- Default rule to reduce with in state
@@ -493,59 +497,61 @@ static const yytype_uint8 yyr2[] =
    means the default is an error.  */
 static const yytype_uint8 yydefact[] =
 {
-       2,     0,     1,     0,     0,     7,     8,     3,     5,     6,
-       0,    11,    12,    18,     0,     9,    13,     4,    19,    18,
-       0,    14,    16,    10,    18,    15,    18,    17
+       4,     0,     0,     1,     0,     0,    10,    11,     5,     3,
+       7,     8,     0,     6,    14,    15,    21,     9,    12,    16,
+      22,    21,     0,    17,    19,    13,    21,    18,    21,    20
 };
 
 /* YYDEFGOTO[NTERM-NUM].  */
 static const yytype_int8 yydefgoto[] =
 {
-      -1,     1,     7,     8,    14,    15,    20,    21,    16
+      -1,     1,     2,     8,     9,    10,    17,    18,    22,    23,
+      19
 };
 
 /* YYPACT[STATE-NUM] -- Index in YYTABLE of the portion describing
    STATE-NUM.  */
-#define YYPACT_NINF -17
+#define YYPACT_NINF -18
 static const yytype_int8 yypact[] =
 {
-     -17,     2,   -17,    -5,    -3,   -17,   -17,   -17,   -17,   -17,
-      10,   -17,   -17,   -17,    14,   -17,    12,   -17,   -17,   -17,
-      11,    -4,     6,   -17,   -17,    12,   -17,     6
+     -18,     4,     0,   -18,    -1,     6,   -18,   -18,   -18,     3,
+     -18,   -18,    11,   -18,   -18,   -18,   -18,   -18,   -18,    13,
+     -18,   -18,    12,    10,    17,   -18,   -18,    13,   -18,    17
 };
 
 /* YYPGOTO[NTERM-NUM].  */
 static const yytype_int8 yypgoto[] =
 {
-     -17,   -17,   -17,     9,   -17,   -16,   -17,   -17,   -13
+     -18,   -18,   -18,   -18,   -18,    15,   -18,   -17,   -18,   -18,
+     -14
 };
 
 /* YYTABLE[YYPACT[STATE-NUM]].  What to do in state STATE-NUM.  If
    positive, shift that token.  If negative, reduce the rule which
    number is the opposite.  If zero, do what YYDEFACT says.
    If YYTABLE_NINF, syntax error.  */
-#define YYTABLE_NINF -1
-static const yytype_uint8 yytable[] =
+#define YYTABLE_NINF -3
+static const yytype_int8 yytable[] =
 {
-      19,     9,     2,     3,    10,     4,    22,    24,     5,    26,
-       6,    25,    18,    27,    11,    12,    11,    12,    18,    13,
-       5,    23,     6,    17
+      -2,     4,    21,     5,     3,    11,     6,    24,     7,     6,
+      28,     7,    27,    12,    29,    14,    15,    14,    15,    20,
+      16,    26,    25,    20,    13
 };
 
 static const yytype_uint8 yycheck[] =
 {
-      16,     6,     0,     1,     7,     3,    19,    11,     6,    25,
-       8,    24,     6,    26,     4,     5,     4,     5,     6,     9,
-       6,    10,     8,    14
+       0,     1,    19,     3,     0,     6,     6,    21,     8,     6,
+      27,     8,    26,     7,    28,     4,     5,     4,     5,     6,
+       9,    11,    10,     6,     9
 };
 
 /* YYSTOS[STATE-NUM] -- The (internal number of the) accessing
    symbol of state STATE-NUM.  */
 static const yytype_uint8 yystos[] =
 {
-       0,    13,     0,     1,     3,     6,     8,    14,    15,     6,
-       7,     4,     5,     9,    16,    17,    20,    15,     6,    17,
-      18,    19,    20,    10,    11,    20,    17,    20
+       0,    13,    14,     0,     1,     3,     6,     8,    15,    16,
+      17,     6,     7,    17,     4,     5,     9,    18,    19,    22,
+       6,    19,    20,    21,    22,    10,    11,    22,    19,    22
 };
 
 #define yyerrok		(yyerrstatus = 0)
@@ -1077,7 +1083,7 @@ yydestruct (yymsg, yytype, yyvaluep, yylocationp, ctx)
 	{ free((yyvaluep->string)); };
 
 /* Line 1000 of yacc.c  */
-#line 1081 "libxlu_cfg_y.c"
+#line 1087 "libxlu_cfg_y.c"
 	break;
       case 4: /* "STRING" */
 
@@ -1086,7 +1092,7 @@ yydestruct (yymsg, yytype, yyvaluep, yylocationp, ctx)
 	{ free((yyvaluep->string)); };
 
 /* Line 1000 of yacc.c  */
-#line 1090 "libxlu_cfg_y.c"
+#line 1096 "libxlu_cfg_y.c"
 	break;
       case 5: /* "NUMBER" */
 
@@ -1095,43 +1101,43 @@ yydestruct (yymsg, yytype, yyvaluep, yylocationp, ctx)
 	{ free((yyvaluep->string)); };
 
 /* Line 1000 of yacc.c  */
-#line 1099 "libxlu_cfg_y.c"
+#line 1105 "libxlu_cfg_y.c"
 	break;
-      case 16: /* "value" */
+      case 18: /* "value" */
 
 /* Line 1000 of yacc.c  */
 #line 43 "libxlu_cfg_y.y"
 	{ xlu__cfg_set_free((yyvaluep->setting)); };
 
 /* Line 1000 of yacc.c  */
-#line 1108 "libxlu_cfg_y.c"
+#line 1114 "libxlu_cfg_y.c"
 	break;
-      case 17: /* "atom" */
+      case 19: /* "atom" */
 
 /* Line 1000 of yacc.c  */
 #line 40 "libxlu_cfg_y.y"
 	{ free((yyvaluep->string)); };
 
 /* Line 1000 of yacc.c  */
-#line 1117 "libxlu_cfg_y.c"
+#line 1123 "libxlu_cfg_y.c"
 	break;
-      case 18: /* "valuelist" */
+      case 20: /* "valuelist" */
 
 /* Line 1000 of yacc.c  */
 #line 43 "libxlu_cfg_y.y"
 	{ xlu__cfg_set_free((yyvaluep->setting)); };
 
 /* Line 1000 of yacc.c  */
-#line 1126 "libxlu_cfg_y.c"
+#line 1132 "libxlu_cfg_y.c"
 	break;
-      case 19: /* "values" */
+      case 21: /* "values" */
 
 /* Line 1000 of yacc.c  */
 #line 43 "libxlu_cfg_y.y"
 	{ xlu__cfg_set_free((yyvaluep->setting)); };
 
 /* Line 1000 of yacc.c  */
-#line 1135 "libxlu_cfg_y.c"
+#line 1141 "libxlu_cfg_y.c"
 	break;
 
       default:
@@ -1459,80 +1465,80 @@ yyreduce:
   YY_REDUCE_PRINT (yyn);
   switch (yyn)
     {
-        case 4:
+        case 9:
 
 /* Line 1455 of yacc.c  */
-#line 51 "libxlu_cfg_y.y"
-    { xlu__cfg_set_store(ctx,(yyvsp[(1) - (4)].string),(yyvsp[(3) - (4)].setting),(yylsp[(3) - (4)]).first_line); ;}
+#line 57 "libxlu_cfg_y.y"
+    { xlu__cfg_set_store(ctx,(yyvsp[(1) - (3)].string),(yyvsp[(3) - (3)].setting),(yylsp[(3) - (3)]).first_line); ;}
     break;
 
-  case 9:
+  case 12:
 
 /* Line 1455 of yacc.c  */
-#line 58 "libxlu_cfg_y.y"
+#line 62 "libxlu_cfg_y.y"
     { (yyval.setting)= xlu__cfg_set_mk(ctx,1,(yyvsp[(1) - (1)].string)); ;}
     break;
 
-  case 10:
+  case 13:
 
 /* Line 1455 of yacc.c  */
-#line 59 "libxlu_cfg_y.y"
+#line 63 "libxlu_cfg_y.y"
     { (yyval.setting)= (yyvsp[(3) - (4)].setting); ;}
     break;
 
-  case 11:
+  case 14:
 
 /* Line 1455 of yacc.c  */
-#line 61 "libxlu_cfg_y.y"
+#line 65 "libxlu_cfg_y.y"
     { (yyval.string)= (yyvsp[(1) - (1)].string); ;}
     break;
 
-  case 12:
+  case 15:
 
 /* Line 1455 of yacc.c  */
-#line 62 "libxlu_cfg_y.y"
+#line 66 "libxlu_cfg_y.y"
     { (yyval.string)= (yyvsp[(1) - (1)].string); ;}
     break;
 
-  case 13:
+  case 16:
 
 /* Line 1455 of yacc.c  */
-#line 64 "libxlu_cfg_y.y"
+#line 68 "libxlu_cfg_y.y"
     { (yyval.setting)= xlu__cfg_set_mk(ctx,0,0); ;}
     break;
 
-  case 14:
+  case 17:
 
 /* Line 1455 of yacc.c  */
-#line 65 "libxlu_cfg_y.y"
+#line 69 "libxlu_cfg_y.y"
     { (yyval.setting)= (yyvsp[(1) - (1)].setting); ;}
     break;
 
-  case 15:
+  case 18:
 
 /* Line 1455 of yacc.c  */
-#line 66 "libxlu_cfg_y.y"
+#line 70 "libxlu_cfg_y.y"
     { (yyval.setting)= (yyvsp[(1) - (3)].setting); ;}
     break;
 
-  case 16:
+  case 19:
 
 /* Line 1455 of yacc.c  */
-#line 68 "libxlu_cfg_y.y"
+#line 72 "libxlu_cfg_y.y"
     { (yyval.setting)= xlu__cfg_set_mk(ctx,2,(yyvsp[(1) - (2)].string)); ;}
     break;
 
-  case 17:
+  case 20:
 
 /* Line 1455 of yacc.c  */
-#line 69 "libxlu_cfg_y.y"
+#line 73 "libxlu_cfg_y.y"
     { xlu__cfg_set_add(ctx,(yyvsp[(1) - (5)].setting),(yyvsp[(4) - (5)].string)); (yyval.setting)= (yyvsp[(1) - (5)].setting); ;}
     break;
 
 
 
 /* Line 1455 of yacc.c  */
-#line 1536 "libxlu_cfg_y.c"
+#line 1542 "libxlu_cfg_y.c"
       default: break;
     }
   YY_SYMBOL_PRINT ("-> $$ =", yyr1[yyn], &yyval, &yyloc);
diff --git a/tools/libxl/libxlu_cfg_y.y b/tools/libxl/libxlu_cfg_y.y
index 29aedca..aa9f787 100644
--- a/tools/libxl/libxlu_cfg_y.y
+++ b/tools/libxl/libxlu_cfg_y.y
@@ -44,14 +44,18 @@
 
 %%
 
-file: /* empty */
- |     file assignment
+file:  stmts
+ |     stmts assignment
 
-assignment: IDENT '=' value endstmt
-                            { xlu__cfg_set_store(ctx,$1,$3,@3.first_line); }
+stmts:  /* empty */
+ |      stmts stmt
+
+stmt:   assignment endstmt
  |      endstmt
  |      error NEWLINE
 
+assignment: IDENT '=' value { xlu__cfg_set_store(ctx,$1,$3,@3.first_line); }
+
 endstmt: NEWLINE
  |      ';'
 
-- 
tg: (9153666..) t/xen/xl.cfg.no-final-newline-ok (depends on: t/xen/xl.cfg.mem-fix)

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] Test report: Migration from 4.1 to 4.2 works"):
> The alternative I suppose would be to:
>       * start xend on new host (running 4.2)
>       * migrate the domains you want over to the new system
>       * stop xend on the new host

This would work.

>       * xl migrate localhost for each domain.

This is only necessary to get the domain config file stashed away for
the _next_ save or migration.

Ian.

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Jackson]

Jan Beulich writes ("Re: [Xen-devel] Test report: Migration from 4.1 to 4.2 works"):
> On 31.08.12 at 13:04, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> > Migration 4.1 xend -> 4.2 xl
> >   Needs to be done with xl
> >   Stop xend on source, which leaves domain running and manipulable by xl
> >   xl migrate -C /etc/xen/debian.guest.osstest.cfg domain potato-beetle
> >   Works.
> 
> Is that really an acceptable approach? What if you have multiple
> VMs running, and want to migrate just part of them? All the other
> would remain unmanageable at least for the duration of the
> migration(s).

This is true but in the usual case you'll be wanting to migrate them
all as part of an infrastructure upgrade to 4.2.

> (And I also wonder if 4.1's xl is complete/stable
> enough to recommend such an approach as a general mechanism.)

That is perhaps a worry but I'm pleased to be able to report that it
does work :-).

Thinking about it, I didn't try an HVM domain.  I will see if I can
manage that but probably not today.

Ian.

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Campbell]

On Fri, 2012-08-31 at 15:38 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] Test report: Migration from 4.1 to 4.2 works"):
> > The alternative I suppose would be to:
> >       * start xend on new host (running 4.2)
> >       * migrate the domains you want over to the new system
> >       * stop xend on the new host
> 
> This would work.
> 
> >       * xl migrate localhost for each domain.
> 
> This is only necessary to get the domain config file stashed away for
> the _next_ save or migration.

and general consistency + reduction of later surprises?

Ian.

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Jackson]

I have now managed to redo these tests with an HVM guest (Windows XP,
in this case).

Migration 4.1 xl -> 4.2 xl
  OK

Migration 4.1 xend -> 4.2 xend
  OK

Migration 4.1 xend -> 4.2 xl, plan A:
  Stop xend, use xl to do the migration.

  Fails:

  xc: Saving memory: iter 1 (last sent 134144 skipped 1553): 0/1048576
  0%xc: error: Failed to allocate memory for batch.!: Internal error

  (XEN) page_alloc.c:1284:d0 Over-allocation for domain 23: 131329 > 131328
  (XEN) memory.c:131:d0 Could not allocate order=0 extent: id=23 memflags=0 (288 of 472)

This was always something of a bodge.  It would be nice if we
understood why this doesn't work.

Migration 4.1 xend -> 4.2 xl, plan B:
  Migrate using xm to 4.2 (works).
  Stop xend, say  xenstore-write /local/domain/0/libxl/disable_udev 1
  Check that xl migrate localhost works.  It doesn't - same error
  as above.

Before we call 4.2 final we should think about this migration path.

Also I wrote:

> However, xl fails on config files which are missing the final
> newline.  This should be fixed for 4.2.

My patch for this didn't make it into 4.2 RC4.

Ian.

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Jackson]

Ian Jackson writes ("Re: Test report: Migration from 4.1 to 4.2 works"):
>   xc: Saving memory: iter 1 (last sent 134144 skipped 1553): 0/1048576
>   0%xc: error: Failed to allocate memory for batch.!: Internal error

15:22 <ijc> It might be interesting to compare the actual memory usage
            of the same domain started with xend and xl on 4.1/4.2?

Started with xl create:

root@potato-beetle:~# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   512     4     r-----     641.0
win.guest.osstest                           10   507     2     -b----     223.4
root@potato-beetle:~#

Then switched to xend:

root@potato-beetle:~# /etc/init.d/xend start
root@potato-beetle:~# xenstore-rm /local/domain/0/libxl/disable_udev

Started with xm create:

root@potato-beetle:~# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   512     4     r-----     720.9
win.guest.osstest                           11   515     2     ------     238.0
root@potato-beetle:~#

I think what is happening is this: xend accounts memory differently to
xl, giving the guest actually slightly more than is specified in the
config file.  When xl during incoming migration reads the guest config
file, it assumes that the config file's memory maximum is an accurate
representation of the guest's actual memory use.

I can reproduce this problem by ballooning a PV guest before migrating
it.  Eg,
   xl create /etc/xen/debian.guest.osstest.cfg
with memory=512, maxmem=1024.  Then
   xl migrate debian.guest.osstest localhost
works but
   xl mem-set debian.guest.osstest 1024
   xl migrate debian.guest.osstest localhost

I think we need to fix this in 4.2.1 somehow.

Ian.

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Campbell]

On Mon, 2012-09-10 at 15:42 +0100, Ian Jackson wrote:
> Ian Jackson writes ("Re: Test report: Migration from 4.1 to 4.2 works"):
> >   xc: Saving memory: iter 1 (last sent 134144 skipped 1553): 0/1048576
> >   0%xc: error: Failed to allocate memory for batch.!: Internal error
> 
> 15:22 <ijc> It might be interesting to compare the actual memory usage
>             of the same domain started with xend and xl on 4.1/4.2?
> 
> Started with xl create:
> 
> root@potato-beetle:~# xl list
> Name                                        ID   Mem VCPUs      State   Time(s)
> Domain-0                                     0   512     4     r-----     641.0
> win.guest.osstest                           10   507     2     -b----     223.4
> root@potato-beetle:~#
> 
> Then switched to xend:
> 
> root@potato-beetle:~# /etc/init.d/xend start
> root@potato-beetle:~# xenstore-rm /local/domain/0/libxl/disable_udev
> 
> Started with xm create:
> 
> root@potato-beetle:~# xl list
> Name                                        ID   Mem VCPUs      State   Time(s)
> Domain-0                                     0   512     4     r-----     720.9
> win.guest.osstest                           11   515     2     ------     238.0
> root@potato-beetle:~#
> 
> I think what is happening is this: xend accounts memory differently to
> xl, giving the guest actually slightly more than is specified in the
> config file.  When xl during incoming migration reads the guest config
> file, it assumes that the config file's memory maximum is an accurate
> representation of the guest's actual memory use.
> 
> I can reproduce this problem by ballooning a PV guest before migrating
> it.  Eg,
>    xl create /etc/xen/debian.guest.osstest.cfg
> with memory=512, maxmem=1024.  Then
>    xl migrate debian.guest.osstest localhost
> works but
>    xl mem-set debian.guest.osstest 1024
>    xl migrate debian.guest.osstest localhost
... "does not". I guess?

This is another facet of the problem with migrating based on the stored
config without taking into account dynamic changes made since the domain
was started.

> I think we need to fix this in 4.2.1 somehow.

The official workaround in 4.2.x is to use "xl config-update" to store a
new config file describing the new state of the domain when you change
its properties at runtime.

In 4.3 I'd like to add functionality to libxl to reconstitute a
libxl_domain_config from a running domain and use that instead of
reparsing the config for migrating/reboot etc. I think it's the only way
to handle these sorts of situations sanely.

Ian.

Re: Test report: Migration from 4.1 to 4.2 works

[Ian Jackson]

Ian Jackson writes ("Re: Test report: Migration from 4.1 to 4.2 works"):
> Migration 4.1 xend -> 4.2 xl, plan A:
>   Stop xend, use xl to do the migration.
> 
>   Fails:
> 
>   xc: Saving memory: iter 1 (last sent 134144 skipped 1553): 0/1048576
>   0%xc: error: Failed to allocate memory for batch.!: Internal error
> 
>   (XEN) page_alloc.c:1284:d0 Over-allocation for domain 23: 131329 > 131328
>   (XEN) memory.c:131:d0 Could not allocate order=0 extent: id=23 memflags=0 (288 of 472)

This works if I create a guest config file with a bigger "memory="
value, and pass that to xl migrate -C.

I can also do the migration successfully by migrating the domain to
the new host with xend, and then stopping xend on the target host.
After that an "xl migrate localhost" again works if the config file
I supply with -C (which is mandatory since xend didn't write one) has
an increased memory= value.

Ian.