Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Ian Campbell]

On Mon, 2014-12-01 at 23:41 +0000, Mark Pryor wrote:
> list,

Thanks. If you've identified a buggy changeset then it is fine to post
to the devel lists. I've added a CC. I've also CCd everyone listed in
the commit which you've fingered.

Olaf, does this suggested change look correct? If so then can you turn
it into a patch please.

> 
> 
>  this commit to 4.5: 
> 
> git show 4542ae340d75bd6319e3fcd94e6c9336e210aeef
> 
> 
> 
>  ^^ causes my C7 install to hang at shutdown.
>  xenstored.service will shutdown in parallel with xendomains before it
> should
>  and its broken
> 
> its by Olaf
> I took the time to narrow the commit down exactly.. first time I ever
> did that
> Its a regression. The xen shutdown process in systemd was OK before
> that commit
> 
> 
> 
> xenconsoled.service needs
> After=xenstored.service
> 
> 
> not,
> After=xenstored.socket
> 
> 
> regards,
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Olaf Hering]

On Tue, Dec 02, Ian Campbell wrote:

> On Mon, 2014-12-01 at 23:41 +0000, Mark Pryor wrote:
> > list,
> 
> Thanks. If you've identified a buggy changeset then it is fine to post
> to the devel lists. I've added a CC. I've also CCd everyone listed in
> the commit which you've fingered.
> 
> Olaf, does this suggested change look correct? If so then can you turn
> it into a patch please.

Yes, something like this (sed -i 's@socket@service@g' *.in):


diff --git a/tools/hotplug/Linux/systemd/xen-init-dom0.service.in b/tools/hotplug/Linux/systemd/xen-init-dom0.service.in
index 4d4cb23..3befadc 100644
--- a/tools/hotplug/Linux/systemd/xen-init-dom0.service.in
+++ b/tools/hotplug/Linux/systemd/xen-init-dom0.service.in
@@ -1,7 +1,7 @@
 [Unit]
 Description=xen-init-dom0, initialise Dom0 configuration (xenstore nodes, JSON configuration stub)
-Requires=xenstored.socket proc-xen.mount
-After=xenstored.socket proc-xen.mount
+Requires=xenstored.service proc-xen.mount
+After=xenstored.service proc-xen.mount
 ConditionPathExists=/proc/xen/capabilities
 
 [Service]
diff --git a/tools/hotplug/Linux/systemd/xen-qemu-dom0-disk-backend.service.in b/tools/hotplug/Linux/systemd/xen-qemu-dom0-disk-backend.service.in
index 6b9c96e..0a5807a 100644
--- a/tools/hotplug/Linux/systemd/xen-qemu-dom0-disk-backend.service.in
+++ b/tools/hotplug/Linux/systemd/xen-qemu-dom0-disk-backend.service.in
@@ -1,7 +1,7 @@
 [Unit]
 Description=qemu for xen dom0 disk backend
-Requires=proc-xen.mount xenstored.socket
-After=proc-xen.mount xenstored.socket xenconsoled.service
+Requires=proc-xen.mount xenstored.service
+After=proc-xen.mount xenstored.service xenconsoled.service
 Before=xendomains.service libvirtd.service libvirt-guests.service
 RefuseManualStop=true
 ConditionPathExists=/proc/xen/capabilities
diff --git a/tools/hotplug/Linux/systemd/xenconsoled.service.in b/tools/hotplug/Linux/systemd/xenconsoled.service.in
index 2c5d99f..cb44cd6 100644
--- a/tools/hotplug/Linux/systemd/xenconsoled.service.in
+++ b/tools/hotplug/Linux/systemd/xenconsoled.service.in
@@ -1,7 +1,7 @@
 [Unit]
 Description=Xenconsoled - handles logging from guest consoles and hypervisor
-Requires=proc-xen.mount xenstored.socket
-After=proc-xen.mount xenstored.socket
+Requires=proc-xen.mount xenstored.service
+After=proc-xen.mount xenstored.service
 ConditionPathExists=/proc/xen/capabilities
 
 [Service]
diff --git a/tools/hotplug/Linux/systemd/xendomains.service.in b/tools/hotplug/Linux/systemd/xendomains.service.in
index 757278f..9962671 100644
--- a/tools/hotplug/Linux/systemd/xendomains.service.in
+++ b/tools/hotplug/Linux/systemd/xendomains.service.in
@@ -1,7 +1,7 @@
 [Unit]
 Description=Xendomains - start and stop guests on boot and shutdown
-Requires=proc-xen.mount xenstored.socket
-After=proc-xen.mount xenstored.socket xenconsoled.service xen-init-dom0.service
+Requires=proc-xen.mount xenstored.service
+After=proc-xen.mount xenstored.service xenconsoled.service xen-init-dom0.service
 ConditionPathExists=/proc/xen/capabilities
 
 [Service]

Olaf

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Ian Campbell]

On Tue, 2014-12-02 at 16:17 +0100, Olaf Hering wrote:
> On Tue, Dec 02, Ian Campbell wrote:
> 
> > On Mon, 2014-12-01 at 23:41 +0000, Mark Pryor wrote:
> > > list,
> > 
> > Thanks. If you've identified a buggy changeset then it is fine to post
> > to the devel lists. I've added a CC. I've also CCd everyone listed in
> > the commit which you've fingered.
> > 
> > Olaf, does this suggested change look correct? If so then can you turn
> > it into a patch please.
> 
> Yes, something like this (sed -i 's@socket@service@g' *.in):

Can you submit to -devel with a commit log and S-o-b etc please.

> 
> 
> diff --git a/tools/hotplug/Linux/systemd/xen-init-dom0.service.in b/tools/hotplug/Linux/systemd/xen-init-dom0.service.in
> index 4d4cb23..3befadc 100644
> --- a/tools/hotplug/Linux/systemd/xen-init-dom0.service.in
> +++ b/tools/hotplug/Linux/systemd/xen-init-dom0.service.in
> @@ -1,7 +1,7 @@
>  [Unit]
>  Description=xen-init-dom0, initialise Dom0 configuration (xenstore nodes, JSON configuration stub)
> -Requires=xenstored.socket proc-xen.mount
> -After=xenstored.socket proc-xen.mount
> +Requires=xenstored.service proc-xen.mount
> +After=xenstored.service proc-xen.mount
>  ConditionPathExists=/proc/xen/capabilities
>  
>  [Service]
> diff --git a/tools/hotplug/Linux/systemd/xen-qemu-dom0-disk-backend.service.in b/tools/hotplug/Linux/systemd/xen-qemu-dom0-disk-backend.service.in
> index 6b9c96e..0a5807a 100644
> --- a/tools/hotplug/Linux/systemd/xen-qemu-dom0-disk-backend.service.in
> +++ b/tools/hotplug/Linux/systemd/xen-qemu-dom0-disk-backend.service.in
> @@ -1,7 +1,7 @@
>  [Unit]
>  Description=qemu for xen dom0 disk backend
> -Requires=proc-xen.mount xenstored.socket
> -After=proc-xen.mount xenstored.socket xenconsoled.service
> +Requires=proc-xen.mount xenstored.service
> +After=proc-xen.mount xenstored.service xenconsoled.service
>  Before=xendomains.service libvirtd.service libvirt-guests.service
>  RefuseManualStop=true
>  ConditionPathExists=/proc/xen/capabilities
> diff --git a/tools/hotplug/Linux/systemd/xenconsoled.service.in b/tools/hotplug/Linux/systemd/xenconsoled.service.in
> index 2c5d99f..cb44cd6 100644
> --- a/tools/hotplug/Linux/systemd/xenconsoled.service.in
> +++ b/tools/hotplug/Linux/systemd/xenconsoled.service.in
> @@ -1,7 +1,7 @@
>  [Unit]
>  Description=Xenconsoled - handles logging from guest consoles and hypervisor
> -Requires=proc-xen.mount xenstored.socket
> -After=proc-xen.mount xenstored.socket
> +Requires=proc-xen.mount xenstored.service
> +After=proc-xen.mount xenstored.service
>  ConditionPathExists=/proc/xen/capabilities
>  
>  [Service]
> diff --git a/tools/hotplug/Linux/systemd/xendomains.service.in b/tools/hotplug/Linux/systemd/xendomains.service.in
> index 757278f..9962671 100644
> --- a/tools/hotplug/Linux/systemd/xendomains.service.in
> +++ b/tools/hotplug/Linux/systemd/xendomains.service.in
> @@ -1,7 +1,7 @@
>  [Unit]
>  Description=Xendomains - start and stop guests on boot and shutdown
> -Requires=proc-xen.mount xenstored.socket
> -After=proc-xen.mount xenstored.socket xenconsoled.service xen-init-dom0.service
> +Requires=proc-xen.mount xenstored.service
> +After=proc-xen.mount xenstored.service xenconsoled.service xen-init-dom0.service
>  ConditionPathExists=/proc/xen/capabilities
>  
>  [Service]
> 
> Olaf
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Olaf Hering]

On Tue, Dec 02, Ian Campbell wrote:

> On Tue, 2014-12-02 at 16:17 +0100, Olaf Hering wrote:
> > On Tue, Dec 02, Ian Campbell wrote:
> > 
> > > On Mon, 2014-12-01 at 23:41 +0000, Mark Pryor wrote:
> > > > list,
> > > 
> > > Thanks. If you've identified a buggy changeset then it is fine to post
> > > to the devel lists. I've added a CC. I've also CCd everyone listed in
> > > the commit which you've fingered.
> > > 
> > > Olaf, does this suggested change look correct? If so then can you turn
> > > it into a patch please.
> > 
> > Yes, something like this (sed -i 's@socket@service@g' *.in):
> 
> Can you submit to -devel with a commit log and S-o-b etc please.

Yes, I did already. Looks like git send-email does not process a
Reported-by tag so Mark did not actually got a copy of that submission.
But the actual patch did not change so what I sent him earlier can be
used as is.

Olaf

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Olaf Hering]

On Tue, Dec 02, Olaf Hering wrote:

> On Tue, Dec 02, Ian Campbell wrote:
> 
> > On Mon, 2014-12-01 at 23:41 +0000, Mark Pryor wrote:
> > > list,
> > 
> > Thanks. If you've identified a buggy changeset then it is fine to post
> > to the devel lists. I've added a CC. I've also CCd everyone listed in
> > the commit which you've fingered.
> > 
> > Olaf, does this suggested change look correct? If so then can you turn
> > it into a patch please.
> 
> Yes, something like this (sed -i 's@socket@service@g' *.in):

But even with that change xendomains is hanging if it cant talk to
xenstored for whatever  reason. The result is that the sytem hangs
forever at shutdown. 

I will try to fix that for 4.5.

Olaf

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Olaf Hering]

On Fri, Dec 05, Olaf Hering wrote:

> But even with that change xendomains is hanging if it cant talk to
> xenstored for whatever  reason. The result is that the sytem hangs
> forever at shutdown. 

This looks like a bug in the tools. xl or xenstore-ls hang forever if xenstored
disappears. Looks like xs_open() is the culprit. The socket functions fail,
then it attempts to use xs_domain_dev() which happens to succeed.  Only
xenstore-ls has the '-s' option to force operating on a socket.  I wonder why
libxl does not pass XS_OPEN_SOCKETONLY? It just uses the wrappers for xs_open.

Can xenstored run in another domain?

Olaf

18:02:43.522767 stat("/var/run/xenstored/socket", {st_mode=S_IFSOCK|0600, st_size=0, ...}) = 0
18:02:43.522864 socket(PF_LOCAL, SOCK_STREAM, 0) = 3
18:02:43.522964 fcntl(3, F_GETFD)       = 0
18:02:43.523035 fcntl(3, F_SETFD, FD_CLOEXEC) = 0
18:02:43.523117 connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/xenstored/socket"}, 110) = -1 ECONNREFUSED (Connection refused)
18:02:43.523224 close(3)                = 0
18:02:43.523305 stat("/proc/xen/xenbus", {st_mode=S_IFREG|0400, st_size=0, ...}) = 0
18:02:43.523393 open("/proc/xen/xenbus", O_RDWR) = 3
18:02:43.523562 brk(0)                  = 0x250c000
18:02:43.523638 brk(0x252d000)          = 0x252d000
18:02:43.523748 rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER, 0x7fc9518e3200}, {SIG_DFL, [], 0}, 8) = 0
18:02:43.523834 write(3, "\1\0\0\0\0\0\0\0\0\0\0\0\2\0\0\0", 16) = 16
18:02:43.523938 write(3, "/\0", 2

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Wei Liu]

On Mon, Dec 08, 2014 at 06:19:49PM +0100, Olaf Hering wrote:
> On Fri, Dec 05, Olaf Hering wrote:
> 
> > But even with that change xendomains is hanging if it cant talk to
> > xenstored for whatever  reason. The result is that the sytem hangs
> > forever at shutdown. 
> 
> This looks like a bug in the tools. xl or xenstore-ls hang forever if xenstored
> disappears. Looks like xs_open() is the culprit. The socket functions fail,
> then it attempts to use xs_domain_dev() which happens to succeed.  Only
> xenstore-ls has the '-s' option to force operating on a socket.  I wonder why
> libxl does not pass XS_OPEN_SOCKETONLY? It just uses the wrappers for xs_open.
> 
> Can xenstored run in another domain?
> 

I think we have xenstored stubdom in tree, don't we?

> Olaf
> 
> 18:02:43.522767 stat("/var/run/xenstored/socket", {st_mode=S_IFSOCK|0600, st_size=0, ...}) = 0
> 18:02:43.522864 socket(PF_LOCAL, SOCK_STREAM, 0) = 3
> 18:02:43.522964 fcntl(3, F_GETFD)       = 0
> 18:02:43.523035 fcntl(3, F_SETFD, FD_CLOEXEC) = 0
> 18:02:43.523117 connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/xenstored/socket"}, 110) = -1 ECONNREFUSED (Connection refused)
> 18:02:43.523224 close(3)                = 0
> 18:02:43.523305 stat("/proc/xen/xenbus", {st_mode=S_IFREG|0400, st_size=0, ...}) = 0
> 18:02:43.523393 open("/proc/xen/xenbus", O_RDWR) = 3
> 18:02:43.523562 brk(0)                  = 0x250c000
> 18:02:43.523638 brk(0x252d000)          = 0x252d000
> 18:02:43.523748 rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER, 0x7fc9518e3200}, {SIG_DFL, [], 0}, 8) = 0
> 18:02:43.523834 write(3, "\1\0\0\0\0\0\0\0\0\0\0\0\2\0\0\0", 16) = 16
> 18:02:43.523938 write(3, "/\0", 2

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Olaf Hering]

On Mon, Dec 08, Wei Liu wrote:

> I think we have xenstored stubdom in tree, don't we?

If we do, isnt that a "special" setup which does not appear all by
its own? If so, shouldnt there be some config file knob to let xs_open
use socket only unless told otherwise by the to-be-created knob?

Olaf

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Wei Liu]

On Mon, Dec 08, 2014 at 06:37:08PM +0100, Olaf Hering wrote:
> On Mon, Dec 08, Wei Liu wrote:
> 
> > I think we have xenstored stubdom in tree, don't we?
> 
> If we do, isnt that a "special" setup which does not appear all by
> its own? If so, shouldnt there be some config file knob to let xs_open
> use socket only unless told otherwise by the to-be-created knob?
> 

At the very least it needs to run init-xenstore-domain to set it up. I
have no idea how this supposes to work though. Maybe other people have
better idea.

However I think the problem you're seeing is due to xenstored exits too
early, which should be fixable by rearranging the shutdown order? I.e.
xenstored shuts down after xendomains.

Wei.

> Olaf

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Olaf Hering]

On Mon, Dec 08, Wei Liu wrote:

> However I think the problem you're seeing is due to xenstored exits too
> early, which should be fixable by rearranging the shutdown order? I.e.
> xenstored shuts down after xendomains.

True, and this is addressed by 268b8f1.

Perhaps a crashed or otherwise unavailable xenstored isnt such a problem
because its like that since a very long time.

Olaf

Re: [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS

[Ian Jackson]

Olaf Hering writes ("Re: [Xen-devel] [Xen-users] 4.5 git: regression in xen systemd shutdown hangs the OS"):
> Perhaps a crashed or otherwise unavailable xenstored isnt such a problem
> because its like that since a very long time.

Without xenstored, the system is basically hosed.  And it's not really
restartable.

Ian.