Re: reboot driver domain, vifX.Y = NO-CARRIER?

[Jason Cooper <xen@lakedaemon.net>]

Morning George,

On Tue, May 01, 2018 at 11:25:06AM +0100, George Dunlap wrote:
> On Mon, Apr 30, 2018 at 7:17 PM, Jason Cooper <xen@lakedaemon.net> wrote:
> > On Mon, Apr 30, 2018 at 05:38:55PM +0100, George Dunlap wrote:
> >> On Mon, Apr 30, 2018 at 5:16 PM, Jason Cooper <xen@lakedaemon.net> wrote:
> >> > On Mon, Apr 30, 2018 at 04:22:30PM +0100, Ian Jackson wrote:
> >> >> Wei Liu writes ("Re: [Xen-devel] reboot driver domain, vifX.Y = NO-CARRIER?"):
> >> >> > To implement reuse_domid in a sane way, either the toolstack needs to
> >> >> > manage all domids and always sets domid when creating domain or the
> >> >> > hypervisor needs to cooperate -- to have interface to reserve /
> >> >> > pre-allocate domids.
> >> >>
> >> >> I think this is entirely the wrong approach.
> >> >
> >> > Whew.  Glad I didn't start hacking yet...
> >> >
> >> >> I think the right answer is that this is simply a bug in the
> >> >> frontends.  frontends should cope if the backend path pointer in the
> >> >> frontend directory is updated, and should start reading the new
> >> >> backend instead.
> >> >
> >> > Ok, so I'm new to the guts of Xen.  The bug, at a high level, is that
> >> > "When a driver domain is rebooted (domid changed), previously connected
> >> > client domUs can't gain network connectivity to/through the driver
> >> > domain via 'xl network-attach client_domu mac=... bridge=...
> >> > backend=drv_dom'"
> >>
> >> Hang on -- just to clarify, something like the following doesn't work
> >> (or wouldn't, you suspect, work)?
> >>
> >> * Start driver domain
> >> * Start domU A with no network
> >
> > My setup is different here.  I include the vif = [... backend=...]
> > declaration in my domain config.
> >
> >> * xl network-attach A backend=drv_dom
> >
> > So I don't do this step manually.
> 
> Right, but you do the detach manually (as well as the subsequent
> attach after the driver domain
> 
> >
> >> * [do some stuff]
> >> * xl network-detach A [network devid]
> >> * Restart driver domain
> >> * xl network-attach A backend=drv_dom
> [snip]
> > Sorry, I get NO-CARRIER in the just rebooted driver domain.  And the
> > interface is still UP in domU A.
> 
> Wait, that sounds like a different problem than the one we thought you
> were talking about.  You're saying that the driver domain is losing
> connection to the *physical* network after reboot?

No, this has nothing to do with the physical nic that is
pic-passthrough'd.  It's as my subject line says: vifX.Y gets
NO-CARRIER.  Here's a snippet from 'ip link'

12: vif20.1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br10 qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff

> So what happens if you do the following:
> 
> * Boot your driver domain (but don't connect any guests)
> * From your driver domain, ping an off-host IP
> * Reboot the driver domain
> * Try pinging an off-host IP again
> 
> It sounds like maybe the second ping will fail?

I assume this is for debugging the (hopefully clarified) non-existent
problem with pci-passthrough.  fwiw, this particular driver domain is in
the middle of the diagram I did earlier in the thread.  It's a netfront
client to a driver domain which does have the pci-passthrough.

When I was first digging into this, I started a thread on xen-users [1],
I've attached my xl-reboot.sh script here so you can see exactly what
I'm attempting to do:

------------------------------->8----------------------------------------
#!/bin/bash

if [ $# -ne 1 ]; then
	echo >&2 "Usage: ${0##*/} domain"
	exit 1
fi

DOM="$1"

# get the domain id
DOMID="`xl domid $DOM`"
[[ "$DOMID" =~ (^[0-9]+$) ]] || exit 1

tmp="`mktemp`"

# loop through frontends
while read frontend <&4; do
	while read vif <&5; do
		if [ "x$vif" = "x" ]; then
			# stale frontend
			echo >&2 "WARN: stale frontend ($frontend), removing"
			xenstore-rm /local/domain/$DOMID/backend/vif/$frontend
			continue
		fi

		# store info for afterwards
		front="`xl domname $frontend`"
		bridge="`xenstore-read /local/domain/$DOMID/backend/vif/$frontend/$vif/bridge`"
		if [ "x$front" != "x" ] && [[ "$bridge" =~ (br[0-9][0-9]*) ]]; then
			echo "$front bridge=$bridge backend=$DOM" >>"$tmp"

			# remove the vif
			echo >&2 "Removing $vif from $front"
			xl -f network-detach $front $vif
		fi
	done 5< <(xenstore-list /local/domain/$DOMID/backend/vif/$frontend)
done 4< <(xenstore-list /local/domain/$DOMID/backend/vif)

# reboot the domain
xl shutdown -w $DOM || exit 2
sleep 1
xl create -c $DOM || exit 3

if [ "`cat $tmp | wc -c`" -eq 0 ]; then
	rm -f $tmp
	exit 0
fi

# reattach everything
echo
while read ln <&4; do
	echo >&2 "re-attach [$ln]"
	xl network-attach $ln || exit 4
done 4< <(cat $tmp)

rm -f $tmp

exit 0
------------------------------->8----------------------------------------

thx,

Jason.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel