4.7 qemu regression: HVM guests fail to boot from xvda

4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

With staging-4.6 this domU boots from xvda, qemu creates an emulated
disk. With staging no disk is found, unless the name is changed to hda.
Looks like qemu-2.6 does not handle xvda either.

Is such setup not part of OSS test, does OSS default to 'vdev=hda'?

Olaf

name="domU"
uuid="64dfa382-6365-46aa-88d8-0d9ca9a09ead"
memory=1024
serial="pty"
builder="hvm"
boot="cd"
#dtype="ahci"
disk=[
        'format=qcow2, vdev=xvda, access=rw, target=/disk0.qcow2',
]
keymap="de"
vfb = [
        'type=vnc,vncunused=1,keymap=de'
]
usb=1
usbdevice='tablet'


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On Mon, May 30, 2016 at 9:42 PM, Olaf Hering <olaf@aepfle.de> wrote:
> With staging-4.6 this domU boots from xvda, qemu creates an emulated
> disk. With staging no disk is found, unless the name is changed to hda.
> Looks like qemu-2.6 does not handle xvda either.

This was intentional; see this thread:

https://marc.info/?i=<1444820717-25565-1-git-send-email-anthony.perard@citrix.com>

The idea was to make it possible to create an HVM guest with no
emulated disks for guest booting with OVMF which contains PV drivers.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Tue, May 31, George Dunlap wrote:

> On Mon, May 30, 2016 at 9:42 PM, Olaf Hering <olaf@aepfle.de> wrote:
> > With staging-4.6 this domU boots from xvda, qemu creates an emulated
> > disk. With staging no disk is found, unless the name is changed to hda.
> > Looks like qemu-2.6 does not handle xvda either.
> 
> This was intentional; see this thread:
> 
> https://marc.info/?i=<1444820717-25565-1-git-send-email-anthony.perard@citrix.com>
> 
> The idea was to make it possible to create an HVM guest with no
> emulated disks for guest booting with OVMF which contains PV drivers.

This breaks the domU becasue it changes its device names from 'xvd' to
'hd' if a xenlinux based kernel is used in domU.
In other words: every SUSE domU out there.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On Tue, May 31, 2016 at 12:16 PM, Olaf Hering <olaf@aepfle.de> wrote:
> On Tue, May 31, George Dunlap wrote:
>
>> On Mon, May 30, 2016 at 9:42 PM, Olaf Hering <olaf@aepfle.de> wrote:
>> > With staging-4.6 this domU boots from xvda, qemu creates an emulated
>> > disk. With staging no disk is found, unless the name is changed to hda.
>> > Looks like qemu-2.6 does not handle xvda either.
>>
>> This was intentional; see this thread:
>>
>> https://marc.info/?i=<1444820717-25565-1-git-send-email-anthony.perard@citrix.com>
>>
>> The idea was to make it possible to create an HVM guest with no
>> emulated disks for guest booting with OVMF which contains PV drivers.
>
> This breaks the domU becasue it changes its device names from 'xvd' to
> 'hd' if a xenlinux based kernel is used in domU.
> In other words: every SUSE domU out there.

Sorry, can you expand on this a bit?  Are you saying that on SuSE, if
you specify "vdev=xvda" in your config file, that you'll get PV
devices named "/dev/xvda", but that if you specify "vdev=hda", that
you'll get PV devices but named "/dev/hda"?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Tue, May 31, George Dunlap wrote:

> Sorry, can you expand on this a bit?  Are you saying that on SuSE, if
> you specify "vdev=xvda" in your config file, that you'll get PV
> devices named "/dev/xvda", but that if you specify "vdev=hda", that
> you'll get PV devices but named "/dev/hda"?

Yes, thats exactly what the xenlinux block frontend does.
pvops forces xvda, independent of the name 'vdev' in domU.cfg.
Up to xen-4.2 'vdev=hd*' was required to tell qemu to create an emulated
disk to boot from. Starting with xen-4.3 qemu also recognized
'vdev=xvd*' for the emulated disk. And starting with xen-4.7 qemu
requires 'xvda=hd*' again.

I think if some domU.cfg for xen-4.3+ has 'vdev=xvd*' and the domU uses
for some reason kernel names in config files and the domU uses a
xenlinux kernel, then changing domU.cfg to 'hd*' will allow the guest to
boot again. But its userland will miss the /dev/xvd* device nodes.
That probably remained unnoticed during testing the referenced commit if
a pvops based kernel was used.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On Tue, May 31, 2016 at 12:41 PM, Olaf Hering <olaf@aepfle.de> wrote:
> On Tue, May 31, George Dunlap wrote:
>
>> Sorry, can you expand on this a bit?  Are you saying that on SuSE, if
>> you specify "vdev=xvda" in your config file, that you'll get PV
>> devices named "/dev/xvda", but that if you specify "vdev=hda", that
>> you'll get PV devices but named "/dev/hda"?
>
> Yes, thats exactly what the xenlinux block frontend does.
> pvops forces xvda, independent of the name 'vdev' in domU.cfg.
> Up to xen-4.2 'vdev=hd*' was required to tell qemu to create an emulated
> disk to boot from. Starting with xen-4.3 qemu also recognized
> 'vdev=xvd*' for the emulated disk. And starting with xen-4.7 qemu
> requires 'xvda=hd*' again.
>
> I think if some domU.cfg for xen-4.3+ has 'vdev=xvd*' and the domU uses
> for some reason kernel names in config files and the domU uses a
> xenlinux kernel, then changing domU.cfg to 'hd*' will allow the guest to
> boot again. But its userland will miss the /dev/xvd* device nodes.
> That probably remained unnoticed during testing the referenced commit if
> a pvops based kernel was used.

Or if -- as is the case for most of my own test systems -- filesystem
UUIDs are used rather than device names.  (This means things work the
same on PV with PV disks, HVM with PV disks, and HVM with emulated
disks -- for instance, if you're using nested virtualization and your
L1 dom0 can't access L0 xenbus.)

Do you have a concrete proposal?

Anthony, does the OVMF-with-pv-only-drivers actually still work at the moment?

Really 'vdev' string in the the guest config file is only meant to
tell libxl how it should behave -- it should ideally not have any
effect on what devices you see in the backend.  And furthermore, it
seems to me that when Linux upstream rejected the idea of the pv
drivers stealing the "hd*" namespace, that SuSE's xenlinux should have
followed suit and had the pv drivers only create devices named xvd*.

But I recognize if you're selling an enterprise kernel, those sorts of
"you should have done this X years ago" doesn't really help you keep
your promises to your users now. :-)

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Tue, May 31, George Dunlap wrote:

> Do you have a concrete proposal?

I have to give it some more thought what the impact really is, what
config variants are affected.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Tue, May 31, Olaf Hering wrote:

> On Tue, May 31, George Dunlap wrote:
> 
> > Do you have a concrete proposal?
> 
> I have to give it some more thought what the impact really is, what
> config variants are affected.

Here is a list. Essentially it means that upgrading dom0 to xen-4.7
might break existing domU if they happen to use xvd* in domU.cfg:

 device names which the emulated BIOS can boot from:
  num       tool_ver        vdev_str        bootable
  01        xen-4.2           hd              yes
  02        xen-4.2           xvd             no
  03        xen-4.3           hd              yes
  04        xen-4.3           xvd             yes
  05        xen-4.4           hd              yes
  06        xen-4.4           xvd             yes
  07        xen-4.5           hd              yes
  08        xen-4.5           xvd             yes
  09        xen-4.6           hd              yes
  10        xen-4.6           xvd             yes
  11        xen-4.7-rc        hd              yes
  12        xen-4.7-rc        xvd             no

 block frontend drivers which will use the vdev_str:
  a) pvops    no  (defaults to xvd)
  b) xenlinux yes
  c) solaris  ??? (Konrad suggests yes)
  d) bsd      ???
  e) windows  ??? (likely no due to drive letters)

domU.cfg configuration which will break with xen-4.7:
04|06|08|10 -> 12
after adjusting to 11 to allow boot again:
b + 11

In real life this means existing SLE11/SLE12/openSUSE domU with
vdev_str==xvd when dom0 is upgraded to xen-4.7, and the domU makes use
of kernel device names.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Wed, Jun 01, Olaf Hering wrote:

> Here is a list. Essentially it means that upgrading dom0 to xen-4.7
> might break existing domU if they happen to use xvd* in domU.cfg:

Actually the list goes like shown below after some real testing.
Now sure why xvd/qemu-xen/raw|qcow2 fails with our 4.5.2 based dom0.
In the end the regression remains in xvd+qemu-xen.

tool_ver  vdev_str  qemu   format  usable  bootable
xen-4.5     xvd     qtrad    raw    yes     no
xen-4.5     xvd     qtrad   qcow2   no      -
xen-4.5     xvd     qmain    raw    yes     SUSE:no, staging:yes
xen-4.5     xvd     qmain   qcow2   yes     SUSE:no, staging:yes
xen-4.5     hd      qtrad    raw    yes     yes
xen-4.5     hd      qtrad   qcow2   no      -
xen-4.5     hd      qmain    raw    yes     yes
xen-4.5     hd      qmain   qcow2   yes     yes


xen-4.6     xvd     qtrad    raw    yes     no
xen-4.6     xvd     qtrad   qcow2   no      -
xen-4.6     xvd     qmain    raw    yes     yes
xen-4.6     xvd     qmain   qcow2   yes     yes
xen-4.6     hd      qtrad    raw    yes     yes
xen-4.6     hd      qtrad   qcow2   no      -
xen-4.6     hd      qmain    raw    yes     yes
xen-4.6     hd      qmain   qcow2   yes     yes


xen-4.7     xvd     qtrad    raw    yes     no
xen-4.7     xvd     qtrad   qcow2   no      -
xen-4.7     xvd     qmain    raw    yes     no
xen-4.7     xvd     qmain   qcow2   yes     no
xen-4.7     hd      qtrad    raw    yes     yes
xen-4.7     hd      qtrad   qcow2   no      -
xen-4.7     hd      qmain    raw    yes     yes
xen-4.7     hd      qmain   qcow2   yes     yes

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Wei Liu]

On Wed, Jun 01, 2016 at 11:40:51PM +0200, Olaf Hering wrote:
> On Wed, Jun 01, Olaf Hering wrote:
> 
> > Here is a list. Essentially it means that upgrading dom0 to xen-4.7
> > might break existing domU if they happen to use xvd* in domU.cfg:
> 
> Actually the list goes like shown below after some real testing.
> Now sure why xvd/qemu-xen/raw|qcow2 fails with our 4.5.2 based dom0.
> In the end the regression remains in xvd+qemu-xen.
> 
> tool_ver  vdev_str  qemu   format  usable  bootable
> xen-4.5     xvd     qtrad    raw    yes     no
> xen-4.5     xvd     qtrad   qcow2   no      -
> xen-4.5     xvd     qmain    raw    yes     SUSE:no, staging:yes
> xen-4.5     xvd     qmain   qcow2   yes     SUSE:no, staging:yes

What does "staging" mean?

> xen-4.5     hd      qtrad    raw    yes     yes
> xen-4.5     hd      qtrad   qcow2   no      -
> xen-4.5     hd      qmain    raw    yes     yes
> xen-4.5     hd      qmain   qcow2   yes     yes
> 
> 
> xen-4.6     xvd     qtrad    raw    yes     no
> xen-4.6     xvd     qtrad   qcow2   no      -
> xen-4.6     xvd     qmain    raw    yes     yes
> xen-4.6     xvd     qmain   qcow2   yes     yes
> xen-4.6     hd      qtrad    raw    yes     yes
> xen-4.6     hd      qtrad   qcow2   no      -
> xen-4.6     hd      qmain    raw    yes     yes
> xen-4.6     hd      qmain   qcow2   yes     yes
> 
> 
> xen-4.7     xvd     qtrad    raw    yes     no
> xen-4.7     xvd     qtrad   qcow2   no      -
> xen-4.7     xvd     qmain    raw    yes     no
> xen-4.7     xvd     qmain   qcow2   yes     no
> xen-4.7     hd      qtrad    raw    yes     yes
> xen-4.7     hd      qtrad   qcow2   no      -
> xen-4.7     hd      qmain    raw    yes     yes
> xen-4.7     hd      qmain   qcow2   yes     yes
> 

So the last column only apply to xenlinux kernel?

I expect pvops to be all "yes" because it doesn't use vdev_str. I also
expect freebsd to work because it parses vbd number and translates that
to freebsd-speak device identifier.

Wei.

> Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Thu, Jun 02, Wei Liu wrote:

> On Wed, Jun 01, 2016 at 11:40:51PM +0200, Olaf Hering wrote:
> > tool_ver  vdev_str  qemu   format  usable  bootable
> > xen-4.5     xvd     qmain   qcow2   yes     SUSE:no, staging:yes
> What does "staging" mean?

That should have been staging-4.5.
Some change between our 4.5.1 and 4.5.2 based toolstack broke xvd booting.

> > xen-4.6     xvd     qmain    raw    yes     yes
> > xen-4.6     xvd     qmain   qcow2   yes     yes

> > xen-4.7     xvd     qmain    raw    yes     no
> > xen-4.7     xvd     qmain   qcow2   yes     no

> So the last column only apply to xenlinux kernel?

The relevant row is 'bootable', thats where the regression is. It means
'BIOS finds a disk, can access it and run bootloader'. In my testing the
BIOS did not see a disk, just the 'cdrom' had in domU.cfg as well.

'usable' means a PV driver may actually access the given device. This is
always the case, except qemu-trad has no qcow2 support.

> I expect pvops to be all "yes" because it doesn't use vdev_str. I also
> expect freebsd to work because it parses vbd number and translates that
> to freebsd-speak device identifier.

This list has no column 'makes use of vdev_str' because the list does not
describe domU OS behaviour. ;-)

See my other mail regarding domU os. If we leave xen-4.7 as is then
domU.cfg has to be changed from 'xvd' to 'hd' to allow BIOS boot again.
This will change things for (at least) xenlinux based kernels because
the kernel device name change. I think this needs adjustments to
/boot/grub/device.map and /etc/default/grub_installdevice. The latter is
an odd SUSE thing, describing where grub2 should write things, its
likely similar to device.map.

fstab and root= can be changed to use on-disk identifiers like UUID or
LABEL to work with either xvd or hd. LVM or raid might be unaffected.

Eventually there are more places which need adjustment, like autoinstall
profiles.


Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Jan Beulich]

>>> On 31.05.16 at 14:00, <george.dunlap@citrix.com> wrote:
> Really 'vdev' string in the the guest config file is only meant to
> tell libxl how it should behave -- it should ideally not have any
> effect on what devices you see in the backend.  And furthermore, it
> seems to me that when Linux upstream rejected the idea of the pv
> drivers stealing the "hd*" namespace, that SuSE's xenlinux should have
> followed suit and had the pv drivers only create devices named xvd*.
> 
> But I recognize if you're selling an enterprise kernel, those sorts of
> "you should have done this X years ago" doesn't really help you keep
> your promises to your users now. :-)

Not just that: We'd have had the problems converting guests
back then which we're having now. Yet I think such problems
shouldn't have got introduced in the first place. But yes, since
when would Linux maintainers care much about Xen-specific
issues... (And all of this is not to say that I'm convinced this
stealing of hd* and sd* name spaces was a good idea.)

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Wei Liu]

On Tue, May 31, 2016 at 01:00:45PM +0100, George Dunlap wrote:
> On Tue, May 31, 2016 at 12:41 PM, Olaf Hering <olaf@aepfle.de> wrote:
> > On Tue, May 31, George Dunlap wrote:
> >
> >> Sorry, can you expand on this a bit?  Are you saying that on SuSE, if
> >> you specify "vdev=xvda" in your config file, that you'll get PV
> >> devices named "/dev/xvda", but that if you specify "vdev=hda", that
> >> you'll get PV devices but named "/dev/hda"?
> >
> > Yes, thats exactly what the xenlinux block frontend does.
> > pvops forces xvda, independent of the name 'vdev' in domU.cfg.
> > Up to xen-4.2 'vdev=hd*' was required to tell qemu to create an emulated
> > disk to boot from. Starting with xen-4.3 qemu also recognized
> > 'vdev=xvd*' for the emulated disk. And starting with xen-4.7 qemu
> > requires 'xvda=hd*' again.
> >
> > I think if some domU.cfg for xen-4.3+ has 'vdev=xvd*' and the domU uses
> > for some reason kernel names in config files and the domU uses a
> > xenlinux kernel, then changing domU.cfg to 'hd*' will allow the guest to
> > boot again. But its userland will miss the /dev/xvd* device nodes.
> > That probably remained unnoticed during testing the referenced commit if
> > a pvops based kernel was used.
> 
> Or if -- as is the case for most of my own test systems -- filesystem
> UUIDs are used rather than device names.  (This means things work the
> same on PV with PV disks, HVM with PV disks, and HVM with emulated
> disks -- for instance, if you're using nested virtualization and your
> L1 dom0 can't access L0 xenbus.)
> 
> Do you have a concrete proposal?
> 
> Anthony, does the OVMF-with-pv-only-drivers actually still work at the moment?
> 

It works. We got pushes to our ovmf tree regularly (only now it fails on
merlot).

Having an extra emulated device shouldn't break OVMF, we've had that for
a long time before that patch was applied.

So I think the safest option now is to revert the said patch and figure
out what to do next.

Anthony and Ian, what do you think?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Wed, Jun 01, Wei Liu wrote:

> So I think the safest option now is to revert the said patch and figure
> out what to do next.

What did OSStest report when c0c099d got into the tree? Do these test
domU.cfg files contain vdev=hd instead of vdev=xvd, so the boot breakage
was not noticed?

What are the options we have to define a disk connected to an PIIX or
AHCI controller? I see xl.cfg has hdtype=. Is vdev hd vs. xvd really the
best way to describe a PV-only disk? It looks like the answer is yes.
hdtype= describes the controller variant, and vdev= the disk variant.

So in the end it needs to be documented properly that moving dom0 to
xen-4.7 requires adjustments to vdev= in domU.cfg (xvd -> hd), and that
this MAY have an impact for domU frontend drivers which use the vdev
string as is.
Namely this affects SLE11, SLE12, SLE12SP1, openSUSE up to Leap 42.1.
Upcoming releases use a pvops based kernel, so the device names are
fixed.
Konrad mentioned Solaris, no idea if its frontend driver uses the vdev
string. BSD should be checked as well.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Wei Liu]

On Wed, Jun 01, 2016 at 03:34:08PM +0200, Olaf Hering wrote:
> On Wed, Jun 01, Wei Liu wrote:
> 
> > So I think the safest option now is to revert the said patch and figure
> > out what to do next.
> 
> What did OSStest report when c0c099d got into the tree? Do these test
> domU.cfg files contain vdev=hd instead of vdev=xvd, so the boot breakage
> was not noticed?
> 

OSStest always has hda for hvm guest.

> What are the options we have to define a disk connected to an PIIX or
> AHCI controller? I see xl.cfg has hdtype=. Is vdev hd vs. xvd really the
> best way to describe a PV-only disk? It looks like the answer is yes.

I would think so.

> hdtype= describes the controller variant, and vdev= the disk variant.
> 

Yes, you're correct. hdtype= is for controller.

> So in the end it needs to be documented properly that moving dom0 to
> xen-4.7 requires adjustments to vdev= in domU.cfg (xvd -> hd), and that
> this MAY have an impact for domU frontend drivers which use the vdev
> string as is.
> Namely this affects SLE11, SLE12, SLE12SP1, openSUSE up to Leap 42.1.
> Upcoming releases use a pvops based kernel, so the device names are
> fixed.
> Konrad mentioned Solaris, no idea if its frontend driver uses the vdev
> string. BSD should be checked as well.
> 

So you are fine with documenting without reverting the patch?

Wei.

> Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Wed, Jun 01, Wei Liu wrote:

> > Konrad mentioned Solaris, no idea if its frontend driver uses the vdev
> > string. BSD should be checked as well.
> > 
> 
> So you are fine with documenting without reverting the patch?

I think yes, If Solaris and BSD domU are not affected by the change.
Given that hdtype= is for controller type and vdev= for disk type, one
has to live with the possible breakage.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
> On Tue, May 31, 2016 at 12:41 PM, Olaf Hering <olaf@aepfle.de> wrote:
> > I think if some domU.cfg for xen-4.3+ has 'vdev=xvd*' and the domU uses
> > for some reason kernel names in config files and the domU uses a
> > xenlinux kernel, then changing domU.cfg to 'hd*' will allow the guest to
> > boot again. But its userland will miss the /dev/xvd* device nodes.
> > That probably remained unnoticed during testing the referenced commit if
> > a pvops based kernel was used.

Linux doctrine, at least, nowadays, is that hd* device names are not
stable.  On my own colo machine I find that on some boots the actual
hard disks are sda and sdb and the emergency usb stick is sdc, but on
other boots the disks are sdb and sdc and the usb stick is sda.  So
some would say that what you are doing is inherently unstable.

But I don't think I really agree with that view.  I think we should be
able to do better.

> Really 'vdev' string in the the guest config file is only meant to
> tell libxl how it should behave -- it should ideally not have any
> effect on what devices you see in the backend.

The vdev specifies the virtual block number.  See vbd-interface.txt.
hda and xvda have different numbers.

>  And furthermore, it
> seems to me that when Linux upstream rejected the idea of the pv
> drivers stealing the "hd*" namespace, that SuSE's xenlinux should have
> followed suit and had the pv drivers only create devices named xvd*.

Right.  This is the root cause.  vbd-interface.txt is designed to cope
with modern PV frontends which never steal the hda minor numbers.

I think we should fix this with a syntax for explicitly specifying a
pv-only device with the hd* pv block number.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On 01/06/16 16:36, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
>> On Tue, May 31, 2016 at 12:41 PM, Olaf Hering <olaf@aepfle.de> wrote:
>>> I think if some domU.cfg for xen-4.3+ has 'vdev=xvd*' and the domU uses
>>> for some reason kernel names in config files and the domU uses a
>>> xenlinux kernel, then changing domU.cfg to 'hd*' will allow the guest to
>>> boot again. But its userland will miss the /dev/xvd* device nodes.
>>> That probably remained unnoticed during testing the referenced commit if
>>> a pvops based kernel was used.
> 
> Linux doctrine, at least, nowadays, is that hd* device names are not
> stable.  On my own colo machine I find that on some boots the actual
> hard disks are sda and sdb and the emergency usb stick is sdc, but on
> other boots the disks are sdb and sdc and the usb stick is sda.  So
> some would say that what you are doing is inherently unstable.
> 
> But I don't think I really agree with that view.  I think we should be
> able to do better.
> 
>> Really 'vdev' string in the the guest config file is only meant to
>> tell libxl how it should behave -- it should ideally not have any
>> effect on what devices you see in the backend.
> 
> The vdev specifies the virtual block number.  See vbd-interface.txt.
> hda and xvda have different numbers.
> 
>>  And furthermore, it
>> seems to me that when Linux upstream rejected the idea of the pv
>> drivers stealing the "hd*" namespace, that SuSE's xenlinux should have
>> followed suit and had the pv drivers only create devices named xvd*.
> 
> Right.  This is the root cause.  vbd-interface.txt is designed to cope
> with modern PV frontends which never steal the hda minor numbers.
> 
> I think we should fix this with a syntax for explicitly specifying a
> pv-only device with the hd* pv block number.

Olaf,

Would this be a suitable solution for you, and do you think you
understand the suggestion well enough / have enough time to write up a
patch?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Fri, Jun 03, George Dunlap wrote:

> On 01/06/16 16:36, Ian Jackson wrote:
> > I think we should fix this with a syntax for explicitly specifying a
> > pv-only device with the hd* pv block number.

> Would this be a suitable solution for you, and do you think you
> understand the suggestion well enough / have enough time to write up a
> patch?

In another mail in this thread it was said that there are two things:
one is the controller type (piix,ahci) and another is the disk type
(connected to emulated hardware, pv-only). The controller type can be
adjusted with 'hdtype=(ide|ahci)'.

I think the regression is: 'vdev=xvda' does not result in a disk
connected to the emulated controller. Should we change the way hdtype=
is handled internally? If hdtype= is not given it remains unset and with
vdev=xvd* no disk-on-emulated-controller gets added. If hdtype= is set
then vdev=xvd* will result in an disk-on-emulated-controller, which
fixes the regression. If vdev=hd* and hdtype= was not set, hdtype will
be silently set to ide.

Would that approach be acceptable? As an alternative adding hdtype=pv
and checking for it in the place changed by c0c099d would be an
alternative way for a pvonly disk.

And I think we are talking only about {hd,xvd}[a-d] anyway, right? Or
would the emulated AHCI controller allow more than 4 disks?

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On 03/06/16 12:20, Olaf Hering wrote:
> On Fri, Jun 03, George Dunlap wrote:
> 
>> On 01/06/16 16:36, Ian Jackson wrote:
>>> I think we should fix this with a syntax for explicitly specifying a
>>> pv-only device with the hd* pv block number.
> 
>> Would this be a suitable solution for you, and do you think you
>> understand the suggestion well enough / have enough time to write up a
>> patch?
> 
> In another mail in this thread it was said that there are two things:
> one is the controller type (piix,ahci) and another is the disk type
> (connected to emulated hardware, pv-only). The controller type can be
> adjusted with 'hdtype=(ide|ahci)'.
> 
> I think the regression is: 'vdev=xvda' does not result in a disk
> connected to the emulated controller. Should we change the way hdtype=
> is handled internally? If hdtype= is not given it remains unset and with
> vdev=xvd* no disk-on-emulated-controller gets added. If hdtype= is set
> then vdev=xvd* will result in an disk-on-emulated-controller, which
> fixes the regression. If vdev=hd* and hdtype= was not set, hdtype will
> be silently set to ide.

I'd be OK with this.  But is the "hdtype unset" also available at the
libxl level?

Although I still think that the XenoLinux kernels should as soon as
posssible try to move to following upstream Linux's practice of not
stealing major numbers of other drivers.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
> On 03/06/16 12:20, Olaf Hering wrote:
> > I think the regression is: 'vdev=xvda' does not result in a disk
> > connected to the emulated controller. Should we change the way hdtype=
> > is handled internally? If hdtype= is not given it remains unset and with
> > vdev=xvd* no disk-on-emulated-controller gets added. If hdtype= is set
> > then vdev=xvd* will result in an disk-on-emulated-controller, which
> > fixes the regression. If vdev=hd* and hdtype= was not set, hdtype will
> > be silently set to ide.
> 
> I'd be OK with this.  But is the "hdtype unset" also available at the
> libxl level?

There are two problems with this `hdtype' approach.

Firstly, it is global.  That is, it applies to all disks of the
particular guest.  But then maybe we don't care about that because
this anomalous major-number-stealing behaviour is probably per-guest
rather than per-disk.

Secondly, the proposal above involves changing both the semantics of
existing `hdtype' parameter values, and the default hdtype value.  The
resulting situation would be that even specifying vdev=hda wouldn't
get you an emulated device, by default, unless you specified `hdtype'
too.  I don't think that is right.

The possibilities I see are:

(1) New boolean per-guest parameter for this behaviour, meaning
   `provide emulated devices for all xvd* as if they were hd*'.

(2) New `hdtype=ideforpv' which has the same effect as `hdtype=ide'
   plus the semantics in (1) above.  (I'm open to better naming
   suggestions.)

(3) New disk property parameter `hvm-emulate' in the Deprecated
    section of xl-disk-configuration.txt.

Open questions:

Do we also need `... as if they were sd*' or `provide ide emulated
devices where vdev=sd* is specified?'

If we have `hdtype=ideinclpv' do we also need `hdtype=ahciinclpv' ?

What should happen if these options are enabled for PV guests - should
they be silently ignored, or should they be rejected ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On 03/06/16 12:45, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
>> On 03/06/16 12:20, Olaf Hering wrote:
>>> I think the regression is: 'vdev=xvda' does not result in a disk
>>> connected to the emulated controller. Should we change the way hdtype=
>>> is handled internally? If hdtype= is not given it remains unset and with
>>> vdev=xvd* no disk-on-emulated-controller gets added. If hdtype= is set
>>> then vdev=xvd* will result in an disk-on-emulated-controller, which
>>> fixes the regression. If vdev=hd* and hdtype= was not set, hdtype will
>>> be silently set to ide.
>>
>> I'd be OK with this.  But is the "hdtype unset" also available at the
>> libxl level?
> 
> There are two problems with this `hdtype' approach.
> 
> Firstly, it is global.  That is, it applies to all disks of the
> particular guest.  But then maybe we don't care about that because
> this anomalous major-number-stealing behaviour is probably per-guest
> rather than per-disk.
> 
> Secondly, the proposal above involves changing both the semantics of
> existing `hdtype' parameter values, and the default hdtype value.  The
> resulting situation would be that even specifying vdev=hda wouldn't
> get you an emulated device, by default, unless you specified `hdtype'
> too.  I don't think that is right.

I don't quite understand this.

First of all, if I make a disk with "vdev=xvda,hdtype=ide", what
happens?  I presume that the 'hdtype' field is effectively ignored?

Secondly, why would the "vdev=hda" behavior change under Olaf's suggestion?

I think what he's proposing (and again this is from a xl.cfg level, not
a libxl level) is this:

* "vdev=xvda": You get only a PV device.  Under both XenoLinux and
upstream Linux your PV device is named 'xvda'.  (No change from existing
semantics.)

* "vdev=hda": You get an emulated IDE "backed" by a PV device.  Under
XenoLinux your PV device is named 'hda'.  Under upstream Linux your PV
device is named 'xvda' (No change from existing semantics.)

* "vdev=xvda,hdtype=ide": You get an emulated IDE backed by a PV device.
 Under both XenoLinux and upstream Linux your PV device is named 'xvda'.
 (This is the only change.)

At a libxl level, the exact same functionality is possible to enable, right?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
> On 03/06/16 12:45, Ian Jackson wrote:
> > Secondly, the proposal above involves changing both the semantics of
> > existing `hdtype' parameter values, and the default hdtype value.  The
> > resulting situation would be that even specifying vdev=hda wouldn't
> > get you an emulated device, by default, unless you specified `hdtype'
> > too.  I don't think that is right.
> 
> I don't quite understand this.
> 
> First of all, if I make a disk with "vdev=xvda,hdtype=ide", what
> happens?  I presume that the 'hdtype' field is effectively ignored?

hdtype is not a field in the disk specification.  It is a global
config option specifying what kind of emulated controller to have.

It affects all disks for which the code determines that an emulated
disk should be provided, but does not affect pv-only disks.

It defaults to `ide'.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On 06/06/16 11:52, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
>> On 03/06/16 12:45, Ian Jackson wrote:
>>> Secondly, the proposal above involves changing both the semantics of
>>> existing `hdtype' parameter values, and the default hdtype value.  The
>>> resulting situation would be that even specifying vdev=hda wouldn't
>>> get you an emulated device, by default, unless you specified `hdtype'
>>> too.  I don't think that is right.
>>
>> I don't quite understand this.
>>
>> First of all, if I make a disk with "vdev=xvda,hdtype=ide", what
>> happens?  I presume that the 'hdtype' field is effectively ignored?
> 
> hdtype is not a field in the disk specification.  It is a global
> config option specifying what kind of emulated controller to have.
> 
> It affects all disks for which the code determines that an emulated
> disk should be provided, but does not affect pv-only disks.
> 
> It defaults to `ide'.

Oh, right; sorry I missed that.

/me goes back and re-reads IanJ's mail in that light

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On 03/06/16 12:45, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
>> On 03/06/16 12:20, Olaf Hering wrote:
>>> I think the regression is: 'vdev=xvda' does not result in a disk
>>> connected to the emulated controller. Should we change the way hdtype=
>>> is handled internally? If hdtype= is not given it remains unset and with
>>> vdev=xvd* no disk-on-emulated-controller gets added. If hdtype= is set
>>> then vdev=xvd* will result in an disk-on-emulated-controller, which
>>> fixes the regression. If vdev=hd* and hdtype= was not set, hdtype will
>>> be silently set to ide.
>>
>> I'd be OK with this.  But is the "hdtype unset" also available at the
>> libxl level?
> 
> There are two problems with this `hdtype' approach.
> 
> Firstly, it is global.  That is, it applies to all disks of the
> particular guest.  But then maybe we don't care about that because
> this anomalous major-number-stealing behaviour is probably per-guest
> rather than per-disk.
> 
> Secondly, the proposal above involves changing both the semantics of
> existing `hdtype' parameter values, and the default hdtype value.  The
> resulting situation would be that even specifying vdev=hda wouldn't
> get you an emulated device, by default, unless you specified `hdtype'
> too.  I don't think that is right.
> 
> The possibilities I see are:
> 
> (1) New boolean per-guest parameter for this behaviour, meaning
>    `provide emulated devices for all xvd* as if they were hd*'.
> 
> (2) New `hdtype=ideforpv' which has the same effect as `hdtype=ide'
>    plus the semantics in (1) above.  (I'm open to better naming
>    suggestions.)
> 
> (3) New disk property parameter `hvm-emulate' in the Deprecated
>     section of xl-disk-configuration.txt.
> 
> Open questions:
> 
> Do we also need `... as if they were sd*' or `provide ide emulated
> devices where vdev=sd* is specified?'
> 
> If we have `hdtype=ideinclpv' do we also need `hdtype=ahciinclpv' ?
> 
> What should happen if these options are enabled for PV guests - should
> they be silently ignored, or should they be rejected ?

In general I like the fact that I can use the same config file and just
switch "builder" to either "generic" or "hvm" and have things Just Work.
 So I'd prefer to have things ignored.  (We might want to  thing about
whether we still want this to be done "silently" or not.)

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On 03/06/16 12:45, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
>> On 03/06/16 12:20, Olaf Hering wrote:
>>> I think the regression is: 'vdev=xvda' does not result in a disk
>>> connected to the emulated controller. Should we change the way hdtype=
>>> is handled internally? If hdtype= is not given it remains unset and with
>>> vdev=xvd* no disk-on-emulated-controller gets added. If hdtype= is set
>>> then vdev=xvd* will result in an disk-on-emulated-controller, which
>>> fixes the regression. If vdev=hd* and hdtype= was not set, hdtype will
>>> be silently set to ide.
>>
>> I'd be OK with this.  But is the "hdtype unset" also available at the
>> libxl level?
> 
> There are two problems with this `hdtype' approach.
> 
> Firstly, it is global.  That is, it applies to all disks of the
> particular guest.  But then maybe we don't care about that because
> this anomalous major-number-stealing behaviour is probably per-guest
> rather than per-disk.
> 
> Secondly, the proposal above involves changing both the semantics of
> existing `hdtype' parameter values, and the default hdtype value.  The
> resulting situation would be that even specifying vdev=hda wouldn't
> get you an emulated device, by default, unless you specified `hdtype'
> too.  I don't think that is right.
> 
> The possibilities I see are:
> 
> (1) New boolean per-guest parameter for this behaviour, meaning
>    `provide emulated devices for all xvd* as if they were hd*'.
> 
> (2) New `hdtype=ideforpv' which has the same effect as `hdtype=ide'
>    plus the semantics in (1) above.  (I'm open to better naming
>    suggestions.)
> 
> (3) New disk property parameter `hvm-emulate' in the Deprecated
>     section of xl-disk-configuration.txt.

Why in the Deprecated section?

The current interface is a bit mad.  I've just been running my CentOS
smoke-testing scripts against packages built against 4.7-rc4.  I've got
bits of the scripts which mount filesystems in dom0; bits of it that do
bits in fstab and so on, and bits of it that actually generate the
config file.

In every part of the whole system -- in dom0, in the guest, in
everything -- I use xvda; *except* in the parts dealing with the guest
config, where for some reason I mysteriously put 'hda', which ends up
producing an xvda either when booted PV or when booted HVM.  Does that
make any sense?

What about a per-disk property, emulate={default,always,only}, which for
HVM will do the things we're talking about and be ignored on PV?
'default' will behave as it does now: xvda will get you only PV, hda
will get you PV-backed emulated.  'always' will always give you an
emulated device even if you specify xvda, and 'only' will only give you
an emulated device (with no PV).

I actually think the default for most people who are not using EFI would
be to include "vdev=xvd?,emulate=always" in most config files for
maximum flexibility and consistency.

 -George


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Tue, Jun 07, George Dunlap wrote:

> In every part of the whole system -- in dom0, in the guest, in
> everything -- I use xvda; *except* in the parts dealing with the guest
> config, where for some reason I mysteriously put 'hda', which ends up
> producing an xvda either when booted PV or when booted HVM.  Does that
> make any sense?

hd|xvd in domU.cfg will be ignored by a pvops kernel, it always uses xvd
as prefix.

> What about a per-disk property, emulate={default,always,only}, which for
> HVM will do the things we're talking about and be ignored on PV?
> 'default' will behave as it does now: xvda will get you only PV, hda
> will get you PV-backed emulated.  'always' will always give you an
> emulated device even if you specify xvda, and 'only' will only give you
> an emulated device (with no PV).

I was hoping to use existing xl.cfg statements so that a given domU.cfg
can be shared across toolstack versions.

I will look into this now, as stated last week.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
> In every part of the whole system -- in dom0, in the guest, in
> everything -- I use xvda; *except* in the parts dealing with the guest
> config, where for some reason I mysteriously put 'hda', which ends up
> producing an xvda either when booted PV or when booted HVM.  Does that
> make any sense?
> 
> What about a per-disk property, emulate={default,always,only}, which for
> HVM will do the things we're talking about and be ignored on PV?
> 'default' will behave as it does now: xvda will get you only PV, hda
> will get you PV-backed emulated.  'always' will always give you an
> emulated device even if you specify xvda, and 'only' will only give you
> an emulated device (with no PV).

This is one of my suggestions which Olaf didn't like.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Fri, Jun 03, Ian Jackson wrote:

> There are two problems with this `hdtype' approach.
> 
> Firstly, it is global.  That is, it applies to all disks of the
> particular guest.  But then maybe we don't care about that because
> this anomalous major-number-stealing behaviour is probably per-guest
> rather than per-disk.

I think there is no issue with hdtype being a global knob. The
controller type for disks is either piix or ahci.

> Secondly, the proposal above involves changing both the semantics of
> existing `hdtype' parameter values, and the default hdtype value.  The
> resulting situation would be that even specifying vdev=hda wouldn't
> get you an emulated device, by default, unless you specified `hdtype'
> too.  I don't think that is right.

IMO vdev= should influence hdtype= if it is unset. Just to make sure the
emulated disk with vdev=xvd can be achived with existing config knobs.

I think the situation was like this in xen-4.6:
a) no hdtype=,  vdev=hd   results in an emulated piix controller
b) no hdtype=,  vdev=xvd  results in an emulated piix controller
c) hdtype=ide,  vdev=hd   results in an emulated pixx controller
d) hdtype=ide,  vdev=xvd  results in an emulated piix controller
e) hdtype=ahci, vdev=hd   results in an emulated ahci controller
f) hdtype=ahci, vdev=xvd  results in an emulated ahci controller

Each of the above was bootable because the disk was visible to the BIOS.
It was possible to mix hd and xvd for disks as long as the index
(a,b,c,..) is unique.

With xen-4.7 any of the vdev=xvd results in a non-booting guest.
So something has to be changed in domU.cfg anyway.
Currently its like this:
a) no hdtype=,  vdev=hd   results in an emulated piix controller
b) no hdtype=,  vdev=xvd  results in PV-only disk
c) hdtype=ide,  vdev=hd   results in an emulated pixx controller
d) hdtype=ide,  vdev=xvd  results in PV-only disk
e) hdtype=ahci, vdev=hd   results in an emulated ahci controller
f) hdtype=ahci, vdev=xvd  results in PV-only disk

This could be added:
hdtype=ide and vdev=xvd gives an emulated piix controller
hdtype=ahci and vdev=xvd gives an emulated ahci controller

To achieve this the default hdtype= should become "UNSET", and a vdev=hd
should set it to IDE if it was "UNSET". That means there could be yet
another state "PVONLY".

As a result the possible configs in xen-4.7 are liks this:
a) no hdtype=,  vdev=hd   results in an emulated piix controller
b) no hdtype=,  vdev=xvd  results in PV-only disk
c) hdtype=ide,  vdev=hd   results in an emulated pixx controller
d) hdtype=ide,  vdev=xvd  results in an emulated piix controller
e) hdtype=ahci, vdev=hd   results in an emulated ahci controller
f) hdtype=ahci, vdev=xvd  results in an emulated ahci controller
g) hdtype=pv,   vdev=hd   results in PV-only disk
h) hdtype=pv,   vdev=xvd  results in PV-only disk

Existing domU.cfg with variant b) have to be changed to d) to get them
boot again, or rather connect the disks to an emulated controller.
I'm not sure if case g) and h) are really needed.

In my testing with xen-4.6 using hdtype=ahci resulted in no unplug, so I
think ahci was really just for emulated usage. Perhaps the pvops kernel
does a more complete unplug? I have not tested ahci with xen-4.7.

> The possibilities I see are:
> 
> (1) New boolean per-guest parameter for this behaviour, meaning
>    `provide emulated devices for all xvd* as if they were hd*'.

This would be an backward compatible approach, at least domU.cfg will
work with older toolstack. libvirt needs to know about this.

> (2) New `hdtype=ideforpv' which has the same effect as `hdtype=ide'
>    plus the semantics in (1) above.  (I'm open to better naming
>    suggestions.)

This would be an incompatible change, older toolstacks dont know the
value.

> (3) New disk property parameter `hvm-emulate' in the Deprecated
>     section of xl-disk-configuration.txt.

This would be an incompatible change, older toolstacks dont know the
value.

> Open questions:
> 
> Do we also need `... as if they were sd*' or `provide ide emulated
> devices where vdev=sd* is specified?'

sd results in an emulated LSI SCSI controller.

> If we have `hdtype=ideinclpv' do we also need `hdtype=ahciinclpv' ?

I think that is not needed given that there is no unplug (in my
testing).

> What should happen if these options are enabled for PV guests - should
> they be silently ignored, or should they be rejected ?

IMO no. Do we have such rejects already for PV or HVM only options?


It has to be noted that libvirt does not seem to know about the hdtype=
knob, which was introduced in xen-4.6.


Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Ian Jackson]

Olaf Hering writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
> To achieve this the default hdtype= should become "UNSET", and a vdev=hd
> should set it to IDE if it was "UNSET". That means there could be yet
> another state "PVONLY".

I'm afraid I think this way lies madness.  You are suggesting that the
default for hdtype= should depend in a complicated way on the set of
disk vdevs.  (It also makes hotplug very confusing.)

> > The possibilities I see are:
> > 
> > (1) New boolean per-guest parameter for this behaviour, meaning
> >    `provide emulated devices for all xvd* as if they were hd*'.
> 
> This would be an backward compatible approach, at least domU.cfg will
> work with older toolstack. libvirt needs to know about this.

This is a strange use of the phrase "backward compatible".  What you
mean is that the necessary domU.cfg changes are backward compatible.

> > What should happen if these options are enabled for PV guests - should
> > they be silently ignored, or should they be rejected ?
> 
> IMO no. Do we have such rejects already for PV or HVM only options?

Maybe.

> It has to be noted that libvirt does not seem to know about the hdtype=
> knob, which was introduced in xen-4.6.

Anyway, to conclude: it seems that you don't like any of my other
options.  I don't like your suggestion.  But you seem happy with my
option (1), above.

Personally I prefer George's suggestion:
  What about a per-disk property, emulate={default,always,only}

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On 08/06/16 11:18, Ian Jackson wrote:
> Olaf Hering writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to boot from xvda"):
>> To achieve this the default hdtype= should become "UNSET", and a vdev=hd
>> should set it to IDE if it was "UNSET". That means there could be yet
>> another state "PVONLY".
> 
> I'm afraid I think this way lies madness.  You are suggesting that the
> default for hdtype= should depend in a complicated way on the set of
> disk vdevs.  (It also makes hotplug very confusing.)
> 
>>> The possibilities I see are:
>>>
>>> (1) New boolean per-guest parameter for this behaviour, meaning
>>>    `provide emulated devices for all xvd* as if they were hd*'.
>>
>> This would be an backward compatible approach, at least domU.cfg will
>> work with older toolstack. libvirt needs to know about this.
> 
> This is a strange use of the phrase "backward compatible".  What you
> mean is that the necessary domU.cfg changes are backward compatible.
> 
>>> What should happen if these options are enabled for PV guests - should
>>> they be silently ignored, or should they be rejected ?
>>
>> IMO no. Do we have such rejects already for PV or HVM only options?
> 
> Maybe.
> 
>> It has to be noted that libvirt does not seem to know about the hdtype=
>> knob, which was introduced in xen-4.6.
> 
> Anyway, to conclude: it seems that you don't like any of my other
> options.  I don't like your suggestion.  But you seem happy with my
> option (1), above.
> 
> Personally I prefer George's suggestion:
>   What about a per-disk property, emulate={default,always,only}

We could consider at an xl level having a domain-wide and system-wide
defaults.  That way Olaf could set "disk_emulate_default=always" (or
whatever) in the global xl.conf and everything would work the way it
used to without even needing to change individual guest config files.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to > We could consider at an xl level having a domain-wide and system-wide
> defaults.  That way Olaf could set "disk_emulate_default=always" (or
> whatever) in the global xl.conf and everything would work the way it
> used to without even needing to change individual guest config files.

Yes.

Let me quote in email something I just said in irc:

11:24 <Diziet> liuw: I think we need to decide what to do about Olaf's 
               complaint about vdev semantic regression.
11:25 <Diziet> I know it's late in the release but given that the best answer 
               is probably some per-vdev parameter to control the behaviour, 
               along George's lines, and the reason we're not adopting that 
               right away is compatibility pain: I'm tempted to suggest 
               reverting the original change for 4.7.
11:26 <Diziet> The conversation in the thread about how to fix the problem is 
               making progress but I am very uncomfortable with this amount of 
               detailed API design being done under release pressure.
11:27 <Diziet> Releases are about short-term thinking, which is harmful to API 
               design.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On 08/06/16 11:30, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] 4.7 qemu regression: HVM guests fail to > We could consider at an xl level having a domain-wide and system-wide
>> defaults.  That way Olaf could set "disk_emulate_default=always" (or
>> whatever) in the global xl.conf and everything would work the way it
>> used to without even needing to change individual guest config files.
> 
> Yes.
> 
> Let me quote in email something I just said in irc:
> 
> 11:24 <Diziet> liuw: I think we need to decide what to do about Olaf's 
>                complaint about vdev semantic regression.
> 11:25 <Diziet> I know it's late in the release but given that the best answer 
>                is probably some per-vdev parameter to control the behaviour, 
>                along George's lines, and the reason we're not adopting that 
>                right away is compatibility pain: I'm tempted to suggest 
>                reverting the original change for 4.7.
> 11:26 <Diziet> The conversation in the thread about how to fix the problem is 
>                making progress but I am very uncomfortable with this amount of 
>                detailed API design being done under release pressure.
> 11:27 <Diziet> Releases are about short-term thinking, which is harmful to API 
>                design.

We definitely don't want to be putting this new stuff we're discussing
in 4.7.0.  I'd be OK with reverting the original change for the release.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Wed, Jun 08, George Dunlap wrote:

> We definitely don't want to be putting this new stuff we're discussing
> in 4.7.0.  I'd be OK with reverting the original change for the release.

I'm fine with delaying a change to address the (theoretical) breakage
introduced by c0c099d.

What I just learned a few minutes ago is the fact that c0c099d went
silently into our 4.5 and 4.4 based packages along with XSA-142 fa30003.
That happend already end of last year. Since I heard no reports about
non-booting HVM guest after applying the dom0 updates I think its safe
to assume that there are very few (if any) domU.cfg outthere which have
vdev=xvda.

So from this perspective its enough to mention the impact of c0c099d in
the 4.7 release-note.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Konrad Rzeszutek Wilk]

On Wed, Jun 08, 2016 at 01:13:03PM +0200, Olaf Hering wrote:
> On Wed, Jun 08, George Dunlap wrote:
> 
> > We definitely don't want to be putting this new stuff we're discussing
> > in 4.7.0.  I'd be OK with reverting the original change for the release.
> 
> I'm fine with delaying a change to address the (theoretical) breakage
> introduced by c0c099d.
> 
> What I just learned a few minutes ago is the fact that c0c099d went
> silently into our 4.5 and 4.4 based packages along with XSA-142 fa30003.
> That happend already end of last year. Since I heard no reports about
> non-booting HVM guest after applying the dom0 updates I think its safe

There were. Boris and I reported it a couple of times.

Ah, you meant via internal SuSE bugs!

> to assume that there are very few (if any) domU.cfg outthere which have
> vdev=xvda.

Oracle by default in their product (OVS) has that hard-coded for all
guest configurations it creates. Granted we are still using Xend
and migrating a guest from xm to xl so we have some other issues
to address as well.

> 
> So from this perspective its enough to mention the impact of c0c099d in
> the 4.7 release-note.
> 
> Olaf
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Fri, Jun 03, George Dunlap wrote:

> I'd be OK with this.  But is the "hdtype unset" also available at the
> libxl level?

Yes, it could. Right now the .idl sets it to IDE. Looks like another
enum has to be added and announced via a #define.

> Although I still think that the XenoLinux kernels should as soon as
> posssible try to move to following upstream Linux's practice of not
> stealing major numbers of other drivers.

Existing domUs cant be changed.

I will provided a patch for libxl and libvirt as a base for further
discussion.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Jan Beulich]

>>> On 31.05.16 at 13:32, <george.dunlap@citrix.com> wrote:
> On Tue, May 31, 2016 at 12:16 PM, Olaf Hering <olaf@aepfle.de> wrote:
>> On Tue, May 31, George Dunlap wrote:
>>
>>> On Mon, May 30, 2016 at 9:42 PM, Olaf Hering <olaf@aepfle.de> wrote:
>>> > With staging-4.6 this domU boots from xvda, qemu creates an emulated
>>> > disk. With staging no disk is found, unless the name is changed to hda.
>>> > Looks like qemu-2.6 does not handle xvda either.
>>>
>>> This was intentional; see this thread:
>>>
>>> https://marc.info/?i=<1444820717-25565-1-git-send-email-anthony.perard@citrix.com>
>>>
>>> The idea was to make it possible to create an HVM guest with no
>>> emulated disks for guest booting with OVMF which contains PV drivers.
>>
>> This breaks the domU becasue it changes its device names from 'xvd' to
>> 'hd' if a xenlinux based kernel is used in domU.
>> In other words: every SUSE domU out there.
> 
> Sorry, can you expand on this a bit?  Are you saying that on SuSE, if
> you specify "vdev=xvda" in your config file, that you'll get PV
> devices named "/dev/xvda", but that if you specify "vdev=hda", that
> you'll get PV devices but named "/dev/hda"?

And just to clarify - this isn't really SUSE-specific, this is how the old
XenoLinux blkfront has always behaved. In fact when I saw this code
gone from the upstream (pv-ops) variant, I think I had inquired how
this is expected to be handling upgrade cases, and I don't think I got
much of a useful answer to that.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[George Dunlap]

On Tue, May 31, 2016 at 1:10 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 31.05.16 at 13:32, <george.dunlap@citrix.com> wrote:
>> On Tue, May 31, 2016 at 12:16 PM, Olaf Hering <olaf@aepfle.de> wrote:
>>> On Tue, May 31, George Dunlap wrote:
>>>
>>>> On Mon, May 30, 2016 at 9:42 PM, Olaf Hering <olaf@aepfle.de> wrote:
>>>> > With staging-4.6 this domU boots from xvda, qemu creates an emulated
>>>> > disk. With staging no disk is found, unless the name is changed to hda.
>>>> > Looks like qemu-2.6 does not handle xvda either.
>>>>
>>>> This was intentional; see this thread:
>>>>
>>>> https://marc.info/?i=<1444820717-25565-1-git-send-email-anthony.perard@citrix.com>
>>>>
>>>> The idea was to make it possible to create an HVM guest with no
>>>> emulated disks for guest booting with OVMF which contains PV drivers.
>>>
>>> This breaks the domU becasue it changes its device names from 'xvd' to
>>> 'hd' if a xenlinux based kernel is used in domU.
>>> In other words: every SUSE domU out there.
>>
>> Sorry, can you expand on this a bit?  Are you saying that on SuSE, if
>> you specify "vdev=xvda" in your config file, that you'll get PV
>> devices named "/dev/xvda", but that if you specify "vdev=hda", that
>> you'll get PV devices but named "/dev/hda"?
>
> And just to clarify - this isn't really SUSE-specific, this is how the old
> XenoLinux blkfront has always behaved. In fact when I saw this code
> gone from the upstream (pv-ops) variant, I think I had inquired how
> this is expected to be handling upgrade cases, and I don't think I got
> much of a useful answer to that.

Sure, but I think SuSE are basically the only distribution still using
XenoLinux, right?  Is there actually any open project maintaining the
XenoLinux fork?  If someone wanted to use XenoLinux, wouldn't their
options basically be 1) do all the forward-porting themselves from
some ancient 2.X tree, or 2) use SuSE's port?

Regarding an upgrade: fstab and other tools can be handed UUIDs; and
an enterprise-grade distribution could probably include udev scripts
to create symlinks, right?

The reality is though that the 'vdev=' way of specifying things is a
bit of a mess.  There's no way to specify, "Please make this
emulated-only (i.e., no backing PV drivers)"; there's no way of
hot-plugging in an emulated device; &c &c.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Jan Beulich]

>>> On 31.05.16 at 15:41, <george.dunlap@citrix.com> wrote:
> On Tue, May 31, 2016 at 1:10 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 31.05.16 at 13:32, <george.dunlap@citrix.com> wrote:
>>> On Tue, May 31, 2016 at 12:16 PM, Olaf Hering <olaf@aepfle.de> wrote:
>>>> On Tue, May 31, George Dunlap wrote:
>>>>
>>>>> On Mon, May 30, 2016 at 9:42 PM, Olaf Hering <olaf@aepfle.de> wrote:
>>>>> > With staging-4.6 this domU boots from xvda, qemu creates an emulated
>>>>> > disk. With staging no disk is found, unless the name is changed to hda.
>>>>> > Looks like qemu-2.6 does not handle xvda either.
>>>>>
>>>>> This was intentional; see this thread:
>>>>>
>>>>> 
> https://marc.info/?i=<1444820717-25565-1-git-send-email-anthony.perard@citrix.
> com>
>>>>>
>>>>> The idea was to make it possible to create an HVM guest with no
>>>>> emulated disks for guest booting with OVMF which contains PV drivers.
>>>>
>>>> This breaks the domU becasue it changes its device names from 'xvd' to
>>>> 'hd' if a xenlinux based kernel is used in domU.
>>>> In other words: every SUSE domU out there.
>>>
>>> Sorry, can you expand on this a bit?  Are you saying that on SuSE, if
>>> you specify "vdev=xvda" in your config file, that you'll get PV
>>> devices named "/dev/xvda", but that if you specify "vdev=hda", that
>>> you'll get PV devices but named "/dev/hda"?
>>
>> And just to clarify - this isn't really SUSE-specific, this is how the old
>> XenoLinux blkfront has always behaved. In fact when I saw this code
>> gone from the upstream (pv-ops) variant, I think I had inquired how
>> this is expected to be handling upgrade cases, and I don't think I got
>> much of a useful answer to that.
> 
> Sure, but I think SuSE are basically the only distribution still using
> XenoLinux, right?  Is there actually any open project maintaining the
> XenoLinux fork?  If someone wanted to use XenoLinux, wouldn't their
> options basically be 1) do all the forward-porting themselves from
> some ancient 2.X tree, or 2) use SuSE's port?

Yes.

> Regarding an upgrade: fstab and other tools can be handed UUIDs;

You know how people behave - you tell them "don't use raw
device names" and they still do. (In a few places I'm guilty of this
myself - on kernel command lines, for brevity, I prefer using raw
device names for "root=", but I also know that it's going to be me
to deal with the fallout.)

> and
> an enterprise-grade distribution could probably include udev scripts
> to create symlinks, right?

I guess so. Yet the question is - how would the script know what
symlinks to create? Cross linking between /dev/xvd* and /dev/hd*
(or /dev/sd*) can't be done blindly, as a guest may have e.g. both
xvda and hda specified in its config file (which works with the old
blkfront afaict, but wouldn't work with upstream's).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Konrad Rzeszutek Wilk]

On Tue, May 31, 2016 at 01:16:14PM +0200, Olaf Hering wrote:
> On Tue, May 31, George Dunlap wrote:
> 
> > On Mon, May 30, 2016 at 9:42 PM, Olaf Hering <olaf@aepfle.de> wrote:
> > > With staging-4.6 this domU boots from xvda, qemu creates an emulated
> > > disk. With staging no disk is found, unless the name is changed to hda.
> > > Looks like qemu-2.6 does not handle xvda either.
> > 
> > This was intentional; see this thread:
> > 
> > https://marc.info/?i=<1444820717-25565-1-git-send-email-anthony.perard@citrix.com>
> > 
> > The idea was to make it possible to create an HVM guest with no
> > emulated disks for guest booting with OVMF which contains PV drivers.
> 
> This breaks the domU becasue it changes its device names from 'xvd' to
> 'hd' if a xenlinux based kernel is used in domU.
> In other words: every SUSE domU out there.

And every Solaris domU too.
> 
> Olaf
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Wei Liu]

I had the impression that Olaf was fine with just updating the
documentation so I dropped this thread from my radar. But Ian brought my
attention to it again because there is discussion on detailed API
design, which is bad at this stage of a release.

I think we should just revert the patch in question and solve this issue
properly in 4.8.

I will revert that patch shortly. We're going to need another RC
for 4.7.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Olaf Hering]

On Wed, Jun 08, Wei Liu wrote:

> I think we should just revert the patch in question and solve this issue
> properly in 4.8.

What impact does reverting c0c099d have for OVMF?
I dont know myself, just used it the first time now with staging-4.6.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 4.7 qemu regression: HVM guests fail to boot from xvda

[Wei Liu]

On Wed, Jun 08, 2016 at 02:04:45PM +0200, Olaf Hering wrote:
> On Wed, Jun 08, Wei Liu wrote:
> 
> > I think we should just revert the patch in question and solve this issue
> > properly in 4.8.
> 
> What impact does reverting c0c099d have for OVMF?
> I dont know myself, just used it the first time now with staging-4.6.

OVMF should work as usual.

Wei.

> 
> Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel