Inconsistent console behavior during HVM direct boot

Inconsistent console behavior during HVM direct boot

[Tamas K Lengyel]

While direct booting a Linux kernel into an HVM VM no console output
is received if serial="pty" is set in the VM config but the kernel
command line specifies "console=hvc0". If "console=ttyS0" is specified
then the console output is received as expected. Conversely, if the
serial line is commented out in the VM config but hvc0 is set on the
kernel command line then it also works. It's unclear whether this
behavior is expected or not, after some discussion it sounds like all
kernels ought to understand hvc0 irrespective of the serial="pty" line
being set or not. Reporting this in case someone has more insight
and/or inclination digging into this.

Cheers,
Tamas

Re: Inconsistent console behavior during HVM direct boot

[Roger Pau Monné]

On Tue, Jan 12, 2021 at 09:54:18PM -0500, Tamas K Lengyel wrote:
> While direct booting a Linux kernel into an HVM VM no console output
> is received if serial="pty" is set in the VM config but the kernel
> command line specifies "console=hvc0". If "console=ttyS0" is specified
> then the console output is received as expected. Conversely, if the
> serial line is commented out in the VM config but hvc0 is set on the
> kernel command line then it also works. It's unclear whether this
> behavior is expected or not, after some discussion it sounds like all
> kernels ought to understand hvc0 irrespective of the serial="pty" line
> being set or not. Reporting this in case someone has more insight
> and/or inclination digging into this.

So I think this is (as confusing as it might seem) the intended
behavior.

Using serial='pty' exposes an emulated serial device to the guest and
'xl console' will fetch the output from that device by default. So on
the Linux kernel command line you need to use console=ttyS0 in order
to point Linux to use the emulated serial device.

however if serial='pty' is not used, there will be no emulated serial
device exposed to the guest, only the PV console, and that's what 'xl
console' will attach to in that case. You then need to point the Linux
kernel to use the PV serial console, by adding the console=hvc0
command line.

I'm unsure how to clarify this, maybe some additions could be made to
the xl.cfg man page to note that using serial='pty' will add an
emulated serial device to the guest, and that by default 'xl console'
will fetch the output from that device instead of the paravirtualized
console?

Roger.

Re: Inconsistent console behavior during HVM direct boot

[Tamas K Lengyel]

On Wed, Jan 13, 2021 at 8:18 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Tue, Jan 12, 2021 at 09:54:18PM -0500, Tamas K Lengyel wrote:
> > While direct booting a Linux kernel into an HVM VM no console output
> > is received if serial="pty" is set in the VM config but the kernel
> > command line specifies "console=hvc0". If "console=ttyS0" is specified
> > then the console output is received as expected. Conversely, if the
> > serial line is commented out in the VM config but hvc0 is set on the
> > kernel command line then it also works. It's unclear whether this
> > behavior is expected or not, after some discussion it sounds like all
> > kernels ought to understand hvc0 irrespective of the serial="pty" line
> > being set or not. Reporting this in case someone has more insight
> > and/or inclination digging into this.
>
> So I think this is (as confusing as it might seem) the intended
> behavior.
>
> Using serial='pty' exposes an emulated serial device to the guest and
> 'xl console' will fetch the output from that device by default. So on
> the Linux kernel command line you need to use console=ttyS0 in order
> to point Linux to use the emulated serial device.
>
> however if serial='pty' is not used, there will be no emulated serial
> device exposed to the guest, only the PV console, and that's what 'xl
> console' will attach to in that case. You then need to point the Linux
> kernel to use the PV serial console, by adding the console=hvc0
> command line.
>
> I'm unsure how to clarify this, maybe some additions could be made to
> the xl.cfg man page to note that using serial='pty' will add an
> emulated serial device to the guest, and that by default 'xl console'
> will fetch the output from that device instead of the paravirtualized
> console?

Hi Roger,
that explanation makes sense, I wasn't aware that there would be two
different console's available to the guest if serial="pty" is set and
that xl would pick on over the other. So adding that to the
documentation would be helpful. Perhaps also pointing out under what
scenario is the emulated console vs the pv console is useful. I assume
the point of the emulated console would be to get console output from
software that's not Xen-aware (like GRUB)?

Thanks,
Tamas

Re: Inconsistent console behavior during HVM direct boot

[Roger Pau Monné]

On Wed, Jan 13, 2021 at 10:48:02AM -0500, Tamas K Lengyel wrote:
> On Wed, Jan 13, 2021 at 8:18 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Tue, Jan 12, 2021 at 09:54:18PM -0500, Tamas K Lengyel wrote:
> > > While direct booting a Linux kernel into an HVM VM no console output
> > > is received if serial="pty" is set in the VM config but the kernel
> > > command line specifies "console=hvc0". If "console=ttyS0" is specified
> > > then the console output is received as expected. Conversely, if the
> > > serial line is commented out in the VM config but hvc0 is set on the
> > > kernel command line then it also works. It's unclear whether this
> > > behavior is expected or not, after some discussion it sounds like all
> > > kernels ought to understand hvc0 irrespective of the serial="pty" line
> > > being set or not. Reporting this in case someone has more insight
> > > and/or inclination digging into this.
> >
> > So I think this is (as confusing as it might seem) the intended
> > behavior.
> >
> > Using serial='pty' exposes an emulated serial device to the guest and
> > 'xl console' will fetch the output from that device by default. So on
> > the Linux kernel command line you need to use console=ttyS0 in order
> > to point Linux to use the emulated serial device.
> >
> > however if serial='pty' is not used, there will be no emulated serial
> > device exposed to the guest, only the PV console, and that's what 'xl
> > console' will attach to in that case. You then need to point the Linux
> > kernel to use the PV serial console, by adding the console=hvc0
> > command line.
> >
> > I'm unsure how to clarify this, maybe some additions could be made to
> > the xl.cfg man page to note that using serial='pty' will add an
> > emulated serial device to the guest, and that by default 'xl console'
> > will fetch the output from that device instead of the paravirtualized
> > console?
> 
> Hi Roger,
> that explanation makes sense, I wasn't aware that there would be two
> different console's available to the guest if serial="pty" is set and
> that xl would pick on over the other. So adding that to the
> documentation would be helpful. Perhaps also pointing out under what
> scenario is the emulated console vs the pv console is useful. I assume
> the point of the emulated console would be to get console output from
> software that's not Xen-aware (like GRUB)?

Right, since you can run unmodified OSes (or bootloaders) in HVM mode
it's useful to be able to get serial output from them directly when
using 'xl console' IMO.

Would you be up for writing a patch to xl.cfg man page to clarify
it?

Thanks, Roger.