Xen 4.8.0 dom0pvh=1 causes ext4 filesystem corruption

Xen 4.8.0 dom0pvh=1 causes ext4 filesystem corruption

[Valtteri Kiviniemi]

[-- Attachment #1.1: Type: text/plain, Size: 1413 bytes --]

Hi,

Yesterday I decided to upgrade my Xen version from 4.6.0 to 4.8.0. I
compiled it from source and at the same time I compiled the latest Linux
kernel (4.10.1).

When rebooting I decided to try if dom0 PVH would work (with previous Xen
version it just caused kernel panic). Seemed to boot fine until systemd
started mounting the root filesystem and then the console was filled with
ext4 errors. Couldn't even log in.

Booting with a systemrescuecd and running fsck just caused the whole
filesystem to be re-attached in thousands of small pieces under lost+found.
I was sure that this was a some kind of hardware failure, so I switched my
hard drives and did a clean reinstall for dom0 and tried again. Again,
after a reboot the whole rootfs was completely corrupted.

Second reinstall and this time I disabled dom0 PVH and the system booted
just fine, and no ext4 errors. My root filesystem is just a simple Linux
software raid1 with ext4 on top of it.

Now that I started thinking I have also had strange ext4 errors happening
inside my guests, so I also disabled PVH from all the guests. With guests
the ext4 error is always the same: "EXT4-fs error (device xvda1):
ext4_iget:4665: inode #317: comm find: bogus i_mode (135206)"

Unfortunately I don't have any logs from the dom0 corruption as I can't
even log in to the system when dom0 PVH is enabled. The corruption happens
instantly during system bootup.

[-- Attachment #1.2: Type: text/html, Size: 1581 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen 4.8.0 dom0pvh=1 causes ext4 filesystem corruption

[Steven Haigh]

[-- Attachment #1.1.1: Type: text/plain, Size: 1807 bytes --]

On 10/03/17 02:13, Valtteri Kiviniemi wrote:
> Hi,
> 
> Yesterday I decided to upgrade my Xen version from 4.6.0 to 4.8.0. I
> compiled it from source and at the same time I compiled the latest Linux
> kernel (4.10.1).
> 
> When rebooting I decided to try if dom0 PVH would work (with previous
> Xen version it just caused kernel panic). Seemed to boot fine until
> systemd started mounting the root filesystem and then the console was
> filled with ext4 errors. Couldn't even log in.
> 
> Booting with a systemrescuecd and running fsck just caused the whole
> filesystem to be re-attached in thousands of small pieces under
> lost+found. I was sure that this was a some kind of hardware failure, so
> I switched my hard drives and did a clean reinstall for dom0 and tried
> again. Again, after a reboot the whole rootfs was completely corrupted.
> 
> Second reinstall and this time I disabled dom0 PVH and the system booted
> just fine, and no ext4 errors. My root filesystem is just a simple Linux
> software raid1 with ext4 on top of it.
> 
> Now that I started thinking I have also had strange ext4 errors
> happening inside my guests, so I also disabled PVH from all the guests.
> With guests the ext4 error is always the same: "EXT4-fs error (device
> xvda1): ext4_iget:4665: inode #317: comm find: bogus i_mode (135206)"
> 
> Unfortunately I don't have any logs from the dom0 corruption as I can't
> even log in to the system when dom0 PVH is enabled. The corruption
> happens instantly during system bootup.

I have this happen a lot using pvh mode in previous Xen versions. Is it
supposed to be 'working' yet or is it still not recommended for use?

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen 4.8.0 dom0pvh=1 causes ext4 filesystem corruption

[Andrew Cooper]

On 10/03/2017 00:09, Steven Haigh wrote:
> On 10/03/17 02:13, Valtteri Kiviniemi wrote:
>> Hi,
>>
>> Yesterday I decided to upgrade my Xen version from 4.6.0 to 4.8.0. I
>> compiled it from source and at the same time I compiled the latest Linux
>> kernel (4.10.1).
>>
>> When rebooting I decided to try if dom0 PVH would work (with previous
>> Xen version it just caused kernel panic). Seemed to boot fine until
>> systemd started mounting the root filesystem and then the console was
>> filled with ext4 errors. Couldn't even log in.
>>
>> Booting with a systemrescuecd and running fsck just caused the whole
>> filesystem to be re-attached in thousands of small pieces under
>> lost+found. I was sure that this was a some kind of hardware failure, so
>> I switched my hard drives and did a clean reinstall for dom0 and tried
>> again. Again, after a reboot the whole rootfs was completely corrupted.
>>
>> Second reinstall and this time I disabled dom0 PVH and the system booted
>> just fine, and no ext4 errors. My root filesystem is just a simple Linux
>> software raid1 with ext4 on top of it.
>>
>> Now that I started thinking I have also had strange ext4 errors
>> happening inside my guests, so I also disabled PVH from all the guests.
>> With guests the ext4 error is always the same: "EXT4-fs error (device
>> xvda1): ext4_iget:4665: inode #317: comm find: bogus i_mode (135206)"
>>
>> Unfortunately I don't have any logs from the dom0 corruption as I can't
>> even log in to the system when dom0 PVH is enabled. The corruption
>> happens instantly during system bootup.
> I have this happen a lot using pvh mode in previous Xen versions. Is it
> supposed to be 'working' yet or is it still not recommended for use?

That version of PVH has just been ripped out of Linux, and is in the
process of being ripped out of the hypervisor.

The replacement, accessed by using dom0=pvh is available in master, will
be present in Xen 4.9, and should work with Linux 4.11 and FreeBSD
Current.  (Roger, Boris - please correct me if I am wrong.)

The same Linux and FreeBSD versions should function perfectly well as
plain domUs in PVH mode, by selecting device_model=none in the domain
configuration.  (Dom0 is rather more complicated than domU, due to the
shared responsibility of real hardware with Xen.)

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen 4.8.0 dom0pvh=1 causes ext4 filesystem corruption

[Boris Ostrovsky]

On 03/09/2017 07:19 PM, Andrew Cooper wrote:
> On 10/03/2017 00:09, Steven Haigh wrote:
>> On 10/03/17 02:13, Valtteri Kiviniemi wrote:
>>> Hi,
>>>
>>> Yesterday I decided to upgrade my Xen version from 4.6.0 to 4.8.0. I
>>> compiled it from source and at the same time I compiled the latest Linux
>>> kernel (4.10.1).
>>>
>>> When rebooting I decided to try if dom0 PVH would work (with previous
>>> Xen version it just caused kernel panic). Seemed to boot fine until
>>> systemd started mounting the root filesystem and then the console was
>>> filled with ext4 errors. Couldn't even log in.
>>>
>>> Booting with a systemrescuecd and running fsck just caused the whole
>>> filesystem to be re-attached in thousands of small pieces under
>>> lost+found. I was sure that this was a some kind of hardware failure, so
>>> I switched my hard drives and did a clean reinstall for dom0 and tried
>>> again. Again, after a reboot the whole rootfs was completely corrupted.
>>>
>>> Second reinstall and this time I disabled dom0 PVH and the system booted
>>> just fine, and no ext4 errors. My root filesystem is just a simple Linux
>>> software raid1 with ext4 on top of it.
>>>
>>> Now that I started thinking I have also had strange ext4 errors
>>> happening inside my guests, so I also disabled PVH from all the guests.
>>> With guests the ext4 error is always the same: "EXT4-fs error (device
>>> xvda1): ext4_iget:4665: inode #317: comm find: bogus i_mode (135206)"
>>>
>>> Unfortunately I don't have any logs from the dom0 corruption as I can't
>>> even log in to the system when dom0 PVH is enabled. The corruption
>>> happens instantly during system bootup.
>> I have this happen a lot using pvh mode in previous Xen versions. Is it
>> supposed to be 'working' yet or is it still not recommended for use?
>
> That version of PVH has just been ripped out of Linux, and is in the
> process of being ripped out of the hypervisor.
>
> The replacement, accessed by using dom0=pvh is available in master, will
> be present in Xen 4.9, and should work with Linux 4.11 and FreeBSD
> Current.  (Roger, Boris - please correct me if I am wrong.)


There is nothing in Linux currently to support PVH dom0, only domU. I 
haven't tried booting dom0 in PVH mode yet but I doubt anything good 
will come out of this.

This is something I will be working on next.

-boris


>
> The same Linux and FreeBSD versions should function perfectly well as
> plain domUs in PVH mode, by selecting device_model=none in the domain
> configuration.  (Dom0 is rather more complicated than domU, due to the
> shared responsibility of real hardware with Xen.)
>
> ~Andrew
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen 4.8.0 dom0pvh=1 causes ext4 filesystem corruption

[Roger Pau Monné]

On Thu, Mar 09, 2017 at 05:13:38PM +0200, Valtteri Kiviniemi wrote:
> Hi,
> 
> Yesterday I decided to upgrade my Xen version from 4.6.0 to 4.8.0. I
> compiled it from source and at the same time I compiled the latest Linux
> kernel (4.10.1).
> 
> When rebooting I decided to try if dom0 PVH would work (with previous Xen
> version it just caused kernel panic). Seemed to boot fine until systemd
> started mounting the root filesystem and then the console was filled with
> ext4 errors. Couldn't even log in.
> 
> Booting with a systemrescuecd and running fsck just caused the whole
> filesystem to be re-attached in thousands of small pieces under lost+found.
> I was sure that this was a some kind of hardware failure, so I switched my
> hard drives and did a clean reinstall for dom0 and tried again. Again,
> after a reboot the whole rootfs was completely corrupted.
> 
> Second reinstall and this time I disabled dom0 PVH and the system booted
> just fine, and no ext4 errors. My root filesystem is just a simple Linux
> software raid1 with ext4 on top of it.
> 
> Now that I started thinking I have also had strange ext4 errors happening
> inside my guests, so I also disabled PVH from all the guests. With guests
> the ext4 error is always the same: "EXT4-fs error (device xvda1):
> ext4_iget:4665: inode #317: comm find: bogus i_mode (135206)"

As Andrew already pointed out, this implementation (PVHv1) is already removed
from Linux, and will be removed soon from Xen. FWIW, FreeBSD has been using
that for Dom0, and no one reported a single FS corruption issue, so it seems
like this was something specific to the Linux PVHv1 implementation.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel