[BUG] Hypervisor 4.3 fails to init Marvel SATA III when Vt-d is enabled in the BIOS (failed to IDENTIFY)

[BUG] Hypervisor 4.3 fails to init Marvel SATA III when Vt-d is enabled in the BIOS (failed to IDENTIFY)

[Rodger McIntosh]

[-- Attachment #1: Type: text/plain, Size: 1741 bytes --]

Problem:
When booting with the Xen Hypervisor after enabling Vt-d in the BIOS some 
drives disappear from /dev (see dev.txt and dev-xen.txt).

Environment:
Motherboard -> Asus Sabertooth X79
Processor -> Intel Core i7-3930K Sandy Bridge-E 3.2GHz 
RAM -> 2 x G.SKILL Ripjaws Z Series 16GB (4 x 4GB) = 32GB
PCIe -> HighPoint Rocket 640L PCI-Express 2.0 x4 SATA III (6.0Gb/s) RAID
Controller Card
OS -> Ubuntu 3.11.0-18

Description:
My system has 6 SATA II ports and 2 SATA III ports on board. I have added a
4 port SATA III controller as a PCIex4 device. The 6 SATA II ports are
connected to 6 SAMSUNG HD103SJ 1TB hard drives. The on board SATA III ports
are connected to 2 Blue Ray drives. The 4 port SATA III controller is
connected to 4 Corsair Force LS 60GB SSDs.

When booting without the hypervisor all drives operate as expected. When
booting with the hypervisor both Blue Ray drives and 2 of the SSD drives go
missing. dmesg shows errors 'failed to IDENTIFY (INIT_DEV_PARAMS failed,
err_mask=0x80)' and 'failed to IDENTIFY (I/O error, err_mask=0x4)' and
'COMRESET failed (errno=-16)'; (see dmesg.txt and dmesg-xen.txt). lspci
doesn't show any real difference in the devices; just some interrupt IRQ
numbers.

I've tried all (that I know of) iommu command line parameters with no luck;
no-intremap,pass-through,no-qinval,no-snoop,workaround_bios_bug.

I've tried the latest Ubuntu 13.10 (3.11.0-19) build as well as 12.10.

I've tried Xen-Hypervisor 4.3 on the Ubuntu repo, 4.3.2 RELEASE, 4.4.0
RELEASE, 4.3-stable, 4.4-stable, 4.3-staging, and 4.4-staging.

I've attached as much information as I could think of. Let me know if you
need more. I can try any updates you throw at me.

Thank you in advance for your time on this.
Rodger

[-- Attachment #2: xen-issue.tar.gz --]
[-- Type: application/octet-stream, Size: 64876 bytes --]

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [BUG] Hypervisor 4.3 fails to init Marvel SATA III when Vt-d is enabled in the BIOS (failed to IDENTIFY)

[Jan Beulich]

>>> On 15.04.14 at 05:32, <rodger@geekshake.com> wrote:
> Problem:
> When booting with the Xen Hypervisor after enabling Vt-d in the BIOS some 
> drives disappear from /dev (see dev.txt and dev-xen.txt).
> 
> Environment:
> Motherboard -> Asus Sabertooth X79
> Processor -> Intel Core i7-3930K Sandy Bridge-E 3.2GHz 
> RAM -> 2 x G.SKILL Ripjaws Z Series 16GB (4 x 4GB) = 32GB
> PCIe -> HighPoint Rocket 640L PCI-Express 2.0 x4 SATA III (6.0Gb/s) RAID
> Controller Card
> OS -> Ubuntu 3.11.0-18
> 
> Description:
> My system has 6 SATA II ports and 2 SATA III ports on board. I have added a
> 4 port SATA III controller as a PCIex4 device. The 6 SATA II ports are
> connected to 6 SAMSUNG HD103SJ 1TB hard drives. The on board SATA III ports
> are connected to 2 Blue Ray drives. The 4 port SATA III controller is
> connected to 4 Corsair Force LS 60GB SSDs.
> 
> When booting without the hypervisor all drives operate as expected. When
> booting with the hypervisor both Blue Ray drives and 2 of the SSD drives go
> missing. dmesg shows errors 'failed to IDENTIFY (INIT_DEV_PARAMS failed,
> err_mask=0x80)' and 'failed to IDENTIFY (I/O error, err_mask=0x4)' and
> 'COMRESET failed (errno=-16)'; (see dmesg.txt and dmesg-xen.txt). lspci
> doesn't show any real difference in the devices; just some interrupt IRQ
> numbers.
> 
> I've tried all (that I know of) iommu command line parameters with no luck;
> no-intremap,pass-through,no-qinval,no-snoop,workaround_bios_bug.

But I suppose you didn't try the "pci-phantom=" option that was
specifically added for controllers like this (see
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=4e3c592c93d7dbe02ca36878457515d30fe931d2)?

Jan

Re: [BUG] Hypervisor 4.3 fails to init Marvel SATA III when Vt-d is enabled in the BIOS (failed to IDENTIFY)

[George Dunlap]

On Tue, Apr 15, 2014 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 15.04.14 at 05:32, <rodger@geekshake.com> wrote:
>> Problem:
>> When booting with the Xen Hypervisor after enabling Vt-d in the BIOS some
>> drives disappear from /dev (see dev.txt and dev-xen.txt).
>>
>> Environment:
>> Motherboard -> Asus Sabertooth X79
>> Processor -> Intel Core i7-3930K Sandy Bridge-E 3.2GHz
>> RAM -> 2 x G.SKILL Ripjaws Z Series 16GB (4 x 4GB) = 32GB
>> PCIe -> HighPoint Rocket 640L PCI-Express 2.0 x4 SATA III (6.0Gb/s) RAID
>> Controller Card
>> OS -> Ubuntu 3.11.0-18
>>
>> Description:
>> My system has 6 SATA II ports and 2 SATA III ports on board. I have added a
>> 4 port SATA III controller as a PCIex4 device. The 6 SATA II ports are
>> connected to 6 SAMSUNG HD103SJ 1TB hard drives. The on board SATA III ports
>> are connected to 2 Blue Ray drives. The 4 port SATA III controller is
>> connected to 4 Corsair Force LS 60GB SSDs.
>>
>> When booting without the hypervisor all drives operate as expected. When
>> booting with the hypervisor both Blue Ray drives and 2 of the SSD drives go
>> missing. dmesg shows errors 'failed to IDENTIFY (INIT_DEV_PARAMS failed,
>> err_mask=0x80)' and 'failed to IDENTIFY (I/O error, err_mask=0x4)' and
>> 'COMRESET failed (errno=-16)'; (see dmesg.txt and dmesg-xen.txt). lspci
>> doesn't show any real difference in the devices; just some interrupt IRQ
>> numbers.
>>
>> I've tried all (that I know of) iommu command line parameters with no luck;
>> no-intremap,pass-through,no-qinval,no-snoop,workaround_bios_bug.
>
> But I suppose you didn't try the "pci-phantom=" option that was
> specifically added for controllers like this (see
> http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=4e3c592c93d7dbe02ca36878457515d30fe931d2)?

It probably would have been better to link to the command-line
reference instead of a git commit:

http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html

As it is, the documentation is a bit sparse -- maybe next doc day I'll
try to write up a wiki page that will be easier to follow (and perhaps
more discoverable on Google).

I suppose there's no way we could add devices like this to a "quirks"
file, so that users don't have to figure this out on their own?

 -George

Re: [BUG] Hypervisor 4.3 fails to init Marvel SATA III when Vt-d is enabled in the BIOS (failed to IDENTIFY)

[Jan Beulich]

>>> On 15.04.14 at 10:59, <George.Dunlap@eu.citrix.com> wrote:
> I suppose there's no way we could add devices like this to a "quirks"
> file, so that users don't have to figure this out on their own?

We surely could, but I'd prefer such a list to not get added IDs one
by one, but instead just the full list of affected ones. Which would
require someone to collect such a list. And of course it would need to
be clarified whether for some of the affected IDs revisions exist that
have/don't have the issue.

Jan

Re: [BUG] Hypervisor 4.3 fails to init Marvel SATA III when Vt-d is enabled in the BIOS (failed to IDENTIFY)

[Ian Campbell]

On Tue, 2014-04-15 at 10:38 +0100, Jan Beulich wrote:
> >>> On 15.04.14 at 10:59, <George.Dunlap@eu.citrix.com> wrote:
> > I suppose there's no way we could add devices like this to a "quirks"
> > file, so that users don't have to figure this out on their own?
> 
> We surely could, but I'd prefer such a list to not get added IDs one
> by one, but instead just the full list of affected ones. Which would
> require someone to collect such a list. And of course it would need to
> be clarified whether for some of the affected IDs revisions exist that
> have/don't have the issue.

I appreciate the preference but in the absence of someone willing/able
to collate such a list surely it would be better to take the ones we
know about and allow people to add others as they are discovered though?

Ian.

Re: [BUG] Hypervisor 4.3 fails to init Marvel SATA III when Vt-d is enabled in the BIOS (failed to IDENTIFY)

[Jan Beulich]

>>> On 15.04.14 at 11:45, <Ian.Campbell@citrix.com> wrote:
> On Tue, 2014-04-15 at 10:38 +0100, Jan Beulich wrote:
>> >>> On 15.04.14 at 10:59, <George.Dunlap@eu.citrix.com> wrote:
>> > I suppose there's no way we could add devices like this to a "quirks"
>> > file, so that users don't have to figure this out on their own?
>> 
>> We surely could, but I'd prefer such a list to not get added IDs one
>> by one, but instead just the full list of affected ones. Which would
>> require someone to collect such a list. And of course it would need to
>> be clarified whether for some of the affected IDs revisions exist that
>> have/don't have the issue.
> 
> I appreciate the preference but in the absence of someone willing/able
> to collate such a list surely it would be better to take the ones we
> know about and allow people to add others as they are discovered though?

I'm unlikely to reject patches to this effect; I just wouldn't want to put
something half baked together myself (not the least because I'm often
enough - perhaps indirectly - complaining about half baked stuff
scattered across the tree).

Jan

Re: [BUG] Hypervisor 4.3 fails to init Marvel SATA III when Vt-d is enabled in the BIOS (failed to IDENTIFY)

[Rodger McIntosh]

Jan / George / Ian,

Sorry for the late response. Worked like a charm. 

As a developer I know the hesitation to write "dirty" code, but as a novice Xen user this proved to be quite a challenge. At least a few more key words will be available for search engines now.

Thanks for the help and keep up the great work.
Rodger

-----Original Message-----
From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of George Dunlap
Sent: Tuesday, April 15, 2014 2:59 AM
To: Jan Beulich
Cc: Rodger McIntosh; xen-devel@lists.xen.org
Subject: Re: [Xen-devel] [BUG] Hypervisor 4.3 fails to init Marvel SATA III when Vt-d is enabled in the BIOS (failed to IDENTIFY)

On Tue, Apr 15, 2014 at 8:39 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 15.04.14 at 05:32, <rodger@geekshake.com> wrote:
>> Problem:
>> When booting with the Xen Hypervisor after enabling Vt-d in the BIOS some
>> drives disappear from /dev (see dev.txt and dev-xen.txt).
>>
>> Environment:
>> Motherboard -> Asus Sabertooth X79
>> Processor -> Intel Core i7-3930K Sandy Bridge-E 3.2GHz
>> RAM -> 2 x G.SKILL Ripjaws Z Series 16GB (4 x 4GB) = 32GB
>> PCIe -> HighPoint Rocket 640L PCI-Express 2.0 x4 SATA III (6.0Gb/s) RAID
>> Controller Card
>> OS -> Ubuntu 3.11.0-18
>>
>> Description:
>> My system has 6 SATA II ports and 2 SATA III ports on board. I have added a
>> 4 port SATA III controller as a PCIex4 device. The 6 SATA II ports are
>> connected to 6 SAMSUNG HD103SJ 1TB hard drives. The on board SATA III ports
>> are connected to 2 Blue Ray drives. The 4 port SATA III controller is
>> connected to 4 Corsair Force LS 60GB SSDs.
>>
>> When booting without the hypervisor all drives operate as expected. When
>> booting with the hypervisor both Blue Ray drives and 2 of the SSD drives go
>> missing. dmesg shows errors 'failed to IDENTIFY (INIT_DEV_PARAMS failed,
>> err_mask=0x80)' and 'failed to IDENTIFY (I/O error, err_mask=0x4)' and
>> 'COMRESET failed (errno=-16)'; (see dmesg.txt and dmesg-xen.txt). lspci
>> doesn't show any real difference in the devices; just some interrupt IRQ
>> numbers.
>>
>> I've tried all (that I know of) iommu command line parameters with no luck;
>> no-intremap,pass-through,no-qinval,no-snoop,workaround_bios_bug.
>
> But I suppose you didn't try the "pci-phantom=" option that was
> specifically added for controllers like this (see
> http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=4e3c592c93d7dbe02ca36878457515d30fe931d2)?

It probably would have been better to link to the command-line
reference instead of a git commit:

http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html

As it is, the documentation is a bit sparse -- maybe next doc day I'll
try to write up a wiki page that will be easier to follow (and perhaps
more discoverable on Google).

I suppose there's no way we could add devices like this to a "quirks"
file, so that users don't have to figure this out on their own?

 -George