"iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

"iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Qian Cai]

The linux-next series "iommu/vt-d: Delegate DMA domain to generic iommu" [1]
causes a system with the rootfs on megaraid_sas card unable to boot.

Reverted the whole series on the top of linux-next (next-20190607) fixed the
issue.

The information regards this storage card is,

[  116.466810][  T324] megaraid_sas 0000:06:00.0: FW provided supportMaxExtLDs:
0	max_lds: 32
[  116.476052][  T324] megaraid_sas 0000:06:00.0: controller type	:
iMR(0MB)
[  116.483646][  T324] megaraid_sas 0000:06:00.0: Online Controller Reset(OCR)	
: Enabled
[  116.492403][  T324] megaraid_sas 0000:06:00.0: Secure JBOD support	:
Yes
[  116.499887][  T324] megaraid_sas 0000:06:00.0: NVMe passthru support	:
No
[  116.507480][  T324] megaraid_sas 0000:06:00.0: FW provided
[  116.612523][  T324] megaraid_sas 0000:06:00.0: NVME page size	: (0)
[  116.629991][  T324] megaraid_sas 0000:06:00.0: INIT adapter done
[  116.714789][  T324] megaraid_sas 0000:06:00.0: pci id		:
(0x1000)/(0x0017)/(0x1d49)/(0x0500)
[  116.724228][  T324] megaraid_sas 0000:06:00.0: unevenspan support	: no
[  116.731518][  T324] megaraid_sas 0000:06:00.0: firmware crash dump	:
no
[  116.738981][  T324] megaraid_sas 0000:06:00.0: jbod sync map		:
yes
[  116.787433][  T324] scsi host0: Avago SAS based MegaRAID driver
[  117.081088][  T324] scsi 0:0:0:0: Direct-
Access     LENOVO   ST900MM0168      L587 PQ: 0 ANSI: 6

[1] https://lore.kernel.org/patchwork/cover/1078960/

"iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Qian Cai]

The linux-next series "iommu/vt-d: Delegate DMA domain to generic iommu" [1]
causes a system with the rootfs on megaraid_sas card unable to boot.

Reverted the whole series on the top of linux-next (next-20190607) fixed the
issue.

The information regards this storage card is,

[  116.466810][  T324] megaraid_sas 0000:06:00.0: FW provided supportMaxExtLDs:
0	max_lds: 32
[  116.476052][  T324] megaraid_sas 0000:06:00.0: controller type	:
iMR(0MB)
[  116.483646][  T324] megaraid_sas 0000:06:00.0: Online Controller Reset(OCR)	
: Enabled
[  116.492403][  T324] megaraid_sas 0000:06:00.0: Secure JBOD support	:
Yes
[  116.499887][  T324] megaraid_sas 0000:06:00.0: NVMe passthru support	:
No
[  116.507480][  T324] megaraid_sas 0000:06:00.0: FW provided
[  116.612523][  T324] megaraid_sas 0000:06:00.0: NVME page size	: (0)
[  116.629991][  T324] megaraid_sas 0000:06:00.0: INIT adapter done
[  116.714789][  T324] megaraid_sas 0000:06:00.0: pci id		:
(0x1000)/(0x0017)/(0x1d49)/(0x0500)
[  116.724228][  T324] megaraid_sas 0000:06:00.0: unevenspan support	: no
[  116.731518][  T324] megaraid_sas 0000:06:00.0: firmware crash dump	:
no
[  116.738981][  T324] megaraid_sas 0000:06:00.0: jbod sync map		:
yes
[  116.787433][  T324] scsi host0: Avago SAS based MegaRAID driver
[  117.081088][  T324] scsi 0:0:0:0: Direct-
Access     LENOVO   ST900MM0168      L587 PQ: 0 ANSI: 6

[1] https://lore.kernel.org/patchwork/cover/1078960/
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Lu Baolu]

Hi Qian,

I just posted some fix patches. I cc'ed them in your email inbox as
well. Can you please check whether they happen to fix your issue?
If not, do you mind posting more debug messages?

Best regards,
Baolu


On 6/8/19 5:08 AM, Qian Cai wrote:
> The linux-next series "iommu/vt-d: Delegate DMA domain to generic iommu" [1]
> causes a system with the rootfs on megaraid_sas card unable to boot.
> 
> Reverted the whole series on the top of linux-next (next-20190607) fixed the
> issue.
> 
> The information regards this storage card is,
> 
> [  116.466810][  T324] megaraid_sas 0000:06:00.0: FW provided supportMaxExtLDs:
> 0	max_lds: 32
> [  116.476052][  T324] megaraid_sas 0000:06:00.0: controller type	:
> iMR(0MB)
> [  116.483646][  T324] megaraid_sas 0000:06:00.0: Online Controller Reset(OCR)	
> : Enabled
> [  116.492403][  T324] megaraid_sas 0000:06:00.0: Secure JBOD support	:
> Yes
> [  116.499887][  T324] megaraid_sas 0000:06:00.0: NVMe passthru support	:
> No
> [  116.507480][  T324] megaraid_sas 0000:06:00.0: FW provided
> [  116.612523][  T324] megaraid_sas 0000:06:00.0: NVME page size	: (0)
> [  116.629991][  T324] megaraid_sas 0000:06:00.0: INIT adapter done
> [  116.714789][  T324] megaraid_sas 0000:06:00.0: pci id		:
> (0x1000)/(0x0017)/(0x1d49)/(0x0500)
> [  116.724228][  T324] megaraid_sas 0000:06:00.0: unevenspan support	: no
> [  116.731518][  T324] megaraid_sas 0000:06:00.0: firmware crash dump	:
> no
> [  116.738981][  T324] megaraid_sas 0000:06:00.0: jbod sync map		:
> yes
> [  116.787433][  T324] scsi host0: Avago SAS based MegaRAID driver
> [  117.081088][  T324] scsi 0:0:0:0: Direct-
> Access     LENOVO   ST900MM0168      L587 PQ: 0 ANSI: 6
> 
> [1] https://lore.kernel.org/patchwork/cover/1078960/
> 

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Lu Baolu]

Hi Qian,

I just posted some fix patches. I cc'ed them in your email inbox as
well. Can you please check whether they happen to fix your issue?
If not, do you mind posting more debug messages?

Best regards,
Baolu


On 6/8/19 5:08 AM, Qian Cai wrote:
> The linux-next series "iommu/vt-d: Delegate DMA domain to generic iommu" [1]
> causes a system with the rootfs on megaraid_sas card unable to boot.
> 
> Reverted the whole series on the top of linux-next (next-20190607) fixed the
> issue.
> 
> The information regards this storage card is,
> 
> [  116.466810][  T324] megaraid_sas 0000:06:00.0: FW provided supportMaxExtLDs:
> 0	max_lds: 32
> [  116.476052][  T324] megaraid_sas 0000:06:00.0: controller type	:
> iMR(0MB)
> [  116.483646][  T324] megaraid_sas 0000:06:00.0: Online Controller Reset(OCR)	
> : Enabled
> [  116.492403][  T324] megaraid_sas 0000:06:00.0: Secure JBOD support	:
> Yes
> [  116.499887][  T324] megaraid_sas 0000:06:00.0: NVMe passthru support	:
> No
> [  116.507480][  T324] megaraid_sas 0000:06:00.0: FW provided
> [  116.612523][  T324] megaraid_sas 0000:06:00.0: NVME page size	: (0)
> [  116.629991][  T324] megaraid_sas 0000:06:00.0: INIT adapter done
> [  116.714789][  T324] megaraid_sas 0000:06:00.0: pci id		:
> (0x1000)/(0x0017)/(0x1d49)/(0x0500)
> [  116.724228][  T324] megaraid_sas 0000:06:00.0: unevenspan support	: no
> [  116.731518][  T324] megaraid_sas 0000:06:00.0: firmware crash dump	:
> no
> [  116.738981][  T324] megaraid_sas 0000:06:00.0: jbod sync map		:
> yes
> [  116.787433][  T324] scsi host0: Avago SAS based MegaRAID driver
> [  117.081088][  T324] scsi 0:0:0:0: Direct-
> Access     LENOVO   ST900MM0168      L587 PQ: 0 ANSI: 6
> 
> [1] https://lore.kernel.org/patchwork/cover/1078960/
> 
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Qian Cai]

On Sun, 2019-06-09 at 10:43 +0800, Lu Baolu wrote:
> Hi Qian,
> 
> I just posted some fix patches. I cc'ed them in your email inbox as
> well. Can you please check whether they happen to fix your issue?
> If not, do you mind posting more debug messages?

Unfortunately, it does not work. Here is the dmesg.

https://raw.githubusercontent.com/cailca/tmp/master/dmesg?token=AMC35QKPIZBYUMFU
QKLW4ZC47ZPIK

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Qian Cai]

On Sun, 2019-06-09 at 10:43 +0800, Lu Baolu wrote:
> Hi Qian,
> 
> I just posted some fix patches. I cc'ed them in your email inbox as
> well. Can you please check whether they happen to fix your issue?
> If not, do you mind posting more debug messages?

Unfortunately, it does not work. Here is the dmesg.

https://raw.githubusercontent.com/cailca/tmp/master/dmesg?token=AMC35QKPIZBYUMFU
QKLW4ZC47ZPIK

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Qian Cai]

On Mon, 2019-06-10 at 09:44 -0400, Qian Cai wrote:
> On Sun, 2019-06-09 at 10:43 +0800, Lu Baolu wrote:
> > Hi Qian,
> > 
> > I just posted some fix patches. I cc'ed them in your email inbox as
> > well. Can you please check whether they happen to fix your issue?
> > If not, do you mind posting more debug messages?
> 
> Unfortunately, it does not work. Here is the dmesg.
> 
> https://raw.githubusercontent.com/cailca/tmp/master/dmesg?token=AMC35QKPIZBYUM
> FUQKLW4ZC47ZPIK

This one should be good to view.

https://cailca.github.io/files/dmesg.txt

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Qian Cai]

On Mon, 2019-06-10 at 09:44 -0400, Qian Cai wrote:
> On Sun, 2019-06-09 at 10:43 +0800, Lu Baolu wrote:
> > Hi Qian,
> > 
> > I just posted some fix patches. I cc'ed them in your email inbox as
> > well. Can you please check whether they happen to fix your issue?
> > If not, do you mind posting more debug messages?
> 
> Unfortunately, it does not work. Here is the dmesg.
> 
> https://raw.githubusercontent.com/cailca/tmp/master/dmesg?token=AMC35QKPIZBYUM
> FUQKLW4ZC47ZPIK

This one should be good to view.

https://cailca.github.io/files/dmesg.txt
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Lu Baolu]

[-- Attachment #1: Type: text/plain, Size: 956 bytes --]

Ah, good catch!

The device failed to be attached by a DMA domain. Can you please try the
attached fix patch?

[  101.885468] pci 0000:06:00.0: DMAR: Device is ineligible for IOMMU
domain attach due to platform RMRR requirement.  Contact your platform
vendor.
[  101.900801] pci 0000:06:00.0: Failed to add to iommu group 23: -1

Best regards,
Baolu

On 6/10/19 10:54 PM, Qian Cai wrote:
> On Mon, 2019-06-10 at 09:44 -0400, Qian Cai wrote:
>> On Sun, 2019-06-09 at 10:43 +0800, Lu Baolu wrote:
>>> Hi Qian,
>>>
>>> I just posted some fix patches. I cc'ed them in your email inbox as
>>> well. Can you please check whether they happen to fix your issue?
>>> If not, do you mind posting more debug messages?
>>
>> Unfortunately, it does not work. Here is the dmesg.
>>
>> https://raw.githubusercontent.com/cailca/tmp/master/dmesg?token=AMC35QKPIZBYUM
>> FUQKLW4ZC47ZPIK
> 
> This one should be good to view.
> 
> https://cailca.github.io/files/dmesg.txt
> 

[-- Attachment #2: 0001-iommu-vt-d-Allow-DMA-domain-attaching-to-rmrr-locked.patch --]
[-- Type: text/x-patch, Size: 1361 bytes --]

From ff0b1ae0d8fde0655392fde3a1090b03a7a35394 Mon Sep 17 00:00:00 2001
From: Lu Baolu <baolu.lu@linux.intel.com>
Date: Tue, 11 Jun 2019 09:29:16 +0800
Subject: [PATCH 1/1] iommu/vt-d: Allow DMA domain attaching to rmrr locked
 device

We don't allow a device to be assigned to user level when it is locked
by any RMRR's. Hence, intel_iommu_attach_device() will return error if
a domain of type IOMMU_DOMAIN_UNMANAGED is about to attach to a device
locked by rmrr. But this doesn't apply to a domain of type other than
IOMMU_DOMAIN_UNMANAGED. This adds a check to fix this.

Fixes: fa954e6831789 ("iommu/vt-d: Delegate the dma domain to upper layer")
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
---
 drivers/iommu/intel-iommu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 1dcb6365ddc4..38232220f6ff 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -5281,7 +5281,8 @@ static int intel_iommu_attach_device(struct iommu_domain *domain,
 {
 	int ret;
 
-	if (device_is_rmrr_locked(dev)) {
+	if (domain->type == IOMMU_DOMAIN_UNMANAGED &&
+	    device_is_rmrr_locked(dev)) {
 		dev_warn(dev, "Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.\n");
 		return -EPERM;
 	}
-- 
2.17.1

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Lu Baolu]

[-- Attachment #1: Type: text/plain, Size: 956 bytes --]

Ah, good catch!

The device failed to be attached by a DMA domain. Can you please try the
attached fix patch?

[  101.885468] pci 0000:06:00.0: DMAR: Device is ineligible for IOMMU
domain attach due to platform RMRR requirement.  Contact your platform
vendor.
[  101.900801] pci 0000:06:00.0: Failed to add to iommu group 23: -1

Best regards,
Baolu

On 6/10/19 10:54 PM, Qian Cai wrote:
> On Mon, 2019-06-10 at 09:44 -0400, Qian Cai wrote:
>> On Sun, 2019-06-09 at 10:43 +0800, Lu Baolu wrote:
>>> Hi Qian,
>>>
>>> I just posted some fix patches. I cc'ed them in your email inbox as
>>> well. Can you please check whether they happen to fix your issue?
>>> If not, do you mind posting more debug messages?
>>
>> Unfortunately, it does not work. Here is the dmesg.
>>
>> https://raw.githubusercontent.com/cailca/tmp/master/dmesg?token=AMC35QKPIZBYUM
>> FUQKLW4ZC47ZPIK
> 
> This one should be good to view.
> 
> https://cailca.github.io/files/dmesg.txt
> 

[-- Attachment #2: 0001-iommu-vt-d-Allow-DMA-domain-attaching-to-rmrr-locked.patch --]
[-- Type: text/x-patch, Size: 1361 bytes --]

From ff0b1ae0d8fde0655392fde3a1090b03a7a35394 Mon Sep 17 00:00:00 2001
From: Lu Baolu <baolu.lu@linux.intel.com>
Date: Tue, 11 Jun 2019 09:29:16 +0800
Subject: [PATCH 1/1] iommu/vt-d: Allow DMA domain attaching to rmrr locked
 device

We don't allow a device to be assigned to user level when it is locked
by any RMRR's. Hence, intel_iommu_attach_device() will return error if
a domain of type IOMMU_DOMAIN_UNMANAGED is about to attach to a device
locked by rmrr. But this doesn't apply to a domain of type other than
IOMMU_DOMAIN_UNMANAGED. This adds a check to fix this.

Fixes: fa954e6831789 ("iommu/vt-d: Delegate the dma domain to upper layer")
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
---
 drivers/iommu/intel-iommu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 1dcb6365ddc4..38232220f6ff 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -5281,7 +5281,8 @@ static int intel_iommu_attach_device(struct iommu_domain *domain,
 {
 	int ret;
 
-	if (device_is_rmrr_locked(dev)) {
+	if (domain->type == IOMMU_DOMAIN_UNMANAGED &&
+	    device_is_rmrr_locked(dev)) {
 		dev_warn(dev, "Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.\n");
 		return -EPERM;
 	}
-- 
2.17.1


[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Qian Cai]

> On Jun 10, 2019, at 9:41 PM, Lu Baolu <baolu.lu@linux.intel.com> wrote:
> 
> Ah, good catch!
> 
> The device failed to be attached by a DMA domain. Can you please try the
> attached fix patch?

It works fine.

> 
> [  101.885468] pci 0000:06:00.0: DMAR: Device is ineligible for IOMMU
> domain attach due to platform RMRR requirement.  Contact your platform
> vendor.
> [  101.900801] pci 0000:06:00.0: Failed to add to iommu group 23: -1
> 
> Best regards,
> Baolu
> 
> On 6/10/19 10:54 PM, Qian Cai wrote:
>> On Mon, 2019-06-10 at 09:44 -0400, Qian Cai wrote:
>>> On Sun, 2019-06-09 at 10:43 +0800, Lu Baolu wrote:
>>>> Hi Qian,
>>>> 
>>>> I just posted some fix patches. I cc'ed them in your email inbox as
>>>> well. Can you please check whether they happen to fix your issue?
>>>> If not, do you mind posting more debug messages?
>>> 
>>> Unfortunately, it does not work. Here is the dmesg.
>>> 
>>> https://raw.githubusercontent.com/cailca/tmp/master/dmesg?token=AMC35QKPIZBYUM
>>> FUQKLW4ZC47ZPIK
>> This one should be good to view.
>> https://cailca.github.io/files/dmesg.txt
> <0001-iommu-vt-d-Allow-DMA-domain-attaching-to-rmrr-locked.patch>

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Qian Cai]

> On Jun 10, 2019, at 9:41 PM, Lu Baolu <baolu.lu@linux.intel.com> wrote:
> 
> Ah, good catch!
> 
> The device failed to be attached by a DMA domain. Can you please try the
> attached fix patch?

It works fine.

> 
> [  101.885468] pci 0000:06:00.0: DMAR: Device is ineligible for IOMMU
> domain attach due to platform RMRR requirement.  Contact your platform
> vendor.
> [  101.900801] pci 0000:06:00.0: Failed to add to iommu group 23: -1
> 
> Best regards,
> Baolu
> 
> On 6/10/19 10:54 PM, Qian Cai wrote:
>> On Mon, 2019-06-10 at 09:44 -0400, Qian Cai wrote:
>>> On Sun, 2019-06-09 at 10:43 +0800, Lu Baolu wrote:
>>>> Hi Qian,
>>>> 
>>>> I just posted some fix patches. I cc'ed them in your email inbox as
>>>> well. Can you please check whether they happen to fix your issue?
>>>> If not, do you mind posting more debug messages?
>>> 
>>> Unfortunately, it does not work. Here is the dmesg.
>>> 
>>> https://raw.githubusercontent.com/cailca/tmp/master/dmesg?token=AMC35QKPIZBYUM
>>> FUQKLW4ZC47ZPIK
>> This one should be good to view.
>> https://cailca.github.io/files/dmesg.txt
> <0001-iommu-vt-d-Allow-DMA-domain-attaching-to-rmrr-locked.patch>

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Lu Baolu]

Hi,

On 6/11/19 10:00 PM, Qian Cai wrote:
> 
>> On Jun 10, 2019, at 9:41 PM, Lu Baolu<baolu.lu@linux.intel.com>  wrote:
>>
>> Ah, good catch!
>>
>> The device failed to be attached by a DMA domain. Can you please try the
>> attached fix patch?
> It works fine.
> 

Thanks a lot for the report and verification.

Best regards,
Baolu

Re: "iommu/vt-d: Delegate DMA domain to generic iommu" series breaks megaraid_sas

[Lu Baolu]

Hi,

On 6/11/19 10:00 PM, Qian Cai wrote:
> 
>> On Jun 10, 2019, at 9:41 PM, Lu Baolu<baolu.lu@linux.intel.com>  wrote:
>>
>> Ah, good catch!
>>
>> The device failed to be attached by a DMA domain. Can you please try the
>> attached fix patch?
> It works fine.
> 

Thanks a lot for the report and verification.

Best regards,
Baolu
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu