mptsas/iommu/pciehp : PCIe hotplug of LSISAS1064E fails with intel_iommu=on

mptsas/iommu/pciehp : PCIe hotplug of LSISAS1064E fails with intel_iommu=on

[Isabelle, Francois]

Hi.

An issue has been raised with the SAS1064E when used on an Intel 5520 platform: pcie hot-plug of the controller does not work.

Since I can't access the archive for 'DL-MPTFusionLinux@lsi.com' and since I didn't receive a reply from MPTFusion list yet, I'm posting the thread here. It seems to be the right place to discuss MPT fusion driver anyways.

It appears that the MPT driver misbehaves; I'd like to know if there are any success stories when mixing VT-d technology with PCIe hot plug on a similar platform.

Instead of duplicating the whole thread I'm sending these links to the original posts.

http://marc.info/?l=linux-pci&m=125252723005873&w=2 [original post]
http://marc.info/?l=linux-pci&m=125268717610600&w=2 [reply]
http://marc.info/?l=linux-pci&m=125268984516738&w=2 [follow up post]

I'm also reposting to IOMMU list with a new 'topic' line that might be more meaningful. 

Thank you.

François Isabelle | Software Designer | Kontron Canada | T 450 437 5682 |F 450 437 8053 | E francois.isabelle@ca.kontron.com
 

RE: mptsas/iommu/pciehp : PCIe hotplug of LSISAS1064E fails with intel_iommu=on

[Isabelle, Francois]

>>PrimeIocFifos calls pci_alloc_consistent() and has debug code to dump
>>the DMA resource allocated (both virtual and DMA addresses). Off hand
>>I don't know how to enable that but it would be the next step.

With mpt_debug_level=0x8060 , I was able to get the allocation code to be more verbose, see below.

>>> DRHD: handling fault status reg 2
>>> DMAR:[DMA Read] Request device [06:00.0] fault addr fffc2000

>>fffc2000 seems to be an unusual address to DMA from/to.
>>Is fffc2000 reserved address space for the IOMMU?
>> (ACPI DMAR info should tell us this)

It seems to be the actual address (see below); however I think there might be some kind of page overlap between the 'reply buffer' and 'request buffer' ...

mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
mptbase: ioc0: RequestBuffers @ ffff88003b182800[00000000fffc2800]

> > > > DMAR:[fault reason 06] PTE Read access is not set
> > > It's also odd that "Read Access is not set" for something (ioc_init)
> > > that I think should be bi-directional.

Wouldn't that depend on the IOMMU driver? Or is the device driver supposed to set protections?

When we compare the 2 'init' we see differences in the 'mem_phys' entry, which I think is weird, especially if it should match the PCI memory region of the device
>>      Region 1: Memory at dbffc000 (64-bit, non-prefetchable) [size=16K]
>>      Region 3: Memory at dbfe0000 (64-bit, non-prefetchable) [size=64K]

mptbase: ioc0: mem = ffffc90012e40000, mem_phys = dbffc000
mptbase: ioc0: facts @ ffff88003dc2541c, pfacts[0] @ ffff88003dc2546c
...
mptbase: ioc1: mem = ffffc90012e48000, mem_phys = d8010000
mptbase: ioc1: facts @ ffff88003d21041c, pfacts[0] @ ffff88003d21046c

..and we see differences in the chain buffers..

mptbase: ioc0: ReqToChain alloc  @ ffff88003d4d8800, sz=1108 bytes
mptbase: ioc0: RequestNB alloc  @ ffff88003e3f3000, sz=1108 bytes
mptbase: ioc0: ChainToChain alloc @ ffff88003db14000, sz=4572 bytes

mptbase: ioc1: ReqToChain alloc  @ ffff88003d2aa000, sz=1108 bytes
mptbase: ioc1: RequestNB alloc  @ ffff88003d268800, sz=1108 bytes
mptbase: ioc1: ChainToChain alloc @ ffff88003dfda000, sz=4572 bytes


The other thing that is now obvious is that the device name has changed (ioc1 instead of ioc0); this makes me think that the whole problem is probably more closely related to the resources unallocation, maybe a missing 'unlink' operation of some kind... I'm looking into this, but I could still use some help if anyone has an idea.

Following is the complete MPTSAS init log.

==== First time init  ===

mptsas 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
mptbase: ioc0: : 32 BIT PCI BUS DMA ADDRESSING SUPPORTED
mptbase: ioc0: mem = ffffc90012e40000, mem_phys = dbffc000
mptbase: ioc0: facts @ ffff88003dc2541c, pfacts[0] @ ffff88003dc2546c
mptbase: ioc0: Initiating bringup
mptbase: ioc0: IOC is in READY state
mptbase: ioc0: Sending get IocFacts request req_sz=12 reply_sz=80
mptbase: ioc0: NB_for_64_byte_frame=2 NBShiftFactor=5 BlockSize=8
mptbase: ioc0: reply_sz= 80, reply_depth= 128
mptbase: ioc0: req_sz  =128, req_depth  = 277
mptbase: ioc0: Sending get PortFacts(0) request
ioc0: LSISAS1064E B3: Capabilities={Initiator}
mptsas 0000:06:00.0: setting latency timer to 64
mptbase: ioc0: installed at interrupt 16
mptbase: ioc0: PrimeIocFifos
mptbase: ioc0: ReqToChain alloc  @ ffff88003d4d8800, sz=1108 bytes
mptbase: ioc0: RequestNB alloc  @ ffff88003e3f3000, sz=1108 bytes
mptbase: ioc0: num_sge=25 numSGE=520
mptbase: ioc0: Now numSGE=128 num_sge=130 num_chain=9
mptbase: ioc0: ChainToChain alloc @ ffff88003db14000, sz=4572 bytes
mptbase: ioc0: ReplyBuffer sz=80 bytes, ReplyDepth=128
mptbase: ioc0: ReplyBuffer sz=10240[2800] bytes
mptbase: ioc0: RequestBuffer sz=128 bytes, RequestDepth=277
mptbase: ioc0: RequestBuffer sz=35456[8a80] bytes
mptbase: ioc0: ChainBuffer sz=128 bytes, ChainDepth=1143
mptbase: ioc0: ChainBuffer sz=146304[23b80] bytes num_chain=1143
mptbase: ioc0: Total alloc @ ffff88003b180000[00000000fffc0000], sz=192000[2ee00] bytes
mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
mptbase: ioc0: RequestBuffers @ ffff88003b182800[00000000fffc2800]
mptbase: ioc0: ChainBuffers @ ffff88003b18b280(00000000fffcb280)
mptbase: ioc0: SenseBuffers @ ffff88003d2f8000[00000000fffb8000]
mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
mptbase: ioc0: SendIocInit
mptbase: ioc0: facts.MsgVersion=105
mptbase: ioc0: Sending Port(0)Enable (req @ ffff88003cc63a98)
mptbase: ioc0: Wait IOC_OPERATIONAL state (cnt=0)
mptbase: ioc0: SendEventNotification
scsi15 : ioc0: LSISAS1064E B3, FwRev=011b0000h, Ports=1, MaxQ=277, IRQ=16

==== Reinit after PCIe Hotplug  === 

tsas 0000:06:00.0: enabling device (0000 -> 0002)
mptsas 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
mptbase: ioc1: : 32 BIT PCI BUS DMA ADDRESSING SUPPORTED
mptbase: ioc1: mem = ffffc90012e48000, mem_phys = d8010000
mptbase: ioc1: facts @ ffff88003d21041c, pfacts[0] @ ffff88003d21046c
mptbase: ioc1: Initiating bringup
mptbase: ioc1: IOC is in READY state
mptbase: ioc1: Sending get IocFacts request req_sz=12 reply_sz=80
mptbase: ioc1: NB_for_64_byte_frame=2 NBShiftFactor=5 BlockSize=8
mptbase: ioc1: reply_sz= 80, reply_depth= 128
mptbase: ioc1: req_sz  =128, req_depth  = 277
mptbase: ioc1: Sending get PortFacts(0) request
ioc1: LSISAS1064E B3: Capabilities={Initiator}
mptsas 0000:06:00.0: setting latency timer to 64
mptbase: ioc1: installed at interrupt 16
mptbase: ioc1: PrimeIocFifos
mptbase: ioc1: ReqToChain alloc  @ ffff88003d2aa000, sz=1108 bytes
mptbase: ioc1: RequestNB alloc  @ ffff88003d268800, sz=1108 bytes
mptbase: ioc1: num_sge=25 numSGE=520
mptbase: ioc1: Now numSGE=128 num_sge=130 num_chain=9
mptbase: ioc1: ChainToChain alloc @ ffff88003dfda000, sz=4572 bytes
mptbase: ioc1: ReplyBuffer sz=80 bytes, ReplyDepth=128
mptbase: ioc1: ReplyBuffer sz=10240[2800] bytes                  ==========> (2800h bytes)
mptbase: ioc1: RequestBuffer sz=128 bytes, RequestDepth=277
mptbase: ioc1: RequestBuffer sz=35456[8a80] bytes
mptbase: ioc1: ChainBuffer sz=128 bytes, ChainDepth=1143
mptbase: ioc1: ChainBuffer sz=146304[23b80] bytes num_chain=1143
mptbase: ioc1: Total alloc @ ffff88003b180000[00000000fffc0000], sz=192000[2ee00] bytes
mptbase: ioc1: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
mptbase: ioc1: RequestBuffers @ ffff88003b182800[00000000fffc2800]
mptbase: ioc1: ChainBuffers @ ffff88003b18b280(00000000fffcb280)
mptbase: ioc1: SenseBuffers @ ffff88003d2f8000[00000000fffb8000]
mptbase: ioc1: ReplyBuffers @ ffff88003b180000[00000000fffc0000] ==========>  (2800h bytes)
mptbase: ioc1: SendIocInit
mptbase: ioc1: facts.MsgVersion=105
mptbase: ioc1: Sending Port(0)Enable (req @ ffff88003e28d880)
deb64:~#


b64:~# mptbase: ioc1: WARNING - Issuing Reset from mpt_config!!
mptbase: ioc1: Initiating recovery
mptbase: ioc1: WARNING - IOC is in FAULT state!!!
mptbase: ioc1: WARNING -            FAULT code = 2000h
mptbase: ioc1: Recovered from IOC FAULT
mptbase: ioc1: PrimeIocFifos
mptbase: ioc1: SendIocInit
mptbase: ioc1: SendEventNotification
mptbase: ioc1: Attempting Retry Config request type 0x1, page 0x2, action 0
DRHD: handling fault status reg 102
DMAR:[DMA Read] Request device [06:00.0] fault addr fffc2000
DMAR:[fault reason 06] PTE Read access is not set


Thank you

François Isabelle | Software Designer | Kontron Canada | T 450 437 5682 |F 450 437 8053 | E francois.isabelle@ca.kontron.com

Re: mptsas/iommu/pciehp : PCIe hotplug of LSISAS1064E fails with intel_iommu=on

[Grant Grundler]

On Wed, Sep 16, 2009 at 12:04:27PM -0400, Isabelle, Francois wrote:
> >>PrimeIocFifos calls pci_alloc_consistent() and has debug code to dump
> >>the DMA resource allocated (both virtual and DMA addresses). Off hand
> >>I don't know how to enable that but it would be the next step.
> 
> With mpt_debug_level=0x8060 , I was able to get the allocation code to be more verbose, see below.

Cool - noted.

> >>> DRHD: handling fault status reg 2
> >>> DMAR:[DMA Read] Request device [06:00.0] fault addr fffc2000
> 
> >>fffc2000 seems to be an unusual address to DMA from/to.
> >>Is fffc2000 reserved address space for the IOMMU?
> >> (ACPI DMAR info should tell us this)
> 
> It seems to be the actual address (see below); however I think there might be some kind of page overlap between the 'reply buffer' and 'request buffer' ...
> 
> mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
> mptbase: ioc0: RequestBuffers @ ffff88003b182800[00000000fffc2800]

Excellent. This corresponds with:
                dinitprintk(ioc, printk(MYIOC_s_DEBUG_FMT "ReplyBuffers @ %p[%p]\n",
                        ioc->name, ioc->reply_frames, (void *)(ulong)alloc_dma));

alloc_dma is the DMA Mapped address.

I don't believe the two ranges overlap based on this output:
> mptbase: ioc0: ReplyBuffer sz=80 bytes, ReplyDepth=128
> mptbase: ioc0: ReplyBuffer sz=10240[2800] bytes

that you included at the bottom of the email.


> 
> > > > > DMAR:[fault reason 06] PTE Read access is not set
> > > > It's also odd that "Read Access is not set" for something (ioc_init)
> > > > that I think should be bi-directional.
> 
> Wouldn't that depend on the IOMMU driver? Or is the device driver supposed to set protections?

Yes - IOMMU support owns that. Anything allocated/mapped with
pci_alloc_consistent() needs to have both read and write permissions
for both Host and PCI target. This definitely not a device driver
problem. Just need to verify the access was to that range.
I've lost track of the original address that was the problem.

> When we compare the 2 'init' we see differences in the 'mem_phys' entry, which I think is weird, especially if it should match the PCI memory region of the device
> >>      Region 1: Memory at dbffc000 (64-bit, non-prefetchable) [size=16K]
> >>      Region 3: Memory at dbfe0000 (64-bit, non-prefetchable) [size=64K]
> 
> mptbase: ioc0: mem = ffffc90012e40000, mem_phys = dbffc000
> mptbase: ioc0: facts @ ffff88003dc2541c, pfacts[0] @ ffff88003dc2546c
> ...
> mptbase: ioc1: mem = ffffc90012e48000, mem_phys = d8010000
> mptbase: ioc1: facts @ ffff88003d21041c, pfacts[0] @ ffff88003d21046c

The Region 1 and Region 3 are for the same controller (ioc0).
Not for two different controllers. The second controller should
have it's own pair of Region 1 and Region 3 configuration space registers.

> ..and we see differences in the chain buffers..
> 
> mptbase: ioc0: ReqToChain alloc  @ ffff88003d4d8800, sz=1108 bytes
> mptbase: ioc0: RequestNB alloc  @ ffff88003e3f3000, sz=1108 bytes
> mptbase: ioc0: ChainToChain alloc @ ffff88003db14000, sz=4572 bytes
> 
> mptbase: ioc1: ReqToChain alloc  @ ffff88003d2aa000, sz=1108 bytes
> mptbase: ioc1: RequestNB alloc  @ ffff88003d268800, sz=1108 bytes
> mptbase: ioc1: ChainToChain alloc @ ffff88003dfda000, sz=4572 bytes

I'm not sure what you are comparing with the previous 6 lines of output.
The two controller (ioc0 and ioc1) are required to have different
chain buffers.

> The other thing that is now obvious is that the device name has
> changed (ioc1 instead of ioc0);

Yes - good catch.

> this makes me think that the whole
> problem is probably more closely related to the resources unallocation,
> maybe a missing 'unlink' operation of some kind... I'm looking into
> this, but I could still use some help if anyone has an idea.

I'd be looking for why the ioc number changed. It should be resuming
with the same instance number since it's the same physical device.

hth,
grant

> 
> Following is the complete MPTSAS init log.
> 
> ==== First time init  ===
> 
> mptsas 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> mptbase: ioc0: : 32 BIT PCI BUS DMA ADDRESSING SUPPORTED
> mptbase: ioc0: mem = ffffc90012e40000, mem_phys = dbffc000
> mptbase: ioc0: facts @ ffff88003dc2541c, pfacts[0] @ ffff88003dc2546c
> mptbase: ioc0: Initiating bringup
> mptbase: ioc0: IOC is in READY state
> mptbase: ioc0: Sending get IocFacts request req_sz=12 reply_sz=80
> mptbase: ioc0: NB_for_64_byte_frame=2 NBShiftFactor=5 BlockSize=8
> mptbase: ioc0: reply_sz= 80, reply_depth= 128
> mptbase: ioc0: req_sz  =128, req_depth  = 277
> mptbase: ioc0: Sending get PortFacts(0) request
> ioc0: LSISAS1064E B3: Capabilities={Initiator}
> mptsas 0000:06:00.0: setting latency timer to 64
> mptbase: ioc0: installed at interrupt 16
> mptbase: ioc0: PrimeIocFifos
> mptbase: ioc0: ReqToChain alloc  @ ffff88003d4d8800, sz=1108 bytes
> mptbase: ioc0: RequestNB alloc  @ ffff88003e3f3000, sz=1108 bytes
> mptbase: ioc0: num_sge=25 numSGE=520
> mptbase: ioc0: Now numSGE=128 num_sge=130 num_chain=9
> mptbase: ioc0: ChainToChain alloc @ ffff88003db14000, sz=4572 bytes
> mptbase: ioc0: ReplyBuffer sz=80 bytes, ReplyDepth=128
> mptbase: ioc0: ReplyBuffer sz=10240[2800] bytes
> mptbase: ioc0: RequestBuffer sz=128 bytes, RequestDepth=277
> mptbase: ioc0: RequestBuffer sz=35456[8a80] bytes
> mptbase: ioc0: ChainBuffer sz=128 bytes, ChainDepth=1143
> mptbase: ioc0: ChainBuffer sz=146304[23b80] bytes num_chain=1143
> mptbase: ioc0: Total alloc @ ffff88003b180000[00000000fffc0000], sz=192000[2ee00] bytes
> mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
> mptbase: ioc0: RequestBuffers @ ffff88003b182800[00000000fffc2800]
> mptbase: ioc0: ChainBuffers @ ffff88003b18b280(00000000fffcb280)
> mptbase: ioc0: SenseBuffers @ ffff88003d2f8000[00000000fffb8000]
> mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
> mptbase: ioc0: SendIocInit
> mptbase: ioc0: facts.MsgVersion=105
> mptbase: ioc0: Sending Port(0)Enable (req @ ffff88003cc63a98)
> mptbase: ioc0: Wait IOC_OPERATIONAL state (cnt=0)
> mptbase: ioc0: SendEventNotification
> scsi15 : ioc0: LSISAS1064E B3, FwRev=011b0000h, Ports=1, MaxQ=277, IRQ=16
> 
> ==== Reinit after PCIe Hotplug  === 
> 
> tsas 0000:06:00.0: enabling device (0000 -> 0002)
> mptsas 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> mptbase: ioc1: : 32 BIT PCI BUS DMA ADDRESSING SUPPORTED
> mptbase: ioc1: mem = ffffc90012e48000, mem_phys = d8010000
> mptbase: ioc1: facts @ ffff88003d21041c, pfacts[0] @ ffff88003d21046c
> mptbase: ioc1: Initiating bringup
> mptbase: ioc1: IOC is in READY state
> mptbase: ioc1: Sending get IocFacts request req_sz=12 reply_sz=80
> mptbase: ioc1: NB_for_64_byte_frame=2 NBShiftFactor=5 BlockSize=8
> mptbase: ioc1: reply_sz= 80, reply_depth= 128
> mptbase: ioc1: req_sz  =128, req_depth  = 277
> mptbase: ioc1: Sending get PortFacts(0) request
> ioc1: LSISAS1064E B3: Capabilities={Initiator}
> mptsas 0000:06:00.0: setting latency timer to 64
> mptbase: ioc1: installed at interrupt 16
> mptbase: ioc1: PrimeIocFifos
> mptbase: ioc1: ReqToChain alloc  @ ffff88003d2aa000, sz=1108 bytes
> mptbase: ioc1: RequestNB alloc  @ ffff88003d268800, sz=1108 bytes
> mptbase: ioc1: num_sge=25 numSGE=520
> mptbase: ioc1: Now numSGE=128 num_sge=130 num_chain=9
> mptbase: ioc1: ChainToChain alloc @ ffff88003dfda000, sz=4572 bytes
> mptbase: ioc1: ReplyBuffer sz=80 bytes, ReplyDepth=128
> mptbase: ioc1: ReplyBuffer sz=10240[2800] bytes                  ==========> (2800h bytes)
> mptbase: ioc1: RequestBuffer sz=128 bytes, RequestDepth=277
> mptbase: ioc1: RequestBuffer sz=35456[8a80] bytes
> mptbase: ioc1: ChainBuffer sz=128 bytes, ChainDepth=1143
> mptbase: ioc1: ChainBuffer sz=146304[23b80] bytes num_chain=1143
> mptbase: ioc1: Total alloc @ ffff88003b180000[00000000fffc0000], sz=192000[2ee00] bytes
> mptbase: ioc1: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
> mptbase: ioc1: RequestBuffers @ ffff88003b182800[00000000fffc2800]
> mptbase: ioc1: ChainBuffers @ ffff88003b18b280(00000000fffcb280)
> mptbase: ioc1: SenseBuffers @ ffff88003d2f8000[00000000fffb8000]
> mptbase: ioc1: ReplyBuffers @ ffff88003b180000[00000000fffc0000] ==========>  (2800h bytes)
> mptbase: ioc1: SendIocInit
> mptbase: ioc1: facts.MsgVersion=105
> mptbase: ioc1: Sending Port(0)Enable (req @ ffff88003e28d880)
> deb64:~#
> 
> 
> b64:~# mptbase: ioc1: WARNING - Issuing Reset from mpt_config!!
> mptbase: ioc1: Initiating recovery
> mptbase: ioc1: WARNING - IOC is in FAULT state!!!
> mptbase: ioc1: WARNING -            FAULT code = 2000h
> mptbase: ioc1: Recovered from IOC FAULT
> mptbase: ioc1: PrimeIocFifos
> mptbase: ioc1: SendIocInit
> mptbase: ioc1: SendEventNotification
> mptbase: ioc1: Attempting Retry Config request type 0x1, page 0x2, action 0
> DRHD: handling fault status reg 102
> DMAR:[DMA Read] Request device [06:00.0] fault addr fffc2000
> DMAR:[fault reason 06] PTE Read access is not set
> 
> 
> Thank you
> 
> François Isabelle | Software Designer | Kontron Canada | T 450 437 5682 |F 450 437 8053 | E francois.isabelle@ca.kontron.com

RE: mptsas/iommu/pciehp : PCIe hotplug of LSISAS1064E fails withintel_iommu=on

[Isabelle, Francois]

> Yes - IOMMU support owns that. Anything allocated/mapped with
> pci_alloc_consistent() needs to have both read and write permissions
> for both Host and PCI target. This definitely not a device driver
> problem. Just need to verify the access was to that range.
> I've lost track of the original address that was the problem.

It was 0xfffc2000, which is in the range allocated (alloc_dma)

>>> I'd be looking for why the ioc number changed. It should be resuming
>> with the same instance number since it's the same physical device.

Yes , that's what I though as well.

I tried to trace the expected call sequence , it seems like pci resources are correctly unassigned.

The folling sequence is executed on removal:

pci_remove_bus_device
pci_stop_dev
MPT:detach
pci_destroy_devs
pci_free_resources

I figured out that the ioc 'id' is not decremented in mpt_detach() and that it's used for some kind of reverse lookup by mptsas.c and by mpt_verify_adapter(), mostly used for mptclt. But it appears to be almost only cosmetical...

...
On the IOMMU side, I see that it sets mapping (CONTEXT_TT_MULTI_LEVEL)) for the SAS controller.

#ipmitool  picmg policy set 2 1 0
deb64:~# pciehp 0000:00:04.0:pcie04: Card present on Slot(52)
pciehp 0000:00:04.0:pcie04: Latch close on Slot(52)
pciehp 0000:00:04.0:pcie04: Button pressed on Slot(52)
pciehp 0000:00:04.0:pcie04: PCI slot #52 - powering on due to button press.
Fusion MPT base driver 3.04.12
Copyright (c) 1999-2008 LSI Corporation
Fusion MPT SAS Host driver 3.04.12
mptsas 0000:06:00.0: enabling device (0000 -> 0002)
mptsas 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
mptbase: ioc0: Initiating bringup
ioc0: LSISAS1064E B3: Capabilities={Initiator}
IOMMU: get valid domain for 0000:06:00.0
IOMMU: mapping 0000:06:00.0
IOMMU: lsi 06:00.0
IOMMU: continue mapping for 06:00.0
IOMMU: mapping for 06:00.0 completed
IOMMU: mapping no upstream  0000:06:00.0
scsi7 : ioc0: LSISAS1064E B3, FwRev=011b0000h, Ports=1, MaxQ=277, IRQ=16

...
I looked at the driver correctness for alloc/free of dma memory, and it seems the play by the book as long as the DMA goes. With some extra debugging I get:


ptbase: ioc0: ChainBuffer sz=128 bytes, ChainDepth=1143
mptbase: ioc0: ChainBuffer sz=146304[23b80] bytes num_chain=1143
IOMMU:alloc coherent
IOMMU:succeeded 000000003bc00000(ffff88003bc00000) 192512 ffffffff
mptbase: ioc0: Total alloc @ ffff88003bc00000[00000000fffc0000], sz=192000[2ee00] bytes
mptbase: ioc0: ReplyBuffers @ ffff88003bc00000[00000000fffc0000]
mptbase: ioc0: RequestBuffers @ ffff88003bc02800[00000000fffc2800]
mptbase: ioc0: ChainBuffers @ ffff88003bc0b280(00000000fffcb280)
IOMMU:alloc coherent
IOMMU:succeeded 000000003d1f8000(ffff88003d1f8000) 20480 ffffffff
mptbase: ioc0: SenseBuffers @ ffff88003d1f8000[00000000fffb8000]
mptbase: ioc0: ReplyBuffers @ ffff88003bc00000[00000000fffc0000]
mptbase: ioc0: SendIocInit
mptbase: ioc0: facts.MsgVersion=105
mptbase: ioc0: Sending Port(0)Enable (req @ ffff88003da6da98)
mptbase: ioc0: Wait IOC_OPERATIONAL state (cnt=0)

The DMA memory is freed on removal:

mptbase: ioc0: free  @ ffff88003bc00000, sz=192000 bytes
IOMMU:free coherent
IOMMU:free coherent
mptsas 0000:06:00.0: PCI INT A disabled

However, I see the 'free' called  at mptbase.c:4596 uses the allocated size (192000) rather than the page padded size(192512) really allocated by the IOMMU engine, but I doubt it could be the cause for this problem.
...

So I'm starting to think the IOMMU driver messes up the mapping tables in some way on deallocation/reallocation operations.

I must admist I don't fully understand the impact of this change: http://kerneltrap.org/mailarchive/git-commits-head/2009/7/3/6134073

But seeing these lines 
-			dma_set_pte_readable(pte);
-			dma_set_pte_writable(pte);

I can't believe it's not related to this message:

DMAR:[DMA Read] Request device [06:00.0] fault addr fffc3000
DMAR:[fault reason 06] PTE Read access is not set

... this still needs some investigation.

Thank you

François Isabelle | Software Designer | Kontron Canada | T 450 437 5682 |F 450 437 8053 | E francois.isabelle@ca.kontron.com