Re: [PATCH 1/2] [RFC] ata: ahci: Respect bus DMA constraints

[Marek Vasut <marek.vasut@gmail.com>]

On 3/16/19 10:25 PM, Marek Vasut wrote:
> On 3/13/19 7:30 PM, Christoph Hellwig wrote:
>> On Sat, Mar 09, 2019 at 12:23:15AM +0100, Marek Vasut wrote:
>>> On 3/8/19 8:18 AM, Christoph Hellwig wrote:
>>>> On Thu, Mar 07, 2019 at 12:14:06PM +0100, Marek Vasut wrote:
>>>>>> Right, but whoever *interprets* the device masks after the driver has
>>>>>> overridden them should be taking the (smaller) bus mask into account as
>>>>>> well, so the question is where is *that* not being done correctly?
>>>>>
>>>>> Do you have a hint where I should look for that ?
>>>>
>>>> If this a 32-bit ARM platform it might the complete lack of support
>>>> for bus_dma_mask in arch/arm/mm/dma-mapping.c..
>>>
>>> It's an ARM 64bit platform, just the PCIe controller is limited to 32bit
>>> address range, so the devices on the PCIe bus cannot read the host's
>>> DRAM above the 32bit limit.
>>
>> arm64 should take the mask into account both for the swiotlb and
>> iommu case.  What are the exact symptoms you see?
> 
> With the nvme, the device is recognized, but cannot be used.
> It boils down to PCI BAR access being possible, since that's all below
> the 32bit boundary, but when the device tries to do any sort of DMA,
> that transfer returns nonsense data.
> 
> But when I call dma_set_mask_and_coherent(dev->dev, DMA_BIT_MASK(32) in
> the affected driver (thus far I tried this nvme, xhci-pci and ahci-pci
> drivers), it all starts to work fine.
> 
> Could it be that the driver overwrites the (coherent_)dma_mask and
> that's why the swiotlb/iommu code cannot take this into account ?
> 
>> Does it involve
>> swiotlb not kicking in, or iommu issues?
> 
> How can I check ? I added printks into arch/arm64/mm/dma-mapping.c and
> drivers/iommu/dma-iommu.c , but I suspect I need to look elsewhere.

Digging further ...

drivers/nvme/host/pci.c nvme_map_data() calls dma_map_sg_attrs() and the
resulting sglist contains entry with >32bit PA. This is because
dma_map_sg_attrs() calls dma_direct_map_sg(), which in turn calls
dma_direct_map_sg(), then dma_direct_map_page() and that's where it goes
weird.

dma_direct_map_page() does a dma_direct_possible() check before
triggering swiotlb_map(). The check succeeds, so the later isn't executed.

dma_direct_possible() calls dma_capable() with dev->dma_mask =
DMA_BIT_MASK(64) and dev->dma_bus_mask = 0, so
min_not_zero(*dev->dma_mask, dev->bus_dma_mask) returns DMA_BIT_MASK(64).

Surely enough, if I hack dma_direct_possible() to return 0,
swiotlb_map() kicks in and the nvme driver starts working fine.

I presume the question here is, why is dev->bus_dma_mask = 0 ?

-- 
Best regards,
Marek Vasut