Question about Power8/9, PHB3/4 and setting of DMA mask

Question about Power8/9, PHB3/4 and setting of DMA mask

[Oded Gabbay]

Hello,
I'm working in a startup called HabanaLabs, and we have an ASIC
accelerator for AI called Goya. It is assembled on a PCIe Gen4 card.
Driver is going to be in kernel 5.1

We are trying to plug the card into a Power8 machine and load the
driver, and we get a failure during the loading of the driver in
regard to the driver trying to set the DMA mask.

Due to some limitation in Goya, the driver first need to allocate a
2MB chunk in a DMA-able address under 39 bits and then we would like
to move to using up to 48 bits. Therefore, the driver first tries to
set the DMA mask to 39 bits, allocate the 2MB area and later on,
change the DMA mask to 48 bits. On x86 this works fine.

However, as I said, on Power8 we got a failure when trying to set to
39 bits. After tracking the code, I reached to this function:
pnv_pci_ioda_dma_set_mask()

In that function, there is a check (composed of 4 conditions) about
the requested dma mask, which appears that we fail and I suspect this
is due to the memory_hotplug_max() returning more then 39 bits.

My questions are:
1. Is this logic applies to Power9 as well ?
2. Why this condition is mandatory ? Is there some kind of workaround
available ?

Thanks in advance,
Oded

Re: Question about Power8/9, PHB3/4 and setting of DMA mask

[Christoph Hellwig]

On Sun, Mar 31, 2019 at 12:50:21PM +0300, Oded Gabbay wrote:
> Due to some limitation in Goya, the driver first need to allocate a
> 2MB chunk in a DMA-able address under 39 bits and then we would like
> to move to using up to 48 bits. Therefore, the driver first tries to
> set the DMA mask to 39 bits, allocate the 2MB area and later on,
> change the DMA mask to 48 bits. On x86 this works fine.

You can't just change the DMA mask while you have active allocations,
this will fail for many implementations.