Hi Lu Baolu,

I tried kernel 4.18.0-147.6.el8.x86_64+debug and used the following API sequence for mapping multiple hugepages:

get_user_pages_fast()
sg_alloc_table_from_pages()
// also tried sg_alloc_table() w/ sg_set_page() using 1GB size for each entry
dma_map_sg()

I'm able to DMA upto 1GB successfully and validate the data. Also, DMA above 1GB completes w/o any error, but data isn't correct starting immediately after 1GB i.e. second GB offset 0x40000000 starts showing data mismatches.

I've used get_user_pages_fast() in two ways to no avail:
1. populate page array w/ first page of 1GB hugepage and used sg_set_page() for
   setting 1GB size of the page entry. This debugging effort uses the fact
   that all pages following the first page of huge page start address are contiguous.
   Ideally dma_map_sg() should coalesce contiguous pages, and my intention was to collect
   more data from debugging.
2. populate page array w/ all pages from all hugepages

Thanks,
-am



On Sun, Dec 1, 2019 at 7:33 PM Anand Misra <am.online.edu@gmail.com> wrote:
[+iommu_list]

Application isn't aware of hugepage but a userspace (lower) level stack is aware of the type of memory being allocated on behalf of application, which in turn communicates w/ driver via ioctl. I'm trying to make it more agnostic by using dma_map_sg() when multiple GBs are required by application. Otherwise, I'm using dmap_map_page(). Admittedly, I'm learning these concepts/APIs for Linux along the way.

Thanks,
-am


On Sun, Dec 1, 2019 at 7:12 PM Lu Baolu <baolu.lu@linux.intel.com> wrote:
Hi,

On 12/2/19 11:00 AM, Anand Misra wrote:
> Thanks, Lu Baolu. This is the dev version we've in our company. I can
> try on a Ubuntu with a recent kernel version. Although, do you think I'm  > going in the right direction? Is it possible to have multiple hugepages
> mapped via iommu to get contiguous mapping for DMA?
>

You mentioned:

"
I'm trying to use iommu for multiple hugepages (mmap'ed by process and
pushed to driver via ioctl). The expectation is to have multiple
hugepages mapped via iommu with each huge page having an entry in iommu
(i.e. minimize table walk for DMA). Is this possible?
"

Currently huge page mapping is hidden in iommu driver according to the
iommu capability and size of map range. Why do you want to be aware of
it in driver or even application level?

Best regards,
baolu

> -am
>
>
> On Sun, Dec 1, 2019 at 6:24 PM Lu Baolu <baolu.lu@linux.intel.com
> <mailto:baolu.lu@linux.intel.com>> wrote:
>
>     Hi,
>
>     On 12/2/19 9:46 AM, Anand Misra wrote:
>      > Hello:
>      >
>      > I'm in process of adding iommu support in my driver for a PCIe
>     device.
>      > The device doesn't publish ACS/ATS via its config space. I've
>     following
>      > config:
>      >
>      > Linux cmdline: "intel-iommu=on iommu=pt
>      > vfio_iommu_type1.allow_unsafe_interrupts=1
>     pcie_acs_override=downstream"
>      > Centos kernel: 3.10.0-1062.1.2.el7.x86_64
>      >
>
>     Can you please use the latest kernel for test? 3.10 seems to be pretty
>     old and there are a lot of changes after it.
>
>     Best regards,
>     baolu
>