Hi Lu Baolu,

I tried kernel 4.18.0-147.6.el8.x86_64+debug and used the following API
sequence for mapping multiple hugepages:

get_user_pages_fast()
sg_alloc_table_from_pages()
// also tried sg_alloc_table() w/ sg_set_page() using 1GB size for each
entry
dma_map_sg()

I'm able to DMA upto 1GB successfully and validate the data. Also, DMA
above 1GB completes w/o any error, but data isn't correct starting
immediately after 1GB i.e. second GB offset 0x40000000 starts showing data
mismatches.

I've used get_user_pages_fast() in two ways to no avail:
1. populate page array w/ first page of 1GB hugepage and used sg_set_page()
for
   setting 1GB size of the page entry. This debugging effort uses the fact
   that all pages following the first page of huge page start address are
contiguous.
   Ideally dma_map_sg() should coalesce contiguous pages, and my intention
was to collect
   more data from debugging.
2. populate page array w/ all pages from all hugepages

Thanks,
-am



On Sun, Dec 1, 2019 at 7:33 PM Anand Misra <am.online.edu@gmail.com> wrote:

> [+iommu_list]
>
> Application isn't aware of hugepage but a userspace (lower) level stack is
> aware of the type of memory being allocated on behalf of application, which
> in turn communicates w/ driver via ioctl. I'm trying to make it more
> agnostic by using dma_map_sg() when multiple GBs are required by
> application. Otherwise, I'm using dmap_map_page(). Admittedly, I'm learning
> these concepts/APIs for Linux along the way.
>
> Thanks,
> -am
>
>
> On Sun, Dec 1, 2019 at 7:12 PM Lu Baolu <baolu.lu@linux.intel.com> wrote:
>
>> Hi,
>>
>> On 12/2/19 11:00 AM, Anand Misra wrote:
>> > Thanks, Lu Baolu. This is the dev version we've in our company. I can
>> > try on a Ubuntu with a recent kernel version. Although, do you think
>> I'm  > going in the right direction? Is it possible to have multiple
>> hugepages
>> > mapped via iommu to get contiguous mapping for DMA?
>> >
>>
>> You mentioned:
>>
>> "
>> I'm trying to use iommu for multiple hugepages (mmap'ed by process and
>> pushed to driver via ioctl). The expectation is to have multiple
>> hugepages mapped via iommu with each huge page having an entry in iommu
>> (i.e. minimize table walk for DMA). Is this possible?
>> "
>>
>> Currently huge page mapping is hidden in iommu driver according to the
>> iommu capability and size of map range. Why do you want to be aware of
>> it in driver or even application level?
>>
>> Best regards,
>> baolu
>>
>> > -am
>> >
>> >
>> > On Sun, Dec 1, 2019 at 6:24 PM Lu Baolu <baolu.lu@linux.intel.com
>> > <mailto:baolu.lu@linux.intel.com>> wrote:
>> >
>> >     Hi,
>> >
>> >     On 12/2/19 9:46 AM, Anand Misra wrote:
>> >      > Hello:
>> >      >
>> >      > I'm in process of adding iommu support in my driver for a PCIe
>> >     device.
>> >      > The device doesn't publish ACS/ATS via its config space. I've
>> >     following
>> >      > config:
>> >      >
>> >      > Linux cmdline: "intel-iommu=on iommu=pt
>> >      > vfio_iommu_type1.allow_unsafe_interrupts=1
>> >     pcie_acs_override=downstream"
>> >      > Centos kernel: 3.10.0-1062.1.2.el7.x86_64
>> >      >
>> >
>> >     Can you please use the latest kernel for test? 3.10 seems to be
>> pretty
>> >     old and there are a lot of changes after it.
>> >
>> >     Best regards,
>> >     baolu
>> >
>>
>