Re: [PATCH v4 10/13] lib: test_hmm add module param for zone device type

From: Felix Kuehling <felix.kuehling@amd.com>
To: "Sierra Guiza, Alejandro (Alex)" <alex.sierra@amd.com>,
	Jason Gunthorpe <jgg@nvidia.com>
Cc: akpm@linux-foundation.org, linux-mm@kvack.org,
	rcampbell@nvidia.com, linux-ext4@vger.kernel.org,
	linux-xfs@vger.kernel.org, amd-gfx@lists.freedesktop.org,
	dri-devel@lists.freedesktop.org, hch@lst.de, jglisse@redhat.com
Subject: Re: [PATCH v4 10/13] lib: test_hmm add module param for zone device type
Date: Fri, 30 Jul 2021 15:11:41 -0400	[thread overview]
Message-ID: <4c931ef6-1a52-2cc8-47ff-6077377517af@amd.com> (raw)
In-Reply-To: <596f4387-a896-acdf-acfa-7ddba947b58f@amd.com>

Am 2021-07-28 um 7:45 p.m. schrieb Sierra Guiza, Alejandro (Alex):
>
> On 7/22/2021 12:26 PM, Jason Gunthorpe wrote:
>> On Thu, Jul 22, 2021 at 11:59:17AM -0500, Sierra Guiza, Alejandro
>> (Alex) wrote:
>>> On 7/22/2021 7:23 AM, Jason Gunthorpe wrote:
>>>> On Sat, Jul 17, 2021 at 02:21:32PM -0500, Alex Sierra wrote:
>>>>> In order to configure device generic in test_hmm, two
>>>>> module parameters should be passed, which correspon to the
>>>>> SP start address of each device (2) spm_addr_dev0 &
>>>>> spm_addr_dev1. If no parameters are passed, private device
>>>>> type is configured.
>>>> I don't think tests should need configuration like this, is it really
>>>> necessary? How can people with normal HW run this test?
>>> Hi Jason,
>>> The idea was to add an easy way to validate the codepaths touched by
>>> this
>>> patch series, which make modifications to the migration helpers for
>>> device
>>> generic type pages. We're using CONFIG_EFI_FAKE_MEMMAP to create
>>> fake SPM
>>> devices inside system memory. No special HW needed. And passing the
>>> kernel
>>> parameter efi_fake_mem. Ex. efi_fake_mem=1G@0x100000000:0x40000. I
>>> should
>>> probably need to include a small example of how to set this in the
>>> test_hmm.sh
>>> usage().
>> I don't think anything about hmm is sensitive to how the pages are
>> acquired - you can't create device generic pages without relying on
>> FAKE_MEMMAP?
> The reason we used fake SPM approach was to have a "special memory"
> not managed by Linux (NOT registered as normal system memory). But
> also accessible by the CPU.
>
> For device_generic we cannot allocate new physical addresses.
> We need the physical address to match the actual system memory
> physical address, so that CPU mappings work as expected.
>
> Would you recommend to use a different approach?

Hi Jason,

Sorry it took us so long to respond to your comment. I was on vacation
for a week. Then I tried to brain-storm some ways with Alex to simplify
hmm_test for device_generic memory, but couldn't really come up with
anything simpler.

The problem as I see it is, that DEVICE_GENERIC pages for hmm_test
should be pages in physical system memory, but they should not be
managed by the Linux page allocator. Unlike DEVICE_PRIVATE, we cannot
allocate arbitrary physical addresses for these pages
(request_free_mem_region). Otherwise we'd break all the assumptions that
make those pages directly accessible in user mode virtual address spaces.

We could maybe allocate contiguous memory from the page allocator and
then register those as device generic pages. But that means, you'd now
have two struct pages for the same physical page. I didn't think that
would be a good idea.

IMHO, reserving some memory for this test with efi_fake_mem is the best
way to keep things sane. It does not require any special hardware or
firmware.

Regards,
  Felix

>
> Regards,
> Alex Sierra
>
>>
>> Jason