[Nouveau] Understanding BAR1 Offset to imem/VRAM PA Mapping

[Nouveau] Understanding BAR1 Offset to imem/VRAM PA Mapping

[Joshua Bakita]

Hello,

I'm trying to understand how VRAM PAs are mapped to BAR1 offsets on
Fermi+, but I'm having difficulty digging through the abstractions in
nouveau. I spent the better part of yesterday digging through the
nv50_instobj_*() functions, but I lost track of which page tables are
being modified and where they're coming from somewhere around level 7
of indirection/aliasing from the nvkm_kmap() call (aka
nv50_instobj_acquire()) to the actual nvkm_vmm_iter() logic which I
think does the mapping.

If page tables are used to map BAR1 offsets to VRAM PAs on Fermi+, I'd
like to understand their relation to the normal GPU VA to PA page
tables, and how we tell the hardware which page tables to use for the
BAR1 mappings.

Best regards,

Joshua Bakita
PhD Student
UNC Chapel Hill | Real-Time Systems Group

(Apologies if anyone already received this email, I tried sending it
earlier and I think it got stuck in moderation.)

Re: [Nouveau] Understanding BAR1 Offset to imem/VRAM PA Mapping

[Ilia Mirkin]

Hi Joshua,

It looks like you got most of the way there. The BARs (BAR1 and BAR3)
are initialized by the code in nvkm/subdev/bar/gf100.c. As you can
see, this sets up a vmm per BAR, whose (physical address >> 12) is
written to 0x1704 / 0x1714. A vmm is basically a list of PDE's (and
the PTE lists those PDE's point to). The address, I believe, is the
base of the PDE's (or maybe they start at 0x200 from the base
address). I believe that all addresses relative to the appropriate BAR
are then translated through this vmm and hit the appropriate physical
VRAM addresses. You can see how this is all set up in
nvkm/subdev/mmu/vmmgf100.c. The code is somewhat fancy to generically
deal with different page sizes/etc, but you can mostly ignore that.
Different generations allow different things, which is what you'll
find in the subsequent vmm* files. Pascal gains 48-bit virtual
addresses (up from 40), which I think changes some details around as
well.

Ben - please correct me if I'm wrong. I suspect I missed some details
relative to *actually* making this work, but this should be the gist
of it.

Hope this helps,

  -ilia

On Wed, Sep 29, 2021 at 4:17 PM Joshua Bakita <jbakita@cs.unc.edu> wrote:
>
> Hello,
>
> I'm trying to understand how VRAM PAs are mapped to BAR1 offsets on
> Fermi+, but I'm having difficulty digging through the abstractions in
> nouveau. I spent the better part of yesterday digging through the
> nv50_instobj_*() functions, but I lost track of which page tables are
> being modified and where they're coming from somewhere around level 7
> of indirection/aliasing from the nvkm_kmap() call (aka
> nv50_instobj_acquire()) to the actual nvkm_vmm_iter() logic which I
> think does the mapping.
>
> If page tables are used to map BAR1 offsets to VRAM PAs on Fermi+, I'd
> like to understand their relation to the normal GPU VA to PA page
> tables, and how we tell the hardware which page tables to use for the
> BAR1 mappings.
>
> Best regards,
>
> Joshua Bakita
> PhD Student
> UNC Chapel Hill | Real-Time Systems Group
>
> (Apologies if anyone already received this email, I tried sending it
> earlier and I think it got stuck in moderation.)

[Nouveau] Understanding BAR1 Offset to imem/VRAM PA Mapping

[Joshua Bakita]

Hello,

I'm trying to understand how VRAM PAs are mapped to BAR1 offsets on
Fermi+, but I'm having difficulty digging through the abstractions in
nouveau. I spent the better part of yesterday digging through the
nv50_instobj_*() functions, but I lost track of which page tables are
being modified and where they're coming from somewhere around level 7
of indirection/aliasing from the nvkm_kmap() call (aka
nv50_instobj_acquire()) to the actual nvkm_vmm_iter() logic which I
think does the mapping.

If page tables are used to map BAR1 offsets to VRAM PAs on Fermi+, I'd
like to understand their relation to the normal GPU VA to PA page
tables, and how we tell the hardware which page tables to use for the
BAR1 mappings.

Best regards,

Joshua Bakita
PhD Student
UNC Chapel Hill | Real-Time Systems Group