None of the virtual/physical/bus address matches the (base) BAR-0 register

None of the virtual/physical/bus address matches the (base) BAR-0 register

[Ajay Garg]

Hi All.

I have a SD/MMC reader over PCI, which displays the following (amongst
others) when we do "lspci -vv" :

#########################################################
Region 0: Memory at e2c20000 (32-bit, non-prefetchable) [size=512]
#########################################################

Above shows that e2c20000 is the physical (base-)address of BAR0.



Now, in the device driver, I do the following :

########################################################
.....
struct pci_dev *ptr;
void __iomem *bar0_ptr;
......

......
pci_request_region(ptr, 0, "ajay_sd_mmc_BAR0_region");
bar0_ptr = pci_iomap(ptr, 0, pci_resource_len(ptr, 0));

printk("Base virtual-address = [%p]\n", bar0_ptr);
printk("Base physical-address = [%p]\n", virt_to_phys(bar0_ptr));
printk("Base bus-address = [%p]\n", virt_to_bus(bar0_ptr));
....
########################################################


I have removed error-checking, but I confirm that pci_request_region()
and pci_iomap calls are successful.

Now, in the 3 printk's, none of the value is printed as e2c20000.
I was expecting that the 2nd result, of virt_to_phys() translation,
would be equal to the base-address of BAR0 register, as reported by
lspci.


What am I missing?
Will be grateful for pointers.


Thanks and Regards,
Ajay

Re: None of the virtual/physical/bus address matches the (base) BAR-0 register

[Keith Busch]

On Fri, Oct 01, 2021 at 08:21:06PM +0530, Ajay Garg wrote:
> Hi All.
> 
> I have a SD/MMC reader over PCI, which displays the following (amongst
> others) when we do "lspci -vv" :
> 
> #########################################################
> Region 0: Memory at e2c20000 (32-bit, non-prefetchable) [size=512]
> #########################################################
> 
> Above shows that e2c20000 is the physical (base-)address of BAR0.
> 
> Now, in the device driver, I do the following :
> 
> ########################################################
> .....
> struct pci_dev *ptr;
> void __iomem *bar0_ptr;
> ......
> 
> ......
> pci_request_region(ptr, 0, "ajay_sd_mmc_BAR0_region");
> bar0_ptr = pci_iomap(ptr, 0, pci_resource_len(ptr, 0));
> 
> printk("Base virtual-address = [%p]\n", bar0_ptr);
> printk("Base physical-address = [%p]\n", virt_to_phys(bar0_ptr));
> printk("Base bus-address = [%p]\n", virt_to_bus(bar0_ptr));
>
> I have removed error-checking, but I confirm that pci_request_region()
> and pci_iomap calls are successful.
> 
> Now, in the 3 printk's, none of the value is printed as e2c20000.
> I was expecting that the 2nd result, of virt_to_phys() translation,
> would be equal to the base-address of BAR0 register, as reported by
> lspci.
> 
> 
> What am I missing?
> Will be grateful for pointers.

The CPU address isn't always the same as the PCI address. For example,
some memory resources are added via pci_add_resource_offset(), so the
windows the host sees will be different than the ones the devices use.

Re: None of the virtual/physical/bus address matches the (base) BAR-0 register

[Ajay Garg]

Thanks Keith.

Let's take a x86 world as of now, and let's say the physical address
(returned by virt_to_phys()) is 0661a070.
The pci address (as stated) is e2c20000.


Since the BAR0-region is of size 256 bytes, so the system-agent (as
per x86-terminology) will monitor the highest 24 bits of
address-lines, to sense a MMIO read/write, and then forward the
transaction to the corresponding pci bridge/device.

So, in the present case, would

a)
The system-agent sense address-lines A31-A8 value as 0661a07? If yes,
is it the system-agent that does the translation from 0661a070 =>
e2c20000, before finally forwarding the transaction to pci
bridge/device?

b)
The system-agent sense address-lines A31-A8 value as e2c2000 (and
simply forwards the transaction to pci bridge/device)? If yes,
who/what does the translation from 0661a070 =? e2c20000?


Meanwhile, I am also trying to learn how to do kernel-development for
statically linked modules (like pci).
That would help in a much better understanding of the things-flow :P


Thanks for the help.


Thanks and Regards,
Ajay

On Fri, Oct 1, 2021 at 8:43 PM Keith Busch <kbusch@kernel.org> wrote:
>
> On Fri, Oct 01, 2021 at 08:21:06PM +0530, Ajay Garg wrote:
> > Hi All.
> >
> > I have a SD/MMC reader over PCI, which displays the following (amongst
> > others) when we do "lspci -vv" :
> >
> > #########################################################
> > Region 0: Memory at e2c20000 (32-bit, non-prefetchable) [size=512]
> > #########################################################
> >
> > Above shows that e2c20000 is the physical (base-)address of BAR0.
> >
> > Now, in the device driver, I do the following :
> >
> > ########################################################
> > .....
> > struct pci_dev *ptr;
> > void __iomem *bar0_ptr;
> > ......
> >
> > ......
> > pci_request_region(ptr, 0, "ajay_sd_mmc_BAR0_region");
> > bar0_ptr = pci_iomap(ptr, 0, pci_resource_len(ptr, 0));
> >
> > printk("Base virtual-address = [%p]\n", bar0_ptr);
> > printk("Base physical-address = [%p]\n", virt_to_phys(bar0_ptr));
> > printk("Base bus-address = [%p]\n", virt_to_bus(bar0_ptr));
> >
> > I have removed error-checking, but I confirm that pci_request_region()
> > and pci_iomap calls are successful.
> >
> > Now, in the 3 printk's, none of the value is printed as e2c20000.
> > I was expecting that the 2nd result, of virt_to_phys() translation,
> > would be equal to the base-address of BAR0 register, as reported by
> > lspci.
> >
> >
> > What am I missing?
> > Will be grateful for pointers.
>
> The CPU address isn't always the same as the PCI address. For example,
> some memory resources are added via pci_add_resource_offset(), so the
> windows the host sees will be different than the ones the devices use.

Re: None of the virtual/physical/bus address matches the (base) BAR-0 register

[Ajay Garg]

Forgot to mention one thing, the environment/machine has "intel_iommu=off".

On Sat, Oct 2, 2021 at 9:36 AM Ajay Garg <ajaygargnsit@gmail.com> wrote:
>
> Thanks Keith.
>
> Let's take a x86 world as of now, and let's say the physical address
> (returned by virt_to_phys()) is 0661a070.
> The pci address (as stated) is e2c20000.
>
>
> Since the BAR0-region is of size 256 bytes, so the system-agent (as
> per x86-terminology) will monitor the highest 24 bits of
> address-lines, to sense a MMIO read/write, and then forward the
> transaction to the corresponding pci bridge/device.
>
> So, in the present case, would
>
> a)
> The system-agent sense address-lines A31-A8 value as 0661a07? If yes,
> is it the system-agent that does the translation from 0661a070 =>
> e2c20000, before finally forwarding the transaction to pci
> bridge/device?
>
> b)
> The system-agent sense address-lines A31-A8 value as e2c2000 (and
> simply forwards the transaction to pci bridge/device)? If yes,
> who/what does the translation from 0661a070 =? e2c20000?
>
>
> Meanwhile, I am also trying to learn how to do kernel-development for
> statically linked modules (like pci).
> That would help in a much better understanding of the things-flow :P
>
>
> Thanks for the help.
>
>
> Thanks and Regards,
> Ajay
>

Re: None of the virtual/physical/bus address matches the (base) BAR-0 register

[Bjorn Helgaas]

On Sat, Oct 02, 2021 at 09:36:44AM +0530, Ajay Garg wrote:
> Thanks Keith.
> 
> Let's take a x86 world as of now, and let's say the physical address
> (returned by virt_to_phys()) is 0661a070.
> The pci address (as stated) is e2c20000.

Something's wrong here.  The low-order 12 bits of the CPU virtual, CPU
physical, and PCI bus address should be the same.  Might be the
hashing done by %p, see Documentation/core-api/printk-formats.rst.

> Since the BAR0-region is of size 256 bytes, so the system-agent (as
> per x86-terminology) will monitor the highest 24 bits of
> address-lines, to sense a MMIO read/write, and then forward the
> transaction to the corresponding pci bridge/device.
> 
> So, in the present case, would
> 
> a)
> The system-agent sense address-lines A31-A8 value as 0661a07? If yes,
> is it the system-agent that does the translation from 0661a070 =>
> e2c20000, before finally forwarding the transaction to pci
> bridge/device?
> 
> b)
> The system-agent sense address-lines A31-A8 value as e2c2000 (and
> simply forwards the transaction to pci bridge/device)? If yes,
> who/what does the translation from 0661a070 =? e2c20000?

> On Fri, Oct 1, 2021 at 8:43 PM Keith Busch <kbusch@kernel.org> wrote:
> > On Fri, Oct 01, 2021 at 08:21:06PM +0530, Ajay Garg wrote:
> > > Hi All.
> > >
> > > I have a SD/MMC reader over PCI, which displays the following (amongst
> > > others) when we do "lspci -vv" :
> > >
> > > #########################################################
> > > Region 0: Memory at e2c20000 (32-bit, non-prefetchable) [size=512]
> > > #########################################################
> > >
> > > Above shows that e2c20000 is the physical (base-)address of BAR0.

Yes.  "lspci -vv" shows the CPU physical address.  "lspci -bvv" shows
the bus addresses, i.e., the addresses you would see with a PCI bus
analyzer.  See Documentation/core-api/dma-api-howto.rst for more.

> > > Now, in the device driver, I do the following :
> > >
> > > ########################################################
> > > .....
> > > struct pci_dev *ptr;
> > > void __iomem *bar0_ptr;
> > > ......
> > >
> > > ......
> > > pci_request_region(ptr, 0, "ajay_sd_mmc_BAR0_region");
> > > bar0_ptr = pci_iomap(ptr, 0, pci_resource_len(ptr, 0));
> > >
> > > printk("Base virtual-address = [%p]\n", bar0_ptr);
> > > printk("Base physical-address = [%p]\n", virt_to_phys(bar0_ptr));
> > > printk("Base bus-address = [%p]\n", virt_to_bus(bar0_ptr));

  printk("Base physical-address = [%#lx]\n", ptr->resource[0].start);
  printk("Base virtual-address = [%px]\n", bar0_ptr);
  printk("Base bus-address = [%#lx]\n", pci_bus_address(ptr, 0));
  printk("BAR 0: %pR\n", &ptr->resource[0]);

> > > Now, in the 3 printk's, none of the value is printed as e2c20000.
> > > I was expecting that the 2nd result, of virt_to_phys() translation,
> > > would be equal to the base-address of BAR0 register, as reported by
> > > lspci.
> > >
> > >
> > > What am I missing?
> > > Will be grateful for pointers.
> >
> > The CPU address isn't always the same as the PCI address. For example,
> > some memory resources are added via pci_add_resource_offset(), so the
> > windows the host sees will be different than the ones the devices use.

Re: None of the virtual/physical/bus address matches the (base) BAR-0 register

[Ajay Garg]

Thanks Bjorn, that certainly helped.
Especially, it is nice to see that ptr->resource[0].start does give e2c20000 :)

Also, idiotic of me of making newbie kernel-format-specifier mistakes :-|

So, now the outputs corresponding to

    printk("Base virtual-address = [%lx]\n", bar0_ptr);
    printk("Base physical-address (form-1) = [%lx]\n", dev->resource[0].start);
    printk("Base bus-address = [%lx]\n", pci_bus_address(dev, 0));
    printk("BAR 0: %pR\n", &dev->resource[0]);


are :

    Base virtual-address = [ffffa0c6c02eb000]
    Base physical-address (form-1) = [e2c20000]
    Base bus-address = [e2c20000]
    BAR 0: [mem 0xe2c20000-0xe2c201ff]


All as expected.
Plus, all the lower 12 bits are now same everywhere (due to the 4 KB
page-size alignment in x86, right)?



My major missing understanding regarding this, is that we use the
iowrite*/ioread* functions, using bar0_ptr as the
base-(virtual-)address.
Thus, bar0_ptr *is* very well the kernel-virtual-address, which maps
to some physical-address (hopefully e2c20000), which directly triggers
the write/read with the pci-device.

Right now, the physical-address (form-1) we have printed, is via the
data-structure field.
However, looking from the virtual-address <=> physical-address
translation from the usual memory write/read datapath's perspective, I
am still not able to coalesce things.


In the same run as above, if I add the following statements :

    printk("Base physical-address (form-2) = [%lx]\n", virt_to_phys(bar0_ptr));
    printk("Base physical-address (form-3) = [%lx]\n",
virt_to_phys(*((uint32_t*)bar0_ptr)));
    printk("Base physical-address (form-4) = [%lx]\n",
virt_to_phys(*((uint64_t*)bar0_ptr)));

I get :

    Base physical-address (form-2) = [12e6002eb000]
    Base physical-address (form-3) = [721f4000001e]
    Base physical-address (form-4) = [721f4000001e]


Looking at the function-doc for virt_to_phys(), it states :

########################################################
* This function does not give bus mappings for DMA transfers. In
* almost all conceivable cases a device driver should not be using
* this function
########################################################



So, two queries :

1)
Does the above comment apply here too (in MMIO case)?

2)
If yes, then what is the datapath followed for our case (since
conventional virtual-address <=> physical-address translations via MMU
/ TLB / page-tables is out of the picture I guess)?


Thanks a ton already to everyone, in helping me clearing out my mind.


Thanks and Regards,
Ajay

Re: None of the virtual/physical/bus address matches the (base) BAR-0 register

[Bjorn Helgaas]

On Sat, Oct 02, 2021 at 10:42:16PM +0530, Ajay Garg wrote:
> Thanks Bjorn, that certainly helped.
> Especially, it is nice to see that ptr->resource[0].start does give
> e2c20000 :)
> 
> Also, idiotic of me of making newbie kernel-format-specifier
> mistakes :-|

If you're referring to %p being hashed, that's bitten me more than
once.  I understand the value of it, but it's annoying in the
day-to-day development process ;)

> So, now the outputs corresponding to
> 
>     printk("Base virtual-address = [%lx]\n", bar0_ptr);
>     printk("Base physical-address (form-1) = [%lx]\n", dev->resource[0].start);
>     printk("Base bus-address = [%lx]\n", pci_bus_address(dev, 0));
>     printk("BAR 0: %pR\n", &dev->resource[0]);
> 
> are :
> 
>     Base virtual-address = [ffffa0c6c02eb000]
>     Base physical-address (form-1) = [e2c20000]
>     Base bus-address = [e2c20000]
>     BAR 0: [mem 0xe2c20000-0xe2c201ff]
> 
> All as expected.
> Plus, all the lower 12 bits are now same everywhere (due to the 4 KB
> page-size alignment in x86, right)?

Yes.

> My major missing understanding regarding this, is that we use the
> iowrite*/ioread* functions, using bar0_ptr as the
> base-(virtual-)address.
> Thus, bar0_ptr *is* very well the kernel-virtual-address, which maps
> to some physical-address (hopefully e2c20000), which directly triggers
> the write/read with the pci-device.

Yes.

> Right now, the physical-address (form-1) we have printed, is via the
> data-structure field.
> However, looking from the virtual-address <=> physical-address
> translation from the usual memory write/read datapath's perspective, I
> am still not able to coalesce things.
> 
> 
> In the same run as above, if I add the following statements :
> 
>     printk("Base physical-address (form-2) = [%lx]\n", virt_to_phys(bar0_ptr));
>     printk("Base physical-address (form-3) = [%lx]\n",
> virt_to_phys(*((uint32_t*)bar0_ptr)));
>     printk("Base physical-address (form-4) = [%lx]\n",
> virt_to_phys(*((uint64_t*)bar0_ptr)));

I don't think these last two make any sense.  You're doing an MMIO
read from the address bar0_ptr, so this value (0x721f4000001e) came
from the PCI devices.  There's no reason to think it's a kernel
virtual address, so you shouldn't call virt_to_phys() on it.

> I get :
> 
>     Base physical-address (form-2) = [12e6002eb000]
>     Base physical-address (form-3) = [721f4000001e]
>     Base physical-address (form-4) = [721f4000001e]
>
> Looking at the function-doc for virt_to_phys(), it states :
> 
> ########################################################
> * This function does not give bus mappings for DMA transfers. In
> * almost all conceivable cases a device driver should not be using
> * this function
> ########################################################
> 
> 
> 
> So, two queries :
> 
> 1)
> Does the above comment apply here too (in MMIO case)?

Yes.  You should not need to use virt_to_phys() for PCI MMIO
addresses.

> 2)
> If yes, then what is the datapath followed for our case (since
> conventional virtual-address <=> physical-address translations via MMU
> / TLB / page-tables is out of the picture I guess)?

This path is IN the picture.  This is exactly the path used by drivers
doing MMIO accesses.

The CPU does a load from a virtual address (0xffffa0c6c02eb000).  The
MMU translates that virtual address to a physical address
(0xe2c20000).  The PCI host bridge decodes that address and generates
a PCIe Memory Read transaction to the bus address 0xe2c20000.

Your dmesg log should have lines similar to this:

  pci_bus 0000:00: root bus resource [mem 0x7f800000-0xefffffff window]
  pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]

that show the parts of the CPU physical address space that result in
MMIO to PCI devices.  0xe2c20000 should be inside one of those
windows.

Bjorn

Re: None of the virtual/physical/bus address matches the (base) BAR-0 register

[Ajay Garg]

>
> I don't think these last two make any sense.  You're doing an MMIO
> read from the address bar0_ptr, so this value (0x721f4000001e) came
> from the PCI devices.  There's no reason to think it's a kernel
> virtual address, so you shouldn't call virt_to_phys() on it.

Got it, thanks !

>
> Yes.  You should not need to use virt_to_phys() for PCI MMIO
> addresses.

Hmm, the thing that started it all :P


>
> This path is IN the picture.  This is exactly the path used by drivers
> doing MMIO accesses.
>
> The CPU does a load from a virtual address (0xffffa0c6c02eb000).  The
> MMU translates that virtual address to a physical address
> (0xe2c20000).

Hmm, so I guess the core takeaway is that the virtual-address <=>
physical-address mapping is indeed happening as envisioned. It's just
that there is no programmatic way to prove/display the mapped physical
address, in the case of PCI-MMIO transfers (as virt_to_phys() is not
usable in this case).

May be, someday it will :)

  The PCI host bridge decodes that address and generates
> a PCIe Memory Read transaction to the bus address 0xe2c20000.
>
> Your dmesg log should have lines similar to this:
>
>   pci_bus 0000:00: root bus resource [mem 0x7f800000-0xefffffff window]
>   pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
>
> that show the parts of the CPU physical address space that result in
> MMIO to PCI devices.  0xe2c20000 should be inside one of those
> windows.
>

It indeed is, in one of the pci-bridge range.


Thanks a ton Bjorn, for taking out the time to provide help, taking
the particular program-values as per my run.
I am grateful for your patience; nature bless you !


Thanks and Regards,
Ajay

Re: None of the virtual/physical/bus address matches the (base) BAR-0 register

[Bjorn Helgaas]

On Sun, Oct 03, 2021 at 07:35:41AM +0530, Ajay Garg wrote:
> > Yes.  You should not need to use virt_to_phys() for PCI MMIO
> > addresses.
> 
> Hmm, the thing that started it all :P
> 
> > This path is IN the picture.  This is exactly the path used by drivers
> > doing MMIO accesses.
> >
> > The CPU does a load from a virtual address (0xffffa0c6c02eb000).  The
> > MMU translates that virtual address to a physical address
> > (0xe2c20000).
> 
> Hmm, so I guess the core takeaway is that the virtual-address <=>
> physical-address mapping is indeed happening as envisioned. It's just
> that there is no programmatic way to prove/display the mapped physical
> address, in the case of PCI-MMIO transfers (as virt_to_phys() is not
> usable in this case).

  bar0_ptr = pci_iomap(ptr, 0, pci_resource_len(ptr, 0));

  printk("Base physical-address (form-1) = [%lx]\n", dev->resource[0].start);
  printk("Base physical-address (form-2) = [%lx]\n", virt_to_phys(bar0_ptr));

  Base physical-address (form-1) = [e2c20000]
  Base physical-address (form-2) = [12e6002eb000]

Despite the fact that you shouldn't *need* to use virt_to_phys()
because drivers use virtual addresses to do MMIO, it *does* seem like
form-1 and form-2 should be the same since form-2 is basically the
result of:

  virt_to_phys(ioremap(form-1))

I can't explain the difference.

Re: None of the virtual/physical/bus address matches the (base) BAR-0 register

[Ajay Garg]

Hi Bjorn.

I tried enabling and dumping the kernel-page-tables (via
/sys/kernel/debug/kernel_page_tables), but there we only have the
kernel-virtual-addresses :-\

So, I tried navigating the code (beginning from
arch/x86/mm/debug_pagetables.c), and stumbled upon
slow_virt_to_phys(), and hit the jackpot !
Even the code there looks as simple as the procedure - we fetch the
pte for the vpn, get the mapped pfn, and return the physical-address
after adding offset.

#####################################################################
So, now, the following :

    {
        phys_addr_t mmu_mapped_address;

        printk("original (kernel-virtual) address \t= [%lx]\n",
bar0_ptr);

        mmu_mapped_address = slow_virt_to_phys(bar0_ptr);
        printk("mapped address (via MMU) \t\t= [%lx]\n", mmu_mapped_address);
    }
#####################################################################



gives :

#####################################################################
original (kernel-virtual) address     = [ffffa11040217000]
mapped address (via MMU)         = [e2c20000]
#####################################################################



Thanks a ton Bjorn, this could not have been possible without you !


Thanks and Regards,
Ajay