pci segments/domains

pci segments/domains

[Grover, Andrew]

Well, ACPI calls them "segments" but a previous discussion (c.f. "RFC:
Changes for PCI" from a year ago) called them domains.

I don't care what they're called, but I wanted to bring them up and see what
everyone thought about how best to implement them, or at least if anyone had
an objection to adding a "segment" parameter to pci_scan_root.

I certainly don't have a machine that uses these but some people do, and it
sounds like it would be nice to handle them in an arch-neutral way.

Regards -- Andy

-----------------------------
Andrew Grover
Intel Labs / Mobile Architecture
andrew.grover@intel.com

Re: pci segments/domains

[David S. Miller]

   From: "Grover, Andrew" <andrew.grover@intel.com>
   Date: Mon, 13 May 2002 19:07:23 -0700

   I don't care what they're called, but I wanted to bring them up and see what
   everyone thought about how best to implement them, or at least if anyone had
   an objection to adding a "segment" parameter to pci_scan_root.
   
   I certainly don't have a machine that uses these but some people do, and it
   sounds like it would be nice to handle them in an arch-neutral way.

What exactly doesn't work right now?  I've been using multi-PCI-domain
systems for years under Linux :-)

Re: pci segments/domains

[Jeff Garzik]

Grover, Andrew wrote:

>Well, ACPI calls them "segments" but a previous discussion (c.f. "RFC:
>Changes for PCI" from a year ago) called them domains.
>
>I don't care what they're called, but I wanted to bring them up and see what
>everyone thought about how best to implement them, or at least if anyone had
>an objection to adding a "segment" parameter to pci_scan_root.
>
>I certainly don't have a machine that uses these but some people do, and it
>sounds like it would be nice to handle them in an arch-neutral way.
>

alpha and sparc64 at least already do them.

I wouldn't mind making the PCI domain support a bit more explicit, 
though.  I think it's fair to be able to obtain a pointer to "struct 
pci_domain", which would most likely be defined in asm/pci.h for each arch.

    Jeff

Re: pci segments/domains

[Pavel Machek]

Hi!

> We are going to end up soon with generic devices in 2.5.x
> and we might as well end up with a "struct bus" too, which

We actually had "struct bus" in the past, but we figured it is better to
let devices have subdevices.
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: pci segments/domains

[David S. Miller]

   From: Jeff Garzik <jgarzik@mandrakesoft.com>
   Date: Thu, 16 May 2002 15:33:34 -0400
   
   alpha and sparc64 at least already do them.
   
   I wouldn't mind making the PCI domain support a bit more explicit, 
   though.  I think it's fair to be able to obtain a pointer to "struct 
   pci_domain", which would most likely be defined in asm/pci.h for each arch.
   
We are going to end up soon with generic devices in 2.5.x
and we might as well end up with a "struct bus" too, which
can provide the domain abstraction in a bus-type independant
way.

Re: pci segments/domains

[David S. Miller]

   From: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
   Date: Fri, 17 May 2002 14:47:55 +0400

   I can think of the only case where domain info might be interesting - if
   some device wants to know whether it can talk to another device directly.
   We have pci_controller_num(pdev) for this.

That is an insufficient way to determine this.  Maybe I can make two
PCI devices talk to each other transparently between two domains
using an IOMMU?

Re: pci segments/domains

[Ivan Kokshaysky]

On Thu, May 16, 2002 at 03:33:34PM -0400, Jeff Garzik wrote:
> I wouldn't mind making the PCI domain support a bit more explicit, 
> though.  I think it's fair to be able to obtain a pointer to "struct 
> pci_domain", which would most likely be defined in asm/pci.h for each arch.

We already have it - void *sysdata. Host-to-PCI (domain) controllers might
be totally different even inside any given architecture, so trying to
make this more generic would be pointless - you will end up with a pointer
to arch/device specific data anyway.
I can think of the only case where domain info might be interesting - if
some device wants to know whether it can talk to another device directly.
We have pci_controller_num(pdev) for this.

Ivan.

Re: pci segments/domains

[David S. Miller]

   From: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
   Date: Fri, 17 May 2002 15:11:54 +0400

   Hmm, not sure. I think it's impossible at least on alpha.
   In either case I don't see how "struct pci_domain" etc.
   could help in this case. Probably you'll need some mapping
   function with pdev1, pdev2 as arguments...
   
I'm not saying pci_domain structure will help, I am only
saying that domain number is not the way to figure out
the ability to do DMA between two devices :-)

I know for a fact that you can do PCI-to-PCI DMA between
two PCI domains on Alpha.

   

Re: pci segments/domains

[Ivan Kokshaysky]

On Fri, May 17, 2002 at 03:40:48AM -0700, David S. Miller wrote:
> That is an insufficient way to determine this.  Maybe I can make two
> PCI devices talk to each other transparently between two domains
> using an IOMMU?

Hmm, not sure. I think it's impossible at least on alpha.
In either case I don't see how "struct pci_domain" etc.
could help in this case. Probably you'll need some mapping
function with pdev1, pdev2 as arguments...

Ivan.

Re: pci segments/domains

[Ivan Kokshaysky]

On Fri, May 17, 2002 at 04:04:21AM -0700, David S. Miller wrote:
> I'm not saying pci_domain structure will help, I am only
> saying that domain number is not the way to figure out
> the ability to do DMA between two devices :-)

Agreed. What about
dma_addr_t pci_to_pci_map_single(struct pci_dev *master,
				 struct pci_dev *target,
				 dma_addr_t tgt_addr, size_t size, int dir)

Could be implemented without much pain if there is enough interest. :-)

> I know for a fact that you can do PCI-to-PCI DMA between
> two PCI domains on Alpha.

Ah, indeed. I should check the docs before posting... :-\

Ivan.

Re: pci segments/domains

[David S. Miller]

   From: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
   Date: Fri, 17 May 2002 15:39:03 +0400
   
   What about
   dma_addr_t pci_to_pci_map_single(struct pci_dev *master,
   				 struct pci_dev *target,
   				 dma_addr_t tgt_addr, size_t size, int dir)
   
   Could be implemented without much pain if there is enough interest. :-)
   
tgt_addr does not make any sense, we are trying to DMA to a device
resource, so pass a "struct resource *" and "unsigned long res_offset"
instead of tgt_addr.

So, as a (real life) example, suppose you wanted to portably point
your BTTV card at the framebuffer of a video card, you'd do
something like:

	res = fb_pdev->resource[1];
	res_off = OFFSET_INTO_FRAMEBUFFER;
	bttv_dma_addr = pci_to_pci_map_single(struct pci_dev *bttv_pdev,
					      struct pci_dev *fb_pdev,
					      res, res_off, size,
					      PCI_DMA_FROMDEVICE);

Note the direction is in terms of the master.  The DMA is coming
"from" the master in this "BTTV capture to framebuffer" case.

Re: pci segments/domains

[David S. Miller]

   From: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
   Date: Fri, 17 May 2002 16:26:21 +0400
   
   BTW, I don't see anymore why pci domain info should be exposed to
   users at all. :-)

It allows the X server do what it wants to sort-of sanely.

It wants to be able to assign resources to video devices (even
implicit resources like VGA) and to do that properly it has to know
which PCI domain which devices are on and also the extent of the I/O
and MEM space on that domain.

Re: pci segments/domains

[Ivan Kokshaysky]

On Fri, May 17, 2002 at 04:41:57AM -0700, David S. Miller wrote:
> So, as a (real life) example, suppose you wanted to portably point
> your BTTV card at the framebuffer of a video card, you'd do
> something like:
> 
> 	res = fb_pdev->resource[1];
> 	res_off = OFFSET_INTO_FRAMEBUFFER;
> 	bttv_dma_addr = pci_to_pci_map_single(struct pci_dev *bttv_pdev,
> 					      struct pci_dev *fb_pdev,
> 					      res, res_off, size,
> 					      PCI_DMA_FROMDEVICE);

Exactly. And this function will look like this

	if (domain(bttv_pdev) == domain(fb_pdev))
		return arch_res_to_bus(res->start) + res_off;
	else if (!interdomain_dma_supported)
		return 0;
	else
		return some_iommu_magic(...);

BTW, I don't see anymore why pci domain info should be exposed to
users at all. :-)

Ivan.

Re: pci segments/domains

[David S. Miller]

   From: Jeff Garzik <jgarzik@mandrakesoft.com>
   Date: Fri, 17 May 2002 10:24:39 -0400

   I know -- that's what I mean by being more explicit.  sysdata would 
   become a pointer to struct pci_domain.
   
No thanks, I want to say what the layout is for
this object.  What pci_domain will end up doing is
making for one more dereference to "arch private"
state and that stinks for performance :-)

Franks a lot,
David S. Miller
davem@redhat.com
   
   
   

Re: pci segments/domains

[Jeff Garzik]

Ivan Kokshaysky wrote:

>On Thu, May 16, 2002 at 03:33:34PM -0400, Jeff Garzik wrote:
>
>>I wouldn't mind making the PCI domain support a bit more explicit, 
>>though.  I think it's fair to be able to obtain a pointer to "struct 
>>pci_domain", which would most likely be defined in asm/pci.h for each arch.
>>
>
>We already have it - void *sysdata. Host-to-PCI (domain) controllers might
>be totally different even inside any given architecture, so trying to
>make this more generic would be pointless - you will end up with a pointer
>to arch/device specific data anyway.
>
I know -- that's what I mean by being more explicit.  sysdata would 
become a pointer to struct pci_domain.

>
>I can think of the only case where domain info might be interesting - if
>some device wants to know whether it can talk to another device directly.
>We have pci_controller_num(pdev) for this.
>

Like gets mentioned later in the thread, you don't want to start 
addressing based on number...

    Jeff

Re: pci segments/domains

[David S. Miller]

   From: Jeff Garzik <jgarzik@mandrakesoft.com>
   Date: Fri, 17 May 2002 10:33:26 -0400

   My main want is cosmetic -- call a spade a spade, so to speak. 
    s/sysdata/pci_domain/  But doing so opens the door to increased 
   flexibility.  Later steps can add common members needed by pci-to-pci 
   IOMMU tricks which are common to most platforms.

Since the name really doesn't matter let's call it struct pci_controller
since that is what Alpha and Sparc use already :-)

Re: pci segments/domains

[Jeff Garzik]

David S. Miller wrote:

>   From: Jeff Garzik <jgarzik@mandrakesoft.com>
>   Date: Fri, 17 May 2002 10:24:39 -0400
>
>   I know -- that's what I mean by being more explicit.  sysdata would 
>   become a pointer to struct pci_domain.
>   
>No thanks, I want to say what the layout is for
>this object.  What pci_domain will end up doing is
>making for one more dereference to "arch private"
>state and that stinks for performance :-)
>


See my previous message from the other day... this would be defined in 
each arch's asm/pci.h, which eliminates this problem you describe.  Each 
arch maintainer would indeed decide how to define it, though over time 
I'm sure it would grow commonly-named struct members.

My main want is cosmetic -- call a spade a spade, so to speak. 
 s/sysdata/pci_domain/  But doing so opens the door to increased 
flexibility.  Later steps can add common members needed by pci-to-pci 
IOMMU tricks which are common to most platforms.

    Jeff

Re: pci segments/domains

[Jeff Garzik]

David S. Miller wrote:

>   From: Jeff Garzik <jgarzik@mandrakesoft.com>
>   Date: Fri, 17 May 2002 10:33:26 -0400
>
>   My main want is cosmetic -- call a spade a spade, so to speak. 
>    s/sysdata/pci_domain/  But doing so opens the door to increased 
>   flexibility.  Later steps can add common members needed by pci-to-pci 
>   IOMMU tricks which are common to most platforms.
>
>Since the name really doesn't matter let's call it struct pci_controller
>since that is what Alpha and Sparc use already :-)
>


Makes sense, sure :)  I just want to get rid of the untyped sysdata in 
favor of a struct with a defined type (arch-defined... but named and 
defined nonetheless).

    Jeff

Re: pci segments/domains

[Ivan Kokshaysky]

On Fri, May 17, 2002 at 10:42:44AM -0400, Jeff Garzik wrote:
> Makes sense, sure :)  I just want to get rid of the untyped sysdata in 
> favor of a struct with a defined type (arch-defined... but named and 
> defined nonetheless).

Just to be sure, do you mean both struct pci_bus and struct pci_dev? :-)
On alpha both sysdata's are pointers to the same thing, but not on sparc
as far as I can see.

Ivan.

Re: pci segments/domains

[David S. Miller]

   From: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
   Date: Fri, 17 May 2002 20:24:03 +0400

   On Fri, May 17, 2002 at 10:42:44AM -0400, Jeff Garzik wrote:
   > Makes sense, sure :)  I just want to get rid of the untyped sysdata in 
   > favor of a struct with a defined type (arch-defined... but named and 
   > defined nonetheless).
   
   Just to be sure, do you mean both struct pci_bus and struct pci_dev? :-)
   On alpha both sysdata's are pointers to the same thing, but not on sparc
   as far as I can see.
   
Just do the struct pci_bus one for now.