Re: With Daniel Phillips Patch

Re: With Daniel Phillips Patch

[Van Maren, Kevin]

> Can you enumerate the devices that actually issue a DAC when loaded with
> 64bit address with 0's in the most significant 32bits?

There had better not be any.  It is a violation of the PCI specification
to generate a DAC if the address fits in 32 bits.

DAC is a LOT faster and more efficient than a copy (except perhaps for the
very smallest of transfers, which are already very inefficient).

The problem is that (for most hardware) the 64-bit descriptors take up more
room (and hence more PCI cycles to transfer) than the 32-bit descriptors,
especially with a 32-bit bus.  [Apparently not the case for the 39-bit
AIC7xxx driver, but it is the case for the 64-bit Adaptec.]  So unless
there is the possibility of using 64-bit DMA, you want to use the smaller
descriptors.  So on systems with <= 32bits of memory/dma_addr_t, the driver
should be able to "know" that it should use the smaller descriptors for
efficiency.

I also believe that a dma_addr_t should be determined by the system, not the
driver: the driver should indicate constraints and the OS should ensure that
the dma_addr_t it provides meets the constraints.  Separate 32-bit and
64-bit
DMA routines adds unnecessary complication.  As far as I can tell, the only
reason to have separate APIs is so that 32bit machines with 64 bit DMA
addresses (PAE on ia32) can avoid copying around an "extra" 32bits of
address for the drivers that don't support 64-bit DMA.  I think it makes
more sense to just make the dma_addr_t 64 bits on ia32 if using PAE and
deal with the insignificant "waste" -- you have > 4GB RAM :-)

Kevin Van Maren

Re: With Daniel Phillips Patch

[David S. Miller]

   From: "Van Maren, Kevin" <kevin.vanmaren@unisys.com>
   Date: Wed, 22 Aug 2001 20:06:21 -0500

   There had better not be any.  It is a violation of the PCI specification
   to generate a DAC if the address fits in 32 bits.
   
Then sym53c8xx with Gerard's current scripts code is in violation of
the PCI specification when the chip is told to use DAC :-)

   DAC is a LOT faster and more efficient than a copy (except perhaps for the
   very smallest of transfers, which are already very inefficient).

SAC with IOMMU is faster on some platforms.
   
   Separate 32-bit and 64-bit DMA routines adds unnecessary complication.
   As far as I can tell, the only reason to have separate APIs is so
   that 32bit machines with 64 bit DMA  addresses (PAE on ia32) can
   avoid copying around an "extra" 32bits of address for the drivers
   that don't support 64-bit DMA.

Welcome to the complicated real world.

There are several other reasons.  (Man, people check the archives, I
feel like I've typed this in like 5 times in linux-kernel postings
already)  Let me list one of them, suppose you have a device for which
some transfers can happily use DAC addresses, but some others strictly
need to work with SAC addresses.

pci64_*() would mean "DAC address would be OK".

   I think it makes more sense to just make the dma_addr_t 64 bits on
   ia32 if using PAE and deal with the insignificant "waste" -- you
   have > 4GB RAM :-)

I think for SAC-only devices, it is just dumb wasted space in the
driver image.

I do not want to even go into the abuse I took in my email when I
added the original APIs because people had to keep track of the
damn mappings at all!  These people would strangle me if they learnt
that in HIGHMEM kernels twice as much space was needed to do this
DMA address tracking.

I at least comfort myself that those who maintain drivers on several
platforms, and have an open mind, such as Gerard, for the most part
support the API I have designed.

Later,
David S. Miller
davem@redhat.com

Re: With Daniel Phillips Patch

[Justin T. Gibbs]

>I do not want to even go into the abuse I took in my email when I
>added the original APIs because people had to keep track of the
>damn mappings at all!  These people would strangle me if they learnt
>that in HIGHMEM kernels twice as much space was needed to do this
>DMA address tracking.

You have to keep track of the significant bits in the dma_addr_t
regardless of its size, so you put it into your TX descriptor's (or
what have you) native format that doesn't waste any space.  You don't
need to keep the full dma_addr_t around.  Perhaps this is just sloppy
programming?

>I at least comfort myself that those who maintain drivers on several
>platforms, and have an open mind, such as Gerard, for the most part
>support the API I have designed.

If you don't want to take part in technical discussions, you should
work in closed source. 8-)

--
Justin

Re: With Daniel Phillips Patch

[David S. Miller]

   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
   Date: Wed, 22 Aug 2001 19:40:55 -0600

   You have to keep track of the significant bits in the dma_addr_t
   regardless of its size, so you put it into your TX descriptor's (or
   what have you) native format that doesn't waste any space.  You don't
   need to keep the full dma_addr_t around.  Perhaps this is just sloppy
   programming?
   
Some devices keep these in registers and advance them as the
dma progresses.  The only reliable way is by keeping track
of it in software.

   If you don't want to take part in technical discussions, you should
   work in closed source. 8-)

It's not open source, it's "dumb source" I have problems with.
A lot of discussions here end up being of that variety.

Later,
David S. Miller
davem@redhat.com
   

Re: With Daniel Phillips Patch

[David S. Miller]

   From: "Van Maren, Kevin" <kevin.vanmaren@unisys.com>
   Date: Wed, 22 Aug 2001 21:22:19 -0500

   If the HW generates DAC for addresses < 4GB whenever enabling
   support for 64-bit addresses, then that is very broken.

That is what happens.
   
   Please don't complain that I didn't spend hours searching
   through the archives looking for a message from months? years? ago
   that I didn't know existed.

Weeks, if not days.
   
   > I think for SAC-only devices, it is just dumb wasted space in the
   > driver image.
   
   Perhaps.  But the question is whether it is simpler/better to have
   HIGHMEM x86 kernels (which by definition have memory to spare) waste
   a few bytes to provide "sane" interfaces across all platforms.  And
   whether the kernel bloat for all the additional functions compensates
   for it ;-)

The plain fact is that %95 of PCI devices do not support DAC
addressing.

Later,
David S. Miller
davem@redhat.com


   

RE: With Daniel Phillips Patch

[Van Maren, Kevin]

>    There had better not be any.  It is a violation of the PCI 
> specification
>    to generate a DAC if the address fits in 32 bits.
>    
> Then sym53c8xx with Gerard's current scripts code is in violation of
> the PCI specification when the chip is told to use DAC :-)

If you say so.  You make it sound like the driver can
compensate for the broken hardware by explicitly telling
it to use SAC for a transfer after checking if the
dma_addr < 4GB.  The HW is not in "violation" if it doesn't
generate a DAC < 4GB, and if it takes a driver check to
ensure that, it is the driver's problem: it is trivial to
check the high 32-bits of a 64-bit address for 0.

If the HW generates DAC for addresses < 4GB whenever enabling
support for 64-bit addresses, then that is very broken.

>    DAC is a LOT faster and more efficient than a copy (except 
> perhaps for the
>    very smallest of transfers, which are already very inefficient).
> 
> SAC with IOMMU is faster on some platforms.

Okay, so when the driver asks for the physical address, the arch-
specific code maps it with the iommu and returns a 32-bit address.
In that case, the dma_addr_t is 32 bits (unless it can return
64-bit addresses as well).

> There are several other reasons.  (Man, people check the archives, I
> feel like I've typed this in like 5 times in linux-kernel postings
> already)

If you send pointer to your previous message I will read it.  I am
interested in this subject (and have experience), so I threw in my
2 cents.  Please don't complain that I didn't spend hours searching
through the archives looking for a message from months? years? ago
that I didn't know existed.

> Let me list one of them, suppose you have a device for which
> some transfers can happily use DAC addresses, but some others strictly
> need to work with SAC addresses.

What does that have to do with anything?  That just means that the
DMA constraints have to be specified on a per-mapping/per-allocation
basis, not a per-device basis.  That doesn't mean you need separate
routines for 64-bit PCI addresses and 32-bit PCI addresses.  It just
means you need sane DMA constraints handling.

> I think for SAC-only devices, it is just dumb wasted space in the
> driver image.

Perhaps.  But the question is whether it is simpler/better to have
HIGHMEM x86 kernels (which by definition have memory to spare) waste
a few bytes to provide "sane" interfaces across all platforms.  And
whether the kernel bloat for all the additional functions compensates
for it ;-)

Reasonable people can have different opinions.

Kevin Van Maren

Re: With Daniel Phillips Patch

[Justin T. Gibbs]

>   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
>   Date: Wed, 22 Aug 2001 19:32:46 -0600
>
>   Perhaps its different for SBUS, but its not different for ISA
>   or EISA.
>
>Right, you pass in a NULL pci_dev pointer.  What is the
>problem with that?

I don't have the same lattitude to express dma characteristics of
broken, non-PCI devices.  For instance, I can't set the "dma mask"
for a VLB card (say some early BusLogic 445) that had some DMA bugs.
I have to treat it like an ISA card even if it may have problems
with DMAs below the typical ISA dma limit.

>   Do you believe that it is architecturally correct to have a single
>   api or multiple apis?
>
>I think just plain different entry points are the way to do things,
>because function pointers and/or extra conditional execution rots when
>it's really not needed.

That needent be the case.  If I can use a single API to define the
DMA characteristics of my device, and the system knows where it
is in the bus hierarchy (and all the warts of the bridges along
the way, etc.), the magic to do the mapping can be hidden from me
and I don't need to have multiple APIs or code paths.  I just pass
a "dma descriptor" that has the necessary info for that type of
dma operation on that platform, and the system does the rest.  This
even allows a device to allocate multiple descriptors to handle its
different operations (bulk data is 64bit capable, transaction descriptors
need to be handled with 24bit addresses, etc.).

>   The "pci" api already allows you to express this.
>
>There will be a "struct device" in 2.5.x and lots of unification.

That's good to know.

--
Justin

Re: With Daniel Phillips Patch

[David S. Miller]

   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
   Date: Wed, 22 Aug 2001 19:32:46 -0600

   Perhaps its different for SBUS, but its not different for ISA
   or EISA.

Right, you pass in a NULL pci_dev pointer.  What is the
problem with that?

   Do you believe that it is architecturally correct to have a single
   api or multiple apis?

I think just plain different entry points are the way to do things,
because function pointers and/or extra conditional execution rots when
it's really not needed.

   The "pci" api already allows you to express this.

There will be a "struct device" in 2.5.x and lots of unification.

Frankly, I'd rather not touch the SBUS drivers though.
All the devices are cast in stone, I'm the only person
who maintains or even works on any of the drivers, and
the less I have to change at this point the better.

Later,
David S. Miller
davem@redhat.com

Re: With Daniel Phillips Patch

[Justin T. Gibbs]

>   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
>   Date: Wed, 22 Aug 2001 18:55:21 -0600
>   
>   Then it is poorly named.  How about "pci_dma32_t".  Or better yet,
>   uint32_t.  How do the guys writing SBUS drivers like the fact that
>   all of this mapping stuff is so PCI centric?
>   
>Please actually take a look at a few SBUS drivers before
>you open your big mouth.  SBUS drivers use a totally different
>API.

Perhaps its different for SBUS, but its not different for ISA
or EISA.  The main point here is that if a single driver has
multiple bus attachements they either "luck out" and can use a
"pci api" to talk to their non-pci devices (the aic7xxx driver talks
EISA/VL/PCI) or they have to have different mapping paths (SBUS/PCI driver).
Do you believe that it is architecturally correct to have a single
api or multiple apis?  From your "big mouth" comment above, I assume
the later.  From the driver's standpoint, the task is pretty much the
same, with perhaps different contraints on the types of address that
can be supported by the device.  The "pci" api already allows you
to express this.

--
Justin

Re: With Daniel Phillips Patch

[David S. Miller]

   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
   Date: Wed, 22 Aug 2001 18:55:21 -0600
   
   Then it is poorly named.  How about "pci_dma32_t".  Or better yet,
   uint32_t.  How do the guys writing SBUS drivers like the fact that
   all of this mapping stuff is so PCI centric?
   
Please actually take a look at a few SBUS drivers before
you open your big mouth.  SBUS drivers use a totally different
API.

Later,
David S. Miller
davem@redhat.com

Re: With Daniel Phillips Patch

[Matthew Jacob]

What guys writing SBus drivers? I mean, other than the NetBSD folks?


On Wed, 22 Aug 2001, Justin T. Gibbs wrote:

> >   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
> >   Date: Wed, 22 Aug 2001 18:01:40 -0600
> >
> >   It is opaque and should be able to represent all dma (or I would prefer
> >   bus) addresses in the system.  The examples I've seen where people
> >   assume it to be 32bits in size are, well, broken.
> >
> >It is the type to be used for 32-bit SAC based DMA.
> >DMA-mapping.txt is pretty clear about this.
>
> Then it is poorly named.  How about "pci_dma32_t".  Or better yet,
> uint32_t.  How do the guys writing SBUS drivers like the fact that
> all of this mapping stuff is so PCI centric?
>
> --
> Justin
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: With Daniel Phillips Patch

[Justin T. Gibbs]

>   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
>   Date: Wed, 22 Aug 2001 18:01:40 -0600
>
>   It is opaque and should be able to represent all dma (or I would prefer
>   bus) addresses in the system.  The examples I've seen where people
>   assume it to be 32bits in size are, well, broken.
>
>It is the type to be used for 32-bit SAC based DMA.
>DMA-mapping.txt is pretty clear about this.

Then it is poorly named.  How about "pci_dma32_t".  Or better yet,
uint32_t.  How do the guys writing SBUS drivers like the fact that
all of this mapping stuff is so PCI centric?

--
Justin

Re: With Daniel Phillips Patch

[David S. Miller]

   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
   Date: Wed, 22 Aug 2001 18:01:40 -0600

   It is opaque and should be able to represent all dma (or I would prefer
   bus) addresses in the system.  The examples I've seen where people
   assume it to be 32bits in size are, well, broken.

It is the type to be used for 32-bit SAC based DMA.
DMA-mapping.txt is pretty clear about this.

In fact, because it is well documented, ia64 is in direct violation of
the API.  I've been mentioning things like this since the beginning.

Later,
David S. Miller
davem@redhat.com

Re: With Daniel Phillips Patch

[Justin T. Gibbs]

>Consider network drivers (most PCI ones) that keep track of:
>
>	struct sk_buff *skb;
>	dma_addr_t mapping;
>
>pairs for each transmit packet.  With your suggested change,
>their structures will increase 32-bits in size for each entry
>when CONFIG_HIGHMEM on x86 or on a 64-bit platform.

They already increase by 32bits on IA64.  A driver should use a
fixed sized type for a fixed sized address that corresponds to its
capabilities.  There is no guarantee of the size of dma_addr_t.
It is opaque and should be able to represent all dma (or I would prefer
bus) addresses in the system.  The examples I've seen where people
assume it to be 32bits in size are, well, broken.

--
Justin

Re: With Daniel Phillips Patch

[David S. Miller]

   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
   Date: Wed, 22 Aug 2001 15:40:30 -0600
   
   I've started looking through the network devices for bloat caused
   by the change in size of this type and I haven't found it anywhere.

Consider network drivers (most PCI ones) that keep track of:

	struct sk_buff *skb;
	dma_addr_t mapping;

pairs for each transmit packet.  With your suggested change,
their structures will increase 32-bits in size for each entry
when CONFIG_HIGHMEM on x86 or on a 64-bit platform.

I mean, just grep for dma_addr_t in structures of these networking
drivers to see where the wasted space would be.

Later,
David S. Miller
davem@redhat.com

Re: With Daniel Phillips Patch

[Justin T. Gibbs]

>I seem to understand that Justin's is referring to the DMA related API of
>BSD O/Ses that, I believe, originates from NetBSD.

Not really.  I just don't think that having both a 32bit and a 64bit
type for dma_addr_t makes sense.  I'm not advocating all devices perform
DAC.

I've started looking through the network devices for bloat caused
by the change in size of this type and I haven't found it anywhere.

--
Justin

Re: With Daniel Phillips Patch

[David S. Miller]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: Text/Plain; charset=big5, Size: 705 bytes --]

   From: Gérard Roudier <groudier@free.fr>
   Date: Wed, 22 Aug 2001 23:07:47 +0200 (CEST)
   
   And I seem to understand that David's preferred SUN hardware only
   allows streaming when using SAC with IOMMU.:-)
   
It is true.

   And using DAC is 1 PCI cycle lost per transaction and if we are
   picky on performances ...

True.  This is why I find it mysterious when I run across a device
which does not issue SAC for addresses with only 32-bits of
significance.

Happily, most do behave this way.

Later,
David S. Miller
davem@redhat.com
ý:.žË›±Êâmçë¢kaŠÉb²ßìzwm…ébïîžË›±Êâmébžìÿ‘êçz_âžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣømšSåy«\x1e­æ¶\x17…\x01\x06­†ÛiÿÿðÃ\x0fí»\x1fè®\x0få’i\x7f

Re: With Daniel Phillips Patch

[David S. Miller]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: Text/Plain; charset=big5, Size: 968 bytes --]

   From: Gérard Roudier <groudier@free.fr>
   Date: Wed, 22 Aug 2001 23:07:47 +0200 (CEST)

   Note that the manual says that the device will not use DAC if higher 32
   bits are zero. Nor I remember of any errata about the device behaving
   this way. But, as sym53c8xx device actually trying to do 64 bit PCI
   addressing should have been pretty rare for now, not all errata on this
   point should have been discovered (just trying to guess ...).

If I do not set DDAC in sym53c8xx it issues DAC cycles for everything,
even addresses with no bits set in upper 32-bits of address.

You will remember, we had this issue long ago and had to add #define
for it (which dies in my pci64 changes becuase this portability
issue no longer exists with proper API present).

Later,
David S. Miller
davem@redhat.com
ý:.žË›±Êâmçë¢kaŠÉb²ßìzwm…ébïîžË›±Êâmébžìÿ‘êçz_âžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣømšSåy«\x1e­æ¶\x17…\x01\x06­†ÛiÿÿðÃ\x0fí»\x1fè®\x0få’i\x7f

Re: With Daniel Phillips Patch

[Gérard Roudier]

On Wed, 22 Aug 2001, David S. Miller wrote:

>    From: "Justin T. Gibbs" <gibbs@scsiguy.com>
>    Date: Wed, 22 Aug 2001 12:32:17 -0600
>
>    I would like the change much better if the size of dma_addr_t
>    simply changed to be 64bits wide if high mem support is enabled
>    in your kernel config.
>
> Drivers for SAC only PCI devices shall not be bloated by 64-bit type,
> not in any case whatsoever.
>
>    Sure, you need the other API changes to more finely set dma characteristics,
>    but having two APIs just complicates life for the device driver.
>
> Either your device is 64-bit capable or not, what is so complicated?
> Each driver I converted was like 15 minutes or work, at best!
>
>    From the device driver's point of view, this wasn't the case.
>    The driver asks to have the data mapped into an address that
>    its dma engine can understand and the system is supposed to do that
>    mapping.
>
> What is the virtual address of physical address 0x100000000
> on a 32-bit cpu system if the page is not currently kmap()'d?
>
> Answer: it doesn't exist.

I seem to understand that Justin's is referring to the DMA related API of
BSD O/Ses that, I believe, originates from NetBSD. Just, the Linux
approach seems to make the hypothesis that 32 bits addressing will still
have a long life. Note that my guess is also that most low end servers and
personnal computers may well still use less than 4 GB and so 32 bit
addressable for a long time. This let me prefer the Linux differentiation,
at the moment, given that O/S vendors provides binaries and donnot
encourage users to recompile the kernel and modules. We probably donnot
want more than 99% of real machines to waste uselessly with 64 bit
quantities just for some source program aesthetic considerations.

And I seem to understand that David's preferred SUN hardware only allows
streaming when using SAC with IOMMU.:-)

And using DAC is 1 PCI cycle lost per transaction and if we are
picky on performances ...

> The only portable way is to use pages.  That is what Jens's and
> my work aims to do.  The ia64 API is nonportable and works only
> on 64-bit systems.
>
>    >It also assumed that using SAC or DAC addressing was simply a matter of
>    >"does the device support it", and the world is far from being that simple :-)
>
>    Can you enumerate the devices that actually issue a DAC when loaded with
>    a 64bit address with 0's in the most significant 32bits?
>
> Sym53c8xx does this.  You have to configure it to do SAC or DAC
> for data, descriptors use SAC always.

Note that the manual says that the device will not use DAC if higher 32
bits are zero. Nor I remember of any errata about the device behaving
this way. But, as sym53c8xx device actually trying to do 64 bit PCI
addressing should have been pretty rare for now, not all errata on this
point should have been discovered (just trying to guess ...).

[...]

Later,
  Gérard.

Re: With Daniel Phillips Patch

[David S. Miller]

   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
   Date: Wed, 22 Aug 2001 13:41:16 -0600

   >
   >Drivers for SAC only PCI devices shall not be bloated by 64-bit type,
   >not in any case whatsoever.
   
   Where's the bloat?  The driver's S/G list is in the driver's native format.

All the world is not a scsi driver.  Consider networking
and other types of drivers that need not make use of
scatter lists.

   >Either your device is 64-bit capable or not, what is so complicated?
   
   The complication arrises when there is a performance impact associated
   with 64bit support.  You may want to completely compile out 64bit support.
   If I can use the exact same API if the user decides to configure my
   device to not enable large address support, then my complaint is only
   that it seems supurfluous to have two different APIs and types.

You can indeed use the same API.  There are two situations.

1) The performance impact is "platform specific", so let the
   platform decide for you:

	Use pci64_foo() and set the DMA mask to what you can support.
	Ie. a normal DAC supporting driver.

2) The performance impact is "device specific", so only use
   the 32-bit API.   

   >What is the virtual address of physical address 0x100000000
   >on a 32-bit cpu system if the page is not currently kmap()'d?
   
   Why does the driver care?  The driver asked to have some virtual address
   mapped into a bus address.  Are you saying the system can't understand
   figure out what physical page this is and from that the necessary
   IOMMU magic to make it visible to the device?
   
It is the object that the block and networking systems work with
that is the issue.

Do you know how physical memory is mapped under Linux?  Everything
non-HIGHMEM is directly mapped.  Everything else must be temporarily
"kmap()'d" so that the kernel and perform loads and stores to that
page.

The only object representation that works for all kinds of pages,
HIGHMEM or not, is the "struct page *page; unsigned long offset;"
tuple.

If this wasn't a problem, Jens's would not be doing any of the
work he is doing right now :-)

   It seems to me that you are complaining that the "backend" implementation
   for IA64 sucked.  Okay.  Fine.  But the drivers were never exposed to
   that suckage.

No, I have in fact no problem with IA64's backend, I don't care how
any platform implements anything.  It is the front end that sucked
balls, and this part I care about because it is the APIs drivers have
to deal with.  Specifically, my gripes are:

1) It took virtual addresses.  Result: does not work on 32-bit
   platforms.

2) It did not take into consideration at all the issues surrounding
   DAC usage on some platforms, such as:

	a) transfers using DAC cycles might run slower than
	   those using SAC cycles
	b) DAC cycles may be preferred even in the presence of
	   slower transfers because the device is "DMA mapping
	   hungry" ala. compute cluster cards.

Let me ask you again: Have you tried to write a driver to the new
APIs at all?  I have for 6 totally different devices, on drivers
written by totally different people (including those I wrote myself)
and they all worked out beautifully.

Later,
David S. Miller
davem@redhat.com

Re: With Daniel Phillips Patch

[Justin T. Gibbs]

>   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
>   Date: Wed, 22 Aug 2001 12:32:17 -0600
>
>   I would like the change much better if the size of dma_addr_t
>   simply changed to be 64bits wide if high mem support is enabled
>   in your kernel config.
>
>Drivers for SAC only PCI devices shall not be bloated by 64-bit type,
>not in any case whatsoever.

Where's the bloat?  The driver's S/G list is in the driver's native format.
Are saying that there will be two different mapping implementation back
ends with the 32bit one possibly saving space by using 32bit only address
in whatever storage is required to record the mapping?  Since you have to
use an API call to fetch these values on a per-element basis, the 32bit back
end need only promote its 32bit values to 64bit values upon return.  On most
architectures I've seen, a load of a 32bit value from a 64bit value is still
just a 32bit move, so the fact that the returned value is a 64bit quantity
makes no difference code wise inside the driver.

>   Sure, you need the other API changes to more finely set dma characteristics
>,
>   but having two APIs just complicates life for the device driver.
>
>Either your device is 64-bit capable or not, what is so complicated?

The complication arrises when there is a performance impact associated
with 64bit support.  You may want to completely compile out 64bit support.
If I can use the exact same API if the user decides to configure my
device to not enable large address support, then my complaint is only
that it seems supurfluous to have two different APIs and types.

>   From the device driver's point of view, this wasn't the case.
>   The driver asks to have the data mapped into an address that
>   its dma engine can understand and the system is supposed to do that
>   mapping.
>
>What is the virtual address of physical address 0x100000000
>on a 32-bit cpu system if the page is not currently kmap()'d?

Why does the driver care?  The driver asked to have some virtual address
mapped into a bus address.  Are you saying the system can't understand
figure out what physical page this is and from that the necessary
IOMMU magic to make it visible to the device?

>The only portable way is to use pages.  That is what Jens's and
>my work aims to do.  The ia64 API is nonportable and works only
>on 64-bit systems.

This doesn't follow from your explanation.  Sure, you need to use
pages, but the basic information provided to the mapping calls allows
you to figure out the pages and that basic information is the same in
the new API.

It seems to me that you are complaining that the "backend" implementation
for IA64 sucked.  Okay.  Fine.  But the drivers were never exposed to
that suckage.

--
Justin

Re: With Daniel Phillips Patch

[David S. Miller]

   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
   Date: Wed, 22 Aug 2001 12:32:17 -0600

   I would like the change much better if the size of dma_addr_t
   simply changed to be 64bits wide if high mem support is enabled
   in your kernel config.

Drivers for SAC only PCI devices shall not be bloated by 64-bit type,
not in any case whatsoever.

   Sure, you need the other API changes to more finely set dma characteristics,
   but having two APIs just complicates life for the device driver.

Either your device is 64-bit capable or not, what is so complicated?
Each driver I converted was like 15 minutes or work, at best!
   
   From the device driver's point of view, this wasn't the case.
   The driver asks to have the data mapped into an address that
   its dma engine can understand and the system is supposed to do that
   mapping.

What is the virtual address of physical address 0x100000000
on a 32-bit cpu system if the page is not currently kmap()'d?

Answer: it doesn't exist.

The only portable way is to use pages.  That is what Jens's and
my work aims to do.  The ia64 API is nonportable and works only
on 64-bit systems.

   >It also assumed that using SAC or DAC addressing was simply a matter of
   >"does the device support it", and the world is far from being that simple :-)
   
   Can you enumerate the devices that actually issue a DAC when loaded with
   a 64bit address with 0's in the most significant 32bits?

Sym53c8xx does this.  You have to configure it to do SAC or DAC
for data, descriptors use SAC always.
   
There will certainly be devices in the future which will only
support DAC cycles.

   Now that I'm supposed to use two differnt apis depending on what
   capabilities I enable in my driver, 

I think you are far overcomplicating things.  I mean, look at how
simple the conversion of some of the networking drivers was.  The
sym53c8xx driver conversion was very simple too, even with it's odd
current behavior due to addressing limitations.

It was nothing more than:

1) Adding probe time code to configure DMA attributes correctly.
   Failing the probe is no suitable mode could be determined.

   You know, it allows you to do something like this:

	pci_set_dma_mask(pdev, 0x1fffffffff);
	if (pci_dac_cycles_ok(pdev)) {
		dac_addressing_method = 1;
		goto dma_configured;
	}
	pci_set_dma_mask(pdev, 0x7ffffffffff);
	if (pci_dac_cycles_ok(pdev)) {
		dac_addressing_method = 2;
		goto dma_configured;
	}
	pci_set_dma_mask(pdev, 0xffffffffffffffff);
	if (pci_dac_cycles_ok(pdev)) {
		dac_addressing_method = 3;
		goto dma_configured;
	}
	if (!pci_dma_supported(pdev, 0xffffffff)) {
		probe_fail_msg();
		return -ENODEV;
	}
	dac_addressing_method = 0; /* Use SAC */

2) sed 's/dma_addr_t/dma64_addr_t/'
   sed 's/pci_{map,unmap,dma_sync}*/pci64_{map,unmap,dma_sync}*/'

And doing a cursory glance over the DMA address references
to make sure they weren't being put into u32's or something
similar.

Justin, have you even _TRIED_ to use the new API?

I did, on like 6 drivers, and it works just fine.

Later,
David S. Miller
davem@redhat.com

Re: With Daniel Phillips Patch

[David S. Miller]

   From: G廨ard Roudier <groudier@free.fr>
   Date: Wed, 22 Aug 2001 20:21:50 +0200 (CEST)
   
   First, let me thank you a LLLOOOOOOTTTTT, David, for you PCI 64 bit
   addressing DMA-mapping. I didn't have looked yet into your patch and
   documentation update, but I will do so ASAP.
   
You're very welcome. :-)

   OTOH, it is a great pleasure for me to hear that you didn't forget what I
   told you about the current limitation of the SYM53C8XX driver. For now,
   the driver would be only able to use 40 bit addresses with all upper bits
   set to zero. This doesn't fit PCI 64 bit implementation of the Alpha
   Monster window for example, and probably doesn't fit most other non-Intel
   PCI 64 bit implementations.
   
It is fully known, in fact, I converted the sym53c8xx.c driver as
one of the examples in the patch.

Alpha can use it, in cases where memory in the system is less than
the addressing limitation of device.

Such logic would reside for Alpha port in pci_dac_cycles_ok()
definition.  IA64 and x86 could act similarly.  This was in fact
how I intended ports to implement pci_dac_cycles_ok().

   But there is an alternate solution for the SYM53C8XX driver by using up to
   16 x 32 bit segment registers. This would (will) allow to address for DMA
   16 x 4GB segments with all upper bits being settable for each 4GB segment.

Note, it relies on no 4GB crossing every occuring.  Jens and I have
decided that we will make this guarentee for devices always.  I know
of 2 devices already which have problems with this (Qlogic,FC and some
buggy variants of Tigon3 chips).

   I have this in my todo-list since months, but haven't had strong reasons
   for implementing it. The strongest reasons would be that I had access to
   64 bit machines with 64 bit PCI, but this isn't possible.

Look for someone to borrow a sparc64 system from.  Or, alternatively
send the patch to me for testing.

On sparc64, you will always be "testing all the bits" since each
DAC address to physical memory has:

	0xfffc000000000000

on the top bits.

Later,
David S. Miller
davem@redhat.com

Re: With Daniel Phillips Patch

[Justin T. Gibbs]

>   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
>   Date: Wed, 22 Aug 2001 07:24:29 -0600
>   
>   Is this somehow different than how large DMA is done on the ia64
>   port?  All I do is look at the size of dma_addr_t to decide whether
>   to enable high address support in my driver.  If dma_addr_t's size
>   changes, then 64bit addressing will work the same as on every other
>   Linux port.
>
>It is totally different.

In looking at the documentation, it really doesn't seem much different
at all.  All you've done is force there to be two types and two APIs,
instead of one of each, for accessing dma addresses.  I would like the
change much better if the size of dma_addr_t simply changed to be
64bits wide if high mem support is enabled in your kernel config.
That the high 32bits may be empty or not even looked at by some
device (as you describe in DMA-mapping.txt) isn't much of a concern.
Sure, you need the other API changes to more finely set dma characteristics,
but having two APIs just complicates life for the device driver.  You'll
see why I say this below.

>The ia64 method, while it worked for ia64, could not work properly on
>just about any other platform.  For example, it assumed that any
>physical address could be represented by a kernel virtual address.

>From the device driver's point of view, this wasn't the case.
The driver asks to have the data mapped into an address that
its dma engine can understand and the system is supposed to do that
mapping.  Whether an IOMMU or some other piece of hardware was involved
didn't matter to the driver.  Well, it might matter because, at least
in the ia64 case, resource shortages result in a panic instead of an
error code being returned that you could do something reasonable with.
Now I just need to stick some "64"'s into my API calls to get the same
effect I currently have on IA64.  The fact that the back end that supports
the mapping changed shouldn't effect the driver.

>It also assumed that using SAC or DAC addressing was simply a matter of
>"does the device support it", and the world is far from being that simple :-)

Can you enumerate the devices that actually issue a DAC when loaded with
a 64bit address with 0's in the most significant 32bits?

>I note that the aic7xxx won't be usable for DAC cycles on many
>platforms since not all 64-bits are significant :-(

This isn't true.  The hardware supports all 64bits, but I've only
implemented two of the three expected S/G formats:

1) 4byte address/3byte count/7bits pad/1bit end of list
2) 4byte address/3byte count/7bits extended address/1bit end of list
and NYI
3) 8byte address/3byte count/7bits pad/1bit end of list

The first is the most efficient as the firmware doesn't have to bother
(or have the code) to load the high address bits.  The second works for
many platforms but doesn't take any additional space up for S/G lists.
The last can be implemented and enabled on any platforms that really need it.

With formats 1 and 2, the choice of what to use can easily be done at
driver initalization time.  If you determine that high mappings will
never be needed, why do the extra work?  Now that I'm supposed to use
two differnt apis depending on what capabilities I enable in my driver,
I'll have to add more bloat to my mapping routine (two func calls that do
almost exact the same thing, gated by a test - or use an indirect function
call).  Perhaps I'll just wrap everything into macros and make it a compile
time option.

--
Justin

Re: With Daniel Phillips Patch

[Gérard Roudier]

On Wed, 22 Aug 2001, David S. Miller wrote:

>    From: "Justin T. Gibbs" <gibbs@scsiguy.com>
>    Date: Wed, 22 Aug 2001 07:24:29 -0600
>
>    Is this somehow different than how large DMA is done on the ia64
>    port?  All I do is look at the size of dma_addr_t to decide whether
>    to enable high address support in my driver.  If dma_addr_t's size
>    changes, then 64bit addressing will work the same as on every other
>    Linux port.
>
> It is totally different.
>
> The ia64 method, while it worked for ia64, could not work properly on
> just about any other platform.  For example, it assumed that any
> physical address could be represented by a kernel virtual address.
> This is not true on 32-bit HIGHMEM systems.  It also assumed that
> using SAC or DAC addressing was simply a matter of "does the device
> support it", and the world is far from being that simple :-)
>
> Please see the pci64 patches for details:
>
> ftp://ftp.kernel.org/pub/linux/kernel/people/davem/PCI64/*.gz
>
> There are Documentation/DMA-mapping.txt updates, where you can read
> how to use the interfaces properly.  A handful of net and scsi drivers
> were updated to use the new API, you have examples to work with as
> well.
>
> I note that the aic7xxx won't be usable for DAC cycles on many
> platforms since not all 64-bits are significant :-(  SYM53C8XX
> has a similar limitation.  Surprisingly, the network PCI cards
> have been the absolute best about this, supporting the full 64-bits
> of DAC address in all card instances I delved into.

First, let me thank you a LLLOOOOOOTTTTT, David, for you PCI 64 bit
addressing DMA-mapping. I didn't have looked yet into your patch and
documentation update, but I will do so ASAP.

OTOH, it is a great pleasure for me to hear that you didn't forget what I
told you about the current limitation of the SYM53C8XX driver. For now,
the driver would be only able to use 40 bit addresses with all upper bits
set to zero. This doesn't fit PCI 64 bit implementation of the Alpha
Monster window for example, and probably doesn't fit most other non-Intel
PCI 64 bit implementations.

But there is an alternate solution for the SYM53C8XX driver by using up to
16 x 32 bit segment registers. This would (will) allow to address for DMA
16 x 4GB segments with all upper bits being settable for each 4GB segment.
I have this in my todo-list since months, but haven't had strong reasons
for implementing it. The strongest reasons would be that I had access to
64 bit machines with 64 bit PCI, but this isn't possible. My machine uses
a Supermicro 370 DLE Mobo that offers PCI 64 bit path, but I only have 256
MB of memory. :-(, and anyway, the thing looks like a 15 years old 32 bit
Intel-arch :-(, that doesn't support 64 bit PCI addressing. :-(

  Gérard.

Re: With Daniel Phillips Patch

[David S. Miller]

   From: "Justin T. Gibbs" <gibbs@scsiguy.com>
   Date: Wed, 22 Aug 2001 07:24:29 -0600
   
   Is this somehow different than how large DMA is done on the ia64
   port?  All I do is look at the size of dma_addr_t to decide whether
   to enable high address support in my driver.  If dma_addr_t's size
   changes, then 64bit addressing will work the same as on every other
   Linux port.

It is totally different.

The ia64 method, while it worked for ia64, could not work properly on
just about any other platform.  For example, it assumed that any
physical address could be represented by a kernel virtual address.
This is not true on 32-bit HIGHMEM systems.  It also assumed that
using SAC or DAC addressing was simply a matter of "does the device
support it", and the world is far from being that simple :-)

Please see the pci64 patches for details:

ftp://ftp.kernel.org/pub/linux/kernel/people/davem/PCI64/*.gz

There are Documentation/DMA-mapping.txt updates, where you can read
how to use the interfaces properly.  A handful of net and scsi drivers
were updated to use the new API, you have examples to work with as
well.

I note that the aic7xxx won't be usable for DAC cycles on many
platforms since not all 64-bits are significant :-(  SYM53C8XX
has a similar limitation.  Surprisingly, the network PCI cards
have been the absolute best about this, supporting the full 64-bits
of DAC address in all card instances I delved into.

Later,
David S. Miller
davem@redhat.com