* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-06-28 22:38 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne David S. Miller
@ 2001-06-28 22:45 ` Ben LaHaise
2001-06-28 22:48 ` David S. Miller
` (3 subsequent siblings)
4 siblings, 0 replies; 36+ messages in thread
From: Ben LaHaise @ 2001-06-28 22:45 UTC (permalink / raw)
To: David S. Miller
Cc: Jes Sorensen, MEHTA,HIREN (A-SanJose,ex1),
'linux-kernel@vger.kernel.org'
On Thu, 28 Jun 2001, David S. Miller wrote:
> Shit happens Ben. One has to draw the line somewhere.
Yup, sure does.
> Sure, once 2.5.x has the interfaces, we'll add the "dummy" ones
> to 2.4.x, but only then. I don't even know %100 how I want the
> damn thing to look yet.
Well, better start suggesting fixes to the work that other people are
doing instead of saying "nope, ya gotta wait".
> There are so many issues with 64-bit DAC support, that many of
> the people whining in this thread have not even considered, and
> these very issues will be what shapes the eventual API to use.
>
> For example. I have IOMMU's on my machine, there is no real need to
> use 64-bit DAC in %99 of cases. In fact, DAC transfers run slower
> because they cannot use the DMA caching in the PCI controller.
Well, let me introduce you to some high end fibre channel devices. These
controllers can keep hundreds of thousands of io requests active at a
given time, and those requests can be very large. In fact, they can be so
large that the memory mappings that would need to remain active simply
cannot fit inside of a 32 bit address space.
> How do you represent this with the undocumented API ia64 has decided
> to use? You can't convey this information to the driver, because the
> driver may say "I don't care if it's slower, I want the large
> addressing because otherwise I'd consume or overflow the IOMMU
> resources". How do you say "SAC is preferred for performance" with
> ia64's API? You can't.
How is SAC useful on ia64? All the machines are going to be shipped with
more than 4GB of RAM, and they need an IOMMU.
Like it or not, 64 bit DMA is here, NOW. Not during the 2.6, but during
2.4. We can either start fixing the ia64 APIs and replacing them with
something that's "Right" or we can continue with ad hoc solutions.
-ben
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-06-28 22:38 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne David S. Miller
2001-06-28 22:45 ` Ben LaHaise
@ 2001-06-28 22:48 ` David S. Miller
2001-06-28 22:48 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m Alan Cox
` (2 subsequent siblings)
4 siblings, 0 replies; 36+ messages in thread
From: David S. Miller @ 2001-06-28 22:48 UTC (permalink / raw)
To: Ben LaHaise
Cc: Jes Sorensen, MEHTA,HIREN (A-SanJose,ex1),
'linux-kernel@vger.kernel.org'
Ben LaHaise writes:
> > How do you represent this with the undocumented API ia64 has decided
> > to use? You can't convey this information to the driver, because the
> > driver may say "I don't care if it's slower, I want the large
> > addressing because otherwise I'd consume or overflow the IOMMU
> > resources". How do you say "SAC is preferred for performance" with
> > ia64's API? You can't.
>
> How is SAC useful on ia64? All the machines are going to be shipped with
> more than 4GB of RAM, and they need an IOMMU.
That is all that some devices are able to do, especially sound
cards, some of which even have < 32-bit addressing limitations.
ia64 supports such devices just fine, I know it does, else you
couldn't stick an eepro100 into an ia64 box running Linux :-)
There is a software IOMMU implemented in the ia64 port, and it
handles such SAC situations today.
> Like it or not, 64 bit DMA is here, NOW. Not during the 2.6, but during
> 2.4. We can either start fixing the ia64 APIs and replacing them with
> something that's "Right" or we can continue with ad hoc solutions.
It will be fixed in 2.5.x and backported perhaps to 2.4.x, 2.4.x is
not a place for API experimentation.
Later,
David S. Miller
davem@redhat.com
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m
2001-06-28 22:38 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne David S. Miller
2001-06-28 22:45 ` Ben LaHaise
2001-06-28 22:48 ` David S. Miller
@ 2001-06-28 22:48 ` Alan Cox
2001-06-28 22:55 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne Jes Sorensen
2001-06-29 9:16 ` David S. Miller
4 siblings, 0 replies; 36+ messages in thread
From: Alan Cox @ 2001-06-28 22:48 UTC (permalink / raw)
To: David S. Miller
Cc: Ben LaHaise, Jes Sorensen, "MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
> There are so many issues with 64-bit DAC support, that many of
> the people whining in this thread have not even considered, and
Such as ? - I can see the obvious ones
- Its slower
- Not all host bridges can hit all of RAM
- You want to use SAC and IOMMU when possible
> resources". How do you say "SAC is preferred for performance" with
> ia64's API? You can't.
I doubt they worried about it, SAC isnt terribly useful on IA64 right now.
> This, almost with several other issues, need to be considered and
> handled by whatever API you come up with. If it does not address
> all of these issues somehow, it is unacceptable.
Nod
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-06-28 22:38 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne David S. Miller
` (2 preceding siblings ...)
2001-06-28 22:48 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m Alan Cox
@ 2001-06-28 22:55 ` Jes Sorensen
2001-06-29 9:16 ` David S. Miller
4 siblings, 0 replies; 36+ messages in thread
From: Jes Sorensen @ 2001-06-28 22:55 UTC (permalink / raw)
To: David S. Miller
Cc: Ben LaHaise, MEHTA,HIREN (A-SanJose,ex1),
'linux-kernel@vger.kernel.org'
>>>>> "David" == David S Miller <davem@redhat.com> writes:
David> There are so many issues with 64-bit DAC support, that many of
David> the people whining in this thread have not even considered, and
David> these very issues will be what shapes the eventual API to use.
David> For example. I have IOMMU's on my machine, there is no real
David> need to use 64-bit DAC in %99 of cases. In fact, DAC transfers
David> run slower because they cannot use the DMA caching in the PCI
David> controller.
David> How do you represent this with the undocumented API ia64 has
David> decided to use? You can't convey this information to the
David> driver, because the driver may say "I don't care if it's
David> slower, I want the large addressing because otherwise I'd
David> consume or overflow the IOMMU resources". How do you say "SAC
David> is preferred for performance" with ia64's API? You can't.
Thats easy, you use the IOMMU in pci_alloc_consistent() and friends
and return a 32bit address in that case. Most cards will simply issue
a SAC cycle if the upper 32 bits in the DMA address are zero, the ones
that don't are broken.
This way you automatically get support for the situation Ben mentioned
as well, when doing large allocs and the IOMMU is full you return a
full 64 bit address.
Jes
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-06-28 22:38 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne David S. Miller
` (3 preceding siblings ...)
2001-06-28 22:55 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne Jes Sorensen
@ 2001-06-29 9:16 ` David S. Miller
2001-06-29 9:56 ` Alan Cox
2001-06-29 20:37 ` David S. Miller
4 siblings, 2 replies; 36+ messages in thread
From: David S. Miller @ 2001-06-29 9:16 UTC (permalink / raw)
To: Jes Sorensen
Cc: Ben LaHaise, MEHTA,HIREN (A-SanJose,ex1),
'linux-kernel@vger.kernel.org'
Jes Sorensen writes:
> Thats easy
...
> This way you automatically get support for the situation Ben mentioned
> as well, when doing large allocs and the IOMMU is full you return a
> full 64 bit address.
And when the IOMMU is "full" what happens to all the SAC only
cards in the machine? pci_map_{single,sg}() are not allowed
to fail.
See, it's not so easy.
Later,
David S. Miller
davem@redhat.com
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-06-29 9:16 ` David S. Miller
@ 2001-06-29 9:56 ` Alan Cox
2001-06-29 20:37 ` David S. Miller
1 sibling, 0 replies; 36+ messages in thread
From: Alan Cox @ 2001-06-29 9:56 UTC (permalink / raw)
To: David S. Miller
Cc: Jes Sorensen, Ben LaHaise, "MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
> > as well, when doing large allocs and the IOMMU is full you return a
> > full 64 bit address.
>
> And when the IOMMU is "full" what happens to all the SAC only
> cards in the machine? pci_map_{single,sg}() are not allowed
> to fail.
Thats something we already know has to be fixed. Its true with or without
an IOMMU that there may be cases where there is no free mapping space
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-06-29 9:16 ` David S. Miller
2001-06-29 9:56 ` Alan Cox
@ 2001-06-29 20:37 ` David S. Miller
2001-07-05 21:06 ` Jes Sorensen
2001-07-05 23:47 ` David S. Miller
1 sibling, 2 replies; 36+ messages in thread
From: David S. Miller @ 2001-06-29 20:37 UTC (permalink / raw)
To: Alan Cox
Cc: Jes Sorensen, Ben LaHaise, "MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
Alan Cox writes:
> > And when the IOMMU is "full" what happens to all the SAC only
> > cards in the machine? pci_map_{single,sg}() are not allowed
> > to fail.
>
> Thats something we already know has to be fixed. Its true with or without
> an IOMMU that there may be cases where there is no free mapping space
True, but my intended point is that starving the SAC-only users then
returning DAC addresses to DAC-capable devices is just as unacceptable.
When we have one of these compute cluster cards in the box, and Jes's
suggested algorithm is used, the rest of the SAC devices in the box
would be totally screwed once the IOMMU fills up.
Later,
David S. Miller
davem@redhat.com
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-06-29 20:37 ` David S. Miller
@ 2001-07-05 21:06 ` Jes Sorensen
2001-07-05 23:47 ` David S. Miller
1 sibling, 0 replies; 36+ messages in thread
From: Jes Sorensen @ 2001-07-05 21:06 UTC (permalink / raw)
To: David S. Miller
Cc: Alan Cox, Ben LaHaise, "MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
>>>>> "David" == David S Miller <davem@redhat.com> writes:
David> Alan Cox writes:
>> Thats something we already know has to be fixed. Its true with or
>> without an IOMMU that there may be cases where there is no free
>> mapping space
David> True, but my intended point is that starving the SAC-only users
David> then returning DAC addresses to DAC-capable devices is just as
David> unacceptable.
David> When we have one of these compute cluster cards in the box, and
David> Jes's suggested algorithm is used, the rest of the SAC devices
David> in the box would be totally screwed once the IOMMU fills up.
The dma_mask in struct pci_dev tells you whether you are DAC
capable. We pass a pointer to this struct when we call the pci_*
functions so the required information needed to make the decision
whether to return a SAC or a DAC address is already available.
Cheers
Jes
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-06-29 20:37 ` David S. Miller
2001-07-05 21:06 ` Jes Sorensen
@ 2001-07-05 23:47 ` David S. Miller
2001-07-05 23:50 ` Ben LaHaise
` (2 more replies)
1 sibling, 3 replies; 36+ messages in thread
From: David S. Miller @ 2001-07-05 23:47 UTC (permalink / raw)
To: Jes Sorensen
Cc: Alan Cox, Ben LaHaise, "MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
Jes Sorensen writes:
> The dma_mask in struct pci_dev tells you whether you are DAC
> capable. We pass a pointer to this struct when we call the pci_*
> functions so the required information needed to make the decision
> whether to return a SAC or a DAC address is already available.
The decision is not based upon "device capable of DAC", that is
precisely my point.
The decision must be based upon a number of considerations.
1) Can it do DAC
2) Is DAC more efficient than SAC on this platform
3) Does the devices _need_ DAC even if it is slower because
it requires referencing large portions of the DMA
address space simultaneously
Sure, you could imply all of this complexity in the driver
by making them consider all of these issues when setting the
mask, but that isn't a nice interface at all.
And this still leaves the 64-bit dma_addr_t overhead issue.
Later,
David S. Miller
davem@redhat.com
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-07-05 23:47 ` David S. Miller
@ 2001-07-05 23:50 ` Ben LaHaise
2001-07-06 13:31 ` Jes Sorensen
2001-07-06 23:46 ` David S. Miller
2 siblings, 0 replies; 36+ messages in thread
From: Ben LaHaise @ 2001-07-05 23:50 UTC (permalink / raw)
To: David S. Miller
Cc: Jes Sorensen, Alan Cox, MEHTA,HIREN (A-SanJose,ex1),
'linux-kernel@vger.kernel.org'
On Thu, 5 Jul 2001, David S. Miller wrote:
> And this still leaves the 64-bit dma_addr_t overhead issue.
Huh? It's a config option, just like blkoff_t.
-ben
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-07-05 23:47 ` David S. Miller
2001-07-05 23:50 ` Ben LaHaise
@ 2001-07-06 13:31 ` Jes Sorensen
2001-07-06 23:46 ` David S. Miller
2 siblings, 0 replies; 36+ messages in thread
From: Jes Sorensen @ 2001-07-06 13:31 UTC (permalink / raw)
To: David S. Miller
Cc: Alan Cox, Ben LaHaise, "MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
>>>>> "David" == David S Miller <davem@redhat.com> writes:
David> Jes Sorensen writes:
>> The dma_mask in struct pci_dev tells you whether you are DAC
>> capable. We pass a pointer to this struct when we call the pci_*
>> functions so the required information needed to make the decision
>> whether to return a SAC or a DAC address is already available.
David> The decision is not based upon "device capable of DAC", that is
David> precisely my point.
I understand that, it's part of the equation.
David> The decision must be based upon a number of considerations.
David> 1) Can it do DAC
David> 2) Is DAC more efficient than SAC on this platform
That sounds to me like a 'static' decision at compile time or at least
something decided upon in the pci_* code as it will be the same for
all devices on the bus. If your IOMMU is very complex to program
compared to the overhead of DAC cycles you pick DAC etc.
David> 3) Does the devices _need_ DAC even if it is slower because it
David> requires referencing large portions of the DMA address space
David> simultaneously
Sure or because the IOMMU is starved. Most devices will do SAC/DAC
based on the address you throw at them, ie. it's perfectly valid to
mix and match DAC and SAC addresses handed to a device.
David> Sure, you could imply all of this complexity in the driver by
David> making them consider all of these issues when setting the mask,
David> but that isn't a nice interface at all.
David> And this still leaves the 64-bit dma_addr_t overhead issue.
The 64 bit dma_addr_t is only an issue on 32 bit architectures with
highmem enabled. I never suggested making dma_addr_t 64 bit on 32 bit
architectures as a general thing.
Jes
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-07-05 23:47 ` David S. Miller
2001-07-05 23:50 ` Ben LaHaise
2001-07-06 13:31 ` Jes Sorensen
@ 2001-07-06 23:46 ` David S. Miller
2001-07-07 3:58 ` Ben LaHaise
2001-07-07 5:35 ` David S. Miller
2 siblings, 2 replies; 36+ messages in thread
From: David S. Miller @ 2001-07-06 23:46 UTC (permalink / raw)
To: Jes Sorensen
Cc: Alan Cox, Ben LaHaise, "MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
Jes Sorensen writes:
> David> And this still leaves the 64-bit dma_addr_t overhead issue.
>
> The 64 bit dma_addr_t is only an issue on 32 bit architectures with
> highmem enabled. I never suggested making dma_addr_t 64 bit on 32 bit
> architectures as a general thing.
What about for drivers of SAC-only devices, they eat the overhead
when highmem is enabled too?
This says nothing about the real reason the IA64 solution is
unacceptable, the inputs to the mapping functions which must
be "page+offset+len" triplets as there is no logical "virtual
address" to pass into the mapping routines on 32-bit systems.
Face it, the ia64 stuff is not what we can use across the board. It
simply doesn't deal with all the necessary issues. Therefore,
encouraging driver author's to use this ia64 hacked up scheme is
not such a hot idea until we have a real API implemented.
Later,
David S. Miller
davem@redhat.com
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-07-06 23:46 ` David S. Miller
@ 2001-07-07 3:58 ` Ben LaHaise
2001-07-07 5:35 ` David S. Miller
1 sibling, 0 replies; 36+ messages in thread
From: Ben LaHaise @ 2001-07-07 3:58 UTC (permalink / raw)
To: David S. Miller
Cc: Jes Sorensen, Alan Cox, MEHTA,HIREN (A-SanJose,ex1),
'linux-kernel@vger.kernel.org'
On Fri, 6 Jul 2001, David S. Miller wrote:
> What about for drivers of SAC-only devices, they eat the overhead
> when highmem is enabled too?
Yes. It's not an unreasonable overhead considering that it's configured
out for all the non-highmem kernels that will be shipped. Keep in mind
that the expected lifespan for 32 bit systems is now less than 3 years, so
elaborate planning that delays implementation buys us nothing more than a
smaller window of usefulness.
> This says nothing about the real reason the IA64 solution is
> unacceptable, the inputs to the mapping functions which must
> be "page+offset+len" triplets as there is no logical "virtual
> address" to pass into the mapping routines on 32-bit systems.
On x86 a 64 bit DMA address cookie is fine. If you've got concerns, tell
us what you have in mind for a design.
> Face it, the ia64 stuff is not what we can use across the board. It
> simply doesn't deal with all the necessary issues. Therefore,
> encouraging driver author's to use this ia64 hacked up scheme is
> not such a hot idea until we have a real API implemented.
So what's the API?
-ben
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne
2001-07-06 23:46 ` David S. Miller
2001-07-07 3:58 ` Ben LaHaise
@ 2001-07-07 5:35 ` David S. Miller
2001-07-07 12:06 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m Alan Cox
1 sibling, 1 reply; 36+ messages in thread
From: David S. Miller @ 2001-07-07 5:35 UTC (permalink / raw)
To: Ben LaHaise
Cc: Jes Sorensen, Alan Cox, MEHTA,HIREN (A-SanJose,ex1),
'linux-kernel@vger.kernel.org'
Ben LaHaise writes:
> Yes. It's not an unreasonable overhead considering that it's configured
> out for all the non-highmem kernels that will be shipped. Keep in mind
> that the expected lifespan for 32 bit systems is now less than 3 years, so
> elaborate planning that delays implementation buys us nothing more than a
> smaller window of usefulness.
Maybe by then only 64-bit cpus will matter. Who knows.
> > This says nothing about the real reason the IA64 solution is
> > unacceptable, the inputs to the mapping functions which must
> > be "page+offset+len" triplets as there is no logical "virtual
> > address" to pass into the mapping routines on 32-bit systems.
>
> On x86 a 64 bit DMA address cookie is fine. If you've got concerns, tell
> us what you have in mind for a design.
Things along the lines of what Jens Axboe's patches are what I'm
thinking about.
> So what's the API?
See Jens's patches.
First it has to pass in page/off/len triplets, on all platforms.
This is addressed by Jens's interfaces.
Secondly it has to provide a query mechanism to delineate the
three cases:
1) DAC is faster and always preferred
2) SAC is faster
3) DAC may be slower but more desirable for certain devices
due to large amounts of parallel address space usage
I have not designed an interface for this, but it ought to be
quite simple.
Thirdly seperate 32-bit/64-bit DMA address types. Added to the
overhead concerns, I also think it sucks big donkey balls to cast the
things around, especially since different platforms would potentially
require different casts to eliminate the warnings. In fact, with your
suggested scheme, the setting of highmem would determine a core
type.
In fact, I'm not going to bother to code one single bit of this myself
until I am convinced I have thought the whole problem over properly.
This is the part nobody else wants to do, but it is a prerequisite for
this sort of API.
Later,
David S. Miller
davem@redhat.com
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m
2001-07-07 5:35 ` David S. Miller
@ 2001-07-07 12:06 ` Alan Cox
2001-07-07 12:17 ` Jeff Garzik
0 siblings, 1 reply; 36+ messages in thread
From: Alan Cox @ 2001-07-07 12:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ben LaHaise, Jes Sorensen, Alan Cox,
"MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
> > that the expected lifespan for 32 bit systems is now less than 3 years, so
> > elaborate planning that delays implementation buys us nothing more than a
> > smaller window of usefulness.
> Maybe by then only 64-bit cpus will matter. Who knows.
Reality check.
Embedded PCI 32bit processors are going to be very common
People are only now retiring 486's
So add another seven or eight years to your estimate
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m
2001-07-07 12:06 ` (reposting) how to get DMA'able memory within 4GB on 64-bit m Alan Cox
@ 2001-07-07 12:17 ` Jeff Garzik
2001-07-07 12:21 ` Alan Cox
2001-07-07 13:00 ` David S. Miller
0 siblings, 2 replies; 36+ messages in thread
From: Jeff Garzik @ 2001-07-07 12:17 UTC (permalink / raw)
To: Alan Cox
Cc: David S. Miller, Ben LaHaise, Jes Sorensen,
MEHTA,HIREN (A-SanJose,ex1),
'linux-kernel@vger.kernel.org'
Alan Cox wrote:
>
> > > that the expected lifespan for 32 bit systems is now less than 3 years, so
> > > elaborate planning that delays implementation buys us nothing more than a
> > > smaller window of usefulness.
> > Maybe by then only 64-bit cpus will matter. Who knows.
>
> Reality check.
>
> Embedded PCI 32bit processors are going to be very common
> People are only now retiring 486's
>
> So add another seven or eight years to your estimate
Given a little more context, I thought we were talking specifically
about 64bit-PCI-on-32bit-machines?
Assuming that, AFAICS Ben's statement seems more correct.
And IMHO we definitely should not optimize for 64-bit-on-32-bit case.
Let CONFIG_HIGHMEM grow dma_addr_t to 64-bits, for that case only...
--
Jeff Garzik | A recent study has shown that too much soup
Building 1024 | can cause malaise in laboratory mice.
MandrakeSoft |
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m
2001-07-07 12:17 ` Jeff Garzik
@ 2001-07-07 12:21 ` Alan Cox
2001-07-07 13:00 ` David S. Miller
1 sibling, 0 replies; 36+ messages in thread
From: Alan Cox @ 2001-07-07 12:21 UTC (permalink / raw)
To: Jeff Garzik
Cc: Alan Cox, David S. Miller, Ben LaHaise, Jes Sorensen,
"MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
> > So add another seven or eight years to your estimate
>
> Given a little more context, I thought we were talking specifically
> about 64bit-PCI-on-32bit-machines?
>
> Assuming that, AFAICS Ben's statement seems more correct.
>
> And IMHO we definitely should not optimize for 64-bit-on-32-bit case.
> Let CONFIG_HIGHMEM grow dma_addr_t to 64-bits, for that case only...
I see no good way to optimise for 64bit dma on a 32bit box. The existing API
is extremely clean, easy to understand and I don't agree with Dave's desire
to write another whole concoction.
We do need pci_set_dma_mask_bits(), which in itself lets drivers indicate
SAC/DAC capable. We do want a way for nosey or fine tuned drivers to query
SAC/DAC properties, but most drivers should be letting arch code make arch
policy.
Alan
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m
2001-07-07 12:17 ` Jeff Garzik
2001-07-07 12:21 ` Alan Cox
@ 2001-07-07 13:00 ` David S. Miller
2001-07-11 19:16 ` Jes Sorensen
2001-07-11 23:07 ` David S. Miller
1 sibling, 2 replies; 36+ messages in thread
From: David S. Miller @ 2001-07-07 13:00 UTC (permalink / raw)
To: Alan Cox
Cc: Jeff Garzik, Ben LaHaise, Jes Sorensen,
"MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
Alan Cox writes:
> I see no good way to optimise for 64bit dma on a 32bit box.
I'm actually not only talking about DAC device on 32-bit cpus. Just
as much, I'm talking about drivers for SAC-only devices even on 64-bit
cpus.
I took a lot of crap from driver authors when we started pushing the
PCI dma stuff on people, because of the dma_addr_t people now had to
keep around to unmap the thing later.
To a certain extent I agreed with these folks. I'll be gutting myself
if I make everyone eat twice as much space just to add DAC support to
the kernel :-)
>From yet another perspective, my proposals have also to do with what
API can actually work on all platforms. This is pretty important to
me. I remember yesteryear when I used to give myself the privilege
of being self-arch-centric in my work, a Sparc hack here, a Sparc hack
there. But I simply cannot operate this way anymore. My conscious
will no longer allow me to crap up things like that :-)
> I don't agree with Dave's desire to write another whole concoction.
It needs to be a new set of interfaces (and at that point, why not use
a different dma64_addr_t type and save overhead for SAC-only devices
while we're at it :-) because the proper inputs for a DAC mapping
involve page/off/len pairs.
Ignoring addressing limits of 32-bit cpus for a moment, consider that
this page/off/len triplet is the natural currency the kernel uses for
this kind of stuff anyways.
I think it is interesting to note that Jens noticed this immediately,
him being the first person to actually try and implement something
that would work on 32-bit platforms.
Later,
David S. Miller
davem@redhat.com
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m
2001-07-07 13:00 ` David S. Miller
@ 2001-07-11 19:16 ` Jes Sorensen
2001-07-11 21:54 ` Chris Wedgwood
2001-07-11 23:17 ` David S. Miller
2001-07-11 23:07 ` David S. Miller
1 sibling, 2 replies; 36+ messages in thread
From: Jes Sorensen @ 2001-07-11 19:16 UTC (permalink / raw)
To: David S. Miller
Cc: Alan Cox, Jeff Garzik, Ben LaHaise,
"MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
>>>>> "David" == David S Miller <davem@redhat.com> writes:
David> Alan Cox writes:
>> I see no good way to optimise for 64bit dma on a 32bit box.
David> I'm actually not only talking about DAC device on 32-bit cpus.
David> Just as much, I'm talking about drivers for SAC-only devices
David> even on 64-bit cpus.
David> I took a lot of crap from driver authors when we started
David> pushing the PCI dma stuff on people, because of the dma_addr_t
David> people now had to keep around to unmap the thing later.
David> To a certain extent I agreed with these folks. I'll be gutting
David> myself if I make everyone eat twice as much space just to add
David> DAC support to the kernel :-)
The overhead is going be negligeble, the overhead of highmem itself is
much worse. Not to mention that today some dma_addr_t's might not be
packed properly in data structure hence they ending up taking 8 bytes
anyway.
Jes
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m
2001-07-11 19:16 ` Jes Sorensen
@ 2001-07-11 21:54 ` Chris Wedgwood
2001-07-11 23:17 ` David S. Miller
1 sibling, 0 replies; 36+ messages in thread
From: Chris Wedgwood @ 2001-07-11 21:54 UTC (permalink / raw)
To: Jes Sorensen
Cc: David S. Miller, Alan Cox, Jeff Garzik, Ben LaHaise,
"MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
On Wed, Jul 11, 2001 at 09:16:50PM +0200, Jes Sorensen wrote:
The overhead is going be negligeble, the overhead of highmem itself is
much worse. Not to mention that today some dma_addr_t's might not be
packed properly in data structure hence they ending up taking 8 bytes
anyway.
What kind of packing makes a 32-bit value take 8-bytes on any
currently supported archicture? The worst-case I can think of is
7-bytes in the case of misaligned by 3 (e.g. __attribute__((packed))
struct blah { char foo[3]; long bar }; sort of thing).
--cw
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m
2001-07-11 19:16 ` Jes Sorensen
2001-07-11 21:54 ` Chris Wedgwood
@ 2001-07-11 23:17 ` David S. Miller
1 sibling, 0 replies; 36+ messages in thread
From: David S. Miller @ 2001-07-11 23:17 UTC (permalink / raw)
To: Chris Wedgwood
Cc: Jes Sorensen, Alan Cox, Jeff Garzik, Ben LaHaise,
"MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
Chris Wedgwood writes:
> What kind of packing makes a 32-bit value take 8-bytes on any
> currently supported archicture? The worst-case I can think of is
> 7-bytes in the case of misaligned by 3 (e.g. __attribute__((packed))
> struct blah { char foo[3]; long bar }; sort of thing).
If you have this:
struct {
u32 foo;
void *bar;
};
"bar" will be at offset 8 on a 64-bit platforms since it must be
aligned on a 64-byte boundary, so what Jes is saying is that for:
struct {
dma_addr_t foo;
void *bar;
};
the "dma_addr_t" is already consuming 8 bytes of space on 64-bit
systems.
Later,
David S. Miller
davem@redhat.com
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m
2001-07-07 13:00 ` David S. Miller
2001-07-11 19:16 ` Jes Sorensen
@ 2001-07-11 23:07 ` David S. Miller
1 sibling, 0 replies; 36+ messages in thread
From: David S. Miller @ 2001-07-11 23:07 UTC (permalink / raw)
To: Jes Sorensen
Cc: Alan Cox, Jeff Garzik, Ben LaHaise,
"MEHTA,HIREN (A-SanJose,ex1)",
'linux-kernel@vger.kernel.org'
Jes Sorensen writes:
> The overhead is going be negligeble, the overhead of highmem itself is
> much worse.
Once Jens's block layer stuff goes in, a lot of that overhead simply
disappears since the page cache need not bounce buffers. The
networking can already technically cope with this too.
> Not to mention that today some dma_addr_t's might not be
> packed properly in data structure hence they ending up taking 8 bytes
> anyway.
Not on x86 which is the current main benefactor of highmem.
Later,
David S. Miller
davem@redhat.com
^ permalink raw reply [flat|nested] 36+ messages in thread