RE: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

RE: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[MEHTA,HIREN (A-SanJose,ex1)]

Then why do we have 64-bit dma_addr_t on ia64 ?

-hiren

-----Original Message-----
From: David S. Miller [mailto:davem@redhat.com]
Sent: Wednesday, June 27, 2001 2:26 PM
To: MEHTA,HIREN (A-SanJose,ex1)
Cc: 'linux-kernel@vger.kernel.org'
Subject: Re: (reposting) how to get DMA'able memory within 4GB on 64-bit
machi ne



MEHTA,HIREN (A-SanJose,ex1) writes:
 > Is there a way for a driver to ask kernel to
 > give DMA'able memory within 4GB ? I read about
 > pci_alloc_consistent(). But I could not find out
 > whether that guarantees the DMA'able memory to be
 > within 4GB or not. Is there any other kernel routine
 > that I should call from Driver to get such a memory ?

All of the pci_*() DMA allocation/mapping interfaces give you
32-bit PCI dma addresses.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Jes Sorensen]

>>>>> "Hiren" == MEHTA,HIREN (A-SanJose,ex1) <hiren_mehta@agilent.com> writes:

Hiren> Then why do we have 64-bit dma_addr_t on ia64 ?  -hiren

Because on ia64 you will get back a 64 bit pointer if you use
pci_set_dma_mask() to set a 64 bit mask before calling the pci
functions in question.

Jes

RE: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

MEHTA,HIREN (A-SanJose,ex1) writes:
 > Then why do we have 64-bit dma_addr_t on ia64 ?

Because what the ia64 folks wanted to happen was to make the existing
pci DMA interfaces return 64-bit DAC addresses if the driver made a
certain call into the PCI subsystem first.

I am totally against this architecture, but this is no matter.

If you do nothing other than call the existing PCI dma interfaces, no
matter how things work in the future you will get 32-bit addresses.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

Jes Sorensen writes:
 > >>>>> "Hiren" == MEHTA,HIREN (A-SanJose,ex1) <hiren_mehta@agilent.com> writes:
 > 
 > Hiren> Then why do we have 64-bit dma_addr_t on ia64 ?  -hiren
 > 
 > Because on ia64 you will get back a 64 bit pointer if you use
 > pci_set_dma_mask() to set a 64 bit mask before calling the pci
 > functions in question.

Please note that this is nonstandard and undocumented behavior.

This is not a supported API at all, and the way 64-bit DMA will
eventually be done across all platforms is likely to be different.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Jes Sorensen]

>>>>> "David" == David S Miller <davem@redhat.com> writes:

David> Jes Sorensen writes:
>>  Because on ia64 you will get back a 64 bit pointer if you use
>> pci_set_dma_mask() to set a 64 bit mask before calling the pci
>> functions in question.

David> Please note that this is nonstandard and undocumented behavior.

David> This is not a supported API at all, and the way 64-bit DMA will
David> eventually be done across all platforms is likely to be
David> different.

Well please also note there has been requests for proper 64 bit DMA
support for over 3 years or so by now.

The interface we use works well, so why should it be changed for other
architecures? Instead it would make a lot more sense to support it on
other architectures that can do 64 bit DMA.

Jes

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Ben LaHaise]

On Thu, 28 Jun 2001, David S. Miller wrote:

> Please note that this is nonstandard and undocumented behavior.
>
> This is not a supported API at all, and the way 64-bit DMA will
> eventually be done across all platforms is likely to be different.

Well, what is the standard API to use?  All these 64 bit cards in my
machine really make that 95% cpu usage in bounce buffer copying rather
depressing.

		-ben

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

Jes Sorensen writes:
 > >>>>> "David" == David S Miller <davem@redhat.com> writes:
 > 
 > David> Jes Sorensen writes:
 > >>  Because on ia64 you will get back a 64 bit pointer if you use
 > >> pci_set_dma_mask() to set a 64 bit mask before calling the pci
 > >> functions in question.
 > 
 > David> Please note that this is nonstandard and undocumented behavior.
 > 
 > David> This is not a supported API at all, and the way 64-bit DMA will
 > David> eventually be done across all platforms is likely to be
 > David> different.
 > 
 > Well please also note there has been requests for proper 64 bit DMA
 > support for over 3 years or so by now.

Hey Jes, figure this out, I did all the work to get proper 32 bit DMA
to work.  I was waiting 3 years for that and I did all the work
myself.  I was patient, people needing 64-bit DMA can be patient as
well and wait 1 or 2 weeks for 2.5.x to start up so we can make these
kinds of changes.

 > The interface we use works well, so why should it be changed for other
 > architecures? Instead it would make a lot more sense to support it on
 > other architectures that can do 64 bit DMA.

Because it makes not one iota of sense to have a 64-bit dma_addr_t on
a 32-bit system where none of this DAC crap is relevant.

That is why.

Send Linus a patch which makes dma_addr_t 64-bit on ix86, see how far
you get.  And I would totally agree with him, the overhead of the
larger type is totally stupid for %99 of cases on x86.

Sure, if HIGHMEM or whatever is set, it may make sense.  But I think
eating the 64-bit type in all drivers using dma_addr_t, even one's
only capable of 32-bit PCI addressing, is bogus as well.

This is why I wanted a seperate dma64_addr_t and pci64_*() routines to
match.  So the overhead only existed where it was absolutely needed.
People already bark at me about the overhead of adding dma_addr_t when
"virt_to_bus and bus_to_virt worked perfectly fine without having to
 keep track of these DMA cookies everywhere" and to a certain extent
they are right.  So we should avoid making this worse when possible.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Alan Cox]

> The interface we use works well, so why should it be changed for other
> architecures? Instead it would make a lot more sense to support it on
> other architectures that can do 64 bit DMA.

The changes needed are small IMHO. The big problem wa the pci_dma_mask
not being pci_dma_mask_bit(foo) - a minor oversight we can fix in 2.5 without
breaking anything else.

It can even default to 32 for non DAC cards.

We will also have to address those cards that have 28/30/31 bit limits (yes
they exist) when we start doing direct I/O for 32bits of memory - one reason
I'm very wary of Jens patch ever being in 2.4


Alan

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

Ben LaHaise writes:
 > On Thu, 28 Jun 2001, David S. Miller wrote:
 > 
 > > Please note that this is nonstandard and undocumented behavior.
 > >
 > > This is not a supported API at all, and the way 64-bit DMA will
 > > eventually be done across all platforms is likely to be different.
 > 
 > Well, what is the standard API to use?  All these 64 bit cards in my
 > machine really make that 95% cpu usage in bounce buffer copying rather
 > depressing.

There simply is no standard API for 64-bit DAC, sorry.

This isn't changing in 2.4.x, please face facts.  It simply is not
a feature of 2.4.x, and we are well beyond feature freeze now.

It is certainly changing in 2.5.x which is just around the corner.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Ben LaHaise]

On Thu, 28 Jun 2001, David S. Miller wrote:

> There simply is no standard API for 64-bit DAC, sorry.
>
> This isn't changing in 2.4.x, please face facts.  It simply is not
> a feature of 2.4.x, and we are well beyond feature freeze now.
>
> It is certainly changing in 2.5.x which is just around the corner.

Sorry, but that's not a good enough answer if 2.5 takes the same 2 years
that 2.3 took.  Define the API so that people can at least write their
drivers to the spec, or else suffer the consequences of people doing their
own thing.

		-ben

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

Ben LaHaise writes:
 > Sorry, but that's not a good enough answer if 2.5 takes the same 2 years
 > that 2.3 took.  Define the API so that people can at least write their
 > drivers to the spec, or else suffer the consequences of people doing their
 > own thing.

Shit happens Ben.  One has to draw the line somewhere.

Sure, once 2.5.x has the interfaces, we'll add the "dummy" ones
to 2.4.x, but only then.  I don't even know %100 how I want the
damn thing to look yet.

There are so many issues with 64-bit DAC support, that many of
the people whining in this thread have not even considered, and
these very issues will be what shapes the eventual API to use.

For example.  I have IOMMU's on my machine, there is no real need to
use 64-bit DAC in %99 of cases.  In fact, DAC transfers run slower
because they cannot use the DMA caching in the PCI controller.

How do you represent this with the undocumented API ia64 has decided
to use?  You can't convey this information to the driver, because the
driver may say "I don't care if it's slower, I want the large
addressing because otherwise I'd consume or overflow the IOMMU
resources".  How do you say "SAC is preferred for performance" with
ia64's API?  You can't.

This, almost with several other issues, need to be considered and
handled by whatever API you come up with.  If it does not address
all of these issues somehow, it is unacceptable.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m

[Alan Cox]

> Sorry, but that's not a good enough answer if 2.5 takes the same 2 years
> that 2.3 took.  Define the API so that people can at least write their
> drivers to the spec, or else suffer the consequences of people doing their
> own thing.

At which point we can also then drop it into 2.4 so that all the non IA64
architectures just happen to resolve to the same old 32bit calls via
a tiny set of #defines and the IA64 - which requires it for basically
everything anyway - can use it natively, and if they get it wrong suffer
their own problems without annoying anyone else

Alan

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Ben LaHaise]

On Thu, 28 Jun 2001, David S. Miller wrote:

> Shit happens Ben.  One has to draw the line somewhere.

Yup, sure does.

> Sure, once 2.5.x has the interfaces, we'll add the "dummy" ones
> to 2.4.x, but only then.  I don't even know %100 how I want the
> damn thing to look yet.

Well, better start suggesting fixes to the work that other people are
doing instead of saying "nope, ya gotta wait".

> There are so many issues with 64-bit DAC support, that many of
> the people whining in this thread have not even considered, and
> these very issues will be what shapes the eventual API to use.
>
> For example.  I have IOMMU's on my machine, there is no real need to
> use 64-bit DAC in %99 of cases.  In fact, DAC transfers run slower
> because they cannot use the DMA caching in the PCI controller.

Well, let me introduce you to some high end fibre channel devices.  These
controllers can keep hundreds of thousands of io requests active at a
given time, and those requests can be very large.  In fact, they can be so
large that the memory mappings that would need to remain active simply
cannot fit inside of a 32 bit address space.

> How do you represent this with the undocumented API ia64 has decided
> to use?  You can't convey this information to the driver, because the
> driver may say "I don't care if it's slower, I want the large
> addressing because otherwise I'd consume or overflow the IOMMU
> resources".  How do you say "SAC is preferred for performance" with
> ia64's API?  You can't.

How is SAC useful on ia64?  All the machines are going to be shipped with
more than 4GB of RAM, and they need an IOMMU.

Like it or not, 64 bit DMA is here, NOW.  Not during the 2.6, but during
2.4.  We can either start fixing the ia64 APIs and replacing them with
something that's "Right" or we can continue with ad hoc solutions.

		-ben

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

Ben LaHaise writes:
 > > How do you represent this with the undocumented API ia64 has decided
 > > to use?  You can't convey this information to the driver, because the
 > > driver may say "I don't care if it's slower, I want the large
 > > addressing because otherwise I'd consume or overflow the IOMMU
 > > resources".  How do you say "SAC is preferred for performance" with
 > > ia64's API?  You can't.
 > 
 > How is SAC useful on ia64?  All the machines are going to be shipped with
 > more than 4GB of RAM, and they need an IOMMU.

That is all that some devices are able to do, especially sound
cards, some of which even have < 32-bit addressing limitations.
ia64 supports such devices just fine, I know it does, else you
couldn't stick an eepro100 into an ia64 box running Linux :-)

There is a software IOMMU implemented in the ia64 port, and it
handles such SAC situations today.

 > Like it or not, 64 bit DMA is here, NOW.  Not during the 2.6, but during
 > 2.4.  We can either start fixing the ia64 APIs and replacing them with
 > something that's "Right" or we can continue with ad hoc solutions.

It will be fixed in 2.5.x and backported perhaps to 2.4.x, 2.4.x is
not a place for API experimentation.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m

[Alan Cox]

> There are so many issues with 64-bit DAC support, that many of
> the people whining in this thread have not even considered, and

Such as ? - I can see the obvious ones

-	Its slower
-	Not all host bridges can hit all of RAM
-	You want to use SAC and IOMMU when possible

> resources".  How do you say "SAC is preferred for performance" with
> ia64's API?  You can't.

I doubt they worried about it, SAC isnt terribly useful on IA64 right now.

> This, almost with several other issues, need to be considered and
> handled by whatever API you come up with.  If it does not address
> all of these issues somehow, it is unacceptable.

Nod

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Jes Sorensen]

>>>>> "David" == David S Miller <davem@redhat.com> writes:

David> There are so many issues with 64-bit DAC support, that many of
David> the people whining in this thread have not even considered, and
David> these very issues will be what shapes the eventual API to use.

David> For example.  I have IOMMU's on my machine, there is no real
David> need to use 64-bit DAC in %99 of cases.  In fact, DAC transfers
David> run slower because they cannot use the DMA caching in the PCI
David> controller.

David> How do you represent this with the undocumented API ia64 has
David> decided to use?  You can't convey this information to the
David> driver, because the driver may say "I don't care if it's
David> slower, I want the large addressing because otherwise I'd
David> consume or overflow the IOMMU resources".  How do you say "SAC
David> is preferred for performance" with ia64's API?  You can't.

Thats easy, you use the IOMMU in pci_alloc_consistent() and friends
and return a 32bit address in that case. Most cards will simply issue
a SAC cycle if the upper 32 bits in the DMA address are zero, the ones
that don't are broken.

This way you automatically get support for the situation Ben mentioned
as well, when doing large allocs and the IOMMU is full you return a
full 64 bit address.

Jes

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

Jes Sorensen writes:
 > Thats easy
 ...
 > This way you automatically get support for the situation Ben mentioned
 > as well, when doing large allocs and the IOMMU is full you return a
 > full 64 bit address.

And when the IOMMU is "full" what happens to all the SAC only
cards in the machine?  pci_map_{single,sg}() are not allowed
to fail.

See, it's not so easy.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Alan Cox]

>  > as well, when doing large allocs and the IOMMU is full you return a
>  > full 64 bit address.
> 
> And when the IOMMU is "full" what happens to all the SAC only
> cards in the machine?  pci_map_{single,sg}() are not allowed
> to fail.

Thats something we already know has to be fixed. Its true with or without
an IOMMU that there may be cases where there is no free mapping space

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

Alan Cox writes:
 > > And when the IOMMU is "full" what happens to all the SAC only
 > > cards in the machine?  pci_map_{single,sg}() are not allowed
 > > to fail.
 > 
 > Thats something we already know has to be fixed. Its true with or without
 > an IOMMU that there may be cases where there is no free mapping space

True, but my intended point is that starving the SAC-only users then
returning DAC addresses to DAC-capable devices is just as unacceptable.

When we have one of these compute cluster cards in the box, and Jes's
suggested algorithm is used, the rest of the SAC devices in the box
would be totally screwed once the IOMMU fills up.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Jens Axboe]

On Thu, Jun 28 2001, Alan Cox wrote:
> We will also have to address those cards that have 28/30/31 bit limits (yes
> they exist) when we start doing direct I/O for 32bits of memory - one reason
> I'm very wary of Jens patch ever being in 2.4

The patch can handle those too, FWIW. The fact that it just sets 32-bit
limit now is unrelated, and it is also just set for devices that have
been sort-of tested :-)

Anyhoo, my point is that the bounce limit is variable on a page
granularity.

-- 
Jens Axboe

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Jens Axboe]

On Thu, Jun 28 2001, Ben LaHaise wrote:
> On Thu, 28 Jun 2001, David S. Miller wrote:
> 
> > Please note that this is nonstandard and undocumented behavior.
> >
> > This is not a supported API at all, and the way 64-bit DMA will
> > eventually be done across all platforms is likely to be different.
> 
> Well, what is the standard API to use?  All these 64 bit cards in my
> machine really make that 95% cpu usage in bounce buffer copying rather
> depressing.

The current sg list and single mapping functions are useless on 64-bit
(and highmem) anyway. I've used struct sg_list as a scatterlist
replacement for some time that holds a page/offset/length thing instead,
and also used pci_map_page for single mappings.

Using a virtual address for this stuff seems a bit short-sighted to
me...

-- 
Jens Axboe

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Jes Sorensen]

>>>>> "David" == David S Miller <davem@redhat.com> writes:

David> Alan Cox writes:
>> Thats something we already know has to be fixed. Its true with or
>> without an IOMMU that there may be cases where there is no free
>> mapping space

David> True, but my intended point is that starving the SAC-only users
David> then returning DAC addresses to DAC-capable devices is just as
David> unacceptable.

David> When we have one of these compute cluster cards in the box, and
David> Jes's suggested algorithm is used, the rest of the SAC devices
David> in the box would be totally screwed once the IOMMU fills up.

The dma_mask in struct pci_dev tells you whether you are DAC
capable. We pass a pointer to this struct when we call the pci_*
functions so the required information needed to make the decision
whether to return a SAC or a DAC address is already available.

Cheers
Jes

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

Jes Sorensen writes:
 > The dma_mask in struct pci_dev tells you whether you are DAC
 > capable. We pass a pointer to this struct when we call the pci_*
 > functions so the required information needed to make the decision
 > whether to return a SAC or a DAC address is already available.

The decision is not based upon "device capable of DAC", that is
precisely my point.

The decision must be based upon a number of considerations.

1) Can it do DAC

2) Is DAC more efficient than SAC on this platform

3) Does the devices _need_ DAC even if it is slower because
   it requires referencing large portions of the DMA
   address space simultaneously

Sure, you could imply all of this complexity in the driver
by making them consider all of these issues when setting the
mask, but that isn't a nice interface at all.

And this still leaves the 64-bit dma_addr_t overhead issue.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Ben LaHaise]

On Thu, 5 Jul 2001, David S. Miller wrote:

> And this still leaves the 64-bit dma_addr_t overhead issue.

Huh?  It's a config option, just like blkoff_t.

		-ben

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Jes Sorensen]

>>>>> "David" == David S Miller <davem@redhat.com> writes:

David> Jes Sorensen writes:
>> The dma_mask in struct pci_dev tells you whether you are DAC
>> capable. We pass a pointer to this struct when we call the pci_*
>> functions so the required information needed to make the decision
>> whether to return a SAC or a DAC address is already available.

David> The decision is not based upon "device capable of DAC", that is
David> precisely my point.

I understand that, it's part of the equation.

David> The decision must be based upon a number of considerations.

David> 1) Can it do DAC

David> 2) Is DAC more efficient than SAC on this platform

That sounds to me like a 'static' decision at compile time or at least
something decided upon in the pci_* code as it will be the same for
all devices on the bus. If your IOMMU is very complex to program
compared to the overhead of DAC cycles you pick DAC etc.

David> 3) Does the devices _need_ DAC even if it is slower because it
David> requires referencing large portions of the DMA address space
David> simultaneously

Sure or because the IOMMU is starved. Most devices will do SAC/DAC
based on the address you throw at them, ie. it's perfectly valid to
mix and match DAC and SAC addresses handed to a device.

David> Sure, you could imply all of this complexity in the driver by
David> making them consider all of these issues when setting the mask,
David> but that isn't a nice interface at all.

David> And this still leaves the 64-bit dma_addr_t overhead issue.

The 64 bit dma_addr_t is only an issue on 32 bit architectures with
highmem enabled. I never suggested making dma_addr_t 64 bit on 32 bit
architectures as a general thing.

Jes

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

Jes Sorensen writes:
 > David> And this still leaves the 64-bit dma_addr_t overhead issue.
 > 
 > The 64 bit dma_addr_t is only an issue on 32 bit architectures with
 > highmem enabled. I never suggested making dma_addr_t 64 bit on 32 bit
 > architectures as a general thing.

What about for drivers of SAC-only devices, they eat the overhead
when highmem is enabled too?

This says nothing about the real reason the IA64 solution is
unacceptable, the inputs to the mapping functions which must
be "page+offset+len" triplets as there is no logical "virtual
address" to pass into the mapping routines on 32-bit systems.

Face it, the ia64 stuff is not what we can use across the board.  It
simply doesn't deal with all the necessary issues.  Therefore,
encouraging driver author's to use this ia64 hacked up scheme is
not such a hot idea until we have a real API implemented.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[Ben LaHaise]

On Fri, 6 Jul 2001, David S. Miller wrote:

> What about for drivers of SAC-only devices, they eat the overhead
> when highmem is enabled too?

Yes.  It's not an unreasonable overhead considering that it's configured
out for all the non-highmem kernels that will be shipped.  Keep in mind
that the expected lifespan for 32 bit systems is now less than 3 years, so
elaborate planning that delays implementation buys us nothing more than a
smaller window of usefulness.


> This says nothing about the real reason the IA64 solution is
> unacceptable, the inputs to the mapping functions which must
> be "page+offset+len" triplets as there is no logical "virtual
> address" to pass into the mapping routines on 32-bit systems.

On x86 a 64 bit DMA address cookie is fine.  If you've got concerns, tell
us what you have in mind for a design.


> Face it, the ia64 stuff is not what we can use across the board.  It
> simply doesn't deal with all the necessary issues.  Therefore,
> encouraging driver author's to use this ia64 hacked up scheme is
> not such a hot idea until we have a real API implemented.

So what's the API?

		-ben

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m achi ne

[David S. Miller]

Ben LaHaise writes:
 > Yes.  It's not an unreasonable overhead considering that it's configured
 > out for all the non-highmem kernels that will be shipped.  Keep in mind
 > that the expected lifespan for 32 bit systems is now less than 3 years, so
 > elaborate planning that delays implementation buys us nothing more than a
 > smaller window of usefulness.

Maybe by then only 64-bit cpus will matter.  Who knows.

 > > This says nothing about the real reason the IA64 solution is
 > > unacceptable, the inputs to the mapping functions which must
 > > be "page+offset+len" triplets as there is no logical "virtual
 > > address" to pass into the mapping routines on 32-bit systems.
 > 
 > On x86 a 64 bit DMA address cookie is fine.  If you've got concerns, tell
 > us what you have in mind for a design.

Things along the lines of what Jens Axboe's patches are what I'm
thinking about.

 > So what's the API?

See Jens's patches.

First it has to pass in page/off/len triplets, on all platforms.
This is addressed by Jens's interfaces.

Secondly it has to provide a query mechanism to delineate the
three cases:

1) DAC is faster and always preferred
2) SAC is faster
3) DAC may be slower but more desirable for certain devices
   due to large amounts of parallel address space usage

I have not designed an interface for this, but it ought to be
quite simple.

Thirdly seperate 32-bit/64-bit DMA address types.  Added to the
overhead concerns, I also think it sucks big donkey balls to cast the
things around, especially since different platforms would potentially
require different casts to eliminate the warnings.  In fact, with your
suggested scheme, the setting of highmem would determine a core
type.

In fact, I'm not going to bother to code one single bit of this myself
until I am convinced I have thought the whole problem over properly.
This is the part nobody else wants to do, but it is a prerequisite for
this sort of API.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m

[Alan Cox]

>  > that the expected lifespan for 32 bit systems is now less than 3 years, so
>  > elaborate planning that delays implementation buys us nothing more than a
>  > smaller window of usefulness.
> Maybe by then only 64-bit cpus will matter.  Who knows.

Reality check.

Embedded PCI 32bit processors are going to be very common
People are only now retiring 486's

So add another seven or eight years to your estimate

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m

[Jeff Garzik]

Alan Cox wrote:
> 
> >  > that the expected lifespan for 32 bit systems is now less than 3 years, so
> >  > elaborate planning that delays implementation buys us nothing more than a
> >  > smaller window of usefulness.
> > Maybe by then only 64-bit cpus will matter.  Who knows.
> 
> Reality check.
> 
> Embedded PCI 32bit processors are going to be very common
> People are only now retiring 486's
> 
> So add another seven or eight years to your estimate

Given a little more context, I thought we were talking specifically
about 64bit-PCI-on-32bit-machines?

Assuming that, AFAICS Ben's statement seems more correct.

And IMHO we definitely should not optimize for 64-bit-on-32-bit case. 
Let CONFIG_HIGHMEM grow dma_addr_t to 64-bits, for that case only...

-- 
Jeff Garzik      | A recent study has shown that too much soup
Building 1024    | can cause malaise in laboratory mice.
MandrakeSoft     |

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m

[Alan Cox]

> > So add another seven or eight years to your estimate
> 
> Given a little more context, I thought we were talking specifically
> about 64bit-PCI-on-32bit-machines?
> 
> Assuming that, AFAICS Ben's statement seems more correct.
> 
> And IMHO we definitely should not optimize for 64-bit-on-32-bit case. 
> Let CONFIG_HIGHMEM grow dma_addr_t to 64-bits, for that case only...

I see no good way to optimise for 64bit dma on a 32bit box. The existing API
is extremely clean, easy to understand and I don't agree with Dave's desire
to write another whole concoction.

We do need pci_set_dma_mask_bits(), which in itself lets drivers indicate
SAC/DAC capable. We do want a way for nosey or fine tuned drivers to query
SAC/DAC properties, but most drivers should be letting arch code make arch
policy. 

Alan

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m

[David S. Miller]

Alan Cox writes:
 > I see no good way to optimise for 64bit dma on a 32bit box.

I'm actually not only talking about DAC device on 32-bit cpus.  Just
as much, I'm talking about drivers for SAC-only devices even on 64-bit
cpus.

I took a lot of crap from driver authors when we started pushing the
PCI dma stuff on people, because of the dma_addr_t people now had to
keep around to unmap the thing later.

To a certain extent I agreed with these folks.  I'll be gutting myself
if I make everyone eat twice as much space just to add DAC support to
the kernel :-)

>From yet another perspective, my proposals have also to do with what
API can actually work on all platforms.  This is pretty important to
me.  I remember yesteryear when I used to give myself the privilege
of being self-arch-centric in my work, a Sparc hack here, a Sparc hack
there.  But I simply cannot operate this way anymore.  My conscious
will no longer allow me to crap up things like that :-)

 > I don't agree with Dave's desire to write another whole concoction.

It needs to be a new set of interfaces (and at that point, why not use
a different dma64_addr_t type and save overhead for SAC-only devices
while we're at it :-) because the proper inputs for a DAC mapping
involve page/off/len pairs.

Ignoring addressing limits of 32-bit cpus for a moment, consider that
this page/off/len triplet is the natural currency the kernel uses for
this kind of stuff anyways.

I think it is interesting to note that Jens noticed this immediately,
him being the first person to actually try and implement something
that would work on 32-bit platforms.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m

[Jes Sorensen]

>>>>> "David" == David S Miller <davem@redhat.com> writes:

David> Alan Cox writes:
>> I see no good way to optimise for 64bit dma on a 32bit box.

David> I'm actually not only talking about DAC device on 32-bit cpus.
David> Just as much, I'm talking about drivers for SAC-only devices
David> even on 64-bit cpus.

David> I took a lot of crap from driver authors when we started
David> pushing the PCI dma stuff on people, because of the dma_addr_t
David> people now had to keep around to unmap the thing later.

David> To a certain extent I agreed with these folks.  I'll be gutting
David> myself if I make everyone eat twice as much space just to add
David> DAC support to the kernel :-)

The overhead is going be negligeble, the overhead of highmem itself is
much worse. Not to mention that today some dma_addr_t's might not be
packed properly in data structure hence they ending up taking 8 bytes
anyway.

Jes

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m

[Chris Wedgwood]

On Wed, Jul 11, 2001 at 09:16:50PM +0200, Jes Sorensen wrote:

    The overhead is going be negligeble, the overhead of highmem itself is
    much worse. Not to mention that today some dma_addr_t's might not be
    packed properly in data structure hence they ending up taking 8 bytes
    anyway.

What kind of packing makes a 32-bit value take 8-bytes on any
currently supported archicture? The worst-case I can think of is
7-bytes in the case of misaligned by 3 (e.g. __attribute__((packed))
struct blah { char foo[3]; long bar }; sort of thing).




  --cw

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m

[David S. Miller]

Jes Sorensen writes:
 > The overhead is going be negligeble, the overhead of highmem itself is
 > much worse.

Once Jens's block layer stuff goes in, a lot of that overhead simply
disappears since the page cache need not bounce buffers.  The
networking can already technically cope with this too.

 > Not to mention that today some dma_addr_t's might not be
 > packed properly in data structure hence they ending up taking 8 bytes
 > anyway.

Not on x86 which is the current main benefactor of highmem.

Later,
David S. Miller
davem@redhat.com

Re: (reposting) how to get DMA'able memory within 4GB on 64-bit m

[David S. Miller]

Chris Wedgwood writes:
 > What kind of packing makes a 32-bit value take 8-bytes on any
 > currently supported archicture? The worst-case I can think of is
 > 7-bytes in the case of misaligned by 3 (e.g. __attribute__((packed))
 > struct blah { char foo[3]; long bar }; sort of thing).

If you have this:

struct {
	u32	foo;
	void	*bar;
};

"bar" will be at offset 8 on a 64-bit platforms since it must be
aligned on a 64-byte boundary, so what Jes is saying is that for:

struct {
	dma_addr_t	foo;
	void		*bar;
};

the "dma_addr_t" is already consuming 8 bytes of space on 64-bit
systems.

Later,
David S. Miller
davem@redhat.com