struct page to 36 (or 64) bit bus address?

struct page to 36 (or 64) bit bus address?

[Johannes Erdfelt]

I have a 64 bit PCI card which I'd like to do 64 bit DMA with. I have a
struct page, but I don't see an easy way of determining what the bus
address for that page is.

Is there a way to do it at all?

JE

Re: struct page to 36 (or 64) bit bus address?

[Alan Cox]

> I have a 64 bit PCI card which I'd like to do 64 bit DMA with. I have a
> struct page, but I don't see an easy way of determining what the bus
> address for that page is.
> 
> Is there a way to do it at all?

Yes but its not the right way to do it (bttv does it for example). You want 
to be using the pci or kiovec apis (Documentation/DMA-mapping.txt)

Thats important because it may well be an iommu that handles the mapping to
make the stuff visible - not the cpu

Re: struct page to 36 (or 64) bit bus address?

[Johannes Erdfelt]

On Thu, Aug 09, 2001, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> > I have a 64 bit PCI card which I'd like to do 64 bit DMA with. I have a
> > struct page, but I don't see an easy way of determining what the bus
> > address for that page is.
> > 
> > Is there a way to do it at all?
> 
> Yes but its not the right way to do it (bttv does it for example). You want 
> to be using the pci or kiovec apis (Documentation/DMA-mapping.txt)
> 
> Thats important because it may well be an iommu that handles the mapping to
> make the stuff visible - not the cpu

Unfortunately the PCI DMA API on i386 will only return 32 bit addresses,
which kinda defeats the purpose of what I'm doing.

Obviously the more portable way across architectures is using the PCI
DMA API but when will the implementation be fixed so I can use it to
exploit the full potential of this device?

I'll take a look at the BTTV for a short term solution.

JE

Re: struct page to 36 (or 64) bit bus address?

[Alan Cox]

> Unfortunately the PCI DMA API on i386 will only return 32 bit addresses,
> which kinda defeats the purpose of what I'm doing.

For the short term

> Obviously the more portable way across architectures is using the PCI
> DMA API but when will the implementation be fixed so I can use it to
> exploit the full potential of this device?

2.5 I believe, ping the peacefrog and ask <DaveM@redhat.com>

Re: struct page to 36 (or 64) bit bus address?

[David S. Miller]

Alan Cox writes:
 > > Obviously the more portable way across architectures is using the PCI
 > > DMA API but when will the implementation be fixed so I can use it to
 > > exploit the full potential of this device?
 > 
 > 2.5 I believe, ping the peacefrog and ask <DaveM@redhat.com>

That's the current plan.  There may be a 2.4.x backport, but no
promises.  It all depends upon how straightforward the changes
are.

Note, if you use the "bttv method" (ie. virt_to_bus) your driver will
then fail to compile on several platforms.

Later,
David S. Miller
davem@redhat.com

Re: struct page to 36 (or 64) bit bus address?

[Johannes Erdfelt]

On Thu, Aug 09, 2001, David S. Miller <davem@redhat.com> wrote:
> 
> Alan Cox writes:
>  > > Obviously the more portable way across architectures is using the PCI
>  > > DMA API but when will the implementation be fixed so I can use it to
>  > > exploit the full potential of this device?
>  > 
>  > 2.5 I believe, ping the peacefrog and ask <DaveM@redhat.com>
> 
> That's the current plan.  There may be a 2.4.x backport, but no
> promises.  It all depends upon how straightforward the changes
> are.

It's not a big deal. It's just less efficient which isn't the end of the
world.

> Note, if you use the "bttv method" (ie. virt_to_bus) your driver will
> then fail to compile on several platforms.

So noted. I already have a PCI DMA API version, but I wanted to code up
a "i have an i386 and gigs of memory" version as well.

JE

Re: struct page to 36 (or 64) bit bus address?

[Gerd Knorr]

> > Note, if you use the "bttv method" (ie. virt_to_bus) your driver will
> > then fail to compile on several platforms.
>  
>  So noted. I already have a PCI DMA API version, but I wanted to code up
>  a "i have an i386 and gigs of memory" version as well.

Forgot about virt_to_bus() then, it doesn't work for highmem.

bttv devel versions (0.8.x -> http://bytesex.org/bttv/) use the pci dma
mapping interface.  It can handle DMA highmem pages too, but that
requires a patched kernel as the current 2.4.x simply has no interfaces
to do that.  I'm using the patch below (pulled out of Jens Axboe's bio
patches, i386 only).

  Gerd

---------------------------- cut here ------------------------
--- 2.4.7-pre6/include/asm-i386/page.h.high	Thu Jan  4 23:50:46 2001
+++ 2.4.7-pre6/include/asm-i386/page.h	Wed Jul 18 12:21:48 2001
@@ -116,6 +116,8 @@
 #define __va(x)			((void *)((unsigned long)(x)+PAGE_OFFSET))
 #define virt_to_page(kaddr)	(mem_map + (__pa(kaddr) >> PAGE_SHIFT))
 #define VALID_PAGE(page)	((page - mem_map) < max_mapnr)
+#define page_to_phys(page)	(((page) - mem_map) * PAGE_SIZE)
+#define page_to_bus(page)	page_to_phys((page))
 
 
 #endif /* __KERNEL__ */
--- 2.4.7-pre6/include/asm-i386/pci.h.high	Tue Jul 10 10:57:22 2001
+++ 2.4.7-pre6/include/asm-i386/pci.h	Wed Jul 18 12:22:28 2001
@@ -28,6 +28,7 @@
 
 #include <linux/types.h>
 #include <linux/slab.h>
+#include <linux/highmem.h>
 #include <asm/scatterlist.h>
 #include <linux/string.h>
 #include <asm/io.h>
@@ -84,6 +85,27 @@
 	/* Nothing to do */
 }
 
+/*
+ * pci_{map,unmap}_single_page maps a kernel page to a dma_addr_t. identical
+ * to pci_map_single, but takes a struct page instead of a virtual address
+ */
+extern inline dma_addr_t pci_map_page(struct pci_dev *hwdev, struct page *page,
+				      size_t size, int offset, int direction)
+{
+	if (direction == PCI_DMA_NONE)
+		BUG();
+
+	return (page - mem_map) * PAGE_SIZE + offset;
+}
+
+extern inline void pci_unmap_page(struct pci_dev *hwdev, dma_addr_t dma_address,
+				  size_t size, int direction)
+{
+	if (direction == PCI_DMA_NONE)
+		BUG();
+	/* Nothing to do */
+}
+
 /* Map a set of buffers described by scatterlist in streaming
  * mode for DMA.  This is the scather-gather version of the
  * above pci_map_single interface.  Here the scatter gather list
@@ -119,6 +141,33 @@
 	/* Nothing to do */
 }
 
+/*
+ * meant to replace the pci_map_sg api, new drivers should use this
+ * interface
+ */
+extern inline int pci_map_sgl(struct pci_dev *hwdev, struct sg_list *sg,
+			      int nents, int direction)
+{
+	int i;
+
+	if (direction == PCI_DMA_NONE)
+		BUG();
+
+	for (i = 0; i < nents; i++)
+		sg[i].dma_address = page_to_bus(sg[i].page) + sg[i].offset;
+
+	return nents;
+}
+
+extern inline void pci_unmap_sgl(struct pci_dev *hwdev, struct sg_list *sg,
+				 int nents, int direction)
+{
+	if (direction == PCI_DMA_NONE)
+		BUG();
+	/* Nothing to do */
+}
+
+
 /* Make physical memory consistent for a single
  * streaming mode DMA translation after a transfer.
  *
@@ -178,6 +227,9 @@
  */
 #define sg_dma_address(sg)	(virt_to_bus((sg)->address))
 #define sg_dma_len(sg)		((sg)->length)
+
+#define sgl_dma_address(sg)	((sg)->dma_address)
+#define sgl_dma_len(sg)		((sg)->length)
 
 /* Return the index of the PCI controller for device. */
 static inline int pci_controller_num(struct pci_dev *dev)
--- 2.4.7-pre6/include/asm-i386/scatterlist.h.high	Mon Dec 30 12:01:10 1996
+++ 2.4.7-pre6/include/asm-i386/scatterlist.h	Wed Jul 18 12:18:53 2001
@@ -8,6 +8,29 @@
     unsigned int length;
 };
 
+/*
+ * new style scatter gather list -- move to this completely?
+ */
+#define HAVE_SG_LIST 1
+struct sg_list {
+	/*
+	 * input
+	 */
+	struct page *page;	/* page to do I/O to */
+	unsigned int length;	/* length of I/O */
+	unsigned int offset;	/* offset into page */
+
+	/*
+	 * original page, if bounced
+	 */
+	struct page *bounce_page;
+
+	/*
+	 * output
+	 */
+	dma_addr_t dma_address;	/* mapped address */
+};
+
 #define ISA_DMA_THRESHOLD (0x00ffffff)
 
 #endif /* !(_I386_SCATTERLIST_H) */

Re: struct page to 36 (or 64) bit bus address?

[Johannes Erdfelt]

On Fri, Aug 10, 2001, Gerd Knorr <kraxel@bytesex.org> wrote:
> > > Note, if you use the "bttv method" (ie. virt_to_bus) your driver will
> > > then fail to compile on several platforms.
> >  
> >  So noted. I already have a PCI DMA API version, but I wanted to code up
> >  a "i have an i386 and gigs of memory" version as well.
> 
> Forgot about virt_to_bus() then, it doesn't work for highmem.

I knew that already :)

The thing about 64 bit PCI cards is that there is no such thing as
highmem, so we don't need to worry about mapping it and using bounce
buffers. The device just DMA's. That's why I originally asked.

I took a look at the code in the 2.4.7 version of bttv.c and it wasn't
readily obvious what Alan was referring to. I'll spend some more time
tomorrow looking into it.

JE

Re: struct page to 36 (or 64) bit bus address?

[David S. Miller]

   I'm using the patch below (pulled out of Jens Axboe's bio
   patches, i386 only).

Are you sure you aren't missing anything in this patch?  For example,
one can't use the patch you've provided for 64-bit DMA unless the
dma_addr_t type is made larger.

Later,
David S. Miller
davem@redhat.com

Re: struct page to 36 (or 64) bit bus address?

[Gerd Knorr]

On Fri, Aug 10, 2001 at 02:58:36PM -0700, David S. Miller wrote:
> 
>    I'm using the patch below (pulled out of Jens Axboe's bio
>    patches, i386 only).
> 
> Are you sure you aren't missing anything in this patch?  For example,
> one can't use the patch you've provided for 64-bit DMA unless the
> dma_addr_t type is made larger.

It only allows to use the full 32 bit address range for DMA.  Without
the patch I'm limited to direct mapped memory (i.e. < ~900MB on i386).
With the patch highmem below 4GB can be used too.

Jens's bio patches don't have support for 64-bit DMA (at least not the
version I've used to build that patch).  Because the bt848/878 chips
are 32-bit devices only I havn't even tried to fix that...

  Gerd

-- 
Damn lot people confuse usability and eye-candy.

Re: struct page to 36 (or 64) bit bus address?

[Jens Axboe]

On Fri, Aug 10 2001, David S. Miller wrote:
> 
>    I'm using the patch below (pulled out of Jens Axboe's bio
>    patches, i386 only).
> 
> Are you sure you aren't missing anything in this patch?  For example,
> one can't use the patch you've provided for 64-bit DMA unless the
> dma_addr_t type is made larger.

The dma_addr_high_t was dumped along with pci_map_page_high etc, on your
request in fact :-). So all that is there currently is highmem capable
32-bit DMA stuff.

-- 
Jens Axboe

Re: struct page to 36 (or 64) bit bus address?

[Andrea Arcangeli]

On Thu, Aug 09, 2001 at 04:35:32PM -0400, Johannes Erdfelt wrote:
> It's not a big deal. It's just less efficient which isn't the end of the
> world.

It's not only less efficient, your machine is going to crash as soon as
you ask the iommu to map some giga of ram with the current state of
drivers (the API says that if you get null out of the map call you
should fallback, but no driver checks for this null retval and so in
turn they're all prone to crash, not going to be fixed in 2.4 I guess).
So try to reduce as much as possible the number of simultaneous pci
mappings and it will probably not crash.

Andrea

Re: struct page to 36 (or 64) bit bus address?

[Sandy Harris]

Andrea Arcangeli wrote:

> ... (the API says that if you get null out of the map call you
> should fallback, but no driver checks for this null retval and so in
> turn they're all prone to crash, not going to be fixed in 2.4 I guess).

That strikes me as a pretty basic programming error. Why on Earth is
it not considered a problem urgently needing a fix?

Re: struct page to 36 (or 64) bit bus address?

[David S. Miller]

   From: Sandy Harris <sandy@storm.ca>
   Date: Sat, 11 Aug 2001 12:13:09 -0400

   Andrea Arcangeli wrote:
   
   > ... (the API says that if you get null out of the map call you
   > should fallback, but no driver checks for this null retval and so in
   > turn they're all prone to crash, not going to be fixed in 2.4 I guess).
   
   That strikes me as a pretty basic programming error. Why on Earth is
   it not considered a problem urgently needing a fix?

Andrea has told a white lie, the "API says" that the interfaces may
not fail ever.  That is why there is no failure return value defined
for the PCI DMA interfaces currently.  The DMA-mapping.txt file (which
is the definition of the API) makes no mention of a failure case for
these interfaces.

Andrea has mentioned how we would _like_ the interface to behave.
:-)

There have been a few threads on this issue.  One of the core reasons
the situation is unlikely to change in 2.4.x is that the scsi layer in
it's current form makes the logic required for recovering from a DMA
mapping failure aweful at best.

Later,
David S. Miller
davem@redhat.com

Write Combining (Write Coalescing) on memory mapped I/O on IA64.

[Steffen Persvold]

[-- Attachment #1: Type: text/plain, Size: 1823 bytes --]

Hi,

I have a PCI adapter which does a lot of PIO (sort of network adapter using
host CPU to send data). On IA32 architechtures we could enable Write Combining
to get large burst on the PCI bus, but on IA64 I get at max 8 byte stores (ld8,
st8) at a time.

The driver for this adapter also exports the memory to user-space (with mmap())
and when doing this I do (inside the mmap function) :

int ssci_mmap(struct file *file, struct vm_area_struct *vma)
{
.....
#ifdef __ia64__      
   vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
#endif
   map_page_range(file, vma, vm_start,physaddr,
		      contlen, vma->vm_page_prot);

   remap_page_range(vma->vm_start,
		       phys_addr,
		       vma->vm_end - vma->vm_start,
		       vma->vm_page_prot))
....
}

This way we get Write Combining on IA64, but only for userspace mappings.
Kernel space clients use ioremap() and don't get Write Combining (ioremap on
IA64 use the uncacheable region directly).

To solve this I've tried to make my own __ia64_ioremap() (created
arch/ia64/mm/ioremap.c) wich has a 'flags' input paramenter so that I can set
the _PAGE_MA_WC attribute. I've taken a look on how other architechtures does
this (e.g i386 or s390), and made it nearly identical, but it doesn't seem to
work. The mapping gets set up, but if I try to read or write something to this
address I get a "Unable to handle kernel paging request at virtual address
0xa00000000015c000" which is the address I'm accessing.


I've attached the patch I made. Please let me know if this is doable on IA64 at
all. Feedback greatly appreciated.

Thanks,
-- 
  Steffen Persvold               Systems Engineer
  Email : mailto:sp@scali.no     Scali AS (http://www.scali.com)
  Tlf   : (+47) 22 62 89 50      Olaf Helsets vei 6
  Fax   : (+47) 22 62 89 51      N-0621 Oslo, Norway

[-- Attachment #2: ia64-ioremap.patch --]
[-- Type: text/plain, Size: 5040 bytes --]

--- linux-2.4.4/arch/ia64/mm/Makefile.~1~	Thu Jan  4 21:50:17 2001
+++ linux-2.4.4/arch/ia64/mm/Makefile	Sat Aug 11 21:20:03 2001
@@ -9,6 +9,6 @@
 
 O_TARGET := mm.o
 
-obj-y	 := init.o fault.o tlb.o extable.o
+obj-y	 := init.o fault.o ioremap.o tlb.o extable.o
 
 include $(TOPDIR)/Rules.make
--- linux-2.4.4/arch/ia64/mm/ioremap.c.~1~	Tue May  5 22:32:27 1998
+++ linux-2.4.4/arch/ia64/mm/ioremap.c	Sat Aug 11 23:57:31 2001
@@ -0,0 +1,148 @@
+/*
+ *  arch/s390/mm/ioremap.c
+ *
+ *  IA64 version
+ *    Copyright (C) 2001 Scali AS
+ *    Author(s): Steffen Persvold (sp@scali.com)
+ *
+ *  Derived from "arch/i386/mm/ioremap.c"
+ *    (C) Copyright 1995 1996 Linus Torvalds
+ *
+ * Re-map IO memory to kernel address space so that we can access it.
+ */
+
+#include <linux/vmalloc.h>
+#include <asm/io.h>
+#include <asm/pgalloc.h>
+
+static inline void remap_area_pte(pte_t * pte, unsigned long address, unsigned long size,
+        unsigned long phys_addr, unsigned long flags)
+{
+        unsigned long end;
+
+        address &= ~PMD_MASK;
+        end = address + size;
+        if (end > PMD_SIZE)
+                end = PMD_SIZE;
+	if (address >= end)
+		BUG();
+        do {
+                if (!pte_none(*pte)) {
+                        printk("remap_area_pte: page already exists\n");
+			BUG();
+		}
+                set_pte(pte, mk_pte_phys(phys_addr, __pgprot(_PAGE_A | _PAGE_P | _PAGE_D |
+					_PAGE_AR_RW | _PAGE_PL_0 | flags)));
+                address += PAGE_SIZE;
+                phys_addr += PAGE_SIZE;
+                pte++;
+        } while (address && (address < end));
+}
+
+static inline int remap_area_pmd(pmd_t * pmd, unsigned long address, unsigned long size,
+        unsigned long phys_addr, unsigned long flags)
+{
+	unsigned long end;
+
+	address &= ~PGDIR_MASK;
+	end = address + size;
+	if (end > PGDIR_SIZE)
+		end = PGDIR_SIZE;
+	phys_addr -= address;
+	if (address >= end)
+		BUG();
+	do {
+		pte_t * pte = pte_alloc(&init_mm, pmd, address);
+		if (!pte)
+			return -ENOMEM;
+		remap_area_pte(pte, address, end - address, address + phys_addr, flags);
+		address = (address + PMD_SIZE) & PMD_MASK;
+		pmd++;
+	} while (address && (address < end));
+	return 0;
+}
+
+static int remap_area_pages(unsigned long address, unsigned long phys_addr,
+				 unsigned long size, unsigned long flags)
+{
+	int error;
+	pgd_t * dir;
+	unsigned long end = address + size;
+
+	phys_addr -= address;
+	dir = pgd_offset(&init_mm, address);
+	flush_cache_all();
+	if (address >= end)
+		BUG();
+	spin_lock(&init_mm.page_table_lock);
+	do {
+		pmd_t *pmd;
+		pmd = pmd_alloc(&init_mm, dir, address);
+		error = -ENOMEM;
+		if (!pmd)
+			break;
+		if (remap_area_pmd(pmd, address, end - address,
+					 phys_addr + address, flags))
+			break;
+		error = 0;
+		address = (address + PGDIR_SIZE) & PGDIR_MASK;
+		dir++;
+	} while (address && (address < end));
+	spin_unlock(&init_mm.page_table_lock);
+	flush_tlb_all();
+	return error;
+}
+
+/*
+ * Generic mapping function (not visible outside):
+ */
+
+/*
+ * Remap an arbitrary physical address space into the kernel virtual
+ * address space. Needed when the kernel wants to access high addresses
+ * directly.
+ *
+ * NOTE! We need to allow non-page-aligned mappings too: we will obviously
+ * have to convert them into an offset in a page-aligned mapping, but the
+ * caller shouldn't need to know that small detail.
+ */
+void * __ia64_ioremap(unsigned long phys_addr, unsigned long size, unsigned long flags)
+{
+	void * addr;
+	struct vm_struct * area;
+	unsigned long offset, last_addr;
+
+	/* Don't allow wraparound or zero size */
+	last_addr = phys_addr + size - 1;
+	if (!size || last_addr < phys_addr)
+		return NULL;
+
+	/*
+	 * Mappings have to be page-aligned
+	 */
+	offset = phys_addr & ~PAGE_MASK;
+	phys_addr &= PAGE_MASK;
+	size = PAGE_ALIGN(last_addr) - phys_addr;
+
+	/*
+	 * Ok, go for it..
+	 */
+	area = get_vm_area(size, VM_IOREMAP);
+	if (!area)
+		return NULL;
+	addr = area->addr;
+
+	if (flags == 0)
+	   flags = _PAGE_MA_UC;
+		
+	if (remap_area_pages(VMALLOC_VMADDR(addr), phys_addr, size, flags)) {
+		vfree(addr);
+		return NULL;
+	}
+	return (void *) (offset + (char *)addr);
+}
+
+void __ia64_iounmap(void *addr)
+{
+	return vfree((void *) (PAGE_MASK & (unsigned long) addr));
+}
--- linux-2.4.4/arch/ia64/kernel/ia64_ksyms.c.~1~	Thu Apr  5 21:51:47 2001
+++ linux-2.4.4/arch/ia64/kernel/ia64_ksyms.c	Sat Aug 11 23:57:12 2001
@@ -39,6 +39,8 @@
 EXPORT_SYMBOL(ip_fast_csum);
 
 #include <asm/io.h>
+EXPORT_SYMBOL(__ia64_ioremap);
+EXPORT_SYMBOL(__ia64_iounmap);
 EXPORT_SYMBOL(__ia64_memcpy_fromio);
 EXPORT_SYMBOL(__ia64_memcpy_toio);
 EXPORT_SYMBOL(__ia64_memset_c_io);
--- linux-2.4.4/include/asm-ia64/io.h.~1~	Thu Apr  5 21:51:47 2001
+++ linux-2.4.4/include/asm-ia64/io.h	Sat Aug 11 23:59:36 2001
@@ -386,6 +386,9 @@
 
 #define ioremap_nocache(o,s)	ioremap(o,s)
 
+extern void * __ia64_ioremap(unsigned long offset, unsigned long size, unsigned long flags);
+extern void __ia64_iounmap(void *addr);
+
 # ifdef __KERNEL__
 
 /*

Re: struct page to 36 (or 64) bit bus address?

[David S. Miller]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: Text/Plain; charset=big5, Size: 812 bytes --]

   From: Gérard Roudier <groudier@free.fr>
   Date: Mon, 13 Aug 2001 21:07:50 +0200 (CEST)

   That's the major problem if we ever want to preserve some ordering in the
   queuing of SCSI IOs.

When DMA mapping operation fails, you simply "stop queueing".  Queue
freezes and nothing new is executed.

DMA wakeup makes you start where you left off.  I cannot see any
ordering constraints violated by this as a side effect.  It is like a
"cork" for running scsi commands in the driver.

The purpose of the hypothetical kernel thread is to get out
of interrupt context if that is deemed necessary.

It may not be.

Later,
David S. Miller
davem@redhat.com
ý:.žË›±Êâmçë¢kaŠÉb²ßìzwm…ébïîžË›±Êâmébžìÿ‘êçz_âžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣømšSåy«\x1e­æ¶\x17…\x01\x06­†ÛiÿÿðÃ\x0fí»\x1fè®\x0få’i\x7f

Re: struct page to 36 (or 64) bit bus address?

[Gérard Roudier]

On Mon, 13 Aug 2001, David S. Miller wrote:

>    From: Alan Cox <alan@lxorguk.ukuu.org.uk>
>    Date: Mon, 13 Aug 2001 15:09:31 +0100 (BST)
>
>    > To make sure we're on the same wave length, are you suggesting
>    > this is the kind of thing we'd call in a callback from the PCI
>    > DMA support layer?
>
>    Well that would be an ugly layer violation, but how about
>
>    	scsi_retry_command_waitq(SCpnt, &dma_waitq)
>
>    ?
>
> I don't mean "PCI DMA support layer knows scsi routines to
> call", rather I mean:
>
> ... register_new_scsi_host() ...
>
> 	add_notifier(&pci_dma_freespace_notifier, &host->dma_notifier_block);
>
> And net drivers would do something similar, registering something
> that will do "netdev_wake_queue();" etc.
>
> Also, the DMA support layer must either:
>
> 1) require users of it to provide a kernel thread in which
>    to execute these things in the correct context
>
> 2) have a kernel thread of it's own to do this
>
> 3) or somehow else be able to accept this notification
>    from any kind of execution context
>
> The notifier is capable of happening anywhere, anytime.
> Right?

That's the major problem if we ever want to preserve some ordering in the
queuing of SCSI IOs. It is not clear to me why a thread may help here. A
thread may help if you actually want to thread the queuing and/or if you
want to wait on resource. Note that using a thread for the queuing might
be a solution for preserving queuing ordering, but it is not the sole
solution.

Defunct T10/CAM addresses ordering problems especially during error
recovery by a freeze/unfreeze mechanism of device queues handled by SIMs
(SIMs are low-level drivers under Linux). Using threads, call-backs or any
other evil things on top of this mechanism can be a matter of O/S
dependent issues, but it is not the solution of the problem.

> Later,
> David S. Miller
> davem@redhat.com

Later,
Gérard P. Roudier
groudier@free.fr

:)

Re: struct page to 36 (or 64) bit bus address?

[Alan Cox]

> And net drivers would do something similar, registering something
> that will do "netdev_wake_queue();" etc.

I don't want a callback, or if you give me a callback for scsi I'll just
turn it into a callback to wake_up(). Thats all I happen to need. net
drivers might be different

Alan

Re: struct page to 36 (or 64) bit bus address?

[David S. Miller]

   From: Alan Cox <alan@lxorguk.ukuu.org.uk>
   Date: Mon, 13 Aug 2001 15:09:31 +0100 (BST)

   > To make sure we're on the same wave length, are you suggesting
   > this is the kind of thing we'd call in a callback from the PCI
   > DMA support layer?
   
   Well that would be an ugly layer violation, but how about
   
   	scsi_retry_command_waitq(SCpnt, &dma_waitq)
   
   ?

I don't mean "PCI DMA support layer knows scsi routines to
call", rather I mean:

... register_new_scsi_host() ...

	add_notifier(&pci_dma_freespace_notifier, &host->dma_notifier_block);

And net drivers would do something similar, registering something
that will do "netdev_wake_queue();" etc.

Also, the DMA support layer must either:

1) require users of it to provide a kernel thread in which
   to execute these things in the correct context

2) have a kernel thread of it's own to do this

3) or somehow else be able to accept this notification
   from any kind of execution context

The notifier is capable of happening anywhere, anytime.
Right?

Later,
David S. Miller
davem@redhat.com

Re: struct page to 36 (or 64) bit bus address?

[Alan Cox]

>    Actually its pretty trivial to resolve that I think. We already follow
>    precisely the right procedure for some other scsi events. So 
>    
>    	scsi_retry_command(SCpnt);
>    
>    would map to setting the error and returning.
> 
> To make sure we're on the same wave length, are you suggesting
> this is the kind of thing we'd call in a callback from the PCI
> DMA support layer?

Well that would be an ugly layer violation, but how about

	scsi_retry_command_waitq(SCpnt, &dma_waitq)

?

Re: struct page to 36 (or 64) bit bus address?

[David S. Miller]

   From: Alan Cox <alan@lxorguk.ukuu.org.uk>
   Date: Sun, 12 Aug 2001 12:49:00 +0100 (BST)

   Actually its pretty trivial to resolve that I think. We already follow
   precisely the right procedure for some other scsi events. So 
   
   	scsi_retry_command(SCpnt);
   
   would map to setting the error and returning.

To make sure we're on the same wave length, are you suggesting
this is the kind of thing we'd call in a callback from the PCI
DMA support layer?

That's the core bit, some callback mechanism.  You need this to let
the drivers get "jump started" when IOMMU mappings become available
once more.

Later,
David S. Miller
davem@redhat.com

Re: struct page to 36 (or 64) bit bus address?

[Alan Cox]

> There have been a few threads on this issue.  One of the core reasons
> the situation is unlikely to change in 2.4.x is that the scsi layer in
> it's current form makes the logic required for recovering from a DMA
> mapping failure aweful at best.

Actually its pretty trivial to resolve that I think. We already follow
precisely the right procedure for some other scsi events. So 

	scsi_retry_command(SCpnt);

would map to setting the error and returning.