* [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility
@ 2011-07-22 13:08 Ian Campbell
2011-07-22 13:17 ` [PATCH 01/13] mm: Make some struct page's const Ian Campbell
` (13 more replies)
0 siblings, 14 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:08 UTC (permalink / raw)
To: netdev, linux-nfs
Hi,
This is v2 of my series to enable visibility into SKB paged fragment's
lifecycles, v1 is at [0] and contains some more background and rationale
but basically the series allows entities which inject pages into the
networking stack to receive a notification when the stack has really
finished with those pages (i.e. including retransmissions, clones,
pull-ups etc) and not just when the original skb is finished with, which
is beneficial to many subsystems which wish to inject pages into the
network stack without giving up full ownership of those page's
lifecycle. It implements something broadly along the lines of what was
described in [1].
I have updated based on the feedback from last time. In particular:
* Added destructor directly to sendpage() protocol hooks instead
of inventing sendpage_destructor() (David Miller)
* Dropped skb_frag_pci_map in favour of skb_frag_dma_map on Michał
Mirosław's advice
* Pushed the NFS fix down into the RPC layer. (Trond)
I also split out the patches to make protocols use the new interface
into per-protocol patches. I held back on splitting up the patch to
drivers/* (since that will cause an explosion in the series length) --
I'll do this split for v3.
FYI I'm travelling for the next two weeks, although I expect to have
good access to mail next week I'm less sure about the following week.
Cheers,
Ian.
[0] http://marc.info/?l=linux-netdev&m=131072801125521&w=2
[1] http://marc.info/?l=linux-netdev&m=130925719513084&w=2
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 01/13] mm: Make some struct page's const.
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 02/13] mm: use const struct page for r/o page-flag accessor methods Ian Campbell
` (12 subsequent siblings)
13 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, Andrew Morton, Rik van Riel, Andrea Arcangeli,
Mel Gorman, Michel Lespinasse, linux-mm, linux-kernel
These uses are read-only and in a subsequent patch I have a const struct page
in my hand...
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: Michel Lespinasse <walken@google.com>
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
---
include/linux/mm.h | 10 +++++-----
mm/sparse.c | 2 +-
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 9670f71..550ec8f 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -636,7 +636,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
#define SECTIONS_MASK ((1UL << SECTIONS_WIDTH) - 1)
#define ZONEID_MASK ((1UL << ZONEID_SHIFT) - 1)
-static inline enum zone_type page_zonenum(struct page *page)
+static inline enum zone_type page_zonenum(const struct page *page)
{
return (page->flags >> ZONES_PGSHIFT) & ZONES_MASK;
}
@@ -664,15 +664,15 @@ static inline int zone_to_nid(struct zone *zone)
}
#ifdef NODE_NOT_IN_PAGE_FLAGS
-extern int page_to_nid(struct page *page);
+extern int page_to_nid(const struct page *page);
#else
-static inline int page_to_nid(struct page *page)
+static inline int page_to_nid(const struct page *page)
{
return (page->flags >> NODES_PGSHIFT) & NODES_MASK;
}
#endif
-static inline struct zone *page_zone(struct page *page)
+static inline struct zone *page_zone(const struct page *page)
{
return &NODE_DATA(page_to_nid(page))->node_zones[page_zonenum(page)];
}
@@ -717,7 +717,7 @@ static inline void set_page_links(struct page *page, enum zone_type zone,
*/
#include <linux/vmstat.h>
-static __always_inline void *lowmem_page_address(struct page *page)
+static __always_inline void *lowmem_page_address(const struct page *page)
{
return __va(PFN_PHYS(page_to_pfn(page)));
}
diff --git a/mm/sparse.c b/mm/sparse.c
index aa64b12..858e1df 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -40,7 +40,7 @@ static u8 section_to_node_table[NR_MEM_SECTIONS] __cacheline_aligned;
static u16 section_to_node_table[NR_MEM_SECTIONS] __cacheline_aligned;
#endif
-int page_to_nid(struct page *page)
+int page_to_nid(const struct page *page)
{
return section_to_node_table[page_to_section(page)];
}
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 02/13] mm: use const struct page for r/o page-flag accessor methods
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
2011-07-22 13:17 ` [PATCH 01/13] mm: Make some struct page's const Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 03/13] net: add APIs for manipulating skb page fragments Ian Campbell
` (11 subsequent siblings)
13 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, Andrew Morton, Andrea Arcangeli, Rik van Riel,
Martin Schwidefsky, Michel Lespinasse, linux-kernel
In a subsquent patch I have a const struct page in my hand...
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Michel Lespinasse <walken@google.com>
Cc: linux-kernel@vger.kernel.org
---
include/linux/page-flags.h | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 6081493..7d632cc 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -135,7 +135,7 @@ enum pageflags {
* Macros to create function definitions for page flags
*/
#define TESTPAGEFLAG(uname, lname) \
-static inline int Page##uname(struct page *page) \
+static inline int Page##uname(const struct page *page) \
{ return test_bit(PG_##lname, &page->flags); }
#define SETPAGEFLAG(uname, lname) \
@@ -173,7 +173,7 @@ static inline int __TestClearPage##uname(struct page *page) \
__SETPAGEFLAG(uname, lname) __CLEARPAGEFLAG(uname, lname)
#define PAGEFLAG_FALSE(uname) \
-static inline int Page##uname(struct page *page) \
+static inline int Page##uname(const struct page *page) \
{ return 0; }
#define TESTSCFLAG(uname, lname) \
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 03/13] net: add APIs for manipulating skb page fragments.
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
2011-07-22 13:17 ` [PATCH 01/13] mm: Make some struct page's const Ian Campbell
2011-07-22 13:17 ` [PATCH 02/13] mm: use const struct page for r/o page-flag accessor methods Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 04/13] net: convert core to skb paged frag APIs Ian Campbell
` (10 subsequent siblings)
13 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, David S. Miller, Eric Dumazet, Michał Mirosław
The primary aim is to add skb_frag_(ref|unref) in order to remove the use of
bare get/put_page on SKB pages fragments and to isolate users from subsequent
changes to the skb_frag_t data structure.
The API also includes an accessor for the struct page itself. The default
variant of this returns a *const* struct page in an attempt to catch bare uses
of get/put_page (which take a non-const struct page).
Also included are helper APIs for passing a paged fragment to kmap and
dma_map_page since I was seeing the same pattern a lot. A helper for
pci_map_page is ommitted due to Michał Mirosław's recommendation that users
should transition to pci_map_page instead.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: "Michał Mirosław" <mirq-linux@rere.qmqm.pl>
Cc: netdev@vger.kernel.org
---
include/linux/skbuff.h | 204 +++++++++++++++++++++++++++++++++++++++++++++++-
1 files changed, 202 insertions(+), 2 deletions(-)
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index c0a4f3a..f4034af 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -29,6 +29,8 @@
#include <linux/rcupdate.h>
#include <linux/dmaengine.h>
#include <linux/hrtimer.h>
+#include <linux/highmem.h>
+#include <linux/pci.h>
/* Don't change this without changing skb_csum_unnecessary! */
#define CHECKSUM_NONE 0
@@ -1109,14 +1111,47 @@ static inline int skb_pagelen(const struct sk_buff *skb)
return len + skb_headlen(skb);
}
-static inline void skb_fill_page_desc(struct sk_buff *skb, int i,
- struct page *page, int off, int size)
+/**
+ * __skb_fill_page_desc - initialise a paged fragment in an skb
+ * @skb: buffer containing fragment to be initialised
+ * @i: paged fragment index to initialise
+ * @page: the page to use for this fragment
+ * @off: the offset to the data with @page
+ * @size: the length of the data
+ *
+ * Initialises the @i'th fragment of @skb to point to &size bytes at
+ * offset @off within @page.
+ *
+ * Does not take any additional reference on the fragment.
+ */
+static inline void __skb_fill_page_desc(struct sk_buff *skb, int i,
+ struct page *page, int off, int size)
{
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
frag->page = page;
frag->page_offset = off;
frag->size = size;
+}
+
+/**
+ * skb_fill_page_desc - initialise a paged fragment in an skb
+ * @skb: buffer containing fragment to be initialised
+ * @i: paged fragment index to initialise
+ * @page: the page to use for this fragment
+ * @off: the offset to the data with @page
+ * @size: the length of the data
+ *
+ * As per __skb_fill_page_desc() -- initialises the @i'th fragment of
+ * @skb to point to &size bytes at offset @off within @page. In
+ * addition updates @skb such that @i is the last fragment.
+ *
+ * Does not take any additional reference on the fragment.
+ */
+static inline void skb_fill_page_desc(struct sk_buff *skb, int i,
+ struct page *page, int off, int size)
+{
+ __skb_fill_page_desc(skb, i, page, off, size);
skb_shinfo(skb)->nr_frags = i + 1;
}
@@ -1605,6 +1640,171 @@ static inline void netdev_free_page(struct net_device *dev, struct page *page)
}
/**
+ * __skb_frag_page - retrieve the page refered to by a paged fragment
+ * @frag: the paged fragment
+ *
+ * Returns the &struct page associated with @frag. Where possible you
+ * should use skb_frag_page() which returns a const &struct page.
+ */
+static inline struct page *__skb_frag_page(const skb_frag_t *frag)
+{
+ return frag->page;
+}
+
+/**
+ * __skb_frag_page - retrieve the page refered to by a paged fragment
+ * @frag: the paged fragment
+ *
+ * Returns the &struct page associated with @frag as a const.
+ */
+static inline const struct page *skb_frag_page(const skb_frag_t *frag)
+{
+ return frag->page;
+}
+
+/**
+ * __skb_frag_ref - take an addition reference on a paged fragment.
+ * @frag: the paged fragment
+ *
+ * Takes an additional reference on the paged fragment @frag.
+ */
+static inline void __skb_frag_ref(skb_frag_t *frag)
+{
+ get_page(__skb_frag_page(frag));
+}
+
+/**
+ * skb_frag_ref - take an addition reference on a paged fragment of an skb.
+ * @skb: the buffer
+ * @f: the fragment offset.
+ *
+ * Takes an additional reference on the @f'th paged fragment of @skb.
+ */
+static inline void skb_frag_ref(struct sk_buff *skb, int f)
+{
+ __skb_frag_ref(&skb_shinfo(skb)->frags[f]);
+}
+
+/**
+ * __skb_frag_unref - release a reference on a paged fragment.
+ * @frag: the paged fragment
+ *
+ * Releases a reference on the paged fragment @frag.
+ */
+static inline void __skb_frag_unref(skb_frag_t *frag)
+{
+ put_page(__skb_frag_page(frag));
+}
+
+/**
+ * skb_frag_unref - release a reference on a paged fragment of an skb.
+ * @skb: the buffer
+ * @f: the fragment offset
+ *
+ * Releases a reference on the @f'th paged fragment of @skb.
+ */
+static inline void skb_frag_unref(struct sk_buff *skb, int f)
+{
+ __skb_frag_unref(&skb_shinfo(skb)->frags[f]);
+}
+
+/**
+ * skb_frag_address - gets the address of the data contained in a paged fragment
+ * @frag: the paged fragment buffer
+ *
+ * Returns the address of the data within @frag. The page must already
+ * be mapped.
+ */
+static inline void *skb_frag_address(const skb_frag_t *frag)
+{
+ return page_address(skb_frag_page(frag)) + frag->page_offset;
+}
+
+/**
+ * skb_frag_address_safe - gets the address of the data contained in a paged fragment
+ * @frag: the paged fragment buffer
+ *
+ * Returns the address of the data within @frag. Checks that the page
+ * is mapped and returns %NULL otherwise.
+ */
+static inline void *skb_frag_address_safe(const skb_frag_t *frag)
+{
+ void *ptr = page_address(skb_frag_page(frag));
+ if (unlikely(!ptr))
+ return NULL;
+
+ return ptr + frag->page_offset;
+}
+
+/**
+ * __skb_frag_set_page - sets the page contained in a paged fragment
+ * @frag: the paged fragment
+ * @page: the page to set
+ *
+ * Sets the fragment @frag to contain @page.
+ */
+static inline void __skb_frag_set_page(skb_frag_t *frag, struct page *page)
+{
+ frag->page = page;
+ __skb_frag_ref(frag);
+}
+
+/**
+ * skb_frag_set_page - sets the page contained in a paged fragment of an skb
+ * @skb: the buffer
+ * @f: the fragment offset
+ * @page: the page to set
+ *
+ * Sets the @f'th fragment of @skb to contain @page.
+ */
+static inline void skb_frag_set_page(struct sk_buff *skb, int f,
+ struct page *page)
+{
+ __skb_frag_set_page(&skb_shinfo(skb)->frags[f], page);
+}
+
+/**
+ * skb_frag_kmap - kmaps a paged fragment
+ * @frag: the paged fragment
+ *
+ * kmap()s the paged fragment @frag and returns the virtual address.
+ */
+static inline void *skb_frag_kmap(skb_frag_t *frag)
+{
+ return kmap(__skb_frag_page(frag));
+}
+
+/**
+ * skb_frag_kmap - kunmaps a paged fragment
+ * @frag: the paged fragment
+ *
+ * kunmap()s the paged fragment @frag.
+ */
+static inline void skb_frag_kunmap(skb_frag_t *frag)
+{
+ kunmap(__skb_frag_page(frag));
+}
+
+/**
+ * skb_frag_dma_map - maps a paged fragment via the DMA API
+ * @device: the device to map the fragment to
+ * @frag: the paged fragment to map
+ * @offset: the offset within the fragment (starting at the fragments own offset)
+ * @size: the number of bytes to map
+ * @direction: the direction of the mapping (%PCI_DMA_*)
+ *
+ * Maps the page associated with @frag to @device.
+ */
+static inline dma_addr_t skb_frag_dma_map(struct device *dev,
+ const skb_frag_t *frag,
+ size_t offset, size_t size,
+ enum dma_data_direction dir)
+{
+ return dma_map_page(dev, __skb_frag_page(frag),
+ frag->page_offset + offset, size, dir);
+}
+
+/**
* skb_clone_writable - is the header of a clone writable
* @skb: buffer to check
* @len: length up to which to write
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 04/13] net: convert core to skb paged frag APIs
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (2 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 03/13] net: add APIs for manipulating skb page fragments Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 05/13] net: ipv4: convert to SKB " Ian Campbell
` (9 subsequent siblings)
13 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, David S. Miller, Eric Dumazet, Michał Mirosław
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: "Michał Mirosław" <mirq-linux@rere.qmqm.pl>
Cc: netdev@vger.kernel.org
---
include/linux/skbuff.h | 4 ++--
net/core/datagram.c | 20 ++++++++------------
net/core/dev.c | 7 +++----
net/core/kmap_skb.h | 2 +-
net/core/pktgen.c | 3 +--
net/core/skbuff.c | 31 +++++++++++++++++--------------
net/core/sock.c | 12 +++++-------
net/core/user_dma.c | 2 +-
8 files changed, 38 insertions(+), 43 deletions(-)
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index f4034af..bc6bd24 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -1906,12 +1906,12 @@ static inline int skb_add_data(struct sk_buff *skb,
}
static inline int skb_can_coalesce(struct sk_buff *skb, int i,
- struct page *page, int off)
+ const struct page *page, int off)
{
if (i) {
struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[i - 1];
- return page == frag->page &&
+ return page == skb_frag_page(frag) &&
off == frag->page_offset + frag->size;
}
return 0;
diff --git a/net/core/datagram.c b/net/core/datagram.c
index 18ac112..f0dcaa2 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -332,14 +332,13 @@ int skb_copy_datagram_iovec(const struct sk_buff *skb, int offset,
int err;
u8 *vaddr;
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- struct page *page = frag->page;
if (copy > len)
copy = len;
- vaddr = kmap(page);
+ vaddr = skb_frag_kmap(frag);
err = memcpy_toiovec(to, vaddr + frag->page_offset +
offset - start, copy);
- kunmap(page);
+ skb_frag_kunmap(frag);
if (err)
goto fault;
if (!(len -= copy))
@@ -418,14 +417,13 @@ int skb_copy_datagram_const_iovec(const struct sk_buff *skb, int offset,
int err;
u8 *vaddr;
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- struct page *page = frag->page;
if (copy > len)
copy = len;
- vaddr = kmap(page);
+ vaddr = skb_frag_kmap(frag);
err = memcpy_toiovecend(to, vaddr + frag->page_offset +
offset - start, to_offset, copy);
- kunmap(page);
+ skb_frag_kunmap(frag);
if (err)
goto fault;
if (!(len -= copy))
@@ -508,15 +506,14 @@ int skb_copy_datagram_from_iovec(struct sk_buff *skb, int offset,
int err;
u8 *vaddr;
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- struct page *page = frag->page;
if (copy > len)
copy = len;
- vaddr = kmap(page);
+ vaddr = skb_frag_kmap(frag);
err = memcpy_fromiovecend(vaddr + frag->page_offset +
offset - start,
from, from_offset, copy);
- kunmap(page);
+ skb_frag_kunmap(frag);
if (err)
goto fault;
@@ -594,16 +591,15 @@ static int skb_copy_and_csum_datagram(const struct sk_buff *skb, int offset,
int err = 0;
u8 *vaddr;
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- struct page *page = frag->page;
if (copy > len)
copy = len;
- vaddr = kmap(page);
+ vaddr = skb_frag_kmap(frag);
csum2 = csum_and_copy_to_user(vaddr +
frag->page_offset +
offset - start,
to, copy, 0, &err);
- kunmap(page);
+ skb_frag_kunmap(frag);
if (err)
goto fault;
*csump = csum_block_add(*csump, csum2, pos);
diff --git a/net/core/dev.c b/net/core/dev.c
index 9c58c1e..9ab39c0 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3414,7 +3414,7 @@ pull:
skb_shinfo(skb)->frags[0].size -= grow;
if (unlikely(!skb_shinfo(skb)->frags[0].size)) {
- put_page(skb_shinfo(skb)->frags[0].page);
+ skb_frag_unref(skb, 0);
memmove(skb_shinfo(skb)->frags,
skb_shinfo(skb)->frags + 1,
--skb_shinfo(skb)->nr_frags * sizeof(skb_frag_t));
@@ -3478,10 +3478,9 @@ void skb_gro_reset_offset(struct sk_buff *skb)
NAPI_GRO_CB(skb)->frag0_len = 0;
if (skb->mac_header == skb->tail &&
- !PageHighMem(skb_shinfo(skb)->frags[0].page)) {
+ !PageHighMem(skb_frag_page(&skb_shinfo(skb)->frags[0]))) {
NAPI_GRO_CB(skb)->frag0 =
- page_address(skb_shinfo(skb)->frags[0].page) +
- skb_shinfo(skb)->frags[0].page_offset;
+ skb_frag_address(&skb_shinfo(skb)->frags[0]);
NAPI_GRO_CB(skb)->frag0_len = skb_shinfo(skb)->frags[0].size;
}
}
diff --git a/net/core/kmap_skb.h b/net/core/kmap_skb.h
index 283c2b9..b1e9711 100644
--- a/net/core/kmap_skb.h
+++ b/net/core/kmap_skb.h
@@ -7,7 +7,7 @@ static inline void *kmap_skb_frag(const skb_frag_t *frag)
local_bh_disable();
#endif
- return kmap_atomic(frag->page, KM_SKB_DATA_SOFTIRQ);
+ return kmap_atomic(__skb_frag_page(frag), KM_SKB_DATA_SOFTIRQ);
}
static inline void kunmap_skb_frag(void *vaddr)
diff --git a/net/core/pktgen.c b/net/core/pktgen.c
index f76079c..989b2b6 100644
--- a/net/core/pktgen.c
+++ b/net/core/pktgen.c
@@ -2600,8 +2600,7 @@ static void pktgen_finalize_skb(struct pktgen_dev *pkt_dev, struct sk_buff *skb,
if (!pkt_dev->page)
break;
}
- skb_shinfo(skb)->frags[i].page = pkt_dev->page;
- get_page(pkt_dev->page);
+ skb_frag_set_page(skb, i, pkt_dev->page);
skb_shinfo(skb)->frags[i].page_offset = 0;
/*last fragment, fill rest of data*/
if (i == (frags - 1))
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..2133600 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -326,7 +326,7 @@ static void skb_release_data(struct sk_buff *skb)
if (skb_shinfo(skb)->nr_frags) {
int i;
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++)
- put_page(skb_shinfo(skb)->frags[i].page);
+ skb_frag_unref(skb, i);
}
if (skb_has_frag_list(skb))
@@ -733,7 +733,7 @@ struct sk_buff *pskb_copy(struct sk_buff *skb, gfp_t gfp_mask)
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
skb_shinfo(n)->frags[i] = skb_shinfo(skb)->frags[i];
- get_page(skb_shinfo(n)->frags[i].page);
+ skb_frag_ref(skb, i);
}
skb_shinfo(n)->nr_frags = i;
}
@@ -820,7 +820,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
kfree(skb->head);
} else {
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++)
- get_page(skb_shinfo(skb)->frags[i].page);
+ skb_frag_ref(skb, i);
if (skb_has_frag_list(skb))
skb_clone_fraglist(skb);
@@ -1098,7 +1098,7 @@ drop_pages:
skb_shinfo(skb)->nr_frags = i;
for (; i < nfrags; i++)
- put_page(skb_shinfo(skb)->frags[i].page);
+ skb_frag_unref(skb, i);
if (skb_has_frag_list(skb))
skb_drop_fraglist(skb);
@@ -1267,7 +1267,7 @@ pull_pages:
k = 0;
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
if (skb_shinfo(skb)->frags[i].size <= eat) {
- put_page(skb_shinfo(skb)->frags[i].page);
+ skb_frag_unref(skb, i);
eat -= skb_shinfo(skb)->frags[i].size;
} else {
skb_shinfo(skb)->frags[k] = skb_shinfo(skb)->frags[i];
@@ -1512,7 +1512,9 @@ static int __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe,
for (seg = 0; seg < skb_shinfo(skb)->nr_frags; seg++) {
const skb_frag_t *f = &skb_shinfo(skb)->frags[seg];
- if (__splice_segment(f->page, f->page_offset, f->size,
+ /* XXX */
+ if (__splice_segment(__skb_frag_page(f),
+ f->page_offset, f->size,
offset, len, skb, spd, 0, sk, pipe))
return 1;
}
@@ -2057,7 +2059,7 @@ static inline void skb_split_no_header(struct sk_buff *skb,
* where splitting is expensive.
* 2. Split is accurately. We make this.
*/
- get_page(skb_shinfo(skb)->frags[i].page);
+ skb_frag_ref(skb, i);
skb_shinfo(skb1)->frags[0].page_offset += len - pos;
skb_shinfo(skb1)->frags[0].size -= len - pos;
skb_shinfo(skb)->frags[i].size = len - pos;
@@ -2132,7 +2134,8 @@ int skb_shift(struct sk_buff *tgt, struct sk_buff *skb, int shiftlen)
* commit all, so that we don't have to undo partial changes
*/
if (!to ||
- !skb_can_coalesce(tgt, to, fragfrom->page, fragfrom->page_offset)) {
+ !skb_can_coalesce(tgt, to, skb_frag_page(fragfrom),
+ fragfrom->page_offset)) {
merge = -1;
} else {
merge = to - 1;
@@ -2179,7 +2182,7 @@ int skb_shift(struct sk_buff *tgt, struct sk_buff *skb, int shiftlen)
to++;
} else {
- get_page(fragfrom->page);
+ __skb_frag_ref(fragfrom);
fragto->page = fragfrom->page;
fragto->page_offset = fragfrom->page_offset;
fragto->size = todo;
@@ -2201,7 +2204,7 @@ int skb_shift(struct sk_buff *tgt, struct sk_buff *skb, int shiftlen)
fragto = &skb_shinfo(tgt)->frags[merge];
fragto->size += fragfrom->size;
- put_page(fragfrom->page);
+ __skb_frag_unref(fragfrom);
}
/* Reposition in the original skb */
@@ -2446,8 +2449,7 @@ int skb_append_datato_frags(struct sock *sk, struct sk_buff *skb,
left = PAGE_SIZE - frag->page_offset;
copy = (length > left)? left : length;
- ret = getfrag(from, (page_address(frag->page) +
- frag->page_offset + frag->size),
+ ret = getfrag(from, skb_frag_address(frag) + frag->size,
offset, copy, 0, skb);
if (ret < 0)
return -EFAULT;
@@ -2599,7 +2601,7 @@ struct sk_buff *skb_segment(struct sk_buff *skb, u32 features)
while (pos < offset + len && i < nfrags) {
*frag = skb_shinfo(skb)->frags[i];
- get_page(frag->page);
+ __skb_frag_ref(frag);
size = frag->size;
if (pos < offset) {
@@ -2822,7 +2824,8 @@ __skb_to_sgvec(struct sk_buff *skb, struct scatterlist *sg, int offset, int len)
if (copy > len)
copy = len;
- sg_set_page(&sg[elt], frag->page, copy,
+ /* XXX */
+ sg_set_page(&sg[elt], __skb_frag_page(frag), copy,
frag->page_offset+offset-start);
elt++;
if (!(len -= copy))
diff --git a/net/core/sock.c b/net/core/sock.c
index 6e81978..0fb2160 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1530,7 +1530,6 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
skb_shinfo(skb)->nr_frags = npages;
for (i = 0; i < npages; i++) {
struct page *page;
- skb_frag_t *frag;
page = alloc_pages(sk->sk_allocation, 0);
if (!page) {
@@ -1540,12 +1539,11 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
goto failure;
}
- frag = &skb_shinfo(skb)->frags[i];
- frag->page = page;
- frag->page_offset = 0;
- frag->size = (data_len >= PAGE_SIZE ?
- PAGE_SIZE :
- data_len);
+ __skb_fill_page_desc(skb, i,
+ page, 0,
+ (data_len >= PAGE_SIZE ?
+ PAGE_SIZE :
+ data_len));
data_len -= PAGE_SIZE;
}
diff --git a/net/core/user_dma.c b/net/core/user_dma.c
index 25d717e..d22ec3e 100644
--- a/net/core/user_dma.c
+++ b/net/core/user_dma.c
@@ -78,7 +78,7 @@ int dma_skb_copy_datagram_iovec(struct dma_chan *chan,
copy = end - offset;
if (copy > 0) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- struct page *page = frag->page;
+ struct page *page = __skb_frag_page(frag); /* XXX */
if (copy > len)
copy = len;
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 05/13] net: ipv4: convert to SKB frag APIs
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (3 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 04/13] net: convert core to skb paged frag APIs Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 06/13] net: ipv6: " Ian Campbell
` (8 subsequent siblings)
13 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, David S. Miller, Alexey Kuznetsov,
Pekka Savola (ipv6),
James Morris, Hideaki YOSHIFUJI, Patrick McHardy
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: "Pekka Savola (ipv6)" <pekkas@netcore.fi>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: netdev@vger.kernel.org
---
net/ipv4/inet_lro.c | 2 +-
net/ipv4/ip_output.c | 8 +++++---
net/ipv4/tcp.c | 3 ++-
net/ipv4/tcp_output.c | 2 +-
4 files changed, 9 insertions(+), 6 deletions(-)
diff --git a/net/ipv4/inet_lro.c b/net/ipv4/inet_lro.c
index 85a0f75..63f3def 100644
--- a/net/ipv4/inet_lro.c
+++ b/net/ipv4/inet_lro.c
@@ -449,7 +449,7 @@ static struct sk_buff *__lro_proc_segment(struct net_lro_mgr *lro_mgr,
if (!lro_mgr->get_frag_header ||
lro_mgr->get_frag_header(frags, (void *)&mac_hdr, (void *)&iph,
(void *)&tcph, &flags, priv)) {
- mac_hdr = page_address(frags->page) + frags->page_offset;
+ mac_hdr = skb_frag_address(frags);
goto out1;
}
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 84f26e8..3aa3c91 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -982,14 +982,14 @@ alloc_new_skb:
if (page && (left = PAGE_SIZE - off) > 0) {
if (copy >= left)
copy = left;
- if (page != frag->page) {
+ if (page != skb_frag_page(frag)) {
if (i == MAX_SKB_FRAGS) {
err = -EMSGSIZE;
goto error;
}
- get_page(page);
skb_fill_page_desc(skb, i, page, off, 0);
frag = &skb_shinfo(skb)->frags[i];
+ __skb_frag_ref(frag);
}
} else if (i < MAX_SKB_FRAGS) {
if (copy > PAGE_SIZE)
@@ -1002,13 +1002,15 @@ alloc_new_skb:
cork->page = page;
cork->off = 0;
+ /* XXX no ref ? */
skb_fill_page_desc(skb, i, page, 0, 0);
frag = &skb_shinfo(skb)->frags[i];
} else {
err = -EMSGSIZE;
goto error;
}
- if (getfrag(from, page_address(frag->page)+frag->page_offset+frag->size, offset, copy, skb->len, skb) < 0) {
+ if (getfrag(from, skb_frag_address(frag)+frag->size,
+ offset, copy, skb->len, skb) < 0) {
err = -EFAULT;
goto error;
}
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 46febca..ac47ab3 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -3035,7 +3035,8 @@ int tcp_md5_hash_skb_data(struct tcp_md5sig_pool *hp,
for (i = 0; i < shi->nr_frags; ++i) {
const struct skb_frag_struct *f = &shi->frags[i];
- sg_set_page(&sg, f->page, f->size, f->page_offset);
+ struct page *page = __skb_frag_page(f); /* XXX */
+ sg_set_page(&sg, page, f->size, f->page_offset);
if (crypto_hash_update(desc, &sg, f->size))
return 1;
}
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 882e0b0..0377c06 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -1095,7 +1095,7 @@ static void __pskb_trim_head(struct sk_buff *skb, int len)
k = 0;
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
if (skb_shinfo(skb)->frags[i].size <= eat) {
- put_page(skb_shinfo(skb)->frags[i].page);
+ skb_frag_unref(skb, i);
eat -= skb_shinfo(skb)->frags[i].size;
} else {
skb_shinfo(skb)->frags[k] = skb_shinfo(skb)->frags[i];
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 06/13] net: ipv6: convert to SKB frag APIs
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (4 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 05/13] net: ipv4: convert to SKB " Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 07/13] net: xfrm: " Ian Campbell
` (7 subsequent siblings)
13 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, David S. Miller, Alexey Kuznetsov,
Pekka Savola (ipv6),
James Morris, Hideaki YOSHIFUJI, Patrick McHardy
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: "Pekka Savola (ipv6)" <pekkas@netcore.fi>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: netdev@vger.kernel.org
---
net/ipv6/ip6_output.c | 8 +++++---
1 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 9d4b165..fdd4f61 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1441,14 +1441,14 @@ alloc_new_skb:
if (page && (left = PAGE_SIZE - off) > 0) {
if (copy >= left)
copy = left;
- if (page != frag->page) {
+ if (page != skb_frag_page(frag)) {
if (i == MAX_SKB_FRAGS) {
err = -EMSGSIZE;
goto error;
}
- get_page(page);
skb_fill_page_desc(skb, i, page, sk->sk_sndmsg_off, 0);
frag = &skb_shinfo(skb)->frags[i];
+ __skb_frag_ref(frag);
}
} else if(i < MAX_SKB_FRAGS) {
if (copy > PAGE_SIZE)
@@ -1461,13 +1461,15 @@ alloc_new_skb:
sk->sk_sndmsg_page = page;
sk->sk_sndmsg_off = 0;
+ /* XXX no ref ? */
skb_fill_page_desc(skb, i, page, 0, 0);
frag = &skb_shinfo(skb)->frags[i];
} else {
err = -EMSGSIZE;
goto error;
}
- if (getfrag(from, page_address(frag->page)+frag->page_offset+frag->size, offset, copy, skb->len, skb) < 0) {
+ if (getfrag(from, skb_frag_address(frag)+frag->size,
+ offset, copy, skb->len, skb) < 0) {
err = -EFAULT;
goto error;
}
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 07/13] net: xfrm: convert to SKB frag APIs
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (5 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 06/13] net: ipv6: " Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 08/13] net: convert drivers to paged frag API Ian Campbell
` (6 subsequent siblings)
13 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs; +Cc: Ian Campbell, David S. Miller
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: netdev@vger.kernel.org
---
net/xfrm/xfrm_ipcomp.c | 11 +++++++----
1 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/net/xfrm/xfrm_ipcomp.c b/net/xfrm/xfrm_ipcomp.c
index fc91ad7..f781b9a 100644
--- a/net/xfrm/xfrm_ipcomp.c
+++ b/net/xfrm/xfrm_ipcomp.c
@@ -70,26 +70,29 @@ static int ipcomp_decompress(struct xfrm_state *x, struct sk_buff *skb)
while ((scratch += len, dlen -= len) > 0) {
skb_frag_t *frag;
+ struct page *page;
err = -EMSGSIZE;
if (WARN_ON(skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS))
goto out;
frag = skb_shinfo(skb)->frags + skb_shinfo(skb)->nr_frags;
- frag->page = alloc_page(GFP_ATOMIC);
+ page = alloc_page(GFP_ATOMIC);
err = -ENOMEM;
- if (!frag->page)
+ if (!page)
goto out;
+ __skb_frag_set_page(frag, page);
+
len = PAGE_SIZE;
if (dlen < len)
len = dlen;
- memcpy(page_address(frag->page), scratch, len);
-
frag->page_offset = 0;
frag->size = len;
+ memcpy(skb_frag_address(frag), scratch, len);
+
skb->truesize += len;
skb->data_len += len;
skb->len += len;
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 08/13] net: convert drivers to paged frag API.
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (6 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 07/13] net: xfrm: " Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 14:12 ` David Miller
2011-07-22 13:17 ` [PATCH 09/13] net: add support for per-paged-fragment destructors Ian Campbell
` (5 subsequent siblings)
13 siblings, 1 reply; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs; +Cc: Ian Campbell
Coccinelle was quite useful in the initial stages of this conversion but a) my
spatch was ugly as sin and b) I've done several rounds of updates since then so
they no longer actually represent the resultant changes anyway.
NB: should be split into individual patches to be acked by relevant driver
maintainers.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
[since v1:
Remove skb_frag_pci_map and switch everyone to skb_frag_dma_map (on
recommendation of Michał Mirosław, drivers should be switching to use the
DMA API directly instead of the PCI DMA API)
]
---
drivers/atm/eni.c | 4 +-
drivers/infiniband/hw/amso1100/c2.c | 8 ++----
drivers/infiniband/hw/nes/nes_nic.c | 20 +++++++++---------
drivers/infiniband/ulp/ipoib/ipoib_cm.c | 6 +++-
drivers/infiniband/ulp/ipoib/ipoib_ib.c | 6 +++-
drivers/net/3c59x.c | 4 +-
drivers/net/8139cp.c | 3 +-
drivers/net/acenic.c | 6 ++--
drivers/net/atl1c/atl1c_main.c | 9 +++----
drivers/net/atl1e/atl1e_main.c | 11 ++++-----
drivers/net/atlx/atl1.c | 10 ++++----
drivers/net/benet/be_main.c | 10 ++++----
drivers/net/bna/bnad.c | 4 +-
drivers/net/bnx2.c | 8 +++---
drivers/net/bnx2x/bnx2x_cmn.c | 3 +-
drivers/net/cassini.c | 15 ++++++-------
drivers/net/chelsio/sge.c | 5 +--
drivers/net/cxgb3/sge.c | 6 ++--
drivers/net/cxgb4/sge.c | 14 +++++++-----
drivers/net/cxgb4vf/sge.c | 13 ++++++-----
drivers/net/e1000/e1000_main.c | 9 +++----
drivers/net/e1000e/netdev.c | 7 ++---
drivers/net/enic/enic_main.c | 14 ++++++------
drivers/net/forcedeth.c | 12 +++++++---
drivers/net/gianfar.c | 10 ++++----
drivers/net/greth.c | 9 ++-----
drivers/net/ibmveth.c | 5 +--
drivers/net/igb/igb_main.c | 5 +---
drivers/net/igbvf/netdev.c | 5 +---
drivers/net/ixgb/ixgb_main.c | 6 ++--
drivers/net/ixgbe/ixgbe_main.c | 9 +++----
drivers/net/ixgbevf/ixgbevf_main.c | 10 +++-----
drivers/net/jme.c | 5 ++-
drivers/net/ksz884x.c | 3 +-
drivers/net/mlx4/en_rx.c | 22 +++++++++-----------
drivers/net/mlx4/en_tx.c | 20 +++--------------
drivers/net/mv643xx_eth.c | 8 +++---
drivers/net/myri10ge/myri10ge.c | 12 +++++-----
drivers/net/netxen/netxen_nic_main.c | 4 +-
drivers/net/niu.c | 4 +-
drivers/net/ns83820.c | 5 +--
drivers/net/pasemi_mac.c | 5 +--
drivers/net/qla3xxx.c | 5 +--
drivers/net/qlcnic/qlcnic_main.c | 4 +-
drivers/net/qlge/qlge_main.c | 8 ++----
drivers/net/r8169.c | 2 +-
drivers/net/s2io.c | 8 +++---
drivers/net/sfc/rx.c | 2 +-
drivers/net/sfc/tx.c | 13 +++--------
drivers/net/skge.c | 4 +-
drivers/net/sky2.c | 13 +++++------
drivers/net/starfire.c | 2 +-
drivers/net/stmmac/stmmac_main.c | 5 +--
drivers/net/sungem.c | 6 +---
drivers/net/sunhme.c | 5 +--
drivers/net/tehuti.c | 6 ++--
drivers/net/tg3.c | 6 +---
drivers/net/tsi108_eth.c | 7 +++--
drivers/net/typhoon.c | 3 +-
drivers/net/via-velocity.c | 7 +++--
drivers/net/virtio_net.c | 2 +-
drivers/net/vmxnet3/vmxnet3_drv.c | 8 +++---
drivers/net/vxge/vxge-main.c | 6 ++--
drivers/net/xen-netback/netback.c | 34 +++++++++++++++++++++----------
drivers/net/xen-netfront.c | 28 +++++++++++++++----------
drivers/scsi/bnx2fc/bnx2fc_fcoe.c | 2 +-
drivers/scsi/cxgbi/libcxgbi.c | 6 ++--
drivers/scsi/fcoe/fcoe.c | 2 +-
drivers/scsi/fcoe/fcoe_transport.c | 5 ++-
drivers/staging/et131x/et1310_tx.c | 11 ++++-----
drivers/staging/hv/netvsc_drv.c | 2 +-
71 files changed, 271 insertions(+), 295 deletions(-)
diff --git a/drivers/atm/eni.c b/drivers/atm/eni.c
index 3230ea0..60bc9c5 100644
--- a/drivers/atm/eni.c
+++ b/drivers/atm/eni.c
@@ -1133,8 +1133,8 @@ DPRINTK("doing direct send\n"); /* @@@ well, this doesn't work anyway */
skb->data,
skb_headlen(skb));
else
- put_dma(tx->index,eni_dev->dma,&j,(unsigned long)
- skb_shinfo(skb)->frags[i].page + skb_shinfo(skb)->frags[i].page_offset,
+ put_dma(tx->index,eni_dev->dma,&j,
+ (unsigned long)skb_frag_address(&skb_shinfo(skb)->frags[i]),
skb_shinfo(skb)->frags[i].size);
}
if (skb->len & 3)
diff --git a/drivers/infiniband/hw/amso1100/c2.c b/drivers/infiniband/hw/amso1100/c2.c
index 0cfc455..f1fff58 100644
--- a/drivers/infiniband/hw/amso1100/c2.c
+++ b/drivers/infiniband/hw/amso1100/c2.c
@@ -801,11 +801,9 @@ static int c2_xmit_frame(struct sk_buff *skb, struct net_device *netdev)
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
maplen = frag->size;
- mapaddr =
- pci_map_page(c2dev->pcidev, frag->page,
- frag->page_offset, maplen,
- PCI_DMA_TODEVICE);
-
+ mapaddr = skb_frag_dma_map(&c2dev->pcidev->dev, frag,
+ 0, maplen,
+ PCI_DMA_TODEVICE);
elem = elem->next;
elem->skb = NULL;
elem->mapaddr = mapaddr;
diff --git a/drivers/infiniband/hw/nes/nes_nic.c b/drivers/infiniband/hw/nes/nes_nic.c
index d3a1c41..b60f10f 100644
--- a/drivers/infiniband/hw/nes/nes_nic.c
+++ b/drivers/infiniband/hw/nes/nes_nic.c
@@ -441,11 +441,11 @@ static int nes_nic_send(struct sk_buff *skb, struct net_device *netdev)
nesnic->tx_skb[nesnic->sq_head] = skb;
for (skb_fragment_index = 0; skb_fragment_index < skb_shinfo(skb)->nr_frags;
skb_fragment_index++) {
- bus_address = pci_map_page( nesdev->pcidev,
- skb_shinfo(skb)->frags[skb_fragment_index].page,
- skb_shinfo(skb)->frags[skb_fragment_index].page_offset,
- skb_shinfo(skb)->frags[skb_fragment_index].size,
- PCI_DMA_TODEVICE);
+ bus_address = skb_frag_dma_map(&nesdev->pcidev->dev,
+ &skb_shinfo(skb)->frags[skb_fragment_index],
+ 0,
+ skb_shinfo(skb)->frags[skb_fragment_index].size,
+ PCI_DMA_TODEVICE);
wqe_fragment_length[wqe_fragment_index] =
cpu_to_le16(skb_shinfo(skb)->frags[skb_fragment_index].size);
set_wqe_64bit_value(nic_sqe->wqe_words, NES_NIC_SQ_WQE_FRAG0_LOW_IDX+(2*wqe_fragment_index),
@@ -561,11 +561,11 @@ tso_sq_no_longer_full:
/* Map all the buffers */
for (tso_frag_count=0; tso_frag_count < skb_shinfo(skb)->nr_frags;
tso_frag_count++) {
- tso_bus_address[tso_frag_count] = pci_map_page( nesdev->pcidev,
- skb_shinfo(skb)->frags[tso_frag_count].page,
- skb_shinfo(skb)->frags[tso_frag_count].page_offset,
- skb_shinfo(skb)->frags[tso_frag_count].size,
- PCI_DMA_TODEVICE);
+ tso_bus_address[tso_frag_count] = skb_frag_dma_map(&nesdev->pcidev->dev,
+ &skb_shinfo(skb)->frags[tso_frag_count],
+ 0,
+ skb_shinfo(skb)->frags[tso_frag_count].size,
+ PCI_DMA_TODEVICE);
}
tso_frag_index = 0;
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
index 39913a0..1f20e40 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
@@ -169,7 +169,8 @@ static struct sk_buff *ipoib_cm_alloc_rx_skb(struct net_device *dev,
goto partial_error;
skb_fill_page_desc(skb, i, page, 0, PAGE_SIZE);
- mapping[i + 1] = ib_dma_map_page(priv->ca, skb_shinfo(skb)->frags[i].page,
+ mapping[i + 1] = ib_dma_map_page(priv->ca,
+ __skb_frag_page(&skb_shinfo(skb)->frags[i]),
0, PAGE_SIZE, DMA_FROM_DEVICE);
if (unlikely(ib_dma_mapping_error(priv->ca, mapping[i + 1])))
goto partial_error;
@@ -537,7 +538,8 @@ static void skb_put_frags(struct sk_buff *skb, unsigned int hdr_space,
if (length == 0) {
/* don't need this page */
- skb_fill_page_desc(toskb, i, frag->page, 0, PAGE_SIZE);
+ skb_fill_page_desc(toskb, i, __skb_frag_page(frag),
+ 0, PAGE_SIZE);/* XXX */
--skb_shinfo(skb)->nr_frags;
} else {
size = min(length, (unsigned) PAGE_SIZE);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 81ae61d..f6ef6c2 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -182,7 +182,8 @@ static struct sk_buff *ipoib_alloc_rx_skb(struct net_device *dev, int id)
goto partial_error;
skb_fill_page_desc(skb, 0, page, 0, PAGE_SIZE);
mapping[1] =
- ib_dma_map_page(priv->ca, skb_shinfo(skb)->frags[0].page,
+ ib_dma_map_page(priv->ca,
+ __skb_frag_page(&skb_shinfo(skb)->frags[0]),
0, PAGE_SIZE, DMA_FROM_DEVICE);
if (unlikely(ib_dma_mapping_error(priv->ca, mapping[1])))
goto partial_error;
@@ -323,7 +324,8 @@ static int ipoib_dma_map_tx(struct ib_device *ca,
for (i = 0; i < skb_shinfo(skb)->nr_frags; ++i) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- mapping[i + off] = ib_dma_map_page(ca, frag->page,
+ mapping[i + off] = ib_dma_map_page(ca,
+ __skb_frag_page(frag),
frag->page_offset, frag->size,
DMA_TO_DEVICE);
if (unlikely(ib_dma_mapping_error(ca, mapping[i + off])))
diff --git a/drivers/net/3c59x.c b/drivers/net/3c59x.c
index 8cc2256..fcd3820 100644
--- a/drivers/net/3c59x.c
+++ b/drivers/net/3c59x.c
@@ -2180,8 +2180,8 @@ boomerang_start_xmit(struct sk_buff *skb, struct net_device *dev)
vp->tx_ring[entry].frag[i+1].addr =
cpu_to_le32(pci_map_single(VORTEX_PCI(vp),
- (void*)page_address(frag->page) + frag->page_offset,
- frag->size, PCI_DMA_TODEVICE));
+ (void*)skb_frag_address(frag),
+ frag->size, PCI_DMA_TODEVICE));
if (i == skb_shinfo(skb)->nr_frags-1)
vp->tx_ring[entry].frag[i+1].length = cpu_to_le32(frag->size|LAST_FRAG);
diff --git a/drivers/net/8139cp.c b/drivers/net/8139cp.c
index 10c4505..c418aa3 100644
--- a/drivers/net/8139cp.c
+++ b/drivers/net/8139cp.c
@@ -815,8 +815,7 @@ static netdev_tx_t cp_start_xmit (struct sk_buff *skb,
len = this_frag->size;
mapping = dma_map_single(&cp->pdev->dev,
- ((void *) page_address(this_frag->page) +
- this_frag->page_offset),
+ skb_frag_address(this_frag),
len, PCI_DMA_TODEVICE);
eor = (entry == (CP_TX_RING_SIZE - 1)) ? RingEnd : 0;
diff --git a/drivers/net/acenic.c b/drivers/net/acenic.c
index d7c1bfe4..cc73b49 100644
--- a/drivers/net/acenic.c
+++ b/drivers/net/acenic.c
@@ -2528,9 +2528,9 @@ restart:
info = ap->skb->tx_skbuff + idx;
desc = ap->tx_ring + idx;
- mapping = pci_map_page(ap->pdev, frag->page,
- frag->page_offset, frag->size,
- PCI_DMA_TODEVICE);
+ mapping = skb_frag_dma_map(&ap->pdev->dev, frag, 0,
+ frag->size,
+ PCI_DMA_TODEVICE);
flagsize = (frag->size << 16);
if (skb->ip_summed == CHECKSUM_PARTIAL)
diff --git a/drivers/net/atl1c/atl1c_main.c b/drivers/net/atl1c/atl1c_main.c
index 1269ba5..ce6dbc5 100644
--- a/drivers/net/atl1c/atl1c_main.c
+++ b/drivers/net/atl1c/atl1c_main.c
@@ -2161,11 +2161,10 @@ static void atl1c_tx_map(struct atl1c_adapter *adapter,
buffer_info = atl1c_get_tx_buffer(adapter, use_tpd);
buffer_info->length = frag->size;
- buffer_info->dma =
- pci_map_page(adapter->pdev, frag->page,
- frag->page_offset,
- buffer_info->length,
- PCI_DMA_TODEVICE);
+ buffer_info->dma = skb_frag_dma_map(&adapter->pdev->dev,
+ frag, 0,
+ buffer_info->length,
+ PCI_DMA_TODEVICE);
ATL1C_SET_BUFFER_STATE(buffer_info, ATL1C_BUFFER_BUSY);
ATL1C_SET_PCIMAP_TYPE(buffer_info, ATL1C_PCIMAP_PAGE,
ATL1C_PCIMAP_TODEVICE);
diff --git a/drivers/net/atl1e/atl1e_main.c b/drivers/net/atl1e/atl1e_main.c
index 86a9122..b292c2a 100644
--- a/drivers/net/atl1e/atl1e_main.c
+++ b/drivers/net/atl1e/atl1e_main.c
@@ -1745,12 +1745,11 @@ static void atl1e_tx_map(struct atl1e_adapter *adapter,
MAX_TX_BUF_LEN : buf_len;
buf_len -= tx_buffer->length;
- tx_buffer->dma =
- pci_map_page(adapter->pdev, frag->page,
- frag->page_offset +
- (i * MAX_TX_BUF_LEN),
- tx_buffer->length,
- PCI_DMA_TODEVICE);
+ tx_buffer->dma = skb_frag_dma_map(&adapter->pdev->dev,
+ frag,
+ (i * MAX_TX_BUF_LEN),
+ tx_buffer->length,
+ PCI_DMA_TODEVICE);
ATL1E_SET_PCIMAP_TYPE(tx_buffer, ATL1E_TX_PCIMAP_PAGE);
use_tpd->buffer_addr = cpu_to_le64(tx_buffer->dma);
use_tpd->word2 = (use_tpd->word2 & (~TPD_BUFLEN_MASK)) |
diff --git a/drivers/net/atlx/atl1.c b/drivers/net/atlx/atl1.c
index cd5789f..d76229c 100644
--- a/drivers/net/atlx/atl1.c
+++ b/drivers/net/atlx/atl1.c
@@ -2283,11 +2283,11 @@ static void atl1_tx_map(struct atl1_adapter *adapter, struct sk_buff *skb,
buffer_info->length = (buf_len > ATL1_MAX_TX_BUF_LEN) ?
ATL1_MAX_TX_BUF_LEN : buf_len;
buf_len -= buffer_info->length;
- buffer_info->dma = pci_map_page(adapter->pdev,
- frag->page,
- frag->page_offset + (i * ATL1_MAX_TX_BUF_LEN),
- buffer_info->length, PCI_DMA_TODEVICE);
-
+ buffer_info->dma = skb_frag_dma_map(&adapter->pdev->dev,
+ frag,
+ (i * ATL1_MAX_TX_BUF_LEN),
+ buffer_info->length,
+ PCI_DMA_TODEVICE);
if (++next_to_use == tpd_ring->count)
next_to_use = 0;
}
diff --git a/drivers/net/benet/be_main.c b/drivers/net/benet/be_main.c
index a485f7f..3e4f643 100644
--- a/drivers/net/benet/be_main.c
+++ b/drivers/net/benet/be_main.c
@@ -715,8 +715,8 @@ static int make_tx_wrbs(struct be_adapter *adapter,
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
struct skb_frag_struct *frag =
&skb_shinfo(skb)->frags[i];
- busaddr = dma_map_page(dev, frag->page, frag->page_offset,
- frag->size, DMA_TO_DEVICE);
+ busaddr = skb_frag_dma_map(dev, frag, 0,
+ frag->size, DMA_TO_DEVICE);
if (dma_mapping_error(dev, busaddr))
goto dma_err;
wrb = queue_head_node(txq);
@@ -1122,7 +1122,7 @@ static void skb_fill_rx_data(struct be_adapter *adapter, struct be_rx_obj *rxo,
skb->tail += curr_frag_len;
} else {
skb_shinfo(skb)->nr_frags = 1;
- skb_shinfo(skb)->frags[0].page = page_info->page;
+ skb_frag_set_page(skb, 0, page_info->page);
skb_shinfo(skb)->frags[0].page_offset =
page_info->page_offset + hdr_len;
skb_shinfo(skb)->frags[0].size = curr_frag_len - hdr_len;
@@ -1147,7 +1147,7 @@ static void skb_fill_rx_data(struct be_adapter *adapter, struct be_rx_obj *rxo,
if (page_info->page_offset == 0) {
/* Fresh page */
j++;
- skb_shinfo(skb)->frags[j].page = page_info->page;
+ skb_frag_set_page(skb, j, page_info->page);
skb_shinfo(skb)->frags[j].page_offset =
page_info->page_offset;
skb_shinfo(skb)->frags[j].size = 0;
@@ -1236,7 +1236,7 @@ static void be_rx_compl_process_gro(struct be_adapter *adapter,
if (i == 0 || page_info->page_offset == 0) {
/* First frag or Fresh page */
j++;
- skb_shinfo(skb)->frags[j].page = page_info->page;
+ skb_frag_set_page(skb, j, page_info->page);
skb_shinfo(skb)->frags[j].page_offset =
page_info->page_offset;
skb_shinfo(skb)->frags[j].size = 0;
diff --git a/drivers/net/bna/bnad.c b/drivers/net/bna/bnad.c
index 44e219c..31b0f16 100644
--- a/drivers/net/bna/bnad.c
+++ b/drivers/net/bna/bnad.c
@@ -2635,8 +2635,8 @@ bnad_start_xmit(struct sk_buff *skb, struct net_device *netdev)
BUG_ON(!(size <= BFI_TX_MAX_DATA_PER_VECTOR));
txqent->vector[vect_id].length = htons(size);
- dma_addr = dma_map_page(&bnad->pcidev->dev, frag->page,
- frag->page_offset, size, DMA_TO_DEVICE);
+ dma_addr = skb_frag_dma_map(&bnad->pcidev->dev, frag,
+ 0, size, DMA_TO_DEVICE);
dma_unmap_addr_set(&unmap_q->unmap_array[unmap_prod], dma_addr,
dma_addr);
BNA_SET_DMA_ADDR(dma_addr, &txqent->vector[vect_id].host_addr);
diff --git a/drivers/net/bnx2.c b/drivers/net/bnx2.c
index 57d3293..ff90b13 100644
--- a/drivers/net/bnx2.c
+++ b/drivers/net/bnx2.c
@@ -2880,8 +2880,8 @@ bnx2_reuse_rx_skb_pages(struct bnx2 *bp, struct bnx2_rx_ring_info *rxr,
shinfo = skb_shinfo(skb);
shinfo->nr_frags--;
- page = shinfo->frags[shinfo->nr_frags].page;
- shinfo->frags[shinfo->nr_frags].page = NULL;
+ page = __skb_frag_page(&shinfo->frags[shinfo->nr_frags]);
+ __skb_frag_set_page(&shinfo->frags[shinfo->nr_frags], NULL);
cons_rx_pg->page = page;
dev_kfree_skb(skb);
@@ -6461,8 +6461,8 @@ bnx2_start_xmit(struct sk_buff *skb, struct net_device *dev)
txbd = &txr->tx_desc_ring[ring_prod];
len = frag->size;
- mapping = dma_map_page(&bp->pdev->dev, frag->page, frag->page_offset,
- len, PCI_DMA_TODEVICE);
+ mapping = skb_frag_dma_map(&bp->pdev->dev, frag, 0, len,
+ PCI_DMA_TODEVICE);
if (dma_mapping_error(&bp->pdev->dev, mapping))
goto dma_error;
dma_unmap_addr_set(&txr->tx_buf_ring[ring_prod], mapping,
diff --git a/drivers/net/bnx2x/bnx2x_cmn.c b/drivers/net/bnx2x/bnx2x_cmn.c
index 28904433..dee09d7 100644
--- a/drivers/net/bnx2x/bnx2x_cmn.c
+++ b/drivers/net/bnx2x/bnx2x_cmn.c
@@ -2406,8 +2406,7 @@ netdev_tx_t bnx2x_start_xmit(struct sk_buff *skb, struct net_device *dev)
if (total_pkt_bd == NULL)
total_pkt_bd = &fp->tx_desc_ring[bd_prod].reg_bd;
- mapping = dma_map_page(&bp->pdev->dev, frag->page,
- frag->page_offset,
+ mapping = skb_frag_dma_map(&bp->pdev->dev, frag, 0,
frag->size, DMA_TO_DEVICE);
tx_data_bd->addr_hi = cpu_to_le32(U64_HI(mapping));
diff --git a/drivers/net/cassini.c b/drivers/net/cassini.c
index 22ce03e..903d88d 100644
--- a/drivers/net/cassini.c
+++ b/drivers/net/cassini.c
@@ -2047,8 +2047,8 @@ static int cas_rx_process_pkt(struct cas *cp, struct cas_rx_comp *rxc,
skb->truesize += hlen - swivel;
skb->len += hlen - swivel;
- get_page(page->buffer);
- frag->page = page->buffer;
+ __skb_frag_set_page(frag, page->buffer);
+ __skb_frag_ref(frag);
frag->page_offset = off;
frag->size = hlen - swivel;
@@ -2071,8 +2071,8 @@ static int cas_rx_process_pkt(struct cas *cp, struct cas_rx_comp *rxc,
skb->len += hlen;
frag++;
- get_page(page->buffer);
- frag->page = page->buffer;
+ __skb_frag_set_page(frag, page->buffer);
+ __skb_frag_ref(frag);
frag->page_offset = 0;
frag->size = hlen;
RX_USED_ADD(page, hlen + cp->crc_size);
@@ -2829,9 +2829,8 @@ static inline int cas_xmit_tx_ringN(struct cas *cp, int ring,
skb_frag_t *fragp = &skb_shinfo(skb)->frags[frag];
len = fragp->size;
- mapping = pci_map_page(cp->pdev, fragp->page,
- fragp->page_offset, len,
- PCI_DMA_TODEVICE);
+ mapping = skb_frag_dma_map(&cp->pdev->dev, fragp, 0, len,
+ PCI_DMA_TODEVICE);
tabort = cas_calc_tabort(cp, fragp->page_offset, len);
if (unlikely(tabort)) {
@@ -2842,7 +2841,7 @@ static inline int cas_xmit_tx_ringN(struct cas *cp, int ring,
ctrl, 0);
entry = TX_DESC_NEXT(ring, entry);
- addr = cas_page_map(fragp->page);
+ addr = cas_page_map(__skb_frag_page(fragp));
memcpy(tx_tiny_buf(cp, ring, entry),
addr + fragp->page_offset + len - tabort,
tabort);
diff --git a/drivers/net/chelsio/sge.c b/drivers/net/chelsio/sge.c
index 58380d2..39c06d9 100644
--- a/drivers/net/chelsio/sge.c
+++ b/drivers/net/chelsio/sge.c
@@ -1276,9 +1276,8 @@ static inline void write_tx_descs(struct adapter *adapter, struct sk_buff *skb,
ce = q->centries;
}
- mapping = pci_map_page(adapter->pdev, frag->page,
- frag->page_offset, frag->size,
- PCI_DMA_TODEVICE);
+ mapping = skb_frag_dma_map(&adapter->pdev->dev, frag, 0,
+ frag->size, PCI_DMA_TODEVICE);
desc_mapping = mapping;
desc_len = frag->size;
diff --git a/drivers/net/cxgb3/sge.c b/drivers/net/cxgb3/sge.c
index 76bf589..3f73a5c 100644
--- a/drivers/net/cxgb3/sge.c
+++ b/drivers/net/cxgb3/sge.c
@@ -979,8 +979,8 @@ static inline unsigned int make_sgl(const struct sk_buff *skb,
for (i = 0; i < nfrags; i++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- mapping = pci_map_page(pdev, frag->page, frag->page_offset,
- frag->size, PCI_DMA_TODEVICE);
+ mapping = skb_frag_dma_map(&pdev->dev, frag, 0, frag->size,
+ PCI_DMA_TODEVICE);
sgp->len[j] = cpu_to_be32(frag->size);
sgp->addr[j] = cpu_to_be64(mapping);
j ^= 1;
@@ -2133,7 +2133,7 @@ static void lro_add_page(struct adapter *adap, struct sge_qset *qs,
len -= offset;
rx_frag += nr_frags;
- rx_frag->page = sd->pg_chunk.page;
+ __skb_frag_set_page(rx_frag, sd->pg_chunk.page);
rx_frag->page_offset = sd->pg_chunk.offset + offset;
rx_frag->size = len;
diff --git a/drivers/net/cxgb4/sge.c b/drivers/net/cxgb4/sge.c
index 56adf44..f1813b5 100644
--- a/drivers/net/cxgb4/sge.c
+++ b/drivers/net/cxgb4/sge.c
@@ -215,8 +215,8 @@ static int map_skb(struct device *dev, const struct sk_buff *skb,
end = &si->frags[si->nr_frags];
for (fp = si->frags; fp < end; fp++) {
- *++addr = dma_map_page(dev, fp->page, fp->page_offset, fp->size,
- DMA_TO_DEVICE);
+ *++addr = skb_frag_dma_map(dev, fp, 0, fp->size,
+ DMA_TO_DEVICE);
if (dma_mapping_error(dev, *addr))
goto unwind;
}
@@ -1409,13 +1409,14 @@ int cxgb4_ofld_send(struct net_device *dev, struct sk_buff *skb)
}
EXPORT_SYMBOL(cxgb4_ofld_send);
-static inline void copy_frags(struct skb_shared_info *ssi,
+static inline void copy_frags(struct sk_buff *skb,
const struct pkt_gl *gl, unsigned int offset)
{
+ struct skb_shared_info *ssi = skb_shinfo(skb);
unsigned int n;
/* usually there's just one frag */
- ssi->frags[0].page = gl->frags[0].page;
+ skb_frag_set_page(skb, 0, gl->frags[0].page);
ssi->frags[0].page_offset = gl->frags[0].page_offset + offset;
ssi->frags[0].size = gl->frags[0].size - offset;
ssi->nr_frags = gl->nfrags;
@@ -1459,7 +1460,7 @@ struct sk_buff *cxgb4_pktgl_to_skb(const struct pkt_gl *gl,
__skb_put(skb, pull_len);
skb_copy_to_linear_data(skb, gl->va, pull_len);
- copy_frags(skb_shinfo(skb), gl, pull_len);
+ copy_frags(skb, gl, pull_len);
skb->len = gl->tot_len;
skb->data_len = skb->len - pull_len;
skb->truesize += skb->data_len;
@@ -1522,7 +1523,7 @@ static void do_gro(struct sge_eth_rxq *rxq, const struct pkt_gl *gl,
return;
}
- copy_frags(skb_shinfo(skb), gl, RX_PKT_PAD);
+ copy_frags(skb, gl, RX_PKT_PAD);
skb->len = gl->tot_len - RX_PKT_PAD;
skb->data_len = skb->len;
skb->truesize += skb->data_len;
@@ -1735,6 +1736,7 @@ static int process_responses(struct sge_rspq *q, int budget)
si.va = page_address(si.frags[0].page) +
si.frags[0].page_offset;
+
prefetch(si.va);
si.nfrags = frags + 1;
diff --git a/drivers/net/cxgb4vf/sge.c b/drivers/net/cxgb4vf/sge.c
index 5fd75fd..f4c4480 100644
--- a/drivers/net/cxgb4vf/sge.c
+++ b/drivers/net/cxgb4vf/sge.c
@@ -296,8 +296,8 @@ static int map_skb(struct device *dev, const struct sk_buff *skb,
si = skb_shinfo(skb);
end = &si->frags[si->nr_frags];
for (fp = si->frags; fp < end; fp++) {
- *++addr = dma_map_page(dev, fp->page, fp->page_offset, fp->size,
- DMA_TO_DEVICE);
+ *++addr = skb_frag_dma_map(dev, fp, 0, fp->size,
+ DMA_TO_DEVICE);
if (dma_mapping_error(dev, *addr))
goto unwind;
}
@@ -1397,7 +1397,7 @@ struct sk_buff *t4vf_pktgl_to_skb(const struct pkt_gl *gl,
skb_copy_to_linear_data(skb, gl->va, pull_len);
ssi = skb_shinfo(skb);
- ssi->frags[0].page = gl->frags[0].page;
+ skb_frag_set_page(skb, 0, gl->frags[0].page);
ssi->frags[0].page_offset = gl->frags[0].page_offset + pull_len;
ssi->frags[0].size = gl->frags[0].size - pull_len;
if (gl->nfrags > 1)
@@ -1442,14 +1442,15 @@ void t4vf_pktgl_free(const struct pkt_gl *gl)
* Copy an internal packet gather list into a Linux skb_shared_info
* structure.
*/
-static inline void copy_frags(struct skb_shared_info *si,
+static inline void copy_frags(struct sk_buff *skb,
const struct pkt_gl *gl,
unsigned int offset)
{
+ struct skb_shared_info *si = skb_shinfo(skb);
unsigned int n;
/* usually there's just one frag */
- si->frags[0].page = gl->frags[0].page;
+ skb_frag_set_page(skb, 0, gl->frags[0].page);
si->frags[0].page_offset = gl->frags[0].page_offset + offset;
si->frags[0].size = gl->frags[0].size - offset;
si->nr_frags = gl->nfrags;
@@ -1484,7 +1485,7 @@ static void do_gro(struct sge_eth_rxq *rxq, const struct pkt_gl *gl,
return;
}
- copy_frags(skb_shinfo(skb), gl, PKTSHIFT);
+ copy_frags(skb, gl, PKTSHIFT);
skb->len = gl->tot_len - PKTSHIFT;
skb->data_len = skb->len;
skb->truesize += skb->data_len;
diff --git a/drivers/net/e1000/e1000_main.c b/drivers/net/e1000/e1000_main.c
index 76e8af0..e902cd0 100644
--- a/drivers/net/e1000/e1000_main.c
+++ b/drivers/net/e1000/e1000_main.c
@@ -2861,7 +2861,7 @@ static int e1000_tx_map(struct e1000_adapter *adapter,
frag = &skb_shinfo(skb)->frags[f];
len = frag->size;
- offset = frag->page_offset;
+ offset = 0;
while (len) {
i++;
@@ -2878,7 +2878,7 @@ static int e1000_tx_map(struct e1000_adapter *adapter,
* Avoid terminating buffers within evenly-aligned
* dwords. */
if (unlikely(adapter->pcix_82544 &&
- !((unsigned long)(page_to_phys(frag->page) + offset
+ !((unsigned long)(page_to_phys(__skb_frag_page(frag)) + offset
+ size - 1) & 4) &&
size > 4))
size -= 4;
@@ -2886,9 +2886,8 @@ static int e1000_tx_map(struct e1000_adapter *adapter,
buffer_info->length = size;
buffer_info->time_stamp = jiffies;
buffer_info->mapped_as_page = true;
- buffer_info->dma = dma_map_page(&pdev->dev, frag->page,
- offset, size,
- DMA_TO_DEVICE);
+ buffer_info->dma = skb_frag_dma_map(&pdev->dev, frag,
+ offset, size, DMA_TO_DEVICE);
if (dma_mapping_error(&pdev->dev, buffer_info->dma))
goto dma_error;
buffer_info->next_to_watch = i;
diff --git a/drivers/net/e1000e/netdev.c b/drivers/net/e1000e/netdev.c
index 3310c3d..30f8a5c 100644
--- a/drivers/net/e1000e/netdev.c
+++ b/drivers/net/e1000e/netdev.c
@@ -4599,7 +4599,7 @@ static int e1000_tx_map(struct e1000_adapter *adapter,
frag = &skb_shinfo(skb)->frags[f];
len = frag->size;
- offset = frag->page_offset;
+ offset = 0;
while (len) {
i++;
@@ -4612,9 +4612,8 @@ static int e1000_tx_map(struct e1000_adapter *adapter,
buffer_info->length = size;
buffer_info->time_stamp = jiffies;
buffer_info->next_to_watch = i;
- buffer_info->dma = dma_map_page(&pdev->dev, frag->page,
- offset, size,
- DMA_TO_DEVICE);
+ buffer_info->dma = skb_frag_dma_map(&pdev->dev, frag,
+ offset, size, DMA_TO_DEVICE);
buffer_info->mapped_as_page = true;
if (dma_mapping_error(&pdev->dev, buffer_info->dma))
goto dma_error;
diff --git a/drivers/net/enic/enic_main.c b/drivers/net/enic/enic_main.c
index 2f433fb..31bf8cb 100644
--- a/drivers/net/enic/enic_main.c
+++ b/drivers/net/enic/enic_main.c
@@ -584,9 +584,9 @@ static inline void enic_queue_wq_skb_cont(struct enic *enic,
for (frag = skb_shinfo(skb)->frags; len_left; frag++) {
len_left -= frag->size;
enic_queue_wq_desc_cont(wq, skb,
- pci_map_page(enic->pdev, frag->page,
- frag->page_offset, frag->size,
- PCI_DMA_TODEVICE),
+ skb_frag_dma_map(&enic->pdev->dev,
+ frag, 0, frag->size,
+ PCI_DMA_TODEVICE),
frag->size,
(len_left == 0), /* EOP? */
loopback);
@@ -698,14 +698,14 @@ static inline void enic_queue_wq_skb_tso(struct enic *enic,
for (frag = skb_shinfo(skb)->frags; len_left; frag++) {
len_left -= frag->size;
frag_len_left = frag->size;
- offset = frag->page_offset;
+ offset = 0;
while (frag_len_left) {
len = min(frag_len_left,
(unsigned int)WQ_ENET_MAX_DESC_LEN);
- dma_addr = pci_map_page(enic->pdev, frag->page,
- offset, len,
- PCI_DMA_TODEVICE);
+ dma_addr = skb_frag_dma_map(&enic->pdev->dev, frag,
+ offset, len,
+ PCI_DMA_TODEVICE);
enic_queue_wq_desc_cont(wq, skb,
dma_addr,
len,
diff --git a/drivers/net/forcedeth.c b/drivers/net/forcedeth.c
index 537b695..df0cb1c 100644
--- a/drivers/net/forcedeth.c
+++ b/drivers/net/forcedeth.c
@@ -2149,8 +2149,10 @@ static netdev_tx_t nv_start_xmit(struct sk_buff *skb, struct net_device *dev)
prev_tx = put_tx;
prev_tx_ctx = np->put_tx_ctx;
bcnt = (size > NV_TX2_TSO_MAX_SIZE) ? NV_TX2_TSO_MAX_SIZE : size;
- np->put_tx_ctx->dma = pci_map_page(np->pci_dev, frag->page, frag->page_offset+offset, bcnt,
- PCI_DMA_TODEVICE);
+ np->put_tx_ctx->dma = skb_frag_dma_map(&np->pci_dev->dev,
+ frag, offset,
+ bcnt,
+ PCI_DMA_TODEVICE);
np->put_tx_ctx->dma_len = bcnt;
np->put_tx_ctx->dma_single = 0;
put_tx->buf = cpu_to_le32(np->put_tx_ctx->dma);
@@ -2260,8 +2262,10 @@ static netdev_tx_t nv_start_xmit_optimized(struct sk_buff *skb,
prev_tx = put_tx;
prev_tx_ctx = np->put_tx_ctx;
bcnt = (size > NV_TX2_TSO_MAX_SIZE) ? NV_TX2_TSO_MAX_SIZE : size;
- np->put_tx_ctx->dma = pci_map_page(np->pci_dev, frag->page, frag->page_offset+offset, bcnt,
- PCI_DMA_TODEVICE);
+ np->put_tx_ctx->dma = skb_frag_dma_map(&np->pci_dev->dev,
+ frag, offset,
+ bcnt,
+ PCI_DMA_TODEVICE);
np->put_tx_ctx->dma_len = bcnt;
np->put_tx_ctx->dma_single = 0;
put_tx->bufhigh = cpu_to_le32(dma_high(np->put_tx_ctx->dma));
diff --git a/drivers/net/gianfar.c b/drivers/net/gianfar.c
index 2dfcc80..766e037 100644
--- a/drivers/net/gianfar.c
+++ b/drivers/net/gianfar.c
@@ -2141,11 +2141,11 @@ static int gfar_start_xmit(struct sk_buff *skb, struct net_device *dev)
if (i == nr_frags - 1)
lstatus |= BD_LFLAG(TXBD_LAST | TXBD_INTERRUPT);
- bufaddr = dma_map_page(&priv->ofdev->dev,
- skb_shinfo(skb)->frags[i].page,
- skb_shinfo(skb)->frags[i].page_offset,
- length,
- DMA_TO_DEVICE);
+ bufaddr = skb_frag_dma_map(&priv->ofdev->dev,
+ &skb_shinfo(skb)->frags[i],
+ 0,
+ length,
+ DMA_TO_DEVICE);
/* set the TxBD length and buffer pointer */
txbdp->bufPtr = bufaddr;
diff --git a/drivers/net/greth.c b/drivers/net/greth.c
index 672f096..26acb8d 100644
--- a/drivers/net/greth.c
+++ b/drivers/net/greth.c
@@ -111,7 +111,7 @@ static void greth_print_tx_packet(struct sk_buff *skb)
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
print_hex_dump(KERN_DEBUG, "TX: ", DUMP_PREFIX_OFFSET, 16, 1,
- phys_to_virt(page_to_phys(skb_shinfo(skb)->frags[i].page)) +
+ phys_to_virt(page_to_phys(skb_frag_page(&skb_shinfo(skb)->frags[i]))) +
skb_shinfo(skb)->frags[i].page_offset,
length, true);
}
@@ -526,11 +526,8 @@ greth_start_xmit_gbit(struct sk_buff *skb, struct net_device *dev)
greth_write_bd(&bdp->stat, status);
- dma_addr = dma_map_page(greth->dev,
- frag->page,
- frag->page_offset,
- frag->size,
- DMA_TO_DEVICE);
+ dma_addr = skb_frag_dma_map(greth->dev, frag, 0, frag->size,
+ DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(greth->dev, dma_addr)))
goto frag_map_error;
diff --git a/drivers/net/ibmveth.c b/drivers/net/ibmveth.c
index b388d78..65e5874 100644
--- a/drivers/net/ibmveth.c
+++ b/drivers/net/ibmveth.c
@@ -1001,9 +1001,8 @@ retry_bounce:
unsigned long dma_addr;
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- dma_addr = dma_map_page(&adapter->vdev->dev, frag->page,
- frag->page_offset, frag->size,
- DMA_TO_DEVICE);
+ dma_addr = skb_frag_dma_map(&adapter->vdev->dev, frag, 0,
+ frag->size, DMA_TO_DEVICE);
if (dma_mapping_error(&adapter->vdev->dev, dma_addr))
goto map_failed_frags;
diff --git a/drivers/net/igb/igb_main.c b/drivers/net/igb/igb_main.c
index 2c28621..17f94f4 100644
--- a/drivers/net/igb/igb_main.c
+++ b/drivers/net/igb/igb_main.c
@@ -4132,10 +4132,7 @@ static inline int igb_tx_map_adv(struct igb_ring *tx_ring, struct sk_buff *skb,
buffer_info->time_stamp = jiffies;
buffer_info->next_to_watch = i;
buffer_info->mapped_as_page = true;
- buffer_info->dma = dma_map_page(dev,
- frag->page,
- frag->page_offset,
- len,
+ buffer_info->dma = skb_frag_dma_map(dev, frag, 0, len,
DMA_TO_DEVICE);
if (dma_mapping_error(dev, buffer_info->dma))
goto dma_error;
diff --git a/drivers/net/igbvf/netdev.c b/drivers/net/igbvf/netdev.c
index 1c77fb3..3f6655f 100644
--- a/drivers/net/igbvf/netdev.c
+++ b/drivers/net/igbvf/netdev.c
@@ -2074,10 +2074,7 @@ static inline int igbvf_tx_map_adv(struct igbvf_adapter *adapter,
buffer_info->time_stamp = jiffies;
buffer_info->next_to_watch = i;
buffer_info->mapped_as_page = true;
- buffer_info->dma = dma_map_page(&pdev->dev,
- frag->page,
- frag->page_offset,
- len,
+ buffer_info->dma = skb_frag_dma_map(&pdev->dev, frag, 0, len,
DMA_TO_DEVICE);
if (dma_mapping_error(&pdev->dev, buffer_info->dma))
goto dma_error;
diff --git a/drivers/net/ixgb/ixgb_main.c b/drivers/net/ixgb/ixgb_main.c
index 6a130eb..45c4e90 100644
--- a/drivers/net/ixgb/ixgb_main.c
+++ b/drivers/net/ixgb/ixgb_main.c
@@ -1341,7 +1341,7 @@ ixgb_tx_map(struct ixgb_adapter *adapter, struct sk_buff *skb,
frag = &skb_shinfo(skb)->frags[f];
len = frag->size;
- offset = frag->page_offset;
+ offset = 0;
while (len) {
i++;
@@ -1361,8 +1361,8 @@ ixgb_tx_map(struct ixgb_adapter *adapter, struct sk_buff *skb,
buffer_info->time_stamp = jiffies;
buffer_info->mapped_as_page = true;
buffer_info->dma =
- dma_map_page(&pdev->dev, frag->page,
- offset, size, DMA_TO_DEVICE);
+ skb_frag_dma_map(&pdev->dev, frag, offset, size,
+ DMA_TO_DEVICE);
if (dma_mapping_error(&pdev->dev, buffer_info->dma))
goto dma_error;
buffer_info->next_to_watch = 0;
diff --git a/drivers/net/ixgbe/ixgbe_main.c b/drivers/net/ixgbe/ixgbe_main.c
index 08e8e25..307cf06 100644
--- a/drivers/net/ixgbe/ixgbe_main.c
+++ b/drivers/net/ixgbe/ixgbe_main.c
@@ -6632,7 +6632,7 @@ static int ixgbe_tx_map(struct ixgbe_adapter *adapter,
frag = &skb_shinfo(skb)->frags[f];
len = min((unsigned int)frag->size, total);
- offset = frag->page_offset;
+ offset = 0;
while (len) {
i++;
@@ -6643,10 +6643,9 @@ static int ixgbe_tx_map(struct ixgbe_adapter *adapter,
size = min(len, (uint)IXGBE_MAX_DATA_PER_TXD);
tx_buffer_info->length = size;
- tx_buffer_info->dma = dma_map_page(dev,
- frag->page,
- offset, size,
- DMA_TO_DEVICE);
+ tx_buffer_info->dma =
+ skb_frag_dma_map(dev, frag, offset, size,
+ DMA_TO_DEVICE);
tx_buffer_info->mapped_as_page = true;
if (dma_mapping_error(dev, tx_buffer_info->dma))
goto dma_error;
diff --git a/drivers/net/ixgbevf/ixgbevf_main.c b/drivers/net/ixgbevf/ixgbevf_main.c
index 28d3cb2..ad05ad9 100644
--- a/drivers/net/ixgbevf/ixgbevf_main.c
+++ b/drivers/net/ixgbevf/ixgbevf_main.c
@@ -2951,18 +2951,16 @@ static int ixgbevf_tx_map(struct ixgbevf_adapter *adapter,
frag = &skb_shinfo(skb)->frags[f];
len = min((unsigned int)frag->size, total);
- offset = frag->page_offset;
+ offset = 0;
while (len) {
tx_buffer_info = &tx_ring->tx_buffer_info[i];
size = min(len, (unsigned int)IXGBE_MAX_DATA_PER_TXD);
tx_buffer_info->length = size;
- tx_buffer_info->dma = dma_map_page(&adapter->pdev->dev,
- frag->page,
- offset,
- size,
- DMA_TO_DEVICE);
+ tx_buffer_info->dma =
+ skb_frag_dma_map(&adapter->pdev->dev, frag,
+ offset, size, DMA_TO_DEVICE);
tx_buffer_info->mapped_as_page = true;
if (dma_mapping_error(&pdev->dev, tx_buffer_info->dma))
goto dma_error;
diff --git a/drivers/net/jme.c b/drivers/net/jme.c
index b5b174a..a73e895 100644
--- a/drivers/net/jme.c
+++ b/drivers/net/jme.c
@@ -1928,8 +1928,9 @@ jme_map_tx_skb(struct jme_adapter *jme, struct sk_buff *skb, int idx)
ctxdesc = txdesc + ((idx + i + 2) & (mask));
ctxbi = txbi + ((idx + i + 2) & (mask));
- jme_fill_tx_map(jme->pdev, ctxdesc, ctxbi, frag->page,
- frag->page_offset, frag->size, hidma);
+ jme_fill_tx_map(jme->pdev, ctxdesc, ctxbi,
+ __skb_frag_page(frag),
+ frag->page_offset, frag->size, hidma);
}
len = skb_is_nonlinear(skb) ? skb_headlen(skb) : skb->len;
diff --git a/drivers/net/ksz884x.c b/drivers/net/ksz884x.c
index 41ea592..e610d88 100644
--- a/drivers/net/ksz884x.c
+++ b/drivers/net/ksz884x.c
@@ -4703,8 +4703,7 @@ static void send_packet(struct sk_buff *skb, struct net_device *dev)
dma_buf->dma = pci_map_single(
hw_priv->pdev,
- page_address(this_frag->page) +
- this_frag->page_offset,
+ skb_frag_address(this_frag),
dma_buf->len,
PCI_DMA_TODEVICE);
set_tx_buf(desc, dma_buf->dma);
diff --git a/drivers/net/mlx4/en_rx.c b/drivers/net/mlx4/en_rx.c
index 277215f..21a89e0 100644
--- a/drivers/net/mlx4/en_rx.c
+++ b/drivers/net/mlx4/en_rx.c
@@ -60,20 +60,18 @@ static int mlx4_en_alloc_frag(struct mlx4_en_priv *priv,
if (!page)
return -ENOMEM;
- skb_frags[i].page = page_alloc->page;
+ __skb_frag_set_page(&skb_frags[i], page_alloc->page);
skb_frags[i].page_offset = page_alloc->offset;
page_alloc->page = page;
page_alloc->offset = frag_info->frag_align;
} else {
- page = page_alloc->page;
- get_page(page);
-
- skb_frags[i].page = page;
+ __skb_frag_set_page(&skb_frags[i], page_alloc->page);
+ __skb_frag_ref(&skb_frags[i]);
skb_frags[i].page_offset = page_alloc->offset;
page_alloc->offset += frag_info->frag_stride;
}
- dma = pci_map_single(mdev->pdev, page_address(skb_frags[i].page) +
- skb_frags[i].page_offset, frag_info->frag_size,
+ dma = pci_map_single(mdev->pdev, skb_frag_address(&skb_frags[i]),
+ frag_info->frag_size,
PCI_DMA_FROMDEVICE);
rx_desc->data[i].addr = cpu_to_be64(dma);
return 0;
@@ -169,7 +167,7 @@ static int mlx4_en_prepare_rx_desc(struct mlx4_en_priv *priv,
err:
while (i--)
- put_page(skb_frags[i].page);
+ __skb_frag_unref(&skb_frags[i]);
return -ENOMEM;
}
@@ -196,7 +194,7 @@ static void mlx4_en_free_rx_desc(struct mlx4_en_priv *priv,
en_dbg(DRV, priv, "Unmapping buffer at dma:0x%llx\n", (u64) dma);
pci_unmap_single(mdev->pdev, dma, skb_frags[nr].size,
PCI_DMA_FROMDEVICE);
- put_page(skb_frags[nr].page);
+ __skb_frag_unref(&skb_frags[nr]);
}
}
@@ -420,7 +418,7 @@ static int mlx4_en_complete_rx_desc(struct mlx4_en_priv *priv,
break;
/* Save page reference in skb */
- skb_frags_rx[nr].page = skb_frags[nr].page;
+ __skb_frag_set_page(&skb_frags_rx[nr], skb_frags[nr].page);
skb_frags_rx[nr].size = skb_frags[nr].size;
skb_frags_rx[nr].page_offset = skb_frags[nr].page_offset;
dma = be64_to_cpu(rx_desc->data[nr].addr);
@@ -444,7 +442,7 @@ fail:
* the descriptor) of this packet; remaining fragments are reused... */
while (nr > 0) {
nr--;
- put_page(skb_frags_rx[nr].page);
+ __skb_frag_unref(&skb_frags_rx[nr]);
}
return 0;
}
@@ -474,7 +472,7 @@ static struct sk_buff *mlx4_en_rx_skb(struct mlx4_en_priv *priv,
/* Get pointer to first fragment so we could copy the headers into the
* (linear part of the) skb */
- va = page_address(skb_frags[0].page) + skb_frags[0].page_offset;
+ va = skb_frag_address(&skb_frags[0]);
if (length <= SMALL_PACKET_SIZE) {
/* We are copying all relevant data to the skb - temporarily
diff --git a/drivers/net/mlx4/en_tx.c b/drivers/net/mlx4/en_tx.c
index b229acf..29816d6 100644
--- a/drivers/net/mlx4/en_tx.c
+++ b/drivers/net/mlx4/en_tx.c
@@ -461,26 +461,13 @@ static inline void mlx4_en_xmit_poll(struct mlx4_en_priv *priv, int tx_ind)
}
}
-static void *get_frag_ptr(struct sk_buff *skb)
-{
- struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[0];
- struct page *page = frag->page;
- void *ptr;
-
- ptr = page_address(page);
- if (unlikely(!ptr))
- return NULL;
-
- return ptr + frag->page_offset;
-}
-
static int is_inline(struct sk_buff *skb, void **pfrag)
{
void *ptr;
if (inline_thold && !skb_is_gso(skb) && skb->len <= inline_thold) {
if (skb_shinfo(skb)->nr_frags == 1) {
- ptr = get_frag_ptr(skb);
+ ptr = skb_frag_address_safe(&skb_shinfo(skb)->frags[0]);
if (unlikely(!ptr))
return 0;
@@ -757,8 +744,9 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
/* Map fragments */
for (i = skb_shinfo(skb)->nr_frags - 1; i >= 0; i--) {
frag = &skb_shinfo(skb)->frags[i];
- dma = pci_map_page(mdev->dev->pdev, frag->page, frag->page_offset,
- frag->size, PCI_DMA_TODEVICE);
+ dma = skb_frag_dma_map(&mdev->dev->pdev->dev, frag,
+ 0, frag->size,
+ PCI_DMA_TODEVICE);
data->addr = cpu_to_be64(dma);
data->lkey = cpu_to_be32(mdev->mr.key);
wmb();
diff --git a/drivers/net/mv643xx_eth.c b/drivers/net/mv643xx_eth.c
index a5d9b1c..d02a034 100644
--- a/drivers/net/mv643xx_eth.c
+++ b/drivers/net/mv643xx_eth.c
@@ -752,10 +752,10 @@ static void txq_submit_frag_skb(struct tx_queue *txq, struct sk_buff *skb)
desc->l4i_chk = 0;
desc->byte_cnt = this_frag->size;
- desc->buf_ptr = dma_map_page(mp->dev->dev.parent,
- this_frag->page,
- this_frag->page_offset,
- this_frag->size, DMA_TO_DEVICE);
+ desc->buf_ptr = skb_frag_dma_map(mp->dev->dev.parent,
+ this_frag, 0,
+ this_frag->size,
+ DMA_TO_DEVICE);
}
}
diff --git a/drivers/net/myri10ge/myri10ge.c b/drivers/net/myri10ge/myri10ge.c
index bf84849..35b64f4 100644
--- a/drivers/net/myri10ge/myri10ge.c
+++ b/drivers/net/myri10ge/myri10ge.c
@@ -1339,7 +1339,7 @@ myri10ge_rx_done(struct myri10ge_slice_state *ss, int len, __wsum csum,
/* Fill skb_frag_struct(s) with data from our receive */
for (i = 0, remainder = len; remainder > 0; i++) {
myri10ge_unmap_rx_page(pdev, &rx->info[idx], bytes);
- rx_frags[i].page = rx->info[idx].page;
+ __skb_frag_set_page(&rx_frags[i], rx->info[idx].page); /* XXX */
rx_frags[i].page_offset = rx->info[idx].page_offset;
if (remainder < MYRI10GE_ALLOC_SIZE)
rx_frags[i].size = remainder;
@@ -1372,7 +1372,7 @@ myri10ge_rx_done(struct myri10ge_slice_state *ss, int len, __wsum csum,
ss->stats.rx_dropped++;
do {
i--;
- put_page(rx_frags[i].page);
+ __skb_frag_unref(&rx_frags[i]); /* XXX */
} while (i != 0);
return 0;
}
@@ -1380,7 +1380,7 @@ myri10ge_rx_done(struct myri10ge_slice_state *ss, int len, __wsum csum,
/* Attach the pages to the skb, and trim off any padding */
myri10ge_rx_skb_build(skb, va, rx_frags, len, hlen);
if (skb_shinfo(skb)->frags[0].size <= 0) {
- put_page(skb_shinfo(skb)->frags[0].page);
+ skb_frag_unref(skb, 0);
skb_shinfo(skb)->nr_frags = 0;
}
skb->protocol = eth_type_trans(skb, dev);
@@ -2220,7 +2220,7 @@ myri10ge_get_frag_header(struct skb_frag_struct *frag, void **mac_hdr,
struct ethhdr *eh;
struct vlan_ethhdr *veh;
struct iphdr *iph;
- u8 *va = page_address(frag->page) + frag->page_offset;
+ u8 *va = skb_frag_address(frag);
unsigned long ll_hlen;
/* passed opaque through lro_receive_frags() */
__wsum csum = (__force __wsum) (unsigned long)priv;
@@ -2863,8 +2863,8 @@ again:
frag = &skb_shinfo(skb)->frags[frag_idx];
frag_idx++;
len = frag->size;
- bus = pci_map_page(mgp->pdev, frag->page, frag->page_offset,
- len, PCI_DMA_TODEVICE);
+ bus = skb_frag_dma_map(&mgp->pdev->dev, frag, 0, len,
+ PCI_DMA_TODEVICE);
dma_unmap_addr_set(&tx->info[idx], bus, bus);
dma_unmap_len_set(&tx->info[idx], len, len);
}
diff --git a/drivers/net/netxen/netxen_nic_main.c b/drivers/net/netxen/netxen_nic_main.c
index c0788a3..d9c5864 100644
--- a/drivers/net/netxen/netxen_nic_main.c
+++ b/drivers/net/netxen/netxen_nic_main.c
@@ -1836,8 +1836,8 @@ netxen_map_tx_skb(struct pci_dev *pdev,
frag = &skb_shinfo(skb)->frags[i];
nf = &pbuf->frag_array[i+1];
- map = pci_map_page(pdev, frag->page, frag->page_offset,
- frag->size, PCI_DMA_TODEVICE);
+ map = skb_frag_dma_map(&pdev->dev, frag, 0, frag->size,
+ PCI_DMA_TODEVICE);
if (pci_dma_mapping_error(pdev, map))
goto unwind;
diff --git a/drivers/net/niu.c b/drivers/net/niu.c
index cc25bff..a901193 100644
--- a/drivers/net/niu.c
+++ b/drivers/net/niu.c
@@ -3290,7 +3290,7 @@ static void niu_rx_skb_append(struct sk_buff *skb, struct page *page,
int i = skb_shinfo(skb)->nr_frags;
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- frag->page = page;
+ __skb_frag_set_page(frag, page);
frag->page_offset = offset;
frag->size = size;
@@ -6731,7 +6731,7 @@ static netdev_tx_t niu_start_xmit(struct sk_buff *skb,
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
len = frag->size;
- mapping = np->ops->map_page(np->device, frag->page,
+ mapping = np->ops->map_page(np->device, __skb_frag_page(frag),
frag->page_offset, len,
DMA_TO_DEVICE);
diff --git a/drivers/net/ns83820.c b/drivers/net/ns83820.c
index 3e4040f..6b3508d 100644
--- a/drivers/net/ns83820.c
+++ b/drivers/net/ns83820.c
@@ -1181,9 +1181,8 @@ again:
if (!nr_frags)
break;
- buf = pci_map_page(dev->pci_dev, frag->page,
- frag->page_offset,
- frag->size, PCI_DMA_TODEVICE);
+ buf = skb_frag_dma_map(&dev->pci_dev->dev, frag, 0,
+ frag->size, PCI_DMA_TODEVICE);
dprintk("frag: buf=%08Lx page=%08lx offset=%08lx\n",
(long long)buf, (long) page_to_pfn(frag->page),
frag->page_offset);
diff --git a/drivers/net/pasemi_mac.c b/drivers/net/pasemi_mac.c
index 9ec112c..3cd2cd3 100644
--- a/drivers/net/pasemi_mac.c
+++ b/drivers/net/pasemi_mac.c
@@ -1505,9 +1505,8 @@ static int pasemi_mac_start_tx(struct sk_buff *skb, struct net_device *dev)
for (i = 0; i < nfrags; i++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- map[i+1] = pci_map_page(mac->dma_pdev, frag->page,
- frag->page_offset, frag->size,
- PCI_DMA_TODEVICE);
+ map[i + 1] = skb_frag_dma_map(&mac->dma_pdev->dev, frag, 0,
+ frag->size, PCI_DMA_TODEVICE);
map_size[i+1] = frag->size;
if (pci_dma_mapping_error(mac->dma_pdev, map[i+1])) {
nfrags = i;
diff --git a/drivers/net/qla3xxx.c b/drivers/net/qla3xxx.c
index 771bb61..8a95fc7 100644
--- a/drivers/net/qla3xxx.c
+++ b/drivers/net/qla3xxx.c
@@ -2388,9 +2388,8 @@ static int ql_send_map(struct ql3_adapter *qdev,
seg++;
}
- map = pci_map_page(qdev->pdev, frag->page,
- frag->page_offset, frag->size,
- PCI_DMA_TODEVICE);
+ map = skb_frag_dma_map(&qdev->pdev->dev, frag, 0, frag->size,
+ PCI_DMA_TODEVICE);
err = pci_dma_mapping_error(qdev->pdev, map);
if (err) {
diff --git a/drivers/net/qlcnic/qlcnic_main.c b/drivers/net/qlcnic/qlcnic_main.c
index 0f6af5c..fe8d1f8 100644
--- a/drivers/net/qlcnic/qlcnic_main.c
+++ b/drivers/net/qlcnic/qlcnic_main.c
@@ -2120,8 +2120,8 @@ qlcnic_map_tx_skb(struct pci_dev *pdev,
frag = &skb_shinfo(skb)->frags[i];
nf = &pbuf->frag_array[i+1];
- map = pci_map_page(pdev, frag->page, frag->page_offset,
- frag->size, PCI_DMA_TODEVICE);
+ map = skb_frag_dma_map(&pdev->dev, frag, 0, frag->size,
+ PCI_DMA_TODEVICE);
if (pci_dma_mapping_error(pdev, map))
goto unwind;
diff --git a/drivers/net/qlge/qlge_main.c b/drivers/net/qlge/qlge_main.c
index 6b4ff97..cc04e5b 100644
--- a/drivers/net/qlge/qlge_main.c
+++ b/drivers/net/qlge/qlge_main.c
@@ -1430,10 +1430,8 @@ static int ql_map_send(struct ql_adapter *qdev,
map_idx++;
}
- map =
- pci_map_page(qdev->pdev, frag->page,
- frag->page_offset, frag->size,
- PCI_DMA_TODEVICE);
+ map = skb_frag_dma_map(&qdev->pdev->dev, frag, 0, frag->size,
+ PCI_DMA_TODEVICE);
err = pci_dma_mapping_error(qdev->pdev, map);
if (err) {
@@ -1494,7 +1492,7 @@ static void ql_process_mac_rx_gro_page(struct ql_adapter *qdev,
rx_frag = skb_shinfo(skb)->frags;
nr_frags = skb_shinfo(skb)->nr_frags;
rx_frag += nr_frags;
- rx_frag->page = lbq_desc->p.pg_chunk.page;
+ __skb_frag_set_page(rx_frag, lbq_desc->p.pg_chunk.page);
rx_frag->page_offset = lbq_desc->p.pg_chunk.offset;
rx_frag->size = length;
diff --git a/drivers/net/r8169.c b/drivers/net/r8169.c
index 5990621f..44e27bd 100644
--- a/drivers/net/r8169.c
+++ b/drivers/net/r8169.c
@@ -4630,7 +4630,7 @@ static int rtl8169_xmit_frags(struct rtl8169_private *tp, struct sk_buff *skb,
txd = tp->TxDescArray + entry;
len = frag->size;
- addr = ((void *) page_address(frag->page)) + frag->page_offset;
+ addr = skb_frag_address(frag);
mapping = dma_map_single(d, addr, len, DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(d, mapping))) {
if (net_ratelimit())
diff --git a/drivers/net/s2io.c b/drivers/net/s2io.c
index df0d2c8..93f20fa 100644
--- a/drivers/net/s2io.c
+++ b/drivers/net/s2io.c
@@ -4242,10 +4242,10 @@ static netdev_tx_t s2io_xmit(struct sk_buff *skb, struct net_device *dev)
if (!frag->size)
continue;
txdp++;
- txdp->Buffer_Pointer = (u64)pci_map_page(sp->pdev, frag->page,
- frag->page_offset,
- frag->size,
- PCI_DMA_TODEVICE);
+ txdp->Buffer_Pointer = (u64)skb_frag_dma_map(&sp->pdev->dev,
+ frag, 0,
+ frag->size,
+ PCI_DMA_TODEVICE);
txdp->Control_1 = TXD_BUFFER0_SIZE(frag->size);
if (offload_type == SKB_GSO_UDP)
txdp->Control_1 |= TXD_UFO_EN;
diff --git a/drivers/net/sfc/rx.c b/drivers/net/sfc/rx.c
index 62e4364..91a6b71 100644
--- a/drivers/net/sfc/rx.c
+++ b/drivers/net/sfc/rx.c
@@ -478,7 +478,7 @@ static void efx_rx_packet_gro(struct efx_channel *channel,
if (efx->net_dev->features & NETIF_F_RXHASH)
skb->rxhash = efx_rx_buf_hash(eh);
- skb_shinfo(skb)->frags[0].page = page;
+ skb_frag_set_page(skb, 0, page);
skb_shinfo(skb)->frags[0].page_offset =
efx_rx_buf_offset(efx, rx_buf);
skb_shinfo(skb)->frags[0].size = rx_buf->len;
diff --git a/drivers/net/sfc/tx.c b/drivers/net/sfc/tx.c
index 84eb99e..f2467a1 100644
--- a/drivers/net/sfc/tx.c
+++ b/drivers/net/sfc/tx.c
@@ -137,8 +137,6 @@ netdev_tx_t efx_enqueue_skb(struct efx_tx_queue *tx_queue, struct sk_buff *skb)
struct pci_dev *pci_dev = efx->pci_dev;
struct efx_tx_buffer *buffer;
skb_frag_t *fragment;
- struct page *page;
- int page_offset;
unsigned int len, unmap_len = 0, fill_level, insert_ptr;
dma_addr_t dma_addr, unmap_addr = 0;
unsigned int dma_len;
@@ -241,13 +239,11 @@ netdev_tx_t efx_enqueue_skb(struct efx_tx_queue *tx_queue, struct sk_buff *skb)
break;
fragment = &skb_shinfo(skb)->frags[i];
len = fragment->size;
- page = fragment->page;
- page_offset = fragment->page_offset;
i++;
/* Map for DMA */
unmap_single = false;
- dma_addr = pci_map_page(pci_dev, page, page_offset, len,
- PCI_DMA_TODEVICE);
+ dma_addr = skb_frag_dma_map(&pci_dev->dev, fragment, 0, len,
+ PCI_DMA_TODEVICE);
}
/* Transfer ownership of the skb to the final buffer */
@@ -929,9 +925,8 @@ static void tso_start(struct tso_state *st, const struct sk_buff *skb)
static int tso_get_fragment(struct tso_state *st, struct efx_nic *efx,
skb_frag_t *frag)
{
- st->unmap_addr = pci_map_page(efx->pci_dev, frag->page,
- frag->page_offset, frag->size,
- PCI_DMA_TODEVICE);
+ st->unmap_addr = skb_frag_dma_map(&efx->pci_dev->dev, frag, 0,
+ frag->size, PCI_DMA_TODEVICE);
if (likely(!pci_dma_mapping_error(efx->pci_dev, st->unmap_addr))) {
st->unmap_single = false;
st->unmap_len = frag->size;
diff --git a/drivers/net/skge.c b/drivers/net/skge.c
index f4be5c7..cf635f7 100644
--- a/drivers/net/skge.c
+++ b/drivers/net/skge.c
@@ -2747,8 +2747,8 @@ static netdev_tx_t skge_xmit_frame(struct sk_buff *skb,
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- map = pci_map_page(hw->pdev, frag->page, frag->page_offset,
- frag->size, PCI_DMA_TODEVICE);
+ map = skb_frag_dma_map(&hw->pdev->dev, frag, 0,
+ frag->size, PCI_DMA_TODEVICE);
e = e->next;
e->skb = skb;
diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 3ee41da..e1cf142 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -1143,10 +1143,9 @@ static int sky2_rx_map_skb(struct pci_dev *pdev, struct rx_ring_info *re,
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- re->frag_addr[i] = pci_map_page(pdev, frag->page,
- frag->page_offset,
- frag->size,
- PCI_DMA_FROMDEVICE);
+ re->frag_addr[i] = skb_frag_dma_map(&pdev->dev, frag, 0,
+ frag->size,
+ PCI_DMA_FROMDEVICE);
if (pci_dma_mapping_error(pdev, re->frag_addr[i]))
goto map_page_error;
@@ -1826,8 +1825,8 @@ static netdev_tx_t sky2_xmit_frame(struct sk_buff *skb,
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
const skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- mapping = pci_map_page(hw->pdev, frag->page, frag->page_offset,
- frag->size, PCI_DMA_TODEVICE);
+ mapping = skb_frag_dma_map(&hw->pdev->dev, frag, 0,
+ frag->size, PCI_DMA_TODEVICE);
if (pci_dma_mapping_error(hw->pdev, mapping))
goto mapping_unwind;
@@ -2360,7 +2359,7 @@ static void skb_put_frags(struct sk_buff *skb, unsigned int hdr_space,
if (length == 0) {
/* don't need this page */
- __free_page(frag->page);
+ __skb_frag_unref(frag);
--skb_shinfo(skb)->nr_frags;
} else {
size = min(length, (unsigned) PAGE_SIZE);
diff --git a/drivers/net/starfire.c b/drivers/net/starfire.c
index 36045f3..a0c8f34 100644
--- a/drivers/net/starfire.c
+++ b/drivers/net/starfire.c
@@ -1270,7 +1270,7 @@ static netdev_tx_t start_tx(struct sk_buff *skb, struct net_device *dev)
skb_frag_t *this_frag = &skb_shinfo(skb)->frags[i - 1];
status |= this_frag->size;
np->tx_info[entry].mapping =
- pci_map_single(np->pci_dev, page_address(this_frag->page) + this_frag->page_offset, this_frag->size, PCI_DMA_TODEVICE);
+ pci_map_single(np->pci_dev, skb_frag_address(this_frag), this_frag->size, PCI_DMA_TODEVICE);
}
np->tx_ring[entry].addr = cpu_to_dma(np->tx_info[entry].mapping);
diff --git a/drivers/net/stmmac/stmmac_main.c b/drivers/net/stmmac/stmmac_main.c
index e25e44a..5157624 100644
--- a/drivers/net/stmmac/stmmac_main.c
+++ b/drivers/net/stmmac/stmmac_main.c
@@ -1040,9 +1040,8 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
desc = priv->dma_tx + entry;
TX_DBG("\t[entry %d] segment len: %d\n", entry, len);
- desc->des2 = dma_map_page(priv->device, frag->page,
- frag->page_offset,
- len, DMA_TO_DEVICE);
+ desc->des2 = skb_frag_dma_map(priv->device, frag, 0, len,
+ DMA_TO_DEVICE);
priv->tx_skbuff[entry] = NULL;
priv->hw->desc->prepare_tx_desc(desc, 0, len, csum_insertion);
priv->hw->desc->set_tx_owner(desc);
diff --git a/drivers/net/sungem.c b/drivers/net/sungem.c
index ab59300..6b8a7bf 100644
--- a/drivers/net/sungem.c
+++ b/drivers/net/sungem.c
@@ -1078,10 +1078,8 @@ static netdev_tx_t gem_start_xmit(struct sk_buff *skb,
u64 this_ctrl;
len = this_frag->size;
- mapping = pci_map_page(gp->pdev,
- this_frag->page,
- this_frag->page_offset,
- len, PCI_DMA_TODEVICE);
+ mapping = skb_frag_dma_map(&gp->pdev->dev, this_frag,
+ 0, len, PCI_DMA_TODEVICE);
this_ctrl = ctrl;
if (frag == skb_shinfo(skb)->nr_frags - 1)
this_ctrl |= TXDCTRL_EOF;
diff --git a/drivers/net/sunhme.c b/drivers/net/sunhme.c
index 30aad54..3baef5e 100644
--- a/drivers/net/sunhme.c
+++ b/drivers/net/sunhme.c
@@ -2315,9 +2315,8 @@ static netdev_tx_t happy_meal_start_xmit(struct sk_buff *skb,
u32 len, mapping, this_txflags;
len = this_frag->size;
- mapping = dma_map_page(hp->dma_dev, this_frag->page,
- this_frag->page_offset, len,
- DMA_TO_DEVICE);
+ mapping = skb_frag_dma_map(hp->dma_dev, this_frag,
+ 0, len, DMA_TO_DEVICE);
this_txflags = tx_flags;
if (frag == skb_shinfo(skb)->nr_frags - 1)
this_txflags |= TXFLAG_EOP;
diff --git a/drivers/net/tehuti.c b/drivers/net/tehuti.c
index 80fbee0..d552617 100644
--- a/drivers/net/tehuti.c
+++ b/drivers/net/tehuti.c
@@ -1519,9 +1519,9 @@ bdx_tx_map_skb(struct bdx_priv *priv, struct sk_buff *skb,
frag = &skb_shinfo(skb)->frags[i];
db->wptr->len = frag->size;
- db->wptr->addr.dma =
- pci_map_page(priv->pdev, frag->page, frag->page_offset,
- frag->size, PCI_DMA_TODEVICE);
+ db->wptr->addr.dma = skb_frag_dma_map(&priv->pdev->dev, frag,
+ 0, frag->size,
+ PCI_DMA_TODEVICE);
pbl++;
pbl->len = CPU_CHIP_SWAP32(db->wptr->len);
diff --git a/drivers/net/tg3.c b/drivers/net/tg3.c
index a1f9f9e..c53104d 100644
--- a/drivers/net/tg3.c
+++ b/drivers/net/tg3.c
@@ -6040,10 +6040,8 @@ static netdev_tx_t tg3_start_xmit(struct sk_buff *skb, struct net_device *dev)
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
len = frag->size;
- mapping = pci_map_page(tp->pdev,
- frag->page,
- frag->page_offset,
- len, PCI_DMA_TODEVICE);
+ mapping = skb_frag_dma_map(&tp->pdev->dev, frag, 0,
+ len, PCI_DMA_TODEVICE);
tnapi->tx_buffers[entry].skb = NULL;
dma_unmap_addr_set(&tnapi->tx_buffers[entry], mapping,
diff --git a/drivers/net/tsi108_eth.c b/drivers/net/tsi108_eth.c
index 5c633a3..52f89a5 100644
--- a/drivers/net/tsi108_eth.c
+++ b/drivers/net/tsi108_eth.c
@@ -710,9 +710,10 @@ static int tsi108_send_packet(struct sk_buff * skb, struct net_device *dev)
} else {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i - 1];
- data->txring[tx].buf0 =
- dma_map_page(NULL, frag->page, frag->page_offset,
- frag->size, DMA_TO_DEVICE);
+ data->txring[tx].buf0 = skb_frag_dma_map(NULL, frag,
+ 0,
+ frag->size,
+ DMA_TO_DEVICE);
data->txring[tx].len = frag->size;
}
diff --git a/drivers/net/typhoon.c b/drivers/net/typhoon.c
index 3de4283..eb32147 100644
--- a/drivers/net/typhoon.c
+++ b/drivers/net/typhoon.c
@@ -819,8 +819,7 @@ typhoon_start_tx(struct sk_buff *skb, struct net_device *dev)
typhoon_inc_tx_index(&txRing->lastWrite, 1);
len = frag->size;
- frag_addr = (void *) page_address(frag->page) +
- frag->page_offset;
+ frag_addr = skb_frag_address(frag);
skb_dma = pci_map_single(tp->tx_pdev, frag_addr, len,
PCI_DMA_TODEVICE);
txd->flags = TYPHOON_FRAG_DESC | TYPHOON_DESC_VALID;
diff --git a/drivers/net/via-velocity.c b/drivers/net/via-velocity.c
index 06daa9d..6e34b78 100644
--- a/drivers/net/via-velocity.c
+++ b/drivers/net/via-velocity.c
@@ -2580,9 +2580,10 @@ static netdev_tx_t velocity_xmit(struct sk_buff *skb,
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- tdinfo->skb_dma[i + 1] = pci_map_page(vptr->pdev, frag->page,
- frag->page_offset, frag->size,
- PCI_DMA_TODEVICE);
+ tdinfo->skb_dma[i + 1] = skb_frag_dma_map(&vptr->pdev->dev,
+ frag, 0,
+ frag->size,
+ PCI_DMA_TODEVICE);
td_ptr->td_buf[i + 1].pa_low = cpu_to_le32(tdinfo->skb_dma[i + 1]);
td_ptr->td_buf[i + 1].pa_high = 0;
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index f685324..c35ae8f 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -137,7 +137,7 @@ static void set_skb_frag(struct sk_buff *skb, struct page *page,
f = &skb_shinfo(skb)->frags[i];
f->size = min((unsigned)PAGE_SIZE - offset, *len);
f->page_offset = offset;
- f->page = page;
+ __skb_frag_set_page(f, page);
skb->data_len += f->size;
skb->len += f->size;
diff --git a/drivers/net/vmxnet3/vmxnet3_drv.c b/drivers/net/vmxnet3/vmxnet3_drv.c
index 6740235..c4a8d2e 100644
--- a/drivers/net/vmxnet3/vmxnet3_drv.c
+++ b/drivers/net/vmxnet3/vmxnet3_drv.c
@@ -656,7 +656,7 @@ vmxnet3_append_frag(struct sk_buff *skb, struct Vmxnet3_RxCompDesc *rcd,
BUG_ON(skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS);
- frag->page = rbi->page;
+ __skb_frag_set_page(frag, rbi->page);
frag->page_offset = 0;
frag->size = rcd->len;
skb->data_len += frag->size;
@@ -750,9 +750,9 @@ vmxnet3_map_pkt(struct sk_buff *skb, struct vmxnet3_tx_ctx *ctx,
tbi = tq->buf_info + tq->tx_ring.next2fill;
tbi->map_type = VMXNET3_MAP_PAGE;
- tbi->dma_addr = pci_map_page(adapter->pdev, frag->page,
- frag->page_offset, frag->size,
- PCI_DMA_TODEVICE);
+ tbi->dma_addr = skb_frag_dma_map(&adapter->pdev->dev, frag,
+ 0, frag->size,
+ PCI_DMA_TODEVICE);
tbi->len = frag->size;
diff --git a/drivers/net/vxge/vxge-main.c b/drivers/net/vxge/vxge-main.c
index 8ab870a..63e7797 100644
--- a/drivers/net/vxge/vxge-main.c
+++ b/drivers/net/vxge/vxge-main.c
@@ -921,9 +921,9 @@ vxge_xmit(struct sk_buff *skb, struct net_device *dev)
if (!frag->size)
continue;
- dma_pointer = (u64) pci_map_page(fifo->pdev, frag->page,
- frag->page_offset, frag->size,
- PCI_DMA_TODEVICE);
+ dma_pointer = (u64)skb_frag_dma_map(&fifo->pdev->dev, frag,
+ 0, frag->size,
+ PCI_DMA_TODEVICE);
if (unlikely(pci_dma_mapping_error(fifo->pdev, dma_pointer)))
goto _exit2;
diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index 0e4851b..5c79483 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -215,6 +215,16 @@ static int get_page_ext(struct page *pg,
sizeof(struct iphdr) + MAX_IPOPTLEN + \
sizeof(struct tcphdr) + MAX_TCP_OPTION_SPACE)
+static unsigned long frag_get_pending_idx(skb_frag_t *frag)
+{
+ return (unsigned long)skb_frag_page(frag);
+}
+
+static void frag_set_pending_idx(skb_frag_t *frag, unsigned long pending_idx)
+{
+ __skb_frag_set_page(frag, (void *)pending_idx);
+}
+
static inline pending_ring_idx_t pending_index(unsigned i)
{
return i & (MAX_PENDING_REQS-1);
@@ -512,7 +522,7 @@ static int netbk_gop_skb(struct sk_buff *skb,
for (i = 0; i < nr_frags; i++) {
netbk_gop_frag_copy(vif, skb, npo,
- skb_shinfo(skb)->frags[i].page,
+ __skb_frag_page(&skb_shinfo(skb)->frags[i]),
skb_shinfo(skb)->frags[i].size,
skb_shinfo(skb)->frags[i].page_offset,
&head);
@@ -913,7 +923,7 @@ static struct gnttab_copy *xen_netbk_get_requests(struct xen_netbk *netbk,
int i, start;
/* Skip first skb fragment if it is on same page as header fragment. */
- start = ((unsigned long)shinfo->frags[0].page == pending_idx);
+ start = (frag_get_pending_idx(&shinfo->frags[0]) == pending_idx);
for (i = start; i < shinfo->nr_frags; i++, txp++) {
struct page *page;
@@ -945,7 +955,7 @@ static struct gnttab_copy *xen_netbk_get_requests(struct xen_netbk *netbk,
memcpy(&pending_tx_info[pending_idx].req, txp, sizeof(*txp));
xenvif_get(vif);
pending_tx_info[pending_idx].vif = vif;
- frags[i].page = (void *)pending_idx;
+ frag_set_pending_idx(&frags[i], pending_idx);
}
return gop;
@@ -976,13 +986,13 @@ static int xen_netbk_tx_check_gop(struct xen_netbk *netbk,
}
/* Skip first skb fragment if it is on same page as header fragment. */
- start = ((unsigned long)shinfo->frags[0].page == pending_idx);
+ start = (frag_get_pending_idx(&shinfo->frags[0]) == pending_idx);
for (i = start; i < nr_frags; i++) {
int j, newerr;
pending_ring_idx_t index;
- pending_idx = (unsigned long)shinfo->frags[i].page;
+ pending_idx = frag_get_pending_idx(&shinfo->frags[i]);
/* Check error status: if okay then remember grant handle. */
newerr = (++gop)->status;
@@ -1008,7 +1018,7 @@ static int xen_netbk_tx_check_gop(struct xen_netbk *netbk,
pending_idx = *((u16 *)skb->data);
xen_netbk_idx_release(netbk, pending_idx);
for (j = start; j < i; j++) {
- pending_idx = (unsigned long)shinfo->frags[i].page;
+ pending_idx = frag_get_pending_idx(&shinfo->frags[i]);
xen_netbk_idx_release(netbk, pending_idx);
}
@@ -1029,12 +1039,14 @@ static void xen_netbk_fill_frags(struct xen_netbk *netbk, struct sk_buff *skb)
for (i = 0; i < nr_frags; i++) {
skb_frag_t *frag = shinfo->frags + i;
struct xen_netif_tx_request *txp;
+ struct page *page;
unsigned long pending_idx;
- pending_idx = (unsigned long)frag->page;
+ pending_idx = frag_get_pending_idx(frag);
txp = &netbk->pending_tx_info[pending_idx].req;
- frag->page = virt_to_page(idx_to_kaddr(netbk, pending_idx));
+ page = virt_to_page(idx_to_kaddr(netbk, pending_idx));
+ __skb_frag_set_page(frag, page);
frag->size = txp->size;
frag->page_offset = txp->offset;
@@ -1349,11 +1361,11 @@ static unsigned xen_netbk_tx_build_gops(struct xen_netbk *netbk)
skb_shinfo(skb)->nr_frags = ret;
if (data_len < txreq.size) {
skb_shinfo(skb)->nr_frags++;
- skb_shinfo(skb)->frags[0].page =
- (void *)(unsigned long)pending_idx;
+ frag_set_pending_idx(&skb_shinfo(skb)->frags[0],
+ pending_idx);
} else {
/* Discriminate from any valid pending_idx value. */
- skb_shinfo(skb)->frags[0].page = (void *)~0UL;
+ frag_set_pending_idx(&skb_shinfo(skb)->frags[0], ~0UL);
}
__skb_queue_tail(&netbk->tx_queue, skb);
diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index d29365a..ecc4b4b 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -265,7 +265,7 @@ no_skb:
break;
}
- skb_shinfo(skb)->frags[0].page = page;
+ skb_frag_set_page(skb, 0, page);
skb_shinfo(skb)->nr_frags = 1;
__skb_queue_tail(&np->rx_batch, skb);
}
@@ -299,8 +299,8 @@ no_skb:
BUG_ON((signed short)ref < 0);
np->grant_rx_ref[id] = ref;
- pfn = page_to_pfn(skb_shinfo(skb)->frags[0].page);
- vaddr = page_address(skb_shinfo(skb)->frags[0].page);
+ pfn = page_to_pfn(skb_frag_page(&skb_shinfo(skb)->frags[0]));
+ vaddr = page_address(skb_frag_page(&skb_shinfo(skb)->frags[0]));
req = RING_GET_REQUEST(&np->rx, req_prod + i);
gnttab_grant_foreign_access_ref(ref,
@@ -451,7 +451,7 @@ static void xennet_make_frags(struct sk_buff *skb, struct net_device *dev,
ref = gnttab_claim_grant_reference(&np->gref_tx_head);
BUG_ON((signed short)ref < 0);
- mfn = pfn_to_mfn(page_to_pfn(frag->page));
+ mfn = pfn_to_mfn(page_to_pfn(skb_frag_page(frag)));
gnttab_grant_foreign_access_ref(ref, np->xbdev->otherend_id,
mfn, GNTMAP_readonly);
@@ -755,8 +755,9 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
while ((nskb = __skb_dequeue(list))) {
struct xen_netif_rx_response *rx =
RING_GET_RESPONSE(&np->rx, ++cons);
+ skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
- frag->page = skb_shinfo(nskb)->frags[0].page;
+ __skb_frag_set_page(frag, __skb_frag_page(nfrag));
frag->page_offset = rx->offset;
frag->size = rx->status;
@@ -858,7 +859,7 @@ static int handle_incoming_queue(struct net_device *dev,
memcpy(skb->data, vaddr + offset,
skb_headlen(skb));
- if (page != skb_shinfo(skb)->frags[0].page)
+ if (page != skb_frag_page(&skb_shinfo(skb)->frags[0]))
__free_page(page);
/* Ethernet work: Delayed to here as it peeks the header. */
@@ -937,7 +938,8 @@ err:
}
}
- NETFRONT_SKB_CB(skb)->page = skb_shinfo(skb)->frags[0].page;
+ NETFRONT_SKB_CB(skb)->page =
+ __skb_frag_page(&skb_shinfo(skb)->frags[0]);
NETFRONT_SKB_CB(skb)->offset = rx->offset;
len = rx->status;
@@ -951,7 +953,7 @@ err:
skb_shinfo(skb)->frags[0].size = rx->status - len;
skb->data_len = rx->status - len;
} else {
- skb_shinfo(skb)->frags[0].page = NULL;
+ skb_frag_set_page(skb, 0, NULL);
skb_shinfo(skb)->nr_frags = 0;
}
@@ -1094,7 +1096,8 @@ static void xennet_release_rx_bufs(struct netfront_info *np)
if (!xen_feature(XENFEAT_auto_translated_physmap)) {
/* Remap the page. */
- struct page *page = skb_shinfo(skb)->frags[0].page;
+ const struct page *page =
+ skb_frag_page(&skb_shinfo(skb)->frags[0]);
unsigned long pfn = page_to_pfn(page);
void *vaddr = page_address(page);
@@ -1593,6 +1596,8 @@ static int xennet_connect(struct net_device *dev)
/* Step 2: Rebuild the RX buffer freelist and the RX ring itself. */
for (requeue_idx = 0, i = 0; i < NET_RX_RING_SIZE; i++) {
+ skb_frag_t *frag;
+ const struct page *page;
if (!np->rx_skbs[i])
continue;
@@ -1600,10 +1605,11 @@ static int xennet_connect(struct net_device *dev)
ref = np->grant_rx_ref[requeue_idx] = xennet_get_rx_ref(np, i);
req = RING_GET_REQUEST(&np->rx, requeue_idx);
+ frag = &skb_shinfo(skb)->frags[0];
+ page = skb_frag_page(frag);
gnttab_grant_foreign_access_ref(
ref, np->xbdev->otherend_id,
- pfn_to_mfn(page_to_pfn(skb_shinfo(skb)->
- frags->page)),
+ pfn_to_mfn(page_to_pfn(page)),
0);
req->gref = ref;
req->id = requeue_idx;
diff --git a/drivers/scsi/bnx2fc/bnx2fc_fcoe.c b/drivers/scsi/bnx2fc/bnx2fc_fcoe.c
index ab255fb..f7a3517 100644
--- a/drivers/scsi/bnx2fc/bnx2fc_fcoe.c
+++ b/drivers/scsi/bnx2fc/bnx2fc_fcoe.c
@@ -296,7 +296,7 @@ static int bnx2fc_xmit(struct fc_lport *lport, struct fc_frame *fp)
return -ENOMEM;
}
frag = &skb_shinfo(skb)->frags[skb_shinfo(skb)->nr_frags - 1];
- cp = kmap_atomic(frag->page, KM_SKB_DATA_SOFTIRQ)
+ cp = kmap_atomic(__skb_frag_page(frag), KM_SKB_DATA_SOFTIRQ)
+ frag->page_offset;
} else {
cp = (struct fcoe_crc_eof *)skb_put(skb, tlen);
diff --git a/drivers/scsi/cxgbi/libcxgbi.c b/drivers/scsi/cxgbi/libcxgbi.c
index a2a9c7c..949ee48 100644
--- a/drivers/scsi/cxgbi/libcxgbi.c
+++ b/drivers/scsi/cxgbi/libcxgbi.c
@@ -1812,7 +1812,7 @@ static int sgl_read_to_frags(struct scatterlist *sg, unsigned int sgoffset,
}
copy = min(datalen, sglen);
- if (i && page == frags[i - 1].page &&
+ if (i && page == skb_frag_page(&frags[i - 1]) &&
sgoffset + sg->offset ==
frags[i - 1].page_offset + frags[i - 1].size) {
frags[i - 1].size += copy;
@@ -1948,7 +1948,7 @@ int cxgbi_conn_init_pdu(struct iscsi_task *task, unsigned int offset,
/* data fits in the skb's headroom */
for (i = 0; i < tdata->nr_frags; i++, frag++) {
- char *src = kmap_atomic(frag->page,
+ char *src = kmap_atomic(__skb_frag_page(frag),
KM_SOFTIRQ0);
memcpy(dst, src+frag->page_offset, frag->size);
@@ -1963,7 +1963,7 @@ int cxgbi_conn_init_pdu(struct iscsi_task *task, unsigned int offset,
} else {
/* data fit into frag_list */
for (i = 0; i < tdata->nr_frags; i++)
- get_page(tdata->frags[i].page);
+ __skb_frag_ref(&tdata->frags[i]);
memcpy(skb_shinfo(skb)->frags, tdata->frags,
sizeof(skb_frag_t) * tdata->nr_frags);
diff --git a/drivers/scsi/fcoe/fcoe.c b/drivers/scsi/fcoe/fcoe.c
index 155d7b9..deee71a 100644
--- a/drivers/scsi/fcoe/fcoe.c
+++ b/drivers/scsi/fcoe/fcoe.c
@@ -1425,7 +1425,7 @@ int fcoe_xmit(struct fc_lport *lport, struct fc_frame *fp)
return -ENOMEM;
}
frag = &skb_shinfo(skb)->frags[skb_shinfo(skb)->nr_frags - 1];
- cp = kmap_atomic(frag->page, KM_SKB_DATA_SOFTIRQ)
+ cp = kmap_atomic(__skb_frag_page(frag), KM_SKB_DATA_SOFTIRQ)
+ frag->page_offset;
} else {
cp = (struct fcoe_crc_eof *)skb_put(skb, tlen);
diff --git a/drivers/scsi/fcoe/fcoe_transport.c b/drivers/scsi/fcoe/fcoe_transport.c
index 41068e8..40243ce 100644
--- a/drivers/scsi/fcoe/fcoe_transport.c
+++ b/drivers/scsi/fcoe/fcoe_transport.c
@@ -108,8 +108,9 @@ u32 fcoe_fc_crc(struct fc_frame *fp)
len = frag->size;
while (len > 0) {
clen = min(len, PAGE_SIZE - (off & ~PAGE_MASK));
- data = kmap_atomic(frag->page + (off >> PAGE_SHIFT),
- KM_SKB_DATA_SOFTIRQ);
+ data = kmap_atomic(
+ __skb_frag_page(frag) + (off >> PAGE_SHIFT),
+ KM_SKB_DATA_SOFTIRQ);
crc = crc32(crc, data + (off & ~PAGE_MASK), clen);
kunmap_atomic(data, KM_SKB_DATA_SOFTIRQ);
off += clen;
diff --git a/drivers/staging/et131x/et1310_tx.c b/drivers/staging/et131x/et1310_tx.c
index 4241d2a..63ee8af 100644
--- a/drivers/staging/et131x/et1310_tx.c
+++ b/drivers/staging/et131x/et1310_tx.c
@@ -519,12 +519,11 @@ static int nic_send_packet(struct et131x_adapter *etdev, struct tcb *tcb)
* returned by pci_map_page() is always 32-bit
* addressable (as defined by the pci/dma subsystem)
*/
- desc[frag++].addr_lo =
- pci_map_page(etdev->pdev,
- frags[i - 1].page,
- frags[i - 1].page_offset,
- frags[i - 1].size,
- PCI_DMA_TODEVICE);
+ desc[frag++].addr_lo = skb_frag_dma_map(&etdev->pdev->dev,
+ &frags[i - 1],
+ 0,
+ frags[i - 1].size,
+ PCI_DMA_TODEVICE);
}
}
diff --git a/drivers/staging/hv/netvsc_drv.c b/drivers/staging/hv/netvsc_drv.c
index 7b9c229..80d1f1f 100644
--- a/drivers/staging/hv/netvsc_drv.c
+++ b/drivers/staging/hv/netvsc_drv.c
@@ -172,7 +172,7 @@ static int netvsc_start_xmit(struct sk_buff *skb, struct net_device *net)
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
skb_frag_t *f = &skb_shinfo(skb)->frags[i];
- packet->page_buf[i+2].pfn = page_to_pfn(f->page);
+ packet->page_buf[i+2].pfn = page_to_pfn(skb_frag_page(f));
packet->page_buf[i+2].offset = f->page_offset;
packet->page_buf[i+2].len = f->size;
}
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 09/13] net: add support for per-paged-fragment destructors
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (7 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 08/13] net: convert drivers to paged frag API Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 10/13] net: add paged frag destructor to skb_fill_page_desc() Ian Campbell
` (4 subsequent siblings)
13 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, David S. Miller, James E.J. Bottomley,
Dimitris Michailidis, Casey Leedom, Yevgeny Petrilin,
Eric Dumazet, Michał Mirosław, linux-scsi
Entities which care about the complete lifecycle of pages which they inject
into the network stack via an skb paged fragment can choose to set this
destructor in order to receive a callback when the stack is really finished
with a page (including all clones, retransmits, pull-ups etc etc).
This destructor will always be propagated alongside the struct page when
copying skb_frag_t->page. This is the reason I chose to embed the destructor in
a "struct { } page" within the skb_frag_t, rather than as a separate field,
since it allows existing code which propagates ->frags[N].page to Just
Work(tm).
When the destructor is present the page reference counting is done slightly
differently. No references are held by the network stack on the struct page (it
is up to the caller to manage this as necessary) instead the network stack will
track references via the count embedded in the destructor structure. When this
reference count reaches zero then the destructor will be called and the caller
can take the necesary steps to release the page (i.e. release the struct page
reference itself).
The intention is that callers can use this callback to delay completion to
_their_ callers until the network stack has completely released the page, in
order to prevent use-after-free or modification of data pages which are still
in use by the stack.
It is allowable (indeed expected) for a caller to share a single destructor
instance between multiple pages injected into the stack e.g. a group of pages
included in a single higher level operation might share a destructor which is
used to complete that higher level operation.
NB: a small number of drivers use skb_frag_t independently of struct sk_buff so
this patch is slightly larger than necessary. I did consider leaving skb_frag_t
alone and defining a new (but similar) structure to be used in the struct
sk_buff itself. This would also have the advantage of more clearly separating
the two uses, which is useful since there are now special reference counting
accessors for skb_frag_t within a struct sk_buff but not (necessarily) for
those used outside of an skb.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: "James E.J. Bottomley" <JBottomley@parallels.com>
Cc: Dimitris Michailidis <dm@chelsio.com>
Cc: Casey Leedom <leedom@chelsio.com>
Cc: Yevgeny Petrilin <yevgenyp@mellanox.co.il>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: "Michał Mirosław" <mirq-linux@rere.qmqm.pl>
Cc: netdev@vger.kernel.org
Cc: linux-scsi@vger.kernel.org
---
drivers/net/cxgb4/sge.c | 14 +++++++-------
drivers/net/cxgb4vf/sge.c | 18 +++++++++---------
drivers/net/mlx4/en_rx.c | 2 +-
drivers/scsi/cxgbi/libcxgbi.c | 2 +-
include/linux/skbuff.h | 31 ++++++++++++++++++++++++++-----
net/core/skbuff.c | 17 +++++++++++++++++
6 files changed, 61 insertions(+), 23 deletions(-)
diff --git a/drivers/net/cxgb4/sge.c b/drivers/net/cxgb4/sge.c
index f1813b5..3e7c4b3 100644
--- a/drivers/net/cxgb4/sge.c
+++ b/drivers/net/cxgb4/sge.c
@@ -1416,7 +1416,7 @@ static inline void copy_frags(struct sk_buff *skb,
unsigned int n;
/* usually there's just one frag */
- skb_frag_set_page(skb, 0, gl->frags[0].page);
+ skb_frag_set_page(skb, 0, gl->frags[0].page.p); /* XXX */
ssi->frags[0].page_offset = gl->frags[0].page_offset + offset;
ssi->frags[0].size = gl->frags[0].size - offset;
ssi->nr_frags = gl->nfrags;
@@ -1425,7 +1425,7 @@ static inline void copy_frags(struct sk_buff *skb,
memcpy(&ssi->frags[1], &gl->frags[1], n * sizeof(skb_frag_t));
/* get a reference to the last page, we don't own it */
- get_page(gl->frags[n].page);
+ get_page(gl->frags[n].page.p); /* XXX */
}
/**
@@ -1482,7 +1482,7 @@ static void t4_pktgl_free(const struct pkt_gl *gl)
const skb_frag_t *p;
for (p = gl->frags, n = gl->nfrags - 1; n--; p++)
- put_page(p->page);
+ put_page(p->page.p); /* XXX */
}
/*
@@ -1635,7 +1635,7 @@ static void restore_rx_bufs(const struct pkt_gl *si, struct sge_fl *q,
else
q->cidx--;
d = &q->sdesc[q->cidx];
- d->page = si->frags[frags].page;
+ d->page = si->frags[frags].page.p; /* XXX */
d->dma_addr |= RX_UNMAPPED_BUF;
q->avail++;
}
@@ -1717,7 +1717,7 @@ static int process_responses(struct sge_rspq *q, int budget)
for (frags = 0, fp = si.frags; ; frags++, fp++) {
rsd = &rxq->fl.sdesc[rxq->fl.cidx];
bufsz = get_buf_size(rsd);
- fp->page = rsd->page;
+ fp->page.p = rsd->page; /* XXX */
fp->page_offset = q->offset;
fp->size = min(bufsz, len);
len -= fp->size;
@@ -1734,8 +1734,8 @@ static int process_responses(struct sge_rspq *q, int budget)
get_buf_addr(rsd),
fp->size, DMA_FROM_DEVICE);
- si.va = page_address(si.frags[0].page) +
- si.frags[0].page_offset;
+ si.va = page_address(si.frags[0].page.p) +
+ si.frags[0].page_offset; /* XXX */
prefetch(si.va);
diff --git a/drivers/net/cxgb4vf/sge.c b/drivers/net/cxgb4vf/sge.c
index f4c4480..0a0dda1 100644
--- a/drivers/net/cxgb4vf/sge.c
+++ b/drivers/net/cxgb4vf/sge.c
@@ -1397,7 +1397,7 @@ struct sk_buff *t4vf_pktgl_to_skb(const struct pkt_gl *gl,
skb_copy_to_linear_data(skb, gl->va, pull_len);
ssi = skb_shinfo(skb);
- skb_frag_set_page(skb, 0, gl->frags[0].page);
+ skb_frag_set_page(skb, 0, gl->frags[0].page.p); /* XXX */
ssi->frags[0].page_offset = gl->frags[0].page_offset + pull_len;
ssi->frags[0].size = gl->frags[0].size - pull_len;
if (gl->nfrags > 1)
@@ -1410,7 +1410,7 @@ struct sk_buff *t4vf_pktgl_to_skb(const struct pkt_gl *gl,
skb->truesize += skb->data_len;
/* Get a reference for the last page, we don't own it */
- get_page(gl->frags[gl->nfrags - 1].page);
+ get_page(gl->frags[gl->nfrags - 1].page.p); /* XXX */
}
out:
@@ -1430,7 +1430,7 @@ void t4vf_pktgl_free(const struct pkt_gl *gl)
frag = gl->nfrags - 1;
while (frag--)
- put_page(gl->frags[frag].page);
+ put_page(gl->frags[frag].page.p); /* XXX */
}
/**
@@ -1450,7 +1450,7 @@ static inline void copy_frags(struct sk_buff *skb,
unsigned int n;
/* usually there's just one frag */
- skb_frag_set_page(skb, 0, gl->frags[0].page);
+ skb_frag_set_page(skb, 0, gl->frags[0].page.p); /* XXX */
si->frags[0].page_offset = gl->frags[0].page_offset + offset;
si->frags[0].size = gl->frags[0].size - offset;
si->nr_frags = gl->nfrags;
@@ -1460,7 +1460,7 @@ static inline void copy_frags(struct sk_buff *skb,
memcpy(&si->frags[1], &gl->frags[1], n * sizeof(skb_frag_t));
/* get a reference to the last page, we don't own it */
- get_page(gl->frags[n].page);
+ get_page(gl->frags[n].page.p); /* XXX */
}
/**
@@ -1633,7 +1633,7 @@ static void restore_rx_bufs(const struct pkt_gl *gl, struct sge_fl *fl,
else
fl->cidx--;
sdesc = &fl->sdesc[fl->cidx];
- sdesc->page = gl->frags[frags].page;
+ sdesc->page = gl->frags[frags].page.p; /* XXX */
sdesc->dma_addr |= RX_UNMAPPED_BUF;
fl->avail++;
}
@@ -1721,7 +1721,7 @@ int process_responses(struct sge_rspq *rspq, int budget)
BUG_ON(rxq->fl.avail == 0);
sdesc = &rxq->fl.sdesc[rxq->fl.cidx];
bufsz = get_buf_size(sdesc);
- fp->page = sdesc->page;
+ fp->page.p = sdesc->page; /* XXX */
fp->page_offset = rspq->offset;
fp->size = min(bufsz, len);
len -= fp->size;
@@ -1739,8 +1739,8 @@ int process_responses(struct sge_rspq *rspq, int budget)
dma_sync_single_for_cpu(rspq->adapter->pdev_dev,
get_buf_addr(sdesc),
fp->size, DMA_FROM_DEVICE);
- gl.va = (page_address(gl.frags[0].page) +
- gl.frags[0].page_offset);
+ gl.va = (page_address(gl.frags[0].page.p) +
+ gl.frags[0].page_offset); /* XXX */
prefetch(gl.va);
/*
diff --git a/drivers/net/mlx4/en_rx.c b/drivers/net/mlx4/en_rx.c
index 21a89e0..c5d01ce 100644
--- a/drivers/net/mlx4/en_rx.c
+++ b/drivers/net/mlx4/en_rx.c
@@ -418,7 +418,7 @@ static int mlx4_en_complete_rx_desc(struct mlx4_en_priv *priv,
break;
/* Save page reference in skb */
- __skb_frag_set_page(&skb_frags_rx[nr], skb_frags[nr].page);
+ __skb_frag_set_page(&skb_frags_rx[nr], skb_frags[nr].page.p); /* XXX */
skb_frags_rx[nr].size = skb_frags[nr].size;
skb_frags_rx[nr].page_offset = skb_frags[nr].page_offset;
dma = be64_to_cpu(rx_desc->data[nr].addr);
diff --git a/drivers/scsi/cxgbi/libcxgbi.c b/drivers/scsi/cxgbi/libcxgbi.c
index 949ee48..8d16a74 100644
--- a/drivers/scsi/cxgbi/libcxgbi.c
+++ b/drivers/scsi/cxgbi/libcxgbi.c
@@ -1823,7 +1823,7 @@ static int sgl_read_to_frags(struct scatterlist *sg, unsigned int sgoffset,
return -EINVAL;
}
- frags[i].page = page;
+ frags[i].page.p = page;
frags[i].page_offset = sg->offset + sgoffset;
frags[i].size = copy;
i++;
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index bc6bd24..9818fe2 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -135,8 +135,17 @@ struct sk_buff;
typedef struct skb_frag_struct skb_frag_t;
+struct skb_frag_destructor {
+ atomic_t ref;
+ int (*destroy)(void *data);
+ void *data;
+};
+
struct skb_frag_struct {
- struct page *page;
+ struct {
+ struct page *p;
+ struct skb_frag_destructor *destructor;
+ } page;
#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536)
__u32 page_offset;
__u32 size;
@@ -1129,7 +1138,8 @@ static inline void __skb_fill_page_desc(struct sk_buff *skb, int i,
{
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
- frag->page = page;
+ frag->page.p = page;
+ frag->page.destructor = NULL;
frag->page_offset = off;
frag->size = size;
}
@@ -1648,7 +1658,7 @@ static inline void netdev_free_page(struct net_device *dev, struct page *page)
*/
static inline struct page *__skb_frag_page(const skb_frag_t *frag)
{
- return frag->page;
+ return frag->page.p;
}
/**
@@ -1659,9 +1669,12 @@ static inline struct page *__skb_frag_page(const skb_frag_t *frag)
*/
static inline const struct page *skb_frag_page(const skb_frag_t *frag)
{
- return frag->page;
+ return frag->page.p;
}
+extern void skb_frag_destructor_ref(struct skb_frag_destructor *destroy);
+extern void skb_frag_destructor_unref(struct skb_frag_destructor *destroy);
+
/**
* __skb_frag_ref - take an addition reference on a paged fragment.
* @frag: the paged fragment
@@ -1670,6 +1683,10 @@ static inline const struct page *skb_frag_page(const skb_frag_t *frag)
*/
static inline void __skb_frag_ref(skb_frag_t *frag)
{
+ if (unlikely(frag->page.destructor)) {
+ skb_frag_destructor_ref(frag->page.destructor);
+ return;
+ }
get_page(__skb_frag_page(frag));
}
@@ -1693,6 +1710,10 @@ static inline void skb_frag_ref(struct sk_buff *skb, int f)
*/
static inline void __skb_frag_unref(skb_frag_t *frag)
{
+ if (unlikely(frag->page.destructor)) {
+ skb_frag_destructor_unref(frag->page.destructor);
+ return;
+ }
put_page(__skb_frag_page(frag));
}
@@ -1745,7 +1766,7 @@ static inline void *skb_frag_address_safe(const skb_frag_t *frag)
*/
static inline void __skb_frag_set_page(skb_frag_t *frag, struct page *page)
{
- frag->page = page;
+ frag->page.p = page;
__skb_frag_ref(frag);
}
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 2133600..bdc6f6e 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -292,6 +292,23 @@ struct sk_buff *dev_alloc_skb(unsigned int length)
}
EXPORT_SYMBOL(dev_alloc_skb);
+void skb_frag_destructor_ref(struct skb_frag_destructor *destroy)
+{
+ BUG_ON(destroy == NULL);
+ atomic_inc(&destroy->ref);
+}
+EXPORT_SYMBOL(skb_frag_destructor_ref);
+
+void skb_frag_destructor_unref(struct skb_frag_destructor *destroy)
+{
+ if (destroy == NULL)
+ return;
+
+ if (atomic_dec_and_test(&destroy->ref))
+ destroy->destroy(destroy->data);
+}
+EXPORT_SYMBOL(skb_frag_destructor_unref);
+
static void skb_drop_list(struct sk_buff **listp)
{
struct sk_buff *list = *listp;
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 10/13] net: add paged frag destructor to skb_fill_page_desc()
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (8 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 09/13] net: add support for per-paged-fragment destructors Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 19:58 ` Michał Mirosław
2011-07-22 13:17 ` [PATCH 11/13] net: only allow paged fragments with the same destructor to be coalesced Ian Campbell
` (3 subsequent siblings)
13 siblings, 1 reply; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, David S. Miller, James E.J. Bottomley,
Alexey Kuznetsov, Pekka Savola (ipv6),
James Morris, Hideaki YOSHIFUJI, Patrick McHardy, linux-rdma,
linux-s390, linux-scsi, devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: "James E.J. Bottomley" <JBottomley@parallels.com>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: "Pekka Savola (ipv6)" <pekkas@netcore.fi>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: netdev@vger.kernel.org
Cc: linux-rdma@vger.kernel.org
Cc: linux-s390@vger.kernel.org
Cc: linux-scsi@vger.kernel.org
Cc: devel@open-fcoe.org
---
drivers/block/aoe/aoecmd.c | 6 ++++--
drivers/infiniband/ulp/ipoib/ipoib_cm.c | 4 ++--
drivers/infiniband/ulp/ipoib/ipoib_ib.c | 2 +-
drivers/net/bnx2.c | 3 ++-
drivers/net/bnx2x/bnx2x_cmn.c | 2 +-
drivers/net/cxgb3/sge.c | 4 ++--
drivers/net/e1000/e1000_main.c | 25 +++++++++++++++++--------
drivers/net/e1000e/netdev.c | 28 +++++++++++++++++++---------
drivers/net/ftmac100.c | 2 +-
drivers/net/igb/igb_main.c | 5 ++---
drivers/net/igbvf/netdev.c | 5 ++---
drivers/net/ixgbe/ixgbe_main.c | 2 +-
drivers/net/ixgbevf/ixgbevf_main.c | 2 +-
drivers/net/qlge/qlge_main.c | 24 +++++++++++-------------
drivers/net/sky2.c | 2 +-
drivers/s390/net/qeth_core_main.c | 7 ++++---
drivers/scsi/cxgbi/libcxgbi.c | 8 ++++----
drivers/scsi/fcoe/fcoe_transport.c | 2 +-
drivers/scsi/libfc/fc_fcp.c | 5 ++++-
drivers/target/tcm_fc/tfc_io.c | 5 ++++-
include/linux/skbuff.h | 12 ++++++++----
net/core/skbuff.c | 4 ++--
net/core/sock.c | 2 +-
net/ipv4/ip_output.c | 7 ++++---
net/ipv4/tcp.c | 7 ++++---
net/ipv6/ip6_output.c | 7 +++++--
net/packet/af_packet.c | 2 +-
27 files changed, 109 insertions(+), 75 deletions(-)
diff --git a/drivers/block/aoe/aoecmd.c b/drivers/block/aoe/aoecmd.c
index de0435e..05c13a0 100644
--- a/drivers/block/aoe/aoecmd.c
+++ b/drivers/block/aoe/aoecmd.c
@@ -252,7 +252,8 @@ aoecmd_ata_rw(struct aoedev *d)
ah->lba3 |= 0xe0; /* LBA bit + obsolete 0xa0 */
}
if (bio_data_dir(buf->bio) == WRITE) {
- skb_fill_page_desc(skb, 0, bv->bv_page, buf->bv_off, bcnt);
+ skb_fill_page_desc(skb, 0, bv->bv_page, NULL, buf->bv_off,
+ bcnt);
ah->aflags |= AOEAFL_WRITE;
skb->len += bcnt;
skb->data_len = bcnt;
@@ -369,7 +370,8 @@ resend(struct aoedev *d, struct aoetgt *t, struct frame *f)
ah->scnt = n >> 9;
if (ah->aflags & AOEAFL_WRITE) {
skb_fill_page_desc(skb, 0, virt_to_page(f->bufaddr),
- offset_in_page(f->bufaddr), n);
+ NULL, offset_in_page(f->bufaddr),
+ n);
skb->len = sizeof *h + sizeof *ah + n;
skb->data_len = n;
}
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
index 1f20e40..0d12366 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
@@ -167,7 +167,7 @@ static struct sk_buff *ipoib_cm_alloc_rx_skb(struct net_device *dev,
if (!page)
goto partial_error;
- skb_fill_page_desc(skb, i, page, 0, PAGE_SIZE);
+ skb_fill_page_desc(skb, i, page, NULL, 0, PAGE_SIZE);
mapping[i + 1] = ib_dma_map_page(priv->ca,
__skb_frag_page(&skb_shinfo(skb)->frags[i]),
@@ -539,7 +539,7 @@ static void skb_put_frags(struct sk_buff *skb, unsigned int hdr_space,
if (length == 0) {
/* don't need this page */
skb_fill_page_desc(toskb, i, __skb_frag_page(frag),
- 0, PAGE_SIZE);/* XXX */
+ NULL, 0, PAGE_SIZE);/* XXX */
--skb_shinfo(skb)->nr_frags;
} else {
size = min(length, (unsigned) PAGE_SIZE);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index f6ef6c2..96657c2 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -180,7 +180,7 @@ static struct sk_buff *ipoib_alloc_rx_skb(struct net_device *dev, int id)
struct page *page = alloc_page(GFP_ATOMIC);
if (!page)
goto partial_error;
- skb_fill_page_desc(skb, 0, page, 0, PAGE_SIZE);
+ skb_fill_page_desc(skb, 0, page, NULL, 0, PAGE_SIZE);
mapping[1] =
ib_dma_map_page(priv->ca,
__skb_frag_page(&skb_shinfo(skb)->frags[0]),
diff --git a/drivers/net/bnx2.c b/drivers/net/bnx2.c
index ff90b13..101bac8 100644
--- a/drivers/net/bnx2.c
+++ b/drivers/net/bnx2.c
@@ -3014,7 +3014,8 @@ bnx2_rx_skb(struct bnx2 *bp, struct bnx2_rx_ring_info *rxr, struct sk_buff *skb,
if (i == pages - 1)
frag_len -= 4;
- skb_fill_page_desc(skb, i, rx_pg->page, 0, frag_len);
+ skb_fill_page_desc(skb, i, rx_pg->page, NULL, 0,
+ frag_len);
rx_pg->page = NULL;
err = bnx2_alloc_rx_page(bp, rxr,
diff --git a/drivers/net/bnx2x/bnx2x_cmn.c b/drivers/net/bnx2x/bnx2x_cmn.c
index dee09d7..5521077 100644
--- a/drivers/net/bnx2x/bnx2x_cmn.c
+++ b/drivers/net/bnx2x/bnx2x_cmn.c
@@ -394,7 +394,7 @@ static int bnx2x_fill_frag_skb(struct bnx2x *bp, struct bnx2x_fastpath *fp,
SGE_PAGE_SIZE*PAGES_PER_SGE, DMA_FROM_DEVICE);
/* Add one frag and update the appropriate fields in the skb */
- skb_fill_page_desc(skb, j, old_rx_pg.page, 0, frag_len);
+ skb_fill_page_desc(skb, j, old_rx_pg.page, NULL, 0, frag_len);
skb->data_len += frag_len;
skb->truesize += frag_len;
diff --git a/drivers/net/cxgb3/sge.c b/drivers/net/cxgb3/sge.c
index 3f73a5c..6e90ca6 100644
--- a/drivers/net/cxgb3/sge.c
+++ b/drivers/net/cxgb3/sge.c
@@ -889,7 +889,7 @@ recycle:
if (!skb) {
__skb_put(newskb, SGE_RX_PULL_LEN);
memcpy(newskb->data, sd->pg_chunk.va, SGE_RX_PULL_LEN);
- skb_fill_page_desc(newskb, 0, sd->pg_chunk.page,
+ skb_fill_page_desc(newskb, 0, sd->pg_chunk.page, NULL,
sd->pg_chunk.offset + SGE_RX_PULL_LEN,
len - SGE_RX_PULL_LEN);
newskb->len = len;
@@ -897,7 +897,7 @@ recycle:
newskb->truesize += newskb->data_len;
} else {
skb_fill_page_desc(newskb, skb_shinfo(newskb)->nr_frags,
- sd->pg_chunk.page,
+ sd->pg_chunk.page, NULL,
sd->pg_chunk.offset, len);
newskb->len += len;
newskb->data_len += len;
diff --git a/drivers/net/e1000/e1000_main.c b/drivers/net/e1000/e1000_main.c
index e902cd0..93a6898 100644
--- a/drivers/net/e1000/e1000_main.c
+++ b/drivers/net/e1000/e1000_main.c
@@ -3808,13 +3808,17 @@ static bool e1000_clean_jumbo_rx_irq(struct e1000_adapter *adapter,
if (!rxtop) {
/* this is the beginning of a chain */
rxtop = skb;
- skb_fill_page_desc(rxtop, 0, buffer_info->page,
- 0, length);
+ skb_fill_page_desc(rxtop, 0,
+ buffer_info->page, NULL,
+ 0, length);
} else {
/* this is the middle of a chain */
skb_fill_page_desc(rxtop,
- skb_shinfo(rxtop)->nr_frags,
- buffer_info->page, 0, length);
+ skb_shinfo(rxtop)->nr_frags,
+ buffer_info->page,
+ NULL,
+ 0,
+ length);
/* re-use the skb, only consumed the page */
buffer_info->skb = skb;
}
@@ -3824,8 +3828,11 @@ static bool e1000_clean_jumbo_rx_irq(struct e1000_adapter *adapter,
if (rxtop) {
/* end of the chain */
skb_fill_page_desc(rxtop,
- skb_shinfo(rxtop)->nr_frags,
- buffer_info->page, 0, length);
+ skb_shinfo(rxtop)->nr_frags,
+ buffer_info->page,
+ NULL,
+ 0,
+ length);
/* re-use the current skb, we only consumed the
* page */
buffer_info->skb = skb;
@@ -3848,8 +3855,10 @@ static bool e1000_clean_jumbo_rx_irq(struct e1000_adapter *adapter,
skb_put(skb, length);
} else {
skb_fill_page_desc(skb, 0,
- buffer_info->page, 0,
- length);
+ buffer_info->page,
+ NULL,
+ 0,
+ length);
e1000_consume_page(buffer_info, skb,
length);
}
diff --git a/drivers/net/e1000e/netdev.c b/drivers/net/e1000e/netdev.c
index 30f8a5c..28da1a7 100644
--- a/drivers/net/e1000e/netdev.c
+++ b/drivers/net/e1000e/netdev.c
@@ -1196,7 +1196,8 @@ static bool e1000_clean_rx_irq_ps(struct e1000_adapter *adapter,
dma_unmap_page(&pdev->dev, ps_page->dma, PAGE_SIZE,
DMA_FROM_DEVICE);
ps_page->dma = 0;
- skb_fill_page_desc(skb, j, ps_page->page, 0, length);
+ skb_fill_page_desc(skb, j, ps_page->page, NULL, 0,
+ length);
ps_page->page = NULL;
skb->len += length;
skb->data_len += length;
@@ -1336,13 +1337,17 @@ static bool e1000_clean_jumbo_rx_irq(struct e1000_adapter *adapter,
if (!rxtop) {
/* this is the beginning of a chain */
rxtop = skb;
- skb_fill_page_desc(rxtop, 0, buffer_info->page,
- 0, length);
+ skb_fill_page_desc(rxtop, 0,
+ buffer_info->page, NULL,
+ 0, length);
} else {
/* this is the middle of a chain */
skb_fill_page_desc(rxtop,
- skb_shinfo(rxtop)->nr_frags,
- buffer_info->page, 0, length);
+ skb_shinfo(rxtop)->nr_frags,
+ buffer_info->page,
+ NULL,
+ 0,
+ length);
/* re-use the skb, only consumed the page */
buffer_info->skb = skb;
}
@@ -1352,8 +1357,11 @@ static bool e1000_clean_jumbo_rx_irq(struct e1000_adapter *adapter,
if (rxtop) {
/* end of the chain */
skb_fill_page_desc(rxtop,
- skb_shinfo(rxtop)->nr_frags,
- buffer_info->page, 0, length);
+ skb_shinfo(rxtop)->nr_frags,
+ buffer_info->page,
+ NULL,
+ 0,
+ length);
/* re-use the current skb, we only consumed the
* page */
buffer_info->skb = skb;
@@ -1377,8 +1385,10 @@ static bool e1000_clean_jumbo_rx_irq(struct e1000_adapter *adapter,
skb_put(skb, length);
} else {
skb_fill_page_desc(skb, 0,
- buffer_info->page, 0,
- length);
+ buffer_info->page,
+ NULL,
+ 0,
+ length);
e1000_consume_page(buffer_info, skb,
length);
}
diff --git a/drivers/net/ftmac100.c b/drivers/net/ftmac100.c
index 9bd7746..d8497dd 100644
--- a/drivers/net/ftmac100.c
+++ b/drivers/net/ftmac100.c
@@ -436,7 +436,7 @@ static bool ftmac100_rx_packet(struct ftmac100 *priv, int *processed)
length = ftmac100_rxdes_frame_length(rxdes);
page = ftmac100_rxdes_get_page(rxdes);
- skb_fill_page_desc(skb, 0, page, 0, length);
+ skb_fill_page_desc(skb, 0, page, NULL, 0, length);
skb->len += length;
skb->data_len += length;
skb->truesize += length;
diff --git a/drivers/net/igb/igb_main.c b/drivers/net/igb/igb_main.c
index 17f94f4..ac48baf 100644
--- a/drivers/net/igb/igb_main.c
+++ b/drivers/net/igb/igb_main.c
@@ -5835,9 +5835,8 @@ static bool igb_clean_rx_irq_adv(struct igb_q_vector *q_vector,
buffer_info->page_dma = 0;
skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,
- buffer_info->page,
- buffer_info->page_offset,
- length);
+ buffer_info->page, NULL,
+ buffer_info->page_offset, length);
if ((page_count(buffer_info->page) != 1) ||
(page_to_nid(buffer_info->page) != current_node))
diff --git a/drivers/net/igbvf/netdev.c b/drivers/net/igbvf/netdev.c
index 3f6655f..9360f0a 100644
--- a/drivers/net/igbvf/netdev.c
+++ b/drivers/net/igbvf/netdev.c
@@ -300,9 +300,8 @@ static bool igbvf_clean_rx_irq(struct igbvf_adapter *adapter,
buffer_info->page_dma = 0;
skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,
- buffer_info->page,
- buffer_info->page_offset,
- length);
+ buffer_info->page, NULL,
+ buffer_info->page_offset, length);
if ((adapter->rx_buffer_len > (PAGE_SIZE / 2)) ||
(page_count(buffer_info->page) != 1))
diff --git a/drivers/net/ixgbe/ixgbe_main.c b/drivers/net/ixgbe/ixgbe_main.c
index 307cf06..d481bea 100644
--- a/drivers/net/ixgbe/ixgbe_main.c
+++ b/drivers/net/ixgbe/ixgbe_main.c
@@ -1409,7 +1409,7 @@ static void ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector,
DMA_FROM_DEVICE);
rx_buffer_info->page_dma = 0;
skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,
- rx_buffer_info->page,
+ rx_buffer_info->page, NULL,
rx_buffer_info->page_offset,
upper_len);
diff --git a/drivers/net/ixgbevf/ixgbevf_main.c b/drivers/net/ixgbevf/ixgbevf_main.c
index ad05ad9..347ae21 100644
--- a/drivers/net/ixgbevf/ixgbevf_main.c
+++ b/drivers/net/ixgbevf/ixgbevf_main.c
@@ -508,7 +508,7 @@ static bool ixgbevf_clean_rx_irq(struct ixgbevf_q_vector *q_vector,
PAGE_SIZE / 2, DMA_FROM_DEVICE);
rx_buffer_info->page_dma = 0;
skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,
- rx_buffer_info->page,
+ rx_buffer_info->page, NULL,
rx_buffer_info->page_offset,
upper_len);
diff --git a/drivers/net/qlge/qlge_main.c b/drivers/net/qlge/qlge_main.c
index cc04e5b..d2a420c 100644
--- a/drivers/net/qlge/qlge_main.c
+++ b/drivers/net/qlge/qlge_main.c
@@ -1558,9 +1558,9 @@ static void ql_process_mac_rx_page(struct ql_adapter *qdev,
netif_printk(qdev, rx_status, KERN_DEBUG, qdev->ndev,
"%d bytes of headers and data in large. Chain page to new skb and pull tail.\n",
length);
- skb_fill_page_desc(skb, 0, lbq_desc->p.pg_chunk.page,
- lbq_desc->p.pg_chunk.offset+ETH_HLEN,
- length-ETH_HLEN);
+ skb_fill_page_desc(skb, 0, lbq_desc->p.pg_chunk.page, NULL,
+ lbq_desc->p.pg_chunk.offset + ETH_HLEN,
+ length - ETH_HLEN);
skb->len += length-ETH_HLEN;
skb->data_len += length-ETH_HLEN;
skb->truesize += length-ETH_HLEN;
@@ -1838,8 +1838,8 @@ static struct sk_buff *ql_build_rx_skb(struct ql_adapter *qdev,
"Chaining page at offset = %d, for %d bytes to skb.\n",
lbq_desc->p.pg_chunk.offset, length);
skb_fill_page_desc(skb, 0, lbq_desc->p.pg_chunk.page,
- lbq_desc->p.pg_chunk.offset,
- length);
+ NULL, lbq_desc->p.pg_chunk.offset,
+ length);
skb->len += length;
skb->data_len += length;
skb->truesize += length;
@@ -1865,10 +1865,9 @@ static struct sk_buff *ql_build_rx_skb(struct ql_adapter *qdev,
netif_printk(qdev, rx_status, KERN_DEBUG, qdev->ndev,
"%d bytes of headers and data in large. Chain page to new skb and pull tail.\n",
length);
- skb_fill_page_desc(skb, 0,
- lbq_desc->p.pg_chunk.page,
- lbq_desc->p.pg_chunk.offset,
- length);
+ skb_fill_page_desc(skb, 0, lbq_desc->p.pg_chunk.page,
+ NULL, lbq_desc->p.pg_chunk.offset,
+ length);
skb->len += length;
skb->data_len += length;
skb->truesize += length;
@@ -1920,10 +1919,9 @@ static struct sk_buff *ql_build_rx_skb(struct ql_adapter *qdev,
netif_printk(qdev, rx_status, KERN_DEBUG, qdev->ndev,
"Adding page %d to skb for %d bytes.\n",
i, size);
- skb_fill_page_desc(skb, i,
- lbq_desc->p.pg_chunk.page,
- lbq_desc->p.pg_chunk.offset,
- size);
+ skb_fill_page_desc(skb, i, lbq_desc->p.pg_chunk.page,
+ NULL, lbq_desc->p.pg_chunk.offset,
+ size);
skb->len += size;
skb->data_len += size;
skb->truesize += size;
diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index e1cf142..80c88be 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -1388,7 +1388,7 @@ static struct sk_buff *sky2_rx_alloc(struct sky2_port *sky2)
if (!page)
goto free_partial;
- skb_fill_page_desc(skb, i, page, 0, PAGE_SIZE);
+ skb_fill_page_desc(skb, i, page, NULL, 0, PAGE_SIZE);
}
return skb;
diff --git a/drivers/s390/net/qeth_core_main.c b/drivers/s390/net/qeth_core_main.c
index dd08f7b..c50a5b7 100644
--- a/drivers/s390/net/qeth_core_main.c
+++ b/drivers/s390/net/qeth_core_main.c
@@ -4163,8 +4163,8 @@ static inline int qeth_create_skb_frag(struct qdio_buffer_element *element,
} else {
get_page(page);
memcpy(skb_put(*pskb, 64), element->addr + offset, 64);
- skb_fill_page_desc(*pskb, *pfrag, page, offset + 64,
- data_len - 64);
+ skb_fill_page_desc(*pskb, *pfrag, page, NULL,
+ offset + 64, data_len - 64);
(*pskb)->data_len += data_len - 64;
(*pskb)->len += data_len - 64;
(*pskb)->truesize += data_len - 64;
@@ -4172,7 +4172,8 @@ static inline int qeth_create_skb_frag(struct qdio_buffer_element *element,
}
} else {
get_page(page);
- skb_fill_page_desc(*pskb, *pfrag, page, offset, data_len);
+ skb_fill_page_desc(*pskb, *pfrag, page, NULL, offset,
+ data_len);
(*pskb)->data_len += data_len;
(*pskb)->len += data_len;
(*pskb)->truesize += data_len;
diff --git a/drivers/scsi/cxgbi/libcxgbi.c b/drivers/scsi/cxgbi/libcxgbi.c
index 8d16a74..94185f9 100644
--- a/drivers/scsi/cxgbi/libcxgbi.c
+++ b/drivers/scsi/cxgbi/libcxgbi.c
@@ -1977,8 +1977,8 @@ int cxgbi_conn_init_pdu(struct iscsi_task *task, unsigned int offset,
pg = virt_to_page(task->data);
get_page(pg);
- skb_fill_page_desc(skb, 0, pg, offset_in_page(task->data),
- count);
+ skb_fill_page_desc(skb, 0, pg, NULL,
+ offset_in_page(task->data), count);
skb->len += count;
skb->data_len += count;
skb->truesize += count;
@@ -1987,8 +1987,8 @@ int cxgbi_conn_init_pdu(struct iscsi_task *task, unsigned int offset,
if (padlen) {
i = skb_shinfo(skb)->nr_frags;
skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,
- virt_to_page(padding), offset_in_page(padding),
- padlen);
+ virt_to_page(padding), NULL,
+ offset_in_page(padding), padlen);
skb->data_len += padlen;
skb->truesize += padlen;
diff --git a/drivers/scsi/fcoe/fcoe_transport.c b/drivers/scsi/fcoe/fcoe_transport.c
index 40243ce..e957823 100644
--- a/drivers/scsi/fcoe/fcoe_transport.c
+++ b/drivers/scsi/fcoe/fcoe_transport.c
@@ -266,7 +266,7 @@ int fcoe_get_paged_crc_eof(struct sk_buff *skb, int tlen,
}
get_page(page);
- skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags, page,
+ skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags, page, NULL,
fps->crc_eof_offset, tlen);
skb->len += tlen;
skb->data_len += tlen;
diff --git a/drivers/scsi/libfc/fc_fcp.c b/drivers/scsi/libfc/fc_fcp.c
index 9cd2149..6689515 100644
--- a/drivers/scsi/libfc/fc_fcp.c
+++ b/drivers/scsi/libfc/fc_fcp.c
@@ -640,7 +640,10 @@ static int fc_fcp_send_data(struct fc_fcp_pkt *fsp, struct fc_seq *seq,
get_page(page);
skb_fill_page_desc(fp_skb(fp),
skb_shinfo(fp_skb(fp))->nr_frags,
- page, off & ~PAGE_MASK, sg_bytes);
+ page,
+ NULL,
+ off & ~PAGE_MASK,
+ sg_bytes);
fp_skb(fp)->data_len += sg_bytes;
fr_len(fp) += sg_bytes;
fp_skb(fp)->truesize += PAGE_SIZE;
diff --git a/drivers/target/tcm_fc/tfc_io.c b/drivers/target/tcm_fc/tfc_io.c
index 8c4a240..53dc0ff 100644
--- a/drivers/target/tcm_fc/tfc_io.c
+++ b/drivers/target/tcm_fc/tfc_io.c
@@ -164,7 +164,10 @@ int ft_queue_data_in(struct se_cmd *se_cmd)
get_page(page);
skb_fill_page_desc(fp_skb(fp),
skb_shinfo(fp_skb(fp))->nr_frags,
- page, off_in_page, tlen);
+ page,
+ NULL,
+ off_in_page,
+ tlen);
fr_len(fp) += tlen;
fp_skb(fp)->data_len += tlen;
fp_skb(fp)->truesize +=
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 9818fe2..faee8d3 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -1134,12 +1134,14 @@ static inline int skb_pagelen(const struct sk_buff *skb)
* Does not take any additional reference on the fragment.
*/
static inline void __skb_fill_page_desc(struct sk_buff *skb, int i,
- struct page *page, int off, int size)
+ struct page *page,
+ struct skb_frag_destructor *destroy,
+ int off, int size)
{
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
frag->page.p = page;
- frag->page.destructor = NULL;
+ frag->page.destructor = destroy;
frag->page_offset = off;
frag->size = size;
}
@@ -1159,9 +1161,11 @@ static inline void __skb_fill_page_desc(struct sk_buff *skb, int i,
* Does not take any additional reference on the fragment.
*/
static inline void skb_fill_page_desc(struct sk_buff *skb, int i,
- struct page *page, int off, int size)
+ struct page *page,
+ struct skb_frag_destructor *destroy,
+ int off, int size)
{
- __skb_fill_page_desc(skb, i, page, off, size);
+ __skb_fill_page_desc(skb, i, page, destroy, off, size);
skb_shinfo(skb)->nr_frags = i + 1;
}
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index bdc6f6e..101c9cc 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -263,7 +263,7 @@ EXPORT_SYMBOL(__netdev_alloc_skb);
void skb_add_rx_frag(struct sk_buff *skb, int i, struct page *page, int off,
int size)
{
- skb_fill_page_desc(skb, i, page, off, size);
+ skb_fill_page_desc(skb, i, page, NULL, off, size);
skb->len += size;
skb->data_len += size;
skb->truesize += size;
@@ -2454,7 +2454,7 @@ int skb_append_datato_frags(struct sock *sk, struct sk_buff *skb,
return -ENOMEM;
/* initialize the next frag */
- skb_fill_page_desc(skb, frg_cnt, page, 0, 0);
+ skb_fill_page_desc(skb, frg_cnt, page, NULL, 0, 0);
skb->truesize += PAGE_SIZE;
atomic_add(PAGE_SIZE, &sk->sk_wmem_alloc);
diff --git a/net/core/sock.c b/net/core/sock.c
index 0fb2160..be55676 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1540,7 +1540,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
}
__skb_fill_page_desc(skb, i,
- page, 0,
+ page, NULL, 0,
(data_len >= PAGE_SIZE ?
PAGE_SIZE :
data_len));
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 3aa3c91..5946de2 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -987,7 +987,8 @@ alloc_new_skb:
err = -EMSGSIZE;
goto error;
}
- skb_fill_page_desc(skb, i, page, off, 0);
+ skb_fill_page_desc(skb, i, page,
+ NULL, off, 0);
frag = &skb_shinfo(skb)->frags[i];
__skb_frag_ref(frag);
}
@@ -1003,7 +1004,7 @@ alloc_new_skb:
cork->off = 0;
/* XXX no ref ? */
- skb_fill_page_desc(skb, i, page, 0, 0);
+ skb_fill_page_desc(skb, i, page, NULL, 0, 0);
frag = &skb_shinfo(skb)->frags[i];
} else {
err = -EMSGSIZE;
@@ -1227,7 +1228,7 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page,
skb_shinfo(skb)->frags[i-1].size += len;
} else if (i < MAX_SKB_FRAGS) {
get_page(page);
- skb_fill_page_desc(skb, i, page, offset, len);
+ skb_fill_page_desc(skb, i, page, NULL, offset, len);
} else {
err = -EMSGSIZE;
goto error;
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index ac47ab3..2f0e985 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -804,7 +804,7 @@ new_segment:
copy = size;
i = skb_shinfo(skb)->nr_frags;
- can_coalesce = skb_can_coalesce(skb, i, page, offset);
+ can_coalesce = skb_can_coalesce(skb, i, page, NULL, offset);
if (!can_coalesce && i >= MAX_SKB_FRAGS) {
tcp_mark_push(tp, skb);
goto new_segment;
@@ -816,7 +816,7 @@ new_segment:
skb_shinfo(skb)->frags[i - 1].size += copy;
} else {
get_page(page);
- skb_fill_page_desc(skb, i, page, offset, copy);
+ skb_fill_page_desc(skb, i, page, NULL, offset, copy);
}
skb->len += copy;
@@ -1061,7 +1061,8 @@ new_segment:
skb_shinfo(skb)->frags[i - 1].size +=
copy;
} else {
- skb_fill_page_desc(skb, i, page, off, copy);
+ skb_fill_page_desc(skb, i, page,
+ NULL, off, copy);
if (TCP_PAGE(sk)) {
get_page(page);
} else if (off + copy < PAGE_SIZE) {
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index fdd4f61..c019865 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1446,7 +1446,10 @@ alloc_new_skb:
err = -EMSGSIZE;
goto error;
}
- skb_fill_page_desc(skb, i, page, sk->sk_sndmsg_off, 0);
+ skb_fill_page_desc(skb, i, page,
+ NULL,
+ sk->sk_sndmsg_off,
+ 0);
frag = &skb_shinfo(skb)->frags[i];
__skb_frag_ref(frag);
}
@@ -1462,7 +1465,7 @@ alloc_new_skb:
sk->sk_sndmsg_off = 0;
/* XXX no ref ? */
- skb_fill_page_desc(skb, i, page, 0, 0);
+ skb_fill_page_desc(skb, i, page, NULL, 0, 0);
frag = &skb_shinfo(skb)->frags[i];
} else {
err = -EMSGSIZE;
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index c0c3cda..1053c28 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -960,7 +960,7 @@ static int tpacket_fill_skb(struct packet_sock *po, struct sk_buff *skb,
data += len;
flush_dcache_page(page);
get_page(page);
- skb_fill_page_desc(skb, nr_frags, page, offset, len);
+ skb_fill_page_desc(skb, nr_frags, page, NULL, offset, len);
to_write -= len;
offset = 0;
len_max = PAGE_SIZE;
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 11/13] net: only allow paged fragments with the same destructor to be coalesced.
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (9 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 10/13] net: add paged frag destructor to skb_fill_page_desc() Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 12/13] net: add paged frag destructor support to kernel_sendpage Ian Campbell
` (2 subsequent siblings)
13 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, David S. Miller, Alexey Kuznetsov,
Pekka Savola (ipv6),
James Morris, Hideaki YOSHIFUJI, Patrick McHardy, Eric Dumazet,
Michał Mirosław
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: "Pekka Savola (ipv6)" <pekkas@netcore.fi>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: "Michał Mirosław" <mirq-linux@rere.qmqm.pl>
Cc: netdev@vger.kernel.org
---
include/linux/skbuff.h | 7 +++++--
net/core/skbuff.c | 1 +
net/ipv4/ip_output.c | 2 +-
net/ipv4/tcp.c | 2 +-
4 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index faee8d3..fb64a0e 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -1931,13 +1931,16 @@ static inline int skb_add_data(struct sk_buff *skb,
}
static inline int skb_can_coalesce(struct sk_buff *skb, int i,
- const struct page *page, int off)
+ const struct page *page,
+ const struct skb_frag_destructor *destroy,
+ int off)
{
if (i) {
struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[i - 1];
return page == skb_frag_page(frag) &&
- off == frag->page_offset + frag->size;
+ off == frag->page_offset + frag->size &&
+ frag->page.destructor == destroy;
}
return 0;
}
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 101c9cc..3d89b1a 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -2152,6 +2152,7 @@ int skb_shift(struct sk_buff *tgt, struct sk_buff *skb, int shiftlen)
*/
if (!to ||
!skb_can_coalesce(tgt, to, skb_frag_page(fragfrom),
+ fragfrom->page.destructor,
fragfrom->page_offset)) {
merge = -1;
} else {
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 5946de2..c4326fb 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -1224,7 +1224,7 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page,
i = skb_shinfo(skb)->nr_frags;
if (len > size)
len = size;
- if (skb_can_coalesce(skb, i, page, offset)) {
+ if (skb_can_coalesce(skb, i, page, NULL, offset)) {
skb_shinfo(skb)->frags[i-1].size += len;
} else if (i < MAX_SKB_FRAGS) {
get_page(page);
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 2f0e985..a1a0ccd 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1008,7 +1008,7 @@ new_segment:
struct page *page = TCP_PAGE(sk);
int off = TCP_OFF(sk);
- if (skb_can_coalesce(skb, i, page, off) &&
+ if (skb_can_coalesce(skb, i, page, NULL, off) &&
off != PAGE_SIZE) {
/* We can extend the last page
* fragment. */
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 12/13] net: add paged frag destructor support to kernel_sendpage.
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (10 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 11/13] net: only allow paged fragments with the same destructor to be coalesced Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 13/13] sunrpc: use SKB fragment destructors to delay completion until page is released by network stack Ian Campbell
2011-07-22 14:13 ` [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility David Miller
13 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, David S. Miller, Alexey Kuznetsov,
Pekka Savola (ipv6),
James Morris, Hideaki YOSHIFUJI, Patrick McHardy,
Trond Myklebust, Greg Kroah-Hartman, drbd-user, devel,
cluster-devel, ocfs2-devel, ceph-devel, rds-devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: "Pekka Savola (ipv6)" <pekkas@netcore.fi>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Greg Kroah-Hartman <gregkh@suse.de>
Cc: drbd-user@lists.linbit.com
Cc: devel@driverdev.osuosl.org
Cc: cluster-devel@redhat.com
Cc: ocfs2-devel@oss.oracle.com
Cc: netdev@vger.kernel.org
Cc: ceph-devel@vger.kernel.org
Cc: rds-devel@oss.oracle.com
Cc: linux-nfs@vger.kernel.org
[since v1:
Drop sendpage_destructor and just add an argument to sendpage protocol hooks
]
---
drivers/block/drbd/drbd_main.c | 1 +
drivers/staging/pohmelfs/trans.c | 2 +-
fs/dlm/lowcomms.c | 2 +-
fs/ocfs2/cluster/tcp.c | 1 +
include/linux/net.h | 6 +++++-
include/net/inet_common.h | 4 +++-
include/net/ip.h | 4 +++-
include/net/sock.h | 2 ++
include/net/tcp.h | 4 +++-
net/ceph/messenger.c | 2 +-
net/core/sock.c | 6 +++++-
net/ipv4/af_inet.c | 9 ++++++---
net/ipv4/ip_output.c | 7 ++++---
net/ipv4/tcp.c | 25 ++++++++++++++++---------
net/ipv4/udp.c | 11 ++++++-----
net/ipv4/udp_impl.h | 5 +++--
net/rds/tcp_send.c | 1 +
net/socket.c | 11 +++++++----
net/sunrpc/svcsock.c | 6 +++---
net/sunrpc/xprtsock.c | 2 +-
20 files changed, 73 insertions(+), 38 deletions(-)
diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c
index 0358e55..49c7346 100644
--- a/drivers/block/drbd/drbd_main.c
+++ b/drivers/block/drbd/drbd_main.c
@@ -2584,6 +2584,7 @@ static int _drbd_send_page(struct drbd_conf *mdev, struct page *page,
set_fs(KERNEL_DS);
do {
sent = mdev->data.socket->ops->sendpage(mdev->data.socket, page,
+ NULL,
offset, len,
msg_flags);
if (sent == -EAGAIN) {
diff --git a/drivers/staging/pohmelfs/trans.c b/drivers/staging/pohmelfs/trans.c
index 36a2535..b5d8411 100644
--- a/drivers/staging/pohmelfs/trans.c
+++ b/drivers/staging/pohmelfs/trans.c
@@ -104,7 +104,7 @@ static int netfs_trans_send_pages(struct netfs_trans *t, struct netfs_state *st)
msg.msg_flags = MSG_WAITALL | (attached_pages == 1 ? 0 :
MSG_MORE);
- err = kernel_sendpage(st->socket, page, 0, size, msg.msg_flags);
+ err = kernel_sendpage(st->socket, page, NULL, 0, size, msg.msg_flags);
if (err <= 0) {
printk("%s: %d/%d failed to send transaction page: t: %p, gen: %u, size: %u, err: %d.\n",
__func__, i, t->page_num, t, t->gen, size, err);
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 5e2c71f..64933ff 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -1341,7 +1341,7 @@ static void send_to_sock(struct connection *con)
ret = 0;
if (len) {
- ret = kernel_sendpage(con->sock, e->page, offset, len,
+ ret = kernel_sendpage(con->sock, e->page, NULL, offset, len,
msg_flags);
if (ret == -EAGAIN || ret == 0) {
if (ret == -EAGAIN &&
diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c
index db5ee4b..81366a0 100644
--- a/fs/ocfs2/cluster/tcp.c
+++ b/fs/ocfs2/cluster/tcp.c
@@ -982,6 +982,7 @@ static void o2net_sendpage(struct o2net_sock_container *sc,
mutex_lock(&sc->sc_send_lock);
ret = sc->sc_sock->ops->sendpage(sc->sc_sock,
virt_to_page(kmalloced_virt),
+ NULL,
(long)kmalloced_virt & ~PAGE_MASK,
size, MSG_DONTWAIT);
mutex_unlock(&sc->sc_send_lock);
diff --git a/include/linux/net.h b/include/linux/net.h
index b299230..db562ba 100644
--- a/include/linux/net.h
+++ b/include/linux/net.h
@@ -157,6 +157,7 @@ struct kiocb;
struct sockaddr;
struct msghdr;
struct module;
+struct skb_frag_destructor;
struct proto_ops {
int family;
@@ -203,6 +204,7 @@ struct proto_ops {
int (*mmap) (struct file *file, struct socket *sock,
struct vm_area_struct * vma);
ssize_t (*sendpage) (struct socket *sock, struct page *page,
+ struct skb_frag_destructor *destroy,
int offset, size_t size, int flags);
ssize_t (*splice_read)(struct socket *sock, loff_t *ppos,
struct pipe_inode_info *pipe, size_t len, unsigned int flags);
@@ -273,7 +275,9 @@ extern int kernel_getsockopt(struct socket *sock, int level, int optname,
char *optval, int *optlen);
extern int kernel_setsockopt(struct socket *sock, int level, int optname,
char *optval, unsigned int optlen);
-extern int kernel_sendpage(struct socket *sock, struct page *page, int offset,
+extern int kernel_sendpage(struct socket *sock, struct page *page,
+ struct skb_frag_destructor *destroy,
+ int offset,
size_t size, int flags);
extern int kernel_sock_ioctl(struct socket *sock, int cmd, unsigned long arg);
extern int kernel_sock_shutdown(struct socket *sock,
diff --git a/include/net/inet_common.h b/include/net/inet_common.h
index 22fac98..91cd8d0 100644
--- a/include/net/inet_common.h
+++ b/include/net/inet_common.h
@@ -21,7 +21,9 @@ extern int inet_dgram_connect(struct socket *sock, struct sockaddr * uaddr,
extern int inet_accept(struct socket *sock, struct socket *newsock, int flags);
extern int inet_sendmsg(struct kiocb *iocb, struct socket *sock,
struct msghdr *msg, size_t size);
-extern ssize_t inet_sendpage(struct socket *sock, struct page *page, int offset,
+extern ssize_t inet_sendpage(struct socket *sock, struct page *page,
+ struct skb_frag_destructor *frag,
+ int offset,
size_t size, int flags);
extern int inet_recvmsg(struct kiocb *iocb, struct socket *sock,
struct msghdr *msg, size_t size, int flags);
diff --git a/include/net/ip.h b/include/net/ip.h
index 66dd491..887a834 100644
--- a/include/net/ip.h
+++ b/include/net/ip.h
@@ -114,7 +114,9 @@ extern int ip_append_data(struct sock *sk, struct flowi4 *fl4,
struct rtable **rt,
unsigned int flags);
extern int ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb);
-extern ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page,
+extern ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4,
+ struct page *page,
+ struct skb_frag_destructor *destroy,
int offset, size_t size, int flags);
extern struct sk_buff *__ip_make_skb(struct sock *sk,
struct flowi4 *fl4,
diff --git a/include/net/sock.h b/include/net/sock.h
index c0b938c..c1ab674 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -763,6 +763,7 @@ struct proto {
size_t len, int noblock, int flags,
int *addr_len);
int (*sendpage)(struct sock *sk, struct page *page,
+ struct skb_frag_destructor *destroy,
int offset, size_t size, int flags);
int (*bind)(struct sock *sk,
struct sockaddr *uaddr, int addr_len);
@@ -1152,6 +1153,7 @@ extern int sock_no_mmap(struct file *file,
struct vm_area_struct *vma);
extern ssize_t sock_no_sendpage(struct socket *sock,
struct page *page,
+ struct skb_frag_destructor *destroy,
int offset, size_t size,
int flags);
diff --git a/include/net/tcp.h b/include/net/tcp.h
index cda30ea..1f43c0d 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -317,7 +317,9 @@ extern void *tcp_v4_tw_get_peer(struct sock *sk);
extern int tcp_v4_tw_remember_stamp(struct inet_timewait_sock *tw);
extern int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
size_t size);
-extern int tcp_sendpage(struct sock *sk, struct page *page, int offset,
+extern int tcp_sendpage(struct sock *sk, struct page *page,
+ struct skb_frag_destructor *destroy,
+ int offset,
size_t size, int flags);
extern int tcp_ioctl(struct sock *sk, int cmd, unsigned long arg);
extern int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb,
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index 78b55f4..ec7955b 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -852,7 +852,7 @@ static int write_partial_msg_pages(struct ceph_connection *con)
cpu_to_le32(crc32c(tmpcrc, base, len));
con->out_msg_pos.did_page_crc = 1;
}
- ret = kernel_sendpage(con->sock, page,
+ ret = kernel_sendpage(con->sock, page, NULL,
con->out_msg_pos.page_pos + page_shift,
len,
MSG_DONTWAIT | MSG_NOSIGNAL |
diff --git a/net/core/sock.c b/net/core/sock.c
index be55676..87d04db 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1858,7 +1858,9 @@ int sock_no_mmap(struct file *file, struct socket *sock, struct vm_area_struct *
}
EXPORT_SYMBOL(sock_no_mmap);
-ssize_t sock_no_sendpage(struct socket *sock, struct page *page, int offset, size_t size, int flags)
+ssize_t sock_no_sendpage(struct socket *sock, struct page *page,
+ struct skb_frag_destructor *destroy,
+ int offset, size_t size, int flags)
{
ssize_t res;
struct msghdr msg = {.msg_flags = flags};
@@ -1868,6 +1870,8 @@ ssize_t sock_no_sendpage(struct socket *sock, struct page *page, int offset, siz
iov.iov_len = size;
res = kernel_sendmsg(sock, &msg, &iov, 1, size);
kunmap(page);
+ /* kernel_sendmsg copies so we can destroy immediately */
+ skb_frag_destructor_unref(destroy);
return res;
}
EXPORT_SYMBOL(sock_no_sendpage);
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index ef1528a..45c0876 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -740,7 +740,9 @@ int inet_sendmsg(struct kiocb *iocb, struct socket *sock, struct msghdr *msg,
}
EXPORT_SYMBOL(inet_sendmsg);
-ssize_t inet_sendpage(struct socket *sock, struct page *page, int offset,
+ssize_t inet_sendpage(struct socket *sock, struct page *page,
+ struct skb_frag_destructor *destroy,
+ int offset,
size_t size, int flags)
{
struct sock *sk = sock->sk;
@@ -753,8 +755,9 @@ ssize_t inet_sendpage(struct socket *sock, struct page *page, int offset,
return -EAGAIN;
if (sk->sk_prot->sendpage)
- return sk->sk_prot->sendpage(sk, page, offset, size, flags);
- return sock_no_sendpage(sock, page, offset, size, flags);
+ return sk->sk_prot->sendpage(sk, page, destroy,
+ offset, size, flags);
+ return sock_no_sendpage(sock, page, destroy, offset, size, flags);
}
EXPORT_SYMBOL(inet_sendpage);
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index c4326fb..b35b728 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -1111,6 +1111,7 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4,
}
ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page,
+ struct skb_frag_destructor *destroy,
int offset, size_t size, int flags)
{
struct inet_sock *inet = inet_sk(sk);
@@ -1224,11 +1225,11 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page,
i = skb_shinfo(skb)->nr_frags;
if (len > size)
len = size;
- if (skb_can_coalesce(skb, i, page, NULL, offset)) {
+ if (skb_can_coalesce(skb, i, page, destroy, offset)) {
skb_shinfo(skb)->frags[i-1].size += len;
} else if (i < MAX_SKB_FRAGS) {
- get_page(page);
- skb_fill_page_desc(skb, i, page, NULL, offset, len);
+ skb_fill_page_desc(skb, i, page, destroy, offset, len);
+ skb_frag_ref(skb, i);
} else {
err = -EMSGSIZE;
goto error;
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index a1a0ccd..2f590e5 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -757,7 +757,10 @@ static int tcp_send_mss(struct sock *sk, int *size_goal, int flags)
return mss_now;
}
-static ssize_t do_tcp_sendpages(struct sock *sk, struct page **pages, int poffset,
+static ssize_t do_tcp_sendpages(struct sock *sk,
+ struct page **pages,
+ struct skb_frag_destructor **destructors,
+ int poffset,
size_t psize, int flags)
{
struct tcp_sock *tp = tcp_sk(sk);
@@ -783,6 +786,8 @@ static ssize_t do_tcp_sendpages(struct sock *sk, struct page **pages, int poffse
while (psize > 0) {
struct sk_buff *skb = tcp_write_queue_tail(sk);
struct page *page = pages[poffset / PAGE_SIZE];
+ struct skb_frag_destructor *destroy =
+ destructors ? destructors[poffset / PAGE_SIZE] : NULL;
int copy, i, can_coalesce;
int offset = poffset % PAGE_SIZE;
int size = min_t(size_t, psize, PAGE_SIZE - offset);
@@ -804,7 +809,7 @@ new_segment:
copy = size;
i = skb_shinfo(skb)->nr_frags;
- can_coalesce = skb_can_coalesce(skb, i, page, NULL, offset);
+ can_coalesce = skb_can_coalesce(skb, i, page, destroy, offset);
if (!can_coalesce && i >= MAX_SKB_FRAGS) {
tcp_mark_push(tp, skb);
goto new_segment;
@@ -815,8 +820,8 @@ new_segment:
if (can_coalesce) {
skb_shinfo(skb)->frags[i - 1].size += copy;
} else {
- get_page(page);
- skb_fill_page_desc(skb, i, page, NULL, offset, copy);
+ skb_fill_page_desc(skb, i, page, destroy, offset, copy);
+ skb_frag_ref(skb, i);
}
skb->len += copy;
@@ -871,18 +876,20 @@ out_err:
return sk_stream_error(sk, flags, err);
}
-int tcp_sendpage(struct sock *sk, struct page *page, int offset,
- size_t size, int flags)
+int tcp_sendpage(struct sock *sk, struct page *page,
+ struct skb_frag_destructor *destroy,
+ int offset, size_t size, int flags)
{
ssize_t res;
if (!(sk->sk_route_caps & NETIF_F_SG) ||
!(sk->sk_route_caps & NETIF_F_ALL_CSUM))
- return sock_no_sendpage(sk->sk_socket, page, offset, size,
- flags);
+ return sock_no_sendpage(sk->sk_socket, page, destroy,
+ offset, size, flags);
lock_sock(sk);
- res = do_tcp_sendpages(sk, &page, offset, size, flags);
+ res = do_tcp_sendpages(sk, &page, &destroy,
+ offset, size, flags);
release_sock(sk);
return res;
}
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index 198f75b..ebdc8ea 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -1027,8 +1027,9 @@ do_confirm:
}
EXPORT_SYMBOL(udp_sendmsg);
-int udp_sendpage(struct sock *sk, struct page *page, int offset,
- size_t size, int flags)
+int udp_sendpage(struct sock *sk, struct page *page,
+ struct skb_frag_destructor *destroy,
+ int offset, size_t size, int flags)
{
struct inet_sock *inet = inet_sk(sk);
struct udp_sock *up = udp_sk(sk);
@@ -1056,11 +1057,11 @@ int udp_sendpage(struct sock *sk, struct page *page, int offset,
}
ret = ip_append_page(sk, &inet->cork.fl.u.ip4,
- page, offset, size, flags);
+ page, destroy, offset, size, flags);
if (ret == -EOPNOTSUPP) {
release_sock(sk);
- return sock_no_sendpage(sk->sk_socket, page, offset,
- size, flags);
+ return sock_no_sendpage(sk->sk_socket, page, destroy,
+ offset, size, flags);
}
if (ret < 0) {
udp_flush_pending_frames(sk);
diff --git a/net/ipv4/udp_impl.h b/net/ipv4/udp_impl.h
index aaad650..4923d82 100644
--- a/net/ipv4/udp_impl.h
+++ b/net/ipv4/udp_impl.h
@@ -23,8 +23,9 @@ extern int compat_udp_getsockopt(struct sock *sk, int level, int optname,
#endif
extern int udp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
size_t len, int noblock, int flags, int *addr_len);
-extern int udp_sendpage(struct sock *sk, struct page *page, int offset,
- size_t size, int flags);
+extern int udp_sendpage(struct sock *sk, struct page *page,
+ struct skb_frag_destructor *destroy,
+ int offset, size_t size, int flags);
extern int udp_queue_rcv_skb(struct sock * sk, struct sk_buff *skb);
extern void udp_destroy_sock(struct sock *sk);
diff --git a/net/rds/tcp_send.c b/net/rds/tcp_send.c
index 1b4fd68..e0f03be 100644
--- a/net/rds/tcp_send.c
+++ b/net/rds/tcp_send.c
@@ -121,6 +121,7 @@ int rds_tcp_xmit(struct rds_connection *conn, struct rds_message *rm,
sg_page(&rm->data.op_sg[sg]),
rm->data.op_sg[sg].offset + off,
rm->data.op_sg[sg].length - off,
+ NULL,
MSG_DONTWAIT|MSG_NOSIGNAL);
rdsdebug("tcp sendpage %p:%u:%u ret %d\n", (void *)sg_page(&rm->data.op_sg[sg]),
rm->data.op_sg[sg].offset + off, rm->data.op_sg[sg].length - off,
diff --git a/net/socket.c b/net/socket.c
index 02dc82d..4b77658 100644
--- a/net/socket.c
+++ b/net/socket.c
@@ -795,7 +795,7 @@ static ssize_t sock_sendpage(struct file *file, struct page *page,
if (more)
flags |= MSG_MORE;
- return kernel_sendpage(sock, page, offset, size, flags);
+ return kernel_sendpage(sock, page, NULL, offset, size, flags);
}
static ssize_t sock_splice_read(struct file *file, loff_t *ppos,
@@ -3343,15 +3343,18 @@ int kernel_setsockopt(struct socket *sock, int level, int optname,
}
EXPORT_SYMBOL(kernel_setsockopt);
-int kernel_sendpage(struct socket *sock, struct page *page, int offset,
+int kernel_sendpage(struct socket *sock, struct page *page,
+ struct skb_frag_destructor *destroy,
+ int offset,
size_t size, int flags)
{
sock_update_classid(sock->sk);
if (sock->ops->sendpage)
- return sock->ops->sendpage(sock, page, offset, size, flags);
+ return sock->ops->sendpage(sock, page, destroy,
+ offset, size, flags);
- return sock_no_sendpage(sock, page, offset, size, flags);
+ return sock_no_sendpage(sock, page, destroy, offset, size, flags);
}
EXPORT_SYMBOL(kernel_sendpage);
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index af04f77..a80b1d3 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -181,7 +181,7 @@ int svc_send_common(struct socket *sock, struct xdr_buf *xdr,
/* send head */
if (slen == xdr->head[0].iov_len)
flags = 0;
- len = kernel_sendpage(sock, headpage, headoffset,
+ len = kernel_sendpage(sock, headpage, NULL, headoffset,
xdr->head[0].iov_len, flags);
if (len != xdr->head[0].iov_len)
goto out;
@@ -194,7 +194,7 @@ int svc_send_common(struct socket *sock, struct xdr_buf *xdr,
while (pglen > 0) {
if (slen == size)
flags = 0;
- result = kernel_sendpage(sock, *ppage, base, size, flags);
+ result = kernel_sendpage(sock, *ppage, NULL, base, size, flags);
if (result > 0)
len += result;
if (result != size)
@@ -208,7 +208,7 @@ int svc_send_common(struct socket *sock, struct xdr_buf *xdr,
/* send tail */
if (xdr->tail[0].iov_len) {
- result = kernel_sendpage(sock, tailpage, tailoffset,
+ result = kernel_sendpage(sock, tailpage, NULL, tailoffset,
xdr->tail[0].iov_len, 0);
if (result > 0)
len += result;
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 72abb73..d027621 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -397,7 +397,7 @@ static int xs_send_pagedata(struct socket *sock, struct xdr_buf *xdr, unsigned i
remainder -= len;
if (remainder != 0 || more)
flags |= MSG_MORE;
- err = sock->ops->sendpage(sock, *ppage, base, len, flags);
+ err = sock->ops->sendpage(sock, *ppage, NULL, base, len, flags);
if (remainder == 0 || err != len)
break;
sent += err;
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 13/13] sunrpc: use SKB fragment destructors to delay completion until page is released by network stack.
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (11 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 12/13] net: add paged frag destructor support to kernel_sendpage Ian Campbell
@ 2011-07-22 13:17 ` Ian Campbell
2011-07-22 18:39 ` Trond Myklebust
2011-07-22 14:13 ` [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility David Miller
13 siblings, 1 reply; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 13:17 UTC (permalink / raw)
To: netdev, linux-nfs
Cc: Ian Campbell, Trond Myklebust, David S. Miller, Neil Brown,
J. Bruce Fields
This prevents an issue where an ACK is delayed, a retransmit is queued (either
at the RPC or TCP level) and the ACK arrives before the retransmission hits the
wire. If this happens to an NFS WRITE RPC then the write() system call
completes and the userspace process can continue, potentially modifying data
referenced by the retransmission before the retransmission occurs.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Neil Brown <neilb@suse.de>
Cc: "J. Bruce Fields" <bfields@fieldses.org>
Cc: linux-nfs@vger.kernel.org
Cc: netdev@vger.kernel.org
[since v1:
Push down from NFS layer into RPM layer
]
---
include/linux/sunrpc/xdr.h | 2 ++
include/linux/sunrpc/xprt.h | 5 ++++-
net/sunrpc/clnt.c | 27 ++++++++++++++++++++++-----
net/sunrpc/svcsock.c | 2 +-
net/sunrpc/xprt.c | 13 +++++++++++++
net/sunrpc/xprtsock.c | 2 +-
6 files changed, 43 insertions(+), 8 deletions(-)
diff --git a/include/linux/sunrpc/xdr.h b/include/linux/sunrpc/xdr.h
index a20970e..172f81e 100644
--- a/include/linux/sunrpc/xdr.h
+++ b/include/linux/sunrpc/xdr.h
@@ -16,6 +16,7 @@
#include <asm/byteorder.h>
#include <asm/unaligned.h>
#include <linux/scatterlist.h>
+#include <linux/skbuff.h>
/*
* Buffer adjustment
@@ -57,6 +58,7 @@ struct xdr_buf {
tail[1]; /* Appended after page data */
struct page ** pages; /* Array of contiguous pages */
+ struct skb_frag_destructor *destructor;
unsigned int page_base, /* Start of page data */
page_len, /* Length of page data */
flags; /* Flags for data disposition */
diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h
index 81cce3b..0de6bc3 100644
--- a/include/linux/sunrpc/xprt.h
+++ b/include/linux/sunrpc/xprt.h
@@ -91,7 +91,10 @@ struct rpc_rqst {
/* A cookie used to track the
state of the transport
connection */
-
+ struct skb_frag_destructor destructor; /* SKB paged fragment
+ * destructor for
+ * transmitted pages*/
+
/*
* Partial send handling
*/
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 8c91415..1145a19 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -61,6 +61,7 @@ static void call_reserve(struct rpc_task *task);
static void call_reserveresult(struct rpc_task *task);
static void call_allocate(struct rpc_task *task);
static void call_decode(struct rpc_task *task);
+static void call_complete(struct rpc_task *task);
static void call_bind(struct rpc_task *task);
static void call_bind_status(struct rpc_task *task);
static void call_transmit(struct rpc_task *task);
@@ -1114,6 +1115,8 @@ rpc_xdr_encode(struct rpc_task *task)
(char *)req->rq_buffer + req->rq_callsize,
req->rq_rcvsize);
+ req->rq_snd_buf.destructor = &req->destructor;
+
p = rpc_encode_header(task);
if (p == NULL) {
printk(KERN_INFO "RPC: couldn't encode RPC header, exit EIO\n");
@@ -1277,6 +1280,7 @@ call_connect_status(struct rpc_task *task)
static void
call_transmit(struct rpc_task *task)
{
+ struct rpc_rqst *req = task->tk_rqstp;
dprint_status(task);
task->tk_action = call_status;
@@ -1310,8 +1314,8 @@ call_transmit(struct rpc_task *task)
call_transmit_status(task);
if (rpc_reply_expected(task))
return;
- task->tk_action = rpc_exit_task;
- rpc_wake_up_queued_task(&task->tk_xprt->pending, task);
+ task->tk_action = call_complete;
+ skb_frag_destructor_unref(&req->destructor);
}
/*
@@ -1384,7 +1388,8 @@ call_bc_transmit(struct rpc_task *task)
return;
}
- task->tk_action = rpc_exit_task;
+ task->tk_action = call_complete;
+ skb_frag_destructor_unref(&req->destructor);
if (task->tk_status < 0) {
printk(KERN_NOTICE "RPC: Could not send backchannel reply "
"error: %d\n", task->tk_status);
@@ -1424,7 +1429,6 @@ call_bc_transmit(struct rpc_task *task)
"error: %d\n", task->tk_status);
break;
}
- rpc_wake_up_queued_task(&req->rq_xprt->pending, task);
}
#endif /* CONFIG_NFS_V4_1 */
@@ -1591,12 +1595,14 @@ call_decode(struct rpc_task *task)
return;
}
- task->tk_action = rpc_exit_task;
+ task->tk_action = call_complete;
if (decode) {
task->tk_status = rpcauth_unwrap_resp(task, decode, req, p,
task->tk_msg.rpc_resp);
}
+ rpc_sleep_on(&req->rq_xprt->pending, task, NULL);
+ skb_frag_destructor_unref(&req->destructor);
dprintk("RPC: %5u call_decode result %d\n", task->tk_pid,
task->tk_status);
return;
@@ -1611,6 +1617,17 @@ out_retry:
}
}
+/*
+ * 8. Wait for pages to be released by the network stack.
+ */
+static void
+call_complete(struct rpc_task *task)
+{
+ struct rpc_rqst *req = task->tk_rqstp;
+ dprintk("RPC: %5u call_complete result %d\n", task->tk_pid, task->tk_status);
+ task->tk_action = rpc_exit_task;
+}
+
static __be32 *
rpc_encode_header(struct rpc_task *task)
{
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index a80b1d3..40c2420 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -194,7 +194,7 @@ int svc_send_common(struct socket *sock, struct xdr_buf *xdr,
while (pglen > 0) {
if (slen == size)
flags = 0;
- result = kernel_sendpage(sock, *ppage, NULL, base, size, flags);
+ result = kernel_sendpage(sock, *ppage, xdr->destructor, base, size, flags);
if (result > 0)
len += result;
if (result != size)
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index ce5eb68..62f52a3 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1017,6 +1017,16 @@ static inline void xprt_init_xid(struct rpc_xprt *xprt)
xprt->xid = net_random();
}
+static int xprt_complete_skb_pages(void *calldata)
+{
+ struct rpc_task *task = calldata;
+ struct rpc_rqst *req = task->tk_rqstp;
+
+ dprintk("RPC: %5u completing skb pages\n", task->tk_pid);
+ rpc_wake_up_queued_task(&req->rq_xprt->pending, task);
+ return 0;
+}
+
static void xprt_request_init(struct rpc_task *task, struct rpc_xprt *xprt)
{
struct rpc_rqst *req = task->tk_rqstp;
@@ -1028,6 +1038,9 @@ static void xprt_request_init(struct rpc_task *task, struct rpc_xprt *xprt)
req->rq_xid = xprt_alloc_xid(xprt);
req->rq_release_snd_buf = NULL;
xprt_reset_majortimeo(req);
+ atomic_set(&req->destructor.ref, 1);
+ req->destructor.destroy = &xprt_complete_skb_pages;
+ req->destructor.data = task;
dprintk("RPC: %5u reserved req %p xid %08x\n", task->tk_pid,
req, ntohl(req->rq_xid));
}
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index d027621..ca1643b 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -397,7 +397,7 @@ static int xs_send_pagedata(struct socket *sock, struct xdr_buf *xdr, unsigned i
remainder -= len;
if (remainder != 0 || more)
flags |= MSG_MORE;
- err = sock->ops->sendpage(sock, *ppage, NULL, base, len, flags);
+ err = sock->ops->sendpage(sock, *ppage, xdr->destructor, base, len, flags);
if (remainder == 0 || err != len)
break;
sent += err;
--
1.7.2.5
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH 08/13] net: convert drivers to paged frag API.
2011-07-22 13:17 ` [PATCH 08/13] net: convert drivers to paged frag API Ian Campbell
@ 2011-07-22 14:12 ` David Miller
2011-07-22 14:16 ` Ian Campbell
0 siblings, 1 reply; 22+ messages in thread
From: David Miller @ 2011-07-22 14:12 UTC (permalink / raw)
To: ian.campbell; +Cc: netdev, linux-nfs
From: Ian Campbell <ian.campbell@citrix.com>
Date: Fri, 22 Jul 2011 14:17:28 +0100
> - put_dma(tx->index,eni_dev->dma,&j,(unsigned long)
> - skb_shinfo(skb)->frags[i].page + skb_shinfo(skb)->frags[i].page_offset,
> + put_dma(tx->index,eni_dev->dma,&j,
> + (unsigned long)skb_frag_address(&skb_shinfo(skb)->frags[i]),
This is not an equivalent transformation.
skb_frag_address() does a page_address() on the frag page, but that is
not what the code was doing here previously.
It's possible the code was buggy, but you can't do a fix like that
amidst what is supposed to be a semantically NOP transformation.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
` (12 preceding siblings ...)
2011-07-22 13:17 ` [PATCH 13/13] sunrpc: use SKB fragment destructors to delay completion until page is released by network stack Ian Campbell
@ 2011-07-22 14:13 ` David Miller
2011-07-22 14:18 ` Ian Campbell
13 siblings, 1 reply; 22+ messages in thread
From: David Miller @ 2011-07-22 14:13 UTC (permalink / raw)
To: Ian.Campbell; +Cc: netdev, linux-nfs
Well, Ian, because you put all of these "struct page *" MM layer
const changes in here I can't just apply this series once you
get it ready enough from a networking perspective.
Why not do the const crap later, so it can be done independently
of these changes and not be a dependency upon them?
I know you want to pass const page structs down as far as possible,
but that can wait for later, make the networking bits work on
non-const pointers for now.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 08/13] net: convert drivers to paged frag API.
2011-07-22 14:12 ` David Miller
@ 2011-07-22 14:16 ` Ian Campbell
0 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 14:16 UTC (permalink / raw)
To: David Miller; +Cc: netdev, linux-nfs, Chas Williams, linux-atm-general
On Fri, 2011-07-22 at 15:12 +0100, David Miller wrote:
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Fri, 22 Jul 2011 14:17:28 +0100
>
> > - put_dma(tx->index,eni_dev->dma,&j,(unsigned long)
> > - skb_shinfo(skb)->frags[i].page + skb_shinfo(skb)->frags[i].page_offset,
> > + put_dma(tx->index,eni_dev->dma,&j,
> > + (unsigned long)skb_frag_address(&skb_shinfo(skb)->frags[i]),
>
> This is not an equivalent transformation.
>
> skb_frag_address() does a page_address() on the frag page, but that is
> not what the code was doing here previously.
>
> It's possible the code was buggy, but you can't do a fix like that
> amidst what is supposed to be a semantically NOP transformation.
Ouch, you are absolutely right, I didn't spot that, sorry.
The original code does look pretty bogus though, indexing off a struct
page * like that -- CC'ing the ATM maintainer + list.
Ian.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility
2011-07-22 14:13 ` [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility David Miller
@ 2011-07-22 14:18 ` Ian Campbell
0 siblings, 0 replies; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 14:18 UTC (permalink / raw)
To: David Miller; +Cc: netdev, linux-nfs
On Fri, 2011-07-22 at 15:13 +0100, David Miller wrote:
> Well, Ian, because you put all of these "struct page *" MM layer
> const changes in here I can't just apply this series once you
> get it ready enough from a networking perspective.
>
> Why not do the const crap later, so it can be done independently
> of these changes and not be a dependency upon them?
Initially it was to help me find locations which needed consideration
(since it caused build failures) but I can flip it round now, sure.
> I know you want to pass const page structs down as far as possible,
> but that can wait for later, make the networking bits work on
> non-const pointers for now.
Will do.
Ian.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 13/13] sunrpc: use SKB fragment destructors to delay completion until page is released by network stack.
2011-07-22 13:17 ` [PATCH 13/13] sunrpc: use SKB fragment destructors to delay completion until page is released by network stack Ian Campbell
@ 2011-07-22 18:39 ` Trond Myklebust
0 siblings, 0 replies; 22+ messages in thread
From: Trond Myklebust @ 2011-07-22 18:39 UTC (permalink / raw)
To: Ian Campbell
Cc: netdev, linux-nfs, David S. Miller, Neil Brown, J. Bruce Fields
On Fri, 2011-07-22 at 14:17 +0100, Ian Campbell wrote:
> This prevents an issue where an ACK is delayed, a retransmit is queued (either
> at the RPC or TCP level) and the ACK arrives before the retransmission hits the
> wire. If this happens to an NFS WRITE RPC then the write() system call
> completes and the userspace process can continue, potentially modifying data
> referenced by the retransmission before the retransmission occurs.
>
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
> Cc: "David S. Miller" <davem@davemloft.net>
> Cc: Neil Brown <neilb@suse.de>
> Cc: "J. Bruce Fields" <bfields@fieldses.org>
> Cc: linux-nfs@vger.kernel.org
> Cc: netdev@vger.kernel.org
> [since v1:
> Push down from NFS layer into RPM layer
> ]
> ---
> include/linux/sunrpc/xdr.h | 2 ++
> include/linux/sunrpc/xprt.h | 5 ++++-
> net/sunrpc/clnt.c | 27 ++++++++++++++++++++++-----
> net/sunrpc/svcsock.c | 2 +-
> net/sunrpc/xprt.c | 13 +++++++++++++
> net/sunrpc/xprtsock.c | 2 +-
> 6 files changed, 43 insertions(+), 8 deletions(-)
This looks good to me. Thanks, Ian!
Acked-by: Trond Myklebust <Trond.Myklebust@netapp.com>
--
Trond Myklebust
Linux NFS client maintainer
NetApp
Trond.Myklebust@netapp.com
www.netapp.com
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 10/13] net: add paged frag destructor to skb_fill_page_desc()
2011-07-22 13:17 ` [PATCH 10/13] net: add paged frag destructor to skb_fill_page_desc() Ian Campbell
@ 2011-07-22 19:58 ` Michał Mirosław
2011-07-22 21:07 ` Ian Campbell
0 siblings, 1 reply; 22+ messages in thread
From: Michał Mirosław @ 2011-07-22 19:58 UTC (permalink / raw)
To: Ian Campbell
Cc: netdev, linux-nfs, David S. Miller, James E.J. Bottomley,
Alexey Kuznetsov, Pekka Savola (ipv6),
James Morris, Hideaki YOSHIFUJI, Patrick McHardy, linux-rdma,
linux-s390, linux-scsi, devel
MjAxMS83LzIyIElhbiBDYW1wYmVsbCA8aWFuLmNhbXBiZWxsQGNpdHJpeC5jb20+Ogo+IGRpZmYg
LS1naXQgYS9pbmNsdWRlL2xpbnV4L3NrYnVmZi5oIGIvaW5jbHVkZS9saW51eC9za2J1ZmYuaAo+
IGluZGV4IDk4MThmZTIuLmZhZWU4ZDMgMTAwNjQ0Cj4gLS0tIGEvaW5jbHVkZS9saW51eC9za2J1
ZmYuaAo+ICsrKyBiL2luY2x1ZGUvbGludXgvc2tidWZmLmgKPiBAQCAtMTEzNCwxMiArMTEzNCwx
NCBAQCBzdGF0aWMgaW5saW5lIGludCBza2JfcGFnZWxlbihjb25zdCBzdHJ1Y3Qgc2tfYnVmZiAq
c2tiKQo+IMKgKiBEb2VzIG5vdCB0YWtlIGFueSBhZGRpdGlvbmFsIHJlZmVyZW5jZSBvbiB0aGUg
ZnJhZ21lbnQuCj4gwqAqLwo+IMKgc3RhdGljIGlubGluZSB2b2lkIF9fc2tiX2ZpbGxfcGFnZV9k
ZXNjKHN0cnVjdCBza19idWZmICpza2IsIGludCBpLAo+IC0gwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgc3RydWN0IHBhZ2UgKnBhZ2UsIGlu
dCBvZmYsIGludCBzaXplKQo+ICsgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgc3RydWN0IHBhZ2UgKnBhZ2UsCj4gKyDCoCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCBzdHJ1Y3Qgc2tiX2Zy
YWdfZGVzdHJ1Y3RvciAqZGVzdHJveSwKPiArIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIGludCBvZmYsIGludCBzaXplKQo+IMKgewo+IMKg
IMKgIMKgIMKgc2tiX2ZyYWdfdCAqZnJhZyA9ICZza2Jfc2hpbmZvKHNrYiktPmZyYWdzW2ldOwo+
Cj4gwqAgwqAgwqAgwqBmcmFnLT5wYWdlLnAgwqAgwqAgwqAgwqAgwqAgwqAgwqA9IHBhZ2U7Cj4g
LSDCoCDCoCDCoCBmcmFnLT5wYWdlLmRlc3RydWN0b3IgwqAgwqAgPSBOVUxMOwo+ICsgwqAgwqAg
wqAgZnJhZy0+cGFnZS5kZXN0cnVjdG9yIMKgIMKgID0gZGVzdHJveTsKPiDCoCDCoCDCoCDCoGZy
YWctPnBhZ2Vfb2Zmc2V0IMKgIMKgIMKgIMKgID0gb2ZmOwo+IMKgIMKgIMKgIMKgZnJhZy0+c2l6
ZSDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoD0gc2l6ZTsKPiDCoH0KCllvdSBjb3VsZCBqdXN0IHJl
bmFtZSB0aGlzIGZ1bmN0aW9uIHRvIGUuZy4KX19za2JfZmlsbF9mcmFnaWxlX3BhZ2VfZGVzYygp
IChvciB3aGF0ZXZlciBuYW1lKSBhZGQgYW4gaW5saW5lCndyYXBwZXIgY2FsbGluZyBpdCB3aXRo
IGRlc3Ryb3kgPT0gTlVMTC4gVGhpcyB3aWxsIGF2b2lkIHRvdWNoaW5nIGFsbAp0aG9zZSBkcml2
ZXJzIHdoaWNoIHdvbid0IGV2ZXIgbmVlZCB0aGlzIGZ1bmN0aW9uYWxpdHkuCgpCZXN0IFJlZ2Fy
ZHMsCk1pY2hhxYIgTWlyb3PFgmF3CgTvv717Lm7vv70r77+977+977+977+977+977+977+9KyXv
v73vv71sendt77+977+9Yu+/veunsu+/ve+/vXLvv73vv716WO+/ve+/vRnfsinvv73vv73vv713
Kh9qZ++/ve+/ve+/vR7vv73vv73vv73vv73vv73domov77+977+977+9eu+/vd6W77+977+9Mu+/
vd6Z77+977+977+9Ju+/vSnfoe+/vWHvv73vv71/77+977+9Hu+/vUfvv73vv73vv71o77+9D++/
vWo6K3bvv73vv73vv71377+92aU=
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 10/13] net: add paged frag destructor to skb_fill_page_desc()
2011-07-22 19:58 ` Michał Mirosław
@ 2011-07-22 21:07 ` Ian Campbell
2011-07-22 21:44 ` Michał Mirosław
0 siblings, 1 reply; 22+ messages in thread
From: Ian Campbell @ 2011-07-22 21:07 UTC (permalink / raw)
To: Michał Mirosław
Cc: netdev, linux-nfs, David S. Miller, James E.J. Bottomley,
Alexey Kuznetsov, Pekka Savola (ipv6),
James Morris, Hideaki YOSHIFUJI, Patrick McHardy, linux-rdma,
linux-s390, linux-scsi, devel
On Fri, 2011-07-22 at 20:58 +0100, Michał Mirosław wrote:
> 2011/7/22 Ian Campbell <ian.campbell@citrix.com>:
> > diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
> > index 9818fe2..faee8d3 100644
> > --- a/include/linux/skbuff.h
> > +++ b/include/linux/skbuff.h
> > @@ -1134,12 +1134,14 @@ static inline int skb_pagelen(const struct sk_buff *skb)
> > * Does not take any additional reference on the fragment.
> > */
> > static inline void __skb_fill_page_desc(struct sk_buff *skb, int i,
> > - struct page *page, int off, int size)
> > + struct page *page,
> > + struct skb_frag_destructor *destroy,
> > + int off, int size)
> > {
> > skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
> >
> > frag->page.p = page;
> > - frag->page.destructor = NULL;
> > + frag->page.destructor = destroy;
> > frag->page_offset = off;
> > frag->size = size;
> > }
>
> You could just rename this function to e.g.
> __skb_fill_fragile_page_desc() (or whatever name) add an inline
> wrapper calling it with destroy == NULL. This will avoid touching all
> those drivers which won't ever need this functionality.
I could call this variant __skb_frag_init (which I think better fits
into the pattern of the new functions) and leave the existing
__skb_fill_page_desc as a compat wrapper if that's preferred but I was
trying to avoid duplicating up constructors just for different sets of
defaults.
Ian.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 10/13] net: add paged frag destructor to skb_fill_page_desc()
2011-07-22 21:07 ` Ian Campbell
@ 2011-07-22 21:44 ` Michał Mirosław
0 siblings, 0 replies; 22+ messages in thread
From: Michał Mirosław @ 2011-07-22 21:44 UTC (permalink / raw)
To: Ian Campbell
Cc: netdev, linux-nfs, David S. Miller, James E.J. Bottomley,
Alexey Kuznetsov, Pekka Savola (ipv6),
James Morris, Hideaki YOSHIFUJI, Patrick McHardy, linux-rdma,
linux-s390, linux-scsi, devel
W dniu 22 lipca 2011 23:07 użytkownik Ian Campbell
<Ian.Campbell@eu.citrix.com> napisał:
> On Fri, 2011-07-22 at 20:58 +0100, Michał Mirosław wrote:
>> 2011/7/22 Ian Campbell <ian.campbell@citrix.com>:
>> > diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
>> > index 9818fe2..faee8d3 100644
>> > --- a/include/linux/skbuff.h
>> > +++ b/include/linux/skbuff.h
>> > @@ -1134,12 +1134,14 @@ static inline int skb_pagelen(const struct sk_buff *skb)
>> > * Does not take any additional reference on the fragment.
>> > */
>> > static inline void __skb_fill_page_desc(struct sk_buff *skb, int i,
>> > - struct page *page, int off, int size)
>> > + struct page *page,
>> > + struct skb_frag_destructor *destroy,
>> > + int off, int size)
>> > {
>> > skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
>> >
>> > frag->page.p = page;
>> > - frag->page.destructor = NULL;
>> > + frag->page.destructor = destroy;
>> > frag->page_offset = off;
>> > frag->size = size;
>> > }
>>
>> You could just rename this function to e.g.
>> __skb_fill_fragile_page_desc() (or whatever name) add an inline
>> wrapper calling it with destroy == NULL. This will avoid touching all
>> those drivers which won't ever need this functionality.
>
> I could call this variant __skb_frag_init (which I think better fits
> into the pattern of the new functions) and leave the existing
> __skb_fill_page_desc as a compat wrapper if that's preferred but I was
> trying to avoid duplicating up constructors just for different sets of
> defaults.
It's just Huffman coding: since most users need destroy = NULL, it's
good to have a wrapper for this case as it will then take less time to
write and understand the code (you won't need to think what is this
NULL for in all those places).
Best Regards,
Michał Mirosław
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2011-07-22 21:44 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-07-22 13:08 [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility Ian Campbell
2011-07-22 13:17 ` [PATCH 01/13] mm: Make some struct page's const Ian Campbell
2011-07-22 13:17 ` [PATCH 02/13] mm: use const struct page for r/o page-flag accessor methods Ian Campbell
2011-07-22 13:17 ` [PATCH 03/13] net: add APIs for manipulating skb page fragments Ian Campbell
2011-07-22 13:17 ` [PATCH 04/13] net: convert core to skb paged frag APIs Ian Campbell
2011-07-22 13:17 ` [PATCH 05/13] net: ipv4: convert to SKB " Ian Campbell
2011-07-22 13:17 ` [PATCH 06/13] net: ipv6: " Ian Campbell
2011-07-22 13:17 ` [PATCH 07/13] net: xfrm: " Ian Campbell
2011-07-22 13:17 ` [PATCH 08/13] net: convert drivers to paged frag API Ian Campbell
2011-07-22 14:12 ` David Miller
2011-07-22 14:16 ` Ian Campbell
2011-07-22 13:17 ` [PATCH 09/13] net: add support for per-paged-fragment destructors Ian Campbell
2011-07-22 13:17 ` [PATCH 10/13] net: add paged frag destructor to skb_fill_page_desc() Ian Campbell
2011-07-22 19:58 ` Michał Mirosław
2011-07-22 21:07 ` Ian Campbell
2011-07-22 21:44 ` Michał Mirosław
2011-07-22 13:17 ` [PATCH 11/13] net: only allow paged fragments with the same destructor to be coalesced Ian Campbell
2011-07-22 13:17 ` [PATCH 12/13] net: add paged frag destructor support to kernel_sendpage Ian Campbell
2011-07-22 13:17 ` [PATCH 13/13] sunrpc: use SKB fragment destructors to delay completion until page is released by network stack Ian Campbell
2011-07-22 18:39 ` Trond Myklebust
2011-07-22 14:13 ` [PATCH/RFC v2 0/13] enable SKB paged fragment lifetime visibility David Miller
2011-07-22 14:18 ` Ian Campbell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).