* [PATCH v3 01/11] arch: introduce __pfn_t for persistenti/device memory
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
@ 2015-05-12 4:29 ` Dan Williams
2015-05-12 4:29 ` [PATCH v3 02/11] block: add helpers for accessing a bio_vec page Dan Williams
` (10 subsequent siblings)
11 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:29 UTC (permalink / raw)
To: linux-kernel
Cc: axboe, linux-arch, riel, linux-nvdimm, david, hch, linux-fsdevel,
mgorman, j.glisse, H. Peter Anvin, Tejun Heo, akpm,
Linus Torvalds, mingo
Introduce a type that encapsulates a page-frame-number that is
optionally backed by memmap (struct page). This type will be used in
place of 'struct page *' instances in contexts where device-backed
memory (usually persistent memory) is being referenced (scatterlists for
drivers, biovecs for the block layer, etc). The operations in those i/o
paths that formerly required a 'struct page *' are to be converted to
use __pfn_t aware equivalent helpers. Otherwise, in the absence of
persistent memory, there is no functional change and __pfn_t is an alias
for a normal memory page.
It turns out that while 'struct page' references are used broadly in the
kernel I/O stacks the usage of 'struct page' based capabilities is very
shallow for block-i/o. It is only used for populating bio_vecs and
scatterlists for the retrieval of dma addresses, and for temporary
kernel mappings (kmap). Aside from kmap, these usages can be trivially
converted to operate on a pfn.
Indeed, kmap_atomic() is more problematic as it uses mm infrastructure,
via struct page, to setup and track temporary kernel mappings. It would
be unfortunate if the kmap infrastructure escaped its 32-bit/HIGHMEM
bonds and leaked into 64-bit code. Thankfully, it seems all that is
needed here is to convert kmap_atomic() callers, that want to opt-in to
supporting persistent memory, to use a new kmap_atomic_pfn_t(). Where
kmap_atomic_pfn_t() is enabled to re-use the existing ioremap() mapping
established by the driver for persistent memory.
Note, that as far as conceptually understanding __pfn_t is concerned,
'persistent memory' is really any address range in host memory not
covered by memmap. Contrast this with pure iomem that is on an mmio
mapped bus like PCI and cannot be converted to a dma_addr_t by "pfn <<
PAGE_SHIFT".
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Tejun Heo <tj@kernel.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
include/asm-generic/memory_model.h | 1
include/asm-generic/pfn.h | 84 ++++++++++++++++++++++++++++++++++++
include/linux/mm.h | 1
init/Kconfig | 13 ++++++
4 files changed, 98 insertions(+), 1 deletion(-)
create mode 100644 include/asm-generic/pfn.h
diff --git a/include/asm-generic/memory_model.h b/include/asm-generic/memory_model.h
index 14909b0b9cae..1b0ae21fd8ff 100644
--- a/include/asm-generic/memory_model.h
+++ b/include/asm-generic/memory_model.h
@@ -70,7 +70,6 @@
#endif /* CONFIG_FLATMEM/DISCONTIGMEM/SPARSEMEM */
#define page_to_pfn __page_to_pfn
-#define pfn_to_page __pfn_to_page
#endif /* __ASSEMBLY__ */
diff --git a/include/asm-generic/pfn.h b/include/asm-generic/pfn.h
new file mode 100644
index 000000000000..ee1363e3c67c
--- /dev/null
+++ b/include/asm-generic/pfn.h
@@ -0,0 +1,84 @@
+#ifndef __ASM_PFN_H
+#define __ASM_PFN_H
+
+/*
+ * Default pfn to physical address conversion, like most arch
+ * page_to_phys() implementations this resolves to a dma_addr_t as it
+ * should be the size needed for a device to reference this address.
+ */
+#ifndef __pfn_to_phys
+#define __pfn_to_phys(pfn) ((dma_addr_t)(pfn) << PAGE_SHIFT)
+#endif
+
+static inline struct page *pfn_to_page(unsigned long pfn)
+{
+ return __pfn_to_page(pfn);
+}
+
+/*
+ * __pfn_t: encapsulates a page-frame number that is optionally backed
+ * by memmap (struct page). This type will be used in place of a
+ * 'struct page *' instance in contexts where unmapped memory (usually
+ * persistent memory) is being referenced (scatterlists for drivers,
+ * biovecs for the block layer, etc). Whether a __pfn_t has a struct
+ * page backing is indicated by flags in the low bits of @data.
+ */
+typedef struct {
+ union {
+ unsigned long data;
+ struct page *page;
+ };
+} __pfn_t;
+
+enum {
+#if BITS_PER_LONG == 64
+ PFN_SHIFT = 3,
+#else
+ PFN_SHIFT = 2,
+#endif
+ PFN_MASK = (1 << PFN_SHIFT) - 1,
+ /* device-pfn not covered by memmap */
+ PFN_DEV = (1 << 0),
+};
+
+#ifdef CONFIG_DEV_PFN
+static inline bool __pfn_t_has_page(__pfn_t pfn)
+{
+ return (pfn.data & PFN_MASK) == 0;
+}
+
+#else
+static inline bool __pfn_t_has_page(__pfn_t pfn)
+{
+ return true;
+}
+#endif
+
+static inline struct page *__pfn_t_to_page(__pfn_t pfn)
+{
+ if (!__pfn_t_has_page(pfn))
+ return NULL;
+ return pfn.page;
+}
+
+static inline unsigned long __pfn_t_to_pfn(__pfn_t pfn)
+{
+ if (__pfn_t_has_page(pfn))
+ return page_to_pfn(pfn.page);
+ return pfn.data >> PFN_SHIFT;
+}
+
+static inline dma_addr_t __pfn_t_to_phys(__pfn_t pfn)
+{
+ if (!__pfn_t_has_page(pfn))
+ return __pfn_to_phys(__pfn_t_to_pfn(pfn));
+ return __pfn_to_phys(page_to_pfn(pfn.page));
+}
+
+static inline __pfn_t page_to_pfn_t(struct page *page)
+{
+ __pfn_t pfn = { .page = page };
+
+ return pfn;
+}
+#endif /* __ASM_PFN_H */
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0755b9fd03a7..9d35cff41c12 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -52,6 +52,7 @@ extern int sysctl_legacy_va_layout;
#include <asm/page.h>
#include <asm/pgtable.h>
#include <asm/processor.h>
+#include <asm-generic/pfn.h>
#ifndef __pa_symbol
#define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0))
diff --git a/init/Kconfig b/init/Kconfig
index dc24dec60232..b5b8a6ed0d97 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1764,6 +1764,19 @@ config PROFILING
Say Y here to enable the extended profiling support mechanisms used
by profilers such as OProfile.
+config DEV_PFN
+ default n
+ bool "Support for device provided (pmem, graphics, etc) memory" if EXPERT
+ help
+ Say Y here to enable I/O to/from device provided memory,
+ i.e. reference memory that is not mapped. This is usually
+ the case if you have large quantities of persistent memory
+ relative to DRAM. Enabling this option may increase the
+ kernel size by a few kilobytes as it instructs the kernel
+ that a __pfn_t may reference unmapped memory. Disabling
+ this option instructs the kernel that a __pfn_t always
+ references mapped platform memory.
+
#
# Place an empty function call at each tracepoint site. Can be
# dynamically changed for a probe function.
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH v3 02/11] block: add helpers for accessing a bio_vec page
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
2015-05-12 4:29 ` [PATCH v3 01/11] arch: introduce __pfn_t for persistenti/device memory Dan Williams
@ 2015-05-12 4:29 ` Dan Williams
2015-05-12 4:29 ` [PATCH v3 03/11] block: convert .bv_page to .bv_pfn bio_vec Dan Williams
` (9 subsequent siblings)
11 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:29 UTC (permalink / raw)
To: linux-kernel
Cc: Boaz Harrosh, Jan Kara, Mike Snitzer, Neil Brown, david,
Chris Mason, j.glisse, hch, Alasdair Kergon, linux-arch,
linux-nvdimm, mingo, mgorman, Matthew Wilcox, Ross Zwisler, riel,
axboe, Theodore Ts'o, Martin K. Petersen, Julia Lawall,
linux-fsdevel, akpm
In preparation for converting struct bio_vec to carry a __pfn_t instead
of struct page.
This change is prompted by the desire to add in-kernel DMA support
(O_DIRECT, hierarchical storage, RDMA, etc) for persistent memory which
lacks struct page coverage.
Alternatives:
1/ Provide struct page coverage for persistent memory in DRAM. The
expectation is that persistent memory capacities make this untenable
in the long term.
2/ Provide struct page coverage for persistent memory with persistent
memory. While persistent memory may have near DRAM performance
characteristics it may not have the same write-endurance of DRAM.
Given the update frequency of struct page objects it may not be
suitable for persistent memory.
3/ Dynamically allocate struct page. This appears to be on the order
of the complexity of converting code paths to use __pfn_t references
instead of struct page, and the amount of setup required to establish
a valid struct page reference is mostly wasted when the only usage in
the block stack is to perform a page_to_pfn() conversion for
dma-mapping. Instances of kmap() / kmap_atomic() usage appear to be
the only occasions in the block stack where struct page is
non-trivially used. A new kmap_atomic_pfn_t() is proposed to handle
those cases.
Generated with the following semantic patch:
// bv_page.cocci: convert usage of ->bv_page to use set/get helpers
// usage: make coccicheck COCCI=bv_page.cocci MODE=patch
virtual patch
virtual report
virtual org
@@
struct bio_vec bvec;
expression E;
type T;
@@
- bvec.bv_page = (T)E
+ bvec_set_page(&bvec, E)
@@
struct bio_vec *bvec;
expression E;
type T;
@@
- bvec->bv_page = (T)E
+ bvec_set_page(bvec, E)
@@
struct bio_vec bvec;
type T;
@@
- (T)bvec.bv_page
+ bvec_page(&bvec)
@@
struct bio_vec *bvec;
type T;
@@
- (T)bvec->bv_page
+ bvec_page(bvec)
@@
struct bio *bio;
expression E;
expression F;
type T;
@@
- bio->bi_io_vec[F].bv_page = (T)E
+ bvec_set_page(&bio->bi_io_vec[F], E)
@@
struct bio *bio;
expression E;
type T;
@@
- bio->bi_io_vec->bv_page = (T)E
+ bvec_set_page(bio->bi_io_vec, E)
@@
struct cached_dev *dc;
expression E;
type T;
@@
- dc->sb_bio.bi_io_vec->bv_page = (T)E
+ bvec_set_page(dc->sb_bio.bi_io_vec, E)
@@
struct cache *ca;
expression E;
expression F;
type T;
@@
- ca->sb_bio.bi_io_vec[F].bv_page = (T)E
+ bvec_set_page(&ca->sb_bio.bi_io_vec[F], E)
@@
struct cache *ca;
expression F;
@@
- ca->sb_bio.bi_io_vec[F].bv_page
+ bvec_page(&ca->sb_bio.bi_io_vec[F])
@@
struct cache *ca;
expression E;
expression F;
type T;
@@
- ca->sb_bio.bi_inline_vecs[F].bv_page = (T)E
+ bvec_set_page(&ca->sb_bio.bi_inline_vecs[F], E)
@@
struct cache *ca;
expression F;
@@
- ca->sb_bio.bi_inline_vecs[F].bv_page
+ bvec_page(&ca->sb_bio.bi_inline_vecs[F])
@@
struct cache *ca;
expression E;
type T;
@@
- ca->sb_bio.bi_io_vec->bv_page = (T)E
+ bvec_set_page(ca->sb_bio.bi_io_vec, E)
@@
struct bio *bio;
expression F;
@@
- bio->bi_io_vec[F].bv_page
+ bvec_page(&bio->bi_io_vec[F])
@@
struct bio bio;
expression F;
@@
- bio.bi_io_vec[F].bv_page
+ bvec_page(&bio.bi_io_vec[F])
@@
struct bio *bio;
@@
- bio->bi_io_vec->bv_page
+ bvec_page(bio->bi_io_vec)
@@
struct cached_dev *dc;
@@
- dc->sb_bio.bi_io_vec->bv_page
+ bvec_page(&dc->sb_bio->bi_io_vec)
@@
struct bio bio;
@@
- bio.bi_io_vec->bv_page
+ bvec_page(bio.bi_io_vec)
@@
struct bio_integrity_payload *bip;
expression E;
type T;
@@
- bip->bip_vec->bv_page = (T)E
+ bvec_set_page(bip->bip_vec, E)
@@
struct bio_integrity_payload *bip;
@@
- bip->bip_vec->bv_page
+ bvec_page(bip->bip_vec)
@@
struct bio_integrity_payload bip;
@@
- bip.bip_vec->bv_page
+ bvec_page(bip.bip_vec)
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Matthew Wilcox <willy@linux.intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Alasdair Kergon <agk@redhat.com>
Cc: Mike Snitzer <snitzer@redhat.com>
Cc: Chris Mason <clm@fb.com>
Cc: Boaz Harrosh <boaz@plexistor.com>
Cc: Theodore Ts'o <tytso@mit.edu>
Cc: Jan Kara <jack@suse.cz>
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Cc: Martin K. Petersen <martin.petersen@oracle.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
arch/powerpc/sysdev/axonram.c | 2 +
block/bio-integrity.c | 8 ++--
block/bio.c | 40 +++++++++++-----------
block/blk-core.c | 4 +-
block/blk-integrity.c | 3 +-
block/blk-lib.c | 2 +
block/blk-merge.c | 7 ++--
block/bounce.c | 24 ++++++-------
drivers/block/aoe/aoecmd.c | 8 ++--
drivers/block/brd.c | 2 +
drivers/block/drbd/drbd_bitmap.c | 5 ++-
drivers/block/drbd/drbd_main.c | 6 ++-
drivers/block/drbd/drbd_receiver.c | 4 +-
drivers/block/drbd/drbd_worker.c | 3 +-
drivers/block/floppy.c | 6 ++-
drivers/block/loop.c | 13 ++++---
drivers/block/nbd.c | 8 ++--
drivers/block/nvme-core.c | 2 +
drivers/block/pktcdvd.c | 11 +++---
drivers/block/pmem.c | 2 +
drivers/block/ps3disk.c | 2 +
drivers/block/ps3vram.c | 2 +
drivers/block/rbd.c | 2 +
drivers/block/rsxx/dma.c | 2 +
drivers/block/umem.c | 2 +
drivers/block/zram/zram_drv.c | 10 +++--
drivers/md/bcache/btree.c | 2 +
drivers/md/bcache/debug.c | 6 ++-
drivers/md/bcache/movinggc.c | 2 +
drivers/md/bcache/request.c | 6 ++-
drivers/md/bcache/super.c | 10 +++--
drivers/md/bcache/util.c | 5 +--
drivers/md/bcache/writeback.c | 2 +
drivers/md/dm-crypt.c | 12 +++---
drivers/md/dm-io.c | 2 +
drivers/md/dm-log-writes.c | 14 ++++----
drivers/md/dm-verity.c | 2 +
drivers/md/raid1.c | 50 ++++++++++++++-------------
drivers/md/raid10.c | 38 ++++++++++-----------
drivers/md/raid5.c | 6 ++-
drivers/s390/block/dasd_diag.c | 2 +
drivers/s390/block/dasd_eckd.c | 14 ++++----
drivers/s390/block/dasd_fba.c | 6 ++-
drivers/s390/block/dcssblk.c | 2 +
drivers/s390/block/scm_blk.c | 2 +
drivers/s390/block/scm_blk_cluster.c | 2 +
drivers/s390/block/xpram.c | 2 +
drivers/scsi/mpt2sas/mpt2sas_transport.c | 6 ++-
drivers/scsi/mpt3sas/mpt3sas_transport.c | 6 ++-
drivers/scsi/sd_dif.c | 4 +-
drivers/staging/lustre/lustre/llite/lloop.c | 2 +
drivers/target/target_core_file.c | 4 +-
drivers/xen/biomerge.c | 4 +-
fs/9p/vfs_addr.c | 2 +
fs/btrfs/check-integrity.c | 6 ++-
fs/btrfs/compression.c | 12 +++---
fs/btrfs/disk-io.c | 5 ++-
fs/btrfs/extent_io.c | 8 ++--
fs/btrfs/file-item.c | 8 ++--
fs/btrfs/inode.c | 19 ++++++----
fs/btrfs/raid56.c | 4 +-
fs/btrfs/volumes.c | 2 +
fs/buffer.c | 4 +-
fs/direct-io.c | 2 +
fs/exofs/ore.c | 4 +-
fs/exofs/ore_raid.c | 2 +
fs/ext4/page-io.c | 2 +
fs/ext4/readpage.c | 4 +-
fs/f2fs/data.c | 4 +-
fs/f2fs/segment.c | 2 +
fs/gfs2/lops.c | 4 +-
fs/jfs/jfs_logmgr.c | 4 +-
fs/logfs/dev_bdev.c | 10 +++--
fs/mpage.c | 2 +
fs/splice.c | 2 +
include/linux/blk_types.h | 10 +++++
kernel/power/block_io.c | 2 +
mm/page_io.c | 6 ++-
net/ceph/messenger.c | 2 +
79 files changed, 275 insertions(+), 250 deletions(-)
diff --git a/arch/powerpc/sysdev/axonram.c b/arch/powerpc/sysdev/axonram.c
index ee90db17b097..9bb5da7f2c0c 100644
--- a/arch/powerpc/sysdev/axonram.c
+++ b/arch/powerpc/sysdev/axonram.c
@@ -123,7 +123,7 @@ axon_ram_make_request(struct request_queue *queue, struct bio *bio)
return;
}
- user_mem = page_address(vec.bv_page) + vec.bv_offset;
+ user_mem = page_address(bvec_page(&vec)) + vec.bv_offset;
if (bio_data_dir(bio) == READ)
memcpy(user_mem, (void *) phys_mem, vec.bv_len);
else
diff --git a/block/bio-integrity.c b/block/bio-integrity.c
index 5cbd5d9ea61d..3add34cba048 100644
--- a/block/bio-integrity.c
+++ b/block/bio-integrity.c
@@ -101,7 +101,7 @@ void bio_integrity_free(struct bio *bio)
struct bio_set *bs = bio->bi_pool;
if (bip->bip_flags & BIP_BLOCK_INTEGRITY)
- kfree(page_address(bip->bip_vec->bv_page) +
+ kfree(page_address(bvec_page(bip->bip_vec)) +
bip->bip_vec->bv_offset);
if (bs) {
@@ -140,7 +140,7 @@ int bio_integrity_add_page(struct bio *bio, struct page *page,
iv = bip->bip_vec + bip->bip_vcnt;
- iv->bv_page = page;
+ bvec_set_page(iv, page);
iv->bv_len = len;
iv->bv_offset = offset;
bip->bip_vcnt++;
@@ -220,7 +220,7 @@ static int bio_integrity_process(struct bio *bio,
struct bio_vec bv;
struct bio_integrity_payload *bip = bio_integrity(bio);
unsigned int ret = 0;
- void *prot_buf = page_address(bip->bip_vec->bv_page) +
+ void *prot_buf = page_address(bvec_page(bip->bip_vec)) +
bip->bip_vec->bv_offset;
iter.disk_name = bio->bi_bdev->bd_disk->disk_name;
@@ -229,7 +229,7 @@ static int bio_integrity_process(struct bio *bio,
iter.prot_buf = prot_buf;
bio_for_each_segment(bv, bio, bviter) {
- void *kaddr = kmap_atomic(bv.bv_page);
+ void *kaddr = kmap_atomic(bvec_page(&bv));
iter.data_buf = kaddr + bv.bv_offset;
iter.data_size = bv.bv_len;
diff --git a/block/bio.c b/block/bio.c
index f66a4eae16ee..7100fd6d5898 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -508,7 +508,7 @@ void zero_fill_bio(struct bio *bio)
bio_for_each_segment(bv, bio, iter) {
char *data = bvec_kmap_irq(&bv, &flags);
memset(data, 0, bv.bv_len);
- flush_dcache_page(bv.bv_page);
+ flush_dcache_page(bvec_page(&bv));
bvec_kunmap_irq(data, &flags);
}
}
@@ -723,7 +723,7 @@ static int __bio_add_page(struct request_queue *q, struct bio *bio, struct page
if (bio->bi_vcnt > 0) {
struct bio_vec *prev = &bio->bi_io_vec[bio->bi_vcnt - 1];
- if (page == prev->bv_page &&
+ if (page == bvec_page(prev) &&
offset == prev->bv_offset + prev->bv_len) {
unsigned int prev_bv_len = prev->bv_len;
prev->bv_len += len;
@@ -768,7 +768,7 @@ static int __bio_add_page(struct request_queue *q, struct bio *bio, struct page
* cannot add the page
*/
bvec = &bio->bi_io_vec[bio->bi_vcnt];
- bvec->bv_page = page;
+ bvec_set_page(bvec, page);
bvec->bv_len = len;
bvec->bv_offset = offset;
bio->bi_vcnt++;
@@ -818,7 +818,7 @@ static int __bio_add_page(struct request_queue *q, struct bio *bio, struct page
return len;
failed:
- bvec->bv_page = NULL;
+ bvec_set_page(bvec, NULL);
bvec->bv_len = 0;
bvec->bv_offset = 0;
bio->bi_vcnt--;
@@ -948,10 +948,10 @@ int bio_alloc_pages(struct bio *bio, gfp_t gfp_mask)
struct bio_vec *bv;
bio_for_each_segment_all(bv, bio, i) {
- bv->bv_page = alloc_page(gfp_mask);
- if (!bv->bv_page) {
+ bvec_set_page(bv, alloc_page(gfp_mask));
+ if (!bvec_page(bv)) {
while (--bv >= bio->bi_io_vec)
- __free_page(bv->bv_page);
+ __free_page(bvec_page(bv));
return -ENOMEM;
}
}
@@ -1004,8 +1004,8 @@ void bio_copy_data(struct bio *dst, struct bio *src)
bytes = min(src_bv.bv_len, dst_bv.bv_len);
- src_p = kmap_atomic(src_bv.bv_page);
- dst_p = kmap_atomic(dst_bv.bv_page);
+ src_p = kmap_atomic(bvec_page(&src_bv));
+ dst_p = kmap_atomic(bvec_page(&dst_bv));
memcpy(dst_p + dst_bv.bv_offset,
src_p + src_bv.bv_offset,
@@ -1052,7 +1052,7 @@ static int bio_copy_from_iter(struct bio *bio, struct iov_iter iter)
bio_for_each_segment_all(bvec, bio, i) {
ssize_t ret;
- ret = copy_page_from_iter(bvec->bv_page,
+ ret = copy_page_from_iter(bvec_page(bvec),
bvec->bv_offset,
bvec->bv_len,
&iter);
@@ -1083,7 +1083,7 @@ static int bio_copy_to_iter(struct bio *bio, struct iov_iter iter)
bio_for_each_segment_all(bvec, bio, i) {
ssize_t ret;
- ret = copy_page_to_iter(bvec->bv_page,
+ ret = copy_page_to_iter(bvec_page(bvec),
bvec->bv_offset,
bvec->bv_len,
&iter);
@@ -1104,7 +1104,7 @@ static void bio_free_pages(struct bio *bio)
int i;
bio_for_each_segment_all(bvec, bio, i)
- __free_page(bvec->bv_page);
+ __free_page(bvec_page(bvec));
}
/**
@@ -1406,9 +1406,9 @@ static void __bio_unmap_user(struct bio *bio)
*/
bio_for_each_segment_all(bvec, bio, i) {
if (bio_data_dir(bio) == READ)
- set_page_dirty_lock(bvec->bv_page);
+ set_page_dirty_lock(bvec_page(bvec));
- page_cache_release(bvec->bv_page);
+ page_cache_release(bvec_page(bvec));
}
bio_put(bio);
@@ -1499,7 +1499,7 @@ static void bio_copy_kern_endio_read(struct bio *bio, int err)
int i;
bio_for_each_segment_all(bvec, bio, i) {
- memcpy(p, page_address(bvec->bv_page), bvec->bv_len);
+ memcpy(p, page_address(bvec_page(bvec)), bvec->bv_len);
p += bvec->bv_len;
}
@@ -1611,7 +1611,7 @@ void bio_set_pages_dirty(struct bio *bio)
int i;
bio_for_each_segment_all(bvec, bio, i) {
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
if (page && !PageCompound(page))
set_page_dirty_lock(page);
@@ -1624,7 +1624,7 @@ static void bio_release_pages(struct bio *bio)
int i;
bio_for_each_segment_all(bvec, bio, i) {
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
if (page)
put_page(page);
@@ -1678,11 +1678,11 @@ void bio_check_pages_dirty(struct bio *bio)
int i;
bio_for_each_segment_all(bvec, bio, i) {
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
if (PageDirty(page) || PageCompound(page)) {
page_cache_release(page);
- bvec->bv_page = NULL;
+ bvec_set_page(bvec, NULL);
} else {
nr_clean_pages++;
}
@@ -1736,7 +1736,7 @@ void bio_flush_dcache_pages(struct bio *bi)
struct bvec_iter iter;
bio_for_each_segment(bvec, bi, iter)
- flush_dcache_page(bvec.bv_page);
+ flush_dcache_page(bvec_page(&bvec));
}
EXPORT_SYMBOL(bio_flush_dcache_pages);
#endif
diff --git a/block/blk-core.c b/block/blk-core.c
index fd154b94447a..94d2c6ccf801 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1442,7 +1442,7 @@ void blk_add_request_payload(struct request *rq, struct page *page,
{
struct bio *bio = rq->bio;
- bio->bi_io_vec->bv_page = page;
+ bvec_set_page(bio->bi_io_vec, page);
bio->bi_io_vec->bv_offset = 0;
bio->bi_io_vec->bv_len = len;
@@ -2868,7 +2868,7 @@ void rq_flush_dcache_pages(struct request *rq)
struct bio_vec bvec;
rq_for_each_segment(bvec, rq, iter)
- flush_dcache_page(bvec.bv_page);
+ flush_dcache_page(bvec_page(&bvec));
}
EXPORT_SYMBOL_GPL(rq_flush_dcache_pages);
#endif
diff --git a/block/blk-integrity.c b/block/blk-integrity.c
index 79ffb4855af0..0458f31f075a 100644
--- a/block/blk-integrity.c
+++ b/block/blk-integrity.c
@@ -117,7 +117,8 @@ new_segment:
sg = sg_next(sg);
}
- sg_set_page(sg, iv.bv_page, iv.bv_len, iv.bv_offset);
+ sg_set_page(sg, bvec_page(&iv),
+ iv.bv_len, iv.bv_offset);
segments++;
}
diff --git a/block/blk-lib.c b/block/blk-lib.c
index 7688ee3f5d72..7931a09f86d6 100644
--- a/block/blk-lib.c
+++ b/block/blk-lib.c
@@ -187,7 +187,7 @@ int blkdev_issue_write_same(struct block_device *bdev, sector_t sector,
bio->bi_bdev = bdev;
bio->bi_private = &bb;
bio->bi_vcnt = 1;
- bio->bi_io_vec->bv_page = page;
+ bvec_set_page(bio->bi_io_vec, page);
bio->bi_io_vec->bv_offset = 0;
bio->bi_io_vec->bv_len = bdev_logical_block_size(bdev);
diff --git a/block/blk-merge.c b/block/blk-merge.c
index fd3fee81c23c..47ceefacd320 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -51,7 +51,7 @@ static unsigned int __blk_recalc_rq_segments(struct request_queue *q,
* never considered part of another segment, since
* that might change with the bounce page.
*/
- high = page_to_pfn(bv.bv_page) > queue_bounce_pfn(q);
+ high = page_to_pfn(bvec_page(&bv)) > queue_bounce_pfn(q);
if (!high && !highprv && cluster) {
if (seg_size + bv.bv_len
> queue_max_segment_size(q))
@@ -192,7 +192,7 @@ new_segment:
*sg = sg_next(*sg);
}
- sg_set_page(*sg, bvec->bv_page, nbytes, bvec->bv_offset);
+ sg_set_page(*sg, bvec_page(bvec), nbytes, bvec->bv_offset);
(*nsegs)++;
}
*bvprv = *bvec;
@@ -228,7 +228,8 @@ static int __blk_bios_map_sg(struct request_queue *q, struct bio *bio,
single_segment:
*sg = sglist;
bvec = bio_iovec(bio);
- sg_set_page(*sg, bvec.bv_page, bvec.bv_len, bvec.bv_offset);
+ sg_set_page(*sg, bvec_page(&bvec),
+ bvec.bv_len, bvec.bv_offset);
return 1;
}
diff --git a/block/bounce.c b/block/bounce.c
index ab21ba203d5c..0390e44d6e1b 100644
--- a/block/bounce.c
+++ b/block/bounce.c
@@ -55,7 +55,7 @@ static void bounce_copy_vec(struct bio_vec *to, unsigned char *vfrom)
unsigned char *vto;
local_irq_save(flags);
- vto = kmap_atomic(to->bv_page);
+ vto = kmap_atomic(bvec_page(to));
memcpy(vto + to->bv_offset, vfrom, to->bv_len);
kunmap_atomic(vto);
local_irq_restore(flags);
@@ -105,17 +105,17 @@ static void copy_to_high_bio_irq(struct bio *to, struct bio *from)
struct bvec_iter iter;
bio_for_each_segment(tovec, to, iter) {
- if (tovec.bv_page != fromvec->bv_page) {
+ if (bvec_page(&tovec) != bvec_page(fromvec)) {
/*
* fromvec->bv_offset and fromvec->bv_len might have
* been modified by the block layer, so use the original
* copy, bounce_copy_vec already uses tovec->bv_len
*/
- vfrom = page_address(fromvec->bv_page) +
+ vfrom = page_address(bvec_page(fromvec)) +
tovec.bv_offset;
bounce_copy_vec(&tovec, vfrom);
- flush_dcache_page(tovec.bv_page);
+ flush_dcache_page(bvec_page(&tovec));
}
fromvec++;
@@ -136,11 +136,11 @@ static void bounce_end_io(struct bio *bio, mempool_t *pool, int err)
*/
bio_for_each_segment_all(bvec, bio, i) {
org_vec = bio_orig->bi_io_vec + i;
- if (bvec->bv_page == org_vec->bv_page)
+ if (bvec_page(bvec) == bvec_page(org_vec))
continue;
- dec_zone_page_state(bvec->bv_page, NR_BOUNCE);
- mempool_free(bvec->bv_page, pool);
+ dec_zone_page_state(bvec_page(bvec), NR_BOUNCE);
+ mempool_free(bvec_page(bvec), pool);
}
bio_endio(bio_orig, err);
@@ -208,7 +208,7 @@ static void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig,
if (force)
goto bounce;
bio_for_each_segment(from, *bio_orig, iter)
- if (page_to_pfn(from.bv_page) > queue_bounce_pfn(q))
+ if (page_to_pfn(bvec_page(&from)) > queue_bounce_pfn(q))
goto bounce;
return;
@@ -216,20 +216,20 @@ bounce:
bio = bio_clone_bioset(*bio_orig, GFP_NOIO, fs_bio_set);
bio_for_each_segment_all(to, bio, i) {
- struct page *page = to->bv_page;
+ struct page *page = bvec_page(to);
if (page_to_pfn(page) <= queue_bounce_pfn(q) && !force)
continue;
- inc_zone_page_state(to->bv_page, NR_BOUNCE);
- to->bv_page = mempool_alloc(pool, q->bounce_gfp);
+ inc_zone_page_state(bvec_page(to), NR_BOUNCE);
+ bvec_set_page(to, mempool_alloc(pool, q->bounce_gfp));
if (rw == WRITE) {
char *vto, *vfrom;
flush_dcache_page(page);
- vto = page_address(to->bv_page) + to->bv_offset;
+ vto = page_address(bvec_page(to)) + to->bv_offset;
vfrom = kmap_atomic(page) + to->bv_offset;
memcpy(vto, vfrom, to->bv_len);
kunmap_atomic(vfrom);
diff --git a/drivers/block/aoe/aoecmd.c b/drivers/block/aoe/aoecmd.c
index 422b7d84f686..f0cbfe8c4bd8 100644
--- a/drivers/block/aoe/aoecmd.c
+++ b/drivers/block/aoe/aoecmd.c
@@ -300,7 +300,7 @@ skb_fillup(struct sk_buff *skb, struct bio *bio, struct bvec_iter iter)
struct bio_vec bv;
__bio_for_each_segment(bv, bio, iter, iter)
- skb_fill_page_desc(skb, frag++, bv.bv_page,
+ skb_fill_page_desc(skb, frag++, bvec_page(&bv),
bv.bv_offset, bv.bv_len);
}
@@ -874,7 +874,7 @@ bio_pageinc(struct bio *bio)
/* Non-zero page count for non-head members of
* compound pages is no longer allowed by the kernel.
*/
- page = compound_head(bv.bv_page);
+ page = compound_head(bvec_page(&bv));
atomic_inc(&page->_count);
}
}
@@ -887,7 +887,7 @@ bio_pagedec(struct bio *bio)
struct bvec_iter iter;
bio_for_each_segment(bv, bio, iter) {
- page = compound_head(bv.bv_page);
+ page = compound_head(bvec_page(&bv));
atomic_dec(&page->_count);
}
}
@@ -1092,7 +1092,7 @@ bvcpy(struct sk_buff *skb, struct bio *bio, struct bvec_iter iter, long cnt)
iter.bi_size = cnt;
__bio_for_each_segment(bv, bio, iter, iter) {
- char *p = page_address(bv.bv_page) + bv.bv_offset;
+ char *p = page_address(bvec_page(&bv)) + bv.bv_offset;
skb_copy_bits(skb, soff, p, bv.bv_len);
soff += bv.bv_len;
}
diff --git a/drivers/block/brd.c b/drivers/block/brd.c
index 64ab4951e9d6..115c6cf9cb43 100644
--- a/drivers/block/brd.c
+++ b/drivers/block/brd.c
@@ -349,7 +349,7 @@ static void brd_make_request(struct request_queue *q, struct bio *bio)
bio_for_each_segment(bvec, bio, iter) {
unsigned int len = bvec.bv_len;
- err = brd_do_bvec(brd, bvec.bv_page, len,
+ err = brd_do_bvec(brd, bvec_page(&bvec), len,
bvec.bv_offset, rw, sector);
if (err)
break;
diff --git a/drivers/block/drbd/drbd_bitmap.c b/drivers/block/drbd/drbd_bitmap.c
index 434c77dcc99e..37ba0f533e4b 100644
--- a/drivers/block/drbd/drbd_bitmap.c
+++ b/drivers/block/drbd/drbd_bitmap.c
@@ -946,7 +946,7 @@ static void drbd_bm_endio(struct bio *bio, int error)
struct drbd_bm_aio_ctx *ctx = bio->bi_private;
struct drbd_device *device = ctx->device;
struct drbd_bitmap *b = device->bitmap;
- unsigned int idx = bm_page_to_idx(bio->bi_io_vec[0].bv_page);
+ unsigned int idx = bm_page_to_idx(bvec_page(&bio->bi_io_vec[0]));
int uptodate = bio_flagged(bio, BIO_UPTODATE);
@@ -979,7 +979,8 @@ static void drbd_bm_endio(struct bio *bio, int error)
bm_page_unlock_io(device, idx);
if (ctx->flags & BM_AIO_COPY_PAGES)
- mempool_free(bio->bi_io_vec[0].bv_page, drbd_md_io_page_pool);
+ mempool_free(bvec_page(&bio->bi_io_vec[0]),
+ drbd_md_io_page_pool);
bio_put(bio);
diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c
index 81fde9ef7f8e..dc759609b2a6 100644
--- a/drivers/block/drbd/drbd_main.c
+++ b/drivers/block/drbd/drbd_main.c
@@ -1554,7 +1554,8 @@ static int _drbd_send_bio(struct drbd_peer_device *peer_device, struct bio *bio)
bio_for_each_segment(bvec, bio, iter) {
int err;
- err = _drbd_no_send_page(peer_device, bvec.bv_page,
+ err = _drbd_no_send_page(peer_device,
+ bvec_page(&bvec),
bvec.bv_offset, bvec.bv_len,
bio_iter_last(bvec, iter)
? 0 : MSG_MORE);
@@ -1573,7 +1574,8 @@ static int _drbd_send_zc_bio(struct drbd_peer_device *peer_device, struct bio *b
bio_for_each_segment(bvec, bio, iter) {
int err;
- err = _drbd_send_page(peer_device, bvec.bv_page,
+ err = _drbd_send_page(peer_device,
+ bvec_page(&bvec),
bvec.bv_offset, bvec.bv_len,
bio_iter_last(bvec, iter) ? 0 : MSG_MORE);
if (err)
diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c
index cee20354ac37..b4f16c6a0d73 100644
--- a/drivers/block/drbd/drbd_receiver.c
+++ b/drivers/block/drbd/drbd_receiver.c
@@ -1729,10 +1729,10 @@ static int recv_dless_read(struct drbd_peer_device *peer_device, struct drbd_req
D_ASSERT(peer_device->device, sector == bio->bi_iter.bi_sector);
bio_for_each_segment(bvec, bio, iter) {
- void *mapped = kmap(bvec.bv_page) + bvec.bv_offset;
+ void *mapped = kmap(bvec_page(&bvec)) + bvec.bv_offset;
expect = min_t(int, data_size, bvec.bv_len);
err = drbd_recv_all_warn(peer_device->connection, mapped, expect);
- kunmap(bvec.bv_page);
+ kunmap(bvec_page(&bvec));
if (err)
return err;
data_size -= expect;
diff --git a/drivers/block/drbd/drbd_worker.c b/drivers/block/drbd/drbd_worker.c
index d0fae55d871d..d4b6e432bf35 100644
--- a/drivers/block/drbd/drbd_worker.c
+++ b/drivers/block/drbd/drbd_worker.c
@@ -332,7 +332,8 @@ void drbd_csum_bio(struct crypto_hash *tfm, struct bio *bio, void *digest)
crypto_hash_init(&desc);
bio_for_each_segment(bvec, bio, iter) {
- sg_set_page(&sg, bvec.bv_page, bvec.bv_len, bvec.bv_offset);
+ sg_set_page(&sg, bvec_page(&bvec),
+ bvec.bv_len, bvec.bv_offset);
crypto_hash_update(&desc, &sg, sg.length);
}
crypto_hash_final(&desc, digest);
diff --git a/drivers/block/floppy.c b/drivers/block/floppy.c
index a08cda955285..6eae02e31731 100644
--- a/drivers/block/floppy.c
+++ b/drivers/block/floppy.c
@@ -2374,7 +2374,7 @@ static int buffer_chain_size(void)
size = 0;
rq_for_each_segment(bv, current_req, iter) {
- if (page_address(bv.bv_page) + bv.bv_offset != base + size)
+ if (page_address(bvec_page(&bv)) + bv.bv_offset != base + size)
break;
size += bv.bv_len;
@@ -2444,7 +2444,7 @@ static void copy_buffer(int ssize, int max_sector, int max_sector_2)
size = bv.bv_len;
SUPBOUND(size, remaining);
- buffer = page_address(bv.bv_page) + bv.bv_offset;
+ buffer = page_address(bvec_page(&bv)) + bv.bv_offset;
if (dma_buffer + size >
floppy_track_buffer + (max_buffer_sectors << 10) ||
dma_buffer < floppy_track_buffer) {
@@ -3805,7 +3805,7 @@ static int __floppy_read_block_0(struct block_device *bdev, int drive)
bio_init(&bio);
bio.bi_io_vec = &bio_vec;
- bio_vec.bv_page = page;
+ bvec_set_page(&bio_vec, page);
bio_vec.bv_len = size;
bio_vec.bv_offset = 0;
bio.bi_vcnt = 1;
diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index ae3fcb4199e9..08a52b42126a 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -261,12 +261,13 @@ static int lo_write_transfer(struct loop_device *lo, struct request *rq,
return -ENOMEM;
rq_for_each_segment(bvec, rq, iter) {
- ret = lo_do_transfer(lo, WRITE, page, 0, bvec.bv_page,
+ ret = lo_do_transfer(lo, WRITE, page, 0,
+ bvec_page(&bvec),
bvec.bv_offset, bvec.bv_len, pos >> 9);
if (unlikely(ret))
break;
- b.bv_page = page;
+ bvec_set_page(&b, page);
b.bv_offset = 0;
b.bv_len = bvec.bv_len;
ret = lo_write_bvec(lo->lo_backing_file, &b, &pos);
@@ -292,7 +293,7 @@ static int lo_read_simple(struct loop_device *lo, struct request *rq,
if (len < 0)
return len;
- flush_dcache_page(bvec.bv_page);
+ flush_dcache_page(bvec_page(&bvec));
if (len != bvec.bv_len) {
struct bio *bio;
@@ -324,7 +325,7 @@ static int lo_read_transfer(struct loop_device *lo, struct request *rq,
rq_for_each_segment(bvec, rq, iter) {
loff_t offset = pos;
- b.bv_page = page;
+ bvec_set_page(&b, page);
b.bv_offset = 0;
b.bv_len = bvec.bv_len;
@@ -335,12 +336,12 @@ static int lo_read_transfer(struct loop_device *lo, struct request *rq,
goto out_free_page;
}
- ret = lo_do_transfer(lo, READ, page, 0, bvec.bv_page,
+ ret = lo_do_transfer(lo, READ, page, 0, bvec_page(&bvec),
bvec.bv_offset, len, offset >> 9);
if (ret)
goto out_free_page;
- flush_dcache_page(bvec.bv_page);
+ flush_dcache_page(bvec_page(&bvec));
if (len != bvec.bv_len) {
struct bio *bio;
diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index 39e5f7fae3ef..dbab11437d2e 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -217,10 +217,10 @@ static inline int sock_send_bvec(struct nbd_device *nbd, struct bio_vec *bvec,
int flags)
{
int result;
- void *kaddr = kmap(bvec->bv_page);
+ void *kaddr = kmap(bvec_page(bvec));
result = sock_xmit(nbd, 1, kaddr + bvec->bv_offset,
bvec->bv_len, flags);
- kunmap(bvec->bv_page);
+ kunmap(bvec_page(bvec));
return result;
}
@@ -303,10 +303,10 @@ static struct request *nbd_find_request(struct nbd_device *nbd,
static inline int sock_recv_bvec(struct nbd_device *nbd, struct bio_vec *bvec)
{
int result;
- void *kaddr = kmap(bvec->bv_page);
+ void *kaddr = kmap(bvec_page(bvec));
result = sock_xmit(nbd, 0, kaddr + bvec->bv_offset, bvec->bv_len,
MSG_WAITALL);
- kunmap(bvec->bv_page);
+ kunmap(bvec_page(bvec));
return result;
}
diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c
index 85b8036deaa3..2727840266bf 100644
--- a/drivers/block/nvme-core.c
+++ b/drivers/block/nvme-core.c
@@ -516,7 +516,7 @@ static void nvme_dif_remap(struct request *req,
if (!bip)
return;
- pmap = kmap_atomic(bip->bip_vec->bv_page) + bip->bip_vec->bv_offset;
+ pmap = kmap_atomic(bvec_page(bip->bip_vec)) + bip->bip_vec->bv_offset;
p = pmap;
virt = bip_get_seed(bip);
diff --git a/drivers/block/pktcdvd.c b/drivers/block/pktcdvd.c
index 09e628dafd9d..c873290bd8bb 100644
--- a/drivers/block/pktcdvd.c
+++ b/drivers/block/pktcdvd.c
@@ -958,12 +958,12 @@ static void pkt_make_local_copy(struct packet_data *pkt, struct bio_vec *bvec)
p = 0;
offs = 0;
for (f = 0; f < pkt->frames; f++) {
- if (bvec[f].bv_page != pkt->pages[p]) {
- void *vfrom = kmap_atomic(bvec[f].bv_page) + bvec[f].bv_offset;
+ if (bvec_page(&bvec[f]) != pkt->pages[p]) {
+ void *vfrom = kmap_atomic(bvec_page(&bvec[f])) + bvec[f].bv_offset;
void *vto = page_address(pkt->pages[p]) + offs;
memcpy(vto, vfrom, CD_FRAMESIZE);
kunmap_atomic(vfrom);
- bvec[f].bv_page = pkt->pages[p];
+ bvec_set_page(&bvec[f], pkt->pages[p]);
bvec[f].bv_offset = offs;
} else {
BUG_ON(bvec[f].bv_offset != offs);
@@ -1307,9 +1307,10 @@ static void pkt_start_write(struct pktcdvd_device *pd, struct packet_data *pkt)
/* XXX: locking? */
for (f = 0; f < pkt->frames; f++) {
- bvec[f].bv_page = pkt->pages[(f * CD_FRAMESIZE) / PAGE_SIZE];
+ bvec_set_page(&bvec[f],
+ pkt->pages[(f * CD_FRAMESIZE) / PAGE_SIZE]);
bvec[f].bv_offset = (f * CD_FRAMESIZE) % PAGE_SIZE;
- if (!bio_add_page(pkt->w_bio, bvec[f].bv_page, CD_FRAMESIZE, bvec[f].bv_offset))
+ if (!bio_add_page(pkt->w_bio, bvec_page(&bvec[f]), CD_FRAMESIZE, bvec[f].bv_offset))
BUG();
}
pkt_dbg(2, pd, "vcnt=%d\n", pkt->w_bio->bi_vcnt);
diff --git a/drivers/block/pmem.c b/drivers/block/pmem.c
index eabf4a8d0085..41bb424533e6 100644
--- a/drivers/block/pmem.c
+++ b/drivers/block/pmem.c
@@ -77,7 +77,7 @@ static void pmem_make_request(struct request_queue *q, struct bio *bio)
rw = bio_data_dir(bio);
sector = bio->bi_iter.bi_sector;
bio_for_each_segment(bvec, bio, iter) {
- pmem_do_bvec(pmem, bvec.bv_page, bvec.bv_len, bvec.bv_offset,
+ pmem_do_bvec(pmem, bvec_page(&bvec), bvec.bv_len, bvec.bv_offset,
rw, sector);
sector += bvec.bv_len >> 9;
}
diff --git a/drivers/block/ps3disk.c b/drivers/block/ps3disk.c
index c120d70d3fb3..07ad0d9d9480 100644
--- a/drivers/block/ps3disk.c
+++ b/drivers/block/ps3disk.c
@@ -112,7 +112,7 @@ static void ps3disk_scatter_gather(struct ps3_storage_device *dev,
else
memcpy(buf, dev->bounce_buf+offset, size);
offset += size;
- flush_kernel_dcache_page(bvec.bv_page);
+ flush_kernel_dcache_page(bvec_page(&bvec));
bvec_kunmap_irq(buf, &flags);
i++;
}
diff --git a/drivers/block/ps3vram.c b/drivers/block/ps3vram.c
index ef45cfb98fd2..5db3311c2865 100644
--- a/drivers/block/ps3vram.c
+++ b/drivers/block/ps3vram.c
@@ -561,7 +561,7 @@ static struct bio *ps3vram_do_bio(struct ps3_system_bus_device *dev,
bio_for_each_segment(bvec, bio, iter) {
/* PS3 is ppc64, so we don't handle highmem */
- char *ptr = page_address(bvec.bv_page) + bvec.bv_offset;
+ char *ptr = page_address(bvec_page(&bvec)) + bvec.bv_offset;
size_t len = bvec.bv_len, retlen;
dev_dbg(&dev->core, " %s %zu bytes at offset %llu\n", op,
diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index ec6c5c6e1ac9..8aa209d929d4 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -1257,7 +1257,7 @@ static void zero_bio_chain(struct bio *chain, int start_ofs)
buf = bvec_kmap_irq(&bv, &flags);
memset(buf + remainder, 0,
bv.bv_len - remainder);
- flush_dcache_page(bv.bv_page);
+ flush_dcache_page(bvec_page(&bv));
bvec_kunmap_irq(buf, &flags);
}
pos += bv.bv_len;
diff --git a/drivers/block/rsxx/dma.c b/drivers/block/rsxx/dma.c
index cf8cd293abb5..6a7e128f9c32 100644
--- a/drivers/block/rsxx/dma.c
+++ b/drivers/block/rsxx/dma.c
@@ -737,7 +737,7 @@ int rsxx_dma_queue_bio(struct rsxx_cardinfo *card,
st = rsxx_queue_dma(card, &dma_list[tgt],
bio_data_dir(bio),
dma_off, dma_len,
- laddr, bvec.bv_page,
+ laddr, bvec_page(&bvec),
bv_off, cb, cb_data);
if (st)
goto bvec_err;
diff --git a/drivers/block/umem.c b/drivers/block/umem.c
index 4cf81b5bf0f7..c7f65e4ec874 100644
--- a/drivers/block/umem.c
+++ b/drivers/block/umem.c
@@ -366,7 +366,7 @@ static int add_bio(struct cardinfo *card)
vec = bio_iter_iovec(bio, card->current_iter);
dma_handle = pci_map_page(card->dev,
- vec.bv_page,
+ bvec_page(&vec),
vec.bv_offset,
vec.bv_len,
(rw == READ) ?
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index c94386aa563d..79e3b33c736c 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -409,7 +409,7 @@ static int page_zero_filled(void *ptr)
static void handle_zero_page(struct bio_vec *bvec)
{
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
void *user_mem;
user_mem = kmap_atomic(page);
@@ -497,7 +497,7 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
struct page *page;
unsigned char *user_mem, *uncmem = NULL;
struct zram_meta *meta = zram->meta;
- page = bvec->bv_page;
+ page = bvec_page(bvec);
bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value);
if (unlikely(!meta->table[index].handle) ||
@@ -568,7 +568,7 @@ static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, u32 index,
bool locked = false;
unsigned long alloced_pages;
- page = bvec->bv_page;
+ page = bvec_page(bvec);
if (is_partial_io(bvec)) {
/*
* This is a partial IO. We need to read the full page
@@ -924,7 +924,7 @@ static void __zram_make_request(struct zram *zram, struct bio *bio)
*/
struct bio_vec bv;
- bv.bv_page = bvec.bv_page;
+ bvec_set_page(&bv, bvec_page(&bvec));
bv.bv_len = max_transfer_size;
bv.bv_offset = bvec.bv_offset;
@@ -1011,7 +1011,7 @@ static int zram_rw_page(struct block_device *bdev, sector_t sector,
index = sector >> SECTORS_PER_PAGE_SHIFT;
offset = sector & (SECTORS_PER_PAGE - 1) << SECTOR_SHIFT;
- bv.bv_page = page;
+ bvec_set_page(&bv, page);
bv.bv_len = PAGE_SIZE;
bv.bv_offset = 0;
diff --git a/drivers/md/bcache/btree.c b/drivers/md/bcache/btree.c
index 00cde40db572..2e76e8b62902 100644
--- a/drivers/md/bcache/btree.c
+++ b/drivers/md/bcache/btree.c
@@ -366,7 +366,7 @@ static void btree_node_write_done(struct closure *cl)
int n;
bio_for_each_segment_all(bv, b->bio, n)
- __free_page(bv->bv_page);
+ __free_page(bvec_page(bv));
__btree_node_write_done(cl);
}
diff --git a/drivers/md/bcache/debug.c b/drivers/md/bcache/debug.c
index 8b1f1d5c1819..c355a02b94dd 100644
--- a/drivers/md/bcache/debug.c
+++ b/drivers/md/bcache/debug.c
@@ -120,8 +120,8 @@ void bch_data_verify(struct cached_dev *dc, struct bio *bio)
submit_bio_wait(READ_SYNC, check);
bio_for_each_segment(bv, bio, iter) {
- void *p1 = kmap_atomic(bv.bv_page);
- void *p2 = page_address(check->bi_io_vec[iter.bi_idx].bv_page);
+ void *p1 = kmap_atomic(bvec_page(&bv));
+ void *p2 = page_address(bvec_page(&check->bi_io_vec[iter.bi_idx]));
cache_set_err_on(memcmp(p1 + bv.bv_offset,
p2 + bv.bv_offset,
@@ -135,7 +135,7 @@ void bch_data_verify(struct cached_dev *dc, struct bio *bio)
}
bio_for_each_segment_all(bv2, check, i)
- __free_page(bv2->bv_page);
+ __free_page(bvec_page(bv2));
out_put:
bio_put(check);
}
diff --git a/drivers/md/bcache/movinggc.c b/drivers/md/bcache/movinggc.c
index cd7490311e51..744e7af4b160 100644
--- a/drivers/md/bcache/movinggc.c
+++ b/drivers/md/bcache/movinggc.c
@@ -48,7 +48,7 @@ static void write_moving_finish(struct closure *cl)
int i;
bio_for_each_segment_all(bv, bio, i)
- __free_page(bv->bv_page);
+ __free_page(bvec_page(bv));
if (io->op.replace_collision)
trace_bcache_gc_copy_collision(&io->w->key);
diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c
index ab43faddb447..e6378a998618 100644
--- a/drivers/md/bcache/request.c
+++ b/drivers/md/bcache/request.c
@@ -42,9 +42,9 @@ static void bio_csum(struct bio *bio, struct bkey *k)
uint64_t csum = 0;
bio_for_each_segment(bv, bio, iter) {
- void *d = kmap(bv.bv_page) + bv.bv_offset;
+ void *d = kmap(bvec_page(&bv)) + bv.bv_offset;
csum = bch_crc64_update(csum, d, bv.bv_len);
- kunmap(bv.bv_page);
+ kunmap(bvec_page(&bv));
}
k->ptr[KEY_PTRS(k)] = csum & (~0ULL >> 1);
@@ -690,7 +690,7 @@ static void cached_dev_cache_miss_done(struct closure *cl)
struct bio_vec *bv;
bio_for_each_segment_all(bv, s->iop.bio, i)
- __free_page(bv->bv_page);
+ __free_page(bvec_page(bv));
}
cached_dev_bio_complete(cl);
diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
index 4dd2bb7167f0..8d7cbba7ff7e 100644
--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -231,7 +231,7 @@ static void write_bdev_super_endio(struct bio *bio, int error)
static void __write_super(struct cache_sb *sb, struct bio *bio)
{
- struct cache_sb *out = page_address(bio->bi_io_vec[0].bv_page);
+ struct cache_sb *out = page_address(bvec_page(&bio->bi_io_vec[0]));
unsigned i;
bio->bi_iter.bi_sector = SB_SECTOR;
@@ -1172,7 +1172,7 @@ static void register_bdev(struct cache_sb *sb, struct page *sb_page,
bio_init(&dc->sb_bio);
dc->sb_bio.bi_max_vecs = 1;
dc->sb_bio.bi_io_vec = dc->sb_bio.bi_inline_vecs;
- dc->sb_bio.bi_io_vec[0].bv_page = sb_page;
+ bvec_set_page(dc->sb_bio.bi_io_vec, sb_page);
get_page(sb_page);
if (cached_dev_init(dc, sb->block_size << 9))
@@ -1811,8 +1811,8 @@ void bch_cache_release(struct kobject *kobj)
for (i = 0; i < RESERVE_NR; i++)
free_fifo(&ca->free[i]);
- if (ca->sb_bio.bi_inline_vecs[0].bv_page)
- put_page(ca->sb_bio.bi_io_vec[0].bv_page);
+ if (bvec_page(&ca->sb_bio.bi_inline_vecs[0]))
+ put_page(bvec_page(&ca->sb_bio.bi_io_vec[0]));
if (!IS_ERR_OR_NULL(ca->bdev))
blkdev_put(ca->bdev, FMODE_READ|FMODE_WRITE|FMODE_EXCL);
@@ -1870,7 +1870,7 @@ static void register_cache(struct cache_sb *sb, struct page *sb_page,
bio_init(&ca->sb_bio);
ca->sb_bio.bi_max_vecs = 1;
ca->sb_bio.bi_io_vec = ca->sb_bio.bi_inline_vecs;
- ca->sb_bio.bi_io_vec[0].bv_page = sb_page;
+ bvec_set_page(&ca->sb_bio.bi_io_vec[0], sb_page);
get_page(sb_page);
if (blk_queue_discard(bdev_get_queue(ca->bdev)))
diff --git a/drivers/md/bcache/util.c b/drivers/md/bcache/util.c
index db3ae4c2b223..d02f6d626529 100644
--- a/drivers/md/bcache/util.c
+++ b/drivers/md/bcache/util.c
@@ -238,9 +238,8 @@ void bch_bio_map(struct bio *bio, void *base)
start: bv->bv_len = min_t(size_t, PAGE_SIZE - bv->bv_offset,
size);
if (base) {
- bv->bv_page = is_vmalloc_addr(base)
- ? vmalloc_to_page(base)
- : virt_to_page(base);
+ bvec_set_page(bv,
+ is_vmalloc_addr(base) ? vmalloc_to_page(base) : virt_to_page(base));
base += bv->bv_len;
}
diff --git a/drivers/md/bcache/writeback.c b/drivers/md/bcache/writeback.c
index f1986bcd1bf0..6e9901c5dd66 100644
--- a/drivers/md/bcache/writeback.c
+++ b/drivers/md/bcache/writeback.c
@@ -133,7 +133,7 @@ static void write_dirty_finish(struct closure *cl)
int i;
bio_for_each_segment_all(bv, &io->bio, i)
- __free_page(bv->bv_page);
+ __free_page(bvec_page(bv));
/* This is kind of a dumb way of signalling errors. */
if (KEY_DIRTY(&w->key)) {
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 9eeea196328a..61784d3e9ac3 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -849,11 +849,11 @@ static int crypt_convert_block(struct crypt_config *cc,
dmreq->iv_sector = ctx->cc_sector;
dmreq->ctx = ctx;
sg_init_table(&dmreq->sg_in, 1);
- sg_set_page(&dmreq->sg_in, bv_in.bv_page, 1 << SECTOR_SHIFT,
+ sg_set_page(&dmreq->sg_in, bvec_page(&bv_in), 1 << SECTOR_SHIFT,
bv_in.bv_offset);
sg_init_table(&dmreq->sg_out, 1);
- sg_set_page(&dmreq->sg_out, bv_out.bv_page, 1 << SECTOR_SHIFT,
+ sg_set_page(&dmreq->sg_out, bvec_page(&bv_out), 1 << SECTOR_SHIFT,
bv_out.bv_offset);
bio_advance_iter(ctx->bio_in, &ctx->iter_in, 1 << SECTOR_SHIFT);
@@ -1002,7 +1002,7 @@ retry:
len = (remaining_size > PAGE_SIZE) ? PAGE_SIZE : remaining_size;
bvec = &clone->bi_io_vec[clone->bi_vcnt++];
- bvec->bv_page = page;
+ bvec_set_page(bvec, page);
bvec->bv_len = len;
bvec->bv_offset = 0;
@@ -1024,9 +1024,9 @@ static void crypt_free_buffer_pages(struct crypt_config *cc, struct bio *clone)
struct bio_vec *bv;
bio_for_each_segment_all(bv, clone, i) {
- BUG_ON(!bv->bv_page);
- mempool_free(bv->bv_page, cc->page_pool);
- bv->bv_page = NULL;
+ BUG_ON(!bvec_page(bv));
+ mempool_free(bvec_page(bv), cc->page_pool);
+ bvec_set_page(bv, NULL);
}
}
diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c
index 74adcd2c967e..b0537d3073a2 100644
--- a/drivers/md/dm-io.c
+++ b/drivers/md/dm-io.c
@@ -204,7 +204,7 @@ static void bio_get_page(struct dpages *dp, struct page **p,
unsigned long *len, unsigned *offset)
{
struct bio_vec *bvec = dp->context_ptr;
- *p = bvec->bv_page;
+ *p = bvec_page(bvec);
*len = bvec->bv_len - dp->context_u;
*offset = bvec->bv_offset + dp->context_u;
}
diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
index 93e08446a87d..d015f29b4a1c 100644
--- a/drivers/md/dm-log-writes.c
+++ b/drivers/md/dm-log-writes.c
@@ -162,7 +162,7 @@ static void log_end_io(struct bio *bio, int err)
}
bio_for_each_segment_all(bvec, bio, i)
- __free_page(bvec->bv_page);
+ __free_page(bvec_page(bvec));
put_io_block(lc);
bio_put(bio);
@@ -178,8 +178,8 @@ static void free_pending_block(struct log_writes_c *lc,
int i;
for (i = 0; i < block->vec_cnt; i++) {
- if (block->vecs[i].bv_page)
- __free_page(block->vecs[i].bv_page);
+ if (bvec_page(&block->vecs[i]))
+ __free_page(bvec_page(&block->vecs[i]));
}
kfree(block->data);
kfree(block);
@@ -277,7 +277,7 @@ static int log_one_block(struct log_writes_c *lc,
* The page offset is always 0 because we allocate a new page
* for every bvec in the original bio for simplicity sake.
*/
- ret = bio_add_page(bio, block->vecs[i].bv_page,
+ ret = bio_add_page(bio, bvec_page(&block->vecs[i]),
block->vecs[i].bv_len, 0);
if (ret != block->vecs[i].bv_len) {
atomic_inc(&lc->io_blocks);
@@ -294,7 +294,7 @@ static int log_one_block(struct log_writes_c *lc,
bio->bi_private = lc;
set_bit(BIO_UPTODATE, &bio->bi_flags);
- ret = bio_add_page(bio, block->vecs[i].bv_page,
+ ret = bio_add_page(bio, bvec_page(&block->vecs[i]),
block->vecs[i].bv_len, 0);
if (ret != block->vecs[i].bv_len) {
DMERR("Couldn't add page on new bio?");
@@ -641,12 +641,12 @@ static int log_writes_map(struct dm_target *ti, struct bio *bio)
return -ENOMEM;
}
- src = kmap_atomic(bv.bv_page);
+ src = kmap_atomic(bvec_page(&bv));
dst = kmap_atomic(page);
memcpy(dst, src + bv.bv_offset, bv.bv_len);
kunmap_atomic(dst);
kunmap_atomic(src);
- block->vecs[i].bv_page = page;
+ bvec_set_page(&block->vecs[i], page);
block->vecs[i].bv_len = bv.bv_len;
block->vec_cnt++;
i++;
diff --git a/drivers/md/dm-verity.c b/drivers/md/dm-verity.c
index 66616db33e6f..d56914eac6f2 100644
--- a/drivers/md/dm-verity.c
+++ b/drivers/md/dm-verity.c
@@ -408,7 +408,7 @@ test_block_hash:
unsigned len;
struct bio_vec bv = bio_iter_iovec(bio, io->iter);
- page = kmap_atomic(bv.bv_page);
+ page = kmap_atomic(bvec_page(&bv));
len = bv.bv_len;
if (likely(len >= todo))
len = todo;
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 9157a29c8dbf..78bc83fab933 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -134,8 +134,8 @@ static void * r1buf_pool_alloc(gfp_t gfp_flags, void *data)
if (!test_bit(MD_RECOVERY_REQUESTED, &pi->mddev->recovery)) {
for (i=0; i<RESYNC_PAGES ; i++)
for (j=1; j<pi->raid_disks; j++)
- r1_bio->bios[j]->bi_io_vec[i].bv_page =
- r1_bio->bios[0]->bi_io_vec[i].bv_page;
+ bvec_set_page(&r1_bio->bios[j]->bi_io_vec[i],
+ bvec_page(&r1_bio->bios[0]->bi_io_vec[i]));
}
r1_bio->master_bio = NULL;
@@ -147,7 +147,7 @@ out_free_pages:
struct bio_vec *bv;
bio_for_each_segment_all(bv, r1_bio->bios[j], i)
- __free_page(bv->bv_page);
+ __free_page(bvec_page(bv));
}
out_free_bio:
@@ -166,9 +166,9 @@ static void r1buf_pool_free(void *__r1_bio, void *data)
for (i = 0; i < RESYNC_PAGES; i++)
for (j = pi->raid_disks; j-- ;) {
if (j == 0 ||
- r1bio->bios[j]->bi_io_vec[i].bv_page !=
- r1bio->bios[0]->bi_io_vec[i].bv_page)
- safe_put_page(r1bio->bios[j]->bi_io_vec[i].bv_page);
+ bvec_page(&r1bio->bios[j]->bi_io_vec[i]) !=
+ bvec_page(&r1bio->bios[0]->bi_io_vec[i]))
+ safe_put_page(bvec_page(&r1bio->bios[j]->bi_io_vec[i]));
}
for (i=0 ; i < pi->raid_disks; i++)
bio_put(r1bio->bios[i]);
@@ -369,7 +369,7 @@ static void close_write(struct r1bio *r1_bio)
/* free extra copy of the data pages */
int i = r1_bio->behind_page_count;
while (i--)
- safe_put_page(r1_bio->behind_bvecs[i].bv_page);
+ safe_put_page(bvec_page(&r1_bio->behind_bvecs[i]));
kfree(r1_bio->behind_bvecs);
r1_bio->behind_bvecs = NULL;
}
@@ -1010,13 +1010,13 @@ static void alloc_behind_pages(struct bio *bio, struct r1bio *r1_bio)
bio_for_each_segment_all(bvec, bio, i) {
bvecs[i] = *bvec;
- bvecs[i].bv_page = alloc_page(GFP_NOIO);
- if (unlikely(!bvecs[i].bv_page))
+ bvec_set_page(&bvecs[i], alloc_page(GFP_NOIO));
+ if (unlikely(!bvec_page(&bvecs[i])))
goto do_sync_io;
- memcpy(kmap(bvecs[i].bv_page) + bvec->bv_offset,
- kmap(bvec->bv_page) + bvec->bv_offset, bvec->bv_len);
- kunmap(bvecs[i].bv_page);
- kunmap(bvec->bv_page);
+ memcpy(kmap(bvec_page(&bvecs[i])) + bvec->bv_offset,
+ kmap(bvec_page(bvec)) + bvec->bv_offset, bvec->bv_len);
+ kunmap(bvec_page(&bvecs[i]));
+ kunmap(bvec_page(bvec));
}
r1_bio->behind_bvecs = bvecs;
r1_bio->behind_page_count = bio->bi_vcnt;
@@ -1025,8 +1025,8 @@ static void alloc_behind_pages(struct bio *bio, struct r1bio *r1_bio)
do_sync_io:
for (i = 0; i < bio->bi_vcnt; i++)
- if (bvecs[i].bv_page)
- put_page(bvecs[i].bv_page);
+ if (bvec_page(&bvecs[i]))
+ put_page(bvec_page(&bvecs[i]));
kfree(bvecs);
pr_debug("%dB behind alloc failed, doing sync I/O\n",
bio->bi_iter.bi_size);
@@ -1397,7 +1397,8 @@ read_again:
* We trimmed the bio, so _all is legit
*/
bio_for_each_segment_all(bvec, mbio, j)
- bvec->bv_page = r1_bio->behind_bvecs[j].bv_page;
+ bvec_set_page(bvec,
+ bvec_page(&r1_bio->behind_bvecs[j]));
if (test_bit(WriteMostly, &conf->mirrors[i].rdev->flags))
atomic_inc(&r1_bio->behind_remaining);
}
@@ -1861,7 +1862,7 @@ static int fix_sync_read_error(struct r1bio *r1_bio)
*/
rdev = conf->mirrors[d].rdev;
if (sync_page_io(rdev, sect, s<<9,
- bio->bi_io_vec[idx].bv_page,
+ bvec_page(&bio->bi_io_vec[idx]),
READ, false)) {
success = 1;
break;
@@ -1917,7 +1918,7 @@ static int fix_sync_read_error(struct r1bio *r1_bio)
continue;
rdev = conf->mirrors[d].rdev;
if (r1_sync_page_io(rdev, sect, s,
- bio->bi_io_vec[idx].bv_page,
+ bvec_page(&bio->bi_io_vec[idx]),
WRITE) == 0) {
r1_bio->bios[d]->bi_end_io = NULL;
rdev_dec_pending(rdev, mddev);
@@ -1932,7 +1933,7 @@ static int fix_sync_read_error(struct r1bio *r1_bio)
continue;
rdev = conf->mirrors[d].rdev;
if (r1_sync_page_io(rdev, sect, s,
- bio->bi_io_vec[idx].bv_page,
+ bvec_page(&bio->bi_io_vec[idx]),
READ) != 0)
atomic_add(s, &rdev->corrected_errors);
}
@@ -2016,8 +2017,8 @@ static void process_checks(struct r1bio *r1_bio)
if (uptodate) {
for (j = vcnt; j-- ; ) {
struct page *p, *s;
- p = pbio->bi_io_vec[j].bv_page;
- s = sbio->bi_io_vec[j].bv_page;
+ p = bvec_page(&pbio->bi_io_vec[j]);
+ s = bvec_page(&sbio->bi_io_vec[j]);
if (memcmp(page_address(p),
page_address(s),
sbio->bi_io_vec[j].bv_len))
@@ -2226,7 +2227,7 @@ static int narrow_write_error(struct r1bio *r1_bio, int i)
unsigned vcnt = r1_bio->behind_page_count;
struct bio_vec *vec = r1_bio->behind_bvecs;
- while (!vec->bv_page) {
+ while (!bvec_page(vec)) {
vec++;
vcnt--;
}
@@ -2700,10 +2701,11 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr, int *skipp
for (i = 0 ; i < conf->raid_disks * 2; i++) {
bio = r1_bio->bios[i];
if (bio->bi_end_io) {
- page = bio->bi_io_vec[bio->bi_vcnt].bv_page;
+ page = bvec_page(&bio->bi_io_vec[bio->bi_vcnt]);
if (bio_add_page(bio, page, len, 0) == 0) {
/* stop here */
- bio->bi_io_vec[bio->bi_vcnt].bv_page = page;
+ bvec_set_page(&bio->bi_io_vec[bio->bi_vcnt],
+ page);
while (i > 0) {
i--;
bio = r1_bio->bios[i];
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index e793ab6b3570..61e0e6d415c7 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -181,16 +181,16 @@ static void * r10buf_pool_alloc(gfp_t gfp_flags, void *data)
/* we can share bv_page's during recovery
* and reshape */
struct bio *rbio = r10_bio->devs[0].bio;
- page = rbio->bi_io_vec[i].bv_page;
+ page = bvec_page(&rbio->bi_io_vec[i]);
get_page(page);
} else
page = alloc_page(gfp_flags);
if (unlikely(!page))
goto out_free_pages;
- bio->bi_io_vec[i].bv_page = page;
+ bvec_set_page(&bio->bi_io_vec[i], page);
if (rbio)
- rbio->bi_io_vec[i].bv_page = page;
+ bvec_set_page(&rbio->bi_io_vec[i], page);
}
}
@@ -198,10 +198,10 @@ static void * r10buf_pool_alloc(gfp_t gfp_flags, void *data)
out_free_pages:
for ( ; i > 0 ; i--)
- safe_put_page(bio->bi_io_vec[i-1].bv_page);
+ safe_put_page(bvec_page(&bio->bi_io_vec[i - 1]));
while (j--)
for (i = 0; i < RESYNC_PAGES ; i++)
- safe_put_page(r10_bio->devs[j].bio->bi_io_vec[i].bv_page);
+ safe_put_page(bvec_page(&r10_bio->devs[j].bio->bi_io_vec[i]));
j = 0;
out_free_bio:
for ( ; j < nalloc; j++) {
@@ -225,8 +225,8 @@ static void r10buf_pool_free(void *__r10_bio, void *data)
struct bio *bio = r10bio->devs[j].bio;
if (bio) {
for (i = 0; i < RESYNC_PAGES; i++) {
- safe_put_page(bio->bi_io_vec[i].bv_page);
- bio->bi_io_vec[i].bv_page = NULL;
+ safe_put_page(bvec_page(&bio->bi_io_vec[i]));
+ bvec_set_page(&bio->bi_io_vec[i], NULL);
}
bio_put(bio);
}
@@ -2074,8 +2074,8 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)
int len = PAGE_SIZE;
if (sectors < (len / 512))
len = sectors * 512;
- if (memcmp(page_address(fbio->bi_io_vec[j].bv_page),
- page_address(tbio->bi_io_vec[j].bv_page),
+ if (memcmp(page_address(bvec_page(&fbio->bi_io_vec[j])),
+ page_address(bvec_page(&tbio->bi_io_vec[j])),
len))
break;
sectors -= len/512;
@@ -2104,8 +2104,8 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)
tbio->bi_io_vec[j].bv_offset = 0;
tbio->bi_io_vec[j].bv_len = PAGE_SIZE;
- memcpy(page_address(tbio->bi_io_vec[j].bv_page),
- page_address(fbio->bi_io_vec[j].bv_page),
+ memcpy(page_address(bvec_page(&tbio->bi_io_vec[j])),
+ page_address(bvec_page(&fbio->bi_io_vec[j])),
PAGE_SIZE);
}
tbio->bi_end_io = end_sync_write;
@@ -2132,8 +2132,8 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)
if (r10_bio->devs[i].bio->bi_end_io != end_sync_write
&& r10_bio->devs[i].bio != fbio)
for (j = 0; j < vcnt; j++)
- memcpy(page_address(tbio->bi_io_vec[j].bv_page),
- page_address(fbio->bi_io_vec[j].bv_page),
+ memcpy(page_address(bvec_page(&tbio->bi_io_vec[j])),
+ page_address(bvec_page(&fbio->bi_io_vec[j])),
PAGE_SIZE);
d = r10_bio->devs[i].devnum;
atomic_inc(&r10_bio->remaining);
@@ -2191,7 +2191,7 @@ static void fix_recovery_read_error(struct r10bio *r10_bio)
ok = sync_page_io(rdev,
addr,
s << 9,
- bio->bi_io_vec[idx].bv_page,
+ bvec_page(&bio->bi_io_vec[idx]),
READ, false);
if (ok) {
rdev = conf->mirrors[dw].rdev;
@@ -2199,7 +2199,7 @@ static void fix_recovery_read_error(struct r10bio *r10_bio)
ok = sync_page_io(rdev,
addr,
s << 9,
- bio->bi_io_vec[idx].bv_page,
+ bvec_page(&bio->bi_io_vec[idx]),
WRITE, false);
if (!ok) {
set_bit(WriteErrorSeen, &rdev->flags);
@@ -3355,12 +3355,12 @@ static sector_t sync_request(struct mddev *mddev, sector_t sector_nr,
break;
for (bio= biolist ; bio ; bio=bio->bi_next) {
struct bio *bio2;
- page = bio->bi_io_vec[bio->bi_vcnt].bv_page;
+ page = bvec_page(&bio->bi_io_vec[bio->bi_vcnt]);
if (bio_add_page(bio, page, len, 0))
continue;
/* stop here */
- bio->bi_io_vec[bio->bi_vcnt].bv_page = page;
+ bvec_set_page(&bio->bi_io_vec[bio->bi_vcnt], page);
for (bio2 = biolist;
bio2 && bio2 != bio;
bio2 = bio2->bi_next) {
@@ -4430,7 +4430,7 @@ read_more:
nr_sectors = 0;
for (s = 0 ; s < max_sectors; s += PAGE_SIZE >> 9) {
- struct page *page = r10_bio->devs[0].bio->bi_io_vec[s/(PAGE_SIZE>>9)].bv_page;
+ struct page *page = bvec_page(&r10_bio->devs[0].bio->bi_io_vec[s / (PAGE_SIZE >> 9)]);
int len = (max_sectors - s) << 9;
if (len > PAGE_SIZE)
len = PAGE_SIZE;
@@ -4587,7 +4587,7 @@ static int handle_reshape_read_error(struct mddev *mddev,
success = sync_page_io(rdev,
addr,
s << 9,
- bvec[idx].bv_page,
+ bvec_page(&bvec[idx]),
READ, false);
if (success)
break;
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 77dfd720aaa0..6ec297699621 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -1006,7 +1006,7 @@ again:
if (test_bit(R5_SkipCopy, &sh->dev[i].flags))
WARN_ON(test_bit(R5_UPTODATE, &sh->dev[i].flags));
- sh->dev[i].vec.bv_page = sh->dev[i].page;
+ bvec_set_page(&sh->dev[i].vec, sh->dev[i].page);
bi->bi_vcnt = 1;
bi->bi_io_vec[0].bv_len = STRIPE_SIZE;
bi->bi_io_vec[0].bv_offset = 0;
@@ -1055,7 +1055,7 @@ again:
+ rrdev->data_offset);
if (test_bit(R5_SkipCopy, &sh->dev[i].flags))
WARN_ON(test_bit(R5_UPTODATE, &sh->dev[i].flags));
- sh->dev[i].rvec.bv_page = sh->dev[i].page;
+ bvec_set_page(&sh->dev[i].rvec, sh->dev[i].page);
rbi->bi_vcnt = 1;
rbi->bi_io_vec[0].bv_len = STRIPE_SIZE;
rbi->bi_io_vec[0].bv_offset = 0;
@@ -1132,7 +1132,7 @@ async_copy_data(int frombio, struct bio *bio, struct page **page,
if (clen > 0) {
b_offset += bvl.bv_offset;
- bio_page = bvl.bv_page;
+ bio_page = bvec_page(&bvl);
if (frombio) {
if (sh->raid_conf->skip_copy &&
b_offset == 0 && page_offset == 0 &&
diff --git a/drivers/s390/block/dasd_diag.c b/drivers/s390/block/dasd_diag.c
index c062f1620c58..89f39d00077d 100644
--- a/drivers/s390/block/dasd_diag.c
+++ b/drivers/s390/block/dasd_diag.c
@@ -545,7 +545,7 @@ static struct dasd_ccw_req *dasd_diag_build_cp(struct dasd_device *memdev,
dbio = dreq->bio;
recid = first_rec;
rq_for_each_segment(bv, req, iter) {
- dst = page_address(bv.bv_page) + bv.bv_offset;
+ dst = page_address(bvec_page(&bv)) + bv.bv_offset;
for (off = 0; off < bv.bv_len; off += blksize) {
memset(dbio, 0, sizeof (struct dasd_diag_bio));
dbio->type = rw_cmd;
diff --git a/drivers/s390/block/dasd_eckd.c b/drivers/s390/block/dasd_eckd.c
index 6215f6455eb8..926d458e5376 100644
--- a/drivers/s390/block/dasd_eckd.c
+++ b/drivers/s390/block/dasd_eckd.c
@@ -2612,7 +2612,7 @@ static struct dasd_ccw_req *dasd_eckd_build_cp_cmd_single(
/* Eckd can only do full blocks. */
return ERR_PTR(-EINVAL);
count += bv.bv_len >> (block->s2b_shift + 9);
- if (idal_is_needed (page_address(bv.bv_page), bv.bv_len))
+ if (idal_is_needed (page_address(bvec_page(&bv)), bv.bv_len))
cidaw += bv.bv_len >> (block->s2b_shift + 9);
}
/* Paranoia. */
@@ -2683,7 +2683,7 @@ static struct dasd_ccw_req *dasd_eckd_build_cp_cmd_single(
last_rec - recid + 1, cmd, basedev, blksize);
}
rq_for_each_segment(bv, req, iter) {
- dst = page_address(bv.bv_page) + bv.bv_offset;
+ dst = page_address(bvec_page(&bv)) + bv.bv_offset;
if (dasd_page_cache) {
char *copy = kmem_cache_alloc(dasd_page_cache,
GFP_DMA | __GFP_NOWARN);
@@ -2846,7 +2846,7 @@ static struct dasd_ccw_req *dasd_eckd_build_cp_cmd_track(
idaw_dst = NULL;
idaw_len = 0;
rq_for_each_segment(bv, req, iter) {
- dst = page_address(bv.bv_page) + bv.bv_offset;
+ dst = page_address(bvec_page(&bv)) + bv.bv_offset;
seg_len = bv.bv_len;
while (seg_len) {
if (new_track) {
@@ -3158,7 +3158,7 @@ static struct dasd_ccw_req *dasd_eckd_build_cp_tpm_track(
new_track = 1;
recid = first_rec;
rq_for_each_segment(bv, req, iter) {
- dst = page_address(bv.bv_page) + bv.bv_offset;
+ dst = page_address(bvec_page(&bv)) + bv.bv_offset;
seg_len = bv.bv_len;
while (seg_len) {
if (new_track) {
@@ -3191,7 +3191,7 @@ static struct dasd_ccw_req *dasd_eckd_build_cp_tpm_track(
}
} else {
rq_for_each_segment(bv, req, iter) {
- dst = page_address(bv.bv_page) + bv.bv_offset;
+ dst = page_address(bvec_page(&bv)) + bv.bv_offset;
last_tidaw = itcw_add_tidaw(itcw, 0x00,
dst, bv.bv_len);
if (IS_ERR(last_tidaw)) {
@@ -3411,7 +3411,7 @@ static struct dasd_ccw_req *dasd_raw_build_cp(struct dasd_device *startdev,
idaws = idal_create_words(idaws, rawpadpage, PAGE_SIZE);
}
rq_for_each_segment(bv, req, iter) {
- dst = page_address(bv.bv_page) + bv.bv_offset;
+ dst = page_address(bvec_page(&bv)) + bv.bv_offset;
seg_len = bv.bv_len;
if (cmd == DASD_ECKD_CCW_READ_TRACK)
memset(dst, 0, seg_len);
@@ -3475,7 +3475,7 @@ dasd_eckd_free_cp(struct dasd_ccw_req *cqr, struct request *req)
if (private->uses_cdl == 0 || recid > 2*blk_per_trk)
ccw++;
rq_for_each_segment(bv, req, iter) {
- dst = page_address(bv.bv_page) + bv.bv_offset;
+ dst = page_address(bvec_page(&bv)) + bv.bv_offset;
for (off = 0; off < bv.bv_len; off += blksize) {
/* Skip locate record. */
if (private->uses_cdl && recid <= 2*blk_per_trk)
diff --git a/drivers/s390/block/dasd_fba.c b/drivers/s390/block/dasd_fba.c
index c9262e78938b..a51cdc5db6dc 100644
--- a/drivers/s390/block/dasd_fba.c
+++ b/drivers/s390/block/dasd_fba.c
@@ -287,7 +287,7 @@ static struct dasd_ccw_req *dasd_fba_build_cp(struct dasd_device * memdev,
/* Fba can only do full blocks. */
return ERR_PTR(-EINVAL);
count += bv.bv_len >> (block->s2b_shift + 9);
- if (idal_is_needed (page_address(bv.bv_page), bv.bv_len))
+ if (idal_is_needed (page_address(bvec_page(&bv)), bv.bv_len))
cidaw += bv.bv_len / blksize;
}
/* Paranoia. */
@@ -324,7 +324,7 @@ static struct dasd_ccw_req *dasd_fba_build_cp(struct dasd_device * memdev,
}
recid = first_rec;
rq_for_each_segment(bv, req, iter) {
- dst = page_address(bv.bv_page) + bv.bv_offset;
+ dst = page_address(bvec_page(&bv)) + bv.bv_offset;
if (dasd_page_cache) {
char *copy = kmem_cache_alloc(dasd_page_cache,
GFP_DMA | __GFP_NOWARN);
@@ -397,7 +397,7 @@ dasd_fba_free_cp(struct dasd_ccw_req *cqr, struct request *req)
if (private->rdc_data.mode.bits.data_chain != 0)
ccw++;
rq_for_each_segment(bv, req, iter) {
- dst = page_address(bv.bv_page) + bv.bv_offset;
+ dst = page_address(bvec_page(&bv)) + bv.bv_offset;
for (off = 0; off < bv.bv_len; off += blksize) {
/* Skip locate record. */
if (private->rdc_data.mode.bits.data_chain == 0)
diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c
index da212813f2d5..5da8515b8fb9 100644
--- a/drivers/s390/block/dcssblk.c
+++ b/drivers/s390/block/dcssblk.c
@@ -857,7 +857,7 @@ dcssblk_make_request(struct request_queue *q, struct bio *bio)
index = (bio->bi_iter.bi_sector >> 3);
bio_for_each_segment(bvec, bio, iter) {
page_addr = (unsigned long)
- page_address(bvec.bv_page) + bvec.bv_offset;
+ page_address(bvec_page(&bvec)) + bvec.bv_offset;
source_addr = dev_info->start + (index<<12) + bytes_done;
if (unlikely((page_addr & 4095) != 0) || (bvec.bv_len & 4095) != 0)
// More paranoia.
diff --git a/drivers/s390/block/scm_blk.c b/drivers/s390/block/scm_blk.c
index 75d9896deccb..9bf2d42c1946 100644
--- a/drivers/s390/block/scm_blk.c
+++ b/drivers/s390/block/scm_blk.c
@@ -203,7 +203,7 @@ static int scm_request_prepare(struct scm_request *scmrq)
rq_for_each_segment(bv, req, iter) {
WARN_ON(bv.bv_offset);
msb->blk_count += bv.bv_len >> 12;
- aidaw->data_addr = (u64) page_address(bv.bv_page);
+ aidaw->data_addr = (u64) page_address(bvec_page(&bv));
aidaw++;
}
diff --git a/drivers/s390/block/scm_blk_cluster.c b/drivers/s390/block/scm_blk_cluster.c
index 7497ddde2dd6..a7e2fcb8f185 100644
--- a/drivers/s390/block/scm_blk_cluster.c
+++ b/drivers/s390/block/scm_blk_cluster.c
@@ -181,7 +181,7 @@ static int scm_prepare_cluster_request(struct scm_request *scmrq)
i++;
}
rq_for_each_segment(bv, req, iter) {
- aidaw->data_addr = (u64) page_address(bv.bv_page);
+ aidaw->data_addr = (u64) page_address(bvec_page(&bv));
aidaw++;
i++;
}
diff --git a/drivers/s390/block/xpram.c b/drivers/s390/block/xpram.c
index 7d4e9397ac31..44e80e13b643 100644
--- a/drivers/s390/block/xpram.c
+++ b/drivers/s390/block/xpram.c
@@ -202,7 +202,7 @@ static void xpram_make_request(struct request_queue *q, struct bio *bio)
index = (bio->bi_iter.bi_sector >> 3) + xdev->offset;
bio_for_each_segment(bvec, bio, iter) {
page_addr = (unsigned long)
- kmap(bvec.bv_page) + bvec.bv_offset;
+ kmap(bvec_page(&bvec)) + bvec.bv_offset;
bytes = bvec.bv_len;
if ((page_addr & 4095) != 0 || (bytes & 4095) != 0)
/* More paranoia. */
diff --git a/drivers/scsi/mpt2sas/mpt2sas_transport.c b/drivers/scsi/mpt2sas/mpt2sas_transport.c
index ff2500ab9ba4..788de1c250a3 100644
--- a/drivers/scsi/mpt2sas/mpt2sas_transport.c
+++ b/drivers/scsi/mpt2sas/mpt2sas_transport.c
@@ -1956,7 +1956,7 @@ _transport_smp_handler(struct Scsi_Host *shost, struct sas_rphy *rphy,
bio_for_each_segment(bvec, req->bio, iter) {
memcpy(pci_addr_out + offset,
- page_address(bvec.bv_page) + bvec.bv_offset,
+ page_address(bvec_page(&bvec)) + bvec.bv_offset,
bvec.bv_len);
offset += bvec.bv_len;
}
@@ -2107,12 +2107,12 @@ _transport_smp_handler(struct Scsi_Host *shost, struct sas_rphy *rphy,
le16_to_cpu(mpi_reply->ResponseDataLength);
bio_for_each_segment(bvec, rsp->bio, iter) {
if (bytes_to_copy <= bvec.bv_len) {
- memcpy(page_address(bvec.bv_page) +
+ memcpy(page_address(bvec_page(&bvec)) +
bvec.bv_offset, pci_addr_in +
offset, bytes_to_copy);
break;
} else {
- memcpy(page_address(bvec.bv_page) +
+ memcpy(page_address(bvec_page(&bvec)) +
bvec.bv_offset, pci_addr_in +
offset, bvec.bv_len);
bytes_to_copy -= bvec.bv_len;
diff --git a/drivers/scsi/mpt3sas/mpt3sas_transport.c b/drivers/scsi/mpt3sas/mpt3sas_transport.c
index efb98afc46e0..f187a1a05b9b 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_transport.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_transport.c
@@ -1939,7 +1939,7 @@ _transport_smp_handler(struct Scsi_Host *shost, struct sas_rphy *rphy,
bio_for_each_segment(bvec, req->bio, iter) {
memcpy(pci_addr_out + offset,
- page_address(bvec.bv_page) + bvec.bv_offset,
+ page_address(bvec_page(&bvec)) + bvec.bv_offset,
bvec.bv_len);
offset += bvec.bv_len;
}
@@ -2068,12 +2068,12 @@ _transport_smp_handler(struct Scsi_Host *shost, struct sas_rphy *rphy,
le16_to_cpu(mpi_reply->ResponseDataLength);
bio_for_each_segment(bvec, rsp->bio, iter) {
if (bytes_to_copy <= bvec.bv_len) {
- memcpy(page_address(bvec.bv_page) +
+ memcpy(page_address(bvec_page(&bvec)) +
bvec.bv_offset, pci_addr_in +
offset, bytes_to_copy);
break;
} else {
- memcpy(page_address(bvec.bv_page) +
+ memcpy(page_address(bvec_page(&bvec)) +
bvec.bv_offset, pci_addr_in +
offset, bvec.bv_len);
bytes_to_copy -= bvec.bv_len;
diff --git a/drivers/scsi/sd_dif.c b/drivers/scsi/sd_dif.c
index 5c06d292b94c..9e838bd5f2c3 100644
--- a/drivers/scsi/sd_dif.c
+++ b/drivers/scsi/sd_dif.c
@@ -134,7 +134,7 @@ void sd_dif_prepare(struct scsi_cmnd *scmd)
virt = bip_get_seed(bip) & 0xffffffff;
bip_for_each_vec(iv, bip, iter) {
- pi = kmap_atomic(iv.bv_page) + iv.bv_offset;
+ pi = kmap_atomic(bvec_page(&iv)) + iv.bv_offset;
for (j = 0; j < iv.bv_len; j += tuple_sz, pi++) {
@@ -181,7 +181,7 @@ void sd_dif_complete(struct scsi_cmnd *scmd, unsigned int good_bytes)
virt = bip_get_seed(bip) & 0xffffffff;
bip_for_each_vec(iv, bip, iter) {
- pi = kmap_atomic(iv.bv_page) + iv.bv_offset;
+ pi = kmap_atomic(bvec_page(&iv)) + iv.bv_offset;
for (j = 0; j < iv.bv_len; j += tuple_sz, pi++) {
diff --git a/drivers/staging/lustre/lustre/llite/lloop.c b/drivers/staging/lustre/lustre/llite/lloop.c
index 413a8408e3f5..044c435fae28 100644
--- a/drivers/staging/lustre/lustre/llite/lloop.c
+++ b/drivers/staging/lustre/lustre/llite/lloop.c
@@ -221,7 +221,7 @@ static int do_bio_lustrebacked(struct lloop_device *lo, struct bio *head)
BUG_ON(bvec.bv_offset != 0);
BUG_ON(bvec.bv_len != PAGE_CACHE_SIZE);
- pages[page_count] = bvec.bv_page;
+ pages[page_count] = bvec_page(&bvec);
offsets[page_count] = offset;
page_count++;
offset += bvec.bv_len;
diff --git a/drivers/target/target_core_file.c b/drivers/target/target_core_file.c
index f7e6e51aed36..47fffb9522fc 100644
--- a/drivers/target/target_core_file.c
+++ b/drivers/target/target_core_file.c
@@ -336,7 +336,7 @@ static int fd_do_rw(struct se_cmd *cmd, struct scatterlist *sgl,
}
for_each_sg(sgl, sg, sgl_nents, i) {
- bvec[i].bv_page = sg_page(sg);
+ bvec_set_page(&bvec[i], sg_page(sg));
bvec[i].bv_len = sg->length;
bvec[i].bv_offset = sg->offset;
@@ -462,7 +462,7 @@ fd_execute_write_same(struct se_cmd *cmd)
return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
for (i = 0; i < nolb; i++) {
- bvec[i].bv_page = sg_page(&cmd->t_data_sg[0]);
+ bvec_set_page(&bvec[i], sg_page(&cmd->t_data_sg[0]));
bvec[i].bv_len = cmd->t_data_sg[0].length;
bvec[i].bv_offset = cmd->t_data_sg[0].offset;
diff --git a/drivers/xen/biomerge.c b/drivers/xen/biomerge.c
index 0edb91c0de6b..7fcdcb2265f1 100644
--- a/drivers/xen/biomerge.c
+++ b/drivers/xen/biomerge.c
@@ -6,8 +6,8 @@
bool xen_biovec_phys_mergeable(const struct bio_vec *vec1,
const struct bio_vec *vec2)
{
- unsigned long mfn1 = pfn_to_mfn(page_to_pfn(vec1->bv_page));
- unsigned long mfn2 = pfn_to_mfn(page_to_pfn(vec2->bv_page));
+ unsigned long mfn1 = pfn_to_mfn(page_to_pfn(bvec_page(vec1)));
+ unsigned long mfn2 = pfn_to_mfn(page_to_pfn(bvec_page(vec2)));
return __BIOVEC_PHYS_MERGEABLE(vec1, vec2) &&
((mfn1 == mfn2) || ((mfn1+1) == mfn2));
diff --git a/fs/9p/vfs_addr.c b/fs/9p/vfs_addr.c
index e9e04376c52c..14b65a2c0d99 100644
--- a/fs/9p/vfs_addr.c
+++ b/fs/9p/vfs_addr.c
@@ -171,7 +171,7 @@ static int v9fs_vfs_writepage_locked(struct page *page)
else
len = PAGE_CACHE_SIZE;
- bvec.bv_page = page;
+ bvec_set_page(&bvec, page);
bvec.bv_offset = 0;
bvec.bv_len = len;
iov_iter_bvec(&from, ITER_BVEC | WRITE, &bvec, 1, len);
diff --git a/fs/btrfs/check-integrity.c b/fs/btrfs/check-integrity.c
index ce7dec88f4b8..bf6fec07b276 100644
--- a/fs/btrfs/check-integrity.c
+++ b/fs/btrfs/check-integrity.c
@@ -2997,11 +2997,11 @@ static void __btrfsic_submit_bio(int rw, struct bio *bio)
cur_bytenr = dev_bytenr;
for (i = 0; i < bio->bi_vcnt; i++) {
BUG_ON(bio->bi_io_vec[i].bv_len != PAGE_CACHE_SIZE);
- mapped_datav[i] = kmap(bio->bi_io_vec[i].bv_page);
+ mapped_datav[i] = kmap(bvec_page(&bio->bi_io_vec[i]));
if (!mapped_datav[i]) {
while (i > 0) {
i--;
- kunmap(bio->bi_io_vec[i].bv_page);
+ kunmap(bvec_page(&bio->bi_io_vec[i]));
}
kfree(mapped_datav);
goto leave;
@@ -3020,7 +3020,7 @@ static void __btrfsic_submit_bio(int rw, struct bio *bio)
NULL, rw);
while (i > 0) {
i--;
- kunmap(bio->bi_io_vec[i].bv_page);
+ kunmap(bvec_page(&bio->bi_io_vec[i]));
}
kfree(mapped_datav);
} else if (NULL != dev_state && (rw & REQ_FLUSH)) {
diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
index ce62324c78e7..8573fed0e8cb 100644
--- a/fs/btrfs/compression.c
+++ b/fs/btrfs/compression.c
@@ -208,7 +208,7 @@ csum_failed:
* checked so the end_io handlers know about it
*/
bio_for_each_segment_all(bvec, cb->orig_bio, i)
- SetPageChecked(bvec->bv_page);
+ SetPageChecked(bvec_page(bvec));
bio_endio(cb->orig_bio, 0);
}
@@ -459,7 +459,7 @@ static noinline int add_ra_bio_pages(struct inode *inode,
u64 end;
int misses = 0;
- page = cb->orig_bio->bi_io_vec[cb->orig_bio->bi_vcnt - 1].bv_page;
+ page = bvec_page(&cb->orig_bio->bi_io_vec[cb->orig_bio->bi_vcnt - 1]);
last_offset = (page_offset(page) + PAGE_CACHE_SIZE);
em_tree = &BTRFS_I(inode)->extent_tree;
tree = &BTRFS_I(inode)->io_tree;
@@ -592,7 +592,7 @@ int btrfs_submit_compressed_read(struct inode *inode, struct bio *bio,
/* we need the actual starting offset of this extent in the file */
read_lock(&em_tree->lock);
em = lookup_extent_mapping(em_tree,
- page_offset(bio->bi_io_vec->bv_page),
+ page_offset(bvec_page(bio->bi_io_vec)),
PAGE_CACHE_SIZE);
read_unlock(&em_tree->lock);
if (!em)
@@ -986,7 +986,7 @@ int btrfs_decompress_buf2page(char *buf, unsigned long buf_start,
unsigned long working_bytes = total_out - buf_start;
unsigned long bytes;
char *kaddr;
- struct page *page_out = bvec[*pg_index].bv_page;
+ struct page *page_out = bvec_page(&bvec[*pg_index]);
/*
* start byte is the first byte of the page we're currently
@@ -1031,7 +1031,7 @@ int btrfs_decompress_buf2page(char *buf, unsigned long buf_start,
if (*pg_index >= vcnt)
return 0;
- page_out = bvec[*pg_index].bv_page;
+ page_out = bvec_page(&bvec[*pg_index]);
*pg_offset = 0;
start_byte = page_offset(page_out) - disk_start;
@@ -1071,7 +1071,7 @@ void btrfs_clear_biovec_end(struct bio_vec *bvec, int vcnt,
unsigned long pg_offset)
{
while (pg_index < vcnt) {
- struct page *page = bvec[pg_index].bv_page;
+ struct page *page = bvec_page(&bvec[pg_index]);
unsigned long off = bvec[pg_index].bv_offset;
unsigned long len = bvec[pg_index].bv_len;
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 2ef9a4b72d06..a9ec0c6cfb81 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -876,8 +876,9 @@ static int btree_csum_one_bio(struct bio *bio)
int i, ret = 0;
bio_for_each_segment_all(bvec, bio, i) {
- root = BTRFS_I(bvec->bv_page->mapping->host)->root;
- ret = csum_dirty_buffer(root->fs_info, bvec->bv_page);
+ root = BTRFS_I(bvec_page(bvec)->mapping->host)->root;
+ ret = csum_dirty_buffer(root->fs_info,
+ bvec_page(bvec));
if (ret)
break;
}
diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 43af5a61ad25..9d5062f298c6 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -2489,7 +2489,7 @@ static void end_bio_extent_writepage(struct bio *bio, int err)
int i;
bio_for_each_segment_all(bvec, bio, i) {
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
/* We always issue full-page reads, but if some block
* in a page fails to read, blk_update_request() will
@@ -2563,7 +2563,7 @@ static void end_bio_extent_readpage(struct bio *bio, int err)
uptodate = 0;
bio_for_each_segment_all(bvec, bio, i) {
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
struct inode *inode = page->mapping->host;
pr_debug("end_bio_extent_readpage: bi_sector=%llu, err=%d, "
@@ -2751,7 +2751,7 @@ static int __must_check submit_one_bio(int rw, struct bio *bio,
{
int ret = 0;
struct bio_vec *bvec = bio->bi_io_vec + bio->bi_vcnt - 1;
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
struct extent_io_tree *tree = bio->bi_private;
u64 start;
@@ -3700,7 +3700,7 @@ static void end_bio_extent_buffer_writepage(struct bio *bio, int err)
int i, done;
bio_for_each_segment_all(bvec, bio, i) {
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
eb = (struct extent_buffer *)page->private;
BUG_ON(!eb);
diff --git a/fs/btrfs/file-item.c b/fs/btrfs/file-item.c
index 58ece6558430..835284e230d4 100644
--- a/fs/btrfs/file-item.c
+++ b/fs/btrfs/file-item.c
@@ -222,7 +222,7 @@ static int __btrfs_lookup_bio_sums(struct btrfs_root *root,
offset = logical_offset;
while (bio_index < bio->bi_vcnt) {
if (!dio)
- offset = page_offset(bvec->bv_page) + bvec->bv_offset;
+ offset = page_offset(bvec_page(bvec)) + bvec->bv_offset;
count = btrfs_find_ordered_sum(inode, offset, disk_bytenr,
(u32 *)csum, nblocks);
if (count)
@@ -448,7 +448,7 @@ int btrfs_csum_one_bio(struct btrfs_root *root, struct inode *inode,
if (contig)
offset = file_start;
else
- offset = page_offset(bvec->bv_page) + bvec->bv_offset;
+ offset = page_offset(bvec_page(bvec)) + bvec->bv_offset;
ordered = btrfs_lookup_ordered_extent(inode, offset);
BUG_ON(!ordered); /* Logic error */
@@ -457,7 +457,7 @@ int btrfs_csum_one_bio(struct btrfs_root *root, struct inode *inode,
while (bio_index < bio->bi_vcnt) {
if (!contig)
- offset = page_offset(bvec->bv_page) + bvec->bv_offset;
+ offset = page_offset(bvec_page(bvec)) + bvec->bv_offset;
if (offset >= ordered->file_offset + ordered->len ||
offset < ordered->file_offset) {
@@ -480,7 +480,7 @@ int btrfs_csum_one_bio(struct btrfs_root *root, struct inode *inode,
index = 0;
}
- data = kmap_atomic(bvec->bv_page);
+ data = kmap_atomic(bvec_page(bvec));
sums->sums[index] = ~(u32)0;
sums->sums[index] = btrfs_csum_data(data + bvec->bv_offset,
sums->sums[index],
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 8bb013672aee..89f2dc525859 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -7682,7 +7682,7 @@ static void btrfs_retry_endio_nocsum(struct bio *bio, int err)
done->uptodate = 1;
bio_for_each_segment_all(bvec, bio, i)
- clean_io_failure(done->inode, done->start, bvec->bv_page, 0);
+ clean_io_failure(done->inode, done->start, bvec_page(bvec), 0);
end:
complete(&done->done);
bio_put(bio);
@@ -7706,7 +7706,9 @@ try_again:
done.start = start;
init_completion(&done.done);
- ret = dio_read_error(inode, &io_bio->bio, bvec->bv_page, start,
+ ret = dio_read_error(inode, &io_bio->bio,
+ bvec_page(bvec),
+ start,
start + bvec->bv_len - 1,
io_bio->mirror_num,
btrfs_retry_endio_nocsum, &done);
@@ -7741,11 +7743,11 @@ static void btrfs_retry_endio(struct bio *bio, int err)
uptodate = 1;
bio_for_each_segment_all(bvec, bio, i) {
ret = __readpage_endio_check(done->inode, io_bio, i,
- bvec->bv_page, 0,
+ bvec_page(bvec), 0,
done->start, bvec->bv_len);
if (!ret)
clean_io_failure(done->inode, done->start,
- bvec->bv_page, 0);
+ bvec_page(bvec), 0);
else
uptodate = 0;
}
@@ -7771,7 +7773,8 @@ static int __btrfs_subio_endio_read(struct inode *inode,
done.inode = inode;
bio_for_each_segment_all(bvec, &io_bio->bio, i) {
- ret = __readpage_endio_check(inode, io_bio, i, bvec->bv_page,
+ ret = __readpage_endio_check(inode, io_bio, i,
+ bvec_page(bvec),
0, start, bvec->bv_len);
if (likely(!ret))
goto next;
@@ -7780,7 +7783,9 @@ try_again:
done.start = start;
init_completion(&done.done);
- ret = dio_read_error(inode, &io_bio->bio, bvec->bv_page, start,
+ ret = dio_read_error(inode, &io_bio->bio,
+ bvec_page(bvec),
+ start,
start + bvec->bv_len - 1,
io_bio->mirror_num,
btrfs_retry_endio, &done);
@@ -8076,7 +8081,7 @@ static int btrfs_submit_direct_hook(int rw, struct btrfs_dio_private *dip,
while (bvec <= (orig_bio->bi_io_vec + orig_bio->bi_vcnt - 1)) {
if (map_length < submit_len + bvec->bv_len ||
- bio_add_page(bio, bvec->bv_page, bvec->bv_len,
+ bio_add_page(bio, bvec_page(bvec), bvec->bv_len,
bvec->bv_offset) < bvec->bv_len) {
/*
* inc the count before we submit the bio so
diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c
index fa72068bd256..fc94998fee9b 100644
--- a/fs/btrfs/raid56.c
+++ b/fs/btrfs/raid56.c
@@ -1147,7 +1147,7 @@ static void index_rbio_pages(struct btrfs_raid_bio *rbio)
page_index = stripe_offset >> PAGE_CACHE_SHIFT;
for (i = 0; i < bio->bi_vcnt; i++) {
- p = bio->bi_io_vec[i].bv_page;
+ p = bvec_page(&bio->bi_io_vec[i]);
rbio->bio_pages[page_index + i] = p;
}
}
@@ -1428,7 +1428,7 @@ static void set_bio_pages_uptodate(struct bio *bio)
struct page *p;
for (i = 0; i < bio->bi_vcnt; i++) {
- p = bio->bi_io_vec[i].bv_page;
+ p = bvec_page(&bio->bi_io_vec[i]);
SetPageUptodate(p);
}
}
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index 96aebf3bcd5b..ed579c40d0e5 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -5791,7 +5791,7 @@ again:
return -ENOMEM;
while (bvec <= (first_bio->bi_io_vec + first_bio->bi_vcnt - 1)) {
- if (bio_add_page(bio, bvec->bv_page, bvec->bv_len,
+ if (bio_add_page(bio, bvec_page(bvec), bvec->bv_len,
bvec->bv_offset) < bvec->bv_len) {
u64 len = bio->bi_iter.bi_size;
diff --git a/fs/buffer.c b/fs/buffer.c
index c7a5602d01ee..e691107060f9 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -2992,7 +2992,7 @@ void guard_bio_eod(int rw, struct bio *bio)
/* ..and clear the end of the buffer for reads */
if ((rw & RW_MASK) == READ) {
- zero_user(bvec->bv_page, bvec->bv_offset + bvec->bv_len,
+ zero_user(bvec_page(bvec), bvec->bv_offset + bvec->bv_len,
truncated_bytes);
}
}
@@ -3022,7 +3022,7 @@ int _submit_bh(int rw, struct buffer_head *bh, unsigned long bio_flags)
bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9);
bio->bi_bdev = bh->b_bdev;
- bio->bi_io_vec[0].bv_page = bh->b_page;
+ bvec_set_page(&bio->bi_io_vec[0], bh->b_page);
bio->bi_io_vec[0].bv_len = bh->b_size;
bio->bi_io_vec[0].bv_offset = bh_offset(bh);
diff --git a/fs/direct-io.c b/fs/direct-io.c
index 745d2342651a..6c0e8c2b8217 100644
--- a/fs/direct-io.c
+++ b/fs/direct-io.c
@@ -468,7 +468,7 @@ static int dio_bio_complete(struct dio *dio, struct bio *bio)
bio_check_pages_dirty(bio); /* transfers ownership */
} else {
bio_for_each_segment_all(bvec, bio, i) {
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
if (dio->rw == READ && !PageCompound(page))
set_page_dirty_lock(page);
diff --git a/fs/exofs/ore.c b/fs/exofs/ore.c
index 7bd8ac8dfb28..4bd44bfed847 100644
--- a/fs/exofs/ore.c
+++ b/fs/exofs/ore.c
@@ -411,9 +411,9 @@ static void _clear_bio(struct bio *bio)
unsigned this_count = bv->bv_len;
if (likely(PAGE_SIZE == this_count))
- clear_highpage(bv->bv_page);
+ clear_highpage(bvec_page(bv));
else
- zero_user(bv->bv_page, bv->bv_offset, this_count);
+ zero_user(bvec_page(bv), bv->bv_offset, this_count);
}
}
diff --git a/fs/exofs/ore_raid.c b/fs/exofs/ore_raid.c
index 27cbdb697649..da76728824e6 100644
--- a/fs/exofs/ore_raid.c
+++ b/fs/exofs/ore_raid.c
@@ -438,7 +438,7 @@ static void _mark_read4write_pages_uptodate(struct ore_io_state *ios, int ret)
continue;
bio_for_each_segment_all(bv, bio, i) {
- struct page *page = bv->bv_page;
+ struct page *page = bvec_page(bv);
SetPageUptodate(page);
if (PageError(page))
diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c
index 5765f88b3904..1951399f54ec 100644
--- a/fs/ext4/page-io.c
+++ b/fs/ext4/page-io.c
@@ -65,7 +65,7 @@ static void ext4_finish_bio(struct bio *bio)
struct bio_vec *bvec;
bio_for_each_segment_all(bvec, bio, i) {
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
#ifdef CONFIG_EXT4_FS_ENCRYPTION
struct page *data_page = NULL;
struct ext4_crypto_ctx *ctx = NULL;
diff --git a/fs/ext4/readpage.c b/fs/ext4/readpage.c
index 171b9ac4b45e..9b58ce079f7d 100644
--- a/fs/ext4/readpage.c
+++ b/fs/ext4/readpage.c
@@ -60,7 +60,7 @@ static void completion_pages(struct work_struct *work)
int i;
bio_for_each_segment_all(bv, bio, i) {
- struct page *page = bv->bv_page;
+ struct page *page = bvec_page(bv);
int ret = ext4_decrypt(ctx, page);
if (ret) {
@@ -116,7 +116,7 @@ static void mpage_end_io(struct bio *bio, int err)
}
}
bio_for_each_segment_all(bv, bio, i) {
- struct page *page = bv->bv_page;
+ struct page *page = bvec_page(bv);
if (!err) {
SetPageUptodate(page);
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index b91b0e10678e..8aef4873c5d7 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -34,7 +34,7 @@ static void f2fs_read_end_io(struct bio *bio, int err)
int i;
bio_for_each_segment_all(bvec, bio, i) {
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
if (!err) {
SetPageUptodate(page);
@@ -54,7 +54,7 @@ static void f2fs_write_end_io(struct bio *bio, int err)
int i;
bio_for_each_segment_all(bvec, bio, i) {
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
if (unlikely(err)) {
set_page_dirty(page);
diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
index f939660941bb..3ae89adeb346 100644
--- a/fs/f2fs/segment.c
+++ b/fs/f2fs/segment.c
@@ -1314,7 +1314,7 @@ static inline bool is_merged_page(struct f2fs_sb_info *sbi,
goto out;
bio_for_each_segment_all(bvec, io->bio, i) {
- if (page == bvec->bv_page) {
+ if (page == bvec_page(bvec)) {
up_read(&io->io_rwsem);
return true;
}
diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c
index 2c1ae861dc94..2c1e14ca5971 100644
--- a/fs/gfs2/lops.c
+++ b/fs/gfs2/lops.c
@@ -173,7 +173,7 @@ static void gfs2_end_log_write_bh(struct gfs2_sbd *sdp, struct bio_vec *bvec,
int error)
{
struct buffer_head *bh, *next;
- struct page *page = bvec->bv_page;
+ struct page *page = bvec_page(bvec);
unsigned size;
bh = page_buffers(page);
@@ -215,7 +215,7 @@ static void gfs2_end_log_write(struct bio *bio, int error)
}
bio_for_each_segment_all(bvec, bio, i) {
- page = bvec->bv_page;
+ page = bvec_page(bvec);
if (page_has_buffers(page))
gfs2_end_log_write_bh(sdp, bvec, error);
else
diff --git a/fs/jfs/jfs_logmgr.c b/fs/jfs/jfs_logmgr.c
index bc462dcd7a40..4effe870b5aa 100644
--- a/fs/jfs/jfs_logmgr.c
+++ b/fs/jfs/jfs_logmgr.c
@@ -1999,7 +1999,7 @@ static int lbmRead(struct jfs_log * log, int pn, struct lbuf ** bpp)
bio->bi_iter.bi_sector = bp->l_blkno << (log->l2bsize - 9);
bio->bi_bdev = log->bdev;
- bio->bi_io_vec[0].bv_page = bp->l_page;
+ bvec_set_page(&bio->bi_io_vec[0], bp->l_page);
bio->bi_io_vec[0].bv_len = LOGPSIZE;
bio->bi_io_vec[0].bv_offset = bp->l_offset;
@@ -2145,7 +2145,7 @@ static void lbmStartIO(struct lbuf * bp)
bio = bio_alloc(GFP_NOFS, 1);
bio->bi_iter.bi_sector = bp->l_blkno << (log->l2bsize - 9);
bio->bi_bdev = log->bdev;
- bio->bi_io_vec[0].bv_page = bp->l_page;
+ bvec_set_page(&bio->bi_io_vec[0], bp->l_page);
bio->bi_io_vec[0].bv_len = LOGPSIZE;
bio->bi_io_vec[0].bv_offset = bp->l_offset;
diff --git a/fs/logfs/dev_bdev.c b/fs/logfs/dev_bdev.c
index 76279e11982d..7daa0e336fdf 100644
--- a/fs/logfs/dev_bdev.c
+++ b/fs/logfs/dev_bdev.c
@@ -22,7 +22,7 @@ static int sync_request(struct page *page, struct block_device *bdev, int rw)
bio_init(&bio);
bio.bi_max_vecs = 1;
bio.bi_io_vec = &bio_vec;
- bio_vec.bv_page = page;
+ bvec_set_page(&bio_vec, page);
bio_vec.bv_len = PAGE_SIZE;
bio_vec.bv_offset = 0;
bio.bi_vcnt = 1;
@@ -65,8 +65,8 @@ static void writeseg_end_io(struct bio *bio, int err)
BUG_ON(err);
bio_for_each_segment_all(bvec, bio, i) {
- end_page_writeback(bvec->bv_page);
- page_cache_release(bvec->bv_page);
+ end_page_writeback(bvec_page(bvec));
+ page_cache_release(bvec_page(bvec));
}
bio_put(bio);
if (atomic_dec_and_test(&super->s_pending_writes))
@@ -110,7 +110,7 @@ static int __bdev_writeseg(struct super_block *sb, u64 ofs, pgoff_t index,
}
page = find_lock_page(mapping, index + i);
BUG_ON(!page);
- bio->bi_io_vec[i].bv_page = page;
+ bvec_set_page(&bio->bi_io_vec[i], page);
bio->bi_io_vec[i].bv_len = PAGE_SIZE;
bio->bi_io_vec[i].bv_offset = 0;
@@ -200,7 +200,7 @@ static int do_erase(struct super_block *sb, u64 ofs, pgoff_t index,
bio = bio_alloc(GFP_NOFS, max_pages);
BUG_ON(!bio);
}
- bio->bi_io_vec[i].bv_page = super->s_erase_page;
+ bvec_set_page(&bio->bi_io_vec[i], super->s_erase_page);
bio->bi_io_vec[i].bv_len = PAGE_SIZE;
bio->bi_io_vec[i].bv_offset = 0;
}
diff --git a/fs/mpage.c b/fs/mpage.c
index 3e79220babac..c570a63e0913 100644
--- a/fs/mpage.c
+++ b/fs/mpage.c
@@ -48,7 +48,7 @@ static void mpage_end_io(struct bio *bio, int err)
int i;
bio_for_each_segment_all(bv, bio, i) {
- struct page *page = bv->bv_page;
+ struct page *page = bvec_page(bv);
page_endio(page, bio_data_dir(bio), err);
}
diff --git a/fs/splice.c b/fs/splice.c
index 476024bb6546..b627c2c55047 100644
--- a/fs/splice.c
+++ b/fs/splice.c
@@ -1000,7 +1000,7 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out,
goto done;
}
- array[n].bv_page = buf->page;
+ bvec_set_page(&array[n], buf->page);
array[n].bv_len = this_len;
array[n].bv_offset = buf->offset;
left -= this_len;
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index a1b25e35ea5f..d7167b50299f 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -26,6 +26,16 @@ struct bio_vec {
unsigned int bv_offset;
};
+static inline struct page *bvec_page(const struct bio_vec *bvec)
+{
+ return bvec->bv_page;
+}
+
+static inline void bvec_set_page(struct bio_vec *bvec, struct page *page)
+{
+ bvec->bv_page = page;
+}
+
#ifdef CONFIG_BLOCK
struct bvec_iter {
diff --git a/kernel/power/block_io.c b/kernel/power/block_io.c
index 9a58bc258810..f2824bacb84d 100644
--- a/kernel/power/block_io.c
+++ b/kernel/power/block_io.c
@@ -90,7 +90,7 @@ int hib_wait_on_bio_chain(struct bio **bio_chain)
struct page *page;
next_bio = bio->bi_private;
- page = bio->bi_io_vec[0].bv_page;
+ page = bvec_page(&bio->bi_io_vec[0]);
wait_on_page_locked(page);
if (!PageUptodate(page) || PageError(page))
ret = -EIO;
diff --git a/mm/page_io.c b/mm/page_io.c
index 6424869e275e..75738896b691 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -33,7 +33,7 @@ static struct bio *get_swap_bio(gfp_t gfp_flags,
if (bio) {
bio->bi_iter.bi_sector = map_swap_page(page, &bio->bi_bdev);
bio->bi_iter.bi_sector <<= PAGE_SHIFT - 9;
- bio->bi_io_vec[0].bv_page = page;
+ bvec_set_page(&bio->bi_io_vec[0], page);
bio->bi_io_vec[0].bv_len = PAGE_SIZE;
bio->bi_io_vec[0].bv_offset = 0;
bio->bi_vcnt = 1;
@@ -46,7 +46,7 @@ static struct bio *get_swap_bio(gfp_t gfp_flags,
void end_swap_bio_write(struct bio *bio, int err)
{
const int uptodate = test_bit(BIO_UPTODATE, &bio->bi_flags);
- struct page *page = bio->bi_io_vec[0].bv_page;
+ struct page *page = bvec_page(&bio->bi_io_vec[0]);
if (!uptodate) {
SetPageError(page);
@@ -72,7 +72,7 @@ void end_swap_bio_write(struct bio *bio, int err)
void end_swap_bio_read(struct bio *bio, int err)
{
const int uptodate = test_bit(BIO_UPTODATE, &bio->bi_flags);
- struct page *page = bio->bi_io_vec[0].bv_page;
+ struct page *page = bvec_page(&bio->bi_io_vec[0]);
if (!uptodate) {
SetPageError(page);
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index 967080a9f043..41e77e7813a2 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -842,7 +842,7 @@ static struct page *ceph_msg_data_bio_next(struct ceph_msg_data_cursor *cursor,
BUG_ON(*length > cursor->resid);
BUG_ON(*page_offset + *length > PAGE_SIZE);
- return bio_vec.bv_page;
+ return bvec_page(&bio_vec);
}
static bool ceph_msg_data_bio_advance(struct ceph_msg_data_cursor *cursor,
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH v3 03/11] block: convert .bv_page to .bv_pfn bio_vec
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
2015-05-12 4:29 ` [PATCH v3 01/11] arch: introduce __pfn_t for persistenti/device memory Dan Williams
2015-05-12 4:29 ` [PATCH v3 02/11] block: add helpers for accessing a bio_vec page Dan Williams
@ 2015-05-12 4:29 ` Dan Williams
2015-05-12 4:29 ` [PATCH v3 04/11] dma-mapping: allow archs to optionally specify a ->map_pfn() operation Dan Williams
` (8 subsequent siblings)
11 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:29 UTC (permalink / raw)
To: linux-kernel
Cc: axboe, linux-arch, riel, linux-nvdimm, Dave Hansen, david, mingo,
Julia Lawall, mgorman, j.glisse, linux-fsdevel, Matthew Wilcox,
akpm, hch
Carry an __pfn_t in a bio_vec rather than a 'struct page *' in support
of allowing a bio to reference unmapped (not struct page backed)
persistent memory.
This also fixes up the macros and static initializers that we were not
automatically converted by the Coccinelle script that introduced the
bvec_page() and bvec_set_page() helpers.
If CONFIG_DEV_PFN=n this is functionally equivalent to the status quo as
the __pfn_t helpers can assume that a __pfn_t always has a corresponding
struct page.
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Matthew Wilcox <willy@linux.intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
Documentation/block/biodoc.txt | 4 ++--
block/blk-integrity.c | 4 ++--
block/blk-merge.c | 6 +++---
block/bounce.c | 2 +-
drivers/md/bcache/btree.c | 2 +-
include/linux/bio.h | 23 ++++++++++++-----------
include/linux/blk_types.h | 24 +++++++++++++++++++++---
lib/iov_iter.c | 22 +++++++++++-----------
mm/page_io.c | 4 ++--
9 files changed, 55 insertions(+), 36 deletions(-)
diff --git a/Documentation/block/biodoc.txt b/Documentation/block/biodoc.txt
index fd12c0d835fd..3a10fd91e890 100644
--- a/Documentation/block/biodoc.txt
+++ b/Documentation/block/biodoc.txt
@@ -412,7 +412,7 @@ mapped to bio structures.
2.2 The bio struct
The bio structure uses a vector representation pointing to an array of tuples
-of <page, offset, len> to describe the i/o buffer, and has various other
+of <pfn, offset, len> to describe the i/o buffer, and has various other
fields describing i/o parameters and state that needs to be maintained for
performing the i/o.
@@ -420,7 +420,7 @@ Notice that this representation means that a bio has no virtual address
mapping at all (unlike buffer heads).
struct bio_vec {
- struct page *bv_page;
+ __pfn_t bv_pfn;
unsigned short bv_len;
unsigned short bv_offset;
};
diff --git a/block/blk-integrity.c b/block/blk-integrity.c
index 0458f31f075a..351198fbda3c 100644
--- a/block/blk-integrity.c
+++ b/block/blk-integrity.c
@@ -43,7 +43,7 @@ static const char *bi_unsupported_name = "unsupported";
*/
int blk_rq_count_integrity_sg(struct request_queue *q, struct bio *bio)
{
- struct bio_vec iv, ivprv = { NULL };
+ struct bio_vec iv, ivprv = BIO_VEC_INIT(ivprv);
unsigned int segments = 0;
unsigned int seg_size = 0;
struct bvec_iter iter;
@@ -89,7 +89,7 @@ EXPORT_SYMBOL(blk_rq_count_integrity_sg);
int blk_rq_map_integrity_sg(struct request_queue *q, struct bio *bio,
struct scatterlist *sglist)
{
- struct bio_vec iv, ivprv = { NULL };
+ struct bio_vec iv, ivprv = BIO_VEC_INIT(ivprv);
struct scatterlist *sg = NULL;
unsigned int segments = 0;
struct bvec_iter iter;
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 47ceefacd320..218ad1e57a49 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -13,7 +13,7 @@ static unsigned int __blk_recalc_rq_segments(struct request_queue *q,
struct bio *bio,
bool no_sg_merge)
{
- struct bio_vec bv, bvprv = { NULL };
+ struct bio_vec bv, bvprv = BIO_VEC_INIT(bvprv);
int cluster, high, highprv = 1;
unsigned int seg_size, nr_phys_segs;
struct bio *fbio, *bbio;
@@ -123,7 +123,7 @@ EXPORT_SYMBOL(blk_recount_segments);
static int blk_phys_contig_segment(struct request_queue *q, struct bio *bio,
struct bio *nxt)
{
- struct bio_vec end_bv = { NULL }, nxt_bv;
+ struct bio_vec end_bv = BIO_VEC_INIT(end_bv), nxt_bv;
struct bvec_iter iter;
if (!blk_queue_cluster(q))
@@ -202,7 +202,7 @@ static int __blk_bios_map_sg(struct request_queue *q, struct bio *bio,
struct scatterlist *sglist,
struct scatterlist **sg)
{
- struct bio_vec bvec, bvprv = { NULL };
+ struct bio_vec bvec, bvprv = BIO_VEC_INIT(bvprv);
struct bvec_iter iter;
int nsegs, cluster;
diff --git a/block/bounce.c b/block/bounce.c
index 0390e44d6e1b..4a3098067c81 100644
--- a/block/bounce.c
+++ b/block/bounce.c
@@ -64,7 +64,7 @@ static void bounce_copy_vec(struct bio_vec *to, unsigned char *vfrom)
#else /* CONFIG_HIGHMEM */
#define bounce_copy_vec(to, vfrom) \
- memcpy(page_address((to)->bv_page) + (to)->bv_offset, vfrom, (to)->bv_len)
+ memcpy(page_address(bvec_page(to)) + (to)->bv_offset, vfrom, (to)->bv_len)
#endif /* CONFIG_HIGHMEM */
diff --git a/drivers/md/bcache/btree.c b/drivers/md/bcache/btree.c
index 2e76e8b62902..36bbe29a806b 100644
--- a/drivers/md/bcache/btree.c
+++ b/drivers/md/bcache/btree.c
@@ -426,7 +426,7 @@ static void do_btree_node_write(struct btree *b)
void *base = (void *) ((unsigned long) i & ~(PAGE_SIZE - 1));
bio_for_each_segment_all(bv, b->bio, j)
- memcpy(page_address(bv->bv_page),
+ memcpy(page_address(bvec_page(bv)),
base + j * PAGE_SIZE, PAGE_SIZE);
bch_submit_bbio(b->bio, b->c, &k.key, 0);
diff --git a/include/linux/bio.h b/include/linux/bio.h
index da3a127c9958..a569e6ea1cd2 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -63,8 +63,8 @@
*/
#define __bvec_iter_bvec(bvec, iter) (&(bvec)[(iter).bi_idx])
-#define bvec_iter_page(bvec, iter) \
- (__bvec_iter_bvec((bvec), (iter))->bv_page)
+#define bvec_iter_pfn(bvec, iter) \
+ (__bvec_iter_bvec((bvec), (iter))->bv_pfn)
#define bvec_iter_len(bvec, iter) \
min((iter).bi_size, \
@@ -75,7 +75,7 @@
#define bvec_iter_bvec(bvec, iter) \
((struct bio_vec) { \
- .bv_page = bvec_iter_page((bvec), (iter)), \
+ .bv_pfn = bvec_iter_pfn((bvec), (iter)), \
.bv_len = bvec_iter_len((bvec), (iter)), \
.bv_offset = bvec_iter_offset((bvec), (iter)), \
})
@@ -83,14 +83,15 @@
#define bio_iter_iovec(bio, iter) \
bvec_iter_bvec((bio)->bi_io_vec, (iter))
-#define bio_iter_page(bio, iter) \
- bvec_iter_page((bio)->bi_io_vec, (iter))
+#define bio_iter_pfn(bio, iter) \
+ bvec_iter_pfn((bio)->bi_io_vec, (iter))
#define bio_iter_len(bio, iter) \
bvec_iter_len((bio)->bi_io_vec, (iter))
#define bio_iter_offset(bio, iter) \
bvec_iter_offset((bio)->bi_io_vec, (iter))
-#define bio_page(bio) bio_iter_page((bio), (bio)->bi_iter)
+#define bio_pfn(bio) bio_iter_pfn((bio), (bio)->bi_iter)
+#define bio_page(bio) __pfn_t_to_page(bio_pfn(bio))
#define bio_offset(bio) bio_iter_offset((bio), (bio)->bi_iter)
#define bio_iovec(bio) bio_iter_iovec((bio), (bio)->bi_iter)
@@ -150,8 +151,8 @@ static inline void *bio_data(struct bio *bio)
/*
* will die
*/
-#define bio_to_phys(bio) (page_to_phys(bio_page((bio))) + (unsigned long) bio_offset((bio)))
-#define bvec_to_phys(bv) (page_to_phys((bv)->bv_page) + (unsigned long) (bv)->bv_offset)
+#define bio_to_phys(bio) (__pfn_t_to_phys(bio_pfn((bio))) + (unsigned long) bio_offset((bio)))
+#define bvec_to_phys(bv) (__pfn_t_to_phys((bv)->bv_pfn) + (unsigned long) (bv)->bv_offset)
/*
* queues that have highmem support enabled may still need to revert to
@@ -160,7 +161,7 @@ static inline void *bio_data(struct bio *bio)
* I/O completely on that queue (see ide-dma for example)
*/
#define __bio_kmap_atomic(bio, iter) \
- (kmap_atomic(bio_iter_iovec((bio), (iter)).bv_page) + \
+ (kmap_atomic(bvec_page(bio_iter_iovec((bio), (iter)))) + \
bio_iter_iovec((bio), (iter)).bv_offset)
#define __bio_kunmap_atomic(addr) kunmap_atomic(addr)
@@ -490,7 +491,7 @@ static inline char *bvec_kmap_irq(struct bio_vec *bvec, unsigned long *flags)
* balancing is a lot nicer this way
*/
local_irq_save(*flags);
- addr = (unsigned long) kmap_atomic(bvec->bv_page);
+ addr = (unsigned long) kmap_atomic(bvec_page(bvec));
BUG_ON(addr & ~PAGE_MASK);
@@ -508,7 +509,7 @@ static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags)
#else
static inline char *bvec_kmap_irq(struct bio_vec *bvec, unsigned long *flags)
{
- return page_address(bvec->bv_page) + bvec->bv_offset;
+ return page_address(bvec_page(bvec)) + bvec->bv_offset;
}
static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags)
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index d7167b50299f..f50464e167b4 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -6,6 +6,7 @@
#define __LINUX_BLK_TYPES_H
#include <linux/types.h>
+#include <linux/mm.h>
struct bio_set;
struct bio;
@@ -21,19 +22,36 @@ typedef void (bio_destructor_t) (struct bio *);
* was unsigned short, but we might as well be ready for > 64kB I/O pages
*/
struct bio_vec {
- struct page *bv_page;
+#ifdef CONFIG_DEV_PFN
+ __pfn_t bv_pfn;
+#else
+ /*
+ * This union to be removed after v4.2-rc1 after all bv_page
+ * users are confirmed converted to prevent bisection errors
+ */
+ union {
+ __pfn_t bv_pfn;
+ struct page *bv_page;
+ };
+#endif
unsigned int bv_len;
unsigned int bv_offset;
};
+#define BIO_VEC_INIT(name) { .bv_pfn = { .data = 0 }, .bv_len = 0, \
+ .bv_offset = 0 }
+
+#define BIO_VEC(name) \
+ struct bio_vec name = BIO_VEC_INIT(name)
+
static inline struct page *bvec_page(const struct bio_vec *bvec)
{
- return bvec->bv_page;
+ return __pfn_t_to_page(bvec->bv_pfn);
}
static inline void bvec_set_page(struct bio_vec *bvec, struct page *page)
{
- bvec->bv_page = page;
+ bvec->bv_pfn = page_to_pfn_t(page);
}
#ifdef CONFIG_BLOCK
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 75232ad0a5e7..4276f6d1826c 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -61,7 +61,7 @@
__p = i->bvec; \
__v.bv_len = min_t(size_t, n, __p->bv_len - skip); \
if (likely(__v.bv_len)) { \
- __v.bv_page = __p->bv_page; \
+ __v.bv_pfn = __p->bv_pfn; \
__v.bv_offset = __p->bv_offset + skip; \
(void)(STEP); \
skip += __v.bv_len; \
@@ -72,7 +72,7 @@
__v.bv_len = min_t(size_t, n, __p->bv_len); \
if (unlikely(!__v.bv_len)) \
continue; \
- __v.bv_page = __p->bv_page; \
+ __v.bv_pfn = __p->bv_pfn; \
__v.bv_offset = __p->bv_offset; \
(void)(STEP); \
skip = __v.bv_len; \
@@ -395,7 +395,7 @@ size_t copy_to_iter(void *addr, size_t bytes, struct iov_iter *i)
iterate_and_advance(i, bytes, v,
__copy_to_user(v.iov_base, (from += v.iov_len) - v.iov_len,
v.iov_len),
- memcpy_to_page(v.bv_page, v.bv_offset,
+ memcpy_to_page(bvec_page(&v), v.bv_offset,
(from += v.bv_len) - v.bv_len, v.bv_len),
memcpy(v.iov_base, (from += v.iov_len) - v.iov_len, v.iov_len)
)
@@ -416,7 +416,7 @@ size_t copy_from_iter(void *addr, size_t bytes, struct iov_iter *i)
iterate_and_advance(i, bytes, v,
__copy_from_user((to += v.iov_len) - v.iov_len, v.iov_base,
v.iov_len),
- memcpy_from_page((to += v.bv_len) - v.bv_len, v.bv_page,
+ memcpy_from_page((to += v.bv_len) - v.bv_len, bvec_page(&v),
v.bv_offset, v.bv_len),
memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)
)
@@ -437,7 +437,7 @@ size_t copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i)
iterate_and_advance(i, bytes, v,
__copy_from_user_nocache((to += v.iov_len) - v.iov_len,
v.iov_base, v.iov_len),
- memcpy_from_page((to += v.bv_len) - v.bv_len, v.bv_page,
+ memcpy_from_page((to += v.bv_len) - v.bv_len, bvec_page(&v),
v.bv_offset, v.bv_len),
memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)
)
@@ -482,7 +482,7 @@ size_t iov_iter_zero(size_t bytes, struct iov_iter *i)
iterate_and_advance(i, bytes, v,
__clear_user(v.iov_base, v.iov_len),
- memzero_page(v.bv_page, v.bv_offset, v.bv_len),
+ memzero_page(bvec_page(&v), v.bv_offset, v.bv_len),
memset(v.iov_base, 0, v.iov_len)
)
@@ -497,7 +497,7 @@ size_t iov_iter_copy_from_user_atomic(struct page *page,
iterate_all_kinds(i, bytes, v,
__copy_from_user_inatomic((p += v.iov_len) - v.iov_len,
v.iov_base, v.iov_len),
- memcpy_from_page((p += v.bv_len) - v.bv_len, v.bv_page,
+ memcpy_from_page((p += v.bv_len) - v.bv_len, bvec_page(&v),
v.bv_offset, v.bv_len),
memcpy((p += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)
)
@@ -596,7 +596,7 @@ ssize_t iov_iter_get_pages(struct iov_iter *i,
0;}),({
/* can't be more than PAGE_SIZE */
*start = v.bv_offset;
- get_page(*pages = v.bv_page);
+ get_page(*pages = bvec_page(&v));
return v.bv_len;
}),({
return -EFAULT;
@@ -650,7 +650,7 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i,
*pages = p = get_pages_array(1);
if (!p)
return -ENOMEM;
- get_page(*p = v.bv_page);
+ get_page(*p = bvec_page(&v));
return v.bv_len;
}),({
return -EFAULT;
@@ -684,7 +684,7 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum,
}
err ? v.iov_len : 0;
}), ({
- char *p = kmap_atomic(v.bv_page);
+ char *p = kmap_atomic(bvec_page(&v));
next = csum_partial_copy_nocheck(p + v.bv_offset,
(to += v.bv_len) - v.bv_len,
v.bv_len, 0);
@@ -728,7 +728,7 @@ size_t csum_and_copy_to_iter(void *addr, size_t bytes, __wsum *csum,
}
err ? v.iov_len : 0;
}), ({
- char *p = kmap_atomic(v.bv_page);
+ char *p = kmap_atomic(bvec_page(&v));
next = csum_partial_copy_nocheck((from += v.bv_len) - v.bv_len,
p + v.bv_offset,
v.bv_len, 0);
diff --git a/mm/page_io.c b/mm/page_io.c
index 75738896b691..33c248527fd2 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -265,8 +265,8 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc,
struct file *swap_file = sis->swap_file;
struct address_space *mapping = swap_file->f_mapping;
struct bio_vec bv = {
- .bv_page = page,
- .bv_len = PAGE_SIZE,
+ .bv_pfn = page_to_pfn_t(page),
+ .bv_len = PAGE_SIZE,
.bv_offset = 0
};
struct iov_iter from;
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH v3 04/11] dma-mapping: allow archs to optionally specify a ->map_pfn() operation
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
` (2 preceding siblings ...)
2015-05-12 4:29 ` [PATCH v3 03/11] block: convert .bv_page to .bv_pfn bio_vec Dan Williams
@ 2015-05-12 4:29 ` Dan Williams
2015-05-12 4:29 ` [PATCH v3 05/11] scatterlist: use sg_phys() Dan Williams
` (7 subsequent siblings)
11 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:29 UTC (permalink / raw)
To: linux-kernel
Cc: linux-arch, axboe, riel, linux-nvdimm, david, hch, j.glisse,
mgorman, linux-fsdevel, akpm, mingo
This is in support of enabling block device drivers to perform DMA
to/from persistent memory which may not have a backing struct page
entry.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
arch/Kconfig | 3 +++
include/asm-generic/dma-mapping-common.h | 30 ++++++++++++++++++++++++++++++
include/linux/dma-debug.h | 23 +++++++++++++++++++----
include/linux/dma-mapping.h | 8 +++++++-
lib/dma-debug.c | 10 ++++++----
5 files changed, 65 insertions(+), 9 deletions(-)
diff --git a/arch/Kconfig b/arch/Kconfig
index a65eafb24997..f7f800860c00 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -203,6 +203,9 @@ config HAVE_DMA_ATTRS
config HAVE_DMA_CONTIGUOUS
bool
+config HAVE_DMA_PFN
+ bool
+
config GENERIC_SMP_IDLE_THREAD
bool
diff --git a/include/asm-generic/dma-mapping-common.h b/include/asm-generic/dma-mapping-common.h
index 940d5ec122c9..7305efb1bac6 100644
--- a/include/asm-generic/dma-mapping-common.h
+++ b/include/asm-generic/dma-mapping-common.h
@@ -17,9 +17,15 @@ static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr,
kmemcheck_mark_initialized(ptr, size);
BUG_ON(!valid_dma_direction(dir));
+#ifdef CONFIG_HAVE_DMA_PFN
+ addr = ops->map_pfn(dev, page_to_pfn_typed(virt_to_page(ptr)),
+ (unsigned long)ptr & ~PAGE_MASK, size,
+ dir, attrs);
+#else
addr = ops->map_page(dev, virt_to_page(ptr),
(unsigned long)ptr & ~PAGE_MASK, size,
dir, attrs);
+#endif
debug_dma_map_page(dev, virt_to_page(ptr),
(unsigned long)ptr & ~PAGE_MASK, size,
dir, addr, true);
@@ -73,6 +79,29 @@ static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg
ops->unmap_sg(dev, sg, nents, dir, attrs);
}
+#ifdef CONFIG_HAVE_DMA_PFN
+static inline dma_addr_t dma_map_pfn(struct device *dev, __pfn_t pfn,
+ size_t offset, size_t size,
+ enum dma_data_direction dir)
+{
+ struct dma_map_ops *ops = get_dma_ops(dev);
+ dma_addr_t addr;
+
+ BUG_ON(!valid_dma_direction(dir));
+ addr = ops->map_pfn(dev, pfn, offset, size, dir, NULL);
+ debug_dma_map_pfn(dev, pfn, offset, size, dir, addr, false);
+
+ return addr;
+}
+
+static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
+ size_t offset, size_t size,
+ enum dma_data_direction dir)
+{
+ kmemcheck_mark_initialized(page_address(page) + offset, size);
+ return dma_map_pfn(dev, page_to_pfn_typed(page), offset, size, dir);
+}
+#else
static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
size_t offset, size_t size,
enum dma_data_direction dir)
@@ -87,6 +116,7 @@ static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
return addr;
}
+#endif /* CONFIG_HAVE_DMA_PFN */
static inline void dma_unmap_page(struct device *dev, dma_addr_t addr,
size_t size, enum dma_data_direction dir)
diff --git a/include/linux/dma-debug.h b/include/linux/dma-debug.h
index fe8cb610deac..a3b4c8c0cd68 100644
--- a/include/linux/dma-debug.h
+++ b/include/linux/dma-debug.h
@@ -34,10 +34,18 @@ extern void dma_debug_init(u32 num_entries);
extern int dma_debug_resize_entries(u32 num_entries);
-extern void debug_dma_map_page(struct device *dev, struct page *page,
- size_t offset, size_t size,
- int direction, dma_addr_t dma_addr,
- bool map_single);
+extern void debug_dma_map_pfn(struct device *dev, __pfn_t pfn, size_t offset,
+ size_t size, int direction, dma_addr_t dma_addr,
+ bool map_single);
+
+static inline void debug_dma_map_page(struct device *dev, struct page *page,
+ size_t offset, size_t size,
+ int direction, dma_addr_t dma_addr,
+ bool map_single)
+{
+ return debug_dma_map_pfn(dev, page_to_pfn_t(page), offset, size,
+ direction, dma_addr, map_single);
+}
extern void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr);
@@ -109,6 +117,13 @@ static inline void debug_dma_map_page(struct device *dev, struct page *page,
{
}
+static inline void debug_dma_map_pfn(struct device *dev, __pfn_t pfn,
+ size_t offset, size_t size,
+ int direction, dma_addr_t dma_addr,
+ bool map_single)
+{
+}
+
static inline void debug_dma_mapping_error(struct device *dev,
dma_addr_t dma_addr)
{
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index ac07ff090919..d6437b493300 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -26,11 +26,17 @@ struct dma_map_ops {
int (*get_sgtable)(struct device *dev, struct sg_table *sgt, void *,
dma_addr_t, size_t, struct dma_attrs *attrs);
-
+#ifdef CONFIG_HAVE_DMA_PFN
+ dma_addr_t (*map_pfn)(struct device *dev, __pfn_t pfn,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir,
+ struct dma_attrs *attrs);
+#else
dma_addr_t (*map_page)(struct device *dev, struct page *page,
unsigned long offset, size_t size,
enum dma_data_direction dir,
struct dma_attrs *attrs);
+#endif
void (*unmap_page)(struct device *dev, dma_addr_t dma_handle,
size_t size, enum dma_data_direction dir,
struct dma_attrs *attrs);
diff --git a/lib/dma-debug.c b/lib/dma-debug.c
index ae4b65e17e64..c24de1cd8f81 100644
--- a/lib/dma-debug.c
+++ b/lib/dma-debug.c
@@ -1250,11 +1250,12 @@ out:
put_hash_bucket(bucket, &flags);
}
-void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
+void debug_dma_map_pfn(struct device *dev, __pfn_t pfn, size_t offset,
size_t size, int direction, dma_addr_t dma_addr,
bool map_single)
{
struct dma_debug_entry *entry;
+ struct page *page;
if (unlikely(dma_debug_disabled()))
return;
@@ -1268,7 +1269,7 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
entry->dev = dev;
entry->type = dma_debug_page;
- entry->pfn = page_to_pfn(page);
+ entry->pfn = __pfn_t_to_pfn(pfn);
entry->offset = offset,
entry->dev_addr = dma_addr;
entry->size = size;
@@ -1278,7 +1279,8 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
if (map_single)
entry->type = dma_debug_single;
- if (!PageHighMem(page)) {
+ page = __pfn_t_to_page(pfn);
+ if (page && !PageHighMem(page)) {
void *addr = page_address(page) + offset;
check_for_stack(dev, addr);
@@ -1287,7 +1289,7 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
add_dma_entry(entry);
}
-EXPORT_SYMBOL(debug_dma_map_page);
+EXPORT_SYMBOL(debug_dma_map_pfn);
void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
{
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH v3 05/11] scatterlist: use sg_phys()
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
` (3 preceding siblings ...)
2015-05-12 4:29 ` [PATCH v3 04/11] dma-mapping: allow archs to optionally specify a ->map_pfn() operation Dan Williams
@ 2015-05-12 4:29 ` Dan Williams
2015-05-12 5:24 ` Julia Lawall
2015-05-12 4:30 ` [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries Dan Williams
` (6 subsequent siblings)
11 siblings, 1 reply; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:29 UTC (permalink / raw)
To: linux-kernel
Cc: linux-arch, axboe, riel, linux-nvdimm, david, hch, Julia Lawall,
mgorman, j.glisse, linux-fsdevel, akpm, mingo
Coccinelle cleanup to replace open coded sg to physical address
translations. This is in preparation for introducing scatterlists that
reference pfn(s) without a backing struct page.
// sg_phys.cocci: convert usage page_to_phys(sg_page(sg)) to sg_phys(sg)
// usage: make coccicheck COCCI=sg_phys.cocci MODE=patch
virtual patch
virtual report
virtual org
@@
struct scatterlist *sg;
@@
- page_to_phys(sg_page(sg)) + sg->offset
+ sg_phys(sg)
@@
struct scatterlist *sg;
@@
- page_to_phys(sg_page(sg))
+ sg_phys(sg) - sg->offset
Cc: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
arch/arm/mm/dma-mapping.c | 2 +-
arch/microblaze/kernel/dma.c | 2 +-
drivers/iommu/intel-iommu.c | 4 ++--
drivers/iommu/iommu.c | 2 +-
drivers/staging/android/ion/ion_chunk_heap.c | 4 ++--
5 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 09c5fe3d30c2..43cc6a8fdacc 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -1502,7 +1502,7 @@ static int __map_sg_chunk(struct device *dev, struct scatterlist *sg,
return -ENOMEM;
for (count = 0, s = sg; count < (size >> PAGE_SHIFT); s = sg_next(s)) {
- phys_addr_t phys = page_to_phys(sg_page(s));
+ phys_addr_t phys = sg_phys(s) - s->offset;
unsigned int len = PAGE_ALIGN(s->offset + s->length);
if (!is_coherent &&
diff --git a/arch/microblaze/kernel/dma.c b/arch/microblaze/kernel/dma.c
index ed7ba8a11822..dcb3c594d626 100644
--- a/arch/microblaze/kernel/dma.c
+++ b/arch/microblaze/kernel/dma.c
@@ -61,7 +61,7 @@ static int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl,
/* FIXME this part of code is untested */
for_each_sg(sgl, sg, nents, i) {
sg->dma_address = sg_phys(sg);
- __dma_sync(page_to_phys(sg_page(sg)) + sg->offset,
+ __dma_sync(sg_phys(sg),
sg->length, direction);
}
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 68d43beccb7e..9b9ada71e0d3 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -1998,7 +1998,7 @@ static int __domain_mapping(struct dmar_domain *domain, unsigned long iov_pfn,
sg_res = aligned_nrpages(sg->offset, sg->length);
sg->dma_address = ((dma_addr_t)iov_pfn << VTD_PAGE_SHIFT) + sg->offset;
sg->dma_length = sg->length;
- pteval = page_to_phys(sg_page(sg)) | prot;
+ pteval = (sg_phys(sg) - sg->offset) | prot;
phys_pfn = pteval >> VTD_PAGE_SHIFT;
}
@@ -3302,7 +3302,7 @@ static int intel_nontranslate_map_sg(struct device *hddev,
for_each_sg(sglist, sg, nelems, i) {
BUG_ON(!sg_page(sg));
- sg->dma_address = page_to_phys(sg_page(sg)) + sg->offset;
+ sg->dma_address = sg_phys(sg);
sg->dma_length = sg->length;
}
return nelems;
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index d4f527e56679..59808fc9110d 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -1147,7 +1147,7 @@ size_t default_iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
min_pagesz = 1 << __ffs(domain->ops->pgsize_bitmap);
for_each_sg(sg, s, nents, i) {
- phys_addr_t phys = page_to_phys(sg_page(s)) + s->offset;
+ phys_addr_t phys = sg_phys(s);
/*
* We are mapping on IOMMU page boundaries, so offset within
diff --git a/drivers/staging/android/ion/ion_chunk_heap.c b/drivers/staging/android/ion/ion_chunk_heap.c
index 3e6ec2ee6802..b7da5d142aa9 100644
--- a/drivers/staging/android/ion/ion_chunk_heap.c
+++ b/drivers/staging/android/ion/ion_chunk_heap.c
@@ -81,7 +81,7 @@ static int ion_chunk_heap_allocate(struct ion_heap *heap,
err:
sg = table->sgl;
for (i -= 1; i >= 0; i--) {
- gen_pool_free(chunk_heap->pool, page_to_phys(sg_page(sg)),
+ gen_pool_free(chunk_heap->pool, sg_phys(sg) - sg->offset,
sg->length);
sg = sg_next(sg);
}
@@ -109,7 +109,7 @@ static void ion_chunk_heap_free(struct ion_buffer *buffer)
DMA_BIDIRECTIONAL);
for_each_sg(table->sgl, sg, table->nents, i) {
- gen_pool_free(chunk_heap->pool, page_to_phys(sg_page(sg)),
+ gen_pool_free(chunk_heap->pool, sg_phys(sg) - sg->offset,
sg->length);
}
chunk_heap->allocated -= allocated_size;
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v3 05/11] scatterlist: use sg_phys()
2015-05-12 4:29 ` [PATCH v3 05/11] scatterlist: use sg_phys() Dan Williams
@ 2015-05-12 5:24 ` Julia Lawall
2015-05-12 5:44 ` Dan Williams
0 siblings, 1 reply; 20+ messages in thread
From: Julia Lawall @ 2015-05-12 5:24 UTC (permalink / raw)
To: Dan Williams
Cc: linux-kernel, linux-arch, axboe, riel, linux-nvdimm, david, hch,
mgorman, j.glisse, linux-fsdevel, akpm, mingo
On Tue, 12 May 2015, Dan Williams wrote:
> Coccinelle cleanup to replace open coded sg to physical address
> translations. This is in preparation for introducing scatterlists that
> reference pfn(s) without a backing struct page.
>
> // sg_phys.cocci: convert usage page_to_phys(sg_page(sg)) to sg_phys(sg)
> // usage: make coccicheck COCCI=sg_phys.cocci MODE=patch
>
> virtual patch
> virtual report
> virtual org
Just for information, you don't need the three lines above. There are ony
useful when you want the semantic patch to support several kinds of
output.
julia
> @@
> struct scatterlist *sg;
> @@
>
> - page_to_phys(sg_page(sg)) + sg->offset
> + sg_phys(sg)
>
> @@
> struct scatterlist *sg;
> @@
>
> - page_to_phys(sg_page(sg))
> + sg_phys(sg) - sg->offset
>
> Cc: Julia Lawall <Julia.Lawall@lip6.fr>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> ---
> arch/arm/mm/dma-mapping.c | 2 +-
> arch/microblaze/kernel/dma.c | 2 +-
> drivers/iommu/intel-iommu.c | 4 ++--
> drivers/iommu/iommu.c | 2 +-
> drivers/staging/android/ion/ion_chunk_heap.c | 4 ++--
> 5 files changed, 7 insertions(+), 7 deletions(-)
>
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index 09c5fe3d30c2..43cc6a8fdacc 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -1502,7 +1502,7 @@ static int __map_sg_chunk(struct device *dev, struct scatterlist *sg,
> return -ENOMEM;
>
> for (count = 0, s = sg; count < (size >> PAGE_SHIFT); s = sg_next(s)) {
> - phys_addr_t phys = page_to_phys(sg_page(s));
> + phys_addr_t phys = sg_phys(s) - s->offset;
> unsigned int len = PAGE_ALIGN(s->offset + s->length);
>
> if (!is_coherent &&
> diff --git a/arch/microblaze/kernel/dma.c b/arch/microblaze/kernel/dma.c
> index ed7ba8a11822..dcb3c594d626 100644
> --- a/arch/microblaze/kernel/dma.c
> +++ b/arch/microblaze/kernel/dma.c
> @@ -61,7 +61,7 @@ static int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl,
> /* FIXME this part of code is untested */
> for_each_sg(sgl, sg, nents, i) {
> sg->dma_address = sg_phys(sg);
> - __dma_sync(page_to_phys(sg_page(sg)) + sg->offset,
> + __dma_sync(sg_phys(sg),
> sg->length, direction);
> }
>
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index 68d43beccb7e..9b9ada71e0d3 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -1998,7 +1998,7 @@ static int __domain_mapping(struct dmar_domain *domain, unsigned long iov_pfn,
> sg_res = aligned_nrpages(sg->offset, sg->length);
> sg->dma_address = ((dma_addr_t)iov_pfn << VTD_PAGE_SHIFT) + sg->offset;
> sg->dma_length = sg->length;
> - pteval = page_to_phys(sg_page(sg)) | prot;
> + pteval = (sg_phys(sg) - sg->offset) | prot;
> phys_pfn = pteval >> VTD_PAGE_SHIFT;
> }
>
> @@ -3302,7 +3302,7 @@ static int intel_nontranslate_map_sg(struct device *hddev,
>
> for_each_sg(sglist, sg, nelems, i) {
> BUG_ON(!sg_page(sg));
> - sg->dma_address = page_to_phys(sg_page(sg)) + sg->offset;
> + sg->dma_address = sg_phys(sg);
> sg->dma_length = sg->length;
> }
> return nelems;
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index d4f527e56679..59808fc9110d 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -1147,7 +1147,7 @@ size_t default_iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
> min_pagesz = 1 << __ffs(domain->ops->pgsize_bitmap);
>
> for_each_sg(sg, s, nents, i) {
> - phys_addr_t phys = page_to_phys(sg_page(s)) + s->offset;
> + phys_addr_t phys = sg_phys(s);
>
> /*
> * We are mapping on IOMMU page boundaries, so offset within
> diff --git a/drivers/staging/android/ion/ion_chunk_heap.c b/drivers/staging/android/ion/ion_chunk_heap.c
> index 3e6ec2ee6802..b7da5d142aa9 100644
> --- a/drivers/staging/android/ion/ion_chunk_heap.c
> +++ b/drivers/staging/android/ion/ion_chunk_heap.c
> @@ -81,7 +81,7 @@ static int ion_chunk_heap_allocate(struct ion_heap *heap,
> err:
> sg = table->sgl;
> for (i -= 1; i >= 0; i--) {
> - gen_pool_free(chunk_heap->pool, page_to_phys(sg_page(sg)),
> + gen_pool_free(chunk_heap->pool, sg_phys(sg) - sg->offset,
> sg->length);
> sg = sg_next(sg);
> }
> @@ -109,7 +109,7 @@ static void ion_chunk_heap_free(struct ion_buffer *buffer)
> DMA_BIDIRECTIONAL);
>
> for_each_sg(table->sgl, sg, table->nents, i) {
> - gen_pool_free(chunk_heap->pool, page_to_phys(sg_page(sg)),
> + gen_pool_free(chunk_heap->pool, sg_phys(sg) - sg->offset,
> sg->length);
> }
> chunk_heap->allocated -= allocated_size;
>
>
^ permalink raw reply [flat|nested] 20+ messages in thread* Re: [PATCH v3 05/11] scatterlist: use sg_phys()
2015-05-12 5:24 ` Julia Lawall
@ 2015-05-12 5:44 ` Dan Williams
0 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-12 5:44 UTC (permalink / raw)
To: Julia Lawall
Cc: linux-kernel, linux-arch, Jens Axboe, Rik van Riel, linux-nvdimm,
david, Christoph Hellwig, Mel Gorman, j.glisse, linux-fsdevel,
Andrew Morton, Ingo Molnar
On Mon, May 11, 2015 at 10:24 PM, Julia Lawall <julia.lawall@lip6.fr> wrote:
>
>
> On Tue, 12 May 2015, Dan Williams wrote:
>
>> Coccinelle cleanup to replace open coded sg to physical address
>> translations. This is in preparation for introducing scatterlists that
>> reference pfn(s) without a backing struct page.
>>
>> // sg_phys.cocci: convert usage page_to_phys(sg_page(sg)) to sg_phys(sg)
>> // usage: make coccicheck COCCI=sg_phys.cocci MODE=patch
>>
>> virtual patch
>> virtual report
>> virtual org
>
> Just for information, you don't need the three lines above. There are ony
> useful when you want the semantic patch to support several kinds of
> output.
>
Ok, I think I added them from copying a coccicheck script, and if I
delete virtual patch I get
"virtual rule patch not supported"
when running:
make coccicheck COCCI=sg_phys.cocci MODE=patch
I suspect I am invoking it wrong.
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
` (4 preceding siblings ...)
2015-05-12 4:29 ` [PATCH v3 05/11] scatterlist: use sg_phys() Dan Williams
@ 2015-05-12 4:30 ` Dan Williams
2015-05-13 18:35 ` Williams, Dan J
2015-05-12 4:30 ` [PATCH v3 07/11] x86: support dma_map_pfn() Dan Williams
` (5 subsequent siblings)
11 siblings, 1 reply; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:30 UTC (permalink / raw)
To: linux-kernel
Cc: linux-arch, axboe, riel, linux-nvdimm, david, hch, j.glisse,
mgorman, linux-fsdevel, Matthew Wilcox, akpm, mingo
From: Matthew Wilcox <willy@linux.intel.com>
Given that an offset will never be more than PAGE_SIZE, steal the unused
bits of the offset to implement a flags field. Move the existing "this
is a sg_chain() entry" flag to the new flags field, and add a new flag
(SG_FLAGS_PAGE) to indicate that there is a struct page backing for the
entry.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Matthew Wilcox <willy@linux.intel.com>
---
block/blk-merge.c | 2 -
drivers/dma/ste_dma40.c | 5 --
drivers/mmc/card/queue.c | 4 +-
include/asm-generic/scatterlist.h | 9 ++++
include/crypto/scatterwalk.h | 10 ++++
include/linux/scatterlist.h | 91 +++++++++++++++++++++++++++++++++----
6 files changed, 105 insertions(+), 16 deletions(-)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 218ad1e57a49..82a688551b72 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -267,7 +267,7 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq,
if (rq->cmd_flags & REQ_WRITE)
memset(q->dma_drain_buffer, 0, q->dma_drain_size);
- sg->page_link &= ~0x02;
+ sg_unmark_end(sg);
sg = sg_next(sg);
sg_set_page(sg, virt_to_page(q->dma_drain_buffer),
q->dma_drain_size,
diff --git a/drivers/dma/ste_dma40.c b/drivers/dma/ste_dma40.c
index 3c10f034d4b9..e8c00642cacb 100644
--- a/drivers/dma/ste_dma40.c
+++ b/drivers/dma/ste_dma40.c
@@ -2562,10 +2562,7 @@ dma40_prep_dma_cyclic(struct dma_chan *chan, dma_addr_t dma_addr,
dma_addr += period_len;
}
- sg[periods].offset = 0;
- sg_dma_len(&sg[periods]) = 0;
- sg[periods].page_link =
- ((unsigned long)sg | 0x01) & ~0x02;
+ sg_chain(sg, periods + 1, sg);
txd = d40_prep_sg(chan, sg, sg, periods, direction,
DMA_PREP_INTERRUPT);
diff --git a/drivers/mmc/card/queue.c b/drivers/mmc/card/queue.c
index 236d194c2883..127f76294e71 100644
--- a/drivers/mmc/card/queue.c
+++ b/drivers/mmc/card/queue.c
@@ -469,7 +469,7 @@ static unsigned int mmc_queue_packed_map_sg(struct mmc_queue *mq,
sg_set_buf(__sg, buf + offset, len);
offset += len;
remain -= len;
- (__sg++)->page_link &= ~0x02;
+ sg_unmark_end(__sg++);
sg_len++;
} while (remain);
}
@@ -477,7 +477,7 @@ static unsigned int mmc_queue_packed_map_sg(struct mmc_queue *mq,
list_for_each_entry(req, &packed->list, queuelist) {
sg_len += blk_rq_map_sg(mq->queue, req, __sg);
__sg = sg + (sg_len - 1);
- (__sg++)->page_link &= ~0x02;
+ sg_unmark_end(__sg++);
}
sg_mark_end(sg + (sg_len - 1));
return sg_len;
diff --git a/include/asm-generic/scatterlist.h b/include/asm-generic/scatterlist.h
index 5de07355fad4..959f51572a8e 100644
--- a/include/asm-generic/scatterlist.h
+++ b/include/asm-generic/scatterlist.h
@@ -7,8 +7,17 @@ struct scatterlist {
#ifdef CONFIG_DEBUG_SG
unsigned long sg_magic;
#endif
+#ifdef CONFIG_HAVE_DMA_PFN
+ union {
+ __pfn_t pfn;
+ struct scatterlist *next;
+ };
+ unsigned short offset;
+ unsigned short sg_flags;
+#else
unsigned long page_link;
unsigned int offset;
+#endif
unsigned int length;
dma_addr_t dma_address;
#ifdef CONFIG_NEED_SG_DMA_LENGTH
diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h
index 20e4226a2e14..7296d89a50b2 100644
--- a/include/crypto/scatterwalk.h
+++ b/include/crypto/scatterwalk.h
@@ -25,6 +25,15 @@
#include <linux/scatterlist.h>
#include <linux/sched.h>
+#ifdef CONFIG_HAVE_DMA_PFN
+/*
+ * If we're using PFNs, the architecture must also have been converted to
+ * support SG_CHAIN. So we can use the generic code instead of custom
+ * code.
+ */
+#define scatterwalk_sg_chain(prv, num, sgl) sg_chain(prv, num, sgl)
+#define scatterwalk_sg_next(sgl) sg_next(sgl)
+#else
static inline void scatterwalk_sg_chain(struct scatterlist *sg1, int num,
struct scatterlist *sg2)
{
@@ -32,6 +41,7 @@ static inline void scatterwalk_sg_chain(struct scatterlist *sg1, int num,
sg1[num - 1].page_link &= ~0x02;
sg1[num - 1].page_link |= 0x01;
}
+#endif
static inline void scatterwalk_crypto_chain(struct scatterlist *head,
struct scatterlist *sg,
diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h
index ed8f9e70df9b..9d423e559bdb 100644
--- a/include/linux/scatterlist.h
+++ b/include/linux/scatterlist.h
@@ -5,6 +5,7 @@
#include <linux/bug.h>
#include <linux/mm.h>
+#include <asm/page.h>
#include <asm/types.h>
#include <asm/scatterlist.h>
#include <asm/io.h>
@@ -18,8 +19,14 @@ struct sg_table {
/*
* Notes on SG table design.
*
- * Architectures must provide an unsigned long page_link field in the
- * scatterlist struct. We use that to place the page pointer AND encode
+ * Architectures may define CONFIG_HAVE_DMA_PFN to indicate that they wish
+ * to support SGLs that point to pages which do not have a struct page to
+ * describe them. If so, they should provide an sg_flags field in their
+ * scatterlist struct (see asm-generic for an example) as well as a pfn
+ * field.
+ *
+ * Otherwise, architectures must provide an unsigned long page_link field in
+ * the scatterlist struct. We use that to place the page pointer AND encode
* information about the sg table as well. The two lower bits are reserved
* for this information.
*
@@ -33,16 +40,25 @@ struct sg_table {
*/
#define SG_MAGIC 0x87654321
-
+#define SG_FLAGS_CHAIN 0x0001
+#define SG_FLAGS_LAST 0x0002
+#define SG_FLAGS_PAGE 0x0004
+
+#ifdef CONFIG_HAVE_DMA_PFN
+#define sg_is_chain(sg) ((sg)->sg_flags & SG_FLAGS_CHAIN)
+#define sg_is_last(sg) ((sg)->sg_flags & SG_FLAGS_LAST)
+#define sg_chain_ptr(sg) ((sg)->next)
+#else /* !CONFIG_HAVE_DMA_PFN */
/*
* We overload the LSB of the page pointer to indicate whether it's
* a valid sg entry, or whether it points to the start of a new scatterlist.
* Those low bits are there for everyone! (thanks mason :-)
*/
-#define sg_is_chain(sg) ((sg)->page_link & 0x01)
-#define sg_is_last(sg) ((sg)->page_link & 0x02)
+#define sg_is_chain(sg) ((sg)->page_link & SG_FLAGS_CHAIN)
+#define sg_is_last(sg) ((sg)->page_link & SG_FLAGS_LAST)
#define sg_chain_ptr(sg) \
((struct scatterlist *) ((sg)->page_link & ~0x03))
+#endif /* !CONFIG_HAVE_DMA_PFN */
/**
* sg_assign_page - Assign a given page to an SG entry
@@ -56,6 +72,14 @@ struct sg_table {
**/
static inline void sg_assign_page(struct scatterlist *sg, struct page *page)
{
+#ifdef CONFIG_HAVE_DMA_PFN
+#ifdef CONFIG_DEBUG_SG
+ BUG_ON(sg->sg_magic != SG_MAGIC);
+ BUG_ON(sg_is_chain(sg));
+#endif
+ sg->pfn = page_to_pfn_t(page);
+ sg->sg_flags |= SG_FLAGS_PAGE;
+#else /* !CONFIG_HAVE_DMA_PFN */
unsigned long page_link = sg->page_link & 0x3;
/*
@@ -68,6 +92,7 @@ static inline void sg_assign_page(struct scatterlist *sg, struct page *page)
BUG_ON(sg_is_chain(sg));
#endif
sg->page_link = page_link | (unsigned long) page;
+#endif /* !CONFIG_HAVE_DMA_PFN */
}
/**
@@ -88,17 +113,39 @@ static inline void sg_set_page(struct scatterlist *sg, struct page *page,
unsigned int len, unsigned int offset)
{
sg_assign_page(sg, page);
+ BUG_ON(offset > 65535);
sg->offset = offset;
sg->length = len;
}
+#ifdef CONFIG_HAVE_DMA_PFN
+static inline void sg_set_pfn(struct scatterlist *sg, __pfn_t pfn,
+ unsigned int len, unsigned int offset)
+{
+#ifdef CONFIG_DEBUG_SG
+ BUG_ON(sg->sg_magic != SG_MAGIC);
+ BUG_ON(sg_is_chain(sg));
+#endif
+ sg->pfn = pfn;
+ BUG_ON(offset > 65535);
+ sg->offset = offset;
+ sg->sg_flags = 0;
+ sg->length = len;
+}
+#endif
+
static inline struct page *sg_page(struct scatterlist *sg)
{
#ifdef CONFIG_DEBUG_SG
BUG_ON(sg->sg_magic != SG_MAGIC);
BUG_ON(sg_is_chain(sg));
#endif
+#ifdef CONFIG_HAVE_DMA_PFN
+ BUG_ON(!(sg->sg_flags & SG_FLAGS_PAGE));
+ return __pfn_t_to_page(sg->pfn);
+#else
return (struct page *)((sg)->page_link & ~0x3);
+#endif
}
/**
@@ -150,7 +197,12 @@ static inline void sg_chain(struct scatterlist *prv, unsigned int prv_nents,
* Set lowest bit to indicate a link pointer, and make sure to clear
* the termination bit if it happens to be set.
*/
+#ifdef CONFIG_HAVE_DMA_PFN
+ prv[prv_nents - 1].next = sgl;
+ prv[prv_nents - 1].sg_flags = SG_FLAGS_CHAIN;
+#else
prv[prv_nents - 1].page_link = ((unsigned long) sgl | 0x01) & ~0x02;
+#endif
}
/**
@@ -170,8 +222,13 @@ static inline void sg_mark_end(struct scatterlist *sg)
/*
* Set termination bit, clear potential chain bit
*/
- sg->page_link |= 0x02;
- sg->page_link &= ~0x01;
+#ifdef CONFIG_HAVE_DMA_PFN
+ sg->sg_flags |= SG_FLAGS_LAST;
+ sg->sg_flags &= ~SG_FLAGS_CHAIN;
+#else
+ sg->page_link |= SG_FLAGS_LAST;
+ sg->page_link &= ~SG_FLAGS_CHAIN;
+#endif
}
/**
@@ -187,7 +244,11 @@ static inline void sg_unmark_end(struct scatterlist *sg)
#ifdef CONFIG_DEBUG_SG
BUG_ON(sg->sg_magic != SG_MAGIC);
#endif
- sg->page_link &= ~0x02;
+#ifdef CONFIG_HAVE_DMA_PFN
+ sg->sg_flags &= ~SG_FLAGS_LAST;
+#else
+ sg->page_link &= ~SG_FLAGS_LAST;
+#endif
}
/**
@@ -202,7 +263,11 @@ static inline void sg_unmark_end(struct scatterlist *sg)
**/
static inline dma_addr_t sg_phys(struct scatterlist *sg)
{
+#ifdef CONFIG_HAVE_DMA_PFN
+ return __pfn_t_to_phys(sg->pfn) + sg->offset;
+#else
return page_to_phys(sg_page(sg)) + sg->offset;
+#endif
}
/**
@@ -217,7 +282,15 @@ static inline dma_addr_t sg_phys(struct scatterlist *sg)
**/
static inline void *sg_virt(struct scatterlist *sg)
{
- return page_address(sg_page(sg)) + sg->offset;
+ struct page *page;
+
+#ifdef CONFIG_HAVE_DMA_PFN
+ page = __pfn_t_to_page(sg->pfn) + sg->offset;
+ BUG_ON(!page); /* don't use sg_virt() on unmapped memory */
+#else
+ page = sg_page(sg);
+#endif
+ return page_address(page) + sg->offset;
}
int sg_nents(struct scatterlist *sg);
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries
2015-05-12 4:30 ` [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries Dan Williams
@ 2015-05-13 18:35 ` Williams, Dan J
2015-05-19 4:10 ` Vinod Koul
2015-05-23 14:12 ` hch
0 siblings, 2 replies; 20+ messages in thread
From: Williams, Dan J @ 2015-05-13 18:35 UTC (permalink / raw)
To: hch, axboe
Cc: mingo, linux-kernel, j.glisse, axboe, akpm, riel,
linux-nvdimm@lists.01.org, dmaengine, willy, linux-fsdevel,
mgorman, linux-arch, david
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 11844 bytes --]
[ adding willy (yes, need to fix my scripts), and dmaengine folks]
Jens, Christoph,
I've rebased this patch series block/for-next. With commit 84be456f883c
"remove <asm/scatterlist.h>" I think we can take the next step of
removing all references to page_link and just use __pfn_t by default.
v4 patch below.
Jens, I'm wondering if you want to take this series(.) as patches or
prepare a git branch to pull?
8<--------
Subject: scatterlist: support "page-less" (__pfn_t only) entries
From: Matthew Wilcox <willy@linux.intel.com>
Given that an offset will never be more than PAGE_SIZE, steal the unused
bits of the offset to implement a flags field. Move the existing "this
is a sg_chain() or sg_last() entry" flags to the new 'flags' field.
[djbw: rebase on block/for-4.2]
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Matthew Wilcox <willy@linux.intel.com>
---
block/blk-merge.c | 2 -
drivers/crypto/omap-sham.c | 2 -
drivers/dma/imx-dma.c | 7 +--
drivers/dma/ste_dma40.c | 5 --
drivers/mmc/card/queue.c | 4 +-
include/crypto/scatterwalk.h | 9 +---
include/linux/scatterlist.h | 92 ++++++++++++++++++++++++++----------------
samples/kfifo/dma-example.c | 8 ++--
8 files changed, 69 insertions(+), 60 deletions(-)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 218ad1e57a49..82a688551b72 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -267,7 +267,7 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq,
if (rq->cmd_flags & REQ_WRITE)
memset(q->dma_drain_buffer, 0, q->dma_drain_size);
- sg->page_link &= ~0x02;
+ sg_unmark_end(sg);
sg = sg_next(sg);
sg_set_page(sg, virt_to_page(q->dma_drain_buffer),
q->dma_drain_size,
diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c
index 4d63e0d4da9a..21736afd3320 100644
--- a/drivers/crypto/omap-sham.c
+++ b/drivers/crypto/omap-sham.c
@@ -582,7 +582,7 @@ static int omap_sham_xmit_dma(struct omap_sham_dev *dd, dma_addr_t dma_addr,
* the dmaengine may try to DMA the incorrect amount of data.
*/
sg_init_table(&ctx->sgl, 1);
- ctx->sgl.page_link = ctx->sg->page_link;
+ ctx->sgl.pfn = ctx->sg->pfn;
ctx->sgl.offset = ctx->sg->offset;
sg_dma_len(&ctx->sgl) = len32;
sg_dma_address(&ctx->sgl) = sg_dma_address(ctx->sg);
diff --git a/drivers/dma/imx-dma.c b/drivers/dma/imx-dma.c
index eed405976ea9..a767727bcfef 100644
--- a/drivers/dma/imx-dma.c
+++ b/drivers/dma/imx-dma.c
@@ -886,7 +886,7 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_cyclic(
sg_init_table(imxdmac->sg_list, periods);
for (i = 0; i < periods; i++) {
- imxdmac->sg_list[i].page_link = 0;
+ imxdmac->sg_list[i].pfn.page = NULL;
imxdmac->sg_list[i].offset = 0;
imxdmac->sg_list[i].dma_address = dma_addr;
sg_dma_len(&imxdmac->sg_list[i]) = period_len;
@@ -894,10 +894,7 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_cyclic(
}
/* close the loop */
- imxdmac->sg_list[periods].offset = 0;
- sg_dma_len(&imxdmac->sg_list[periods]) = 0;
- imxdmac->sg_list[periods].page_link =
- ((unsigned long)imxdmac->sg_list | 0x01) & ~0x02;
+ sg_chain(imxdmac->sg_list, periods + 1, imxdmac->sg_list);
desc->type = IMXDMA_DESC_CYCLIC;
desc->sg = imxdmac->sg_list;
diff --git a/drivers/dma/ste_dma40.c b/drivers/dma/ste_dma40.c
index 3c10f034d4b9..e8c00642cacb 100644
--- a/drivers/dma/ste_dma40.c
+++ b/drivers/dma/ste_dma40.c
@@ -2562,10 +2562,7 @@ dma40_prep_dma_cyclic(struct dma_chan *chan, dma_addr_t dma_addr,
dma_addr += period_len;
}
- sg[periods].offset = 0;
- sg_dma_len(&sg[periods]) = 0;
- sg[periods].page_link =
- ((unsigned long)sg | 0x01) & ~0x02;
+ sg_chain(sg, periods + 1, sg);
txd = d40_prep_sg(chan, sg, sg, periods, direction,
DMA_PREP_INTERRUPT);
diff --git a/drivers/mmc/card/queue.c b/drivers/mmc/card/queue.c
index 236d194c2883..127f76294e71 100644
--- a/drivers/mmc/card/queue.c
+++ b/drivers/mmc/card/queue.c
@@ -469,7 +469,7 @@ static unsigned int mmc_queue_packed_map_sg(struct mmc_queue *mq,
sg_set_buf(__sg, buf + offset, len);
offset += len;
remain -= len;
- (__sg++)->page_link &= ~0x02;
+ sg_unmark_end(__sg++);
sg_len++;
} while (remain);
}
@@ -477,7 +477,7 @@ static unsigned int mmc_queue_packed_map_sg(struct mmc_queue *mq,
list_for_each_entry(req, &packed->list, queuelist) {
sg_len += blk_rq_map_sg(mq->queue, req, __sg);
__sg = sg + (sg_len - 1);
- (__sg++)->page_link &= ~0x02;
+ sg_unmark_end(__sg++);
}
sg_mark_end(sg + (sg_len - 1));
return sg_len;
diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h
index 20e4226a2e14..4529889b0f07 100644
--- a/include/crypto/scatterwalk.h
+++ b/include/crypto/scatterwalk.h
@@ -25,13 +25,8 @@
#include <linux/scatterlist.h>
#include <linux/sched.h>
-static inline void scatterwalk_sg_chain(struct scatterlist *sg1, int num,
- struct scatterlist *sg2)
-{
- sg_set_page(&sg1[num - 1], (void *)sg2, 0, 0);
- sg1[num - 1].page_link &= ~0x02;
- sg1[num - 1].page_link |= 0x01;
-}
+#define scatterwalk_sg_chain(prv, num, sgl) sg_chain(prv, num, sgl)
+#define scatterwalk_sg_next(sgl) sg_next(sgl)
static inline void scatterwalk_crypto_chain(struct scatterlist *head,
struct scatterlist *sg,
diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h
index eca1ec93775c..98c21b460292 100644
--- a/include/linux/scatterlist.h
+++ b/include/linux/scatterlist.h
@@ -9,14 +9,18 @@
struct scatterlist {
#ifdef CONFIG_DEBUG_SG
- unsigned long sg_magic;
+ unsigned long sg_magic;
#endif
- unsigned long page_link;
- unsigned int offset;
- unsigned int length;
- dma_addr_t dma_address;
+ union {
+ __pfn_t pfn;
+ struct scatterlist *next;
+ };
+ unsigned short offset;
+ unsigned short sg_flags;
+ unsigned int length;
+ dma_addr_t dma_address;
#ifdef CONFIG_NEED_SG_DMA_LENGTH
- unsigned int dma_length;
+ unsigned int dma_length;
#endif
};
@@ -44,30 +48,29 @@ struct sg_table {
/*
* Notes on SG table design.
*
- * We use the unsigned long page_link field in the scatterlist struct to place
- * the page pointer AND encode information about the sg table as well. The two
- * lower bits are reserved for this information.
+ * We use the fact that a given sg entry will never be larger than a
+ * page to pack 'offset' and 'sg_flags' into 32-bits. This supports a
+ * PAGE_SIZE up to 64K and flags to encode information about the sg
+ * entry.
*
- * If bit 0 is set, then the page_link contains a pointer to the next sg
- * table list. Otherwise the next entry is at sg + 1.
+ * If SG_FLAGS_CHAIN is set, then the 'next' member of the entry is a
+ * pointer to the next sg table list. Otherwise the next entry is at sg
+ * + 1.
*
- * If bit 1 is set, then this sg entry is the last element in a list.
+ * If SG_FLAGS_LAST is set, then this sg entry is the last element in a
+ * list.
*
* See sg_next().
*
*/
#define SG_MAGIC 0x87654321
+#define SG_FLAGS_CHAIN 0x0001
+#define SG_FLAGS_LAST 0x0002
-/*
- * We overload the LSB of the page pointer to indicate whether it's
- * a valid sg entry, or whether it points to the start of a new scatterlist.
- * Those low bits are there for everyone! (thanks mason :-)
- */
-#define sg_is_chain(sg) ((sg)->page_link & 0x01)
-#define sg_is_last(sg) ((sg)->page_link & 0x02)
-#define sg_chain_ptr(sg) \
- ((struct scatterlist *) ((sg)->page_link & ~0x03))
+#define sg_is_chain(sg) ((sg)->sg_flags & SG_FLAGS_CHAIN)
+#define sg_is_last(sg) ((sg)->sg_flags & SG_FLAGS_LAST)
+#define sg_chain_ptr(sg) ((sg)->next)
/**
* sg_assign_page - Assign a given page to an SG entry
@@ -81,18 +84,15 @@ struct sg_table {
**/
static inline void sg_assign_page(struct scatterlist *sg, struct page *page)
{
- unsigned long page_link = sg->page_link & 0x3;
-
- /*
- * In order for the low bit stealing approach to work, pages
- * must be aligned at a 32-bit boundary as a minimum.
- */
- BUG_ON((unsigned long) page & 0x03);
#ifdef CONFIG_DEBUG_SG
BUG_ON(sg->sg_magic != SG_MAGIC);
BUG_ON(sg_is_chain(sg));
#endif
- sg->page_link = page_link | (unsigned long) page;
+ sg->pfn = page_to_pfn_t(page);
+#ifdef CONFIG_DEBUG_SG
+ BUG_ON(sg->sg_magic != SG_MAGIC);
+ BUG_ON(sg_is_chain(sg));
+#endif
}
/**
@@ -113,17 +113,32 @@ static inline void sg_set_page(struct scatterlist *sg, struct page *page,
unsigned int len, unsigned int offset)
{
sg_assign_page(sg, page);
+ BUG_ON(offset > 65535);
sg->offset = offset;
sg->length = len;
}
+static inline void sg_set_pfn(struct scatterlist *sg, __pfn_t pfn,
+ unsigned int len, unsigned int offset)
+{
+#ifdef CONFIG_DEBUG_SG
+ BUG_ON(sg->sg_magic != SG_MAGIC);
+ BUG_ON(sg_is_chain(sg));
+#endif
+ sg->pfn = pfn;
+ BUG_ON(offset > 65535);
+ sg->offset = offset;
+ sg->sg_flags = 0;
+ sg->length = len;
+}
+
static inline struct page *sg_page(struct scatterlist *sg)
{
#ifdef CONFIG_DEBUG_SG
BUG_ON(sg->sg_magic != SG_MAGIC);
BUG_ON(sg_is_chain(sg));
#endif
- return (struct page *)((sg)->page_link & ~0x3);
+ return __pfn_t_to_page(sg->pfn);
}
/**
@@ -175,7 +190,8 @@ static inline void sg_chain(struct scatterlist *prv, unsigned int prv_nents,
* Set lowest bit to indicate a link pointer, and make sure to clear
* the termination bit if it happens to be set.
*/
- prv[prv_nents - 1].page_link = ((unsigned long) sgl | 0x01) & ~0x02;
+ prv[prv_nents - 1].next = sgl;
+ prv[prv_nents - 1].sg_flags = SG_FLAGS_CHAIN;
}
/**
@@ -195,8 +211,8 @@ static inline void sg_mark_end(struct scatterlist *sg)
/*
* Set termination bit, clear potential chain bit
*/
- sg->page_link |= 0x02;
- sg->page_link &= ~0x01;
+ sg->sg_flags |= SG_FLAGS_LAST;
+ sg->sg_flags &= ~SG_FLAGS_CHAIN;
}
/**
@@ -212,7 +228,7 @@ static inline void sg_unmark_end(struct scatterlist *sg)
#ifdef CONFIG_DEBUG_SG
BUG_ON(sg->sg_magic != SG_MAGIC);
#endif
- sg->page_link &= ~0x02;
+ sg->sg_flags &= ~SG_FLAGS_LAST;
}
/**
@@ -227,7 +243,7 @@ static inline void sg_unmark_end(struct scatterlist *sg)
**/
static inline dma_addr_t sg_phys(struct scatterlist *sg)
{
- return page_to_phys(sg_page(sg)) + sg->offset;
+ return __pfn_t_to_phys(sg->pfn) + sg->offset;
}
/**
@@ -242,7 +258,11 @@ static inline dma_addr_t sg_phys(struct scatterlist *sg)
**/
static inline void *sg_virt(struct scatterlist *sg)
{
- return page_address(sg_page(sg)) + sg->offset;
+ struct page *page;
+
+ page = __pfn_t_to_page(sg->pfn) + sg->offset;
+ BUG_ON(!page); /* don't use sg_virt() on unmapped memory */
+ return page_address(page) + sg->offset;
}
int sg_nents(struct scatterlist *sg);
diff --git a/samples/kfifo/dma-example.c b/samples/kfifo/dma-example.c
index aa243db93f01..3eeff9a56e0e 100644
--- a/samples/kfifo/dma-example.c
+++ b/samples/kfifo/dma-example.c
@@ -75,8 +75,8 @@ static int __init example_init(void)
for (i = 0; i < nents; i++) {
printk(KERN_INFO
"sg[%d] -> "
- "page_link 0x%.8lx offset 0x%.8x length 0x%.8x\n",
- i, sg[i].page_link, sg[i].offset, sg[i].length);
+ "pfn_data 0x%.8lx offset 0x%.8x length 0x%.8x\n",
+ i, sg[i].pfn.data, sg[i].offset, sg[i].length);
if (sg_is_last(&sg[i]))
break;
@@ -104,8 +104,8 @@ static int __init example_init(void)
for (i = 0; i < nents; i++) {
printk(KERN_INFO
"sg[%d] -> "
- "page_link 0x%.8lx offset 0x%.8x length 0x%.8x\n",
- i, sg[i].page_link, sg[i].offset, sg[i].length);
+ "pfn_data 0x%.8lx offset 0x%.8x length 0x%.8x\n",
+ i, sg[i].pfn.data, sg[i].offset, sg[i].length);
if (sg_is_last(&sg[i]))
break;
ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries
2015-05-13 18:35 ` Williams, Dan J
@ 2015-05-19 4:10 ` Vinod Koul
2015-05-20 16:03 ` Dan Williams
2015-05-23 14:12 ` hch
1 sibling, 1 reply; 20+ messages in thread
From: Vinod Koul @ 2015-05-19 4:10 UTC (permalink / raw)
To: Williams, Dan J
Cc: hch, axboe, mingo, linux-kernel, j.glisse, axboe, akpm, riel,
linux-nvdimm@lists.01.org, dmaengine, willy, linux-fsdevel,
mgorman, linux-arch, david
On Wed, May 13, 2015 at 06:35:55PM +0000, Williams, Dan J wrote:
> [ adding willy (yes, need to fix my scripts), and dmaengine folks]
>
> Jens, Christoph,
>
> I've rebased this patch series block/for-next. With commit 84be456f883c
> "remove <asm/scatterlist.h>" I think we can take the next step of
> removing all references to page_link and just use __pfn_t by default.
> v4 patch below.
>
> Jens, I'm wondering if you want to take this series(.) as patches or
> prepare a git branch to pull?
>
> diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c
> index 4d63e0d4da9a..21736afd3320 100644
> --- a/drivers/crypto/omap-sham.c
> +++ b/drivers/crypto/omap-sham.c
> @@ -582,7 +582,7 @@ static int omap_sham_xmit_dma(struct omap_sham_dev *dd, dma_addr_t dma_addr,
> * the dmaengine may try to DMA the incorrect amount of data.
> */
> sg_init_table(&ctx->sgl, 1);
> - ctx->sgl.page_link = ctx->sg->page_link;
> + ctx->sgl.pfn = ctx->sg->pfn;
Do you want drivers to tinker with internals of sg, I think driver should be
agnostic here, perhpas a helper to get pfn, sg_get_pfn()
> ctx->sgl.offset = ctx->sg->offset;
> sg_dma_len(&ctx->sgl) = len32;
> sg_dma_address(&ctx->sgl) = sg_dma_address(ctx->sg);
> diff --git a/drivers/dma/imx-dma.c b/drivers/dma/imx-dma.c
> index eed405976ea9..a767727bcfef 100644
> --- a/drivers/dma/imx-dma.c
> +++ b/drivers/dma/imx-dma.c
> @@ -886,7 +886,7 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_cyclic(
> sg_init_table(imxdmac->sg_list, periods);
>
> for (i = 0; i < periods; i++) {
> - imxdmac->sg_list[i].page_link = 0;
> + imxdmac->sg_list[i].pfn.page = NULL;
same here, sg_set_pfn_page() ?
> imxdmac->sg_list[i].offset = 0;
> imxdmac->sg_list[i].dma_address = dma_addr;
> sg_dma_len(&imxdmac->sg_list[i]) = period_len;
> @@ -894,10 +894,7 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_cyclic(
> }
>
> /* close the loop */
> - imxdmac->sg_list[periods].offset = 0;
> - sg_dma_len(&imxdmac->sg_list[periods]) = 0;
> - imxdmac->sg_list[periods].page_link =
> - ((unsigned long)imxdmac->sg_list | 0x01) & ~0x02;
> + sg_chain(imxdmac->sg_list, periods + 1, imxdmac->sg_list);
This looks right...
--
~Vinod
^ permalink raw reply [flat|nested] 20+ messages in thread* Re: [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries
2015-05-19 4:10 ` Vinod Koul
@ 2015-05-20 16:03 ` Dan Williams
0 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-20 16:03 UTC (permalink / raw)
To: Vinod Koul
Cc: hch, axboe, mingo, linux-kernel, j.glisse, axboe, akpm, riel,
linux-nvdimm@lists.01.org, dmaengine, willy, linux-fsdevel,
mgorman, linux-arch, david
On Mon, May 18, 2015 at 9:10 PM, Vinod Koul <vinod.koul@intel.com> wrote:
> On Wed, May 13, 2015 at 06:35:55PM +0000, Williams, Dan J wrote:
>> [ adding willy (yes, need to fix my scripts), and dmaengine folks]
>>
>> Jens, Christoph,
>>
>> I've rebased this patch series block/for-next. With commit 84be456f883c
>> "remove <asm/scatterlist.h>" I think we can take the next step of
>> removing all references to page_link and just use __pfn_t by default.
>> v4 patch below.
>>
>> Jens, I'm wondering if you want to take this series(.) as patches or
>> prepare a git branch to pull?
>>
>
>> diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c
>> index 4d63e0d4da9a..21736afd3320 100644
>> --- a/drivers/crypto/omap-sham.c
>> +++ b/drivers/crypto/omap-sham.c
>> @@ -582,7 +582,7 @@ static int omap_sham_xmit_dma(struct omap_sham_dev *dd, dma_addr_t dma_addr,
>> * the dmaengine may try to DMA the incorrect amount of data.
>> */
>> sg_init_table(&ctx->sgl, 1);
>> - ctx->sgl.page_link = ctx->sg->page_link;
>> + ctx->sgl.pfn = ctx->sg->pfn;
> Do you want drivers to tinker with internals of sg, I think driver should be
> agnostic here, perhpas a helper to get pfn, sg_get_pfn()
Hmm, we have sg_page() seems like we also need an sg_pfn(). Thanks, will fix.
>
>> ctx->sgl.offset = ctx->sg->offset;
>> sg_dma_len(&ctx->sgl) = len32;
>> sg_dma_address(&ctx->sgl) = sg_dma_address(ctx->sg);
>> diff --git a/drivers/dma/imx-dma.c b/drivers/dma/imx-dma.c
>> index eed405976ea9..a767727bcfef 100644
>> --- a/drivers/dma/imx-dma.c
>> +++ b/drivers/dma/imx-dma.c
>> @@ -886,7 +886,7 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_cyclic(
>> sg_init_table(imxdmac->sg_list, periods);
>>
>> for (i = 0; i < periods; i++) {
>> - imxdmac->sg_list[i].page_link = 0;
>> + imxdmac->sg_list[i].pfn.page = NULL;
> same here, sg_set_pfn_page() ?
sg_set_page() would work here.
>
>> imxdmac->sg_list[i].offset = 0;
>> imxdmac->sg_list[i].dma_address = dma_addr;
>> sg_dma_len(&imxdmac->sg_list[i]) = period_len;
>> @@ -894,10 +894,7 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_cyclic(
>> }
>>
>> /* close the loop */
>> - imxdmac->sg_list[periods].offset = 0;
>> - sg_dma_len(&imxdmac->sg_list[periods]) = 0;
>> - imxdmac->sg_list[periods].page_link =
>> - ((unsigned long)imxdmac->sg_list | 0x01) & ~0x02;
>> + sg_chain(imxdmac->sg_list, periods + 1, imxdmac->sg_list);
> This looks right...
>
Cool, thanks for taking a look.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries
2015-05-13 18:35 ` Williams, Dan J
2015-05-19 4:10 ` Vinod Koul
@ 2015-05-23 14:12 ` hch
2015-05-23 16:41 ` Dan Williams
1 sibling, 1 reply; 20+ messages in thread
From: hch @ 2015-05-23 14:12 UTC (permalink / raw)
To: Williams, Dan J
Cc: hch, axboe, mingo, linux-kernel, j.glisse, axboe, akpm, riel,
linux-nvdimm@lists.01.org, dmaengine, willy, linux-fsdevel,
mgorman, linux-arch, david
On Wed, May 13, 2015 at 06:35:55PM +0000, Williams, Dan J wrote:
> Jens, I'm wondering if you want to take this series(.) as patches or
> prepare a git branch to pull?
Honestly I don't think it should go anyway. It makes a big mess of
a structure without providing a real user for it. Given how we are
using the bio_vec for in-kernel page based I/O these days it seems
like a very dangerous idea.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries
2015-05-23 14:12 ` hch
@ 2015-05-23 16:41 ` Dan Williams
0 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-23 16:41 UTC (permalink / raw)
To: hch
Cc: axboe, linux-arch, riel, linux-nvdimm@lists.01.org,
linux-fsdevel, david, linux-kernel, mingo, axboe, dmaengine,
mgorman, j.glisse, akpm
On Sat, May 23, 2015 at 7:12 AM, hch@lst.de <hch@lst.de> wrote:
> On Wed, May 13, 2015 at 06:35:55PM +0000, Williams, Dan J wrote:
>> Jens, I'm wondering if you want to take this series(.) as patches or
>> prepare a git branch to pull?
>
> Honestly I don't think it should go anyway. It makes a big mess of
> a structure without providing a real user for it. Given how we are
> using the bio_vec for in-kernel page based I/O these days it seems
> like a very dangerous idea.
There's nothing dangerous about the __pfn_t conversion of the block
layer in the !CONFIG_DEV_PFN case a __pfn_t based bio_vec is
bit-for-bit identical to a struct page based bio_vec. However, you're
right, I can't make the same claim about a scatterlist before and
after the change.
Hmm, we're missing a pfn-only block I/O user and we're missing the
second half of the implementation that provides __pfn_t_to_page() for
persistent memory. I'm looking to have a solution __pfn_t_to_page()
shortly, maybe that will allow the scatterlist changes to be
skipped... we'll see.
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v3 07/11] x86: support dma_map_pfn()
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
` (5 preceding siblings ...)
2015-05-12 4:30 ` [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries Dan Williams
@ 2015-05-12 4:30 ` Dan Williams
2015-05-12 4:30 ` [PATCH v3 08/11] x86: support kmap_atomic_pfn_t() for persistent memory Dan Williams
` (4 subsequent siblings)
11 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:30 UTC (permalink / raw)
To: linux-kernel
Cc: linux-arch, axboe, riel, linux-nvdimm, david, hch, j.glisse,
mgorman, linux-fsdevel, akpm, mingo
Fix up x86 dma_map_ops to allow pfn-only mappings.
As long as a dma_map_sg() implementation uses the generic sg_phys()
helpers it can support scatterlists that use __pfn_t instead of struct
page.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
arch/x86/Kconfig | 5 +++++
arch/x86/kernel/amd_gart_64.c | 22 +++++++++++++++++-----
arch/x86/kernel/pci-nommu.c | 22 +++++++++++++++++-----
arch/x86/kernel/pci-swiotlb.c | 4 ++++
arch/x86/pci/sta2x11-fixup.c | 4 ++++
arch/x86/xen/pci-swiotlb-xen.c | 4 ++++
drivers/iommu/amd_iommu.c | 21 ++++++++++++++++-----
drivers/iommu/intel-iommu.c | 22 +++++++++++++++++-----
drivers/xen/swiotlb-xen.c | 29 +++++++++++++++++++----------
include/asm-generic/dma-mapping-common.h | 4 ++--
include/asm-generic/scatterlist.h | 1 +
include/linux/swiotlb.h | 4 ++++
lib/swiotlb.c | 20 +++++++++++++++-----
13 files changed, 125 insertions(+), 37 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 226d5696e1d1..c626ffa5c01e 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -796,6 +796,7 @@ config CALGARY_IOMMU
bool "IBM Calgary IOMMU support"
select SWIOTLB
depends on X86_64 && PCI
+ depends on !HAVE_DMA_PFN
---help---
Support for hardware IOMMUs in IBM's xSeries x366 and x460
systems. Needed to run systems with more than 3GB of memory
@@ -1432,6 +1433,10 @@ config X86_PMEM_LEGACY
Say Y if unsure.
+config X86_PMEM_DMA
+ def_bool DEV_PFN
+ select HAVE_DMA_PFN
+
config HIGHPTE
bool "Allocate 3rd-level pagetables from highmem"
depends on HIGHMEM
diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c
index 8e3842fc8bea..8fad83c8dfd2 100644
--- a/arch/x86/kernel/amd_gart_64.c
+++ b/arch/x86/kernel/amd_gart_64.c
@@ -239,13 +239,13 @@ static dma_addr_t dma_map_area(struct device *dev, dma_addr_t phys_mem,
}
/* Map a single area into the IOMMU */
-static dma_addr_t gart_map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size,
- enum dma_data_direction dir,
- struct dma_attrs *attrs)
+static dma_addr_t gart_map_pfn(struct device *dev, __pfn_t pfn,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir,
+ struct dma_attrs *attrs)
{
unsigned long bus;
- phys_addr_t paddr = page_to_phys(page) + offset;
+ phys_addr_t paddr = __pfn_t_to_phys(pfn) + offset;
if (!dev)
dev = &x86_dma_fallback_dev;
@@ -259,6 +259,14 @@ static dma_addr_t gart_map_page(struct device *dev, struct page *page,
return bus;
}
+static __maybe_unused dma_addr_t gart_map_page(struct device *dev,
+ struct page *page, unsigned long offset, size_t size,
+ enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+ return gart_map_pfn(dev, page_to_pfn_t(page), offset, size, dir,
+ attrs);
+}
+
/*
* Free a DMA mapping.
*/
@@ -699,7 +707,11 @@ static __init int init_amd_gatt(struct agp_kern_info *info)
static struct dma_map_ops gart_dma_ops = {
.map_sg = gart_map_sg,
.unmap_sg = gart_unmap_sg,
+#ifdef CONFIG_HAVE_DMA_PFN
+ .map_pfn = gart_map_pfn,
+#else
.map_page = gart_map_page,
+#endif
.unmap_page = gart_unmap_page,
.alloc = gart_alloc_coherent,
.free = gart_free_coherent,
diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c
index da15918d1c81..876dacfbabf6 100644
--- a/arch/x86/kernel/pci-nommu.c
+++ b/arch/x86/kernel/pci-nommu.c
@@ -25,12 +25,12 @@ check_addr(char *name, struct device *hwdev, dma_addr_t bus, size_t size)
return 1;
}
-static dma_addr_t nommu_map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size,
- enum dma_data_direction dir,
- struct dma_attrs *attrs)
+static dma_addr_t nommu_map_pfn(struct device *dev, __pfn_t pfn,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir,
+ struct dma_attrs *attrs)
{
- dma_addr_t bus = page_to_phys(page) + offset;
+ dma_addr_t bus = __pfn_t_to_phys(pfn) + offset;
WARN_ON(size == 0);
if (!check_addr("map_single", dev, bus, size))
return DMA_ERROR_CODE;
@@ -38,6 +38,14 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page,
return bus;
}
+static __maybe_unused dma_addr_t nommu_map_page(struct device *dev,
+ struct page *page, unsigned long offset, size_t size,
+ enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+ return nommu_map_pfn(dev, page_to_pfn_t(page), offset, size, dir,
+ attrs);
+}
+
/* Map a set of buffers described by scatterlist in streaming
* mode for DMA. This is the scatter-gather version of the
* above pci_map_single interface. Here the scatter gather list
@@ -92,7 +100,11 @@ struct dma_map_ops nommu_dma_ops = {
.alloc = dma_generic_alloc_coherent,
.free = dma_generic_free_coherent,
.map_sg = nommu_map_sg,
+#ifdef CONFIG_HAVE_DMA_PFN
+ .map_pfn = nommu_map_pfn,
+#else
.map_page = nommu_map_page,
+#endif
.sync_single_for_device = nommu_sync_single_for_device,
.sync_sg_for_device = nommu_sync_sg_for_device,
.is_phys = 1,
diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c
index 77dd0ad58be4..5351eb8c8f7f 100644
--- a/arch/x86/kernel/pci-swiotlb.c
+++ b/arch/x86/kernel/pci-swiotlb.c
@@ -48,7 +48,11 @@ static struct dma_map_ops swiotlb_dma_ops = {
.sync_sg_for_device = swiotlb_sync_sg_for_device,
.map_sg = swiotlb_map_sg_attrs,
.unmap_sg = swiotlb_unmap_sg_attrs,
+#ifdef CONFIG_HAVE_DMA_PFN
+ .map_pfn = swiotlb_map_pfn,
+#else
.map_page = swiotlb_map_page,
+#endif
.unmap_page = swiotlb_unmap_page,
.dma_supported = NULL,
};
diff --git a/arch/x86/pci/sta2x11-fixup.c b/arch/x86/pci/sta2x11-fixup.c
index 5ceda85b8687..d1c6e3808bb5 100644
--- a/arch/x86/pci/sta2x11-fixup.c
+++ b/arch/x86/pci/sta2x11-fixup.c
@@ -182,7 +182,11 @@ static void *sta2x11_swiotlb_alloc_coherent(struct device *dev,
static struct dma_map_ops sta2x11_dma_ops = {
.alloc = sta2x11_swiotlb_alloc_coherent,
.free = x86_swiotlb_free_coherent,
+#ifdef CONFIG_HAVE_DMA_PFN
+ .map_pfn = swiotlb_map_pfn,
+#else
.map_page = swiotlb_map_page,
+#endif
.unmap_page = swiotlb_unmap_page,
.map_sg = swiotlb_map_sg_attrs,
.unmap_sg = swiotlb_unmap_sg_attrs,
diff --git a/arch/x86/xen/pci-swiotlb-xen.c b/arch/x86/xen/pci-swiotlb-xen.c
index 0e98e5d241d0..e65ea48d7aed 100644
--- a/arch/x86/xen/pci-swiotlb-xen.c
+++ b/arch/x86/xen/pci-swiotlb-xen.c
@@ -28,7 +28,11 @@ static struct dma_map_ops xen_swiotlb_dma_ops = {
.sync_sg_for_device = xen_swiotlb_sync_sg_for_device,
.map_sg = xen_swiotlb_map_sg_attrs,
.unmap_sg = xen_swiotlb_unmap_sg_attrs,
+#ifdef CONFIG_HAVE_DMA_PFN
+ .map_pfn = xen_swiotlb_map_pfn,
+#else
.map_page = xen_swiotlb_map_page,
+#endif
.unmap_page = xen_swiotlb_unmap_page,
.dma_supported = xen_swiotlb_dma_supported,
};
diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index e43d48956dea..ee8f70224b73 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -2754,16 +2754,15 @@ static void __unmap_single(struct dma_ops_domain *dma_dom,
/*
* The exported map_single function for dma_ops.
*/
-static dma_addr_t map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size,
- enum dma_data_direction dir,
- struct dma_attrs *attrs)
+static dma_addr_t map_pfn(struct device *dev, __pfn_t pfn, unsigned long offset,
+ size_t size, enum dma_data_direction dir,
+ struct dma_attrs *attrs)
{
unsigned long flags;
struct protection_domain *domain;
dma_addr_t addr;
u64 dma_mask;
- phys_addr_t paddr = page_to_phys(page) + offset;
+ phys_addr_t paddr = __pfn_t_to_phys(pfn) + offset;
INC_STATS_COUNTER(cnt_map_single);
@@ -2788,6 +2787,14 @@ out:
spin_unlock_irqrestore(&domain->lock, flags);
return addr;
+
+}
+
+static __maybe_unused dma_addr_t map_page(struct device *dev, struct page *page,
+ unsigned long offset, size_t size, enum dma_data_direction dir,
+ struct dma_attrs *attrs)
+{
+ return map_pfn(dev, page_to_pfn_t(page), offset, size, dir, attrs);
}
/*
@@ -3062,7 +3069,11 @@ static void __init prealloc_protection_domains(void)
static struct dma_map_ops amd_iommu_dma_ops = {
.alloc = alloc_coherent,
.free = free_coherent,
+#ifdef CONFIG_HAVE_DMA_PFN
+ .map_pfn = map_pfn,
+#else
.map_page = map_page,
+#endif
.unmap_page = unmap_page,
.map_sg = map_sg,
.unmap_sg = unmap_sg,
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 9b9ada71e0d3..6d9a0f85b827 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -3086,15 +3086,23 @@ error:
return 0;
}
-static dma_addr_t intel_map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size,
- enum dma_data_direction dir,
- struct dma_attrs *attrs)
+static dma_addr_t intel_map_pfn(struct device *dev, __pfn_t pfn,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir,
+ struct dma_attrs *attrs)
{
- return __intel_map_single(dev, page_to_phys(page) + offset, size,
+ return __intel_map_single(dev, __pfn_t_to_phys(pfn) + offset, size,
dir, *dev->dma_mask);
}
+static __maybe_unused dma_addr_t intel_map_page(struct device *dev,
+ struct page *page, unsigned long offset, size_t size,
+ enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+ return intel_map_pfn(dev, page_to_pfn_t(page), offset, size, dir,
+ attrs);
+}
+
static void flush_unmaps(void)
{
int i, j;
@@ -3380,7 +3388,11 @@ struct dma_map_ops intel_dma_ops = {
.free = intel_free_coherent,
.map_sg = intel_map_sg,
.unmap_sg = intel_unmap_sg,
+#ifdef CONFIG_HAVE_DMA_PFN
+ .map_pfn = intel_map_pfn,
+#else
.map_page = intel_map_page,
+#endif
.unmap_page = intel_unmap_page,
.mapping_error = intel_mapping_error,
};
diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 810ad419e34c..bd29d09bbacc 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -382,10 +382,10 @@ EXPORT_SYMBOL_GPL(xen_swiotlb_free_coherent);
* Once the device is given the dma address, the device owns this memory until
* either xen_swiotlb_unmap_page or xen_swiotlb_dma_sync_single is performed.
*/
-dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size,
- enum dma_data_direction dir,
- struct dma_attrs *attrs)
+dma_addr_t xen_swiotlb_map_pfn(struct device *dev, unsigned long pfn,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir,
+ struct dma_attrs *attrs)
{
phys_addr_t map, phys = page_to_phys(page) + offset;
dma_addr_t dev_addr = xen_phys_to_bus(phys);
@@ -429,6 +429,16 @@ dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
}
return dev_addr;
}
+EXPORT_SYMBOL_GPL(xen_swiotlb_map_pfn);
+
+dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir,
+ struct dma_attrs *attrs)
+{
+ return xen_swiotlb_map_pfn(dev, page_to_pfn(page), offset, size, dir,
+ attrs);
+}
EXPORT_SYMBOL_GPL(xen_swiotlb_map_page);
/*
@@ -582,15 +592,14 @@ xen_swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
attrs);
sg->dma_address = xen_phys_to_bus(map);
} else {
+ __pfn_t pfn = { .pfn = paddr >> PAGE_SHIFT };
+ unsigned long offset = paddr & ~PAGE_MASK;
+
/* we are not interested in the dma_addr returned by
* xen_dma_map_page, only in the potential cache flushes executed
* by the function. */
- xen_dma_map_page(hwdev, pfn_to_page(paddr >> PAGE_SHIFT),
- dev_addr,
- paddr & ~PAGE_MASK,
- sg->length,
- dir,
- attrs);
+ xen_dma_map_pfn(hwdev, pfn, dev_addr, offset,
+ sg->length, attrs);
sg->dma_address = dev_addr;
}
sg_dma_len(sg) = sg->length;
diff --git a/include/asm-generic/dma-mapping-common.h b/include/asm-generic/dma-mapping-common.h
index 7305efb1bac6..e031b079ce4e 100644
--- a/include/asm-generic/dma-mapping-common.h
+++ b/include/asm-generic/dma-mapping-common.h
@@ -18,7 +18,7 @@ static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr,
kmemcheck_mark_initialized(ptr, size);
BUG_ON(!valid_dma_direction(dir));
#ifdef CONFIG_HAVE_DMA_PFN
- addr = ops->map_pfn(dev, page_to_pfn_typed(virt_to_page(ptr)),
+ addr = ops->map_pfn(dev, page_to_pfn_t(virt_to_page(ptr)),
(unsigned long)ptr & ~PAGE_MASK, size,
dir, attrs);
#else
@@ -99,7 +99,7 @@ static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
enum dma_data_direction dir)
{
kmemcheck_mark_initialized(page_address(page) + offset, size);
- return dma_map_pfn(dev, page_to_pfn_typed(page), offset, size, dir);
+ return dma_map_pfn(dev, page_to_pfn_t(page), offset, size, dir);
}
#else
static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
diff --git a/include/asm-generic/scatterlist.h b/include/asm-generic/scatterlist.h
index 959f51572a8e..ddc743513e55 100644
--- a/include/asm-generic/scatterlist.h
+++ b/include/asm-generic/scatterlist.h
@@ -2,6 +2,7 @@
#define __ASM_GENERIC_SCATTERLIST_H
#include <linux/types.h>
+#include <linux/mm.h>
struct scatterlist {
#ifdef CONFIG_DEBUG_SG
diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index e7a018eaf3a2..5093fc8d2825 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -66,6 +66,10 @@ extern dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
unsigned long offset, size_t size,
enum dma_data_direction dir,
struct dma_attrs *attrs);
+extern dma_addr_t swiotlb_map_pfn(struct device *dev, __pfn_t pfn,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir,
+ struct dma_attrs *attrs);
extern void swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr,
size_t size, enum dma_data_direction dir,
struct dma_attrs *attrs);
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 4abda074ea45..4183d691bf2e 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -727,12 +727,12 @@ swiotlb_full(struct device *dev, size_t size, enum dma_data_direction dir,
* Once the device is given the dma address, the device owns this memory until
* either swiotlb_unmap_page or swiotlb_dma_sync_single is performed.
*/
-dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size,
- enum dma_data_direction dir,
- struct dma_attrs *attrs)
+dma_addr_t swiotlb_map_pfn(struct device *dev, __pfn_t pfn,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir,
+ struct dma_attrs *attrs)
{
- phys_addr_t map, phys = page_to_phys(page) + offset;
+ phys_addr_t map, phys = __pfn_t_to_phys(pfn) + offset;
dma_addr_t dev_addr = phys_to_dma(dev, phys);
BUG_ON(dir == DMA_NONE);
@@ -763,6 +763,16 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
return dev_addr;
}
+EXPORT_SYMBOL_GPL(swiotlb_map_pfn);
+
+dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir,
+ struct dma_attrs *attrs)
+{
+ return swiotlb_map_pfn(dev, page_to_pfn_t(page), offset, size, dir,
+ attrs);
+}
EXPORT_SYMBOL_GPL(swiotlb_map_page);
/*
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH v3 08/11] x86: support kmap_atomic_pfn_t() for persistent memory
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
` (6 preceding siblings ...)
2015-05-12 4:30 ` [PATCH v3 07/11] x86: support dma_map_pfn() Dan Williams
@ 2015-05-12 4:30 ` Dan Williams
2015-05-12 4:30 ` [PATCH v3 09/11] block: convert kmap helpers to kmap_atomic_pfn_t() Dan Williams
` (3 subsequent siblings)
11 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:30 UTC (permalink / raw)
To: linux-kernel
Cc: linux-arch, axboe, riel, linux-nvdimm, david, hch, j.glisse,
mgorman, linux-fsdevel, akpm, mingo
It would be unfortunate if the kmap infrastructure escaped its current
32-bit/HIGHMEM bonds and leaked into 64-bit code. Instead, if the user
has enabled CONFIG_DEV_PFN we direct the kmap_atomic_pfn_t()
implementation to scan a list of pre-mapped persistent memory address
ranges inserted by the pmem driver.
The __pfn_t to resource lookup is indeed inefficient walking of a linked list,
but there are two mitigating factors:
1/ The number of persistent memory ranges is bounded by the number of
DIMMs which is on the order of 10s of DIMMs, not hundreds.
2/ The lookup yields the entire range, if it becomes inefficient to do a
kmap_atomic_pfn_t() a PAGE_SIZE at a time the caller can take
advantage of the fact that the lookup can be amortized for all kmap
operations it needs to perform in a given range.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
arch/Kconfig | 3 +
arch/x86/Kconfig | 2 +
drivers/block/pmem.c | 6 +++
include/linux/highmem.h | 23 +++++++++++
mm/Makefile | 1
mm/pfn.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++
6 files changed, 133 insertions(+)
create mode 100644 mm/pfn.c
diff --git a/arch/Kconfig b/arch/Kconfig
index f7f800860c00..69d3a3fa21af 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -206,6 +206,9 @@ config HAVE_DMA_CONTIGUOUS
config HAVE_DMA_PFN
bool
+config HAVE_KMAP_PFN
+ bool
+
config GENERIC_SMP_IDLE_THREAD
bool
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c626ffa5c01e..2fd7690ed0e2 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1434,7 +1434,9 @@ config X86_PMEM_LEGACY
Say Y if unsure.
config X86_PMEM_DMA
+ depends on !HIGHMEM
def_bool DEV_PFN
+ select HAVE_KMAP_PFN
select HAVE_DMA_PFN
config HIGHPTE
diff --git a/drivers/block/pmem.c b/drivers/block/pmem.c
index 41bb424533e6..2a847651f8de 100644
--- a/drivers/block/pmem.c
+++ b/drivers/block/pmem.c
@@ -23,6 +23,7 @@
#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/slab.h>
+#include <linux/highmem.h>
#define PMEM_MINORS 16
@@ -147,6 +148,11 @@ static struct pmem_device *pmem_alloc(struct device *dev, struct resource *res)
if (!pmem->virt_addr)
goto out_release_region;
+ err = devm_register_kmap_pfn_range(dev, res, pmem->virt_addr);
+ if (err)
+ goto out_unmap;
+
+ err = -ENOMEM;
pmem->pmem_queue = blk_alloc_queue(GFP_KERNEL);
if (!pmem->pmem_queue)
goto out_unmap;
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 9286a46b7d69..85fd52d43a9a 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -83,6 +83,29 @@ static inline void __kunmap_atomic(void *addr)
#endif /* CONFIG_HIGHMEM */
+#ifdef CONFIG_HAVE_KMAP_PFN
+extern void *kmap_atomic_pfn_t(__pfn_t pfn);
+extern void kunmap_atomic_pfn_t(void *addr);
+extern int devm_register_kmap_pfn_range(struct device *dev,
+ struct resource *res, void *base);
+#else
+static inline void *kmap_atomic_pfn_t(__pfn_t pfn)
+{
+ return kmap_atomic(__pfn_t_to_page(pfn));
+}
+
+static inline void kunmap_atomic_pfn_t(void *addr)
+{
+ __kunmap_atomic(addr);
+}
+
+static inline int devm_register_kmap_pfn_range(struct device *dev,
+ struct resource *res, void *base)
+{
+ return 0;
+}
+#endif /* CONFIG_HAVE_KMAP_PFN */
+
#if defined(CONFIG_HIGHMEM) || defined(CONFIG_X86_32)
DECLARE_PER_CPU(int, __kmap_atomic_idx);
diff --git a/mm/Makefile b/mm/Makefile
index 98c4eaeabdcb..66e30c2addfe 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -78,3 +78,4 @@ obj-$(CONFIG_CMA) += cma.o
obj-$(CONFIG_MEMORY_BALLOON) += balloon_compaction.o
obj-$(CONFIG_PAGE_EXTENSION) += page_ext.o
obj-$(CONFIG_CMA_DEBUGFS) += cma_debug.o
+obj-$(CONFIG_HAVE_KMAP_PFN) += pfn.o
diff --git a/mm/pfn.c b/mm/pfn.c
new file mode 100644
index 000000000000..0e046b49aebf
--- /dev/null
+++ b/mm/pfn.c
@@ -0,0 +1,98 @@
+/*
+ * Copyright(c) 2015 Intel Corporation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#include <linux/rcupdate.h>
+#include <linux/rculist.h>
+#include <linux/highmem.h>
+#include <linux/device.h>
+#include <linux/slab.h>
+#include <linux/mm.h>
+
+static LIST_HEAD(ranges);
+
+struct kmap {
+ struct list_head list;
+ struct resource *res;
+ struct device *dev;
+ void *base;
+};
+
+static void teardown_kmap(void *data)
+{
+ struct kmap *kmap = data;
+
+ dev_dbg(kmap->dev, "kmap unregister %pr\n", kmap->res);
+ list_del_rcu(&kmap->list);
+ synchronize_rcu();
+ kfree(kmap);
+}
+
+int devm_register_kmap_pfn_range(struct device *dev, struct resource *res,
+ void *base)
+{
+ struct kmap *kmap = kzalloc(sizeof(*kmap), GFP_KERNEL);
+ int rc;
+
+ if (!kmap)
+ return -ENOMEM;
+
+ INIT_LIST_HEAD(&kmap->list);
+ kmap->res = res;
+ kmap->base = base;
+ kmap->dev = dev;
+ rc = devm_add_action(dev, teardown_kmap, kmap);
+ if (rc) {
+ kfree(kmap);
+ return rc;
+ }
+ dev_dbg(kmap->dev, "kmap register %pr\n", kmap->res);
+ list_add_rcu(&kmap->list, &ranges);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(devm_register_kmap_pfn_range);
+
+void *kmap_atomic_pfn_t(__pfn_t pfn)
+{
+ struct page *page = __pfn_t_to_page(pfn);
+ resource_size_t addr;
+ struct kmap *kmap;
+
+ if (page)
+ return kmap_atomic(page);
+ addr = __pfn_t_to_phys(pfn);
+ rcu_read_lock();
+ list_for_each_entry_rcu(kmap, &ranges, list)
+ if (addr >= kmap->res->start && addr <= kmap->res->end)
+ return kmap->base + addr - kmap->res->start;
+
+ /* only unlock in the error case */
+ rcu_read_unlock();
+ return NULL;
+}
+EXPORT_SYMBOL(kmap_atomic_pfn_t);
+
+void kunmap_atomic_pfn_t(void *addr)
+{
+ /*
+ * If the original __pfn_t had an entry in the memmap then
+ * 'addr' will be outside of vmalloc space i.e. it came from
+ * page_address()
+ */
+ if (!is_vmalloc_addr(addr)) {
+ kunmap_atomic(addr);
+ return;
+ }
+
+ /* signal that we are done with the range */
+ rcu_read_unlock();
+}
+EXPORT_SYMBOL(kunmap_atomic_pfn_t);
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH v3 09/11] block: convert kmap helpers to kmap_atomic_pfn_t()
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
` (7 preceding siblings ...)
2015-05-12 4:30 ` [PATCH v3 08/11] x86: support kmap_atomic_pfn_t() for persistent memory Dan Williams
@ 2015-05-12 4:30 ` Dan Williams
2015-05-12 4:30 ` [PATCH v3 10/11] dax: convert to __pfn_t Dan Williams
` (2 subsequent siblings)
11 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:30 UTC (permalink / raw)
To: linux-kernel
Cc: linux-arch, axboe, riel, linux-nvdimm, david, hch, j.glisse,
mgorman, linux-fsdevel, akpm, mingo
Convert the generic helpers to the __pfn_t version of kmap_atomic() in
support of generically enabling "page-less" block i/o.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
include/linux/bio.h | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/include/linux/bio.h b/include/linux/bio.h
index a569e6ea1cd2..6537d78e78b3 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -161,7 +161,7 @@ static inline void *bio_data(struct bio *bio)
* I/O completely on that queue (see ide-dma for example)
*/
#define __bio_kmap_atomic(bio, iter) \
- (kmap_atomic(bvec_page(bio_iter_iovec((bio), (iter)))) + \
+ (kmap_atomic_pfn_t(bio_iter_iovec((bio), (iter)).pfn) + \
bio_iter_iovec((bio), (iter)).bv_offset)
#define __bio_kunmap_atomic(addr) kunmap_atomic(addr)
@@ -491,7 +491,7 @@ static inline char *bvec_kmap_irq(struct bio_vec *bvec, unsigned long *flags)
* balancing is a lot nicer this way
*/
local_irq_save(*flags);
- addr = (unsigned long) kmap_atomic(bvec_page(bvec));
+ addr = (unsigned long) kmap_atomic_pfn_t(bvec.pfn);
BUG_ON(addr & ~PAGE_MASK);
@@ -502,18 +502,21 @@ static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags)
{
unsigned long ptr = (unsigned long) buffer & PAGE_MASK;
- kunmap_atomic((void *) ptr);
+ kunmap_atomic_pfn_t((void *) ptr);
local_irq_restore(*flags);
}
#else
static inline char *bvec_kmap_irq(struct bio_vec *bvec, unsigned long *flags)
{
- return page_address(bvec_page(bvec)) + bvec->bv_offset;
+ return kmap_atomic_pfn_t(bvec->bv_pfn) + bvec->bv_offset;
}
static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags)
{
+ unsigned long ptr = (unsigned long) buffer & PAGE_MASK;
+
+ kunmap_atomic_pfn_t((void *) ptr);
*flags = 0;
}
#endif
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH v3 10/11] dax: convert to __pfn_t
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
` (8 preceding siblings ...)
2015-05-12 4:30 ` [PATCH v3 09/11] block: convert kmap helpers to kmap_atomic_pfn_t() Dan Williams
@ 2015-05-12 4:30 ` Dan Williams
2015-05-12 4:30 ` [PATCH v3 11/11] block: base support for pfn i/o Dan Williams
2015-05-23 14:32 ` [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Christoph Hellwig
11 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:30 UTC (permalink / raw)
To: linux-kernel
Cc: axboe, Boaz Harrosh, riel, akpm, j.glisse, linux-nvdimm,
Benjamin Herrenschmidt, Heiko Carstens, hch, linux-fsdevel,
Paul Mackerras, mgorman, david, linux-arch, Martin Schwidefsky,
Matthew Wilcox, Ross Zwisler, mingo
The primary source for non-page-backed page-frames to enter the system
is via the pmem driver's ->direct_access() method. The pfns returned by
the top-level bdev_direct_access() may be passed to any other subsystem
in the kernel and those sub-systems either need to assume that the pfn
is page backed (CONFIG_DEV_PFN=n) or be prepared to handle non-page
backed case (CONFIG_DEV_PFN=y). Currently the pfns returned by
->direct_access() are only ever used by vm_insert_mixed() which does not
care if the pfn is mapped. As we go to add more usages of these pfns
add the type-safety of __pfn_t.
This also simplifies the calling convention of ->direct_access() by not
returning the virtual address in the same call. This annotates cases
where the kernel is directly accessing pmem outside the driver, and
makes the valid lifetime of the reference explicit. This property may
be useful in the future for invalidating mappings to pmem, but for now
it provides some protection against the "pmem disable vs still-in-use"
race.
Cc: Matthew Wilcox <willy@linux.intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Boaz Harrosh <boaz@plexistor.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
arch/powerpc/sysdev/axonram.c | 11 +++++--
drivers/block/brd.c | 5 +--
drivers/block/pmem.c | 11 +++++--
drivers/s390/block/dcssblk.c | 13 ++++++---
fs/block_dev.c | 4 +--
fs/dax.c | 62 ++++++++++++++++++++++++++++++++---------
include/asm-generic/pfn.h | 11 +++++++
include/linux/blkdev.h | 7 ++---
8 files changed, 91 insertions(+), 33 deletions(-)
diff --git a/arch/powerpc/sysdev/axonram.c b/arch/powerpc/sysdev/axonram.c
index 9bb5da7f2c0c..91c40a300797 100644
--- a/arch/powerpc/sysdev/axonram.c
+++ b/arch/powerpc/sysdev/axonram.c
@@ -139,22 +139,27 @@ axon_ram_make_request(struct request_queue *queue, struct bio *bio)
* axon_ram_direct_access - direct_access() method for block device
* @device, @sector, @data: see block_device_operations method
*/
+#ifdef CONFIG_DEV_PFN
static long
axon_ram_direct_access(struct block_device *device, sector_t sector,
- void **kaddr, unsigned long *pfn, long size)
+ __pfn_t *pfn, long size)
{
struct axon_ram_bank *bank = device->bd_disk->private_data;
loff_t offset = (loff_t)sector << AXON_RAM_SECTOR_SHIFT;
+ void *kaddr;
- *kaddr = (void *)(bank->ph_addr + offset);
- *pfn = virt_to_phys(*kaddr) >> PAGE_SHIFT;
+ kaddr = (void *)(bank->ph_addr + offset);
+ *pfn = phys_to_pfn_t(virt_to_phys(*kaddr));
return bank->size - offset;
}
+#endif
static const struct block_device_operations axon_ram_devops = {
.owner = THIS_MODULE,
+#ifdef CONFIG_DEV_PFN
.direct_access = axon_ram_direct_access
+#endif
};
/**
diff --git a/drivers/block/brd.c b/drivers/block/brd.c
index 115c6cf9cb43..3be31a2aed20 100644
--- a/drivers/block/brd.c
+++ b/drivers/block/brd.c
@@ -371,7 +371,7 @@ static int brd_rw_page(struct block_device *bdev, sector_t sector,
#ifdef CONFIG_BLK_DEV_RAM_DAX
static long brd_direct_access(struct block_device *bdev, sector_t sector,
- void **kaddr, unsigned long *pfn, long size)
+ __pfn_t *pfn, long size)
{
struct brd_device *brd = bdev->bd_disk->private_data;
struct page *page;
@@ -381,8 +381,7 @@ static long brd_direct_access(struct block_device *bdev, sector_t sector,
page = brd_insert_page(brd, sector);
if (!page)
return -ENOSPC;
- *kaddr = page_address(page);
- *pfn = page_to_pfn(page);
+ *pfn = page_to_pfn_t(page);
/*
* TODO: If size > PAGE_SIZE, we could look to see if the next page in
diff --git a/drivers/block/pmem.c b/drivers/block/pmem.c
index 2a847651f8de..0cf34fba308c 100644
--- a/drivers/block/pmem.c
+++ b/drivers/block/pmem.c
@@ -98,8 +98,9 @@ static int pmem_rw_page(struct block_device *bdev, sector_t sector,
return 0;
}
-static long pmem_direct_access(struct block_device *bdev, sector_t sector,
- void **kaddr, unsigned long *pfn, long size)
+#ifdef CONFIG_DEV_PFN
+static long pmem_direct_access(struct block_device *bdev,
+ sector_t sector, __pfn_t *pfn, long size)
{
struct pmem_device *pmem = bdev->bd_disk->private_data;
size_t offset = sector << 9;
@@ -107,16 +108,18 @@ static long pmem_direct_access(struct block_device *bdev, sector_t sector,
if (!pmem)
return -ENODEV;
- *kaddr = pmem->virt_addr + offset;
- *pfn = (pmem->phys_addr + offset) >> PAGE_SHIFT;
+ *pfn = phys_to_pfn_t(pmem->phys_addr + offset);
return pmem->size - offset;
}
+#endif
static const struct block_device_operations pmem_fops = {
.owner = THIS_MODULE,
.rw_page = pmem_rw_page,
+#ifdef CONFIG_DEV_PFN
.direct_access = pmem_direct_access,
+#endif
};
static struct pmem_device *pmem_alloc(struct device *dev, struct resource *res)
diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c
index 5da8515b8fb9..41031e715152 100644
--- a/drivers/s390/block/dcssblk.c
+++ b/drivers/s390/block/dcssblk.c
@@ -29,7 +29,7 @@ static int dcssblk_open(struct block_device *bdev, fmode_t mode);
static void dcssblk_release(struct gendisk *disk, fmode_t mode);
static void dcssblk_make_request(struct request_queue *q, struct bio *bio);
static long dcssblk_direct_access(struct block_device *bdev, sector_t secnum,
- void **kaddr, unsigned long *pfn, long size);
+ __pfn_t *pfn, long size);
static char dcssblk_segments[DCSSBLK_PARM_LEN] = "\0";
@@ -38,7 +38,9 @@ static const struct block_device_operations dcssblk_devops = {
.owner = THIS_MODULE,
.open = dcssblk_open,
.release = dcssblk_release,
+#ifdef CONFIG_DEV_PFN
.direct_access = dcssblk_direct_access,
+#endif
};
struct dcssblk_dev_info {
@@ -877,23 +879,26 @@ fail:
bio_io_error(bio);
}
+#ifdef CONFIG_DEV_PFN
static long
dcssblk_direct_access (struct block_device *bdev, sector_t secnum,
- void **kaddr, unsigned long *pfn, long size)
+ __pfn_t *pfn, long size)
{
struct dcssblk_dev_info *dev_info;
unsigned long offset, dev_sz;
+ void *kaddr;
dev_info = bdev->bd_disk->private_data;
if (!dev_info)
return -ENODEV;
dev_sz = dev_info->end - dev_info->start;
offset = secnum * 512;
- *kaddr = (void *) (dev_info->start + offset);
- *pfn = virt_to_phys(*kaddr) >> PAGE_SHIFT;
+ kaddr = (void *) (dev_info->start + offset);
+ *pfn = phys_to_pfn_t(virt_to_phys(*kaddr));
return dev_sz - offset;
}
+#endif
static void
dcssblk_check_params(void)
diff --git a/fs/block_dev.c b/fs/block_dev.c
index c7e4163ede87..100749b51524 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -437,7 +437,7 @@ EXPORT_SYMBOL_GPL(bdev_write_page);
* accessible at this address.
*/
long bdev_direct_access(struct block_device *bdev, sector_t sector,
- void **addr, unsigned long *pfn, long size)
+ __pfn_t *pfn, long size)
{
long avail;
const struct block_device_operations *ops = bdev->bd_disk->fops;
@@ -452,7 +452,7 @@ long bdev_direct_access(struct block_device *bdev, sector_t sector,
sector += get_start_sect(bdev);
if (sector % (PAGE_SIZE / 512))
return -EINVAL;
- avail = ops->direct_access(bdev, sector, addr, pfn, size);
+ avail = ops->direct_access(bdev, sector, pfn, size);
if (!avail)
return -ERANGE;
return min(avail, size);
diff --git a/fs/dax.c b/fs/dax.c
index 6f65f00e58ec..7d302bfff48a 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -35,13 +35,16 @@ int dax_clear_blocks(struct inode *inode, sector_t block, long size)
might_sleep();
do {
void *addr;
- unsigned long pfn;
+ __pfn_t pfn;
long count;
- count = bdev_direct_access(bdev, sector, &addr, &pfn, size);
+ count = bdev_direct_access(bdev, sector, &pfn, size);
if (count < 0)
return count;
BUG_ON(size < count);
+ addr = kmap_atomic_pfn_t(pfn);
+ if (!addr)
+ return -EIO;
while (count > 0) {
unsigned pgsz = PAGE_SIZE - offset_in_page(addr);
if (pgsz > count)
@@ -57,17 +60,17 @@ int dax_clear_blocks(struct inode *inode, sector_t block, long size)
sector += pgsz / 512;
cond_resched();
}
+ kunmap_atomic_pfn_t(addr);
} while (size);
return 0;
}
EXPORT_SYMBOL_GPL(dax_clear_blocks);
-static long dax_get_addr(struct buffer_head *bh, void **addr, unsigned blkbits)
+static long dax_get_pfn(struct buffer_head *bh, __pfn_t *pfn, unsigned blkbits)
{
- unsigned long pfn;
sector_t sector = bh->b_blocknr << (blkbits - 9);
- return bdev_direct_access(bh->b_bdev, sector, addr, &pfn, bh->b_size);
+ return bdev_direct_access(bh->b_bdev, sector, pfn, bh->b_size);
}
static void dax_new_buf(void *addr, unsigned size, unsigned first, loff_t pos,
@@ -106,7 +109,8 @@ static ssize_t dax_io(struct inode *inode, struct iov_iter *iter,
loff_t pos = start;
loff_t max = start;
loff_t bh_max = start;
- void *addr;
+ void *addr = NULL, *kmap = NULL;
+ __pfn_t pfn;
bool hole = false;
if (iov_iter_rw(iter) != WRITE)
@@ -142,9 +146,19 @@ static ssize_t dax_io(struct inode *inode, struct iov_iter *iter,
addr = NULL;
size = bh->b_size - first;
} else {
- retval = dax_get_addr(bh, &addr, blkbits);
+ if (kmap) {
+ kunmap_atomic_pfn_t(kmap);
+ kmap = NULL;
+ }
+ retval = dax_get_pfn(bh, &pfn, blkbits);
if (retval < 0)
break;
+ kmap = kmap_atomic_pfn_t(pfn);
+ if (!kmap) {
+ retval = -EIO;
+ break;
+ }
+ addr = kmap;
if (buffer_unwritten(bh) || buffer_new(bh))
dax_new_buf(addr, retval, first, pos,
end);
@@ -168,6 +182,9 @@ static ssize_t dax_io(struct inode *inode, struct iov_iter *iter,
addr += len;
}
+ if (kmap)
+ kunmap_atomic_pfn_t(kmap);
+
return (pos == start) ? retval : pos - start;
}
@@ -259,11 +276,17 @@ static int copy_user_bh(struct page *to, struct buffer_head *bh,
unsigned blkbits, unsigned long vaddr)
{
void *vfrom, *vto;
- if (dax_get_addr(bh, &vfrom, blkbits) < 0)
+ __pfn_t pfn;
+
+ if (dax_get_pfn(bh, &pfn, blkbits) < 0)
+ return -EIO;
+ vfrom = kmap_atomic_pfn_t(pfn);
+ if (!vfrom)
return -EIO;
vto = kmap_atomic(to);
copy_user_page(vto, vfrom, vaddr, to);
kunmap_atomic(vto);
+ kunmap_atomic_pfn_t(vfrom);
return 0;
}
@@ -274,7 +297,7 @@ static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh,
sector_t sector = bh->b_blocknr << (inode->i_blkbits - 9);
unsigned long vaddr = (unsigned long)vmf->virtual_address;
void *addr;
- unsigned long pfn;
+ __pfn_t pfn;
pgoff_t size;
int error;
@@ -293,7 +316,7 @@ static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh,
goto out;
}
- error = bdev_direct_access(bh->b_bdev, sector, &addr, &pfn, bh->b_size);
+ error = bdev_direct_access(bh->b_bdev, sector, &pfn, bh->b_size);
if (error < 0)
goto out;
if (error < PAGE_SIZE) {
@@ -301,10 +324,17 @@ static int dax_insert_mapping(struct inode *inode, struct buffer_head *bh,
goto out;
}
- if (buffer_unwritten(bh) || buffer_new(bh))
+ if (buffer_unwritten(bh) || buffer_new(bh)) {
+ addr = kmap_atomic_pfn_t(pfn);
+ if (!addr) {
+ error = -EIO;
+ goto out;
+ }
clear_page(addr);
+ kunmap_atomic_pfn_t(addr);
+ }
- error = vm_insert_mixed(vma, vaddr, pfn);
+ error = vm_insert_mixed(vma, vaddr, __pfn_t_to_pfn(pfn));
out:
i_mmap_unlock_read(mapping);
@@ -517,10 +547,16 @@ int dax_zero_page_range(struct inode *inode, loff_t from, unsigned length,
return err;
if (buffer_written(&bh)) {
void *addr;
- err = dax_get_addr(&bh, &addr, inode->i_blkbits);
+ __pfn_t pfn;
+
+ err = dax_get_pfn(&bh, &pfn, inode->i_blkbits);
if (err < 0)
return err;
+ addr = kmap_atomic_pfn_t(pfn);
+ if (!addr)
+ return -EIO;
memset(addr + offset, 0, length);
+ kunmap_atomic_pfn_t(addr);
}
return 0;
diff --git a/include/asm-generic/pfn.h b/include/asm-generic/pfn.h
index ee1363e3c67c..78ec84282334 100644
--- a/include/asm-generic/pfn.h
+++ b/include/asm-generic/pfn.h
@@ -47,6 +47,17 @@ static inline bool __pfn_t_has_page(__pfn_t pfn)
return (pfn.data & PFN_MASK) == 0;
}
+static inline __pfn_t pfn_to_pfn_t(unsigned long pfn)
+{
+ __pfn_t pfn_t = { .data = (pfn << PFN_SHIFT) | PFN_DEV };
+
+ return pfn_t;
+}
+
+static inline __pfn_t phys_to_pfn_t(dma_addr_t addr)
+{
+ return pfn_to_pfn_t(addr >> PAGE_SHIFT);
+}
#else
static inline bool __pfn_t_has_page(__pfn_t pfn)
{
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 7f9a516f24de..42bcaf2b9311 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1604,8 +1604,7 @@ struct block_device_operations {
int (*rw_page)(struct block_device *, sector_t, struct page *, int rw);
int (*ioctl) (struct block_device *, fmode_t, unsigned, unsigned long);
int (*compat_ioctl) (struct block_device *, fmode_t, unsigned, unsigned long);
- long (*direct_access)(struct block_device *, sector_t,
- void **, unsigned long *pfn, long size);
+ long (*direct_access)(struct block_device *, sector_t, __pfn_t *pfn, long size);
unsigned int (*check_events) (struct gendisk *disk,
unsigned int clearing);
/* ->media_changed() is DEPRECATED, use ->check_events() instead */
@@ -1623,8 +1622,8 @@ extern int __blkdev_driver_ioctl(struct block_device *, fmode_t, unsigned int,
extern int bdev_read_page(struct block_device *, sector_t, struct page *);
extern int bdev_write_page(struct block_device *, sector_t, struct page *,
struct writeback_control *);
-extern long bdev_direct_access(struct block_device *, sector_t, void **addr,
- unsigned long *pfn, long size);
+extern long bdev_direct_access(struct block_device *, sector_t,
+ __pfn_t *pfn, long size);
#else /* CONFIG_BLOCK */
struct block_device;
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH v3 11/11] block: base support for pfn i/o
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
` (9 preceding siblings ...)
2015-05-12 4:30 ` [PATCH v3 10/11] dax: convert to __pfn_t Dan Williams
@ 2015-05-12 4:30 ` Dan Williams
2015-05-23 14:32 ` [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Christoph Hellwig
11 siblings, 0 replies; 20+ messages in thread
From: Dan Williams @ 2015-05-12 4:30 UTC (permalink / raw)
To: linux-kernel
Cc: axboe, linux-arch, riel, linux-nvdimm, david, hch, linux-fsdevel,
mgorman, j.glisse, Tejun Heo, akpm, mingo
Allow block device drivers to opt-in to receiving bio(s) where the
bio_vec(s) point to memory that is not backed by struct page entries.
When a driver opts in it asserts that it will use the __pfn_t versions of the
dma_map/kmap/scatterlist apis in its bio submission path.
Cc: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
block/bio.c | 46 ++++++++++++++++++++++++++++++++++++++-------
block/blk-core.c | 9 +++++++++
include/linux/blk_types.h | 1 +
include/linux/blkdev.h | 2 ++
4 files changed, 51 insertions(+), 7 deletions(-)
diff --git a/block/bio.c b/block/bio.c
index 7100fd6d5898..58553dfd777e 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -567,6 +567,7 @@ void __bio_clone_fast(struct bio *bio, struct bio *bio_src)
bio->bi_rw = bio_src->bi_rw;
bio->bi_iter = bio_src->bi_iter;
bio->bi_io_vec = bio_src->bi_io_vec;
+ bio->bi_flags |= bio_src->bi_flags & (1 << BIO_PFN);
}
EXPORT_SYMBOL(__bio_clone_fast);
@@ -658,6 +659,8 @@ struct bio *bio_clone_bioset(struct bio *bio_src, gfp_t gfp_mask,
goto integrity_clone;
}
+ bio->bi_flags |= bio_src->bi_flags & (1 << BIO_PFN);
+
bio_for_each_segment(bv, bio_src, iter)
bio->bi_io_vec[bio->bi_vcnt++] = bv;
@@ -699,9 +702,9 @@ int bio_get_nr_vecs(struct block_device *bdev)
}
EXPORT_SYMBOL(bio_get_nr_vecs);
-static int __bio_add_page(struct request_queue *q, struct bio *bio, struct page
- *page, unsigned int len, unsigned int offset,
- unsigned int max_sectors)
+static int __bio_add_pfn(struct request_queue *q, struct bio *bio,
+ __pfn_t pfn, unsigned int len, unsigned int offset,
+ unsigned int max_sectors)
{
int retried_segments = 0;
struct bio_vec *bvec;
@@ -723,7 +726,7 @@ static int __bio_add_page(struct request_queue *q, struct bio *bio, struct page
if (bio->bi_vcnt > 0) {
struct bio_vec *prev = &bio->bi_io_vec[bio->bi_vcnt - 1];
- if (page == bvec_page(prev) &&
+ if (__pfn_t_to_pfn(pfn) == __pfn_t_to_pfn(prev->bv_pfn) &&
offset == prev->bv_offset + prev->bv_len) {
unsigned int prev_bv_len = prev->bv_len;
prev->bv_len += len;
@@ -768,7 +771,7 @@ static int __bio_add_page(struct request_queue *q, struct bio *bio, struct page
* cannot add the page
*/
bvec = &bio->bi_io_vec[bio->bi_vcnt];
- bvec_set_page(bvec, page);
+ bvec->bv_pfn = pfn;
bvec->bv_len = len;
bvec->bv_offset = offset;
bio->bi_vcnt++;
@@ -845,7 +848,7 @@ static int __bio_add_page(struct request_queue *q, struct bio *bio, struct page
int bio_add_pc_page(struct request_queue *q, struct bio *bio, struct page *page,
unsigned int len, unsigned int offset)
{
- return __bio_add_page(q, bio, page, len, offset,
+ return __bio_add_pfn(q, bio, page_to_pfn_t(page), len, offset,
queue_max_hw_sectors(q));
}
EXPORT_SYMBOL(bio_add_pc_page);
@@ -872,10 +875,39 @@ int bio_add_page(struct bio *bio, struct page *page, unsigned int len,
if ((max_sectors < (len >> 9)) && !bio->bi_iter.bi_size)
max_sectors = len >> 9;
- return __bio_add_page(q, bio, page, len, offset, max_sectors);
+ return __bio_add_pfn(q, bio, page_to_pfn_t(page), len, offset,
+ max_sectors);
}
EXPORT_SYMBOL(bio_add_page);
+/**
+ * bio_add_pfn - attempt to add pfn to bio
+ * @bio: destination bio
+ * @pfn: pfn to add
+ * @len: vec entry length
+ * @offset: vec entry offset
+ *
+ * Identical to bio_add_page() except this variant flags the bio as
+ * not have struct page backing. A given request_queue must assert
+ * that it is prepared to handle this constraint before bio(s)
+ * flagged in the manner can be passed.
+ */
+int bio_add_pfn(struct bio *bio, __pfn_t pfn, unsigned int len,
+ unsigned int offset)
+{
+ struct request_queue *q = bdev_get_queue(bio->bi_bdev);
+ unsigned int max_sectors;
+
+ if (!blk_queue_pfn(q))
+ return 0;
+ set_bit(BIO_PFN, &bio->bi_flags);
+ max_sectors = blk_max_size_offset(q, bio->bi_iter.bi_sector);
+ if ((max_sectors < (len >> 9)) && !bio->bi_iter.bi_size)
+ max_sectors = len >> 9;
+
+ return __bio_add_pfn(q, bio, pfn, len, offset, max_sectors);
+}
+
struct submit_bio_ret {
struct completion event;
int error;
diff --git a/block/blk-core.c b/block/blk-core.c
index 94d2c6ccf801..1275e2c08c16 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1856,6 +1856,15 @@ generic_make_request_checks(struct bio *bio)
goto end_io;
}
+ if (bio_flagged(bio, BIO_PFN)) {
+ if (IS_ENABLED(CONFIG_DEV_PFN) && blk_queue_pfn(q))
+ /* pass */;
+ else {
+ err = -EOPNOTSUPP;
+ goto end_io;
+ }
+ }
+
/*
* Various block parts want %current->io_context and lazy ioc
* allocation ends up trading a lot of pain for a small amount of
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index f50464e167b4..ccde0b2d689d 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -150,6 +150,7 @@ struct bio {
#define BIO_NULL_MAPPED 8 /* contains invalid user pages */
#define BIO_QUIET 9 /* Make BIO Quiet */
#define BIO_SNAP_STABLE 10 /* bio data must be snapshotted during write */
+#define BIO_PFN 11 /* bio_vec references memory without struct page */
/*
* Flags starting here get preserved by bio_reset() - this includes
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 42bcaf2b9311..d3f9b8cc50f2 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -513,6 +513,7 @@ struct request_queue {
#define QUEUE_FLAG_INIT_DONE 20 /* queue is initialized */
#define QUEUE_FLAG_NO_SG_MERGE 21 /* don't attempt to merge SG segments*/
#define QUEUE_FLAG_SG_GAPS 22 /* queue doesn't support SG gaps */
+#define QUEUE_FLAG_PFN 23 /* queue supports pfn-only bio_vec(s) */
#define QUEUE_FLAG_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \
(1 << QUEUE_FLAG_STACKABLE) | \
@@ -594,6 +595,7 @@ static inline void queue_flag_clear(unsigned int flag, struct request_queue *q)
#define blk_queue_noxmerges(q) \
test_bit(QUEUE_FLAG_NOXMERGES, &(q)->queue_flags)
#define blk_queue_nonrot(q) test_bit(QUEUE_FLAG_NONROT, &(q)->queue_flags)
+#define blk_queue_pfn(q) test_bit(QUEUE_FLAG_PFN, &(q)->queue_flags)
#define blk_queue_io_stat(q) test_bit(QUEUE_FLAG_IO_STAT, &(q)->queue_flags)
#define blk_queue_add_random(q) test_bit(QUEUE_FLAG_ADD_RANDOM, &(q)->queue_flags)
#define blk_queue_stackable(q) \
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t
2015-05-12 4:29 [PATCH v3 00/11] evacuate struct page from the block layer, introduce __pfn_t Dan Williams
` (10 preceding siblings ...)
2015-05-12 4:30 ` [PATCH v3 11/11] block: base support for pfn i/o Dan Williams
@ 2015-05-23 14:32 ` Christoph Hellwig
11 siblings, 0 replies; 20+ messages in thread
From: Christoph Hellwig @ 2015-05-23 14:32 UTC (permalink / raw)
To: Dan Williams
Cc: linux-kernel, Boaz Harrosh, Jan Kara, Mike Snitzer, Neil Brown,
Benjamin Herrenschmidt, Dave Hansen, Heiko Carstens, Chris Mason,
Paul Mackerras, H. Peter Anvin, j.glisse, mingo, Alasdair Kergon,
linux-arch, linux-nvdimm, hch, mgorman, Matthew Wilcox,
Ross Zwisler, riel, david, Tejun Heo, axboe, Theodore Ts'o,
Martin K. Petersen, Julia Lawall, Martin Schwidefsky,
linux-fsdevel, akpm, Linus Torvalds
I don't like this series at all, it does too much and too little at
the same time.
There's three totally different parts to it that are mixed up:
(1) cleanups to use accessors for struct scatterlist instead of exposing
the intricate details of chained S/G list to users
(2) a switch of struct scatterlist to store a PFN instead of a page
(3) switch struct bio_vec to store a struct PFN instead of a page
(1) are pretty obvious cleanups, and they should have been submited
separately long time ago.
(2) seems like a good idea, but only when done properly, that is a full
conversion to it. Not a you need a config option, in which case maybe
some architectures and can sometimes deal with it if they driver says:
meh, okay.
Given that nature of SGLs most consumer want a physical address anyway,
and it shouldn't be a problem to convert all others that need a kernel
mapping to proper helpers.
(3) I'm not sure about in it's current form. The bio_vec sees all kinds
of highlevel use and we need to be a lot more careful about it, due to
the way we e.g. use the in the iov_iter based read/write interfaces.
It could be that pfn_t based approach there makes sense, but only if
we ensure all consumer can always handle them.
Because of that I'd suggest you try to get (1) and (2) done properly
first, at that point we'll have how we can do (3) without causing
a big mess.
^ permalink raw reply [flat|nested] 20+ messages in thread