[PATCH 0/7] [RFC] iosched: make a difference between read/readahead requests

[PATCH 0/7] [RFC] iosched: make a difference between read/readahead requests

[Fengguang Wu]

Hi Jens,

This patchset does two jobs:
	1) do io schedule differently on READ/READA requests.
		- to help improve I/O latency and throughput
	2) do notification/action on READA => READ events
		- to make the elevators better informed
		- to prevent the priority inversion problem
		- also brings some CPU overheads*
(*) I'm not able to provide the numbers at the moment.
    But sure for the next time.


The patches come in two groups:

1) explicitly schedule READA requests
Note: currently only the deadline elevator is touched.

[PATCH 1/7] iosched: introduce WRITEA                                                  
[PATCH 2/7] iosched: introduce parameter deadline.reada_expire                         
[PATCH 3/7] iosched: introduce deadline_add_drq_fifo()                                 
[PATCH 4/7] iosched: submit READA requests on possible readahead code path             

2) notify/act on pending reads
Naming issue: how about pending_read/need_page/... for kick_page?

[PATCH 5/7] iosched: introduce elv_kick_page()                                         
[PATCH 6/7] iosched: run elv_kick_page() on sync read                                  
[PATCH 7/7] iosched: introduce deadline_kick_page()                                    

Most overheads should be in functions deadline_kick_page() and
deadline_add_drq_fifo(). I'll explore the details later.

Any comments are welcome, thanks.

Fengguang Wu
--
Dept. Automation                University of Science and Technology of China

[PATCH 1/7] iosched: introduce WRITEA

[Fengguang Wu]

[-- Attachment #1: iosched-reada-redef.patch --]
[-- Type: text/plain, Size: 911 bytes --]

Introduce WRITEA as 3, and redefine SWRITE as 5.

I'm not sure if WRITEA will ever be used, though it would be better to
redefine SWRITE to avoid possible conflict with BIO_RW_AHEAD.

Signed-off-by: Wu Fengguang <wfg@mail.ustc.edu.cn>
---


--- linux-2.6.17-rc6-mm2.orig/include/linux/fs.h
+++ linux-2.6.17-rc6-mm2/include/linux/fs.h
@@ -74,8 +74,9 @@ extern int dir_notify_enable;
 #define READ 0
 #define WRITE 1
 #define READA 2		/* read-ahead  - don't block if no resources */
-#define SWRITE 3	/* for ll_rw_block() - wait for buffer lock */
+#define WRITEA 3	/* write-ahead - don't block if no resources */
 #define SPECIAL 4	/* For non-blockdevice requests in request queue */
+#define SWRITE 5	/* for ll_rw_block() - wait for buffer lock */
 #define READ_SYNC	(READ | (1 << BIO_RW_SYNC))
 #define WRITE_SYNC	(WRITE | (1 << BIO_RW_SYNC))
 #define WRITE_BARRIER	((1 << BIO_RW) | (1 << BIO_RW_BARRIER))

--

[PATCH 2/7] iosched: introduce parameter deadline.reada_expire

[Fengguang Wu]

[-- Attachment #1: iosched-reada-deadline.patch --]
[-- Type: text/plain, Size: 2760 bytes --]

Introduce parameter reada_expire to the deadline elevator.

It avoids readahead requests contending with read requests,
and helps improve latency/throughput when there's many concurrent readers.

Signed-off-by: Wu Fengguang <wfg@mail.ustc.edu.cn>
---


--- linux-2.6.17-rc6-mm2.orig/block/deadline-iosched.c
+++ linux-2.6.17-rc6-mm2/block/deadline-iosched.c
@@ -19,7 +19,8 @@
 /*
  * See Documentation/block/deadline-iosched.txt
  */
-static const int read_expire = HZ / 2;  /* max time before a read is submitted. */
+static const int read_expire = HZ / 2;	/* max time before an _impending_ read is submitted. */
+static const int reada_expire = 60 * HZ;/* max time before a read-ahead is submitted. */
 static const int write_expire = 5 * HZ; /* ditto for writes, these limits are SOFT! */
 static const int writes_starved = 2;    /* max times reads can starve a write */
 static const int fifo_batch = 16;       /* # of sequential requests treated as one
@@ -56,7 +57,7 @@ struct deadline_data {
 	/*
 	 * settings that change how the i/o scheduler behaves
 	 */
-	int fifo_expire[2];
+	int fifo_expire[4];
 	int fifo_batch;
 	int writes_starved;
 	int front_merges;
@@ -711,7 +712,9 @@ static void *deadline_init_queue(request
 	dd->sort_list[READ] = RB_ROOT;
 	dd->sort_list[WRITE] = RB_ROOT;
 	dd->fifo_expire[READ] = read_expire;
+	dd->fifo_expire[READA] = reada_expire;
 	dd->fifo_expire[WRITE] = write_expire;
+	dd->fifo_expire[WRITEA] = write_expire;
 	dd->writes_starved = writes_starved;
 	dd->front_merges = 1;
 	dd->fifo_batch = fifo_batch;
@@ -780,6 +783,7 @@ static ssize_t __FUNC(elevator_t *e, cha
 	return deadline_var_show(__data, (page));			\
 }
 SHOW_FUNCTION(deadline_read_expire_show, dd->fifo_expire[READ], 1);
+SHOW_FUNCTION(deadline_reada_expire_show, dd->fifo_expire[READA], 1);
 SHOW_FUNCTION(deadline_write_expire_show, dd->fifo_expire[WRITE], 1);
 SHOW_FUNCTION(deadline_writes_starved_show, dd->writes_starved, 0);
 SHOW_FUNCTION(deadline_front_merges_show, dd->front_merges, 0);
@@ -803,6 +807,7 @@ static ssize_t __FUNC(elevator_t *e, con
 	return ret;							\
 }
 STORE_FUNCTION(deadline_read_expire_store, &dd->fifo_expire[READ], 0, INT_MAX, 1);
+STORE_FUNCTION(deadline_reada_expire_store, &dd->fifo_expire[READA], 0, INT_MAX, 1);
 STORE_FUNCTION(deadline_write_expire_store, &dd->fifo_expire[WRITE], 0, INT_MAX, 1);
 STORE_FUNCTION(deadline_writes_starved_store, &dd->writes_starved, INT_MIN, INT_MAX, 0);
 STORE_FUNCTION(deadline_front_merges_store, &dd->front_merges, 0, 1, 0);
@@ -815,6 +820,7 @@ STORE_FUNCTION(deadline_fifo_batch_store
 
 static struct elv_fs_entry deadline_attrs[] = {
 	DD_ATTR(read_expire),
+	DD_ATTR(reada_expire),
 	DD_ATTR(write_expire),
 	DD_ATTR(writes_starved),
 	DD_ATTR(front_merges),

--

[PATCH 4/7] iosched: submit READA requests on possible readahead code path

[Fengguang Wu]

[-- Attachment #1: iosched-submit-reada-on-possible-readahead-path.patch --]
[-- Type: text/plain, Size: 2119 bytes --]

Change the request type from READ to READA in possible readahead code paths.

- call mpage_bio_submit(READA) in mpage_readpages()
- call submit_bio(READA) in swap_readpage()

Signed-off-by: Wu Fengguang <wfg@mail.ustc.edu.cn>
---


 fs/mpage.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

--- linux-2.6.17-mm1.orig/fs/mpage.c
+++ linux-2.6.17-mm1/fs/mpage.c
@@ -302,7 +302,7 @@ do_mpage_readpage(struct bio *bio, struc
 	 * This page will go to BIO.  Do we need to send this BIO off first?
 	 */
 	if (bio && (*last_block_in_bio != blocks[0] - 1))
-		bio = mpage_bio_submit(READ, bio);
+		bio = mpage_bio_submit(READA, bio);
 
 alloc_new:
 	if (bio == NULL) {
@@ -315,12 +315,12 @@ alloc_new:
 
 	length = first_hole << blkbits;
 	if (bio_add_page(bio, page, length, 0) < length) {
-		bio = mpage_bio_submit(READ, bio);
+		bio = mpage_bio_submit(READA, bio);
 		goto alloc_new;
 	}
 
 	if (buffer_boundary(map_bh) || (first_hole != blocks_per_page))
-		bio = mpage_bio_submit(READ, bio);
+		bio = mpage_bio_submit(READA, bio);
 	else
 		*last_block_in_bio = blocks[blocks_per_page - 1];
 out:
@@ -328,7 +328,7 @@ out:
 
 confused:
 	if (bio)
-		bio = mpage_bio_submit(READ, bio);
+		bio = mpage_bio_submit(READA, bio);
 	if (!PageUptodate(page))
 	        block_read_full_page(page, get_block);
 	else
@@ -418,7 +418,7 @@ mpage_readpages(struct address_space *ma
 	pagevec_lru_add(&lru_pvec);
 	BUG_ON(!list_empty(pages));
 	if (bio)
-		mpage_bio_submit(READ, bio);
+		mpage_bio_submit(READA, bio);
 	return 0;
 }
 EXPORT_SYMBOL(mpage_readpages);
@@ -437,7 +437,7 @@ int mpage_readpage(struct page *page, ge
 	bio = do_mpage_readpage(bio, page, 1, &last_block_in_bio,
 			&map_bh, &first_logical_block, get_block);
 	if (bio)
-		mpage_bio_submit(READ, bio);
+		mpage_bio_submit(READA, bio);
 	return 0;
 }
 EXPORT_SYMBOL(mpage_readpage);
--- linux-2.6.17-mm1.orig/mm/page_io.c
+++ linux-2.6.17-mm1/mm/page_io.c
@@ -124,7 +124,7 @@ int swap_readpage(struct file *file, str
 		goto out;
 	}
 	inc_page_state(pswpin);
-	submit_bio(READ, bio);
+	submit_bio(READA, bio);
 out:
 	return ret;
 }

--

[PATCH 6/7] iosched: run elv_kick_page() on sync read

[Fengguang Wu]

[-- Attachment #1: iosched-kick-page-on-sync-read.patch --]
[-- Type: text/plain, Size: 939 bytes --]

Call elv_kick_page() to notify the elevator of a pending read.

Requests of priority IOPRIO_CLASS_IDLE are ignored.

Signed-off-by: Wu Fengguang <wfg@mail.ustc.edu.cn>
---


--- linux-2.6.17-rc5-mm3.orig/block/ll_rw_blk.c
+++ linux-2.6.17-rc5-mm3/block/ll_rw_blk.c
@@ -1619,6 +1619,9 @@ static void blk_backing_dev_unplug(struc
 {
 	request_queue_t *q = bdi->unplug_io_data;
 
+	if (IOPRIO_PRIO_CLASS(current->ioprio) != IOPRIO_CLASS_IDLE)
+		elv_kick_page(q, page);
+
 	/*
 	 * devices don't necessarily have an ->unplug_fn defined
 	 */
--- linux-2.6.17-rc5-mm3.orig/fs/buffer.c
+++ linux-2.6.17-rc5-mm3/fs/buffer.c
@@ -63,8 +63,9 @@ static int sync_buffer(void *word)
 
 	smp_mb();
 	bd = bh->b_bdev;
-	if (bd)
-		blk_run_address_space(bd->bd_inode->i_mapping);
+	if (bd && bd->bd_inode && bd->bd_inode->i_mapping)
+		blk_run_backing_dev(bd->bd_inode->i_mapping->backing_dev_info,
+					bh->b_page);
 	io_schedule();
 	return 0;
 }

--

[PATCH 7/7] iosched: introduce deadline_kick_page()

[Fengguang Wu]

[-- Attachment #1: iosched-kick-page-deadline.patch --]
[-- Type: text/plain, Size: 1891 bytes --]

Introduce deadline_kick_page() to
	- find the request containing the page
	- remove its BIO_RW_AHEAD flag
	- reschedule if it was of type READA

Signed-off-by: Wu Fengguang <wfg@mail.ustc.edu.cn>
---


 block/deadline-iosched.c |   45 +++++++++++++++++++++++++++++++++++++++++++--
 1 files changed, 43 insertions(+), 2 deletions(-)

--- linux-2.6.17-rc6-mm2.orig/block/deadline-iosched.c
+++ linux-2.6.17-rc6-mm2/block/deadline-iosched.c
@@ -317,6 +317,44 @@ deadline_add_request(struct request_queu
 }
 
 /*
+ * We have a pending read on @page,
+ * find the corresponding request of type READA,
+ * promote it to READ, and reschedule it.
+ */
+static int
+deadline_kick_page(struct request_queue *q, struct page *page)
+{
+	struct deadline_data *dd = q->elevator->elevator_data;
+	struct deadline_rq *drq;
+	struct request *rq;
+	struct list_head *pos;
+	struct bio_vec *bvec;
+	struct bio *bio;
+	int i;
+
+	list_for_each(pos, &dd->fifo_list[READ]) {
+		drq = list_entry_fifo(pos);
+		rq = drq->request;
+		if (rq->flags & (1 << BIO_RW_AHEAD)) {
+			rq_for_each_bio(bio, rq) {
+				bio_for_each_segment(bvec, bio, i) {
+					if (page == bvec->bv_page)
+						goto found;
+				}
+			}
+		}
+	}
+
+	return -1;
+
+found:
+	rq->flags &= ~(1 << BIO_RW_AHEAD);
+	list_del(&drq->fifo);
+	deadline_add_drq_fifo(dd, rq);
+	return 0;
+}
+
+/*
  * remove rq from rbtree, fifo, and hash
  */
 static void deadline_remove_request(request_queue_t *q, struct request *rq)
@@ -794,6 +832,7 @@ static struct elevator_type iosched_dead
 		.elevator_merge_req_fn =	deadline_merged_requests,
 		.elevator_dispatch_fn =		deadline_dispatch_requests,
 		.elevator_add_req_fn =		deadline_add_request,
+		.elevator_kick_page_fn =	deadline_kick_page,
 		.elevator_queue_empty_fn =	deadline_queue_empty,
 		.elevator_former_req_fn =	deadline_former_request,
 		.elevator_latter_req_fn =	deadline_latter_request,

--

[PATCH 1/7] iosched: introduce WRITEA

[Fengguang Wu]

[-- Attachment #1: iosched-reada-redef.patch --]
[-- Type: text/plain, Size: 911 bytes --]

Introduce WRITEA as 3, and redefine SWRITE as 5.

I'm not sure if WRITEA will ever be used, though it would be better to
redefine SWRITE to avoid possible conflict with BIO_RW_AHEAD.

Signed-off-by: Wu Fengguang <wfg@mail.ustc.edu.cn>
---


--- linux-2.6.17-rc6-mm2.orig/include/linux/fs.h
+++ linux-2.6.17-rc6-mm2/include/linux/fs.h
@@ -74,8 +74,9 @@ extern int dir_notify_enable;
 #define READ 0
 #define WRITE 1
 #define READA 2		/* read-ahead  - don't block if no resources */
-#define SWRITE 3	/* for ll_rw_block() - wait for buffer lock */
+#define WRITEA 3	/* write-ahead - don't block if no resources */
 #define SPECIAL 4	/* For non-blockdevice requests in request queue */
+#define SWRITE 5	/* for ll_rw_block() - wait for buffer lock */
 #define READ_SYNC	(READ | (1 << BIO_RW_SYNC))
 #define WRITE_SYNC	(WRITE | (1 << BIO_RW_SYNC))
 #define WRITE_BARRIER	((1 << BIO_RW) | (1 << BIO_RW_BARRIER))

--

[PATCH 1/7] iosched: introduce WRITEA

[Fengguang Wu]

[-- Attachment #1: iosched-reada-redef.patch --]
[-- Type: text/plain, Size: 911 bytes --]

Introduce WRITEA as 3, and redefine SWRITE as 5.

I'm not sure if WRITEA will ever be used, though it would be better to
redefine SWRITE to avoid possible conflict with BIO_RW_AHEAD.

Signed-off-by: Wu Fengguang <wfg@mail.ustc.edu.cn>
---


--- linux-2.6.17-rc6-mm2.orig/include/linux/fs.h
+++ linux-2.6.17-rc6-mm2/include/linux/fs.h
@@ -74,8 +74,9 @@ extern int dir_notify_enable;
 #define READ 0
 #define WRITE 1
 #define READA 2		/* read-ahead  - don't block if no resources */
-#define SWRITE 3	/* for ll_rw_block() - wait for buffer lock */
+#define WRITEA 3	/* write-ahead - don't block if no resources */
 #define SPECIAL 4	/* For non-blockdevice requests in request queue */
+#define SWRITE 5	/* for ll_rw_block() - wait for buffer lock */
 #define READ_SYNC	(READ | (1 << BIO_RW_SYNC))
 #define WRITE_SYNC	(WRITE | (1 << BIO_RW_SYNC))
 #define WRITE_BARRIER	((1 << BIO_RW) | (1 << BIO_RW_BARRIER))

--