* [PATCH 01/10] lightnvm: pblk: fix bad le64 assignations
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
@ 2017-06-30 15:56 ` Javier González
2017-06-30 15:56 ` [PATCH 02/10] lightnvm: pblk: fix double-free on pblk init Javier González
` (9 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Javier González @ 2017-06-30 15:56 UTC (permalink / raw)
To: mb, axboe
Cc: linux-block, linux-kernel, Javier González, Matias Bjørling
Use the right types and conversions on le64 variables. Reported by
sparse.
Signed-off-by: Javier González <javier@cnexlabs.com>
Signed-off-by: Matias Bjørling <matias@cnexlabs.com>
---
drivers/lightnvm/pblk-core.c | 2 +-
drivers/lightnvm/pblk-gc.c | 5 ++++-
drivers/lightnvm/pblk-map.c | 2 +-
drivers/lightnvm/pblk-recovery.c | 2 +-
4 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/drivers/lightnvm/pblk-core.c b/drivers/lightnvm/pblk-core.c
index 7648186bd1b1..a654b34f6f86 100644
--- a/drivers/lightnvm/pblk-core.c
+++ b/drivers/lightnvm/pblk-core.c
@@ -765,7 +765,7 @@ static int pblk_line_submit_smeta_io(struct pblk *pblk, struct pblk_line *line,
rqd.ppa_list[i] = addr_to_gen_ppa(pblk, paddr, line->id);
if (dir == WRITE) {
- u64 addr_empty = cpu_to_le64(ADDR_EMPTY);
+ __le64 addr_empty = cpu_to_le64(ADDR_EMPTY);
meta_list[i].lba = lba_list[paddr] = addr_empty;
}
diff --git a/drivers/lightnvm/pblk-gc.c b/drivers/lightnvm/pblk-gc.c
index 1d289242ab92..9b4059b93855 100644
--- a/drivers/lightnvm/pblk-gc.c
+++ b/drivers/lightnvm/pblk-gc.c
@@ -314,10 +314,13 @@ static struct pblk_line *pblk_gc_get_victim_line(struct pblk *pblk,
struct list_head *group_list)
{
struct pblk_line *line, *victim;
+ int line_vsc, victim_vsc;
victim = list_first_entry(group_list, struct pblk_line, list);
list_for_each_entry(line, group_list, list) {
- if (*line->vsc < *victim->vsc)
+ line_vsc = le32_to_cpu(*line->vsc);
+ victim_vsc = le32_to_cpu(*victim->vsc);
+ if (line_vsc < victim_vsc)
victim = line;
}
diff --git a/drivers/lightnvm/pblk-map.c b/drivers/lightnvm/pblk-map.c
index a9be03cd07a8..fddb924f6dde 100644
--- a/drivers/lightnvm/pblk-map.c
+++ b/drivers/lightnvm/pblk-map.c
@@ -53,7 +53,7 @@ static void pblk_map_page_data(struct pblk *pblk, unsigned int sentry,
lba_list[paddr] = cpu_to_le64(w_ctx->lba);
line->nr_valid_lbas++;
} else {
- u64 addr_empty = cpu_to_le64(ADDR_EMPTY);
+ __le64 addr_empty = cpu_to_le64(ADDR_EMPTY);
lba_list[paddr] = meta_list[i].lba = addr_empty;
__pblk_map_invalidate(pblk, line, paddr);
diff --git a/drivers/lightnvm/pblk-recovery.c b/drivers/lightnvm/pblk-recovery.c
index abf36f587477..7e1c314f2766 100644
--- a/drivers/lightnvm/pblk-recovery.c
+++ b/drivers/lightnvm/pblk-recovery.c
@@ -395,7 +395,7 @@ static int pblk_recov_pad_oob(struct pblk *pblk, struct pblk_line *line,
for (j = 0; j < pblk->min_write_pgs; j++, i++, w_ptr++) {
struct ppa_addr dev_ppa;
- u64 addr_empty = cpu_to_le64(ADDR_EMPTY);
+ __le64 addr_empty = cpu_to_le64(ADDR_EMPTY);
dev_ppa = addr_to_gen_ppa(pblk, w_ptr, line->id);
--
2.7.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 02/10] lightnvm: pblk: fix double-free on pblk init
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
2017-06-30 15:56 ` [PATCH 01/10] lightnvm: pblk: fix bad le64 assignations Javier González
@ 2017-06-30 15:56 ` Javier González
2017-06-30 15:56 ` [PATCH 03/10] lightnvm: pblk: remove unused return variable Javier González
` (8 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Javier González @ 2017-06-30 15:56 UTC (permalink / raw)
To: mb, axboe
Cc: linux-block, linux-kernel, Javier González, Matias Bjørling
Prevent pblk->lines being double freed in case of an error during pblk
initialization.
Fixes: dd2a43437337: "lightnvm: pblk: sched. metadata on write thread"
Reported-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Javier González <javier@cnexlabs.com>
Signed-off-by: Matias Bjørling <matias@cnexlabs.com>
---
drivers/lightnvm/pblk-init.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/drivers/lightnvm/pblk-init.c b/drivers/lightnvm/pblk-init.c
index 8bdaf7e0e00b..b3fc310aa51c 100644
--- a/drivers/lightnvm/pblk-init.c
+++ b/drivers/lightnvm/pblk-init.c
@@ -812,8 +812,6 @@ static int pblk_lines_init(struct pblk *pblk)
fail_free_lines:
while (--i >= 0)
pblk_free_line_bitmaps(&pblk->lines[i]);
-
- kfree(pblk->lines);
fail_free_bb_aux:
kfree(l_mg->bb_aux);
fail_free_bb_template:
--
2.7.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 03/10] lightnvm: pblk: remove unused return variable
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
2017-06-30 15:56 ` [PATCH 01/10] lightnvm: pblk: fix bad le64 assignations Javier González
2017-06-30 15:56 ` [PATCH 02/10] lightnvm: pblk: fix double-free on pblk init Javier González
@ 2017-06-30 15:56 ` Javier González
2017-06-30 15:56 ` [PATCH 04/10] lightnvm: pblk: schedule if data is not ready Javier González
` (7 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Javier González @ 2017-06-30 15:56 UTC (permalink / raw)
To: mb, axboe
Cc: linux-block, linux-kernel, Javier González, Matias Bjørling
Remove unused variable.
Signed-off-by: Javier González <javier@cnexlabs.com>
Signed-off-by: Matias Bjørling <matias@cnexlabs.com>
---
drivers/lightnvm/pblk-sysfs.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/lightnvm/pblk-sysfs.c b/drivers/lightnvm/pblk-sysfs.c
index 22e6f2ad4aee..95fb434e2f01 100644
--- a/drivers/lightnvm/pblk-sysfs.c
+++ b/drivers/lightnvm/pblk-sysfs.c
@@ -322,7 +322,7 @@ static ssize_t pblk_sysfs_gc_force(struct pblk *pblk, const char *page,
size_t len)
{
size_t c_len;
- int ret, force;
+ int force;
c_len = strcspn(page, "\n");
if (c_len >= len)
@@ -331,7 +331,7 @@ static ssize_t pblk_sysfs_gc_force(struct pblk *pblk, const char *page,
if (kstrtouint(page, 0, &force))
return -EINVAL;
- ret = pblk_gc_sysfs_force(pblk, force);
+ pblk_gc_sysfs_force(pblk, force);
return len;
}
--
2.7.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 04/10] lightnvm: pblk: schedule if data is not ready
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
` (2 preceding siblings ...)
2017-06-30 15:56 ` [PATCH 03/10] lightnvm: pblk: remove unused return variable Javier González
@ 2017-06-30 15:56 ` Javier González
2017-06-30 15:56 ` [PATCH 05/10] lightnvm: pblk: use right metadata buffer for recovery Javier González
` (6 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Javier González @ 2017-06-30 15:56 UTC (permalink / raw)
To: mb, axboe
Cc: linux-block, linux-kernel, Javier González, Matias Bjørling
When user threads place data into the write buffer, they reserve space
and do the memory copy out of the lock. As a consequence, when the write
thread starts persisting data, there is a chance that it is not copied
yet. In this case, avoid polling, and schedule before retrying.
Signed-off-by: Javier González <javier@cnexlabs.com>
Signed-off-by: Matias Bjørling <matias@cnexlabs.com>
---
drivers/lightnvm/pblk-rb.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/lightnvm/pblk-rb.c b/drivers/lightnvm/pblk-rb.c
index 665a4ccfe7f5..2dda874af890 100644
--- a/drivers/lightnvm/pblk-rb.c
+++ b/drivers/lightnvm/pblk-rb.c
@@ -578,8 +578,10 @@ unsigned int pblk_rb_read_to_bio(struct pblk_rb *rb, struct nvm_rq *rqd,
*/
try:
flags = READ_ONCE(entry->w_ctx.flags);
- if (!(flags & PBLK_WRITTEN_DATA))
+ if (!(flags & PBLK_WRITTEN_DATA)) {
+ io_schedule();
goto try;
+ }
page = virt_to_page(entry->data);
if (!page) {
--
2.7.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 05/10] lightnvm: pblk: use right metadata buffer for recovery
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
` (3 preceding siblings ...)
2017-06-30 15:56 ` [PATCH 04/10] lightnvm: pblk: schedule if data is not ready Javier González
@ 2017-06-30 15:56 ` Javier González
2017-06-30 15:56 ` [PATCH 06/10] lightnvm: pblk: use vmalloc for GC data buffer Javier González
` (5 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Javier González @ 2017-06-30 15:56 UTC (permalink / raw)
To: mb, axboe
Cc: linux-block, linux-kernel, Javier González, Matias Bjørling
Fix bad metadata buffer assignations introduced when refactoring the
medatada write path.
Fixes: dd2a43437337 lightnvm: pblk: sched. metadata on write thread
Signed-off-by: Javier González <javier@cnexlabs.com>
Signed-off-by: Matias Bjørling <matias@cnexlabs.com>
---
drivers/lightnvm/pblk-recovery.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/lightnvm/pblk-recovery.c b/drivers/lightnvm/pblk-recovery.c
index 7e1c314f2766..6d58659fa3da 100644
--- a/drivers/lightnvm/pblk-recovery.c
+++ b/drivers/lightnvm/pblk-recovery.c
@@ -801,7 +801,7 @@ struct pblk_line *pblk_recov_l2p(struct pblk *pblk)
set_bit(meta_line, &l_mg->meta_bitmap);
smeta = l_mg->sline_meta[meta_line];
emeta = l_mg->eline_meta[meta_line];
- smeta_buf = smeta->buf;
+ smeta_buf = (struct line_smeta *)smeta;
spin_unlock(&l_mg->free_lock);
/* Order data lines using their sequence number */
@@ -888,9 +888,9 @@ struct pblk_line *pblk_recov_l2p(struct pblk *pblk)
nr_bb = bitmap_weight(line->blk_bitmap, lm->blk_per_line);
off -= nr_bb * geo->sec_per_pl;
- memset(&emeta->buf, 0, lm->emeta_len[0]);
- line->emeta = emeta;
line->emeta_ssec = off;
+ line->emeta = emeta;
+ memset(line->emeta->buf, 0, lm->emeta_len[0]);
if (pblk_line_read_emeta(pblk, line, line->emeta->buf)) {
pblk_recov_l2p_from_oob(pblk, line);
--
2.7.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 06/10] lightnvm: pblk: use vmalloc for GC data buffer
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
` (4 preceding siblings ...)
2017-06-30 15:56 ` [PATCH 05/10] lightnvm: pblk: use right metadata buffer for recovery Javier González
@ 2017-06-30 15:56 ` Javier González
2017-06-30 15:56 ` [PATCH 07/10] lightnvm: pblk: remove target using async. I/Os Javier González
` (4 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Javier González @ 2017-06-30 15:56 UTC (permalink / raw)
To: mb, axboe
Cc: linux-block, linux-kernel, Javier González, Matias Bjørling
For now, we allocate a per I/O buffer for GC data. Since the potential
size of the buffer is 256KB and GC is not in the fast path, do this
allocation with vmalloc. This puts lets pressure on the memory
allocator at no performance cost.
Signed-off-by: Javier González <javier@cnexlabs.com>
Signed-off-by: Matias Bjørling <matias@cnexlabs.com>
---
drivers/lightnvm/pblk-core.c | 9 +++++----
drivers/lightnvm/pblk-gc.c | 6 +++---
drivers/lightnvm/pblk-read.c | 4 ++--
drivers/lightnvm/pblk-write.c | 3 ++-
drivers/lightnvm/pblk.h | 4 ++--
5 files changed, 14 insertions(+), 12 deletions(-)
diff --git a/drivers/lightnvm/pblk-core.c b/drivers/lightnvm/pblk-core.c
index a654b34f6f86..74b8d9db05e1 100644
--- a/drivers/lightnvm/pblk-core.c
+++ b/drivers/lightnvm/pblk-core.c
@@ -425,16 +425,15 @@ int pblk_submit_io(struct pblk *pblk, struct nvm_rq *rqd)
struct bio *pblk_bio_map_addr(struct pblk *pblk, void *data,
unsigned int nr_secs, unsigned int len,
- gfp_t gfp_mask)
+ int alloc_type, gfp_t gfp_mask)
{
struct nvm_tgt_dev *dev = pblk->dev;
- struct pblk_line_mgmt *l_mg = &pblk->l_mg;
void *kaddr = data;
struct page *page;
struct bio *bio;
int i, ret;
- if (l_mg->emeta_alloc_type == PBLK_KMALLOC_META)
+ if (alloc_type == PBLK_KMALLOC_META)
return bio_map_kern(dev->q, kaddr, len, gfp_mask);
bio = bio_kmalloc(gfp_mask, nr_secs);
@@ -552,6 +551,7 @@ static int pblk_line_submit_emeta_io(struct pblk *pblk, struct pblk_line *line,
{
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
struct pblk_line_meta *lm = &pblk->lm;
void *ppa_list, *meta_list;
struct bio *bio;
@@ -589,7 +589,8 @@ static int pblk_line_submit_emeta_io(struct pblk *pblk, struct pblk_line *line,
rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
rq_len = rq_ppas * geo->sec_size;
- bio = pblk_bio_map_addr(pblk, emeta_buf, rq_ppas, rq_len, GFP_KERNEL);
+ bio = pblk_bio_map_addr(pblk, emeta_buf, rq_ppas, rq_len,
+ l_mg->emeta_alloc_type, GFP_KERNEL);
if (IS_ERR(bio)) {
ret = PTR_ERR(bio);
goto free_rqd_dma;
diff --git a/drivers/lightnvm/pblk-gc.c b/drivers/lightnvm/pblk-gc.c
index 9b4059b93855..6090d28f7995 100644
--- a/drivers/lightnvm/pblk-gc.c
+++ b/drivers/lightnvm/pblk-gc.c
@@ -20,7 +20,7 @@
static void pblk_gc_free_gc_rq(struct pblk_gc_rq *gc_rq)
{
- kfree(gc_rq->data);
+ vfree(gc_rq->data);
kfree(gc_rq);
}
@@ -72,7 +72,7 @@ static int pblk_gc_move_valid_secs(struct pblk *pblk, struct pblk_gc_rq *gc_rq)
unsigned int secs_to_gc;
int ret = 0;
- data = kmalloc(gc_rq->nr_secs * geo->sec_size, GFP_KERNEL);
+ data = vmalloc(gc_rq->nr_secs * geo->sec_size);
if (!data) {
ret = -ENOMEM;
goto out;
@@ -110,7 +110,7 @@ static int pblk_gc_move_valid_secs(struct pblk *pblk, struct pblk_gc_rq *gc_rq)
free_rq:
kfree(gc_rq);
free_data:
- kfree(data);
+ vfree(data);
out:
kref_put(&line->ref, pblk_line_put);
return ret;
diff --git a/drivers/lightnvm/pblk-read.c b/drivers/lightnvm/pblk-read.c
index ed2ea01a0a38..31d4869b0500 100644
--- a/drivers/lightnvm/pblk-read.c
+++ b/drivers/lightnvm/pblk-read.c
@@ -462,7 +462,6 @@ int pblk_submit_read_gc(struct pblk *pblk, u64 *lba_list, void *data,
{
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
- struct request_queue *q = dev->q;
struct bio *bio;
struct nvm_rq rqd;
int ret, data_len;
@@ -491,7 +490,8 @@ int pblk_submit_read_gc(struct pblk *pblk, u64 *lba_list, void *data,
goto out;
data_len = (*secs_to_gc) * geo->sec_size;
- bio = bio_map_kern(q, data, data_len, GFP_KERNEL);
+ bio = pblk_bio_map_addr(pblk, data, *secs_to_gc, data_len,
+ PBLK_KMALLOC_META, GFP_KERNEL);
if (IS_ERR(bio)) {
pr_err("pblk: could not allocate GC bio (%lu)\n", PTR_ERR(bio));
goto err_free_dma;
diff --git a/drivers/lightnvm/pblk-write.c b/drivers/lightnvm/pblk-write.c
index 3e0b84937b90..8151bf4bb945 100644
--- a/drivers/lightnvm/pblk-write.c
+++ b/drivers/lightnvm/pblk-write.c
@@ -389,7 +389,8 @@ int pblk_submit_meta_io(struct pblk *pblk, struct pblk_line *meta_line)
rq_len = rq_ppas * geo->sec_size;
data = ((void *)emeta->buf) + emeta->mem;
- bio = pblk_bio_map_addr(pblk, data, rq_ppas, rq_len, GFP_KERNEL);
+ bio = pblk_bio_map_addr(pblk, data, rq_ppas, rq_len,
+ l_mg->emeta_alloc_type, GFP_KERNEL);
if (IS_ERR(bio)) {
ret = PTR_ERR(bio);
goto fail_free_rqd;
diff --git a/drivers/lightnvm/pblk.h b/drivers/lightnvm/pblk.h
index 36c5f5999324..cdad2c9edbdf 100644
--- a/drivers/lightnvm/pblk.h
+++ b/drivers/lightnvm/pblk.h
@@ -698,7 +698,7 @@ int pblk_submit_io(struct pblk *pblk, struct nvm_rq *rqd);
int pblk_submit_meta_io(struct pblk *pblk, struct pblk_line *meta_line);
struct bio *pblk_bio_map_addr(struct pblk *pblk, void *data,
unsigned int nr_secs, unsigned int len,
- gfp_t gfp_mask);
+ int alloc_type, gfp_t gfp_mask);
struct pblk_line *pblk_line_get(struct pblk *pblk);
struct pblk_line *pblk_line_get_first_data(struct pblk *pblk);
void pblk_line_replace_data(struct pblk *pblk);
@@ -805,7 +805,7 @@ int pblk_recov_setup_rq(struct pblk *pblk, struct pblk_c_ctx *c_ctx,
* pblk gc
*/
#define PBLK_GC_MAX_READERS 8 /* Max number of outstanding GC reader jobs */
-#define PBLK_GC_W_QD 1024 /* Queue depth for inflight GC write I/Os */
+#define PBLK_GC_W_QD 128 /* Queue depth for inflight GC write I/Os */
#define PBLK_GC_L_QD 4 /* Queue depth for inflight GC lines */
#define PBLK_GC_RSV_LINE 1 /* Reserved lines for GC */
--
2.7.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 07/10] lightnvm: pblk: remove target using async. I/Os
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
` (5 preceding siblings ...)
2017-06-30 15:56 ` [PATCH 06/10] lightnvm: pblk: use vmalloc for GC data buffer Javier González
@ 2017-06-30 15:56 ` Javier González
2017-06-30 15:56 ` [PATCH 08/10] lightnvm: pblk: add initialization check Javier González
` (3 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Javier González @ 2017-06-30 15:56 UTC (permalink / raw)
To: mb, axboe
Cc: linux-block, linux-kernel, Javier González, Matias Bjørling
When removing a pblk instance, pad the current line using asynchronous
I/O. This reduces the removal time from ~1 minute in the worst case to a
couple of seconds.
Signed-off-by: Javier González <javier@cnexlabs.com>
Signed-off-by: Matias Bjørling <matias@cnexlabs.com>
---
drivers/lightnvm/pblk-core.c | 5 +-
drivers/lightnvm/pblk-init.c | 9 +++
drivers/lightnvm/pblk-rb.c | 8 ++
drivers/lightnvm/pblk-recovery.c | 165 ++++++++++++++++++++++-----------------
drivers/lightnvm/pblk-write.c | 2 +-
drivers/lightnvm/pblk.h | 8 ++
6 files changed, 123 insertions(+), 74 deletions(-)
diff --git a/drivers/lightnvm/pblk-core.c b/drivers/lightnvm/pblk-core.c
index 74b8d9db05e1..e6f42cddc8ec 100644
--- a/drivers/lightnvm/pblk-core.c
+++ b/drivers/lightnvm/pblk-core.c
@@ -273,9 +273,10 @@ static void pblk_flush_writer(struct pblk *pblk)
{
pblk_rb_flush(&pblk->rwb);
do {
- if (!pblk_rb_read_count(&pblk->rwb))
+ if (!pblk_rb_sync_count(&pblk->rwb))
break;
+ pblk_write_kick(pblk);
schedule();
} while (1);
}
@@ -1350,6 +1351,7 @@ void pblk_pipeline_stop(struct pblk *pblk)
return;
}
+ flush_workqueue(pblk->bb_wq);
pblk_line_close_meta_sync(pblk);
spin_lock(&l_mg->free_lock);
@@ -1547,6 +1549,7 @@ void pblk_line_close_meta_sync(struct pblk *pblk)
}
pblk_wait_for_meta(pblk);
+ flush_workqueue(pblk->close_wq);
}
static void pblk_line_should_sync_meta(struct pblk *pblk)
diff --git a/drivers/lightnvm/pblk-init.c b/drivers/lightnvm/pblk-init.c
index b3fc310aa51c..025d8fe52154 100644
--- a/drivers/lightnvm/pblk-init.c
+++ b/drivers/lightnvm/pblk-init.c
@@ -841,6 +841,15 @@ static int pblk_writer_init(struct pblk *pblk)
static void pblk_writer_stop(struct pblk *pblk)
{
+ /* The pipeline must be stopped and the write buffer emptied before the
+ * write thread is stopped
+ */
+ WARN(pblk_rb_read_count(&pblk->rwb),
+ "Stopping not fully persisted write buffer\n");
+
+ WARN(pblk_rb_sync_count(&pblk->rwb),
+ "Stopping not fully synced write buffer\n");
+
if (pblk->writer_ts)
kthread_stop(pblk->writer_ts);
del_timer(&pblk->wtimer);
diff --git a/drivers/lightnvm/pblk-rb.c b/drivers/lightnvm/pblk-rb.c
index 2dda874af890..7300be98e831 100644
--- a/drivers/lightnvm/pblk-rb.c
+++ b/drivers/lightnvm/pblk-rb.c
@@ -180,6 +180,14 @@ unsigned int pblk_rb_read_count(struct pblk_rb *rb)
return pblk_rb_ring_count(mem, subm, rb->nr_entries);
}
+unsigned int pblk_rb_sync_count(struct pblk_rb *rb)
+{
+ unsigned int mem = READ_ONCE(rb->mem);
+ unsigned int sync = READ_ONCE(rb->sync);
+
+ return pblk_rb_ring_count(mem, sync, rb->nr_entries);
+}
+
unsigned int pblk_rb_read_commit(struct pblk_rb *rb, unsigned int nr_entries)
{
unsigned int subm;
diff --git a/drivers/lightnvm/pblk-recovery.c b/drivers/lightnvm/pblk-recovery.c
index 6d58659fa3da..0e48d3e4e143 100644
--- a/drivers/lightnvm/pblk-recovery.c
+++ b/drivers/lightnvm/pblk-recovery.c
@@ -327,47 +327,94 @@ static int pblk_recov_read_oob(struct pblk *pblk, struct pblk_line *line,
return 0;
}
+static void pblk_recov_complete(struct kref *ref)
+{
+ struct pblk_pad_rq *pad_rq = container_of(ref, struct pblk_pad_rq, ref);
+
+ complete(&pad_rq->wait);
+}
+
+static void pblk_end_io_recov(struct nvm_rq *rqd)
+{
+ struct pblk_pad_rq *pad_rq = rqd->private;
+ struct pblk *pblk = pad_rq->pblk;
+ struct nvm_tgt_dev *dev = pblk->dev;
+
+ kref_put(&pad_rq->ref, pblk_recov_complete);
+ nvm_dev_dma_free(dev->parent, rqd->meta_list, rqd->dma_meta_list);
+ pblk_free_rqd(pblk, rqd, WRITE);
+}
+
static int pblk_recov_pad_oob(struct pblk *pblk, struct pblk_line *line,
- struct pblk_recov_alloc p, int left_ppas)
+ int left_ppas)
{
struct nvm_tgt_dev *dev = pblk->dev;
struct nvm_geo *geo = &dev->geo;
struct ppa_addr *ppa_list;
struct pblk_sec_meta *meta_list;
+ struct pblk_pad_rq *pad_rq;
struct nvm_rq *rqd;
struct bio *bio;
void *data;
dma_addr_t dma_ppa_list, dma_meta_list;
__le64 *lba_list = emeta_to_lbas(pblk, line->emeta->buf);
u64 w_ptr = line->cur_sec;
- int left_line_ppas = line->left_msecs;
- int rq_ppas, rq_len;
+ int left_line_ppas, rq_ppas, rq_len;
int i, j;
int ret = 0;
- DECLARE_COMPLETION_ONSTACK(wait);
-
- ppa_list = p.ppa_list;
- meta_list = p.meta_list;
- rqd = p.rqd;
- data = p.data;
- dma_ppa_list = p.dma_ppa_list;
- dma_meta_list = p.dma_meta_list;
+
+ spin_lock(&line->lock);
+ left_line_ppas = line->left_msecs;
+ spin_unlock(&line->lock);
+
+ pad_rq = kmalloc(sizeof(struct pblk_pad_rq), GFP_KERNEL);
+ if (!pad_rq)
+ return -ENOMEM;
+
+ data = vzalloc(pblk->max_write_pgs * geo->sec_size);
+ if (!data) {
+ ret = -ENOMEM;
+ goto free_rq;
+ }
+
+ pad_rq->pblk = pblk;
+ init_completion(&pad_rq->wait);
+ kref_init(&pad_rq->ref);
next_pad_rq:
rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
- if (!rq_ppas)
- rq_ppas = pblk->min_write_pgs;
+ if (rq_ppas < pblk->min_write_pgs) {
+ pr_err("pblk: corrupted pad line %d\n", line->id);
+ goto free_rq;
+ }
+
rq_len = rq_ppas * geo->sec_size;
+ meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL, &dma_meta_list);
+ if (!meta_list) {
+ ret = -ENOMEM;
+ goto free_data;
+ }
+
+ ppa_list = (void *)(meta_list) + pblk_dma_meta_size;
+ dma_ppa_list = dma_meta_list + pblk_dma_meta_size;
+
+ rqd = pblk_alloc_rqd(pblk, WRITE);
+ if (IS_ERR(rqd)) {
+ ret = PTR_ERR(rqd);
+ goto fail_free_meta;
+ }
+ memset(rqd, 0, pblk_w_rq_size);
+
bio = bio_map_kern(dev->q, data, rq_len, GFP_KERNEL);
- if (IS_ERR(bio))
- return PTR_ERR(bio);
+ if (IS_ERR(bio)) {
+ ret = PTR_ERR(bio);
+ goto fail_free_rqd;
+ }
bio->bi_iter.bi_sector = 0; /* internal bio */
bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
- memset(rqd, 0, pblk_g_rq_size);
-
rqd->bio = bio;
rqd->opcode = NVM_OP_PWRITE;
rqd->flags = pblk_set_progr_mode(pblk, WRITE);
@@ -376,8 +423,8 @@ static int pblk_recov_pad_oob(struct pblk *pblk, struct pblk_line *line,
rqd->ppa_list = ppa_list;
rqd->dma_ppa_list = dma_ppa_list;
rqd->dma_meta_list = dma_meta_list;
- rqd->end_io = pblk_end_io_sync;
- rqd->private = &wait;
+ rqd->end_io = pblk_end_io_recov;
+ rqd->private = pad_rq;
for (i = 0; i < rqd->nr_ppas; ) {
struct ppa_addr ppa;
@@ -405,25 +452,41 @@ static int pblk_recov_pad_oob(struct pblk *pblk, struct pblk_line *line,
}
}
+ kref_get(&pad_rq->ref);
+
ret = pblk_submit_io(pblk, rqd);
if (ret) {
pr_err("pblk: I/O submission failed: %d\n", ret);
- return ret;
+ goto free_data;
}
- if (!wait_for_completion_io_timeout(&wait,
- msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
- pr_err("pblk: L2P recovery write timed out\n");
- }
atomic_dec(&pblk->inflight_io);
- reinit_completion(&wait);
left_line_ppas -= rq_ppas;
left_ppas -= rq_ppas;
- if (left_ppas > 0 && left_line_ppas)
+ if (left_ppas && left_line_ppas)
goto next_pad_rq;
- return 0;
+ kref_put(&pad_rq->ref, pblk_recov_complete);
+
+ if (!wait_for_completion_io_timeout(&pad_rq->wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: pad write timed out\n");
+ ret = -ETIME;
+ }
+
+free_rq:
+ kfree(pad_rq);
+free_data:
+ vfree(data);
+ return ret;
+
+fail_free_rqd:
+ pblk_free_rqd(pblk, rqd, WRITE);
+fail_free_meta:
+ nvm_dev_dma_free(dev->parent, meta_list, dma_meta_list);
+ kfree(pad_rq);
+ return ret;
}
/* When this function is called, it means that not all upper pages have been
@@ -555,7 +618,7 @@ static int pblk_recov_scan_all_oob(struct pblk *pblk, struct pblk_line *line,
if (pad_secs > line->left_msecs)
pad_secs = line->left_msecs;
- ret = pblk_recov_pad_oob(pblk, line, p, pad_secs);
+ ret = pblk_recov_pad_oob(pblk, line, pad_secs);
if (ret)
pr_err("pblk: OOB padding failed (err:%d)\n", ret);
@@ -961,64 +1024,22 @@ struct pblk_line *pblk_recov_l2p(struct pblk *pblk)
*/
int pblk_recov_pad(struct pblk *pblk)
{
- struct nvm_tgt_dev *dev = pblk->dev;
- struct nvm_geo *geo = &dev->geo;
struct pblk_line *line;
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
- struct nvm_rq *rqd;
- struct pblk_recov_alloc p;
- struct ppa_addr *ppa_list;
- struct pblk_sec_meta *meta_list;
- void *data;
int left_msecs;
int ret = 0;
- dma_addr_t dma_ppa_list, dma_meta_list;
spin_lock(&l_mg->free_lock);
line = l_mg->data_line;
left_msecs = line->left_msecs;
spin_unlock(&l_mg->free_lock);
- rqd = pblk_alloc_rqd(pblk, READ);
- if (IS_ERR(rqd))
- return PTR_ERR(rqd);
-
- meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL, &dma_meta_list);
- if (!meta_list) {
- ret = -ENOMEM;
- goto free_rqd;
- }
-
- ppa_list = (void *)(meta_list) + pblk_dma_meta_size;
- dma_ppa_list = dma_meta_list + pblk_dma_meta_size;
-
- data = kcalloc(pblk->max_write_pgs, geo->sec_size, GFP_KERNEL);
- if (!data) {
- ret = -ENOMEM;
- goto free_meta_list;
- }
-
- p.ppa_list = ppa_list;
- p.meta_list = meta_list;
- p.rqd = rqd;
- p.data = data;
- p.dma_ppa_list = dma_ppa_list;
- p.dma_meta_list = dma_meta_list;
-
- ret = pblk_recov_pad_oob(pblk, line, p, left_msecs);
+ ret = pblk_recov_pad_oob(pblk, line, left_msecs);
if (ret) {
pr_err("pblk: Tear down padding failed (%d)\n", ret);
- goto free_data;
+ return ret;
}
pblk_line_close_meta(pblk, line);
-
-free_data:
- kfree(data);
-free_meta_list:
- nvm_dev_dma_free(dev->parent, meta_list, dma_meta_list);
-free_rqd:
- pblk_free_rqd(pblk, rqd, READ);
-
return ret;
}
diff --git a/drivers/lightnvm/pblk-write.c b/drivers/lightnvm/pblk-write.c
index 8151bf4bb945..d62a8f4faaf4 100644
--- a/drivers/lightnvm/pblk-write.c
+++ b/drivers/lightnvm/pblk-write.c
@@ -190,7 +190,7 @@ static void pblk_end_io_write_meta(struct nvm_rq *rqd)
if (rqd->error) {
pblk_log_write_err(pblk, rqd);
- pr_err("pblk: metadata I/O failed\n");
+ pr_err("pblk: metadata I/O failed. Line %d\n", line->id);
}
#ifdef CONFIG_NVM_DEBUG
else
diff --git a/drivers/lightnvm/pblk.h b/drivers/lightnvm/pblk.h
index cdad2c9edbdf..bf5b73fb345f 100644
--- a/drivers/lightnvm/pblk.h
+++ b/drivers/lightnvm/pblk.h
@@ -111,6 +111,13 @@ struct pblk_g_ctx {
void *private;
};
+/* Pad context */
+struct pblk_pad_rq {
+ struct pblk *pblk;
+ struct completion wait;
+ struct kref ref;
+};
+
/* Recovery context */
struct pblk_rec_ctx {
struct pblk *pblk;
@@ -674,6 +681,7 @@ void pblk_rb_sync_end(struct pblk_rb *rb, unsigned long *flags);
unsigned int pblk_rb_sync_point_count(struct pblk_rb *rb);
unsigned int pblk_rb_read_count(struct pblk_rb *rb);
+unsigned int pblk_rb_sync_count(struct pblk_rb *rb);
unsigned int pblk_rb_wrap_pos(struct pblk_rb *rb, unsigned int pos);
int pblk_rb_tear_down_check(struct pblk_rb *rb);
--
2.7.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 08/10] lightnvm: pblk: add initialization check
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
` (6 preceding siblings ...)
2017-06-30 15:56 ` [PATCH 07/10] lightnvm: pblk: remove target using async. I/Os Javier González
@ 2017-06-30 15:56 ` Javier González
2017-06-30 15:56 ` [PATCH 09/10] lightnvm: pblk: verify that cache read is still valid Javier González
` (2 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Javier González @ 2017-06-30 15:56 UTC (permalink / raw)
To: mb, axboe
Cc: linux-block, linux-kernel, Javier González, Matias Bjørling
Add a sanity check to the pblk initialization sequence in order to
ensure that enough LUNs have been allocated to store the line metadata.
Signed-off-by: Javier González <javier@cnexlabs.com>
Signed-off-by: Matias Bjørling <matias@cnexlabs.com>
---
drivers/lightnvm/pblk-init.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/lightnvm/pblk-init.c b/drivers/lightnvm/pblk-init.c
index 025d8fe52154..1b0f61233c21 100644
--- a/drivers/lightnvm/pblk-init.c
+++ b/drivers/lightnvm/pblk-init.c
@@ -716,6 +716,12 @@ static int pblk_lines_init(struct pblk *pblk)
lm->emeta_bb = geo->nr_luns - i;
lm->min_blk_line = 1 + DIV_ROUND_UP(lm->smeta_sec + lm->emeta_sec[0],
geo->sec_per_blk);
+ if (lm->min_blk_line > lm->blk_per_line) {
+ pr_err("pblk: config. not supported. Min. LUN in line:%d\n",
+ lm->blk_per_line);
+ ret = -EINVAL;
+ goto fail;
+ }
ret = pblk_lines_alloc_metadata(pblk);
if (ret)
--
2.7.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 09/10] lightnvm: pblk: verify that cache read is still valid
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
` (7 preceding siblings ...)
2017-06-30 15:56 ` [PATCH 08/10] lightnvm: pblk: add initialization check Javier González
@ 2017-06-30 15:56 ` Javier González
2017-06-30 15:56 ` [PATCH 10/10] lightnvm: pblk: set line bitmap check under debug Javier González
2017-06-30 17:08 ` [PATCH 00/10] lightnvm: pblk fixes for 4.13 Jens Axboe
10 siblings, 0 replies; 12+ messages in thread
From: Javier González @ 2017-06-30 15:56 UTC (permalink / raw)
To: mb, axboe
Cc: linux-block, linux-kernel, Javier González, Matias Bjørling
When a read is directed to the cache, we risk that the lba has been
updated during the time we made the L2P table lookup and the time we are
actually reading form the cache. We intentionally not hold the L2P lock
not to block other threads.
While strict ordering is not a guarantee at this level (unless REQ_FLUSH
has been previously issued), we have experience that some databases that
have recently implemented direct I/O support, issue metadata reads very
close to the writes, without issuing a fsync in the middle. An easy way
to support them while they is to make an extra effort and check the L2P
map right before reading the cache.
Signed-off-by: Javier González <javier@cnexlabs.com>
Signed-off-by: Matias Bjørling <matias@cnexlabs.com>
---
drivers/lightnvm/pblk-rb.c | 15 ++++++++++++---
drivers/lightnvm/pblk-read.c | 3 +--
drivers/lightnvm/pblk.h | 10 +++++++++-
3 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/drivers/lightnvm/pblk-rb.c b/drivers/lightnvm/pblk-rb.c
index 7300be98e831..5ecc154f6831 100644
--- a/drivers/lightnvm/pblk-rb.c
+++ b/drivers/lightnvm/pblk-rb.c
@@ -150,6 +150,7 @@ static void clean_wctx(struct pblk_w_ctx *w_ctx)
/* Release flags on context. Protect from writes and reads */
smp_store_release(&w_ctx->flags, PBLK_WRITABLE_ENTRY);
pblk_ppa_set_empty(&w_ctx->ppa);
+ w_ctx->lba = ADDR_EMPTY;
}
#define pblk_rb_ring_count(head, tail, size) CIRC_CNT(head, tail, size)
@@ -656,15 +657,17 @@ unsigned int pblk_rb_read_to_bio(struct pblk_rb *rb, struct nvm_rq *rqd,
* be directed to disk.
*/
int pblk_rb_copy_to_bio(struct pblk_rb *rb, struct bio *bio, sector_t lba,
- u64 pos, int bio_iter)
+ struct ppa_addr ppa, int bio_iter)
{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
struct pblk_rb_entry *entry;
struct pblk_w_ctx *w_ctx;
+ struct ppa_addr l2p_ppa;
+ u64 pos = pblk_addr_to_cacheline(ppa);
void *data;
int flags;
int ret = 1;
- spin_lock(&rb->w_lock);
#ifdef CONFIG_NVM_DEBUG
/* Caller must ensure that the access will not cause an overflow */
@@ -674,8 +677,14 @@ int pblk_rb_copy_to_bio(struct pblk_rb *rb, struct bio *bio, sector_t lba,
w_ctx = &entry->w_ctx;
flags = READ_ONCE(w_ctx->flags);
+ spin_lock(&rb->w_lock);
+ spin_lock(&pblk->trans_lock);
+ l2p_ppa = pblk_trans_map_get(pblk, lba);
+ spin_unlock(&pblk->trans_lock);
+
/* Check if the entry has been overwritten or is scheduled to be */
- if (w_ctx->lba != lba || flags & PBLK_WRITABLE_ENTRY) {
+ if (!pblk_ppa_comp(l2p_ppa, ppa) || w_ctx->lba != lba ||
+ flags & PBLK_WRITABLE_ENTRY) {
ret = 0;
goto out;
}
diff --git a/drivers/lightnvm/pblk-read.c b/drivers/lightnvm/pblk-read.c
index 31d4869b0500..4e5c48f3de62 100644
--- a/drivers/lightnvm/pblk-read.c
+++ b/drivers/lightnvm/pblk-read.c
@@ -34,8 +34,7 @@ static int pblk_read_from_cache(struct pblk *pblk, struct bio *bio,
BUG_ON(!pblk_addr_in_cache(ppa));
#endif
- return pblk_rb_copy_to_bio(&pblk->rwb, bio, lba,
- pblk_addr_to_cacheline(ppa), bio_iter);
+ return pblk_rb_copy_to_bio(&pblk->rwb, bio, lba, ppa, bio_iter);
}
static void pblk_read_ppalist_rq(struct pblk *pblk, struct nvm_rq *rqd,
diff --git a/drivers/lightnvm/pblk.h b/drivers/lightnvm/pblk.h
index bf5b73fb345f..15931381348c 100644
--- a/drivers/lightnvm/pblk.h
+++ b/drivers/lightnvm/pblk.h
@@ -670,7 +670,7 @@ unsigned int pblk_rb_read_to_bio_list(struct pblk_rb *rb, struct bio *bio,
struct list_head *list,
unsigned int max);
int pblk_rb_copy_to_bio(struct pblk_rb *rb, struct bio *bio, sector_t lba,
- u64 pos, int bio_iter);
+ struct ppa_addr ppa, int bio_iter);
unsigned int pblk_rb_read_commit(struct pblk_rb *rb, unsigned int entries);
unsigned int pblk_rb_sync_init(struct pblk_rb *rb, unsigned long *flags);
@@ -1037,6 +1037,14 @@ static inline void pblk_ppa_set_empty(struct ppa_addr *ppa_addr)
ppa_addr->ppa = ADDR_EMPTY;
}
+static inline bool pblk_ppa_comp(struct ppa_addr lppa, struct ppa_addr rppa)
+{
+ if (lppa.ppa == rppa.ppa)
+ return true;
+
+ return false;
+}
+
static inline int pblk_addr_in_cache(struct ppa_addr ppa)
{
return (ppa.ppa != ADDR_EMPTY && ppa.c.is_cached);
--
2.7.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH 10/10] lightnvm: pblk: set line bitmap check under debug
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
` (8 preceding siblings ...)
2017-06-30 15:56 ` [PATCH 09/10] lightnvm: pblk: verify that cache read is still valid Javier González
@ 2017-06-30 15:56 ` Javier González
2017-06-30 17:08 ` [PATCH 00/10] lightnvm: pblk fixes for 4.13 Jens Axboe
10 siblings, 0 replies; 12+ messages in thread
From: Javier González @ 2017-06-30 15:56 UTC (permalink / raw)
To: mb, axboe
Cc: linux-block, linux-kernel, Javier González, Matias Bjørling
Do bitmap checks only when debug mode is enable. The line bitmap used
for mapping to physical addresses is fairly large (~512KB) and it is
expensive to do this checks on the fast path.
Signed-off-by: Javier González <javier@cnexlabs.com>
Signed-off-by: Matias Bjørling <matias@cnexlabs.com>
---
drivers/lightnvm/pblk-core.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/lightnvm/pblk-core.c b/drivers/lightnvm/pblk-core.c
index e6f42cddc8ec..11fe0c5b2a9c 100644
--- a/drivers/lightnvm/pblk-core.c
+++ b/drivers/lightnvm/pblk-core.c
@@ -1561,11 +1561,14 @@ static void pblk_line_should_sync_meta(struct pblk *pblk)
void pblk_line_close(struct pblk *pblk, struct pblk_line *line)
{
struct pblk_line_mgmt *l_mg = &pblk->l_mg;
- struct pblk_line_meta *lm = &pblk->lm;
struct list_head *move_list;
+#ifdef CONFIG_NVM_DEBUG
+ struct pblk_line_meta *lm = &pblk->lm;
+
WARN(!bitmap_full(line->map_bitmap, lm->sec_per_line),
"pblk: corrupt closed line %d\n", line->id);
+#endif
spin_lock(&l_mg->free_lock);
WARN_ON(!test_and_clear_bit(line->meta_line, &l_mg->meta_bitmap));
--
2.7.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH 00/10] lightnvm: pblk fixes for 4.13
2017-06-30 15:56 [PATCH 00/10] lightnvm: pblk fixes for 4.13 Javier González
` (9 preceding siblings ...)
2017-06-30 15:56 ` [PATCH 10/10] lightnvm: pblk: set line bitmap check under debug Javier González
@ 2017-06-30 17:08 ` Jens Axboe
10 siblings, 0 replies; 12+ messages in thread
From: Jens Axboe @ 2017-06-30 17:08 UTC (permalink / raw)
To: Javier González, mb; +Cc: linux-block, linux-kernel, Javier González
On 06/30/2017 09:56 AM, Javier González wrote:
> Hi Jens,
>
> Here you have a second round of fixes for pblk. They are in essence bug
> fixes including a double-free reported by Dan.
>
> There is also regression fix for pblk removal, which was introduced with
> the new metadata scheduler. This fix makes that removing a pblk instance
> takes again at most 2 seconds.
>
> Finally, I also included a patch for double checking the L2P to verify a
> cache hit before reading from the write cache. We have experienced a
> database not issuing fsync before reading recently written metadata
> triggering a bad cache read. We verified it this week. This patch is an
> easy fix that we can make in pblk.
Applied for 4.13, thanks.
--
Jens Axboe
^ permalink raw reply [flat|nested] 12+ messages in thread