* [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device
@ 2019-05-17 8:42 Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 1/8] nvme: move device parameters to separate struct Klaus Birkelund Jensen
` (8 more replies)
0 siblings, 9 replies; 15+ messages in thread
From: Klaus Birkelund Jensen @ 2019-05-17 8:42 UTC (permalink / raw)
To: qemu-block; +Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
Hi,
This series of patches contains a number of refactorings to the emulated
nvme device, adds additional features, such as support for metadata and
scatter gather lists, and bumps the supported NVMe version to 1.3.
Lastly, it contains a new 'ocssd' device.
The motivation for the first seven patches is to set everything up for
the final patch that adds a new 'ocssd' device and associated block
driver that implements the OpenChannel 2.0 specification[1]. Many of us
in the OpenChannel comunity have used a qemu fork[2] for emulation of
OpenChannel devices. The fork is itself based on Keith's qemu-nvme
tree[3] and we recently merged mainline qemu into it, but the result is
still a "hybrid" nvme device that supports both conventional nvme and
the OCSSD 2.0 spec through a 'dialect' mechanism. Merging instead of
rebasing also created a pretty messy commit history and my efforts to
try and rebase our work onto mainline was getting hairy to say the
least. And I was never really happy with the dialect approach anyway.
I have instead prepared this series of fresh patches that incrementally
adds additional features to the nvme device to bring it into shape for
finally introducing a new (and separate) 'ocssd' device that emulates an
OpenChannel 2.0 device by reusing core functionality from the nvme
device. Providing a separate ocssd device ensures that no ocssd specific
stuff creeps into the nvme device.
The ocssd device is backed by a new 'ocssd' block driver that holds
internal meta data and keeps state permanent across power cycles. In the
future I think we could use the same approach for the nvme device to
keep internal metadata such as utilization and deallocated blocks. For
now, the nvme device does not support the Deallocated and Unwritten
Logical Block Error (DULBE) feature or the Data Set Management command
as this would require such support.
I have tried to make the patches to the nvme device in this series as
digestible as possible, but I undestand that commit 310fcd5965e5 ("nvme:
bump supported spec to 1.3") is pretty huge. I can try to chop it up if
required, but the changes pretty much needs to be done in bulk to
actually implement v1.3.
This version was recently used to find a bug in use of SGLs in the Linux
kernel, so I believe there is some value in introducing these new
features. As for the ocssd device I believe that it is time it is
included upstream and not kept seperately. I have knowledge of at least
one other qemu fork implementing OCSSD 2.0 used by the SPDK team and I
think we could all benefit from using a common implementation. The ocssd
device is feature complete with respect to the OCSSD 2.0 spec (mandatory
as well as optional features).
[1]: http://lightnvm.io/docs/OCSSD-2_0-20180129.pdf
[2]: https://github.com/OpenChannelSSD/qemu-nvme
[3]: http://git.infradead.org/users/kbusch/qemu-nvme.git
Klaus Birkelund Jensen (8):
nvme: move device parameters to separate struct
nvme: bump supported spec to 1.3
nvme: simplify PRP mappings
nvme: allow multiple i/o's per request
nvme: add support for metadata
nvme: add support for scatter gather lists
nvme: keep a copy of the NVMe command in request
nvme: add an OpenChannel 2.0 NVMe device (ocssd)
MAINTAINERS | 14 +-
Makefile.objs | 1 +
block.c | 2 +-
block/Makefile.objs | 2 +-
block/nvme.c | 20 +-
block/ocssd.c | 690 ++++++++++
hw/block/Makefile.objs | 2 +-
hw/block/nvme.c | 1405 -------------------
hw/block/nvme.h | 92 --
hw/block/nvme/nvme.c | 2485 +++++++++++++++++++++++++++++++++
hw/block/nvme/ocssd.c | 2647 ++++++++++++++++++++++++++++++++++++
hw/block/nvme/ocssd.h | 140 ++
hw/block/nvme/trace-events | 136 ++
hw/block/trace-events | 91 --
include/block/block_int.h | 3 +
include/block/nvme.h | 152 ++-
include/block/ocssd.h | 231 ++++
include/hw/block/nvme.h | 233 ++++
include/hw/pci/pci_ids.h | 2 +
qapi/block-core.json | 47 +-
20 files changed, 6774 insertions(+), 1621 deletions(-)
create mode 100644 block/ocssd.c
delete mode 100644 hw/block/nvme.c
delete mode 100644 hw/block/nvme.h
create mode 100644 hw/block/nvme/nvme.c
create mode 100644 hw/block/nvme/ocssd.c
create mode 100644 hw/block/nvme/ocssd.h
create mode 100644 hw/block/nvme/trace-events
create mode 100644 include/block/ocssd.h
create mode 100644 include/hw/block/nvme.h
--
2.21.0
^ permalink raw reply [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH 1/8] nvme: move device parameters to separate struct
2019-05-17 8:42 [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Klaus Birkelund Jensen
@ 2019-05-17 8:42 ` Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 2/8] nvme: bump supported spec to 1.3 Klaus Birkelund Jensen
` (7 subsequent siblings)
8 siblings, 0 replies; 15+ messages in thread
From: Klaus Birkelund Jensen @ 2019-05-17 8:42 UTC (permalink / raw)
To: qemu-block; +Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
Move device configuration parameters to separate struct to make it
explicit what is configurable and what is set internally.
Also, clean up some includes.
Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
---
hw/block/nvme.c | 53 +++++++++++++++++++++++--------------------------
hw/block/nvme.h | 16 ++++++++++++---
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index 7caf92532a09..b689c0776e72 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -27,17 +27,14 @@
#include "qemu/osdep.h"
#include "qemu/units.h"
+#include "qemu/cutils.h"
+#include "qemu/log.h"
#include "hw/block/block.h"
-#include "hw/hw.h"
#include "hw/pci/msix.h"
-#include "hw/pci/pci.h"
#include "sysemu/sysemu.h"
-#include "qapi/error.h"
-#include "qapi/visitor.h"
#include "sysemu/block-backend.h"
+#include "qapi/error.h"
-#include "qemu/log.h"
-#include "qemu/cutils.h"
#include "trace.h"
#include "nvme.h"
@@ -62,12 +59,12 @@ static void nvme_addr_read(NvmeCtrl *n, hwaddr addr, void *buf, int size)
static int nvme_check_sqid(NvmeCtrl *n, uint16_t sqid)
{
- return sqid < n->num_queues && n->sq[sqid] != NULL ? 0 : -1;
+ return sqid < n->params.num_queues && n->sq[sqid] != NULL ? 0 : -1;
}
static int nvme_check_cqid(NvmeCtrl *n, uint16_t cqid)
{
- return cqid < n->num_queues && n->cq[cqid] != NULL ? 0 : -1;
+ return cqid < n->params.num_queues && n->cq[cqid] != NULL ? 0 : -1;
}
static void nvme_inc_cq_tail(NvmeCQueue *cq)
@@ -605,7 +602,7 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeCmd *cmd)
trace_nvme_err_invalid_create_cq_addr(prp1);
return NVME_INVALID_FIELD | NVME_DNR;
}
- if (unlikely(vector > n->num_queues)) {
+ if (unlikely(vector > n->params.num_queues)) {
trace_nvme_err_invalid_create_cq_vector(vector);
return NVME_INVALID_IRQ_VECTOR | NVME_DNR;
}
@@ -707,7 +704,8 @@ static uint16_t nvme_get_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
trace_nvme_getfeat_vwcache(result ? "enabled" : "disabled");
break;
case NVME_NUMBER_OF_QUEUES:
- result = cpu_to_le32((n->num_queues - 2) | ((n->num_queues - 2) << 16));
+ result = cpu_to_le32((n->params.num_queues - 2) |
+ ((n->params.num_queues - 2) << 16));
trace_nvme_getfeat_numq(result);
break;
default:
@@ -731,9 +729,10 @@ static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
case NVME_NUMBER_OF_QUEUES:
trace_nvme_setfeat_numq((dw11 & 0xFFFF) + 1,
((dw11 >> 16) & 0xFFFF) + 1,
- n->num_queues - 1, n->num_queues - 1);
- req->cqe.result =
- cpu_to_le32((n->num_queues - 2) | ((n->num_queues - 2) << 16));
+ n->params.num_queues - 1,
+ n->params.num_queues - 1);
+ req->cqe.result = cpu_to_le32((n->params.num_queues - 2) |
+ ((n->params.num_queues - 2) << 16));
break;
default:
trace_nvme_err_invalid_setfeat(dw10);
@@ -802,12 +801,12 @@ static void nvme_clear_ctrl(NvmeCtrl *n)
blk_drain(n->conf.blk);
- for (i = 0; i < n->num_queues; i++) {
+ for (i = 0; i < n->params.num_queues; i++) {
if (n->sq[i] != NULL) {
nvme_free_sq(n->sq[i], n);
}
}
- for (i = 0; i < n->num_queues; i++) {
+ for (i = 0; i < n->params.num_queues; i++) {
if (n->cq[i] != NULL) {
nvme_free_cq(n->cq[i], n);
}
@@ -1208,7 +1207,7 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
int64_t bs_size;
uint8_t *pci_conf;
- if (!n->num_queues) {
+ if (!n->params.num_queues) {
error_setg(errp, "num_queues can't be zero");
return;
}
@@ -1224,7 +1223,7 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
return;
}
- if (!n->serial) {
+ if (!n->params.serial) {
error_setg(errp, "serial property not set");
return;
}
@@ -1241,25 +1240,25 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
pcie_endpoint_cap_init(pci_dev, 0x80);
n->num_namespaces = 1;
- n->reg_size = pow2ceil(0x1004 + 2 * (n->num_queues + 1) * 4);
+ n->reg_size = pow2ceil(0x1004 + 2 * (n->params.num_queues + 1) * 4);
n->ns_size = bs_size / (uint64_t)n->num_namespaces;
n->namespaces = g_new0(NvmeNamespace, n->num_namespaces);
- n->sq = g_new0(NvmeSQueue *, n->num_queues);
- n->cq = g_new0(NvmeCQueue *, n->num_queues);
+ n->sq = g_new0(NvmeSQueue *, n->params.num_queues);
+ n->cq = g_new0(NvmeCQueue *, n->params.num_queues);
memory_region_init_io(&n->iomem, OBJECT(n), &nvme_mmio_ops, n,
"nvme", n->reg_size);
pci_register_bar(pci_dev, 0,
PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64,
&n->iomem);
- msix_init_exclusive_bar(pci_dev, n->num_queues, 4, NULL);
+ msix_init_exclusive_bar(pci_dev, n->params.num_queues, 4, NULL);
id->vid = cpu_to_le16(pci_get_word(pci_conf + PCI_VENDOR_ID));
id->ssvid = cpu_to_le16(pci_get_word(pci_conf + PCI_SUBSYSTEM_VENDOR_ID));
strpadcpy((char *)id->mn, sizeof(id->mn), "QEMU NVMe Ctrl", ' ');
strpadcpy((char *)id->fr, sizeof(id->fr), "1.0", ' ');
- strpadcpy((char *)id->sn, sizeof(id->sn), n->serial, ' ');
+ strpadcpy((char *)id->sn, sizeof(id->sn), n->params.serial, ' ');
id->rab = 6;
id->ieee[0] = 0x00;
id->ieee[1] = 0x02;
@@ -1289,7 +1288,7 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
n->bar.vs = 0x00010200;
n->bar.intmc = n->bar.intms = 0;
- if (n->cmb_size_mb) {
+ if (n->params.cmb_size_mb) {
NVME_CMBLOC_SET_BIR(n->bar.cmbloc, 2);
NVME_CMBLOC_SET_OFST(n->bar.cmbloc, 0);
@@ -1300,7 +1299,7 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
NVME_CMBSZ_SET_RDS(n->bar.cmbsz, 1);
NVME_CMBSZ_SET_WDS(n->bar.cmbsz, 1);
NVME_CMBSZ_SET_SZU(n->bar.cmbsz, 2); /* MBs */
- NVME_CMBSZ_SET_SZ(n->bar.cmbsz, n->cmb_size_mb);
+ NVME_CMBSZ_SET_SZ(n->bar.cmbsz, n->params.cmb_size_mb);
n->cmbloc = n->bar.cmbloc;
n->cmbsz = n->bar.cmbsz;
@@ -1339,7 +1338,7 @@ static void nvme_exit(PCIDevice *pci_dev)
g_free(n->cq);
g_free(n->sq);
- if (n->cmb_size_mb) {
+ if (n->params.cmb_size_mb) {
g_free(n->cmbuf);
}
msix_uninit_exclusive_bar(pci_dev);
@@ -1347,9 +1346,7 @@ static void nvme_exit(PCIDevice *pci_dev)
static Property nvme_props[] = {
DEFINE_BLOCK_PROPERTIES(NvmeCtrl, conf),
- DEFINE_PROP_STRING("serial", NvmeCtrl, serial),
- DEFINE_PROP_UINT32("cmb_size_mb", NvmeCtrl, cmb_size_mb, 0),
- DEFINE_PROP_UINT32("num_queues", NvmeCtrl, num_queues, 64),
+ DEFINE_NVME_PROPERTIES(NvmeCtrl, params),
DEFINE_PROP_END_OF_LIST(),
};
diff --git a/hw/block/nvme.h b/hw/block/nvme.h
index 56c9d4b4b136..8866373058f6 100644
--- a/hw/block/nvme.h
+++ b/hw/block/nvme.h
@@ -1,7 +1,19 @@
#ifndef HW_NVME_H
#define HW_NVME_H
+
#include "block/nvme.h"
+#define DEFINE_NVME_PROPERTIES(_state, _props) \
+ DEFINE_PROP_STRING("serial", _state, _props.serial), \
+ DEFINE_PROP_UINT32("cmb_size_mb", _state, _props.cmb_size_mb, 0), \
+ DEFINE_PROP_UINT32("num_queues", _state, _props.num_queues, 64)
+
+typedef struct NvmeParams {
+ char *serial;
+ uint32_t num_queues;
+ uint32_t cmb_size_mb;
+} NvmeParams;
+
typedef struct NvmeAsyncEvent {
QSIMPLEQ_ENTRY(NvmeAsyncEvent) entry;
NvmeAerResult result;
@@ -63,6 +75,7 @@ typedef struct NvmeCtrl {
MemoryRegion ctrl_mem;
NvmeBar bar;
BlockConf conf;
+ NvmeParams params;
uint32_t page_size;
uint16_t page_bits;
@@ -71,16 +84,13 @@ typedef struct NvmeCtrl {
uint16_t sqe_size;
uint32_t reg_size;
uint32_t num_namespaces;
- uint32_t num_queues;
uint32_t max_q_ents;
uint64_t ns_size;
- uint32_t cmb_size_mb;
uint32_t cmbsz;
uint32_t cmbloc;
uint8_t *cmbuf;
uint64_t irq_status;
- char *serial;
NvmeNamespace *namespaces;
NvmeSQueue **sq;
NvmeCQueue **cq;
--
2.21.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH 2/8] nvme: bump supported spec to 1.3
2019-05-17 8:42 [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 1/8] nvme: move device parameters to separate struct Klaus Birkelund Jensen
@ 2019-05-17 8:42 ` Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 3/8] nvme: simplify PRP mappings Klaus Birkelund Jensen
` (6 subsequent siblings)
8 siblings, 0 replies; 15+ messages in thread
From: Klaus Birkelund Jensen @ 2019-05-17 8:42 UTC (permalink / raw)
To: qemu-block; +Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
Bump the supported NVMe version to 1.3. To do so, this patch adds a
number of missing 'Mandatory' features from the spec:
* Support for returning a Namespace Identification Descriptor List in
the Identify command (CNS 03h).
* Support for the Asynchronous Event Request command.
* Support for the Get Log Page command and the mandatory Error
Information, Smart / Health Information and Firmware Slot
Information log pages.
* Support for the Abort command.
As a side-effect, this bump also fixes support for multiple namespaces.
The implementation of AER, Get Log Page and Abort commands has been
imported and slightly modified from Keith's qemu-nvme tree[1]. Thanks!
[1]: http://git.infradead.org/users/kbusch/qemu-nvme.git
Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
---
hw/block/nvme.c | 792 ++++++++++++++++++++++++++++++++++++------
hw/block/nvme.h | 31 +-
hw/block/trace-events | 16 +-
include/block/nvme.h | 58 +++-
4 files changed, 783 insertions(+), 114 deletions(-)
diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index b689c0776e72..65dfc04f71e5 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -9,17 +9,35 @@
*/
/**
- * Reference Specs: http://www.nvmexpress.org, 1.2, 1.1, 1.0e
+ * Reference Specs: http://www.nvmexpress.org, 1.3d, 1.2, 1.1, 1.0e
*
* http://www.nvmexpress.org/resources/
*/
/**
* Usage: add options:
- * -drive file=<file>,if=none,id=<drive_id>
- * -device nvme,drive=<drive_id>,serial=<serial>,id=<id[optional]>, \
- * cmb_size_mb=<cmb_size_mb[optional]>, \
- * num_queues=<N[optional]>
+ * -drive file=<file>,if=none,id=<drive_id>
+ * -device nvme,drive=<drive_id>,serial=<serial>,id=<id[optional]>
+ *
+ * The "file" option must point to a path to a real file that you will use as
+ * the backing storage for your NVMe device. It must be a non-zero length, as
+ * this will be the disk image that your nvme controller will use to carve up
+ * namespaces for storage.
+ *
+ * Note the "drive" option's "id" name must match the "device nvme" drive's
+ * name to link the block device used for backing storage to the nvme
+ * interface.
+ *
+ * Advanced optional options:
+ *
+ * num_ns=<int> : Namespaces to make out of the backing storage,
+ * Default:1
+ * num_queues=<int> : Number of possible IO Queues, Default:64
+ * cmb_size_mb=<int> : Size of CMB in MBs, Default:0
+ *
+ * Parameters will be verified against conflicting capabilities and attributes
+ * and fail to load if there is a conflict or a configuration the emulated
+ * device is unable to handle.
*
* Note cmb_size_mb denotes size of CMB in MB. CMB is assumed to be at
* offset 0 in BAR2 and supports only WDS, RDS and SQS for now.
@@ -38,6 +56,12 @@
#include "trace.h"
#include "nvme.h"
+#define NVME_MAX_QS PCI_MSIX_FLAGS_QSIZE
+#define NVME_TEMPERATURE 0x143
+#define NVME_ELPE 3
+#define NVME_AERL 3
+#define NVME_OP_ABORTED 0xff
+
#define NVME_GUEST_ERR(trace, fmt, ...) \
do { \
(trace_##trace)(__VA_ARGS__); \
@@ -57,6 +81,16 @@ static void nvme_addr_read(NvmeCtrl *n, hwaddr addr, void *buf, int size)
}
}
+static void nvme_addr_write(NvmeCtrl *n, hwaddr addr, void *buf, int size)
+{
+ if (n->cmbsz && addr >= n->ctrl_mem.addr &&
+ addr < (n->ctrl_mem.addr + int128_get64(n->ctrl_mem.size))) {
+ memcpy((void *)&n->cmbuf[addr - n->ctrl_mem.addr], buf, size);
+ return;
+ }
+ pci_dma_write(&n->parent_obj, addr, buf, size);
+}
+
static int nvme_check_sqid(NvmeCtrl *n, uint16_t sqid)
{
return sqid < n->params.num_queues && n->sq[sqid] != NULL ? 0 : -1;
@@ -244,6 +278,24 @@ static uint16_t nvme_dma_read_prp(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
return status;
}
+static void nvme_post_cqe(NvmeCQueue *cq, NvmeRequest *req)
+{
+ NvmeCtrl *n = cq->ctrl;
+ NvmeSQueue *sq = req->sq;
+ NvmeCqe *cqe = &req->cqe;
+ uint8_t phase = cq->phase;
+ hwaddr addr;
+
+ addr = cq->dma_addr + cq->tail * n->cqe_size;
+ cqe->status = cpu_to_le16((req->status << 1) | phase);
+ cqe->sq_id = cpu_to_le16(sq->sqid);
+ cqe->sq_head = cpu_to_le16(sq->head);
+ nvme_addr_write(n, addr, (void *) cqe, sizeof(*cqe));
+ nvme_inc_cq_tail(cq);
+
+ QTAILQ_INSERT_TAIL(&sq->req_list, req, entry);
+}
+
static void nvme_post_cqes(void *opaque)
{
NvmeCQueue *cq = opaque;
@@ -251,24 +303,14 @@ static void nvme_post_cqes(void *opaque)
NvmeRequest *req, *next;
QTAILQ_FOREACH_SAFE(req, &cq->req_list, entry, next) {
- NvmeSQueue *sq;
- hwaddr addr;
-
if (nvme_cq_full(cq)) {
break;
}
QTAILQ_REMOVE(&cq->req_list, req, entry);
- sq = req->sq;
- req->cqe.status = cpu_to_le16((req->status << 1) | cq->phase);
- req->cqe.sq_id = cpu_to_le16(sq->sqid);
- req->cqe.sq_head = cpu_to_le16(sq->head);
- addr = cq->dma_addr + cq->tail * n->cqe_size;
- nvme_inc_cq_tail(cq);
- pci_dma_write(&n->parent_obj, addr, (void *)&req->cqe,
- sizeof(req->cqe));
- QTAILQ_INSERT_TAIL(&sq->req_list, req, entry);
+ nvme_post_cqe(cq, req);
}
+
if (cq->tail != cq->head) {
nvme_irq_assert(n, cq);
}
@@ -277,11 +319,88 @@ static void nvme_post_cqes(void *opaque)
static void nvme_enqueue_req_completion(NvmeCQueue *cq, NvmeRequest *req)
{
assert(cq->cqid == req->sq->cqid);
+
+ trace_nvme_enqueue_req_completion(req->cqe.cid, cq->cqid);
QTAILQ_REMOVE(&req->sq->out_req_list, req, entry);
QTAILQ_INSERT_TAIL(&cq->req_list, req, entry);
timer_mod(cq->timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + 500);
}
+static void nvme_enqueue_event(NvmeCtrl *n, uint8_t event_type,
+ uint8_t event_info, uint8_t log_page)
+{
+ NvmeAsyncEvent *event;
+
+ trace_nvme_enqueue_event(event_type, event_info, log_page);
+
+ /*
+ * Do not enqueue the event if something of this type is already queued.
+ * This bounds the size of the event queue and makes sure it does not grow
+ * indefinitely when events are not processed by the host (i.e. does not
+ * issue any AERs).
+ */
+ if (n->aer_mask_queued & (1 << event_type)) {
+ return;
+ }
+ n->aer_mask_queued |= (1 << event_type);
+
+ event = g_new(NvmeAsyncEvent, 1);
+ event->result = (NvmeAerResult) {
+ .event_type = event_type,
+ .event_info = event_info,
+ .log_page = log_page,
+ };
+
+ QSIMPLEQ_INSERT_TAIL(&n->aer_queue, event, entry);
+
+ timer_mod(n->aer_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + 500);
+}
+
+static void nvme_process_aers(void *opaque)
+{
+ NvmeCtrl *n = opaque;
+ NvmeRequest *req;
+ NvmeAerResult *result;
+ NvmeAsyncEvent *event, *next;
+
+ trace_nvme_process_aers();
+
+ QSIMPLEQ_FOREACH_SAFE(event, &n->aer_queue, entry, next) {
+ /* can't post cqe if there is nothing to complete */
+ if (!n->outstanding_aers) {
+ trace_nvme_no_outstanding_aers();
+ break;
+ }
+
+ /* ignore if masked (cqe posted, but event not cleared) */
+ if (n->aer_mask & (1 << event->result.event_type)) {
+ trace_nvme_aer_masked(event->result.event_type, n->aer_mask);
+ continue;
+ }
+
+ QSIMPLEQ_REMOVE_HEAD(&n->aer_queue, entry);
+
+ n->aer_mask |= 1 << event->result.event_type;
+ n->aer_mask_queued &= ~(1 << event->result.event_type);
+ n->outstanding_aers--;
+
+ req = n->aer_reqs[n->outstanding_aers];
+
+ result = (NvmeAerResult *) &req->cqe.result;
+ result->event_type = event->result.event_type;
+ result->event_info = event->result.event_info;
+ result->log_page = event->result.log_page;
+ g_free(event);
+
+ req->status = NVME_SUCCESS;
+
+ trace_nvme_aer_post_cqe(result->event_type, result->event_info,
+ result->log_page);
+
+ nvme_enqueue_req_completion(&n->admin_cq, req);
+ }
+}
+
static void nvme_rw_cb(void *opaque, int ret)
{
NvmeRequest *req = opaque;
@@ -318,7 +437,7 @@ static uint16_t nvme_write_zeros(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd,
{
NvmeRwCmd *rw = (NvmeRwCmd *)cmd;
const uint8_t lba_index = NVME_ID_NS_FLBAS_INDEX(ns->id_ns.flbas);
- const uint8_t data_shift = ns->id_ns.lbaf[lba_index].ds;
+ const uint8_t data_shift = ns->id_ns.lbaf[lba_index].lbads;
uint64_t slba = le64_to_cpu(rw->slba);
uint32_t nlb = le16_to_cpu(rw->nlb) + 1;
uint64_t offset = slba << data_shift;
@@ -347,9 +466,9 @@ static uint16_t nvme_rw(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd,
uint64_t prp2 = le64_to_cpu(rw->prp2);
uint8_t lba_index = NVME_ID_NS_FLBAS_INDEX(ns->id_ns.flbas);
- uint8_t data_shift = ns->id_ns.lbaf[lba_index].ds;
+ uint8_t data_shift = ns->id_ns.lbaf[lba_index].lbads;
uint64_t data_size = (uint64_t)nlb << data_shift;
- uint64_t data_offset = slba << data_shift;
+ uint64_t data_offset = ns->blk_offset + (slba << data_shift);
int is_write = rw->opcode == NVME_CMD_WRITE ? 1 : 0;
enum BlockAcctType acct = is_write ? BLOCK_ACCT_WRITE : BLOCK_ACCT_READ;
@@ -391,8 +510,8 @@ static uint16_t nvme_io_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
NvmeNamespace *ns;
uint32_t nsid = le32_to_cpu(cmd->nsid);
- if (unlikely(nsid == 0 || nsid > n->num_namespaces)) {
- trace_nvme_err_invalid_ns(nsid, n->num_namespaces);
+ if (unlikely(nsid == 0 || nsid > n->params.num_ns)) {
+ trace_nvme_err_invalid_ns(nsid, n->params.num_ns);
return NVME_INVALID_NSID | NVME_DNR;
}
@@ -420,6 +539,7 @@ static void nvme_free_sq(NvmeSQueue *sq, NvmeCtrl *n)
if (sq->sqid) {
g_free(sq);
}
+ n->qs_created--;
}
static uint16_t nvme_del_sq(NvmeCtrl *n, NvmeCmd *cmd)
@@ -486,6 +606,7 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr,
cq = n->cq[cqid];
QTAILQ_INSERT_TAIL(&(cq->sq_list), sq, entry);
n->sq[sqid] = sq;
+ n->qs_created++;
}
static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeCmd *cmd)
@@ -535,6 +656,7 @@ static void nvme_free_cq(NvmeCQueue *cq, NvmeCtrl *n)
if (cq->cqid) {
g_free(cq);
}
+ n->qs_created--;
}
static uint16_t nvme_del_cq(NvmeCtrl *n, NvmeCmd *cmd)
@@ -575,6 +697,7 @@ static void nvme_init_cq(NvmeCQueue *cq, NvmeCtrl *n, uint64_t dma_addr,
msix_vector_use(&n->parent_obj, cq->vector);
n->cq[cqid] = cq;
cq->timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, nvme_post_cqes, cq);
+ n->qs_created++;
}
static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeCmd *cmd)
@@ -637,8 +760,8 @@ static uint16_t nvme_identify_ns(NvmeCtrl *n, NvmeIdentify *c)
trace_nvme_identify_ns(nsid);
- if (unlikely(nsid == 0 || nsid > n->num_namespaces)) {
- trace_nvme_err_invalid_ns(nsid, n->num_namespaces);
+ if (unlikely(nsid == 0 || nsid > n->params.num_ns)) {
+ trace_nvme_err_invalid_ns(nsid, n->params.num_ns);
return NVME_INVALID_NSID | NVME_DNR;
}
@@ -648,7 +771,7 @@ static uint16_t nvme_identify_ns(NvmeCtrl *n, NvmeIdentify *c)
prp1, prp2);
}
-static uint16_t nvme_identify_nslist(NvmeCtrl *n, NvmeIdentify *c)
+static uint16_t nvme_identify_ns_list(NvmeCtrl *n, NvmeIdentify *c)
{
static const int data_len = 4 * KiB;
uint32_t min_nsid = le32_to_cpu(c->nsid);
@@ -658,10 +781,10 @@ static uint16_t nvme_identify_nslist(NvmeCtrl *n, NvmeIdentify *c)
uint16_t ret;
int i, j = 0;
- trace_nvme_identify_nslist(min_nsid);
+ trace_nvme_identify_ns_list(min_nsid);
list = g_malloc0(data_len);
- for (i = 0; i < n->num_namespaces; i++) {
+ for (i = 0; i < n->params.num_ns; i++) {
if (i < min_nsid) {
continue;
}
@@ -675,6 +798,25 @@ static uint16_t nvme_identify_nslist(NvmeCtrl *n, NvmeIdentify *c)
return ret;
}
+static uint16_t nvme_identify_ns_descriptor_list(NvmeCtrl *n, NvmeCmd *c)
+{
+ static const int data_len = sizeof(NvmeIdentifyNamespaceDescriptor) + 0x10;
+ uint32_t nsid = le32_to_cpu(c->nsid);
+ uint64_t prp1 = le64_to_cpu(c->prp1);
+ uint64_t prp2 = le64_to_cpu(c->prp2);
+ NvmeIdentifyNamespaceDescriptor *list;
+ uint16_t ret;
+
+ trace_nvme_identify_ns_descriptor_list(nsid);
+
+ list = g_malloc0(data_len);
+ list->nidt = 0x3;
+ list->nidl = 0x10;
+
+ ret = nvme_dma_read_prp(n, (uint8_t *) list, data_len, prp1, prp2);
+ g_free(list);
+ return ret;
+}
static uint16_t nvme_identify(NvmeCtrl *n, NvmeCmd *cmd)
{
@@ -686,7 +828,9 @@ static uint16_t nvme_identify(NvmeCtrl *n, NvmeCmd *cmd)
case 0x01:
return nvme_identify_ctrl(n, c);
case 0x02:
- return nvme_identify_nslist(n, c);
+ return nvme_identify_ns_list(n, c);
+ case 0x03:
+ return nvme_identify_ns_descriptor_list(n, cmd);
default:
trace_nvme_err_invalid_identify_cns(le32_to_cpu(c->cns));
return NVME_INVALID_FIELD | NVME_DNR;
@@ -696,18 +840,49 @@ static uint16_t nvme_identify(NvmeCtrl *n, NvmeCmd *cmd)
static uint16_t nvme_get_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
uint32_t dw10 = le32_to_cpu(cmd->cdw10);
+ uint32_t dw11 = le32_to_cpu(cmd->cdw11);
uint32_t result;
+ trace_nvme_getfeat(dw10);
+
switch (dw10) {
+ case NVME_ARBITRATION:
+ result = cpu_to_le32(n->features.arbitration);
+ break;
+ case NVME_POWER_MANAGEMENT:
+ result = cpu_to_le32(n->features.power_mgmt);
+ break;
+ case NVME_TEMPERATURE_THRESHOLD:
+ result = cpu_to_le32(n->features.temp_thresh);
+ break;
+ case NVME_ERROR_RECOVERY:
+ result = cpu_to_le32(n->features.err_rec);
+ break;
case NVME_VOLATILE_WRITE_CACHE:
result = blk_enable_write_cache(n->conf.blk);
trace_nvme_getfeat_vwcache(result ? "enabled" : "disabled");
break;
case NVME_NUMBER_OF_QUEUES:
result = cpu_to_le32((n->params.num_queues - 2) |
- ((n->params.num_queues - 2) << 16));
+ ((n->params.num_queues - 2) << 16));
trace_nvme_getfeat_numq(result);
break;
+ case NVME_INTERRUPT_COALESCING:
+ result = cpu_to_le32(n->features.int_coalescing);
+ break;
+ case NVME_INTERRUPT_VECTOR_CONF:
+ if ((dw11 & 0xffff) > n->params.num_queues) {
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ result = cpu_to_le32(n->features.int_vector_config[dw11 & 0xffff]);
+ break;
+ case NVME_WRITE_ATOMICITY:
+ result = cpu_to_le32(n->features.write_atomicity);
+ break;
+ case NVME_ASYNCHRONOUS_EVENT_CONF:
+ result = cpu_to_le32(n->features.async_config);
+ break;
default:
trace_nvme_err_invalid_getfeat(dw10);
return NVME_INVALID_FIELD | NVME_DNR;
@@ -722,22 +897,244 @@ static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
uint32_t dw10 = le32_to_cpu(cmd->cdw10);
uint32_t dw11 = le32_to_cpu(cmd->cdw11);
+ trace_nvme_setfeat(dw10, dw11);
+
switch (dw10) {
+ case NVME_TEMPERATURE_THRESHOLD:
+ n->features.temp_thresh = dw11;
+ if (n->features.temp_thresh <= n->temperature) {
+ nvme_enqueue_event(n, NVME_AER_TYPE_SMART,
+ NVME_AER_INFO_SMART_TEMP_THRESH, NVME_LOG_SMART_INFO);
+ }
+ break;
case NVME_VOLATILE_WRITE_CACHE:
blk_set_enable_write_cache(n->conf.blk, dw11 & 1);
break;
case NVME_NUMBER_OF_QUEUES:
+ if (n->qs_created > 2) {
+ return NVME_CMD_SEQ_ERROR | NVME_DNR;
+ }
+
+ if ((dw11 & 0xffff) == 0xffff || ((dw11 >> 16) & 0xffff) == 0xffff) {
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
trace_nvme_setfeat_numq((dw11 & 0xFFFF) + 1,
((dw11 >> 16) & 0xFFFF) + 1,
n->params.num_queues - 1,
n->params.num_queues - 1);
req->cqe.result = cpu_to_le32((n->params.num_queues - 2) |
- ((n->params.num_queues - 2) << 16));
+ ((n->params.num_queues - 2) << 16));
break;
+ case NVME_ASYNCHRONOUS_EVENT_CONF:
+ n->features.async_config = dw11;
+ break;
+ case NVME_ARBITRATION:
+ case NVME_POWER_MANAGEMENT:
+ case NVME_ERROR_RECOVERY:
+ case NVME_INTERRUPT_COALESCING:
+ case NVME_INTERRUPT_VECTOR_CONF:
+ case NVME_WRITE_ATOMICITY:
+ return NVME_FEAT_NOT_CHANGABLE | NVME_DNR;
default:
trace_nvme_err_invalid_setfeat(dw10);
return NVME_INVALID_FIELD | NVME_DNR;
}
+
+ return NVME_SUCCESS;
+}
+
+static void nvme_clear_events(NvmeCtrl *n, uint8_t event_type)
+{
+ n->aer_mask &= ~(1 << event_type);
+ if (!QSIMPLEQ_EMPTY(&n->aer_queue)) {
+ timer_mod(n->aer_timer,
+ qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + 500);
+ }
+}
+
+static uint16_t nvme_error_log_info(NvmeCtrl *n, NvmeCmd *cmd, uint8_t rae,
+ uint32_t buf_len, uint64_t off, NvmeRequest *req)
+{
+ uint32_t trans_len;
+ uint64_t prp1 = le64_to_cpu(cmd->prp1);
+ uint64_t prp2 = le64_to_cpu(cmd->prp2);
+
+ if (off > sizeof(*n->elpes) * (NVME_ELPE + 1)) {
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ trans_len = MIN(sizeof(*n->elpes) * (NVME_ELPE + 1) - off, buf_len);
+
+ if (!rae) {
+ nvme_clear_events(n, NVME_AER_TYPE_ERROR);
+ }
+
+ return nvme_dma_read_prp(n, (uint8_t *) n->elpes + off, trans_len, prp1,
+ prp2);
+}
+
+static uint16_t nvme_smart_info(NvmeCtrl *n, NvmeCmd *cmd, uint8_t rae,
+ uint32_t buf_len, uint64_t off, NvmeRequest *req)
+{
+ uint64_t prp1 = le64_to_cpu(cmd->prp1);
+ uint64_t prp2 = le64_to_cpu(cmd->prp2);
+
+ uint32_t trans_len;
+ time_t current_ms;
+ NvmeSmartLog smart;
+
+ if (cmd->nsid != 0 && cmd->nsid != 0xffffffff) {
+ trace_nvme_err(req->cqe.cid, "smart log not supported for namespace",
+ NVME_INVALID_FIELD | NVME_DNR);
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ if (off > sizeof(smart)) {
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ trans_len = MIN(sizeof(smart) - off, buf_len);
+
+ memset(&smart, 0x0, sizeof(smart));
+ smart.number_of_error_log_entries[0] = cpu_to_le64(0);
+ smart.temperature[0] = n->temperature & 0xff;
+ smart.temperature[1] = (n->temperature >> 8) & 0xff;
+
+ if (n->features.temp_thresh <= n->temperature) {
+ smart.critical_warning |= NVME_SMART_TEMPERATURE;
+ }
+
+ current_ms = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
+ smart.power_on_hours[0] = cpu_to_le64(
+ (((current_ms - n->starttime_ms) / 1000) / 60) / 60);
+
+ if (!rae) {
+ nvme_clear_events(n, NVME_AER_TYPE_SMART);
+ }
+
+ return nvme_dma_read_prp(n, (uint8_t *) &smart + off, trans_len, prp1,
+ prp2);
+}
+
+static uint16_t nvme_fw_log_info(NvmeCtrl *n, NvmeCmd *cmd, uint32_t buf_len,
+ uint64_t off, NvmeRequest *req)
+{
+ uint32_t trans_len;
+ uint64_t prp1 = le64_to_cpu(cmd->prp1);
+ uint64_t prp2 = le64_to_cpu(cmd->prp2);
+ NvmeFwSlotInfoLog fw_log;
+
+ if (off > sizeof(fw_log)) {
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ memset(&fw_log, 0, sizeof(NvmeFwSlotInfoLog));
+
+ trans_len = MIN(sizeof(fw_log) - off, buf_len);
+
+ return nvme_dma_read_prp(n, (uint8_t *) &fw_log + off, trans_len, prp1,
+ prp2);
+}
+
+static uint16_t nvme_get_log(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+{
+ uint32_t dw10 = le32_to_cpu(cmd->cdw10);
+ uint32_t dw11 = le32_to_cpu(cmd->cdw11);
+ uint32_t dw12 = le32_to_cpu(cmd->cdw12);
+ uint32_t dw13 = le32_to_cpu(cmd->cdw13);
+ uint16_t lid = dw10 & 0xff;
+ uint8_t rae = (dw10 >> 15) & 0x1;
+ uint32_t numdl, numdu, len;
+ uint64_t off, lpol, lpou;
+
+ numdl = (dw10 >> 16);
+ numdu = (dw11 & 0xffff);
+ lpol = dw12;
+ lpou = dw13;
+
+ len = (((numdu << 16) | numdl) + 1) << 2;
+ off = (lpou << 32ULL) | lpol;
+
+ if (off & 0x3) {
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ trace_nvme_get_log(req->cqe.cid, lid);
+
+ switch (lid) {
+ case NVME_LOG_ERROR_INFO:
+ return nvme_error_log_info(n, cmd, rae, len, off, req);
+ case NVME_LOG_SMART_INFO:
+ return nvme_smart_info(n, cmd, rae, len, off, req);
+ case NVME_LOG_FW_SLOT_INFO:
+ return nvme_fw_log_info(n, cmd, len, off, req);
+ default:
+ trace_nvme_err_invalid_log_page(req->cqe.cid, lid);
+ return NVME_INVALID_LOG_ID | NVME_DNR;
+ }
+}
+
+static uint16_t nvme_aer(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+{
+ trace_nvme_aer(req->cqe.cid);
+
+ if (n->outstanding_aers > NVME_AERL) {
+ trace_nvme_aer_aerl_exceeded();
+ return NVME_AER_LIMIT_EXCEEDED;
+ }
+
+ n->aer_reqs[n->outstanding_aers] = req;
+ timer_mod(n->aer_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + 500);
+ n->outstanding_aers++;
+
+ return NVME_NO_COMPLETE;
+}
+
+static uint16_t nvme_abort(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+{
+ NvmeSQueue *sq;
+ NvmeRequest *new;
+ uint32_t index = 0;
+ uint16_t sqid = cmd->cdw10 & 0xffff;
+ uint16_t cid = (cmd->cdw10 >> 16) & 0xffff;
+
+ req->cqe.result = 1;
+ if (nvme_check_sqid(n, sqid)) {
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ sq = n->sq[sqid];
+
+ /* only consider queued (and not executing) commands for abort */
+ while ((sq->head + index) % sq->size != sq->tail) {
+ NvmeCmd abort_cmd;
+ hwaddr addr;
+
+ addr = sq->dma_addr + ((sq->head + index) % sq->size) * n->sqe_size;
+
+ nvme_addr_read(n, addr, (void *) &abort_cmd, sizeof(abort_cmd));
+ if (abort_cmd.cid == cid) {
+ req->cqe.result = 0;
+ new = QTAILQ_FIRST(&sq->req_list);
+ QTAILQ_REMOVE(&sq->req_list, new, entry);
+ QTAILQ_INSERT_TAIL(&sq->out_req_list, new, entry);
+
+ memset(&new->cqe, 0, sizeof(new->cqe));
+ new->cqe.cid = cid;
+ new->status = NVME_CMD_ABORT_REQ;
+
+ abort_cmd.opcode = NVME_OP_ABORTED;
+ nvme_addr_write(n, addr, (void *) &abort_cmd, sizeof(abort_cmd));
+
+ nvme_enqueue_req_completion(n->cq[sq->cqid], new);
+
+ return NVME_SUCCESS;
+ }
+
+ ++index;
+ }
+
return NVME_SUCCESS;
}
@@ -758,6 +1155,12 @@ static uint16_t nvme_admin_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
return nvme_set_feature(n, cmd, req);
case NVME_ADM_CMD_GET_FEATURES:
return nvme_get_feature(n, cmd, req);
+ case NVME_ADM_CMD_GET_LOG_PAGE:
+ return nvme_get_log(n, cmd, req);
+ case NVME_ADM_CMD_ASYNC_EV_REQ:
+ return nvme_aer(n, cmd, req);
+ case NVME_ADM_CMD_ABORT:
+ return nvme_abort(n, cmd, req);
default:
trace_nvme_err_invalid_admin_opc(cmd->opcode);
return NVME_INVALID_OPCODE | NVME_DNR;
@@ -780,6 +1183,10 @@ static void nvme_process_sq(void *opaque)
nvme_addr_read(n, addr, (void *)&cmd, sizeof(cmd));
nvme_inc_sq_head(sq);
+ if (cmd.opcode == NVME_OP_ABORTED) {
+ continue;
+ }
+
req = QTAILQ_FIRST(&sq->req_list);
QTAILQ_REMOVE(&sq->req_list, req, entry);
QTAILQ_INSERT_TAIL(&sq->out_req_list, req, entry);
@@ -797,6 +1204,7 @@ static void nvme_process_sq(void *opaque)
static void nvme_clear_ctrl(NvmeCtrl *n)
{
+ NvmeAsyncEvent *event;
int i;
blk_drain(n->conf.blk);
@@ -812,8 +1220,19 @@ static void nvme_clear_ctrl(NvmeCtrl *n)
}
}
+ if (n->aer_timer) {
+ timer_del(n->aer_timer);
+ timer_free(n->aer_timer);
+ n->aer_timer = NULL;
+ }
+ while ((event = QSIMPLEQ_FIRST(&n->aer_queue)) != NULL) {
+ QSIMPLEQ_REMOVE_HEAD(&n->aer_queue, entry);
+ g_free(event);
+ }
+
blk_flush(n->conf.blk);
n->bar.cc = 0;
+ n->outstanding_aers = 0;
}
static int nvme_start_ctrl(NvmeCtrl *n)
@@ -906,6 +1325,9 @@ static int nvme_start_ctrl(NvmeCtrl *n)
nvme_init_sq(&n->admin_sq, n, n->bar.asq, 0, 0,
NVME_AQA_ASQS(n->bar.aqa) + 1);
+ n->aer_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, nvme_process_aers, n);
+ QSIMPLEQ_INIT(&n->aer_queue);
+
return 0;
}
@@ -1098,6 +1520,13 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val)
"completion queue doorbell write"
" for nonexistent queue,"
" sqid=%"PRIu32", ignoring", qid);
+
+ if (n->outstanding_aers) {
+ nvme_enqueue_event(n, NVME_AER_TYPE_ERROR,
+ NVME_AER_INFO_ERR_INVALID_DB_REGISTER,
+ NVME_LOG_ERROR_INFO);
+ }
+
return;
}
@@ -1108,6 +1537,12 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val)
" beyond queue size, sqid=%"PRIu32","
" new_head=%"PRIu16", ignoring",
qid, new_head);
+
+ if (n->outstanding_aers) {
+ nvme_enqueue_event(n, NVME_AER_TYPE_ERROR,
+ NVME_AER_INFO_ERR_INVALID_DB_VALUE, NVME_LOG_ERROR_INFO);
+ }
+
return;
}
@@ -1136,6 +1571,13 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val)
"submission queue doorbell write"
" for nonexistent queue,"
" sqid=%"PRIu32", ignoring", qid);
+
+ if (n->outstanding_aers) {
+ nvme_enqueue_event(n, NVME_AER_TYPE_ERROR,
+ NVME_AER_INFO_ERR_INVALID_DB_REGISTER,
+ NVME_LOG_ERROR_INFO);
+ }
+
return;
}
@@ -1146,6 +1588,12 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val)
" beyond queue size, sqid=%"PRIu32","
" new_tail=%"PRIu16", ignoring",
qid, new_tail);
+
+ if (n->outstanding_aers) {
+ nvme_enqueue_event(n, NVME_AER_TYPE_ERROR,
+ NVME_AER_INFO_ERR_INVALID_DB_VALUE, NVME_LOG_ERROR_INFO);
+ }
+
return;
}
@@ -1198,134 +1646,271 @@ static const MemoryRegionOps nvme_cmb_ops = {
},
};
-static void nvme_realize(PCIDevice *pci_dev, Error **errp)
+static int nvme_check_constraints(NvmeCtrl *n, Error **errp)
{
- NvmeCtrl *n = NVME(pci_dev);
- NvmeIdCtrl *id = &n->id_ctrl;
-
- int i;
- int64_t bs_size;
- uint8_t *pci_conf;
-
- if (!n->params.num_queues) {
- error_setg(errp, "num_queues can't be zero");
- return;
- }
+ NvmeParams *params = &n->params;
if (!n->conf.blk) {
- error_setg(errp, "drive property not set");
- return;
+ error_setg(errp, "nvme: block backend not configured");
+ return 1;
}
- bs_size = blk_getlength(n->conf.blk);
- if (bs_size < 0) {
- error_setg(errp, "could not get backing file size");
- return;
+ if (!params->serial) {
+ error_setg(errp, "nvme: serial not configured");
+ return 1;
}
- if (!n->params.serial) {
- error_setg(errp, "serial property not set");
- return;
+ if ((params->num_queues < 1 || params->num_queues > NVME_MAX_QS)) {
+ error_setg(errp, "nvme: invalid queue configuration");
+ return 1;
}
+
+ return 0;
+}
+
+static int nvme_init_blk(NvmeCtrl *n, Error **errp)
+{
blkconf_blocksizes(&n->conf);
if (!blkconf_apply_backend_options(&n->conf, blk_is_read_only(n->conf.blk),
- false, errp)) {
- return;
+ false, errp)) {
+ return 1;
}
- pci_conf = pci_dev->config;
- pci_conf[PCI_INTERRUPT_PIN] = 1;
- pci_config_set_prog_interface(pci_dev->config, 0x2);
- pci_config_set_class(pci_dev->config, PCI_CLASS_STORAGE_EXPRESS);
- pcie_endpoint_cap_init(pci_dev, 0x80);
+ return 0;
+}
+
+static int nvme_init_state(NvmeCtrl *n, Error **errp)
+{
+ int64_t bs_size;
+ Error *local_err = NULL;
+
+ if (!n->params.serial) {
+ error_setg(errp, "serial property not set");
+ return 1;
+ }
+
+ if (nvme_check_constraints(n, &local_err)) {
+ error_propagate_prepend(errp, local_err,
+ "nvme_check_constraints failed");
+ return 1;
+ }
- n->num_namespaces = 1;
n->reg_size = pow2ceil(0x1004 + 2 * (n->params.num_queues + 1) * 4);
- n->ns_size = bs_size / (uint64_t)n->num_namespaces;
- n->namespaces = g_new0(NvmeNamespace, n->num_namespaces);
+ bs_size = blk_getlength(n->conf.blk);
+ if (bs_size < 0) {
+ error_setg(errp, "could not get backing file size");
+ return 1;
+ }
+
+ n->ns_size = bs_size / (uint64_t) n->params.num_ns;
+
+ n->starttime_ms = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
n->sq = g_new0(NvmeSQueue *, n->params.num_queues);
n->cq = g_new0(NvmeCQueue *, n->params.num_queues);
+ n->elpes = g_new0(NvmeErrorLog, NVME_ELPE + 1);
+ n->aer_reqs = g_new0(NvmeRequest *, NVME_AERL + 1);
+ n->features.int_vector_config = g_malloc0_n(n->params.num_queues,
+ sizeof(*n->features.int_vector_config));
- memory_region_init_io(&n->iomem, OBJECT(n), &nvme_mmio_ops, n,
- "nvme", n->reg_size);
+ return 0;
+}
+
+static void nvme_init_cmb(NvmeCtrl *n, PCIDevice *pci_dev)
+{
+ NVME_CMBLOC_SET_BIR(n->bar.cmbloc, 2);
+ NVME_CMBLOC_SET_OFST(n->bar.cmbloc, 0);
+
+ NVME_CMBSZ_SET_SQS(n->bar.cmbsz, 1);
+ NVME_CMBSZ_SET_CQS(n->bar.cmbsz, 0);
+ NVME_CMBSZ_SET_LISTS(n->bar.cmbsz, 0);
+ NVME_CMBSZ_SET_RDS(n->bar.cmbsz, 1);
+ NVME_CMBSZ_SET_WDS(n->bar.cmbsz, 1);
+ NVME_CMBSZ_SET_SZU(n->bar.cmbsz, 2);
+ NVME_CMBSZ_SET_SZ(n->bar.cmbsz, n->params.cmb_size_mb);
+
+ n->cmbloc = n->bar.cmbloc;
+ n->cmbsz = n->bar.cmbsz;
+
+ n->cmbuf = g_malloc0(NVME_CMBSZ_GETSIZE(n->bar.cmbsz));
+ memory_region_init_io(&n->ctrl_mem, OBJECT(n), &nvme_cmb_ops, n,
+ "nvme-cmb", NVME_CMBSZ_GETSIZE(n->bar.cmbsz));
+ pci_register_bar(pci_dev, NVME_CMBLOC_BIR(n->bar.cmbloc),
+ PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64 |
+ PCI_BASE_ADDRESS_MEM_PREFETCH, &n->ctrl_mem);
+}
+
+static void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev)
+{
+ uint8_t *pci_conf = pci_dev->config;
+
+ pci_conf[PCI_INTERRUPT_PIN] = 1;
+ pci_config_set_prog_interface(pci_conf, 0x2);
+ pci_config_set_vendor_id(pci_conf, PCI_VENDOR_ID_INTEL);
+ pci_config_set_device_id(pci_conf, 0x5845);
+ pci_config_set_class(pci_conf, PCI_CLASS_STORAGE_EXPRESS);
+ pcie_endpoint_cap_init(pci_dev, 0x80);
+
+ memory_region_init_io(&n->iomem, OBJECT(n), &nvme_mmio_ops, n, "nvme",
+ n->reg_size);
pci_register_bar(pci_dev, 0,
PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64,
&n->iomem);
msix_init_exclusive_bar(pci_dev, n->params.num_queues, 4, NULL);
+ if (n->params.cmb_size_mb) {
+ nvme_init_cmb(n, pci_dev);
+ }
+}
+
+static void nvme_init_ctrl(NvmeCtrl *n)
+{
+ NvmeIdCtrl *id = &n->id_ctrl;
+ NvmeParams *params = &n->params;
+ uint8_t *pci_conf = n->parent_obj.config;
+
id->vid = cpu_to_le16(pci_get_word(pci_conf + PCI_VENDOR_ID));
id->ssvid = cpu_to_le16(pci_get_word(pci_conf + PCI_SUBSYSTEM_VENDOR_ID));
strpadcpy((char *)id->mn, sizeof(id->mn), "QEMU NVMe Ctrl", ' ');
strpadcpy((char *)id->fr, sizeof(id->fr), "1.0", ' ');
- strpadcpy((char *)id->sn, sizeof(id->sn), n->params.serial, ' ');
+ strpadcpy((char *)id->sn, sizeof(id->sn), params->serial, ' ');
id->rab = 6;
id->ieee[0] = 0x00;
id->ieee[1] = 0x02;
id->ieee[2] = 0xb3;
+ id->cmic = 0;
+ id->ver = cpu_to_le32(0x00010300);
id->oacs = cpu_to_le16(0);
- id->frmw = 7 << 1;
- id->lpa = 1 << 0;
+ id->acl = 3;
+ id->aerl = NVME_AERL;
+ id->frmw = 7 << 1 | 1;
+ id->lpa = 1 << 2;
+ id->elpe = NVME_ELPE;
+ id->npss = 0;
id->sqes = (0x6 << 4) | 0x6;
id->cqes = (0x4 << 4) | 0x4;
- id->nn = cpu_to_le32(n->num_namespaces);
+ id->nn = cpu_to_le32(params->num_ns);
id->oncs = cpu_to_le16(NVME_ONCS_WRITE_ZEROS);
+ id->fuses = cpu_to_le16(0);
+ id->fna = 0;
+ if (blk_enable_write_cache(n->conf.blk)) {
+ id->vwc = 1;
+ }
+ id->awun = cpu_to_le16(0);
+ id->awupf = cpu_to_le16(0);
+ id->sgls = cpu_to_le32(0);
+
+ strcpy((char *) id->subnqn, "nqn.2014-08.org.nvmexpress:uuid:");
+ qemu_uuid_unparse(&qemu_uuid,
+ (char *) id->subnqn + strlen((char *) id->subnqn));
+
id->psd[0].mp = cpu_to_le16(0x9c4);
id->psd[0].enlat = cpu_to_le32(0x10);
id->psd[0].exlat = cpu_to_le32(0x4);
- if (blk_enable_write_cache(n->conf.blk)) {
- id->vwc = 1;
+
+ n->temperature = NVME_TEMPERATURE;
+ n->features.temp_thresh = 0x14d;
+
+ for (int i = 0; i < n->params.num_queues; i++) {
+ n->features.int_vector_config[i] = i;
}
n->bar.cap = 0;
NVME_CAP_SET_MQES(n->bar.cap, 0x7ff);
NVME_CAP_SET_CQR(n->bar.cap, 1);
- NVME_CAP_SET_AMS(n->bar.cap, 1);
NVME_CAP_SET_TO(n->bar.cap, 0xf);
NVME_CAP_SET_CSS(n->bar.cap, 1);
NVME_CAP_SET_MPSMAX(n->bar.cap, 4);
- n->bar.vs = 0x00010200;
+ n->bar.vs = 0x00010300;
n->bar.intmc = n->bar.intms = 0;
+}
- if (n->params.cmb_size_mb) {
+static uint64_t nvme_ns_calc_blks(NvmeCtrl *n, NvmeNamespace *ns)
+{
+ return n->ns_size / nvme_ns_lbads_bytes(ns);
+}
- NVME_CMBLOC_SET_BIR(n->bar.cmbloc, 2);
- NVME_CMBLOC_SET_OFST(n->bar.cmbloc, 0);
+static void nvme_ns_init_identify(NvmeCtrl *n, NvmeIdNs *id_ns)
+{
+ id_ns->nlbaf = 0;
+ id_ns->flbas = 0;
+ id_ns->mc = 0;
+ id_ns->dpc = 0;
+ id_ns->dps = 0;
+ id_ns->lbaf[0].lbads = BDRV_SECTOR_BITS;
+}
- NVME_CMBSZ_SET_SQS(n->bar.cmbsz, 1);
- NVME_CMBSZ_SET_CQS(n->bar.cmbsz, 0);
- NVME_CMBSZ_SET_LISTS(n->bar.cmbsz, 0);
- NVME_CMBSZ_SET_RDS(n->bar.cmbsz, 1);
- NVME_CMBSZ_SET_WDS(n->bar.cmbsz, 1);
- NVME_CMBSZ_SET_SZU(n->bar.cmbsz, 2); /* MBs */
- NVME_CMBSZ_SET_SZ(n->bar.cmbsz, n->params.cmb_size_mb);
+static int nvme_init_namespace(NvmeCtrl *n, NvmeNamespace *ns, Error **errp)
+{
+ NvmeIdNs *id_ns = &ns->id_ns;
- n->cmbloc = n->bar.cmbloc;
- n->cmbsz = n->bar.cmbsz;
+ nvme_ns_init_identify(n, id_ns);
- n->cmbuf = g_malloc0(NVME_CMBSZ_GETSIZE(n->bar.cmbsz));
- memory_region_init_io(&n->ctrl_mem, OBJECT(n), &nvme_cmb_ops, n,
- "nvme-cmb", NVME_CMBSZ_GETSIZE(n->bar.cmbsz));
- pci_register_bar(pci_dev, NVME_CMBLOC_BIR(n->bar.cmbloc),
- PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64 |
- PCI_BASE_ADDRESS_MEM_PREFETCH, &n->ctrl_mem);
+ ns->ns_blks = nvme_ns_calc_blks(n, ns);
+ id_ns->nuse = id_ns->ncap = id_ns->nsze = cpu_to_le64(ns->ns_blks);
+ return 0;
+}
+
+static int nvme_init_namespaces(NvmeCtrl *n, Error **errp)
+{
+ int64_t bs_size;
+ Error *local_err = NULL;
+ NvmeNamespace *ns;
+
+ n->namespaces = g_new0(NvmeNamespace, n->params.num_ns);
+
+ bs_size = blk_getlength(n->conf.blk);
+ if (bs_size < 0) {
+ error_setg_errno(errp, -bs_size, "blk_getlength");
+ return 1;
+ }
+
+ n->ns_size = bs_size / (uint64_t) n->params.num_ns;
+
+ for (int i = 0; i < n->params.num_ns; i++) {
+ ns = &n->namespaces[i];
+ ns->id = i + 1;
+ ns->blk_offset = i * n->ns_size;
+
+ if (nvme_init_namespace(n, ns, &local_err)) {
+ error_propagate_prepend(errp, local_err,
+ "nvme_init_namespace: ");
+ return 1;
+ }
+ }
+
+ return 0;
+}
+
+static void nvme_realize(PCIDevice *pci_dev, Error **errp)
+{
+ NvmeCtrl *n = NVME(pci_dev);
+ Error *local_err = NULL;
+
+ if (nvme_check_constraints(n, &local_err)) {
+ error_propagate_prepend(errp, local_err, "nvme_check_constraints: ");
+ return;
+ }
+
+ if (nvme_init_blk(n, &local_err)) {
+ error_propagate_prepend(errp, local_err, "nvme_init_blk: ");
+ return;
+ }
+
+ if (nvme_init_state(n, &local_err)) {
+ error_propagate_prepend(errp, local_err, "nvme_init_state: ");
+ return;
}
- for (i = 0; i < n->num_namespaces; i++) {
- NvmeNamespace *ns = &n->namespaces[i];
- NvmeIdNs *id_ns = &ns->id_ns;
- id_ns->nsfeat = 0;
- id_ns->nlbaf = 0;
- id_ns->flbas = 0;
- id_ns->mc = 0;
- id_ns->dpc = 0;
- id_ns->dps = 0;
- id_ns->lbaf[0].ds = BDRV_SECTOR_BITS;
- id_ns->ncap = id_ns->nuse = id_ns->nsze =
- cpu_to_le64(n->ns_size >>
- id_ns->lbaf[NVME_ID_NS_FLBAS_INDEX(ns->id_ns.flbas)].ds);
+ nvme_init_pci(n, pci_dev);
+ nvme_init_ctrl(n);
+
+ if (nvme_init_namespaces(n, &local_err)) {
+ error_propagate_prepend(errp, local_err,
+ "nvme_init_namespaces: ");
+ return;
}
}
@@ -1337,6 +1922,9 @@ static void nvme_exit(PCIDevice *pci_dev)
g_free(n->namespaces);
g_free(n->cq);
g_free(n->sq);
+ g_free(n->elpes);
+ g_free(n->aer_reqs);
+ g_free(n->features.int_vector_config);
if (n->params.cmb_size_mb) {
g_free(n->cmbuf);
diff --git a/hw/block/nvme.h b/hw/block/nvme.h
index 8866373058f6..8925a05445da 100644
--- a/hw/block/nvme.h
+++ b/hw/block/nvme.h
@@ -6,11 +6,13 @@
#define DEFINE_NVME_PROPERTIES(_state, _props) \
DEFINE_PROP_STRING("serial", _state, _props.serial), \
DEFINE_PROP_UINT32("cmb_size_mb", _state, _props.cmb_size_mb, 0), \
- DEFINE_PROP_UINT32("num_queues", _state, _props.num_queues, 64)
+ DEFINE_PROP_UINT32("num_queues", _state, _props.num_queues, 64), \
+ DEFINE_PROP_UINT32("num_ns", _state, _props.num_ns, 1)
typedef struct NvmeParams {
char *serial;
uint32_t num_queues;
+ uint32_t num_ns;
uint32_t cmb_size_mb;
} NvmeParams;
@@ -63,6 +65,9 @@ typedef struct NvmeCQueue {
typedef struct NvmeNamespace {
NvmeIdNs id_ns;
+ uint32_t id;
+ uint64_t ns_blks;
+ uint64_t blk_offset;
} NvmeNamespace;
#define TYPE_NVME "nvme"
@@ -77,26 +82,48 @@ typedef struct NvmeCtrl {
BlockConf conf;
NvmeParams params;
+ uint64_t starttime_ms;
+ uint16_t temperature;
uint32_t page_size;
uint16_t page_bits;
uint16_t max_prp_ents;
uint16_t cqe_size;
uint16_t sqe_size;
uint32_t reg_size;
- uint32_t num_namespaces;
uint32_t max_q_ents;
uint64_t ns_size;
+ uint8_t outstanding_aers;
uint32_t cmbsz;
uint32_t cmbloc;
uint8_t *cmbuf;
uint64_t irq_status;
+ uint32_t qs_created;
+ QSIMPLEQ_HEAD(, NvmeAsyncEvent) aer_queue;
+ QEMUTimer *aer_timer;
+ uint8_t aer_mask;
+ uint8_t aer_mask_queued;
+
+ NvmeErrorLog *elpes;
+ NvmeRequest **aer_reqs;
NvmeNamespace *namespaces;
NvmeSQueue **sq;
NvmeCQueue **cq;
NvmeSQueue admin_sq;
NvmeCQueue admin_cq;
+ NvmeFeatureVal features;
NvmeIdCtrl id_ctrl;
} NvmeCtrl;
+static inline uint8_t nvme_ns_lbads(NvmeNamespace *ns)
+{
+ NvmeIdNs *id = &ns->id_ns;
+ return id->lbaf[NVME_ID_NS_FLBAS_INDEX(id->flbas)].lbads;
+}
+
+static inline size_t nvme_ns_lbads_bytes(NvmeNamespace *ns)
+{
+ return 1 << nvme_ns_lbads(ns);
+}
+
#endif /* HW_NVME_H */
diff --git a/hw/block/trace-events b/hw/block/trace-events
index b92039a5739f..abec518167d0 100644
--- a/hw/block/trace-events
+++ b/hw/block/trace-events
@@ -42,10 +42,22 @@ nvme_del_sq(uint16_t qid) "deleting submission queue sqid=%"PRIu16""
nvme_del_cq(uint16_t cqid) "deleted completion queue, sqid=%"PRIu16""
nvme_identify_ctrl(void) "identify controller"
nvme_identify_ns(uint16_t ns) "identify namespace, nsid=%"PRIu16""
-nvme_identify_nslist(uint16_t ns) "identify namespace list, nsid=%"PRIu16""
+nvme_identify_ns_list(uint16_t ns) "identify namespace list, nsid=%"PRIu16""
+nvme_identify_ns_descriptor_list(uint16_t ns) "identify namespace descriptor list, nsid=%"PRIu16""
+nvme_getfeat(uint32_t fid) "fid 0x%"PRIx32""
+nvme_setfeat(uint32_t fid, uint32_t val) "fid 0x%"PRIx32" val 0x%"PRIx32""
nvme_getfeat_vwcache(const char* result) "get feature volatile write cache, result=%s"
nvme_getfeat_numq(int result) "get feature number of queues, result=%d"
nvme_setfeat_numq(int reqcq, int reqsq, int gotcq, int gotsq) "requested cq_count=%d sq_count=%d, responding with cq_count=%d sq_count=%d"
+nvme_get_log(uint16_t cid, uint16_t lid) "cid %"PRIu16" lid 0x%"PRIx16""
+nvme_process_aers(void) "processing aers"
+nvme_aer(uint16_t cid) "cid %"PRIu16""
+nvme_aer_aerl_exceeded(void) "aerl exceeded"
+nvme_aer_masked(uint8_t type, uint8_t mask) "type 0x%"PRIx8" mask 0x%"PRIx8""
+nvme_aer_post_cqe(uint8_t typ, uint8_t info, uint8_t log_page) "type 0x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8""
+nvme_enqueue_req_completion(uint16_t cid, uint16_t cqid) "cid %"PRIu16" cqid %"PRIu16""
+nvme_enqueue_event(uint8_t typ, uint8_t info, uint8_t log_page) "type 0x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8""
+nvme_no_outstanding_aers(void) "ignoring event; no outstanding AERs"
nvme_mmio_intm_set(uint64_t data, uint64_t new_mask) "wrote MMIO, interrupt mask set, data=0x%"PRIx64", new_mask=0x%"PRIx64""
nvme_mmio_intm_clr(uint64_t data, uint64_t new_mask) "wrote MMIO, interrupt mask clr, data=0x%"PRIx64", new_mask=0x%"PRIx64""
nvme_mmio_cfg(uint64_t data) "wrote MMIO, config controller config=0x%"PRIx64""
@@ -60,6 +72,7 @@ nvme_mmio_shutdown_set(void) "shutdown bit set"
nvme_mmio_shutdown_cleared(void) "shutdown bit cleared"
# nvme traces for error conditions
+nvme_err(uint16_t cid, const char *s, uint16_t status) "cid %"PRIu16" \"%s\" status 0x%"PRIx16""
nvme_err_invalid_dma(void) "PRP/SGL is too small for transfer size"
nvme_err_invalid_prplist_ent(uint64_t prplist) "PRP list entry is null or not page aligned: 0x%"PRIx64""
nvme_err_invalid_prp2_align(uint64_t prp2) "PRP2 is not page aligned: 0x%"PRIx64""
@@ -85,6 +98,7 @@ nvme_err_invalid_create_cq_qflags(uint16_t qflags) "failed creating completion q
nvme_err_invalid_identify_cns(uint16_t cns) "identify, invalid cns=0x%"PRIx16""
nvme_err_invalid_getfeat(int dw10) "invalid get features, dw10=0x%"PRIx32""
nvme_err_invalid_setfeat(uint32_t dw10) "invalid set features, dw10=0x%"PRIx32""
+nvme_err_invalid_log_page(uint16_t cid, uint16_t lid) "cid %"PRIu16" lid 0x%"PRIx16""
nvme_err_startfail_cq(void) "nvme_start_ctrl failed because there are non-admin completion queues"
nvme_err_startfail_sq(void) "nvme_start_ctrl failed because there are non-admin submission queues"
nvme_err_startfail_nbarasq(void) "nvme_start_ctrl failed because the admin submission queue address is null"
diff --git a/include/block/nvme.h b/include/block/nvme.h
index 849a6f3fa346..5a169e7ed7ac 100644
--- a/include/block/nvme.h
+++ b/include/block/nvme.h
@@ -386,8 +386,8 @@ enum NvmeAsyncEventRequest {
NVME_AER_TYPE_SMART = 1,
NVME_AER_TYPE_IO_SPECIFIC = 6,
NVME_AER_TYPE_VENDOR_SPECIFIC = 7,
- NVME_AER_INFO_ERR_INVALID_SQ = 0,
- NVME_AER_INFO_ERR_INVALID_DB = 1,
+ NVME_AER_INFO_ERR_INVALID_DB_REGISTER = 0,
+ NVME_AER_INFO_ERR_INVALID_DB_VALUE = 1,
NVME_AER_INFO_ERR_DIAG_FAIL = 2,
NVME_AER_INFO_ERR_PERS_INTERNAL_ERR = 3,
NVME_AER_INFO_ERR_TRANS_INTERNAL_ERR = 4,
@@ -445,7 +445,8 @@ enum NvmeStatusCodes {
NVME_FW_REQ_RESET = 0x010b,
NVME_INVALID_QUEUE_DEL = 0x010c,
NVME_FID_NOT_SAVEABLE = 0x010d,
- NVME_FID_NOT_NSID_SPEC = 0x010f,
+ NVME_FEAT_NOT_CHANGABLE = 0x010e,
+ NVME_FEAT_NOT_NSID_SPEC = 0x010f,
NVME_FW_REQ_SUSYSTEM_RESET = 0x0110,
NVME_CONFLICTING_ATTRS = 0x0180,
NVME_INVALID_PROT_INFO = 0x0181,
@@ -462,6 +463,13 @@ enum NvmeStatusCodes {
NVME_NO_COMPLETE = 0xffff,
};
+typedef struct NvmeIdentifyNamespaceDescriptor {
+ uint8_t nidt;
+ uint8_t nidl;
+ uint8_t rsvd[2];
+ uint8_t nid[];
+} NvmeIdentifyNamespaceDescriptor;
+
typedef struct NvmeFwSlotInfoLog {
uint8_t afi;
uint8_t reserved1[7];
@@ -543,7 +551,15 @@ typedef struct NvmeIdCtrl {
uint8_t ieee[3];
uint8_t cmic;
uint8_t mdts;
- uint8_t rsvd255[178];
+ uint16_t cntlid;
+ uint32_t ver;
+ uint16_t rtd3r;
+ uint32_t rtd3e;
+ uint32_t oaes;
+ uint32_t ctratt;
+ uint8_t rsvd111[12];
+ uint8_t fguid[16];
+ uint8_t rsvd255[128];
uint16_t oacs;
uint8_t acl;
uint8_t aerl;
@@ -551,10 +567,28 @@ typedef struct NvmeIdCtrl {
uint8_t lpa;
uint8_t elpe;
uint8_t npss;
- uint8_t rsvd511[248];
+ uint8_t avscc;
+ uint8_t apsta;
+ uint16_t wctemp;
+ uint16_t cctemp;
+ uint16_t mtfa;
+ uint32_t hmpre;
+ uint32_t hmmin;
+ uint8_t tnvmcap[16];
+ uint8_t unvmcap[16];
+ uint32_t rpmbs;
+ uint16_t edstt;
+ uint8_t dsto;
+ uint8_t fwug;
+ uint16_t kas;
+ uint16_t hctma;
+ uint16_t mntmt;
+ uint16_t mxtmt;
+ uint32_t sanicap;
+ uint8_t rsvd511[180];
uint8_t sqes;
uint8_t cqes;
- uint16_t rsvd515;
+ uint16_t maxcmd;
uint32_t nn;
uint16_t oncs;
uint16_t fuses;
@@ -562,8 +596,14 @@ typedef struct NvmeIdCtrl {
uint8_t vwc;
uint16_t awun;
uint16_t awupf;
- uint8_t rsvd703[174];
- uint8_t rsvd2047[1344];
+ uint8_t nvscc;
+ uint8_t rsvd531;
+ uint16_t acwu;
+ uint16_t rsvd535;
+ uint32_t sgls;
+ uint8_t rsvd767[228];
+ uint8_t subnqn[256];
+ uint8_t rsvd2047[1024];
NvmePSD psd[32];
uint8_t vs[1024];
} NvmeIdCtrl;
@@ -637,7 +677,7 @@ typedef struct NvmeRangeType {
typedef struct NvmeLBAF {
uint16_t ms;
- uint8_t ds;
+ uint8_t lbads;
uint8_t rp;
} NvmeLBAF;
--
2.21.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH 3/8] nvme: simplify PRP mappings
2019-05-17 8:42 [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 1/8] nvme: move device parameters to separate struct Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 2/8] nvme: bump supported spec to 1.3 Klaus Birkelund Jensen
@ 2019-05-17 8:42 ` Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 4/8] nvme: allow multiple i/o's per request Klaus Birkelund Jensen
` (5 subsequent siblings)
8 siblings, 0 replies; 15+ messages in thread
From: Klaus Birkelund Jensen @ 2019-05-17 8:42 UTC (permalink / raw)
To: qemu-block; +Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
Instead of handling both QSGs and IOVs in multiple places, simply use
QSGs everywhere by assuming that the request does not involve the
controller memory buffer (CMB). If the request is found to involve the
CMB, convert the QSG to an IOV and issue the I/O.
The QSG is converted to an IOV by the dma helpers anyway, so it is not
like the CMB path is unfairly affected by this simplifying change.
Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
---
hw/block/nvme.c | 205 +++++++++++++++++++++++++++---------------
hw/block/nvme.h | 3 +-
hw/block/trace-events | 1 +
include/block/nvme.h | 1 +
4 files changed, 138 insertions(+), 72 deletions(-)
diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index 65dfc04f71e5..453213f9abb4 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -71,14 +71,21 @@
static void nvme_process_sq(void *opaque);
+static inline uint8_t nvme_addr_is_cmb(NvmeCtrl *n, hwaddr addr)
+{
+ return n->cmbsz && addr >= n->ctrl_mem.addr &&
+ addr < (n->ctrl_mem.addr + int128_get64(n->ctrl_mem.size));
+}
+
static void nvme_addr_read(NvmeCtrl *n, hwaddr addr, void *buf, int size)
{
- if (n->cmbsz && addr >= n->ctrl_mem.addr &&
- addr < (n->ctrl_mem.addr + int128_get64(n->ctrl_mem.size))) {
+ if (nvme_addr_is_cmb(n, addr)) {
memcpy(buf, (void *)&n->cmbuf[addr - n->ctrl_mem.addr], size);
- } else {
- pci_dma_read(&n->parent_obj, addr, buf, size);
+
+ return;
}
+
+ pci_dma_read(&n->parent_obj, addr, buf, size);
}
static void nvme_addr_write(NvmeCtrl *n, hwaddr addr, void *buf, int size)
@@ -167,31 +174,48 @@ static void nvme_irq_deassert(NvmeCtrl *n, NvmeCQueue *cq)
}
}
-static uint16_t nvme_map_prp(QEMUSGList *qsg, QEMUIOVector *iov, uint64_t prp1,
- uint64_t prp2, uint32_t len, NvmeCtrl *n)
+static uint16_t nvme_map_prp(NvmeCtrl *n, QEMUSGList *qsg, uint64_t prp1,
+ uint64_t prp2, uint32_t len, NvmeRequest *req)
{
hwaddr trans_len = n->page_size - (prp1 % n->page_size);
trans_len = MIN(len, trans_len);
int num_prps = (len >> n->page_bits) + 1;
+ uint16_t status = NVME_SUCCESS;
+
+ trace_nvme_map_prp(req->cmd_opcode, trans_len, len, prp1, prp2, num_prps);
if (unlikely(!prp1)) {
trace_nvme_err_invalid_prp();
return NVME_INVALID_FIELD | NVME_DNR;
- } else if (n->cmbsz && prp1 >= n->ctrl_mem.addr &&
- prp1 < n->ctrl_mem.addr + int128_get64(n->ctrl_mem.size)) {
- qsg->nsg = 0;
- qemu_iovec_init(iov, num_prps);
- qemu_iovec_add(iov, (void *)&n->cmbuf[prp1 - n->ctrl_mem.addr], trans_len);
+ }
+
+ if (nvme_addr_is_cmb(n, prp1)) {
+ NvmeSQueue *sq = req->sq;
+ if (!nvme_addr_is_cmb(n, sq->dma_addr)) {
+ return NVME_INVALID_USE_OF_CMB | NVME_DNR;
+ }
+
+ req->is_cmb = true;
} else {
- pci_dma_sglist_init(qsg, &n->parent_obj, num_prps);
- qemu_sglist_add(qsg, prp1, trans_len);
+ req->is_cmb = false;
}
+
+ pci_dma_sglist_init(qsg, &n->parent_obj, num_prps);
+ qemu_sglist_add(qsg, prp1, trans_len);
+
len -= trans_len;
if (len) {
if (unlikely(!prp2)) {
trace_nvme_err_invalid_prp2_missing();
+ status = NVME_INVALID_FIELD | NVME_DNR;
goto unmap;
}
+
+ if (req->is_cmb && !nvme_addr_is_cmb(n, prp2)) {
+ status = NVME_INVALID_USE_OF_CMB | NVME_DNR;
+ goto unmap;
+ }
+
if (len > n->page_size) {
uint64_t prp_list[n->max_prp_ents];
uint32_t nents, prp_trans;
@@ -203,79 +227,99 @@ static uint16_t nvme_map_prp(QEMUSGList *qsg, QEMUIOVector *iov, uint64_t prp1,
while (len != 0) {
uint64_t prp_ent = le64_to_cpu(prp_list[i]);
+ if (req->is_cmb && !nvme_addr_is_cmb(n, prp_ent)) {
+ status = NVME_INVALID_USE_OF_CMB | NVME_DNR;
+ goto unmap;
+ }
+
if (i == n->max_prp_ents - 1 && len > n->page_size) {
if (unlikely(!prp_ent || prp_ent & (n->page_size - 1))) {
trace_nvme_err_invalid_prplist_ent(prp_ent);
+ status = NVME_INVALID_FIELD | NVME_DNR;
goto unmap;
}
i = 0;
nents = (len + n->page_size - 1) >> n->page_bits;
prp_trans = MIN(n->max_prp_ents, nents) * sizeof(uint64_t);
- nvme_addr_read(n, prp_ent, (void *)prp_list,
- prp_trans);
+ nvme_addr_read(n, prp_ent, (void *)prp_list, prp_trans);
prp_ent = le64_to_cpu(prp_list[i]);
}
if (unlikely(!prp_ent || prp_ent & (n->page_size - 1))) {
trace_nvme_err_invalid_prplist_ent(prp_ent);
+ status = NVME_INVALID_FIELD | NVME_DNR;
goto unmap;
}
trans_len = MIN(len, n->page_size);
- if (qsg->nsg){
- qemu_sglist_add(qsg, prp_ent, trans_len);
- } else {
- qemu_iovec_add(iov, (void *)&n->cmbuf[prp_ent - n->ctrl_mem.addr], trans_len);
- }
+ qemu_sglist_add(qsg, prp_ent, trans_len);
+
len -= trans_len;
i++;
}
} else {
if (unlikely(prp2 & (n->page_size - 1))) {
trace_nvme_err_invalid_prp2_align(prp2);
+ status = NVME_INVALID_FIELD | NVME_DNR;
goto unmap;
}
- if (qsg->nsg) {
- qemu_sglist_add(qsg, prp2, len);
- } else {
- qemu_iovec_add(iov, (void *)&n->cmbuf[prp2 - n->ctrl_mem.addr], trans_len);
- }
+
+ qemu_sglist_add(qsg, prp2, len);
}
}
+
return NVME_SUCCESS;
- unmap:
+unmap:
qemu_sglist_destroy(qsg);
- return NVME_INVALID_FIELD | NVME_DNR;
+
+ return status;
+}
+
+static void dma_to_cmb(NvmeCtrl *n, QEMUSGList *qsg, QEMUIOVector *iov)
+{
+ for (int i = 0; i < qsg->nsg; i++) {
+ void *addr = &n->cmbuf[qsg->sg[i].base - n->ctrl_mem.addr];
+ qemu_iovec_add(iov, addr, qsg->sg[i].len);
+ }
}
static uint16_t nvme_dma_read_prp(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
- uint64_t prp1, uint64_t prp2)
+ uint64_t prp1, uint64_t prp2, NvmeRequest *req)
{
QEMUSGList qsg;
- QEMUIOVector iov;
- uint16_t status = NVME_SUCCESS;
+ uint16_t err = NVME_SUCCESS;
- trace_nvme_dma_read(prp1, prp2);
-
- if (nvme_map_prp(&qsg, &iov, prp1, prp2, len, n)) {
- return NVME_INVALID_FIELD | NVME_DNR;
+ err = nvme_map_prp(n, &qsg, prp1, prp2, len, req);
+ if (err) {
+ return err;
}
- if (qsg.nsg > 0) {
- if (unlikely(dma_buf_read(ptr, len, &qsg))) {
+
+ if (req->is_cmb) {
+ QEMUIOVector iov;
+
+ qemu_iovec_init(&iov, qsg.nsg);
+ dma_to_cmb(n, &qsg, &iov);
+
+ if (unlikely(qemu_iovec_from_buf(&iov, 0, ptr, len) != len)) {
trace_nvme_err_invalid_dma();
- status = NVME_INVALID_FIELD | NVME_DNR;
- }
- qemu_sglist_destroy(&qsg);
- } else {
- if (unlikely(qemu_iovec_to_buf(&iov, 0, ptr, len) != len)) {
- trace_nvme_err_invalid_dma();
- status = NVME_INVALID_FIELD | NVME_DNR;
+ err = NVME_INVALID_FIELD | NVME_DNR;
}
+
qemu_iovec_destroy(&iov);
+
+ return err;
}
- return status;
+
+ if (unlikely(dma_buf_read(ptr, len, &qsg))) {
+ trace_nvme_err_invalid_dma();
+ err = NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ qemu_sglist_destroy(&qsg);
+
+ return err;
}
static void nvme_post_cqe(NvmeCQueue *cq, NvmeRequest *req)
@@ -415,16 +459,20 @@ static void nvme_rw_cb(void *opaque, int ret)
block_acct_failed(blk_get_stats(n->conf.blk), &req->acct);
req->status = NVME_INTERNAL_DEV_ERROR;
}
- if (req->has_sg) {
+
+ if (req->qsg.nalloc) {
qemu_sglist_destroy(&req->qsg);
}
+ if (req->iov.nalloc) {
+ qemu_iovec_destroy(&req->iov);
+ }
+
nvme_enqueue_req_completion(cq, req);
}
static uint16_t nvme_flush(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd,
NvmeRequest *req)
{
- req->has_sg = false;
block_acct_start(blk_get_stats(n->conf.blk), &req->acct, 0,
BLOCK_ACCT_FLUSH);
req->aiocb = blk_aio_flush(n->conf.blk, nvme_rw_cb, req);
@@ -448,7 +496,6 @@ static uint16_t nvme_write_zeros(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd,
return NVME_LBA_RANGE | NVME_DNR;
}
- req->has_sg = false;
block_acct_start(blk_get_stats(n->conf.blk), &req->acct, 0,
BLOCK_ACCT_WRITE);
req->aiocb = blk_aio_pwrite_zeroes(n->conf.blk, offset, count,
@@ -480,21 +527,21 @@ static uint16_t nvme_rw(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd,
return NVME_LBA_RANGE | NVME_DNR;
}
- if (nvme_map_prp(&req->qsg, &req->iov, prp1, prp2, data_size, n)) {
+ if (nvme_map_prp(n, &req->qsg, prp1, prp2, data_size, req)) {
block_acct_invalid(blk_get_stats(n->conf.blk), acct);
return NVME_INVALID_FIELD | NVME_DNR;
}
dma_acct_start(n->conf.blk, &req->acct, &req->qsg, acct);
- if (req->qsg.nsg > 0) {
- req->has_sg = true;
+ if (!req->is_cmb) {
req->aiocb = is_write ?
dma_blk_write(n->conf.blk, &req->qsg, data_offset, BDRV_SECTOR_SIZE,
nvme_rw_cb, req) :
dma_blk_read(n->conf.blk, &req->qsg, data_offset, BDRV_SECTOR_SIZE,
nvme_rw_cb, req);
} else {
- req->has_sg = false;
+ qemu_iovec_init(&req->iov, req->qsg.nsg);
+ dma_to_cmb(n, &req->qsg, &req->iov);
req->aiocb = is_write ?
blk_aio_pwritev(n->conf.blk, data_offset, &req->iov, 0, nvme_rw_cb,
req) :
@@ -592,7 +639,7 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr,
sq->size = size;
sq->cqid = cqid;
sq->head = sq->tail = 0;
- sq->io_req = g_new(NvmeRequest, sq->size);
+ sq->io_req = g_new0(NvmeRequest, sq->size);
QTAILQ_INIT(&sq->req_list);
QTAILQ_INIT(&sq->out_req_list);
@@ -740,7 +787,8 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeCmd *cmd)
return NVME_SUCCESS;
}
-static uint16_t nvme_identify_ctrl(NvmeCtrl *n, NvmeIdentify *c)
+static uint16_t nvme_identify_ctrl(NvmeCtrl *n, NvmeIdentify *c,
+ NvmeRequest *req)
{
uint64_t prp1 = le64_to_cpu(c->prp1);
uint64_t prp2 = le64_to_cpu(c->prp2);
@@ -748,10 +796,11 @@ static uint16_t nvme_identify_ctrl(NvmeCtrl *n, NvmeIdentify *c)
trace_nvme_identify_ctrl();
return nvme_dma_read_prp(n, (uint8_t *)&n->id_ctrl, sizeof(n->id_ctrl),
- prp1, prp2);
+ prp1, prp2, req);
}
-static uint16_t nvme_identify_ns(NvmeCtrl *n, NvmeIdentify *c)
+static uint16_t nvme_identify_ns(NvmeCtrl *n, NvmeIdentify *c,
+ NvmeRequest *req)
{
NvmeNamespace *ns;
uint32_t nsid = le32_to_cpu(c->nsid);
@@ -768,10 +817,11 @@ static uint16_t nvme_identify_ns(NvmeCtrl *n, NvmeIdentify *c)
ns = &n->namespaces[nsid - 1];
return nvme_dma_read_prp(n, (uint8_t *)&ns->id_ns, sizeof(ns->id_ns),
- prp1, prp2);
+ prp1, prp2, req);
}
-static uint16_t nvme_identify_ns_list(NvmeCtrl *n, NvmeIdentify *c)
+static uint16_t nvme_identify_ns_list(NvmeCtrl *n, NvmeIdentify *c,
+ NvmeRequest *req)
{
static const int data_len = 4 * KiB;
uint32_t min_nsid = le32_to_cpu(c->nsid);
@@ -793,12 +843,13 @@ static uint16_t nvme_identify_ns_list(NvmeCtrl *n, NvmeIdentify *c)
break;
}
}
- ret = nvme_dma_read_prp(n, (uint8_t *)list, data_len, prp1, prp2);
+ ret = nvme_dma_read_prp(n, (uint8_t *)list, data_len, prp1, prp2, req);
g_free(list);
return ret;
}
-static uint16_t nvme_identify_ns_descriptor_list(NvmeCtrl *n, NvmeCmd *c)
+static uint16_t nvme_identify_ns_descriptor_list(NvmeCtrl *n, NvmeCmd *c,
+ NvmeRequest *req)
{
static const int data_len = sizeof(NvmeIdentifyNamespaceDescriptor) + 0x10;
uint32_t nsid = le32_to_cpu(c->nsid);
@@ -813,24 +864,24 @@ static uint16_t nvme_identify_ns_descriptor_list(NvmeCtrl *n, NvmeCmd *c)
list->nidt = 0x3;
list->nidl = 0x10;
- ret = nvme_dma_read_prp(n, (uint8_t *) list, data_len, prp1, prp2);
+ ret = nvme_dma_read_prp(n, (uint8_t *) list, data_len, prp1, prp2, req);
g_free(list);
return ret;
}
-static uint16_t nvme_identify(NvmeCtrl *n, NvmeCmd *cmd)
+static uint16_t nvme_identify(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
NvmeIdentify *c = (NvmeIdentify *)cmd;
switch (le32_to_cpu(c->cns)) {
case 0x00:
- return nvme_identify_ns(n, c);
+ return nvme_identify_ns(n, c, req);
case 0x01:
- return nvme_identify_ctrl(n, c);
+ return nvme_identify_ctrl(n, c, req);
case 0x02:
- return nvme_identify_ns_list(n, c);
+ return nvme_identify_ns_list(n, c, req);
case 0x03:
- return nvme_identify_ns_descriptor_list(n, cmd);
+ return nvme_identify_ns_descriptor_list(n, cmd, req);
default:
trace_nvme_err_invalid_identify_cns(le32_to_cpu(c->cns));
return NVME_INVALID_FIELD | NVME_DNR;
@@ -971,7 +1022,7 @@ static uint16_t nvme_error_log_info(NvmeCtrl *n, NvmeCmd *cmd, uint8_t rae,
}
return nvme_dma_read_prp(n, (uint8_t *) n->elpes + off, trans_len, prp1,
- prp2);
+ prp2, req);
}
static uint16_t nvme_smart_info(NvmeCtrl *n, NvmeCmd *cmd, uint8_t rae,
@@ -1014,7 +1065,7 @@ static uint16_t nvme_smart_info(NvmeCtrl *n, NvmeCmd *cmd, uint8_t rae,
}
return nvme_dma_read_prp(n, (uint8_t *) &smart + off, trans_len, prp1,
- prp2);
+ prp2, req);
}
static uint16_t nvme_fw_log_info(NvmeCtrl *n, NvmeCmd *cmd, uint32_t buf_len,
@@ -1034,9 +1085,10 @@ static uint16_t nvme_fw_log_info(NvmeCtrl *n, NvmeCmd *cmd, uint32_t buf_len,
trans_len = MIN(sizeof(fw_log) - off, buf_len);
return nvme_dma_read_prp(n, (uint8_t *) &fw_log + off, trans_len, prp1,
- prp2);
+ prp2, req);
}
+
static uint16_t nvme_get_log(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
uint32_t dw10 = le32_to_cpu(cmd->cdw10);
@@ -1150,7 +1202,7 @@ static uint16_t nvme_admin_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
case NVME_ADM_CMD_CREATE_CQ:
return nvme_create_cq(n, cmd);
case NVME_ADM_CMD_IDENTIFY:
- return nvme_identify(n, cmd);
+ return nvme_identify(n, cmd, req);
case NVME_ADM_CMD_SET_FEATURES:
return nvme_set_feature(n, cmd, req);
case NVME_ADM_CMD_GET_FEATURES:
@@ -1167,6 +1219,17 @@ static uint16_t nvme_admin_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
}
}
+static void nvme_init_req(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+{
+ memset(&req->cqe, 0, sizeof(req->cqe));
+ req->cqe.cid = le16_to_cpu(cmd->cid);
+
+ req->cmd_opcode = cmd->opcode;
+ req->is_cmb = false;
+
+ req->status = NVME_SUCCESS;
+}
+
static void nvme_process_sq(void *opaque)
{
NvmeSQueue *sq = opaque;
@@ -1190,8 +1253,8 @@ static void nvme_process_sq(void *opaque)
req = QTAILQ_FIRST(&sq->req_list);
QTAILQ_REMOVE(&sq->req_list, req, entry);
QTAILQ_INSERT_TAIL(&sq->out_req_list, req, entry);
- memset(&req->cqe, 0, sizeof(req->cqe));
- req->cqe.cid = cmd.cid;
+
+ nvme_init_req(n, &cmd, req);
status = sq->sqid ? nvme_io_cmd(n, &cmd, req) :
nvme_admin_cmd(n, &cmd, req);
diff --git a/hw/block/nvme.h b/hw/block/nvme.h
index 8925a05445da..05217257ca3f 100644
--- a/hw/block/nvme.h
+++ b/hw/block/nvme.h
@@ -25,7 +25,8 @@ typedef struct NvmeRequest {
struct NvmeSQueue *sq;
BlockAIOCB *aiocb;
uint16_t status;
- bool has_sg;
+ bool is_cmb;
+ uint8_t cmd_opcode;
NvmeCqe cqe;
BlockAcctCookie acct;
QEMUSGList qsg;
diff --git a/hw/block/trace-events b/hw/block/trace-events
index abec518167d0..676a3a615c9d 100644
--- a/hw/block/trace-events
+++ b/hw/block/trace-events
@@ -35,6 +35,7 @@ nvme_irq_msix(uint32_t vector) "raising MSI-X IRQ vector %u"
nvme_irq_pin(void) "pulsing IRQ pin"
nvme_irq_masked(void) "IRQ is masked"
nvme_dma_read(uint64_t prp1, uint64_t prp2) "DMA read, prp1=0x%"PRIx64" prp2=0x%"PRIx64""
+nvme_map_prp(uint8_t cmd_opcode, uint64_t trans_len, uint32_t len, uint64_t prp1, uint64_t prp2, int num_prps) "cmd_opcode=0x%"PRIx8", trans_len=%"PRIu64", len=%"PRIu32", prp1=0x%"PRIx64", prp2=0x%"PRIx64", num_prps=%d"
nvme_rw(const char *verb, uint32_t blk_count, uint64_t byte_count, uint64_t lba) "%s %"PRIu32" blocks (%"PRIu64" bytes) from LBA %"PRIu64""
nvme_create_sq(uint64_t addr, uint16_t sqid, uint16_t cqid, uint16_t qsize, uint16_t qflags) "create submission queue, addr=0x%"PRIx64", sqid=%"PRIu16", cqid=%"PRIu16", qsize=%"PRIu16", qflags=%"PRIu16""
nvme_create_cq(uint64_t addr, uint16_t cqid, uint16_t vector, uint16_t size, uint16_t qflags, int ien) "create completion queue, addr=0x%"PRIx64", cqid=%"PRIu16", vector=%"PRIu16", qsize=%"PRIu16", qflags=%"PRIu16", ien=%d"
diff --git a/include/block/nvme.h b/include/block/nvme.h
index 5a169e7ed7ac..c90c36b66971 100644
--- a/include/block/nvme.h
+++ b/include/block/nvme.h
@@ -427,6 +427,7 @@ enum NvmeStatusCodes {
NVME_CMD_ABORT_MISSING_FUSE = 0x000a,
NVME_INVALID_NSID = 0x000b,
NVME_CMD_SEQ_ERROR = 0x000c,
+ NVME_INVALID_USE_OF_CMB = 0x0012,
NVME_LBA_RANGE = 0x0080,
NVME_CAP_EXCEEDED = 0x0081,
NVME_NS_NOT_READY = 0x0082,
--
2.21.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH 4/8] nvme: allow multiple i/o's per request
2019-05-17 8:42 [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Klaus Birkelund Jensen
` (2 preceding siblings ...)
2019-05-17 8:42 ` [Qemu-devel] [PATCH 3/8] nvme: simplify PRP mappings Klaus Birkelund Jensen
@ 2019-05-17 8:42 ` Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 5/8] nvme: add support for metadata Klaus Birkelund Jensen
` (4 subsequent siblings)
8 siblings, 0 replies; 15+ messages in thread
From: Klaus Birkelund Jensen @ 2019-05-17 8:42 UTC (permalink / raw)
To: qemu-block; +Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
Introduce a new NvmeBlockBackendRequest and move the QEMUSGList and
QEMUIOVector from the NvmeRequest.
This is in preparation for metadata support and makes it easier to
handle multiple block backend requests to different offsets.
Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
---
hw/block/nvme.c | 319 ++++++++++++++++++++++++++++++++----------
hw/block/nvme.h | 47 +++++--
hw/block/trace-events | 2 +
3 files changed, 286 insertions(+), 82 deletions(-)
diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index 453213f9abb4..c514f93f3867 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -322,6 +322,88 @@ static uint16_t nvme_dma_read_prp(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
return err;
}
+static void nvme_blk_req_destroy(NvmeBlockBackendRequest *blk_req)
+{
+ if (blk_req->qsg.nalloc) {
+ qemu_sglist_destroy(&blk_req->qsg);
+ }
+
+ if (blk_req->iov.nalloc) {
+ qemu_iovec_destroy(&blk_req->iov);
+ }
+
+ g_free(blk_req);
+}
+
+static void nvme_blk_req_put(NvmeCtrl *n, NvmeBlockBackendRequest *blk_req)
+{
+ nvme_blk_req_destroy(blk_req);
+}
+
+static NvmeBlockBackendRequest *nvme_blk_req_get(NvmeCtrl *n, NvmeRequest *req,
+ QEMUSGList *qsg)
+{
+ NvmeBlockBackendRequest *blk_req = g_malloc0(sizeof(*blk_req));
+
+ blk_req->req = req;
+
+ if (qsg) {
+ pci_dma_sglist_init(&blk_req->qsg, &n->parent_obj, qsg->nsg);
+ memcpy(blk_req->qsg.sg, qsg->sg, qsg->nsg * sizeof(ScatterGatherEntry));
+
+ blk_req->qsg.nsg = qsg->nsg;
+ blk_req->qsg.size = qsg->size;
+ }
+
+ return blk_req;
+}
+
+static uint16_t nvme_blk_setup(NvmeCtrl *n, NvmeNamespace *ns, QEMUSGList *qsg,
+ uint64_t blk_offset, uint32_t unit_len, NvmeRequest *req)
+{
+ NvmeBlockBackendRequest *blk_req = nvme_blk_req_get(n, req, qsg);
+ if (!blk_req) {
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "nvme_blk_req_get: %s",
+ "could not allocate memory");
+ return NVME_INTERNAL_DEV_ERROR;
+ }
+
+ blk_req->slba = req->slba;
+ blk_req->nlb = req->nlb;
+ blk_req->blk_offset = blk_offset + req->slba * unit_len;
+
+ QTAILQ_INSERT_TAIL(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ return NVME_SUCCESS;
+}
+
+static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+{
+ NvmeNamespace *ns = req->ns;
+ uint16_t err;
+
+ QEMUSGList qsg;
+
+ uint32_t unit_len = nvme_ns_lbads_bytes(ns);
+ uint32_t len = req->nlb * unit_len;
+ uint64_t prp1 = le64_to_cpu(cmd->prp1);
+ uint64_t prp2 = le64_to_cpu(cmd->prp2);
+
+ err = nvme_map_prp(n, &qsg, prp1, prp2, len, req);
+ if (err) {
+ return err;
+ }
+
+ err = nvme_blk_setup(n, ns, &qsg, ns->blk_offset, unit_len, req);
+ if (err) {
+ return err;
+ }
+
+ qemu_sglist_destroy(&qsg);
+
+ return NVME_SUCCESS;
+}
+
static void nvme_post_cqe(NvmeCQueue *cq, NvmeRequest *req)
{
NvmeCtrl *n = cq->ctrl;
@@ -447,114 +529,190 @@ static void nvme_process_aers(void *opaque)
static void nvme_rw_cb(void *opaque, int ret)
{
- NvmeRequest *req = opaque;
+ NvmeBlockBackendRequest *blk_req = opaque;
+ NvmeRequest *req = blk_req->req;
NvmeSQueue *sq = req->sq;
NvmeCtrl *n = sq->ctrl;
NvmeCQueue *cq = n->cq[sq->cqid];
+ NvmeNamespace *ns = req->ns;
+
+ QTAILQ_REMOVE(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ trace_nvme_rw_cb(req->cqe.cid, ns->id);
if (!ret) {
- block_acct_done(blk_get_stats(n->conf.blk), &req->acct);
- req->status = NVME_SUCCESS;
+ block_acct_done(blk_get_stats(n->conf.blk), &blk_req->acct);
} else {
- block_acct_failed(blk_get_stats(n->conf.blk), &req->acct);
- req->status = NVME_INTERNAL_DEV_ERROR;
+ block_acct_failed(blk_get_stats(n->conf.blk), &blk_req->acct);
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "block request failed: %s",
+ strerror(-ret));
+ req->status = NVME_INTERNAL_DEV_ERROR | NVME_DNR;
}
- if (req->qsg.nalloc) {
- qemu_sglist_destroy(&req->qsg);
- }
- if (req->iov.nalloc) {
- qemu_iovec_destroy(&req->iov);
+ if (QTAILQ_EMPTY(&req->blk_req_tailq)) {
+ nvme_enqueue_req_completion(cq, req);
}
- nvme_enqueue_req_completion(cq, req);
+ nvme_blk_req_put(n, blk_req);
}
-static uint16_t nvme_flush(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd,
- NvmeRequest *req)
+static uint16_t nvme_flush(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
- block_acct_start(blk_get_stats(n->conf.blk), &req->acct, 0,
+ NvmeBlockBackendRequest *blk_req = nvme_blk_req_get(n, req, NULL);
+ if (!blk_req) {
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "nvme_blk_req_get: %s",
+ "could not allocate memory");
+ return NVME_INTERNAL_DEV_ERROR;
+ }
+
+ block_acct_start(blk_get_stats(n->conf.blk), &blk_req->acct, 0,
BLOCK_ACCT_FLUSH);
- req->aiocb = blk_aio_flush(n->conf.blk, nvme_rw_cb, req);
+ blk_req->aiocb = blk_aio_flush(n->conf.blk, nvme_rw_cb, blk_req);
+
+ QTAILQ_INSERT_TAIL(&req->blk_req_tailq, blk_req, tailq_entry);
return NVME_NO_COMPLETE;
}
-static uint16_t nvme_write_zeros(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd,
- NvmeRequest *req)
+static uint16_t nvme_write_zeros(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
NvmeRwCmd *rw = (NvmeRwCmd *)cmd;
- const uint8_t lba_index = NVME_ID_NS_FLBAS_INDEX(ns->id_ns.flbas);
- const uint8_t data_shift = ns->id_ns.lbaf[lba_index].lbads;
+ NvmeBlockBackendRequest *blk_req;
+ const uint8_t lbads = nvme_ns_lbads(req->ns);
uint64_t slba = le64_to_cpu(rw->slba);
uint32_t nlb = le16_to_cpu(rw->nlb) + 1;
- uint64_t offset = slba << data_shift;
- uint32_t count = nlb << data_shift;
+ uint64_t offset = slba << lbads;
+ uint32_t count = nlb << lbads;
- if (unlikely(slba + nlb > ns->id_ns.nsze)) {
- trace_nvme_err_invalid_lba_range(slba, nlb, ns->id_ns.nsze);
+ if (unlikely(slba + nlb > req->ns->id_ns.nsze)) {
+ trace_nvme_err_invalid_lba_range(slba, nlb, req->ns->id_ns.nsze);
return NVME_LBA_RANGE | NVME_DNR;
}
- block_acct_start(blk_get_stats(n->conf.blk), &req->acct, 0,
- BLOCK_ACCT_WRITE);
- req->aiocb = blk_aio_pwrite_zeroes(n->conf.blk, offset, count,
- BDRV_REQ_MAY_UNMAP, nvme_rw_cb, req);
+ blk_req = nvme_blk_req_get(n, req, NULL);
+ if (!blk_req) {
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "nvme_blk_req_get: %s",
+ "could not allocate memory");
+ return NVME_INTERNAL_DEV_ERROR;
+ }
+
+ block_acct_start(blk_get_stats(n->conf.blk), &blk_req->acct, 0,
+ BLOCK_ACCT_WRITE);
+
+ blk_req->aiocb = blk_aio_pwrite_zeroes(n->conf.blk, offset, count,
+ BDRV_REQ_MAY_UNMAP, nvme_rw_cb, blk_req);
+
+ QTAILQ_INSERT_TAIL(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ return NVME_NO_COMPLETE;
+}
+
+static uint16_t nvme_rw_check_req(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+{
+ NvmeNamespace *ns = req->ns;
+ NvmeRwCmd *rw = (NvmeRwCmd *) cmd;
+
+ uint16_t ctrl = le16_to_cpu(rw->control);
+ uint32_t data_size = req->nlb << nvme_ns_lbads(ns);
+
+ if (n->params.mdts && data_size > n->page_size * (1 << n->params.mdts)) {
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ if ((ctrl & NVME_RW_PRINFO_PRACT) && !(ns->id_ns.dps & DPS_TYPE_MASK)) {
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ return NVME_SUCCESS;
+}
+
+static void nvme_blk_submit_dma(NvmeCtrl *n, NvmeBlockBackendRequest *blk_req,
+ BlockCompletionFunc *cb)
+{
+ NvmeRequest *req = blk_req->req;
+
+ if (req->is_write) {
+ dma_acct_start(n->conf.blk, &blk_req->acct, &blk_req->qsg,
+ BLOCK_ACCT_WRITE);
+
+ blk_req->aiocb = dma_blk_write(n->conf.blk, &blk_req->qsg,
+ blk_req->blk_offset, BDRV_SECTOR_SIZE, cb, blk_req);
+ } else {
+ dma_acct_start(n->conf.blk, &blk_req->acct, &blk_req->qsg,
+ BLOCK_ACCT_READ);
+
+ blk_req->aiocb = dma_blk_read(n->conf.blk, &blk_req->qsg,
+ blk_req->blk_offset, BDRV_SECTOR_SIZE, cb, blk_req);
+ }
+}
+
+static void nvme_blk_submit_cmb(NvmeCtrl *n, NvmeBlockBackendRequest *blk_req,
+ BlockCompletionFunc *cb)
+{
+ NvmeRequest *req = blk_req->req;
+
+ qemu_iovec_init(&blk_req->iov, blk_req->qsg.nsg);
+ dma_to_cmb(n, &blk_req->qsg, &blk_req->iov);
+
+ if (req->is_write) {
+ block_acct_start(blk_get_stats(n->conf.blk), &blk_req->acct,
+ blk_req->iov.size, BLOCK_ACCT_WRITE);
+
+ blk_req->aiocb = blk_aio_pwritev(n->conf.blk, blk_req->blk_offset,
+ &blk_req->iov, 0, cb, blk_req);
+ } else {
+ block_acct_start(blk_get_stats(n->conf.blk), &blk_req->acct,
+ blk_req->iov.size, BLOCK_ACCT_READ);
+
+ blk_req->aiocb = blk_aio_preadv(n->conf.blk, blk_req->blk_offset,
+ &blk_req->iov, 0, cb, blk_req);
+ }
+}
+
+static uint16_t nvme_blk_submit_io(NvmeCtrl *n, NvmeRequest *req,
+ BlockCompletionFunc *cb)
+{
+ NvmeBlockBackendRequest *blk_req;
+
+ if (QTAILQ_EMPTY(&req->blk_req_tailq)) {
+ return NVME_SUCCESS;
+ }
+
+ QTAILQ_FOREACH(blk_req, &req->blk_req_tailq, tailq_entry) {
+ if (req->is_cmb) {
+ nvme_blk_submit_cmb(n, blk_req, cb);
+ } else {
+ nvme_blk_submit_dma(n, blk_req, cb);
+ }
+ }
+
return NVME_NO_COMPLETE;
}
-static uint16_t nvme_rw(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd,
- NvmeRequest *req)
+static uint16_t nvme_rw(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
NvmeRwCmd *rw = (NvmeRwCmd *)cmd;
uint32_t nlb = le32_to_cpu(rw->nlb) + 1;
uint64_t slba = le64_to_cpu(rw->slba);
- uint64_t prp1 = le64_to_cpu(rw->prp1);
- uint64_t prp2 = le64_to_cpu(rw->prp2);
- uint8_t lba_index = NVME_ID_NS_FLBAS_INDEX(ns->id_ns.flbas);
- uint8_t data_shift = ns->id_ns.lbaf[lba_index].lbads;
- uint64_t data_size = (uint64_t)nlb << data_shift;
- uint64_t data_offset = ns->blk_offset + (slba << data_shift);
- int is_write = rw->opcode == NVME_CMD_WRITE ? 1 : 0;
- enum BlockAcctType acct = is_write ? BLOCK_ACCT_WRITE : BLOCK_ACCT_READ;
+ req->is_write = nvme_rw_is_write(req);
- trace_nvme_rw(is_write ? "write" : "read", nlb, data_size, slba);
+ trace_nvme_rw(req->is_write ? "write" : "read", nlb,
+ nlb << nvme_ns_lbads(req->ns), slba);
- if (unlikely((slba + nlb) > ns->id_ns.nsze)) {
- block_acct_invalid(blk_get_stats(n->conf.blk), acct);
- trace_nvme_err_invalid_lba_range(slba, nlb, ns->id_ns.nsze);
- return NVME_LBA_RANGE | NVME_DNR;
+ int err = nvme_blk_map(n, cmd, req);
+ if (err) {
+ return err;
}
- if (nvme_map_prp(n, &req->qsg, prp1, prp2, data_size, req)) {
- block_acct_invalid(blk_get_stats(n->conf.blk), acct);
- return NVME_INVALID_FIELD | NVME_DNR;
- }
-
- dma_acct_start(n->conf.blk, &req->acct, &req->qsg, acct);
- if (!req->is_cmb) {
- req->aiocb = is_write ?
- dma_blk_write(n->conf.blk, &req->qsg, data_offset, BDRV_SECTOR_SIZE,
- nvme_rw_cb, req) :
- dma_blk_read(n->conf.blk, &req->qsg, data_offset, BDRV_SECTOR_SIZE,
- nvme_rw_cb, req);
- } else {
- qemu_iovec_init(&req->iov, req->qsg.nsg);
- dma_to_cmb(n, &req->qsg, &req->iov);
- req->aiocb = is_write ?
- blk_aio_pwritev(n->conf.blk, data_offset, &req->iov, 0, nvme_rw_cb,
- req) :
- blk_aio_preadv(n->conf.blk, data_offset, &req->iov, 0, nvme_rw_cb,
- req);
- }
-
- return NVME_NO_COMPLETE;
+ return nvme_blk_submit_io(n, req, nvme_rw_cb);
}
static uint16_t nvme_io_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
- NvmeNamespace *ns;
+ NvmeRwCmd *rw;
+ int err;
+
uint32_t nsid = le32_to_cpu(cmd->nsid);
if (unlikely(nsid == 0 || nsid > n->params.num_ns)) {
@@ -562,15 +720,26 @@ static uint16_t nvme_io_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
return NVME_INVALID_NSID | NVME_DNR;
}
- ns = &n->namespaces[nsid - 1];
+ req->ns = &n->namespaces[nsid - 1];
+
switch (cmd->opcode) {
case NVME_CMD_FLUSH:
- return nvme_flush(n, ns, cmd, req);
+ return nvme_flush(n, cmd, req);
case NVME_CMD_WRITE_ZEROS:
- return nvme_write_zeros(n, ns, cmd, req);
+ return nvme_write_zeros(n, cmd, req);
case NVME_CMD_WRITE:
case NVME_CMD_READ:
- return nvme_rw(n, ns, cmd, req);
+ rw = (NvmeRwCmd *)cmd;
+
+ req->nlb = le16_to_cpu(rw->nlb) + 1;
+ req->slba = le64_to_cpu(rw->slba);
+
+ err = nvme_rw_check_req(n, cmd, req);
+ if (err) {
+ return err;
+ }
+
+ return nvme_rw(n, cmd, req);
default:
trace_nvme_err_invalid_opc(cmd->opcode);
return NVME_INVALID_OPCODE | NVME_DNR;
@@ -595,6 +764,7 @@ static uint16_t nvme_del_sq(NvmeCtrl *n, NvmeCmd *cmd)
NvmeRequest *req, *next;
NvmeSQueue *sq;
NvmeCQueue *cq;
+ NvmeBlockBackendRequest *blk_req;
uint16_t qid = le16_to_cpu(c->qid);
if (unlikely(!qid || nvme_check_sqid(n, qid))) {
@@ -607,8 +777,11 @@ static uint16_t nvme_del_sq(NvmeCtrl *n, NvmeCmd *cmd)
sq = n->sq[qid];
while (!QTAILQ_EMPTY(&sq->out_req_list)) {
req = QTAILQ_FIRST(&sq->out_req_list);
- assert(req->aiocb);
- blk_aio_cancel(req->aiocb);
+ while (!QTAILQ_EMPTY(&req->blk_req_tailq)) {
+ blk_req = QTAILQ_FIRST(&req->blk_req_tailq);
+ assert(blk_req->aiocb);
+ blk_aio_cancel(blk_req->aiocb);
+ }
}
if (!nvme_check_cqid(n, sq->cqid)) {
cq = n->cq[sq->cqid];
@@ -645,6 +818,7 @@ static void nvme_init_sq(NvmeSQueue *sq, NvmeCtrl *n, uint64_t dma_addr,
QTAILQ_INIT(&sq->out_req_list);
for (i = 0; i < sq->size; i++) {
sq->io_req[i].sq = sq;
+ QTAILQ_INIT(&(sq->io_req[i].blk_req_tailq));
QTAILQ_INSERT_TAIL(&(sq->req_list), &sq->io_req[i], entry);
}
sq->timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, nvme_process_sq, sq);
@@ -1842,6 +2016,7 @@ static void nvme_init_ctrl(NvmeCtrl *n)
id->ieee[1] = 0x02;
id->ieee[2] = 0xb3;
id->cmic = 0;
+ id->mdts = params->mdts;
id->ver = cpu_to_le32(0x00010300);
id->oacs = cpu_to_le16(0);
id->acl = 3;
diff --git a/hw/block/nvme.h b/hw/block/nvme.h
index 05217257ca3f..711ca249eac5 100644
--- a/hw/block/nvme.h
+++ b/hw/block/nvme.h
@@ -7,12 +7,14 @@
DEFINE_PROP_STRING("serial", _state, _props.serial), \
DEFINE_PROP_UINT32("cmb_size_mb", _state, _props.cmb_size_mb, 0), \
DEFINE_PROP_UINT32("num_queues", _state, _props.num_queues, 64), \
- DEFINE_PROP_UINT32("num_ns", _state, _props.num_ns, 1)
+ DEFINE_PROP_UINT32("num_ns", _state, _props.num_ns, 1), \
+ DEFINE_PROP_UINT8("mdts", _state, _props.mdts, 7)
typedef struct NvmeParams {
char *serial;
uint32_t num_queues;
uint32_t num_ns;
+ uint8_t mdts;
uint32_t cmb_size_mb;
} NvmeParams;
@@ -21,16 +23,36 @@ typedef struct NvmeAsyncEvent {
NvmeAerResult result;
} NvmeAsyncEvent;
+typedef struct NvmeBlockBackendRequest {
+ uint64_t slba;
+ uint16_t nlb;
+ uint64_t blk_offset;
+
+ struct NvmeRequest *req;
+
+ BlockAIOCB *aiocb;
+ BlockAcctCookie acct;
+
+ QEMUSGList qsg;
+ QEMUIOVector iov;
+
+ QTAILQ_ENTRY(NvmeBlockBackendRequest) tailq_entry;
+ QSLIST_ENTRY(NvmeBlockBackendRequest) slist_entry;
+} NvmeBlockBackendRequest;
+
typedef struct NvmeRequest {
- struct NvmeSQueue *sq;
- BlockAIOCB *aiocb;
- uint16_t status;
- bool is_cmb;
- uint8_t cmd_opcode;
- NvmeCqe cqe;
- BlockAcctCookie acct;
- QEMUSGList qsg;
- QEMUIOVector iov;
+ struct NvmeSQueue *sq;
+ struct NvmeNamespace *ns;
+ NvmeCqe cqe;
+
+ uint64_t slba;
+ uint16_t nlb;
+ uint16_t status;
+ bool is_cmb;
+ bool is_write;
+ uint8_t cmd_opcode;
+
+ QTAILQ_HEAD(, NvmeBlockBackendRequest) blk_req_tailq;
QTAILQ_ENTRY(NvmeRequest)entry;
} NvmeRequest;
@@ -116,6 +138,11 @@ typedef struct NvmeCtrl {
NvmeIdCtrl id_ctrl;
} NvmeCtrl;
+static inline bool nvme_rw_is_write(NvmeRequest *req)
+{
+ return req->cmd_opcode == NVME_CMD_WRITE;
+}
+
static inline uint8_t nvme_ns_lbads(NvmeNamespace *ns)
{
NvmeIdNs *id = &ns->id_ns;
diff --git a/hw/block/trace-events b/hw/block/trace-events
index 676a3a615c9d..56fec40d130c 100644
--- a/hw/block/trace-events
+++ b/hw/block/trace-events
@@ -37,6 +37,7 @@ nvme_irq_masked(void) "IRQ is masked"
nvme_dma_read(uint64_t prp1, uint64_t prp2) "DMA read, prp1=0x%"PRIx64" prp2=0x%"PRIx64""
nvme_map_prp(uint8_t cmd_opcode, uint64_t trans_len, uint32_t len, uint64_t prp1, uint64_t prp2, int num_prps) "cmd_opcode=0x%"PRIx8", trans_len=%"PRIu64", len=%"PRIu32", prp1=0x%"PRIx64", prp2=0x%"PRIx64", num_prps=%d"
nvme_rw(const char *verb, uint32_t blk_count, uint64_t byte_count, uint64_t lba) "%s %"PRIu32" blocks (%"PRIu64" bytes) from LBA %"PRIu64""
+nvme_rw_cb(uint16_t cid, uint32_t nsid) "cid %"PRIu16" nsid %"PRIu32""
nvme_create_sq(uint64_t addr, uint16_t sqid, uint16_t cqid, uint16_t qsize, uint16_t qflags) "create submission queue, addr=0x%"PRIx64", sqid=%"PRIu16", cqid=%"PRIu16", qsize=%"PRIu16", qflags=%"PRIu16""
nvme_create_cq(uint64_t addr, uint16_t cqid, uint16_t vector, uint16_t size, uint16_t qflags, int ien) "create completion queue, addr=0x%"PRIx64", cqid=%"PRIu16", vector=%"PRIu16", qsize=%"PRIu16", qflags=%"PRIu16", ien=%d"
nvme_del_sq(uint16_t qid) "deleting submission queue sqid=%"PRIu16""
@@ -115,6 +116,7 @@ nvme_err_startfail_sqent_too_large(uint8_t log2ps, uint8_t maxlog2ps) "nvme_star
nvme_err_startfail_asqent_sz_zero(void) "nvme_start_ctrl failed because the admin submission queue size is zero"
nvme_err_startfail_acqent_sz_zero(void) "nvme_start_ctrl failed because the admin completion queue size is zero"
nvme_err_startfail(void) "setting controller enable bit failed"
+nvme_err_internal_dev_error(const char *reason) "%s"
# Traces for undefined behavior
nvme_ub_mmiowr_misaligned32(uint64_t offset) "MMIO write not 32-bit aligned, offset=0x%"PRIx64""
--
2.21.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH 5/8] nvme: add support for metadata
2019-05-17 8:42 [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Klaus Birkelund Jensen
` (3 preceding siblings ...)
2019-05-17 8:42 ` [Qemu-devel] [PATCH 4/8] nvme: allow multiple i/o's per request Klaus Birkelund Jensen
@ 2019-05-17 8:42 ` Klaus Birkelund Jensen
2019-05-22 6:12 ` [Qemu-devel] [Qemu-block] " Klaus Birkelund
2019-05-17 8:42 ` [Qemu-devel] [PATCH 6/8] nvme: add support for scatter gather lists Klaus Birkelund Jensen
` (3 subsequent siblings)
8 siblings, 1 reply; 15+ messages in thread
From: Klaus Birkelund Jensen @ 2019-05-17 8:42 UTC (permalink / raw)
To: qemu-block; +Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
The new `ms` parameter may be used to indicate the number of metadata
bytes provided per LBA.
Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
---
hw/block/nvme.c | 31 +++++++++++++++++++++++++++++--
hw/block/nvme.h | 11 ++++++++++-
2 files changed, 39 insertions(+), 3 deletions(-)
diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index c514f93f3867..675967a596d1 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -33,6 +33,8 @@
* num_ns=<int> : Namespaces to make out of the backing storage,
* Default:1
* num_queues=<int> : Number of possible IO Queues, Default:64
+ * ms=<int> : Number of metadata bytes provided per LBA,
+ * Default:0
* cmb_size_mb=<int> : Size of CMB in MBs, Default:0
*
* Parameters will be verified against conflicting capabilities and attributes
@@ -386,6 +388,8 @@ static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
uint32_t unit_len = nvme_ns_lbads_bytes(ns);
uint32_t len = req->nlb * unit_len;
+ uint32_t meta_unit_len = nvme_ns_ms(ns);
+ uint32_t meta_len = req->nlb * meta_unit_len;
uint64_t prp1 = le64_to_cpu(cmd->prp1);
uint64_t prp2 = le64_to_cpu(cmd->prp2);
@@ -399,6 +403,19 @@ static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
return err;
}
+ qsg.nsg = 0;
+ qsg.size = 0;
+
+ if (cmd->mptr && n->params.ms) {
+ qemu_sglist_add(&qsg, le64_to_cpu(cmd->mptr), meta_len);
+
+ err = nvme_blk_setup(n, ns, &qsg, ns->blk_offset_md, meta_unit_len,
+ req);
+ if (err) {
+ return err;
+ }
+ }
+
qemu_sglist_destroy(&qsg);
return NVME_SUCCESS;
@@ -1902,6 +1919,11 @@ static int nvme_check_constraints(NvmeCtrl *n, Error **errp)
return 1;
}
+ if (params->ms && !is_power_of_2(params->ms)) {
+ error_setg(errp, "nvme: invalid metadata configuration");
+ return 1;
+ }
+
return 0;
}
@@ -2066,17 +2088,20 @@ static void nvme_init_ctrl(NvmeCtrl *n)
static uint64_t nvme_ns_calc_blks(NvmeCtrl *n, NvmeNamespace *ns)
{
- return n->ns_size / nvme_ns_lbads_bytes(ns);
+ return n->ns_size / (nvme_ns_lbads_bytes(ns) + nvme_ns_ms(ns));
}
static void nvme_ns_init_identify(NvmeCtrl *n, NvmeIdNs *id_ns)
{
+ NvmeParams *params = &n->params;
+
id_ns->nlbaf = 0;
id_ns->flbas = 0;
- id_ns->mc = 0;
+ id_ns->mc = params->ms ? 0x2 : 0;
id_ns->dpc = 0;
id_ns->dps = 0;
id_ns->lbaf[0].lbads = BDRV_SECTOR_BITS;
+ id_ns->lbaf[0].ms = params->ms;
}
static int nvme_init_namespace(NvmeCtrl *n, NvmeNamespace *ns, Error **errp)
@@ -2086,6 +2111,8 @@ static int nvme_init_namespace(NvmeCtrl *n, NvmeNamespace *ns, Error **errp)
nvme_ns_init_identify(n, id_ns);
ns->ns_blks = nvme_ns_calc_blks(n, ns);
+ ns->blk_offset_md = ns->blk_offset + nvme_ns_lbads_bytes(ns) * ns->ns_blks;
+
id_ns->nuse = id_ns->ncap = id_ns->nsze = cpu_to_le64(ns->ns_blks);
return 0;
diff --git a/hw/block/nvme.h b/hw/block/nvme.h
index 711ca249eac5..81ee0c5173d5 100644
--- a/hw/block/nvme.h
+++ b/hw/block/nvme.h
@@ -8,13 +8,15 @@
DEFINE_PROP_UINT32("cmb_size_mb", _state, _props.cmb_size_mb, 0), \
DEFINE_PROP_UINT32("num_queues", _state, _props.num_queues, 64), \
DEFINE_PROP_UINT32("num_ns", _state, _props.num_ns, 1), \
- DEFINE_PROP_UINT8("mdts", _state, _props.mdts, 7)
+ DEFINE_PROP_UINT8("mdts", _state, _props.mdts, 7), \
+ DEFINE_PROP_UINT8("ms", _state, _props.ms, 0)
typedef struct NvmeParams {
char *serial;
uint32_t num_queues;
uint32_t num_ns;
uint8_t mdts;
+ uint8_t ms;
uint32_t cmb_size_mb;
} NvmeParams;
@@ -91,6 +93,7 @@ typedef struct NvmeNamespace {
uint32_t id;
uint64_t ns_blks;
uint64_t blk_offset;
+ uint64_t blk_offset_md;
} NvmeNamespace;
#define TYPE_NVME "nvme"
@@ -154,4 +157,10 @@ static inline size_t nvme_ns_lbads_bytes(NvmeNamespace *ns)
return 1 << nvme_ns_lbads(ns);
}
+static inline uint16_t nvme_ns_ms(NvmeNamespace *ns)
+{
+ NvmeIdNs *id = &ns->id_ns;
+ return le16_to_cpu(id->lbaf[NVME_ID_NS_FLBAS_INDEX(id->flbas)].ms);
+}
+
#endif /* HW_NVME_H */
--
2.21.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH 6/8] nvme: add support for scatter gather lists
2019-05-17 8:42 [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Klaus Birkelund Jensen
` (4 preceding siblings ...)
2019-05-17 8:42 ` [Qemu-devel] [PATCH 5/8] nvme: add support for metadata Klaus Birkelund Jensen
@ 2019-05-17 8:42 ` Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 7/8] nvme: keep a copy of the NVMe command in request Klaus Birkelund Jensen
` (2 subsequent siblings)
8 siblings, 0 replies; 15+ messages in thread
From: Klaus Birkelund Jensen @ 2019-05-17 8:42 UTC (permalink / raw)
To: qemu-block; +Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
Add partial SGL support. For now, only support a single data block or
last segment descriptor. This is in line with what, for instance, SPDK
currently supports.
Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
---
block/nvme.c | 18 ++--
hw/block/nvme.c | 242 +++++++++++++++++++++++++++++++++---------
hw/block/nvme.h | 6 ++
hw/block/trace-events | 1 +
include/block/nvme.h | 81 +++++++++++++-
5 files changed, 285 insertions(+), 63 deletions(-)
diff --git a/block/nvme.c b/block/nvme.c
index 0684bbd077dd..12d98c0d0be6 100644
--- a/block/nvme.c
+++ b/block/nvme.c
@@ -437,7 +437,7 @@ static void nvme_identify(BlockDriverState *bs, int namespace, Error **errp)
error_setg(errp, "Cannot map buffer for DMA");
goto out;
}
- cmd.prp1 = cpu_to_le64(iova);
+ cmd.dptr.prp.prp1 = cpu_to_le64(iova);
if (nvme_cmd_sync(bs, s->queues[0], &cmd)) {
error_setg(errp, "Failed to identify controller");
@@ -511,7 +511,7 @@ static bool nvme_add_io_queue(BlockDriverState *bs, Error **errp)
}
cmd = (NvmeCmd) {
.opcode = NVME_ADM_CMD_CREATE_CQ,
- .prp1 = cpu_to_le64(q->cq.iova),
+ .dptr.prp.prp1 = cpu_to_le64(q->cq.iova),
.cdw10 = cpu_to_le32(((queue_size - 1) << 16) | (n & 0xFFFF)),
.cdw11 = cpu_to_le32(0x3),
};
@@ -522,7 +522,7 @@ static bool nvme_add_io_queue(BlockDriverState *bs, Error **errp)
}
cmd = (NvmeCmd) {
.opcode = NVME_ADM_CMD_CREATE_SQ,
- .prp1 = cpu_to_le64(q->sq.iova),
+ .dptr.prp.prp1 = cpu_to_le64(q->sq.iova),
.cdw10 = cpu_to_le32(((queue_size - 1) << 16) | (n & 0xFFFF)),
.cdw11 = cpu_to_le32(0x1 | (n << 16)),
};
@@ -857,16 +857,16 @@ try_map:
case 0:
abort();
case 1:
- cmd->prp1 = pagelist[0];
- cmd->prp2 = 0;
+ cmd->dptr.prp.prp1 = pagelist[0];
+ cmd->dptr.prp.prp2 = 0;
break;
case 2:
- cmd->prp1 = pagelist[0];
- cmd->prp2 = pagelist[1];
+ cmd->dptr.prp.prp1 = pagelist[0];
+ cmd->dptr.prp.prp2 = pagelist[1];
break;
default:
- cmd->prp1 = pagelist[0];
- cmd->prp2 = cpu_to_le64(req->prp_list_iova + sizeof(uint64_t));
+ cmd->dptr.prp.prp1 = pagelist[0];
+ cmd->dptr.prp.prp2 = cpu_to_le64(req->prp_list_iova + sizeof(uint64_t));
break;
}
trace_nvme_cmd_map_qiov(s, cmd, req, qiov, entries);
diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index 675967a596d1..81201a8b4834 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -279,6 +279,96 @@ unmap:
return status;
}
+static uint16_t nvme_map_sgl(NvmeCtrl *n, QEMUSGList *qsg,
+ NvmeSglDescriptor sgl, uint32_t len, NvmeRequest *req)
+{
+ NvmeSglDescriptor *sgl_descriptors;
+ uint64_t nsgld;
+ uint16_t status = NVME_SUCCESS;
+
+ trace_nvme_map_sgl(req->cqe.cid, le64_to_cpu(sgl.generic.type), req->nlb,
+ len);
+
+ int cmb = 0;
+
+ switch (le64_to_cpu(sgl.generic.type)) {
+ case SGL_DESCR_TYPE_DATA_BLOCK:
+ sgl_descriptors = &sgl;
+ nsgld = 1;
+
+ break;
+
+ case SGL_DESCR_TYPE_LAST_SEGMENT:
+ sgl_descriptors = g_malloc0(le64_to_cpu(sgl.unkeyed.len));
+ nsgld = le64_to_cpu(sgl.unkeyed.len) / sizeof(NvmeSglDescriptor);
+
+ if (nvme_addr_is_cmb(n, sgl.addr)) {
+ cmb = 1;
+ }
+
+ nvme_addr_read(n, le64_to_cpu(sgl.addr), sgl_descriptors,
+ le64_to_cpu(sgl.unkeyed.len));
+
+ break;
+
+ default:
+ return NVME_SGL_DESCRIPTOR_TYPE_INVALID | NVME_DNR;
+ }
+
+ if (nvme_addr_is_cmb(n, le64_to_cpu(sgl_descriptors[0].addr))) {
+ if (!cmb) {
+ status = NVME_INVALID_USE_OF_CMB | NVME_DNR;
+ goto maybe_free;
+ }
+
+ req->is_cmb = true;
+ } else {
+ if (cmb) {
+ status = NVME_INVALID_USE_OF_CMB | NVME_DNR;
+ goto maybe_free;
+ }
+
+ req->is_cmb = false;
+ }
+
+ pci_dma_sglist_init(qsg, &n->parent_obj, nsgld);
+
+ for (int i = 0; i < nsgld; i++) {
+ uint64_t addr;
+ uint32_t trans_len;
+
+ if (len == 0) {
+ if (!NVME_CTRL_SGLS_EXCESS_LENGTH(n->id_ctrl.sgls)) {
+ status = NVME_DATA_SGL_LENGTH_INVALID | NVME_DNR;
+ qemu_sglist_destroy(qsg);
+ goto maybe_free;
+ }
+
+ break;
+ }
+
+ addr = le64_to_cpu(sgl_descriptors[i].addr);
+ trans_len = MIN(len, le64_to_cpu(sgl_descriptors[i].unkeyed.len));
+
+ if (req->is_cmb && !nvme_addr_is_cmb(n, addr)) {
+ status = NVME_INVALID_USE_OF_CMB | NVME_DNR;
+ qemu_sglist_destroy(qsg);
+ goto maybe_free;
+ }
+
+ qemu_sglist_add(qsg, addr, trans_len);
+
+ len -= trans_len;
+ }
+
+maybe_free:
+ if (nsgld > 1) {
+ g_free(sgl_descriptors);
+ }
+
+ return status;
+}
+
static void dma_to_cmb(NvmeCtrl *n, QEMUSGList *qsg, QEMUIOVector *iov)
{
for (int i = 0; i < qsg->nsg; i++) {
@@ -324,6 +414,56 @@ static uint16_t nvme_dma_read_prp(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
return err;
}
+static uint16_t nvme_dma_read_sgl(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
+ NvmeSglDescriptor sgl, NvmeCmd *cmd, NvmeRequest *req)
+{
+ QEMUSGList qsg;
+ uint16_t err = NVME_SUCCESS;
+
+ err = nvme_map_sgl(n, &qsg, sgl, len, req);
+ if (err) {
+ return err;
+ }
+
+ if (req->is_cmb) {
+ QEMUIOVector iov;
+
+ qemu_iovec_init(&iov, qsg.nsg);
+ dma_to_cmb(n, &qsg, &iov);
+
+ if (unlikely(qemu_iovec_from_buf(&iov, 0, ptr, len) != len)) {
+ trace_nvme_err_invalid_dma();
+ err = NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ qemu_iovec_destroy(&iov);
+
+ return err;
+ }
+
+ if (unlikely(dma_buf_read(ptr, len, &qsg))) {
+ trace_nvme_err_invalid_dma();
+ err = NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ qemu_sglist_destroy(&qsg);
+
+ return err;
+}
+
+static uint16_t nvme_dma_read(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
+ NvmeCmd *cmd, NvmeRequest *req)
+{
+ if (cmd->psdt) {
+ return nvme_dma_read_sgl(n, ptr, len, cmd->dptr.sgl, cmd, req);
+ }
+
+ uint64_t prp1 = le64_to_cpu(cmd->dptr.prp.prp1);
+ uint64_t prp2 = le64_to_cpu(cmd->dptr.prp.prp2);
+
+ return nvme_dma_read_prp(n, ptr, len, prp1, prp2, req);
+}
+
static void nvme_blk_req_destroy(NvmeBlockBackendRequest *blk_req)
{
if (blk_req->qsg.nalloc) {
@@ -385,17 +525,26 @@ static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
uint16_t err;
QEMUSGList qsg;
+ NvmeSglDescriptor sgl;
uint32_t unit_len = nvme_ns_lbads_bytes(ns);
uint32_t len = req->nlb * unit_len;
uint32_t meta_unit_len = nvme_ns_ms(ns);
uint32_t meta_len = req->nlb * meta_unit_len;
- uint64_t prp1 = le64_to_cpu(cmd->prp1);
- uint64_t prp2 = le64_to_cpu(cmd->prp2);
- err = nvme_map_prp(n, &qsg, prp1, prp2, len, req);
- if (err) {
- return err;
+ if (cmd->psdt) {
+ err = nvme_map_sgl(n, &qsg, cmd->dptr.sgl, len, req);
+ if (err) {
+ return err;
+ }
+ } else {
+ uint64_t prp1 = le64_to_cpu(cmd->dptr.prp.prp1);
+ uint64_t prp2 = le64_to_cpu(cmd->dptr.prp.prp2);
+
+ err = nvme_map_prp(n, &qsg, prp1, prp2, len, req);
+ if (err) {
+ return err;
+ }
}
err = nvme_blk_setup(n, ns, &qsg, ns->blk_offset, unit_len, req);
@@ -407,7 +556,25 @@ static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
qsg.size = 0;
if (cmd->mptr && n->params.ms) {
- qemu_sglist_add(&qsg, le64_to_cpu(cmd->mptr), meta_len);
+ if (cmd->psdt == PSDT_SGL_MPTR_SGL) {
+ nvme_addr_read(n, le64_to_cpu(cmd->mptr), &sgl,
+ sizeof(NvmeSglDescriptor));
+
+ err = nvme_map_sgl(n, &qsg, sgl, meta_len, req);
+ if (err) {
+ /*
+ * nvme_map_sgl does not know if it was mapping a data or meta
+ * data SGL, so fix the error code if needed.
+ */
+ if (nvme_is_error(err, NVME_DATA_SGL_LENGTH_INVALID)) {
+ err = NVME_METADATA_SGL_LENGTH_INVALID | NVME_DNR;
+ }
+
+ return err;
+ }
+ } else {
+ qemu_sglist_add(&qsg, le64_to_cpu(cmd->mptr), meta_len);
+ }
err = nvme_blk_setup(n, ns, &qsg, ns->blk_offset_md, meta_unit_len,
req);
@@ -416,8 +583,6 @@ static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
}
}
- qemu_sglist_destroy(&qsg);
-
return NVME_SUCCESS;
}
@@ -978,25 +1143,18 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeCmd *cmd)
return NVME_SUCCESS;
}
-static uint16_t nvme_identify_ctrl(NvmeCtrl *n, NvmeIdentify *c,
- NvmeRequest *req)
+static uint16_t nvme_identify_ctrl(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
- uint64_t prp1 = le64_to_cpu(c->prp1);
- uint64_t prp2 = le64_to_cpu(c->prp2);
-
trace_nvme_identify_ctrl();
- return nvme_dma_read_prp(n, (uint8_t *)&n->id_ctrl, sizeof(n->id_ctrl),
- prp1, prp2, req);
+ return nvme_dma_read(n, (uint8_t *) &n->id_ctrl, sizeof(n->id_ctrl), cmd,
+ req);
}
-static uint16_t nvme_identify_ns(NvmeCtrl *n, NvmeIdentify *c,
- NvmeRequest *req)
+static uint16_t nvme_identify_ns(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
NvmeNamespace *ns;
- uint32_t nsid = le32_to_cpu(c->nsid);
- uint64_t prp1 = le64_to_cpu(c->prp1);
- uint64_t prp2 = le64_to_cpu(c->prp2);
+ uint32_t nsid = le32_to_cpu(cmd->nsid);
trace_nvme_identify_ns(nsid);
@@ -1007,17 +1165,15 @@ static uint16_t nvme_identify_ns(NvmeCtrl *n, NvmeIdentify *c,
ns = &n->namespaces[nsid - 1];
- return nvme_dma_read_prp(n, (uint8_t *)&ns->id_ns, sizeof(ns->id_ns),
- prp1, prp2, req);
+ return nvme_dma_read(n, (uint8_t *) &ns->id_ns, sizeof(ns->id_ns), cmd,
+ req);
}
-static uint16_t nvme_identify_ns_list(NvmeCtrl *n, NvmeIdentify *c,
+static uint16_t nvme_identify_ns_list(NvmeCtrl *n, NvmeCmd *cmd,
NvmeRequest *req)
{
static const int data_len = 4 * KiB;
- uint32_t min_nsid = le32_to_cpu(c->nsid);
- uint64_t prp1 = le64_to_cpu(c->prp1);
- uint64_t prp2 = le64_to_cpu(c->prp2);
+ uint32_t min_nsid = le32_to_cpu(cmd->nsid);
uint32_t *list;
uint16_t ret;
int i, j = 0;
@@ -1034,18 +1190,16 @@ static uint16_t nvme_identify_ns_list(NvmeCtrl *n, NvmeIdentify *c,
break;
}
}
- ret = nvme_dma_read_prp(n, (uint8_t *)list, data_len, prp1, prp2, req);
+ ret = nvme_dma_read(n, (uint8_t *) list, data_len, cmd, req);
g_free(list);
return ret;
}
-static uint16_t nvme_identify_ns_descriptor_list(NvmeCtrl *n, NvmeCmd *c,
+static uint16_t nvme_identify_ns_descriptor_list(NvmeCtrl *n, NvmeCmd *cmd,
NvmeRequest *req)
{
static const int data_len = sizeof(NvmeIdentifyNamespaceDescriptor) + 0x10;
- uint32_t nsid = le32_to_cpu(c->nsid);
- uint64_t prp1 = le64_to_cpu(c->prp1);
- uint64_t prp2 = le64_to_cpu(c->prp2);
+ uint32_t nsid = le32_to_cpu(cmd->nsid);
NvmeIdentifyNamespaceDescriptor *list;
uint16_t ret;
@@ -1055,7 +1209,7 @@ static uint16_t nvme_identify_ns_descriptor_list(NvmeCtrl *n, NvmeCmd *c,
list->nidt = 0x3;
list->nidl = 0x10;
- ret = nvme_dma_read_prp(n, (uint8_t *) list, data_len, prp1, prp2, req);
+ ret = nvme_dma_read(n, (uint8_t *) list, data_len, cmd, req);
g_free(list);
return ret;
}
@@ -1066,11 +1220,11 @@ static uint16_t nvme_identify(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
switch (le32_to_cpu(c->cns)) {
case 0x00:
- return nvme_identify_ns(n, c, req);
+ return nvme_identify_ns(n, cmd, req);
case 0x01:
- return nvme_identify_ctrl(n, c, req);
+ return nvme_identify_ctrl(n, cmd, req);
case 0x02:
- return nvme_identify_ns_list(n, c, req);
+ return nvme_identify_ns_list(n, cmd, req);
case 0x03:
return nvme_identify_ns_descriptor_list(n, cmd, req);
default:
@@ -1199,8 +1353,6 @@ static uint16_t nvme_error_log_info(NvmeCtrl *n, NvmeCmd *cmd, uint8_t rae,
uint32_t buf_len, uint64_t off, NvmeRequest *req)
{
uint32_t trans_len;
- uint64_t prp1 = le64_to_cpu(cmd->prp1);
- uint64_t prp2 = le64_to_cpu(cmd->prp2);
if (off > sizeof(*n->elpes) * (NVME_ELPE + 1)) {
return NVME_INVALID_FIELD | NVME_DNR;
@@ -1212,16 +1364,12 @@ static uint16_t nvme_error_log_info(NvmeCtrl *n, NvmeCmd *cmd, uint8_t rae,
nvme_clear_events(n, NVME_AER_TYPE_ERROR);
}
- return nvme_dma_read_prp(n, (uint8_t *) n->elpes + off, trans_len, prp1,
- prp2, req);
+ return nvme_dma_read(n, (uint8_t *) n->elpes + off, trans_len, cmd, req);
}
static uint16_t nvme_smart_info(NvmeCtrl *n, NvmeCmd *cmd, uint8_t rae,
uint32_t buf_len, uint64_t off, NvmeRequest *req)
{
- uint64_t prp1 = le64_to_cpu(cmd->prp1);
- uint64_t prp2 = le64_to_cpu(cmd->prp2);
-
uint32_t trans_len;
time_t current_ms;
NvmeSmartLog smart;
@@ -1255,16 +1403,13 @@ static uint16_t nvme_smart_info(NvmeCtrl *n, NvmeCmd *cmd, uint8_t rae,
nvme_clear_events(n, NVME_AER_TYPE_SMART);
}
- return nvme_dma_read_prp(n, (uint8_t *) &smart + off, trans_len, prp1,
- prp2, req);
+ return nvme_dma_read(n, (uint8_t *) &smart + off, trans_len, cmd, req);
}
static uint16_t nvme_fw_log_info(NvmeCtrl *n, NvmeCmd *cmd, uint32_t buf_len,
uint64_t off, NvmeRequest *req)
{
uint32_t trans_len;
- uint64_t prp1 = le64_to_cpu(cmd->prp1);
- uint64_t prp2 = le64_to_cpu(cmd->prp2);
NvmeFwSlotInfoLog fw_log;
if (off > sizeof(fw_log)) {
@@ -1275,8 +1420,7 @@ static uint16_t nvme_fw_log_info(NvmeCtrl *n, NvmeCmd *cmd, uint32_t buf_len,
trans_len = MIN(sizeof(fw_log) - off, buf_len);
- return nvme_dma_read_prp(n, (uint8_t *) &fw_log + off, trans_len, prp1,
- prp2, req);
+ return nvme_dma_read(n, (uint8_t *) &fw_log + off, trans_len, cmd, req);
}
@@ -2058,7 +2202,7 @@ static void nvme_init_ctrl(NvmeCtrl *n)
}
id->awun = cpu_to_le16(0);
id->awupf = cpu_to_le16(0);
- id->sgls = cpu_to_le32(0);
+ id->sgls = cpu_to_le32(params->ms ? 0xa00001 : 0x1);
strcpy((char *) id->subnqn, "nqn.2014-08.org.nvmexpress:uuid:");
qemu_uuid_unparse(&qemu_uuid,
diff --git a/hw/block/nvme.h b/hw/block/nvme.h
index 81ee0c5173d5..70f4781a1b61 100644
--- a/hw/block/nvme.h
+++ b/hw/block/nvme.h
@@ -146,6 +146,12 @@ static inline bool nvme_rw_is_write(NvmeRequest *req)
return req->cmd_opcode == NVME_CMD_WRITE;
}
+static inline bool nvme_is_error(uint16_t status, uint16_t err)
+{
+ /* strip DNR and MORE */
+ return (status & 0xfff) == err;
+}
+
static inline uint8_t nvme_ns_lbads(NvmeNamespace *ns)
{
NvmeIdNs *id = &ns->id_ns;
diff --git a/hw/block/trace-events b/hw/block/trace-events
index 56fec40d130c..3324aac41dbb 100644
--- a/hw/block/trace-events
+++ b/hw/block/trace-events
@@ -36,6 +36,7 @@ nvme_irq_pin(void) "pulsing IRQ pin"
nvme_irq_masked(void) "IRQ is masked"
nvme_dma_read(uint64_t prp1, uint64_t prp2) "DMA read, prp1=0x%"PRIx64" prp2=0x%"PRIx64""
nvme_map_prp(uint8_t cmd_opcode, uint64_t trans_len, uint32_t len, uint64_t prp1, uint64_t prp2, int num_prps) "cmd_opcode=0x%"PRIx8", trans_len=%"PRIu64", len=%"PRIu32", prp1=0x%"PRIx64", prp2=0x%"PRIx64", num_prps=%d"
+nvme_map_sgl(uint16_t cid, uint64_t typ, uint16_t nlb, uint64_t len) "cid %"PRIu16" type %"PRIu64" nlb %"PRIu16" len %"PRIu64""
nvme_rw(const char *verb, uint32_t blk_count, uint64_t byte_count, uint64_t lba) "%s %"PRIu32" blocks (%"PRIu64" bytes) from LBA %"PRIu64""
nvme_rw_cb(uint16_t cid, uint32_t nsid) "cid %"PRIu16" nsid %"PRIu32""
nvme_create_sq(uint64_t addr, uint16_t sqid, uint16_t cqid, uint16_t qsize, uint16_t qflags) "create submission queue, addr=0x%"PRIx64", sqid=%"PRIu16", cqid=%"PRIu16", qsize=%"PRIu16", qflags=%"PRIu16""
diff --git a/include/block/nvme.h b/include/block/nvme.h
index c90c36b66971..583b61a76570 100644
--- a/include/block/nvme.h
+++ b/include/block/nvme.h
@@ -205,15 +205,71 @@ enum NvmeCmbszMask {
#define NVME_CMBSZ_GETSIZE(cmbsz) \
(NVME_CMBSZ_SZ(cmbsz) * (1 << (12 + 4 * NVME_CMBSZ_SZU(cmbsz))))
+enum NvmeSglDescriptorType {
+ SGL_DESCR_TYPE_DATA_BLOCK = 0x0,
+ SGL_DESCR_TYPE_BIT_BUCKET = 0x1,
+ SGL_DESCR_TYPE_SEGMENT = 0x2,
+ SGL_DESCR_TYPE_LAST_SEGMENT = 0x3,
+ SGL_DESCR_TYPE_KEYED_DATA_BLOCK = 0x4,
+
+ SGL_DESCR_TYPE_VENDOR_SPECIFIC = 0xf,
+};
+
+enum NvmeSglDescriptorSubtype {
+ SGL_DESCR_SUBTYPE_ADDRESS = 0x0,
+ SGL_DESCR_SUBTYPE_OFFSET = 0x1,
+};
+
+typedef struct NvmeSglDescriptor {
+ uint64_t addr;
+
+ union {
+ struct {
+ uint64_t rsvd:56;
+ uint64_t subtype:4;
+ uint64_t type:4;
+ } generic;
+ struct {
+ uint64_t len:32;
+ uint64_t rsvd:24;
+ uint64_t subtype:4;
+ uint64_t type:4;
+ } unkeyed;
+
+ struct {
+ uint64_t len:24;
+ uint64_t key:32;
+ uint64_t subtype:4;
+ uint64_t type:4;
+ } keyed;
+ };
+} NvmeSglDescriptor;
+
+typedef union NvmeCmdDptr {
+ struct {
+ uint64_t prp1;
+ uint64_t prp2;
+ } prp;
+
+ NvmeSglDescriptor sgl;
+} NvmeCmdDptr;
+
+enum NvmePsdt {
+ PSDT_PRP = 0x0,
+ PSDT_SGL_MPTR_CONTIGUOUS = 0x1,
+ PSDT_SGL_MPTR_SGL = 0x2,
+};
+
typedef struct NvmeCmd {
- uint8_t opcode;
- uint8_t fuse;
+ uint16_t opcode:8;
+ uint16_t fuse:2;
+ uint16_t rsvd1:4;
+ uint16_t psdt:2;
uint16_t cid;
uint32_t nsid;
- uint64_t res1;
+ uint64_t rsvd2;
uint64_t mptr;
- uint64_t prp1;
- uint64_t prp2;
+ NvmeCmdDptr dptr;
uint32_t cdw10;
uint32_t cdw11;
uint32_t cdw12;
@@ -427,6 +483,11 @@ enum NvmeStatusCodes {
NVME_CMD_ABORT_MISSING_FUSE = 0x000a,
NVME_INVALID_NSID = 0x000b,
NVME_CMD_SEQ_ERROR = 0x000c,
+ NVME_INVALID_SGL_SEG_DESCRIPTOR = 0x000d,
+ NVME_INVALID_NUM_SGL_DESCRIPTORS = 0x000e,
+ NVME_DATA_SGL_LENGTH_INVALID = 0x000f,
+ NVME_METADATA_SGL_LENGTH_INVALID = 0x0010,
+ NVME_SGL_DESCRIPTOR_TYPE_INVALID = 0x0011,
NVME_INVALID_USE_OF_CMB = 0x0012,
NVME_LBA_RANGE = 0x0080,
NVME_CAP_EXCEEDED = 0x0081,
@@ -629,6 +690,16 @@ enum NvmeIdCtrlOncs {
#define NVME_CTRL_CQES_MIN(cqes) ((cqes) & 0xf)
#define NVME_CTRL_CQES_MAX(cqes) (((cqes) >> 4) & 0xf)
+#define NVME_CTRL_SGLS_SUPPORTED(sgls) ((sgls) & 0x3)
+#define NVME_CTRL_SGLS_SUPPORTED_NO_ALIGNMENT(sgls) ((sgls) & (0x1 << 0))
+#define NVME_CTRL_SGLS_SUPPORTED_DWORD_ALIGNMENT(sgls) ((sgls) & (0x1 << 1))
+#define NVME_CTRL_SGLS_KEYED(sgls) ((sgls) & (0x1 << 2))
+#define NVME_CTRL_SGLS_BITBUCKET(sgls) ((sgls) & (0x1 << 16))
+#define NVME_CTRL_SGLS_MPTR_CONTIGUOUS(sgls) ((sgls) & (0x1 << 17))
+#define NVME_CTRL_SGLS_EXCESS_LENGTH(sgls) ((sgls) & (0x1 << 18))
+#define NVME_CTRL_SGLS_MPTR_SGL(sgls) ((sgls) & (0x1 << 19))
+#define NVME_CTRL_SGLS_ADDR_OFFSET(sgls) ((sgls) & (0x1 << 20))
+
typedef struct NvmeFeatureVal {
uint32_t arbitration;
uint32_t power_mgmt;
--
2.21.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH 7/8] nvme: keep a copy of the NVMe command in request
2019-05-17 8:42 [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Klaus Birkelund Jensen
` (5 preceding siblings ...)
2019-05-17 8:42 ` [Qemu-devel] [PATCH 6/8] nvme: add support for scatter gather lists Klaus Birkelund Jensen
@ 2019-05-17 8:42 ` Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 8/8] nvme: add an OpenChannel 2.0 NVMe device (ocssd) Klaus Birkelund Jensen
2019-05-20 13:01 ` [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Kevin Wolf
8 siblings, 0 replies; 15+ messages in thread
From: Klaus Birkelund Jensen @ 2019-05-17 8:42 UTC (permalink / raw)
To: qemu-block; +Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
---
hw/block/nvme.c | 4 ++--
hw/block/nvme.h | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index 81201a8b4834..5cd593806701 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -184,7 +184,7 @@ static uint16_t nvme_map_prp(NvmeCtrl *n, QEMUSGList *qsg, uint64_t prp1,
int num_prps = (len >> n->page_bits) + 1;
uint16_t status = NVME_SUCCESS;
- trace_nvme_map_prp(req->cmd_opcode, trans_len, len, prp1, prp2, num_prps);
+ trace_nvme_map_prp(req->cmd.opcode, trans_len, len, prp1, prp2, num_prps);
if (unlikely(!prp1)) {
trace_nvme_err_invalid_prp();
@@ -1559,7 +1559,7 @@ static void nvme_init_req(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
memset(&req->cqe, 0, sizeof(req->cqe));
req->cqe.cid = le16_to_cpu(cmd->cid);
- req->cmd_opcode = cmd->opcode;
+ memcpy(&req->cmd, cmd, sizeof(NvmeCmd));
req->is_cmb = false;
req->status = NVME_SUCCESS;
diff --git a/hw/block/nvme.h b/hw/block/nvme.h
index 70f4781a1b61..7e1e026d90e6 100644
--- a/hw/block/nvme.h
+++ b/hw/block/nvme.h
@@ -52,7 +52,7 @@ typedef struct NvmeRequest {
uint16_t status;
bool is_cmb;
bool is_write;
- uint8_t cmd_opcode;
+ NvmeCmd cmd;
QTAILQ_HEAD(, NvmeBlockBackendRequest) blk_req_tailq;
QTAILQ_ENTRY(NvmeRequest)entry;
@@ -143,7 +143,7 @@ typedef struct NvmeCtrl {
static inline bool nvme_rw_is_write(NvmeRequest *req)
{
- return req->cmd_opcode == NVME_CMD_WRITE;
+ return req->cmd.opcode == NVME_CMD_WRITE;
}
static inline bool nvme_is_error(uint16_t status, uint16_t err)
--
2.21.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Qemu-devel] [PATCH 8/8] nvme: add an OpenChannel 2.0 NVMe device (ocssd)
2019-05-17 8:42 [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Klaus Birkelund Jensen
` (6 preceding siblings ...)
2019-05-17 8:42 ` [Qemu-devel] [PATCH 7/8] nvme: keep a copy of the NVMe command in request Klaus Birkelund Jensen
@ 2019-05-17 8:42 ` Klaus Birkelund Jensen
2019-05-20 16:45 ` Eric Blake
2019-05-20 13:01 ` [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Kevin Wolf
8 siblings, 1 reply; 15+ messages in thread
From: Klaus Birkelund Jensen @ 2019-05-17 8:42 UTC (permalink / raw)
To: qemu-block; +Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
This adds a new 'ocssd' block device that emulates an OpenChannel 2.0
device. The device is backed by a new 'ocssd' block backend that is
based on the raw format driver but includes a header that holds the
device geometry and write data requirements. This new block backend is
special in that the size is not specified explicitly but in terms of
sector size, number of chunks, number of parallel units, etc. This
called for the addition of the `no_size_required` field in `struct
BlockDriver` to not fail image creation when the size parameter is
missing.
The ocssd device is an individual device but shares a lot of code with
the nvme device. Thus, some core functionality of nvme/nvme.c has been
exported for use by nvme/ocssd.c.
Thank you to the following people for their contributions to the
original qemu-nvme (github.com/OpenChannelSSD/qemu-nvme) implementation.
Matias Bjørling <mb@lightnvm.io>
Javier González <javier@javigon.com>
Simon Andreas Frimann Lund <ocssd@safl.dk>
Hans Holmberg <hans@owltronix.com>
Jesper Devantier <contact@pseudonymous.me>
Young Tack Jin <youngtack.jin@circuitblvd.com>
Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
---
MAINTAINERS | 14 +-
Makefile.objs | 1 +
block.c | 2 +-
block/Makefile.objs | 2 +-
block/nvme.c | 2 +-
block/ocssd.c | 690 ++++++++
hw/block/Makefile.objs | 2 +-
hw/block/{ => nvme}/nvme.c | 192 ++-
hw/block/nvme/ocssd.c | 2647 +++++++++++++++++++++++++++++++
hw/block/nvme/ocssd.h | 140 ++
hw/block/nvme/trace-events | 136 ++
hw/block/trace-events | 109 --
include/block/block_int.h | 3 +
include/block/nvme.h | 12 +-
include/block/ocssd.h | 231 +++
{hw => include/hw}/block/nvme.h | 61 +
include/hw/pci/pci_ids.h | 2 +
qapi/block-core.json | 47 +-
18 files changed, 4121 insertions(+), 172 deletions(-)
create mode 100644 block/ocssd.c
rename hw/block/{ => nvme}/nvme.c (94%)
create mode 100644 hw/block/nvme/ocssd.c
create mode 100644 hw/block/nvme/ocssd.h
create mode 100644 hw/block/nvme/trace-events
create mode 100644 include/block/ocssd.h
rename {hw => include/hw}/block/nvme.h (63%)
diff --git a/MAINTAINERS b/MAINTAINERS
index a73a61a54654..c2e89fbabeac 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1561,9 +1561,20 @@ nvme
M: Keith Busch <keith.busch@intel.com>
L: qemu-block@nongnu.org
S: Supported
-F: hw/block/nvme*
+F: hw/block/nvme/nvme.c
+F: include/block/nvme.h
+F: include/hw/block/nvme/nvme.h
F: tests/nvme-test.c
+ocssd
+M: Klaus Birkelund <klaus@birkelund.eu>
+L: qemu-block@nongnu.org
+S: Supported
+F: block/ocssd.c
+F: include/block/ocssd.h
+F: hw/block/nvme/ocssd.*
+T: git https://github.com/birkelund/qemu.git ocssd
+
megasas
M: Hannes Reinecke <hare@suse.com>
L: qemu-block@nongnu.org
@@ -2434,6 +2445,7 @@ M: Fam Zheng <fam@euphon.net>
L: qemu-block@nongnu.org
S: Supported
F: block/nvme*
+F: include/block/nvme.h
Bootdevice
M: Gonglei <arei.gonglei@huawei.com>
diff --git a/Makefile.objs b/Makefile.objs
index cf065de5ed44..32b75635dcba 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -139,6 +139,7 @@ trace-events-subdirs += hw/arm
trace-events-subdirs += hw/audio
trace-events-subdirs += hw/block
trace-events-subdirs += hw/block/dataplane
+trace-events-subdirs += hw/block/nvme
trace-events-subdirs += hw/char
trace-events-subdirs += hw/display
trace-events-subdirs += hw/dma
diff --git a/block.c b/block.c
index 6999aad4460b..a9189b090c5f 100644
--- a/block.c
+++ b/block.c
@@ -5617,7 +5617,7 @@ void bdrv_img_create(const char *filename, const char *fmt,
}
} /* (backing_file && !(flags & BDRV_O_NO_BACKING)) */
- if (size == -1) {
+ if (size == -1 && !drv->no_size_required) {
error_setg(errp, "Image creation needs a size parameter");
goto out;
}
diff --git a/block/Makefile.objs b/block/Makefile.objs
index 7a81892a5281..4fc56cae3a14 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -1,4 +1,4 @@
-block-obj-y += raw-format.o vmdk.o vpc.o
+block-obj-y += raw-format.o ocssd.o vmdk.o vpc.o
block-obj-$(CONFIG_QCOW1) += qcow.o
block-obj-$(CONFIG_VDI) += vdi.o
block-obj-$(CONFIG_CLOOP) += cloop.o
diff --git a/block/nvme.c b/block/nvme.c
index 12d98c0d0be6..edb077ef17d5 100644
--- a/block/nvme.c
+++ b/block/nvme.c
@@ -278,7 +278,7 @@ static inline int nvme_translate_error(const NvmeCqe *c)
{
uint16_t status = (le16_to_cpu(c->status) >> 1) & 0xFF;
if (status) {
- trace_nvme_error(le32_to_cpu(c->result),
+ trace_nvme_error(le32_to_cpu(c->cdw0),
le16_to_cpu(c->sq_head),
le16_to_cpu(c->sq_id),
le16_to_cpu(c->cid),
diff --git a/block/ocssd.c b/block/ocssd.c
new file mode 100644
index 000000000000..260f9f81b3a2
--- /dev/null
+++ b/block/ocssd.c
@@ -0,0 +1,690 @@
+/*
+ * BlockDriver implementation for "ocssd" format driver
+ *
+ * Based on the "raw" format driver (raw-format.c).
+ *
+ * Copyright (c) 2019 CNEX Labs, Inc.
+ * Copyright (C) 2010-2016 Red Hat, Inc.
+ * Copyright (C) 2010, Blue Swirl <blauwirbel@gmail.com>
+ * Copyright (C) 2009, Anthony Liguori <aliguori@us.ibm.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/option.h"
+#include "sysemu/block-backend.h"
+#include "qapi/error.h"
+
+#include "hw/block/nvme/ocssd.h"
+
+typedef struct BDRVOcssdState {
+ OcssdFormatHeader hdr;
+ OcssdIdentity *namespaces;
+ uint64_t size, real_size;
+} BDRVOcssdState;
+
+static QemuOptsList ocssd_create_opts = {
+ .name = "ocssd-create-opts",
+ .head = QTAILQ_HEAD_INITIALIZER(ocssd_create_opts.head),
+ .desc = {
+ {
+ .name = "num_grp",
+ .type = QEMU_OPT_NUMBER,
+ .help = "number of groups (default: 2)"
+ },
+ {
+ .name = "num_pu",
+ .type = QEMU_OPT_NUMBER,
+ .help = "number of parallel units per group (default: 4)"
+ },
+ {
+ .name = "num_chk",
+ .type = QEMU_OPT_NUMBER,
+ .help = "number of chunks per parallel unit (defaut: 60)"
+ },
+ {
+ .name = "num_sec",
+ .type = QEMU_OPT_NUMBER,
+ .help = "number of sectors per chunk (default: 4096)"
+ },
+ {
+ .name = "sec_size",
+ .type = QEMU_OPT_SIZE,
+ .help = "sector size (default: 4096)"
+ },
+ {
+ .name = "md_size",
+ .type = QEMU_OPT_SIZE,
+ .help = "metadata size (default: 16)"
+ },
+ {
+ .name = "num_ns",
+ .type = QEMU_OPT_NUMBER,
+ .help = "number of namespaces (default: 1)",
+ },
+ {
+ .name = "mccap",
+ .type = QEMU_OPT_NUMBER,
+ .help = "media and controller capabilities (default: 0x1)",
+ },
+ {
+ .name = "wit",
+ .type = QEMU_OPT_NUMBER,
+ .help = "wear-level index delta threshold (default: 10)",
+ },
+ {
+ .name = "ws_min",
+ .type = QEMU_OPT_NUMBER,
+ .help = "minimum write size (default: 4)",
+ },
+ {
+ .name = "ws_opt",
+ .type = QEMU_OPT_NUMBER,
+ .help = "optimal write size (default: 8)",
+ },
+ {
+ .name = "mw_cunits",
+ .type = QEMU_OPT_NUMBER,
+ .help = "cache minimum write size units (default: 24)",
+ },
+ {
+ .name = "pe_cycles",
+ .type = QEMU_OPT_NUMBER,
+ .help = "program/erase cycles per chunk (default: 1000)",
+ },
+ { /* end of list */ }
+ }
+};
+
+static int ocssd_reopen_prepare(BDRVReopenState *reopen_state,
+ BlockReopenQueue *queue, Error **errp)
+{
+ assert(reopen_state != NULL);
+ assert(reopen_state->bs != NULL);
+ return 0;
+}
+
+static int coroutine_fn ocssd_co_preadv(BlockDriverState *bs, uint64_t offset,
+ uint64_t bytes, QEMUIOVector *qiov, int flags)
+{
+ BDRVOcssdState *s = bs->opaque;
+
+ BLKDBG_EVENT(bs->file, BLKDBG_READ_AIO);
+
+ /*
+ * Return predefined (deterministic) data for reads that are out of bounds
+ * of the real physical size of the device. See ocssd_getlength for why
+ * reads might be issued out of bounds.
+ */
+ if (offset > s->real_size || s->real_size - offset < bytes) {
+ return qemu_iovec_memset(qiov, 0, 0x0, bytes);
+ }
+
+ return bdrv_co_preadv(bs->file, offset, bytes, qiov, flags);
+}
+
+static int coroutine_fn ocssd_co_pwritev(BlockDriverState *bs, uint64_t offset,
+ uint64_t bytes, QEMUIOVector *qiov, int flags)
+{
+ void *buf = NULL;
+ BlockDriver *drv;
+ QEMUIOVector local_qiov;
+ int ret;
+
+ if (bs->probed && offset < BLOCK_PROBE_BUF_SIZE && bytes) {
+ /*
+ * Handling partial writes would be a pain - so we just
+ * require that guests have 512-byte request alignment if
+ * probing occurred
+ */
+ QEMU_BUILD_BUG_ON(BLOCK_PROBE_BUF_SIZE != 512);
+ QEMU_BUILD_BUG_ON(BDRV_SECTOR_SIZE != 512);
+ assert(offset == 0 && bytes >= BLOCK_PROBE_BUF_SIZE);
+
+ buf = qemu_try_blockalign(bs->file->bs, 512);
+ if (!buf) {
+ ret = -ENOMEM;
+ goto fail;
+ }
+
+ ret = qemu_iovec_to_buf(qiov, 0, buf, 512);
+ if (ret != 512) {
+ ret = -EINVAL;
+ goto fail;
+ }
+
+ drv = bdrv_probe_all(buf, 512, NULL);
+ if (drv != bs->drv) {
+ ret = -EPERM;
+ goto fail;
+ }
+
+ /*
+ * Use the checked buffer, a malicious guest might be overwriting its
+ * original buffer in the background.
+ */
+ qemu_iovec_init(&local_qiov, qiov->niov + 1);
+ qemu_iovec_add(&local_qiov, buf, 512);
+ qemu_iovec_concat(&local_qiov, qiov, 512, qiov->size - 512);
+ qiov = &local_qiov;
+ }
+
+ BLKDBG_EVENT(bs->file, BLKDBG_WRITE_AIO);
+ ret = bdrv_co_pwritev(bs->file, offset, bytes, qiov, flags);
+
+fail:
+ if (qiov == &local_qiov) {
+ qemu_iovec_destroy(&local_qiov);
+ }
+ qemu_vfree(buf);
+ return ret;
+}
+
+static int coroutine_fn ocssd_co_block_status(BlockDriverState *bs,
+ bool want_zero, int64_t offset, int64_t bytes, int64_t *pnum, int64_t *map,
+ BlockDriverState **file)
+{
+ *pnum = bytes;
+ *file = bs->file->bs;
+ *map = offset;
+ return BDRV_BLOCK_RAW | BDRV_BLOCK_OFFSET_VALID;
+}
+
+static int coroutine_fn ocssd_co_pwrite_zeroes(BlockDriverState *bs,
+ int64_t offset, int bytes, BdrvRequestFlags flags)
+{
+ return bdrv_co_pwrite_zeroes(bs->file, offset, bytes, flags);
+}
+
+static int coroutine_fn ocssd_co_pdiscard(BlockDriverState *bs, int64_t offset,
+ int bytes)
+{
+ return bdrv_co_pdiscard(bs->file, offset, bytes);
+}
+
+static int64_t ocssd_getlength(BlockDriverState *bs)
+{
+ /*
+ * Return the size of the full physical address space. It may be larger
+ * than the real size due to the LBA address format. For reads, predefined
+ * (deterministic) data is returned for addresses that are out of bounds of
+ * the real size (see ocssd_co_preadv).
+ */
+
+ BDRVOcssdState *s = bs->opaque;
+ return s->size;
+}
+
+static BlockMeasureInfo *ocssd_measure(QemuOpts *opts, BlockDriverState *in_bs,
+ Error **errp)
+{
+ BlockMeasureInfo *info;
+ int64_t required;
+
+ if (in_bs) {
+ required = bdrv_getlength(in_bs);
+ if (required < 0) {
+ error_setg_errno(errp, -required, "Unable to get image size");
+ return NULL;
+ }
+ } else {
+ required = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
+ BDRV_SECTOR_SIZE);
+ }
+
+ info = g_new(BlockMeasureInfo, 1);
+ info->required = required;
+
+ /* unallocated sectors count towards the file size in ocssd images */
+ info->fully_allocated = info->required;
+ return info;
+}
+
+static int ocssd_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)
+{
+ return bdrv_get_info(bs->file->bs, bdi);
+}
+
+static ImageInfoSpecificOcssdNS *ocssd_get_namespace_info(
+ OcssdIdentity *ns)
+{
+ ImageInfoSpecificOcssdNS *info = g_new0(ImageInfoSpecificOcssdNS, 1);
+
+ *info = (ImageInfoSpecificOcssdNS) {
+ .num_grp = ns->geo.num_grp,
+ .num_pu = ns->geo.num_pu,
+ .num_chk = ns->geo.num_chk,
+ .num_sec = ns->geo.clba,
+ };
+
+ return info;
+}
+
+static ImageInfoSpecific *ocssd_get_specific_info(BlockDriverState *bs,
+ Error **errp)
+{
+ BDRVOcssdState *s = bs->opaque;
+ ImageInfoSpecific *spec_info;
+ ImageInfoSpecificOcssdNSList **next;
+
+ spec_info = g_new0(ImageInfoSpecific, 1);
+ *spec_info = (ImageInfoSpecific){
+ .type = IMAGE_INFO_SPECIFIC_KIND_OCSSD,
+ .u.ocssd.data = g_new0(ImageInfoSpecificOcssd, 1),
+ };
+
+ *spec_info->u.ocssd.data = (ImageInfoSpecificOcssd){
+ .num_ns = s->hdr.num_ns,
+ .sector_size = s->hdr.sector_size,
+ .metadata_size = s->hdr.md_size,
+ };
+
+ next = &spec_info->u.ocssd.data->namespaces;
+ for (int i = 0; i < s->hdr.num_ns; i++) {
+ *next = g_new0(ImageInfoSpecificOcssdNSList, 1);
+ (*next)->value = ocssd_get_namespace_info(&s->namespaces[i]);
+ (*next)->next = NULL;
+ next = &(*next)->next;
+ }
+
+ return spec_info;
+}
+
+static void ocssd_refresh_limits(BlockDriverState *bs, Error **errp)
+{
+ if (bs->probed) {
+ /*
+ * To make it easier to protect the first sector, any probed
+ * image is restricted to read-modify-write on sub-sector
+ * operations.
+ */
+ bs->bl.request_alignment = BDRV_SECTOR_SIZE;
+ }
+}
+
+static int coroutine_fn ocssd_co_truncate(BlockDriverState *bs, int64_t offset,
+ PreallocMode prealloc, Error **errp)
+{
+ return bdrv_co_truncate(bs->file, offset, prealloc, errp);
+}
+
+static void ocssd_eject(BlockDriverState *bs, bool eject_flag)
+{
+ bdrv_eject(bs->file->bs, eject_flag);
+}
+
+static void ocssd_lock_medium(BlockDriverState *bs, bool locked)
+{
+ bdrv_lock_medium(bs->file->bs, locked);
+}
+
+static int ocssd_co_ioctl(BlockDriverState *bs, unsigned long int req,
+ void *buf)
+{
+ return bdrv_co_ioctl(bs->file->bs, req, buf);
+}
+
+static int ocssd_has_zero_init(BlockDriverState *bs)
+{
+ return bdrv_has_zero_init(bs->file->bs);
+}
+
+static int coroutine_fn ocssd_co_create_opts(const char *filename,
+ QemuOpts *opts, Error **errp)
+{
+ BlockBackend *blk = NULL;
+ BlockDriverState *bs = NULL;
+ Error *local_err = NULL;
+ OcssdFormatHeader hdr;
+ OcssdIdentity id;
+ OcssdIdLBAF lbaf;
+ OcssdAddrF addrf;
+ OcssdChunkDescriptor *chk;
+ OcssdChunkAcctDescriptor *acct;
+ uint8_t wit;
+ uint16_t num_grp, num_pu;
+ uint32_t num_chk, num_sec, mccap, ws_min, ws_opt, mw_cunits, pe_cycles;
+ uint64_t sec_size, md_size, num_ns, logpage_size, acct_size;
+ uint64_t chks_total, secs_total, usable_size, ns_size, size, offset;
+ int ret;
+
+ num_grp = qemu_opt_get_number(opts, "num_grp", 2);
+ num_pu = qemu_opt_get_number(opts, "num_pu", 8);
+ num_chk = qemu_opt_get_number(opts, "num_chk", 60);
+ num_sec = qemu_opt_get_number(opts, "num_sec", 4096);
+ num_ns = qemu_opt_get_number(opts, "num_ns", 1);
+ mccap = qemu_opt_get_number(opts, "mccap", 0x1);
+ wit = qemu_opt_get_number(opts, "wit", 10);
+ ws_min = qemu_opt_get_number(opts, "ws_min", 4);
+ ws_opt = qemu_opt_get_number(opts, "ws_opt", 8);
+ mw_cunits = qemu_opt_get_number(opts, "mw_cunits", 24);
+ pe_cycles = qemu_opt_get_number(opts, "pe_cycles", 1000);
+
+ sec_size = qemu_opt_get_size(opts, "sec_size", 4096);
+ md_size = qemu_opt_get_size(opts, "md_size", 16);
+
+ chks_total = num_grp * num_pu * num_chk;
+ logpage_size = QEMU_ALIGN_UP(chks_total * sizeof(OcssdChunkDescriptor),
+ sec_size);
+ acct_size = QEMU_ALIGN_UP(chks_total * sizeof(OcssdChunkAcctDescriptor),
+ sec_size);
+
+ secs_total = chks_total * num_sec;
+
+ /*
+ * The ocssd format begins with a 4k format header. Namespaces are laid out
+ * contiguously in sections after the header.
+ *
+ * A namespace section consists of a 4k OcssdIdentify block, an accounting
+ * region and a chunk info region. The accounting and chunk info regions
+ * are of variable size (in multiples of the sector size). Then comes a
+ * data section that contains a region dedicated to data dn a region for
+ * metadata afterwards.
+ *
+ * [Format header ] 4096 bytes
+ * [OCSSD identity/geometry] 4096 bytes
+ * [Accounting ] sector_size * n
+ * [Chunk info ] sector_size * m
+ * [Namespace data ] sector_size * k
+ * [Namespace meta data ] md_size * k
+ *
+ * , where 'n' is the number of sectors required to hold accounting
+ * information on all chunks, 'm' is the number of sectors required to hold
+ * chunk information and 'k' is the number of available LBAs.
+ *
+ */
+
+ usable_size = secs_total * (sec_size + md_size);
+ ns_size = usable_size + sizeof(OcssdIdentity) + acct_size + logpage_size;
+
+ size = sizeof(OcssdFormatHeader) + ns_size * num_ns;
+
+ qemu_opt_set_number(opts, "size", size, errp);
+
+ ret = bdrv_create_file(filename, opts, errp);
+ if (ret < 0) {
+ error_propagate(errp, local_err);
+ goto fail;
+ }
+
+ bs = bdrv_open(filename, NULL, NULL,
+ BDRV_O_RDWR | BDRV_O_RESIZE | BDRV_O_PROTOCOL, errp);
+ if (bs == NULL) {
+ ret = -EIO;
+ goto fail;
+ }
+
+ blk = blk_new(BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL);
+ ret = blk_insert_bs(blk, bs, errp);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ blk_set_allow_write_beyond_eof(blk, true);
+
+ ret = blk_truncate(blk, 0, PREALLOC_MODE_OFF, errp);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ /*
+ * Calculate an "optimal" LBA address format that uses as few bits as
+ * possible.
+ */
+ lbaf = (OcssdIdLBAF) {
+ .sec_len = 32 - clz32(num_sec - 1),
+ .chk_len = 32 - clz32(num_chk - 1),
+ .pu_len = 32 - clz32(num_pu - 1),
+ .grp_len = 32 - clz32(num_grp - 1),
+ };
+
+ ocssd_ns_optimal_addrf(&addrf, &lbaf);
+
+ hdr = (OcssdFormatHeader) {
+ .magic = OCSSD_MAGIC,
+ .version = 0x1,
+ .num_ns = num_ns,
+ .md_size = md_size,
+ .sector_size = sec_size,
+ .ns_size = ns_size,
+ .pe_cycles = pe_cycles,
+ .lbaf = lbaf,
+ };
+
+ ret = blk_pwrite(blk, 0, &hdr, sizeof(OcssdFormatHeader), 0);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ offset = sizeof(OcssdFormatHeader);
+ for (int i = 0; i < num_ns; i++) {
+ id = (OcssdIdentity) {
+ .ver.major = 2,
+ .ver.minor = 0,
+ .lbaf = lbaf,
+ .mccap = mccap,
+ .wit = wit,
+ .geo = (OcssdIdGeo) {
+ .num_grp = num_grp,
+ .num_pu = num_pu,
+ .num_chk = num_chk,
+ .clba = num_sec,
+ },
+ .wrt = (OcssdIdWrt) {
+ .ws_min = ws_min,
+ .ws_opt = ws_opt,
+ .mw_cunits = mw_cunits,
+ },
+ .perf = (OcssdIdPerf) {
+ .trdt = cpu_to_le32(70000),
+ .trdm = cpu_to_le32(100000),
+ .tprt = cpu_to_le32(1900000),
+ .tprm = cpu_to_le32(3500000),
+ .tbet = cpu_to_le32(3000000),
+ .tbem = cpu_to_le32(3000000),
+ },
+ };
+
+ ret = blk_pwrite(blk, offset, &id, sizeof(OcssdIdentity), 0);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ offset += sizeof(OcssdIdentity);
+
+ acct = g_malloc0(acct_size);
+ ret = blk_pwrite(blk, offset, acct, acct_size, 0);
+ g_free(acct);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ offset += acct_size;
+
+ chk = g_malloc0(logpage_size);
+ for (int i = 0; i < chks_total; i++) {
+ chk[i].state = OCSSD_CHUNK_FREE;
+ chk[i].type = OCSSD_CHUNK_TYPE_SEQUENTIAL;
+ chk[i].wear_index = 0;
+ chk[i].slba = (i / (num_chk * num_pu)) << addrf.grp_offset
+ | (i % (num_chk * num_pu) / num_chk) << addrf.pu_offset
+ | (i % num_chk) << addrf.chk_offset;
+ chk[i].cnlb = num_sec;
+ chk[i].wp = 0;
+ }
+
+ ret = blk_pwrite(blk, offset, chk, logpage_size, 0);
+ g_free(chk);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ offset += logpage_size + usable_size;
+ }
+
+ ret = blk_truncate(blk, size, PREALLOC_MODE_OFF, errp);
+ if (ret < 0) {
+ goto fail;
+ }
+
+fail:
+ bdrv_unref(bs);
+ return ret;
+}
+
+static int ocssd_open(BlockDriverState *bs, QDict *options, int flags,
+ Error **errp)
+{
+ BDRVOcssdState *s = bs->opaque;
+ OcssdIdLBAF *lbaf;
+ int ret;
+ int64_t len;
+ uint64_t nblks;
+
+ bs->file = bdrv_open_child(NULL, options, "file", bs, &child_file,
+ false, errp);
+ if (!bs->file) {
+ return -EINVAL;
+ }
+
+ bs->sg = bs->file->bs->sg;
+ bs->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED |
+ (BDRV_REQ_FUA & bs->file->bs->supported_write_flags);
+ bs->supported_zero_flags = BDRV_REQ_WRITE_UNCHANGED |
+ ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP) &
+ bs->file->bs->supported_zero_flags);
+
+ len = bdrv_getlength(bs->file->bs);
+ if (len < 0) {
+ error_setg_errno(errp, -len, "could not get image size");
+ return len;
+ }
+ s->real_size = len;
+
+ ret = bdrv_pread(bs->file, 0, &s->hdr, sizeof(OcssdFormatHeader));
+ if (ret < 0) {
+ return ret;
+ }
+
+ /*
+ * Calculate the size according to the address space. This is returned from
+ * ocssd_getlength. See ocssd_getlength and ocssd_co_preadv.
+ */
+ lbaf = &s->hdr.lbaf;
+ nblks = 1 << (lbaf->grp_len + lbaf->pu_len + lbaf->chk_len +
+ lbaf->sec_len);
+ s->size = s->hdr.num_ns * nblks * (s->hdr.sector_size + s->hdr.md_size);
+
+ s->namespaces = g_new0(OcssdIdentity, s->hdr.num_ns);
+
+ for (int i = 0; i < s->hdr.num_ns; i++) {
+ ret = bdrv_pread(bs->file, s->hdr.sector_size + i * s->hdr.ns_size,
+ &s->namespaces[i], sizeof(OcssdIdentity));
+ if (ret < 0) {
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int ocssd_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+ const OcssdFormatHeader *hdr = (const void *) buf;
+
+ if (buf_size < sizeof(*hdr)) {
+ return 0;
+ }
+
+ if (hdr->magic == OCSSD_MAGIC && hdr->version == 1) {
+ return 100;
+ }
+
+ return 0;
+}
+
+static int ocssd_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
+{
+ return bdrv_probe_blocksizes(bs->file->bs, bsz);
+}
+
+static int ocssd_probe_geometry(BlockDriverState *bs, HDGeometry *geo)
+{
+ return bdrv_probe_geometry(bs->file->bs, geo);
+}
+
+static int coroutine_fn ocssd_co_copy_range_from(BlockDriverState *bs,
+ BdrvChild *src, uint64_t src_offset, BdrvChild *dst, uint64_t dst_offset,
+ uint64_t bytes, BdrvRequestFlags read_flags, BdrvRequestFlags write_flags)
+{
+ return bdrv_co_copy_range_from(bs->file, src_offset, dst, dst_offset,
+ bytes, read_flags, write_flags);
+}
+
+static int coroutine_fn ocssd_co_copy_range_to(BlockDriverState *bs,
+ BdrvChild *src, uint64_t src_offset, BdrvChild *dst, uint64_t dst_offset,
+ uint64_t bytes, BdrvRequestFlags read_flags, BdrvRequestFlags write_flags)
+{
+ return bdrv_co_copy_range_to(src, src_offset, bs->file, dst_offset, bytes,
+ read_flags, write_flags);
+}
+
+BlockDriver bdrv_ocssd = {
+ .format_name = "ocssd",
+ .instance_size = sizeof(BDRVOcssdState),
+
+ .bdrv_probe = &ocssd_probe,
+ .bdrv_open = &ocssd_open,
+ .bdrv_reopen_prepare = &ocssd_reopen_prepare,
+ .bdrv_child_perm = bdrv_filter_default_perms,
+
+ .bdrv_co_create_opts = &ocssd_co_create_opts,
+ .bdrv_co_preadv = &ocssd_co_preadv,
+ .bdrv_co_pwritev = &ocssd_co_pwritev,
+ .bdrv_co_pwrite_zeroes = &ocssd_co_pwrite_zeroes,
+ .bdrv_co_pdiscard = &ocssd_co_pdiscard,
+ .bdrv_co_block_status = &ocssd_co_block_status,
+ .bdrv_co_copy_range_from = &ocssd_co_copy_range_from,
+ .bdrv_co_copy_range_to = &ocssd_co_copy_range_to,
+ .bdrv_co_truncate = &ocssd_co_truncate,
+ .bdrv_co_ioctl = &ocssd_co_ioctl,
+
+ .bdrv_getlength = &ocssd_getlength,
+ .bdrv_measure = &ocssd_measure,
+ .bdrv_get_info = &ocssd_get_info,
+ .bdrv_get_specific_info = &ocssd_get_specific_info,
+ .bdrv_refresh_limits = &ocssd_refresh_limits,
+ .bdrv_probe_blocksizes = &ocssd_probe_blocksizes,
+ .bdrv_probe_geometry = &ocssd_probe_geometry,
+ .bdrv_eject = &ocssd_eject,
+ .bdrv_lock_medium = &ocssd_lock_medium,
+ .bdrv_has_zero_init = &ocssd_has_zero_init,
+
+ .create_opts = &ocssd_create_opts,
+
+ .no_size_required = true,
+};
+
+static void bdrv_ocssd_init(void)
+{
+ bdrv_register(&bdrv_ocssd);
+}
+
+block_init(bdrv_ocssd_init);
diff --git a/hw/block/Makefile.objs b/hw/block/Makefile.objs
index f5f643f0cc06..724bf20df90a 100644
--- a/hw/block/Makefile.objs
+++ b/hw/block/Makefile.objs
@@ -7,7 +7,7 @@ common-obj-$(CONFIG_PFLASH_CFI02) += pflash_cfi02.o
common-obj-$(CONFIG_XEN) += xen-block.o
common-obj-$(CONFIG_ECC) += ecc.o
common-obj-$(CONFIG_ONENAND) += onenand.o
-common-obj-$(CONFIG_NVME_PCI) += nvme.o
+common-obj-$(CONFIG_NVME_PCI) += nvme/nvme.o nvme/ocssd.o
obj-$(CONFIG_SH4) += tc58128.o
diff --git a/hw/block/nvme.c b/hw/block/nvme/nvme.c
similarity index 94%
rename from hw/block/nvme.c
rename to hw/block/nvme/nvme.c
index 5cd593806701..0ad102e487db 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme/nvme.c
@@ -48,15 +48,13 @@
#include "qemu/osdep.h"
#include "qemu/units.h"
#include "qemu/cutils.h"
-#include "qemu/log.h"
-#include "hw/block/block.h"
+#include "hw/block/nvme.h"
#include "hw/pci/msix.h"
#include "sysemu/sysemu.h"
#include "sysemu/block-backend.h"
#include "qapi/error.h"
#include "trace.h"
-#include "nvme.h"
#define NVME_MAX_QS PCI_MSIX_FLAGS_QSIZE
#define NVME_TEMPERATURE 0x143
@@ -64,13 +62,6 @@
#define NVME_AERL 3
#define NVME_OP_ABORTED 0xff
-#define NVME_GUEST_ERR(trace, fmt, ...) \
- do { \
- (trace_##trace)(__VA_ARGS__); \
- qemu_log_mask(LOG_GUEST_ERROR, #trace \
- " in %s: " fmt "\n", __func__, ## __VA_ARGS__); \
- } while (0)
-
static void nvme_process_sq(void *opaque);
static inline uint8_t nvme_addr_is_cmb(NvmeCtrl *n, hwaddr addr)
@@ -79,7 +70,7 @@ static inline uint8_t nvme_addr_is_cmb(NvmeCtrl *n, hwaddr addr)
addr < (n->ctrl_mem.addr + int128_get64(n->ctrl_mem.size));
}
-static void nvme_addr_read(NvmeCtrl *n, hwaddr addr, void *buf, int size)
+void nvme_addr_read(NvmeCtrl *n, hwaddr addr, void *buf, int size)
{
if (nvme_addr_is_cmb(n, addr)) {
memcpy(buf, (void *)&n->cmbuf[addr - n->ctrl_mem.addr], size);
@@ -90,13 +81,14 @@ static void nvme_addr_read(NvmeCtrl *n, hwaddr addr, void *buf, int size)
pci_dma_read(&n->parent_obj, addr, buf, size);
}
-static void nvme_addr_write(NvmeCtrl *n, hwaddr addr, void *buf, int size)
+void nvme_addr_write(NvmeCtrl *n, hwaddr addr, void *buf, int size)
{
- if (n->cmbsz && addr >= n->ctrl_mem.addr &&
- addr < (n->ctrl_mem.addr + int128_get64(n->ctrl_mem.size))) {
+ if (nvme_addr_is_cmb(n, addr)) {
memcpy((void *)&n->cmbuf[addr - n->ctrl_mem.addr], buf, size);
+
return;
}
+
pci_dma_write(&n->parent_obj, addr, buf, size);
}
@@ -377,6 +369,93 @@ static void dma_to_cmb(NvmeCtrl *n, QEMUSGList *qsg, QEMUIOVector *iov)
}
}
+static uint16_t nvme_dma_write_prp(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
+ uint64_t prp1, uint64_t prp2, NvmeRequest *req)
+{
+ QEMUSGList qsg;
+ uint16_t err = NVME_SUCCESS;
+
+ err = nvme_map_prp(n, &qsg, prp1, prp2, len, req);
+ if (err) {
+ return err;
+ }
+
+ if (req->is_cmb) {
+ QEMUIOVector iov;
+
+ qemu_iovec_init(&iov, qsg.nsg);
+ dma_to_cmb(n, &qsg, &iov);
+
+ if (unlikely(qemu_iovec_to_buf(&iov, 0, ptr, len) != len)) {
+ trace_nvme_err_invalid_dma();
+ err = NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ qemu_iovec_destroy(&iov);
+
+ return err;
+ }
+
+ if (unlikely(dma_buf_write(ptr, len, &qsg))) {
+ trace_nvme_err_invalid_dma();
+ err = NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ qemu_sglist_destroy(&qsg);
+
+ return err;
+}
+
+static uint16_t nvme_dma_write_sgl(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
+ NvmeSglDescriptor sgl, NvmeRequest *req)
+{
+ QEMUSGList qsg;
+ uint16_t err = NVME_SUCCESS;
+
+ err = nvme_map_sgl(n, &qsg, sgl, len, req);
+ if (err) {
+ return err;
+ }
+
+ if (req->is_cmb) {
+ QEMUIOVector iov;
+
+ qemu_iovec_init(&iov, qsg.nsg);
+ dma_to_cmb(n, &qsg, &iov);
+
+ if (unlikely(qemu_iovec_to_buf(&iov, 0, ptr, len) != len)) {
+ trace_nvme_err_invalid_dma();
+ err = NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ qemu_iovec_destroy(&iov);
+
+ return err;
+ }
+
+ if (unlikely(dma_buf_write(ptr, len, &qsg))) {
+ trace_nvme_err_invalid_dma();
+ err = NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ qemu_sglist_destroy(&qsg);
+
+ return err;
+}
+
+uint16_t nvme_dma_write(NvmeCtrl *n, uint8_t *ptr, uint32_t len, NvmeCmd *cmd,
+ NvmeRequest *req)
+{
+ if (cmd->psdt) {
+ return nvme_dma_write_sgl(n, ptr, len, cmd->dptr.sgl, req);
+ }
+
+ uint64_t prp1 = le64_to_cpu(cmd->dptr.prp.prp1);
+ uint64_t prp2 = le64_to_cpu(cmd->dptr.prp.prp2);
+
+ return nvme_dma_write_prp(n, ptr, len, prp1, prp2, req);
+}
+
static uint16_t nvme_dma_read_prp(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
uint64_t prp1, uint64_t prp2, NvmeRequest *req)
{
@@ -414,7 +493,7 @@ static uint16_t nvme_dma_read_prp(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
return err;
}
-static uint16_t nvme_dma_read_sgl(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
+uint16_t nvme_dma_read_sgl(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
NvmeSglDescriptor sgl, NvmeCmd *cmd, NvmeRequest *req)
{
QEMUSGList qsg;
@@ -451,7 +530,7 @@ static uint16_t nvme_dma_read_sgl(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
return err;
}
-static uint16_t nvme_dma_read(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
+uint16_t nvme_dma_read(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
NvmeCmd *cmd, NvmeRequest *req)
{
if (cmd->psdt) {
@@ -477,12 +556,12 @@ static void nvme_blk_req_destroy(NvmeBlockBackendRequest *blk_req)
g_free(blk_req);
}
-static void nvme_blk_req_put(NvmeCtrl *n, NvmeBlockBackendRequest *blk_req)
+void nvme_blk_req_put(NvmeCtrl *n, NvmeBlockBackendRequest *blk_req)
{
nvme_blk_req_destroy(blk_req);
}
-static NvmeBlockBackendRequest *nvme_blk_req_get(NvmeCtrl *n, NvmeRequest *req,
+NvmeBlockBackendRequest *nvme_blk_req_get(NvmeCtrl *n, NvmeRequest *req,
QEMUSGList *qsg)
{
NvmeBlockBackendRequest *blk_req = g_malloc0(sizeof(*blk_req));
@@ -519,7 +598,8 @@ static uint16_t nvme_blk_setup(NvmeCtrl *n, NvmeNamespace *ns, QEMUSGList *qsg,
return NVME_SUCCESS;
}
-static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req,
+ NvmeBlockSetupFn blk_setup)
{
NvmeNamespace *ns = req->ns;
uint16_t err;
@@ -547,7 +627,7 @@ static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
}
}
- err = nvme_blk_setup(n, ns, &qsg, ns->blk_offset, unit_len, req);
+ err = blk_setup(n, ns, &qsg, ns->blk_offset, unit_len, req);
if (err) {
return err;
}
@@ -576,8 +656,7 @@ static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
qemu_sglist_add(&qsg, le64_to_cpu(cmd->mptr), meta_len);
}
- err = nvme_blk_setup(n, ns, &qsg, ns->blk_offset_md, meta_unit_len,
- req);
+ err = blk_setup(n, ns, &qsg, ns->blk_offset_md, meta_unit_len, req);
if (err) {
return err;
}
@@ -624,7 +703,7 @@ static void nvme_post_cqes(void *opaque)
}
}
-static void nvme_enqueue_req_completion(NvmeCQueue *cq, NvmeRequest *req)
+void nvme_enqueue_req_completion(NvmeCQueue *cq, NvmeRequest *req)
{
assert(cq->cqid == req->sq->cqid);
@@ -634,8 +713,8 @@ static void nvme_enqueue_req_completion(NvmeCQueue *cq, NvmeRequest *req)
timer_mod(cq->timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + 500);
}
-static void nvme_enqueue_event(NvmeCtrl *n, uint8_t event_type,
- uint8_t event_info, uint8_t log_page)
+void nvme_enqueue_event(NvmeCtrl *n, uint8_t event_type, uint8_t event_info,
+ uint8_t log_page)
{
NvmeAsyncEvent *event;
@@ -693,8 +772,7 @@ static void nvme_process_aers(void *opaque)
n->outstanding_aers--;
req = n->aer_reqs[n->outstanding_aers];
-
- result = (NvmeAerResult *) &req->cqe.result;
+ result = (NvmeAerResult *) &req->cqe.cdw0;
result->event_type = event->result.event_type;
result->event_info = event->result.event_info;
result->log_page = event->result.log_page;
@@ -709,7 +787,7 @@ static void nvme_process_aers(void *opaque)
}
}
-static void nvme_rw_cb(void *opaque, int ret)
+void nvme_rw_cb(void *opaque, int ret)
{
NvmeBlockBackendRequest *blk_req = opaque;
NvmeRequest *req = blk_req->req;
@@ -789,7 +867,7 @@ static uint16_t nvme_write_zeros(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
return NVME_NO_COMPLETE;
}
-static uint16_t nvme_rw_check_req(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+uint16_t nvme_rw_check_req(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
NvmeNamespace *ns = req->ns;
NvmeRwCmd *rw = (NvmeRwCmd *) cmd;
@@ -851,7 +929,7 @@ static void nvme_blk_submit_cmb(NvmeCtrl *n, NvmeBlockBackendRequest *blk_req,
}
}
-static uint16_t nvme_blk_submit_io(NvmeCtrl *n, NvmeRequest *req,
+uint16_t nvme_blk_submit_io(NvmeCtrl *n, NvmeRequest *req,
BlockCompletionFunc *cb)
{
NvmeBlockBackendRequest *blk_req;
@@ -882,7 +960,7 @@ static uint16_t nvme_rw(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
trace_nvme_rw(req->is_write ? "write" : "read", nlb,
nlb << nvme_ns_lbads(req->ns), slba);
- int err = nvme_blk_map(n, cmd, req);
+ int err = nvme_blk_map(n, cmd, req, nvme_blk_setup);
if (err) {
return err;
}
@@ -890,7 +968,7 @@ static uint16_t nvme_rw(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
return nvme_blk_submit_io(n, req, nvme_rw_cb);
}
-static uint16_t nvme_io_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+uint16_t nvme_io_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
NvmeRwCmd *rw;
int err;
@@ -1233,7 +1311,7 @@ static uint16_t nvme_identify(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
}
}
-static uint16_t nvme_get_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+uint16_t nvme_get_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
uint32_t dw10 = le32_to_cpu(cmd->cdw10);
uint32_t dw11 = le32_to_cpu(cmd->cdw11);
@@ -1284,11 +1362,11 @@ static uint16_t nvme_get_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
return NVME_INVALID_FIELD | NVME_DNR;
}
- req->cqe.result = result;
+ req->cqe.cdw0 = result;
return NVME_SUCCESS;
}
-static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
uint32_t dw10 = le32_to_cpu(cmd->cdw10);
uint32_t dw11 = le32_to_cpu(cmd->cdw11);
@@ -1319,8 +1397,8 @@ static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
((dw11 >> 16) & 0xFFFF) + 1,
n->params.num_queues - 1,
n->params.num_queues - 1);
- req->cqe.result = cpu_to_le32((n->params.num_queues - 2) |
- ((n->params.num_queues - 2) << 16));
+ req->cqe.cdw0 = cpu_to_le32((n->params.num_queues - 2) |
+ ((n->params.num_queues - 2) << 16));
break;
case NVME_ASYNCHRONOUS_EVENT_CONF:
n->features.async_config = dw11;
@@ -1340,7 +1418,7 @@ static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
return NVME_SUCCESS;
}
-static void nvme_clear_events(NvmeCtrl *n, uint8_t event_type)
+void nvme_clear_events(NvmeCtrl *n, uint8_t event_type)
{
n->aer_mask &= ~(1 << event_type);
if (!QSIMPLEQ_EMPTY(&n->aer_queue)) {
@@ -1423,8 +1501,7 @@ static uint16_t nvme_fw_log_info(NvmeCtrl *n, NvmeCmd *cmd, uint32_t buf_len,
return nvme_dma_read(n, (uint8_t *) &fw_log + off, trans_len, cmd, req);
}
-
-static uint16_t nvme_get_log(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+uint16_t nvme_get_log(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
uint32_t dw10 = le32_to_cpu(cmd->cdw10);
uint32_t dw11 = le32_to_cpu(cmd->cdw11);
@@ -1486,7 +1563,7 @@ static uint16_t nvme_abort(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
uint16_t sqid = cmd->cdw10 & 0xffff;
uint16_t cid = (cmd->cdw10 >> 16) & 0xffff;
- req->cqe.result = 1;
+ req->cqe.cdw0 = 1;
if (nvme_check_sqid(n, sqid)) {
return NVME_INVALID_FIELD | NVME_DNR;
}
@@ -1502,7 +1579,7 @@ static uint16_t nvme_abort(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
nvme_addr_read(n, addr, (void *) &abort_cmd, sizeof(abort_cmd));
if (abort_cmd.cid == cid) {
- req->cqe.result = 0;
+ req->cqe.cdw0 = 0;
new = QTAILQ_FIRST(&sq->req_list);
QTAILQ_REMOVE(&sq->req_list, new, entry);
QTAILQ_INSERT_TAIL(&sq->out_req_list, new, entry);
@@ -1525,7 +1602,7 @@ static uint16_t nvme_abort(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
return NVME_SUCCESS;
}
-static uint16_t nvme_admin_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+uint16_t nvme_admin_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
{
switch (cmd->opcode) {
case NVME_ADM_CMD_DELETE_SQ:
@@ -1591,8 +1668,8 @@ static void nvme_process_sq(void *opaque)
nvme_init_req(n, &cmd, req);
- status = sq->sqid ? nvme_io_cmd(n, &cmd, req) :
- nvme_admin_cmd(n, &cmd, req);
+ status = sq->sqid ? n->io_cmd(n, &cmd, req) :
+ n->admin_cmd(n, &cmd, req);
if (status != NVME_NO_COMPLETE) {
req->status = status;
nvme_enqueue_req_completion(cq, req);
@@ -2071,7 +2148,7 @@ static int nvme_check_constraints(NvmeCtrl *n, Error **errp)
return 0;
}
-static int nvme_init_blk(NvmeCtrl *n, Error **errp)
+int nvme_init_blk(NvmeCtrl *n, Error **errp)
{
blkconf_blocksizes(&n->conf);
if (!blkconf_apply_backend_options(&n->conf, blk_is_read_only(n->conf.blk),
@@ -2082,7 +2159,7 @@ static int nvme_init_blk(NvmeCtrl *n, Error **errp)
return 0;
}
-static int nvme_init_state(NvmeCtrl *n, Error **errp)
+int nvme_init_state(NvmeCtrl *n, Error **errp)
{
int64_t bs_size;
Error *local_err = NULL;
@@ -2143,7 +2220,7 @@ static void nvme_init_cmb(NvmeCtrl *n, PCIDevice *pci_dev)
PCI_BASE_ADDRESS_MEM_PREFETCH, &n->ctrl_mem);
}
-static void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev)
+void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev)
{
uint8_t *pci_conf = pci_dev->config;
@@ -2166,7 +2243,7 @@ static void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev)
}
}
-static void nvme_init_ctrl(NvmeCtrl *n)
+void nvme_init_ctrl(NvmeCtrl *n)
{
NvmeIdCtrl *id = &n->id_ctrl;
NvmeParams *params = &n->params;
@@ -2235,7 +2312,7 @@ static uint64_t nvme_ns_calc_blks(NvmeCtrl *n, NvmeNamespace *ns)
return n->ns_size / (nvme_ns_lbads_bytes(ns) + nvme_ns_ms(ns));
}
-static void nvme_ns_init_identify(NvmeCtrl *n, NvmeIdNs *id_ns)
+void nvme_ns_init_identify(NvmeCtrl *n, NvmeIdNs *id_ns)
{
NvmeParams *params = &n->params;
@@ -2298,6 +2375,9 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
NvmeCtrl *n = NVME(pci_dev);
Error *local_err = NULL;
+ n->admin_cmd = nvme_admin_cmd;
+ n->io_cmd = nvme_io_cmd;
+
if (nvme_check_constraints(n, &local_err)) {
error_propagate_prepend(errp, local_err, "nvme_check_constraints: ");
return;
@@ -2323,10 +2403,8 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
}
}
-static void nvme_exit(PCIDevice *pci_dev)
+void nvme_free_ctrl(NvmeCtrl *n, PCIDevice *pci_dev)
{
- NvmeCtrl *n = NVME(pci_dev);
-
nvme_clear_ctrl(n);
g_free(n->namespaces);
g_free(n->cq);
@@ -2338,9 +2416,17 @@ static void nvme_exit(PCIDevice *pci_dev)
if (n->params.cmb_size_mb) {
g_free(n->cmbuf);
}
+
msix_uninit_exclusive_bar(pci_dev);
}
+static void nvme_exit(PCIDevice *pci_dev)
+{
+ NvmeCtrl *n = NVME(pci_dev);
+
+ nvme_free_ctrl(n, pci_dev);
+}
+
static Property nvme_props[] = {
DEFINE_BLOCK_PROPERTIES(NvmeCtrl, conf),
DEFINE_NVME_PROPERTIES(NvmeCtrl, params),
diff --git a/hw/block/nvme/ocssd.c b/hw/block/nvme/ocssd.c
new file mode 100644
index 000000000000..c2e45edfa9b8
--- /dev/null
+++ b/hw/block/nvme/ocssd.c
@@ -0,0 +1,2647 @@
+/*
+ * QEMU OpenChannel 2.0 Controller
+ *
+ * Copyright (c) 2019 CNEX Labs, Inc.
+ *
+ * Written by Klaus Birkelund Abildgaard Jensen <klaus@birkelund.eu>
+ *
+ * Thank you to the following people for their contributions to the original
+ * qemu-nvme (github.com/OpenChannelSSD/qemu-nvme) implementation.
+ *
+ * Matias Bjørling <mb@lightnvm.io>
+ * Javier González <javier@javigon.com>
+ * Simon Andreas Frimann Lund <ocssd@safl.dk>
+ * Hans Holmberg <hans@owltronix.com>
+ * Jesper Devantier <contact@pseudonymous.me>
+ * Young Tack Jin <youngtack.jin@circuitblvd.com>
+ *
+ * This code is licensed under the GNU GPL v2 or later.
+ */
+
+/*
+ * This device emulates an OpenChannel 2.0 compliant NVMe controller.
+ *
+ * Reference docs: http://lightnvm.io/docs/OCSSD-2_0-20180129.pdf
+ *
+ *
+ * Usage
+ * -----
+ *
+ * The device must have a backing file to store its data. An initialized OCSSD
+ * backing file must be created using qemu-img:
+ *
+ * qemu-img create -f ocssd -o num_grp=2,num_pu=4,num_chk=60 ocssd.img
+ *
+ * Besides the geometry options (num_{grp,pu,chk,sec}), qemu-img also supports
+ * options related to write characteristics (ws_min, ws_opt and mw_cunits) and
+ * lbads and ms sizes. These options can also be overwritten as parameters to
+ * the device. Issue
+ *
+ * qemu-img create -f ocssd -o help
+ *
+ * to see the full list of supported options.
+ *
+ * To add the OCSSD NVMe device, extend the QEMU arguments with something like
+ *
+ * -blockdev ocssd,node-name=ocssd01,file.driver=file,file.filename=ocssd.img
+ * -device nvme,drive=ocssd01,serial=deadbeef,id=ocssd01
+ *
+ * All of the standard nvme device options are supported, except 'ms', which is
+ * configured at image creation time.
+ *
+ * Additional advanced -device options.
+ *
+ * mccap=<int> : Media and Controller Capabilities (MCCAP). OR'ed
+ * value of the following:
+ * multiple resets 0x2
+ * early resets (non-standard) 0x4
+ * ws_min=<int> : Mininum write size for device in sectors.
+ * ws_opt=<int> : Optimal write size for device in sectors.
+ * mw_cunits=<int> : Cache minimum write size units. If DULBE is enabled,
+ * an error will be reported if reads are within this
+ * window.
+ * wit=<int> : Wear-level index delta threshold.
+ * chunkinfo=<file> : Overwrite chunk states from file.
+ * resetfail=<file> : Reset fail injection configuration file.
+ * writefail=<file> : Write fail injection configuration file.
+ * early_reset : Allow early resets (reset open chunks).
+ *
+ * NOTE: mccap, ws_min, ws_opt, mw_cunits and wit defaults to whatever was
+ * defined at image creation time.
+ *
+ * The emulated device maintains a Chunk Info Log Page on the backing block
+ * device. When the device is brought up any state will be restored. The
+ * restored chunk states may be overwritten using the `chunkinfo` parameter. An
+ * example chunk state file follows (note the use of the '*' wildcard to match
+ * all groups, punits or chunks).
+ *
+ * # "reset" all chunks
+ * ns=1 group=* punit=* chunk=* state=FREE type=SEQUENTIAL pe_cycles=0
+ *
+ * # first chunk on all luns has type random
+ * ns=1 group=* punit=* chunk=0 type=RANDOM
+ *
+ * # add an open chunk
+ * ns=1 group=0 punit=0 chunk=1 state=OPEN type=SEQ wp=0x800
+ *
+ * # add a closed chunk
+ * ns=1 group=0 punit=0 chunk=2 state=CLOSED type=SEQ wp=0x1000
+ *
+ * # add an offline chunk
+ * ns=1 group=0 punit=0 chunk=3 state=OFFLINE type=SEQ
+ *
+ *
+ * The `resetfail` and `writefail` QEMU parameters can be used to do
+ * probabilistic error injection. The parameters points to text files and they
+ * also support the '*' wildcard.
+ *
+ * Write error injection is done per sector.
+ *
+ * # always fail writes for this sector
+ * ns=1 group=0 punit=3 chunk=0 sectr=53 prob=100
+ *
+ *
+ * Reset error injection is done per chunk, so exclude the `sec` parameter.
+ *
+ * # always fail resets for this chunk
+ * ns=1 group=0 punit=3 chunk=0 prob=100
+ *
+ *
+ * You probably want to make sure the following options are enabled in the
+ * kernel you are going to use.
+ *
+ * CONFIG_BLK_DEV_INTEGRITY=y
+ * CONFIG_HOTPLUG_PCI_PCIE=y
+ * CONFIG_HOTPLUG_PCI_ACPI=y
+ *
+ *
+ * It is assumed that when using vector write requests, then the LBAs for
+ * different chunks are laid out contiguously and sorted with increasing
+ * addresses.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/cutils.h"
+#include "sysemu/block-backend.h"
+#include "sysemu/sysemu.h"
+#include "qapi/error.h"
+
+#include "trace.h"
+#include "ocssd.h"
+
+/* #define OCSSD_CTRL_DEBUG */
+
+#ifdef OCSSD_CTRL_DEBUG
+#define _dprintf(fmt, ...) \
+ do { \
+ fprintf(stderr, "ocssd: " fmt, ## __VA_ARGS__); \
+ } while (0)
+
+static inline void _dprint_lba(OcssdCtrl *o, OcssdNamespace *ons, uint64_t lba)
+{
+ OcssdAddrF *addrf = &ons->addrf;
+
+ uint8_t group, punit;
+ uint16_t chunk;
+ uint32_t sectr;
+
+ group = _group(addrf, lba);
+ punit = _punit(addrf, lba);
+ chunk = _chunk(addrf, lba);
+ sectr = _sectr(addrf, lba);
+
+ _dprintf("lba 0x%016"PRIx64" group %"PRIu8" punit %"PRIu8" chunk %"PRIu16
+ " sectr %"PRIu32"\n", lba, group, punit, chunk, sectr);
+}
+
+static inline void _dprint_vector_rw(OcssdCtrl *o, NvmeRequest *req)
+{
+ OcssdNamespace *ons = _ons(o, req->ns->id);
+ _dprintf("vector %s request: cid %d nlb %d\n",
+ req->is_write ? "write" : "read", req->cqe.cid, req->nlb);
+ _dprintf("lba list:\n");
+ for (uint16_t i = 0; i < req->nlb; i++) {
+ _dprint_lba(o, ons, _vlba(req, i));
+ }
+}
+#else
+static void _dprint_lba(OcssdCtrl *o, OcssdNamespace *ons, uint64_t lba) {}
+static void _dprint_vector_rw(OcssdCtrl *o, NvmeRequest *req) {}
+#endif
+
+
+static inline bool _is_write(NvmeRequest *req)
+{
+ return req->cmd.opcode == OCSSD_CMD_VECT_WRITE || nvme_rw_is_write(req);
+}
+
+static inline bool _is_vector_request(NvmeRequest *req)
+{
+ switch (req->cmd.opcode) {
+ case OCSSD_CMD_VECT_RESET:
+ case OCSSD_CMD_VECT_WRITE:
+ case OCSSD_CMD_VECT_READ:
+ case OCSSD_CMD_VECT_COPY:
+ return true;
+ }
+
+ return false;
+}
+
+static inline OcssdNamespace *_ons(OcssdCtrl *o, uint32_t nsid)
+{
+ if (unlikely(nsid == 0 || nsid > o->nvme.params.num_ns)) {
+ return NULL;
+ }
+
+ return &o->namespaces[nsid - 1];
+}
+
+static inline uint64_t _sectr(OcssdAddrF *addrf, uint64_t lba)
+{
+ return (lba & addrf->sec_mask) >> addrf->sec_offset;
+}
+
+static inline uint64_t _chunk(OcssdAddrF *addrf, uint64_t lba)
+{
+ return (lba & addrf->chk_mask) >> addrf->chk_offset;
+}
+
+static inline uint64_t _punit(OcssdAddrF *addrf, uint64_t lba)
+{
+ return (lba & addrf->pu_mask) >> addrf->pu_offset;
+}
+
+static inline uint64_t _group(OcssdAddrF *addrf, uint64_t lba)
+{
+ return (lba & addrf->grp_mask) >> addrf->grp_offset;
+}
+
+static inline uint64_t _make_lba(OcssdAddrF *addrf, uint16_t group,
+ uint16_t punit, uint32_t chunk, uint32_t sectr)
+{
+ return sectr << addrf->sec_offset
+ | chunk << addrf->chk_offset
+ | punit << addrf->pu_offset
+ | group << addrf->grp_offset;
+}
+
+static inline int _valid(OcssdCtrl *o, OcssdNamespace *ons, uint64_t lba)
+{
+ OcssdIdGeo *geo = &ons->id.geo;
+ OcssdAddrF *addrf = &ons->addrf;
+
+ return _sectr(addrf, lba) < geo->clba &&
+ _chunk(addrf, lba) < geo->num_chk &&
+ _punit(addrf, lba) < geo->num_pu &&
+ _group(addrf, lba) < geo->num_grp;
+}
+
+static inline uint64_t _sectr_idx(OcssdCtrl *o, OcssdNamespace *ons,
+ uint64_t lba)
+{
+ OcssdAddrF *addrf = &ons->addrf;
+
+ return _sectr(addrf, lba) +
+ _chunk(addrf, lba) * ons->secs_per_chk +
+ _punit(addrf, lba) * ons->secs_per_pu +
+ _group(addrf, lba) * ons->secs_per_grp;
+}
+
+static inline uint64_t _chk_idx(OcssdCtrl *o, OcssdNamespace *ons,
+ uint64_t lba)
+{
+ OcssdIdGeo *geo = &ons->id.geo;
+ OcssdAddrF *addrf = &ons->addrf;
+
+ return _chunk(addrf, lba) +
+ _punit(addrf, lba) * geo->num_chk +
+ _group(addrf, lba) * ons->chks_per_grp;
+}
+
+static inline uint64_t _vlba(NvmeRequest *req, uint16_t n)
+{
+ return req->nlb > 1 ? ((uint64_t *) req->slba)[n] : req->slba;
+}
+
+static inline void _sglist_to_iov(NvmeCtrl *n, QEMUSGList *qsg,
+ QEMUIOVector *iov)
+{
+ for (int i = 0; i < qsg->nsg; i++) {
+ qemu_iovec_add(iov, (void *) qsg->sg[i].base, qsg->sg[i].len);
+ }
+}
+
+/*
+ * _sglist_copy_from copies `len` bytes from the `idx`'th scatter gather entry
+ * at `offset` in the `to` QEMUSGList into the `to` QEMUSGList. `idx` and
+ * `offset` are updated to mark the position in `to` at which the function
+ * reached `len` bytes.
+ */
+static void _sglist_copy_from(QEMUSGList *to, QEMUSGList *from, int *idx,
+ size_t *offset, size_t len)
+{
+ dma_addr_t curr_addr, curr_len;
+
+ while (len) {
+ curr_addr = from->sg[*idx].base + *offset;
+ curr_len = from->sg[*idx].len - *offset;
+
+ curr_len = MIN(curr_len, len);
+
+ if (to) {
+ qemu_sglist_add(to, curr_addr, curr_len);
+ }
+
+ *offset += curr_len;
+ len -= curr_len;
+
+ if (*offset == from->sg[*idx].len) {
+ *offset = 0;
+ (*idx)++;
+ }
+ }
+}
+
+static inline bool _wi_outside_threshold(OcssdNamespace *ons,
+ OcssdChunkDescriptor *chk)
+{
+ return chk->wear_index < ons->wear_index_avg - ons->id.wit ||
+ chk->wear_index > ons->wear_index_avg + ons->id.wit;
+}
+
+static inline uint8_t _calc_wi(OcssdCtrl *o, uint32_t pe_cycles)
+{
+ return (pe_cycles * 255) / o->hdr.pe_cycles;
+}
+
+static OcssdChunkDescriptor *_get_chunk(OcssdCtrl *o, OcssdNamespace *ons,
+ uint64_t lba)
+{
+ if (!_valid(o, ons, lba)) {
+ return NULL;
+ }
+
+ return &ons->info.descr[_chk_idx(o, ons, lba)];
+}
+
+static OcssdChunkAcctDescriptor *_get_chunk_acct(OcssdCtrl *o,
+ OcssdNamespace *ons, uint64_t lba)
+{
+ if (!_valid(o, ons, lba)) {
+ return NULL;
+ }
+
+ return &ons->acct.descr[_chk_idx(o, ons, lba)];
+}
+
+static void _get_lba_list(OcssdCtrl *o, hwaddr addr, uint64_t **lbal,
+ NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+ uint32_t len = req->nlb * sizeof(uint64_t);
+
+ if (req->nlb > 1) {
+ *lbal = g_malloc_n(req->nlb, sizeof(uint64_t));
+ nvme_addr_read(n, addr, *lbal, len);
+ } else {
+ *lbal = (uint64_t *) addr;
+ }
+}
+
+static inline OcssdChunkState _str_to_chunk_state(char *s)
+{
+ if (!strcmp(s, "FREE")) {
+ return OCSSD_CHUNK_FREE;
+ }
+
+ if (!strcmp(s, "OFFLINE")) {
+ return OCSSD_CHUNK_OFFLINE;
+ }
+
+ if (!strcmp(s, "OPEN")) {
+ return OCSSD_CHUNK_OPEN;
+ }
+
+ if (!strcmp(s, "CLOSED")) {
+ return OCSSD_CHUNK_CLOSED;
+ }
+
+ return -1;
+}
+
+static inline OcssdChunkType _str_to_chunk_type(char *s)
+{
+ if (!strcmp(s, "SEQ") || !strcmp(s, "SEQUENTIAL")) {
+ return OCSSD_CHUNK_TYPE_SEQUENTIAL;
+ }
+
+ if (!strcmp(s, "RAN") || !strcmp(s, "RANDOM")) {
+ return OCSSD_CHUNK_TYPE_RANDOM;
+ }
+
+ return -1;
+}
+
+static int _parse_string(const char *s, const char *k, char **v)
+{
+ char *p = strstr(s, k);
+ if (!p) {
+ return 0;
+ }
+
+ return sscanf(p + strlen(k), "%ms", v);
+}
+
+static int _parse_uint8(const char *s, const char *k, uint8_t *v)
+{
+ char *p = strstr(s, k);
+ if (!p) {
+ return 0;
+ }
+
+ return sscanf(p + strlen(k), "0x%"SCNx8, v) ||
+ sscanf(p + strlen(k), "%"SCNu8, v);
+}
+
+static int _parse_uint16(const char *s, const char *k, uint16_t *v)
+{
+ char *p = strstr(s, k);
+ if (!p) {
+ return 0;
+ }
+
+ return sscanf(p + strlen(k), "0x%"SCNx16, v) ||
+ sscanf(p + strlen(k), "%"SCNu16, v);
+}
+
+static int _parse_uint32(const char *s, const char *k, uint32_t *v)
+{
+ char *p = strstr(s, k);
+ if (!p) {
+ return 0;
+ }
+
+ return sscanf(p + strlen(k), "0x%"SCNx32, v) ||
+ sscanf(p + strlen(k), "%"SCNu32, v);
+}
+
+static int _parse_uint64(const char *s, const char *k, uint64_t *v)
+{
+ char *p = strstr(s, k);
+ if (!p) {
+ return 0;
+ }
+
+ return sscanf(p + strlen(k), "0x%"SCNx64, v) ||
+ sscanf(p + strlen(k), "%"SCNu64, v);
+}
+
+static bool _parse_wildcard(const char *s, const char *k)
+{
+ char *v;
+ bool found = false;
+ if (!_parse_string(s, k, &v)) {
+ return false;
+ }
+
+ if (strcmp(v, "*") == 0) {
+ found = true;
+ }
+
+ free(v);
+
+ return found;
+}
+
+static int _parse_lba_part_uint16(const char *s, const char *k,
+ uint16_t *bgn, uint16_t *end, uint16_t end_defval)
+{
+ if (!bgn || !end) {
+ return 1;
+ }
+
+ if (_parse_wildcard(s, k)) {
+ *bgn = 0;
+ *end = end_defval;
+
+ return 1;
+ }
+
+ if (!_parse_uint16(s, k, bgn)) {
+ return 0;
+ }
+
+ *end = *bgn + 1;
+
+ return 1;
+}
+
+static int _parse_lba_part_uint32(const char *s, const char *k,
+ uint32_t *bgn, uint32_t *end, uint32_t end_defval)
+{
+ if (!bgn || !end) {
+ return 1;
+ }
+
+ if (_parse_wildcard(s, k)) {
+ *bgn = 0;
+ *end = end_defval;
+
+ return 1;
+ }
+
+ if (!_parse_uint32(s, k, bgn)) {
+ return 0;
+ }
+
+ *end = *bgn + 1;
+
+ return 1;
+}
+
+static int _parse_lba_parts(OcssdIdGeo *geo, const char *s,
+ uint16_t *grp_bgn, uint16_t *grp_end, uint16_t *pu_bgn,
+ uint16_t *pu_end, uint32_t *chk_bgn, uint32_t *chk_end,
+ uint32_t *sec_bgn, uint32_t *sec_end, Error **errp)
+{
+ if (!_parse_lba_part_uint16(s, "group=", grp_bgn, grp_end, geo->num_grp)) {
+ error_setg(errp, "could not parse group");
+ return 0;
+ }
+
+ if (!_parse_lba_part_uint16(s, "punit=", pu_bgn, pu_end, geo->num_pu)) {
+ error_setg(errp, "could not parse punit");
+ return 0;
+ }
+
+ if (!_parse_lba_part_uint32(s, "chunk=", chk_bgn, chk_end, geo->num_chk)) {
+ error_setg(errp, "could not parse chunk");
+ return 0;
+ }
+
+ if (!_parse_lba_part_uint32(s, "sectr=", sec_bgn, sec_end, geo->clba)) {
+ error_setg(errp, "could not parse sectr");
+ return 0;
+ }
+
+ return 1;
+}
+
+static int _parse_and_update_reset_error_injection(OcssdCtrl *o, const char *s,
+ Error **errp)
+{
+ OcssdNamespace *ons;
+ OcssdIdGeo *geo;
+ uint16_t group, group_end, punit, punit_end;
+ uint32_t nsid, chunk, chunk_end;
+ uint64_t idx;
+ uint8_t prob;
+ Error *local_err = NULL;
+
+ size_t slen = strlen(s);
+ if (slen == 1 || (slen > 1 && s[0] == '#')) {
+ return 0;
+ }
+
+ if (!_parse_uint32(s, "ns=", &nsid)) {
+ error_setg(errp, "could not parse namespace id");
+ return 1;
+ }
+
+ ons = &o->namespaces[nsid - 1];
+ geo = &ons->id.geo;
+
+ if (!_parse_lba_parts(geo, s, &group, &group_end, &punit, &punit_end,
+ &chunk, &chunk_end, NULL, NULL, &local_err)) {
+ error_propagate_prepend(errp, local_err, "could not parse chunk slba");
+ return 1;
+ }
+
+ if (!_parse_uint8(s, "prob=", &prob)) {
+ error_setg(errp, "could not parse probability");
+ return 1;
+ }
+
+ if (prob > 100) {
+ error_setg(errp, "invalid probability");
+ return 1;
+ }
+
+ for (uint16_t g = group; g < group_end; g++) {
+ for (uint16_t p = punit; p < punit_end; p++) {
+ for (uint32_t c = chunk; c < chunk_end; c++) {
+ idx = _chk_idx(o, ons, _make_lba(&ons->addrf, g, p, c, 0));
+ ons->resetfail[idx] = prob;
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int _parse_and_update_write_error_injection(OcssdCtrl *o, const char *s,
+ Error **errp)
+{
+ OcssdNamespace *ons;
+ OcssdIdGeo *geo;
+ uint16_t group, group_end, punit, punit_end;
+ uint32_t nsid, chunk, chunk_end, sectr, sectr_end;
+ uint64_t sectr_idx;
+ uint8_t prob;
+ Error *local_err = NULL;
+
+ size_t slen = strlen(s);
+ if (slen == 1 || (slen > 1 && s[0] == '#')) {
+ return 0;
+ }
+
+ if (!_parse_uint32(s, "ns=", &nsid)) {
+ error_setg(errp, "could not parse namespace id");
+ return 1;
+ }
+
+ ons = &o->namespaces[nsid - 1];
+ geo = &ons->id.geo;
+
+ if (!_parse_lba_parts(geo, s, &group, &group_end, &punit, &punit_end,
+ &chunk, &chunk_end, §r, §r_end, &local_err)) {
+ error_propagate_prepend(errp, local_err, "could not parse lba");
+ return 1;
+ }
+
+ if (!_parse_uint8(s, "prob=", &prob)) {
+ error_setg(errp, "could not parse probability");
+ return 1;
+ }
+
+ if (prob > 100) {
+ error_setg(errp, "invalid probability");
+ return 1;
+ }
+
+ for (uint16_t g = group; g < group_end; g++) {
+ for (uint16_t p = punit; p < punit_end; p++) {
+ for (uint32_t c = chunk; c < chunk_end; c++) {
+ for (uint32_t s = sectr; s < sectr_end; s++) {
+ sectr_idx = _sectr_idx(o, ons, _make_lba(
+ &ons->addrf, g, p, c, s));
+ ons->writefail[sectr_idx] = prob;
+ }
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int _parse_and_update_chunk_info(OcssdCtrl *o, const char *s,
+ Error **errp)
+{
+ char *v;
+ OcssdChunkDescriptor *chk;
+ OcssdChunkAcctDescriptor *chk_acct;
+ OcssdNamespace *ons;
+ OcssdIdGeo *geo;
+ uint16_t group, group_end, punit, punit_end;
+ uint32_t nsid, chunk, chunk_end, pe_cycles;
+ uint64_t cnlb, wp, slba;
+ int state = 0, type = 0;
+ bool cnlb_parsed = false, wp_parsed = false, pe_cycles_parsed = false;
+ bool state_parsed = false, type_parsed = false;
+ Error *local_err = NULL;
+
+ size_t slen = strlen(s);
+ if (slen == 1 || (slen > 1 && s[0] == '#')) {
+ return 0;
+ }
+
+ if (!_parse_uint32(s, "ns=", &nsid)) {
+ error_setg(errp, "could not parse namespace id");
+ return 1;
+ }
+
+ ons = &o->namespaces[nsid - 1];
+ geo = &ons->id.geo;
+
+ if (!_parse_lba_parts(geo, s, &group, &group_end, &punit, &punit_end,
+ &chunk, &chunk_end, NULL, NULL, &local_err)) {
+ error_propagate_prepend(errp, local_err, "could not parse chunk slba");
+ return 1;
+ }
+
+ if (_parse_string(s, "state=", &v)) {
+ state_parsed = true;
+ state = _str_to_chunk_state(v);
+ free(v);
+
+ if (state < 0) {
+ error_setg(errp, "invalid chunk state");
+ return 1;
+ }
+ }
+
+ if (_parse_string(s, "type=", &v)) {
+ type_parsed = true;
+ type = _str_to_chunk_type(v);
+ free(v);
+
+ if (type < 0) {
+ error_setg(errp, "invalid chunk type");
+ return 1;
+ }
+ }
+
+ if (_parse_uint64(s, "cnlb=", &cnlb)) {
+ cnlb_parsed = true;
+ }
+
+ if (_parse_uint64(s, "wp=", &wp)) {
+ wp_parsed = true;
+ }
+
+ if (_parse_uint32(s, "pe_cycles=", &pe_cycles)) {
+ pe_cycles = true;
+ }
+
+ if (state_parsed) {
+ if (state == OCSSD_CHUNK_OFFLINE && wp_parsed) {
+ error_setg(errp, "invalid wp; state is offline");
+ return 1;
+ }
+ }
+
+ if (type_parsed) {
+ if (type == OCSSD_CHUNK_TYPE_RANDOM && wp_parsed) {
+ error_setg(errp, "invalid wp; type has random write capability");
+ return 1;
+ }
+ }
+
+ for (uint16_t g = group; g < group_end; g++) {
+ for (uint16_t p = punit; p < punit_end; p++) {
+ for (uint32_t c = chunk; c < chunk_end; c++) {
+ slba = _make_lba(&ons->addrf, g, p, c, 0);
+ chk = _get_chunk(o, ons, slba);
+ chk_acct = _get_chunk_acct(o, ons, slba);
+ if (!chk) {
+ error_setg(errp, "invalid lba");
+ return 1;
+ }
+
+ if (state_parsed) {
+ /*
+ * Reset the wear index and pe_cycles to zero if the
+ * persisted state is OFFLINE and we move to another state.
+ * If the number of pe_cycles is also changed, it will be
+ * updated subsequently.
+ */
+ if (chk->state == OCSSD_CHUNK_OFFLINE &&
+ state != OCSSD_CHUNK_OFFLINE) {
+ chk->wear_index = 0;
+ chk_acct->pe_cycles = 0;
+ }
+
+ if (state == OCSSD_CHUNK_OFFLINE) {
+ chk->wp = UINT64_MAX;
+ }
+
+ if (state == OCSSD_CHUNK_FREE) {
+ chk->wp = 0;
+ }
+
+ chk->state = state;
+ }
+
+ if (type_parsed) {
+ chk->type = type;
+ if (chk->type == OCSSD_CHUNK_TYPE_RANDOM) {
+ chk->wp = UINT64_MAX;
+ }
+ }
+
+ if (cnlb_parsed) {
+ chk->cnlb = cnlb;
+ if (chk->cnlb > ons->id.geo.clba) {
+ error_setg(errp, "invalid chunk cnlb");
+ return 1;
+ }
+
+ if (chk->cnlb != ons->id.geo.clba) {
+ chk->type |= OCSSD_CHUNK_TYPE_SHRINKED;
+ }
+ }
+
+ if (wp_parsed) {
+ chk->wp = wp;
+ if (chk->wp > chk->cnlb) {
+ error_setg(errp, "invalid chunk wp");
+ return 1;
+ }
+ }
+
+ if (pe_cycles_parsed) {
+ if (pe_cycles > o->hdr.pe_cycles) {
+ error_setg(errp, "invalid number of pe_cycles");
+ return 1;
+ }
+
+ chk->wear_index = _calc_wi(o, pe_cycles);
+ chk_acct->pe_cycles = pe_cycles;
+ }
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int ocssd_load_write_error_injection_from_file(OcssdCtrl *o,
+ const char *fname, Error **errp)
+{
+ ssize_t n;
+ size_t len = 0;
+ int line_num = 0;
+ char *line;
+ Error *local_err = NULL;
+ FILE *fp;
+
+ fp = fopen(fname, "r");
+ if (!fp) {
+ error_setg_errno(errp, errno,
+ "could not open write error injection file (%s): ", fname);
+ return 1;
+ }
+
+ while ((n = getline(&line, &len, fp)) != -1) {
+ line_num++;
+ if (_parse_and_update_write_error_injection(o, line, &local_err)) {
+ error_propagate_prepend(errp, local_err,
+ "could not parse write error injection (line %d): ", line_num);
+ return 1;
+ }
+ }
+
+ fclose(fp);
+
+ return 0;
+}
+
+static int ocssd_load_reset_error_injection_from_file(OcssdCtrl *o,
+ const char *fname, Error **errp)
+{
+ ssize_t n;
+ size_t len = 0;
+ int line_num = 0;
+ char *line;
+ Error *local_err = NULL;
+ FILE *fp;
+
+ fp = fopen(fname, "r");
+ if (!fp) {
+ error_setg_errno(errp, errno,
+ "could not open reset error injection file (%s): ", fname);
+ return 1;
+ }
+
+ while ((n = getline(&line, &len, fp)) != -1) {
+ line_num++;
+ if (_parse_and_update_reset_error_injection(o, line, &local_err)) {
+ error_propagate_prepend(errp, local_err,
+ "could not parse reset error injection (line %d): ", line_num);
+ return 1;
+ }
+ }
+
+ fclose(fp);
+
+ return 0;
+}
+
+static int ocssd_load_chunk_info_from_file(OcssdCtrl *o, const char *fname,
+ Error **errp)
+{
+ ssize_t n;
+ size_t len = 0;
+ int line_num = 0;
+ char *line;
+ Error *local_err = NULL;
+ FILE *fp;
+
+ fp = fopen(fname, "r");
+ if (!fp) {
+ error_setg_errno(errp, errno, "could not open chunk info file");
+ return 1;
+ }
+
+ while ((n = getline(&line, &len, fp)) != -1) {
+ line_num++;
+ if (_parse_and_update_chunk_info(o, line, &local_err)) {
+ error_propagate_prepend(errp, local_err,
+ "could not parse chunk info (line %d): ", line_num);
+ return 1;
+ }
+ }
+
+ fclose(fp);
+
+ return 0;
+}
+
+static void ocssd_ns_commit_chunk_acct(OcssdCtrl *o, OcssdNamespace *ons,
+ NvmeRequest *req, OcssdChunkDescriptor *chk,
+ OcssdChunkAcctDescriptor *chk_acct)
+{
+ NvmeCtrl *n = &o->nvme;
+ NvmeBlockBackendRequest *blk_req = nvme_blk_req_get(n, req, NULL);
+
+ blk_req->blk_offset = ons->acct.blk_offset;
+
+ qemu_iovec_init(&blk_req->iov, 1);
+ if (chk) {
+ qemu_iovec_add(&blk_req->iov, chk_acct,
+ sizeof(OcssdChunkAcctDescriptor));
+ blk_req->blk_offset += _chk_idx(o, ons, chk->slba) *
+ sizeof(OcssdChunkAcctDescriptor);
+ } else {
+ qemu_iovec_add(&blk_req->iov, ons->acct.descr, ons->acct.size);
+ }
+
+ QTAILQ_INSERT_TAIL(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ block_acct_start(blk_get_stats(n->conf.blk), &blk_req->acct,
+ blk_req->iov.size, BLOCK_ACCT_WRITE);
+
+ blk_req->aiocb = blk_aio_pwritev(n->conf.blk, blk_req->blk_offset,
+ &blk_req->iov, 0, nvme_rw_cb, blk_req);
+}
+
+static void ocssd_ns_commit_chunk_state(OcssdCtrl *o, OcssdNamespace *ons,
+ NvmeRequest *req, OcssdChunkDescriptor *chk)
+{
+ NvmeCtrl *n = &o->nvme;
+ NvmeBlockBackendRequest *blk_req = nvme_blk_req_get(n, req, NULL);
+
+ blk_req->blk_offset = ons->info.blk_offset;
+
+ qemu_iovec_init(&blk_req->iov, 1);
+ if (chk) {
+ qemu_iovec_add(&blk_req->iov, chk, sizeof(OcssdChunkDescriptor));
+ blk_req->blk_offset += _chk_idx(o, ons, chk->slba) *
+ sizeof(OcssdChunkDescriptor);
+ } else {
+ qemu_iovec_add(&blk_req->iov, ons->info.descr, ons->info.size);
+ }
+
+ QTAILQ_INSERT_TAIL(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ block_acct_start(blk_get_stats(n->conf.blk), &blk_req->acct,
+ blk_req->iov.size, BLOCK_ACCT_WRITE);
+
+ blk_req->aiocb = blk_aio_pwritev(n->conf.blk, blk_req->blk_offset,
+ &blk_req->iov, 0, nvme_rw_cb, blk_req);
+}
+
+static int ocssd_ns_load_chunk_acct(OcssdCtrl *o, OcssdNamespace *ons)
+{
+ BlockBackend *blk = o->nvme.conf.blk;
+ return blk_pread(blk, ons->acct.blk_offset, ons->acct.descr,
+ ons->acct.size);
+}
+
+static int ocssd_ns_load_chunk_info(OcssdCtrl *o, OcssdNamespace *ons)
+{
+ BlockBackend *blk = o->nvme.conf.blk;
+ return blk_pread(blk, ons->info.blk_offset, ons->info.descr,
+ ons->info.size);
+}
+
+static uint16_t ocssd_do_get_chunk_info(OcssdCtrl *o, NvmeCmd *cmd,
+ uint32_t buf_len, uint64_t off, NvmeRequest *req)
+{
+ uint8_t *log_page;
+ uint32_t log_len, trans_len;
+
+ OcssdNamespace *ons = _ons(o, le32_to_cpu(cmd->nsid));
+ if (!ons) {
+ trace_ocssd_err(req->cqe.cid, "chunk info requires nsid",
+ NVME_INVALID_FIELD | NVME_DNR);
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ log_len = ons->chks_total * sizeof(OcssdChunkDescriptor);
+
+ if (off > log_len) {
+ trace_ocssd_err(req->cqe.cid, "invalid log page offset",
+ NVME_INVALID_FIELD | NVME_DNR);
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ trans_len = MIN(log_len - off, buf_len);
+ log_page = (uint8_t *) ons->info.descr + off;
+
+ return nvme_dma_read(&o->nvme, log_page, trans_len, cmd, req);
+}
+
+static uint16_t ocssd_do_get_chunk_notification(OcssdCtrl *o, NvmeCmd *cmd,
+ uint32_t buf_len, uint64_t off, uint8_t rae, NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+
+ uint8_t *log_page;
+ uint32_t log_len, trans_len;
+
+ log_len = OCSSD_MAX_CHUNK_NOTIFICATIONS * sizeof(OcssdChunkNotification);
+
+ if (off > log_len) {
+ trace_ocssd_err(req->cqe.cid, "invalid log page offset",
+ NVME_INVALID_FIELD | NVME_DNR);
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ trans_len = MIN(log_len - off, buf_len);
+ log_page = (uint8_t *) &o->notifications[off];
+
+ if (!rae) {
+ nvme_clear_events(n, NVME_AER_TYPE_VENDOR_SPECIFIC);
+ }
+
+ return nvme_dma_read(&o->nvme, log_page, trans_len, cmd, req);
+}
+
+static void ocssd_add_chunk_notification(OcssdCtrl *o, OcssdNamespace *ons,
+ uint64_t lba, uint16_t state, uint8_t mask, uint16_t nlb)
+{
+ NvmeNamespace *ns = ons->ns;
+ OcssdChunkNotification *notice;
+
+ notice = &o->notifications[o->notifications_index];
+ notice->nc = cpu_to_le64(++(o->notifications_count));
+ notice->lba = cpu_to_le64(lba);
+ notice->nsid = cpu_to_le32(ns->id);
+ notice->state = cpu_to_le16(state);
+ notice->mask = mask;
+ notice->nlb = cpu_to_le16(nlb);
+
+ o->notifications_index = (o->notifications_index + 1) %
+ OCSSD_MAX_CHUNK_NOTIFICATIONS;
+}
+
+static uint16_t ocssd_rw_check_chunk_read(OcssdCtrl *o, NvmeCmd *cmd,
+ NvmeRequest *req, uint64_t lba)
+{
+ NvmeNamespace *ns = req->ns;
+ OcssdNamespace *ons = &o->namespaces[ns->id - 1];
+ OcssdAddrF *addrf = &ons->addrf;
+ OcssdIdWrt *wrt = &ons->id.wrt;
+
+ OcssdChunkDescriptor *chk;
+ uint64_t sectr, mw_cunits, wp;
+ uint8_t state;
+
+ chk = _get_chunk(o, ons, lba);
+ if (!chk) {
+ trace_ocssd_err_invalid_chunk(req->cqe.cid,
+ lba & ~ons->addrf.sec_mask);
+ return NVME_DULB;
+ }
+
+ sectr = _sectr(addrf, lba);
+ mw_cunits = wrt->mw_cunits;
+ wp = chk->wp;
+ state = chk->state;
+
+ if (chk->type == OCSSD_CHUNK_TYPE_RANDOM) {
+ /*
+ * For OCSSD_CHUNK_TYPE_RANDOM it is sufficient to ensure that the
+ * chunk is OPEN and that we are reading a valid address.
+ */
+ if (state != OCSSD_CHUNK_OPEN || sectr >= chk->cnlb) {
+ trace_ocssd_err_invalid_chunk_state(req->cqe.cid,
+ lba & ~(ons->addrf.sec_mask), chk->state);
+ return NVME_DULB;
+ }
+
+ return NVME_SUCCESS;
+ }
+
+ if (state == OCSSD_CHUNK_CLOSED && sectr < wp) {
+ return NVME_SUCCESS;
+ }
+
+ if (state == OCSSD_CHUNK_OPEN) {
+ if (wp < mw_cunits) {
+ return NVME_DULB;
+ }
+
+ if (sectr < (wp - mw_cunits)) {
+ return NVME_SUCCESS;
+ }
+ }
+
+ return NVME_DULB;
+}
+
+static uint16_t ocssd_rw_check_chunk_write(OcssdCtrl *o, NvmeCmd *cmd,
+ uint64_t lba, uint32_t ws, NvmeRequest *req)
+{
+ OcssdChunkDescriptor *chk;
+ NvmeNamespace *ns = req->ns;
+ OcssdNamespace *ons = &o->namespaces[ns->id - 1];
+
+ OcssdIdWrt *wrt = &ons->id.wrt;
+
+ chk = _get_chunk(o, ons, lba);
+ if (!chk) {
+ trace_ocssd_err_invalid_chunk(req->cqe.cid,
+ lba & ~ons->addrf.sec_mask);
+ return NVME_WRITE_FAULT | NVME_DNR;
+ }
+
+ uint32_t start_sectr = lba & ons->addrf.sec_mask;
+ uint32_t end_sectr = start_sectr + ws;
+
+ /* check if we are at all allowed to write to the chunk */
+ if (chk->state == OCSSD_CHUNK_OFFLINE ||
+ chk->state == OCSSD_CHUNK_CLOSED) {
+ trace_ocssd_err_invalid_chunk_state(req->cqe.cid,
+ lba & ~(ons->addrf.sec_mask), chk->state);
+ return NVME_WRITE_FAULT | NVME_DNR;
+ }
+
+ if (end_sectr > chk->cnlb) {
+ trace_ocssd_err_out_of_bounds(req->cqe.cid, end_sectr, chk->cnlb);
+ return NVME_WRITE_FAULT | NVME_DNR;
+ }
+
+
+ if (chk->type == OCSSD_CHUNK_TYPE_RANDOM) {
+ return NVME_SUCCESS;
+ }
+
+ if (ws < wrt->ws_min || (ws % wrt->ws_min) != 0) {
+ trace_ocssd_err_write_constraints(req->cqe.cid, ws, wrt->ws_min);
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ /* check that the write begins at the current wp */
+ if (start_sectr != chk->wp) {
+ trace_ocssd_err_out_of_order(req->cqe.cid, start_sectr, chk->wp);
+ return OCSSD_OUT_OF_ORDER_WRITE | NVME_DNR;
+ }
+
+ return NVME_SUCCESS;
+}
+
+static uint16_t ocssd_rw_check_vector_read_req(OcssdCtrl *o, NvmeCmd *cmd,
+ NvmeRequest *req, uint64_t *dulbe)
+{
+ uint16_t status;
+
+ assert(dulbe);
+
+ for (int i = 0; i < req->nlb; i++) {
+ status = ocssd_rw_check_chunk_read(o, cmd, req, _vlba(req, i));
+
+ if (status) {
+ if (nvme_is_error(status, NVME_DULB)) {
+ *dulbe |= (1 << i);
+ continue;
+ }
+
+ return status;
+ }
+ }
+
+ return NVME_SUCCESS;
+}
+
+static uint16_t ocssd_rw_check_vector_write_req(OcssdCtrl *o, NvmeCmd *cmd,
+ NvmeRequest *req)
+{
+ NvmeNamespace *ns = req->ns;
+ OcssdNamespace *ons = &o->namespaces[ns->id - 1];
+ OcssdAddrF *addrf = &ons->addrf;
+
+ uint64_t prev_lba = _vlba(req, 0);
+ uint64_t prev_chk_idx = _chk_idx(o, ons, prev_lba);
+ uint32_t sectr = _sectr(addrf, prev_lba);
+ uint16_t ws = 1, status;
+
+ for (uint16_t i = 1; i < req->nlb; i++) {
+ uint64_t lba = _vlba(req, i);
+ uint64_t chk_idx = _chk_idx(o, ons, lba);
+
+ /*
+ * It is assumed that LBAs for different chunks are laid out
+ * contiguously and sorted with increasing addresses.
+ */
+ if (prev_chk_idx != chk_idx) {
+ status = ocssd_rw_check_chunk_write(o, cmd, prev_lba, ws, req);
+ if (status) {
+ req->cqe.res64 = cpu_to_le64((1 << req->nlb) - 1);
+ return status;
+ }
+
+ prev_lba = lba;
+ prev_chk_idx = chk_idx;
+ sectr = _sectr(addrf, prev_lba);
+ ws = 1;
+
+ continue;
+ }
+
+ if (++sectr != _sectr(addrf, lba)) {
+ return OCSSD_OUT_OF_ORDER_WRITE | NVME_DNR;
+ }
+
+ ws++;
+ }
+
+ return ocssd_rw_check_chunk_write(o, cmd, prev_lba, ws, req);
+}
+
+static uint16_t ocssd_rw_check_scalar_req(OcssdCtrl *o, NvmeCmd *cmd,
+ NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+ uint16_t status;
+
+ status = nvme_rw_check_req(n, cmd, req);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "nvme_rw_check_req", status);
+ return status;
+ }
+
+ if (req->is_write) {
+ return ocssd_rw_check_chunk_write(o, cmd, req->slba, req->nlb, req);
+ }
+
+ for (uint16_t i = 0; i < req->nlb; i++) {
+ status = ocssd_rw_check_chunk_read(o, cmd, req, req->slba + i);
+ if (nvme_is_error(status, NVME_DULB)) {
+ if (NVME_ERR_REC_DULBE(n->features.err_rec)) {
+ return NVME_DULB | NVME_DNR;
+ }
+
+ break;
+ }
+
+ return status;
+ }
+
+ return NVME_SUCCESS;
+}
+
+static uint16_t ocssd_rw_check_vector_req(OcssdCtrl *o, NvmeCmd *cmd,
+ NvmeRequest *req, uint64_t *dulbe)
+{
+ NvmeCtrl *n = &o->nvme;
+ uint16_t status;
+
+ status = nvme_rw_check_req(n, cmd, req);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "nvme_rw_check_req", status);
+ return status;
+ }
+
+ if (req->is_write) {
+ return ocssd_rw_check_vector_write_req(o, cmd, req);
+ }
+
+ return ocssd_rw_check_vector_read_req(o, cmd, req, dulbe);
+}
+
+static uint16_t ocssd_blk_setup_scalar(NvmeCtrl *n, NvmeNamespace *ns,
+ QEMUSGList *qsg, uint64_t blk_offset, uint32_t unit_len, NvmeRequest *req)
+{
+ OcssdCtrl *o = OCSSD(n);
+ OcssdNamespace *ons = _ons(o, req->ns->id);
+
+ NvmeBlockBackendRequest *blk_req = nvme_blk_req_get(n, req, qsg);
+ if (!blk_req) {
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "nvme_blk_req_get: %s",
+ "could not allocate memory");
+ return NVME_INTERNAL_DEV_ERROR;
+ }
+
+ blk_req->slba = req->slba;
+ blk_req->nlb = req->nlb;
+ blk_req->blk_offset = blk_offset + _sectr_idx(o, ons, req->slba) *
+ unit_len;
+
+ QTAILQ_INSERT_TAIL(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ return NVME_SUCCESS;
+}
+
+static uint16_t ocssd_blk_setup_vector(NvmeCtrl *n, NvmeNamespace *ns,
+ QEMUSGList *qsg, uint64_t blk_offset, uint32_t unit_len, NvmeRequest *req)
+{
+ OcssdCtrl *o = OCSSD(n);
+ OcssdNamespace *ons = _ons(o, req->ns->id);
+
+ size_t curr_byte = 0;
+ uint64_t lba, chk_idx, prev_chk_idx;
+ int curr_sge = 0;
+
+ NvmeBlockBackendRequest *blk_req = nvme_blk_req_get(n, req, NULL);
+ pci_dma_sglist_init(&blk_req->qsg, &n->parent_obj, 1);
+
+ /*
+ * Similar to ocssd_rw_check_vector_write_req, it is assumed that LBAs for
+ * different chunks are laid out contiguously and sorted with increasing
+ * addresses. Thus, split request into multiple NvmeBlockBackendRequest for
+ * each chunk involved unconditionally, even if the last sector of chunk N
+ * has address K and the first address of chunk N+1 has address K+1 and
+ * would be contiguous on the block backend. The invariant that a single
+ * NvmeBlockBackendRequest corresponds to at most one chunk is used in
+ * e.g. write error injection.
+ */
+
+ lba = _vlba(req, 0);
+ prev_chk_idx = _chk_idx(o, ons, lba);
+
+ blk_req->blk_offset = blk_offset + _sectr_idx(o, ons, lba) * unit_len;
+ blk_req->slba = lba;
+ blk_req->nlb = 1;
+
+ QTAILQ_INSERT_TAIL(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ for (uint16_t i = 1; i < req->nlb; i++) {
+ lba = _vlba(req, i);
+ chk_idx = _chk_idx(o, ons, lba);
+
+ if (prev_chk_idx != chk_idx) {
+ _sglist_copy_from(&blk_req->qsg, qsg, &curr_sge, &curr_byte,
+ blk_req->nlb * unit_len);
+
+ blk_req = nvme_blk_req_get(n, req, NULL);
+ if (!blk_req) {
+ NVME_GUEST_ERR(nvme_err_internal_dev_error,
+ "nvme_blk_req_get: %s", "could not allocate memory");
+ return NVME_INTERNAL_DEV_ERROR;
+ }
+
+ pci_dma_sglist_init(&blk_req->qsg, &n->parent_obj, 1);
+
+ blk_req->blk_offset = blk_offset + _sectr_idx(o, ons, lba) *
+ unit_len;
+ blk_req->slba = lba;
+
+ QTAILQ_INSERT_TAIL(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ prev_chk_idx = chk_idx;
+ }
+
+ blk_req->nlb++;
+ }
+
+ _sglist_copy_from(&blk_req->qsg, qsg, &curr_sge, &curr_byte,
+ blk_req->nlb * unit_len);
+
+ return NVME_SUCCESS;
+}
+
+static uint16_t ocssd_do_chunk_reset(OcssdCtrl *o, OcssdNamespace *ons,
+ uint64_t lba, hwaddr mptr, NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+ OcssdChunkDescriptor *chk;
+ OcssdChunkAcctDescriptor *chk_acct;
+ uint8_t p;
+
+ chk = _get_chunk(o, ons, lba);
+ if (!chk) {
+ trace_ocssd_err_invalid_chunk(req->cqe.cid,
+ lba & ~ons->addrf.sec_mask);
+ return OCSSD_INVALID_RESET | NVME_DNR;
+ }
+
+ if (chk->state & OCSSD_CHUNK_RESETABLE) {
+ switch (chk->state) {
+ case OCSSD_CHUNK_FREE:
+ trace_ocssd_notice_double_reset(req->cqe.cid, lba);
+
+ if (!(ons->id.mccap & OCSSD_IDENTITY_MCCAP_MULTIPLE_RESETS)) {
+ return OCSSD_INVALID_RESET | NVME_DNR;
+ }
+
+ break;
+
+ case OCSSD_CHUNK_OPEN:
+ trace_ocssd_notice_early_reset(req->cqe.cid, lba, chk->wp);
+ if (!(ons->id.mccap & OCSSD_IDENTITY_MCCAP_EARLY_RESET)) {
+ return OCSSD_INVALID_RESET | NVME_DNR;
+ }
+
+ break;
+ }
+
+ if (ons->resetfail) {
+ p = ons->resetfail[_chk_idx(o, ons, lba)];
+
+ if (p == 100 || (rand() % 100) < p) {
+ chk->state = OCSSD_CHUNK_OFFLINE;
+ chk->wp = UINT64_MAX;
+ trace_ocssd_inject_reset_err(req->cqe.cid, p, lba);
+ return OCSSD_INVALID_RESET | NVME_DNR;
+ }
+ }
+
+ chk->state = OCSSD_CHUNK_FREE;
+
+ if (chk->type == OCSSD_CHUNK_TYPE_SEQUENTIAL) {
+ chk->wp = 0;
+
+ chk_acct = _get_chunk_acct(o, ons, lba);
+
+ if (chk_acct->pe_cycles < o->hdr.pe_cycles) {
+ chk_acct->pe_cycles++;
+
+ ons->wear_index_total++;
+ ons->wear_index_avg = ons->wear_index_total / ons->chks_total;
+
+ chk->wear_index = _calc_wi(o, chk_acct->pe_cycles);
+
+ if (_wi_outside_threshold(ons, chk)) {
+ ocssd_add_chunk_notification(o, ons, chk->slba,
+ OCSSD_CHUNK_NOTIFICATION_STATE_WLI,
+ OCSSD_CHUNK_NOTIFICATION_MASK_CHUNK, 0);
+
+ nvme_enqueue_event(n, NVME_AER_TYPE_VENDOR_SPECIFIC, 0x0,
+ OCSSD_CHUNK_NOTIFICATION);
+ }
+ }
+
+ if (chk->wear_index == 255) {
+ chk->state = OCSSD_CHUNK_OFFLINE;
+ }
+
+ ocssd_ns_commit_chunk_acct(o, ons, req, chk, chk_acct);
+ }
+
+ if (mptr) {
+ nvme_addr_write(n, mptr, chk, sizeof(*chk));
+ }
+
+ ocssd_ns_commit_chunk_state(o, ons, req, chk);
+
+ return NVME_SUCCESS;
+ }
+
+ trace_ocssd_err_offline_chunk(req->cqe.cid, lba);
+
+ return OCSSD_OFFLINE_CHUNK | NVME_DNR;
+}
+
+static uint16_t ocssd_do_advance_wp(OcssdCtrl *o, OcssdNamespace *ons,
+ uint64_t lba, uint16_t nlb, NvmeRequest *req)
+{
+ OcssdChunkDescriptor *chk;
+
+ trace_ocssd_advance_wp(req->cqe.cid, lba, nlb);
+ _dprint_lba(o, ons, lba);
+
+ chk = _get_chunk(o, ons, lba);
+ if (!chk) {
+ NVME_GUEST_ERR(ocssd_err_invalid_chunk,
+ "invalid chunk; cid %d slba 0x%lx", req->cqe.cid,
+ lba & ~ons->addrf.sec_mask);
+ return NVME_INTERNAL_DEV_ERROR | NVME_DNR;
+ }
+
+ if (chk->state == OCSSD_CHUNK_FREE) {
+ chk->state = OCSSD_CHUNK_OPEN;
+ }
+
+ if (chk->type == OCSSD_CHUNK_TYPE_RANDOM) {
+ goto commit;
+ }
+
+ if (chk->state != OCSSD_CHUNK_OPEN) {
+ NVME_GUEST_ERR(ocssd_err_invalid_chunk_state,
+ "invalid chunk state; cid %d slba 0x%lx state 0x%x",
+ req->cqe.cid, lba, chk->state);
+ return NVME_INTERNAL_DEV_ERROR | NVME_DNR;
+ }
+
+ chk->wp += nlb;
+ if (chk->wp == chk->cnlb) {
+ chk->state = OCSSD_CHUNK_CLOSED;
+ }
+
+commit:
+ ocssd_ns_commit_chunk_state(o, ons, req, chk);
+
+ return NVME_SUCCESS;
+}
+
+static void ocssd_dsm_cb(void *opaque, int ret)
+{
+ NvmeBlockBackendRequest *blk_req = opaque;
+ NvmeRequest *req = blk_req->req;
+ NvmeSQueue *sq = req->sq;
+ NvmeCtrl *n = sq->ctrl;
+ NvmeCQueue *cq = n->cq[sq->cqid];
+ NvmeNamespace *ns = req->ns;
+
+ OcssdCtrl *o = OCSSD(n);
+ OcssdNamespace *ons = &o->namespaces[ns->id - 1];
+
+ uint16_t status;
+
+ QTAILQ_REMOVE(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ if (!ret) {
+ status = ocssd_do_chunk_reset(o, ons, blk_req->slba, 0x0, req);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "ocssd_do_chunk_reset", status);
+ req->status = status;
+ goto out;
+ }
+ } else {
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "block request failed: %s",
+ strerror(-ret));
+ req->status = NVME_INTERNAL_DEV_ERROR;
+ }
+
+out:
+ if (QTAILQ_EMPTY(&req->blk_req_tailq)) {
+ nvme_enqueue_req_completion(cq, req);
+ }
+
+ nvme_blk_req_put(n, blk_req);
+}
+
+
+static uint16_t ocssd_dsm(OcssdCtrl *o, NvmeCmd *cmd, NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+ NvmeNamespace *ns = req->ns;
+ NvmeDsmCmd *dsm = (NvmeDsmCmd *) cmd;
+
+ OcssdNamespace *ons = &o->namespaces[ns->id - 1];
+
+ uint16_t status;
+
+ if (dsm->attributes & NVME_DSMGMT_AD) {
+ NvmeBlockBackendRequest *blk_req;
+ OcssdChunkDescriptor *chk;
+
+ uint16_t nr = (dsm->nr & 0xff) + 1;
+ uint8_t lbads = nvme_ns_lbads(ns);
+
+ NvmeDsmRange range[nr];
+
+ status = nvme_dma_write(n, (uint8_t *) range, sizeof(range), cmd, req);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "nvme_dma_write", status);
+ return status;
+ }
+
+ for (int i = 0; i < nr; i++) {
+ chk = _get_chunk(o, ons, range[i].slba);
+
+ if (!chk) {
+ trace_ocssd_err_invalid_chunk(req->cqe.cid,
+ range[i].slba & ~ons->addrf.sec_mask);
+ return OCSSD_INVALID_RESET | NVME_DNR;
+ }
+
+ if (range[i].nlb != chk->cnlb) {
+ trace_ocssd_err(req->cqe.cid, "invalid reset size",
+ NVME_LBA_RANGE);
+ return NVME_LBA_RANGE | NVME_DNR;
+ }
+
+ blk_req = nvme_blk_req_get(n, req, NULL);
+ if (!blk_req) {
+ NVME_GUEST_ERR(nvme_err_internal_dev_error,
+ "nvme_blk_req_get: %s", "could not allocate memory");
+ return NVME_INTERNAL_DEV_ERROR;
+ }
+
+ blk_req->slba = range[i].slba;
+
+ QTAILQ_INSERT_TAIL(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ blk_req->aiocb = blk_aio_pdiscard(n->conf.blk,
+ ns->blk_offset + _sectr_idx(o, ons, range[i].slba),
+ range[i].nlb << lbads,
+ ocssd_dsm_cb, blk_req);
+ }
+
+ return NVME_NO_COMPLETE;
+ }
+
+ return NVME_SUCCESS;
+}
+
+static void ocssd_reset_cb(void *opaque, int ret)
+{
+ NvmeBlockBackendRequest *blk_req = opaque;
+ NvmeRequest *req = blk_req->req;
+ NvmeSQueue *sq = req->sq;
+ NvmeCtrl *n = sq->ctrl;
+ NvmeCQueue *cq = n->cq[sq->cqid];
+ NvmeNamespace *ns = req->ns;
+
+ OcssdCtrl *o = OCSSD(n);
+ OcssdNamespace *ons = &o->namespaces[ns->id - 1];
+
+ hwaddr mptr;
+ uint16_t status;
+
+ QTAILQ_REMOVE(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ if (!ret) {
+ /*
+ * blk_req->nlb has been hijacked to store the index that this entry
+ * held in the LBA list, so use that to calculate the MPTR offset.
+ */
+ mptr = req->mptr ? req->mptr +
+ blk_req->nlb * sizeof(OcssdChunkDescriptor) : 0x0;
+ status = ocssd_do_chunk_reset(o, ons, blk_req->slba, mptr, req);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "ocssd_do_chunk_reset", status);
+ req->status = status;
+ goto out;
+ }
+ } else {
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "block request failed: %s",
+ strerror(-ret));
+ req->status = NVME_INTERNAL_DEV_ERROR;
+ }
+
+out:
+ if (QTAILQ_EMPTY(&req->blk_req_tailq)) {
+ nvme_enqueue_req_completion(cq, req);
+ }
+
+ nvme_blk_req_put(n, blk_req);
+}
+
+static uint16_t ocssd_reset(OcssdCtrl *o, NvmeCmd *cmd, NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+ OcssdRwCmd *rst = (OcssdRwCmd *) cmd;
+ OcssdNamespace *ons = _ons(o, req->ns->id);
+ hwaddr lbal_addr = le64_to_cpu(rst->lbal);
+ uint16_t nlb = le16_to_cpu(rst->nlb) + 1;
+ uint8_t lbads = nvme_ns_lbads(req->ns);
+ uint16_t status = NVME_NO_COMPLETE;
+ uint64_t *lbal;
+
+ trace_ocssd_reset(req->cqe.cid, nlb);
+
+ req->nlb = nlb;
+ req->mptr = le64_to_cpu(cmd->mptr);
+
+ _get_lba_list(o, lbal_addr, &lbal, req);
+ req->slba = (uint64_t) lbal;
+
+ /*
+ * The resetting of multiple chunks is done asynchronously, so hijack
+ * blk_req->nlb to store the LBAL index which is required for the callback
+ * to know the index in MPTR at which to store the updated chunk
+ * descriptor.
+ */
+ for (int i = 0; i < nlb; i++) {
+ OcssdChunkDescriptor *chk;
+ NvmeBlockBackendRequest *blk_req = nvme_blk_req_get(n, req, NULL);
+ if (!blk_req) {
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "nvme_blk_req_get: %s",
+ "could not allocate memory");
+ status = NVME_INTERNAL_DEV_ERROR | NVME_DNR;
+ goto out;
+ }
+
+ blk_req->slba = _vlba(req, i);
+ blk_req->nlb = i;
+
+ chk = _get_chunk(o, ons, blk_req->slba);
+ if (!chk) {
+ trace_ocssd_err_invalid_chunk(req->cqe.cid,
+ blk_req->slba & ~ons->addrf.sec_mask);
+ status = OCSSD_INVALID_RESET | NVME_DNR;
+ goto out;
+ }
+
+ QTAILQ_INSERT_TAIL(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ blk_req->aiocb = blk_aio_pdiscard(n->conf.blk,
+ req->ns->blk_offset + (_sectr_idx(o, ons, blk_req->slba) << lbads),
+ chk->cnlb << lbads, ocssd_reset_cb, blk_req);
+ }
+
+out:
+ if (req->nlb > 1) {
+ g_free((uint64_t *) req->slba);
+ }
+
+ return status;
+}
+
+static uint16_t ocssd_maybe_write_error_inject(OcssdCtrl *o,
+ NvmeBlockBackendRequest *blk_req)
+{
+ NvmeRequest *req = blk_req->req;
+ NvmeNamespace *ns = req->ns;
+ OcssdNamespace *ons = &o->namespaces[ns->id - 1];
+ OcssdChunkDescriptor *chk;
+ uint8_t p;
+ uint64_t cidx, slba = blk_req->slba;
+
+ if (!ons->writefail || !req->is_write) {
+ return NVME_SUCCESS;
+ }
+
+ for (uint16_t i = 0; i < blk_req->nlb; i++) {
+ p = ons->writefail[_sectr_idx(o, ons, slba + i)];
+
+ if (p && (p == 100 || (rand() % 100) < p)) {
+ trace_ocssd_inject_write_err(req->cqe.cid, p, slba + i);
+
+ chk = _get_chunk(o, ons, slba);
+ if (!chk) {
+ NVME_GUEST_ERR(ocssd_err_invalid_chunk,
+ "invalid chunk; cid %d addr 0x%lx", req->cqe.cid,
+ slba & ~ons->addrf.sec_mask);
+ return NVME_INTERNAL_DEV_ERROR | NVME_DNR;
+ }
+
+ cidx = _chk_idx(o, ons, slba + i);
+ chk->state = OCSSD_CHUNK_CLOSED;
+
+ ocssd_ns_commit_chunk_state(o, ons, req, chk);
+ ons->resetfail[cidx] = 100;
+
+ if (_is_vector_request(req)) {
+ for (uint16_t j = 0; j < req->nlb; j++) {
+ if (cidx == _chk_idx(o, ons, slba)) {
+ bitmap_set(&req->cqe.res64, j, 1);
+ }
+ }
+ }
+
+ return OCSSD_CHUNK_EARLY_CLOSE | NVME_DNR;
+ }
+ }
+
+ return NVME_SUCCESS;
+}
+
+static void ocssd_rwc_aio_complete(OcssdCtrl *o,
+ NvmeBlockBackendRequest *blk_req, int ret)
+{
+ NvmeCtrl *n = &o->nvme;
+ NvmeRequest *req = blk_req->req;
+ NvmeNamespace *ns = req->ns;
+ OcssdNamespace *ons = &o->namespaces[ns->id - 1];
+ uint16_t status;
+
+ if (!ret) {
+ block_acct_done(blk_get_stats(n->conf.blk), &blk_req->acct);
+
+ if (req->is_write && blk_req->blk_offset >= ns->blk_offset &&
+ blk_req->blk_offset < ns->blk_offset_md) {
+
+ /*
+ * We know that each NvmeBlockBackendRequest corresponds to a write
+ * to at most one chunk (one contiguous write). This way, we can
+ * allow a write to a single chunk to fail (while leaving the write
+ * pointer intact), but allow writes to other chunks to proceed.
+ */
+ status = ocssd_maybe_write_error_inject(o, blk_req);
+ if (!status) {
+ status = ocssd_do_advance_wp(o, ons, blk_req->slba,
+ blk_req->nlb, req);
+ }
+
+ /*
+ * An internal device error trumps all other errors, but there is
+ * no way of triaging other errors, so only set an error if one has
+ * not already been set.
+ */
+ if (status) {
+ if (nvme_is_error(status, NVME_INTERNAL_DEV_ERROR)) {
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "%s",
+ "internal device error");
+ req->status = status;
+ }
+
+ if (!req->status) {
+ req->status = status;
+ }
+ }
+ }
+ } else {
+ block_acct_failed(blk_get_stats(n->conf.blk), &blk_req->acct);
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "block request failed: %s",
+ strerror(-ret));
+ req->status = NVME_INTERNAL_DEV_ERROR | NVME_DNR;
+ }
+}
+
+static void ocssd_copy_out_cb(void *opaque, int ret)
+{
+ NvmeBlockBackendRequest *blk_req = opaque;
+ NvmeRequest *req = blk_req->req;
+ NvmeSQueue *sq = req->sq;
+ NvmeCtrl *n = sq->ctrl;
+ NvmeCQueue *cq = n->cq[sq->cqid];
+
+ OcssdCtrl *o = OCSSD(n);
+ hwaddr addr;
+
+ trace_ocssd_copy_out_cb(req->cqe.cid, req->ns->id);
+
+ QTAILQ_REMOVE(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ ocssd_rwc_aio_complete(o, blk_req, ret);
+ nvme_blk_req_put(n, blk_req);
+
+ if (QTAILQ_EMPTY(&req->blk_req_tailq)) {
+ /* free the bounce buffers */
+ addr = req->cmd.cdw12;
+ addr = (addr << 32) | req->cmd.cdw13;
+ g_free((void *) addr);
+ g_free((void *) req->cmd.mptr);
+
+ nvme_enqueue_req_completion(cq, req);
+ }
+}
+
+static void ocssd_copy_in_cb(void *opaque, int ret)
+{
+ NvmeBlockBackendRequest *blk_req = opaque;
+ NvmeRequest *req = blk_req->req;
+ NvmeSQueue *sq = req->sq;
+ NvmeCtrl *n = sq->ctrl;
+ NvmeCQueue *cq = n->cq[sq->cqid];
+ NvmeNamespace *ns = req->ns;
+
+ OcssdCtrl *o = OCSSD(n);
+ OcssdCopyCmd *cpy = (OcssdCopyCmd *) &req->cmd;
+
+ hwaddr addr = le64_to_cpu(cpy->dlbal);
+ uint64_t *dlbal;
+ size_t unit_len = nvme_ns_lbads_bytes(ns);
+ size_t unit_len_meta = nvme_ns_ms(ns);
+ uint16_t status;
+
+ QEMUSGList qsg;
+
+ QTAILQ_REMOVE(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ trace_ocssd_copy_in_cb(req->cqe.cid, req->ns->id);
+
+ if (!ret) {
+ block_acct_done(blk_get_stats(n->conf.blk), &blk_req->acct);
+ } else {
+ block_acct_failed(blk_get_stats(n->conf.blk), &blk_req->acct);
+ NVME_GUEST_ERR(nvme_err_internal_dev_error, "block request failed: %s",
+ strerror(-ret));
+ req->status = NVME_INTERNAL_DEV_ERROR | NVME_DNR;
+ }
+
+ nvme_blk_req_put(n, blk_req);
+
+ if (QTAILQ_EMPTY(&req->blk_req_tailq)) {
+ _get_lba_list(o, addr, &dlbal, req);
+ req->slba = (uint64_t) dlbal;
+
+ /* second phase of copy is a write */
+ req->is_write = true;
+
+ status = ocssd_rw_check_vector_req(o, &req->cmd, req, NULL);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "ocssd_rw_check_vector_req",
+ status);
+ goto out;
+ }
+
+ addr = req->cmd.cdw12;
+ addr = (addr << 32) | req->cmd.cdw13;
+
+ pci_dma_sglist_init(&qsg, &n->parent_obj, 1);
+ qemu_sglist_add(&qsg, addr, req->nlb * unit_len);
+
+ status = ocssd_blk_setup_vector(n, ns, &qsg, ns->blk_offset, unit_len,
+ req);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "ocssd_blk_setup_vector", status);
+ goto out_sglist_destroy;
+ }
+
+ if (n->params.ms) {
+ qsg.nsg = 0;
+ qsg.size = 0;
+
+ qemu_sglist_add(&qsg, req->cmd.mptr, req->nlb * unit_len_meta);
+
+ status = ocssd_blk_setup_vector(n, ns, &qsg, ns->blk_offset_md,
+ unit_len_meta, req);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "ocssd_blk_setup_vector", status);
+ goto out_sglist_destroy;
+ }
+ }
+
+ QTAILQ_FOREACH(blk_req, &req->blk_req_tailq, tailq_entry) {
+ qemu_iovec_init(&blk_req->iov, blk_req->qsg.nsg);
+ _sglist_to_iov(n, &blk_req->qsg, &blk_req->iov);
+
+ block_acct_start(blk_get_stats(n->conf.blk), &blk_req->acct,
+ blk_req->iov.size, BLOCK_ACCT_WRITE);
+
+ blk_req->aiocb = blk_aio_pwritev(n->conf.blk, blk_req->blk_offset,
+ &blk_req->iov, 0, ocssd_copy_out_cb, blk_req);
+ }
+
+out_sglist_destroy:
+ qemu_sglist_destroy(&qsg);
+
+out:
+ if (req->nlb > 1) {
+ g_free(dlbal);
+ }
+
+ if (status != NVME_SUCCESS) {
+ req->status = status;
+ nvme_enqueue_req_completion(cq, req);
+ }
+ }
+}
+
+static uint16_t ocssd_copy(OcssdCtrl *o, NvmeCmd *cmd, NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+ NvmeNamespace *ns = req->ns;
+ OcssdCopyCmd *cpy = (OcssdCopyCmd *) cmd;
+ NvmeBlockBackendRequest *blk_req;
+
+ hwaddr addr = 0x0;
+ uint64_t *lbal;
+ uint64_t dulbe = 0;
+ size_t unit_len = nvme_ns_lbads_bytes(ns);
+ size_t unit_len_meta = nvme_ns_ms(ns);
+ uint16_t status;
+
+ trace_ocssd_copy(req->cqe.cid, req->nlb);
+
+ if (req->nlb > OCSSD_CMD_MAX_LBAS) {
+ trace_ocssd_err(req->cqe.cid, "OCSSD_CMD_MAX_LBAS exceeded",
+ NVME_INVALID_FIELD | NVME_DNR);
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ _get_lba_list(o, le64_to_cpu(cpy->lbal), &lbal, req);
+ req->slba = (uint64_t) lbal;
+
+ status = ocssd_rw_check_vector_req(o, cmd, req, &dulbe);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "ocssd_rw_check_vector_req",
+ status);
+ goto out;
+ }
+
+ if (NVME_ERR_REC_DULBE(n->features.err_rec)) {
+ for (uint32_t i = 0; i < req->nlb; i++) {
+ if (dulbe & (1 << i)) {
+ status = NVME_DULB | NVME_DNR;
+ goto out;
+ }
+ }
+ }
+
+ /*
+ * For now, use bounce buffers to do the copy. Store the bounce buffer
+ * addresses in the unused cdw12/cdw13 and mptr fields so it can be
+ * referred to in the callback.
+ */
+ addr = (hwaddr) g_malloc_n(req->nlb, unit_len);
+ req->cmd.cdw12 = addr >> 32;
+ req->cmd.cdw13 = addr & 0xffffffff;
+
+ QEMUSGList qsg;
+ pci_dma_sglist_init(&qsg, &n->parent_obj, 1);
+ qemu_sglist_add(&qsg, addr, req->nlb * unit_len);
+
+ status = ocssd_blk_setup_vector(n, ns, &qsg, ns->blk_offset, unit_len,
+ req);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "ocssd_blk_setup_vector", status);
+ goto out_sglist_destroy;
+ }
+
+ if (n->params.ms) {
+ req->cmd.mptr = (hwaddr) g_malloc_n(req->nlb, unit_len_meta);
+
+ qsg.nsg = 0;
+ qsg.size = 0;
+
+ qemu_sglist_add(&qsg, req->cmd.mptr, req->nlb * unit_len_meta);
+
+ status = ocssd_blk_setup_vector(n, ns, &qsg, ns->blk_offset_md,
+ unit_len_meta, req);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "ocssd_blk_setup_vector", status);
+ goto out_sglist_destroy;
+ }
+ }
+
+ QTAILQ_FOREACH(blk_req, &req->blk_req_tailq, tailq_entry) {
+ qemu_iovec_init(&blk_req->iov, blk_req->qsg.nsg);
+ _sglist_to_iov(n, &blk_req->qsg, &blk_req->iov);
+
+ block_acct_start(blk_get_stats(n->conf.blk), &blk_req->acct,
+ blk_req->iov.size, BLOCK_ACCT_READ);
+
+ blk_req->aiocb = blk_aio_preadv(n->conf.blk, blk_req->blk_offset,
+ &blk_req->iov, 0, ocssd_copy_in_cb, blk_req);
+ }
+
+out_sglist_destroy:
+ qemu_sglist_destroy(&qsg);
+
+out:
+ if (req->nlb > 1) {
+ g_free(lbal);
+ }
+
+ if (status) {
+ g_free((void *) addr);
+ g_free((void *) req->cmd.mptr);
+ return status;
+ }
+
+ return NVME_NO_COMPLETE;
+}
+
+
+static void ocssd_rw_cb(void *opaque, int ret)
+{
+ NvmeBlockBackendRequest *blk_req = opaque;
+ NvmeRequest *req = blk_req->req;
+ NvmeSQueue *sq = req->sq;
+ NvmeCtrl *n = sq->ctrl;
+ NvmeCQueue *cq = n->cq[sq->cqid];
+
+ OcssdCtrl *o = OCSSD(n);
+
+ trace_ocssd_rw_cb(req->cqe.cid, req->ns->id);
+
+ QTAILQ_REMOVE(&req->blk_req_tailq, blk_req, tailq_entry);
+
+ ocssd_rwc_aio_complete(o, blk_req, ret);
+ nvme_blk_req_put(n, blk_req);
+
+ if (QTAILQ_EMPTY(&req->blk_req_tailq)) {
+ trace_nvme_enqueue_req_completion(req->cqe.cid, cq->cqid);
+ nvme_enqueue_req_completion(cq, req);
+ }
+}
+
+static uint16_t ocssd_rw(OcssdCtrl *o, NvmeCmd *cmd, NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+ OcssdRwCmd *orw = (OcssdRwCmd *) cmd;
+
+ uint64_t dulbe = 0;
+ uint64_t *lbal;
+ uint64_t lbal_addr = le64_to_cpu(orw->lbal);
+ uint16_t status = NVME_SUCCESS;
+
+ if (req->nlb > OCSSD_CMD_MAX_LBAS) {
+ trace_ocssd_err(req->cqe.cid, "OCSSD_CMD_MAX_LBAS exceeded",
+ NVME_INVALID_FIELD | NVME_DNR);
+ return NVME_INVALID_FIELD | NVME_DNR;
+ }
+
+ _get_lba_list(o, lbal_addr, &lbal, req);
+ req->slba = (uint64_t) lbal;
+
+ _dprint_vector_rw(o, req);
+
+ status = ocssd_rw_check_vector_req(o, cmd, req, &dulbe);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "ocssd_rw_check_vector_req", status);
+ goto out;
+ }
+
+ if (!req->is_write && NVME_ERR_REC_DULBE(n->features.err_rec)) {
+ for (uint32_t i = 0; i < req->nlb; i++) {
+ if (dulbe & (1 << i)) {
+ status = NVME_DULB | NVME_DNR;
+ goto out;
+ }
+ }
+ }
+
+ status = nvme_blk_map(n, cmd, req, ocssd_blk_setup_vector);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "nvme_blk_map", status);
+ goto out;
+ }
+
+out:
+ if (req->nlb > 1) {
+ g_free((uint64_t *) req->slba);
+ }
+
+ if (status) {
+ return status;
+ }
+
+ return nvme_blk_submit_io(n, req, ocssd_rw_cb);
+}
+
+static uint16_t ocssd_geometry(OcssdCtrl *o, NvmeCmd *cmd, NvmeRequest *req)
+{
+ OcssdNamespace *ons;
+
+ uint32_t nsid = le32_to_cpu(cmd->nsid);
+ if (unlikely(nsid == 0 || nsid > o->nvme.params.num_ns)) {
+ return NVME_INVALID_NSID | NVME_DNR;
+ }
+
+ ons = &o->namespaces[nsid - 1];
+
+ return nvme_dma_read(&o->nvme, (uint8_t *) &ons->id, sizeof(OcssdIdentity),
+ cmd, req);
+}
+
+static uint16_t ocssd_get_log(OcssdCtrl *o, NvmeCmd *cmd, NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+
+ uint32_t dw10 = le32_to_cpu(cmd->cdw10);
+ uint32_t dw11 = le32_to_cpu(cmd->cdw11);
+ uint32_t dw12 = le32_to_cpu(cmd->cdw12);
+ uint32_t dw13 = le32_to_cpu(cmd->cdw13);
+ uint16_t lid = dw10 & 0xff;
+ uint8_t rae = (dw10 >> 15) & 0x1;
+ uint32_t numdl, numdu, len;
+ uint64_t off, lpol, lpou;
+
+ numdl = (dw10 >> 16);
+ numdu = (dw11 & 0xffff);
+ lpol = dw12;
+ lpou = dw13;
+
+ len = (((numdu << 16) | numdl) + 1) << 2;
+ off = (lpou << 32ULL) | lpol;
+
+ switch (lid) {
+ case OCSSD_CHUNK_INFO:
+ return ocssd_do_get_chunk_info(o, cmd, len, off, req);
+ case OCSSD_CHUNK_NOTIFICATION:
+ return ocssd_do_get_chunk_notification(o, cmd, len, off, rae, req);
+ default:
+ return nvme_get_log(n, cmd, req);
+ }
+}
+
+static uint16_t ocssd_get_feature(OcssdCtrl *o, NvmeCmd *cmd, NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+
+ uint32_t dw10 = le32_to_cpu(cmd->cdw10);
+
+ trace_ocssd_getfeat(dw10);
+
+ switch (dw10) {
+ case OCSSD_MEDIA_FEEDBACK:
+ req->cqe.cdw0 = cpu_to_le32(o->features.media_feedback);
+ break;
+ default:
+ return nvme_get_feature(n, cmd, req);
+ }
+
+ return NVME_SUCCESS;
+}
+
+static uint16_t ocssd_set_feature(OcssdCtrl *o, NvmeCmd *cmd, NvmeRequest *req)
+{
+ NvmeCtrl *n = &o->nvme;
+
+ uint32_t dw10 = le32_to_cpu(cmd->cdw10);
+ uint32_t dw11 = le32_to_cpu(cmd->cdw11);
+
+ trace_ocssd_setfeat(dw10, dw11);
+
+ switch (dw10) {
+ case NVME_ERROR_RECOVERY:
+ n->features.err_rec = dw11;
+ break;
+ case OCSSD_MEDIA_FEEDBACK:
+ o->features.media_feedback = dw11;
+ break;
+ default:
+ return nvme_set_feature(n, cmd, req);
+ }
+
+ return NVME_SUCCESS;
+}
+
+static uint16_t ocssd_admin_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+{
+ OcssdCtrl *o = OCSSD(n);
+
+ switch (cmd->opcode) {
+ case NVME_ADM_CMD_SET_FEATURES:
+ return ocssd_set_feature(o, cmd, req);
+ case NVME_ADM_CMD_GET_FEATURES:
+ return ocssd_get_feature(o, cmd, req);
+ case OCSSD_ADM_CMD_GEOMETRY:
+ return ocssd_geometry(o, cmd, req);
+ case NVME_ADM_CMD_GET_LOG_PAGE:
+ return ocssd_get_log(o, cmd, req);
+ default:
+ return nvme_admin_cmd(n, cmd, req);
+ }
+}
+
+static uint16_t ocssd_io_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
+{
+ OcssdCtrl *o = OCSSD(n);
+ NvmeRwCmd *rw;
+ uint16_t status;
+
+ uint32_t nsid = le32_to_cpu(cmd->nsid);
+
+ if (unlikely(nsid == 0 || nsid > n->params.num_ns)) {
+ trace_nvme_err_invalid_ns(nsid, n->params.num_ns);
+ return NVME_INVALID_NSID | NVME_DNR;
+ }
+
+ trace_ocssd_io_cmd(req->cqe.cid, nsid, cmd->opcode);
+
+ req->ns = &n->namespaces[nsid - 1];
+
+ switch (cmd->opcode) {
+ case NVME_CMD_READ:
+ case NVME_CMD_WRITE:
+ rw = (NvmeRwCmd *) cmd;
+
+ req->nlb = le16_to_cpu(rw->nlb) + 1;
+ req->is_write = nvme_rw_is_write(req);
+ req->slba = le64_to_cpu(rw->slba);
+
+ trace_nvme_rw(req->is_write ? "write" : "read", req->nlb,
+ req->nlb << nvme_ns_lbads(req->ns), req->slba);
+
+ status = ocssd_rw_check_scalar_req(o, cmd, req);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "ocssd_rw_check_scalar_req", status);
+ return status;
+ }
+
+ status = nvme_blk_map(n, cmd, req, ocssd_blk_setup_scalar);
+ if (status) {
+ trace_ocssd_err(req->cqe.cid, "nvme_blk_map", status);
+ return status;
+ }
+
+ return nvme_blk_submit_io(n, req, ocssd_rw_cb);
+
+ case NVME_CMD_DSM:
+ return ocssd_dsm(o, cmd, req);
+
+ case OCSSD_CMD_VECT_READ:
+ case OCSSD_CMD_VECT_WRITE:
+ rw = (NvmeRwCmd *) cmd;
+
+ req->nlb = le16_to_cpu(rw->nlb) + 1;
+ req->is_write = _is_write(req);
+
+ trace_ocssd_rw(req->cqe.cid, nsid, req->cmd.opcode, req->nlb);
+
+ return ocssd_rw(o, cmd, req);
+
+ case OCSSD_CMD_VECT_COPY:
+ rw = (NvmeRwCmd *) cmd;
+ req->nlb = le16_to_cpu(rw->nlb) + 1;
+
+ /* first phase of copy is a read */
+ req->is_write = false;
+
+ return ocssd_copy(o, cmd, req);
+
+ case OCSSD_CMD_VECT_RESET:
+ return ocssd_reset(o, cmd, req);
+
+ default:
+ return nvme_io_cmd(n, cmd, req);
+ }
+}
+
+static uint64_t ocssd_ns_calc_blks(OcssdCtrl *o, OcssdNamespace *ons)
+{
+ NvmeNamespace *ns = ons->ns;
+ return o->hdr.ns_size / (nvme_ns_lbads_bytes(ns) + nvme_ns_ms(ns));
+}
+
+static uint64_t ocssd_ns_calc_info_size(OcssdCtrl *o,
+ OcssdNamespace *ons)
+{
+ OcssdIdGeo *geo = &ons->id.geo;
+ uint64_t chks_total = geo->num_grp * geo->num_pu * geo->num_chk;
+
+ return QEMU_ALIGN_UP(chks_total * sizeof(OcssdChunkDescriptor),
+ o->hdr.sector_size);
+}
+
+static uint64_t ocssd_ns_calc_acct_size(OcssdCtrl *o, OcssdNamespace *ons)
+{
+ OcssdIdGeo *geo = &ons->id.geo;
+ uint64_t chks_total = geo->num_grp * geo->num_pu * geo->num_chk;
+
+ return QEMU_ALIGN_UP(chks_total * sizeof(OcssdChunkAcctDescriptor),
+ o->hdr.sector_size);
+}
+
+static void ocssd_free_namespace(OcssdCtrl *o, OcssdNamespace *ons)
+{
+ g_free(ons->info.descr);
+ g_free(ons->acct.descr);
+ g_free(ons->resetfail);
+ g_free(ons->writefail);
+}
+
+static void ocssd_free_namespaces(OcssdCtrl *o)
+{
+ for (int i = 0; i < o->hdr.num_ns; i++) {
+ ocssd_free_namespace(o, &o->namespaces[i]);
+ }
+}
+
+static int ocssd_init_namespace(OcssdCtrl *o, OcssdNamespace *ons,
+ Error **errp)
+{
+ NvmeCtrl *n = &o->nvme;
+ NvmeNamespace *ns = ons->ns;
+ NvmeIdNs *id_ns = &ons->ns->id_ns;
+ OcssdParams *params = &o->params;
+ BlockBackend *blk = n->conf.blk;
+ OcssdIdentity *id = &ons->id;
+ OcssdIdGeo *geo = &id->geo;
+ OcssdAddrF *addrf = &ons->addrf;
+ Error *local_err = NULL;
+
+ int ret;
+
+ nvme_ns_init_identify(n, id_ns);
+
+ /*
+ * In addition to checking if the device has the NVME_QUIRK_LIGHTNVM quirk,
+ * the Linux NVMe driver also checks if the first byte of the
+ * vendor specific area in the identify namespace structure is set to 0x1.
+ *
+ * This is non-standard and Linux specific.
+ */
+ id_ns->vs[0] = 0x1;
+
+ ret = blk_pread(blk, ns->blk_offset, id, sizeof(OcssdIdentity));
+ if (ret < 0) {
+ error_setg_errno(errp, -ret,
+ "could not read namespace identity structure: ");
+ return 1;
+ }
+ ns->blk_offset += sizeof(OcssdIdentity);
+
+ if (params->ws_min != UINT32_MAX) {
+ id->wrt.ws_min = cpu_to_le32(params->ws_min);
+ }
+
+ if (params->ws_opt != UINT32_MAX) {
+ id->wrt.ws_opt = cpu_to_le32(params->ws_opt);
+ }
+
+ if (params->mw_cunits != UINT32_MAX) {
+ id->wrt.mw_cunits = cpu_to_le32(params->mw_cunits);
+ }
+
+ if (params->mccap != UINT32_MAX) {
+ id->mccap = params->mccap;
+ }
+
+ if (params->early_reset) {
+ id->mccap |= OCSSD_IDENTITY_MCCAP_EARLY_RESET;
+ }
+
+ if (params->wit != UINT8_MAX) {
+ id->wit = params->wit;
+ }
+
+ id_ns->lbaf[0].lbads = 63 - clz64(o->hdr.sector_size);
+ id_ns->lbaf[0].ms = o->hdr.md_size;
+ id_ns->nlbaf = 0;
+ id_ns->flbas = 0;
+ id_ns->mc = o->hdr.md_size ? 0x2 : 0;
+
+ ons->acct.size = ocssd_ns_calc_acct_size(o, ons);
+ ons->acct.descr = g_malloc0(ons->acct.size);
+ ons->acct.blk_offset = ns->blk_offset;
+ ns->blk_offset += ons->acct.size;
+
+ ons->info.size = ocssd_ns_calc_info_size(o, ons);
+ ons->info.descr = g_malloc0(ons->info.size);
+ ons->info.blk_offset = ns->blk_offset;
+ ns->blk_offset += ons->info.size;
+
+ ns->ns_blks = ocssd_ns_calc_blks(o, ons);
+ ns->ns_blks -= (sizeof(OcssdIdentity) + ons->info.size) /
+ nvme_ns_lbads_bytes(ns);
+
+ ns->blk_offset_md = ns->blk_offset + nvme_ns_lbads_bytes(ns) * ns->ns_blks;
+
+ ons->chks_per_grp = geo->num_chk * geo->num_pu;
+ ons->chks_total = ons->chks_per_grp * geo->num_grp;
+ ons->secs_per_chk = geo->clba;
+ ons->secs_per_pu = ons->secs_per_chk * geo->num_chk;
+ ons->secs_per_grp = ons->secs_per_pu * geo->num_pu;
+ ons->secs_total = ons->secs_per_grp * geo->clba;
+
+ ocssd_ns_optimal_addrf(addrf, &id->lbaf);
+
+ /*
+ * Size of device (NSZE) is the entire address space (though some space is
+ * not usable).
+ */
+ id_ns->nuse = id_ns->nsze =
+ 1ULL << (id->lbaf.sec_len + id->lbaf.chk_len +
+ id->lbaf.pu_len + id->lbaf.grp_len);
+
+ /*
+ * Namespace capacity (NCAP) is set to the actual usable size in logical
+ * blocks.
+ */
+ id_ns->ncap = ns->ns_blks;
+
+ ret = ocssd_ns_load_chunk_info(o, ons);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "could not load chunk info");
+ return 1;
+ }
+
+ ret = ocssd_ns_load_chunk_acct(o, ons);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "could not load chunk acct");
+ return 1;
+ }
+
+ if (params->chunkinfo_fname) {
+ if (ocssd_load_chunk_info_from_file(o, params->chunkinfo_fname,
+ &local_err)) {
+ error_propagate_prepend(errp, local_err,
+ "could not load chunk info from file");
+ return 1;
+ }
+
+ for (int i = 0; i < o->hdr.num_ns; i++) {
+ ret = blk_pwrite(o->nvme.conf.blk, ons->info.blk_offset,
+ ons->info.descr, ons->info.size, 0);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "could not commit chunk info");
+ return 1;
+ }
+
+ ret = blk_pwrite(o->nvme.conf.blk, ons->acct.blk_offset,
+ ons->acct.descr, ons->acct.size, 0);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "could not commit chunk acct");
+ return 1;
+ }
+ }
+ }
+
+ for (int i = 0; i < ons->chks_total; i++) {
+ OcssdChunkDescriptor *cnk = &ons->info.descr[i];
+ ons->wear_index_total += cnk->wear_index;
+ }
+
+ ons->wear_index_avg = ons->wear_index_total / ons->chks_total;
+
+ ons->resetfail = NULL;
+ if (params->resetfail_fname) {
+ ons->resetfail = g_malloc0_n(ons->chks_total, sizeof(*ons->resetfail));
+ if (!ons->resetfail) {
+ error_setg_errno(errp, ENOMEM, "could not allocate memory");
+ return 1;
+ }
+
+ if (ocssd_load_reset_error_injection_from_file(o,
+ params->resetfail_fname, &local_err)) {
+ error_propagate_prepend(errp, local_err,
+ "could not load reset error injection from file");
+ return 1;
+ }
+ }
+
+ ons->writefail = NULL;
+ if (params->writefail_fname) {
+ ons->writefail = g_malloc0_n(ons->secs_total, sizeof(*ons->writefail));
+ if (!ons->writefail) {
+ error_setg_errno(errp, ENOMEM, "could not allocate memory");
+ return 1;
+ }
+
+ if (ocssd_load_write_error_injection_from_file(o,
+ params->writefail_fname, &local_err)) {
+ error_propagate_prepend(errp, local_err,
+ "could not load write error injection from file");
+ return 1;
+ }
+
+ /*
+ * We fail resets for a chunk after a write failure to it, so make sure
+ * to allocate the resetfailure buffer if it has not been already.
+ */
+ if (!ons->resetfail) {
+ ons->resetfail = g_malloc0_n(ons->chks_total,
+ sizeof(*ons->resetfail));
+ }
+ }
+
+ return 0;
+}
+
+static int ocssd_init_namespaces(OcssdCtrl *o, Error **errp)
+{
+ NvmeCtrl *n = &o->nvme;
+ Error *local_err = NULL;
+
+ n->namespaces = g_new0(NvmeNamespace, o->hdr.num_ns);
+ o->namespaces = g_new0(OcssdNamespace, o->hdr.num_ns);
+ for (int i = 0; i < o->hdr.num_ns; i++) {
+ OcssdNamespace *ons = &o->namespaces[i];
+ NvmeNamespace *ns = ons->ns = &n->namespaces[i];
+
+ ns->id = i + 1;
+ ns->blk_offset = o->hdr.sector_size + i * o->hdr.ns_size;
+
+ if (ocssd_init_namespace(o, ons, &local_err)) {
+ error_propagate_prepend(errp, local_err,
+ "init namespaces failed: ");
+ return 1;
+ }
+ }
+
+ return 0;
+}
+
+static void ocssd_realize(PCIDevice *pci_dev, Error **errp)
+{
+ int ret;
+
+ OcssdCtrl *o = OCSSD(pci_dev);
+ NvmeCtrl *n = &o->nvme;
+ NvmeIdCtrl *id_ctrl = &n->id_ctrl;
+ Error *local_err = NULL;
+
+ n->namespaces = NULL;
+ n->admin_cmd = ocssd_admin_cmd;
+ n->io_cmd = ocssd_io_cmd;
+
+ if (nvme_init_blk(n, &local_err)) {
+ error_propagate_prepend(errp, local_err, "nvme_init_blk failed: ");
+ return;
+ }
+
+ ret = blk_pread(n->conf.blk, 0, &o->hdr, sizeof(OcssdFormatHeader));
+ if (ret < 0) {
+ error_setg(errp, "could not read block format header");
+ return;
+ }
+
+ n->params.num_ns = o->hdr.num_ns;
+ n->params.ms = o->hdr.md_size;
+
+ if (nvme_init_state(n, &local_err)) {
+ error_propagate_prepend(errp, local_err, "nvme_init_state failed: ");
+ return;
+ }
+
+ nvme_init_pci(n, pci_dev);
+
+ pci_config_set_vendor_id(pci_dev->config, PCI_VENDOR_ID_CNEX);
+ pci_config_set_device_id(pci_dev->config, 0x1f1f);
+
+ ocssd_init_namespaces(o, errp);
+
+ nvme_init_ctrl(n);
+
+ n->id_ctrl.oncs |= cpu_to_le16(NVME_ONCS_DSM);
+
+ strpadcpy((char *)id_ctrl->mn, sizeof(id_ctrl->mn),
+ "QEMU NVM Express LightNVM Controller", ' ');
+}
+
+static void ocssd_exit(PCIDevice *pci_dev)
+{
+ OcssdCtrl *o = OCSSD(pci_dev);
+
+ ocssd_free_namespaces(o);
+ nvme_free_ctrl(&o->nvme, pci_dev);
+}
+
+static Property ocssd_props[] = {
+ DEFINE_BLOCK_PROPERTIES(OcssdCtrl, nvme.conf),
+ DEFINE_NVME_PROPERTIES(OcssdCtrl, nvme.params),
+ DEFINE_OCSSD_PROPERTIES(OcssdCtrl, params),
+ DEFINE_PROP_END_OF_LIST(),
+};
+
+static const VMStateDescription ocssd_vmstate = {
+ .name = "ocssd",
+ .unmigratable = 1,
+};
+
+static void ocssd_class_init(ObjectClass *oc, void *data)
+{
+ DeviceClass *dc = DEVICE_CLASS(oc);
+ PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+
+ pc->realize = ocssd_realize;
+ pc->exit = ocssd_exit;
+ pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
+ pc->vendor_id = PCI_VENDOR_ID_CNEX;
+ pc->device_id = 0x1f1f;
+ pc->revision = 2;
+
+ set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+ dc->desc = "OpenChannel 2.0 NVMe";
+ dc->props = ocssd_props;
+ dc->vmsd = &ocssd_vmstate;
+}
+
+static void ocssd_instance_init(Object *obj)
+{
+ OcssdCtrl *s = OCSSD(obj);
+
+ device_add_bootindex_property(obj, &s->nvme.conf.bootindex,
+ "bootindex", "/namespace@1,0",
+ DEVICE(obj), &error_abort);
+}
+
+static const TypeInfo ocssd_info = {
+ .name = TYPE_OCSSD,
+ .parent = TYPE_PCI_DEVICE,
+ .instance_size = sizeof(OcssdCtrl),
+ .class_init = ocssd_class_init,
+ .instance_init = ocssd_instance_init,
+ .interfaces = (InterfaceInfo[]) {
+ { INTERFACE_PCIE_DEVICE },
+ { }
+ },
+};
+
+static void ocssd_register_types(void)
+{
+ type_register_static(&ocssd_info);
+}
+
+type_init(ocssd_register_types)
diff --git a/hw/block/nvme/ocssd.h b/hw/block/nvme/ocssd.h
new file mode 100644
index 000000000000..31dadc0a1034
--- /dev/null
+++ b/hw/block/nvme/ocssd.h
@@ -0,0 +1,140 @@
+/*
+ * QEMU OpenChannel 2.0 Controller
+ *
+ * Copyright (c) 2019 CNEX Labs, Inc.
+ *
+ * Thank you to the following people for their contributions to the original
+ * qemu-nvme (github.com/OpenChannelSSD/qemu-nvme) implementation.
+ *
+ * Matias Bjørling <mb@lightnvm.io>
+ * Javier González <javier@javigon.com>
+ * Simon Andreas Frimann Lund <ocssd@safl.dk>
+ * Hans Holmberg <hans@owltronix.com>
+ * Jesper Devantier <contact@pseudonymous.me>
+ * Young Tack Jin <youngtack.jin@circuitblvd.com>
+ *
+ * This code is licensed under the GNU GPL v2 or later.
+ */
+
+#ifndef HW_NVME_OCSSD_H
+#define HW_NVME_OCSSD_H
+
+#include "block/ocssd.h"
+#include "hw/block/nvme.h"
+
+#define TYPE_OCSSD "ocssd"
+#define OCSSD(obj) \
+ OBJECT_CHECK(OcssdCtrl, (obj), TYPE_OCSSD)
+
+#define OCSSD_MAX_CHUNK_NOTIFICATIONS 64
+
+#define DEFINE_OCSSD_PROPERTIES(_state, _props) \
+ DEFINE_PROP_UINT32("mccap", _state, _props.mccap, UINT32_MAX), \
+ DEFINE_PROP_UINT32("ws_min", _state, _props.ws_min, UINT32_MAX), \
+ DEFINE_PROP_UINT32("ws_opt", _state, _props.ws_opt, UINT32_MAX), \
+ DEFINE_PROP_UINT32("mw_cunits", _state, _props.mw_cunits, UINT32_MAX), \
+ DEFINE_PROP_UINT8("wit", _state, _props.wit, UINT8_MAX), \
+ DEFINE_PROP_BOOL("early_reset", _state, _props.early_reset, true), \
+ DEFINE_PROP_STRING("resetfail", _state, _props.resetfail_fname), \
+ DEFINE_PROP_STRING("writefail", _state, _props.writefail_fname), \
+ DEFINE_PROP_STRING("chunkinfo", _state, _props.chunkinfo_fname)
+
+typedef struct OcssdParams {
+ /* qemu configurable device characteristics */
+ uint32_t mccap;
+ uint32_t ws_min;
+ uint32_t ws_opt;
+ uint32_t mw_cunits;
+ uint8_t wit;
+ bool early_reset;
+
+ char *chunkinfo_fname;
+ char *resetfail_fname;
+ char *writefail_fname;
+} OcssdParams;
+
+#define OCSSD_CMD_MAX_LBAS 64
+
+typedef struct OcssdAddrF {
+ uint64_t grp_mask;
+ uint64_t pu_mask;
+ uint64_t chk_mask;
+ uint64_t sec_mask;
+ uint8_t grp_offset;
+ uint8_t pu_offset;
+ uint8_t chk_offset;
+ uint8_t sec_offset;
+} OcssdAddrF;
+
+typedef struct OcssdChunkAcctDescriptor {
+ uint32_t pe_cycles;
+} OcssdChunkAcctDescriptor;
+
+typedef struct OcssdChunkAcct {
+ uint64_t blk_offset;
+ uint64_t size;
+
+ OcssdChunkAcctDescriptor *descr;
+} OcssdChunkAcct;
+
+typedef struct OcssdChunkInfo {
+ uint64_t blk_offset;
+ uint64_t size;
+
+ OcssdChunkDescriptor *descr;
+} OcssdChunkInfo;
+
+typedef struct OcssdNamespace {
+ NvmeNamespace *ns;
+
+ OcssdIdentity id;
+ OcssdAddrF addrf;
+
+ /* reset and write fail error probabilities indexed by namespace */
+ uint8_t *resetfail;
+ uint8_t *writefail;
+
+ /* derived values (convenience) */
+ uint32_t chks_per_grp;
+ uint32_t chks_total;
+ uint32_t secs_per_chk;
+ uint32_t secs_per_pu;
+ uint32_t secs_per_grp;
+ uint32_t secs_total;
+
+ /* wear index tracking */
+ uint8_t wear_index_avg;
+ uint64_t wear_index_total;
+
+ OcssdChunkInfo info;
+ OcssdChunkAcct acct;
+} OcssdNamespace;
+
+typedef struct OcssdCtrl {
+ NvmeCtrl nvme;
+
+ OcssdFormatHeader hdr;
+ OcssdParams params;
+ OcssdNamespace *namespaces;
+ OcssdFeatureVal features;
+
+ uint64_t notifications_count;
+ uint16_t notifications_index;
+ uint16_t notifications_max;
+ OcssdChunkNotification notifications[OCSSD_MAX_CHUNK_NOTIFICATIONS];
+} OcssdCtrl;
+
+static inline void ocssd_ns_optimal_addrf(OcssdAddrF *addrf, OcssdIdLBAF *lbaf)
+{
+ addrf->sec_offset = 0;
+ addrf->chk_offset = lbaf->sec_len;
+ addrf->pu_offset = lbaf->sec_len + lbaf->chk_len;
+ addrf->grp_offset = lbaf->sec_len + lbaf->chk_len + lbaf->pu_len;
+
+ addrf->grp_mask = ((1 << lbaf->grp_len) - 1) << addrf->grp_offset;
+ addrf->pu_mask = ((1 << lbaf->pu_len) - 1) << addrf->pu_offset;
+ addrf->chk_mask = ((1 << lbaf->chk_len) - 1) << addrf->chk_offset;
+ addrf->sec_mask = ((1 << lbaf->sec_len) - 1) << addrf->sec_offset;
+}
+
+#endif /* HW_NVME_OCSSD_H */
diff --git a/hw/block/nvme/trace-events b/hw/block/nvme/trace-events
new file mode 100644
index 000000000000..444c75e34c59
--- /dev/null
+++ b/hw/block/nvme/trace-events
@@ -0,0 +1,136 @@
+# nvme.c
+# nvme traces for successful events
+nvme_irq_msix(uint32_t vector) "raising MSI-X IRQ vector %u"
+nvme_irq_pin(void) "pulsing IRQ pin"
+nvme_irq_masked(void) "IRQ is masked"
+nvme_dma_read(uint64_t prp1, uint64_t prp2) "DMA read, prp1=0x%"PRIx64" prp2=0x%"PRIx64""
+nvme_map_prp(uint8_t cmd_opcode, uint64_t trans_len, uint32_t len, uint64_t prp1, uint64_t prp2, int num_prps) "cmd_opcode=0x%"PRIx8", trans_len=%"PRIu64", len=%"PRIu32", prp1=0x%"PRIx64", prp2=0x%"PRIx64", num_prps=%d"
+nvme_map_sgl(uint16_t cid, uint64_t typ, uint16_t nlb, uint64_t len) "cid %"PRIu16" type %"PRIu64" nlb %"PRIu16" len %"PRIu64""
+nvme_rw(const char *verb, uint32_t blk_count, uint64_t byte_count, uint64_t lba) "%s %"PRIu32" blocks (%"PRIu64" bytes) from LBA %"PRIu64""
+nvme_rw_cb(uint16_t cid, uint32_t nsid) "cid %"PRIu16" nsid %"PRIu32""
+nvme_create_sq(uint64_t addr, uint16_t sqid, uint16_t cqid, uint16_t qsize, uint16_t qflags) "create submission queue, addr=0x%"PRIx64", sqid=%"PRIu16", cqid=%"PRIu16", qsize=%"PRIu16", qflags=%"PRIu16""
+nvme_create_cq(uint64_t addr, uint16_t cqid, uint16_t vector, uint16_t size, uint16_t qflags, int ien) "create completion queue, addr=0x%"PRIx64", cqid=%"PRIu16", vector=%"PRIu16", qsize=%"PRIu16", qflags=%"PRIu16", ien=%d"
+nvme_del_sq(uint16_t qid) "deleting submission queue sqid=%"PRIu16""
+nvme_del_cq(uint16_t cqid) "deleted completion queue, sqid=%"PRIu16""
+nvme_identify_ctrl(void) "identify controller"
+nvme_identify_ns(uint16_t ns) "identify namespace, nsid=%"PRIu16""
+nvme_identify_ns_list(uint16_t ns) "identify namespace list, nsid=%"PRIu16""
+nvme_identify_ns_descriptor_list(uint16_t ns) "identify namespace descriptor list, nsid=%"PRIu16""
+nvme_getfeat(uint32_t fid) "fid %"PRIu32""
+nvme_setfeat(uint32_t fid, uint32_t val) "fid %"PRIu32" val %"PRIu32""
+nvme_getfeat_vwcache(const char* result) "get feature volatile write cache, result=%s"
+nvme_getfeat_numq(int result) "get feature number of queues, result=%d"
+nvme_setfeat_numq(int reqcq, int reqsq, int gotcq, int gotsq) "requested cq_count=%d sq_count=%d, responding with cq_count=%d sq_count=%d"
+nvme_get_log(uint16_t cid, uint16_t lid) "cid %"PRIu16" lid 0x%"PRIx16""
+nvme_process_aers(void) "processing aers"
+nvme_aer(uint16_t cid) "cid %"PRIu16""
+nvme_aer_aerl_exceeded(void) "aerl exceeded"
+nvme_aer_masked(uint8_t type, uint8_t mask) "type 0x%"PRIx8" mask 0x%"PRIx8""
+nvme_aer_post_cqe(uint8_t typ, uint8_t info, uint8_t log_page) "type 0x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8""
+nvme_enqueue_req_completion(uint16_t cid, uint16_t cqid) "cid %"PRIu16" cqid %"PRIu16""
+nvme_enqueue_event(uint8_t typ, uint8_t info, uint8_t log_page) "type 0x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8""
+nvme_no_outstanding_aers(void) "ignoring event; no outstanding AERs"
+nvme_mmio_intm_set(uint64_t data, uint64_t new_mask) "wrote MMIO, interrupt mask set, data=0x%"PRIx64", new_mask=0x%"PRIx64""
+nvme_mmio_intm_clr(uint64_t data, uint64_t new_mask) "wrote MMIO, interrupt mask clr, data=0x%"PRIx64", new_mask=0x%"PRIx64""
+nvme_mmio_cfg(uint64_t data) "wrote MMIO, config controller config=0x%"PRIx64""
+nvme_mmio_aqattr(uint64_t data) "wrote MMIO, admin queue attributes=0x%"PRIx64""
+nvme_mmio_asqaddr(uint64_t data) "wrote MMIO, admin submission queue address=0x%"PRIx64""
+nvme_mmio_acqaddr(uint64_t data) "wrote MMIO, admin completion queue address=0x%"PRIx64""
+nvme_mmio_asqaddr_hi(uint64_t data, uint64_t new_addr) "wrote MMIO, admin submission queue high half=0x%"PRIx64", new_address=0x%"PRIx64""
+nvme_mmio_acqaddr_hi(uint64_t data, uint64_t new_addr) "wrote MMIO, admin completion queue high half=0x%"PRIx64", new_address=0x%"PRIx64""
+nvme_mmio_start_success(void) "setting controller enable bit succeeded"
+nvme_mmio_stopped(void) "cleared controller enable bit"
+nvme_mmio_shutdown_set(void) "shutdown bit set"
+nvme_mmio_shutdown_cleared(void) "shutdown bit cleared"
+
+# nvme traces for error conditions
+nvme_err(uint16_t cid, const char *s, uint16_t status) "cid %"PRIu16" \"%s\" status 0x%"PRIx16""
+nvme_err_invalid_dma(void) "PRP/SGL is too small for transfer size"
+nvme_err_invalid_prplist_ent(uint64_t prplist) "PRP list entry is null or not page aligned: 0x%"PRIx64""
+nvme_err_invalid_prp2_align(uint64_t prp2) "PRP2 is not page aligned: 0x%"PRIx64""
+nvme_err_invalid_prp2_missing(void) "PRP2 is null and more data to be transferred"
+nvme_err_invalid_prp(void) "invalid PRP"
+nvme_err_invalid_ns(uint32_t ns, uint32_t limit) "invalid namespace %u not within 1-%u"
+nvme_err_invalid_opc(uint8_t opc) "invalid opcode 0x%"PRIx8""
+nvme_err_invalid_admin_opc(uint8_t opc) "invalid admin opcode 0x%"PRIx8""
+nvme_err_invalid_lba_range(uint64_t start, uint64_t len, uint64_t limit) "Invalid LBA start=%"PRIu64" len=%"PRIu64" limit=%"PRIu64""
+nvme_err_invalid_del_sq(uint16_t qid) "invalid submission queue deletion, sid=%"PRIu16""
+nvme_err_invalid_create_sq_cqid(uint16_t cqid) "failed creating submission queue, invalid cqid=%"PRIu16""
+nvme_err_invalid_create_sq_sqid(uint16_t sqid) "failed creating submission queue, invalid sqid=%"PRIu16""
+nvme_err_invalid_create_sq_size(uint16_t qsize) "failed creating submission queue, invalid qsize=%"PRIu16""
+nvme_err_invalid_create_sq_addr(uint64_t addr) "failed creating submission queue, addr=0x%"PRIx64""
+nvme_err_invalid_create_sq_qflags(uint16_t qflags) "failed creating submission queue, qflags=%"PRIu16""
+nvme_err_invalid_del_cq_cqid(uint16_t cqid) "failed deleting completion queue, cqid=%"PRIu16""
+nvme_err_invalid_del_cq_notempty(uint16_t cqid) "failed deleting completion queue, it is not empty, cqid=%"PRIu16""
+nvme_err_invalid_create_cq_cqid(uint16_t cqid) "failed creating completion queue, cqid=%"PRIu16""
+nvme_err_invalid_create_cq_size(uint16_t size) "failed creating completion queue, size=%"PRIu16""
+nvme_err_invalid_create_cq_addr(uint64_t addr) "failed creating completion queue, addr=0x%"PRIx64""
+nvme_err_invalid_create_cq_vector(uint16_t vector) "failed creating completion queue, vector=%"PRIu16""
+nvme_err_invalid_create_cq_qflags(uint16_t qflags) "failed creating completion queue, qflags=%"PRIu16""
+nvme_err_invalid_identify_cns(uint16_t cns) "identify, invalid cns=0x%"PRIx16""
+nvme_err_invalid_getfeat(int dw10) "invalid get features, dw10=0x%"PRIx32""
+nvme_err_invalid_setfeat(uint32_t dw10) "invalid set features, dw10=0x%"PRIx32""
+nvme_err_invalid_log_page(uint16_t cid, uint16_t lid) "cid %"PRIu16" lid 0x%"PRIx16""
+nvme_err_startfail_cq(void) "nvme_start_ctrl failed because there are non-admin completion queues"
+nvme_err_startfail_sq(void) "nvme_start_ctrl failed because there are non-admin submission queues"
+nvme_err_startfail_nbarasq(void) "nvme_start_ctrl failed because the admin submission queue address is null"
+nvme_err_startfail_nbaracq(void) "nvme_start_ctrl failed because the admin completion queue address is null"
+nvme_err_startfail_asq_misaligned(uint64_t addr) "nvme_start_ctrl failed because the admin submission queue address is misaligned: 0x%"PRIx64""
+nvme_err_startfail_acq_misaligned(uint64_t addr) "nvme_start_ctrl failed because the admin completion queue address is misaligned: 0x%"PRIx64""
+nvme_err_startfail_page_too_small(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the page size is too small: log2size=%u, min=%u"
+nvme_err_startfail_page_too_large(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the page size is too large: log2size=%u, max=%u"
+nvme_err_startfail_cqent_too_small(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the completion queue entry size is too small: log2size=%u, min=%u"
+nvme_err_startfail_cqent_too_large(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the completion queue entry size is too large: log2size=%u, max=%u"
+nvme_err_startfail_sqent_too_small(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the submission queue entry size is too small: log2size=%u, min=%u"
+nvme_err_startfail_sqent_too_large(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the submission queue entry size is too large: log2size=%u, max=%u"
+nvme_err_startfail_asqent_sz_zero(void) "nvme_start_ctrl failed because the admin submission queue size is zero"
+nvme_err_startfail_acqent_sz_zero(void) "nvme_start_ctrl failed because the admin completion queue size is zero"
+nvme_err_startfail(void) "setting controller enable bit failed"
+nvme_err_internal_dev_error(const char *reason) "%s"
+
+# Traces for undefined behavior
+nvme_ub_mmiowr_misaligned32(uint64_t offset) "MMIO write not 32-bit aligned, offset=0x%"PRIx64""
+nvme_ub_mmiowr_toosmall(uint64_t offset, unsigned size) "MMIO write smaller than 32 bits, offset=0x%"PRIx64", size=%u"
+nvme_ub_mmiowr_intmask_with_msix(void) "undefined access to interrupt mask set when MSI-X is enabled"
+nvme_ub_mmiowr_ro_csts(void) "attempted to set a read only bit of controller status"
+nvme_ub_mmiowr_ssreset_w1c_unsupported(void) "attempted to W1C CSTS.NSSRO but CAP.NSSRS is zero (not supported)"
+nvme_ub_mmiowr_ssreset_unsupported(void) "attempted NVM subsystem reset but CAP.NSSRS is zero (not supported)"
+nvme_ub_mmiowr_cmbloc_reserved(void) "invalid write to reserved CMBLOC when CMBSZ is zero, ignored"
+nvme_ub_mmiowr_cmbsz_readonly(void) "invalid write to read only CMBSZ, ignored"
+nvme_ub_mmiowr_invalid(uint64_t offset, uint64_t data) "invalid MMIO write, offset=0x%"PRIx64", data=0x%"PRIx64""
+nvme_ub_mmiord_misaligned32(uint64_t offset) "MMIO read not 32-bit aligned, offset=0x%"PRIx64""
+nvme_ub_mmiord_toosmall(uint64_t offset) "MMIO read smaller than 32-bits, offset=0x%"PRIx64""
+nvme_ub_mmiord_invalid_ofs(uint64_t offset) "MMIO read beyond last register, offset=0x%"PRIx64", returning 0"
+nvme_ub_db_wr_misaligned(uint64_t offset) "doorbell write not 32-bit aligned, offset=0x%"PRIx64", ignoring"
+nvme_ub_db_wr_invalid_cq(uint32_t qid) "completion queue doorbell write for nonexistent queue, cqid=%"PRIu32", ignoring"
+nvme_ub_db_wr_invalid_cqhead(uint32_t qid, uint16_t new_head) "completion queue doorbell write value beyond queue size, cqid=%"PRIu32", new_head=%"PRIu16", ignoring"
+nvme_ub_db_wr_invalid_sq(uint32_t qid) "submission queue doorbell write for nonexistent queue, sqid=%"PRIu32", ignoring"
+nvme_ub_db_wr_invalid_sqtail(uint32_t qid, uint16_t new_tail) "submission queue doorbell write value beyond queue size, sqid=%"PRIu32", new_head=%"PRIu16", ignoring"
+
+# ocssd traces
+ocssd_rw(uint16_t cid, uint32_t nsid, uint16_t opcode, uint16_t nlb) "cid %"PRIu16" nsid %"PRIu32" opcode 0x%"PRIx16" nlb %"PRIu16""
+ocssd_rw_cb(uint16_t cid, uint32_t nsid) "cid %"PRIu16" nsid %"PRIu32""
+ocssd_copy(uint16_t cid, uint16_t nlb) "cid %"PRIu16" nlb %"PRIu16""
+ocssd_copy_in_cb(uint16_t cid, uint32_t nsid) "cid %"PRIu16" nsid %"PRIu32""
+ocssd_copy_out_cb(uint16_t cid, uint32_t nsid) "cid %"PRIu16" nsid %"PRIu32""
+ocssd_getfeat(uint32_t fid) "fid %"PRIu32""
+ocssd_setfeat(uint32_t fid, uint32_t val) "fid %"PRIu32" val %"PRIu32""
+ocssd_addr(uint16_t cid, char *lba) "cid %"PRIu16" %s"
+ocssd_advance_wp(uint16_t cid, uint64_t lba, uint16_t nlb) "cid %"PRIu16" lba 0x%"PRIx64" nlb %"PRIu16""
+ocssd_reset(uint16_t cid, uint16_t nlb) "cid %"PRIu16" nlb 0x%"PRIu16""
+ocssd_inject_write_err(uint16_t cid, uint8_t p, uint64_t lba) "cid %"PRIu16" p %"PRIu8" lba 0x%"PRIx64""
+ocssd_inject_reset_err(uint16_t cid, uint8_t p, uint64_t lba) "cid %"PRIu16" p %"PRIu8" lba 0x%"PRIx64""
+ocssd_io_cmd(uint16_t cid, uint32_t nsid, uint16_t opcode) "cid %"PRIu16" nsid %"PRIu32" opcode 0x%"PRIx16""
+
+# ocssd traces (notices)
+ocssd_notice_double_reset(uint16_t cid, uint64_t lba) "cid %"PRIu16" lba 0x%"PRIx64""
+ocssd_notice_early_reset(uint16_t cid, uint64_t lba, uint64_t wp) "cid %"PRIu16" lba 0x%"PRIx64" wp %"PRIu64""
+
+# ocssd traces (error conditions)
+ocssd_err(uint16_t cid, const char *s, uint16_t err) "cid %"PRIu16" \"%s\" err 0x%"PRIx16""
+ocssd_err_invalid_chunk(uint16_t cid, uint64_t lba) "cid %"PRIu16" lba 0x%"PRIx64""
+ocssd_err_invalid_chunk_state(uint16_t cid, uint64_t lba, uint8_t state) "cid %"PRIu16" lba 0x%"PRIx64" state 0x%"PRIx8""
+ocssd_err_offline_chunk(uint16_t cid, uint64_t lba) "cid %"PRIu16" lba 0x%"PRIx64""
+ocssd_err_write_constraints(uint16_t cid, uint16_t nlb, uint32_t ws_min) "cid %"PRIu16" nlb %"PRIu16" ws_min %"PRIu32""
+ocssd_err_out_of_bounds(uint16_t cid, uint32_t sectr, uint64_t cnlb) "cid %"PRIu16" sectr %"PRIu32" cnlb %"PRIu64""
+ocssd_err_out_of_order(uint16_t cid, uint32_t sectr, uint64_t wp) "cid %"PRIu16" sectr %"PRIu32" wp %"PRIu64""
diff --git a/hw/block/trace-events b/hw/block/trace-events
index 3324aac41dbb..98b36f60dcd6 100644
--- a/hw/block/trace-events
+++ b/hw/block/trace-events
@@ -29,115 +29,6 @@ virtio_blk_submit_multireq(void *vdev, void *mrb, int start, int num_reqs, uint6
hd_geometry_lchs_guess(void *blk, int cyls, int heads, int secs) "blk %p LCHS %d %d %d"
hd_geometry_guess(void *blk, uint32_t cyls, uint32_t heads, uint32_t secs, int trans) "blk %p CHS %u %u %u trans %d"
-# nvme.c
-# nvme traces for successful events
-nvme_irq_msix(uint32_t vector) "raising MSI-X IRQ vector %u"
-nvme_irq_pin(void) "pulsing IRQ pin"
-nvme_irq_masked(void) "IRQ is masked"
-nvme_dma_read(uint64_t prp1, uint64_t prp2) "DMA read, prp1=0x%"PRIx64" prp2=0x%"PRIx64""
-nvme_map_prp(uint8_t cmd_opcode, uint64_t trans_len, uint32_t len, uint64_t prp1, uint64_t prp2, int num_prps) "cmd_opcode=0x%"PRIx8", trans_len=%"PRIu64", len=%"PRIu32", prp1=0x%"PRIx64", prp2=0x%"PRIx64", num_prps=%d"
-nvme_map_sgl(uint16_t cid, uint64_t typ, uint16_t nlb, uint64_t len) "cid %"PRIu16" type %"PRIu64" nlb %"PRIu16" len %"PRIu64""
-nvme_rw(const char *verb, uint32_t blk_count, uint64_t byte_count, uint64_t lba) "%s %"PRIu32" blocks (%"PRIu64" bytes) from LBA %"PRIu64""
-nvme_rw_cb(uint16_t cid, uint32_t nsid) "cid %"PRIu16" nsid %"PRIu32""
-nvme_create_sq(uint64_t addr, uint16_t sqid, uint16_t cqid, uint16_t qsize, uint16_t qflags) "create submission queue, addr=0x%"PRIx64", sqid=%"PRIu16", cqid=%"PRIu16", qsize=%"PRIu16", qflags=%"PRIu16""
-nvme_create_cq(uint64_t addr, uint16_t cqid, uint16_t vector, uint16_t size, uint16_t qflags, int ien) "create completion queue, addr=0x%"PRIx64", cqid=%"PRIu16", vector=%"PRIu16", qsize=%"PRIu16", qflags=%"PRIu16", ien=%d"
-nvme_del_sq(uint16_t qid) "deleting submission queue sqid=%"PRIu16""
-nvme_del_cq(uint16_t cqid) "deleted completion queue, sqid=%"PRIu16""
-nvme_identify_ctrl(void) "identify controller"
-nvme_identify_ns(uint16_t ns) "identify namespace, nsid=%"PRIu16""
-nvme_identify_ns_list(uint16_t ns) "identify namespace list, nsid=%"PRIu16""
-nvme_identify_ns_descriptor_list(uint16_t ns) "identify namespace descriptor list, nsid=%"PRIu16""
-nvme_getfeat(uint32_t fid) "fid 0x%"PRIx32""
-nvme_setfeat(uint32_t fid, uint32_t val) "fid 0x%"PRIx32" val 0x%"PRIx32""
-nvme_getfeat_vwcache(const char* result) "get feature volatile write cache, result=%s"
-nvme_getfeat_numq(int result) "get feature number of queues, result=%d"
-nvme_setfeat_numq(int reqcq, int reqsq, int gotcq, int gotsq) "requested cq_count=%d sq_count=%d, responding with cq_count=%d sq_count=%d"
-nvme_get_log(uint16_t cid, uint16_t lid) "cid %"PRIu16" lid 0x%"PRIx16""
-nvme_process_aers(void) "processing aers"
-nvme_aer(uint16_t cid) "cid %"PRIu16""
-nvme_aer_aerl_exceeded(void) "aerl exceeded"
-nvme_aer_masked(uint8_t type, uint8_t mask) "type 0x%"PRIx8" mask 0x%"PRIx8""
-nvme_aer_post_cqe(uint8_t typ, uint8_t info, uint8_t log_page) "type 0x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8""
-nvme_enqueue_req_completion(uint16_t cid, uint16_t cqid) "cid %"PRIu16" cqid %"PRIu16""
-nvme_enqueue_event(uint8_t typ, uint8_t info, uint8_t log_page) "type 0x%"PRIx8" info 0x%"PRIx8" lid 0x%"PRIx8""
-nvme_no_outstanding_aers(void) "ignoring event; no outstanding AERs"
-nvme_mmio_intm_set(uint64_t data, uint64_t new_mask) "wrote MMIO, interrupt mask set, data=0x%"PRIx64", new_mask=0x%"PRIx64""
-nvme_mmio_intm_clr(uint64_t data, uint64_t new_mask) "wrote MMIO, interrupt mask clr, data=0x%"PRIx64", new_mask=0x%"PRIx64""
-nvme_mmio_cfg(uint64_t data) "wrote MMIO, config controller config=0x%"PRIx64""
-nvme_mmio_aqattr(uint64_t data) "wrote MMIO, admin queue attributes=0x%"PRIx64""
-nvme_mmio_asqaddr(uint64_t data) "wrote MMIO, admin submission queue address=0x%"PRIx64""
-nvme_mmio_acqaddr(uint64_t data) "wrote MMIO, admin completion queue address=0x%"PRIx64""
-nvme_mmio_asqaddr_hi(uint64_t data, uint64_t new_addr) "wrote MMIO, admin submission queue high half=0x%"PRIx64", new_address=0x%"PRIx64""
-nvme_mmio_acqaddr_hi(uint64_t data, uint64_t new_addr) "wrote MMIO, admin completion queue high half=0x%"PRIx64", new_address=0x%"PRIx64""
-nvme_mmio_start_success(void) "setting controller enable bit succeeded"
-nvme_mmio_stopped(void) "cleared controller enable bit"
-nvme_mmio_shutdown_set(void) "shutdown bit set"
-nvme_mmio_shutdown_cleared(void) "shutdown bit cleared"
-
-# nvme traces for error conditions
-nvme_err(uint16_t cid, const char *s, uint16_t status) "cid %"PRIu16" \"%s\" status 0x%"PRIx16""
-nvme_err_invalid_dma(void) "PRP/SGL is too small for transfer size"
-nvme_err_invalid_prplist_ent(uint64_t prplist) "PRP list entry is null or not page aligned: 0x%"PRIx64""
-nvme_err_invalid_prp2_align(uint64_t prp2) "PRP2 is not page aligned: 0x%"PRIx64""
-nvme_err_invalid_prp2_missing(void) "PRP2 is null and more data to be transferred"
-nvme_err_invalid_prp(void) "invalid PRP"
-nvme_err_invalid_ns(uint32_t ns, uint32_t limit) "invalid namespace %u not within 1-%u"
-nvme_err_invalid_opc(uint8_t opc) "invalid opcode 0x%"PRIx8""
-nvme_err_invalid_admin_opc(uint8_t opc) "invalid admin opcode 0x%"PRIx8""
-nvme_err_invalid_lba_range(uint64_t start, uint64_t len, uint64_t limit) "Invalid LBA start=%"PRIu64" len=%"PRIu64" limit=%"PRIu64""
-nvme_err_invalid_del_sq(uint16_t qid) "invalid submission queue deletion, sid=%"PRIu16""
-nvme_err_invalid_create_sq_cqid(uint16_t cqid) "failed creating submission queue, invalid cqid=%"PRIu16""
-nvme_err_invalid_create_sq_sqid(uint16_t sqid) "failed creating submission queue, invalid sqid=%"PRIu16""
-nvme_err_invalid_create_sq_size(uint16_t qsize) "failed creating submission queue, invalid qsize=%"PRIu16""
-nvme_err_invalid_create_sq_addr(uint64_t addr) "failed creating submission queue, addr=0x%"PRIx64""
-nvme_err_invalid_create_sq_qflags(uint16_t qflags) "failed creating submission queue, qflags=%"PRIu16""
-nvme_err_invalid_del_cq_cqid(uint16_t cqid) "failed deleting completion queue, cqid=%"PRIu16""
-nvme_err_invalid_del_cq_notempty(uint16_t cqid) "failed deleting completion queue, it is not empty, cqid=%"PRIu16""
-nvme_err_invalid_create_cq_cqid(uint16_t cqid) "failed creating completion queue, cqid=%"PRIu16""
-nvme_err_invalid_create_cq_size(uint16_t size) "failed creating completion queue, size=%"PRIu16""
-nvme_err_invalid_create_cq_addr(uint64_t addr) "failed creating completion queue, addr=0x%"PRIx64""
-nvme_err_invalid_create_cq_vector(uint16_t vector) "failed creating completion queue, vector=%"PRIu16""
-nvme_err_invalid_create_cq_qflags(uint16_t qflags) "failed creating completion queue, qflags=%"PRIu16""
-nvme_err_invalid_identify_cns(uint16_t cns) "identify, invalid cns=0x%"PRIx16""
-nvme_err_invalid_getfeat(int dw10) "invalid get features, dw10=0x%"PRIx32""
-nvme_err_invalid_setfeat(uint32_t dw10) "invalid set features, dw10=0x%"PRIx32""
-nvme_err_invalid_log_page(uint16_t cid, uint16_t lid) "cid %"PRIu16" lid 0x%"PRIx16""
-nvme_err_startfail_cq(void) "nvme_start_ctrl failed because there are non-admin completion queues"
-nvme_err_startfail_sq(void) "nvme_start_ctrl failed because there are non-admin submission queues"
-nvme_err_startfail_nbarasq(void) "nvme_start_ctrl failed because the admin submission queue address is null"
-nvme_err_startfail_nbaracq(void) "nvme_start_ctrl failed because the admin completion queue address is null"
-nvme_err_startfail_asq_misaligned(uint64_t addr) "nvme_start_ctrl failed because the admin submission queue address is misaligned: 0x%"PRIx64""
-nvme_err_startfail_acq_misaligned(uint64_t addr) "nvme_start_ctrl failed because the admin completion queue address is misaligned: 0x%"PRIx64""
-nvme_err_startfail_page_too_small(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the page size is too small: log2size=%u, min=%u"
-nvme_err_startfail_page_too_large(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the page size is too large: log2size=%u, max=%u"
-nvme_err_startfail_cqent_too_small(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the completion queue entry size is too small: log2size=%u, min=%u"
-nvme_err_startfail_cqent_too_large(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the completion queue entry size is too large: log2size=%u, max=%u"
-nvme_err_startfail_sqent_too_small(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the submission queue entry size is too small: log2size=%u, min=%u"
-nvme_err_startfail_sqent_too_large(uint8_t log2ps, uint8_t maxlog2ps) "nvme_start_ctrl failed because the submission queue entry size is too large: log2size=%u, max=%u"
-nvme_err_startfail_asqent_sz_zero(void) "nvme_start_ctrl failed because the admin submission queue size is zero"
-nvme_err_startfail_acqent_sz_zero(void) "nvme_start_ctrl failed because the admin completion queue size is zero"
-nvme_err_startfail(void) "setting controller enable bit failed"
-nvme_err_internal_dev_error(const char *reason) "%s"
-
-# Traces for undefined behavior
-nvme_ub_mmiowr_misaligned32(uint64_t offset) "MMIO write not 32-bit aligned, offset=0x%"PRIx64""
-nvme_ub_mmiowr_toosmall(uint64_t offset, unsigned size) "MMIO write smaller than 32 bits, offset=0x%"PRIx64", size=%u"
-nvme_ub_mmiowr_intmask_with_msix(void) "undefined access to interrupt mask set when MSI-X is enabled"
-nvme_ub_mmiowr_ro_csts(void) "attempted to set a read only bit of controller status"
-nvme_ub_mmiowr_ssreset_w1c_unsupported(void) "attempted to W1C CSTS.NSSRO but CAP.NSSRS is zero (not supported)"
-nvme_ub_mmiowr_ssreset_unsupported(void) "attempted NVM subsystem reset but CAP.NSSRS is zero (not supported)"
-nvme_ub_mmiowr_cmbloc_reserved(void) "invalid write to reserved CMBLOC when CMBSZ is zero, ignored"
-nvme_ub_mmiowr_cmbsz_readonly(void) "invalid write to read only CMBSZ, ignored"
-nvme_ub_mmiowr_invalid(uint64_t offset, uint64_t data) "invalid MMIO write, offset=0x%"PRIx64", data=0x%"PRIx64""
-nvme_ub_mmiord_misaligned32(uint64_t offset) "MMIO read not 32-bit aligned, offset=0x%"PRIx64""
-nvme_ub_mmiord_toosmall(uint64_t offset) "MMIO read smaller than 32-bits, offset=0x%"PRIx64""
-nvme_ub_mmiord_invalid_ofs(uint64_t offset) "MMIO read beyond last register, offset=0x%"PRIx64", returning 0"
-nvme_ub_db_wr_misaligned(uint64_t offset) "doorbell write not 32-bit aligned, offset=0x%"PRIx64", ignoring"
-nvme_ub_db_wr_invalid_cq(uint32_t qid) "completion queue doorbell write for nonexistent queue, cqid=%"PRIu32", ignoring"
-nvme_ub_db_wr_invalid_cqhead(uint32_t qid, uint16_t new_head) "completion queue doorbell write value beyond queue size, cqid=%"PRIu32", new_head=%"PRIu16", ignoring"
-nvme_ub_db_wr_invalid_sq(uint32_t qid) "submission queue doorbell write for nonexistent queue, sqid=%"PRIu32", ignoring"
-nvme_ub_db_wr_invalid_sqtail(uint32_t qid, uint16_t new_tail) "submission queue doorbell write value beyond queue size, sqid=%"PRIu32", new_head=%"PRIu16", ignoring"
-
# xen-block.c
xen_block_realize(const char *type, uint32_t disk, uint32_t partition) "%s d%up%u"
xen_block_connect(const char *type, uint32_t disk, uint32_t partition) "%s d%up%u"
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 94d45c970894..0dced3ca7f08 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -120,6 +120,9 @@ struct BlockDriver {
/* Set if a driver can support backing files */
bool supports_backing;
+ /* Set if a driver does not require a size parameter */
+ bool no_size_required;
+
/* For handling image reopen for split or non-split files */
int (*bdrv_reopen_prepare)(BDRVReopenState *reopen_state,
BlockReopenQueue *queue, Error **errp);
diff --git a/include/block/nvme.h b/include/block/nvme.h
index 583b61a76570..392fc7f087c0 100644
--- a/include/block/nvme.h
+++ b/include/block/nvme.h
@@ -461,8 +461,13 @@ typedef struct NvmeAerResult {
} NvmeAerResult;
typedef struct NvmeCqe {
- uint32_t result;
- uint32_t rsvd;
+ union {
+ struct {
+ uint32_t cdw0;
+ uint32_t rsvd;
+ };
+ uint64_t res64;
+ };
uint16_t sq_head;
uint16_t sq_id;
uint16_t cid;
@@ -520,6 +525,7 @@ enum NvmeStatusCodes {
NVME_E2E_REF_ERROR = 0x0284,
NVME_CMP_FAILURE = 0x0285,
NVME_ACCESS_DENIED = 0x0286,
+ NVME_DULB = 0x0287,
NVME_MORE = 0x2000,
NVME_DNR = 0x4000,
NVME_NO_COMPLETE = 0xffff,
@@ -722,6 +728,8 @@ typedef struct NvmeFeatureVal {
#define NVME_INTC_THR(intc) (intc & 0xff)
#define NVME_INTC_TIME(intc) ((intc >> 8) & 0xff)
+#define NVME_ERR_REC_DULBE(err_rec) (err_rec & 0x10000)
+
enum NvmeFeatureIds {
NVME_ARBITRATION = 0x1,
NVME_POWER_MANAGEMENT = 0x2,
diff --git a/include/block/ocssd.h b/include/block/ocssd.h
new file mode 100644
index 000000000000..b3174a687401
--- /dev/null
+++ b/include/block/ocssd.h
@@ -0,0 +1,231 @@
+/*
+ * QEMU OpenChannel 2.0 Controller
+ *
+ * Copyright (c) 2019 CNEX Labs, Inc.
+ *
+ * Thank you to the following people for their contributions to the original
+ * qemu-nvme (github.com/OpenChannelSSD/qemu-nvme) implementation.
+ *
+ * Matias Bjørling <mb@lightnvm.io>
+ * Javier González <javier@javigon.com>
+ * Simon Andreas Frimann Lund <ocssd@safl.dk>
+ * Hans Holmberg <hans@owltronix.com>
+ * Jesper Devantier <contact@pseudonymous.me>
+ * Young Tack Jin <youngtack.jin@circuitblvd.com>
+ *
+ * This code is licensed under the GNU GPL v2 or later.
+ */
+
+#ifndef BLOCK_OCSSD_H
+#define BLOCK_OCSSD_H
+
+#include "block/nvme.h"
+
+#define OCSSD_MAGIC ('O' << 24 | 'C' << 16 | '2' << 8 | '0')
+
+enum OcssdAdminCommands {
+ OCSSD_ADM_CMD_GEOMETRY = 0xe2,
+};
+
+enum OcssdIoCommands {
+ OCSSD_CMD_VECT_RESET = 0x90,
+ OCSSD_CMD_VECT_WRITE = 0x91,
+ OCSSD_CMD_VECT_READ = 0x92,
+ OCSSD_CMD_VECT_COPY = 0x93,
+};
+
+typedef enum OcssdChunkState {
+ OCSSD_CHUNK_FREE = 1 << 0,
+ OCSSD_CHUNK_CLOSED = 1 << 1,
+ OCSSD_CHUNK_OPEN = 1 << 2,
+ OCSSD_CHUNK_OFFLINE = 1 << 3,
+} OcssdChunkState;
+
+#define OCSSD_CHUNK_RESETABLE \
+ (OCSSD_CHUNK_FREE | OCSSD_CHUNK_CLOSED | OCSSD_CHUNK_OPEN)
+
+typedef enum OcssdChunkType {
+ OCSSD_CHUNK_TYPE_SEQUENTIAL = 1 << 0,
+ OCSSD_CHUNK_TYPE_RANDOM = 1 << 1,
+ OCSSD_CHUNK_TYPE_SHRINKED = 1 << 4,
+} OcssdChunkType;
+
+enum OcssdStatusCodes {
+ OCSSD_LBAL_SGL_LENGTH_INVALID = 0x01c1,
+
+ OCSSD_WRITE_NEXT_UNIT = 0x02f0,
+ OCSSD_CHUNK_EARLY_CLOSE = 0x02f1,
+ OCSSD_OUT_OF_ORDER_WRITE = 0x02f2,
+ OCSSD_OFFLINE_CHUNK = 0x02c0,
+ OCSSD_INVALID_RESET = 0x02c1,
+};
+
+typedef struct OcssdFeatureVal {
+ uint32_t media_feedback;
+} OcssdFeatureVal;
+
+#define OCSSD_MEDIA_FEEDBACK_VHECC(media_feedback) (media_feedback & 0x2)
+#define OCSSD_MEDIA_FEEDBACK_HECC(media_feedback) (media_feedback & 0x1)
+
+enum OcssdFeatureIds {
+ OCSSD_MEDIA_FEEDBACK = 0xca,
+};
+
+typedef struct OcssdChunkDescriptor {
+ uint8_t state;
+ uint8_t type;
+ uint8_t wear_index;
+ uint8_t rsvd7[5];
+ uint64_t slba;
+ uint64_t cnlb;
+ uint64_t wp;
+} OcssdChunkDescriptor;
+
+enum OcssdChunkNotificationState {
+ OCSSD_CHUNK_NOTIFICATION_STATE_LOW = 1 << 0,
+ OCSSD_CHUNK_NOTIFICATION_STATE_MID = 1 << 1,
+ OCSSD_CHUNK_NOTIFICATION_STATE_HIGH = 1 << 2,
+ OCSSD_CHUNK_NOTIFICATION_STATE_UNREC = 1 << 3,
+ OCSSD_CHUNK_NOTIFICATION_STATE_REFRESHED = 1 << 4,
+ OCSSD_CHUNK_NOTIFICATION_STATE_WLI = 1 << 8
+};
+
+enum OcssdChunkNotificationMask {
+ OCSSD_CHUNK_NOTIFICATION_MASK_SECTOR = 1 << 0,
+ OCSSD_CHUNK_NOTIFICATION_MASK_CHUNK = 1 << 1,
+ OCSSD_CHUNK_NOTIFICATION_MASK_PUNIT = 1 << 2
+};
+
+typedef struct OcssdChunkNotification {
+ uint64_t nc;
+ uint64_t lba;
+ uint32_t nsid;
+ uint16_t state;
+ uint8_t mask;
+ uint8_t rsvd31[9];
+ uint16_t nlb;
+ uint8_t rsvd63[30];
+} OcssdChunkNotification;
+
+typedef struct OcssdRwCmd {
+ uint16_t opcode:8;
+ uint16_t fuse:2;
+ uint16_t rsvd1:4;
+ uint16_t psdt:2;
+ uint16_t cid;
+ uint32_t nsid;
+ uint64_t rsvd2;
+ uint64_t metadata;
+ NvmeCmdDptr dptr;
+ uint64_t lbal;
+ uint16_t nlb;
+ uint16_t control;
+ uint32_t rsvd3;
+ uint64_t rsvd4;
+} OcssdRwCmd;
+
+typedef struct OcssdCopyCmd {
+ uint16_t opcode:8;
+ uint16_t fuse:2;
+ uint16_t rsvd1:4;
+ uint16_t psdt:2;
+ uint16_t cid;
+ uint32_t nsid;
+ uint64_t rsvd2;
+ uint64_t metadata;
+ NvmeCmdDptr dptr;
+ uint64_t lbal;
+ uint16_t nlb;
+ uint16_t control;
+ uint32_t rsvd3;
+ uint64_t dlbal;
+} OcssdCopyCmd;
+
+typedef struct OcssdIdGeo {
+ uint16_t num_grp;
+ uint16_t num_pu;
+ uint32_t num_chk;
+ uint32_t clba;
+ uint8_t rsvd63[52];
+} OcssdIdGeo;
+
+typedef struct OcssdIdWrt {
+ uint32_t ws_min;
+ uint32_t ws_opt;
+ uint32_t mw_cunits;
+ uint8_t rsvd63[52];
+} OcssdIdWrt;
+
+typedef struct OcssdIdPerf {
+ uint32_t trdt;
+ uint32_t trdm;
+ uint32_t tprt;
+ uint32_t tprm;
+ uint32_t tbet;
+ uint32_t tbem;
+ uint8_t rsvd63[40];
+} OcssdIdPerf;
+
+typedef struct OcssdIdLBAF {
+ uint8_t grp_len;
+ uint8_t pu_len;
+ uint8_t chk_len;
+ uint8_t sec_len;
+ uint8_t rsvd7[4];
+} OcssdIdLBAF;
+
+typedef struct OcssdFormatHeader {
+ uint32_t magic;
+ uint32_t version;
+ uint32_t num_ns;
+ uint32_t md_size;
+ uint64_t sector_size;
+ uint64_t ns_size;
+ uint32_t pe_cycles;
+ OcssdIdLBAF lbaf;
+ uint8_t rsvd4095[4052];
+} OcssdFormatHeader;
+
+typedef struct OcssdIdentity {
+ struct {
+ uint8_t major;
+ uint8_t minor;
+ } ver;
+ uint8_t rsvd1[6];
+ OcssdIdLBAF lbaf;
+ uint32_t mccap;
+ uint8_t rsvd2[12];
+ uint8_t wit;
+ uint8_t rsvd3[31];
+ OcssdIdGeo geo;
+ OcssdIdWrt wrt;
+ OcssdIdPerf perf;
+ uint8_t rsvd4[3840];
+} OcssdIdentity;
+
+enum OcssdIdentityMccap {
+ OCSSD_IDENTITY_MCCAP_MULTIPLE_RESETS = 0x1 << 1,
+
+ /* OCSSD 2.0 spec de-facto extension */
+ OCSSD_IDENTITY_MCCAP_EARLY_RESET = 0x1 << 2,
+};
+
+enum OcssdLogPage {
+ OCSSD_CHUNK_INFO = 0xCA,
+ OCSSD_CHUNK_NOTIFICATION = 0xD0,
+};
+
+static inline void _ocssd_check_sizes(void)
+{
+ QEMU_BUILD_BUG_ON(sizeof(OcssdIdLBAF) != 8);
+ QEMU_BUILD_BUG_ON(sizeof(OcssdIdGeo) != 64);
+ QEMU_BUILD_BUG_ON(sizeof(OcssdIdWrt) != 64);
+ QEMU_BUILD_BUG_ON(sizeof(OcssdIdPerf) != 64);
+ QEMU_BUILD_BUG_ON(sizeof(OcssdRwCmd) != 64);
+ QEMU_BUILD_BUG_ON(sizeof(OcssdIdentity) != 4096);
+ QEMU_BUILD_BUG_ON(sizeof(OcssdChunkDescriptor) != 32);
+ QEMU_BUILD_BUG_ON(sizeof(OcssdChunkNotification) != 64);
+ QEMU_BUILD_BUG_ON(sizeof(OcssdFormatHeader) != 4096);
+}
+
+#endif
diff --git a/hw/block/nvme.h b/include/hw/block/nvme.h
similarity index 63%
rename from hw/block/nvme.h
rename to include/hw/block/nvme.h
index 7e1e026d90e6..db8ea0b6d8ac 100644
--- a/hw/block/nvme.h
+++ b/include/hw/block/nvme.h
@@ -1,7 +1,17 @@
#ifndef HW_NVME_H
#define HW_NVME_H
+#include "qemu/log.h"
#include "block/nvme.h"
+#include "hw/block/block.h"
+#include "hw/pci/pci.h"
+
+#define NVME_GUEST_ERR(trace, fmt, ...) \
+ do { \
+ (trace_##trace)(__VA_ARGS__); \
+ qemu_log_mask(LOG_GUEST_ERROR, #trace \
+ " in %s: " fmt "\n", __func__, ## __VA_ARGS__); \
+ } while (0)
#define DEFINE_NVME_PROPERTIES(_state, _props) \
DEFINE_PROP_STRING("serial", _state, _props.serial), \
@@ -49,6 +59,7 @@ typedef struct NvmeRequest {
uint64_t slba;
uint16_t nlb;
+ hwaddr mptr;
uint16_t status;
bool is_cmb;
bool is_write;
@@ -139,8 +150,14 @@ typedef struct NvmeCtrl {
NvmeCQueue admin_cq;
NvmeFeatureVal features;
NvmeIdCtrl id_ctrl;
+
+ uint16_t (*admin_cmd)(struct NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req);
+ uint16_t (*io_cmd)(struct NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req);
} NvmeCtrl;
+typedef uint16_t (*NvmeBlockSetupFn)(NvmeCtrl *n, NvmeNamespace *ns,
+ QEMUSGList *qsg, uint64_t blk_offset, uint32_t unit_len, NvmeRequest *req);
+
static inline bool nvme_rw_is_write(NvmeRequest *req)
{
return req->cmd.opcode == NVME_CMD_WRITE;
@@ -169,4 +186,48 @@ static inline uint16_t nvme_ns_ms(NvmeNamespace *ns)
return le16_to_cpu(id->lbaf[NVME_ID_NS_FLBAS_INDEX(id->flbas)].ms);
}
+void nvme_addr_read(NvmeCtrl *n, hwaddr addr, void *buf, int size);
+void nvme_addr_write(NvmeCtrl *n, hwaddr addr, void *buf, int size);
+
+uint16_t nvme_dma_read(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
+ NvmeCmd *cmd, NvmeRequest *req);
+uint16_t nvme_dma_read_sgl(NvmeCtrl *n, uint8_t *ptr, uint32_t len,
+ NvmeSglDescriptor sgl, NvmeCmd *cmd, NvmeRequest *req);
+uint16_t nvme_dma_write(NvmeCtrl *n, uint8_t *ptr, uint32_t len, NvmeCmd *cmd,
+ NvmeRequest *req);
+
+void nvme_rw_cb(void *opaque, int ret);
+uint16_t nvme_rw_check_req(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req);
+
+void nvme_clear_events(NvmeCtrl *n, uint8_t event_type);
+void nvme_enqueue_event(NvmeCtrl *n, uint8_t event_type, uint8_t event_info,
+ uint8_t log_page);
+
+void nvme_enqueue_req_completion(NvmeCQueue *cq, NvmeRequest *req);
+
+NvmeBlockBackendRequest *nvme_blk_req_get(NvmeCtrl *n, NvmeRequest *req,
+ QEMUSGList *qsg);
+void nvme_blk_req_put(NvmeCtrl *n, NvmeBlockBackendRequest *blk_req);
+
+uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req,
+ NvmeBlockSetupFn blk_setup);
+uint16_t nvme_blk_submit_io(NvmeCtrl *n, NvmeRequest *req,
+ BlockCompletionFunc *cb);
+
+uint16_t nvme_io_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req);
+
+uint16_t nvme_get_log(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req);
+uint16_t nvme_get_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req);
+uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req);
+uint16_t nvme_admin_cmd(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req);
+
+void nvme_ns_init_identify(NvmeCtrl *n, NvmeIdNs *id_ns);
+
+int nvme_init_blk(NvmeCtrl *n, Error **errp);
+int nvme_init_state(NvmeCtrl *n, Error **errp);
+void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev);
+void nvme_init_ctrl(NvmeCtrl *n);
+
+void nvme_free_ctrl(NvmeCtrl *n, PCIDevice *pci_dev);
+
#endif /* HW_NVME_H */
diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
index 0abe27a53a38..ece1f37cbe01 100644
--- a/include/hw/pci/pci_ids.h
+++ b/include/hw/pci/pci_ids.h
@@ -273,4 +273,6 @@
#define PCI_VENDOR_ID_NVIDIA 0x10de
+#define PCI_VENDOR_ID_CNEX 0x1d1d
+
#endif
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 7ccbfff9d0b4..5e13808530c3 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -113,6 +113,44 @@
'extents': ['ImageInfo']
} }
+##
+# @ImageInfoSpecificOcssd:
+#
+# @num-ns: number of namespaces
+#
+# @namespaces: List of namespsaces
+#
+# Since: 4.1
+##
+{ 'struct': 'ImageInfoSpecificOcssd',
+ 'data': {
+ 'num-ns': 'int',
+ 'sector-size': 'int',
+ 'metadata-size': 'int',
+ 'namespaces': ['ImageInfoSpecificOcssdNS']
+ } }
+
+##
+# @ImageInfoSpecificOcssdNS:
+#
+# @num-grp: number of groups
+#
+# @num-pu: number of parallel units per group
+#
+# @num-chk: number of chunks per parallel unit
+#
+# @num-sec: number of sectors per chunk
+#
+# Since: 3.1
+##
+{ 'struct': 'ImageInfoSpecificOcssdNS',
+ 'data': {
+ 'num-grp': 'int',
+ 'num-pu': 'int',
+ 'num-chk': 'int',
+ 'num-sec': 'int'
+ } }
+
##
# @ImageInfoSpecific:
#
@@ -124,6 +162,7 @@
'data': {
'qcow2': 'ImageInfoSpecificQCow2',
'vmdk': 'ImageInfoSpecificVmdk',
+ 'ocssd': 'ImageInfoSpecificOcssd',
# If we need to add block driver specific parameters for
# LUKS in future, then we'll subclass QCryptoBlockInfoLUKS
# to define a ImageInfoSpecificLUKS
@@ -282,7 +321,7 @@
# @drv: the name of the block format used to open the backing device. As of
# 0.14.0 this can be: 'blkdebug', 'bochs', 'cloop', 'cow', 'dmg',
# 'file', 'file', 'ftp', 'ftps', 'host_cdrom', 'host_device',
-# 'http', 'https', 'luks', 'nbd', 'parallels', 'qcow',
+# 'http', 'https', 'luks', 'nbd', 'ocssd', 'parallels', 'qcow',
# 'qcow2', 'raw', 'vdi', 'vmdk', 'vpc', 'vvfat'
# 2.2: 'archipelago' added, 'cow' dropped
# 2.3: 'host_floppy' deprecated
@@ -290,6 +329,7 @@
# 2.6: 'luks' added
# 2.8: 'replication' added, 'tftp' dropped
# 2.9: 'archipelago' dropped
+# 4.1: 'ocssd' added
#
# @backing_file: the name of the backing file (for copy-on-write)
#
@@ -2815,8 +2855,8 @@
'data': [ 'blkdebug', 'blklogwrites', 'blkverify', 'bochs', 'cloop',
'copy-on-read', 'dmg', 'file', 'ftp', 'ftps', 'gluster',
'host_cdrom', 'host_device', 'http', 'https', 'iscsi', 'luks',
- 'nbd', 'nfs', 'null-aio', 'null-co', 'nvme', 'parallels', 'qcow',
- 'qcow2', 'qed', 'quorum', 'raw', 'rbd',
+ 'nbd', 'nfs', 'null-aio', 'null-co', 'nvme', 'ocssd', 'parallels',
+ 'qcow', 'qcow2', 'qed', 'quorum', 'raw', 'rbd',
{ 'name': 'replication', 'if': 'defined(CONFIG_REPLICATION)' },
'sheepdog',
'ssh', 'throttle', 'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat', 'vxhs' ] }
@@ -3917,6 +3957,7 @@
'null-aio': 'BlockdevOptionsNull',
'null-co': 'BlockdevOptionsNull',
'nvme': 'BlockdevOptionsNVMe',
+ 'ocssd': 'BlockdevOptionsGenericFormat',
'parallels': 'BlockdevOptionsGenericFormat',
'qcow2': 'BlockdevOptionsQcow2',
'qcow': 'BlockdevOptionsQcow',
--
2.21.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device
2019-05-17 8:42 [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Klaus Birkelund Jensen
` (7 preceding siblings ...)
2019-05-17 8:42 ` [Qemu-devel] [PATCH 8/8] nvme: add an OpenChannel 2.0 NVMe device (ocssd) Klaus Birkelund Jensen
@ 2019-05-20 13:01 ` Kevin Wolf
2019-05-20 13:32 ` Klaus Birkelund
[not found] ` <20190520193445.GA22742@apples.localdomain>
8 siblings, 2 replies; 15+ messages in thread
From: Kevin Wolf @ 2019-05-20 13:01 UTC (permalink / raw)
To: Klaus Birkelund Jensen
Cc: Keith Busch, armbru, qemu-devel, qemu-block, Max Reitz
Am 17.05.2019 um 10:42 hat Klaus Birkelund Jensen geschrieben:
> Hi,
>
> This series of patches contains a number of refactorings to the emulated
> nvme device, adds additional features, such as support for metadata and
> scatter gather lists, and bumps the supported NVMe version to 1.3.
> Lastly, it contains a new 'ocssd' device.
>
> The motivation for the first seven patches is to set everything up for
> the final patch that adds a new 'ocssd' device and associated block
> driver that implements the OpenChannel 2.0 specification[1]. Many of us
> in the OpenChannel comunity have used a qemu fork[2] for emulation of
> OpenChannel devices. The fork is itself based on Keith's qemu-nvme
> tree[3] and we recently merged mainline qemu into it, but the result is
> still a "hybrid" nvme device that supports both conventional nvme and
> the OCSSD 2.0 spec through a 'dialect' mechanism. Merging instead of
> rebasing also created a pretty messy commit history and my efforts to
> try and rebase our work onto mainline was getting hairy to say the
> least. And I was never really happy with the dialect approach anyway.
>
> I have instead prepared this series of fresh patches that incrementally
> adds additional features to the nvme device to bring it into shape for
> finally introducing a new (and separate) 'ocssd' device that emulates an
> OpenChannel 2.0 device by reusing core functionality from the nvme
> device. Providing a separate ocssd device ensures that no ocssd specific
> stuff creeps into the nvme device.
>
> The ocssd device is backed by a new 'ocssd' block driver that holds
> internal meta data and keeps state permanent across power cycles. In the
> future I think we could use the same approach for the nvme device to
> keep internal metadata such as utilization and deallocated blocks.
A backend driver that is specific for a guest device model (i.e. the
device model requires this driver, and the backend is useless without
the device) sounds like a very questionable design.
Metadata like OcssdFormatHeader that is considered part of the image
data, which means that the _actual_ image content without metadata isn't
directly accessible any more feels like a bad idea, too. Simple things
like what a resize operation means (change only the actual disk size as
usual, or is the new size disk + metadata?) become confusing. Attaching
an image to a different device becomes impossible.
The block format driver doesn't seem to actually add much functionality
to a specially crafted raw image: It provides a convenient way to create
such special images and it dumps some values in 'qemu-img info', but the
actual interpretation of the data is left to the device model.
Looking at the options it does provide, my impression is that these
should really be qdev properties, and the place to store them
persistently is something like the libvirt XML. The device doesn't
change any of the values, so there is nothing that QEMU actually needs
to store. What you invented is a one-off way to pass a config file to a
device, but only for one specific device type.
I think this needs to use a much more standard approach to be mergable.
Markus (CCed) as the maintainer for the configuration mechanisms may
have an opinion on this, too.
> For now, the nvme device does not support the Deallocated and
> Unwritten Logical Block Error (DULBE) feature or the Data Set
> Management command as this would require such support.
Doesn't bdrv_co_block_status() provide all the information you need for
that?
Kevin
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device
2019-05-20 13:01 ` [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Kevin Wolf
@ 2019-05-20 13:32 ` Klaus Birkelund
[not found] ` <20190520193445.GA22742@apples.localdomain>
1 sibling, 0 replies; 15+ messages in thread
From: Klaus Birkelund @ 2019-05-20 13:32 UTC (permalink / raw)
To: Kevin Wolf; +Cc: Keith Busch, armbru, qemu-devel, qemu-block, Max Reitz
On Mon, May 20, 2019 at 03:01:24PM +0200, Kevin Wolf wrote:
> Am 17.05.2019 um 10:42 hat Klaus Birkelund Jensen geschrieben:
> > Hi,
> >
> > This series of patches contains a number of refactorings to the emulated
> > nvme device, adds additional features, such as support for metadata and
> > scatter gather lists, and bumps the supported NVMe version to 1.3.
> > Lastly, it contains a new 'ocssd' device.
> >
> > The motivation for the first seven patches is to set everything up for
> > the final patch that adds a new 'ocssd' device and associated block
> > driver that implements the OpenChannel 2.0 specification[1]. Many of us
> > in the OpenChannel comunity have used a qemu fork[2] for emulation of
> > OpenChannel devices. The fork is itself based on Keith's qemu-nvme
> > tree[3] and we recently merged mainline qemu into it, but the result is
> > still a "hybrid" nvme device that supports both conventional nvme and
> > the OCSSD 2.0 spec through a 'dialect' mechanism. Merging instead of
> > rebasing also created a pretty messy commit history and my efforts to
> > try and rebase our work onto mainline was getting hairy to say the
> > least. And I was never really happy with the dialect approach anyway.
> >
> > I have instead prepared this series of fresh patches that incrementally
> > adds additional features to the nvme device to bring it into shape for
> > finally introducing a new (and separate) 'ocssd' device that emulates an
> > OpenChannel 2.0 device by reusing core functionality from the nvme
> > device. Providing a separate ocssd device ensures that no ocssd specific
> > stuff creeps into the nvme device.
> >
> > The ocssd device is backed by a new 'ocssd' block driver that holds
> > internal meta data and keeps state permanent across power cycles. In the
> > future I think we could use the same approach for the nvme device to
> > keep internal metadata such as utilization and deallocated blocks.
>
> A backend driver that is specific for a guest device model (i.e. the
> device model requires this driver, and the backend is useless without
> the device) sounds like a very questionable design.
>
> Metadata like OcssdFormatHeader that is considered part of the image
> data, which means that the _actual_ image content without metadata isn't
> directly accessible any more feels like a bad idea, too. Simple things
> like what a resize operation means (change only the actual disk size as
> usual, or is the new size disk + metadata?) become confusing. Attaching
> an image to a different device becomes impossible.
>
> The block format driver doesn't seem to actually add much functionality
> to a specially crafted raw image: It provides a convenient way to create
> such special images and it dumps some values in 'qemu-img info', but the
> actual interpretation of the data is left to the device model.
>
> Looking at the options it does provide, my impression is that these
> should really be qdev properties, and the place to store them
> persistently is something like the libvirt XML. The device doesn't
> change any of the values, so there is nothing that QEMU actually needs
> to store. What you invented is a one-off way to pass a config file to a
> device, but only for one specific device type.
>
> I think this needs to use a much more standard approach to be mergable.
>
> Markus (CCed) as the maintainer for the configuration mechanisms may
> have an opinion on this, too.
Hi Kevin,
Thank you for going through my motivations. I see what you mean. And
yes, the main reason I did it like that was for the convenience of being
able to `qemu-img create`'ing the image. I'll reconsider how to do this.
>
> > For now, the nvme device does not support the Deallocated and
> > Unwritten Logical Block Error (DULBE) feature or the Data Set
> > Management command as this would require such support.
>
> Doesn't bdrv_co_block_status() provide all the information you need for
> that?
>
That does look useful. I'll look into it.
Thanks!
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] [PATCH 8/8] nvme: add an OpenChannel 2.0 NVMe device (ocssd)
2019-05-17 8:42 ` [Qemu-devel] [PATCH 8/8] nvme: add an OpenChannel 2.0 NVMe device (ocssd) Klaus Birkelund Jensen
@ 2019-05-20 16:45 ` Eric Blake
2019-05-20 17:33 ` Klaus Birkelund
0 siblings, 1 reply; 15+ messages in thread
From: Eric Blake @ 2019-05-20 16:45 UTC (permalink / raw)
To: Klaus Birkelund Jensen, qemu-block
Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
[-- Attachment #1: Type: text/plain, Size: 6889 bytes --]
On 5/17/19 3:42 AM, Klaus Birkelund Jensen wrote:
> This adds a new 'ocssd' block device that emulates an OpenChannel 2.0
> device. The device is backed by a new 'ocssd' block backend that is
> based on the raw format driver but includes a header that holds the
> device geometry and write data requirements. This new block backend is
> special in that the size is not specified explicitly but in terms of
> sector size, number of chunks, number of parallel units, etc. This
> called for the addition of the `no_size_required` field in `struct
> BlockDriver` to not fail image creation when the size parameter is
> missing.
>
> The ocssd device is an individual device but shares a lot of code with
> the nvme device. Thus, some core functionality of nvme/nvme.c has been
> exported for use by nvme/ocssd.c.
>
> Thank you to the following people for their contributions to the
> original qemu-nvme (github.com/OpenChannelSSD/qemu-nvme) implementation.
>
> Matias Bjørling <mb@lightnvm.io>
> Javier González <javier@javigon.com>
> Simon Andreas Frimann Lund <ocssd@safl.dk>
> Hans Holmberg <hans@owltronix.com>
> Jesper Devantier <contact@pseudonymous.me>
> Young Tack Jin <youngtack.jin@circuitblvd.com>
>
> Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
> ---
> MAINTAINERS | 14 +-
> Makefile.objs | 1 +
> block.c | 2 +-
> block/Makefile.objs | 2 +-
> block/nvme.c | 2 +-
> block/ocssd.c | 690 ++++++++
> hw/block/Makefile.objs | 2 +-
> hw/block/{ => nvme}/nvme.c | 192 ++-
> hw/block/nvme/ocssd.c | 2647 +++++++++++++++++++++++++++++++
> hw/block/nvme/ocssd.h | 140 ++
> hw/block/nvme/trace-events | 136 ++
> hw/block/trace-events | 109 --
> include/block/block_int.h | 3 +
> include/block/nvme.h | 12 +-
> include/block/ocssd.h | 231 +++
> {hw => include/hw}/block/nvme.h | 61 +
> include/hw/pci/pci_ids.h | 2 +
> qapi/block-core.json | 47 +-
> 18 files changed, 4121 insertions(+), 172 deletions(-)
> create mode 100644 block/ocssd.c
> rename hw/block/{ => nvme}/nvme.c (94%)
> create mode 100644 hw/block/nvme/ocssd.c
> create mode 100644 hw/block/nvme/ocssd.h
> create mode 100644 hw/block/nvme/trace-events
> create mode 100644 include/block/ocssd.h
> rename {hw => include/hw}/block/nvme.h (63%)
Feels big; are you sure this can't be split into smaller pieces to ease
review?
I'm focusing just on the qapi portions:
> +++ b/qapi/block-core.json
> @@ -113,6 +113,44 @@
> 'extents': ['ImageInfo']
> } }
>
> +##
> +# @ImageInfoSpecificOcssd:
> +#
> +# @num-ns: number of namespaces
> +#
> +# @namespaces: List of namespsaces
Inconsistent on whether you start with lower or upper case.
> +#
> +# Since: 4.1
> +##
> +{ 'struct': 'ImageInfoSpecificOcssd',
> + 'data': {
> + 'num-ns': 'int',
> + 'sector-size': 'int',
> + 'metadata-size': 'int',
> + 'namespaces': ['ImageInfoSpecificOcssdNS']
Is num-ns redundant information (that is, if I count the length of the
array in namespaces, I don't need num-ns)?
You failed to document sector-size and metadata-size.
> + } }
> +
> +##
> +# @ImageInfoSpecificOcssdNS:
> +#
> +# @num-grp: number of groups
> +#
> +# @num-pu: number of parallel units per group
> +#
> +# @num-chk: number of chunks per parallel unit
> +#
> +# @num-sec: number of sectors per chunk
Why the abbreviation? Machines don't care if they have to pass something
slightly longer, and if it makes the job easier for the human reading
the machine traces, then spelling things out (such as 'num-groups'),
then abbreviation didn't buy us anything useful.
> +#
> +# Since: 3.1
You missed 3.1 by a long shot. The next release is 4.1.
> +##
> +{ 'struct': 'ImageInfoSpecificOcssdNS',
> + 'data': {
> + 'num-grp': 'int',
> + 'num-pu': 'int',
> + 'num-chk': 'int',
> + 'num-sec': 'int'
> + } }
> +
> ##
> # @ImageInfoSpecific:
> #
> @@ -124,6 +162,7 @@
> 'data': {
> 'qcow2': 'ImageInfoSpecificQCow2',
> 'vmdk': 'ImageInfoSpecificVmdk',
> + 'ocssd': 'ImageInfoSpecificOcssd',
Perhaps missing a doc comment that ocssd is new to 4.1; although it may
be implied by the documentation adding 'occsd' to the list of overall
block types elsewhere....
> # If we need to add block driver specific parameters for
> # LUKS in future, then we'll subclass QCryptoBlockInfoLUKS
> # to define a ImageInfoSpecificLUKS
> @@ -282,7 +321,7 @@
> # @drv: the name of the block format used to open the backing device. As of
> # 0.14.0 this can be: 'blkdebug', 'bochs', 'cloop', 'cow', 'dmg',
> # 'file', 'file', 'ftp', 'ftps', 'host_cdrom', 'host_device',
> -# 'http', 'https', 'luks', 'nbd', 'parallels', 'qcow',
> +# 'http', 'https', 'luks', 'nbd', 'ocssd', 'parallels', 'qcow',
> # 'qcow2', 'raw', 'vdi', 'vmdk', 'vpc', 'vvfat'
> # 2.2: 'archipelago' added, 'cow' dropped
> # 2.3: 'host_floppy' deprecated
> @@ -290,6 +329,7 @@
> # 2.6: 'luks' added
> # 2.8: 'replication' added, 'tftp' dropped
> # 2.9: 'archipelago' dropped
> +# 4.1: 'ocssd' added
...here
> #
> # @backing_file: the name of the backing file (for copy-on-write)
> #
> @@ -2815,8 +2855,8 @@
> 'data': [ 'blkdebug', 'blklogwrites', 'blkverify', 'bochs', 'cloop',
> 'copy-on-read', 'dmg', 'file', 'ftp', 'ftps', 'gluster',
> 'host_cdrom', 'host_device', 'http', 'https', 'iscsi', 'luks',
> - 'nbd', 'nfs', 'null-aio', 'null-co', 'nvme', 'parallels', 'qcow',
> - 'qcow2', 'qed', 'quorum', 'raw', 'rbd',
> + 'nbd', 'nfs', 'null-aio', 'null-co', 'nvme', 'ocssd', 'parallels',
> + 'qcow', 'qcow2', 'qed', 'quorum', 'raw', 'rbd',
> { 'name': 'replication', 'if': 'defined(CONFIG_REPLICATION)' },
> 'sheepdog',
> 'ssh', 'throttle', 'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat', 'vxhs' ] }
> @@ -3917,6 +3957,7 @@
> 'null-aio': 'BlockdevOptionsNull',
> 'null-co': 'BlockdevOptionsNull',
> 'nvme': 'BlockdevOptionsNVMe',
> + 'ocssd': 'BlockdevOptionsGenericFormat',
I guess you don't support qemu creating one of these devices yet?
> 'parallels': 'BlockdevOptionsGenericFormat',
> 'qcow2': 'BlockdevOptionsQcow2',
> 'qcow': 'BlockdevOptionsQcow',
>
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] [PATCH 8/8] nvme: add an OpenChannel 2.0 NVMe device (ocssd)
2019-05-20 16:45 ` Eric Blake
@ 2019-05-20 17:33 ` Klaus Birkelund
0 siblings, 0 replies; 15+ messages in thread
From: Klaus Birkelund @ 2019-05-20 17:33 UTC (permalink / raw)
To: Eric Blake; +Cc: Keith Busch, Kevin Wolf, qemu-devel, qemu-block, Max Reitz
On Mon, May 20, 2019 at 11:45:00AM -0500, Eric Blake wrote:
> On 5/17/19 3:42 AM, Klaus Birkelund Jensen wrote:
> > This adds a new 'ocssd' block device that emulates an OpenChannel 2.0
> > device. The device is backed by a new 'ocssd' block backend that is
> > based on the raw format driver but includes a header that holds the
> > device geometry and write data requirements. This new block backend is
> > special in that the size is not specified explicitly but in terms of
> > sector size, number of chunks, number of parallel units, etc. This
> > called for the addition of the `no_size_required` field in `struct
> > BlockDriver` to not fail image creation when the size parameter is
> > missing.
> >
> > The ocssd device is an individual device but shares a lot of code with
> > the nvme device. Thus, some core functionality of nvme/nvme.c has been
> > exported for use by nvme/ocssd.c.
> >
> > Thank you to the following people for their contributions to the
> > original qemu-nvme (github.com/OpenChannelSSD/qemu-nvme) implementation.
> >
> > Matias Bjørling <mb@lightnvm.io>
> > Javier González <javier@javigon.com>
> > Simon Andreas Frimann Lund <ocssd@safl.dk>
> > Hans Holmberg <hans@owltronix.com>
> > Jesper Devantier <contact@pseudonymous.me>
> > Young Tack Jin <youngtack.jin@circuitblvd.com>
> >
> > Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
> > ---
> > MAINTAINERS | 14 +-
> > Makefile.objs | 1 +
> > block.c | 2 +-
> > block/Makefile.objs | 2 +-
> > block/nvme.c | 2 +-
> > block/ocssd.c | 690 ++++++++
> > hw/block/Makefile.objs | 2 +-
> > hw/block/{ => nvme}/nvme.c | 192 ++-
> > hw/block/nvme/ocssd.c | 2647 +++++++++++++++++++++++++++++++
> > hw/block/nvme/ocssd.h | 140 ++
> > hw/block/nvme/trace-events | 136 ++
> > hw/block/trace-events | 109 --
> > include/block/block_int.h | 3 +
> > include/block/nvme.h | 12 +-
> > include/block/ocssd.h | 231 +++
> > {hw => include/hw}/block/nvme.h | 61 +
> > include/hw/pci/pci_ids.h | 2 +
> > qapi/block-core.json | 47 +-
> > 18 files changed, 4121 insertions(+), 172 deletions(-)
> > create mode 100644 block/ocssd.c
> > rename hw/block/{ => nvme}/nvme.c (94%)
> > create mode 100644 hw/block/nvme/ocssd.c
> > create mode 100644 hw/block/nvme/ocssd.h
> > create mode 100644 hw/block/nvme/trace-events
> > create mode 100644 include/block/ocssd.h
> > rename {hw => include/hw}/block/nvme.h (63%)
>
> Feels big; are you sure this can't be split into smaller pieces to ease
> review?
>
I know, but I'm not sure how to meaningfully split it up. Would you
prefer that I move files in one commit? Changed stuff to nvme.{c,h} is
mostly removing static from functions and creating a prototype in the
header files to allow the ocssd device to use the functions. The commit
should be restricted to just adding the ocssd device. Any features and
additions required in the nvme device are added in previous commits.
> I'm focusing just on the qapi portions:
>
Thank you for the review of that, but it looks like this will all be
dropped from a v2 (see mail from Kevin), because it's simply bad
design to have the driver and device depend so closely on each other.
Thanks,
Klaus
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device
[not found] ` <20190521080115.GA4971@linux.fritz.box>
@ 2019-05-21 20:14 ` Klaus Birkelund
0 siblings, 0 replies; 15+ messages in thread
From: Klaus Birkelund @ 2019-05-21 20:14 UTC (permalink / raw)
To: Kevin Wolf; +Cc: Keith Busch, armbru, qemu-devel, qemu-block, Max Reitz
On Tue, May 21, 2019 at 10:01:15AM +0200, Kevin Wolf wrote:
> Am 20.05.2019 um 21:34 hat Klaus Birkelund geschrieben:
> > On Mon, May 20, 2019 at 03:01:24PM +0200, Kevin Wolf wrote:
> > > Am 17.05.2019 um 10:42 hat Klaus Birkelund Jensen geschrieben:
> > > > Hi,
> > > >
> > > > This series of patches contains a number of refactorings to the emulated
> > > > nvme device, adds additional features, such as support for metadata and
> > > > scatter gather lists, and bumps the supported NVMe version to 1.3.
> > > > Lastly, it contains a new 'ocssd' device.
> > > >
> > > > The motivation for the first seven patches is to set everything up for
> > > > the final patch that adds a new 'ocssd' device and associated block
> > > > driver that implements the OpenChannel 2.0 specification[1]. Many of us
> > > > in the OpenChannel comunity have used a qemu fork[2] for emulation of
> > > > OpenChannel devices. The fork is itself based on Keith's qemu-nvme
> > > > tree[3] and we recently merged mainline qemu into it, but the result is
> > > > still a "hybrid" nvme device that supports both conventional nvme and
> > > > the OCSSD 2.0 spec through a 'dialect' mechanism. Merging instead of
> > > > rebasing also created a pretty messy commit history and my efforts to
> > > > try and rebase our work onto mainline was getting hairy to say the
> > > > least. And I was never really happy with the dialect approach anyway.
> > > >
> > > > I have instead prepared this series of fresh patches that incrementally
> > > > adds additional features to the nvme device to bring it into shape for
> > > > finally introducing a new (and separate) 'ocssd' device that emulates an
> > > > OpenChannel 2.0 device by reusing core functionality from the nvme
> > > > device. Providing a separate ocssd device ensures that no ocssd specific
> > > > stuff creeps into the nvme device.
> > > >
> > > > The ocssd device is backed by a new 'ocssd' block driver that holds
> > > > internal meta data and keeps state permanent across power cycles. In the
> > > > future I think we could use the same approach for the nvme device to
> > > > keep internal metadata such as utilization and deallocated blocks.
> > >
> > > A backend driver that is specific for a guest device model (i.e. the
> > > device model requires this driver, and the backend is useless without
> > > the device) sounds like a very questionable design.
> > >
> > > Metadata like OcssdFormatHeader that is considered part of the image
> > > data, which means that the _actual_ image content without metadata isn't
> > > directly accessible any more feels like a bad idea, too. Simple things
> > > like what a resize operation means (change only the actual disk size as
> > > usual, or is the new size disk + metadata?) become confusing. Attaching
> > > an image to a different device becomes impossible.
> > >
> > > The block format driver doesn't seem to actually add much functionality
> > > to a specially crafted raw image: It provides a convenient way to create
> > > such special images and it dumps some values in 'qemu-img info', but the
> > > actual interpretation of the data is left to the device model.
> > >
> > > Looking at the options it does provide, my impression is that these
> > > should really be qdev properties, and the place to store them
> > > persistently is something like the libvirt XML. The device doesn't
> > > change any of the values, so there is nothing that QEMU actually needs
> > > to store. What you invented is a one-off way to pass a config file to a
> > > device, but only for one specific device type.
> > >
> > > I think this needs to use a much more standard approach to be mergable.
> > >
> > > Markus (CCed) as the maintainer for the configuration mechanisms may
> > > have an opinion on this, too.
> > >
> > > > For now, the nvme device does not support the Deallocated and
> > > > Unwritten Logical Block Error (DULBE) feature or the Data Set
> > > > Management command as this would require such support.
> > >
> > > Doesn't bdrv_co_block_status() provide all the information you need for
> > > that?
> >
> > Is it wrong for a device to store such "internal" metadata on the image?
> > Before implementing the ocssd block driver, the device just used a raw
> > image that it initialized with internal metadata and would error out if
> > the size of the raw image was too small to accomodate the chosen
> > geometry. Is that an acceptable way forward?
>
> No, I think if you read from the image normally, it should produce
> whatever the guest sees. So I can take the same image and attach it to
> IDE, virtio-blk or ocssd without modifying the image file.
>
Hmm, I am not sure how that should work for an OpenChannel device due to
the difference in adressing. An LBA that is valid on an OpenChannel
device may not be valid on a regular NVMe device and vice versa.
The physical layout of the NAND is exposed to the host (ehm, guest in
this case...) in such a way that there is no linear address space. The
address space is hierarchical such that an OCSSD LBA encodes group,
parallel unit, chunk and sector into a 64 bit value. The spec has an
example. Consider this physical geometry:
* 16 groups: 4 bits required
* 4 PUs within each group: 2 bits required
* 1004 chunks within each PU: 10 bits required
* 4096 logical blocks within a chunk: 12 bits required
In other words, there are 263192576 (16 * 4 * 1004 * 4096) logical
blocks available. In a linear world, we'd have an LBA range of
0-263192575, but with the hierarchical adressing, the 64 bits are used
like this:
* Bits 11-0 specify a logical block within a chunk
* Bits 21-12 specify a chunk within a PU
* Bits 23-22 specify a PU within a group
* Bits 27-24 specify a group
* Bits 63-28 are unused
So, the last logical block in the last chunk on the last pu in the last
group has LBA:
(0b1111 << 24) | (0b11 << 22) | (0b1111101011 << 12) | 0b111111111111
[ group ] | [ punit ] | [ chunk ] | [ sector ]
which has the value 268353535 != 263192575. Bottomline: the addresses of
data change depending on whether we are working with an nvme or an ocssd
device.
I assume you mean that if an ide device reads all sectors on the image
starting from 0, it should give the same result as if we did it when
connected to the ocssd device. Even if we assume that I do not write any
additional internal metadata to the image such that sector 0 on the
image is LBA 0, then this is still not possible because the address
space when used with the ocssd device has "holes". All the addresses
describing the non-existing chunks with chunk number 1004-1023 has an
address in the address space, but there is conceptually no NAND backing
it, so when we try to write it we get a write fault. On the other hand,
when we read it, we read predefined data (say, zeroes).
So, say we just include these "holes" by making the disk larger (i.e.
more LBAs). This does not affect the ocssd device because it wont allow
writes to these fillers and when reading them we just read the data that
they are initialized with. We are then able to read the same raw data
from the image when connected to an nvme device. But this would also
mean that the nvme device could write these "filler" LBAs that are
invalid under the geometry that the ocssd device knows about. So it only
works one-way. My v2 actually works like that, e.g. truncating the image
file to the size of the address space.
> If we want to store such data in the image file (I'm not convinced of
> this idea yet, but maybe that's because I don't know the use case),
> adding an block layer API for storing metadata and extending qcow2 to
> support that would be a cleaner design. Or if you don't mind having a
> second file, you could have a qdev option for the device to pass a
> separate config file.
>
Yes - it would be awesome if we could just hide the internal metadata by
having raw/qcow2 support and a block api. We could also just move the
internal metadata to an additional file, but we would still be left with
the issue on linear vs hierarchical adressing that I described above.
Maybe the conclusion here is to recognize that an ocssd device is not a
block device? It just looks like it in most circumstances.
> Anyway, if I understand correctly, this is a convenience function. So
> I'd suggest to split this off, so we can move forward with the actual
> device (assuming it gets sufficient review) while discussing the
> configuration mechanisms.
>
I really don't think the block driver is a good idea anymore - it does
spooky stuff, so I have a v2 ready that no longer contains it. Instead,
the image is initialized on the basis of ocssd device parameters. It
still holds a bunch of internal metadata, which I think is justifiable.
As explained above, we can't make the image work on other devices
anyway.
> I'm also sure that others such as Markus can provide valuable input, so
> let's try to keep the discussion on the list. (Is this intentionally
> off-list? If not, feel free to add the list again in your reply.)
>
Adding in the list again. I was unsure how much back and forth
discussion goes to the list vs off-list, but I'll keep it on-list.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [Qemu-devel] [Qemu-block] [PATCH 5/8] nvme: add support for metadata
2019-05-17 8:42 ` [Qemu-devel] [PATCH 5/8] nvme: add support for metadata Klaus Birkelund Jensen
@ 2019-05-22 6:12 ` Klaus Birkelund
0 siblings, 0 replies; 15+ messages in thread
From: Klaus Birkelund @ 2019-05-22 6:12 UTC (permalink / raw)
To: qemu-block; +Cc: Keith Busch, Kevin Wolf, qemu-devel, Max Reitz
On Fri, May 17, 2019 at 10:42:31AM +0200, Klaus Birkelund Jensen wrote:
> The new `ms` parameter may be used to indicate the number of metadata
> bytes provided per LBA.
>
> Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>
> ---
> hw/block/nvme.c | 31 +++++++++++++++++++++++++++++--
> hw/block/nvme.h | 11 ++++++++++-
> 2 files changed, 39 insertions(+), 3 deletions(-)
>
> diff --git a/hw/block/nvme.c b/hw/block/nvme.c
> index c514f93f3867..675967a596d1 100644
> --- a/hw/block/nvme.c
> +++ b/hw/block/nvme.c
> @@ -33,6 +33,8 @@
> * num_ns=<int> : Namespaces to make out of the backing storage,
> * Default:1
> * num_queues=<int> : Number of possible IO Queues, Default:64
> + * ms=<int> : Number of metadata bytes provided per LBA,
> + * Default:0
> * cmb_size_mb=<int> : Size of CMB in MBs, Default:0
> *
> * Parameters will be verified against conflicting capabilities and attributes
> @@ -386,6 +388,8 @@ static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
>
> uint32_t unit_len = nvme_ns_lbads_bytes(ns);
> uint32_t len = req->nlb * unit_len;
> + uint32_t meta_unit_len = nvme_ns_ms(ns);
> + uint32_t meta_len = req->nlb * meta_unit_len;
> uint64_t prp1 = le64_to_cpu(cmd->prp1);
> uint64_t prp2 = le64_to_cpu(cmd->prp2);
>
> @@ -399,6 +403,19 @@ static uint16_t nvme_blk_map(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req)
> return err;
> }
>
> + qsg.nsg = 0;
> + qsg.size = 0;
> +
> + if (cmd->mptr && n->params.ms) {
> + qemu_sglist_add(&qsg, le64_to_cpu(cmd->mptr), meta_len);
> +
> + err = nvme_blk_setup(n, ns, &qsg, ns->blk_offset_md, meta_unit_len,
> + req);
> + if (err) {
> + return err;
> + }
> + }
> +
> qemu_sglist_destroy(&qsg);
>
> return NVME_SUCCESS;
> @@ -1902,6 +1919,11 @@ static int nvme_check_constraints(NvmeCtrl *n, Error **errp)
> return 1;
> }
>
> + if (params->ms && !is_power_of_2(params->ms)) {
> + error_setg(errp, "nvme: invalid metadata configuration");
> + return 1;
> + }
> +
> return 0;
> }
>
> @@ -2066,17 +2088,20 @@ static void nvme_init_ctrl(NvmeCtrl *n)
>
> static uint64_t nvme_ns_calc_blks(NvmeCtrl *n, NvmeNamespace *ns)
> {
> - return n->ns_size / nvme_ns_lbads_bytes(ns);
> + return n->ns_size / (nvme_ns_lbads_bytes(ns) + nvme_ns_ms(ns));
> }
>
> static void nvme_ns_init_identify(NvmeCtrl *n, NvmeIdNs *id_ns)
> {
> + NvmeParams *params = &n->params;
> +
> id_ns->nlbaf = 0;
> id_ns->flbas = 0;
> - id_ns->mc = 0;
> + id_ns->mc = params->ms ? 0x2 : 0;
> id_ns->dpc = 0;
> id_ns->dps = 0;
> id_ns->lbaf[0].lbads = BDRV_SECTOR_BITS;
> + id_ns->lbaf[0].ms = params->ms;
> }
>
> static int nvme_init_namespace(NvmeCtrl *n, NvmeNamespace *ns, Error **errp)
> @@ -2086,6 +2111,8 @@ static int nvme_init_namespace(NvmeCtrl *n, NvmeNamespace *ns, Error **errp)
> nvme_ns_init_identify(n, id_ns);
>
> ns->ns_blks = nvme_ns_calc_blks(n, ns);
> + ns->blk_offset_md = ns->blk_offset + nvme_ns_lbads_bytes(ns) * ns->ns_blks;
> +
> id_ns->nuse = id_ns->ncap = id_ns->nsze = cpu_to_le64(ns->ns_blks);
>
> return 0;
> diff --git a/hw/block/nvme.h b/hw/block/nvme.h
> index 711ca249eac5..81ee0c5173d5 100644
> --- a/hw/block/nvme.h
> +++ b/hw/block/nvme.h
> @@ -8,13 +8,15 @@
> DEFINE_PROP_UINT32("cmb_size_mb", _state, _props.cmb_size_mb, 0), \
> DEFINE_PROP_UINT32("num_queues", _state, _props.num_queues, 64), \
> DEFINE_PROP_UINT32("num_ns", _state, _props.num_ns, 1), \
> - DEFINE_PROP_UINT8("mdts", _state, _props.mdts, 7)
> + DEFINE_PROP_UINT8("mdts", _state, _props.mdts, 7), \
> + DEFINE_PROP_UINT8("ms", _state, _props.ms, 0)
>
> typedef struct NvmeParams {
> char *serial;
> uint32_t num_queues;
> uint32_t num_ns;
> uint8_t mdts;
> + uint8_t ms;
> uint32_t cmb_size_mb;
> } NvmeParams;
>
> @@ -91,6 +93,7 @@ typedef struct NvmeNamespace {
> uint32_t id;
> uint64_t ns_blks;
> uint64_t blk_offset;
> + uint64_t blk_offset_md;
> } NvmeNamespace;
>
> #define TYPE_NVME "nvme"
> @@ -154,4 +157,10 @@ static inline size_t nvme_ns_lbads_bytes(NvmeNamespace *ns)
> return 1 << nvme_ns_lbads(ns);
> }
>
> +static inline uint16_t nvme_ns_ms(NvmeNamespace *ns)
> +{
> + NvmeIdNs *id = &ns->id_ns;
> + return le16_to_cpu(id->lbaf[NVME_ID_NS_FLBAS_INDEX(id->flbas)].ms);
> +}
> +
> #endif /* HW_NVME_H */
> --
> 2.21.0
>
Hi Kevin,
I feel this patch that has nothing to do with ocssd also touches upon
the core issue we are discussing in the ocssd thread.
As I understand it, QEMU does not support metadata pr LBA in the block
layer. Thus, the approach in this patch is also flawed because it breaks
the assumptions of other devices?
The approach suggested in this patch is to split the image into a large
data part and a trailing part for storing metadata bytes per data
sector.
[ data ][ meta ]
How about I drop this patch and the ocssd patch from this series and
just move forward with the other changes to the nvme device (i.e. v1.3
and sgls) for now and we can continue the discussion on how to get
metadata and ocssd support merged in another thread?
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2019-05-22 6:13 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-17 8:42 [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 1/8] nvme: move device parameters to separate struct Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 2/8] nvme: bump supported spec to 1.3 Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 3/8] nvme: simplify PRP mappings Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 4/8] nvme: allow multiple i/o's per request Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 5/8] nvme: add support for metadata Klaus Birkelund Jensen
2019-05-22 6:12 ` [Qemu-devel] [Qemu-block] " Klaus Birkelund
2019-05-17 8:42 ` [Qemu-devel] [PATCH 6/8] nvme: add support for scatter gather lists Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 7/8] nvme: keep a copy of the NVMe command in request Klaus Birkelund Jensen
2019-05-17 8:42 ` [Qemu-devel] [PATCH 8/8] nvme: add an OpenChannel 2.0 NVMe device (ocssd) Klaus Birkelund Jensen
2019-05-20 16:45 ` Eric Blake
2019-05-20 17:33 ` Klaus Birkelund
2019-05-20 13:01 ` [Qemu-devel] [PATCH 0/8] nvme: v1.3, sgls, metadata and new 'ocssd' device Kevin Wolf
2019-05-20 13:32 ` Klaus Birkelund
[not found] ` <20190520193445.GA22742@apples.localdomain>
[not found] ` <20190521080115.GA4971@linux.fritz.box>
2019-05-21 20:14 ` Klaus Birkelund
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.