[PATCH v2 0/3] Add OCP extended log to nvme QEMU

[PATCH v2 0/3] Add OCP extended log to nvme QEMU

[Joel Granados]

The motivation and description are contained in the last patch in this set.
Will copy paste it here for convenience:

    In order to evaluate write amplification factor (WAF) within the storage
    stack it is important to know the number of bytes written to the
    controller. The existing SMART log value of Data Units Written is too
    coarse (given in units of 500 Kb) and so we add the SMART health
    information extended from the OCP specification (given in units of bytes).

    To accommodate different vendor specific specifications like OCP, we add a
    multiplexing function (nvme_vendor_specific_log) which will route to the
    different log functions based on arguments and log ids. We only return the
    OCP extended smart log when the command is 0xC0 and ocp has been turned on
    in the args.

    Though we add the whole nvme smart log extended structure, we only populate
    the physical_media_units_{read,written}, log_page_version and
    log_page_uuid.

V1 changes:
1. I moved the ocp parameter from the namespace to the subsystem as it is
   defined there in the OCP specification
2. I now accumulate statistics from all namespaces and report them back on
   the extended log as per the spec.
3. I removed the default case in the switch in nvme_vendor_specific_log as
   it does not have any special function.

Joel Granados (3):
  nvme: Move adjustment of data_units{read,written}
  nvme: Add ocp to the subsys
  nvme: Add physical writes/reads from OCP log

 hw/nvme/ctrl.c       | 70 ++++++++++++++++++++++++++++++++++++++++----
 hw/nvme/nvme.h       |  1 +
 hw/nvme/subsys.c     |  4 +--
 include/block/nvme.h | 36 +++++++++++++++++++++++
 4 files changed, 103 insertions(+), 8 deletions(-)

-- 
2.30.2

[PATCH v2 1/3] nvme: Move adjustment of data_units{read,written}

[Joel Granados]

In order to return the units_{read/written} required by the SMART log we
need to shift the number of bytes value by BDRV_SECTORS_BITS and multiply
by 1000. This is a prep patch that moves this adjustment to where the SMART
log is calculated in order to use the stats struct for calculating OCP
extended smart log values.

Signed-off-by: Joel Granados <j.granados@samsung.com>
---
 hw/nvme/ctrl.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c
index 87aeba0564..220683201a 100644
--- a/hw/nvme/ctrl.c
+++ b/hw/nvme/ctrl.c
@@ -4449,8 +4449,8 @@ static void nvme_set_blk_stats(NvmeNamespace *ns, struct nvme_stats *stats)
 {
     BlockAcctStats *s = blk_get_stats(ns->blkconf.blk);
 
-    stats->units_read += s->nr_bytes[BLOCK_ACCT_READ] >> BDRV_SECTOR_BITS;
-    stats->units_written += s->nr_bytes[BLOCK_ACCT_WRITE] >> BDRV_SECTOR_BITS;
+    stats->units_read += s->nr_bytes[BLOCK_ACCT_READ];
+    stats->units_written += s->nr_bytes[BLOCK_ACCT_WRITE];
     stats->read_commands += s->nr_ops[BLOCK_ACCT_READ];
     stats->write_commands += s->nr_ops[BLOCK_ACCT_WRITE];
 }
@@ -4490,10 +4490,12 @@ static uint16_t nvme_smart_info(NvmeCtrl *n, uint8_t rae, uint32_t buf_len,
     trans_len = MIN(sizeof(smart) - off, buf_len);
     smart.critical_warning = n->smart_critical_warning;
 
-    smart.data_units_read[0] = cpu_to_le64(DIV_ROUND_UP(stats.units_read,
-                                                        1000));
-    smart.data_units_written[0] = cpu_to_le64(DIV_ROUND_UP(stats.units_written,
-                                                           1000));
+    smart.data_units_read[0] = cpu_to_le64(DIV_ROUND_UP(
+                                           stats.units_read >> BDRV_SECTOR_BITS,
+                                           1000));
+    smart.data_units_written[0] = cpu_to_le64(DIV_ROUND_UP(
+                                              stats.units_written >> BDRV_SECTOR_BITS,
+                                              1000));
     smart.host_read_commands[0] = cpu_to_le64(stats.read_commands);
     smart.host_write_commands[0] = cpu_to_le64(stats.write_commands);
 
-- 
2.30.2

[PATCH v2 2/3] nvme: Add ocp to the subsys

[Joel Granados]

The Open Compute Project defines a Datacenter NVMe SSD Spec that sits on
top of the NVMe spec. Additional commands and NVMe behaviors specific for
the Datacenter. This is a preparation patch that introduces an argument to
activate OCP in nvme.

Signed-off-by: Joel Granados <j.granados@samsung.com>
---
 hw/nvme/nvme.h   | 1 +
 hw/nvme/subsys.c | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h
index 79f5c281c2..aa99c0c57c 100644
--- a/hw/nvme/nvme.h
+++ b/hw/nvme/nvme.h
@@ -56,6 +56,7 @@ typedef struct NvmeSubsystem {
 
     struct {
         char *nqn;
+        bool ocp;
     } params;
 } NvmeSubsystem;
 
diff --git a/hw/nvme/subsys.c b/hw/nvme/subsys.c
index 9d2643678b..ecca28449c 100644
--- a/hw/nvme/subsys.c
+++ b/hw/nvme/subsys.c
@@ -129,8 +129,8 @@ static void nvme_subsys_realize(DeviceState *dev, Error **errp)
 
 static Property nvme_subsystem_props[] = {
     DEFINE_PROP_STRING("nqn", NvmeSubsystem, params.nqn),
-    DEFINE_PROP_END_OF_LIST(),
-};
+    DEFINE_PROP_BOOL("ocp", NvmeSubsystem, params.ocp, false),
+    DEFINE_PROP_END_OF_LIST(), };
 
 static void nvme_subsys_class_init(ObjectClass *oc, void *data)
 {
-- 
2.30.2

[PATCH v2 3/3] nvme: Add physical writes/reads from OCP log

[Joel Granados]

In order to evaluate write amplification factor (WAF) within the storage
stack it is important to know the number of bytes written to the
controller. The existing SMART log value of Data Units Written is too
coarse (given in units of 500 Kb) and so we add the SMART health
information extended from the OCP specification (given in units of bytes).

To accomodate different vendor specific specifications like OCP, we add a
multiplexing function (nvme_vendor_specific_log) which will route to the
different log functions based on arguments and log ids. We only return the
OCP extended smart log when the command is 0xC0 and ocp has been turned on
in the args.

Though we add the whole nvme smart log extended structure, we only populate
the physical_media_units_{read,written}, log_page_version and
log_page_uuid.

Signed-off-by: Joel Granados <j.granados@samsung.com>

squash with main

Signed-off-by: Joel Granados <j.granados@samsung.com>
---
 hw/nvme/ctrl.c       | 56 ++++++++++++++++++++++++++++++++++++++++++++
 include/block/nvme.h | 36 ++++++++++++++++++++++++++++
 2 files changed, 92 insertions(+)

diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c
index 220683201a..5e6a8150a2 100644
--- a/hw/nvme/ctrl.c
+++ b/hw/nvme/ctrl.c
@@ -4455,6 +4455,42 @@ static void nvme_set_blk_stats(NvmeNamespace *ns, struct nvme_stats *stats)
     stats->write_commands += s->nr_ops[BLOCK_ACCT_WRITE];
 }
 
+static uint16_t nvme_ocp_extended_smart_info(NvmeCtrl *n, uint8_t rae,
+                                             uint32_t buf_len, uint64_t off,
+                                             NvmeRequest *req)
+{
+    NvmeNamespace *ns = NULL;
+    NvmeSmartLogExtended smart_ext = { 0 };
+    struct nvme_stats stats = { 0 };
+    uint32_t trans_len;
+
+    if (off >= sizeof(smart_ext)) {
+        return NVME_INVALID_FIELD | NVME_DNR;
+    }
+
+    // Accumulate all stats from all namespaces
+    for (int i = 1; i <= NVME_MAX_NAMESPACES; i++) {
+        ns = nvme_ns(n, i);
+        if (ns)
+        {
+            nvme_set_blk_stats(ns, &stats);
+        }
+    }
+
+    smart_ext.physical_media_units_written[0] = cpu_to_le32(stats.units_written);
+    smart_ext.physical_media_units_read[0] = cpu_to_le32(stats.units_read);
+    smart_ext.log_page_version = 0x0003;
+    smart_ext.log_page_uuid[0] = 0xA4F2BFEA2810AFC5;
+    smart_ext.log_page_uuid[1] = 0xAFD514C97C6F4F9C;
+
+    if (!rae) {
+        nvme_clear_events(n, NVME_AER_TYPE_SMART);
+    }
+
+    trans_len = MIN(sizeof(smart_ext) - off, buf_len);
+    return nvme_c2h(n, (uint8_t *) &smart_ext + off, trans_len, req);
+}
+
 static uint16_t nvme_smart_info(NvmeCtrl *n, uint8_t rae, uint32_t buf_len,
                                 uint64_t off, NvmeRequest *req)
 {
@@ -4642,6 +4678,24 @@ static uint16_t nvme_cmd_effects(NvmeCtrl *n, uint8_t csi, uint32_t buf_len,
     return nvme_c2h(n, ((uint8_t *)&log) + off, trans_len, req);
 }
 
+static uint16_t nvme_vendor_specific_log(uint8_t lid, NvmeCtrl *n, uint8_t rae,
+                                         uint32_t buf_len, uint64_t off,
+                                         NvmeRequest *req)
+{
+    NvmeSubsystem *subsys = n->subsys;
+    switch (lid) {
+        case NVME_LOG_VENDOR_START:
+            if (subsys->params.ocp) {
+                return nvme_ocp_extended_smart_info(n, rae, buf_len, off, req);
+            }
+            break;
+            /* Add a case for each additional vendor specific log id */
+    }
+
+    trace_pci_nvme_err_invalid_log_page(nvme_cid(req), lid);
+    return NVME_INVALID_FIELD | NVME_DNR;
+}
+
 static uint16_t nvme_get_log(NvmeCtrl *n, NvmeRequest *req)
 {
     NvmeCmd *cmd = &req->cmd;
@@ -4683,6 +4737,8 @@ static uint16_t nvme_get_log(NvmeCtrl *n, NvmeRequest *req)
         return nvme_error_info(n, rae, len, off, req);
     case NVME_LOG_SMART_INFO:
         return nvme_smart_info(n, rae, len, off, req);
+    case NVME_LOG_VENDOR_START...NVME_LOG_VENDOR_END:
+        return nvme_vendor_specific_log(lid, n, rae, len, off, req);
     case NVME_LOG_FW_SLOT_INFO:
         return nvme_fw_log_info(n, len, off, req);
     case NVME_LOG_CHANGED_NSLIST:
diff --git a/include/block/nvme.h b/include/block/nvme.h
index 8027b7126b..2ab0dca529 100644
--- a/include/block/nvme.h
+++ b/include/block/nvme.h
@@ -978,6 +978,40 @@ typedef struct QEMU_PACKED NvmeSmartLog {
     uint8_t     reserved2[320];
 } NvmeSmartLog;
 
+typedef struct QEMU_PACKED NvmeSmartLogExtended {
+    uint64_t    physical_media_units_written[2];
+    uint64_t    physical_media_units_read[2];
+    uint64_t    bad_user_blocks;
+    uint64_t    bad_system_nand_blocks;
+    uint64_t    xor_recovery_count;
+    uint64_t    uncorrectable_read_error_count;
+    uint64_t    soft_ecc_error_count;
+    uint64_t    end2end_correction_counts;
+    uint8_t     system_data_percent_used;
+    uint8_t     refresh_counts[7];
+    uint64_t    user_data_erase_counts;
+    uint16_t    thermal_throttling_stat_and_count;
+    uint16_t    dssd_spec_version[3];
+    uint64_t    pcie_correctable_error_count;
+    uint32_t    incomplete_shutdowns;
+    uint32_t    reserved0;
+    uint8_t     percent_free_blocks;
+    uint8_t     reserved1[7];
+    uint16_t    capacity_health;
+    uint8_t     nvme_errata_ver;
+    uint8_t     reserved2[5];
+    uint64_t    unaligned_io;
+    uint64_t    security_ver_num;
+    uint64_t    total_nuse;
+    uint64_t    plp_start_count[2];
+    uint64_t    endurance_estimate[2];
+    uint64_t    pcie_retraining_count;
+    uint64_t    power_state_change_count;
+    uint8_t     reserved3[286];
+    uint16_t    log_page_version;
+    uint64_t    log_page_uuid[2];
+} NvmeSmartLogExtended;
+
 #define NVME_SMART_WARN_MAX     6
 enum NvmeSmartWarn {
     NVME_SMART_SPARE                  = 1 << 0,
@@ -1010,6 +1044,8 @@ enum NvmeLogIdentifier {
     NVME_LOG_FW_SLOT_INFO   = 0x03,
     NVME_LOG_CHANGED_NSLIST = 0x04,
     NVME_LOG_CMD_EFFECTS    = 0x05,
+    NVME_LOG_VENDOR_START   = 0xC0,
+    NVME_LOG_VENDOR_END     = 0xFF,
 };
 
 typedef struct QEMU_PACKED NvmePSD {
-- 
2.30.2

Re: [PATCH v2 1/3] nvme: Move adjustment of data_units{read,written}

[Klaus Jensen]

[-- Attachment #1: Type: text/plain, Size: 2398 bytes --]

On Nov 14 14:50, Joel Granados wrote:
> In order to return the units_{read/written} required by the SMART log we
> need to shift the number of bytes value by BDRV_SECTORS_BITS and multiply
> by 1000. This is a prep patch that moves this adjustment to where the SMART
> log is calculated in order to use the stats struct for calculating OCP
> extended smart log values.
> 
> Signed-off-by: Joel Granados <j.granados@samsung.com>
> ---
>  hw/nvme/ctrl.c | 14 ++++++++------
>  1 file changed, 8 insertions(+), 6 deletions(-)
> 
> diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c
> index 87aeba0564..220683201a 100644
> --- a/hw/nvme/ctrl.c
> +++ b/hw/nvme/ctrl.c
> @@ -4449,8 +4449,8 @@ static void nvme_set_blk_stats(NvmeNamespace *ns, struct nvme_stats *stats)
>  {
>      BlockAcctStats *s = blk_get_stats(ns->blkconf.blk);
>  
> -    stats->units_read += s->nr_bytes[BLOCK_ACCT_READ] >> BDRV_SECTOR_BITS;
> -    stats->units_written += s->nr_bytes[BLOCK_ACCT_WRITE] >> BDRV_SECTOR_BITS;
> +    stats->units_read += s->nr_bytes[BLOCK_ACCT_READ];
> +    stats->units_written += s->nr_bytes[BLOCK_ACCT_WRITE];
>      stats->read_commands += s->nr_ops[BLOCK_ACCT_READ];
>      stats->write_commands += s->nr_ops[BLOCK_ACCT_WRITE];
>  }
> @@ -4490,10 +4490,12 @@ static uint16_t nvme_smart_info(NvmeCtrl *n, uint8_t rae, uint32_t buf_len,
>      trans_len = MIN(sizeof(smart) - off, buf_len);
>      smart.critical_warning = n->smart_critical_warning;
>  
> -    smart.data_units_read[0] = cpu_to_le64(DIV_ROUND_UP(stats.units_read,
> -                                                        1000));
> -    smart.data_units_written[0] = cpu_to_le64(DIV_ROUND_UP(stats.units_written,
> -                                                           1000));
> +    smart.data_units_read[0] = cpu_to_le64(DIV_ROUND_UP(
> +                                           stats.units_read >> BDRV_SECTOR_BITS,
> +                                           1000));
> +    smart.data_units_written[0] = cpu_to_le64(DIV_ROUND_UP(
> +                                              stats.units_written >> BDRV_SECTOR_BITS,
> +                                              1000));
>      smart.host_read_commands[0] = cpu_to_le64(stats.read_commands);
>      smart.host_write_commands[0] = cpu_to_le64(stats.write_commands);
>  

Reviewed-by: Klaus Jensen <k.jensen@samsung.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 2/3] nvme: Add ocp to the subsys

[Klaus Jensen]

[-- Attachment #1: Type: text/plain, Size: 1640 bytes --]

On Nov 14 14:50, Joel Granados wrote:
> The Open Compute Project defines a Datacenter NVMe SSD Spec that sits on
> top of the NVMe spec. Additional commands and NVMe behaviors specific for
> the Datacenter. This is a preparation patch that introduces an argument to
> activate OCP in nvme.
> 
> Signed-off-by: Joel Granados <j.granados@samsung.com>
> ---
>  hw/nvme/nvme.h   | 1 +
>  hw/nvme/subsys.c | 4 ++--
>  2 files changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h
> index 79f5c281c2..aa99c0c57c 100644
> --- a/hw/nvme/nvme.h
> +++ b/hw/nvme/nvme.h
> @@ -56,6 +56,7 @@ typedef struct NvmeSubsystem {
>  
>      struct {
>          char *nqn;
> +        bool ocp;
>      } params;
>  } NvmeSubsystem;
>  
> diff --git a/hw/nvme/subsys.c b/hw/nvme/subsys.c
> index 9d2643678b..ecca28449c 100644
> --- a/hw/nvme/subsys.c
> +++ b/hw/nvme/subsys.c
> @@ -129,8 +129,8 @@ static void nvme_subsys_realize(DeviceState *dev, Error **errp)
>  
>  static Property nvme_subsystem_props[] = {
>      DEFINE_PROP_STRING("nqn", NvmeSubsystem, params.nqn),
> -    DEFINE_PROP_END_OF_LIST(),
> -};
> +    DEFINE_PROP_BOOL("ocp", NvmeSubsystem, params.ocp, false),

It is the controller that implements the OCP specification, not the
namespace or the subsystem. The parameter should be on the controller
device.

We discussed that the Get Log Page was subsystem scoped and not
namespace scoped, but that is unrelated to this.

> +    DEFINE_PROP_END_OF_LIST(), };
>  
>  static void nvme_subsys_class_init(ObjectClass *oc, void *data)
>  {
> -- 
> 2.30.2
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 3/3] nvme: Add physical writes/reads from OCP log

[Klaus Jensen]

[-- Attachment #1: Type: text/plain, Size: 7496 bytes --]

On Nov 14 14:50, Joel Granados wrote:
> In order to evaluate write amplification factor (WAF) within the storage
> stack it is important to know the number of bytes written to the
> controller. The existing SMART log value of Data Units Written is too
> coarse (given in units of 500 Kb) and so we add the SMART health
> information extended from the OCP specification (given in units of bytes).
> 
> To accomodate different vendor specific specifications like OCP, we add a
> multiplexing function (nvme_vendor_specific_log) which will route to the
> different log functions based on arguments and log ids. We only return the
> OCP extended smart log when the command is 0xC0 and ocp has been turned on
> in the args.
> 
> Though we add the whole nvme smart log extended structure, we only populate
> the physical_media_units_{read,written}, log_page_version and
> log_page_uuid.
> 
> Signed-off-by: Joel Granados <j.granados@samsung.com>
> 
> squash with main
> 
> Signed-off-by: Joel Granados <j.granados@samsung.com>

Looks like you slightly messed up the squash ;)

Also, squash the previous patch (adding the ocp parameter) into this.
Please add a note in the documentation (docs/system/devices/nvme.rst)
about this parameter.

> ---
>  hw/nvme/ctrl.c       | 56 ++++++++++++++++++++++++++++++++++++++++++++
>  include/block/nvme.h | 36 ++++++++++++++++++++++++++++
>  2 files changed, 92 insertions(+)
> 
> diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c
> index 220683201a..5e6a8150a2 100644
> --- a/hw/nvme/ctrl.c
> +++ b/hw/nvme/ctrl.c
> @@ -4455,6 +4455,42 @@ static void nvme_set_blk_stats(NvmeNamespace *ns, struct nvme_stats *stats)
>      stats->write_commands += s->nr_ops[BLOCK_ACCT_WRITE];
>  }
>  
> +static uint16_t nvme_ocp_extended_smart_info(NvmeCtrl *n, uint8_t rae,
> +                                             uint32_t buf_len, uint64_t off,
> +                                             NvmeRequest *req)
> +{
> +    NvmeNamespace *ns = NULL;
> +    NvmeSmartLogExtended smart_ext = { 0 };
> +    struct nvme_stats stats = { 0 };
> +    uint32_t trans_len;
> +
> +    if (off >= sizeof(smart_ext)) {
> +        return NVME_INVALID_FIELD | NVME_DNR;
> +    }
> +
> +    // Accumulate all stats from all namespaces

Use /* lower-case and no period */ for one sentence, one line comments.

I think scripts/checkpatch.pl picks this up.

> +    for (int i = 1; i <= NVME_MAX_NAMESPACES; i++) {
> +        ns = nvme_ns(n, i);
> +        if (ns)
> +        {

Paranthesis go on the same line as the `if`.

> +            nvme_set_blk_stats(ns, &stats);
> +        }
> +    }
> +
> +    smart_ext.physical_media_units_written[0] = cpu_to_le32(stats.units_written);
> +    smart_ext.physical_media_units_read[0] = cpu_to_le32(stats.units_read);
> +    smart_ext.log_page_version = 0x0003;
> +    smart_ext.log_page_uuid[0] = 0xA4F2BFEA2810AFC5;
> +    smart_ext.log_page_uuid[1] = 0xAFD514C97C6F4F9C;
> +
> +    if (!rae) {
> +        nvme_clear_events(n, NVME_AER_TYPE_SMART);
> +    }
> +
> +    trans_len = MIN(sizeof(smart_ext) - off, buf_len);
> +    return nvme_c2h(n, (uint8_t *) &smart_ext + off, trans_len, req);
> +}
> +
>  static uint16_t nvme_smart_info(NvmeCtrl *n, uint8_t rae, uint32_t buf_len,
>                                  uint64_t off, NvmeRequest *req)
>  {
> @@ -4642,6 +4678,24 @@ static uint16_t nvme_cmd_effects(NvmeCtrl *n, uint8_t csi, uint32_t buf_len,
>      return nvme_c2h(n, ((uint8_t *)&log) + off, trans_len, req);
>  }
>  
> +static uint16_t nvme_vendor_specific_log(uint8_t lid, NvmeCtrl *n, uint8_t rae,
> +                                         uint32_t buf_len, uint64_t off,
> +                                         NvmeRequest *req)

`NvmeCtrl *n` must be first parameter.

> +{
> +    NvmeSubsystem *subsys = n->subsys;
> +    switch (lid) {
> +        case NVME_LOG_VENDOR_START:

In this particular case, I think it is more clear if you simply use the
hex value directly. The "meaning" of the log page id depends on if or
not this is an controller implementing the OCP spec.

> +            if (subsys->params.ocp) {
> +                return nvme_ocp_extended_smart_info(n, rae, buf_len, off, req);
> +            }
> +            break;
> +            /* Add a case for each additional vendor specific log id */

Lower-case the comment.

> +    }
> +
> +    trace_pci_nvme_err_invalid_log_page(nvme_cid(req), lid);
> +    return NVME_INVALID_FIELD | NVME_DNR;
> +}
> +
>  static uint16_t nvme_get_log(NvmeCtrl *n, NvmeRequest *req)
>  {
>      NvmeCmd *cmd = &req->cmd;
> @@ -4683,6 +4737,8 @@ static uint16_t nvme_get_log(NvmeCtrl *n, NvmeRequest *req)
>          return nvme_error_info(n, rae, len, off, req);
>      case NVME_LOG_SMART_INFO:
>          return nvme_smart_info(n, rae, len, off, req);
> +    case NVME_LOG_VENDOR_START...NVME_LOG_VENDOR_END:
> +        return nvme_vendor_specific_log(lid, n, rae, len, off, req);
>      case NVME_LOG_FW_SLOT_INFO:
>          return nvme_fw_log_info(n, len, off, req);
>      case NVME_LOG_CHANGED_NSLIST:
> diff --git a/include/block/nvme.h b/include/block/nvme.h
> index 8027b7126b..2ab0dca529 100644
> --- a/include/block/nvme.h
> +++ b/include/block/nvme.h
> @@ -978,6 +978,40 @@ typedef struct QEMU_PACKED NvmeSmartLog {
>      uint8_t     reserved2[320];
>  } NvmeSmartLog;
>  
> +typedef struct QEMU_PACKED NvmeSmartLogExtended {
> +    uint64_t    physical_media_units_written[2];
> +    uint64_t    physical_media_units_read[2];
> +    uint64_t    bad_user_blocks;
> +    uint64_t    bad_system_nand_blocks;
> +    uint64_t    xor_recovery_count;
> +    uint64_t    uncorrectable_read_error_count;
> +    uint64_t    soft_ecc_error_count;
> +    uint64_t    end2end_correction_counts;
> +    uint8_t     system_data_percent_used;
> +    uint8_t     refresh_counts[7];
> +    uint64_t    user_data_erase_counts;
> +    uint16_t    thermal_throttling_stat_and_count;
> +    uint16_t    dssd_spec_version[3];
> +    uint64_t    pcie_correctable_error_count;
> +    uint32_t    incomplete_shutdowns;
> +    uint32_t    reserved0;

I know that it is not totally consistent across the code, but please use
`rsvd<BYTEOFFSET>` for the reserved field names. The above would be
`rsvd116` if I am not mistaken.

> +    uint8_t     percent_free_blocks;
> +    uint8_t     reserved1[7];
> +    uint16_t    capacity_health;
> +    uint8_t     nvme_errata_ver;
> +    uint8_t     reserved2[5];
> +    uint64_t    unaligned_io;
> +    uint64_t    security_ver_num;
> +    uint64_t    total_nuse;
> +    uint64_t    plp_start_count[2];
> +    uint64_t    endurance_estimate[2];
> +    uint64_t    pcie_retraining_count;
> +    uint64_t    power_state_change_count;
> +    uint8_t     reserved3[286];
> +    uint16_t    log_page_version;
> +    uint64_t    log_page_uuid[2];
> +} NvmeSmartLogExtended;
> +
>  #define NVME_SMART_WARN_MAX     6
>  enum NvmeSmartWarn {
>      NVME_SMART_SPARE                  = 1 << 0,
> @@ -1010,6 +1044,8 @@ enum NvmeLogIdentifier {
>      NVME_LOG_FW_SLOT_INFO   = 0x03,
>      NVME_LOG_CHANGED_NSLIST = 0x04,
>      NVME_LOG_CMD_EFFECTS    = 0x05,
> +    NVME_LOG_VENDOR_START   = 0xC0,
> +    NVME_LOG_VENDOR_END     = 0xFF,

Existing style is generally lower-case hex.

>  };
>  
>  typedef struct QEMU_PACKED NvmePSD {
> -- 
> 2.30.2
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 2/3] nvme: Add ocp to the subsys

[Joel Granados]

[-- Attachment #1: Type: text/plain, Size: 1915 bytes --]

On Tue, Nov 15, 2022 at 12:11:50PM +0100, Klaus Jensen wrote:
> On Nov 14 14:50, Joel Granados wrote:
> > The Open Compute Project defines a Datacenter NVMe SSD Spec that sits on
> > top of the NVMe spec. Additional commands and NVMe behaviors specific for
> > the Datacenter. This is a preparation patch that introduces an argument to
> > activate OCP in nvme.
> > 
> > Signed-off-by: Joel Granados <j.granados@samsung.com>
> > ---
> >  hw/nvme/nvme.h   | 1 +
> >  hw/nvme/subsys.c | 4 ++--
> >  2 files changed, 3 insertions(+), 2 deletions(-)
> > 
> > diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h
> > index 79f5c281c2..aa99c0c57c 100644
> > --- a/hw/nvme/nvme.h
> > +++ b/hw/nvme/nvme.h
> > @@ -56,6 +56,7 @@ typedef struct NvmeSubsystem {
> >  
> >      struct {
> >          char *nqn;
> > +        bool ocp;
> >      } params;
> >  } NvmeSubsystem;
> >  
> > diff --git a/hw/nvme/subsys.c b/hw/nvme/subsys.c
> > index 9d2643678b..ecca28449c 100644
> > --- a/hw/nvme/subsys.c
> > +++ b/hw/nvme/subsys.c
> > @@ -129,8 +129,8 @@ static void nvme_subsys_realize(DeviceState *dev, Error **errp)
> >  
> >  static Property nvme_subsystem_props[] = {
> >      DEFINE_PROP_STRING("nqn", NvmeSubsystem, params.nqn),
> > -    DEFINE_PROP_END_OF_LIST(),
> > -};
> > +    DEFINE_PROP_BOOL("ocp", NvmeSubsystem, params.ocp, false),
> 
> It is the controller that implements the OCP specification, not the
> namespace or the subsystem. The parameter should be on the controller
> device.
Makes sense. I'll put the option in hw/nvme/ctrl.c

> 
> We discussed that the Get Log Page was subsystem scoped and not
> namespace scoped, but that is unrelated to this.
Yep, this was the confusion. Thx for clarifying.

> 
> > +    DEFINE_PROP_END_OF_LIST(), };
> >  
> >  static void nvme_subsys_class_init(ObjectClass *oc, void *data)
> >  {
> > -- 
> > 2.30.2
> > 
> > 



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 3/3] nvme: Add physical writes/reads from OCP log

[Joel Granados]

[-- Attachment #1: Type: text/plain, Size: 8545 bytes --]

On Tue, Nov 15, 2022 at 12:26:17PM +0100, Klaus Jensen wrote:
> On Nov 14 14:50, Joel Granados wrote:
> > In order to evaluate write amplification factor (WAF) within the storage
> > stack it is important to know the number of bytes written to the
> > controller. The existing SMART log value of Data Units Written is too
> > coarse (given in units of 500 Kb) and so we add the SMART health
> > information extended from the OCP specification (given in units of bytes).
> > 
> > To accomodate different vendor specific specifications like OCP, we add a
> > multiplexing function (nvme_vendor_specific_log) which will route to the
> > different log functions based on arguments and log ids. We only return the
> > OCP extended smart log when the command is 0xC0 and ocp has been turned on
> > in the args.
> > 
> > Though we add the whole nvme smart log extended structure, we only populate
> > the physical_media_units_{read,written}, log_page_version and
> > log_page_uuid.
> > 
> > Signed-off-by: Joel Granados <j.granados@samsung.com>
> > 
> > squash with main
> > 
> > Signed-off-by: Joel Granados <j.granados@samsung.com>
> 
> Looks like you slightly messed up the squash ;)
oops. that is my bad

> 
> Also, squash the previous patch (adding the ocp parameter) into this.
Here I wanted to keep the introduction of the argument separate. In any
case, I'll squash it with the other one.

> Please add a note in the documentation (docs/system/devices/nvme.rst)
> about this parameter.
Of course. I always forget documentation. I'll add it under the
"Controller Emulation" section and I'll call it ``ocp``

> 
> > ---
> >  hw/nvme/ctrl.c       | 56 ++++++++++++++++++++++++++++++++++++++++++++
> >  include/block/nvme.h | 36 ++++++++++++++++++++++++++++
> >  2 files changed, 92 insertions(+)
> > 
> > diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c
> > index 220683201a..5e6a8150a2 100644
> > --- a/hw/nvme/ctrl.c
> > +++ b/hw/nvme/ctrl.c
> > @@ -4455,6 +4455,42 @@ static void nvme_set_blk_stats(NvmeNamespace *ns, struct nvme_stats *stats)
> >      stats->write_commands += s->nr_ops[BLOCK_ACCT_WRITE];
> >  }
> >  
> > +static uint16_t nvme_ocp_extended_smart_info(NvmeCtrl *n, uint8_t rae,
> > +                                             uint32_t buf_len, uint64_t off,
> > +                                             NvmeRequest *req)
> > +{
> > +    NvmeNamespace *ns = NULL;
> > +    NvmeSmartLogExtended smart_ext = { 0 };
> > +    struct nvme_stats stats = { 0 };
> > +    uint32_t trans_len;
> > +
> > +    if (off >= sizeof(smart_ext)) {
> > +        return NVME_INVALID_FIELD | NVME_DNR;
> > +    }
> > +
> > +    // Accumulate all stats from all namespaces
> 
> Use /* lower-case and no period */ for one sentence, one line comments.
> 
> I think scripts/checkpatch.pl picks this up.
There is a checkpatch like in the kernel. Fantastic! I'll make a note to
use it from now on.


> 
> > +    for (int i = 1; i <= NVME_MAX_NAMESPACES; i++) {
> > +        ns = nvme_ns(n, i);
> > +        if (ns)
> > +        {
> 
> Paranthesis go on the same line as the `if`.
of course

> 
> > +            nvme_set_blk_stats(ns, &stats);
> > +        }
> > +    }
> > +
> > +    smart_ext.physical_media_units_written[0] = cpu_to_le32(stats.units_written);
> > +    smart_ext.physical_media_units_read[0] = cpu_to_le32(stats.units_read);
> > +    smart_ext.log_page_version = 0x0003;
> > +    smart_ext.log_page_uuid[0] = 0xA4F2BFEA2810AFC5;
> > +    smart_ext.log_page_uuid[1] = 0xAFD514C97C6F4F9C;
> > +
> > +    if (!rae) {
> > +        nvme_clear_events(n, NVME_AER_TYPE_SMART);
> > +    }
> > +
> > +    trans_len = MIN(sizeof(smart_ext) - off, buf_len);
> > +    return nvme_c2h(n, (uint8_t *) &smart_ext + off, trans_len, req);
> > +}
> > +
> >  static uint16_t nvme_smart_info(NvmeCtrl *n, uint8_t rae, uint32_t buf_len,
> >                                  uint64_t off, NvmeRequest *req)
> >  {
> > @@ -4642,6 +4678,24 @@ static uint16_t nvme_cmd_effects(NvmeCtrl *n, uint8_t csi, uint32_t buf_len,
> >      return nvme_c2h(n, ((uint8_t *)&log) + off, trans_len, req);
> >  }
> >  
> > +static uint16_t nvme_vendor_specific_log(uint8_t lid, NvmeCtrl *n, uint8_t rae,
> > +                                         uint32_t buf_len, uint64_t off,
> > +                                         NvmeRequest *req)
> 
> `NvmeCtrl *n` must be first parameter.
Any reason why this is the case? I'll change it in my code, but would be
nice to understand the reason.


> 
> > +{
> > +    NvmeSubsystem *subsys = n->subsys;
> > +    switch (lid) {
> > +        case NVME_LOG_VENDOR_START:
> 
> In this particular case, I think it is more clear if you simply use the
> hex value directly. The "meaning" of the log page id depends on if or
> not this is an controller implementing the OCP spec.
Agreed

> 
> > +            if (subsys->params.ocp) {
> > +                return nvme_ocp_extended_smart_info(n, rae, buf_len, off, req);
> > +            }
> > +            break;
> > +            /* Add a case for each additional vendor specific log id */
> 
> Lower-case the comment.
ok

> 
> > +    }
> > +
> > +    trace_pci_nvme_err_invalid_log_page(nvme_cid(req), lid);
> > +    return NVME_INVALID_FIELD | NVME_DNR;
> > +}
> > +
> >  static uint16_t nvme_get_log(NvmeCtrl *n, NvmeRequest *req)
> >  {
> >      NvmeCmd *cmd = &req->cmd;
> > @@ -4683,6 +4737,8 @@ static uint16_t nvme_get_log(NvmeCtrl *n, NvmeRequest *req)
> >          return nvme_error_info(n, rae, len, off, req);
> >      case NVME_LOG_SMART_INFO:
> >          return nvme_smart_info(n, rae, len, off, req);
> > +    case NVME_LOG_VENDOR_START...NVME_LOG_VENDOR_END:
> > +        return nvme_vendor_specific_log(lid, n, rae, len, off, req);
> >      case NVME_LOG_FW_SLOT_INFO:
> >          return nvme_fw_log_info(n, len, off, req);
> >      case NVME_LOG_CHANGED_NSLIST:
> > diff --git a/include/block/nvme.h b/include/block/nvme.h
> > index 8027b7126b..2ab0dca529 100644
> > --- a/include/block/nvme.h
> > +++ b/include/block/nvme.h
> > @@ -978,6 +978,40 @@ typedef struct QEMU_PACKED NvmeSmartLog {
> >      uint8_t     reserved2[320];
> >  } NvmeSmartLog;
> >  
> > +typedef struct QEMU_PACKED NvmeSmartLogExtended {
> > +    uint64_t    physical_media_units_written[2];
> > +    uint64_t    physical_media_units_read[2];
> > +    uint64_t    bad_user_blocks;
> > +    uint64_t    bad_system_nand_blocks;
> > +    uint64_t    xor_recovery_count;
> > +    uint64_t    uncorrectable_read_error_count;
> > +    uint64_t    soft_ecc_error_count;
> > +    uint64_t    end2end_correction_counts;
> > +    uint8_t     system_data_percent_used;
> > +    uint8_t     refresh_counts[7];
> > +    uint64_t    user_data_erase_counts;
> > +    uint16_t    thermal_throttling_stat_and_count;
> > +    uint16_t    dssd_spec_version[3];
> > +    uint64_t    pcie_correctable_error_count;
> > +    uint32_t    incomplete_shutdowns;
> > +    uint32_t    reserved0;
> 
> I know that it is not totally consistent across the code, but please use
> `rsvd<BYTEOFFSET>` for the reserved field names. The above would be
> `rsvd116` if I am not mistaken.
ok

> 
> > +    uint8_t     percent_free_blocks;
> > +    uint8_t     reserved1[7];
> > +    uint16_t    capacity_health;
> > +    uint8_t     nvme_errata_ver;
> > +    uint8_t     reserved2[5];
> > +    uint64_t    unaligned_io;
> > +    uint64_t    security_ver_num;
> > +    uint64_t    total_nuse;
> > +    uint64_t    plp_start_count[2];
> > +    uint64_t    endurance_estimate[2];
> > +    uint64_t    pcie_retraining_count;
> > +    uint64_t    power_state_change_count;
> > +    uint8_t     reserved3[286];
> > +    uint16_t    log_page_version;
> > +    uint64_t    log_page_uuid[2];
> > +} NvmeSmartLogExtended;
> > +
> >  #define NVME_SMART_WARN_MAX     6
> >  enum NvmeSmartWarn {
> >      NVME_SMART_SPARE                  = 1 << 0,
> > @@ -1010,6 +1044,8 @@ enum NvmeLogIdentifier {
> >      NVME_LOG_FW_SLOT_INFO   = 0x03,
> >      NVME_LOG_CHANGED_NSLIST = 0x04,
> >      NVME_LOG_CMD_EFFECTS    = 0x05,
> > +    NVME_LOG_VENDOR_START   = 0xC0,
> > +    NVME_LOG_VENDOR_END     = 0xFF,
> 
> Existing style is generally lower-case hex.
No problem

Thx for the review. Will post V3 after running checkpatch
> 
> >  };
> >  
> >  typedef struct QEMU_PACKED NvmePSD {
> > -- 
> > 2.30.2
> > 
> > 



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 3/3] nvme: Add physical writes/reads from OCP log

[Klaus Jensen]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

On Nov 16 17:19, Joel Granados wrote:
> On Tue, Nov 15, 2022 at 12:26:17PM +0100, Klaus Jensen wrote:
> > On Nov 14 14:50, Joel Granados wrote:
> > >  
> > > +static uint16_t nvme_vendor_specific_log(uint8_t lid, NvmeCtrl *n, uint8_t rae,
> > > +                                         uint32_t buf_len, uint64_t off,
> > > +                                         NvmeRequest *req)
> > 
> > `NvmeCtrl *n` must be first parameter.
> Any reason why this is the case? I'll change it in my code, but would be
> nice to understand the reason.
> 

No other reason than consistency with existing code.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 0/3] Add OCP extended log to nvme QEMU

[Christoph Hellwig]

Please don't do this.  OCP is acting as a counter standard to the
proper NVMe standard here and should in absolutely no way be supported
by open source projects that needs to stick to the actual standards.

Please work with the NVMe technical working group to add this (very
useful) functionality to NVMe proper first.

On Mon, Nov 14, 2022 at 02:50:40PM +0100, Joel Granados wrote:
> The motivation and description are contained in the last patch in this set.
> Will copy paste it here for convenience:
> 
>     In order to evaluate write amplification factor (WAF) within the storage
>     stack it is important to know the number of bytes written to the
>     controller. The existing SMART log value of Data Units Written is too
>     coarse (given in units of 500 Kb) and so we add the SMART health
>     information extended from the OCP specification (given in units of bytes).
> 
>     To accommodate different vendor specific specifications like OCP, we add a
>     multiplexing function (nvme_vendor_specific_log) which will route to the
>     different log functions based on arguments and log ids. We only return the
>     OCP extended smart log when the command is 0xC0 and ocp has been turned on
>     in the args.
> 
>     Though we add the whole nvme smart log extended structure, we only populate
>     the physical_media_units_{read,written}, log_page_version and
>     log_page_uuid.
> 
> V1 changes:
> 1. I moved the ocp parameter from the namespace to the subsystem as it is
>    defined there in the OCP specification
> 2. I now accumulate statistics from all namespaces and report them back on
>    the extended log as per the spec.
> 3. I removed the default case in the switch in nvme_vendor_specific_log as
>    it does not have any special function.
> 
> Joel Granados (3):
>   nvme: Move adjustment of data_units{read,written}
>   nvme: Add ocp to the subsys
>   nvme: Add physical writes/reads from OCP log
> 
>  hw/nvme/ctrl.c       | 70 ++++++++++++++++++++++++++++++++++++++++----
>  hw/nvme/nvme.h       |  1 +
>  hw/nvme/subsys.c     |  4 +--
>  include/block/nvme.h | 36 +++++++++++++++++++++++
>  4 files changed, 103 insertions(+), 8 deletions(-)
> 
> -- 
> 2.30.2
> 
> 
---end quoted text---

Re: [PATCH v2 0/3] Add OCP extended log to nvme QEMU

[Joel Granados]

[-- Attachment #1: Type: text/plain, Size: 2765 bytes --]

On Thu, Dec 22, 2022 at 11:39:54PM -0800, Christoph Hellwig wrote:
> Please don't do this.  OCP is acting as a counter standard to the
> proper NVMe standard here and should in absolutely no way be supported
> by open source projects that needs to stick to the actual standards.
> 
> Please work with the NVMe technical working group to add this (very
> useful) functionality to NVMe proper first.

This is a very good point. Regardless of what OCP's ultimate objective,
having this in the NVMe specification would reach more cases. We can
even use existing values like the "Media Bytes with Metadata Written" in
the statistics log page of the newly ratified FDP TP.

Thx for the review

Best

Joel

> 
> On Mon, Nov 14, 2022 at 02:50:40PM +0100, Joel Granados wrote:
> > The motivation and description are contained in the last patch in this set.
> > Will copy paste it here for convenience:
> > 
> >     In order to evaluate write amplification factor (WAF) within the storage
> >     stack it is important to know the number of bytes written to the
> >     controller. The existing SMART log value of Data Units Written is too
> >     coarse (given in units of 500 Kb) and so we add the SMART health
> >     information extended from the OCP specification (given in units of bytes).
> > 
> >     To accommodate different vendor specific specifications like OCP, we add a
> >     multiplexing function (nvme_vendor_specific_log) which will route to the
> >     different log functions based on arguments and log ids. We only return the
> >     OCP extended smart log when the command is 0xC0 and ocp has been turned on
> >     in the args.
> > 
> >     Though we add the whole nvme smart log extended structure, we only populate
> >     the physical_media_units_{read,written}, log_page_version and
> >     log_page_uuid.
> > 
> > V1 changes:
> > 1. I moved the ocp parameter from the namespace to the subsystem as it is
> >    defined there in the OCP specification
> > 2. I now accumulate statistics from all namespaces and report them back on
> >    the extended log as per the spec.
> > 3. I removed the default case in the switch in nvme_vendor_specific_log as
> >    it does not have any special function.
> > 
> > Joel Granados (3):
> >   nvme: Move adjustment of data_units{read,written}
> >   nvme: Add ocp to the subsys
> >   nvme: Add physical writes/reads from OCP log
> > 
> >  hw/nvme/ctrl.c       | 70 ++++++++++++++++++++++++++++++++++++++++----
> >  hw/nvme/nvme.h       |  1 +
> >  hw/nvme/subsys.c     |  4 +--
> >  include/block/nvme.h | 36 +++++++++++++++++++++++
> >  4 files changed, 103 insertions(+), 8 deletions(-)
> > 
> > -- 
> > 2.30.2
> > 
> > 
> ---end quoted text---

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]