* [PATCH 1/5] lsblk: add -N/--nvme
2022-06-03 12:23 [PATCH 0/5] Support --nvme/--virtio and MQ zhenwei pi
@ 2022-06-03 12:23 ` zhenwei pi
2022-06-03 12:23 ` [PATCH 2/5] lsblk: support virtio block zhenwei pi
` (4 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: zhenwei pi @ 2022-06-03 12:23 UTC (permalink / raw)
To: kzak; +Cc: util-linux, zhenwei pi
Add -N/--nvme to filter NVMe device only, NVMe usually has a larger
I/O depth, also show COL_RQ_SIZE by default.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
bash-completion/lsblk | 1 +
misc-utils/lsblk.8.adoc | 3 +++
misc-utils/lsblk.c | 24 +++++++++++++++++++++++-
misc-utils/lsblk.h | 1 +
4 files changed, 28 insertions(+), 1 deletion(-)
diff --git a/bash-completion/lsblk b/bash-completion/lsblk
index ca0ad39d7..011af41c2 100644
--- a/bash-completion/lsblk
+++ b/bash-completion/lsblk
@@ -79,6 +79,7 @@ _lsblk_module()
--inverse
--topology
--scsi
+ --nvme
--sort
--width
--help
diff --git a/misc-utils/lsblk.8.adoc b/misc-utils/lsblk.8.adoc
index 8ffc2cd79..8c6d50f79 100644
--- a/misc-utils/lsblk.8.adoc
+++ b/misc-utils/lsblk.8.adoc
@@ -72,6 +72,9 @@ Group parents of sub-trees to provide more readable output for RAIDs and Multi-p
*-m*, *--perms*::
Output info about device owner, group and mode. This option is equivalent to *-o NAME,SIZE,OWNER,GROUP,MODE*.
+*-N*, *--nvme*::
+Output info about NVMe devices only.
+
*-n*, *--noheadings*::
Do not print a header line.
diff --git a/misc-utils/lsblk.c b/misc-utils/lsblk.c
index fbaa1797f..047f744be 100644
--- a/misc-utils/lsblk.c
+++ b/misc-utils/lsblk.c
@@ -1349,6 +1349,16 @@ static int initialize_device(struct lsblk_device *dev,
return -1;
}
+ /* ignore non-NVMe devices */
+ if (lsblk->nvme) {
+ char *transport = get_transport(dev);
+
+ if (!transport || strcmp(transport, "nvme")) {
+ DBG(DEV, ul_debugobj(dev, "non-nvme device -- ignore"));
+ return -1;
+ }
+ }
+
DBG(DEV, ul_debugobj(dev, "%s: context successfully initialized", dev->name));
return 0;
}
@@ -1925,6 +1935,7 @@ static void __attribute__((__noreturn__)) usage(void)
fputs(_(" -O, --output-all output all columns\n"), out);
fputs(_(" -P, --pairs use key=\"value\" output format\n"), out);
fputs(_(" -S, --scsi output info about SCSI devices\n"), out);
+ fputs(_(" -N, --nvme output info about NVMe devices\n"), out);
fputs(_(" -T, --tree[=<column>] use tree format output\n"), out);
fputs(_(" -a, --all print all devices\n"), out);
fputs(_(" -b, --bytes print SIZE in bytes rather than in human readable format\n"), out);
@@ -2010,6 +2021,7 @@ int main(int argc, char *argv[])
{ "paths", no_argument, NULL, 'p' },
{ "pairs", no_argument, NULL, 'P' },
{ "scsi", no_argument, NULL, 'S' },
+ { "nvme", no_argument, NULL, 'N' },
{ "sort", required_argument, NULL, 'x' },
{ "sysroot", required_argument, NULL, OPT_SYSROOT },
{ "shell", no_argument, NULL, 'y' },
@@ -2043,7 +2055,7 @@ int main(int argc, char *argv[])
lsblk_init_debug();
while((c = getopt_long(argc, argv,
- "AabdDzE:e:fhJlnMmo:OpPiI:rstVST::w:x:y",
+ "AabdDzE:e:fhJlNnMmo:OpPiI:rstVST::w:x:y",
longopts, NULL)) != -1) {
err_exclusive_options(c, longopts, excl, excl_st);
@@ -2165,6 +2177,16 @@ int main(int argc, char *argv[])
add_uniq_column(COL_SERIAL);
add_uniq_column(COL_TRANSPORT);
break;
+ case 'N':
+ lsblk->nodeps = 1;
+ lsblk->nvme = 1;
+ add_uniq_column(COL_NAME);
+ add_uniq_column(COL_TYPE);
+ add_uniq_column(COL_MODEL);
+ add_uniq_column(COL_SERIAL);
+ add_uniq_column(COL_TRANSPORT);
+ add_uniq_column(COL_RQ_SIZE);
+ break;
case 'T':
force_tree = 1;
if (optarg) {
diff --git a/misc-utils/lsblk.h b/misc-utils/lsblk.h
index b20aa6be9..536120a9c 100644
--- a/misc-utils/lsblk.h
+++ b/misc-utils/lsblk.h
@@ -51,6 +51,7 @@ struct lsblk {
unsigned int merge:1; /* merge sub-trees */
unsigned int nodeps:1; /* don't print slaves/holders */
unsigned int scsi:1; /* print only device with HCTL (SCSI) */
+ unsigned int nvme:1; /* print NVMe device only */
unsigned int paths:1; /* print devnames with "/dev" prefix */
unsigned int sort_hidden:1; /* sort column not between output columns */
unsigned int dedup_hidden :1; /* deduplication column not between output columns */
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 2/5] lsblk: support virtio block
2022-06-03 12:23 [PATCH 0/5] Support --nvme/--virtio and MQ zhenwei pi
2022-06-03 12:23 ` [PATCH 1/5] lsblk: add -N/--nvme zhenwei pi
@ 2022-06-03 12:23 ` zhenwei pi
2022-06-03 12:24 ` [PATCH 3/5] lsblk: add -v/--virtio zhenwei pi
` (3 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: zhenwei pi @ 2022-06-03 12:23 UTC (permalink / raw)
To: kzak; +Cc: util-linux, zhenwei pi
virtio blk is quite popular in the virtual machines, support 'TRAN'
column for a virtio block device.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
misc-utils/lsblk.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/misc-utils/lsblk.c b/misc-utils/lsblk.c
index 047f744be..dfa317970 100644
--- a/misc-utils/lsblk.c
+++ b/misc-utils/lsblk.c
@@ -514,8 +514,10 @@ static char *get_transport(struct lsblk_device *dev)
trans = "ata";
free(attr);
- } else if (strncmp(dev->name, "nvme", 4) == 0)
+ } else if (strncmp(dev->name, "nvme", 4) == 0) {
trans = "nvme";
+ } else if (strncmp(dev->name, "vd", 2) == 0)
+ trans = "virtio";
return trans ? xstrdup(trans) : NULL;
}
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 3/5] lsblk: add -v/--virtio
2022-06-03 12:23 [PATCH 0/5] Support --nvme/--virtio and MQ zhenwei pi
2022-06-03 12:23 ` [PATCH 1/5] lsblk: add -N/--nvme zhenwei pi
2022-06-03 12:23 ` [PATCH 2/5] lsblk: support virtio block zhenwei pi
@ 2022-06-03 12:24 ` zhenwei pi
2022-06-03 12:24 ` [PATCH 4/5] lsblk: introduce 'MQ' column zhenwei pi
` (2 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: zhenwei pi @ 2022-06-03 12:24 UTC (permalink / raw)
To: kzak; +Cc: util-linux, zhenwei pi
Add -v/--virtio to filter the virtio block devices.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
bash-completion/lsblk | 1 +
misc-utils/lsblk.8.adoc | 3 +++
misc-utils/lsblk.c | 23 ++++++++++++++++++++++-
misc-utils/lsblk.h | 1 +
4 files changed, 27 insertions(+), 1 deletion(-)
diff --git a/bash-completion/lsblk b/bash-completion/lsblk
index 011af41c2..697dd23d2 100644
--- a/bash-completion/lsblk
+++ b/bash-completion/lsblk
@@ -80,6 +80,7 @@ _lsblk_module()
--topology
--scsi
--nvme
+ --virtio
--sort
--width
--help
diff --git a/misc-utils/lsblk.8.adoc b/misc-utils/lsblk.8.adoc
index 8c6d50f79..6a9770ef2 100644
--- a/misc-utils/lsblk.8.adoc
+++ b/misc-utils/lsblk.8.adoc
@@ -75,6 +75,9 @@ Output info about device owner, group and mode. This option is equivalent to *-o
*-N*, *--nvme*::
Output info about NVMe devices only.
+*-v*, *--virtio*::
+Output info about virtio devices only.
+
*-n*, *--noheadings*::
Do not print a header line.
diff --git a/misc-utils/lsblk.c b/misc-utils/lsblk.c
index dfa317970..64d7edad6 100644
--- a/misc-utils/lsblk.c
+++ b/misc-utils/lsblk.c
@@ -1361,6 +1361,16 @@ static int initialize_device(struct lsblk_device *dev,
}
}
+ /* ignore non-virtio devices */
+ if (lsblk->virtio) {
+ char *transport = get_transport(dev);
+
+ if (!transport || strcmp(transport, "virtio")) {
+ DBG(DEV, ul_debugobj(dev, "non-virtio device -- ignore"));
+ return -1;
+ }
+ }
+
DBG(DEV, ul_debugobj(dev, "%s: context successfully initialized", dev->name));
return 0;
}
@@ -1938,6 +1948,7 @@ static void __attribute__((__noreturn__)) usage(void)
fputs(_(" -P, --pairs use key=\"value\" output format\n"), out);
fputs(_(" -S, --scsi output info about SCSI devices\n"), out);
fputs(_(" -N, --nvme output info about NVMe devices\n"), out);
+ fputs(_(" -v, --virtio output info about virtio devices\n"), out);
fputs(_(" -T, --tree[=<column>] use tree format output\n"), out);
fputs(_(" -a, --all print all devices\n"), out);
fputs(_(" -b, --bytes print SIZE in bytes rather than in human readable format\n"), out);
@@ -2024,6 +2035,7 @@ int main(int argc, char *argv[])
{ "pairs", no_argument, NULL, 'P' },
{ "scsi", no_argument, NULL, 'S' },
{ "nvme", no_argument, NULL, 'N' },
+ { "virtio", no_argument, NULL, 'v' },
{ "sort", required_argument, NULL, 'x' },
{ "sysroot", required_argument, NULL, OPT_SYSROOT },
{ "shell", no_argument, NULL, 'y' },
@@ -2057,7 +2069,7 @@ int main(int argc, char *argv[])
lsblk_init_debug();
while((c = getopt_long(argc, argv,
- "AabdDzE:e:fhJlNnMmo:OpPiI:rstVST::w:x:y",
+ "AabdDzE:e:fhJlNnMmo:OpPiI:rstVvST::w:x:y",
longopts, NULL)) != -1) {
err_exclusive_options(c, longopts, excl, excl_st);
@@ -2189,6 +2201,15 @@ int main(int argc, char *argv[])
add_uniq_column(COL_TRANSPORT);
add_uniq_column(COL_RQ_SIZE);
break;
+ case 'v':
+ lsblk->nodeps = 1;
+ lsblk->virtio = 1;
+ add_uniq_column(COL_NAME);
+ add_uniq_column(COL_TYPE);
+ add_uniq_column(COL_TRANSPORT);
+ add_uniq_column(COL_SIZE);
+ add_uniq_column(COL_RQ_SIZE);
+ break;
case 'T':
force_tree = 1;
if (optarg) {
diff --git a/misc-utils/lsblk.h b/misc-utils/lsblk.h
index 536120a9c..31c9ecad5 100644
--- a/misc-utils/lsblk.h
+++ b/misc-utils/lsblk.h
@@ -52,6 +52,7 @@ struct lsblk {
unsigned int nodeps:1; /* don't print slaves/holders */
unsigned int scsi:1; /* print only device with HCTL (SCSI) */
unsigned int nvme:1; /* print NVMe device only */
+ unsigned int virtio:1; /* print virtio device only */
unsigned int paths:1; /* print devnames with "/dev" prefix */
unsigned int sort_hidden:1; /* sort column not between output columns */
unsigned int dedup_hidden :1; /* deduplication column not between output columns */
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH 4/5] lsblk: introduce 'MQ' column
2022-06-03 12:23 [PATCH 0/5] Support --nvme/--virtio and MQ zhenwei pi
` (2 preceding siblings ...)
2022-06-03 12:24 ` [PATCH 3/5] lsblk: add -v/--virtio zhenwei pi
@ 2022-06-03 12:24 ` zhenwei pi
2022-06-06 10:41 ` Karel Zak
2022-06-03 12:24 ` [PATCH 5/5] lsblk: enable 'MQ' for NVMe/virtio by default zhenwei pi
2022-06-06 10:37 ` [PATCH 0/5] Support --nvme/--virtio and MQ Karel Zak
5 siblings, 1 reply; 9+ messages in thread
From: zhenwei pi @ 2022-06-03 12:24 UTC (permalink / raw)
To: kzak; +Cc: util-linux, zhenwei pi
Typically a modern block device supports mutil queues feature, count
queues by walking '$sysfs/mq' directory. If no '$sysfs/mq' exists, it
is a legacy single queue.
~# lsblk --nvme -o NAME,TYPE,MODEL,TRAN,RQ-SIZE,MQ
NAME TYPE MODEL TRAN RQ-SIZE MQ
nvme0n1 disk INTEL SSDPF2KX038TZ nvme 1023 135
nvme3n1 disk INTEL SSDPE2KX020T8 nvme 1023 128
nvme1n1 disk SAMSUNG MZQL23T8HCLS-00A07 nvme 1023 129
nvme2n2 disk RP2A03T8RK004LX nvme 1023 64
nvme2n3 disk RP2A03T8RK004LX nvme 1023 64
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
bash-completion/lsblk | 2 +-
misc-utils/lsblk.c | 33 +++++++++++++++++++++++++++++++++
2 files changed, 34 insertions(+), 1 deletion(-)
diff --git a/bash-completion/lsblk b/bash-completion/lsblk
index 697dd23d2..6756764b2 100644
--- a/bash-completion/lsblk
+++ b/bash-completion/lsblk
@@ -11,7 +11,7 @@ _lsblk_module()
RO RM HOTPLUG MODEL SERIAL SIZE STATE OWNER GROUP MODE ALIGNMENT MIN-IO OPT-IO
PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE TYPE DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
WSAME WWN RAND PKNAME HCTL TRAN SUBSYSTEMS REV VENDOR ZONED ZONE-SZ ZONE-WGRAN
- ZONE-APP ZONE-NR ZONE-OMAX ZONE-AMAX DAX
+ ZONE-APP ZONE-NR ZONE-OMAX ZONE-AMAX DAX MQ
"
case $prev in
diff --git a/misc-utils/lsblk.c b/misc-utils/lsblk.c
index 64d7edad6..d9c4ee982 100644
--- a/misc-utils/lsblk.c
+++ b/misc-utils/lsblk.c
@@ -90,6 +90,7 @@ enum {
COL_MINIO,
COL_MODE,
COL_MODEL,
+ COL_MQ,
COL_NAME,
COL_OPTIO,
COL_OWNER,
@@ -189,6 +190,7 @@ static struct colinfo infos[] = {
[COL_MINIO] = { "MIN-IO", 6, SCOLS_FL_RIGHT, N_("minimum I/O size"), COLTYPE_NUM },
[COL_MODEL] = { "MODEL", 0.1, SCOLS_FL_TRUNC, N_("device identifier") },
[COL_MODE] = { "MODE", 10, 0, N_("device node permissions") },
+ [COL_MQ] = { "MQ", 3, SCOLS_FL_RIGHT, N_("device queues") },
[COL_NAME] = { "NAME", 0.25, SCOLS_FL_NOEXTREMES, N_("device name") },
[COL_OPTIO] = { "OPT-IO", 6, SCOLS_FL_RIGHT, N_("optimal I/O size"), COLTYPE_NUM },
[COL_OWNER] = { "OWNER", 0.1, SCOLS_FL_TRUNC, N_("user name"), },
@@ -746,6 +748,34 @@ static void device_read_bytes(struct lsblk_device *dev, char *path, char **str,
}
}
+static void process_mq(struct lsblk_device *dev, char **str)
+{
+ DIR *dir;
+ struct dirent *d;
+ unsigned int queues = 0;
+
+ DBG(DEV, ul_debugobj(dev, "%s: process mq", dev->name));
+
+ dir = ul_path_opendir(dev->sysfs, "mq");
+ if (!dir) {
+ *str = xstrdup("1");
+ DBG(DEV, ul_debugobj(dev, "%s: no mq supported, use a single queue", dev->name));
+ return;
+ }
+
+ while ((d = xreaddir(dir))) {
+ if (!strcmp(d->d_name, ".") || !strcmp(d->d_name, ".."))
+ continue;
+
+ queues++;
+ }
+
+ closedir(dir);
+
+ DBG(DEV, ul_debugobj(dev, "%s: has %d queues", dev->name, queues));
+ xasprintf(str, "%3u", queues);
+}
+
/*
* Generates data (string) for column specified by column ID for specified device. If sortdata
* is not NULL then returns number usable to sort the column if the data are available for the
@@ -1143,6 +1173,9 @@ static char *device_get_data(
case COL_DAX:
ul_path_read_string(dev->sysfs, &str, "queue/dax");
break;
+ case COL_MQ:
+ process_mq(dev, &str);
+ break;
};
return str;
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH 4/5] lsblk: introduce 'MQ' column
2022-06-03 12:24 ` [PATCH 4/5] lsblk: introduce 'MQ' column zhenwei pi
@ 2022-06-06 10:41 ` Karel Zak
2022-06-06 10:45 ` zhenwei pi
0 siblings, 1 reply; 9+ messages in thread
From: Karel Zak @ 2022-06-06 10:41 UTC (permalink / raw)
To: zhenwei pi; +Cc: util-linux
On Fri, Jun 03, 2022 at 08:24:01PM +0800, zhenwei pi wrote:
> +static void process_mq(struct lsblk_device *dev, char **str)
> +{
> + DIR *dir;
> + struct dirent *d;
> + unsigned int queues = 0;
> +
> + DBG(DEV, ul_debugobj(dev, "%s: process mq", dev->name));
> +
> + dir = ul_path_opendir(dev->sysfs, "mq");
> + if (!dir) {
> + *str = xstrdup("1");
> + DBG(DEV, ul_debugobj(dev, "%s: no mq supported, use a single queue", dev->name));
> + return;
> + }
> +
> + while ((d = xreaddir(dir))) {
> + if (!strcmp(d->d_name, ".") || !strcmp(d->d_name, ".."))
> + continue;
xreaddir() filters "." and "..".
Anyway, there is a function ul_path_count_dirents() that does all the
work. I made the change in the code.
Karel
--
Karel Zak <kzak@redhat.com>
http://karelzak.blogspot.com
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Re: [PATCH 4/5] lsblk: introduce 'MQ' column
2022-06-06 10:41 ` Karel Zak
@ 2022-06-06 10:45 ` zhenwei pi
0 siblings, 0 replies; 9+ messages in thread
From: zhenwei pi @ 2022-06-06 10:45 UTC (permalink / raw)
To: Karel Zak; +Cc: util-linux
On 6/6/22 18:41, Karel Zak wrote:
> On Fri, Jun 03, 2022 at 08:24:01PM +0800, zhenwei pi wrote:
>> +static void process_mq(struct lsblk_device *dev, char **str)
>> +{
>> + DIR *dir;
>> + struct dirent *d;
>> + unsigned int queues = 0;
>> +
>> + DBG(DEV, ul_debugobj(dev, "%s: process mq", dev->name));
>> +
>> + dir = ul_path_opendir(dev->sysfs, "mq");
>> + if (!dir) {
>> + *str = xstrdup("1");
>> + DBG(DEV, ul_debugobj(dev, "%s: no mq supported, use a single queue", dev->name));
>> + return;
>> + }
>> +
>> + while ((d = xreaddir(dir))) {
>> + if (!strcmp(d->d_name, ".") || !strcmp(d->d_name, ".."))
>> + continue;
>
> xreaddir() filters "." and "..".
>
> Anyway, there is a function ul_path_count_dirents() that does all the
> work. I made the change in the code.
>
> Karel
>
Hi, Karel
Thanks!
--
zhenwei pi
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 5/5] lsblk: enable 'MQ' for NVMe/virtio by default
2022-06-03 12:23 [PATCH 0/5] Support --nvme/--virtio and MQ zhenwei pi
` (3 preceding siblings ...)
2022-06-03 12:24 ` [PATCH 4/5] lsblk: introduce 'MQ' column zhenwei pi
@ 2022-06-03 12:24 ` zhenwei pi
2022-06-06 10:37 ` [PATCH 0/5] Support --nvme/--virtio and MQ Karel Zak
5 siblings, 0 replies; 9+ messages in thread
From: zhenwei pi @ 2022-06-03 12:24 UTC (permalink / raw)
To: kzak; +Cc: util-linux, zhenwei pi
NVMe/virtio devices typically has MQ feature, enable this column by
default for option '--nvme/--virtio'.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
misc-utils/lsblk.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/misc-utils/lsblk.c b/misc-utils/lsblk.c
index d9c4ee982..a09ff2a1f 100644
--- a/misc-utils/lsblk.c
+++ b/misc-utils/lsblk.c
@@ -2233,6 +2233,7 @@ int main(int argc, char *argv[])
add_uniq_column(COL_SERIAL);
add_uniq_column(COL_TRANSPORT);
add_uniq_column(COL_RQ_SIZE);
+ add_uniq_column(COL_MQ);
break;
case 'v':
lsblk->nodeps = 1;
@@ -2242,6 +2243,7 @@ int main(int argc, char *argv[])
add_uniq_column(COL_TRANSPORT);
add_uniq_column(COL_SIZE);
add_uniq_column(COL_RQ_SIZE);
+ add_uniq_column(COL_MQ);
break;
case 'T':
force_tree = 1;
--
2.20.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH 0/5] Support --nvme/--virtio and MQ
2022-06-03 12:23 [PATCH 0/5] Support --nvme/--virtio and MQ zhenwei pi
` (4 preceding siblings ...)
2022-06-03 12:24 ` [PATCH 5/5] lsblk: enable 'MQ' for NVMe/virtio by default zhenwei pi
@ 2022-06-06 10:37 ` Karel Zak
5 siblings, 0 replies; 9+ messages in thread
From: Karel Zak @ 2022-06-06 10:37 UTC (permalink / raw)
To: zhenwei pi; +Cc: util-linux
On Fri, Jun 03, 2022 at 08:23:57PM +0800, zhenwei pi wrote:
> zhenwei pi (5):
> lsblk: add -N/--nvme
> lsblk: support virtio block
> lsblk: add -v/--virtio
> lsblk: introduce 'MQ' column
> lsblk: enable 'MQ' for NVMe/virtio by default
>
> bash-completion/lsblk | 4 +-
> misc-utils/lsblk.8.adoc | 6 +++
> misc-utils/lsblk.c | 84 ++++++++++++++++++++++++++++++++++++++++-
> misc-utils/lsblk.h | 2 +
> 4 files changed, 93 insertions(+), 3 deletions(-)
Applied, thanks!
Karel
--
Karel Zak <kzak@redhat.com>
http://karelzak.blogspot.com
^ permalink raw reply [flat|nested] 9+ messages in thread