[NDCTL PATCH v2 1/2] cxl: Save the number of decoders committed to a port

[NDCTL PATCH v2 1/2] cxl: Save the number of decoders committed to a port

[Dave Jiang]

Save the number of decoders committed to a port exposed by the kernel to the
libcxl cxl_port context. The attribute is helpful for determing if a region is
active.  Add libcxl API to retrieve the number of decoders committed.
Add the decoders_committed attribute to the port for cxl list command.

Link: https://lore.kernel.org/linux-cxl/169645700414.623072.3893376765415910289.stgit@djiang5-mobl3/T/#t
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v2:
- Rebase against latest pending branch
---
 Documentation/cxl/lib/libcxl.txt |    1 +
 cxl/json.c                       |    4 ++++
 cxl/lib/libcxl.c                 |    9 +++++++++
 cxl/lib/libcxl.sym               |    1 +
 cxl/lib/private.h                |    1 +
 cxl/libcxl.h                     |    1 +
 6 files changed, 17 insertions(+)

diff --git a/Documentation/cxl/lib/libcxl.txt b/Documentation/cxl/lib/libcxl.txt
index bcb89288afaf..93c32b54f19e 100644
--- a/Documentation/cxl/lib/libcxl.txt
+++ b/Documentation/cxl/lib/libcxl.txt
@@ -295,6 +295,7 @@ bool cxl_port_is_endpoint(struct cxl_port *port);
 int cxl_port_get_depth(struct cxl_port *port);
 bool cxl_port_hosts_memdev(struct cxl_port *port, struct cxl_memdev *memdev);
 int cxl_port_get_nr_dports(struct cxl_port *port);
+struct cxl_port *cxl_port_decoders_committed(struct cxl_port *port);
 ----
 The port type is communicated via cxl_port_is_<type>(). An 'enabled' port
 is one that has succeeded in discovering the CXL component registers in
diff --git a/cxl/json.c b/cxl/json.c
index 6fb17582a1cb..477e35a5157a 100644
--- a/cxl/json.c
+++ b/cxl/json.c
@@ -1331,6 +1331,10 @@ static struct json_object *__util_cxl_port_to_json(struct cxl_port *port,
 			json_object_object_add(jport, "state", jobj);
 	}
 
+	jobj = json_object_new_int(cxl_port_decoders_committed(port));
+	if (jobj)
+		json_object_object_add(jport, "decoders_committed", jobj);
+
 	json_object_set_userdata(jport, port, NULL);
 	return jport;
 }
diff --git a/cxl/lib/libcxl.c b/cxl/lib/libcxl.c
index bdec2959508b..c1a127f66a1d 100644
--- a/cxl/lib/libcxl.c
+++ b/cxl/lib/libcxl.c
@@ -1880,6 +1880,10 @@ static int cxl_port_init(struct cxl_port *port, struct cxl_port *parent_port,
 	if (sysfs_read_attr(ctx, path, buf) == 0)
 		port->module = util_modalias_to_module(ctx, buf);
 
+	sprintf(path, "%s/decoders_committed", cxlport_base);
+	if (sysfs_read_attr(ctx, path, buf) == 0)
+		port->decoders_committed = strtoul(buf, NULL, 0);
+
 	free(path);
 	return 0;
 err:
@@ -3121,6 +3125,11 @@ cxl_port_get_dport_by_memdev(struct cxl_port *port, struct cxl_memdev *memdev)
 	return NULL;
 }
 
+CXL_EXPORT int cxl_port_decoders_committed(struct cxl_port *port)
+{
+	return port->decoders_committed;
+}
+
 static void *add_cxl_bus(void *parent, int id, const char *cxlbus_base)
 {
 	const char *devname = devpath_to_devname(cxlbus_base);
diff --git a/cxl/lib/libcxl.sym b/cxl/lib/libcxl.sym
index 42f523fda16d..16eca09b3d8b 100644
--- a/cxl/lib/libcxl.sym
+++ b/cxl/lib/libcxl.sym
@@ -277,4 +277,5 @@ global:
 	cxl_cmd_new_set_alert_config;
 	cxl_memdev_trigger_poison_list;
 	cxl_region_trigger_poison_list;
+	cxl_port_decoders_committed;
 } LIBCXL_6;
diff --git a/cxl/lib/private.h b/cxl/lib/private.h
index b26a8629e047..30c898940dec 100644
--- a/cxl/lib/private.h
+++ b/cxl/lib/private.h
@@ -87,6 +87,7 @@ struct cxl_port {
 	int dports_init;
 	int nr_dports;
 	int depth;
+	int decoders_committed;
 	struct cxl_ctx *ctx;
 	struct cxl_bus *bus;
 	enum cxl_port_type type;
diff --git a/cxl/libcxl.h b/cxl/libcxl.h
index 1154f4ce34d1..f7db400f574a 100644
--- a/cxl/libcxl.h
+++ b/cxl/libcxl.h
@@ -145,6 +145,7 @@ bool cxl_port_hosts_memdev(struct cxl_port *port, struct cxl_memdev *memdev);
 int cxl_port_get_nr_dports(struct cxl_port *port);
 int cxl_port_disable_invalidate(struct cxl_port *port);
 int cxl_port_enable(struct cxl_port *port);
+int cxl_port_decoders_committed(struct cxl_port *port);
 struct cxl_port *cxl_port_get_next_all(struct cxl_port *port,
 				       const struct cxl_port *top);
 

[NDCTL PATCH v2 2/2] cxl: Add check for regions before disabling memdev

[Dave Jiang]

Add a check for memdev disable to see if there are active regions present
before disabling the device. This is necessary now regions are present to
fulfill the TODO that was left there. The best way to determine if a
region is active is to see if there are decoders enabled for the mem
device. This is also best effort as the state is only a snapshot the
kernel provides and is not atomic WRT the memdev disable operation. The
expectation is the admin issuing the command has full control of the mem
device and there are no other agents also attempt to control the device.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v2:
- Warn if active region regardless of -f. (Alison)
- Expound on -f behavior in man page. (Vishal)
---
 Documentation/cxl/cxl-disable-memdev.txt |    4 +++-
 cxl/memdev.c                             |   17 ++++++++++++++---
 2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/Documentation/cxl/cxl-disable-memdev.txt b/Documentation/cxl/cxl-disable-memdev.txt
index c4edb93ee94a..34b720288705 100644
--- a/Documentation/cxl/cxl-disable-memdev.txt
+++ b/Documentation/cxl/cxl-disable-memdev.txt
@@ -27,7 +27,9 @@ include::bus-option.txt[]
 	a device if the tool determines the memdev is in active usage. Recall
 	that CXL memory ranges might have been established by platform
 	firmware and disabling an active device is akin to force removing
-	memory from a running system.
+	memory from a running system. Going down this path does not offline
+	active memory if they are currently online. User is recommended to
+	offline and disable the appropriate regions before disabling the memdevs.
 
 -v::
 	Turn on verbose debug messages in the library (if libcxl was built with
diff --git a/cxl/memdev.c b/cxl/memdev.c
index 2dd2e7fcc4dd..1d3121915284 100644
--- a/cxl/memdev.c
+++ b/cxl/memdev.c
@@ -437,14 +437,25 @@ static int action_free_dpa(struct cxl_memdev *memdev,
 
 static int action_disable(struct cxl_memdev *memdev, struct action_context *actx)
 {
+	struct cxl_endpoint *ep;
+	struct cxl_port *port;
+
 	if (!cxl_memdev_is_enabled(memdev))
 		return 0;
 
-	if (!param.force) {
-		/* TODO: actually detect rather than assume active */
+	ep = cxl_memdev_get_endpoint(memdev);
+	if (!ep)
+		return -ENODEV;
+
+	port = cxl_endpoint_get_port(ep);
+	if (!port)
+		return -ENODEV;
+
+	if (cxl_port_decoders_committed(port)) {
 		log_err(&ml, "%s is part of an active region\n",
 			cxl_memdev_get_devname(memdev));
-		return -EBUSY;
+		if (!param.force)
+			return -EBUSY;
 	}
 
 	return cxl_memdev_disable_invalidate(memdev);

Re: [NDCTL PATCH v2 2/2] cxl: Add check for regions before disabling memdev

[Cao, Quanquan/曹 全全]

> Add a check for memdev disable to see if there are active regions present
> before disabling the device. This is necessary now regions are present to
> fulfill the TODO that was left there. The best way to determine if a
> region is active is to see if there are decoders enabled for the mem
> device. This is also best effort as the state is only a snapshot the
> kernel provides and is not atomic WRT the memdev disable operation. The
> expectation is the admin issuing the command has full control of the mem
> device and there are no other agents also attempt to control the device.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
> v2:
> - Warn if active region regardless of -f. (Alison)
> - Expound on -f behavior in man page. (Vishal)
> ---
>   Documentation/cxl/cxl-disable-memdev.txt |    4 +++-
>   cxl/memdev.c                             |   17 ++++++++++++++---
>   2 files changed, 17 insertions(+), 4 deletions(-)
> 
> diff --git a/Documentation/cxl/cxl-disable-memdev.txt b/Documentation/cxl/cxl-disable-memdev.txt
> index c4edb93ee94a..34b720288705 100644
> --- a/Documentation/cxl/cxl-disable-memdev.txt
> +++ b/Documentation/cxl/cxl-disable-memdev.txt
> @@ -27,7 +27,9 @@ include::bus-option.txt[]
>   	a device if the tool determines the memdev is in active usage. Recall
>   	that CXL memory ranges might have been established by platform
>   	firmware and disabling an active device is akin to force removing
> -	memory from a running system.
> +	memory from a running system. Going down this path does not offline
> +	active memory if they are currently online. User is recommended to
> +	offline and disable the appropriate regions before disabling the memdevs.
>   
>   -v::
>   	Turn on verbose debug messages in the library (if libcxl was built with
> diff --git a/cxl/memdev.c b/cxl/memdev.c
> index 2dd2e7fcc4dd..1d3121915284 100644
> --- a/cxl/memdev.c
> +++ b/cxl/memdev.c
> @@ -437,14 +437,25 @@ static int action_free_dpa(struct cxl_memdev *memdev,
>   
>   static int action_disable(struct cxl_memdev *memdev, struct action_context *actx)
>   {
> +	struct cxl_endpoint *ep;
> +	struct cxl_port *port;
> +
>   	if (!cxl_memdev_is_enabled(memdev))
>   		return 0;
>   
> -	if (!param.force) {
> -		/* TODO: actually detect rather than assume active */
> +	ep = cxl_memdev_get_endpoint(memdev);
> +	if (!ep)
> +		return -ENODEV;
> +
> +	port = cxl_endpoint_get_port(ep);
> +	if (!port)
> +		return -ENODEV;
> +
> +	if (cxl_port_decoders_committed(port)) {
>   		log_err(&ml, "%s is part of an active region\n",
>   			cxl_memdev_get_devname(memdev));
> -		return -EBUSY;
> +		if (!param.force)
> +			return -EBUSY;
>   	}
>   
>   	return cxl_memdev_disable_invalidate(memdev);
> 
> 

Reviewed-by: Quanquan Cao <caoqq@fujitsu.com>

Re: [NDCTL PATCH v2 2/2] cxl: Add check for regions before disabling memdev

[Cao, Quanquan/曹 全全]

>   static int action_disable(struct cxl_memdev *memdev, struct action_context *actx)
>   {
> +	struct cxl_endpoint *ep;
> +	struct cxl_port *port;
> +
>   	if (!cxl_memdev_is_enabled(memdev))
>   		return 0;
>   
> -	if (!param.force) {
> -		/* TODO: actually detect rather than assume active */
> +	ep = cxl_memdev_get_endpoint(memdev);
> +	if (!ep)
> +		return -ENODEV;
> +
> +	port = cxl_endpoint_get_port(ep);
> +	if (!port)
> +		return -ENODEV;
> +
> +	if (cxl_port_decoders_committed(port)) {
>   		log_err(&ml, "%s is part of an active region\n",
>   			cxl_memdev_get_devname(memdev));
> -		return -EBUSY;
> +		if (!param.force)
> +			return -EBUSY;
>   	}
>   
>   	return cxl_memdev_disable_invalidate(memdev);
> 
> 
Hi Dave,
Do you think adding one more prompt message would be more user-friendly?

code:
         if (cxl_port_decoders_committed(port)) {
                 log_err(&ml, "%s is part of an active region\n",
                         cxl_memdev_get_devname(memdev));
                 if (!param.force)
                         return -EBUSY;
                 else
                         log_err(&ml,"Forcing memdev disable with an 
active region\n");
         }

output:
[root@fedora-37-client ndctl]# cxl disable-memdev mem0 -f
cxl memdev: action_disable: mem0 is part of an active region
cxl memdev: action_disable: Forcing memdev disable with an active region
cxl memdev: cmd_disable_memdev: disabled 1 mem

Re: [NDCTL PATCH v2 2/2] cxl: Add check for regions before disabling memdev

[Dave Jiang]

On 11/30/23 01:29, Cao, Quanquan/曹 全全 wrote:
> 
>>   static int action_disable(struct cxl_memdev *memdev, struct action_context *actx)
>>   {
>> +    struct cxl_endpoint *ep;
>> +    struct cxl_port *port;
>> +
>>       if (!cxl_memdev_is_enabled(memdev))
>>           return 0;
>>   -    if (!param.force) {
>> -        /* TODO: actually detect rather than assume active */
>> +    ep = cxl_memdev_get_endpoint(memdev);
>> +    if (!ep)
>> +        return -ENODEV;
>> +
>> +    port = cxl_endpoint_get_port(ep);
>> +    if (!port)
>> +        return -ENODEV;
>> +
>> +    if (cxl_port_decoders_committed(port)) {
>>           log_err(&ml, "%s is part of an active region\n",
>>               cxl_memdev_get_devname(memdev));
>> -        return -EBUSY;
>> +        if (!param.force)
>> +            return -EBUSY;
>>       }
>>         return cxl_memdev_disable_invalidate(memdev);
>>
>>
> Hi Dave,
> Do you think adding one more prompt message would be more user-friendly?

Yes good idea. I'll add.

> 
> code:
>         if (cxl_port_decoders_committed(port)) {
>                 log_err(&ml, "%s is part of an active region\n",
>                         cxl_memdev_get_devname(memdev));
>                 if (!param.force)
>                         return -EBUSY;
>                 else
>                         log_err(&ml,"Forcing memdev disable with an active region\n");
>         }
> 
> output:
> [root@fedora-37-client ndctl]# cxl disable-memdev mem0 -f
> cxl memdev: action_disable: mem0 is part of an active region
> cxl memdev: action_disable: Forcing memdev disable with an active region
> cxl memdev: cmd_disable_memdev: disabled 1 mem
> 

[NDCTL PATCH v3 2/2] cxl: Add check for regions before disabling memdev

[Dave Jiang]

Add a check for memdev disable to see if there are active regions present
before disabling the device. This is necessary now regions are present to
fulfill the TODO that was left there. The best way to determine if a
region is active is to see if there are decoders enabled for the mem
device. This is also best effort as the state is only a snapshot the
kernel provides and is not atomic WRT the memdev disable operation. The
expectation is the admin issuing the command has full control of the mem
device and there are no other agents also attempt to control the device.

Reviewed-by: Quanquan Cao <caoqq@fujitsu.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
v3:
- Add emission of warning for forcing operation. (Quanquan)
v2:
- Warn if active region regardless of -f. (Alison)
- Expound on -f behavior in man page. (Vishal)
---
 Documentation/cxl/cxl-disable-memdev.txt |    4 +++-
 cxl/memdev.c                             |   20 +++++++++++++++++---
 2 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/Documentation/cxl/cxl-disable-memdev.txt b/Documentation/cxl/cxl-disable-memdev.txt
index c4edb93ee94a..34b720288705 100644
--- a/Documentation/cxl/cxl-disable-memdev.txt
+++ b/Documentation/cxl/cxl-disable-memdev.txt
@@ -27,7 +27,9 @@ include::bus-option.txt[]
 	a device if the tool determines the memdev is in active usage. Recall
 	that CXL memory ranges might have been established by platform
 	firmware and disabling an active device is akin to force removing
-	memory from a running system.
+	memory from a running system. Going down this path does not offline
+	active memory if they are currently online. User is recommended to
+	offline and disable the appropriate regions before disabling the memdevs.
 
 -v::
 	Turn on verbose debug messages in the library (if libcxl was built with
diff --git a/cxl/memdev.c b/cxl/memdev.c
index 2dd2e7fcc4dd..ed962d478048 100644
--- a/cxl/memdev.c
+++ b/cxl/memdev.c
@@ -437,14 +437,28 @@ static int action_free_dpa(struct cxl_memdev *memdev,
 
 static int action_disable(struct cxl_memdev *memdev, struct action_context *actx)
 {
+	struct cxl_endpoint *ep;
+	struct cxl_port *port;
+
 	if (!cxl_memdev_is_enabled(memdev))
 		return 0;
 
-	if (!param.force) {
-		/* TODO: actually detect rather than assume active */
+	ep = cxl_memdev_get_endpoint(memdev);
+	if (!ep)
+		return -ENODEV;
+
+	port = cxl_endpoint_get_port(ep);
+	if (!port)
+		return -ENODEV;
+
+	if (cxl_port_decoders_committed(port)) {
 		log_err(&ml, "%s is part of an active region\n",
 			cxl_memdev_get_devname(memdev));
-		return -EBUSY;
+		if (!param.force)
+			return -EBUSY;
+
+		log_err(&ml, "Forcing %s disable with an active region!\n",
+			cxl_memdev_get_devname(memdev));
 	}
 
 	return cxl_memdev_disable_invalidate(memdev);

(no subject)

[Yao Xingtao]

Hi Dave,
  I have applied this patch in my env, and done a lot of testing, this
feature is currently working fine. 
  But it is not merged into master branch yet, are there any updates
on this feature?

Associated patches:
https://lore.kernel.org/linux-cxl/170112921107.2687457.2741231995154639197.stgit@djiang5-mobl3/
https://lore.kernel.org/linux-cxl/170120423159.2725915.14670830315829916850.stgit@djiang5-mobl3/

Thanks
Xingtao

Re:

[Verma, Vishal L]

On Wed, 2024-04-17 at 02:46 -0400, Yao Xingtao wrote:
> 
> Hi Dave,
>   I have applied this patch in my env, and done a lot of testing,
> this
> feature is currently working fine. 
>   But it is not merged into master branch yet, are there any updates
> on this feature?

Hi Xingtao,

Turns out that I had applied this to a branch but forgot to merge and
push it. Thanks for the ping - done now, and pushed to pending.

> 
> Associated patches:
> https://lore.kernel.org/linux-cxl/170112921107.2687457.2741231995154639197.stgit@djiang5-mobl3/
> https://lore.kernel.org/linux-cxl/170120423159.2725915.14670830315829916850.stgit@djiang5-mobl3/
> 
> Thanks
> Xingtao

RE: Re:

[Xingtao Yao (Fujitsu)]

> -----Original Message-----
> From: Verma, Vishal L <vishal.l.verma@intel.com>
> Sent: Thursday, April 18, 2024 2:14 AM
> To: Jiang, Dave <dave.jiang@intel.com>; Yao, Xingtao/姚 幸涛
> <yaoxt.fnst@fujitsu.com>
> Cc: Cao, Quanquan/曹 全全 <caoqq@fujitsu.com>; linux-cxl@vger.kernel.org;
> nvdimm@lists.linux.dev
> Subject: Re:
> 
> On Wed, 2024-04-17 at 02:46 -0400, Yao Xingtao wrote:
> >
> > Hi Dave,
> >   I have applied this patch in my env, and done a lot of testing,
> > this
> > feature is currently working fine.
> >   But it is not merged into master branch yet, are there any updates
> > on this feature?
> 
> Hi Xingtao,
> 
> Turns out that I had applied this to a branch but forgot to merge and
> push it. Thanks for the ping - done now, and pushed to pending.
Awesome, many thanks!!!

> 
> >
> > Associated patches:
> >
> https://lore.kernel.org/linux-cxl/170112921107.2687457.2741231995154639197.st
> git@djiang5-mobl3/
> >
> https://lore.kernel.org/linux-cxl/170120423159.2725915.14670830315829916850.s
> tgit@djiang5-mobl3/
> >
> > Thanks
> > Xingtao

Re: [NDCTL PATCH v3 2/2] cxl: Add check for regions before disabling memdev

[Zhijian Li (Fujitsu)]

Hi Verma,


On 18/04/2024 02:14, Verma, Vishal L wrote:
> On Wed, 2024-04-17 at 02:46 -0400, Yao Xingtao wrote:
>>
>> Hi Dave,
>>    I have applied this patch in my env, and done a lot of testing,
>> this
>> feature is currently working fine.
>>    But it is not merged into master branch yet, are there any updates
>> on this feature?
> 
> Hi Xingtao,
> 
> Turns out that I had applied this to a branch but forgot to merge and
> push it. Thanks for the ping - done now, and pushed to pending.


May I know when the next version of NDCTL will be released. It seems like
it's been a very long time since the last release.



Thanks
Zhijian




> 
>>
>> Associated patches:
>> https://lore.kernel.org/linux-cxl/170112921107.2687457.2741231995154639197.stgit@djiang5-mobl3/
>> https://lore.kernel.org/linux-cxl/170120423159.2725915.14670830315829916850.stgit@djiang5-mobl3/
>>
>> Thanks
>> Xingtao
> 

Re: [NDCTL PATCH v3 2/2] cxl: Add check for regions before disabling memdev

[Alison Schofield]

On Thu, Apr 25, 2024 at 05:30:40AM +0000, Zhijian Li (Fujitsu) wrote:
> Hi Verma,
> 
> 
> On 18/04/2024 02:14, Verma, Vishal L wrote:
> > On Wed, 2024-04-17 at 02:46 -0400, Yao Xingtao wrote:
> >>
> >> Hi Dave,
> >>    I have applied this patch in my env, and done a lot of testing,
> >> this
> >> feature is currently working fine.
> >>    But it is not merged into master branch yet, are there any updates
> >> on this feature?
> > 
> > Hi Xingtao,
> > 
> > Turns out that I had applied this to a branch but forgot to merge and
> > push it. Thanks for the ping - done now, and pushed to pending.
> 
> 
> May I know when the next version of NDCTL will be released. It seems like
> it's been a very long time since the last release.

Hi Zhijian,

We appreciate you working with the pending branch while waiting
for the next release. We are aiming to get the poison list feature
merged in the next release, and are getting closer. That's all I
can offer, no date prediction.

-- Alison

> 
> 
> Thanks
> Zhijian
> 
> 
> 
> 
> > 
> >>
> >> Associated patches:
> >> https://lore.kernel.org/linux-cxl/170112921107.2687457.2741231995154639197.stgit@djiang5-mobl3/
> >> https://lore.kernel.org/linux-cxl/170120423159.2725915.14670830315829916850.stgit@djiang5-mobl3/
> >>
> >> Thanks
> >> Xingtao
> > 

Re: [NDCTL PATCH v3 2/2] cxl: Add check for regions before disabling memdev

[Zhijian Li (Fujitsu)]

On 26/04/2024 00:52, Alison Schofield wrote:
> On Thu, Apr 25, 2024 at 05:30:40AM +0000, Zhijian Li (Fujitsu) wrote:
>> Hi Verma,
>>
>>
>> On 18/04/2024 02:14, Verma, Vishal L wrote:
>>> On Wed, 2024-04-17 at 02:46 -0400, Yao Xingtao wrote:
>>>>
>>>> Hi Dave,
>>>>     I have applied this patch in my env, and done a lot of testing,
>>>> this
>>>> feature is currently working fine.
>>>>     But it is not merged into master branch yet, are there any updates
>>>> on this feature?
>>>
>>> Hi Xingtao,
>>>
>>> Turns out that I had applied this to a branch but forgot to merge and
>>> push it. Thanks for the ping - done now, and pushed to pending.
>>
>>
>> May I know when the next version of NDCTL will be released. It seems like
>> it's been a very long time since the last release.
> 
> Hi Zhijian,
> 
> We appreciate you working with the pending branch while waiting
> for the next release. We are aiming to get the poison list feature
> merged in the next release, and are getting closer. That's all I
> can offer, no date prediction.

Understood, thanks for your information.


Thanks
Zhijian

> 
> -- Alison
> 
>>
>>
>> Thanks
>> Zhijian
>>
>>
>>
>>
>>>
>>>>
>>>> Associated patches:
>>>> https://lore.kernel.org/linux-cxl/170112921107.2687457.2741231995154639197.stgit@djiang5-mobl3/
>>>> https://lore.kernel.org/linux-cxl/170120423159.2725915.14670830315829916850.stgit@djiang5-mobl3/
>>>>
>>>> Thanks
>>>> Xingtao
>>>

RE:

[SGV INVESTMENT]

Did you receive our business proposal email ?
_______________________________________________
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-leave@lists.01.org

Re:

[USPS Parcels Delivery]

Hello,

Your item has arrived at the USPS Post Office at  Wed, 17 May 2017 00:46:58
+0200, but the courier was unable to deliver parcel to you. 
You can download the shipment label attached!

Yours respectfully.
Mellicent Northan -  USPS Operation Manager.

Re:

[USPS Station Management]

Hello,

Your item has arrived at the USPS Post Office at  Sat, 29 Apr 2017 15:53:09
-0700, but the courier was unable to deliver parcel to you. 
You can find more details in this e-mail attachment!

With thanks and appreciation.
Fermina Khan -  USPS Support Agent.

Re:

[USPS Ground Support]

Hello,

Your item has arrived at the USPS Post Office at  Fri, 28 Apr 2017 11:27:27
-0700, but the courier was unable to deliver parcel to you. 
You can download the shipment label attached!

With gratitude.
Lashan Simmering -  USPS Support Manager.

Re:

[USPS Priority Delivery]

Hello,

We can not deliver your parcel arrived at  Tue, 11 Apr 2017 15:37:40 +0100.

Please click on the link for more details.
http://uspswoiugue62677104.ideliverys.com/iq5866671

With anticipation.
Sophie Wadkins -  USPS Chief Office Manager.

Re:

[USPS Delivery]

Hello,
Your item has arrived at Sat, 01 Apr 2017 06:31:34 +0100, but our courier
was not able to deliver the parcel. 
Review the document that is attached to this e-mail!

Most sincerely.
Gertruda Hendry -  USPS Mail Delivery Agent.