[PATCH] libnvdimm, dimm: Maximize label transfer size

[PATCH] libnvdimm, dimm: Maximize label transfer size

[Dan Williams]

Use kvzalloc() to bypass the arbitrary PAGE_SIZE limit of label transfer
operations. Given the expense of calling into firmware, maximize the
amount of label data we transfer per call to be up to the total label
space if allowed by the firmware.

Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
 drivers/nvdimm/dimm_devs.c |   12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/nvdimm/dimm_devs.c b/drivers/nvdimm/dimm_devs.c
index 863cabc35215..82ab96a9f560 100644
--- a/drivers/nvdimm/dimm_devs.c
+++ b/drivers/nvdimm/dimm_devs.c
@@ -111,8 +111,8 @@ int nvdimm_init_config_data(struct nvdimm_drvdata *ndd)
 	if (!ndd->data)
 		return -ENOMEM;
 
-	max_cmd_size = min_t(u32, PAGE_SIZE, ndd->nsarea.max_xfer);
-	cmd = kzalloc(max_cmd_size + sizeof(*cmd), GFP_KERNEL);
+	max_cmd_size = min_t(u32, ndd->nsarea.config_size, ndd->nsarea.max_xfer);
+	cmd = kvzalloc(max_cmd_size + sizeof(*cmd), GFP_KERNEL);
 	if (!cmd)
 		return -ENOMEM;
 
@@ -134,7 +134,7 @@ int nvdimm_init_config_data(struct nvdimm_drvdata *ndd)
 		memcpy(ndd->data + offset, cmd->out_buf, cmd->in_length);
 	}
 	dev_dbg(ndd->dev, "len: %zu rc: %d\n", offset, rc);
-	kfree(cmd);
+	kvfree(cmd);
 
 	return rc;
 }
@@ -157,9 +157,9 @@ int nvdimm_set_config_data(struct nvdimm_drvdata *ndd, size_t offset,
 	if (offset + len > ndd->nsarea.config_size)
 		return -ENXIO;
 
-	max_cmd_size = min_t(u32, PAGE_SIZE, len);
+	max_cmd_size = min_t(u32, ndd->nsarea.config_size, len);
 	max_cmd_size = min_t(u32, max_cmd_size, ndd->nsarea.max_xfer);
-	cmd = kzalloc(max_cmd_size + sizeof(*cmd) + sizeof(u32), GFP_KERNEL);
+	cmd = kvzalloc(max_cmd_size + sizeof(*cmd) + sizeof(u32), GFP_KERNEL);
 	if (!cmd)
 		return -ENOMEM;
 
@@ -183,7 +183,7 @@ int nvdimm_set_config_data(struct nvdimm_drvdata *ndd, size_t offset,
 			break;
 		}
 	}
-	kfree(cmd);
+	kvfree(cmd);
 
 	return rc;
 }

_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH] libnvdimm, dimm: Maximize label transfer size

[Alexander Duyck]

On 9/28/2018 11:56 AM, Dan Williams wrote:
> Use kvzalloc() to bypass the arbitrary PAGE_SIZE limit of label transfer
> operations. Given the expense of calling into firmware, maximize the
> amount of label data we transfer per call to be up to the total label
> space if allowed by the firmware.
> 
> Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>

This looks mostly fine to me.

The only thing I am not certain about is if we want to use the raw 
max_cmd_size value or not since there is no guarantee it is aligned to 
any sort of boundary. It might be beneficial to either cache line align 
it or align it to 256B if it is larger than that since that way we at 
least would be getting full labels instead of partial ones.

> ---
>   drivers/nvdimm/dimm_devs.c |   12 ++++++------
>   1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/nvdimm/dimm_devs.c b/drivers/nvdimm/dimm_devs.c
> index 863cabc35215..82ab96a9f560 100644
> --- a/drivers/nvdimm/dimm_devs.c
> +++ b/drivers/nvdimm/dimm_devs.c
> @@ -111,8 +111,8 @@ int nvdimm_init_config_data(struct nvdimm_drvdata *ndd)
>   	if (!ndd->data)
>   		return -ENOMEM;
>   
> -	max_cmd_size = min_t(u32, PAGE_SIZE, ndd->nsarea.max_xfer);
> -	cmd = kzalloc(max_cmd_size + sizeof(*cmd), GFP_KERNEL);
> +	max_cmd_size = min_t(u32, ndd->nsarea.config_size, ndd->nsarea.max_xfer);
> +	cmd = kvzalloc(max_cmd_size + sizeof(*cmd), GFP_KERNEL);
>   	if (!cmd)
>   		return -ENOMEM;
>   
> @@ -134,7 +134,7 @@ int nvdimm_init_config_data(struct nvdimm_drvdata *ndd)
>   		memcpy(ndd->data + offset, cmd->out_buf, cmd->in_length);
>   	}
>   	dev_dbg(ndd->dev, "len: %zu rc: %d\n", offset, rc);
> -	kfree(cmd);
> +	kvfree(cmd);
>   
>   	return rc;
>   }
> @@ -157,9 +157,9 @@ int nvdimm_set_config_data(struct nvdimm_drvdata *ndd, size_t offset,
>   	if (offset + len > ndd->nsarea.config_size)
>   		return -ENXIO;
>   
> -	max_cmd_size = min_t(u32, PAGE_SIZE, len);
> +	max_cmd_size = min_t(u32, ndd->nsarea.config_size, len);
>   	max_cmd_size = min_t(u32, max_cmd_size, ndd->nsarea.max_xfer);
> -	cmd = kzalloc(max_cmd_size + sizeof(*cmd) + sizeof(u32), GFP_KERNEL);
> +	cmd = kvzalloc(max_cmd_size + sizeof(*cmd) + sizeof(u32), GFP_KERNEL);
>   	if (!cmd)
>   		return -ENOMEM;
>   
> @@ -183,7 +183,7 @@ int nvdimm_set_config_data(struct nvdimm_drvdata *ndd, size_t offset,
>   			break;
>   		}
>   	}
> -	kfree(cmd);
> +	kvfree(cmd);
>   
>   	return rc;
>   }
> 
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH] libnvdimm, dimm: Maximize label transfer size

[Dan Williams]

On Fri, Sep 28, 2018 at 3:44 PM Alexander Duyck
<alexander.h.duyck@linux.intel.com> wrote:
>
> On 9/28/2018 11:56 AM, Dan Williams wrote:
> > Use kvzalloc() to bypass the arbitrary PAGE_SIZE limit of label transfer
> > operations. Given the expense of calling into firmware, maximize the
> > amount of label data we transfer per call to be up to the total label
> > space if allowed by the firmware.
> >
> > Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com>
> > Signed-off-by: Dan Williams <dan.j.williams@intel.com>
>
> This looks mostly fine to me.
>
> The only thing I am not certain about is if we want to use the raw
> max_cmd_size value or not since there is no guarantee it is aligned to
> any sort of boundary. It might be beneficial to either cache line align
> it or align it to 256B if it is larger than that since that way we at
> least would be getting full labels instead of partial ones.

If we leave any portion of the max transfer size unused it just means
we're that much more likely to need an additional call into the ASL
interpreter. That said It might be worthwhile capping the max transfer
size in case the BIOS specifies something extreme (> 1MB) in the
future.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH] libnvdimm, dimm: Maximize label transfer size

[Alexander Duyck]

On 9/28/2018 4:03 PM, Dan Williams wrote:
> On Fri, Sep 28, 2018 at 3:44 PM Alexander Duyck
> <alexander.h.duyck@linux.intel.com> wrote:
>>
>> On 9/28/2018 11:56 AM, Dan Williams wrote:
>>> Use kvzalloc() to bypass the arbitrary PAGE_SIZE limit of label transfer
>>> operations. Given the expense of calling into firmware, maximize the
>>> amount of label data we transfer per call to be up to the total label
>>> space if allowed by the firmware.
>>>
>>> Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com>
>>> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
>>
>> This looks mostly fine to me.
>>
>> The only thing I am not certain about is if we want to use the raw
>> max_cmd_size value or not since there is no guarantee it is aligned to
>> any sort of boundary. It might be beneficial to either cache line align
>> it or align it to 256B if it is larger than that since that way we at
>> least would be getting full labels instead of partial ones.
> 
> If we leave any portion of the max transfer size unused it just means
> we're that much more likely to need an additional call into the ASL
> interpreter. That said It might be worthwhile capping the max transfer
> size in case the BIOS specifies something extreme (> 1MB) in the
> future.

Actually it all depends on the size of the max transfer versus the size 
of the label area. So for example if the label area is 128K and the max 
transfer size is 64K minus 32 then it doesn't really make much 
difference until we get below 44K for our max read size.

My main concern is that I don't want to waste resources on allocating 
memory that isn't needed. For smaller allocations I agree we need to use 
everything we have to minimize calls to the ASL interpreter, but when 
max transfer sizes start getting large we are just allocating multiple 
pages of memory that do not need to be used.

If nothing else I can look at doing the logic to reduce our memory 
allocations as a set of follow-up patches.

- Alex
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: [PATCH] libnvdimm, dimm: Maximize label transfer size

[Dan Williams]

On Mon, Oct 1, 2018 at 8:41 AM Alexander Duyck
<alexander.h.duyck@linux.intel.com> wrote:
>
> On 9/28/2018 4:03 PM, Dan Williams wrote:
> > On Fri, Sep 28, 2018 at 3:44 PM Alexander Duyck
> > <alexander.h.duyck@linux.intel.com> wrote:
> >>
> >> On 9/28/2018 11:56 AM, Dan Williams wrote:
> >>> Use kvzalloc() to bypass the arbitrary PAGE_SIZE limit of label transfer
> >>> operations. Given the expense of calling into firmware, maximize the
> >>> amount of label data we transfer per call to be up to the total label
> >>> space if allowed by the firmware.
> >>>
> >>> Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com>
> >>> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> >>
> >> This looks mostly fine to me.
> >>
> >> The only thing I am not certain about is if we want to use the raw
> >> max_cmd_size value or not since there is no guarantee it is aligned to
> >> any sort of boundary. It might be beneficial to either cache line align
> >> it or align it to 256B if it is larger than that since that way we at
> >> least would be getting full labels instead of partial ones.
> >
> > If we leave any portion of the max transfer size unused it just means
> > we're that much more likely to need an additional call into the ASL
> > interpreter. That said It might be worthwhile capping the max transfer
> > size in case the BIOS specifies something extreme (> 1MB) in the
> > future.
>
> Actually it all depends on the size of the max transfer versus the size
> of the label area. So for example if the label area is 128K and the max
> transfer size is 64K minus 32 then it doesn't really make much
> difference until we get below 44K for our max read size.
>
> My main concern is that I don't want to waste resources on allocating
> memory that isn't needed. For smaller allocations I agree we need to use
> everything we have to minimize calls to the ASL interpreter, but when
> max transfer sizes start getting large we are just allocating multiple
> pages of memory that do not need to be used.
>
> If nothing else I can look at doing the logic to reduce our memory
> allocations as a set of follow-up patches.

Sounds good, I would like to get smarter about only needing to cache
the portion of the label area that is in active use. However there is
at least one bug around simultaneous label updates that I want to
squash before we make the label handling more clever.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm