* [PATCH 1/7] libnvdimm: cleanup nvdimm_namespace_common_probe(), kill 'host'
2016-05-08 22:35 [PATCH 0/7] "Device DAX" for persistent memory Dan Williams
@ 2016-05-08 22:35 ` Dan Williams
2016-05-09 7:55 ` Johannes Thumshirn
2016-05-08 22:35 ` [PATCH 2/7] libnvdimm, dax: introduce device-dax infrastructure Dan Williams
` (6 subsequent siblings)
7 siblings, 1 reply; 11+ messages in thread
From: Dan Williams @ 2016-05-08 22:35 UTC (permalink / raw)
To: linux-nvdimm; +Cc: linux-block, linux-kernel
The 'host' variable can be killed as it is always the same as the passed
in device.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/nvdimm/namespace_devs.c | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/drivers/nvdimm/namespace_devs.c b/drivers/nvdimm/namespace_devs.c
index f5cb88601359..e5ad5162bf34 100644
--- a/drivers/nvdimm/namespace_devs.c
+++ b/drivers/nvdimm/namespace_devs.c
@@ -1379,21 +1379,16 @@ struct nd_namespace_common *nvdimm_namespace_common_probe(struct device *dev)
{
struct nd_btt *nd_btt = is_nd_btt(dev) ? to_nd_btt(dev) : NULL;
struct nd_pfn *nd_pfn = is_nd_pfn(dev) ? to_nd_pfn(dev) : NULL;
- struct nd_namespace_common *ndns;
+ struct nd_namespace_common *ndns = NULL;
resource_size_t size;
if (nd_btt || nd_pfn) {
- struct device *host = NULL;
-
- if (nd_btt) {
- host = &nd_btt->dev;
+ if (nd_btt)
ndns = nd_btt->ndns;
- } else if (nd_pfn) {
- host = &nd_pfn->dev;
+ else if (nd_pfn)
ndns = nd_pfn->ndns;
- }
- if (!ndns || !host)
+ if (!ndns)
return ERR_PTR(-ENODEV);
/*
@@ -1404,12 +1399,12 @@ struct nd_namespace_common *nvdimm_namespace_common_probe(struct device *dev)
device_unlock(&ndns->dev);
if (ndns->dev.driver) {
dev_dbg(&ndns->dev, "is active, can't bind %s\n",
- dev_name(host));
+ dev_name(dev));
return ERR_PTR(-EBUSY);
}
- if (dev_WARN_ONCE(&ndns->dev, ndns->claim != host,
+ if (dev_WARN_ONCE(&ndns->dev, ndns->claim != dev,
"host (%s) vs claim (%s) mismatch\n",
- dev_name(host),
+ dev_name(dev),
dev_name(ndns->claim)))
return ERR_PTR(-ENXIO);
} else {
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH 1/7] libnvdimm: cleanup nvdimm_namespace_common_probe(), kill 'host'
2016-05-08 22:35 ` [PATCH 1/7] libnvdimm: cleanup nvdimm_namespace_common_probe(), kill 'host' Dan Williams
@ 2016-05-09 7:55 ` Johannes Thumshirn
0 siblings, 0 replies; 11+ messages in thread
From: Johannes Thumshirn @ 2016-05-09 7:55 UTC (permalink / raw)
To: Dan Williams; +Cc: linux-nvdimm, linux-block, linux-kernel
On Sun, May 08, 2016 at 03:35:15PM -0700, Dan Williams wrote:
> The 'host' variable can be killed as it is always the same as the passed
> in device.
>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
--
Johannes Thumshirn Storage
jthumshirn@suse.de +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 2/7] libnvdimm, dax: introduce device-dax infrastructure
2016-05-08 22:35 [PATCH 0/7] "Device DAX" for persistent memory Dan Williams
2016-05-08 22:35 ` [PATCH 1/7] libnvdimm: cleanup nvdimm_namespace_common_probe(), kill 'host' Dan Williams
@ 2016-05-08 22:35 ` Dan Williams
2016-05-08 22:35 ` [PATCH 3/7] libnvdimm, dax: reserve space to store labels for device-dax Dan Williams
` (5 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Dan Williams @ 2016-05-08 22:35 UTC (permalink / raw)
To: linux-nvdimm; +Cc: linux-block, linux-kernel
Device DAX is the device-centric analogue of Filesystem DAX
(CONFIG_FS_DAX). It allows persistent memory ranges to be allocated and
mapped without need of an intervening file system. This initial
infrastructure arranges for a libnvdimm pfn-device to be represented as
a different device-type so that it can be attached to a driver other
than the pmem driver.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/nvdimm/Kconfig | 13 +++++
drivers/nvdimm/Makefile | 1
drivers/nvdimm/bus.c | 4 ++
drivers/nvdimm/claim.c | 2 +
drivers/nvdimm/dax_devs.c | 99 +++++++++++++++++++++++++++++++++++++++
drivers/nvdimm/namespace_devs.c | 19 +++++++
drivers/nvdimm/nd-core.h | 1
drivers/nvdimm/nd.h | 25 ++++++++++
drivers/nvdimm/pfn_devs.c | 100 ++++++++++++++++++++++++++-------------
drivers/nvdimm/region.c | 2 +
drivers/nvdimm/region_devs.c | 29 +++++++++++
include/uapi/linux/ndctl.h | 2 +
tools/testing/nvdimm/Kbuild | 1
13 files changed, 264 insertions(+), 34 deletions(-)
create mode 100644 drivers/nvdimm/dax_devs.c
diff --git a/drivers/nvdimm/Kconfig b/drivers/nvdimm/Kconfig
index 53c11621d5b1..7c8a3bf07884 100644
--- a/drivers/nvdimm/Kconfig
+++ b/drivers/nvdimm/Kconfig
@@ -88,4 +88,17 @@ config NVDIMM_PFN
Select Y if unsure
+config NVDIMM_DAX
+ bool "NVDIMM DAX: Raw access to persistent memory"
+ default LIBNVDIMM
+ depends on NVDIMM_PFN
+ help
+ Support raw device dax access to a persistent memory
+ namespace. For environments that want to hard partition
+ peristent memory, this capability provides a mechanism to
+ sub-divide a namespace into character devices that can only be
+ accessed via DAX (mmap(2)).
+
+ Select Y if unsure
+
endif
diff --git a/drivers/nvdimm/Makefile b/drivers/nvdimm/Makefile
index ea84d3c4e8e5..909554c3f955 100644
--- a/drivers/nvdimm/Makefile
+++ b/drivers/nvdimm/Makefile
@@ -23,3 +23,4 @@ libnvdimm-y += label.o
libnvdimm-$(CONFIG_ND_CLAIM) += claim.o
libnvdimm-$(CONFIG_BTT) += btt_devs.o
libnvdimm-$(CONFIG_NVDIMM_PFN) += pfn_devs.o
+libnvdimm-$(CONFIG_NVDIMM_DAX) += dax_devs.o
diff --git a/drivers/nvdimm/bus.c b/drivers/nvdimm/bus.c
index 19f822d7f652..97589e3cb852 100644
--- a/drivers/nvdimm/bus.c
+++ b/drivers/nvdimm/bus.c
@@ -40,6 +40,8 @@ static int to_nd_device_type(struct device *dev)
return ND_DEVICE_REGION_PMEM;
else if (is_nd_blk(dev))
return ND_DEVICE_REGION_BLK;
+ else if (is_nd_dax(dev))
+ return ND_DEVICE_DAX_PMEM;
else if (is_nd_pmem(dev->parent) || is_nd_blk(dev->parent))
return nd_region_to_nstype(to_nd_region(dev->parent));
@@ -246,6 +248,8 @@ static void nd_async_device_unregister(void *d, async_cookie_t cookie)
void __nd_device_register(struct device *dev)
{
+ if (!dev)
+ return;
dev->bus = &nvdimm_bus_type;
get_device(dev);
async_schedule_domain(nd_async_device_register, dev,
diff --git a/drivers/nvdimm/claim.c b/drivers/nvdimm/claim.c
index 6bbd0a36994a..5f53db59a058 100644
--- a/drivers/nvdimm/claim.c
+++ b/drivers/nvdimm/claim.c
@@ -85,6 +85,8 @@ static bool is_idle(struct device *dev, struct nd_namespace_common *ndns)
seed = nd_region->btt_seed;
else if (is_nd_pfn(dev))
seed = nd_region->pfn_seed;
+ else if (is_nd_dax(dev))
+ seed = nd_region->dax_seed;
if (seed == dev || ndns || dev->driver)
return false;
diff --git a/drivers/nvdimm/dax_devs.c b/drivers/nvdimm/dax_devs.c
new file mode 100644
index 000000000000..f90f7549e7f4
--- /dev/null
+++ b/drivers/nvdimm/dax_devs.c
@@ -0,0 +1,99 @@
+/*
+ * Copyright(c) 2013-2016 Intel Corporation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#include <linux/device.h>
+#include <linux/sizes.h>
+#include <linux/slab.h>
+#include <linux/mm.h>
+#include "nd-core.h"
+#include "nd.h"
+
+static void nd_dax_release(struct device *dev)
+{
+ struct nd_region *nd_region = to_nd_region(dev->parent);
+ struct nd_dax *nd_dax = to_nd_dax(dev);
+ struct nd_pfn *nd_pfn = &nd_dax->nd_pfn;
+
+ dev_dbg(dev, "%s\n", __func__);
+ nd_detach_ndns(dev, &nd_pfn->ndns);
+ ida_simple_remove(&nd_region->dax_ida, nd_pfn->id);
+ kfree(nd_pfn->uuid);
+ kfree(nd_dax);
+}
+
+static struct device_type nd_dax_device_type = {
+ .name = "nd_dax",
+ .release = nd_dax_release,
+};
+
+bool is_nd_dax(struct device *dev)
+{
+ return dev ? dev->type == &nd_dax_device_type : false;
+}
+EXPORT_SYMBOL(is_nd_dax);
+
+struct nd_dax *to_nd_dax(struct device *dev)
+{
+ struct nd_dax *nd_dax = container_of(dev, struct nd_dax, nd_pfn.dev);
+
+ WARN_ON(!is_nd_dax(dev));
+ return nd_dax;
+}
+EXPORT_SYMBOL(to_nd_dax);
+
+static const struct attribute_group *nd_dax_attribute_groups[] = {
+ &nd_pfn_attribute_group,
+ &nd_device_attribute_group,
+ &nd_numa_attribute_group,
+ NULL,
+};
+
+static struct nd_dax *nd_dax_alloc(struct nd_region *nd_region)
+{
+ struct nd_pfn *nd_pfn;
+ struct nd_dax *nd_dax;
+ struct device *dev;
+
+ nd_dax = kzalloc(sizeof(*nd_dax), GFP_KERNEL);
+ if (!nd_dax)
+ return NULL;
+
+ nd_pfn = &nd_dax->nd_pfn;
+ nd_pfn->id = ida_simple_get(&nd_region->dax_ida, 0, 0, GFP_KERNEL);
+ if (nd_pfn->id < 0) {
+ kfree(nd_dax);
+ return NULL;
+ }
+
+ dev = &nd_pfn->dev;
+ dev_set_name(dev, "dax%d.%d", nd_region->id, nd_pfn->id);
+ dev->groups = nd_dax_attribute_groups;
+ dev->type = &nd_dax_device_type;
+ dev->parent = &nd_region->dev;
+
+ return nd_dax;
+}
+
+struct device *nd_dax_create(struct nd_region *nd_region)
+{
+ struct device *dev = NULL;
+ struct nd_dax *nd_dax;
+
+ if (!is_nd_pmem(&nd_region->dev))
+ return NULL;
+
+ nd_dax = nd_dax_alloc(nd_region);
+ if (nd_dax)
+ dev = nd_pfn_devinit(&nd_dax->nd_pfn, NULL);
+ __nd_device_register(dev);
+ return dev;
+}
diff --git a/drivers/nvdimm/namespace_devs.c b/drivers/nvdimm/namespace_devs.c
index e5ad5162bf34..c5e3196c45b0 100644
--- a/drivers/nvdimm/namespace_devs.c
+++ b/drivers/nvdimm/namespace_devs.c
@@ -1288,6 +1288,8 @@ static ssize_t mode_show(struct device *dev,
mode = "safe";
else if (claim && is_nd_pfn(claim))
mode = "memory";
+ else if (claim && is_nd_dax(claim))
+ mode = "dax";
else if (!claim && pmem_should_map_pages(dev))
mode = "memory";
else
@@ -1379,14 +1381,17 @@ struct nd_namespace_common *nvdimm_namespace_common_probe(struct device *dev)
{
struct nd_btt *nd_btt = is_nd_btt(dev) ? to_nd_btt(dev) : NULL;
struct nd_pfn *nd_pfn = is_nd_pfn(dev) ? to_nd_pfn(dev) : NULL;
+ struct nd_dax *nd_dax = is_nd_dax(dev) ? to_nd_dax(dev) : NULL;
struct nd_namespace_common *ndns = NULL;
resource_size_t size;
- if (nd_btt || nd_pfn) {
+ if (nd_btt || nd_pfn || nd_dax) {
if (nd_btt)
ndns = nd_btt->ndns;
else if (nd_pfn)
ndns = nd_pfn->ndns;
+ else if (nd_dax)
+ ndns = nd_dax->nd_pfn.ndns;
if (!ndns)
return ERR_PTR(-ENODEV);
@@ -1779,6 +1784,18 @@ void nd_region_create_blk_seed(struct nd_region *nd_region)
nd_device_register(nd_region->ns_seed);
}
+void nd_region_create_dax_seed(struct nd_region *nd_region)
+{
+ WARN_ON(!is_nvdimm_bus_locked(&nd_region->dev));
+ nd_region->dax_seed = nd_dax_create(nd_region);
+ /*
+ * Seed creation failures are not fatal, provisioning is simply
+ * disabled until memory becomes available
+ */
+ if (!nd_region->dax_seed)
+ dev_err(&nd_region->dev, "failed to create dax namespace\n");
+}
+
void nd_region_create_pfn_seed(struct nd_region *nd_region)
{
WARN_ON(!is_nvdimm_bus_locked(&nd_region->dev));
diff --git a/drivers/nvdimm/nd-core.h b/drivers/nvdimm/nd-core.h
index 1d1500f3d8b5..cb65308c0329 100644
--- a/drivers/nvdimm/nd-core.h
+++ b/drivers/nvdimm/nd-core.h
@@ -54,6 +54,7 @@ struct nd_region;
void nd_region_create_blk_seed(struct nd_region *nd_region);
void nd_region_create_btt_seed(struct nd_region *nd_region);
void nd_region_create_pfn_seed(struct nd_region *nd_region);
+void nd_region_create_dax_seed(struct nd_region *nd_region);
void nd_region_disable(struct nvdimm_bus *nvdimm_bus, struct device *dev);
int nvdimm_bus_create_ndctl(struct nvdimm_bus *nvdimm_bus);
void nvdimm_bus_destroy_ndctl(struct nvdimm_bus *nvdimm_bus);
diff --git a/drivers/nvdimm/nd.h b/drivers/nvdimm/nd.h
index 6c36509662e4..46910b8f32b1 100644
--- a/drivers/nvdimm/nd.h
+++ b/drivers/nvdimm/nd.h
@@ -101,10 +101,12 @@ struct nd_region {
struct ida ns_ida;
struct ida btt_ida;
struct ida pfn_ida;
+ struct ida dax_ida;
unsigned long flags;
struct device *ns_seed;
struct device *btt_seed;
struct device *pfn_seed;
+ struct device *dax_seed;
u16 ndr_mappings;
u64 ndr_size;
u64 ndr_start;
@@ -161,6 +163,10 @@ struct nd_pfn {
struct nd_namespace_common *ndns;
};
+struct nd_dax {
+ struct nd_pfn nd_pfn;
+};
+
enum nd_async_mode {
ND_SYNC,
ND_ASYNC,
@@ -224,7 +230,10 @@ struct nd_pfn *to_nd_pfn(struct device *dev);
int nd_pfn_probe(struct device *dev, struct nd_namespace_common *ndns);
bool is_nd_pfn(struct device *dev);
struct device *nd_pfn_create(struct nd_region *nd_region);
+struct device *nd_pfn_devinit(struct nd_pfn *nd_pfn,
+ struct nd_namespace_common *ndns);
int nd_pfn_validate(struct nd_pfn *nd_pfn);
+extern struct attribute_group nd_pfn_attribute_group;
#else
static inline int nd_pfn_probe(struct device *dev,
struct nd_namespace_common *ndns)
@@ -248,6 +257,22 @@ static inline int nd_pfn_validate(struct nd_pfn *nd_pfn)
}
#endif
+struct nd_dax *to_nd_dax(struct device *dev);
+#if IS_ENABLED(CONFIG_NVDIMM_DAX)
+bool is_nd_dax(struct device *dev);
+struct device *nd_dax_create(struct nd_region *nd_region);
+#else
+static inline bool is_nd_dax(struct device *dev)
+{
+ return false;
+}
+
+static inline struct device *nd_dax_create(struct nd_region *nd_region)
+{
+ return NULL;
+}
+#endif
+
struct nd_region *to_nd_region(struct device *dev);
int nd_region_to_nstype(struct nd_region *nd_region);
int nd_region_register_namespaces(struct nd_region *nd_region, int *err);
diff --git a/drivers/nvdimm/pfn_devs.c b/drivers/nvdimm/pfn_devs.c
index e8693fe65e49..6ade2eb7615d 100644
--- a/drivers/nvdimm/pfn_devs.c
+++ b/drivers/nvdimm/pfn_devs.c
@@ -1,5 +1,5 @@
/*
- * Copyright(c) 2013-2015 Intel Corporation. All rights reserved.
+ * Copyright(c) 2013-2016 Intel Corporation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -54,10 +54,29 @@ struct nd_pfn *to_nd_pfn(struct device *dev)
}
EXPORT_SYMBOL(to_nd_pfn);
+static struct nd_pfn *to_nd_pfn_safe(struct device *dev)
+{
+ /*
+ * pfn device attributes are re-used by dax device instances, so we
+ * need to be careful to correct device-to-nd_pfn conversion.
+ */
+ if (is_nd_pfn(dev))
+ return to_nd_pfn(dev);
+
+ if (is_nd_dax(dev)) {
+ struct nd_dax *nd_dax = to_nd_dax(dev);
+
+ return &nd_dax->nd_pfn;
+ }
+
+ WARN_ON(1);
+ return NULL;
+}
+
static ssize_t mode_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- struct nd_pfn *nd_pfn = to_nd_pfn(dev);
+ struct nd_pfn *nd_pfn = to_nd_pfn_safe(dev);
switch (nd_pfn->mode) {
case PFN_MODE_RAM:
@@ -72,7 +91,7 @@ static ssize_t mode_show(struct device *dev,
static ssize_t mode_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
- struct nd_pfn *nd_pfn = to_nd_pfn(dev);
+ struct nd_pfn *nd_pfn = to_nd_pfn_safe(dev);
ssize_t rc = 0;
device_lock(dev);
@@ -106,7 +125,7 @@ static DEVICE_ATTR_RW(mode);
static ssize_t align_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- struct nd_pfn *nd_pfn = to_nd_pfn(dev);
+ struct nd_pfn *nd_pfn = to_nd_pfn_safe(dev);
return sprintf(buf, "%lx\n", nd_pfn->align);
}
@@ -134,7 +153,7 @@ static ssize_t __align_store(struct nd_pfn *nd_pfn, const char *buf)
static ssize_t align_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
- struct nd_pfn *nd_pfn = to_nd_pfn(dev);
+ struct nd_pfn *nd_pfn = to_nd_pfn_safe(dev);
ssize_t rc;
device_lock(dev);
@@ -152,7 +171,7 @@ static DEVICE_ATTR_RW(align);
static ssize_t uuid_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- struct nd_pfn *nd_pfn = to_nd_pfn(dev);
+ struct nd_pfn *nd_pfn = to_nd_pfn_safe(dev);
if (nd_pfn->uuid)
return sprintf(buf, "%pUb\n", nd_pfn->uuid);
@@ -162,7 +181,7 @@ static ssize_t uuid_show(struct device *dev,
static ssize_t uuid_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
- struct nd_pfn *nd_pfn = to_nd_pfn(dev);
+ struct nd_pfn *nd_pfn = to_nd_pfn_safe(dev);
ssize_t rc;
device_lock(dev);
@@ -178,7 +197,7 @@ static DEVICE_ATTR_RW(uuid);
static ssize_t namespace_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- struct nd_pfn *nd_pfn = to_nd_pfn(dev);
+ struct nd_pfn *nd_pfn = to_nd_pfn_safe(dev);
ssize_t rc;
nvdimm_bus_lock(dev);
@@ -191,7 +210,7 @@ static ssize_t namespace_show(struct device *dev,
static ssize_t namespace_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
- struct nd_pfn *nd_pfn = to_nd_pfn(dev);
+ struct nd_pfn *nd_pfn = to_nd_pfn_safe(dev);
ssize_t rc;
device_lock(dev);
@@ -209,7 +228,7 @@ static DEVICE_ATTR_RW(namespace);
static ssize_t resource_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- struct nd_pfn *nd_pfn = to_nd_pfn(dev);
+ struct nd_pfn *nd_pfn = to_nd_pfn_safe(dev);
ssize_t rc;
device_lock(dev);
@@ -235,7 +254,7 @@ static DEVICE_ATTR_RO(resource);
static ssize_t size_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- struct nd_pfn *nd_pfn = to_nd_pfn(dev);
+ struct nd_pfn *nd_pfn = to_nd_pfn_safe(dev);
ssize_t rc;
device_lock(dev);
@@ -270,7 +289,7 @@ static struct attribute *nd_pfn_attributes[] = {
NULL,
};
-static struct attribute_group nd_pfn_attribute_group = {
+struct attribute_group nd_pfn_attribute_group = {
.attrs = nd_pfn_attributes,
};
@@ -281,15 +300,31 @@ static const struct attribute_group *nd_pfn_attribute_groups[] = {
NULL,
};
-static struct device *__nd_pfn_create(struct nd_region *nd_region,
+struct device *nd_pfn_devinit(struct nd_pfn *nd_pfn,
struct nd_namespace_common *ndns)
{
- struct nd_pfn *nd_pfn;
- struct device *dev;
+ struct device *dev = &nd_pfn->dev;
- /* we can only create pages for contiguous ranged of pmem */
- if (!is_nd_pmem(&nd_region->dev))
+ if (!nd_pfn)
+ return NULL;
+
+ nd_pfn->mode = PFN_MODE_NONE;
+ nd_pfn->align = HPAGE_SIZE;
+ dev = &nd_pfn->dev;
+ device_initialize(&nd_pfn->dev);
+ if (ndns && !__nd_attach_ndns(&nd_pfn->dev, ndns, &nd_pfn->ndns)) {
+ dev_dbg(&ndns->dev, "%s failed, already claimed by %s\n",
+ __func__, dev_name(ndns->claim));
+ put_device(dev);
return NULL;
+ }
+ return dev;
+}
+
+static struct nd_pfn *nd_pfn_alloc(struct nd_region *nd_region)
+{
+ struct nd_pfn *nd_pfn;
+ struct device *dev;
nd_pfn = kzalloc(sizeof(*nd_pfn), GFP_KERNEL);
if (!nd_pfn)
@@ -301,29 +336,27 @@ static struct device *__nd_pfn_create(struct nd_region *nd_region,
return NULL;
}
- nd_pfn->mode = PFN_MODE_NONE;
- nd_pfn->align = HPAGE_SIZE;
dev = &nd_pfn->dev;
dev_set_name(dev, "pfn%d.%d", nd_region->id, nd_pfn->id);
- dev->parent = &nd_region->dev;
- dev->type = &nd_pfn_device_type;
dev->groups = nd_pfn_attribute_groups;
- device_initialize(&nd_pfn->dev);
- if (ndns && !__nd_attach_ndns(&nd_pfn->dev, ndns, &nd_pfn->ndns)) {
- dev_dbg(&ndns->dev, "%s failed, already claimed by %s\n",
- __func__, dev_name(ndns->claim));
- put_device(dev);
- return NULL;
- }
- return dev;
+ dev->type = &nd_pfn_device_type;
+ dev->parent = &nd_region->dev;
+
+ return nd_pfn;
}
struct device *nd_pfn_create(struct nd_region *nd_region)
{
- struct device *dev = __nd_pfn_create(nd_region, NULL);
+ struct nd_pfn *nd_pfn;
+ struct device *dev;
+
+ if (!is_nd_pmem(&nd_region->dev))
+ return NULL;
+
+ nd_pfn = nd_pfn_alloc(nd_region);
+ dev = nd_pfn_devinit(nd_pfn, NULL);
- if (dev)
- __nd_device_register(dev);
+ __nd_device_register(dev);
return dev;
}
@@ -423,7 +456,8 @@ int nd_pfn_probe(struct device *dev, struct nd_namespace_common *ndns)
return -ENODEV;
nvdimm_bus_lock(&ndns->dev);
- pfn_dev = __nd_pfn_create(nd_region, ndns);
+ nd_pfn = nd_pfn_alloc(nd_region);
+ pfn_dev = nd_pfn_devinit(nd_pfn, ndns);
nvdimm_bus_unlock(&ndns->dev);
if (!pfn_dev)
return -ENOMEM;
diff --git a/drivers/nvdimm/region.c b/drivers/nvdimm/region.c
index 4b7715e29cff..05a912359939 100644
--- a/drivers/nvdimm/region.c
+++ b/drivers/nvdimm/region.c
@@ -54,6 +54,7 @@ static int nd_region_probe(struct device *dev)
nd_region->btt_seed = nd_btt_create(nd_region);
nd_region->pfn_seed = nd_pfn_create(nd_region);
+ nd_region->dax_seed = nd_dax_create(nd_region);
if (err == 0)
return 0;
@@ -86,6 +87,7 @@ static int nd_region_remove(struct device *dev)
nd_region->ns_seed = NULL;
nd_region->btt_seed = NULL;
nd_region->pfn_seed = NULL;
+ nd_region->dax_seed = NULL;
dev_set_drvdata(dev, NULL);
nvdimm_bus_unlock(dev);
diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c
index 139bf71ca549..9e1b054e0e61 100644
--- a/drivers/nvdimm/region_devs.c
+++ b/drivers/nvdimm/region_devs.c
@@ -306,6 +306,23 @@ static ssize_t pfn_seed_show(struct device *dev,
}
static DEVICE_ATTR_RO(pfn_seed);
+static ssize_t dax_seed_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct nd_region *nd_region = to_nd_region(dev);
+ ssize_t rc;
+
+ nvdimm_bus_lock(dev);
+ if (nd_region->dax_seed)
+ rc = sprintf(buf, "%s\n", dev_name(nd_region->dax_seed));
+ else
+ rc = sprintf(buf, "\n");
+ nvdimm_bus_unlock(dev);
+
+ return rc;
+}
+static DEVICE_ATTR_RO(dax_seed);
+
static ssize_t read_only_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
@@ -335,6 +352,7 @@ static struct attribute *nd_region_attributes[] = {
&dev_attr_mappings.attr,
&dev_attr_btt_seed.attr,
&dev_attr_pfn_seed.attr,
+ &dev_attr_dax_seed.attr,
&dev_attr_read_only.attr,
&dev_attr_set_cookie.attr,
&dev_attr_available_size.attr,
@@ -353,6 +371,9 @@ static umode_t region_visible(struct kobject *kobj, struct attribute *a, int n)
if (!is_nd_pmem(dev) && a == &dev_attr_pfn_seed.attr)
return 0;
+ if (!is_nd_pmem(dev) && a == &dev_attr_dax_seed.attr)
+ return 0;
+
if (a != &dev_attr_set_cookie.attr
&& a != &dev_attr_available_size.attr)
return a->mode;
@@ -441,6 +462,13 @@ static void nd_region_notify_driver_action(struct nvdimm_bus *nvdimm_bus,
nd_region_create_pfn_seed(nd_region);
nvdimm_bus_unlock(dev);
}
+ if (is_nd_dax(dev) && probe) {
+ nd_region = to_nd_region(dev->parent);
+ nvdimm_bus_lock(dev);
+ if (nd_region->dax_seed == dev)
+ nd_region_create_dax_seed(nd_region);
+ nvdimm_bus_unlock(dev);
+ }
}
void nd_region_probe_success(struct nvdimm_bus *nvdimm_bus, struct device *dev)
@@ -718,6 +746,7 @@ static struct nd_region *nd_region_create(struct nvdimm_bus *nvdimm_bus,
ida_init(&nd_region->ns_ida);
ida_init(&nd_region->btt_ida);
ida_init(&nd_region->pfn_ida);
+ ida_init(&nd_region->dax_ida);
dev = &nd_region->dev;
dev_set_name(dev, "region%d", nd_region->id);
dev->parent = &nvdimm_bus->dev;
diff --git a/include/uapi/linux/ndctl.h b/include/uapi/linux/ndctl.h
index 7cc28ab05b87..4f29d247f709 100644
--- a/include/uapi/linux/ndctl.h
+++ b/include/uapi/linux/ndctl.h
@@ -206,6 +206,7 @@ static inline const char *nvdimm_cmd_name(unsigned cmd)
#define ND_DEVICE_NAMESPACE_IO 4 /* legacy persistent memory */
#define ND_DEVICE_NAMESPACE_PMEM 5 /* PMEM namespace (may alias with BLK) */
#define ND_DEVICE_NAMESPACE_BLK 6 /* BLK namespace (may alias with PMEM) */
+#define ND_DEVICE_DAX_PMEM 7 /* Device DAX interface to pmem */
enum nd_driver_flags {
ND_DRIVER_DIMM = 1 << ND_DEVICE_DIMM,
@@ -214,6 +215,7 @@ enum nd_driver_flags {
ND_DRIVER_NAMESPACE_IO = 1 << ND_DEVICE_NAMESPACE_IO,
ND_DRIVER_NAMESPACE_PMEM = 1 << ND_DEVICE_NAMESPACE_PMEM,
ND_DRIVER_NAMESPACE_BLK = 1 << ND_DEVICE_NAMESPACE_BLK,
+ ND_DRIVER_DAX_PMEM = 1 << ND_DEVICE_DAX_PMEM,
};
enum {
diff --git a/tools/testing/nvdimm/Kbuild b/tools/testing/nvdimm/Kbuild
index d5bc8c080b44..5ff6d3c126a9 100644
--- a/tools/testing/nvdimm/Kbuild
+++ b/tools/testing/nvdimm/Kbuild
@@ -50,6 +50,7 @@ libnvdimm-y += $(NVDIMM_SRC)/label.o
libnvdimm-$(CONFIG_ND_CLAIM) += $(NVDIMM_SRC)/claim.o
libnvdimm-$(CONFIG_BTT) += $(NVDIMM_SRC)/btt_devs.o
libnvdimm-$(CONFIG_NVDIMM_PFN) += $(NVDIMM_SRC)/pfn_devs.o
+libnvdimm-$(CONFIG_NVDIMM_DAX) += $(NVDIMM_SRC)/dax_devs.o
libnvdimm-y += config_check.o
obj-m += test/
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 3/7] libnvdimm, dax: reserve space to store labels for device-dax
2016-05-08 22:35 [PATCH 0/7] "Device DAX" for persistent memory Dan Williams
2016-05-08 22:35 ` [PATCH 1/7] libnvdimm: cleanup nvdimm_namespace_common_probe(), kill 'host' Dan Williams
2016-05-08 22:35 ` [PATCH 2/7] libnvdimm, dax: introduce device-dax infrastructure Dan Williams
@ 2016-05-08 22:35 ` Dan Williams
2016-05-08 22:35 ` [PATCH 4/7] libnvdimm, dax: record the specified alignment of a dax-device instance Dan Williams
` (4 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Dan Williams @ 2016-05-08 22:35 UTC (permalink / raw)
To: linux-nvdimm; +Cc: linux-block, linux-kernel
We may want to subdivide a device-dax range into multiple devices so
that each can have separate permissions or naming. Reserve 128K of
label space by default so we have the capability of making allocation
decisions persistent. This reservation is not something we can add
later since it would result in the default size of a device-dax range
changing between kernel versions.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/nvdimm/pfn_devs.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/drivers/nvdimm/pfn_devs.c b/drivers/nvdimm/pfn_devs.c
index 6ade2eb7615d..ca396c8f2cd5 100644
--- a/drivers/nvdimm/pfn_devs.c
+++ b/drivers/nvdimm/pfn_devs.c
@@ -540,6 +540,7 @@ static struct vmem_altmap *__nvdimm_setup_pfn(struct nd_pfn *nd_pfn,
static int nd_pfn_init(struct nd_pfn *nd_pfn)
{
+ u32 dax_label_reserve = is_nd_dax(&nd_pfn->dev) ? SZ_128K : 0;
struct nd_namespace_common *ndns = nd_pfn->ndns;
u32 start_pad = 0, end_trunc = 0;
resource_size_t start, size;
@@ -606,10 +607,11 @@ static int nd_pfn_init(struct nd_pfn *nd_pfn)
size = resource_size(&nsio->res);
npfns = (size - start_pad - end_trunc - SZ_8K) / SZ_4K;
if (nd_pfn->mode == PFN_MODE_PMEM)
- offset = ALIGN(start + SZ_8K + 64 * npfns, nd_pfn->align)
- - start;
+ offset = ALIGN(start + SZ_8K + 64 * npfns + dax_label_reserve,
+ nd_pfn->align) - start;
else if (nd_pfn->mode == PFN_MODE_RAM)
- offset = ALIGN(start + SZ_8K, nd_pfn->align) - start;
+ offset = ALIGN(start + SZ_8K + dax_label_reserve,
+ nd_pfn->align) - start;
else
return -ENXIO;
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 4/7] libnvdimm, dax: record the specified alignment of a dax-device instance
2016-05-08 22:35 [PATCH 0/7] "Device DAX" for persistent memory Dan Williams
` (2 preceding siblings ...)
2016-05-08 22:35 ` [PATCH 3/7] libnvdimm, dax: reserve space to store labels for device-dax Dan Williams
@ 2016-05-08 22:35 ` Dan Williams
2016-05-08 22:35 ` [PATCH 5/7] /dev/dax, pmem: direct access to persistent memory Dan Williams
` (3 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Dan Williams @ 2016-05-08 22:35 UTC (permalink / raw)
To: linux-nvdimm; +Cc: linux-block, linux-kernel
We want to use the alignment as the allocation and mapping unit.
Previously this information was only useful for establishing the data
offset, but now it is important to remember the granularity for the
later use.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/nvdimm/pfn.h | 4 +++-
drivers/nvdimm/pfn_devs.c | 8 ++++++--
2 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/drivers/nvdimm/pfn.h b/drivers/nvdimm/pfn.h
index 8e343a3ca873..9d2704c83fa7 100644
--- a/drivers/nvdimm/pfn.h
+++ b/drivers/nvdimm/pfn.h
@@ -33,7 +33,9 @@ struct nd_pfn_sb {
/* minor-version-1 additions for section alignment */
__le32 start_pad;
__le32 end_trunc;
- u8 padding[4004];
+ /* minor-version-2 record the base alignment of the mapping */
+ __le32 align;
+ u8 padding[4000];
__le64 checksum;
};
diff --git a/drivers/nvdimm/pfn_devs.c b/drivers/nvdimm/pfn_devs.c
index ca396c8f2cd5..58740d7ce81b 100644
--- a/drivers/nvdimm/pfn_devs.c
+++ b/drivers/nvdimm/pfn_devs.c
@@ -394,6 +394,9 @@ int nd_pfn_validate(struct nd_pfn *nd_pfn)
pfn_sb->end_trunc = 0;
}
+ if (__le16_to_cpu(pfn_sb->version_minor) < 2)
+ pfn_sb->align = 0;
+
switch (le32_to_cpu(pfn_sb->mode)) {
case PFN_MODE_RAM:
case PFN_MODE_PMEM:
@@ -433,7 +436,7 @@ int nd_pfn_validate(struct nd_pfn *nd_pfn)
return -EBUSY;
}
- nd_pfn->align = 1UL << ilog2(offset);
+ nd_pfn->align = le32_to_cpu(pfn_sb->align);
if (!is_power_of_2(offset) || offset < PAGE_SIZE) {
dev_err(&nd_pfn->dev, "bad offset: %#llx dax disabled\n",
offset);
@@ -629,9 +632,10 @@ static int nd_pfn_init(struct nd_pfn *nd_pfn)
memcpy(pfn_sb->uuid, nd_pfn->uuid, 16);
memcpy(pfn_sb->parent_uuid, nd_dev_to_uuid(&ndns->dev), 16);
pfn_sb->version_major = cpu_to_le16(1);
- pfn_sb->version_minor = cpu_to_le16(1);
+ pfn_sb->version_minor = cpu_to_le16(2);
pfn_sb->start_pad = cpu_to_le32(start_pad);
pfn_sb->end_trunc = cpu_to_le32(end_trunc);
+ pfn_sb->align = cpu_to_le32(nd_pfn->align);
checksum = nd_sb_checksum((struct nd_gen_sb *) pfn_sb);
pfn_sb->checksum = cpu_to_le64(checksum);
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 5/7] /dev/dax, pmem: direct access to persistent memory
2016-05-08 22:35 [PATCH 0/7] "Device DAX" for persistent memory Dan Williams
` (3 preceding siblings ...)
2016-05-08 22:35 ` [PATCH 4/7] libnvdimm, dax: record the specified alignment of a dax-device instance Dan Williams
@ 2016-05-08 22:35 ` Dan Williams
2016-05-08 22:35 ` [PATCH 6/7] /dev/dax, core: file operations and dax-mmap Dan Williams
` (2 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Dan Williams @ 2016-05-08 22:35 UTC (permalink / raw)
To: linux-nvdimm
Cc: Dave Hansen, linux-kernel, linux-block, Jeff Moyer, Ross Zwisler,
Andrew Morton, Christoph Hellwig
Device DAX is the device-centric analogue of Filesystem DAX
(CONFIG_FS_DAX). It allows memory ranges to be allocated and mapped
without need of an intervening file system. Device DAX is strict,
precise and predictable. Specifically this interface:
1/ Guarantees fault granularity with respect to a given page size (pte,
pmd, or pud) set at configuration time.
2/ Enforces deterministic behavior by being strict about what fault
scenarios are supported.
For example, by forcing MADV_DONTFORK semantics and omitting MAP_PRIVATE
support device-dax guarantees that a mapping always behaves/performs the
same once established. It is the "what you see is what you get" access
mechanism to differentiated memory vs filesystem DAX which has
filesystem specific implementation semantics.
Persistent memory is the first target, but the mechanism is also
targeted for exclusive allocations of performance differentiated memory
ranges.
This commit is limited to the base device driver infrastructure to
associate a dax device with pmem range.
Cc: Jeff Moyer <jmoyer@redhat.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/Kconfig | 2
drivers/Makefile | 1
drivers/dax/Kconfig | 25 ++++
drivers/dax/Makefile | 4 +
drivers/dax/dax.c | 223 +++++++++++++++++++++++++++++++++++
drivers/dax/dax.h | 24 ++++
drivers/dax/pmem.c | 168 ++++++++++++++++++++++++++
tools/testing/nvdimm/Kbuild | 9 +
tools/testing/nvdimm/config_check.c | 2
9 files changed, 458 insertions(+)
create mode 100644 drivers/dax/Kconfig
create mode 100644 drivers/dax/Makefile
create mode 100644 drivers/dax/dax.c
create mode 100644 drivers/dax/dax.h
create mode 100644 drivers/dax/pmem.c
diff --git a/drivers/Kconfig b/drivers/Kconfig
index d2ac339de85f..8298eab84a6f 100644
--- a/drivers/Kconfig
+++ b/drivers/Kconfig
@@ -190,6 +190,8 @@ source "drivers/android/Kconfig"
source "drivers/nvdimm/Kconfig"
+source "drivers/dax/Kconfig"
+
source "drivers/nvmem/Kconfig"
source "drivers/hwtracing/stm/Kconfig"
diff --git a/drivers/Makefile b/drivers/Makefile
index 8f5d076baeb0..0b6f3d60193d 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -66,6 +66,7 @@ obj-$(CONFIG_PARPORT) += parport/
obj-$(CONFIG_NVM) += lightnvm/
obj-y += base/ block/ misc/ mfd/ nfc/
obj-$(CONFIG_LIBNVDIMM) += nvdimm/
+obj-$(CONFIG_DEV_DAX) += dax/
obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/
obj-$(CONFIG_NUBUS) += nubus/
obj-y += macintosh/
diff --git a/drivers/dax/Kconfig b/drivers/dax/Kconfig
new file mode 100644
index 000000000000..86ffbaa891ad
--- /dev/null
+++ b/drivers/dax/Kconfig
@@ -0,0 +1,25 @@
+menuconfig DEV_DAX
+ tristate "DAX: direct access to differentiated memory"
+ default m if NVDIMM_DAX
+ help
+ Support raw access to differentiated (persistence, bandwidth,
+ latency...) memory via an mmap(2) capable character
+ device. Platform firmware or a device driver may identify a
+ platform memory resource that is differentiated from the
+ baseline memory pool. Mappings of a /dev/daxX.Y device impose
+ restrictions that make the mapping behavior deterministic.
+
+if DEV_DAX
+
+config DEV_DAX_PMEM
+ tristate "PMEM DAX: direct access to persistent memory"
+ depends on NVDIMM_DAX
+ default DEV_DAX
+ help
+ Support raw access to persistent memory. Note that this
+ driver consumes memory ranges allocated and exported by the
+ libnvdimm sub-system.
+
+ Say Y if unsure
+
+endif
diff --git a/drivers/dax/Makefile b/drivers/dax/Makefile
new file mode 100644
index 000000000000..27c54e38478a
--- /dev/null
+++ b/drivers/dax/Makefile
@@ -0,0 +1,4 @@
+obj-$(CONFIG_DEV_DAX) += dax.o
+obj-$(CONFIG_DEV_DAX_PMEM) += dax_pmem.o
+
+dax_pmem-y := pmem.o
diff --git a/drivers/dax/dax.c b/drivers/dax/dax.c
new file mode 100644
index 000000000000..e95af4ead357
--- /dev/null
+++ b/drivers/dax/dax.c
@@ -0,0 +1,223 @@
+/*
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#include <linux/pagemap.h>
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/pfn_t.h>
+#include <linux/slab.h>
+#include <linux/dax.h>
+#include <linux/fs.h>
+#include <linux/mm.h>
+
+static int dax_major;
+static struct class *dax_class;
+static DEFINE_IDA(dax_minor_ida);
+
+struct dax_region {
+ int id;
+ struct ida ida;
+ void *base;
+ struct kref kref;
+ struct device *dev;
+ unsigned int align;
+ struct resource res;
+ unsigned long pfn_flags;
+};
+
+struct dax_dev {
+ struct dax_region *region;
+ struct device *dev;
+ int num_resources;
+ struct resource res[0];
+};
+
+static void dax_region_release(struct kref *kref)
+{
+ struct dax_region *dax_region;
+
+ dax_region = container_of(kref, struct dax_region, kref);
+ kfree(dax_region);
+}
+
+void dax_region_put(struct dax_region *dax_region)
+{
+ kref_put(&dax_region->kref, dax_region_release);
+}
+EXPORT_SYMBOL_GPL(dax_region_put);
+
+static void dax_release(struct device *dev)
+{
+ struct dax_dev *dax_dev = dev_get_drvdata(dev);
+ struct dax_region *dax_region = dax_dev->region;
+ int region_id, id, rc, minor;
+
+ dev_dbg(dev, "%s\n", __func__);
+ rc = sscanf(dev_name(dev), "dax%d.%d", ®ion_id, &id);
+ WARN_ON(rc != 2 || dax_region->id != region_id);
+
+ ida_simple_remove(&dax_region->ida, id);
+ minor = MINOR(dev->devt);
+ ida_simple_remove(&dax_minor_ida, minor);
+ dax_region_put(dax_region);
+}
+
+struct dax_region *alloc_dax_region(struct device *parent,
+ int region_id, struct resource *res, unsigned int align,
+ void *addr, unsigned long pfn_flags)
+{
+ struct dax_region *dax_region;
+
+ dax_region = kzalloc(sizeof(*dax_region), GFP_KERNEL);
+
+ if (!dax_region)
+ return NULL;
+
+ memcpy(&dax_region->res, res, sizeof(*res));
+ dax_region->pfn_flags = pfn_flags;
+ kref_init(&dax_region->kref);
+ dax_region->id = region_id;
+ ida_init(&dax_region->ida);
+ dax_region->align = align;
+ dax_region->dev = parent;
+ dax_region->base = addr;
+
+ return dax_region;
+}
+EXPORT_SYMBOL_GPL(alloc_dax_region);
+
+static ssize_t size_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct dax_dev *dax_dev = dev_get_drvdata(dev);
+ unsigned long long size = 0;
+ int i;
+
+ for (i = 0; i < dax_dev->num_resources; i++)
+ size += resource_size(&dax_dev->res[i]);
+
+ return sprintf(buf, "%llu\n", size);
+}
+static DEVICE_ATTR_RO(size);
+
+static struct attribute *dax_device_attributes[] = {
+ &dev_attr_size.attr,
+ NULL,
+};
+
+static const struct attribute_group dax_device_attribute_group = {
+ .attrs = dax_device_attributes,
+};
+
+static const struct attribute_group *dax_attribute_groups[] = {
+ &dax_device_attribute_group,
+ NULL,
+};
+
+static void destroy_dax_dev(void *dev)
+{
+ dev_dbg(dev, "%s\n", __func__);
+ device_unregister(dev);
+}
+
+int devm_create_dax_dev(struct dax_region *dax_region, struct resource *res,
+ int count)
+{
+ struct device *parent = dax_region->dev;
+ struct dax_dev *dax_dev;
+ struct device *dev;
+ int id, rc, minor;
+ dev_t dev_t;
+
+ dax_dev = kzalloc(sizeof(*dax_dev) + sizeof(*res) * count, GFP_KERNEL);
+ if (!dax_dev)
+ return -ENOMEM;
+ memcpy(dax_dev->res, res, sizeof(*res) * count);
+ dax_dev->num_resources = count;
+ dax_dev->region = dax_region;
+ kref_get(&dax_region->kref);
+
+ id = ida_simple_get(&dax_region->ida, 0, 0, GFP_KERNEL);
+ if (id < 0) {
+ rc = id;
+ goto err_id;
+ }
+
+ minor = ida_simple_get(&dax_minor_ida, 0, 0, GFP_KERNEL);
+ if (minor < 0) {
+ rc = minor;
+ goto err_minor;
+ }
+
+ dev_t = MKDEV(dax_major, id);
+ dev = device_create_with_groups(dax_class, parent, dev_t, dax_dev,
+ dax_attribute_groups, "dax%d.%d", dax_region->id, id);
+ if (IS_ERR(dev)) {
+ rc = PTR_ERR(dev);
+ goto err_create;
+ }
+ dax_dev->dev = dev;
+
+ rc = devm_add_action(dax_region->dev, destroy_dax_dev, dev);
+ if (rc) {
+ destroy_dax_dev(dev);
+ return rc;
+ }
+
+ return 0;
+
+ err_create:
+ ida_simple_remove(&dax_minor_ida, minor);
+ err_minor:
+ ida_simple_remove(&dax_region->ida, id);
+ err_id:
+ dax_region_put(dax_region);
+ kfree(dax_dev);
+
+ return rc;
+}
+EXPORT_SYMBOL_GPL(devm_create_dax_dev);
+
+static const struct file_operations dax_fops = {
+ .llseek = noop_llseek,
+ .owner = THIS_MODULE,
+};
+
+static int __init dax_init(void)
+{
+ int rc;
+
+ rc = register_chrdev(0, "dax", &dax_fops);
+ if (rc < 0)
+ return rc;
+ dax_major = rc;
+
+ dax_class = class_create(THIS_MODULE, "dax");
+ if (IS_ERR(dax_class)) {
+ unregister_chrdev(dax_major, "dax");
+ return PTR_ERR(dax_class);
+ }
+ dax_class->dev_release = dax_release;
+
+ return 0;
+}
+
+static void __exit dax_exit(void)
+{
+ class_destroy(dax_class);
+ unregister_chrdev(dax_major, "dax");
+}
+
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+subsys_initcall(dax_init);
+module_exit(dax_exit);
diff --git a/drivers/dax/dax.h b/drivers/dax/dax.h
new file mode 100644
index 000000000000..d8b8f1f25054
--- /dev/null
+++ b/drivers/dax/dax.h
@@ -0,0 +1,24 @@
+/*
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#ifndef __DAX_H__
+#define __DAX_H__
+struct device;
+struct resource;
+struct dax_region;
+void dax_region_put(struct dax_region *dax_region);
+struct dax_region *alloc_dax_region(struct device *parent,
+ int region_id, struct resource *res, unsigned int align,
+ void *addr, unsigned long flags);
+int devm_create_dax_dev(struct dax_region *dax_region, struct resource *res,
+ int count);
+#endif /* __DAX_H__ */
diff --git a/drivers/dax/pmem.c b/drivers/dax/pmem.c
new file mode 100644
index 000000000000..4e97555e1cab
--- /dev/null
+++ b/drivers/dax/pmem.c
@@ -0,0 +1,168 @@
+/*
+ * Copyright(c) 2016 Intel Corporation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#include <linux/percpu-refcount.h>
+#include <linux/memremap.h>
+#include <linux/module.h>
+#include <linux/pfn_t.h>
+#include "../nvdimm/pfn.h"
+#include "../nvdimm/nd.h"
+#include "dax.h"
+
+struct dax_pmem {
+ struct device *dev;
+ struct percpu_ref ref;
+ struct completion cmp;
+};
+
+struct dax_pmem *to_dax_pmem(struct percpu_ref *ref)
+{
+ return container_of(ref, struct dax_pmem, ref);
+}
+
+static void dax_pmem_percpu_release(struct percpu_ref *ref)
+{
+ struct dax_pmem *dax_pmem = to_dax_pmem(ref);
+
+ dev_dbg(dax_pmem->dev, "%s\n", __func__);
+ complete(&dax_pmem->cmp);
+}
+
+static void dax_pmem_percpu_exit(void *data)
+{
+ struct percpu_ref *ref = data;
+ struct dax_pmem *dax_pmem = to_dax_pmem(ref);
+
+ dev_dbg(dax_pmem->dev, "%s\n", __func__);
+ percpu_ref_exit(ref);
+ wait_for_completion(&dax_pmem->cmp);
+}
+
+static void dax_pmem_percpu_kill(void *data)
+{
+ struct percpu_ref *ref = data;
+ struct dax_pmem *dax_pmem = to_dax_pmem(ref);
+
+ dev_dbg(dax_pmem->dev, "%s\n", __func__);
+ percpu_ref_kill(ref);
+}
+
+static int dax_pmem_probe(struct device *dev)
+{
+ int rc;
+ void *addr;
+ struct resource res;
+ struct nd_pfn_sb *pfn_sb;
+ struct dax_pmem *dax_pmem;
+ struct nd_region *nd_region;
+ struct nd_namespace_io *nsio;
+ struct dax_region *dax_region;
+ struct nd_namespace_common *ndns;
+ struct nd_dax *nd_dax = to_nd_dax(dev);
+ struct nd_pfn *nd_pfn = &nd_dax->nd_pfn;
+ struct vmem_altmap __altmap, *altmap = NULL;
+
+ ndns = nvdimm_namespace_common_probe(dev);
+ if (IS_ERR(ndns))
+ return PTR_ERR(ndns);
+ nsio = to_nd_namespace_io(&ndns->dev);
+
+ /* parse the 'pfn' info block via ->rw_bytes */
+ devm_nsio_enable(dev, nsio);
+ altmap = nvdimm_setup_pfn(nd_pfn, &res, &__altmap);
+ if (IS_ERR(altmap))
+ return PTR_ERR(altmap);
+ devm_nsio_disable(dev, nsio);
+
+ pfn_sb = nd_pfn->pfn_sb;
+
+ if (!devm_request_mem_region(dev, nsio->res.start,
+ resource_size(&nsio->res), dev_name(dev))) {
+ dev_warn(dev, "could not reserve region %pR\n", &nsio->res);
+ return -EBUSY;
+ }
+
+ dax_pmem = devm_kzalloc(dev, sizeof(*dax_pmem), GFP_KERNEL);
+ if (!dax_pmem)
+ return -ENOMEM;
+
+ dax_pmem->dev = dev;
+ init_completion(&dax_pmem->cmp);
+ rc = percpu_ref_init(&dax_pmem->ref, dax_pmem_percpu_release, 0,
+ GFP_KERNEL);
+ if (rc)
+ return rc;
+
+ rc = devm_add_action(dev, dax_pmem_percpu_exit, &dax_pmem->ref);
+ if (rc) {
+ dax_pmem_percpu_exit(&dax_pmem->ref);
+ return rc;
+ }
+
+ addr = devm_memremap_pages(dev, &res, &dax_pmem->ref, altmap);
+ if (IS_ERR(addr))
+ return PTR_ERR(addr);
+
+ rc = devm_add_action(dev, dax_pmem_percpu_kill, &dax_pmem->ref);
+ if (rc) {
+ dax_pmem_percpu_kill(&dax_pmem->ref);
+ return rc;
+ }
+
+ nd_region = to_nd_region(dev->parent);
+ dax_region = alloc_dax_region(dev, nd_region->id, &res,
+ le32_to_cpu(pfn_sb->align), addr, PFN_DEV|PFN_MAP);
+ if (!dax_region)
+ return -ENOMEM;
+
+ /* TODO: support for subdividing a dax region... */
+ rc = devm_create_dax_dev(dax_region, &res, 1);
+
+ /* child dax_dev instances now own the lifetime of the dax_region */
+ dax_region_put(dax_region);
+
+ return rc;
+}
+
+static int dax_pmem_remove(struct device *dev)
+{
+ /*
+ * The init path is fully devm-enabled, or child devices
+ * otherwise hold references on parent resources.
+ */
+ return 0;
+}
+
+static struct nd_device_driver dax_pmem_driver = {
+ .probe = dax_pmem_probe,
+ .remove = dax_pmem_remove,
+ .drv = {
+ .name = "dax_pmem",
+ },
+ .type = ND_DRIVER_DAX_PMEM,
+};
+
+static int __init dax_pmem_init(void)
+{
+ return nd_driver_register(&dax_pmem_driver);
+}
+module_init(dax_pmem_init);
+
+static void __exit dax_pmem_exit(void)
+{
+ driver_unregister(&dax_pmem_driver.drv);
+}
+module_exit(dax_pmem_exit);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_ALIAS_ND_DEVICE(ND_DEVICE_DAX_PMEM);
diff --git a/tools/testing/nvdimm/Kbuild b/tools/testing/nvdimm/Kbuild
index 5ff6d3c126a9..785985677159 100644
--- a/tools/testing/nvdimm/Kbuild
+++ b/tools/testing/nvdimm/Kbuild
@@ -16,6 +16,7 @@ ldflags-y += --wrap=phys_to_pfn_t
DRIVERS := ../../../drivers
NVDIMM_SRC := $(DRIVERS)/nvdimm
ACPI_SRC := $(DRIVERS)/acpi
+DAX_SRC := $(DRIVERS)/dax
obj-$(CONFIG_LIBNVDIMM) += libnvdimm.o
obj-$(CONFIG_BLK_DEV_PMEM) += nd_pmem.o
@@ -23,6 +24,8 @@ obj-$(CONFIG_ND_BTT) += nd_btt.o
obj-$(CONFIG_ND_BLK) += nd_blk.o
obj-$(CONFIG_X86_PMEM_LEGACY) += nd_e820.o
obj-$(CONFIG_ACPI_NFIT) += nfit.o
+obj-$(CONFIG_DEV_DAX) += dax.o
+obj-$(CONFIG_DEV_DAX_PMEM) += dax_pmem.o
nfit-y := $(ACPI_SRC)/nfit.o
nfit-y += config_check.o
@@ -39,6 +42,12 @@ nd_blk-y += config_check.o
nd_e820-y := $(NVDIMM_SRC)/e820.o
nd_e820-y += config_check.o
+dax-y := $(DAX_SRC)/dax.o
+dax-y += config_check.o
+
+dax_pmem-y := $(DAX_SRC)/pmem.o
+dax_pmem-y += config_check.o
+
libnvdimm-y := $(NVDIMM_SRC)/core.o
libnvdimm-y += $(NVDIMM_SRC)/bus.o
libnvdimm-y += $(NVDIMM_SRC)/dimm_devs.o
diff --git a/tools/testing/nvdimm/config_check.c b/tools/testing/nvdimm/config_check.c
index f2c7615554eb..adf18bfeca00 100644
--- a/tools/testing/nvdimm/config_check.c
+++ b/tools/testing/nvdimm/config_check.c
@@ -12,4 +12,6 @@ void check(void)
BUILD_BUG_ON(!IS_MODULE(CONFIG_ND_BTT));
BUILD_BUG_ON(!IS_MODULE(CONFIG_ND_BLK));
BUILD_BUG_ON(!IS_MODULE(CONFIG_ACPI_NFIT));
+ BUILD_BUG_ON(!IS_MODULE(CONFIG_DEV_DAX));
+ BUILD_BUG_ON(!IS_MODULE(CONFIG_DEV_DAX_PMEM));
}
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 6/7] /dev/dax, core: file operations and dax-mmap
2016-05-08 22:35 [PATCH 0/7] "Device DAX" for persistent memory Dan Williams
` (4 preceding siblings ...)
2016-05-08 22:35 ` [PATCH 5/7] /dev/dax, pmem: direct access to persistent memory Dan Williams
@ 2016-05-08 22:35 ` Dan Williams
2016-05-08 22:35 ` [PATCH 7/7] Revert "block: enable dax for raw block devices" Dan Williams
2016-05-09 12:57 ` [PATCH 0/7] "Device DAX" for persistent memory Christoph Hellwig
7 siblings, 0 replies; 11+ messages in thread
From: Dan Williams @ 2016-05-08 22:35 UTC (permalink / raw)
To: linux-nvdimm
Cc: Dave Hansen, linux-kernel, linux-block, Jeff Moyer, Ross Zwisler,
Andrew Morton, Christoph Hellwig
The "Device DAX" core enables dax mappings of performance / feature
differentiated memory. An open mapping or file handle keeps the backing
struct device live, but new mappings are only possible while the device
is enabled. Faults are handled under the device lock to synchronize
with the enabled state of the device. Per the standard device model,
device->driver_data is non-NULL while the device is enabled, and device
state transitions happen under the device lock.
Similar to the filesystem-dax case the backing memory may optionally
have struct page entries. However, unlike fs-dax there is no support
for private mappings, or mappings that are not backed by media (see
use of zero-page in fs-dax).
Mappings are always guaranteed to match the alignment of the dax_region.
If the dax_region is configured to have a 2MB alignment, all mappings
are guaranteed to be backed by a pmd entry. Contrast this determinism
with the fs-dax case where pmd mappings are opportunistic. If userspace
attempts to force a misaligned mapping, the driver will fail the mmap
attempt. See dax_dev_check_vma() for other scenarios that are rejected,
like MAP_PRIVATE mappings.
Cc: Jeff Moyer <jmoyer@redhat.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dax/dax.c | 320 +++++++++++++++++++++++++++++++++++++++++++++++++++++
mm/huge_memory.c | 1
mm/hugetlb.c | 1
3 files changed, 322 insertions(+)
diff --git a/drivers/dax/dax.c b/drivers/dax/dax.c
index e95af4ead357..4198054e703c 100644
--- a/drivers/dax/dax.c
+++ b/drivers/dax/dax.c
@@ -187,9 +187,329 @@ int devm_create_dax_dev(struct dax_region *dax_region, struct resource *res,
}
EXPORT_SYMBOL_GPL(devm_create_dax_dev);
+/* return an unmapped area aligned to the dax region specified alignment */
+static unsigned long dax_dev_get_unmapped_area(struct file *filp,
+ unsigned long addr, unsigned long len, unsigned long pgoff,
+ unsigned long flags)
+{
+ struct dax_dev *dax_dev;
+ struct device *dev = filp ? filp->private_data : NULL;
+ unsigned long off, off_end, off_align, len_align, addr_align, align = 0;
+
+ if (!filp || addr)
+ goto out;
+
+ device_lock(dev);
+ dax_dev = dev_get_drvdata(dev);
+ if (dax_dev) {
+ struct dax_region *dax_region = dax_dev->region;
+
+ align = dax_region->align;
+ }
+ device_unlock(dev);
+
+ if (!align)
+ goto out;
+
+ off = pgoff << PAGE_SHIFT;
+ off_end = off + len;
+ off_align = round_up(off, align);
+
+ if ((off_end <= off_align) || ((off_end - off_align) < align))
+ goto out;
+
+ len_align = len + align;
+ if ((off + len_align) < off)
+ goto out;
+
+ addr_align = current->mm->get_unmapped_area(filp, addr, len_align,
+ pgoff, flags);
+ if (!IS_ERR_VALUE(addr_align)) {
+ addr_align += (off - addr_align) & (align - 1);
+ return addr_align;
+ }
+ out:
+ return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags);
+}
+
+static int __match_devt(struct device *dev, const void *data)
+{
+ const dev_t *devt = data;
+
+ return dev->devt == *devt;
+}
+
+static int dax_dev_open(struct inode *inode, struct file *filp)
+{
+ struct device *dev;
+
+ dev = class_find_device(dax_class, NULL, &inode->i_rdev, __match_devt);
+ if (dev) {
+ dev_dbg(dev, "%s\n", __func__);
+ filp->private_data = dev;
+ inode->i_flags = S_DAX;
+ return 0;
+ }
+ return -ENXIO;
+}
+
+static int dax_dev_release(struct inode *inode, struct file *filp)
+{
+ struct device *dev = filp->private_data;
+
+ dev_dbg(dev, "%s\n", __func__);
+ put_device(dev);
+ return 0;
+}
+
+static struct dax_dev *to_dax_dev(struct device *dev)
+{
+ WARN_ON(dev->class != dax_class);
+ device_lock_assert(dev);
+ return dev_get_drvdata(dev);
+}
+
+static int dax_dev_check_vma(struct device *dev, struct vm_area_struct *vma,
+ const char *func)
+{
+ struct dax_dev *dax_dev = to_dax_dev(dev);
+ struct dax_region *dax_region;
+ unsigned long mask;
+
+ if (!dax_dev)
+ return -ENXIO;
+
+ /* prevent private / writable mappings from being established */
+ if ((vma->vm_flags & (VM_NORESERVE|VM_SHARED|VM_WRITE)) == VM_WRITE) {
+ dev_dbg(dev, "%s: fail, attempted private mapping\n", func);
+ return -EINVAL;
+ }
+
+ dax_region = dax_dev->region;
+ mask = dax_region->align - 1;
+ if (vma->vm_start & mask || vma->vm_end & mask) {
+ dev_dbg(dev, "%s: fail, unaligned vma (%#lx - %#lx, %#lx)\n",
+ func, vma->vm_start, vma->vm_end, mask);
+ return -EINVAL;
+ }
+
+ if ((dax_region->pfn_flags & (PFN_DEV|PFN_MAP)) == PFN_DEV
+ && (vma->vm_flags & VM_DONTCOPY) == 0) {
+ dev_dbg(dev, "%s: fail, dax range requires MADV_DONTFORK\n",
+ func);
+ return -EINVAL;
+ }
+
+ if (!vma_is_dax(vma)) {
+ dev_dbg(dev, "%s: fail, vma is not DAX capable\n", func);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static phys_addr_t pgoff_to_phys(struct dax_dev *dax_dev, pgoff_t pgoff,
+ unsigned long size)
+{
+ struct resource *res;
+ phys_addr_t phys;
+ int i;
+
+ for (i = 0; i < dax_dev->num_resources; i++) {
+ res = &dax_dev->res[i];
+ phys = pgoff * PAGE_SIZE + res->start;
+ if (phys >= res->start && phys <= res->end)
+ break;
+ pgoff -= PHYS_PFN(resource_size(res));
+ }
+
+ if (i < dax_dev->num_resources) {
+ res = &dax_dev->res[i];
+ if (phys + size - 1 <= res->end)
+ return phys;
+ }
+
+ return -1;
+}
+
+static int __dax_dev_fault(struct address_space *mapping, struct device *dev,
+ struct vm_area_struct *vma, struct vm_fault *vmf)
+{
+ unsigned long vaddr = (unsigned long)vmf->virtual_address;
+ struct dax_dev *dax_dev = to_dax_dev(dev);
+ struct dax_region *dax_region;
+ phys_addr_t phys;
+ pfn_t pfn;
+ int rc;
+
+ if (!dax_dev)
+ return VM_FAULT_SIGBUS;
+
+ if (dax_dev_check_vma(dev, vma, __func__))
+ return VM_FAULT_SIGBUS;
+
+ dax_region = dax_dev->region;
+ if (dax_region->align > PAGE_SIZE) {
+ dev_dbg(dev, "%s: alignment > fault size\n", __func__);
+ return VM_FAULT_SIGBUS;
+ }
+
+ phys = pgoff_to_phys(dax_dev, vmf->pgoff, PAGE_SIZE);
+ if (phys == -1) {
+ dev_dbg(dev, "%s: phys_to_pgoff(%#lx) failed\n", __func__,
+ vmf->pgoff);
+ return VM_FAULT_SIGBUS;
+ }
+
+ pfn = phys_to_pfn_t(phys, dax_region->pfn_flags);
+
+ i_mmap_lock_read(mapping);
+ rc = vm_insert_mixed(vma, vaddr, pfn);
+ i_mmap_unlock_read(mapping);
+
+ if (rc == -ENOMEM)
+ return VM_FAULT_OOM;
+ if (rc < 0 && rc != -EBUSY)
+ return VM_FAULT_SIGBUS;
+
+ return VM_FAULT_NOPAGE;
+}
+
+static int dax_dev_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
+{
+ int rc;
+ struct file *filp = vma->vm_file;
+ struct device *dev = filp->private_data;
+ struct address_space *mapping = filp->f_mapping;
+
+ dev_dbg(dev, "%s: %s (%#lx - %#lx)\n", __func__,
+ (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read",
+ vma->vm_start, vma->vm_end);
+ device_lock(dev);
+ rc = __dax_dev_fault(mapping, dev, vma, vmf);
+ device_unlock(dev);
+
+ return rc;
+}
+
+static int __dax_dev_pmd_fault(struct address_space *mapping,
+ struct device *dev, struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmd, unsigned int flags)
+{
+ struct dax_dev *dax_dev = to_dax_dev(dev);
+ unsigned long pmd_addr = addr & PMD_MASK;
+ struct dax_region *dax_region;
+ phys_addr_t phys;
+ pgoff_t pgoff;
+ pfn_t pfn;
+ int rc;
+
+ if (!dax_dev)
+ return VM_FAULT_SIGBUS;
+
+ if (dax_dev_check_vma(dev, vma, __func__))
+ return VM_FAULT_SIGBUS;
+
+ dax_region = dax_dev->region;
+ if (dax_region->align > HPAGE_SIZE) {
+ dev_dbg(dev, "%s: alignment > fault size\n", __func__);
+ return VM_FAULT_SIGBUS;
+ }
+
+ /* dax pmd mappings require pfn_t_devmap() */
+ if ((dax_region->pfn_flags & (PFN_DEV|PFN_MAP)) != (PFN_DEV|PFN_MAP)) {
+ dev_dbg(dev, "%s: alignment > fault size\n", __func__);
+ return VM_FAULT_SIGBUS;
+ }
+
+ pgoff = linear_page_index(vma, pmd_addr);
+ phys = pgoff_to_phys(dax_dev, pgoff, PAGE_SIZE);
+ if (phys == -1) {
+ dev_dbg(dev, "%s: phys_to_pgoff(%#lx) failed\n", __func__,
+ pgoff);
+ return VM_FAULT_SIGBUS;
+ }
+
+ pfn = phys_to_pfn_t(phys, dax_region->pfn_flags);
+
+ i_mmap_lock_read(mapping);
+ rc = vmf_insert_pfn_pmd(vma, addr, pmd, pfn,
+ flags & FAULT_FLAG_WRITE);
+ i_mmap_unlock_read(mapping);
+
+ return rc;
+}
+
+static int dax_dev_pmd_fault(struct vm_area_struct *vma, unsigned long addr,
+ pmd_t *pmd, unsigned int flags)
+{
+ int rc;
+ struct file *filp = vma->vm_file;
+ struct device *dev = filp->private_data;
+ struct address_space *mapping = filp->f_mapping;
+
+ dev_dbg(dev, "%s: %s (%#lx - %#lx)\n", __func__,
+ (flags & FAULT_FLAG_WRITE) ? "write" : "read",
+ vma->vm_start, vma->vm_end);
+ device_lock(dev);
+ rc = __dax_dev_pmd_fault(mapping, dev, vma, addr, pmd, flags);
+ device_unlock(dev);
+
+ return rc;
+}
+
+static void dax_dev_vm_open(struct vm_area_struct *vma)
+{
+ struct file *filp = vma->vm_file;
+ struct device *dev = filp->private_data;
+
+ dev_dbg(dev, "%s\n", __func__);
+ get_device(dev);
+}
+
+static void dax_dev_vm_close(struct vm_area_struct *vma)
+{
+ struct file *filp = vma->vm_file;
+ struct device *dev = filp->private_data;
+
+ dev_dbg(dev, "%s\n", __func__);
+ put_device(dev);
+}
+
+static const struct vm_operations_struct dax_dev_vm_ops = {
+ .fault = dax_dev_fault,
+ .pmd_fault = dax_dev_pmd_fault,
+ .open = dax_dev_vm_open,
+ .close = dax_dev_vm_close,
+};
+
+static int dax_dev_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+ struct device *dev = filp->private_data;
+ int rc;
+
+ dev_dbg(dev, "%s\n", __func__);
+
+ device_lock(dev);
+ rc = dax_dev_check_vma(dev, vma, __func__);
+ device_unlock(dev);
+ if (rc)
+ return rc;
+
+ get_device(dev);
+ vma->vm_ops = &dax_dev_vm_ops;
+ vma->vm_flags |= VM_MIXEDMAP | VM_HUGEPAGE;
+ return 0;
+
+}
+
static const struct file_operations dax_fops = {
.llseek = noop_llseek,
.owner = THIS_MODULE,
+ .open = dax_dev_open,
+ .release = dax_dev_release,
+ .get_unmapped_area = dax_dev_get_unmapped_area,
+ .mmap = dax_dev_mmap,
};
static int __init dax_init(void)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 86f9f8b82f8e..52ea012d8a80 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1013,6 +1013,7 @@ int vmf_insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
insert_pfn_pmd(vma, addr, pmd, pfn, pgprot, write);
return VM_FAULT_NOPAGE;
}
+EXPORT_SYMBOL_GPL(vmf_insert_pfn_pmd);
static void touch_pmd(struct vm_area_struct *vma, unsigned long addr,
pmd_t *pmd)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 19d0d08b396f..b14e98129b07 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -624,6 +624,7 @@ pgoff_t linear_hugepage_index(struct vm_area_struct *vma,
{
return vma_hugecache_offset(hstate_vma(vma), vma, address);
}
+EXPORT_SYMBOL_GPL(linear_hugepage_index);
/*
* Return the size of the pages allocated when backing a VMA. In the majority
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 7/7] Revert "block: enable dax for raw block devices"
2016-05-08 22:35 [PATCH 0/7] "Device DAX" for persistent memory Dan Williams
` (5 preceding siblings ...)
2016-05-08 22:35 ` [PATCH 6/7] /dev/dax, core: file operations and dax-mmap Dan Williams
@ 2016-05-08 22:35 ` Dan Williams
2016-05-09 12:57 ` [PATCH 0/7] "Device DAX" for persistent memory Christoph Hellwig
7 siblings, 0 replies; 11+ messages in thread
From: Dan Williams @ 2016-05-08 22:35 UTC (permalink / raw)
To: linux-nvdimm
Cc: Jan Kara, Dave Chinner, linux-kernel, linux-block, Jeff Moyer,
Ross Zwisler, Andrew Morton, Christoph Hellwig
This reverts commit 5a023cdba50c5f5f2bc351783b3131699deb3937.
The functionality is superseded by the new "Device DAX" facility.
Cc: Jeff Moyer <jmoyer@redhat.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Dave Chinner <david@fromorbit.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
Cc: Jan Kara <jack@suse.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
block/ioctl.c | 32 ----------------
fs/block_dev.c | 96 ++++++++++++++---------------------------------
include/linux/fs.h | 8 ----
include/uapi/linux/fs.h | 1
4 files changed, 29 insertions(+), 108 deletions(-)
diff --git a/block/ioctl.c b/block/ioctl.c
index 4ff1f92f89ca..698c7933d582 100644
--- a/block/ioctl.c
+++ b/block/ioctl.c
@@ -407,35 +407,6 @@ static inline int is_unrecognized_ioctl(int ret)
ret == -ENOIOCTLCMD;
}
-#ifdef CONFIG_FS_DAX
-bool blkdev_dax_capable(struct block_device *bdev)
-{
- struct gendisk *disk = bdev->bd_disk;
-
- if (!disk->fops->direct_access)
- return false;
-
- /*
- * If the partition is not aligned on a page boundary, we can't
- * do dax I/O to it.
- */
- if ((bdev->bd_part->start_sect % (PAGE_SIZE / 512))
- || (bdev->bd_part->nr_sects % (PAGE_SIZE / 512)))
- return false;
-
- /*
- * If the device has known bad blocks, force all I/O through the
- * driver / page cache.
- *
- * TODO: support finer grained dax error handling
- */
- if (disk->bb && disk->bb->count)
- return false;
-
- return true;
-}
-#endif
-
static int blkdev_flushbuf(struct block_device *bdev, fmode_t mode,
unsigned cmd, unsigned long arg)
{
@@ -598,9 +569,6 @@ int blkdev_ioctl(struct block_device *bdev, fmode_t mode, unsigned cmd,
case BLKTRACESETUP:
case BLKTRACETEARDOWN:
return blk_trace_ioctl(bdev, cmd, argp);
- case BLKDAXGET:
- return put_int(arg, !!(bdev->bd_inode->i_flags & S_DAX));
- break;
case IOC_PR_REGISTER:
return blkdev_pr_register(bdev, argp);
case IOC_PR_RESERVE:
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 20a2c02b77c4..36ee10ca503e 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -29,6 +29,7 @@
#include <linux/log2.h>
#include <linux/cleancache.h>
#include <linux/dax.h>
+#include <linux/badblocks.h>
#include <asm/uaccess.h>
#include "internal.h"
@@ -1159,6 +1160,33 @@ void bd_set_size(struct block_device *bdev, loff_t size)
}
EXPORT_SYMBOL(bd_set_size);
+static bool blkdev_dax_capable(struct block_device *bdev)
+{
+ struct gendisk *disk = bdev->bd_disk;
+
+ if (!disk->fops->direct_access || !IS_ENABLED(CONFIG_FS_DAX))
+ return false;
+
+ /*
+ * If the partition is not aligned on a page boundary, we can't
+ * do dax I/O to it.
+ */
+ if ((bdev->bd_part->start_sect % (PAGE_SIZE / 512))
+ || (bdev->bd_part->nr_sects % (PAGE_SIZE / 512)))
+ return false;
+
+ /*
+ * If the device has known bad blocks, force all I/O through the
+ * driver / page cache.
+ *
+ * TODO: support finer grained dax error handling
+ */
+ if (disk->bb && disk->bb->count)
+ return false;
+
+ return true;
+}
+
static void __blkdev_put(struct block_device *bdev, fmode_t mode, int for_part);
/*
@@ -1724,79 +1752,13 @@ static const struct address_space_operations def_blk_aops = {
.is_dirty_writeback = buffer_check_dirty_writeback,
};
-#ifdef CONFIG_FS_DAX
-/*
- * In the raw block case we do not need to contend with truncation nor
- * unwritten file extents. Without those concerns there is no need for
- * additional locking beyond the mmap_sem context that these routines
- * are already executing under.
- *
- * Note, there is no protection if the block device is dynamically
- * resized (partition grow/shrink) during a fault. A stable block device
- * size is already not enforced in the blkdev_direct_IO path.
- *
- * For DAX, it is the responsibility of the block device driver to
- * ensure the whole-disk device size is stable while requests are in
- * flight.
- *
- * Finally, unlike the filemap_page_mkwrite() case there is no
- * filesystem superblock to sync against freezing. We still include a
- * pfn_mkwrite callback for dax drivers to receive write fault
- * notifications.
- */
-static int blkdev_dax_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
-{
- return __dax_fault(vma, vmf, blkdev_get_block, NULL);
-}
-
-static int blkdev_dax_pfn_mkwrite(struct vm_area_struct *vma,
- struct vm_fault *vmf)
-{
- return dax_pfn_mkwrite(vma, vmf);
-}
-
-static int blkdev_dax_pmd_fault(struct vm_area_struct *vma, unsigned long addr,
- pmd_t *pmd, unsigned int flags)
-{
- return __dax_pmd_fault(vma, addr, pmd, flags, blkdev_get_block, NULL);
-}
-
-static const struct vm_operations_struct blkdev_dax_vm_ops = {
- .fault = blkdev_dax_fault,
- .pmd_fault = blkdev_dax_pmd_fault,
- .pfn_mkwrite = blkdev_dax_pfn_mkwrite,
-};
-
-static const struct vm_operations_struct blkdev_default_vm_ops = {
- .fault = filemap_fault,
- .map_pages = filemap_map_pages,
-};
-
-static int blkdev_mmap(struct file *file, struct vm_area_struct *vma)
-{
- struct inode *bd_inode = bdev_file_inode(file);
-
- file_accessed(file);
- if (IS_DAX(bd_inode)) {
- vma->vm_ops = &blkdev_dax_vm_ops;
- vma->vm_flags |= VM_MIXEDMAP | VM_HUGEPAGE;
- } else {
- vma->vm_ops = &blkdev_default_vm_ops;
- }
-
- return 0;
-}
-#else
-#define blkdev_mmap generic_file_mmap
-#endif
-
const struct file_operations def_blk_fops = {
.open = blkdev_open,
.release = blkdev_close,
.llseek = block_llseek,
.read_iter = blkdev_read_iter,
.write_iter = blkdev_write_iter,
- .mmap = blkdev_mmap,
+ .mmap = generic_file_mmap,
.fsync = blkdev_fsync,
.unlocked_ioctl = block_ioctl,
#ifdef CONFIG_COMPAT
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 70e61b58baaf..8363a10660f6 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -2320,14 +2320,6 @@ extern struct super_block *freeze_bdev(struct block_device *);
extern void emergency_thaw_all(void);
extern int thaw_bdev(struct block_device *bdev, struct super_block *sb);
extern int fsync_bdev(struct block_device *);
-#ifdef CONFIG_FS_DAX
-extern bool blkdev_dax_capable(struct block_device *bdev);
-#else
-static inline bool blkdev_dax_capable(struct block_device *bdev)
-{
- return false;
-}
-#endif
extern struct super_block *blockdev_superblock;
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index a079d50376e1..fbff8b28aa35 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -222,7 +222,6 @@ struct fsxattr {
#define BLKSECDISCARD _IO(0x12,125)
#define BLKROTATIONAL _IO(0x12,126)
#define BLKZEROOUT _IO(0x12,127)
-#define BLKDAXGET _IO(0x12,129)
#define BMAP_IOCTL 1 /* obsolete - kept for compatibility */
#define FIBMAP _IO(0x00,1) /* bmap access */
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH 0/7] "Device DAX" for persistent memory
2016-05-08 22:35 [PATCH 0/7] "Device DAX" for persistent memory Dan Williams
` (6 preceding siblings ...)
2016-05-08 22:35 ` [PATCH 7/7] Revert "block: enable dax for raw block devices" Dan Williams
@ 2016-05-09 12:57 ` Christoph Hellwig
2016-05-09 20:40 ` Dan Williams
7 siblings, 1 reply; 11+ messages in thread
From: Christoph Hellwig @ 2016-05-09 12:57 UTC (permalink / raw)
To: Dan Williams
Cc: linux-nvdimm, Dave Hansen, Dave Chinner, linux-kernel,
linux-block, Jeff Moyer, Jan Kara, Andrew Morton, Ross Zwisler
On Sun, May 08, 2016 at 03:35:10PM -0700, Dan Williams wrote:
> Device DAX is the device-centric analogue of Filesystem DAX
> (CONFIG_FS_DAX). It allows memory ranges to be allocated and mapped
> without need of an intervening file system or being bound to block
> device semantics. Device DAX is strict and predictable. Specifically
> this interface:
Can you explain the "why" a little more?
And please, if you decide to Cc me on some of the patches do it for the
whole series or none of it, but never just for some patches as that make
the cc pretty pointless.
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH 0/7] "Device DAX" for persistent memory
2016-05-09 12:57 ` [PATCH 0/7] "Device DAX" for persistent memory Christoph Hellwig
@ 2016-05-09 20:40 ` Dan Williams
0 siblings, 0 replies; 11+ messages in thread
From: Dan Williams @ 2016-05-09 20:40 UTC (permalink / raw)
To: Christoph Hellwig
Cc: linux-nvdimm, Dave Hansen, Dave Chinner, linux-kernel,
linux-block, Jeff Moyer, Jan Kara, Andrew Morton, Ross Zwisler
On Mon, May 9, 2016 at 5:57 AM, Christoph Hellwig <hch@infradead.org> wrote:
> On Sun, May 08, 2016 at 03:35:10PM -0700, Dan Williams wrote:
>> Device DAX is the device-centric analogue of Filesystem DAX
>> (CONFIG_FS_DAX). It allows memory ranges to be allocated and mapped
>> without need of an intervening file system or being bound to block
>> device semantics. Device DAX is strict and predictable. Specifically
>> this interface:
>
> Can you explain the "why" a little more?
1/ As I mentioned at LSF [1] we're starting to see platforms with
performance and feature differentiated memory ranges. Environments
like high-performance-computing and usages like in-memory databases
want 100% exclusive allocation of a memory range with zero conflicting
kernel/metadata allocations. For dedicated applications of high
bandwidth or low latency memory device-DAX provides a predictable
direct map mechanism.
Note that this is only for the small number of "crazy" applications
that are willing to re-write to get every bit of performance. For
everyone else we, Dave Hansen and I, are looking to add a mechanism to
hot-plug device-DAX ranges into the mm to get general memory
management services (oversubscribe / migration, etc) with the
understanding that it may sacrifice some predictability.
2/ For persistent memory there are similar applications that are
willing to re-write to take full advantage of byte-addressable
persistence. This mechanism satisfies those usages that only need a
pre-allocated file to mmap.
3/ It answers Dave Chinner's call to start thinking about pmem-native
solutions. Device DAX specifically avoids block-device and file
system conflicts.
> And please, if you decide to Cc me on some of the patches do it for the
> whole series or none of it, but never just for some patches as that make
> the cc pretty pointless.
Sorry, you've told me this before. I'll update my scripts to
auto-include you on the whole series if you ever appear in the cc of
any patch in the set.
[1]: https://lwn.net/Articles/685107/
^ permalink raw reply [flat|nested] 11+ messages in thread