* [v2 0/2] "Hotremove" persistent memory @ 2019-04-21 1:44 Pavel Tatashin 2019-04-21 1:44 ` [v2 1/2] device-dax: fix memory and resource leak if hotplug fails Pavel Tatashin 2019-04-21 1:44 ` [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM Pavel Tatashin 0 siblings, 2 replies; 9+ messages in thread From: Pavel Tatashin @ 2019-04-21 1:44 UTC (permalink / raw) To: pasha.tatashin, jmorris, sashal, linux-kernel, linux-mm, linux-nvdimm, akpm, mhocko, dave.hansen, dan.j.williams, keith.busch, vishal.l.verma, dave.jiang, zwisler, thomas.lendacky, ying.huang, fengguang.wu, bp, bhelgaas, baiyaowei, tiwai, jglisse Changelog: v2 - Dan Williams mentioned that drv->remove() return is ignored by unbind. Unbind always succeeds. Because we cannot guarantee that memory can be offlined from the driver, don't even attempt to do so. Simply check that every section is offlined beforehand and only then proceed with removing dax memory. --- Recently, adding a persistent memory to be used like a regular RAM was added to Linux. This work extends this functionality to also allow hot removing persistent memory. We (Microsoft) have an important use case for this functionality. The requirement is for physical machines with small amount of RAM (~8G) to be able to reboot in a very short period of time (<1s). Yet, there is a userland state that is expensive to recreate (~2G). The solution is to boot machines with 2G preserved for persistent memory. Copy the state, and hotadd the persistent memory so machine still has all 8G available for runtime. Before reboot, offline and hotremove device-dax 2G, copy the memory that is needed to be preserved to pmem0 device, and reboot. The series of operations look like this: 1. After boot restore /dev/pmem0 to ramdisk to be consumed by apps. and free ramdisk. 2. Convert raw pmem0 to devdax ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f 3. Hotadd to System RAM echo dax0.0 > /sys/bus/dax/drivers/device_dax/unbind echo dax0.0 > /sys/bus/dax/drivers/kmem/new_id echo online_movable > /sys/devices/system/memoryXXX/state 4. Before reboot hotremove device-dax memory from System RAM echo offline > /sys/devices/system/memoryXXX/state echo dax0.0 > /sys/bus/dax/drivers/kmem/unbind 5. Create raw pmem0 device ndctl create-namespace --mode raw -e namespace0.0 -f 6. Copy the state that was stored by apps to ramdisk to pmem device 7. Do kexec reboot or reboot through firmware if firmware does not zero memory in pmem0 region (These machines have only regular volatile memory). So to have pmem0 device either memmap kernel parameter is used, or devices nodes in dtb are specified. Pavel Tatashin (2): device-dax: fix memory and resource leak if hotplug fails device-dax: "Hotremove" persistent memory that is used like normal RAM drivers/dax/dax-private.h | 2 + drivers/dax/kmem.c | 96 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 93 insertions(+), 5 deletions(-) -- 2.21.0 _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply [flat|nested] 9+ messages in thread
* [v2 1/2] device-dax: fix memory and resource leak if hotplug fails 2019-04-21 1:44 [v2 0/2] "Hotremove" persistent memory Pavel Tatashin @ 2019-04-21 1:44 ` Pavel Tatashin 2019-04-21 1:44 ` [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM Pavel Tatashin 1 sibling, 0 replies; 9+ messages in thread From: Pavel Tatashin @ 2019-04-21 1:44 UTC (permalink / raw) To: pasha.tatashin, jmorris, sashal, linux-kernel, linux-mm, linux-nvdimm, akpm, mhocko, dave.hansen, dan.j.williams, keith.busch, vishal.l.verma, dave.jiang, zwisler, thomas.lendacky, ying.huang, fengguang.wu, bp, bhelgaas, baiyaowei, tiwai, jglisse When add_memory() function fails, the resource and the memory should be freed. Fixes: c221c0b0308f ("device-dax: "Hotplug" persistent memory for use like normal RAM") Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com> --- drivers/dax/kmem.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c index a02318c6d28a..4c0131857133 100644 --- a/drivers/dax/kmem.c +++ b/drivers/dax/kmem.c @@ -66,8 +66,11 @@ int dev_dax_kmem_probe(struct device *dev) new_res->name = dev_name(dev); rc = add_memory(numa_node, new_res->start, resource_size(new_res)); - if (rc) + if (rc) { + release_resource(new_res); + kfree(new_res); return rc; + } return 0; } -- 2.21.0 _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM 2019-04-21 1:44 [v2 0/2] "Hotremove" persistent memory Pavel Tatashin 2019-04-21 1:44 ` [v2 1/2] device-dax: fix memory and resource leak if hotplug fails Pavel Tatashin @ 2019-04-21 1:44 ` Pavel Tatashin 2019-04-24 20:55 ` David Hildenbrand 1 sibling, 1 reply; 9+ messages in thread From: Pavel Tatashin @ 2019-04-21 1:44 UTC (permalink / raw) To: pasha.tatashin, jmorris, sashal, linux-kernel, linux-mm, linux-nvdimm, akpm, mhocko, dave.hansen, dan.j.williams, keith.busch, vishal.l.verma, dave.jiang, zwisler, thomas.lendacky, ying.huang, fengguang.wu, bp, bhelgaas, baiyaowei, tiwai, jglisse It is now allowed to use persistent memory like a regular RAM, but currently there is no way to remove this memory until machine is rebooted. This work expands the functionality to also allows hotremoving previously hotplugged persistent memory, and recover the device for use for other purposes. To hotremove persistent memory, the management software must first offline all memory blocks of dax region, and than unbind it from device-dax/kmem driver. So, operations should look like this: echo offline > echo offline > /sys/devices/system/memory/memoryN/state ... echo dax0.0 > /sys/bus/dax/drivers/kmem/unbind Note: if unbind is done without offlining memory beforehand, it won't be possible to do dax0.0 hotremove, and dax's memory is going to be part of System RAM until reboot. Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com> --- drivers/dax/dax-private.h | 2 + drivers/dax/kmem.c | 91 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 89 insertions(+), 4 deletions(-) diff --git a/drivers/dax/dax-private.h b/drivers/dax/dax-private.h index a45612148ca0..999aaf3a29b3 100644 --- a/drivers/dax/dax-private.h +++ b/drivers/dax/dax-private.h @@ -53,6 +53,7 @@ struct dax_region { * @pgmap - pgmap for memmap setup / lifetime (driver owned) * @ref: pgmap reference count (driver owned) * @cmp: @ref final put completion (driver owned) + * @dax_mem_res: physical address range of hotadded DAX memory */ struct dev_dax { struct dax_region *region; @@ -62,6 +63,7 @@ struct dev_dax { struct dev_pagemap pgmap; struct percpu_ref ref; struct completion cmp; + struct resource *dax_kmem_res; }; static inline struct dev_dax *to_dev_dax(struct device *dev) diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c index 4c0131857133..d4896b281036 100644 --- a/drivers/dax/kmem.c +++ b/drivers/dax/kmem.c @@ -71,21 +71,104 @@ int dev_dax_kmem_probe(struct device *dev) kfree(new_res); return rc; } + dev_dax->dax_kmem_res = new_res; return 0; } +#ifdef CONFIG_MEMORY_HOTREMOVE +/* + * Check that device-dax's memory_blocks are offline. If a memory_block is not + * offline a warning is printed and an error is returned. dax hotremove can + * succeed only when every memory_block is offlined beforehand. + */ +static int +offline_memblock_cb(struct memory_block *mem, void *arg) +{ + struct device *mem_dev = &mem->dev; + bool is_offline; + + device_lock(mem_dev); + is_offline = mem_dev->offline; + device_unlock(mem_dev); + + if (!is_offline) { + struct device *dev = (struct device *)arg; + unsigned long spfn = section_nr_to_pfn(mem->start_section_nr); + unsigned long epfn = section_nr_to_pfn(mem->end_section_nr); + phys_addr_t spa = spfn << PAGE_SHIFT; + phys_addr_t epa = epfn << PAGE_SHIFT; + + dev_warn(dev, "memory block [%pa-%pa] is not offline\n", + &spa, &epa); + + return -EBUSY; + } + + return 0; +} + +static int dev_dax_kmem_remove(struct device *dev) +{ + struct dev_dax *dev_dax = to_dev_dax(dev); + struct resource *res = dev_dax->dax_kmem_res; + resource_size_t kmem_start; + resource_size_t kmem_size; + unsigned long start_pfn; + unsigned long end_pfn; + int rc; + + /* + * dax kmem resource does not exist, means memory was never hotplugged. + * So, nothing to do here. + */ + if (!res) + return 0; + + kmem_start = res->start; + kmem_size = resource_size(res); + start_pfn = kmem_start >> PAGE_SHIFT; + end_pfn = start_pfn + (kmem_size >> PAGE_SHIFT) - 1; + + /* + * Walk and check that every singe memory_block of dax region is + * offline + */ + lock_device_hotplug(); + rc = walk_memory_range(start_pfn, end_pfn, dev, offline_memblock_cb); + unlock_device_hotplug(); + + /* + * If admin has not offlined memory beforehand, we cannot hotremove dax. + * Unfortunately, because unbind will still succeed there is no way for + * user to hotremove dax after this. + */ + if (rc) + return rc; + + /* Hotremove memory, cannot fail because memory is already offlined */ + remove_memory(dev_dax->target_node, kmem_start, kmem_size); + + /* Release and free dax resources */ + release_resource(res); + kfree(res); + dev_dax->dax_kmem_res = NULL; + + return 0; +} +#else static int dev_dax_kmem_remove(struct device *dev) { /* - * Purposely leak the request_mem_region() for the device-dax - * range and return '0' to ->remove() attempts. The removal of - * the device from the driver always succeeds, but the region - * is permanently pinned as reserved by the unreleased + * Without hotremove purposely leak the request_mem_region() for the + * device-dax range and return '0' to ->remove() attempts. The removal + * of the device from the driver always succeeds, but the region is + * permanently pinned as reserved by the unreleased * request_mem_region(). */ return 0; } +#endif /* CONFIG_MEMORY_HOTREMOVE */ static struct dax_device_driver device_dax_kmem_driver = { .drv = { -- 2.21.0 _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM 2019-04-21 1:44 ` [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM Pavel Tatashin @ 2019-04-24 20:55 ` David Hildenbrand 2019-04-24 21:02 ` Dan Williams 0 siblings, 1 reply; 9+ messages in thread From: David Hildenbrand @ 2019-04-24 20:55 UTC (permalink / raw) To: Pavel Tatashin, jmorris, sashal, linux-kernel, linux-mm, linux-nvdimm, akpm, mhocko, dave.hansen, dan.j.williams, keith.busch, vishal.l.verma, dave.jiang, zwisler, thomas.lendacky, ying.huang, fengguang.wu, bp, bhelgaas, baiyaowei, tiwai, jglisse On 21.04.19 03:44, Pavel Tatashin wrote: > It is now allowed to use persistent memory like a regular RAM, but > currently there is no way to remove this memory until machine is > rebooted. > > This work expands the functionality to also allows hotremoving > previously hotplugged persistent memory, and recover the device for use > for other purposes. > > To hotremove persistent memory, the management software must first > offline all memory blocks of dax region, and than unbind it from > device-dax/kmem driver. So, operations should look like this: > > echo offline > echo offline > /sys/devices/system/memory/memoryN/state > ... > echo dax0.0 > /sys/bus/dax/drivers/kmem/unbind > > Note: if unbind is done without offlining memory beforehand, it won't be > possible to do dax0.0 hotremove, and dax's memory is going to be part of > System RAM until reboot. > > Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com> > --- > drivers/dax/dax-private.h | 2 + > drivers/dax/kmem.c | 91 +++++++++++++++++++++++++++++++++++++-- > 2 files changed, 89 insertions(+), 4 deletions(-) > > diff --git a/drivers/dax/dax-private.h b/drivers/dax/dax-private.h > index a45612148ca0..999aaf3a29b3 100644 > --- a/drivers/dax/dax-private.h > +++ b/drivers/dax/dax-private.h > @@ -53,6 +53,7 @@ struct dax_region { > * @pgmap - pgmap for memmap setup / lifetime (driver owned) > * @ref: pgmap reference count (driver owned) > * @cmp: @ref final put completion (driver owned) > + * @dax_mem_res: physical address range of hotadded DAX memory > */ > struct dev_dax { > struct dax_region *region; > @@ -62,6 +63,7 @@ struct dev_dax { > struct dev_pagemap pgmap; > struct percpu_ref ref; > struct completion cmp; > + struct resource *dax_kmem_res; > }; > > static inline struct dev_dax *to_dev_dax(struct device *dev) > diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c > index 4c0131857133..d4896b281036 100644 > --- a/drivers/dax/kmem.c > +++ b/drivers/dax/kmem.c > @@ -71,21 +71,104 @@ int dev_dax_kmem_probe(struct device *dev) > kfree(new_res); > return rc; > } > + dev_dax->dax_kmem_res = new_res; > > return 0; > } > > +#ifdef CONFIG_MEMORY_HOTREMOVE > +/* > + * Check that device-dax's memory_blocks are offline. If a memory_block is not > + * offline a warning is printed and an error is returned. dax hotremove can > + * succeed only when every memory_block is offlined beforehand. > + */ > +static int > +offline_memblock_cb(struct memory_block *mem, void *arg) Function name suggests that you are actually trying to offline memory here. Maybe check_memblocks_offline_cb(), just like we have in mm/memory_hotplug.c. > +{ > + struct device *mem_dev = &mem->dev; > + bool is_offline; > + > + device_lock(mem_dev); > + is_offline = mem_dev->offline; > + device_unlock(mem_dev); > + > + if (!is_offline) { > + struct device *dev = (struct device *)arg; > + unsigned long spfn = section_nr_to_pfn(mem->start_section_nr); > + unsigned long epfn = section_nr_to_pfn(mem->end_section_nr); > + phys_addr_t spa = spfn << PAGE_SHIFT; > + phys_addr_t epa = epfn << PAGE_SHIFT; > + > + dev_warn(dev, "memory block [%pa-%pa] is not offline\n", > + &spa, &epa); > + > + return -EBUSY; > + } > + > + return 0; > +} > + > +static int dev_dax_kmem_remove(struct device *dev) > +{ > + struct dev_dax *dev_dax = to_dev_dax(dev); > + struct resource *res = dev_dax->dax_kmem_res; > + resource_size_t kmem_start; > + resource_size_t kmem_size; > + unsigned long start_pfn; > + unsigned long end_pfn; > + int rc; > + > + /* > + * dax kmem resource does not exist, means memory was never hotplugged. > + * So, nothing to do here. > + */ > + if (!res) > + return 0; > + > + kmem_start = res->start; > + kmem_size = resource_size(res); > + start_pfn = kmem_start >> PAGE_SHIFT; > + end_pfn = start_pfn + (kmem_size >> PAGE_SHIFT) - 1; > + > + /* > + * Walk and check that every singe memory_block of dax region is > + * offline > + */ > + lock_device_hotplug(); > + rc = walk_memory_range(start_pfn, end_pfn, dev, offline_memblock_cb); > + unlock_device_hotplug(); > + > + /* > + * If admin has not offlined memory beforehand, we cannot hotremove dax. > + * Unfortunately, because unbind will still succeed there is no way for > + * user to hotremove dax after this. > + */ > + if (rc) > + return rc; Can't it happen that there is a race between you checking if memory is offline and an admin onlining memory again? maybe pull the remove_memory() into the locked region, using __remove_memory() instead. > + > + /* Hotremove memory, cannot fail because memory is already offlined */ > + remove_memory(dev_dax->target_node, kmem_start, kmem_size); > + > + /* Release and free dax resources */ > + release_resource(res); > + kfree(res); > + dev_dax->dax_kmem_res = NULL; > + > + return 0; > +} > +#else > static int dev_dax_kmem_remove(struct device *dev) > { > /* > - * Purposely leak the request_mem_region() for the device-dax > - * range and return '0' to ->remove() attempts. The removal of > - * the device from the driver always succeeds, but the region > - * is permanently pinned as reserved by the unreleased > + * Without hotremove purposely leak the request_mem_region() for the > + * device-dax range and return '0' to ->remove() attempts. The removal > + * of the device from the driver always succeeds, but the region is > + * permanently pinned as reserved by the unreleased > * request_mem_region(). > */ > return 0; > } > +#endif /* CONFIG_MEMORY_HOTREMOVE */ > > static struct dax_device_driver device_dax_kmem_driver = { > .drv = { > -- Thanks, David / dhildenb ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM 2019-04-24 20:55 ` David Hildenbrand @ 2019-04-24 21:02 ` Dan Williams 2019-04-24 21:34 ` Pavel Tatashin 0 siblings, 1 reply; 9+ messages in thread From: Dan Williams @ 2019-04-24 21:02 UTC (permalink / raw) To: David Hildenbrand Cc: Sasha Levin, Tom Lendacky, Michal Hocko, Pavel Tatashin, linux-nvdimm, Takashi Iwai, Dave Hansen, Huang, Ying <ying.huang@intel.com>, Linux Kernel Mailing List, James Morris, Linux MM, Jérôme Glisse, Borislav Petkov, Yaowei Bai, Ross Zwisler, Bjorn Helgaas, Andrew Morton, Fengguang Wu On Wed, Apr 24, 2019 at 1:55 PM David Hildenbrand <david@redhat.com> wrote: > > On 21.04.19 03:44, Pavel Tatashin wrote: > > It is now allowed to use persistent memory like a regular RAM, but > > currently there is no way to remove this memory until machine is > > rebooted. > > > > This work expands the functionality to also allows hotremoving > > previously hotplugged persistent memory, and recover the device for use > > for other purposes. > > > > To hotremove persistent memory, the management software must first > > offline all memory blocks of dax region, and than unbind it from > > device-dax/kmem driver. So, operations should look like this: > > > > echo offline > echo offline > /sys/devices/system/memory/memoryN/state > > ... > > echo dax0.0 > /sys/bus/dax/drivers/kmem/unbind > > > > Note: if unbind is done without offlining memory beforehand, it won't be > > possible to do dax0.0 hotremove, and dax's memory is going to be part of > > System RAM until reboot. > > > > Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com> > > --- > > drivers/dax/dax-private.h | 2 + > > drivers/dax/kmem.c | 91 +++++++++++++++++++++++++++++++++++++-- > > 2 files changed, 89 insertions(+), 4 deletions(-) > > > > diff --git a/drivers/dax/dax-private.h b/drivers/dax/dax-private.h > > index a45612148ca0..999aaf3a29b3 100644 > > --- a/drivers/dax/dax-private.h > > +++ b/drivers/dax/dax-private.h > > @@ -53,6 +53,7 @@ struct dax_region { > > * @pgmap - pgmap for memmap setup / lifetime (driver owned) > > * @ref: pgmap reference count (driver owned) > > * @cmp: @ref final put completion (driver owned) > > + * @dax_mem_res: physical address range of hotadded DAX memory > > */ > > struct dev_dax { > > struct dax_region *region; > > @@ -62,6 +63,7 @@ struct dev_dax { > > struct dev_pagemap pgmap; > > struct percpu_ref ref; > > struct completion cmp; > > + struct resource *dax_kmem_res; > > }; > > > > static inline struct dev_dax *to_dev_dax(struct device *dev) > > diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c > > index 4c0131857133..d4896b281036 100644 > > --- a/drivers/dax/kmem.c > > +++ b/drivers/dax/kmem.c > > @@ -71,21 +71,104 @@ int dev_dax_kmem_probe(struct device *dev) > > kfree(new_res); > > return rc; > > } > > + dev_dax->dax_kmem_res = new_res; > > > > return 0; > > } > > > > +#ifdef CONFIG_MEMORY_HOTREMOVE > > +/* > > + * Check that device-dax's memory_blocks are offline. If a memory_block is not > > + * offline a warning is printed and an error is returned. dax hotremove can > > + * succeed only when every memory_block is offlined beforehand. > > + */ > > +static int > > +offline_memblock_cb(struct memory_block *mem, void *arg) > > Function name suggests that you are actually trying to offline memory > here. Maybe check_memblocks_offline_cb(), just like we have in > mm/memory_hotplug.c. > > > +{ > > + struct device *mem_dev = &mem->dev; > > + bool is_offline; > > + > > + device_lock(mem_dev); > > + is_offline = mem_dev->offline; > > + device_unlock(mem_dev); > > + > > + if (!is_offline) { > > + struct device *dev = (struct device *)arg; > > + unsigned long spfn = section_nr_to_pfn(mem->start_section_nr); > > + unsigned long epfn = section_nr_to_pfn(mem->end_section_nr); > > + phys_addr_t spa = spfn << PAGE_SHIFT; > > + phys_addr_t epa = epfn << PAGE_SHIFT; > > + > > + dev_warn(dev, "memory block [%pa-%pa] is not offline\n", > > + &spa, &epa); > > + > > + return -EBUSY; > > + } > > + > > + return 0; > > +} > > + > > +static int dev_dax_kmem_remove(struct device *dev) > > +{ > > + struct dev_dax *dev_dax = to_dev_dax(dev); > > + struct resource *res = dev_dax->dax_kmem_res; > > + resource_size_t kmem_start; > > + resource_size_t kmem_size; > > + unsigned long start_pfn; > > + unsigned long end_pfn; > > + int rc; > > + > > + /* > > + * dax kmem resource does not exist, means memory was never hotplugged. > > + * So, nothing to do here. > > + */ > > + if (!res) > > + return 0; > > + > > + kmem_start = res->start; > > + kmem_size = resource_size(res); > > + start_pfn = kmem_start >> PAGE_SHIFT; > > + end_pfn = start_pfn + (kmem_size >> PAGE_SHIFT) - 1; > > + > > + /* > > + * Walk and check that every singe memory_block of dax region is > > + * offline > > + */ > > + lock_device_hotplug(); > > + rc = walk_memory_range(start_pfn, end_pfn, dev, offline_memblock_cb); > > + unlock_device_hotplug(); > > + > > + /* > > + * If admin has not offlined memory beforehand, we cannot hotremove dax. > > + * Unfortunately, because unbind will still succeed there is no way for > > + * user to hotremove dax after this. > > + */ > > + if (rc) > > + return rc; > > Can't it happen that there is a race between you checking if memory is > offline and an admin onlining memory again? maybe pull the > remove_memory() into the locked region, using __remove_memory() instead. I think the race is ok. The admin gets to keep the pieces of allowing racing updates to the state and the kernel will keep the range active until the next reboot. _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM 2019-04-24 21:02 ` Dan Williams @ 2019-04-24 21:34 ` Pavel Tatashin 2019-04-25 7:41 ` David Hildenbrand 0 siblings, 1 reply; 9+ messages in thread From: Pavel Tatashin @ 2019-04-24 21:34 UTC (permalink / raw) To: Dan Williams Cc: Sasha Levin, Tom Lendacky, Michal Hocko, linux-nvdimm, Takashi Iwai, David Hildenbrand, Huang, Ying <ying.huang@intel.com>, Linux Kernel Mailing List, James Morris, Linux MM, Jérôme Glisse, Dave Hansen, Borislav Petkov, Yaowei Bai, Ross Zwisler, Bjorn Helgaas, Andrew Morton, Fengguang Wu > > > +static int > > > +offline_memblock_cb(struct memory_block *mem, void *arg) > > > > Function name suggests that you are actually trying to offline memory > > here. Maybe check_memblocks_offline_cb(), just like we have in > > mm/memory_hotplug.c. Makes sense, I will rename to check_memblocks_offline_cb() > > > + lock_device_hotplug(); > > > + rc = walk_memory_range(start_pfn, end_pfn, dev, offline_memblock_cb); > > > + unlock_device_hotplug(); > > > + > > > + /* > > > + * If admin has not offlined memory beforehand, we cannot hotremove dax. > > > + * Unfortunately, because unbind will still succeed there is no way for > > > + * user to hotremove dax after this. > > > + */ > > > + if (rc) > > > + return rc; > > > > Can't it happen that there is a race between you checking if memory is > > offline and an admin onlining memory again? maybe pull the > > remove_memory() into the locked region, using __remove_memory() instead. > > I think the race is ok. The admin gets to keep the pieces of allowing > racing updates to the state and the kernel will keep the range active > until the next reboot. Thank you for noticing this. I will pull it into locking region. Because, __remove_memory() has this code: 1868 ret = walk_memory_range(PFN_DOWN(start), PFN_UP(start + size - 1), NULL, 1869 check_memblock_offlined_cb); 1870 if (ret) 1871 BUG(); Basically, panic if at least something is not offlined. Pasha _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM 2019-04-24 21:34 ` Pavel Tatashin @ 2019-04-25 7:41 ` David Hildenbrand 2019-04-25 12:30 ` Pavel Tatashin 0 siblings, 1 reply; 9+ messages in thread From: David Hildenbrand @ 2019-04-25 7:41 UTC (permalink / raw) To: Pavel Tatashin, Dan Williams Cc: Sasha Levin, Tom Lendacky, Michal Hocko, linux-nvdimm, Takashi Iwai, Dave Hansen, Huang, Ying, James Morris, Linux Kernel Mailing List, Linux MM, Jérôme Glisse, Borislav Petkov, Yaowei Bai, Ross Zwisler, Bjorn Helgaas, Andrew Morton, Fengguang Wu On 24.04.19 23:34, Pavel Tatashin wrote: >>>> +static int >>>> +offline_memblock_cb(struct memory_block *mem, void *arg) >>> >>> Function name suggests that you are actually trying to offline memory >>> here. Maybe check_memblocks_offline_cb(), just like we have in >>> mm/memory_hotplug.c. > > Makes sense, I will rename to check_memblocks_offline_cb() > >>>> + lock_device_hotplug(); >>>> + rc = walk_memory_range(start_pfn, end_pfn, dev, offline_memblock_cb); >>>> + unlock_device_hotplug(); >>>> + >>>> + /* >>>> + * If admin has not offlined memory beforehand, we cannot hotremove dax. >>>> + * Unfortunately, because unbind will still succeed there is no way for >>>> + * user to hotremove dax after this. >>>> + */ >>>> + if (rc) >>>> + return rc; >>> >>> Can't it happen that there is a race between you checking if memory is >>> offline and an admin onlining memory again? maybe pull the >>> remove_memory() into the locked region, using __remove_memory() instead. >> >> I think the race is ok. The admin gets to keep the pieces of allowing >> racing updates to the state and the kernel will keep the range active >> until the next reboot. > > Thank you for noticing this. I will pull it into locking region. > Because, __remove_memory() has this code: > > 1868 ret = walk_memory_range(PFN_DOWN(start), PFN_UP(start + size - 1), NULL, > 1869 check_memblock_offlined_cb); > 1870 if (ret) > 1871 BUG(); > Yes, also I think you can let go of the device_lock in check_memblocks_offline_cb, lock_device_hotplug() should take care of this (see Documentation/core-api/memory-hotplug.rst - "locking internals") Cheers! -- Thanks, David / dhildenb _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM 2019-04-25 7:41 ` David Hildenbrand @ 2019-04-25 12:30 ` Pavel Tatashin 2019-04-25 12:38 ` David Hildenbrand 0 siblings, 1 reply; 9+ messages in thread From: Pavel Tatashin @ 2019-04-25 12:30 UTC (permalink / raw) To: David Hildenbrand Cc: Dan Williams, James Morris, Sasha Levin, Linux Kernel Mailing List, Linux MM, linux-nvdimm, Andrew Morton, Michal Hocko, Dave Hansen, Keith Busch, Vishal L Verma, Dave Jiang, Ross Zwisler, Tom Lendacky, Huang, Ying, Fengguang Wu, Borislav Petkov, Bjorn Helgaas, Yaowei Bai, Takashi Iwai, Jérôme Glisse > > Yes, also I think you can let go of the device_lock in > check_memblocks_offline_cb, lock_device_hotplug() should take care of > this (see Documentation/core-api/memory-hotplug.rst - "locking internals") > Hi David, Thank you for your comments. I went through memory-hotplug.rst, and I still think that device_lock() is needed here. In this particular case it can be replaced with something like READ_ONCE(), but for simplicity it is better to have device_lock()/device_unlock() as this is not a performance critical code. I do not see any lock ordering issues with this code, as we are holding lock_device_hotplug() first that prevents userland from adding/removing memory during this check. https://soleen.com/source/xref/linux/arch/powerpc/platforms/powernv/memtrace.c?r=98fa15f3#248 Here we have a similar code: lock_device_hotplug(); online_mem_block(); device_online() device_lock(dev); Pasha ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM 2019-04-25 12:30 ` Pavel Tatashin @ 2019-04-25 12:38 ` David Hildenbrand 0 siblings, 0 replies; 9+ messages in thread From: David Hildenbrand @ 2019-04-25 12:38 UTC (permalink / raw) To: Pavel Tatashin Cc: Dan Williams, James Morris, Sasha Levin, Linux Kernel Mailing List, Linux MM, linux-nvdimm, Andrew Morton, Michal Hocko, Dave Hansen, Keith Busch, Vishal L Verma, Dave Jiang, Ross Zwisler, Tom Lendacky, Huang, Ying, Fengguang Wu, Borislav Petkov, Bjorn Helgaas, Yaowei Bai, Takashi Iwai, Jérôme Glisse On 25.04.19 14:30, Pavel Tatashin wrote: >> >> Yes, also I think you can let go of the device_lock in >> check_memblocks_offline_cb, lock_device_hotplug() should take care of >> this (see Documentation/core-api/memory-hotplug.rst - "locking internals") >> > Hi David, > > Thank you for your comments. I went through memory-hotplug.rst, and I > still think that device_lock() is needed here. In this particular case > it can be replaced with something like READ_ONCE(), but for simplicity > it is better to have device_lock()/device_unlock() as this is not a > performance critical code. > > I do not see any lock ordering issues with this code, as we are > holding lock_device_hotplug() first that prevents userland from > adding/removing memory during this check. Yes, lock ordering is not an issue, I rather think that the device hotplug lock will guard us in all situations. E.g. remove_memory() also does not use it when checking if all blocks are offline. But you can leave it in if you think it is needed. > > https://soleen.com/source/xref/linux/arch/powerpc/platforms/powernv/memtrace.c?r=98fa15f3#248 > > Here we have a similar code: > lock_device_hotplug(); > online_mem_block(); > device_online() > device_lock(dev); > > Pasha > -- Thanks, David / dhildenb ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-04-25 12:38 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-04-21 1:44 [v2 0/2] "Hotremove" persistent memory Pavel Tatashin 2019-04-21 1:44 ` [v2 1/2] device-dax: fix memory and resource leak if hotplug fails Pavel Tatashin 2019-04-21 1:44 ` [v2 2/2] device-dax: "Hotremove" persistent memory that is used like normal RAM Pavel Tatashin 2019-04-24 20:55 ` David Hildenbrand 2019-04-24 21:02 ` Dan Williams 2019-04-24 21:34 ` Pavel Tatashin 2019-04-25 7:41 ` David Hildenbrand 2019-04-25 12:30 ` Pavel Tatashin 2019-04-25 12:38 ` David Hildenbrand
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).