* Powerpath vs dm-multipath - two points of FUD? @ 2014-09-09 16:50 Rob 2014-09-10 10:04 ` Bryn M. Reeves 2014-09-14 8:39 ` Hannes Reinecke 0 siblings, 2 replies; 5+ messages in thread From: Rob @ 2014-09-09 16:50 UTC (permalink / raw) To: dm-devel [-- Attachment #1.1: Type: text/plain, Size: 2070 bytes --] Hi List, Firstly, apologies if this is a common topic and my intentions are not to start a flame war. I've googled extensively but haven't found specific information to address my queries, so I thought I would turn here. We have a rather large multi-tenant infrastructure using PowerPath. Since this inherently comes with increased maintenance costs (recompiling the module, requiring extra steps / care when upgrading etc) we are looking at using dm-multipath as the defacto standard SAN-connection abstraction layer for installations of RHEL 7+. After discussions with our SAN Architect team, we were given the below points to chew over and we were met with stiff resistance to moving away from Powerpath. Since there was little right-of-reply, I'd like to run these points past the minds of this list to understand if these are valid enough to justify a valid business case of keeping Powerpath over Multipath. *Here’s a couple of reasons to stick with powerpath:* Load Balancing: Whilst dm-multipath can make use of more than one of the paths to an array, .i.e with round-robin, this isn’t true load-balancing. Powerpath is able to examine the paths down to the array and balance workload based on how busy the storage controller / ports are. AFAIK Rhel6 has added functionality to make path choices based on queue depth and service time, which does add some improvement over vanilla round-robin. For VMAX and CX/VNX, powerpath uses the following parameters to balance the paths out: Pending I/Os on the path, Size of I/Os, Types of I/Os, and Paths most recently used. * Flakey Path Detection: The latest versions of powerpath can proactively take paths out of service should it observe intermittent IO failures (remember any IO failure can hold a thread for 30-60 seconds whilst the SCSI command further up the stack times out, and a retry is sent). dm-multipath doesn’t have functionality to remove a flakey path, paths can only be marked out of service on hard failure.* Many thanks -- Rob [-- Attachment #1.2: Type: text/html, Size: 2187 bytes --] [-- Attachment #2: Type: text/plain, Size: 0 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Powerpath vs dm-multipath - two points of FUD? 2014-09-09 16:50 Powerpath vs dm-multipath - two points of FUD? Rob @ 2014-09-10 10:04 ` Bryn M. Reeves 2014-09-14 8:39 ` Hannes Reinecke 1 sibling, 0 replies; 5+ messages in thread From: Bryn M. Reeves @ 2014-09-10 10:04 UTC (permalink / raw) To: device-mapper development On Tue, Sep 09, 2014 at 05:50:49PM +0100, Rob wrote: > Firstly, apologies if this is a common topic and my intentions are not to > start a flame war. I've googled extensively but haven't found specific > information to address my queries, so I thought I would turn here. Dr. Freeman, I presume? ;) > *Here’s a couple of reasons to stick with powerpath:* Load > Balancing: Whilst dm-multipath can make use of more than one of the paths > to an array, .i.e with round-robin, this isn’t true load-balancing. Hasn't been true since RHEL5. RHEL6's kernel has request-based device-mapper and the SCSI device_handler infrastructure which allows the dm-multipath target to make better decisions in response to device utilisation and error conditions. In addition to the simple round-robin path selector RHEL6 (and later) device-mapper-multipath supports the queue-length and service-time path selectors. These route IO to the device with the shortest queue or best service time (shortest queue relative to the device's throughput). It's relatively easy to add a new path selector - the difficult part is to define generally useful selection algorithms. One thing to bear in mind is that all of the Linux multipath solutions sit on top of the same SCSI infrastructure; I've seen situations where hosts using multipath-tools, PowerPath and Veritas VxDMP all suffered a similar failure due to a hardware fualt condition that was not handled well by the Linux midlayer at the time. > Powerpath is able to examine the paths down to the array and balance > workload based on how busy the storage controller / ports are. AFAIK Rhel6 > has added functionality to make path choices based on queue depth and > service time, which does add some improvement over vanilla round-robin. For > VMAX and CX/VNX, powerpath uses the following parameters to balance the > paths out: Pending I/Os on the path, Size of I/Os, Types of I/Os, and Paths PowerPath does support various modes of optimisation that are specific to EMC arrays (iirc separate policies exist for Symmetrix and other product lines). These are based on an understanding of the device internals and rely on information that is not publicly available - e.g. I've seen cases where PowerPath decides to route all IO to a single port of a Symmetrix presumably because this gives improved cache behaviour. > most recently used. * Flakey Path Detection: The latest versions of > powerpath can proactively take paths out of service should it observe > intermittent IO failures (remember any IO failure can hold a thread for > 30-60 seconds whilst the SCSI command further up the stack times out, and a > retry is sent). dm-multipath doesn’t have functionality to remove a flakey > path, paths can only be marked out of service on hard failure.* This is something I've discussed a few times. It's a useful feature but one that mainly comes into play when dealing with failing or marginal hardware. Regards, Bryn. -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Powerpath vs dm-multipath - two points of FUD? 2014-09-09 16:50 Powerpath vs dm-multipath - two points of FUD? Rob 2014-09-10 10:04 ` Bryn M. Reeves @ 2014-09-14 8:39 ` Hannes Reinecke 2019-03-31 23:16 ` least-pending multipath path selector WAS (9/14/14 !!!!): " Xose Vazquez Perez 1 sibling, 1 reply; 5+ messages in thread From: Hannes Reinecke @ 2014-09-14 8:39 UTC (permalink / raw) To: dm-devel On 09/09/2014 06:50 PM, Rob wrote: > Hi List, > > Firstly, apologies if this is a common topic and my intentions are not > to start a flame war. I've googled extensively but haven't found > specific information to address my queries, so I thought I would turn here. > > We have a rather large multi-tenant infrastructure using PowerPath. > Since this inherently comes with increased maintenance costs > (recompiling the module, requiring extra steps / care when upgrading > etc) we are looking at using dm-multipath as the defacto standard > SAN-connection abstraction layer for installations of RHEL 7+. > > After discussions with our SAN Architect team, we were given the below > points to chew over and we were met with stiff resistance to moving away > from Powerpath. Since there was little right-of-reply, I'd like to run > these points past the minds of this list to understand if these are > valid enough to justify a valid business case of keeping Powerpath over > Multipath. > Hehe. PowerPath again. Mind you, device-mapper multipathing is fully supported by EMC ... > > /Here’s a couple of reasons to stick with powerpath: > > * Load Balancing: > > Whilst dm-multipath can make use of more than one of the paths to an > array, .i.e with round-robin, this isn’t true load-balancing. Powerpath > is able to examine the paths down to the array and balance workload > based on how busy the storage controller / ports are. AFAIK Rhel6 has > added functionality to make path choices based on queue depth and > service time, which does add some improvement over vanilla round-robin. > We do this with the switch to request-based multipathing. Using one of the other load balancers (eg least-pending) and set rr_min_io to '1' will give you exactly that behaviour. > For VMAX and CX/VNX, powerpath uses the following parameters to > balance the paths out: Pending I/Os on the path, Size of I/Os, Types of > I/Os, and Paths most recently used. > pending I/O is covered with the 'least-pending' I/O scheduler; I fail to see the value in any of the others (where would be the point in switching I/O based on the _size_ of the I/O request ?) > * Flakey Path Detection: > > The latest versions of powerpath can proactively take paths out of > service should it observe intermittent IO failures (remember any IO > failure can hold a thread for 30-60 seconds whilst the SCSI command > further up the stack times out, and a retry is sent). dm-multipath > doesn’t have functionality to remove a flakey path, paths can only be > marked out of service on hard failure./ > Wrong. I've added flakey path detection a while back. I'll be looking at the sources and will be checking the current status; might be I've not gotten around to send it upstream. So you _might_ need to switch to SLES :-) Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) ^ permalink raw reply [flat|nested] 5+ messages in thread
* least-pending multipath path selector WAS (9/14/14 !!!!): Re: Powerpath vs dm-multipath - two points of FUD? 2014-09-14 8:39 ` Hannes Reinecke @ 2019-03-31 23:16 ` Xose Vazquez Perez 2019-04-01 11:05 ` Martin Wilck 0 siblings, 1 reply; 5+ messages in thread From: Xose Vazquez Perez @ 2019-03-31 23:16 UTC (permalink / raw) To: Hannes Reinecke, Martin Wilck, device-mapper development On 9/14/14 10:39 AM, Hannes Reinecke wrote: > We do this with the switch to request-based multipathing. > Using one of the other load balancers (eg least-pending) and set rr_min_io to '1' will give you exactly that behaviour. As the original authors team, at HP, was dissolved ( https://bugzilla.redhat.com/show_bug.cgi?id=438760#c14 ) Hannes, do you consider least-pending mp selector helpful for upstream? https://raw.githubusercontent.com/kdave/kernel-source/master/patches.suse/dm-mpath-leastpending-path-update ---cut--- Subject: Update least-pending-IO dynamic load balancer From: Hannes Reinecke <hare@suse.de> Date: Wed Jan 7 09:26:30 2009 +0100: References: bnc#444199 Patch-mainline: not yet, <hare: will resubmit, Oct 4, 2017> Attached patch provides "Least pending IO" dynamic load balancing policy for bio based device mapper multipath. This load balancing policy considers the number of unserviced requests pending on a path and selects the path with least count for pending service request. We find this policy more useful especially when the SAN environment has heterogeneous components. Ex, when there is one 8GB HBA and one 2GB HBA connected to the same server, 8GB HBA could be utilized better with this algorithm. This patch includes the update as posted in the bugzilla, based on the review comments received in the dm-devel mailing list. Signed-off-by: Sakshi Chaitanya Veni <vsakshi@hp.com> Signed-off-by: Vijayakumar Balasubramanian <vijayakumar@hp.com> Signed-off-by: Senthil Kumar V <senthil-kumar.veluswamy@hp.com> Signed-off-by: Hannes Reinecke <hare@suse.de> --- drivers/md/Makefile | 2 drivers/md/dm-least-pending.c | 265 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 266 insertions(+), 1 deletion(-) --- a/drivers/md/Makefile +++ b/drivers/md/Makefile @@ -41,7 +41,7 @@ obj-$(CONFIG_DM_BIO_PRISON) += dm-bio-pr obj-$(CONFIG_DM_CRYPT) += dm-crypt.o obj-$(CONFIG_DM_DELAY) += dm-delay.o obj-$(CONFIG_DM_FLAKEY) += dm-flakey.o -obj-$(CONFIG_DM_MULTIPATH) += dm-multipath.o dm-round-robin.o +obj-$(CONFIG_DM_MULTIPATH) += dm-multipath.o dm-round-robin.o dm-least-pending.o obj-$(CONFIG_DM_MULTIPATH_QL) += dm-queue-length.o obj-$(CONFIG_DM_MULTIPATH_ST) += dm-service-time.o obj-$(CONFIG_DM_SWITCH) += dm-switch.o --- /dev/null +++ b/drivers/md/dm-least-pending.c @@ -0,0 +1,265 @@ +/* + * (C) Copyright 2008 Hewlett-Packard Development Company, L.P + * + * This file is released under the GPL. + */ + +#include "dm-path-selector.h" + +#include <linux/slab.h> +#include <linux/module.h> + +#define DM_MSG_PREFIX "multipath least-pending" + +/*----------------------------------------------------------------- +* Path-handling code, paths are held in lists +*---------------------------------------------------------------*/ +struct path_info { + struct list_head list; + struct dm_path *path; + atomic_t io_count; +}; + +static void free_paths(struct list_head *paths) +{ + struct path_info *pi, *next; + + list_for_each_entry_safe(pi, next, paths, list) { + list_del(&pi->list); + kfree(pi); + } +} + +/*----------------------------------------------------------------- + * Least-pending selector + *---------------------------------------------------------------*/ + +#define LPP_MIN_IO 1 + +struct selector { + struct list_head valid_paths; + struct list_head invalid_paths; +}; + +static struct selector *alloc_selector(void) +{ + struct selector *s = kmalloc(sizeof(*s), GFP_KERNEL); + + if (s) { + INIT_LIST_HEAD(&s->valid_paths); + INIT_LIST_HEAD(&s->invalid_paths); + } + + return s; +} + +static int lpp_create(struct path_selector *ps, unsigned argc, char **argv) +{ + struct selector *s; + + s = alloc_selector(); + if (!s) + return -ENOMEM; + + ps->context = s; + return 0; +} + +static void lpp_destroy(struct path_selector *ps) +{ + struct selector *s = ps->context; + + free_paths(&s->valid_paths); + free_paths(&s->invalid_paths); + kfree(s); + ps->context = NULL; +} + +static int lpp_status(struct path_selector *ps, struct dm_path *path, + status_type_t type, char *result, unsigned int maxlen) +{ + struct path_info *pi; + int sz = 0; + + if (!path) + switch (type) { + case STATUSTYPE_INFO: + DMEMIT("1 "); + break; + case STATUSTYPE_TABLE: + DMEMIT("0 "); + break; + } + else { + pi = path->pscontext; + switch (type) { + case STATUSTYPE_INFO: + /* + * Even though repeat_count isn't used anymore, + * status info expects it. It's always 1. + */ + DMEMIT("1:%u ", atomic_read(&pi->io_count)); + break; + case STATUSTYPE_TABLE: + break; + } + } + + return sz; +} + +/* + * Called during initialisation to register each path + */ +static int lpp_add_path(struct path_selector *ps, struct dm_path *path, + int argc, char **argv, char **error) +{ + struct selector *s = ps->context; + struct path_info *pi; + + if (argc > 1) { + *error = "least-pending ps: incorrect number of arguments"; + return -EINVAL; + } + + /* + * Sanity check the optional repeat_count argument. It must + * always be 1 and isn't actually variable. We can just issue + * a warning and adjust automatically rather than error out. + */ + if (argc == 1) { + unsigned repeat_count = 1; + if (sscanf(argv[0], "%u", &repeat_count) != 1) { + *error = "least-pending ps: invalid repeat count"; + return -EINVAL; + } + if (repeat_count > 1) + DMWARN_LIMIT("repeat_count > 1 is deprecated, using 1 instead"); + } + + /* allocate the path */ + pi = kmalloc(sizeof(*pi), GFP_KERNEL); + if (!pi) { + *error = "least-pending ps: Error allocating path context"; + return -ENOMEM; + } + + pi->path = path; + atomic_set(&pi->io_count, 0); + + path->pscontext = pi; + + list_add(&pi->list, &s->valid_paths); + + return 0; +} + +static void lpp_fail_path(struct path_selector *ps, struct dm_path *p) +{ + struct selector *s = ps->context; + struct path_info *pi = p->pscontext; + + if (!pi) + return; + + atomic_set(&pi->io_count, 0); + + list_move(&pi->list, &s->invalid_paths); +} + +static int lpp_reinstate_path(struct path_selector *ps, struct dm_path *p) +{ + struct selector *s = ps->context; + struct path_info *pi = p->pscontext; + + if (!pi) + return 1; + + list_move(&pi->list, &s->valid_paths); + + return 0; +} + +static struct dm_path *lpp_select_path(struct path_selector *ps, + size_t nr_bytes) +{ + struct selector *s = ps->context; + struct path_info *pi, *next, *least_io_path = NULL; + struct list_head *paths; + + if (list_empty(&s->valid_paths)) + return NULL; + + paths = &s->valid_paths; + + list_for_each_entry_safe(pi, next, paths, list) { + if (!least_io_path || atomic_read(&least_io_path->io_count) < atomic_read(&pi->io_count)) + least_io_path = pi; + if (!atomic_read(&least_io_path->io_count)) + break; + } + + if (!least_io_path) + return NULL; + + atomic_inc(&least_io_path->io_count); + + return least_io_path->path; +} + +static int lpp_end_io(struct path_selector *ps, struct dm_path *path, + size_t nr_bytes) +{ + struct path_info *pi = NULL; + + pi = path->pscontext; + if (!pi) + return 1; + + atomic_dec(&pi->io_count); + + return 0; +} + +static struct path_selector_type lpp_ps = { + .name = "least-pending", + .module = THIS_MODULE, + .table_args = 1, + .info_args = 0, + .create = lpp_create, + .destroy = lpp_destroy, + .status = lpp_status, + .add_path = lpp_add_path, + .fail_path = lpp_fail_path, + .reinstate_path = lpp_reinstate_path, + .select_path = lpp_select_path, + .end_io = lpp_end_io, +}; + +static int __init dm_lpp_init(void) +{ + int r = dm_register_path_selector(&lpp_ps); + + if (r < 0) + DMERR("register failed %d", r); + + DMINFO("version 1.0.0 loaded"); + + return r; +} + +static void __exit dm_lpp_exit(void) +{ + int r = dm_unregister_path_selector(&lpp_ps); + + if (r < 0) + DMERR("unregister failed %d", r); +} + +module_init(dm_lpp_init); +module_exit(dm_lpp_exit); + +MODULE_DESCRIPTION(DM_NAME " least-pending multipath path selector"); +MODULE_AUTHOR("Sakshi Chaitanya Veni <vsakshi@hp.com>"); +MODULE_LICENSE("GPL"); + ---end--- Missing Kconfig option: diff -pNaur linux-2.6.28-rc7-org//drivers/md/Kconfig linux-2.6.28-rc7-mod//drivers/md/Kconfig --- linux-2.6.28-rc7-org//drivers/md/Kconfig 2008-12-04 14:50:35.000000000 +0530 +++ linux-2.6.28-rc7-mod//drivers/md/Kconfig 2008-12-05 15:32:56.000000000 +0530 @@ -192,6 +192,16 @@ config MD_MULTIPATH If unsure, say N. +config MD_MULTIPATH_LPP + tristate "Multipath - Least Pending IO Path Selector" + depends on MD_MULTIPATH + help + Least Pending IO Path Selector - This load balancing policy + considers the number of unserviced requests pending on a path + and selects the path with least count for pending service request. + + If unsure, say N. + config MD_FAULTY tristate "Faulty test module for MD" depends on BLK_DEV_MD ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: least-pending multipath path selector WAS (9/14/14 !!!!): Re: Powerpath vs dm-multipath - two points of FUD? 2019-03-31 23:16 ` least-pending multipath path selector WAS (9/14/14 !!!!): " Xose Vazquez Perez @ 2019-04-01 11:05 ` Martin Wilck 0 siblings, 0 replies; 5+ messages in thread From: Martin Wilck @ 2019-04-01 11:05 UTC (permalink / raw) To: Xose Vazquez Perez, device-mapper development, Hannes Reinecke Hi Xose, On Mon, 2019-04-01 at 01:16 +0200, Xose Vazquez Perez wrote: > On 9/14/14 10:39 AM, Hannes Reinecke wrote: > > > We do this with the switch to request-based multipathing. > > Using one of the other load balancers (eg least-pending) and set > > rr_min_io to '1' will give you exactly that behaviour. > > As the original authors team, at HP, was dissolved ( > https://bugzilla.redhat.com/show_bug.cgi?id=438760#c14 ) > Hannes, do you consider least-pending mp selector helpful for > upstream? > I can't speak for Hannes, but I believe that the functionality provided by this patch is mostly provided by the "queue-length" path selector today. "service-time" has been the default since 2013, and most users seem to be fine with that. It's true that SLES kernels still support it, but I don't think it's used much, if at all. Even if the "least pending" selector was still needed, the patch would need updating for current upstream, to accomodate the changes from 9659f811446a "dm mpath: push path selector locking down to path selectors", and possibly other changes in the general dm/path selector framework. Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-04-01 11:05 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-09-09 16:50 Powerpath vs dm-multipath - two points of FUD? Rob 2014-09-10 10:04 ` Bryn M. Reeves 2014-09-14 8:39 ` Hannes Reinecke 2019-03-31 23:16 ` least-pending multipath path selector WAS (9/14/14 !!!!): " Xose Vazquez Perez 2019-04-01 11:05 ` Martin Wilck
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.