PR API fixes for multipathing

PR API fixes for multipathing

[Christoph Hellwig]

I was a bit overeager to thing ALL_TG_PT would solve all our multipathing
woes in respect to persistent reservation.  Turns out that there are
lots of possible setups where it doesn't work, and we'll have to ask
device mapper to register all underlying devices instead.

[PATCH 1/2] sd: don't use the ALL_TG_PT bit for reservations

[Christoph Hellwig]

These only work if the we use the same initiator ID for all path,
which might not be true if we use different protocols, or even just
different HBAs.

Instead dm-mpath will grow support to register all path manually
later in this series.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 drivers/scsi/sd.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index 60bff78..42b4d05 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -1613,8 +1613,7 @@ static int sd_pr_register(struct block_device *bdev, u64 old_key, u64 new_key,
 		return -EOPNOTSUPP;
 	return sd_pr_command(bdev, (flags & PR_FL_IGNORE_KEY) ? 0x06 : 0x00,
 			old_key, new_key, 0,
-			(1 << 0) /* APTPL */ |
-			(1 << 2) /* ALL_TG_PT */);
+			(1 << 0) /* APTPL */);
 }
 
 static int sd_pr_reserve(struct block_device *bdev, u64 key, enum pr_type type,
-- 
2.1.4

Re: [PATCH 1/2] sd: don't use the ALL_TG_PT bit for reservations

[Mike Christie]

On 07/08/2016 07:23 AM, Christoph Hellwig wrote:
> These only work if the we use the same initiator ID for all path,
> which might not be true if we use different protocols, or even just
> different HBAs.
> 
> Instead dm-mpath will grow support to register all path manually
> later in this series.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>  drivers/scsi/sd.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
> index 60bff78..42b4d05 100644
> --- a/drivers/scsi/sd.c
> +++ b/drivers/scsi/sd.c
> @@ -1613,8 +1613,7 @@ static int sd_pr_register(struct block_device *bdev, u64 old_key, u64 new_key,
>  		return -EOPNOTSUPP;
>  	return sd_pr_command(bdev, (flags & PR_FL_IGNORE_KEY) ? 0x06 : 0x00,
>  			old_key, new_key, 0,
> -			(1 << 0) /* APTPL */ |
> -			(1 << 2) /* ALL_TG_PT */);
> +			(1 << 0) /* APTPL */);
>  }
>  
>  static int sd_pr_reserve(struct block_device *bdev, u64 key, enum pr_type type,
> 

Reviewed-by: Mike Christie <mchristi@redhat.com>

[PATCH 2/2] dm: call PR reserve/unreserve on each underlying device

[Christoph Hellwig]

So far we tried to rely on the SCSI 'all target ports' bit to register
all path, but for many setups this didn't work properly as the different
path aren seen as separate initiators to the target instead of multiple
ports of the same initiator.  Because of that we'll stop setting the
'all target ports' bit in SCSI, and let device mapper handle iterating
over the device for each path and register it manually.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 drivers/md/dm.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 70 insertions(+), 15 deletions(-)

diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 1b2f962..e4e98b7 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -3601,26 +3601,81 @@ void dm_free_md_mempools(struct dm_md_mempools *pools)
 	kfree(pools);
 }
 
-static int dm_pr_register(struct block_device *bdev, u64 old_key, u64 new_key,
-			  u32 flags)
+struct dm_pr {
+	u64	old_key;
+	u64	new_key;
+	u32	flags;
+	bool	fail_early;
+};
+
+static int dm_call_pr(struct block_device *bdev, iterate_devices_callout_fn fn,
+		void *data)
 {
 	struct mapped_device *md = bdev->bd_disk->private_data;
-	const struct pr_ops *ops;
-	fmode_t mode;
-	int r;
+	struct dm_table *table;
+	struct dm_target *ti;
+	int ret = 0, srcu_idx;
 
-	r = dm_grab_bdev_for_ioctl(md, &bdev, &mode);
-	if (r < 0)
-		return r;
+	table = dm_get_live_table(md, &srcu_idx);
+	if (!table || !dm_table_get_size(table))
+		return -ENOTTY;
 
-	ops = bdev->bd_disk->fops->pr_ops;
-	if (ops && ops->pr_register)
-		r = ops->pr_register(bdev, old_key, new_key, flags);
-	else
-		r = -EOPNOTSUPP;
+	/* We only support devices that have a single target */
+	ret = -ENOTTY;
+	if (dm_table_get_num_targets(table) != 1)
+		goto out;
+	ti = dm_table_get_target(table, 0);
 
-	bdput(bdev);
-	return r;
+	ret = -EINVAL;
+	if (!ti->type->iterate_devices)
+		goto out;
+
+	ret = ti->type->iterate_devices(ti, fn, data);
+	if (ret)
+		goto out;
+
+	ret = 0;
+out:
+	dm_put_live_table(md, srcu_idx);
+	return ret;
+}
+
+/*
+ * For register / unregister we need to manually call out to every path.
+ */
+static int __dm_pr_register(struct dm_target *ti, struct dm_dev *dev,
+		sector_t start, sector_t len, void *data)
+{
+	struct dm_pr *pr = data;
+	const struct pr_ops *ops = dev->bdev->bd_disk->fops->pr_ops;
+
+	if (!ops || !ops->pr_register)
+		return -EOPNOTSUPP;
+	return ops->pr_register(dev->bdev, pr->old_key, pr->new_key, pr->flags);
+}
+
+static int dm_pr_register(struct block_device *bdev, u64 old_key, u64 new_key,
+			  u32 flags)
+{
+	struct dm_pr pr = {
+		.old_key	= old_key,
+		.new_key	= new_key,
+		.flags		= flags,
+		.fail_early	= true,
+	};
+	int ret;
+
+	ret = dm_call_pr(bdev, __dm_pr_register, &pr);
+	if (ret && new_key) {
+		/* unregister all paths if we failed to register any path */
+		pr.old_key = new_key;
+		pr.new_key = 0;
+		pr.flags = 0;
+		pr.fail_early = false;
+		dm_call_pr(bdev, __dm_pr_register, &pr);
+	}
+
+	return ret;
 }
 
 static int dm_pr_reserve(struct block_device *bdev, u64 key, enum pr_type type,
-- 
2.1.4

Re: [PATCH 2/2] dm: call PR reserve/unreserve on each underlying device

[Mike Christie]

On 07/08/2016 07:23 AM, Christoph Hellwig wrote:
> So far we tried to rely on the SCSI 'all target ports' bit to register
> all path, but for many setups this didn't work properly as the different
> path aren seen as separate initiators to the target instead of multiple
> ports of the same initiator.  Because of that we'll stop setting the
> 'all target ports' bit in SCSI, and let device mapper handle iterating
> over the device for each path and register it manually.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>  drivers/md/dm.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++----------
>  1 file changed, 70 insertions(+), 15 deletions(-)
> 
> diff --git a/drivers/md/dm.c b/drivers/md/dm.c
> index 1b2f962..e4e98b7 100644
> --- a/drivers/md/dm.c
> +++ b/drivers/md/dm.c
> @@ -3601,26 +3601,81 @@ void dm_free_md_mempools(struct dm_md_mempools *pools)
>  	kfree(pools);
>  }
>  
> -static int dm_pr_register(struct block_device *bdev, u64 old_key, u64 new_key,
> -			  u32 flags)
> +struct dm_pr {
> +	u64	old_key;
> +	u64	new_key;
> +	u32	flags;
> +	bool	fail_early;
> +};
> +
> +static int dm_call_pr(struct block_device *bdev, iterate_devices_callout_fn fn,
> +		void *data)
>  {
>  	struct mapped_device *md = bdev->bd_disk->private_data;
> -	const struct pr_ops *ops;
> -	fmode_t mode;
> -	int r;
> +	struct dm_table *table;
> +	struct dm_target *ti;
> +	int ret = 0, srcu_idx;
>  
> -	r = dm_grab_bdev_for_ioctl(md, &bdev, &mode);
> -	if (r < 0)
> -		return r;
> +	table = dm_get_live_table(md, &srcu_idx);
> +	if (!table || !dm_table_get_size(table))
> +		return -ENOTTY;
>  
> -	ops = bdev->bd_disk->fops->pr_ops;
> -	if (ops && ops->pr_register)
> -		r = ops->pr_register(bdev, old_key, new_key, flags);
> -	else
> -		r = -EOPNOTSUPP;
> +	/* We only support devices that have a single target */
> +	ret = -ENOTTY;
> +	if (dm_table_get_num_targets(table) != 1)
> +		goto out;
> +	ti = dm_table_get_target(table, 0);
>  
> -	bdput(bdev);
> -	return r;
> +	ret = -EINVAL;
> +	if (!ti->type->iterate_devices)
> +		goto out;
> +
> +	ret = ti->type->iterate_devices(ti, fn, data);
> +	if (ret)
> +		goto out;
> +
> +	ret = 0;
> +out:
> +	dm_put_live_table(md, srcu_idx);
> +	return ret;
> +}
> +
> +/*
> + * For register / unregister we need to manually call out to every path.
> + */
> +static int __dm_pr_register(struct dm_target *ti, struct dm_dev *dev,
> +		sector_t start, sector_t len, void *data)
> +{
> +	struct dm_pr *pr = data;
> +	const struct pr_ops *ops = dev->bdev->bd_disk->fops->pr_ops;
> +
> +	if (!ops || !ops->pr_register)
> +		return -EOPNOTSUPP;
> +	return ops->pr_register(dev->bdev, pr->old_key, pr->new_key, pr->flags);
> +}
> +
> +static int dm_pr_register(struct block_device *bdev, u64 old_key, u64 new_key,
> +			  u32 flags)
> +{
> +	struct dm_pr pr = {
> +		.old_key	= old_key,
> +		.new_key	= new_key,
> +		.flags		= flags,
> +		.fail_early	= true,
> +	};
> +	int ret;
> +
> +	ret = dm_call_pr(bdev, __dm_pr_register, &pr);
> +	if (ret && new_key) {
> +		/* unregister all paths if we failed to register any path */
> +		pr.old_key = new_key;
> +		pr.new_key = 0;
> +		pr.flags = 0;
> +		pr.fail_early = false;
> +		dm_call_pr(bdev, __dm_pr_register, &pr);
> +	}
> +
> +	return ret;
>  }
>  
>  static int dm_pr_reserve(struct block_device *bdev, u64 key, enum pr_type type,
> 



Reviewed-by: Mike Christie <mchristi@redhat.com>

Re: PR API fixes for multipathing

[Martin K. Petersen]

>>>>> "Christoph" == Christoph Hellwig <hch@lst.de> writes:

Christoph> I was a bit overeager to thing ALL_TG_PT would solve all our
Christoph> multipathing woes in respect to persistent reservation.
Christoph> Turns out that there are lots of possible setups where it
Christoph> doesn't work, and we'll have to ask device mapper to register
Christoph> all underlying devices instead.

Should I queue the sd patch or let Mike take both through the DM tree?

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: PR API fixes for multipathing

[Christoph Hellwig]

On Fri, Jul 15, 2016 at 03:03:54PM -0400, Martin K. Petersen wrote:
> >>>>> "Christoph" == Christoph Hellwig <hch@lst.de> writes:
> 
> Christoph> I was a bit overeager to thing ALL_TG_PT would solve all our
> Christoph> multipathing woes in respect to persistent reservation.
> Christoph> Turns out that there are lots of possible setups where it
> Christoph> doesn't work, and we'll have to ask device mapper to register
> Christoph> all underlying devices instead.
> 
> Should I queue the sd patch or let Mike take both through the DM tree?

I think having both in the same tree would be very useful.  I don't care
which one that is.

Re: PR API fixes for multipathing

[Mike Snitzer]

On Fri, Jul 15 2016 at  9:08pm -0400,
Christoph Hellwig <hch@lst.de> wrote:

> On Fri, Jul 15, 2016 at 03:03:54PM -0400, Martin K. Petersen wrote:
> > >>>>> "Christoph" == Christoph Hellwig <hch@lst.de> writes:
> > 
> > Christoph> I was a bit overeager to thing ALL_TG_PT would solve all our
> > Christoph> multipathing woes in respect to persistent reservation.
> > Christoph> Turns out that there are lots of possible setups where it
> > Christoph> doesn't work, and we'll have to ask device mapper to register
> > Christoph> all underlying devices instead.
> > 
> > Should I queue the sd patch or let Mike take both through the DM tree?
> 
> I think having both in the same tree would be very useful.  I don't care
> which one that is.

I've picked both of them up.  Staged for 4.8 merge and in linux-next via
linux-dm.git's 'for-next'.

(I added Martin's Acked-by to the sd patch header, Martin: if not OK, or
if you'd prefer Reviewed-by just let me know)

Christoph, I had to fix this:

drivers/md/dm.c:2564:12: warning: context imbalance in 'dm_call_pr' - different lock contexts for basic block

here is the incremental I folded in to the dm patch:

diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 214fa03..4dca5a7 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -2567,14 +2567,13 @@ static int dm_call_pr(struct block_device *bdev, iterate_devices_callout_fn fn,
        struct mapped_device *md = bdev->bd_disk->private_data;
        struct dm_table *table;
        struct dm_target *ti;
-       int ret = 0, srcu_idx;
+       int ret = -ENOTTY, srcu_idx;

        table = dm_get_live_table(md, &srcu_idx);
        if (!table || !dm_table_get_size(table))
-               return -ENOTTY;
+               goto out;

        /* We only support devices that have a single target */
-       ret = -ENOTTY;
        if (dm_table_get_num_targets(table) != 1)
                goto out;
        ti = dm_table_get_target(table, 0);
@@ -2584,10 +2583,6 @@ static int dm_call_pr(struct block_device *bdev, iterate_devices_callout_fn fn,
                goto out;

        ret = ti->type->iterate_devices(ti, fn, data);
-       if (ret)
-               goto out;
-
-       ret = 0;
 out:
        dm_put_live_table(md, srcu_idx);
        return ret;

Mike

Re: PR API fixes for multipathing

[James Bottomley]

On Sat, 2016-07-16 at 14:10 -0400, Mike Snitzer wrote:
> On Fri, Jul 15 2016 at  9:08pm -0400,
> Christoph Hellwig <hch@lst.de> wrote:
> 
> > On Fri, Jul 15, 2016 at 03:03:54PM -0400, Martin K. Petersen wrote:
> > > > > > > > "Christoph" == Christoph Hellwig <hch@lst.de> writes:
> > > 
> > > Christoph> I was a bit overeager to thing ALL_TG_PT would solve
> > > all our
> > > Christoph> multipathing woes in respect to persistent
> > > reservation.
> > > Christoph> Turns out that there are lots of possible setups where
> > > it
> > > Christoph> doesn't work, and we'll have to ask device mapper to
> > > register
> > > Christoph> all underlying devices instead.
> > > 
> > > Should I queue the sd patch or let Mike take both through the DM
> > > tree?
> > 
> > I think having both in the same tree would be very useful.  I don't 
> > care which one that is.
> 
> I've picked both of them up.  Staged for 4.8 merge and in linux-next 
> via linux-dm.git's 'for-next'.
> 
> (I added Martin's Acked-by to the sd patch header, Martin: if not OK, 
> or if you'd prefer Reviewed-by just let me know)

You are actually missing the reviewed by tags from Mike Christie on
this, which should be added.

James

Re: PR API fixes for multipathing

[Mike Snitzer]

On Sat, Jul 16 2016 at  2:14pm -0400,
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> On Sat, 2016-07-16 at 14:10 -0400, Mike Snitzer wrote:
> > On Fri, Jul 15 2016 at  9:08pm -0400,
> > Christoph Hellwig <hch@lst.de> wrote:
> > 
> > > On Fri, Jul 15, 2016 at 03:03:54PM -0400, Martin K. Petersen wrote:
> > > > > > > > > "Christoph" == Christoph Hellwig <hch@lst.de> writes:
> > > > 
> > > > Christoph> I was a bit overeager to thing ALL_TG_PT would solve
> > > > all our
> > > > Christoph> multipathing woes in respect to persistent
> > > > reservation.
> > > > Christoph> Turns out that there are lots of possible setups where
> > > > it
> > > > Christoph> doesn't work, and we'll have to ask device mapper to
> > > > register
> > > > Christoph> all underlying devices instead.
> > > > 
> > > > Should I queue the sd patch or let Mike take both through the DM
> > > > tree?
> > > 
> > > I think having both in the same tree would be very useful.  I don't 
> > > care which one that is.
> > 
> > I've picked both of them up.  Staged for 4.8 merge and in linux-next 
> > via linux-dm.git's 'for-next'.
> > 
> > (I added Martin's Acked-by to the sd patch header, Martin: if not OK, 
> > or if you'd prefer Reviewed-by just let me know)
> 
> You are actually missing the reviewed by tags from Mike Christie on
> this, which should be added.

Added, thanks!

Re: PR API fixes for multipathing

[Christoph Hellwig]

On Sat, Jul 16, 2016 at 02:10:07PM -0400, Mike Snitzer wrote:
> Christoph, I had to fix this:
> 
> drivers/md/dm.c:2564:12: warning: context imbalance in 'dm_call_pr' - different lock contexts for basic block

That's coming from sparse?  Guess I need to update my sparse version..

> here is the incremental I folded in to the dm patch:

Looks fine,

Reviewed-by: Christoph Hellwig <hch@lst.de>