All of lore.kernel.org
 help / color / mirror / Atom feed
From: Martin Wilck <martin.wilck@suse.com>
To: "emilne@redhat.com" <emilne@redhat.com>,
	"Ulrich.Windl@rz.uni-regensburg.de" 
	<Ulrich.Windl@rz.uni-regensburg.de>,
	"martin.petersen@oracle.com" <martin.petersen@oracle.com>
Cc: Hannes Reinecke <hare@suse.com>, "hch@lst.de" <hch@lst.de>,
	"dgilbert@interlog.com" <dgilbert@interlog.com>,
	"dm-devel@redhat.com" <dm-devel@redhat.com>,
	"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
	"jejb@linux.vnet.ibm.com" <jejb@linux.vnet.ibm.com>,
	"systemd-devel@lists.freedesktop.org" 
	<systemd-devel@lists.freedesktop.org>,
	"bmarzins@redhat.com" <bmarzins@redhat.com>
Subject: Re: [systemd-devel] RFC: one more time: SCSI device identification
Date: Wed, 28 Apr 2021 06:30:28 +0000	[thread overview]
Message-ID: <9248c6df5484a0f5fe4247a1867945ed3902341b.camel@suse.com> (raw)
In-Reply-To: <c8ede601244e1710dbf320c33c0f7853e249bbee.camel@redhat.com>

On Tue, 2021-04-27 at 16:41 -0400, Ewan D. Milne wrote:
> On Tue, 2021-04-27 at 20:33 +0000, Martin Wilck wrote:
> > On Tue, 2021-04-27 at 16:14 -0400, Ewan D. Milne wrote:
> > > 
> > > There's no way to do that, in principle.  Because there could be
> > > other I/Os in flight.  You might (somehow) avoid retrying an I/O
> > > that got a UA until you figured out if something changed, but other
> > > I/Os can already have been sent to the target, or issued before you
> > > get to look at the status.
> > 
> > Right. But in practice, a WWID change will hardly happen under full
> > IO
> > load. The storage side will probably have to block IO while this
> > happens, at least for a short time period. So blocking and quiescing
> > the queue upon an UA might still work, most of the time. Even if we
> > were too late already, the sooner we stop the queue, the better.
> > 
> > The current algorithm in multipath-tools needs to detect a path going
> > down and being reinstated. The time interval during which a WWID
> > change
> > will go unnoticed is one or more path checker intervals, typically on
> > the order of 5-30 seconds. If we could decrease this interval to a
> > sub-
> > second or even millisecond range by blocking the queue in the kernel
> > quickly, we'd have made a big step forward.
> 
> Yes, and in many situations this may help.  But in the general case
> we can't protect against a storage array misconfiguration,
> where something like this can happen.  So I worry about people
> believing the host software will protect them against a mistake,
> when we can't really do that.

I agree. I expressed a similar notion in the following thread about
multipathd's WWID change detection capabilities in the face of really
bad mistakes on the administrator's (or storage array's, FTM)  part:
https://listman.redhat.com/archives/dm-devel/2021-February/msg00248.html
But others stressed that nonetheless we should try our best to
avoid customer data corruption (which I agree with, too), and thus we
settled on the current algorithm, which suited the needs at least of
the affected user(s) in that specific case.

Personally I think that the current "5-30s" time period for WWID change
detection in multipathd is unsafe both theoretically and practially,
and may lure users into a false feeling of safety. Therefore I'd
strongly welcome a kernel-side solution that might still not be safe
theoretically, but cover most practical problem scenarios much better
than we currently do.

Regards
Martin

-- 
Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107
SUSE Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix Imendörffer



WARNING: multiple messages have this Message-ID (diff)
From: Martin Wilck <martin.wilck@suse.com>
To: "emilne@redhat.com" <emilne@redhat.com>,
	"Ulrich.Windl@rz.uni-regensburg.de"
	<Ulrich.Windl@rz.uni-regensburg.de>,
	"martin.petersen@oracle.com" <martin.petersen@oracle.com>
Cc: Hannes Reinecke <hare@suse.com>,
	"jejb@linux.vnet.ibm.com" <jejb@linux.vnet.ibm.com>,
	"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
	"dm-devel@redhat.com" <dm-devel@redhat.com>,
	"dgilbert@interlog.com" <dgilbert@interlog.com>,
	"systemd-devel@lists.freedesktop.org"
	<systemd-devel@lists.freedesktop.org>, "hch@lst.de" <hch@lst.de>
Subject: Re: [dm-devel] [systemd-devel] RFC: one more time: SCSI device identification
Date: Wed, 28 Apr 2021 06:30:28 +0000	[thread overview]
Message-ID: <9248c6df5484a0f5fe4247a1867945ed3902341b.camel@suse.com> (raw)
In-Reply-To: <c8ede601244e1710dbf320c33c0f7853e249bbee.camel@redhat.com>

On Tue, 2021-04-27 at 16:41 -0400, Ewan D. Milne wrote:
> On Tue, 2021-04-27 at 20:33 +0000, Martin Wilck wrote:
> > On Tue, 2021-04-27 at 16:14 -0400, Ewan D. Milne wrote:
> > > 
> > > There's no way to do that, in principle.  Because there could be
> > > other I/Os in flight.  You might (somehow) avoid retrying an I/O
> > > that got a UA until you figured out if something changed, but other
> > > I/Os can already have been sent to the target, or issued before you
> > > get to look at the status.
> > 
> > Right. But in practice, a WWID change will hardly happen under full
> > IO
> > load. The storage side will probably have to block IO while this
> > happens, at least for a short time period. So blocking and quiescing
> > the queue upon an UA might still work, most of the time. Even if we
> > were too late already, the sooner we stop the queue, the better.
> > 
> > The current algorithm in multipath-tools needs to detect a path going
> > down and being reinstated. The time interval during which a WWID
> > change
> > will go unnoticed is one or more path checker intervals, typically on
> > the order of 5-30 seconds. If we could decrease this interval to a
> > sub-
> > second or even millisecond range by blocking the queue in the kernel
> > quickly, we'd have made a big step forward.
> 
> Yes, and in many situations this may help.  But in the general case
> we can't protect against a storage array misconfiguration,
> where something like this can happen.  So I worry about people
> believing the host software will protect them against a mistake,
> when we can't really do that.

I agree. I expressed a similar notion in the following thread about
multipathd's WWID change detection capabilities in the face of really
bad mistakes on the administrator's (or storage array's, FTM)  part:
https://listman.redhat.com/archives/dm-devel/2021-February/msg00248.html
But others stressed that nonetheless we should try our best to
avoid customer data corruption (which I agree with, too), and thus we
settled on the current algorithm, which suited the needs at least of
the affected user(s) in that specific case.

Personally I think that the current "5-30s" time period for WWID change
detection in multipathd is unsafe both theoretically and practially,
and may lure users into a false feeling of safety. Therefore I'd
strongly welcome a kernel-side solution that might still not be safe
theoretically, but cover most practical problem scenarios much better
than we currently do.

Regards
Martin

-- 
Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107
SUSE Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix Imendörffer



--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel


  parent reply	other threads:[~2021-04-28  6:30 UTC|newest]

Thread overview: 50+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-29  9:58 RFC: one more time: SCSI device identification Martin Wilck
2021-03-29  9:58 ` [dm-devel] " Martin Wilck
2021-04-06  4:47 ` Martin K. Petersen
2021-04-06  4:47   ` [dm-devel] " Martin K. Petersen
2021-04-16 23:28   ` Martin Wilck
2021-04-16 23:28     ` [dm-devel] " Martin Wilck
2021-04-22  2:46     ` Martin K. Petersen
2021-04-22  2:46       ` [dm-devel] " Martin K. Petersen
2021-04-22  9:07       ` Martin Wilck
2021-04-22  9:07         ` [dm-devel] " Martin Wilck
2021-04-22 16:14         ` Benjamin Marzinski
2021-04-22 16:14           ` [dm-devel] " Benjamin Marzinski
2021-04-23  1:40         ` Martin K. Petersen
2021-04-23  1:40           ` [dm-devel] " Martin K. Petersen
2021-04-23 10:28           ` Martin Wilck
2021-04-23 10:28             ` [dm-devel] " Martin Wilck
2021-04-26 11:14             ` Antw: [EXT] Re: [systemd-devel] " Ulrich Windl
2021-04-26 11:14               ` [dm-devel] " Ulrich Windl
2021-04-26 13:16               ` Martin Wilck
2021-04-26 13:16                 ` [dm-devel] " Martin Wilck
2021-04-27  3:48                 ` Erwin van Londen
2021-04-27  7:02                   ` Antw: [EXT] " Ulrich Windl
2021-04-27  7:02                     ` [dm-devel] Antw: [EXT] " Ulrich Windl
2021-04-27  8:10                   ` [dm-devel] " Martin Wilck
2021-04-27  8:10                     ` Martin Wilck
2021-04-27  8:21                     ` Hannes Reinecke
2021-04-27  8:21                       ` Hannes Reinecke
2021-04-27 10:52                       ` Antw: [EXT] " Ulrich Windl
2021-04-27 10:52                         ` [dm-devel] Antw: [EXT] " Ulrich Windl
2021-04-27 20:04                         ` Antw: [EXT] Re: [dm-devel] " Ewan D. Milne
2021-04-27 20:04                           ` [dm-devel] Antw: [EXT] " Ewan D. Milne
2021-05-04  7:32                         ` Antw: [EXT] Re: [dm-devel] " Hannes Reinecke
2021-05-04  7:32                           ` [dm-devel] Antw: [EXT] " Hannes Reinecke
2021-04-28  1:01                       ` [dm-devel] " Erwin van Londen
2021-04-28  6:34                         ` Martin Wilck
2021-04-28  6:34                           ` Martin Wilck
2021-04-29 14:47                           ` Erwin van Londen
2021-04-29 14:47                             ` Erwin van Londen
2021-04-27 20:14                 ` Ewan D. Milne
2021-04-27 20:14                   ` [dm-devel] " Ewan D. Milne
2021-04-27 20:33                   ` Martin Wilck
2021-04-27 20:33                     ` [dm-devel] " Martin Wilck
2021-04-27 20:41                     ` Ewan D. Milne
2021-04-27 20:41                       ` [dm-devel] " Ewan D. Milne
2021-04-28  0:09                       ` Erwin van Londen
2021-04-30 23:44                         ` Ewan D. Milne
2021-05-03  2:34                           ` Erwin van Londen
2021-05-03  2:34                             ` Erwin van Londen
2021-04-28  6:30                       ` Martin Wilck [this message]
2021-04-28  6:30                         ` [dm-devel] [systemd-devel] " Martin Wilck

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9248c6df5484a0f5fe4247a1867945ed3902341b.camel@suse.com \
    --to=martin.wilck@suse.com \
    --cc=Ulrich.Windl@rz.uni-regensburg.de \
    --cc=bmarzins@redhat.com \
    --cc=dgilbert@interlog.com \
    --cc=dm-devel@redhat.com \
    --cc=emilne@redhat.com \
    --cc=hare@suse.com \
    --cc=hch@lst.de \
    --cc=jejb@linux.vnet.ibm.com \
    --cc=linux-scsi@vger.kernel.org \
    --cc=martin.petersen@oracle.com \
    --cc=systemd-devel@lists.freedesktop.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.