LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Steffen Maier <maier@linux.ibm.com>,
	Jens Remus <jremus@linux.ibm.com>,
	"Martin K. Petersen" <martin.petersen@oracle.com>
Subject: [PATCH 3.18 31/47] scsi: zfcp: fix posting too many status read buffers leading to adapter shutdown
Date: Fri, 11 Jan 2019 15:08:16 +0100
Message-ID: <20190111131000.118374438@linuxfoundation.org> (raw)
In-Reply-To: <20190111130956.170952125@linuxfoundation.org>

3.18-stable review patch.  If anyone has any objections, please let me know.

------------------

From: Steffen Maier <maier@linux.ibm.com>

commit 60a161b7e5b2a252ff0d4c622266a7d8da1120ce upstream.

Suppose adapter (open) recovery is between opened QDIO queues and before
(the end of) initial posting of status read buffers (SRBs). This time
window can be seconds long due to FSF_PROT_HOST_CONNECTION_INITIALIZING
causing by design looping with exponential increase sleeps in the function
performing exchange config data during recovery
[zfcp_erp_adapter_strat_fsf_xconf()]. Recovery triggered by local link up.

Suppose an event occurs for which the FCP channel would send an unsolicited
notification to zfcp by means of a previously posted SRB.  We saw it with
local cable pull (link down) in multi-initiator zoning with multiple
NPIV-enabled subchannels of the same shared FCP channel.

As soon as zfcp_erp_adapter_strategy_open_fsf() starts posting the initial
status read buffers from within the adapter's ERP thread, the channel does
send an unsolicited notification.

Since v2.6.27 commit d26ab06ede83 ("[SCSI] zfcp: receiving an unsolicted
status can lead to I/O stall"), zfcp_fsf_status_read_handler() schedules
adapter->stat_work to re-fill the just consumed SRB from a work item.

Now the ERP thread and the work item post SRBs in parallel.  Both contexts
call the helper function zfcp_status_read_refill().  The tracking of
missing (to be posted / re-filled) SRBs is not thread-safe due to separate
atomic_read() and atomic_dec(), in order to depend on posting
success. Hence, both contexts can see
atomic_read(&adapter->stat_miss) == 1. One of the two contexts posts
one too many SRB. Zfcp gets QDIO_ERROR_SLSB_STATE on the output queue
(trace tag "qdireq1") leading to zfcp_erp_adapter_shutdown() in
zfcp_qdio_handler_error().

An obvious and seemingly clean fix would be to schedule stat_work from the
ERP thread and wait for it to finish. This would serialize all SRB
re-fills. However, we already have another work item wait on the ERP
thread: adapter->scan_work runs zfcp_fc_scan_ports() which calls
zfcp_fc_eval_gpn_ft(). The latter calls zfcp_erp_wait() to wait for all the
open port recoveries during zfcp auto port scan, but in fact it waits for
any pending recovery including an adapter recovery. This approach leads to
a deadlock.  [see also v3.19 commit 18f87a67e6d6 ("zfcp: auto port scan
resiliency"); v2.6.37 commit d3e1088d6873
("[SCSI] zfcp: No ERP escalation on gpn_ft eval");
v2.6.28 commit fca55b6fb587
("[SCSI] zfcp: fix deadlock between wq triggered port scan and ERP")
fixing v2.6.27 commit c57a39a45a76
("[SCSI] zfcp: wait until adapter is finished with ERP during auto-port");
v2.6.27 commit cc8c282963bd
("[SCSI] zfcp: Automatically attach remote ports")]

Instead make the accounting of missing SRBs atomic for parallel execution
in both the ERP thread and adapter->stat_work.

Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Fixes: d26ab06ede83 ("[SCSI] zfcp: receiving an unsolicted status can lead to I/O stall")
Cc: <stable@vger.kernel.org> #2.6.27+
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

---
 drivers/s390/scsi/zfcp_aux.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

--- a/drivers/s390/scsi/zfcp_aux.c
+++ b/drivers/s390/scsi/zfcp_aux.c
@@ -275,16 +275,16 @@ static void zfcp_free_low_mem_buffers(st
  */
 int zfcp_status_read_refill(struct zfcp_adapter *adapter)
 {
-	while (atomic_read(&adapter->stat_miss) > 0)
+	while (atomic_add_unless(&adapter->stat_miss, -1, 0))
 		if (zfcp_fsf_status_read(adapter->qdio)) {
+			atomic_inc(&adapter->stat_miss); /* undo add -1 */
 			if (atomic_read(&adapter->stat_miss) >=
 			    adapter->stat_read_buf_num) {
 				zfcp_erp_adapter_reopen(adapter, 0, "axsref1");
 				return 1;
 			}
 			break;
-		} else
-			atomic_dec(&adapter->stat_miss);
+		}
 	return 0;
 }
 



  parent reply index

Thread overview: 52+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-11 14:07 [PATCH 3.18 00/47] 3.18.132-stable review Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 01/47] USB: hso: Fix OOB memory access in hso_probe/hso_get_config_data Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 02/47] USB: serial: option: add HP lt4132 Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 03/47] mmc: core: Reset HPI enabled state during re-init and in case of errors Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 04/47] mmc: omap_hsmmc: fix DMA API warning Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 05/47] gpio: max7301: fix driver for use with CONFIG_VMAP_STACK Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 06/47] Drivers: hv: vmbus: Return -EINVAL for the sys files for unopened channels Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 07/47] x86/mtrr: Dont copy uninitialized gentry fields back to userspace Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 08/47] ax25: fix a use-after-free in ax25_fillin_cb() Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 09/47] ibmveth: fix DMA unmap error in ibmveth_xmit_start error path Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 10/47] ipv6: explicitly initialize udp6_addr in udp_sock_create6() Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 11/47] isdn: fix kernel-infoleak in capi_unlocked_ioctl Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 12/47] netrom: fix locking in nr_find_socket() Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 13/47] packet: validate address length Greg Kroah-Hartman
2019-01-11 14:07 ` [PATCH 3.18 14/47] packet: validate address length if non-zero Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 15/47] sctp: initialize sin6_flowinfo for ipv6 addrs in sctp_inet6addr_event Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 16/47] vhost: make sure used idx is seen before log in vhost_add_used_n() Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 17/47] VSOCK: Send reset control packet when socket is partially bound Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 18/47] xen/netfront: tolerate frags with no data Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 19/47] sock: Make sock->sk_stamp thread-safe Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 20/47] USB: serial: pl2303: add ids for Hewlett-Packard HP POS pole displays Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 21/47] usb: r8a66597: Fix a possible concurrency use-after-free bug in r8a66597_endpoint_disable() Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 22/47] KVM: x86: Use jmp to invoke kvm_spurious_fault() from .fixup Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 23/47] ext4: fix possible use after free in ext4_quota_enable Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 24/47] ext4: missing unlock/put_page() in ext4_try_to_write_inline_data() Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 25/47] ext4: force inode writes when nfsd calls commit_metadata() Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 26/47] cdc-acm: fix abnormal DATA RX issue for Mediatek Preloader Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 27/47] media: vivid: free bitmap_cap when updating std/timings/etc Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 28/47] MIPS: Ensure pmd_present() returns false after pmd_mknotpresent() Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 29/47] MIPS: Align kernel load address to 64KB Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 30/47] CIFS: Fix error mapping for SMB2_LOCK command which caused OFD lock problem Greg Kroah-Hartman
2019-01-11 14:08 ` Greg Kroah-Hartman [this message]
2019-01-11 14:08 ` [PATCH 3.18 32/47] fork: record start_time late Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 33/47] sunrpc: fix cache_head leak due to queued request Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 34/47] sunrpc: use SVC_NET() in svcauth_gss_* functions Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 35/47] ALSA: cs46xx: Potential NULL dereference in probe Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 36/47] ALSA: usb-audio: Avoid access before bLength check in build_audio_procunit() Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 37/47] ALSA: usb-audio: Fix an out-of-bound read in create_composite_quirks Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 38/47] dlm: fixed memory leaks after failed ls_remove_names allocation Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 39/47] dlm: possible memory leak on error path in create_lkb() Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 40/47] dlm: lost put_lkb on error path in receive_convert() and receive_unlock() Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 41/47] dlm: memory leaks on error path in dlm_user_request() Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 42/47] gfs2: Fix loop in gfs2_rbm_find Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 43/47] b43: Fix error in cordic routine Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 44/47] 9p/net: put a lower bound on msize Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 45/47] ceph: dont update importing caps mseq when handing cap export Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 46/47] genwqe: Fix size check Greg Kroah-Hartman
2019-01-11 14:08 ` [PATCH 3.18 47/47] power: supply: olpc_battery: correct the temperature units Greg Kroah-Hartman
2019-01-11 21:50 ` [PATCH 3.18 00/47] 3.18.132-stable review shuah
2019-01-12 14:58 ` Harsh Shandilya
2019-01-12 17:42 ` Guenter Roeck
2019-01-13  7:44   ` Greg Kroah-Hartman

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190111131000.118374438@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=jremus@linux.ibm.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=maier@linux.ibm.com \
    --cc=martin.petersen@oracle.com \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org linux-kernel@archiver.kernel.org
	public-inbox-index lkml


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/ public-inbox