From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A67C7C43214 for ; Wed, 1 Sep 2021 11:19:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 81E6461053 for ; Wed, 1 Sep 2021 11:19:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241773AbhIALUl (ORCPT ); Wed, 1 Sep 2021 07:20:41 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:35100 "EHLO mx0b-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234678AbhIALUi (ORCPT ); Wed, 1 Sep 2021 07:20:38 -0400 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 181B2k2M081119; Wed, 1 Sep 2021 07:19:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : subject : to : cc : message-id : date : mime-version : content-type : content-transfer-encoding; s=pp1; bh=1uLl+JCz4S9e6TN+jLW/Ny/+ozUjBZuusm7jdYXuDbc=; b=aLPlckcqnJDWF2cEBPNEkmUQ+nV6MBNHc8sU0vGUsDHj6b66R8lvhm+fisrpJyBZEq4A UXYS3dqkcXX1UsJT16WwkLyD7HLnvobJq70VwOSjHZxUWdSJ/PUNETFd+wf1JvUurrao K7KyaxpV6B7R8gZOqlZR6zAGImPZyFT5wHse+y9UGoBvJGZdq5L96JVzrhbt5ZZIlnsw 2KKFHYvt7l3K/LML+QvAJuZF+HKImYnPfIiSs+d6mMilL5m3mLYSEacMxdAV82xxx3ak 5rKway2hhuOHardXybapfxwgHNa8D1g8Svpb0YYMfye7BYNXUdEzg10o5U6bToEGxDI+ Zw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3at56b600m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 07:19:27 -0400 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 181B3Bjg083331; Wed, 1 Sep 2021 07:19:27 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 3at56b600b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 07:19:27 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 181B6hFq030147; Wed, 1 Sep 2021 11:19:26 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma03dal.us.ibm.com with ESMTP id 3aqcsdfs7k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 11:19:26 +0000 Received: from b01ledav002.gho.pok.ibm.com (b01ledav002.gho.pok.ibm.com [9.57.199.107]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 181BJPn042205626 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 1 Sep 2021 11:19:25 GMT Received: from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3090A12416D; Wed, 1 Sep 2021 11:19:25 +0000 (GMT) Received: from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 31E67124164; Wed, 1 Sep 2021 11:19:20 +0000 (GMT) Received: from [9.43.110.185] (unknown [9.43.110.185]) by b01ledav002.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 1 Sep 2021 11:19:19 +0000 (GMT) From: Abdul Haleem Subject: [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target To: linux-kernel Cc: linux-scsi , linux-next , hch@lst.de, jack@suse.cz, axboe@kernel.dk, linuxppc-dev , dm-devel@redhat.com, sachinp , Brian King , dougmill@us.ibm.com Message-ID: <68dde454-965a-0c44-374a-a0ca277150ee@linux.vnet.ibm.com> Date: Wed, 1 Sep 2021 16:47:26 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-TM-AS-GCONF: 00 X-Proofpoint-GUID: ODi_EXi2QsMDI05CgcyxWJs4FQ6SWvtN X-Proofpoint-ORIG-GUID: _jbh_IrJ6PcIkJ9w3qX0Kua1qO_ZKvIT X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-09-01_03:2021-09-01,2021-09-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 adultscore=0 malwarescore=0 priorityscore=1501 phishscore=0 suspectscore=0 mlxscore=0 clxscore=1011 mlxlogscore=999 lowpriorityscore=0 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2109010065 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Greeting's multiple task hung while adding the vfc disk back to the multipath on my powerpc box running linux-next kernel Test: $ multipathd -k"remove path sde" $ multipathd -k"add path sde" After above command multipathd task hung with continuous call traces on console, it requires reboot to stop call traces systemd-udevd[180359]: Process '/sbin/mdadm -I /dev/dm-8' failed with exit code 1. multipathd[180272]: mpathf: load table [0 62914560 multipath 1 queue_if_no_path 1 alua 2 1 round-robin 0 2 1 8:64 1 65:0 1 round-robin 0 2 1 8:144 1 65:80 1] systemd[1]: Started Device-Mapper Multipath Device Controller. multipathd[180272]: mpatha: sdl - tur checker reports path is up multipathd[180272]: 8:176: reinstated multipathd[180272]: mpatha: remaining active paths: 2 multipathd[180272]: sde: remove path (operator) multipathd[180272]: mpathf: load table [0 62914560 multipath 1 queue_if_no_path 1 alua 2 1 round-robin 0 1 1 65:0 1 round-robin 0 2 1 8:144 1 65:80 1] multipathd[180272]: sde [8:64]: path removed from map mpathf multipathd[180272]: sdq: remove path (operator) multipathd[180272]: mpathf: load table [0 62914560 multipath 1 queue_if_no_path 1 alua 1 1 round-robin 0 2 1 8:144 1 65:80 1] multipathd[180272]: sdq [65:0]: path removed from map mpathf multipathd[180272]: sdj: remove path (operator) multipathd[180272]: mpathf: load table [0 62914560 multipath 1 queue_if_no_path 1 alua 1 1 round-robin 0 1 1 65:80 1] multipathd[180272]: sdj [8:144]: path removed from map mpathf multipathd[180272]: sdv: remove path (operator) multipathd[180272]: mpathf: map in use multipathd[180272]: mpathf: can't flush multipathd[180272]: mpathf: load table [0 62914560 multipath 1 queue_if_no_path 0 0 0] multipathd[180272]: sdv [65:80]: path removed from map mpathf systemd[1]: Starting system activity accounting tool... systemd[1]: sysstat-collect.service: Succeeded. systemd[1]: Started system activity accounting tool. systemd-udevd[1156]: seq 5678 '/devices/virtual/block/dm-10' is taking a long time systemd-udevd[1156]: seq 5678 '/devices/virtual/block/dm-10' killed multipathd[180272]: sde: add path (operator) systemd[1]: Stopping Device-Mapper Multipath Device Controller... systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing. systemd[1]: multipathd.service: Killing process 180272 (multipathd) with signal SIGKILL. systemd[1]: multipathd.service: Processes still around after SIGKILL. Ignoring. dbus-daemon[1920]: [system] Activating via systemd: service name='net.reactivated.Fprint' unit='fprintd.service' requested by ':1.255' (uid=0 pid=95173 comm="/bin/login -p --      ") systemd[1]: Starting Fingerprint Authentication Daemon... dbus-daemon[1920]: [system] Successfully activated service 'net.reactivated.Fprint' systemd[1]: Started Fingerprint Authentication Daemon. systemd-logind[2032]: New session 6 of user root. systemd[1]: Started Session 6 of user root. kernel: INFO: task multipathd:180274 blocked for more than 122 seconds. kernel:      Not tainted 5.14.0-rc7-next-20210827-autotest #1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kernel: task:multipathd      state:D stack:    0 pid:180274 ppid:     1 flags:0x00040082 kernel: Call Trace: kernel: [c00000002aaa72c0] [c00000002aaa73a0] 0xc00000002aaa73a0 (unreliable) kernel: [c00000002aaa74b0] [c00000000001e638] __switch_to+0x278/0x490 kernel: [c00000002aaa7510] [c000000000c89cfc] __schedule+0x31c/0xa10 kernel: [c00000002aaa75d0] [c000000000c8a468] schedule+0x78/0x130 kernel: [c00000002aaa7600] [c000000000c8aa08] schedule_preempt_disabled+0x18/0x30 kernel: [c00000002aaa7620] [c000000000c8cf0c] __mutex_lock.isra.11+0x36c/0x700 kernel: [c00000002aaa76b0] [c00000000067efe4] bd_link_disk_holder+0x34/0x270 kernel: [c00000002aaa7700] [c0080000003e3b78] dm_get_table_device+0x1e0/0x2c0 [dm_mod] kernel: [c00000002aaa77a0] [c0080000003e7e48] dm_get_device+0x130/0x2f0 [dm_mod] kernel: [c00000002aaa7850] [c008000000745234] multipath_ctr+0x9bc/0xff0 [dm_multipath] kernel: [c00000002aaa79d0] [c0080000003e883c] dm_table_add_target+0x1a4/0x420 [dm_mod] kernel: [c00000002aaa7a90] [c0080000003ee874] table_load+0x15c/0x4a0 [dm_mod] kernel: [c00000002aaa7b40] [c0080000003f1454] ctl_ioctl+0x27c/0x770 [dm_mod] kernel: [c00000002aaa7d40] [c0080000003f1960] dm_ctl_ioctl+0x18/0x30 [dm_mod] kernel: [c00000002aaa7d60] [c000000000481198] sys_ioctl+0xf8/0x150 kernel: [c00000002aaa7db0] [c00000000002ff28] system_call_exception+0x158/0x2c0 kernel: [c00000002aaa7e10] [c00000000000c764] system_call_common+0xf4/0x258 kernel: --- interrupt: c00 at 0x7fffa06f4480 kernel: NIP:  00007fffa06f4480 LR: 00007fffa0ad6714 CTR: 0000000000000000 kernel: REGS: c00000002aaa7e80 TRAP: 0c00   Not tainted (5.14.0-rc7-next-20210827-autotest) kernel: MSR:  800000000000d033   CR: 24042204  XER: 00000000 kernel: IRQMASK: 0 #012GPR00: 0000000000000036 00007fff9fd4c3a0 00007fffa07e7100 0000000000000005 #012GPR04: 00000000c138fd09 00007fff9806a0c0 00007fffa0ad9f18 00007fff9fd4a298 #012GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000 #012GPR12: 0000000000000000 00007fff9fd56300 00007fff9806a0c0 00007fffa0ad9c80 #012GPR16: 00007fffa0ad9c80 00007fffa0ad9c80 00007fffa0b13670 0000000000000000 #012GPR20: 00007fffa0ae3260 00007fffa0b12040 00007fff9806a0f0 00007fff9800eca0 #012GPR24: 00007fffa0ad9c80 00007fffa0ad9c80 00007fffa0ad9c80 0000000000000000 #012GPR28: 00007fffa0ad9c80 00007fffa0ad9c80 0000000000000000 00007fffa0ad9c80 kernel: NIP [00007fffa06f4480] 0x7fffa06f4480 kernel: LR [00007fffa0ad6714] 0x7fffa0ad6714 kernel: --- interrupt: c00 kernel: INFO: task systemd-udevd:180404 blocked for more than 122 seconds. kernel:      Not tainted 5.14.0-rc7-next-20210827-autotest #1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kernel: task:systemd-udevd   state:D stack:    0 pid:180404 ppid:  1156 flags:0x00042482 kernel: Call Trace: Before test ------------ $multipath -ll mpathf (360050768108001b3a8000000000000e8) dm-10 IBM,2145 size=30G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw |-+- policy='round-robin 0' prio=50 status=active | |- 1:0:0:4 sde 8:64  active ready running | `- 2:0:0:4 sdq 65:0  active ready running `-+- policy='round-robin 0' prio=10 status=enabled   |- 1:0:1:4 sdj 8:144 active ready running   `- 2:0:1:4 sdv 65:80 active ready running $ After test fail ------------ $ multipath -ll mpathf (360050768108001b3a8000000000000e8) dm-10 ##,## size=30G features='1 queue_if_no_path' hwhandler='0' wp=rw $ -- Regard's Abdul Haleem IBM Linux Technology Center From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB636C4320A for ; Wed, 1 Sep 2021 11:20:19 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 12B6D61053 for ; Wed, 1 Sep 2021 11:20:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 12B6D61053 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.ozlabs.org Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4H01nF12Hjz2ynB for ; Wed, 1 Sep 2021 21:20:17 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=aLPlckcq; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=linux.vnet.ibm.com (client-ip=148.163.158.5; helo=mx0b-001b2d01.pphosted.com; envelope-from=abdhalee@linux.vnet.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=aLPlckcq; dkim-atps=neutral Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4H01mQ1Hn4z2xvM for ; Wed, 1 Sep 2021 21:19:33 +1000 (AEST) Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 181B2k2M081119; Wed, 1 Sep 2021 07:19:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : subject : to : cc : message-id : date : mime-version : content-type : content-transfer-encoding; s=pp1; bh=1uLl+JCz4S9e6TN+jLW/Ny/+ozUjBZuusm7jdYXuDbc=; b=aLPlckcqnJDWF2cEBPNEkmUQ+nV6MBNHc8sU0vGUsDHj6b66R8lvhm+fisrpJyBZEq4A UXYS3dqkcXX1UsJT16WwkLyD7HLnvobJq70VwOSjHZxUWdSJ/PUNETFd+wf1JvUurrao K7KyaxpV6B7R8gZOqlZR6zAGImPZyFT5wHse+y9UGoBvJGZdq5L96JVzrhbt5ZZIlnsw 2KKFHYvt7l3K/LML+QvAJuZF+HKImYnPfIiSs+d6mMilL5m3mLYSEacMxdAV82xxx3ak 5rKway2hhuOHardXybapfxwgHNa8D1g8Svpb0YYMfye7BYNXUdEzg10o5U6bToEGxDI+ Zw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3at56b600m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 07:19:27 -0400 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 181B3Bjg083331; Wed, 1 Sep 2021 07:19:27 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 3at56b600b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 07:19:27 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 181B6hFq030147; Wed, 1 Sep 2021 11:19:26 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma03dal.us.ibm.com with ESMTP id 3aqcsdfs7k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 11:19:26 +0000 Received: from b01ledav002.gho.pok.ibm.com (b01ledav002.gho.pok.ibm.com [9.57.199.107]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 181BJPn042205626 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 1 Sep 2021 11:19:25 GMT Received: from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3090A12416D; Wed, 1 Sep 2021 11:19:25 +0000 (GMT) Received: from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 31E67124164; Wed, 1 Sep 2021 11:19:20 +0000 (GMT) Received: from [9.43.110.185] (unknown [9.43.110.185]) by b01ledav002.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 1 Sep 2021 11:19:19 +0000 (GMT) From: Abdul Haleem Subject: [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target To: linux-kernel Message-ID: <68dde454-965a-0c44-374a-a0ca277150ee@linux.vnet.ibm.com> Date: Wed, 1 Sep 2021 16:47:26 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-TM-AS-GCONF: 00 X-Proofpoint-GUID: ODi_EXi2QsMDI05CgcyxWJs4FQ6SWvtN X-Proofpoint-ORIG-GUID: _jbh_IrJ6PcIkJ9w3qX0Kua1qO_ZKvIT X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-09-01_03:2021-09-01, 2021-09-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 adultscore=0 malwarescore=0 priorityscore=1501 phishscore=0 suspectscore=0 mlxscore=0 clxscore=1011 mlxlogscore=999 lowpriorityscore=0 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2109010065 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: axboe@kernel.dk, sachinp , jack@suse.cz, linux-scsi , dm-devel@redhat.com, linux-next , dougmill@us.ibm.com, Brian King , linuxppc-dev , hch@lst.de Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" Greeting's multiple task hung while adding the vfc disk back to the multipath on my powerpc box running linux-next kernel Test: $ multipathd -k"remove path sde" $ multipathd -k"add path sde" After above command multipathd task hung with continuous call traces on console, it requires reboot to stop call traces systemd-udevd[180359]: Process '/sbin/mdadm -I /dev/dm-8' failed with exit code 1. multipathd[180272]: mpathf: load table [0 62914560 multipath 1 queue_if_no_path 1 alua 2 1 round-robin 0 2 1 8:64 1 65:0 1 round-robin 0 2 1 8:144 1 65:80 1] systemd[1]: Started Device-Mapper Multipath Device Controller. multipathd[180272]: mpatha: sdl - tur checker reports path is up multipathd[180272]: 8:176: reinstated multipathd[180272]: mpatha: remaining active paths: 2 multipathd[180272]: sde: remove path (operator) multipathd[180272]: mpathf: load table [0 62914560 multipath 1 queue_if_no_path 1 alua 2 1 round-robin 0 1 1 65:0 1 round-robin 0 2 1 8:144 1 65:80 1] multipathd[180272]: sde [8:64]: path removed from map mpathf multipathd[180272]: sdq: remove path (operator) multipathd[180272]: mpathf: load table [0 62914560 multipath 1 queue_if_no_path 1 alua 1 1 round-robin 0 2 1 8:144 1 65:80 1] multipathd[180272]: sdq [65:0]: path removed from map mpathf multipathd[180272]: sdj: remove path (operator) multipathd[180272]: mpathf: load table [0 62914560 multipath 1 queue_if_no_path 1 alua 1 1 round-robin 0 1 1 65:80 1] multipathd[180272]: sdj [8:144]: path removed from map mpathf multipathd[180272]: sdv: remove path (operator) multipathd[180272]: mpathf: map in use multipathd[180272]: mpathf: can't flush multipathd[180272]: mpathf: load table [0 62914560 multipath 1 queue_if_no_path 0 0 0] multipathd[180272]: sdv [65:80]: path removed from map mpathf systemd[1]: Starting system activity accounting tool... systemd[1]: sysstat-collect.service: Succeeded. systemd[1]: Started system activity accounting tool. systemd-udevd[1156]: seq 5678 '/devices/virtual/block/dm-10' is taking a long time systemd-udevd[1156]: seq 5678 '/devices/virtual/block/dm-10' killed multipathd[180272]: sde: add path (operator) systemd[1]: Stopping Device-Mapper Multipath Device Controller... systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing. systemd[1]: multipathd.service: Killing process 180272 (multipathd) with signal SIGKILL. systemd[1]: multipathd.service: Processes still around after SIGKILL. Ignoring. dbus-daemon[1920]: [system] Activating via systemd: service name='net.reactivated.Fprint' unit='fprintd.service' requested by ':1.255' (uid=0 pid=95173 comm="/bin/login -p --      ") systemd[1]: Starting Fingerprint Authentication Daemon... dbus-daemon[1920]: [system] Successfully activated service 'net.reactivated.Fprint' systemd[1]: Started Fingerprint Authentication Daemon. systemd-logind[2032]: New session 6 of user root. systemd[1]: Started Session 6 of user root. kernel: INFO: task multipathd:180274 blocked for more than 122 seconds. kernel:      Not tainted 5.14.0-rc7-next-20210827-autotest #1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kernel: task:multipathd      state:D stack:    0 pid:180274 ppid:     1 flags:0x00040082 kernel: Call Trace: kernel: [c00000002aaa72c0] [c00000002aaa73a0] 0xc00000002aaa73a0 (unreliable) kernel: [c00000002aaa74b0] [c00000000001e638] __switch_to+0x278/0x490 kernel: [c00000002aaa7510] [c000000000c89cfc] __schedule+0x31c/0xa10 kernel: [c00000002aaa75d0] [c000000000c8a468] schedule+0x78/0x130 kernel: [c00000002aaa7600] [c000000000c8aa08] schedule_preempt_disabled+0x18/0x30 kernel: [c00000002aaa7620] [c000000000c8cf0c] __mutex_lock.isra.11+0x36c/0x700 kernel: [c00000002aaa76b0] [c00000000067efe4] bd_link_disk_holder+0x34/0x270 kernel: [c00000002aaa7700] [c0080000003e3b78] dm_get_table_device+0x1e0/0x2c0 [dm_mod] kernel: [c00000002aaa77a0] [c0080000003e7e48] dm_get_device+0x130/0x2f0 [dm_mod] kernel: [c00000002aaa7850] [c008000000745234] multipath_ctr+0x9bc/0xff0 [dm_multipath] kernel: [c00000002aaa79d0] [c0080000003e883c] dm_table_add_target+0x1a4/0x420 [dm_mod] kernel: [c00000002aaa7a90] [c0080000003ee874] table_load+0x15c/0x4a0 [dm_mod] kernel: [c00000002aaa7b40] [c0080000003f1454] ctl_ioctl+0x27c/0x770 [dm_mod] kernel: [c00000002aaa7d40] [c0080000003f1960] dm_ctl_ioctl+0x18/0x30 [dm_mod] kernel: [c00000002aaa7d60] [c000000000481198] sys_ioctl+0xf8/0x150 kernel: [c00000002aaa7db0] [c00000000002ff28] system_call_exception+0x158/0x2c0 kernel: [c00000002aaa7e10] [c00000000000c764] system_call_common+0xf4/0x258 kernel: --- interrupt: c00 at 0x7fffa06f4480 kernel: NIP:  00007fffa06f4480 LR: 00007fffa0ad6714 CTR: 0000000000000000 kernel: REGS: c00000002aaa7e80 TRAP: 0c00   Not tainted (5.14.0-rc7-next-20210827-autotest) kernel: MSR:  800000000000d033   CR: 24042204  XER: 00000000 kernel: IRQMASK: 0 #012GPR00: 0000000000000036 00007fff9fd4c3a0 00007fffa07e7100 0000000000000005 #012GPR04: 00000000c138fd09 00007fff9806a0c0 00007fffa0ad9f18 00007fff9fd4a298 #012GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000 #012GPR12: 0000000000000000 00007fff9fd56300 00007fff9806a0c0 00007fffa0ad9c80 #012GPR16: 00007fffa0ad9c80 00007fffa0ad9c80 00007fffa0b13670 0000000000000000 #012GPR20: 00007fffa0ae3260 00007fffa0b12040 00007fff9806a0f0 00007fff9800eca0 #012GPR24: 00007fffa0ad9c80 00007fffa0ad9c80 00007fffa0ad9c80 0000000000000000 #012GPR28: 00007fffa0ad9c80 00007fffa0ad9c80 0000000000000000 00007fffa0ad9c80 kernel: NIP [00007fffa06f4480] 0x7fffa06f4480 kernel: LR [00007fffa0ad6714] 0x7fffa0ad6714 kernel: --- interrupt: c00 kernel: INFO: task systemd-udevd:180404 blocked for more than 122 seconds. kernel:      Not tainted 5.14.0-rc7-next-20210827-autotest #1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kernel: task:systemd-udevd   state:D stack:    0 pid:180404 ppid:  1156 flags:0x00042482 kernel: Call Trace: Before test ------------ $multipath -ll mpathf (360050768108001b3a8000000000000e8) dm-10 IBM,2145 size=30G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw |-+- policy='round-robin 0' prio=50 status=active | |- 1:0:0:4 sde 8:64  active ready running | `- 2:0:0:4 sdq 65:0  active ready running `-+- policy='round-robin 0' prio=10 status=enabled   |- 1:0:1:4 sdj 8:144 active ready running   `- 2:0:1:4 sdv 65:80 active ready running $ After test fail ------------ $ multipath -ll mpathf (360050768108001b3a8000000000000e8) dm-10 ##,## size=30G features='1 queue_if_no_path' hwhandler='0' wp=rw $ -- Regard's Abdul Haleem IBM Linux Technology Center From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F7DEC432BE for ; Thu, 2 Sep 2021 07:17:06 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DFE2460FDA for ; Thu, 2 Sep 2021 07:17:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DFE2460FDA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.vnet.ibm.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-439-Rz1RrBAePYmFjw7GdIuQuQ-1; Thu, 02 Sep 2021 03:17:01 -0400 X-MC-Unique: Rz1RrBAePYmFjw7GdIuQuQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EB40F6D4E0; Thu, 2 Sep 2021 07:16:55 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1E2D227CA4; Thu, 2 Sep 2021 07:16:54 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 312DA1809C98; Thu, 2 Sep 2021 07:16:51 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 181BJXL7018704 for ; Wed, 1 Sep 2021 07:19:33 -0400 Received: by smtp.corp.redhat.com (Postfix) id 471A6100CA22; Wed, 1 Sep 2021 11:19:33 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast06.extmail.prod.ext.rdu2.redhat.com [10.11.55.22]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4244311558B5 for ; Wed, 1 Sep 2021 11:19:30 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 528E918A0161 for ; Wed, 1 Sep 2021 11:19:30 +0000 (UTC) Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-17-BIlVhXg2N4WshgacL5Xwmg-1; Wed, 01 Sep 2021 07:19:28 -0400 X-MC-Unique: BIlVhXg2N4WshgacL5Xwmg-1 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 181B2kIN081138 for ; Wed, 1 Sep 2021 07:19:28 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3at56b600j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 07:19:27 -0400 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 181B3Bjf083331 for ; Wed, 1 Sep 2021 07:19:27 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 3at56b600b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 07:19:27 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 181B6hFq030147; Wed, 1 Sep 2021 11:19:26 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma03dal.us.ibm.com with ESMTP id 3aqcsdfs7k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 11:19:26 +0000 Received: from b01ledav002.gho.pok.ibm.com (b01ledav002.gho.pok.ibm.com [9.57.199.107]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 181BJPn042205626 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 1 Sep 2021 11:19:25 GMT Received: from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3090A12416D; Wed, 1 Sep 2021 11:19:25 +0000 (GMT) Received: from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 31E67124164; Wed, 1 Sep 2021 11:19:20 +0000 (GMT) Received: from [9.43.110.185] (unknown [9.43.110.185]) by b01ledav002.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 1 Sep 2021 11:19:19 +0000 (GMT) From: Abdul Haleem To: linux-kernel Message-ID: <68dde454-965a-0c44-374a-a0ca277150ee@linux.vnet.ibm.com> Date: Wed, 1 Sep 2021 16:47:26 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: J6Nhi7OSXE_qj8Ne99icUggqGp_3IeKu X-Proofpoint-ORIG-GUID: _jbh_IrJ6PcIkJ9w3qX0Kua1qO_ZKvIT X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-09-01_03:2021-09-01, 2021-09-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 adultscore=0 malwarescore=0 priorityscore=1501 phishscore=0 suspectscore=0 mlxscore=0 clxscore=1011 mlxlogscore=956 lowpriorityscore=0 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2109010065 X-MIME-Autoconverted: from 8bit to quoted-printable by mx0a-001b2d01.pphosted.com id 181B2kIN081138 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-MIME-Autoconverted: from quoted-printable to 8bit by lists01.pubmisc.prod.ext.phx2.redhat.com id 181BJXL7018704 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Thu, 02 Sep 2021 03:11:11 -0400 Cc: axboe@kernel.dk, sachinp , jack@suse.cz, linux-scsi , dm-devel@redhat.com, linux-next , dougmill@us.ibm.com, Brian King , linuxppc-dev , hch@lst.de Subject: [dm-devel] [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" R3JlZXRpbmcncwoKbXVsdGlwbGUgdGFzayBodW5nIHdoaWxlIGFkZGluZyB0aGUgdmZjIGRpc2sg YmFjayB0byB0aGUgbXVsdGlwYXRoIG9uIG15IApwb3dlcnBjIGJveCBydW5uaW5nIGxpbnV4LW5l eHQga2VybmVsCgpUZXN0OgokIG11bHRpcGF0aGQgLWsicmVtb3ZlIHBhdGggc2RlIgokIG11bHRp cGF0aGQgLWsiYWRkIHBhdGggc2RlIgoKQWZ0ZXIgYWJvdmUgY29tbWFuZCBtdWx0aXBhdGhkIHRh c2sgaHVuZyB3aXRoIGNvbnRpbnVvdXMgY2FsbCB0cmFjZXMgb24gCmNvbnNvbGUsIGl0IHJlcXVp cmVzIHJlYm9vdCB0byBzdG9wIGNhbGwgdHJhY2VzCgpzeXN0ZW1kLXVkZXZkWzE4MDM1OV06IFBy b2Nlc3MgJy9zYmluL21kYWRtIC1JIC9kZXYvZG0tOCcgZmFpbGVkIHdpdGggCmV4aXQgY29kZSAx LgptdWx0aXBhdGhkWzE4MDI3Ml06IG1wYXRoZjogbG9hZCB0YWJsZSBbMCA2MjkxNDU2MCBtdWx0 aXBhdGggMSAKcXVldWVfaWZfbm9fcGF0aCAxIGFsdWEgMiAxIHJvdW5kLXJvYmluIDAgMiAxIDg6 NjQgMSA2NTowIDEgcm91bmQtcm9iaW4gCjAgMiAxIDg6MTQ0IDEgNjU6ODAgMV0Kc3lzdGVtZFsx XTogU3RhcnRlZCBEZXZpY2UtTWFwcGVyIE11bHRpcGF0aCBEZXZpY2UgQ29udHJvbGxlci4KbXVs dGlwYXRoZFsxODAyNzJdOiBtcGF0aGE6IHNkbCAtIHR1ciBjaGVja2VyIHJlcG9ydHMgcGF0aCBp cyB1cAptdWx0aXBhdGhkWzE4MDI3Ml06IDg6MTc2OiByZWluc3RhdGVkCm11bHRpcGF0aGRbMTgw MjcyXTogbXBhdGhhOiByZW1haW5pbmcgYWN0aXZlIHBhdGhzOiAyCm11bHRpcGF0aGRbMTgwMjcy XTogc2RlOiByZW1vdmUgcGF0aCAob3BlcmF0b3IpCm11bHRpcGF0aGRbMTgwMjcyXTogbXBhdGhm OiBsb2FkIHRhYmxlIFswIDYyOTE0NTYwIG11bHRpcGF0aCAxIApxdWV1ZV9pZl9ub19wYXRoIDEg YWx1YSAyIDEgcm91bmQtcm9iaW4gMCAxIDEgNjU6MCAxIHJvdW5kLXJvYmluIDAgMiAxIAo4OjE0 NCAxIDY1OjgwIDFdCm11bHRpcGF0aGRbMTgwMjcyXTogc2RlIFs4OjY0XTogcGF0aCByZW1vdmVk IGZyb20gbWFwIG1wYXRoZgptdWx0aXBhdGhkWzE4MDI3Ml06IHNkcTogcmVtb3ZlIHBhdGggKG9w ZXJhdG9yKQptdWx0aXBhdGhkWzE4MDI3Ml06IG1wYXRoZjogbG9hZCB0YWJsZSBbMCA2MjkxNDU2 MCBtdWx0aXBhdGggMSAKcXVldWVfaWZfbm9fcGF0aCAxIGFsdWEgMSAxIHJvdW5kLXJvYmluIDAg MiAxIDg6MTQ0IDEgNjU6ODAgMV0KbXVsdGlwYXRoZFsxODAyNzJdOiBzZHEgWzY1OjBdOiBwYXRo IHJlbW92ZWQgZnJvbSBtYXAgbXBhdGhmCm11bHRpcGF0aGRbMTgwMjcyXTogc2RqOiByZW1vdmUg cGF0aCAob3BlcmF0b3IpCm11bHRpcGF0aGRbMTgwMjcyXTogbXBhdGhmOiBsb2FkIHRhYmxlIFsw IDYyOTE0NTYwIG11bHRpcGF0aCAxIApxdWV1ZV9pZl9ub19wYXRoIDEgYWx1YSAxIDEgcm91bmQt cm9iaW4gMCAxIDEgNjU6ODAgMV0KbXVsdGlwYXRoZFsxODAyNzJdOiBzZGogWzg6MTQ0XTogcGF0 aCByZW1vdmVkIGZyb20gbWFwIG1wYXRoZgptdWx0aXBhdGhkWzE4MDI3Ml06IHNkdjogcmVtb3Zl IHBhdGggKG9wZXJhdG9yKQptdWx0aXBhdGhkWzE4MDI3Ml06IG1wYXRoZjogbWFwIGluIHVzZQpt dWx0aXBhdGhkWzE4MDI3Ml06IG1wYXRoZjogY2FuJ3QgZmx1c2gKbXVsdGlwYXRoZFsxODAyNzJd OiBtcGF0aGY6IGxvYWQgdGFibGUgWzAgNjI5MTQ1NjAgbXVsdGlwYXRoIDEgCnF1ZXVlX2lmX25v X3BhdGggMCAwIDBdCm11bHRpcGF0aGRbMTgwMjcyXTogc2R2IFs2NTo4MF06IHBhdGggcmVtb3Zl ZCBmcm9tIG1hcCBtcGF0aGYKc3lzdGVtZFsxXTogU3RhcnRpbmcgc3lzdGVtIGFjdGl2aXR5IGFj Y291bnRpbmcgdG9vbC4uLgpzeXN0ZW1kWzFdOiBzeXNzdGF0LWNvbGxlY3Quc2VydmljZTogU3Vj Y2VlZGVkLgpzeXN0ZW1kWzFdOiBTdGFydGVkIHN5c3RlbSBhY3Rpdml0eSBhY2NvdW50aW5nIHRv b2wuCnN5c3RlbWQtdWRldmRbMTE1Nl06IHNlcSA1Njc4ICcvZGV2aWNlcy92aXJ0dWFsL2Jsb2Nr L2RtLTEwJyBpcyB0YWtpbmcgYSAKbG9uZyB0aW1lCnN5c3RlbWQtdWRldmRbMTE1Nl06IHNlcSA1 Njc4ICcvZGV2aWNlcy92aXJ0dWFsL2Jsb2NrL2RtLTEwJyBraWxsZWQKbXVsdGlwYXRoZFsxODAy NzJdOiBzZGU6IGFkZCBwYXRoIChvcGVyYXRvcikKc3lzdGVtZFsxXTogU3RvcHBpbmcgRGV2aWNl LU1hcHBlciBNdWx0aXBhdGggRGV2aWNlIENvbnRyb2xsZXIuLi4Kc3lzdGVtZFsxXTogbXVsdGlw YXRoZC5zZXJ2aWNlOiBTdGF0ZSAnc3RvcC1zaWd0ZXJtJyB0aW1lZCBvdXQuIEtpbGxpbmcuCnN5 c3RlbWRbMV06IG11bHRpcGF0aGQuc2VydmljZTogS2lsbGluZyBwcm9jZXNzIDE4MDI3MiAobXVs dGlwYXRoZCkgd2l0aCAKc2lnbmFsIFNJR0tJTEwuCnN5c3RlbWRbMV06IG11bHRpcGF0aGQuc2Vy dmljZTogUHJvY2Vzc2VzIHN0aWxsIGFyb3VuZCBhZnRlciBTSUdLSUxMLiAKSWdub3JpbmcuCmRi dXMtZGFlbW9uWzE5MjBdOiBbc3lzdGVtXSBBY3RpdmF0aW5nIHZpYSBzeXN0ZW1kOiBzZXJ2aWNl IApuYW1lPSduZXQucmVhY3RpdmF0ZWQuRnByaW50JyB1bml0PSdmcHJpbnRkLnNlcnZpY2UnIHJl cXVlc3RlZCBieSAKJzoxLjI1NScgKHVpZD0wIHBpZD05NTE3MyBjb21tPSIvYmluL2xvZ2luIC1w IC0twqDCoMKgwqDCoCAiKQpzeXN0ZW1kWzFdOiBTdGFydGluZyBGaW5nZXJwcmludCBBdXRoZW50 aWNhdGlvbiBEYWVtb24uLi4KZGJ1cy1kYWVtb25bMTkyMF06IFtzeXN0ZW1dIFN1Y2Nlc3NmdWxs eSBhY3RpdmF0ZWQgc2VydmljZSAKJ25ldC5yZWFjdGl2YXRlZC5GcHJpbnQnCnN5c3RlbWRbMV06 IFN0YXJ0ZWQgRmluZ2VycHJpbnQgQXV0aGVudGljYXRpb24gRGFlbW9uLgpzeXN0ZW1kLWxvZ2lu ZFsyMDMyXTogTmV3IHNlc3Npb24gNiBvZiB1c2VyIHJvb3QuCnN5c3RlbWRbMV06IFN0YXJ0ZWQg U2Vzc2lvbiA2IG9mIHVzZXIgcm9vdC4Ka2VybmVsOiBJTkZPOiB0YXNrIG11bHRpcGF0aGQ6MTgw Mjc0IGJsb2NrZWQgZm9yIG1vcmUgdGhhbiAxMjIgc2Vjb25kcy4Ka2VybmVsOsKgwqDCoMKgwqAg Tm90IHRhaW50ZWQgNS4xNC4wLXJjNy1uZXh0LTIwMjEwODI3LWF1dG90ZXN0ICMxCmtlcm5lbDog ImVjaG8gMCA+IC9wcm9jL3N5cy9rZXJuZWwvaHVuZ190YXNrX3RpbWVvdXRfc2VjcyIgZGlzYWJs ZXMgdGhpcyAKbWVzc2FnZS4Ka2VybmVsOiB0YXNrOm11bHRpcGF0aGTCoMKgwqDCoMKgIHN0YXRl OkQgc3RhY2s6wqDCoMKgIDAgcGlkOjE4MDI3NCBwcGlkOsKgwqDCoMKgIDEgCmZsYWdzOjB4MDAw NDAwODIKa2VybmVsOiBDYWxsIFRyYWNlOgprZXJuZWw6IFtjMDAwMDAwMDJhYWE3MmMwXSBbYzAw MDAwMDAyYWFhNzNhMF0gMHhjMDAwMDAwMDJhYWE3M2EwIAoodW5yZWxpYWJsZSkKa2VybmVsOiBb YzAwMDAwMDAyYWFhNzRiMF0gW2MwMDAwMDAwMDAwMWU2MzhdIF9fc3dpdGNoX3RvKzB4Mjc4LzB4 NDkwCmtlcm5lbDogW2MwMDAwMDAwMmFhYTc1MTBdIFtjMDAwMDAwMDAwYzg5Y2ZjXSBfX3NjaGVk dWxlKzB4MzFjLzB4YTEwCmtlcm5lbDogW2MwMDAwMDAwMmFhYTc1ZDBdIFtjMDAwMDAwMDAwYzhh NDY4XSBzY2hlZHVsZSsweDc4LzB4MTMwCmtlcm5lbDogW2MwMDAwMDAwMmFhYTc2MDBdIFtjMDAw MDAwMDAwYzhhYTA4XSAKc2NoZWR1bGVfcHJlZW1wdF9kaXNhYmxlZCsweDE4LzB4MzAKa2VybmVs OiBbYzAwMDAwMDAyYWFhNzYyMF0gW2MwMDAwMDAwMDBjOGNmMGNdIApfX211dGV4X2xvY2suaXNy YS4xMSsweDM2Yy8weDcwMAprZXJuZWw6IFtjMDAwMDAwMDJhYWE3NmIwXSBbYzAwMDAwMDAwMDY3 ZWZlNF0gYmRfbGlua19kaXNrX2hvbGRlcisweDM0LzB4MjcwCmtlcm5lbDogW2MwMDAwMDAwMmFh YTc3MDBdIFtjMDA4MDAwMDAwM2UzYjc4XSAKZG1fZ2V0X3RhYmxlX2RldmljZSsweDFlMC8weDJj MCBbZG1fbW9kXQprZXJuZWw6IFtjMDAwMDAwMDJhYWE3N2EwXSBbYzAwODAwMDAwMDNlN2U0OF0g ZG1fZ2V0X2RldmljZSsweDEzMC8weDJmMCAKW2RtX21vZF0Ka2VybmVsOiBbYzAwMDAwMDAyYWFh Nzg1MF0gW2MwMDgwMDAwMDA3NDUyMzRdIG11bHRpcGF0aF9jdHIrMHg5YmMvMHhmZjAgCltkbV9t dWx0aXBhdGhdCmtlcm5lbDogW2MwMDAwMDAwMmFhYTc5ZDBdIFtjMDA4MDAwMDAwM2U4ODNjXSAK ZG1fdGFibGVfYWRkX3RhcmdldCsweDFhNC8weDQyMCBbZG1fbW9kXQprZXJuZWw6IFtjMDAwMDAw MDJhYWE3YTkwXSBbYzAwODAwMDAwMDNlZTg3NF0gdGFibGVfbG9hZCsweDE1Yy8weDRhMCAKW2Rt X21vZF0Ka2VybmVsOiBbYzAwMDAwMDAyYWFhN2I0MF0gW2MwMDgwMDAwMDAzZjE0NTRdIGN0bF9p b2N0bCsweDI3Yy8weDc3MCBbZG1fbW9kXQprZXJuZWw6IFtjMDAwMDAwMDJhYWE3ZDQwXSBbYzAw ODAwMDAwMDNmMTk2MF0gZG1fY3RsX2lvY3RsKzB4MTgvMHgzMCAKW2RtX21vZF0Ka2VybmVsOiBb YzAwMDAwMDAyYWFhN2Q2MF0gW2MwMDAwMDAwMDA0ODExOThdIHN5c19pb2N0bCsweGY4LzB4MTUw Cmtlcm5lbDogW2MwMDAwMDAwMmFhYTdkYjBdIFtjMDAwMDAwMDAwMDJmZjI4XSAKc3lzdGVtX2Nh bGxfZXhjZXB0aW9uKzB4MTU4LzB4MmMwCmtlcm5lbDogW2MwMDAwMDAwMmFhYTdlMTBdIFtjMDAw MDAwMDAwMDBjNzY0XSBzeXN0ZW1fY2FsbF9jb21tb24rMHhmNC8weDI1OAprZXJuZWw6IC0tLSBp bnRlcnJ1cHQ6IGMwMCBhdCAweDdmZmZhMDZmNDQ4MAprZXJuZWw6IE5JUDrCoCAwMDAwN2ZmZmEw NmY0NDgwIExSOiAwMDAwN2ZmZmEwYWQ2NzE0IENUUjogMDAwMDAwMDAwMDAwMDAwMAprZXJuZWw6 IFJFR1M6IGMwMDAwMDAwMmFhYTdlODAgVFJBUDogMGMwMMKgwqAgTm90IHRhaW50ZWQgCig1LjE0 LjAtcmM3LW5leHQtMjAyMTA4MjctYXV0b3Rlc3QpCmtlcm5lbDogTVNSOsKgIDgwMDAwMDAwMDAw MGQwMzMgPFNGLEVFLFBSLE1FLElSLERSLFJJLExFPsKgIENSOiAyNDA0MjIwNMKgIApYRVI6IDAw MDAwMDAwCmtlcm5lbDogSVJRTUFTSzogMCAjMDEyR1BSMDA6IDAwMDAwMDAwMDAwMDAwMzYgMDAw MDdmZmY5ZmQ0YzNhMCAKMDAwMDdmZmZhMDdlNzEwMCAwMDAwMDAwMDAwMDAwMDA1ICMwMTJHUFIw NDogMDAwMDAwMDBjMTM4ZmQwOSAKMDAwMDdmZmY5ODA2YTBjMCAwMDAwN2ZmZmEwYWQ5ZjE4IDAw MDA3ZmZmOWZkNGEyOTggIzAxMkdQUjA4OiAKMDAwMDAwMDAwMDAwMDAwNSAwMDAwMDAwMDAwMDAw MDAwIDAwMDAwMDAwMDAwMDAwMDAgMDAwMDAwMDAwMDAwMDAwMCAKIzAxMkdQUjEyOiAwMDAwMDAw MDAwMDAwMDAwIDAwMDA3ZmZmOWZkNTYzMDAgMDAwMDdmZmY5ODA2YTBjMCAKMDAwMDdmZmZhMGFk OWM4MCAjMDEyR1BSMTY6IDAwMDA3ZmZmYTBhZDljODAgMDAwMDdmZmZhMGFkOWM4MCAKMDAwMDdm ZmZhMGIxMzY3MCAwMDAwMDAwMDAwMDAwMDAwICMwMTJHUFIyMDogMDAwMDdmZmZhMGFlMzI2MCAK MDAwMDdmZmZhMGIxMjA0MCAwMDAwN2ZmZjk4MDZhMGYwIDAwMDA3ZmZmOTgwMGVjYTAgIzAxMkdQ UjI0OiAKMDAwMDdmZmZhMGFkOWM4MCAwMDAwN2ZmZmEwYWQ5YzgwIDAwMDA3ZmZmYTBhZDljODAg MDAwMDAwMDAwMDAwMDAwMCAKIzAxMkdQUjI4OiAwMDAwN2ZmZmEwYWQ5YzgwIDAwMDA3ZmZmYTBh ZDljODAgMDAwMDAwMDAwMDAwMDAwMCAKMDAwMDdmZmZhMGFkOWM4MAprZXJuZWw6IE5JUCBbMDAw MDdmZmZhMDZmNDQ4MF0gMHg3ZmZmYTA2ZjQ0ODAKa2VybmVsOiBMUiBbMDAwMDdmZmZhMGFkNjcx NF0gMHg3ZmZmYTBhZDY3MTQKa2VybmVsOiAtLS0gaW50ZXJydXB0OiBjMDAKa2VybmVsOiBJTkZP OiB0YXNrIHN5c3RlbWQtdWRldmQ6MTgwNDA0IGJsb2NrZWQgZm9yIG1vcmUgdGhhbiAxMjIgc2Vj b25kcy4Ka2VybmVsOsKgwqDCoMKgwqAgTm90IHRhaW50ZWQgNS4xNC4wLXJjNy1uZXh0LTIwMjEw ODI3LWF1dG90ZXN0ICMxCmtlcm5lbDogImVjaG8gMCA+IC9wcm9jL3N5cy9rZXJuZWwvaHVuZ190 YXNrX3RpbWVvdXRfc2VjcyIgZGlzYWJsZXMgdGhpcyAKbWVzc2FnZS4Ka2VybmVsOiB0YXNrOnN5 c3RlbWQtdWRldmTCoMKgIHN0YXRlOkQgc3RhY2s6wqDCoMKgIDAgcGlkOjE4MDQwNCBwcGlkOsKg IDExNTYgCmZsYWdzOjB4MDAwNDI0ODIKa2VybmVsOiBDYWxsIFRyYWNlOgoKQmVmb3JlIHRlc3QK LS0tLS0tLS0tLS0tCiRtdWx0aXBhdGggLWxsCm1wYXRoZiAoMzYwMDUwNzY4MTA4MDAxYjNhODAw MDAwMDAwMDAwMGU4KSBkbS0xMCBJQk0sMjE0NQpzaXplPTMwRyBmZWF0dXJlcz0nMSBxdWV1ZV9p Zl9ub19wYXRoJyBod2hhbmRsZXI9JzEgYWx1YScgd3A9cncKfC0rLSBwb2xpY3k9J3JvdW5kLXJv YmluIDAnIHByaW89NTAgc3RhdHVzPWFjdGl2ZQp8IHwtIDE6MDowOjQgc2RlIDg6NjTCoCBhY3Rp dmUgcmVhZHkgcnVubmluZwp8IGAtIDI6MDowOjQgc2RxIDY1OjDCoCBhY3RpdmUgcmVhZHkgcnVu bmluZwpgLSstIHBvbGljeT0ncm91bmQtcm9iaW4gMCcgcHJpbz0xMCBzdGF0dXM9ZW5hYmxlZAog wqAgfC0gMTowOjE6NCBzZGogODoxNDQgYWN0aXZlIHJlYWR5IHJ1bm5pbmcKIMKgIGAtIDI6MDox OjQgc2R2IDY1OjgwIGFjdGl2ZSByZWFkeSBydW5uaW5nCiQKCkFmdGVyIHRlc3QgZmFpbAotLS0t LS0tLS0tLS0KJCBtdWx0aXBhdGggLWxsCm1wYXRoZiAoMzYwMDUwNzY4MTA4MDAxYjNhODAwMDAw MDAwMDAwMGU4KSBkbS0xMCAjIywjIwpzaXplPTMwRyBmZWF0dXJlcz0nMSBxdWV1ZV9pZl9ub19w YXRoJyBod2hhbmRsZXI9JzAnIHdwPXJ3CiQKCi0tIApSZWdhcmQncwoKQWJkdWwgSGFsZWVtCklC TSBMaW51eCBUZWNobm9sb2d5IENlbnRlcgoKCi0tCmRtLWRldmVsIG1haWxpbmcgbGlzdApkbS1k ZXZlbEByZWRoYXQuY29tCmh0dHBzOi8vbGlzdG1hbi5yZWRoYXQuY29tL21haWxtYW4vbGlzdGlu Zm8vZG0tZGV2ZWw=