From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB754C4321D for ; Thu, 16 Aug 2018 17:09:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7295420C07 for ; Thu, 16 Aug 2018 17:09:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7295420C07 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=molgen.mpg.de Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728916AbeHPUIz (ORCPT ); Thu, 16 Aug 2018 16:08:55 -0400 Received: from mx3.molgen.mpg.de ([141.14.17.11]:60461 "EHLO mx1.molgen.mpg.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726864AbeHPUIz (ORCPT ); Thu, 16 Aug 2018 16:08:55 -0400 Received: from keineahnung.molgen.mpg.de (keineahnung.molgen.mpg.de [141.14.17.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: pmenzel) by mx.molgen.mpg.de (Postfix) with ESMTPSA id B830E2012D0DC0; Thu, 16 Aug 2018 19:09:07 +0200 (CEST) Subject: Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs* To: Ming Lei , Greg Kroah-Hartman Cc: stable@vger.kernel.org, Christoph Hellwig , Linux Kernel Mailing List , it+linux-scsi@molgen.mpg.de, Adaptec OEM Raid Solutions , linux-scsi@vger.kernel.org References: <06ee23fe-ec9e-d67b-b533-d5151be74a11@molgen.mpg.de> <20180810133628.GA4131@kroah.com> <29b5f3ce-2c23-bf5e-fe50-29bedd4833e1@molgen.mpg.de> <20180810155511.GA17092@kroah.com> <20180811135021.GA2186@kroah.com> <20180813033157.GA18254@ming.t460p> From: Paul Menzel Message-ID: <89f2b622-3d2a-04ce-a6d1-ffb5d9612763@molgen.mpg.de> Date: Thu, 16 Aug 2018 19:09:07 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <20180813033157.GA18254@ming.t460p> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms010901040403090207090108" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This is a cryptographically signed message in MIME format. --------------ms010901040403090207090108 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Dear Ming, On 08/13/18 05:32, Ming Lei wrote: > On Sat, Aug 11, 2018 at 03:50:21PM +0200, Greg Kroah-Hartman wrote: >> On Sat, Aug 11, 2018 at 10:14:18AM +0200, Paul Menzel wrote: >>> Am 10.08.2018 um 17:55 schrieb Greg Kroah-Hartman: >>>> On Fri, Aug 10, 2018 at 04:11:23PM +0200, Paul Menzel wrote: >>> >>>>> On 08/10/18 15:36, Greg Kroah-Hartman wrote: >>>>>> On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote: >>>>>>> Commit ef86f3a7 (genirq/affinity: assign vectors to all possible = CPUs) added >>>>>>> for Linux 4.14.56 causes the aacraid module to not detect the att= ached devices >>>>>>> anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ = 2.30GHz. >>>>>>> >>>>>>> ``` >>>>>>> $ dmesg | grep raid >>>>>>> [ 0.269768] raid6: sse2x1 gen() 7179 MB/s >>>>>>> [ 0.290069] raid6: sse2x1 xor() 5636 MB/s >>>>>>> [ 0.311068] raid6: sse2x2 gen() 9160 MB/s >>>>>>> [ 0.332076] raid6: sse2x2 xor() 6375 MB/s >>>>>>> [ 0.353075] raid6: sse2x4 gen() 11164 MB/s >>>>>>> [ 0.374064] raid6: sse2x4 xor() 7429 MB/s >>>>>>> [ 0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s >>>>>>> [ 0.386001] raid6: .... xor() 7429 MB/s, rmw enabled >>>>>>> [ 0.391008] raid6: using ssse3x2 recovery algorithm >>>>>>> [ 3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 0= 0:01:03 EST 2006) >>>>>>> [ 3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32= :35 EST 2006) >>>>>>> [ 10.725767] Adaptec aacraid driver 1.2.1[50834]-custom >>>>>>> [ 10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn= 't have ASPM control >>>>>>> [ 10.743295] aacraid: Comm Interface type3 enabled >>>>>>> $ lspci -nn | grep Adaptec >>>>>>> 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 = 12G SAS/PCIe 3 [9005:028d] (rev 01) >>>>>>> 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Sto= rage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01) >>>>>>> ``` >>>>>>> >>>>>>> But, it still works with a Dell PowerEdge R715 with two eight cor= e AMD >>>>>>> Opteron 6136, the card below. >>>>>>> >>>>>>> ``` >>>>>>> $ lspci -nn | grep Adaptec >>>>>>> 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 = 12G SAS/PCIe 3 [9005:028d] (rev 01) >>>>>>> ``` >>>>>>> >>>>>>> Reverting the commit fixes the issue. >>>>>>> >>>>>>> commit ef86f3a72adb8a7931f67335560740a7ad696d1d >>>>>>> Author: Christoph Hellwig >>>>>>> Date: Fri Jan 12 10:53:05 2018 +0800 >>>>>>> >>>>>>> genirq/affinity: assign vectors to all possible CPUs >>>>>>> commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream. >>>>>>> Currently we assign managed interrupt vectors to all present= CPUs. This >>>>>>> works fine for systems were we only online/offline CPUs. Bu= t in case of >>>>>>> systems that support physical CPU hotplug (or the virtualize= d version of >>>>>>> it) this means the additional CPUs covered for in the ACPI t= ables or on >>>>>>> the command line are not catered for. To fix this we'd eith= er need to >>>>>>> introduce new hotplug CPU states just for this case, or we c= an start >>>>>>> assining vectors to possible but not present CPUs. >>>>>>> Reported-by: Christian Borntraeger >>>>>>> Tested-by: Christian Borntraeger >>>>>>> Tested-by: Stefan Haberland >>>>>>> Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present C= PU") >>>>>>> Cc: linux-kernel@vger.kernel.org >>>>>>> Cc: Thomas Gleixner >>>>>>> Signed-off-by: Christoph Hellwig >>>>>>> Signed-off-by: Jens Axboe >>>>>>> Signed-off-by: Greg Kroah-Hartman >>>>>>> >>>>>>> The problem doesn=E2=80=99t happen with Linux 4.17.11, so there a= re commits in >>>>>>> Linux master fixing this. Unfortunately, my attempts to find out = failed. >>>>>>> >>>>>>> I was able to cherry-pick the three commits below on top of 4.14.= 62, >>>>>>> but the problem persists. >>>>>>> >>>>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity ma= sks on error >>>>>>> 355d7ecdea35 scsi: hpsa: fix selection of reply queue >>>>>>> e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic i= rq vector affinity >>>>>>> >>>>>>> Trying to cherry-pick the commits below, referencing the commit >>>>>>> in question, gave conflicts. >>>>>>> >>>>>>> 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue >>>>>>> 2. d3056812e7df genirq/affinity: Spread irq vectors among present= CPUs as far as possible >>>>>>> >>>>>>> To avoid further trial and error with the server with a slow firm= ware, >>>>>>> do you know what commits should fix the issue? >>>>>> >>>>>> Look at the email on the stable mailing list: >>>>>> Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y >>>>>> it should help you out here. >>>>> >>>>> Ah, I didn=E2=80=99t see that [1] yet. Also I can=E2=80=99t find th= e original message, and a >>>>> way to reply to that thread. Therefore, here is my reply. >>>>> >>>>>> Can you try the patches listed there? >>>>> >>>>> I tried some of these already without success. >>>>> >>>>> b5b6e8c8d3b4 scsi: virtio_scsi: fix IO hang caused by automatic irq= vector affinity >>>>> 2f31115e940c scsi: core: introduce force_blk_mq >>>>> adbe552349f2 scsi: megaraid_sas: fix selection of reply queue >>>>> >>>>> The commit above is already in v4.14.56. >>>>> >>>>> 8b834bff1b73 scsi: hpsa: fix selection of reply queue >>>>> >>>>> The problem persists. >>>>> >>>>> The problem also persists with the state below. >>>>> >>>>> 3528f73a4e5d scsi: core: introduce force_blk_mq >>>>> 16dc4d8215f3 scsi: hpsa: fix selection of reply queue >>>>> f0a7ab12232d scsi: virtio_scsi: fix IO hang caused by automatic irq= vector affinity >>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity mask= s on error >>>>> 1aa1166eface (tag: v4.14.62, stable/linux-4.14.y) Linux 4.14.62 >>>>> >>>>> So, some more commits are necessary. >>>> >>>> Or I revert the original patch here, and the follow-on ones that wer= e >>>> added to "fix" this issue. I think that might be the better thing >>>> overall here, right? Have you tried that? >>> >>> Yes, reverting the commit fixed the issue for us. If Christoph or Min= g do >>> not have another suggestion for a commit, that would be the way to go= =2E >> >> Christoph or Ming, any ideas here? >> >> In looking at the aacraid code, I don't see anywhere that this is usin= g >> a specific cpu number for queues or anything, but I could be wrong. >> Ideally this should also be failing in 4.17 or 4.18-rc right now as we= ll >> as I don't see anything that would have "fixed" this recently. Unless= >> I'm missing something here? >=20 > From both aac_fib_vector_assign() and aac_src_deliver_message(), looks > every msix vector is used inside driver, so causes this trouble. >=20 > From 84676c1f21e8ff54b (genirq/affinity: assign vectors to all possible= CPUs), > actually it should have been since 9a0ef98e18 (genirq/affinity: Assign = vectors > to all present CPUs), it isn't guaranteed that one vector can be mapped= to > at least one online CPU. >=20 > So it should be fixed in driver by similar approach taken in the > following patches: > scsi: hpsa: fix selection of reply queue > scsi: megaraid_sas: fix selection of reply queue >=20 > And the reason this issue won't be observed is that the following > patches(merged to v4.17) may avoid this issue, but in theory, the issue= > won't be avoided completely by these patches: >=20 > d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as = far as possible > 1a2d0914e23a genirq/affinity: Allow irq spreading from a given starting= point > b3e6aaa8d94d genirq/affinity: Move actual irq vector spreading into a h= elper function > 47778f33dcba genirq/affinity: Rename *node_to_possible_cpumask as *node= _to_cpumask > 0211e12dd0a5 genirq/affinity: Don't return with empty affinity masks on= error >=20 > In short, this issue should have been fixed in upstream aacraid, and th= e idea > is simple, use the current CPU id to retrieve the msix vector offset, t= his > way will make sure that the msix vector can be mapped from one online > CPU. AACRAID folks, are you going to work on that? I looked at the code a bit,= and it=E2=80=99ll take me some days to understand it and to fix it. Just out of curiosity, shouldn=E2=80=99t the subsystems(?) communicate be= tter to avoid such breakage, or was that unexpected? Kind regards, Paul --------------ms010901040403090207090108 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC EFowggUSMIID+qADAgECAgkA4wvV+K8l2YEwDQYJKoZIhvcNAQELBQAwgYIxCzAJBgNVBAYT AkRFMSswKQYDVQQKDCJULVN5c3RlbXMgRW50ZXJwcmlzZSBTZXJ2aWNlcyBHbWJIMR8wHQYD VQQLDBZULVN5c3RlbXMgVHJ1c3QgQ2VudGVyMSUwIwYDVQQDDBxULVRlbGVTZWMgR2xvYmFs Um9vdCBDbGFzcyAyMB4XDTE2MDIyMjEzMzgyMloXDTMxMDIyMjIzNTk1OVowgZUxCzAJBgNV BAYTAkRFMUUwQwYDVQQKEzxWZXJlaW4genVyIEZvZXJkZXJ1bmcgZWluZXMgRGV1dHNjaGVu IEZvcnNjaHVuZ3NuZXR6ZXMgZS4gVi4xEDAOBgNVBAsTB0RGTi1QS0kxLTArBgNVBAMTJERG Ti1WZXJlaW4gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgMjCCASIwDQYJKoZIhvcNAQEBBQAD ggEPADCCAQoCggEBAMtg1/9moUHN0vqHl4pzq5lN6mc5WqFggEcVToyVsuXPztNXS43O+FZs FVV2B+pG/cgDRWM+cNSrVICxI5y+NyipCf8FXRgPxJiZN7Mg9mZ4F4fCnQ7MSjLnFp2uDo0p eQcAIFTcFV9Kltd4tjTTwXS1nem/wHdN6r1ZB+BaL2w8pQDcNb1lDY9/Mm3yWmpLYgHurDg0 WUU2SQXaeMpqbVvAgWsRzNI8qIv4cRrKO+KA3Ra0Z3qLNupOkSk9s1FcragMvp0049ENF4N1 xDkesJQLEvHVaY4l9Lg9K7/AjsMeO6W/VRCrKq4Xl14zzsjz9AkH4wKGMUZrAcUQDBHHWekC AwEAAaOCAXQwggFwMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUk+PYMiba1fFKpZFK4OpL 4qIMz+EwHwYDVR0jBBgwFoAUv1kgNgB5oKAia4zV8mHSuCzLgkowEgYDVR0TAQH/BAgwBgEB /wIBAjAzBgNVHSAELDAqMA8GDSsGAQQBga0hgiwBAQQwDQYLKwYBBAGBrSGCLB4wCAYGZ4EM AQICMEwGA1UdHwRFMEMwQaA/oD2GO2h0dHA6Ly9wa2kwMzM2LnRlbGVzZWMuZGUvcmwvVGVs ZVNlY19HbG9iYWxSb290X0NsYXNzXzIuY3JsMIGGBggrBgEFBQcBAQR6MHgwLAYIKwYBBQUH MAGGIGh0dHA6Ly9vY3NwMDMzNi50ZWxlc2VjLmRlL29jc3ByMEgGCCsGAQUFBzAChjxodHRw Oi8vcGtpMDMzNi50ZWxlc2VjLmRlL2NydC9UZWxlU2VjX0dsb2JhbFJvb3RfQ2xhc3NfMi5j ZXIwDQYJKoZIhvcNAQELBQADggEBAIcL/z4Cm2XIVi3WO5qYi3FP2ropqiH5Ri71sqQPrhE4 eTizDnS6dl2e6BiClmLbTDPo3flq3zK9LExHYFV/53RrtCyD2HlrtrdNUAtmB7Xts5et6u5/ MOaZ/SLick0+hFvu+c+Z6n/XUjkurJgARH5pO7917tALOxrN5fcPImxHhPalR6D90Bo0fa3S PXez7vTXTf/D6OWST1k+kEcQSrCFWMBvf/iu7QhCnh7U3xQuTY+8npTD5+32GPg8SecmqKc2 2CzeIs2LgtjZeOJVEqM7h0S2EQvVDFKvaYwPBt/QolOLV5h7z/0HJPT8vcP9SpIClxvyt7bP ZYoaorVyGTkwggWNMIIEdaADAgECAgwcOtRQhH7u81j4jncwDQYJKoZIhvcNAQELBQAwgZUx CzAJBgNVBAYTAkRFMUUwQwYDVQQKEzxWZXJlaW4genVyIEZvZXJkZXJ1bmcgZWluZXMgRGV1 dHNjaGVuIEZvcnNjaHVuZ3NuZXR6ZXMgZS4gVi4xEDAOBgNVBAsTB0RGTi1QS0kxLTArBgNV BAMTJERGTi1WZXJlaW4gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgMjAeFw0xNjExMDMxNTI0 NDhaFw0zMTAyMjIyMzU5NTlaMGoxCzAJBgNVBAYTAkRFMQ8wDQYDVQQIDAZCYXllcm4xETAP BgNVBAcMCE11ZW5jaGVuMSAwHgYDVQQKDBdNYXgtUGxhbmNrLUdlc2VsbHNjaGFmdDEVMBMG A1UEAwwMTVBHIENBIC0gRzAyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnhx4 59Lh4WqgOs/Md04XxU2yFtfM15ZuJV0PZP7BmqSJKLLPyqmOrADfNdJ5PIGBto2JBhtRRBHd G0GROOvTRHjzOga95WOTeura79T21FWwwAwa29OFnD3ZplQs6HgdwQrZWNi1WHNJxn/4mA19 rNEBUc5urSIpZPvZi5XmlF3v3JHOlx3KWV7mUteB4pwEEfGTg4npPAJbp2o7arxQdoIq+Pu2 OsvqhD7Rk4QeaX+EM1QS4lqd1otW4hE70h/ODPy1xffgbZiuotWQLC6nIwa65Qv6byqlIX0q Zuu99Vsu+r3sWYsL5SBkgecNI7fMJ5tfHrjoxfrKl/ErTAt8GQIDAQABo4ICBTCCAgEwEgYD VR0TAQH/BAgwBgEB/wIBATAOBgNVHQ8BAf8EBAMCAQYwKQYDVR0gBCIwIDANBgsrBgEEAYGt IYIsHjAPBg0rBgEEAYGtIYIsAQEEMB0GA1UdDgQWBBTEiKUH7rh7qgwTv9opdGNSG0lwFjAf BgNVHSMEGDAWgBST49gyJtrV8UqlkUrg6kviogzP4TCBjwYDVR0fBIGHMIGEMECgPqA8hjpo dHRwOi8vY2RwMS5wY2EuZGZuLmRlL2dsb2JhbC1yb290LWcyLWNhL3B1Yi9jcmwvY2Fjcmwu Y3JsMECgPqA8hjpodHRwOi8vY2RwMi5wY2EuZGZuLmRlL2dsb2JhbC1yb290LWcyLWNhL3B1 Yi9jcmwvY2FjcmwuY3JsMIHdBggrBgEFBQcBAQSB0DCBzTAzBggrBgEFBQcwAYYnaHR0cDov L29jc3AucGNhLmRmbi5kZS9PQ1NQLVNlcnZlci9PQ1NQMEoGCCsGAQUFBzAChj5odHRwOi8v Y2RwMS5wY2EuZGZuLmRlL2dsb2JhbC1yb290LWcyLWNhL3B1Yi9jYWNlcnQvY2FjZXJ0LmNy dDBKBggrBgEFBQcwAoY+aHR0cDovL2NkcDIucGNhLmRmbi5kZS9nbG9iYWwtcm9vdC1nMi1j YS9wdWIvY2FjZXJ0L2NhY2VydC5jcnQwDQYJKoZIhvcNAQELBQADggEBABLpeD5FygzqOjj+ /lAOy20UQOGWlx0RMuPcI4nuyFT8SGmK9lD7QCg/HoaJlfU/r78ex+SEide326evlFAoJXIF jVyzNltDhpMKrPIDuh2N12zyn1EtagqPL6hu4pVRzcBpl/F2HCvtmMx5K4WN1L1fmHWLcSap dhXLvAZ9RG/B3rqyULLSNN8xHXYXpmtvG0VGJAndZ+lj+BH7uvd3nHWnXEHC2q7iQlDUqg0a wIqWJgdLlx1Q8Dg/sodv0m+LN0kOzGvVDRCmowBdWGhhusD+duKV66pBl+qhC+4LipariWaM qK5ppMQROATjYeNRvwI+nDcEXr2vDaKmdbxgDVwwggWvMIIEl6ADAgECAgweKlJIhfynPMVG /KIwDQYJKoZIhvcNAQELBQAwajELMAkGA1UEBhMCREUxDzANBgNVBAgMBkJheWVybjERMA8G A1UEBwwITXVlbmNoZW4xIDAeBgNVBAoMF01heC1QbGFuY2stR2VzZWxsc2NoYWZ0MRUwEwYD VQQDDAxNUEcgQ0EgLSBHMDIwHhcNMTcxMTE0MTEzNDE2WhcNMjAxMTEzMTEzNDE2WjCBizEL MAkGA1UEBhMCREUxIDAeBgNVBAoMF01heC1QbGFuY2stR2VzZWxsc2NoYWZ0MTQwMgYDVQQL DCtNYXgtUGxhbmNrLUluc3RpdHV0IGZ1ZXIgbW9sZWt1bGFyZSBHZW5ldGlrMQ4wDAYDVQQL DAVNUElNRzEUMBIGA1UEAwwLUGF1bCBNZW56ZWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQDIh/UR/AX/YQ48VWWDMLTYtXjYJyhRHMc81ZHMMoaoG66lWB9MtKRTnB5lovLZ enTIUyPsCrMhTqV9CWzDf6v9gOTWVxHEYqrUwK5H1gx4XoK81nfV8oGV4EKuVmmikTXiztGz peyDmOY8o/EFNWP7YuRkY/lPQJQBeBHYq9AYIgX4StuXu83nusq4MDydygVOeZC15ts0tv3/ 6WmibmZd1OZRqxDOkoBbY3Djx6lERohs3IKS6RKiI7e90rCSy9rtidJBOvaQS9wvtOSKPx0a +2pAgJEVzZFjOAfBcXydXtqXhcpOi2VCyl+7+LnnTz016JJLsCBuWEcB3kP9nJYNAgMBAAGj ggIxMIICLTAJBgNVHRMEAjAAMA4GA1UdDwEB/wQEAwIF4DAdBgNVHSUEFjAUBggrBgEFBQcD AgYIKwYBBQUHAwQwHQYDVR0OBBYEFHM0Mc3XjMLlhWpp4JufRELL4A/qMB8GA1UdIwQYMBaA FMSIpQfuuHuqDBO/2il0Y1IbSXAWMCAGA1UdEQQZMBeBFXBtZW56ZWxAbW9sZ2VuLm1wZy5k ZTB9BgNVHR8EdjB0MDigNqA0hjJodHRwOi8vY2RwMS5wY2EuZGZuLmRlL21wZy1nMi1jYS9w dWIvY3JsL2NhY3JsLmNybDA4oDagNIYyaHR0cDovL2NkcDIucGNhLmRmbi5kZS9tcGctZzIt Y2EvcHViL2NybC9jYWNybC5jcmwwgc0GCCsGAQUFBwEBBIHAMIG9MDMGCCsGAQUFBzABhido dHRwOi8vb2NzcC5wY2EuZGZuLmRlL09DU1AtU2VydmVyL09DU1AwQgYIKwYBBQUHMAKGNmh0 dHA6Ly9jZHAxLnBjYS5kZm4uZGUvbXBnLWcyLWNhL3B1Yi9jYWNlcnQvY2FjZXJ0LmNydDBC BggrBgEFBQcwAoY2aHR0cDovL2NkcDIucGNhLmRmbi5kZS9tcGctZzItY2EvcHViL2NhY2Vy dC9jYWNlcnQuY3J0MEAGA1UdIAQ5MDcwDwYNKwYBBAGBrSGCLAEBBDARBg8rBgEEAYGtIYIs AQEEAwYwEQYPKwYBBAGBrSGCLAIBBAMGMA0GCSqGSIb3DQEBCwUAA4IBAQCQs6bUDROpFO2F Qz2FMgrdb39VEo8P3DhmpqkaIMC5ZurGbbAL/tAR6lpe4af682nEOJ7VW86ilsIJgm1j0ueY aOuL8jrN4X7IF/8KdZnnNnImW3QVni6TCcc+7+ggci9JHtt0IDCj5vPJBpP/dKXLCN4M+exl GXYpfHgxh8gclJPY1rquhQrihCzHfKB01w9h9tWZDVMtSoy9EUJFhCXw7mYUsvBeJwZesN2B fndPkrXx6XWDdU3S1LyKgHlLIFtarLFm2Hb5zAUR33h+26cN6ohcGqGEEzgIG8tXS8gztEaj 1s2RyzmKd4SXTkKR3GhkZNVWy+gM68J7jP6zzN+cMYIDmjCCA5YCAQEwejBqMQswCQYDVQQG EwJERTEPMA0GA1UECAwGQmF5ZXJuMREwDwYDVQQHDAhNdWVuY2hlbjEgMB4GA1UECgwXTWF4 LVBsYW5jay1HZXNlbGxzY2hhZnQxFTATBgNVBAMMDE1QRyBDQSAtIEcwMgIMHipSSIX8pzzF RvyiMA0GCWCGSAFlAwQCAQUAoIIB8TAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqG SIb3DQEJBTEPFw0xODA4MTYxNzA5MDdaMC8GCSqGSIb3DQEJBDEiBCAuwRsC3mLqNYtRFlrQ 4yEbDQ61LaWyZkgzZrsR7/SGLjBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFlAwQBKjALBglg hkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcG BSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGJBgkrBgEEAYI3EAQxfDB6MGoxCzAJBgNVBAYTAkRF MQ8wDQYDVQQIDAZCYXllcm4xETAPBgNVBAcMCE11ZW5jaGVuMSAwHgYDVQQKDBdNYXgtUGxh bmNrLUdlc2VsbHNjaGFmdDEVMBMGA1UEAwwMTVBHIENBIC0gRzAyAgweKlJIhfynPMVG/KIw gYsGCyqGSIb3DQEJEAILMXygejBqMQswCQYDVQQGEwJERTEPMA0GA1UECAwGQmF5ZXJuMREw DwYDVQQHDAhNdWVuY2hlbjEgMB4GA1UECgwXTWF4LVBsYW5jay1HZXNlbGxzY2hhZnQxFTAT BgNVBAMMDE1QRyBDQSAtIEcwMgIMHipSSIX8pzzFRvyiMA0GCSqGSIb3DQEBAQUABIIBAA/5 ZAr6xk4VV86AhW3tJpe91EAK3V5qiOX80OozopELNbKObp/+LlM/5IXJ/tKUy9aUBmIlYnOA mgnDsHjwXId9bKcEsL4OjOyabNLw0I4wQCTI7DjLlMhnc74B6UiWRmx7GNTvvV12PEufZbTr dYZgz4frgW5kLvjluxTjctmY1ga/Bgf8Ikvyhd+Siwh9OXJqm3yUCP/0LLsTlnx8aYWPRlkt hhkxC0jUYf4th4adi3xaaVA3ubvIS6CEzh/EQjTpQjLwazrm9s+r+gKZoE8qPBzawTRSsuIh ZXCsh87QoMdVgYcgPljIkwjxTPrz51gDGF4SyIDyAuvSRUnNjeYAAAAAAAA= --------------ms010901040403090207090108--