From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Dexuan Cui To: Jens Axboe , Bart Van Assche , "hare@suse.com" , "hare@suse.de" , "Martin K. Petersen" CC: "hch@lst.de" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "jth@kernel.org" Subject: RE: [PATCH] genhd: Do not hold event lock when scheduling workqueue elements Date: Wed, 8 Feb 2017 10:48:13 +0000 Message-ID: References: <1484732896-22941-1-git-send-email-hare@suse.de> <1485822639.2669.16.camel@sandisk.com> <532c55c4-15da-d2f9-401c-36bc4343756b@suse.com> <1486436195.2791.1.camel@sandisk.com> <9199d528-f220-5b77-d657-c510ca210067@kernel.dk> In-Reply-To: <9199d528-f220-5b77-d657-c510ca210067@kernel.dk> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 List-ID: PiBGcm9tOiBKZW5zIEF4Ym9lIFttYWlsdG86YXhib2VAa2VybmVsLmRrXQ0KPiBTZW50OiBXZWRu ZXNkYXksIEZlYnJ1YXJ5IDgsIDIwMTcgMDA6MDkNCj4gVG86IERleHVhbiBDdWkgPGRlY3VpQG1p Y3Jvc29mdC5jb20+OyBCYXJ0IFZhbiBBc3NjaGUNCj4gPEJhcnQuVmFuQXNzY2hlQHNhbmRpc2su Y29tPjsgaGFyZUBzdXNlLmNvbTsgaGFyZUBzdXNlLmRlDQo+IENjOiBoY2hAbHN0LmRlOyBsaW51 eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnOyBsaW51eC1ibG9ja0B2Z2VyLmtlcm5lbC5vcmc7DQo+ IGp0aEBrZXJuZWwub3JnDQo+IFN1YmplY3Q6IFJlOiBbUEFUQ0hdIGdlbmhkOiBEbyBub3QgaG9s ZCBldmVudCBsb2NrIHdoZW4gc2NoZWR1bGluZyB3b3JrcXVldWUNCj4gZWxlbWVudHMNCj4gDQo+ IE9uIDAyLzA2LzIwMTcgMTE6MjkgUE0sIERleHVhbiBDdWkgd3JvdGU6DQo+ID4+IEZyb206IGxp bnV4LWJsb2NrLW93bmVyQHZnZXIua2VybmVsLm9yZyBbbWFpbHRvOmxpbnV4LWJsb2NrLQ0KPiA+ PiBvd25lckB2Z2VyLmtlcm5lbC5vcmddIE9uIEJlaGFsZiBPZiBEZXh1YW4gQ3VpDQo+ID4+IHdp dGggdGhlIGxpbnV4LW5leHQga2VybmVsLg0KPiA+Pg0KPiA+PiBJIGNhbiBib290IHRoZSBndWVz dCB3aXRoIGxpbnV4LW5leHQncyBuZXh0LTIwMTcwMTMwIHdpdGhvdXQgYW55IGlzc3VlLA0KPiA+ PiBidXQgc2luY2UgbmV4dC0yMDE3MDEzMSBJIGhhdmVuJ3Qgc3VjY2VlZGVkIGluIGJvb3Rpbmcg dGhlIGd1ZXN0Lg0KPiA+Pg0KPiA+PiBXaXRoIG5leHQtMjAxNzAyMDMgKG1lbnRpb25lZCBpbiBt eSBtYWlsIGxhc3QgRnJpZGF5KSwgSSBnb3QgdGhlIHNhbWUNCj4gPj4gY2FsbHRyYWNlIGFzIEhh bm5lcy4NCj4gPj4NCj4gPj4gV2l0aCB0b2RheSdzIGxpbnV4LW5leHQgKG5leHQtMjAxNzAyMDYp LCBhY3R1YWxseSB0aGUgY2FsbHRyYWNlIGNoYW5nZWQgdG8NCj4gPj4gdGhlIGJlbG93Lg0KPiA+ PiBbICAxMjIuMDIzMDM2XSAgPyByZW1vdmVfd2FpdF9xdWV1ZSsweDcwLzB4NzANCj4gPj4gWyAg MTIyLjA1MTM4M10gIGFzeW5jX3N5bmNocm9uaXplX2Z1bGwrMHgxNy8weDIwDQo+ID4+IFsgIDEy Mi4wNzY5MjVdICBkb19pbml0X21vZHVsZSsweGMxLzB4MWY5DQo+ID4+IFsgIDEyMi4wOTc1MzBd ICBsb2FkX21vZHVsZSsweDI0YmMvMHgyOTgwDQo+ID4NCj4gPiBJIGRvbid0IGtub3cgd2h5IGl0 IGhhbmdzIGhlcmUsIGJ1dCB0aGlzIGlzIHRoZSBzYW1lIGNhbGx0cmFjZSBpbiBteQ0KPiA+IGxh c3QtRnJpZGF5IG1haWwsIHdoaWNoIGNvbnRhaW5zIDIgY2FsbHRyYWNlcy4gSXQgbG9va3MgdGhl IG90aGVyIGNhbGx0cmFjZSBoYXMNCj4gPiBiZWVuIHJlc29sdmVkIGJ5IHNvbWUgY2hhbmdlcyBi ZXR3ZWVuIG5leHQtMjAxNzAyMDMgYW5kIHRvZGF5Lg0KPiA+DQo+ID4gSGVyZSB0aGUga2VybmVs IGlzIHRyeWluZyB0byBsb2FkIHRoZSBIeXBlci1WIHN0b3JhZ2UgZHJpdmVyIChodl9zdG9ydnNj KSwgYW5kDQo+ID4gdGhlIGRyaXZlcidzIF9faW5pdCBhbmQgLnByb2JlIGhhdmUgZmluaXNoZWQg c3VjY2Vzc2Z1bGx5IGFuZCB0aGVuIHRoZSBrZXJuZWwNCj4gPiBoYW5ncyBoZXJlLg0KPiA+DQo+ ID4gSSBiZWxpZXZlIHNvbWV0aGluZyBpcyBicm9rZW4gcmVjZW50bHksIGJlY2F1c2UgSSBkb24n dCBoYXZlIGFueSBpc3N1ZSBiZWZvcmUNCj4gPiBKYW4gMzEuDQo+IA0KPiBDYW4geW91IHRyeSBh bmQgYmlzZWN0IGl0Pw0KPiANCj4gSmVucyBBeGJvZQ0KDQpJIGJpc2VjdGVkIGl0IG9uIHRoZSBi cmFuY2ggZm9yLTQuMTEvbmV4dCBvZiB0aGUgbGludXgtYmxvY2sgcmVwbyBhbmQgdGhlIGxvZyBz aG93cw0KdGhlIGZpcnN0IGJhZCBjb21taXQgaXMgDQpbZTljNzg3ZTZdIHNjc2k6IGFsbG9jYXRl IHNjc2lfY21uZCBzdHJ1Y3R1cmVzIGFzIHBhcnQgb2Ygc3RydWN0IHJlcXVlc3QNCg0KIyBnaXQg YmlzZWN0IGxvZw0KZ2l0IGJpc2VjdCBzdGFydA0KIyBiYWQ6IFs4MGM2YjE1NzMyZjBkODgzMDAz MjE0OWNiY2JjOGQ2N2UwNzRiNWU4XSBibGstbXEtc2NoZWQ6ICh1bilyZWdpc3RlciBlbGV2YXRv ciB3aGVuICh1bilyZWdpc3RlcmluZyBxdWV1ZQ0KZ2l0IGJpc2VjdCBiYWQgODBjNmIxNTczMmYw ZDg4MzAwMzIxNDljYmNiYzhkNjdlMDc0YjVlOA0KIyBnb29kOiBbMzA5YmQ5NmFmOWUyNmRhMzAz ODY2MWJmNWNkYWQ3ODBlZWY0OWRkOV0gbWQ6IGNsZWFudXAgYmlvIG9wIC8gZmxhZ3MgaGFuZGxp bmcgaW4gcmFpZDFfd3JpdGVfcmVxdWVzdA0KZ2l0IGJpc2VjdCBnb29kIDMwOWJkOTZhZjllMjZk YTMwMzg2NjFiZjVjZGFkNzgwZWVmNDlkZDkNCiMgYmFkOiBbMjc0MTBhODkyN2ZiODliZDE1MGRl MDhkNzQ5YThlZDdmNjdiNzczOV0gbmJkOiByZW1vdmUgUkVRX1RZUEVfRFJWX1BSSVYgbGVmdG92 ZXJzDQpnaXQgYmlzZWN0IGJhZCAyNzQxMGE4OTI3ZmI4OWJkMTUwZGUwOGQ3NDlhOGVkN2Y2N2I3 NzM5DQojIGJhZDogW2U5Yzc4N2U2NWMwYzM2NTI5NzQ1YmU0N2Q0OTBkOTk4YjRiNmU1ODldIHNj c2k6IGFsbG9jYXRlIHNjc2lfY21uZCBzdHJ1Y3R1cmVzIGFzIHBhcnQgb2Ygc3RydWN0IHJlcXVl c3QNCmdpdCBiaXNlY3QgYmFkIGU5Yzc4N2U2NWMwYzM2NTI5NzQ1YmU0N2Q0OTBkOTk4YjRiNmU1 ODkNCiMgZ29vZDogWzMyNzgyNTU3NDEzMjZiNmQ2NmQ4Y2E3ZDFjYjJjNTc2MzNlZTQzZDldIHNj c2lfZGhfcmRhYzogc3dpdGNoIHRvIHNjc2lfZXhlY3V0ZV9yZXFfZmxhZ3MoKQ0KZ2l0IGJpc2Vj dCBnb29kIDMyNzgyNTU3NDEzMjZiNmQ2NmQ4Y2E3ZDFjYjJjNTc2MzNlZTQzZDkNCiMgZ29vZDog WzBmYmMzZTBmZjYyM2YxMDEyZTdjMmFmOTZlNzgxZWViMjZiY2MwZDddIHNjc2k6IHJlbW92ZSBn ZnBfZmxhZ3MgbWVtYmVyIGluIHNjc2lfaG9zdF9jbWRfcG9vbA0KZ2l0IGJpc2VjdCBnb29kIDBm YmMzZTBmZjYyM2YxMDEyZTdjMmFmOTZlNzgxZWViMjZiY2MwZDcNCiMgZ29vZDogW2VlZmY2OGM1 NjE4YzhkMDkyMGIxNDUzM2M3MGIyZGYwMDdiZDk0YjRdIHNjc2k6IHJlbW92ZSBzY3NpX2NtZF9k bWFfcG9vbA0KZ2l0IGJpc2VjdCBnb29kIGVlZmY2OGM1NjE4YzhkMDkyMGIxNDUzM2M3MGIyZGYw MDdiZDk0YjQNCiMgZ29vZDogW2Q0ODc3N2E2MzNkNmZhN2NjZGUwZjBlNjUwOWYwYzAxZmJmYzUy OTldIHNjc2k6IHJlbW92ZSBfX3Njc2lfYWxsb2NfcXVldWUNCmdpdCBiaXNlY3QgZ29vZCBkNDg3 NzdhNjMzZDZmYTdjY2RlMGYwZTY1MDlmMGMwMWZiZmM1Mjk5DQojIGZpcnN0IGJhZCBjb21taXQ6 IFtlOWM3ODdlNjVjMGMzNjUyOTc0NWJlNDdkNDkwZDk5OGI0YjZlNTg5XSBzY3NpOiBhbGxvY2F0 ZSBzY3NpX2NtbmQgc3RydWN0dXJlcyBhcyBwYXJ0IG9mIHN0cnVjdCByZXF1ZXN0DQoNClRoYW5r cywNCi0tIERleHVhbg0K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753945AbdBHKsV (ORCPT ); Wed, 8 Feb 2017 05:48:21 -0500 Received: from mail-sn1nam02on0124.outbound.protection.outlook.com ([104.47.36.124]:3840 "EHLO NAM02-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753604AbdBHKsR (ORCPT ); Wed, 8 Feb 2017 05:48:17 -0500 From: Dexuan Cui To: Jens Axboe , Bart Van Assche , "hare@suse.com" , "hare@suse.de" , "Martin K. Petersen" CC: "hch@lst.de" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "jth@kernel.org" Subject: RE: [PATCH] genhd: Do not hold event lock when scheduling workqueue elements Thread-Topic: [PATCH] genhd: Do not hold event lock when scheduling workqueue elements Thread-Index: AQHSe9062dHV0XX4/EGXRjibwyBPjaFXNu6QgAWiPdCAAArCAIAACQ5ggAAwHGCAAKQ9AIABLBtQ Date: Wed, 8 Feb 2017 10:48:13 +0000 Message-ID: References: <1484732896-22941-1-git-send-email-hare@suse.de> <1485822639.2669.16.camel@sandisk.com> <532c55c4-15da-d2f9-401c-36bc4343756b@suse.com> <1486436195.2791.1.camel@sandisk.com> <9199d528-f220-5b77-d657-c510ca210067@kernel.dk> In-Reply-To: <9199d528-f220-5b77-d657-c510ca210067@kernel.dk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=decui@microsoft.com; x-originating-ip: [2404:f801:9000:19::ed] x-ms-office365-filtering-correlation-id: 5f32a674-2dee-41b6-51e3-08d4500ffb81 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(48565401081);SRVR:MWHPR03MB2671; x-microsoft-exchange-diagnostics: 1;MWHPR03MB2671;7:Lk19lDffKnFBsBWsBQ1ybR3JenUsBga8xNKl6A/Pgs++NHKrQ57h6QTruJWo1YU5BVUW+3AAO5JuwaaWuqT/PTH7COwm3LfXM9gsl2sz9cDb8OhQa0E8WeSHInQP47psIPSchCxmKjvqfGJjITuvzo+kA/O2OaMxxg24MGWJeac87t+BYdDcFRilNAcNRbD9TxVjbOSXRd1+LdxjZGOF9Eo7thytH5yDo2MPnLi9ZbwucM+k7cybl+esq+XgLS8Oq3OynJzXIq7h6t3P8TZ3ZniBRB48SuBkN36ukDnqYHg+D25PmEExtzrkSC76OPLS1xlnjY0BtJ1j3kkJXricoaEKLm8TlNGNpz3mtnJOJyuIC0kT6HrZBOWt7zN2X6J3uK3CPPq+DqybBR+azxqGd6uSVTmhJk8MbWQ4qASSkLLd1YQUYCNzlMaVSOc9CCiFk8xa+MJRkURvYQB522qF6g25rt7MsPf0Pz867t8CozuHTMPYigCNdfxZR52NNm9cBekh06j4ULHK6JeD+B43jDlmAIpglHHwsOyvi9pFyJI= x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(9452136761055)(131327999870524)(42932892334569); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(61425038)(6040375)(601004)(2401047)(8121501046)(5005006)(20170203043)(2017020702029)(10201501046)(3002001)(6055026)(61426038)(61427038)(6041248)(20161123560025)(20161123564025)(20161123562025)(20161123558025)(20161123555025)(6072148);SRVR:MWHPR03MB2671;BCL:0;PCL:0;RULEID:;SRVR:MWHPR03MB2671; x-forefront-prvs: 0212BDE3BE x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(7916002)(39850400002)(39410400002)(39860400002)(39840400002)(39450400003)(189002)(377454003)(199003)(24454002)(99286003)(101416001)(122556002)(106116001)(105586002)(6436002)(5005710100001)(106356001)(25786008)(33656002)(92566002)(54356999)(50986999)(38730400002)(76176999)(53546003)(6246003)(229853002)(77096006)(8656002)(10290500002)(9686003)(55016002)(53936002)(3660700001)(6506006)(93886004)(68736007)(54906002)(86362001)(97736004)(2201001)(7736002)(74316002)(305945005)(4326007)(2906002)(7696004)(8936002)(3280700002)(5660300001)(8676002)(102836003)(8990500004)(189998001)(6116002)(86612001)(2501003)(10090500001)(1691005)(2900100001)(575784001)(81166006)(81156014)(2950100002);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR03MB2671;H:MWHPR03MB2669.namprd03.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Feb 2017 10:48:13.9189 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB2671 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id v18ApamQ012368 > From: Jens Axboe [mailto:axboe@kernel.dk] > Sent: Wednesday, February 8, 2017 00:09 > To: Dexuan Cui ; Bart Van Assche > ; hare@suse.com; hare@suse.de > Cc: hch@lst.de; linux-kernel@vger.kernel.org; linux-block@vger.kernel.org; > jth@kernel.org > Subject: Re: [PATCH] genhd: Do not hold event lock when scheduling workqueue > elements > > On 02/06/2017 11:29 PM, Dexuan Cui wrote: > >> From: linux-block-owner@vger.kernel.org [mailto:linux-block- > >> owner@vger.kernel.org] On Behalf Of Dexuan Cui > >> with the linux-next kernel. > >> > >> I can boot the guest with linux-next's next-20170130 without any issue, > >> but since next-20170131 I haven't succeeded in booting the guest. > >> > >> With next-20170203 (mentioned in my mail last Friday), I got the same > >> calltrace as Hannes. > >> > >> With today's linux-next (next-20170206), actually the calltrace changed to > >> the below. > >> [ 122.023036] ? remove_wait_queue+0x70/0x70 > >> [ 122.051383] async_synchronize_full+0x17/0x20 > >> [ 122.076925] do_init_module+0xc1/0x1f9 > >> [ 122.097530] load_module+0x24bc/0x2980 > > > > I don't know why it hangs here, but this is the same calltrace in my > > last-Friday mail, which contains 2 calltraces. It looks the other calltrace has > > been resolved by some changes between next-20170203 and today. > > > > Here the kernel is trying to load the Hyper-V storage driver (hv_storvsc), and > > the driver's __init and .probe have finished successfully and then the kernel > > hangs here. > > > > I believe something is broken recently, because I don't have any issue before > > Jan 31. > > Can you try and bisect it? > > Jens Axboe I bisected it on the branch for-4.11/next of the linux-block repo and the log shows the first bad commit is [e9c787e6] scsi: allocate scsi_cmnd structures as part of struct request # git bisect log git bisect start # bad: [80c6b15732f0d8830032149cbcbc8d67e074b5e8] blk-mq-sched: (un)register elevator when (un)registering queue git bisect bad 80c6b15732f0d8830032149cbcbc8d67e074b5e8 # good: [309bd96af9e26da3038661bf5cdad780eef49dd9] md: cleanup bio op / flags handling in raid1_write_request git bisect good 309bd96af9e26da3038661bf5cdad780eef49dd9 # bad: [27410a8927fb89bd150de08d749a8ed7f67b7739] nbd: remove REQ_TYPE_DRV_PRIV leftovers git bisect bad 27410a8927fb89bd150de08d749a8ed7f67b7739 # bad: [e9c787e65c0c36529745be47d490d998b4b6e589] scsi: allocate scsi_cmnd structures as part of struct request git bisect bad e9c787e65c0c36529745be47d490d998b4b6e589 # good: [3278255741326b6d66d8ca7d1cb2c57633ee43d9] scsi_dh_rdac: switch to scsi_execute_req_flags() git bisect good 3278255741326b6d66d8ca7d1cb2c57633ee43d9 # good: [0fbc3e0ff623f1012e7c2af96e781eeb26bcc0d7] scsi: remove gfp_flags member in scsi_host_cmd_pool git bisect good 0fbc3e0ff623f1012e7c2af96e781eeb26bcc0d7 # good: [eeff68c5618c8d0920b14533c70b2df007bd94b4] scsi: remove scsi_cmd_dma_pool git bisect good eeff68c5618c8d0920b14533c70b2df007bd94b4 # good: [d48777a633d6fa7ccde0f0e6509f0c01fbfc5299] scsi: remove __scsi_alloc_queue git bisect good d48777a633d6fa7ccde0f0e6509f0c01fbfc5299 # first bad commit: [e9c787e65c0c36529745be47d490d998b4b6e589] scsi: allocate scsi_cmnd structures as part of struct request Thanks, -- Dexuan