From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Alex G." Subject: Re: [RFC PATCH v2 3/4] acpi: apei: Do not panic() when correctable errors are marked as fatal. Date: Thu, 19 Apr 2018 12:40:54 -0500 Message-ID: References: <20180416215903.7318-1-mr.nuke.me@gmail.com> <20180416215903.7318-4-mr.nuke.me@gmail.com> <20180418175415.GJ4795@pd.tnic> <20180419154006.GE3600@pd.tnic> <977608e6-9f5d-c523-a78a-993ac5bfd55f@gmail.com> <20180419164528.GD5635@pd.tnic> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20180419164528.GD5635@pd.tnic> Content-Language: en-US Sender: linux-kernel-owner@vger.kernel.org To: Borislav Petkov Cc: linux-acpi@vger.kernel.org, linux-edac@vger.kernel.org, rjw@rjwysocki.net, lenb@kernel.org, tony.luck@intel.com, tbaicar@codeaurora.org, will.deacon@arm.com, james.morse@arm.com, shiju.jose@huawei.com, zjzhang@codeaurora.org, gengdongjiu@huawei.com, linux-kernel@vger.kernel.org, alex_gagniuc@dellteam.com, austin_bolen@dell.com, shyam_iyer@dell.com, devel@acpica.org, mchehab@kernel.org, robert.moore@intel.com, erik.schmauss@intel.com List-Id: linux-acpi@vger.kernel.org SURPRISE!!! On 04/19/2018 11:45 AM, Borislav Petkov wrote: > On Thu, Apr 19, 2018 at 11:26:57AM -0500, Alex G. wrote: >> At a very high level, I'm working with Dell on improving server >> reliability, with a focus on NVME hotplug and surprise removal. One of >> the features we don't support is surprise removal of NVME drives; >> hotplug is supported with 'prepare to remove'. This is one of the >> reasons NVME is not on feature parity with SAS and SATA. > > Ok, first question: is surprise removal something purely mechanical or > do you need firmware support for it? In the sense that you need to tell > the firmware that you will be removing the drive. SURPRISE!!! removal only means that the system was not expecting the drive to be yanked. An example is removing a USB flash drive without first unmounting it and removing the usb device (echo 0 > /sys/bus/usb/.../authorized). PCIe removal and hotplug is fairly well spec'd, and NVMe rides on that without issue. It's much easier and faster for an OS to just follow the spec and handle things on its own. Interference from firmware only comes in with EFI/ACPI and FFS. From a purely technical point of view, firmware has nothing to do with this. >>From a firmware-centric view, unfortunately, firmware wants the ability to log errors to the BMC... and hotplug events. Does firmware need to know that a drive will be removed? I'm not aware of any such requirement. I think the main purpose of 'prepare to remove' is to shut down any traffic on the link. This way, link removal does not generate PCIe errors which may otherwise end up crashing the OS. > I'm sceptical, though, as it has "surprise" in the name so I'm guessing > the firmware doesn't know about it, the drive physically disappears and > the FW starts spewing PCIe errors... It's not the FW that spews out errors. It's the hardware. It's very likely that a device which is actively used will have several DMA transactions already queued up and lots of traffic going through the link. When the link dies and the traffic can't be delivered, Unsupported Request errors are very common. On the r740xd, FW just hides those errors from the OS with no further notification. On this machine BIOS sets things up such that non-posted requests report fatal (PCIe) errors. FW still tries very hard to hide this from the OS, and I think the heuristic is that if the drive physical presence is gone, don't even report the error. There are a lot of problems with the approach, but one thing to keep in mind is that the FW was written at a time when OSes were more than happy to crash at any PCIe error reported through GHES. Alex >> I'm not sure if this is the example you're looking for, but >> take an r740xd server, and slowly unplug an Intel NVME drives at an >> angle. You're likely to crash the machine. > > No no, that's actually a great example! > > Thx. > From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Subject: [RFC,v2,3/4] acpi: apei: Do not panic() when correctable errors are marked as fatal. From: Alexandru Gagniuc Message-Id: Date: Thu, 19 Apr 2018 12:40:54 -0500 To: Borislav Petkov Cc: linux-acpi@vger.kernel.org, linux-edac@vger.kernel.org, rjw@rjwysocki.net, lenb@kernel.org, tony.luck@intel.com, tbaicar@codeaurora.org, will.deacon@arm.com, james.morse@arm.com, shiju.jose@huawei.com, zjzhang@codeaurora.org, gengdongjiu@huawei.com, linux-kernel@vger.kernel.org, alex_gagniuc@dellteam.com, austin_bolen@dell.com, shyam_iyer@dell.com, devel@acpica.org, mchehab@kernel.org, robert.moore@intel.com, erik.schmauss@intel.com List-ID: U1VSUFJJU0UhISEKCk9uIDA0LzE5LzIwMTggMTE6NDUgQU0sIEJvcmlzbGF2IFBldGtvdiB3cm90 ZToKPiBPbiBUaHUsIEFwciAxOSwgMjAxOCBhdCAxMToyNjo1N0FNIC0wNTAwLCBBbGV4IEcuIHdy b3RlOgo+PiBBdCBhIHZlcnkgaGlnaCBsZXZlbCwgSSdtIHdvcmtpbmcgd2l0aCBEZWxsIG9uIGlt cHJvdmluZyBzZXJ2ZXIKPj4gcmVsaWFiaWxpdHksIHdpdGggYSBmb2N1cyBvbiBOVk1FIGhvdHBs dWcgYW5kIHN1cnByaXNlIHJlbW92YWwuIE9uZSBvZgo+PiB0aGUgZmVhdHVyZXMgd2UgZG9uJ3Qg c3VwcG9ydCBpcyBzdXJwcmlzZSByZW1vdmFsIG9mIE5WTUUgZHJpdmVzOwo+PiBob3RwbHVnIGlz IHN1cHBvcnRlZCB3aXRoICdwcmVwYXJlIHRvIHJlbW92ZScuIFRoaXMgaXMgb25lIG9mIHRoZQo+ PiByZWFzb25zIE5WTUUgaXMgbm90IG9uIGZlYXR1cmUgcGFyaXR5IHdpdGggU0FTIGFuZCBTQVRB Lgo+IAo+IE9rLCBmaXJzdCBxdWVzdGlvbjogaXMgc3VycHJpc2UgcmVtb3ZhbCBzb21ldGhpbmcg cHVyZWx5IG1lY2hhbmljYWwgb3IKPiBkbyB5b3UgbmVlZCBmaXJtd2FyZSBzdXBwb3J0IGZvciBp dD8gSW4gdGhlIHNlbnNlIHRoYXQgeW91IG5lZWQgdG8gdGVsbAo+IHRoZSBmaXJtd2FyZSB0aGF0 IHlvdSB3aWxsIGJlIHJlbW92aW5nIHRoZSBkcml2ZS4KClNVUlBSSVNFISEhIHJlbW92YWwgb25s eSBtZWFucyB0aGF0IHRoZSBzeXN0ZW0gd2FzIG5vdCBleHBlY3RpbmcgdGhlCmRyaXZlIHRvIGJl IHlhbmtlZC4gQW4gZXhhbXBsZSBpcyByZW1vdmluZyBhIFVTQiBmbGFzaCBkcml2ZSB3aXRob3V0 CmZpcnN0IHVubW91bnRpbmcgaXQgYW5kIHJlbW92aW5nIHRoZSB1c2IgZGV2aWNlIChlY2hvIDAg Pgovc3lzL2J1cy91c2IvLi4uL2F1dGhvcml6ZWQpLgoKUENJZSByZW1vdmFsIGFuZCBob3RwbHVn IGlzIGZhaXJseSB3ZWxsIHNwZWMnZCwgYW5kIE5WTWUgcmlkZXMgb24gdGhhdAp3aXRob3V0IGlz c3VlLiBJdCdzIG11Y2ggZWFzaWVyIGFuZCBmYXN0ZXIgZm9yIGFuIE9TIHRvIGp1c3QgZm9sbG93 IHRoZQpzcGVjIGFuZCBoYW5kbGUgdGhpbmdzIG9uIGl0cyBvd24uCgpJbnRlcmZlcmVuY2UgZnJv bSBmaXJtd2FyZSBvbmx5IGNvbWVzIGluIHdpdGggRUZJL0FDUEkgYW5kIEZGUy4gRnJvbSBhCnB1 cmVseSB0ZWNobmljYWwgcG9pbnQgb2YgdmlldywgZmlybXdhcmUgaGFzIG5vdGhpbmcgdG8gZG8g d2l0aCB0aGlzLgpGcm9tIGEgZmlybXdhcmUtY2VudHJpYyB2aWV3LCB1bmZvcnR1bmF0ZWx5LCBm aXJtd2FyZSB3YW50cyB0aGUgYWJpbGl0eQp0byBsb2cgZXJyb3JzIHRvIHRoZSBCTUMuLi4gYW5k IGhvdHBsdWcgZXZlbnRzLgoKRG9lcyBmaXJtd2FyZSBuZWVkIHRvIGtub3cgdGhhdCBhIGRyaXZl IHdpbGwgYmUgcmVtb3ZlZD8gSSdtIG5vdCBhd2FyZQpvZiBhbnkgc3VjaCByZXF1aXJlbWVudC4g SSB0aGluayB0aGUgbWFpbiBwdXJwb3NlIG9mICdwcmVwYXJlIHRvIHJlbW92ZScKaXMgdG8gc2h1 dCBkb3duIGFueSB0cmFmZmljIG9uIHRoZSBsaW5rLiBUaGlzIHdheSwgbGluayByZW1vdmFsIGRv ZXMgbm90CmdlbmVyYXRlIFBDSWUgZXJyb3JzIHdoaWNoIG1heSBvdGhlcndpc2UgZW5kIHVwIGNy YXNoaW5nIHRoZSBPUy4KCgo+IEknbSBzY2VwdGljYWwsIHRob3VnaCwgYXMgaXQgaGFzICJzdXJw cmlzZSIgaW4gdGhlIG5hbWUgc28gSSdtIGd1ZXNzaW5nCj4gdGhlIGZpcm13YXJlIGRvZXNuJ3Qg a25vdyBhYm91dCBpdCwgdGhlIGRyaXZlIHBoeXNpY2FsbHkgZGlzYXBwZWFycyBhbmQKPiB0aGUg Rlcgc3RhcnRzIHNwZXdpbmcgUENJZSBlcnJvcnMuLi4KCkl0J3Mgbm90IHRoZSBGVyB0aGF0IHNw ZXdzIG91dCBlcnJvcnMuIEl0J3MgdGhlIGhhcmR3YXJlLiBJdCdzIHZlcnkKbGlrZWx5IHRoYXQg YSBkZXZpY2Ugd2hpY2ggaXMgYWN0aXZlbHkgdXNlZCB3aWxsIGhhdmUgc2V2ZXJhbCBETUEKdHJh bnNhY3Rpb25zIGFscmVhZHkgcXVldWVkIHVwIGFuZCBsb3RzIG9mIHRyYWZmaWMgZ29pbmcgdGhy b3VnaCB0aGUKbGluay4gV2hlbiB0aGUgbGluayBkaWVzIGFuZCB0aGUgdHJhZmZpYyBjYW4ndCBi ZSBkZWxpdmVyZWQsIFVuc3VwcG9ydGVkClJlcXVlc3QgZXJyb3JzIGFyZSB2ZXJ5IGNvbW1vbi4K Ck9uIHRoZSByNzQweGQsIEZXIGp1c3QgaGlkZXMgdGhvc2UgZXJyb3JzIGZyb20gdGhlIE9TIHdp dGggbm8gZnVydGhlcgpub3RpZmljYXRpb24uIE9uIHRoaXMgbWFjaGluZSBCSU9TIHNldHMgdGhp bmdzIHVwIHN1Y2ggdGhhdCBub24tcG9zdGVkCnJlcXVlc3RzIHJlcG9ydCBmYXRhbCAoUENJZSkg ZXJyb3JzLiBGVyBzdGlsbCB0cmllcyB2ZXJ5IGhhcmQgdG8gaGlkZQp0aGlzIGZyb20gdGhlIE9T LCBhbmQgSSB0aGluayB0aGUgaGV1cmlzdGljIGlzIHRoYXQgaWYgdGhlIGRyaXZlCnBoeXNpY2Fs IHByZXNlbmNlIGlzIGdvbmUsIGRvbid0IGV2ZW4gcmVwb3J0IHRoZSBlcnJvci4KClRoZXJlIGFy ZSBhIGxvdCBvZiBwcm9ibGVtcyB3aXRoIHRoZSBhcHByb2FjaCwgYnV0IG9uZSB0aGluZyB0byBr ZWVwIGluCm1pbmQgaXMgdGhhdCB0aGUgRlcgd2FzIHdyaXR0ZW4gYXQgYSB0aW1lIHdoZW4gT1Nl cyB3ZXJlIG1vcmUgdGhhbiBoYXBweQp0byBjcmFzaCBhdCBhbnkgUENJZSBlcnJvciByZXBvcnRl ZCB0aHJvdWdoIEdIRVMuCgpBbGV4Cgo+PiBJJ20gbm90IHN1cmUgaWYgdGhpcyBpcyB0aGUgZXhh bXBsZSB5b3UncmUgbG9va2luZyBmb3IsIGJ1dAo+PiB0YWtlIGFuIHI3NDB4ZCBzZXJ2ZXIsIGFu ZCBzbG93bHkgdW5wbHVnIGFuIEludGVsIE5WTUUgZHJpdmVzIGF0IGFuCj4+IGFuZ2xlLiBZb3Un cmUgbGlrZWx5IHRvIGNyYXNoIHRoZSBtYWNoaW5lLgo+IAo+IE5vIG5vLCB0aGF0J3MgYWN0dWFs bHkgYSBncmVhdCBleGFtcGxlIQo+IAo+IFRoeC4KPgotLS0KVG8gdW5zdWJzY3JpYmUgZnJvbSB0 aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vic2NyaWJlIGxpbnV4LWVkYWMiIGluCnRoZSBi b2R5IG9mIGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3JnCk1vcmUgbWFqb3Jk b21vIGluZm8gYXQgIGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbAo=