From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E838C49ED6 for ; Wed, 11 Sep 2019 23:42:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DC41B206CD for ; Wed, 11 Sep 2019 23:42:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=dell.com header.i=@dell.com header.b="UVQl5gcL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727782AbfIKXmi (ORCPT ); Wed, 11 Sep 2019 19:42:38 -0400 Received: from mx0a-00154904.pphosted.com ([148.163.133.20]:38836 "EHLO mx0a-00154904.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726954AbfIKXmh (ORCPT ); Wed, 11 Sep 2019 19:42:37 -0400 Received: from pps.filterd (m0170392.ppops.net [127.0.0.1]) by mx0a-00154904.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x8BNOgF4013490 for ; Wed, 11 Sep 2019 19:42:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dell.com; h=from : to : cc : subject : date : message-id; s=smtpout1; bh=UsdcHnQerIrW+zZqSKECSoY4BuSb3CSM/ugV5iEKsw0=; b=UVQl5gcLz6ymaGVbL6BTtQXCJbdXxJJPPJ0USpoAPSlOM04p2K1UVvt27ZaCrhp0f0nM gMPZ79RbDq14s2oRHJhE5ydFp+6fCvYTnbZ4qjMK/RnSiB56teFuZ6tmQZfGy3hFnk/3 OQCcYI4jQjo7vdLzuOsle4ajTXvAK6orUG2ieLEWR53q2o0cXNW6rzC8IFTw1hWcsFZr Pg/4vPEDVsNOVI7duTzcNK/O8R32wRK0WSgDnbxLkA2NrAF9QaJCAGFiuhF2bllkHALM W+zuBerkT6R3BGez+RkZxjFQI7svIEEzmgkIWkbkcbr9YRjf460RVuIsBkIK3ZIWExdf 8g== Received: from mx0a-00154901.pphosted.com (mx0a-00154901.pphosted.com [67.231.149.39]) by mx0a-00154904.pphosted.com with ESMTP id 2uv7qef1xj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 11 Sep 2019 19:42:36 -0400 Received: from pps.filterd (m0142693.ppops.net [127.0.0.1]) by mx0a-00154901.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x8BNNGpf067818 for ; Wed, 11 Sep 2019 19:42:36 -0400 Received: from ausxippc101.us.dell.com (ausxippc101.us.dell.com [143.166.85.207]) by mx0a-00154901.pphosted.com with ESMTP id 2uy92e9aum-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 11 Sep 2019 19:42:35 -0400 X-LoopCount0: from 10.173.37.27 X-PREM-Routing: D-Outbound X-IronPort-AV: E=Sophos;i="5.60,346,1549951200"; d="scan'208";a="1296789607" From: Mario Limonciello To: Keith Busch Cc: Jens Axboe , Christoph Hellwig , Sagi Grimberg , linux-nvme@lists.infradead.org, LKML , Ryan Hong , Crag Wang , sjg@google.com, Jared Dominguez , Mario Limonciello Subject: [PATCH] nvme-pci: Save PCI state before putting drive into deepest state Date: Wed, 11 Sep 2019 18:42:33 -0500 Message-Id: <1568245353-13787-1-git-send-email-mario.limonciello@dell.com> X-Mailer: git-send-email 2.7.4 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-11_11:2019-09-11,2019-09-11 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 clxscore=1015 mlxscore=0 suspectscore=0 spamscore=0 priorityscore=1501 phishscore=0 mlxlogscore=577 adultscore=0 impostorscore=0 lowpriorityscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909110206 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 lowpriorityscore=0 bulkscore=0 phishscore=0 priorityscore=1501 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=710 mlxscore=0 clxscore=1015 impostorscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1909110206 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The action of saving the PCI state will cause numerous PCI configuration space reads which depending upon the vendor implementation may cause the drive to exit the deepest NVMe state. In these cases ASPM will typically resolve the PCIe link state and APST may resolve the NVMe power state. However it has also been observed that this register access after quiesced will cause PC10 failure on some device combinations. To resolve this, move the PCI state saving to before SetFeatures has been called. This has been proven to resolve the issue across a 5000 sample test on previously failing disk/system combinations. Signed-off-by: Mario Limonciello --- drivers/nvme/host/pci.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 732d5b6..9b3fed4 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2894,6 +2894,13 @@ static int nvme_suspend(struct device *dev) if (ret < 0) goto unfreeze; + /* + * A saved state prevents pci pm from generically controlling the + * device's power. If we're using protocol specific settings, we don't + * want pci interfering. + */ + pci_save_state(pdev); + ret = nvme_set_power_state(ctrl, ctrl->npss); if (ret < 0) goto unfreeze; @@ -2908,12 +2915,6 @@ static int nvme_suspend(struct device *dev) ret = 0; goto unfreeze; } - /* - * A saved state prevents pci pm from generically controlling the - * device's power. If we're using protocol specific settings, we don't - * want pci interfering. - */ - pci_save_state(pdev); unfreeze: nvme_unfreeze(ctrl); return ret; -- 2.7.4