From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88B87C2D0A3 for ; Tue, 3 Nov 2020 14:04:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F1DE222264 for ; Tue, 3 Nov 2020 14:04:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="G+BmjeiH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F1DE222264 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 32C896B0036; Tue, 3 Nov 2020 09:04:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E1636B005C; Tue, 3 Nov 2020 09:04:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CD4A6B0068; Tue, 3 Nov 2020 09:04:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id E14DB6B0036 for ; Tue, 3 Nov 2020 09:04:07 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 70EF0180AD801 for ; Tue, 3 Nov 2020 14:04:07 +0000 (UTC) X-FDA: 77443276134.12.cream93_1412f6e272b9 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id AB5E3180054F2 for ; Tue, 3 Nov 2020 14:04:04 +0000 (UTC) X-HE-Tag: cream93_1412f6e272b9 X-Filterd-Recvd-Size: 4924 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Tue, 3 Nov 2020 14:04:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1604412243; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=IsejNI8EYSmowoiAvOvFY33zKF4jozyNUSQ9LBHEIew=; b=G+BmjeiHfQkXjo4My/zO9tdhf0PVaW6LzA6xieVKsT5aLa3SQ+ShQaVSrO9xBSVu7t3APa uBxUBvfLqOmYlO6YTGOT195Iev6lE3DfIyJx5UZxqf+SxQ+wHN+ULxm3QVPctVECVC2ubb ePwmjgOqrpCbmh3SGWKBKM2gK7J2ZpQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-595-cvTYnFueNbGheckDPJMuxw-1; Tue, 03 Nov 2020 09:03:58 -0500 X-MC-Unique: cvTYnFueNbGheckDPJMuxw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7A2E3107465A; Tue, 3 Nov 2020 14:03:56 +0000 (UTC) Received: from localhost (ovpn-12-165.pek2.redhat.com [10.72.12.165]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5383F5C230; Tue, 3 Nov 2020 14:03:55 +0000 (UTC) Date: Tue, 3 Nov 2020 22:03:52 +0800 From: "bhe@redhat.com" To: Rahul Gopakumar Cc: "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , "akpm@linux-foundation.org" , "natechancellor@gmail.com" , "ndesaulniers@google.com" , "clang-built-linux@googlegroups.com" , "rostedt@goodmis.org" , Rajender M , Yiu Cho Lau , Peter Jonasson , Venkatesh Rajaram Subject: Re: Performance regressions in "boot_time" tests in Linux 5.8 Kernel Message-ID: <20201103140352.GB3177@MiWiFi-R3L-srv> References: <20201013131735.GL25604@MiWiFi-R3L-srv> <20201020151814.GU25604@MiWiFi-R3L-srv> <20201022040440.GX25604@MiWiFi-R3L-srv> <20201102143035.GA3177@MiWiFi-R3L-srv> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 11/03/20 at 12:34pm, Rahul Gopakumar wrote: > >> So, you mean with the draft patch applied, the initial performance > regression goes away, just many page corruption errors with call trace > are seen, right? > > Yes, that's right. > > >> And the performance regression is about 2sec delay in > your system? > > The delay due to this new page corruption issue is about > 3 secs. > > Here is the summary > > * Initial problem - 2 secs > * Draft patch - Fixes initial problem (recovers 2 secs) but > brings in new page corruption issue (3 secs) > > >> Could you tell how you setup vmware VM so that I can ask our QA for > help to create a vmware VM for me to test? > > * Use vSphere ESXi 6.7 or 7.0 GA. > * Create VM using vSphere Web Client and specify 1TB VM Memory. > * Install RHEL 8.1, that's the guest used in this test. OK, I see. The draft patch fix the original issue, seems some boundary of memory region is not handled correctly. Thanks for confirmation. The memory layout is important in this case. Not sure if making a VM gesut as you suggested can also create a system with below memory layout. [ 0.008842] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff] [ 0.008842] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0xbfffffff] [ 0.008843] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x55ffffffff] [ 0.008844] ACPI: SRAT: Node 1 PXM 1 [mem 0x5600000000-0xaaffffffff] [ 0.008844] ACPI: SRAT: Node 2 PXM 2 [mem 0xab00000000-0xfcffffffff] [ 0.008845] ACPI: SRAT: Node 2 PXM 2 [mem 0x10000000000-0x1033fffffff] > > With draft patch, you should be able to reproduce the issue. > Let me know if you need more details.