From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 703F3C47095 for ; Wed, 9 Jun 2021 05:30:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4895A61249 for ; Wed, 9 Jun 2021 05:30:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236150AbhFIFcS (ORCPT ); Wed, 9 Jun 2021 01:32:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231443AbhFIFcQ (ORCPT ); Wed, 9 Jun 2021 01:32:16 -0400 Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5440AC061574 for ; Tue, 8 Jun 2021 22:30:06 -0700 (PDT) Received: by mail-qt1-x831.google.com with SMTP id v6so8204940qta.9 for ; Tue, 08 Jun 2021 22:30:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:subject:from:to:cc:date:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=NN7xmTnHOJkFIgJqmIQHYCemTkBXultHJB0UVCrZGJU=; b=lKzfwxvZYmXMio6DxcohP6UZ0VciCjemrQxWd/Wo4qqmgYF2Kw0+5kLAN7z/fwkDKy CMV4dTuWzd4b7yedIkBvfi16W4eG3HU0wDjNVmQrry+FtDpDJmpUKhCjwKWvVedGPIwf +j+c2DPRMfYzvW+kZM4dce49dDqa17muJgH0Vwi+d3ThcjVZ2y25P9KM7fGgM1OQnGwK 1Tpmx6W77SsIcd07ri9QThEVp208q+R33tneycV7Vy4ljxuYwzj0yNAT/7aYbShIon47 v2hSeJu/41txZYKiOjqJDfeAuO6rIEJvsMK2IZT5hkdmRIdg+GQ16XfW/F4gzxPrMyI9 9bcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=NN7xmTnHOJkFIgJqmIQHYCemTkBXultHJB0UVCrZGJU=; b=s6s+xD0NTDgLw/K3wFAdwOaGzhxmGZFmlT7j9QYrOM/a2OWQQsOzywxLQ3enilIaQ1 8uuquYm4s26WkXgfqUh1ucvzvIVrbhYEXIv/PVHm43ucbm+ad7p0wAdWDNpZo3+tM25u UOHul5uNMy8fsKVVJvye2uPLixx6OF3X9gzSyFCle72hXb+w+1BCjc3RQ6G8yrQxHxZw byx8X/SC67iJYzZvQ9W4uPj2mvmm9Rk2IqPxP5J4c+I/quoXUXnciY7En1GvtLmnd3R7 8J+3FG5aFS6sIDgW7sxNvDRUdJs1D+NH/JQA1eAjM8qhev4Ogp+kPBXSyFbSakIS2oZE Zh/A== X-Gm-Message-State: AOAM533EVE7KmBCGnVxxdaUWCCzptP1kJ1Fe9hRKbJUZWjJHOl4VU7EE eI9LF6tasWfpwGVjFLtShd8= X-Google-Smtp-Source: ABdhPJyNarADNIgx5i6021RzYgKXFMSTP9Xz/qNJbnqV2oKGzxx1S0Sz/8dqbhP4KEJgsnRW5BQVUA== X-Received: by 2002:ac8:5202:: with SMTP id r2mr19484392qtn.86.1623216605538; Tue, 08 Jun 2021 22:30:05 -0700 (PDT) Received: from ?IPv6:2804:14c:482:87bb::1000? ([2804:14c:482:87bb::1000]) by smtp.gmail.com with ESMTPSA id y20sm12122386qtv.64.2021.06.08.22.30.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 22:30:05 -0700 (PDT) Message-ID: Subject: Re: [PATCH v2 3/3] powerpc/mm/hash: Avoid multiple HPT resize-downs on memory hotunplug From: Leonardo =?ISO-8859-1?Q?Br=E1s?= To: David Gibson Cc: Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , Sandipan Das , Mike Rapoport , Andrew Morton , "Aneesh Kumar K.V" , Nicholas Piggin , Nathan Lynch , David Hildenbrand , Scott Cheloha , Laurent Dufour , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org Date: Wed, 09 Jun 2021 02:30:36 -0300 In-Reply-To: References: <20210430143607.135005-1-leobras.c@gmail.com> <20210430143607.135005-4-leobras.c@gmail.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.40.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 2021-06-07 at 15:20 +1000, David Gibson wrote: > On Fri, Apr 30, 2021 at 11:36:10AM -0300, Leonardo Bras wrote: > > During memory hotunplug, after each LMB is removed, the HPT may be > > resized-down if it would map a max of 4 times the current amount of > > memory. > > (2 shifts, due to introduced histeresis) > > > > It usually is not an issue, but it can take a lot of time if HPT > > resizing-down fails. This happens  because resize-down failures > > usually repeat at each LMB removal, until there are no more bolted > > entries > > conflict, which can take a while to happen. > > > > This can be solved by doing a single HPT resize at the end of > > memory > > hotunplug, after all requested entries are removed. > > > > To make this happen, it's necessary to temporarily disable all HPT > > resize-downs before hotunplug, re-enable them after hotunplug ends, > > and then resize-down HPT to the current memory size. > > > > As an example, hotunplugging 256GB from a 385GB guest took 621s > > without > > this patch, and 100s after applied. > > > > Signed-off-by: Leonardo Bras > > Hrm.  This looks correct, but it seems overly complicated. > > AFAICT, the resize calls that this adds should in practice be the > *only* times we call resize, all the calls from the lower level code > should be suppressed.  That's correct. > In which case can't we just remove those calls > entirely, and not deal with the clunky locking and exclusion here. > That should also remove the need for the 'shrinking' parameter in > 1/3. If I get your suggestion correctly, you suggest something like: 1 - Never calling resize_hpt_for_hotplug() in hash__remove_section_mapping(), thus not needing the srinking parameter. 2 - Functions in hotplug-memory.c that call dlpar_remove_lmb() would in fact call another function to do the batch resize_hpt_for_hotplug() for them If so, that assumes that no other function that currently calls resize_hpt_for_hotplug() under another path, or if they do, it does not need to actually resize the HPT. Is the above correct? There are some examples of functions that currently call resize_hpt_for_hotplug() by another path: add_memory_driver_managed virtio_mem_add_memory dev_dax_kmem_probe reserve_additional_memory balloon_process add_ballooned_pages __add_memory probe_store __remove_memory pseries_remove_memblock remove_memory dev_dax_kmem_remove virtio_mem_remove_memory memunmap_pages pci_p2pdma_add_resource virtio_fs_setup_dax Best regards, Leonardo Bras From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E9A8C47095 for ; Wed, 9 Jun 2021 05:30:41 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 51096611BD for ; Wed, 9 Jun 2021 05:30:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 51096611BD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4G0G0b2SRPz3bxV for ; Wed, 9 Jun 2021 15:30:39 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=lKzfwxvZ; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::82b; helo=mail-qt1-x82b.google.com; envelope-from=leobras.c@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=lKzfwxvZ; dkim-atps=neutral Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4G0G036881z2ymb for ; Wed, 9 Jun 2021 15:30:09 +1000 (AEST) Received: by mail-qt1-x82b.google.com with SMTP id l17so12973410qtq.12 for ; Tue, 08 Jun 2021 22:30:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:subject:from:to:cc:date:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=NN7xmTnHOJkFIgJqmIQHYCemTkBXultHJB0UVCrZGJU=; b=lKzfwxvZYmXMio6DxcohP6UZ0VciCjemrQxWd/Wo4qqmgYF2Kw0+5kLAN7z/fwkDKy CMV4dTuWzd4b7yedIkBvfi16W4eG3HU0wDjNVmQrry+FtDpDJmpUKhCjwKWvVedGPIwf +j+c2DPRMfYzvW+kZM4dce49dDqa17muJgH0Vwi+d3ThcjVZ2y25P9KM7fGgM1OQnGwK 1Tpmx6W77SsIcd07ri9QThEVp208q+R33tneycV7Vy4ljxuYwzj0yNAT/7aYbShIon47 v2hSeJu/41txZYKiOjqJDfeAuO6rIEJvsMK2IZT5hkdmRIdg+GQ16XfW/F4gzxPrMyI9 9bcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=NN7xmTnHOJkFIgJqmIQHYCemTkBXultHJB0UVCrZGJU=; b=P1BAE/XigqpA4ByhqRrt3PERO0f2X3+nat3bir4ZnSeuecazMEBaPAGc5feqbJkriY j7UeajOIGuLWBLFmtJ0qqrHt2OWE6w79zOtHKUcIfeo+ouAoZYRb0EbJdUIv5jrWtk3j oBS9jOdWOjymKm/rIl+X0S7XV+F1SF93dsxo9OE4seJh+efQmErpu2VYrK+ckS2Y0UJw Pp5VJyWawUylAOyTzUoQ5WMIHlHUTJj7Qw2PZJXwgx9nMM3GZ4qeItzmqwYirkJHrn6N LkqLzspdEVtUyRXoH7YbaDsMvHnAOS762NoSSVEsxRnyreeZ7KVUZlJuNbsELFu/vW1L 1oBw== X-Gm-Message-State: AOAM533iWWHnMWy7kuuMZhdxPCpKSlPuJ9c+UBsayMiTWKV5YXDWZE2o 9cVpL56mDj/ST78DPYMGhts= X-Google-Smtp-Source: ABdhPJyNarADNIgx5i6021RzYgKXFMSTP9Xz/qNJbnqV2oKGzxx1S0Sz/8dqbhP4KEJgsnRW5BQVUA== X-Received: by 2002:ac8:5202:: with SMTP id r2mr19484392qtn.86.1623216605538; Tue, 08 Jun 2021 22:30:05 -0700 (PDT) Received: from ?IPv6:2804:14c:482:87bb::1000? ([2804:14c:482:87bb::1000]) by smtp.gmail.com with ESMTPSA id y20sm12122386qtv.64.2021.06.08.22.30.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 22:30:05 -0700 (PDT) Message-ID: Subject: Re: [PATCH v2 3/3] powerpc/mm/hash: Avoid multiple HPT resize-downs on memory hotunplug From: Leonardo =?ISO-8859-1?Q?Br=E1s?= To: David Gibson Date: Wed, 09 Jun 2021 02:30:36 -0300 In-Reply-To: References: <20210430143607.135005-1-leobras.c@gmail.com> <20210430143607.135005-4-leobras.c@gmail.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.40.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nathan Lynch , "Aneesh Kumar K.V" , David Hildenbrand , Scott Cheloha , linux-kernel@vger.kernel.org, Nicholas Piggin , Paul Mackerras , Sandipan Das , Andrew Morton , Laurent Dufour , linuxppc-dev@lists.ozlabs.org, Mike Rapoport Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Mon, 2021-06-07 at 15:20 +1000, David Gibson wrote: > On Fri, Apr 30, 2021 at 11:36:10AM -0300, Leonardo Bras wrote: > > During memory hotunplug, after each LMB is removed, the HPT may be > > resized-down if it would map a max of 4 times the current amount of > > memory. > > (2 shifts, due to introduced histeresis) > > > > It usually is not an issue, but it can take a lot of time if HPT > > resizing-down fails. This happens  because resize-down failures > > usually repeat at each LMB removal, until there are no more bolted > > entries > > conflict, which can take a while to happen. > > > > This can be solved by doing a single HPT resize at the end of > > memory > > hotunplug, after all requested entries are removed. > > > > To make this happen, it's necessary to temporarily disable all HPT > > resize-downs before hotunplug, re-enable them after hotunplug ends, > > and then resize-down HPT to the current memory size. > > > > As an example, hotunplugging 256GB from a 385GB guest took 621s > > without > > this patch, and 100s after applied. > > > > Signed-off-by: Leonardo Bras > > Hrm.  This looks correct, but it seems overly complicated. > > AFAICT, the resize calls that this adds should in practice be the > *only* times we call resize, all the calls from the lower level code > should be suppressed.  That's correct. > In which case can't we just remove those calls > entirely, and not deal with the clunky locking and exclusion here. > That should also remove the need for the 'shrinking' parameter in > 1/3. If I get your suggestion correctly, you suggest something like: 1 - Never calling resize_hpt_for_hotplug() in hash__remove_section_mapping(), thus not needing the srinking parameter. 2 - Functions in hotplug-memory.c that call dlpar_remove_lmb() would in fact call another function to do the batch resize_hpt_for_hotplug() for them If so, that assumes that no other function that currently calls resize_hpt_for_hotplug() under another path, or if they do, it does not need to actually resize the HPT. Is the above correct? There are some examples of functions that currently call resize_hpt_for_hotplug() by another path: add_memory_driver_managed virtio_mem_add_memory dev_dax_kmem_probe reserve_additional_memory balloon_process add_ballooned_pages __add_memory probe_store __remove_memory pseries_remove_memblock remove_memory dev_dax_kmem_remove virtio_mem_remove_memory memunmap_pages pci_p2pdma_add_resource virtio_fs_setup_dax Best regards, Leonardo Bras