From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6702DC433E7 for ; Tue, 13 Oct 2020 06:55:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0F4B42078A for ; Tue, 13 Oct 2020 06:55:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DVCqZc0K" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732563AbgJMGzf (ORCPT ); Tue, 13 Oct 2020 02:55:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50362 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727353AbgJMGze (ORCPT ); Tue, 13 Oct 2020 02:55:34 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32669C0613D0; Mon, 12 Oct 2020 23:55:34 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id e18so22576610wrw.9; Mon, 12 Oct 2020 23:55:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=tsRtK+RphjJoix8LNFIUyEqIBIL/eXaIC2bZ4t/CBOQ=; b=DVCqZc0KrivZAyxTL0AvxUJIK8ar/eLOwEdWUo/cKCbAoTHgn33/8+mW9OHucS84v6 0imUvQqQAst6kKaHRL53ucde/VGeMSByqHyllJ/uDQpUysG6+31rPcgj26UxVsnLdPw8 bfyqYHaZd6F7DTxDh4M9MqsWd1Fqr7QO6Eib0jiaFCXEG3HkYbI14lKYhDar8x0Cdl8s pB7FD0Rhk2JUG7jdZ45/x7EknGoQjDpzV2vQ7D5UdDPRn5/rJbsCpCJQU5neobKMaKUQ U6QTs0HJiG92p59SleNX/m+xsMzOj0Pb+03mhokRCjuiTEZ/11U7eLxoWvknFy0pXz3f EJ2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=tsRtK+RphjJoix8LNFIUyEqIBIL/eXaIC2bZ4t/CBOQ=; b=dvGKsceSZHrXAvGm8wap9aB1lMJfJn7DxNjAZ4U5adYy66ftRTYeoY49LlHhrl5qV3 j7Xi89dfLgY7McgHT9yK5OPf65n7z9z3W7lT8wolgX3bEJkDrp9uQhBGyqGfjZMvai1g 5IsEJek8s2G4gtDYy6ObD3uMIosl/leTRsKWur1u2q1c6AURct5F3zPrbHeo4Lu3XWgE uvAfatXTNEbj2j+KCLm3A5C6PJ+vPN2vI53cS6jujPX08SgaFd41RJWO/WA+lru1MUH7 TB1KHJ8GzAoglk0hioq4hImnszojbgkibZXrQrUlcoUMCfHhlZWyMIiU1aC3PCbgosGJ SgCQ== X-Gm-Message-State: AOAM530mG2NdPEkYGbbCz/z7nRDJBo1rguo8zJwUAWuW3JEnAadNDhy7 C7sQspuCN1ui1jxjvpscTqs= X-Google-Smtp-Source: ABdhPJyQXupiGXk53k8bnCs10W8RAjL7Zi7PA+tSuK0vnrjR6U4YpJdR1j2LtYICyCerFkByTf/QBw== X-Received: by 2002:adf:e412:: with SMTP id g18mr11015720wrm.211.1602572132746; Mon, 12 Oct 2020 23:55:32 -0700 (PDT) Received: from [192.168.8.147] ([37.171.149.236]) by smtp.gmail.com with ESMTPSA id f14sm26683540wme.22.2020.10.12.23.55.29 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 12 Oct 2020 23:55:32 -0700 (PDT) Subject: Re: [External] Re: [PATCH] mm: proc: add Sock to /proc/meminfo To: Muchun Song , Eric Dumazet Cc: Eric Dumazet , Cong Wang , Greg KH , rafael@kernel.org, "Michael S. Tsirkin" , Jason Wang , David Miller , Jakub Kicinski , Alexey Dobriyan , Andrew Morton , Alexey Kuznetsov , Hideaki YOSHIFUJI , Steffen Klassert , Herbert Xu , Shakeel Butt , Will Deacon , Michal Hocko , Roman Gushchin , Neil Brown , rppt@kernel.org, Sami Tolvanen , "Kirill A. Shutemov" , Feng Tang , Paolo Abeni , Willem de Bruijn , Randy Dunlap , Florian Westphal , gustavoars@kernel.org, Pablo Neira Ayuso , Dexuan Cui , Jakub Sitnicki , Peter Zijlstra , Christian Brauner , "Eric W. Biederman" , Thomas Gleixner , dave@stgolabs.net, Michel Lespinasse , Jann Horn , chenqiwu@xiaomi.com, christophe.leroy@c-s.fr, Minchan Kim , Martin KaFai Lau , Alexei Starovoitov , Daniel Borkmann , Miaohe Lin , Kees Cook , LKML , virtualization@lists.linux-foundation.org, Linux Kernel Network Developers , linux-fsdevel , linux-mm References: <20201010103854.66746-1-songmuchun@bytedance.com> <9262ea44-fc3a-0b30-54dd-526e16df85d1@gmail.com> From: Eric Dumazet Message-ID: <93d5ad90-2bd5-07ad-618e-456ed2e6da87@gmail.com> Date: Tue, 13 Oct 2020 08:55:29 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/12/20 11:53 AM, Muchun Song wrote: > We are not complaining about TCP using too much memory, but how do > we know that TCP uses a lot of memory. When I firstly face this problem, > I do not know who uses the 25GB memory and it is not shown in the /proc/meminfo. > If we can know the amount memory of the socket buffer via /proc/meminfo, we > may not need to spend a lot of time troubleshooting this problem. Not everyone > knows that a lot of memory may be used here. But I believe many people > should know /proc/meminfo to confirm memory users. Adding yet another operations in networking fast path is a high cost to pay just to add one extra line in /proc/meminfo, while /proc/net/sockstat is already a good proxy, with per protocol details, instead of a single bucket. I reiterate that zero copy would make this counter out of sync, unless special support is added (adding yet another operations ?) Also your patch does not address gazillions of page allocations from drivers in RX path. Here at Google the majority of networking mm usage when hosts are under stress is in RX path, when out of order queues start to grow in TCP sockets. Allocations in TX path were greatly reduced and optimally sized with the introduction of /proc/sys/net/ipv4/tcp_notsent_lowat. We have gazillions of put_page()/__free_page()/__free_pages()/alloc_page()/... all over the places, adding yet another tracking of "this page is used by networking stacks" is going to be quite a big project. I thought memcg was a better goal in the long run, lets focus on it. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.8 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9E92C43467 for ; Tue, 13 Oct 2020 06:55:38 +0000 (UTC) Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2B76120797 for ; Tue, 13 Oct 2020 06:55:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DVCqZc0K" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2B76120797 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=virtualization-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 708DB878A1; Tue, 13 Oct 2020 06:55:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WXUSkmSQontc; Tue, 13 Oct 2020 06:55:36 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by whitealder.osuosl.org (Postfix) with ESMTP id 7A0AF87883; Tue, 13 Oct 2020 06:55:36 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 65843C0052; Tue, 13 Oct 2020 06:55:36 +0000 (UTC) Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 78639C0051 for ; Tue, 13 Oct 2020 06:55:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 5505387A5A for ; Tue, 13 Oct 2020 06:55:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AaH5NIBnG8KR for ; Tue, 13 Oct 2020 06:55:34 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by hemlock.osuosl.org (Postfix) with ESMTPS id 68AE487A49 for ; Tue, 13 Oct 2020 06:55:34 +0000 (UTC) Received: by mail-wr1-f66.google.com with SMTP id g12so22576572wrp.10 for ; Mon, 12 Oct 2020 23:55:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=tsRtK+RphjJoix8LNFIUyEqIBIL/eXaIC2bZ4t/CBOQ=; b=DVCqZc0KrivZAyxTL0AvxUJIK8ar/eLOwEdWUo/cKCbAoTHgn33/8+mW9OHucS84v6 0imUvQqQAst6kKaHRL53ucde/VGeMSByqHyllJ/uDQpUysG6+31rPcgj26UxVsnLdPw8 bfyqYHaZd6F7DTxDh4M9MqsWd1Fqr7QO6Eib0jiaFCXEG3HkYbI14lKYhDar8x0Cdl8s pB7FD0Rhk2JUG7jdZ45/x7EknGoQjDpzV2vQ7D5UdDPRn5/rJbsCpCJQU5neobKMaKUQ U6QTs0HJiG92p59SleNX/m+xsMzOj0Pb+03mhokRCjuiTEZ/11U7eLxoWvknFy0pXz3f EJ2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=tsRtK+RphjJoix8LNFIUyEqIBIL/eXaIC2bZ4t/CBOQ=; b=VZpDAYrFVWeosycgLu06IFMEXqmR55a8/jDCfHeal8/WzDnyk4VjpEOY+YWHf8kvEK 8iOylq02Zw/pBbcmxuEevJ8N4Ph9ojvs+7DWrdXG71g5jHGpcbWPb765oHg1KuXxSCJc iYQMn5tqsrS3jzbi3NFc25pXEEe59aH5ZZYa3av4QUsHxx+IBBikTaYjoTcgr3an4xWC XBkEQgrMWIF5wMkQQskzhnt11X1/ron4f/6X7IkAywu17EekLPIkrQBXNLkDY9yud87B igBRRmzBdogkpC9XekxlC+u6TI20NHTmVUKwo9jOGQRo4AqmvC4C1pKN6f7wQfbfkhrF xkAQ== X-Gm-Message-State: AOAM531quyxr9mw2JH/0uziRhxPdwEWfNt4WMlbH6kyGvJkvhu8tfYrg o+DU4rI/hktLwZIWrUwNg24= X-Google-Smtp-Source: ABdhPJyQXupiGXk53k8bnCs10W8RAjL7Zi7PA+tSuK0vnrjR6U4YpJdR1j2LtYICyCerFkByTf/QBw== X-Received: by 2002:adf:e412:: with SMTP id g18mr11015720wrm.211.1602572132746; Mon, 12 Oct 2020 23:55:32 -0700 (PDT) Received: from [192.168.8.147] ([37.171.149.236]) by smtp.gmail.com with ESMTPSA id f14sm26683540wme.22.2020.10.12.23.55.29 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 12 Oct 2020 23:55:32 -0700 (PDT) Subject: Re: [External] Re: [PATCH] mm: proc: add Sock to /proc/meminfo To: Muchun Song , Eric Dumazet References: <20201010103854.66746-1-songmuchun@bytedance.com> <9262ea44-fc3a-0b30-54dd-526e16df85d1@gmail.com> From: Eric Dumazet Message-ID: <93d5ad90-2bd5-07ad-618e-456ed2e6da87@gmail.com> Date: Tue, 13 Oct 2020 08:55:29 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Cc: Miaohe Lin , Feng Tang , Michal Hocko , rafael@kernel.org, Neil Brown , Alexei Starovoitov , LKML , linux-mm , Eric Dumazet , Christian Brauner , Michel Lespinasse , Will Deacon , Steffen Klassert , dave@stgolabs.net, Herbert Xu , Daniel Borkmann , "Michael S. Tsirkin" , Dexuan Cui , Peter Zijlstra , Sami Tolvanen , Jakub Kicinski , Paolo Abeni , Alexey Dobriyan , Pablo Neira Ayuso , "Eric W. Biederman" , Kees Cook , Jann Horn , Shakeel Butt , Alexey Kuznetsov , Cong Wang , Thomas Gleixner , virtualization@lists.linux-foundation.org, chenqiwu@xiaomi.com, Martin KaFai Lau , Jakub Sitnicki , christophe.leroy@c-s.fr, Willem de Bruijn , Hideaki YOSHIFUJI , Greg KH , Randy Dunlap , Florian Westphal , gustavoars@kernel.org, Roman Gushchin , Minchan Kim , rppt@kernel.org, Linux Kernel Network Developers , linux-fsdevel , Andrew Morton , David Miller , "Kirill A. Shutemov" X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" On 10/12/20 11:53 AM, Muchun Song wrote: > We are not complaining about TCP using too much memory, but how do > we know that TCP uses a lot of memory. When I firstly face this problem, > I do not know who uses the 25GB memory and it is not shown in the /proc/meminfo. > If we can know the amount memory of the socket buffer via /proc/meminfo, we > may not need to spend a lot of time troubleshooting this problem. Not everyone > knows that a lot of memory may be used here. But I believe many people > should know /proc/meminfo to confirm memory users. Adding yet another operations in networking fast path is a high cost to pay just to add one extra line in /proc/meminfo, while /proc/net/sockstat is already a good proxy, with per protocol details, instead of a single bucket. I reiterate that zero copy would make this counter out of sync, unless special support is added (adding yet another operations ?) Also your patch does not address gazillions of page allocations from drivers in RX path. Here at Google the majority of networking mm usage when hosts are under stress is in RX path, when out of order queues start to grow in TCP sockets. Allocations in TX path were greatly reduced and optimally sized with the introduction of /proc/sys/net/ipv4/tcp_notsent_lowat. We have gazillions of put_page()/__free_page()/__free_pages()/alloc_page()/... all over the places, adding yet another tracking of "this page is used by networking stacks" is going to be quite a big project. I thought memcg was a better goal in the long run, lets focus on it. _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization