From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CAC7C282C2 for ; Wed, 13 Feb 2019 17:31:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DF0AF218D3 for ; Wed, 13 Feb 2019 17:31:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LGOA3l+D" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2393056AbfBMRbG (ORCPT ); Wed, 13 Feb 2019 12:31:06 -0500 Received: from mail-yb1-f195.google.com ([209.85.219.195]:35359 "EHLO mail-yb1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732968AbfBMRbG (ORCPT ); Wed, 13 Feb 2019 12:31:06 -0500 Received: by mail-yb1-f195.google.com with SMTP id v2so1251628ybm.2 for ; Wed, 13 Feb 2019 09:31:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=l0iJMC3nIJ8svmT1FtwsaSZy7m2+g5rwgWORKmW7r+w=; b=LGOA3l+D603eElafmDTwlVCieXD56HWlDQNEvBWrfEGUelrKThDmaCwphGcUOzD9BW kYw/gFWh8uc00tbrXGKNEZlj3I1N4PZWMYMBAnsVdZIjr5AvUqR1oPAgTHbrNIRuLKB+ sQqHjF1/YEEIZpeIocFjW+QT21N0gExbq0hVBTFjjYWfplM7FebOEn7bLx1Co7b5p+Ry YqGXyKEEYSuakEs9866ZzZQ55RdZJxO8LEBwKH4zUv+wVRgfNT1+3ZLfF4uNZn4Q/BZg Mxi9kehvBKVYxbH45A/cR5e1ns0dPzLhxlD4lipx6AMtQzbU+Y0dSFvV22okUBytjqzr 2TcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=l0iJMC3nIJ8svmT1FtwsaSZy7m2+g5rwgWORKmW7r+w=; b=gwr5DPxOt1oIIqqC795KIA7otEVe0apVbhgR3klKY0nHtxocjSp+mdUnWD40YZrUzB 7/3ai86rjrHZVcBRzIGesvA95qJevHsouHy3imvl+QNAsTXPqSZigFOG6fA8LLEUdeHq oi1zTFn+qULQ7DnPMwkFPei3gwg2l/bo1UWDOqdYiXTQh5ZeaMn8VPqDvODjwqD7ju8P BBjMvzVLn9VNFt50x1eqeVvDKqs+VfoN6nU/S5hhHqujIaUWmOwy9FO35z7vETgMEIRx SBay9yLXN3cFmulYalFwHJdqbLHP+BIyjrn7wHc+2tZADZVNcs/X9v+ib44DJIOWXPVl tnaw== X-Gm-Message-State: AHQUAuaGzR02vElCgfQ0OCNLmLfWZ7ynpJsFCEx8yb2Zvg/fFnWwGojo hZXss1a9EjG+KaZYnbxBmm0= X-Google-Smtp-Source: AHgI3IbXUswKIoqloEMx5qrVwvFRI71MxVcg7sTMTd/qeikwChG726ONgSpBQH7ix9XsRplfQm3FFQ== X-Received: by 2002:a25:3b46:: with SMTP id i67mr1526803yba.522.1550079065692; Wed, 13 Feb 2019 09:31:05 -0800 (PST) Received: from [192.168.86.235] (c-73-241-150-70.hsd1.ca.comcast.net. [73.241.150.70]) by smtp.gmail.com with ESMTPSA id r20sm6625357ywa.13.2019.02.13.09.31.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 09:31:04 -0800 (PST) Subject: Re: [PATCH iproute2 net-next v2 3/4] ss: Buffer raw fields first, then render them as a table To: Stefano Brivio Cc: Stephen Hemminger , netdev@vger.kernel.org, Sabrina Dubroca References: <82f1bc98-df6d-2b0a-17e5-fa057563284e@gmail.com> <20190213093711.13ab560e@redhat.com> From: Eric Dumazet Message-ID: Date: Wed, 13 Feb 2019 09:31:03 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20190213093711.13ab560e@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 02/13/2019 12:37 AM, Stefano Brivio wrote: > On Tue, 12 Feb 2019 16:42:04 -0800 > Eric Dumazet wrote: > >> I do not get it. >> >> "ss -emoi " uses almost 1KB per socket. >> >> 10,000,000 sockets -> we need about 10GB of memory ??? >> >> This is a serious regression. > > I guess this is rather subjective: the worst case I considered back then > was the output of 'ss -tei0' (less than 500 bytes) for one million > sockets, which gives 500M of memory, which should in turn be fine on a > machine handling one million sockets. > > Now, if 'ss -emoi' on 10 million sockets is an actual use case (out of > curiosity: how are you going to process that output? Would JSON help?), > I see two easy options to solve this: ss -temoi | parser (written in shell or awk or whatever...) This is a use case, I just got bitten because using ss command actually OOM my container, while trying to debug a busy GFE. The host itself can have 10,000,000 TCP sockets, but usually sysadmin shells run in a container with no more than 500 MB available. Otherwise, it would be too easy for a buggy program to OOM the whole machine and have angry customers. > > 1. flush the output every time we reach a given buffer size (1M > perhaps). This might make the resulting blocks slightly unaligned, > with occasional loss of readability on lines occurring every 1k to > 10k sockets approximately, even though after 1k sockets column sizes > won't change much (it looks anyway better than the original), and I > don't expect anybody to actually scroll that output > > 2. add a switch for unbuffered output, but then you need to remember to > pass it manually, and the whole output would be as bad as the > original in case you need the switch. > > I'd rather go with 1., it's easy to implement (we already have partial > flushing with '--events') and it looks like a good compromise on > usability. Thoughts? > 1 seems fine, but a switch for 'please do not try to format' would be fine. I wonder why we try to 'format' when stdout is a pipe or a regular file .