From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B37EC43334 for ; Wed, 29 Jun 2022 20:43:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231268AbiF2Un2 (ORCPT ); Wed, 29 Jun 2022 16:43:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37678 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229745AbiF2UnZ (ORCPT ); Wed, 29 Jun 2022 16:43:25 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D84D21825 for ; Wed, 29 Jun 2022 13:43:24 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id r20so24206925wra.1 for ; Wed, 29 Jun 2022 13:43:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=jK9//luGxpfbLffwOAXZvUD0KZmVzGC+4VJ1Mf8Ewyg=; b=k5WSowoVnLxuxKjSPP3cp1XzTEAHY3y45UhQuF8UEDAyCTaVCqG/CT+GI0HDpCJiRr qY92QFJIvnA+icgYFsho+2NHk8VIu8knAAsA4n5acDav+H8Bx/JChuc9/xU2pHxunkqn Xa7V6+8eHKvHEzuihehfP5VnTNMUxG2sVEkn+lldtmnf/cCTkXjtmmUEqEhHB1ghA1+P XiNUznA+U4VlknZfH43Wecpj/M1WX+pi8c7VlQcttS6Tn+4WR79rWAWtr3AWhjkWvAlB 8c14CN87gcOsljgVtquohoB6BqLa9yxRumbngWbW4Zv5Aafipq5gZb9HHF5pygsGcTHe sV5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=jK9//luGxpfbLffwOAXZvUD0KZmVzGC+4VJ1Mf8Ewyg=; b=F/R7pQXR/O5pM+xNMmDIWKXdFECmBvZuiW2xvsdmeSjqhmZaEmmGIgOg9Wiey4FSgH vVpzC9haTx6m7oXcnS1NLVQdAdXx+x15TmiNrgut9Y8oryWMt3PQppuWuucq1c5FKEf3 yUb26xDgr/R8p3s1vYuU/1RYl/6hMZwxL+8sHZOit+OoZQjwdtw8bgafvaJrHavRYbeg Nif5V08EaDjOYXFviK8vXLgD2dQgmTmuVddiik0Zo+2TgnLEKJla7WbhBghybtAk4Xna FtSKS+VsH9RNJJdr0fzp/oiS2Xgh8RX/fy1ontOYoN9pQCVahBEVB98khhYnHmF0pa21 oPxQ== X-Gm-Message-State: AJIora//MAyh52ga/ysUcTBUTyBvxU2/7R4hl486IkGTCtYiIp163s1L NtuSIjAwQ0EtRXWWvBDMA9tdujwyPLrU6MvyMJZG9g== X-Google-Smtp-Source: AGRyM1sdZpRIIt8pzdPqsysENF0+n2K+v1kR9bTNvP8/I1WmweNa1mfdcg2Fg6hW5XcLplZP4kyRd9qLNe68d2DEstQ= X-Received: by 2002:a5d:52c6:0:b0:21b:9f39:78de with SMTP id r6-20020a5d52c6000000b0021b9f3978demr5179522wrv.699.1656535402268; Wed, 29 Jun 2022 13:43:22 -0700 (PDT) MIME-Version: 1.0 References: <20220623220613.3014268-1-kaleshsingh@google.com> <20220623220613.3014268-2-kaleshsingh@google.com> In-Reply-To: From: Kalesh Singh Date: Wed, 29 Jun 2022 13:43:11 -0700 Message-ID: Subject: Re: [PATCH v2 1/2] procfs: Add 'size' to /proc//fdinfo/ To: Brian Foster Cc: =?UTF-8?Q?Christian_K=C3=B6nig?= , =?UTF-8?Q?Christian_K=C3=B6nig?= , Alexander Viro , Christoph Hellwig , Stephen Brennan , David.Laight@aculab.com, Ioannis Ilkos , "T.J. Mercier" , Suren Baghdasaryan , "Cc: Android Kernel" , Jonathan Corbet , Sumit Semwal , Andrew Morton , Johannes Weiner , Christoph Anton Mitterer , Paul Gortmaker , Mike Rapoport , Randy Dunlap , LKML , linux-fsdevel , "open list:DOCUMENTATION" , Linux Media Mailing List , DRI mailing list , "moderated list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jun 29, 2022 at 5:23 AM Brian Foster wrote: > > On Tue, Jun 28, 2022 at 03:38:02PM -0700, Kalesh Singh wrote: > > On Tue, Jun 28, 2022 at 4:54 AM Brian Foster wrote= : > > > > > > On Thu, Jun 23, 2022 at 03:06:06PM -0700, Kalesh Singh wrote: > > > > To be able to account the amount of memory a process is keeping pin= ned > > > > by open file descriptors add a 'size' field to fdinfo output. > > > > > > > > dmabufs fds already expose a 'size' field for this reason, remove t= his > > > > and make it a common field for all fds. This allows tracking of > > > > other types of memory (e.g. memfd and ashmem in Android). > > > > > > > > Signed-off-by: Kalesh Singh > > > > Reviewed-by: Christian K=C3=B6nig > > > > --- > > > > > > > > Changes in v2: > > > > - Add Christian's Reviewed-by > > > > > > > > Changes from rfc: > > > > - Split adding 'size' and 'path' into a separate patches, per Chr= istian > > > > - Split fdinfo seq_printf into separate lines, per Christian > > > > - Fix indentation (use tabs) in documentaion, per Randy > > > > > > > > Documentation/filesystems/proc.rst | 12 ++++++++++-- > > > > drivers/dma-buf/dma-buf.c | 1 - > > > > fs/proc/fd.c | 9 +++++---- > > > > 3 files changed, 15 insertions(+), 7 deletions(-) > > > > > ... > > > > > > Also not sure if it matters that much for your use case, but somethin= g > > > worth noting at least with shmem is that one can do something like: > > > > > > # cat /proc/meminfo | grep Shmem: > > > Shmem: 764 kB > > > # xfs_io -fc "falloc -k 0 10m" ./file > > > # ls -alh file > > > -rw-------. 1 root root 0 Jun 28 07:22 file > > > # stat file > > > File: file > > > Size: 0 Blocks: 20480 IO Block: 4096 regular e= mpty file > > > # cat /proc/meminfo | grep Shmem: > > > Shmem: 11004 kB > > > > > > ... where the resulting memory usage isn't reflected in i_size (but i= s > > > is in i_blocks/bytes). > > > > I tried a similar experiment a few times, but I don't see the same > > results. In my case, there is not any change in shmem. IIUC the > > fallocate is allocating the disk space not shared memory. > > > > Sorry, it was implied in my previous test was that I was running against > tmpfs. So regardless of fs, the fallocate keep_size semantics shown in > both cases is as expected: the underlying blocks are allocated and the > inode size is unchanged. > > What wasn't totally clear to me when I read this patch was 1. whether > tmpfs refers to Shmem and 2. whether tmpfs allowed this sort of > operation. The test above seems to confirm both, however, right? E.g., a > more detailed example: > > # mount | grep /tmp > tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel,nr_inodes=3D1048576,in= ode64) > # cat /proc/meminfo | grep Shmem: > Shmem: 5300 kB > # xfs_io -fc "falloc -k 0 1g" /tmp/file > # stat /tmp/file > File: /tmp/file > Size: 0 Blocks: 2097152 IO Block: 4096 regular empty= file > Device: 22h/34d Inode: 45 Links: 1 > Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root) > Context: unconfined_u:object_r:user_tmp_t:s0 > Access: 2022-06-29 08:04:01.301307154 -0400 > Modify: 2022-06-29 08:04:01.301307154 -0400 > Change: 2022-06-29 08:04:01.451312834 -0400 > Birth: 2022-06-29 08:04:01.301307154 -0400 > # cat /proc/meminfo | grep Shmem: > Shmem: 1053876 kB > # rm -f /tmp/file > # cat /proc/meminfo | grep Shmem: > Shmem: 5300 kB > > So clearly this impacts Shmem.. was your test run against tmpfs or some > other (disk based) fs? Hi Brian, Thanks for clarifying. My issue was tmpfs not mounted at /tmp in my system: =3D=3D> meminfo.start <=3D=3D Shmem: 572 kB =3D=3D> meminfo.stop <=3D=3D Shmem: 51688 kB > > FWIW, I don't have any objection to exposing inode size if it's commonly > useful information. My feedback was more just an fyi that i_size doesn't > necessarily reflect underlying space consumption (whether it's memory or > disk space) in more generic cases, because it sounds like that is really > what you're after here. The opposite example to the above would be > something like an 'xfs_io -fc "truncate 1t" /tmp/file', which shows a > 1TB inode size with zero additional shmem usage. >From these cases, it seems the more generic way to do this is by calculating the actual size consumed using the blocks. (i_blocks * 512). So in the latter example 'xfs_io -fc "truncate 1t" /tmp/file' the size consumed would be zero. Let me know if it sounds ok to you and I can repost the updated version. Thanks, Kalesh > > Brian > > > cat /proc/meminfo > meminfo.start > > xfs_io -fc "falloc -k 0 50m" ./xfs_file > > cat /proc/meminfo > meminfo.stop > > tail -n +1 meminfo.st* | grep -i '=3D=3D\|Shmem:' > > > > =3D=3D> meminfo.start <=3D=3D > > Shmem: 484 kB > > =3D=3D> meminfo.stop <=3D=3D > > Shmem: 484 kB > > > > ls -lh xfs_file > > -rw------- 1 root root 0 Jun 28 15:12 xfs_file > > > > stat xfs_file > > File: xfs_file > > Size: 0 Blocks: 102400 IO Block: 4096 regular emp= ty file > > > > Thanks, > > Kalesh > > > > > > > > Brian > > > > > > > > > > > /* show_fd_locks() never deferences files so a stale value is= safe */ > > > > show_fd_locks(m, file, files); > > > > -- > > > > 2.37.0.rc0.161.g10f37bed90-goog > > > > > > > > > > > -- > To unsubscribe from this group and stop receiving emails from it, send an= email to kernel-team+unsubscribe@android.com. > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6894FC433EF for ; Wed, 29 Jun 2022 20:43:25 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C227310EE36; Wed, 29 Jun 2022 20:43:24 +0000 (UTC) Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by gabe.freedesktop.org (Postfix) with ESMTPS id 10B1E10ED15 for ; Wed, 29 Jun 2022 20:43:24 +0000 (UTC) Received: by mail-wr1-x42d.google.com with SMTP id cl1so472836wrb.4 for ; Wed, 29 Jun 2022 13:43:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=jK9//luGxpfbLffwOAXZvUD0KZmVzGC+4VJ1Mf8Ewyg=; b=k5WSowoVnLxuxKjSPP3cp1XzTEAHY3y45UhQuF8UEDAyCTaVCqG/CT+GI0HDpCJiRr qY92QFJIvnA+icgYFsho+2NHk8VIu8knAAsA4n5acDav+H8Bx/JChuc9/xU2pHxunkqn Xa7V6+8eHKvHEzuihehfP5VnTNMUxG2sVEkn+lldtmnf/cCTkXjtmmUEqEhHB1ghA1+P XiNUznA+U4VlknZfH43Wecpj/M1WX+pi8c7VlQcttS6Tn+4WR79rWAWtr3AWhjkWvAlB 8c14CN87gcOsljgVtquohoB6BqLa9yxRumbngWbW4Zv5Aafipq5gZb9HHF5pygsGcTHe sV5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=jK9//luGxpfbLffwOAXZvUD0KZmVzGC+4VJ1Mf8Ewyg=; b=KWWAV5eZu1hR4EN8CMKKKkuevEjRkALwiqcoTd0wMfxer4b17YLj61FoVKCmuGJaOJ azOOV5Q6eNmyvPyMK4yYf6y/FU/JrFKEtkEpNbNfT+2JaPZR9p5SLGiSOs9iuTL/3TFC sqU4/2ysROENzU417Fo+rZ5xgTgA8hISv0HC80Kaa2j5vTY6mvdWf0LmzgZEp32T/ob2 xucsX+4/mAPCcCsBn+/m9iOuhwOUn+DxR9CTpPCb1IjqMHte3YMzimgs9Rj7L5ld4GNP ehObkEqkh7RjZimGR9yk5S+a+hpFsKazWQCgKL8ipLWw1cNBuJ5GQo57jXpMppdHZaJc lKJw== X-Gm-Message-State: AJIora/bz116yj5GcS2jV4cGTGIivqw/LYEcrBdJIjuKE9t9zIWuAx9s OHMbi8k8X67q6F+hVq+XBtoTNus5QDwS6MwRMwOWTw== X-Google-Smtp-Source: AGRyM1sdZpRIIt8pzdPqsysENF0+n2K+v1kR9bTNvP8/I1WmweNa1mfdcg2Fg6hW5XcLplZP4kyRd9qLNe68d2DEstQ= X-Received: by 2002:a5d:52c6:0:b0:21b:9f39:78de with SMTP id r6-20020a5d52c6000000b0021b9f3978demr5179522wrv.699.1656535402268; Wed, 29 Jun 2022 13:43:22 -0700 (PDT) MIME-Version: 1.0 References: <20220623220613.3014268-1-kaleshsingh@google.com> <20220623220613.3014268-2-kaleshsingh@google.com> In-Reply-To: From: Kalesh Singh Date: Wed, 29 Jun 2022 13:43:11 -0700 Message-ID: Subject: Re: [PATCH v2 1/2] procfs: Add 'size' to /proc//fdinfo/ To: Brian Foster Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "open list:DOCUMENTATION" , DRI mailing list , Stephen Brennan , Paul Gortmaker , Sumit Semwal , Jonathan Corbet , Christoph Anton Mitterer , Christoph Hellwig , "Cc: Android Kernel" , Linux Media Mailing List , =?UTF-8?Q?Christian_K=C3=B6nig?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Alexander Viro , Suren Baghdasaryan , "T.J. Mercier" , Randy Dunlap , Ioannis Ilkos , LKML , David.Laight@aculab.com, Johannes Weiner , linux-fsdevel , Andrew Morton , =?UTF-8?Q?Christian_K=C3=B6nig?= , Mike Rapoport Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Wed, Jun 29, 2022 at 5:23 AM Brian Foster wrote: > > On Tue, Jun 28, 2022 at 03:38:02PM -0700, Kalesh Singh wrote: > > On Tue, Jun 28, 2022 at 4:54 AM Brian Foster wrote= : > > > > > > On Thu, Jun 23, 2022 at 03:06:06PM -0700, Kalesh Singh wrote: > > > > To be able to account the amount of memory a process is keeping pin= ned > > > > by open file descriptors add a 'size' field to fdinfo output. > > > > > > > > dmabufs fds already expose a 'size' field for this reason, remove t= his > > > > and make it a common field for all fds. This allows tracking of > > > > other types of memory (e.g. memfd and ashmem in Android). > > > > > > > > Signed-off-by: Kalesh Singh > > > > Reviewed-by: Christian K=C3=B6nig > > > > --- > > > > > > > > Changes in v2: > > > > - Add Christian's Reviewed-by > > > > > > > > Changes from rfc: > > > > - Split adding 'size' and 'path' into a separate patches, per Chr= istian > > > > - Split fdinfo seq_printf into separate lines, per Christian > > > > - Fix indentation (use tabs) in documentaion, per Randy > > > > > > > > Documentation/filesystems/proc.rst | 12 ++++++++++-- > > > > drivers/dma-buf/dma-buf.c | 1 - > > > > fs/proc/fd.c | 9 +++++---- > > > > 3 files changed, 15 insertions(+), 7 deletions(-) > > > > > ... > > > > > > Also not sure if it matters that much for your use case, but somethin= g > > > worth noting at least with shmem is that one can do something like: > > > > > > # cat /proc/meminfo | grep Shmem: > > > Shmem: 764 kB > > > # xfs_io -fc "falloc -k 0 10m" ./file > > > # ls -alh file > > > -rw-------. 1 root root 0 Jun 28 07:22 file > > > # stat file > > > File: file > > > Size: 0 Blocks: 20480 IO Block: 4096 regular e= mpty file > > > # cat /proc/meminfo | grep Shmem: > > > Shmem: 11004 kB > > > > > > ... where the resulting memory usage isn't reflected in i_size (but i= s > > > is in i_blocks/bytes). > > > > I tried a similar experiment a few times, but I don't see the same > > results. In my case, there is not any change in shmem. IIUC the > > fallocate is allocating the disk space not shared memory. > > > > Sorry, it was implied in my previous test was that I was running against > tmpfs. So regardless of fs, the fallocate keep_size semantics shown in > both cases is as expected: the underlying blocks are allocated and the > inode size is unchanged. > > What wasn't totally clear to me when I read this patch was 1. whether > tmpfs refers to Shmem and 2. whether tmpfs allowed this sort of > operation. The test above seems to confirm both, however, right? E.g., a > more detailed example: > > # mount | grep /tmp > tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel,nr_inodes=3D1048576,in= ode64) > # cat /proc/meminfo | grep Shmem: > Shmem: 5300 kB > # xfs_io -fc "falloc -k 0 1g" /tmp/file > # stat /tmp/file > File: /tmp/file > Size: 0 Blocks: 2097152 IO Block: 4096 regular empty= file > Device: 22h/34d Inode: 45 Links: 1 > Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root) > Context: unconfined_u:object_r:user_tmp_t:s0 > Access: 2022-06-29 08:04:01.301307154 -0400 > Modify: 2022-06-29 08:04:01.301307154 -0400 > Change: 2022-06-29 08:04:01.451312834 -0400 > Birth: 2022-06-29 08:04:01.301307154 -0400 > # cat /proc/meminfo | grep Shmem: > Shmem: 1053876 kB > # rm -f /tmp/file > # cat /proc/meminfo | grep Shmem: > Shmem: 5300 kB > > So clearly this impacts Shmem.. was your test run against tmpfs or some > other (disk based) fs? Hi Brian, Thanks for clarifying. My issue was tmpfs not mounted at /tmp in my system: =3D=3D> meminfo.start <=3D=3D Shmem: 572 kB =3D=3D> meminfo.stop <=3D=3D Shmem: 51688 kB > > FWIW, I don't have any objection to exposing inode size if it's commonly > useful information. My feedback was more just an fyi that i_size doesn't > necessarily reflect underlying space consumption (whether it's memory or > disk space) in more generic cases, because it sounds like that is really > what you're after here. The opposite example to the above would be > something like an 'xfs_io -fc "truncate 1t" /tmp/file', which shows a > 1TB inode size with zero additional shmem usage. >From these cases, it seems the more generic way to do this is by calculating the actual size consumed using the blocks. (i_blocks * 512). So in the latter example 'xfs_io -fc "truncate 1t" /tmp/file' the size consumed would be zero. Let me know if it sounds ok to you and I can repost the updated version. Thanks, Kalesh > > Brian > > > cat /proc/meminfo > meminfo.start > > xfs_io -fc "falloc -k 0 50m" ./xfs_file > > cat /proc/meminfo > meminfo.stop > > tail -n +1 meminfo.st* | grep -i '=3D=3D\|Shmem:' > > > > =3D=3D> meminfo.start <=3D=3D > > Shmem: 484 kB > > =3D=3D> meminfo.stop <=3D=3D > > Shmem: 484 kB > > > > ls -lh xfs_file > > -rw------- 1 root root 0 Jun 28 15:12 xfs_file > > > > stat xfs_file > > File: xfs_file > > Size: 0 Blocks: 102400 IO Block: 4096 regular emp= ty file > > > > Thanks, > > Kalesh > > > > > > > > Brian > > > > > > > > > > > /* show_fd_locks() never deferences files so a stale value is= safe */ > > > > show_fd_locks(m, file, files); > > > > -- > > > > 2.37.0.rc0.161.g10f37bed90-goog > > > > > > > > > > > -- > To unsubscribe from this group and stop receiving emails from it, send an= email to kernel-team+unsubscribe@android.com. >