From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07F16C636D7 for ; Tue, 21 Feb 2023 19:46:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69C8E6B0075; Tue, 21 Feb 2023 14:46:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 672F66B007B; Tue, 21 Feb 2023 14:46:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 514B86B007D; Tue, 21 Feb 2023 14:46:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4229A6B0075 for ; Tue, 21 Feb 2023 14:46:46 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 05F554071F for ; Tue, 21 Feb 2023 19:46:46 +0000 (UTC) X-FDA: 80492331612.17.B4ED2DE Received: from mail-ed1-f46.google.com (mail-ed1-f46.google.com [209.85.208.46]) by imf29.hostedemail.com (Postfix) with ESMTP id 1A999120014 for ; Tue, 21 Feb 2023 19:46:43 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MQHLJl4B; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.46 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677008804; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7S9vWZgZSV5LDTvYeAKdLMWR3Iaz9MZBgTFX637ncOA=; b=XrBUYgHmjMWZ7MLPXmI35M4wbWalE8BH3F1CVbC4Uam2/tUQCdYE0mcjD/A9jTyTma5RfV BOsiSW+WRm4B4dEHX/GUzR9tnHf7GGl+TAaGKirYqm8S1882Uto2YqZPbYQ+dsqhbmfPjH ovZXvbMq5YFMYz6MUUEYk6TlTrkMPOw= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MQHLJl4B; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.46 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677008804; a=rsa-sha256; cv=none; b=vE3i/S2rqUJ8s66Hpx2iXiT7dX2it2h2RnqU0cpGJKIfeKUn4fDYqm+WaO7F3XPOjjEjlj 3vVcq5Ox5vX4g2/sX24gTNK0um2LgThO6DpSwcd/speGjrVkJKVFnNPo/OCOVm2m73LdjU zdV7Vpq3WTe4GS1Yxd3dir4Ui6OCACs= Received: by mail-ed1-f46.google.com with SMTP id eg37so17939800edb.12 for ; Tue, 21 Feb 2023 11:46:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=7S9vWZgZSV5LDTvYeAKdLMWR3Iaz9MZBgTFX637ncOA=; b=MQHLJl4ByucplQ0XEDzx4YSnkR690HJluE3Gi9uKnF1zpwk6YGfzwQMUTEpTe9Q5cM Z3dsdr5An/TIejaSuQzjqFiIIh9U6+TTyInV2nHQoEY7esDc0AHDZXq/sJcE/tpr+Kre wbr7FLzqRMqy4WCyaYz12UO2XQl6JbLMwHgutoqFN4xHRseXZ5FVOp2Dxebzc6EEmyBD XKlBshBPIidps69ewYLaYtxVB78EY+2X63LZzYomQ8btpjRfCRO7UBADMe4P81A5dxZ0 HTPJ9668Vi5bNHlDZIfdVI8p7y453az5fPURnPYASsSKUz+D674R5LDB2kVFZpBxGre7 YS3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=7S9vWZgZSV5LDTvYeAKdLMWR3Iaz9MZBgTFX637ncOA=; b=3U8iOfLaB5wKxpWrtCGQZ8Iwn7iuLytuuX5OPXUagAbdZzgl/AAJNlTht8GI8GSfbh 2GaJioJv2gp5fEpsWT+yFdrjafBM4QyDGLyyR4CfAEyXGkH3u8D5jtnA5NSaVxiv3Wmd 40B+1GefxDE8cVok+/qtYP2Z1P5T9amGUGO79SNVpSgykL6efCM5oPsgsSX9fwlgwHPC fxDGXQ53WfPxZYpmLjavlKttxr41UUJ0t5/3JQdjaLcPBbdJ1Z1MDMx9XF5bcV370Cmz /4bloIGpVX3am4B8LKun2EZspVvA+DOwIODpVgk2HpmfVs5dASQfiABaEas45vTFssGY BvmA== X-Gm-Message-State: AO0yUKXiZiz2FgO6OpYjhiHj78nZdjTcwfK0jS0Fr4ZJaXn+J4YK4fYo ZNWD3Uce1NTmQgHLBQU+bvkU4tIXJt9Rjl1mpVJdYw== X-Google-Smtp-Source: AK7set/I2HFzWIjNa0y+ibN+oQp0UcrsO6YbjrQ9iSs70NvUaykhVPqvzmrnJffApqYvvIkz40bZIyk3RLFlWm/eCfY= X-Received: by 2002:a17:907:c907:b0:8d8:4578:18e0 with SMTP id ui7-20020a170907c90700b008d8457818e0mr2854689ejc.10.1677008802348; Tue, 21 Feb 2023 11:46:42 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Yosry Ahmed Date: Tue, 21 Feb 2023 11:46:05 -0800 Message-ID: Subject: Re: [LSF/MM/BPF TOPIC] Swap Abstraction / Native Zswap To: Yang Shi Cc: lsf-pc@lists.linux-foundation.org, Johannes Weiner , Linux-MM , Michal Hocko , Shakeel Butt , David Rientjes , Hugh Dickins , Seth Jennings , Dan Streetman , Vitaly Wool , Peter Xu , Minchan Kim , Andrew Morton Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 1A999120014 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: c4466fycihj6rrmtzajixwnch3camgae X-HE-Tag: 1677008803-109350 X-HE-Meta: U2FsdGVkX1+/yIwTb4WtI4SQ8GvkJcR2GrR7KAbVdY9D4c1x4fR+nbD12gpzm8cLyDoY+OYAZ2OlAYDh+r7yJD0WMX12PH0EUO4DSk5GJtJk/vnY3ZQpDdQY7P6ReuG4VfynC1ih1BMWs2731daHqsyPnolyBczUvkY8z75q+LK6g6uX/rI4aJFPYCZhvER9T/g714Sge7+FSsU575FF7xsT6bdys3MZb4LkvVbXRf0wRyLEpSTqyiSiLfxaBfQoVynFxF2/Het0FzoLC5kDVZ4IutNYVKWgz1FJQpq1sjspDIRHKL0/481//yr6OXL+Xh3L8OA/UdIYWixI0rYavojK6byP8v3kC09RLiSIx8eDvUGEjqnecoJGObVUogRGkwr2rhsqMvkSJX6Mi12EqkST22BqzH5REdPfBaI6GR7D1LEshJTCmHFN01GOagSfVMT0qr+z4bNpXtrEuCd7Dn9yDYhjNcR7fAVWYB7lRROm/Kt7UaPMFUlVhVTMZY4XWZAtFg1BgI+Dtdf7NlGFyKu9EAIQq+gmyW3o8hfVSuZj9cAP86cOStLUdR4EyZTMtx6G8KvZTzlcZ8IqJdO5wrRdHIpSzTthnWmNRMTGaY8SOlGWB6v13D4l3arKRorLbyTiqPCp6ClUYFW9/2Wi1qVR6FOrF+MStw5Xg5e7l5Y7jda+eN8yAEIL0lGHJdnkU8K1yB9Qif1kL049pkmzWweVus8l2InvNacRmhmj4ENLmK06cChAuaC1T/R9X6C3/YIXrp9BkDLIyifAN96YNMqFQDycGVblRQqPUogmjKMKnTK13j+Mhg1a2H8HBvAe3c6JB70qmucMMrphYjqKc/k90CSItP6DL6cFX7r379yUKaHCUfsneNyDvTZHinYpVmVXVyxRgf7mkcl1W3+CGxKYa05cvBM0ys1DA8UisIO3r3P5YkGojQm2foyJT9Ap5WOMaJY/L7S6Kkyh1PH JwAtcrol hQsSS7R3loRIu2XJothqbynTkY8ZMzaP+1am8bC9L0pUUmA0MjPKFIIFjqKkAB6LI4W5ys+M6x8jQoRXXGPkkzoWnHkPEg/YzKvzS3x4hlLYBTsuErcxtpYJrgN6kyupRFNaGdoSc5Vx510w3z8UOSshj4LE9pKPWmpBeXPCHzPCNh9U9EsJ79PrVQO8N8F0oSQC3ypDXeDjrCQRjGcX61Vn0nTnK4uyQfTUwYHydn9aGHOinJCrxBaIrERlbJMFL4wPPOW5x3PtBz4pin2tcsOuStZV20Sf8V5KTEGCA8jfkTzE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 21, 2023 at 11:26 AM Yang Shi wrote: > > On Tue, Feb 21, 2023 at 10:56 AM Yosry Ahmed wrote: > > > > On Tue, Feb 21, 2023 at 10:40 AM Yang Shi wrote: > > > > > > Hi Yosry, > > > > > > Thanks for proposing this topic. I was thinking about this before but > > > I didn't make too much progress due to some other distractions, and I > > > got a couple of follow up questions about your design. Please see the > > > inline comments below. > > > > Great to see interested folks, thanks! > > > > > > > > > > > On Sat, Feb 18, 2023 at 2:39 PM Yosry Ahmed wrote: > > > > > > > > Hello everyone, > > > > > > > > I would like to propose a topic for the upcoming LSF/MM/BPF in May > > > > 2023 about swap & zswap (hope I am not too late). > > > > > > > > ==================== Intro ==================== > > > > Currently, using zswap is dependent on swapfiles in an unnecessary > > > > way. To use zswap, you need a swapfile configured (even if the space > > > > will not be used) and zswap is restricted by its size. When pages > > > > reside in zswap, the corresponding swap entry in the swapfile cannot > > > > be used, and is essentially wasted. We also go through unnecessary > > > > code paths when using zswap, such as finding and allocating a swap > > > > entry on the swapout path, or readahead in the swapin path. I am > > > > proposing a swapping abstraction layer that would allow us to remove > > > > zswap's dependency on swapfiles. This can be done by introducing a > > > > data structure between the actual swapping implementation (swapfiles, > > > > zswap) and the rest of the MM code. > > > > > > > > ==================== Objective ==================== > > > > Enabling the use of zswap without a backing swapfile, which makes > > > > zswap useful for a wider variety of use cases. Also, when zswap is > > > > used with a swapfile, the pages in zswap do not use up space in the > > > > swapfile, so the overall swapping capacity increases. > > > > > > > > ==================== Idea ==================== > > > > Introduce a data structure, which I currently call a swap_desc, as an > > > > abstraction layer between swapping implementation and the rest of MM > > > > code. Page tables & page caches would store a swap id (encoded as a > > > > swp_entry_t) instead of directly storing the swap entry associated > > > > with the swapfile. This swap id maps to a struct swap_desc, which acts > > > > as our abstraction layer. All MM code not concerned with swapping > > > > details would operate in terms of swap descs. The swap_desc can point > > > > to either a normal swap entry (associated with a swapfile) or a zswap > > > > entry. It can also include all non-backend specific operations, such > > > > as the swapcache (which would be a simple pointer in swap_desc), swap > > > > counting, etc. It creates a clear, nice abstraction layer between MM > > > > code and the actual swapping implementation. > > > > > > How will the swap_desc be allocated? Dynamically or preallocated? Is > > > it 1:1 mapped to the swap slots on swap devices (whatever it is > > > backed, for example, zswap, swap partition, swapfile, etc)? > > > > I imagine swap_desc's would be dynamically allocated when we need to > > swap something out. When allocated, a swap_desc would either point to > > a zswap_entry (if available), or a swap slot otherwise. In this case, > > it would be 1:1 mapped to swapped out pages, not the swap slots on > > devices. > > It makes sense to be 1:1 mapped to swapped out pages if the swapfile > is used as the back of zswap. > > > > > I know that it might not be ideal to make allocations on the reclaim > > path (although it would be a small-ish slab allocation so we might be > > able to get away with it), but otherwise we would have statically > > allocated swap_desc's for all swap slots on a swap device, even unused > > ones, which I imagine is too expensive. Also for things like zswap, it > > doesn't really make sense to preallocate at all. > > Yeah, it is not perfect to allocate memory in the reclamation path. We > do have such cases, but the fewer the better IMHO. Yeah. Perhaps we can preallocate a pool of swap_desc's on top of the slab cache, idk if that makes sense, or if there is a way to tell slab to proactively refill a cache. I am open to suggestions here. I don't think we should/can preallocate the swap_desc's, and we cannot completely eliminate the allocations in the reclaim path. We can only try to minimize them through caching, etc. Right? > > > > > WDYT? > > > > > > > > > > > > > ==================== Benefits ==================== > > > > This work enables using zswap without a backing swapfile and increases > > > > the swap capacity when zswap is used with a swapfile. It also creates > > > > a separation that allows us to skip code paths that don't make sense > > > > in the zswap path (e.g. readahead). We get to drop zswap's rbtree > > > > which might result in better performance (less lookups, less lock > > > > contention). > > > > > > > > The abstraction layer also opens the door for multiple cleanups (e.g. > > > > removing swapper address spaces, removing swap count continuation > > > > code, etc). Another nice cleanup that this work enables would be > > > > separating the overloaded swp_entry_t into two distinct types: one for > > > > things that are stored in page tables / caches, and for actual swap > > > > entries. In the future, we can potentially further optimize how we use > > > > the bits in the page tables instead of sticking everything into the > > > > current type/offset format. > > > > > > > > Another potential win here can be swapoff, which can be more practical > > > > by directly scanning all swap_desc's instead of going through page > > > > tables and shmem page caches. > > > > > > > > Overall zswap becomes more accessible and available to a wider range > > > > of use cases. > > > > > > How will you handle zswap writeback? Zswap may writeback to the backed > > > swap device IIUC. Assuming you have both zswap and swapfile, they are > > > separate devices with this design, right? If so, is the swapfile still > > > the writeback target of zswap? And if it is the writeback target, what > > > if swapfile is full? > > > > When we try to writeback from zswap, we try to allocate a swap slot in > > the swapfile, and switch the swap_desc to point to that instead. The > > process would be transparent to the rest of MM (page tables, page > > cache, etc). If the swapfile is full, then there's really nothing we > > can do, reclaim fails and we start OOMing. I imagine this is the same > > behavior as today when swap is full, the difference would be that we > > have to fill both zswap AND the swapfile to get to the OOMing point, > > so an overall increased swapping capacity. > > When zswap is full, but swapfile is not yet, will the swap try to > writeback zswap to swapfile to make more room for zswap or just swap > out to swapfile directly? > The current behavior is that we swap to swapfile directly in this case, which is far from ideal as we break LRU ordering by skipping zswap. I believe this should be addressed, but not as part of this effort. The work to make zswap respect the LRU ordering by writing back from zswap to make room can be done orthogonal to this effort. I believe Johannes was looking into this at some point. > > > > > > > > Anyway I'm interested in attending the discussion for this topic. > > > > Great! Looking forward to discuss this more! > > > > > > > > > > > > > ==================== Cost ==================== > > > > The obvious downside of this is added memory overhead, specifically > > > > for users that use swapfiles without zswap. Instead of paying one byte > > > > (swap_map) for every potential page in the swapfile (+ swap count > > > > continuation), we pay the size of the swap_desc for every page that is > > > > actually in the swapfile, which I am estimating can be roughly around > > > > 24 bytes or so, so maybe 0.6% of swapped out memory. The overhead only > > > > scales with pages actually swapped out. For zswap users, it should be > > > > a win (or at least even) because we get to drop a lot of fields from > > > > struct zswap_entry (e.g. rbtree, index, etc). > > > > > > > > Another potential concern is readahead. With this design, we have no > > > > way to get a swap_desc given a swap entry (type & offset). We would > > > > need to maintain a reverse mapping, adding a little bit more overhead, > > > > or search all swapped out pages instead :). A reverse mapping might > > > > pump the per-swapped page overhead to ~32 bytes (~0.8% of swapped out > > > > memory). > > > > > > > > ==================== Bottom Line ==================== > > > > It would be nice to discuss the potential here and the tradeoffs. I > > > > know that other folks using zswap (or interested in using it) may find > > > > this very useful. I am sure I am missing some context on why things > > > > are the way they are, and perhaps some obvious holes in my story. > > > > Looking forward to discussing this with anyone interested :) > > > > > > > > I think Johannes may be interested in attending this discussion, since > > > > a lot of ideas here are inspired by discussions I had with him :)