From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx2.suse.de ([195.135.220.15]:44823 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752470AbdK1Xvg (ORCPT ); Tue, 28 Nov 2017 18:51:36 -0500 Date: Wed, 29 Nov 2017 00:49:37 +0100 From: David Sterba To: Nick Terrell Cc: "linux-btrfs@vger.kernel.org" , Martin Steigerwald , Imran Geriskovan Subject: Re: Read before you deploy btrfs + zstd Message-ID: <20171128234937.GG3553@suse.cz> Reply-To: dsterba@suse.cz References: <20171113225046.GD28899@suse.cz> <20171114185331.GJ28899@twin.jikos.cz> <20171115143906.GM28899@twin.jikos.cz> <20171121162239.GO3553@suse.cz> <5BB834FC-A54C-4448-8D86-1D2F22806096@fb.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <5BB834FC-A54C-4448-8D86-1D2F22806096@fb.com> Sender: linux-btrfs-owner@vger.kernel.org List-ID: On Tue, Nov 28, 2017 at 09:31:57PM +0000, Nick Terrell wrote: > > > On Nov 21, 2017, at 8:22 AM, David Sterba wrote: > > > > On Wed, Nov 15, 2017 at 08:09:15PM +0000, Nick Terrell wrote: > >> On 11/15/17, 6:41 AM, "David Sterba" wrote: > >>> The branch is now in a state that can be tested. Turns out the memory > >>> requirements are too much for grub, so the boot fails with "not enough > >>> memory". The calculated value > >>> > >>> ZSTD_BTRFS_MAX_INPUT: 131072 > >>> ZSTD_DStreamWorkspaceBound with ZSTD_BTRFS_MAX_INPUT: 549424 > >>> > >>> This is not something I could fix easily, we'd probalby need a tuned > >>> version of ZSTD for grub constraints. Adding Nick to CC. > >> > >> If I understand the grub code correctly, we only need to read, and we have > >> the entire input and output buffer in one segment. In that case you can use > >> ZSTD_initDCtx(), and ZSTD_decompressDCtx(). ZSTD_DCtxWorkspaceBound() is > >> only 155984. See decompress_single() in > >> https://patchwork.kernel.org/patch/9997909/ for an example. > > > > Does not help, still ENOMEM. > > It looks like XZ had the same issue, and they make the decompression > context a static object (grep for GRUB_EMBED_DECOMPRESSOR). We could > potentially do the same and statically allocate the workspace: > > ``` > /* Could also be size_t */ > #define BTRFS_ZSTD_WORKSPACE_SIZE_U64 (155984 / sizeof(uint64_t)) > static uint64_t workspace[BTRFS_ZSTD_WORKSPACE_SIZE_U64]; > > /* ... */ > > assert(sizeof(workspace) >= ZSTD_DCtxWorkspaceBound()); > ``` Interesting, thanks for the tip, I'll try it next. I've meanwhile tried to tweak the numbers, the maximum block for zstd, that squeezed the DCtx somewhere under 48k, with block size 8k. Still enomem. I've tried to add some debugging prints to see what numbers get actually passed to the allocator, but did not see anything printed. I'm sure there is a more intelligent way to test the grub changes. So far each test loop takes quite some time, as I build the rpm package, test it in a VM and have to recreate the environmet each time.