From: Huang Jianan via Linux-erofs <linux-erofs@lists.ozlabs.org>
To: Gao Xiang <hsiangkao@redhat.com>
Cc: zhangshiming@oppo.com, guoweichao@oppo.com,
linux-erofs@lists.ozlabs.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH] erofs: support adjust lz4 history window size
Date: Tue, 23 Feb 2021 10:03:59 +0800 [thread overview]
Message-ID: <11afa9f6-109a-c660-1664-7a596d6836b6@oppo.com> (raw)
In-Reply-To: <20210222044410.GA1038521@xiangao.remote.csb>
Hi Xiang,
On 2021/2/22 12:44, Gao Xiang wrote:
> Hi Jianan,
>
> On Thu, Feb 18, 2021 at 08:00:49PM +0800, Huang Jianan via Linux-erofs wrote:
>> From: huangjianan <huangjianan@oppo.com>
>>
>> lz4 uses LZ4_DISTANCE_MAX to record history preservation. When
>> using rolling decompression, a block with a higher compression
>> ratio will cause a larger memory allocation (up to 64k). It may
>> cause a large resource burden in extreme cases on devices with
>> small memory and a large number of concurrent IOs. So appropriately
>> reducing this value can improve performance.
>>
>> Decreasing this value will reduce the compression ratio (except
>> when input_size <LZ4_DISTANCE_MAX). But considering that erofs
>> currently only supports 4k output, reducing this value will not
>> significantly reduce the compression benefits.
>>
>> Signed-off-by: Huang Jianan <huangjianan@oppo.com>
>> Signed-off-by: Guo Weichao <guoweichao@oppo.com>
>> ---
>> fs/erofs/decompressor.c | 13 +++++++++----
>> fs/erofs/erofs_fs.h | 3 ++-
>> fs/erofs/internal.h | 3 +++
>> fs/erofs/super.c | 3 +++
>> 4 files changed, 17 insertions(+), 5 deletions(-)
>>
>> diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c
>> index 1cb1ffd10569..94ae56b3ff71 100644
>> --- a/fs/erofs/decompressor.c
>> +++ b/fs/erofs/decompressor.c
>> @@ -36,22 +36,27 @@ static int z_erofs_lz4_prepare_destpages(struct z_erofs_decompress_req *rq,
>> struct page *availables[LZ4_MAX_DISTANCE_PAGES] = { NULL };
>> unsigned long bounced[DIV_ROUND_UP(LZ4_MAX_DISTANCE_PAGES,
>> BITS_PER_LONG)] = { 0 };
>> + unsigned int lz4_distance_pages = LZ4_MAX_DISTANCE_PAGES;
>> void *kaddr = NULL;
>> unsigned int i, j, top;
>>
>> + if (EROFS_SB(rq->sb)->compr_alg)
>> + lz4_distance_pages = DIV_ROUND_UP(EROFS_SB(rq->sb)->compr_alg,
>> + PAGE_SIZE) + 1;
>> +
> Thanks for your patch, I agree that will reduce runtime memory
> footpoint. and keep max sliding window ondisk in bytes (rather
> than in blocks) is better., but could we calculate lz4_distance_pages
> ahead when reading super_block?
Thanks for suggestion, i will update it soon.
> Also, in the next cycle, I'd like to introduce a bitmap for available
> algorithms (maximum 16-bit) for the next LZMA algorithm, and for each
> available algorithm introduces an on-disk variable-array like below:
> bitmap(16-bit) 2 1 0
> ... LZMA LZ4
> __le16 compr_opt_off; /* get the opt array start offset
> (I think also in 4-byte) */
>
> compr alg 0 (lz4) __le16 alg_opt_size;
> /* next opt off = roundup(off + alg_opt_size, 4); */
> __le16 lz4_max_distance;
>
> /* 4-byte aligned */
> compr alg x (if available) u8 alg_opt_size;
> ...
>
> ...
>
> When reading sb, first, it scans the whole bitmap, and get all the
> available algorithms in the image at once. And then read such compr
> opts one-by-one.
>
> Do you have some interest and extra time to implement it? :) That
> makes me work less since I'm debugging mbpcluster compression now...
Sounds good, I will try to do this part of the work.
Thanks,
Jianan
> Thanks,
> Gao Xiang
>
next prev parent reply other threads:[~2021-02-23 2:04 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-18 12:00 [PATCH] erofs: support adjust lz4 history window size Huang Jianan via Linux-erofs
2021-02-22 4:44 ` Gao Xiang
2021-02-23 2:03 ` Huang Jianan via Linux-erofs [this message]
2021-02-23 2:18 ` Gao Xiang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=11afa9f6-109a-c660-1664-7a596d6836b6@oppo.com \
--to=linux-erofs@lists.ozlabs.org \
--cc=guoweichao@oppo.com \
--cc=hsiangkao@redhat.com \
--cc=huangjianan@oppo.com \
--cc=linux-kernel@vger.kernel.org \
--cc=zhangshiming@oppo.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).