From: "jnhuang95@gmail.com" <jnhuang95@gmail.com>
To: "Gao Xiang" <hsiangkao@aol.com>
Cc: zhangshiming <zhangshiming@oppo.com>,
linux-erofs <linux-erofs@lists.ozlabs.org>,
guoweichao <guoweichao@oppo.com>
Subject: Re: Re: [PATCH 5/5] erofs-utils: support read compressed file
Date: Sat, 17 Oct 2020 14:24:13 +0800 [thread overview]
Message-ID: <2020101714241160424912@gmail.com> (raw)
In-Reply-To: 20201016161736.GC32727@hsiangkao-HP-ZHAN-66-Pro-G1
[-- Attachment #1: Type: text/plain, Size: 25299 bytes --]
Hi Gaoxiang,
Thanks for your review and suggestions. It's great to communicate with outstanding community developers. I have learned a lot and look forward to contribute more patches in the future.
Regards,
Jianan
From: Gao Xiang
Date: 2020-10-17 00:17
To: Huang Jianan
CC: linux-erofs; guoweichao; zhangshiming; huangjianan
Subject: Re: [PATCH 5/5] erofs-utils: support read compressed file
Hi Jianan,
On Thu, Oct 15, 2020 at 09:39:59PM +0800, Huang Jianan wrote:
> Signed-off-by: Huang Jianan <huangjianan@oppo.com>
> Signed-off-by: Guo Weichao <guoweichao@oppo.com>
I read the logic below, and generally it looks good! <thumb>
I'm now handling some minor thing about this and I will
update as the following patch to speed up the development.
> ---
...
> diff --git a/fuse/decompress.c b/fuse/decompress.c
> new file mode 100644
> index 0000000..e2df3ce
> --- /dev/null
> +++ b/fuse/decompress.c
we might need to move this file to lib/decompress.c
> @@ -0,0 +1,86 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +/*
> + * Copyright (C), 2008-2020, OPPO Mobile Comm Corp., Ltd.
> + * Created by Huang Jianan <huangjianan@oppo.com>
> + */
> +
> +#include <stdlib.h>
> +#include <lz4.h>
> +
> +#include "erofs/internal.h"
> +#include "erofs/err.h"
> +#include "decompress.h"
> +#include "logging.h"
> +#include "init.h"
> +
> +static int z_erofs_shifted_transform(struct z_erofs_decompress_req *rq)
> +{
> + char *dest = rq->out + rq->ofs_out;
I think ofs_out can be omited, since we only care about the dest buffer here.
so (new) rq->out == (old)rq->out + rq->ofs_out.
> + char *src = rq->in + rq->ofs_head;
need to judge if the outputsize is larger than EROFS_BLKSIZ, since it's
undefined on-disk behavior, and can be implemented in the future
on-disk update.
> +
> + memcpy(dest, src, rq->outputsize - rq->ofs_head);
> +
> + return 0;
> +}
And this function can be folded into z_erofs_decompress(), because the
implementation here is much simple enough.
> +
> +static int z_erofs_decompress_generic(struct z_erofs_decompress_req *rq)
it might be renamed into z_erofs_decompress_lz4.
> +{
> + int ret = 0;
> + char *dest = rq->out + rq->ofs_out;
> + char *src = rq->in;
> + char *buff = NULL;
> + bool support_0padding = false;
> + unsigned int inputmargin = 0;
> +
> + if (sbk->feature_incompat & EROFS_FEATURE_INCOMPAT_LZ4_0PADDING) {
> + support_0padding = true;
> +
> + while (!src[inputmargin & ~PAGE_MASK])
> + if (!(++inputmargin & ~PAGE_MASK))
> + break;
> +
> + if (inputmargin >= rq->inputsize)
> + return -EIO;
> + }
> +
> + if (rq->ofs_head) {
we might need to rethink about the name of ofs_head.
> + buff = malloc(rq->outputsize);
> + if (!buff)
> + return -ENOMEM;
> + dest = buff;
need some cleanup, and dest variable is assigned for several times,
but without making the logic simpler.
> + }
> +
> + if (rq->partial_decoding || !support_0padding)
> + ret = LZ4_decompress_safe_partial(src + inputmargin, dest,
> + rq->inputsize - inputmargin,
> + rq->outputsize, rq->outputsize);
> + else
> + ret = LZ4_decompress_safe(src + inputmargin, dest,
> + rq->inputsize - inputmargin,
> + rq->outputsize);
> +
> + if (ret != (int)rq->outputsize) {
> + ret = -EIO;
> + goto out;
> + }
> +
> + if (rq->ofs_head) {
> + src = dest + rq->ofs_head;
> + dest = rq->out + rq->ofs_out;
> + memcpy(dest, src, rq->outputsize - rq->ofs_head);
> + }
> +
> +out:
> + if (buff)
> + free(buff);
> +
> + return ret;
> +}
> +
> +int z_erofs_decompress(struct z_erofs_decompress_req *rq)
> +{
> + if (rq->alg == Z_EROFS_COMPRESSION_SHIFTED)
> + return z_erofs_shifted_transform(rq);
> +
> + return z_erofs_decompress_generic(rq);
> +}
> diff --git a/fuse/decompress.h b/fuse/decompress.h
> new file mode 100644
> index 0000000..cd395c3
> --- /dev/null
> +++ b/fuse/decompress.h
move this file to include/erofs/decompress.h
> @@ -0,0 +1,37 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +/*
> + * Copyright (C), 2008-2020, OPPO Mobile Comm Corp., Ltd.
> + * Created by Huang Jianan <huangjianan@oppo.com>
> + */
> +
> +#ifndef __EROFS_DECOMPRESS_H
> +#define __EROFS_DECOMPRESS_H
> +
> +#include "erofs/internal.h"
> +
> +enum {
> + Z_EROFS_COMPRESSION_SHIFTED = Z_EROFS_COMPRESSION_MAX,
> + Z_EROFS_COMPRESSION_RUNTIME_MAX
> +};
> +
> +struct z_erofs_decompress_req {
> + char *in, *out;
> +
> + size_t ofs_out, ofs_head;
> + unsigned int inputsize, outputsize;
> +
> + /* indicate the algorithm will be used for decompression */
> + unsigned int alg;
> + bool partial_decoding;
> +};
> +
> +#ifdef LZ4_ENABLED
> +int z_erofs_decompress(struct z_erofs_decompress_req *rq);
> +#else
> +int z_erofs_decompress(struct z_erofs_decompress_req *rq)
> +{
> + return -EOPNOTSUPP;
> +}
> +#endif
> +
> +#endif
> diff --git a/fuse/dentry.h b/fuse/dentry.h
> index ee2144d..f89c506 100644
> --- a/fuse/dentry.h
> +++ b/fuse/dentry.h
> @@ -10,10 +10,11 @@
> #include <stdint.h>
> #include "erofs/internal.h"
>
> +/* fixme: Deal with names that exceed the allocated size */
> #ifdef __64BITS
> -#define DCACHE_ENTRY_NAME_LEN 40
> +#define DCACHE_ENTRY_NAME_LEN EROFS_NAME_LEN
> #else
> -#define DCACHE_ENTRY_NAME_LEN 48
> +#define DCACHE_ENTRY_NAME_LEN EROFS_NAME_LEN
> #endif
not related to this patch, does it relate to another preexist
bug about erofsfuse codebase?
>
> /* This struct declares a node of a k-tree. Every node has a pointer to one of
> diff --git a/fuse/init.c b/fuse/init.c
> index 8198fa7..e9cc9f8 100644
> --- a/fuse/init.c
> +++ b/fuse/init.c
> @@ -17,7 +17,23 @@
>
>
> struct erofs_super_block super;
> -static struct erofs_super_block *sbk = &super;
> +struct erofs_super_block *sbk = &super;
> +
> +static bool check_layout_compatibility(struct erofs_super_block *sb,
> + struct erofs_super_block *dsb)
> +{
> + const unsigned int feature = le32_to_cpu(dsb->feature_incompat);
> +
> + sb->feature_incompat = feature;
> +
> + /* check if current kernel meets all mandatory requirements */
> + if (feature & (~EROFS_ALL_FEATURE_INCOMPAT)) {
> + loge("unidentified incompatible feature %x, please upgrade kernel version",
> + feature & ~EROFS_ALL_FEATURE_INCOMPAT);
> + return false;
> + }
> + return true;
> +}
>
> int erofs_init_super(void)
> {
> @@ -40,6 +56,9 @@ int erofs_init_super(void)
> return -EINVAL;
> }
>
> + if (!check_layout_compatibility(sbk, sb))
> + return -EINVAL;
> +
> sbk->checksum = le32_to_cpu(sb->checksum);
> sbk->feature_compat = le32_to_cpu(sb->feature_compat);
> sbk->blkszbits = sb->blkszbits;
> @@ -56,6 +75,7 @@ int erofs_init_super(void)
> sbk->root_nid = le16_to_cpu(sb->root_nid);
>
> logp("%-15s:0x%X", STR(magic), SUPER_MEM(magic));
> + logp("%-15s:0x%X", STR(feature_incompat), SUPER_MEM(feature_incompat));
> logp("%-15s:0x%X", STR(feature_compat), SUPER_MEM(feature_compat));
> logp("%-15s:%u", STR(blkszbits), SUPER_MEM(blkszbits));
> logp("%-15s:%u", STR(root_nid), SUPER_MEM(root_nid));
> diff --git a/fuse/init.h b/fuse/init.h
> index d7a97b5..3fc4eb5 100644
> --- a/fuse/init.h
> +++ b/fuse/init.h
> @@ -13,6 +13,8 @@
>
> #define BOOT_SECTOR_SIZE 0x400
>
> +extern struct erofs_super_block *sbk;
> +
> int erofs_init_super(void);
> erofs_nid_t erofs_get_root_nid(void);
> erofs_off_t nid2addr(erofs_nid_t nid);
> diff --git a/fuse/namei.c b/fuse/namei.c
> index 7ed1168..510fcfd 100644
> --- a/fuse/namei.c
> +++ b/fuse/namei.c
> @@ -49,7 +49,7 @@ static inline dev_t new_decode_dev(u32 dev)
>
> int erofs_iget_by_nid(erofs_nid_t nid, struct erofs_vnode *vi)
> {
> - int ret;
> + int ret, ifmt;
> char buf[EROFS_BLKSIZ];
> struct erofs_inode_compact *v1;
> const erofs_off_t addr = nid2addr(nid);
> @@ -60,6 +60,11 @@ int erofs_iget_by_nid(erofs_nid_t nid, struct erofs_vnode *vi)
> return -EIO;
>
> v1 = (struct erofs_inode_compact *)buf;
> + /* fixme: support extended inode */
> + ifmt = le16_to_cpu(v1->i_format);
> + if (__inode_version(ifmt) != EROFS_INODE_LAYOUT_COMPACT)
> + return -EOPNOTSUPP;
> +
> vi->datalayout = __inode_data_mapping(le16_to_cpu(v1->i_format));
> vi->inode_isize = sizeof(struct erofs_inode_compact);
> vi->xattr_isize = erofs_xattr_ibody_size(v1->i_xattr_icount);
> @@ -88,6 +93,10 @@ int erofs_iget_by_nid(erofs_nid_t nid, struct erofs_vnode *vi)
> return -EIO;
> }
>
> + vi->z_inited = false;
> + if (erofs_inode_is_data_compressed(vi->datalayout))
> + z_erofs_fill_inode(vi);
> +
> return 0;
> }
>
> diff --git a/fuse/read.c b/fuse/read.c
> index 3ce5c4f..cc0781f 100644
> --- a/fuse/read.c
> +++ b/fuse/read.c
> @@ -16,6 +16,7 @@
> #include "namei.h"
> #include "disk_io.h"
> #include "init.h"
> +#include "decompress.h"
>
> size_t erofs_read_data(struct erofs_vnode *vnode, char *buffer,
> size_t size, off_t offset)
> @@ -78,6 +79,73 @@ finished:
>
> }
>
> +size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,
> + size_t size, off_t offset)
> +{
> + int ret;
> + size_t end, count, ofs, sum = size;
> + struct erofs_map_blocks map = {
> + .index = UINT_MAX,
> + };
> + bool partial;
> + unsigned int algorithmformat;
> + char raw[EROFS_BLKSIZ];
> +
> + while (sum) {
> + end = offset + sum;
> + map.m_la = end - 1;
> +
> + ret = z_erofs_map_blocks_iter(vnode, &map);
> + if (ret)
> + return ret;
> +
> + if (!(map.m_flags & EROFS_MAP_MAPPED)) {
> + sum -= map.m_llen;
> + continue;
> + }
> +
> + ret = dev_read(raw, EROFS_BLKSIZ, map.m_pa);
> + if (ret < 0 || (size_t)ret != EROFS_BLKSIZ)
> + return -EIO;
> +
> + algorithmformat = map.m_flags & EROFS_MAP_ZIPPED ?
> + Z_EROFS_COMPRESSION_LZ4 :
> + Z_EROFS_COMPRESSION_SHIFTED;
> +
> + if (end >= map.m_la + map.m_llen) {
> + count = map.m_llen;
> + partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);
> + } else {
> + count = end - map.m_la;
> + partial = true;
> + }
it would be better to make the logic more explicitly, like:
- if (end >= map.m_la + map.m_llen) {
- count = map.m_llen;
- partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);
- } else {
+ /*
+ * trim to the needed size if the returned extent is quite
+ * larger than requested, and set up partial flag as well.
+ */
+ if (end < map.m_la + map.m_llen) {
count = end - map.m_la;
partial = true;
+ } else {
+ ASSERT(end == map.m_la + map_m_llen);
+ count = map.m_llen;
+ partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);
}
> +
> + if ((off_t)map.m_la < offset) {
> + ofs = offset - map.m_la;
> + sum = 0;
> + } else {
> + ofs = 0;
> + sum -= count;
> + }
a bit weird here, anyway, let me clean up here as well.
> +
> + ret = z_erofs_decompress(&(struct z_erofs_decompress_req) {
> + .in = raw,
> + .out = buffer,
> + .ofs_out = sum,
> + .ofs_head = ofs,
> + .inputsize = EROFS_BLKSIZ,
> + .outputsize = count,
> + .alg = algorithmformat,
> + .partial_decoding = partial
> + });
> + if (ret < 0)
> + return ret;
> + }
> +
> + logi("nid:%u size=%zd offset=%llu done",
> + vnode->nid, size, (long long)offset);
> + return size;
> +}
>
> int erofs_read(const char *path, char *buffer, size_t size, off_t offset,
> struct fuse_file_info *fi)
> @@ -107,7 +175,8 @@ int erofs_read(const char *path, char *buffer, size_t size, off_t offset,
>
> case EROFS_INODE_FLAT_COMPRESSION_LEGACY:
> case EROFS_INODE_FLAT_COMPRESSION:
> - /* Fixme: */
> + return erofs_read_data_compression(&v, buffer, size, offset);
> +
> default:
> return -EINVAL;
> }
> diff --git a/fuse/zmap.c b/fuse/zmap.c
> new file mode 100644
> index 0000000..022ca1b
> --- /dev/null
> +++ b/fuse/zmap.c
let's move this file to lib/ as well.
> @@ -0,0 +1,416 @@
> +// SPDX-License-Identifier: GPL-2.0-only
Let's relicense this file to GPL-2.0+ for erofs-utils
(since I'm the original author, and assume you agree on this as well..)
> +/*
> + * Many parts of codes are copied from Linux kernel/fs/erofs.
> + *
> + * Copyright (C) 2018-2019 HUAWEI, Inc.
> + * https://www.huawei.com/
> + * Created by Gao Xiang <gaoxiang25@huawei.com>
> + * Modified by Huang Jianan <huangjianan@oppo.com>
> + */
> +
> +#include "init.h"
> +#include "disk_io.h"
> +#include "logging.h"
> +
> +int z_erofs_fill_inode(struct erofs_vnode *vi)
> +{
> + if (vi->datalayout == EROFS_INODE_FLAT_COMPRESSION_LEGACY) {
> + vi->z_advise = 0;
> + vi->z_algorithmtype[0] = 0;
> + vi->z_algorithmtype[1] = 0;
> + vi->z_logical_clusterbits = LOG_BLOCK_SIZE;
> + vi->z_physical_clusterbits[0] = vi->z_logical_clusterbits;
> + vi->z_physical_clusterbits[1] = vi->z_logical_clusterbits;
> + vi->z_inited = true;
> + }
> +
> + return 0;
> +}
> +
> +static int z_erofs_fill_inode_lazy(struct erofs_vnode *vi)
> +{
> + int ret;
> + erofs_off_t pos;
> + struct z_erofs_map_header *h;
> + char buf[8];
> +
> + if (vi->z_inited)
> + return 0;
> +
> + DBG_BUGON(vi->datalayout == EROFS_INODE_FLAT_COMPRESSION_LEGACY);
> +
> + pos = round_up(nid2addr(vi->nid) + vi->inode_isize + vi->xattr_isize, 8);
> +
> + ret = dev_read(buf, 8, pos);
> + if (ret < 0 && (uint32_t)ret != 8)
> + return -EIO;
> +
> + h = (struct z_erofs_map_header *)buf;
> + vi->z_advise = le16_to_cpu(h->h_advise);
> + vi->z_algorithmtype[0] = h->h_algorithmtype & 15;
> + vi->z_algorithmtype[1] = h->h_algorithmtype >> 4;
> +
> + if (vi->z_algorithmtype[0] >= Z_EROFS_COMPRESSION_MAX) {
> + loge("unknown compression format %u for nid %llu",
> + vi->z_algorithmtype[0], vi->nid);
> + return -EOPNOTSUPP;
> + }
> +
> + vi->z_logical_clusterbits = LOG_BLOCK_SIZE + (h->h_clusterbits & 7);
> + vi->z_physical_clusterbits[0] = vi->z_logical_clusterbits +
> + ((h->h_clusterbits >> 3) & 3);
> +
> + if (vi->z_physical_clusterbits[0] != LOG_BLOCK_SIZE) {
> + loge("unsupported physical clusterbits %u for nid %llu",
> + vi->z_physical_clusterbits[0], vi->nid);
> + return -EOPNOTSUPP;
> + }
> +
> + vi->z_physical_clusterbits[1] = vi->z_logical_clusterbits +
> + ((h->h_clusterbits >> 5) & 7);
> + vi->z_inited = true;
> +
> + return 0;
> +}
> +
> +struct z_erofs_maprecorder {
> + struct erofs_vnode *vnode;
> + struct erofs_map_blocks *map;
> + void *kaddr;
> +
> + unsigned long lcn;
> + /* compression extent information gathered */
> + u8 type;
> + u16 clusterofs;
> + u16 delta[2];
> + erofs_blk_t pblk;
> +};
> +
> +static int z_erofs_reload_indexes(struct z_erofs_maprecorder *m,
> + erofs_blk_t eblk)
> +{
> + int ret;
> + struct erofs_map_blocks *const map = m->map;
> + char *mpage = map->mpage;
> +
> + if (map->index == eblk)
> + return 0;
> +
> + ret = dev_read(mpage, EROFS_BLKSIZ, blknr_to_addr(eblk));
> + if (ret < 0 && (uint32_t)ret != EROFS_BLKSIZ)
> + return -EIO;
> +
> + map->index = eblk;
> +
> + return 0;
> +}
> +
> +static int legacy_load_cluster_from_disk(struct z_erofs_maprecorder *m,
> + unsigned long lcn)
> +{
> + struct erofs_vnode *const vi = m->vnode;
> + const erofs_off_t ibase = nid2addr(vi->nid);
> + const erofs_off_t pos =
> + Z_EROFS_VLE_LEGACY_INDEX_ALIGN(ibase + vi->inode_isize +
> + vi->xattr_isize) +
> + lcn * sizeof(struct z_erofs_vle_decompressed_index);
> + struct z_erofs_vle_decompressed_index *di;
> + unsigned int advise, type;
> + int err;
> +
> + err = z_erofs_reload_indexes(m, erofs_blknr(pos));
> + if (err)
> + return err;
> +
> + m->lcn = lcn;
> + di = m->kaddr + erofs_blkoff(pos);
> +
> + advise = le16_to_cpu(di->di_advise);
> + type = (advise >> Z_EROFS_VLE_DI_CLUSTER_TYPE_BIT) &
> + ((1 << Z_EROFS_VLE_DI_CLUSTER_TYPE_BITS) - 1);
> + switch (type) {
> + case Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD:
> + m->clusterofs = 1 << vi->z_logical_clusterbits;
> + m->delta[0] = le16_to_cpu(di->di_u.delta[0]);
> + m->delta[1] = le16_to_cpu(di->di_u.delta[1]);
> + break;
> + case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
> + case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:
> + m->clusterofs = le16_to_cpu(di->di_clusterofs);
> + m->pblk = le32_to_cpu(di->di_u.blkaddr);
> + break;
> + default:
> + DBG_BUGON(1);
> + return -EOPNOTSUPP;
> + }
> + m->type = type;
> + return 0;
> +}
> +
> +static unsigned int decode_compactedbits(unsigned int lobits,
> + unsigned int lomask,
> + u8 *in, unsigned int pos, u8 *type)
> +{
> + const unsigned int v = get_unaligned_le32(in + pos / 8) >> (pos & 7);
> + const unsigned int lo = v & lomask;
> +
> + *type = (v >> lobits) & 3;
> + return lo;
> +}
> +
> +static int unpack_compacted_index(struct z_erofs_maprecorder *m,
> + unsigned int amortizedshift,
> + unsigned int eofs)
> +{
> + struct erofs_vnode *const vi = m->vnode;
> + const unsigned int lclusterbits = vi->z_logical_clusterbits;
> + const unsigned int lomask = (1 << lclusterbits) - 1;
> + unsigned int vcnt, base, lo, encodebits, nblk;
> + int i;
> + u8 *in, type;
> +
> + if (1 << amortizedshift == 4)
> + vcnt = 2;
> + else if (1 << amortizedshift == 2 && lclusterbits == 12)
> + vcnt = 16;
> + else
> + return -EOPNOTSUPP;
> +
> + encodebits = ((vcnt << amortizedshift) - sizeof(__le32)) * 8 / vcnt;
> + base = round_down(eofs, vcnt << amortizedshift);
> + in = m->kaddr + base;
> +
> + i = (eofs - base) >> amortizedshift;
> +
> + lo = decode_compactedbits(lclusterbits, lomask,
> + in, encodebits * i, &type);
> + m->type = type;
> + if (type == Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD) {
> + m->clusterofs = 1 << lclusterbits;
> + if (i + 1 != (int)vcnt) {
> + m->delta[0] = lo;
> + return 0;
> + }
> + /*
> + * since the last lcluster in the pack is special,
> + * of which lo saves delta[1] rather than delta[0].
> + * Hence, get delta[0] by the previous lcluster indirectly.
> + */
> + lo = decode_compactedbits(lclusterbits, lomask,
> + in, encodebits * (i - 1), &type);
> + if (type != Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD)
> + lo = 0;
> + m->delta[0] = lo + 1;
> + return 0;
> + }
> + m->clusterofs = lo;
> + m->delta[0] = 0;
> + /* figout out blkaddr (pblk) for HEAD lclusters */
> + nblk = 1;
> + while (i > 0) {
> + --i;
> + lo = decode_compactedbits(lclusterbits, lomask,
> + in, encodebits * i, &type);
> + if (type == Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD)
> + i -= lo;
> +
> + if (i >= 0)
> + ++nblk;
> + }
> + in += (vcnt << amortizedshift) - sizeof(__le32);
> + m->pblk = le32_to_cpu(*(__le32 *)in) + nblk;
> + return 0;
> +}
> +
> +static int compacted_load_cluster_from_disk(struct z_erofs_maprecorder *m,
> + unsigned long lcn)
> +{
> + struct erofs_vnode *const vi = m->vnode;
> + const unsigned int lclusterbits = vi->z_logical_clusterbits;
> + const erofs_off_t ebase = round_up(nid2addr(vi->nid) + vi->inode_isize +
> + vi->xattr_isize, 8) +
> + sizeof(struct z_erofs_map_header);
> + const unsigned int totalidx = DIV_ROUND_UP(vi->i_size, EROFS_BLKSIZ);
> + unsigned int compacted_4b_initial, compacted_2b;
> + unsigned int amortizedshift;
> + erofs_off_t pos;
> + int err;
> +
> + if (lclusterbits != 12)
> + return -EOPNOTSUPP;
> +
> + if (lcn >= totalidx)
> + return -EINVAL;
> +
> + m->lcn = lcn;
> + /* used to align to 32-byte (compacted_2b) alignment */
> + compacted_4b_initial = (32 - ebase % 32) / 4;
> + if (compacted_4b_initial == 32 / 4)
> + compacted_4b_initial = 0;
> +
> + if (vi->z_advise & Z_EROFS_ADVISE_COMPACTED_2B)
> + compacted_2b = rounddown(totalidx - compacted_4b_initial, 16);
> + else
> + compacted_2b = 0;
> +
> + pos = ebase;
> + if (lcn < compacted_4b_initial) {
> + amortizedshift = 2;
> + goto out;
> + }
> + pos += compacted_4b_initial * 4;
> + lcn -= compacted_4b_initial;
> +
> + if (lcn < compacted_2b) {
> + amortizedshift = 1;
> + goto out;
> + }
> + pos += compacted_2b * 2;
> + lcn -= compacted_2b;
> + amortizedshift = 2;
> +out:
> + pos += lcn * (1 << amortizedshift);
> + err = z_erofs_reload_indexes(m, erofs_blknr(pos));
> + if (err)
> + return err;
> + return unpack_compacted_index(m, amortizedshift, erofs_blkoff(pos));
> +}
> +
> +static int z_erofs_load_cluster_from_disk(struct z_erofs_maprecorder *m,
> + unsigned int lcn)
> +{
> + const unsigned int datamode = m->vnode->datalayout;
> +
> + if (datamode == EROFS_INODE_FLAT_COMPRESSION_LEGACY)
> + return legacy_load_cluster_from_disk(m, lcn);
> +
> + if (datamode == EROFS_INODE_FLAT_COMPRESSION)
> + return compacted_load_cluster_from_disk(m, lcn);
> +
> + return -EINVAL;
> +}
> +
> +static int z_erofs_extent_lookback(struct z_erofs_maprecorder *m,
> + unsigned int lookback_distance)
> +{
> + struct erofs_vnode *const vi = m->vnode;
> + struct erofs_map_blocks *const map = m->map;
> + const unsigned int lclusterbits = vi->z_logical_clusterbits;
> + unsigned long lcn = m->lcn;
> + int err;
> +
> + if (lcn < lookback_distance) {
> + loge("bogus lookback distance @ nid %llu", vi->nid);
> + DBG_BUGON(1);
> + return -EFSCORRUPTED;
> + }
> +
> + /* load extent head logical cluster if needed */
> + lcn -= lookback_distance;
> + err = z_erofs_load_cluster_from_disk(m, lcn);
> + if (err)
> + return err;
> +
> + switch (m->type) {
> + case Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD:
> + if (!m->delta[0]) {
> + loge("invalid lookback distance 0 @ nid %llu",
> + vi->nid);
> + DBG_BUGON(1);
> + return -EFSCORRUPTED;
> + }
> + return z_erofs_extent_lookback(m, m->delta[0]);
> + case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
> + map->m_flags &= ~EROFS_MAP_ZIPPED;
> + case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:
> + map->m_la = (lcn << lclusterbits) | m->clusterofs;
> + break;
> + default:
> + loge("unknown type %u @ lcn %lu of nid %llu",
> + m->type, lcn, vi->nid);
> + DBG_BUGON(1);
> + return -EOPNOTSUPP;
> + }
> + return 0;
> +}
> +
> +int z_erofs_map_blocks_iter(struct erofs_vnode *vi,
> + struct erofs_map_blocks *map)
> +{
> + struct z_erofs_maprecorder m = {
> + .vnode = vi,
> + .map = map,
> + .kaddr = map->mpage,
> + };
> + int err = 0;
> + unsigned int lclusterbits, endoff;
> + unsigned long long ofs, end;
> +
> + /* when trying to read beyond EOF, leave it unmapped */
> + if (map->m_la >= vi->i_size) {
> + map->m_llen = map->m_la + 1 - vi->i_size;
> + map->m_la = vi->i_size;
> + map->m_flags = 0;
> + goto out;
> + }
> +
> + err = z_erofs_fill_inode_lazy(vi);
> + if (err)
> + goto out;
> +
> + lclusterbits = vi->z_logical_clusterbits;
> + ofs = map->m_la;
> + m.lcn = ofs >> lclusterbits;
> + endoff = ofs & ((1 << lclusterbits) - 1);
> +
> + err = z_erofs_load_cluster_from_disk(&m, m.lcn);
> + if (err)
> + goto out;
> +
> + map->m_flags = EROFS_MAP_ZIPPED; /* by default, compressed */
> + end = (m.lcn + 1ULL) << lclusterbits;
> + switch (m.type) {
> + case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
> + if (endoff >= m.clusterofs)
> + map->m_flags &= ~EROFS_MAP_ZIPPED;
> + case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:
> + if (endoff >= m.clusterofs) {
> + map->m_la = (m.lcn << lclusterbits) | m.clusterofs;
> + break;
> + }
> + /* m.lcn should be >= 1 if endoff < m.clusterofs */
> + if (!m.lcn) {
> + loge("invalid logical cluster 0 at nid %llu",
> + vi->nid);
> + err = -EFSCORRUPTED;
> + goto out;
> + }
> + end = (m.lcn << lclusterbits) | m.clusterofs;
> + map->m_flags |= EROFS_MAP_FULL_MAPPED;
> + m.delta[0] = 1;
> + case Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD:
> + /* get the correspoinding first chunk */
> + err = z_erofs_extent_lookback(&m, m.delta[0]);
> + if (err)
> + goto out;
> + break;
> + default:
> + loge("unknown type %u @ offset %llu of nid %llu",
> + m.type, ofs, vi->nid);
> + err = -EOPNOTSUPP;
> + goto out;
> + }
> +
> + map->m_llen = end - map->m_la;
> + map->m_plen = 1 << lclusterbits;
> + map->m_pa = blknr_to_addr(m.pblk);
> + map->m_flags |= EROFS_MAP_MAPPED;
> +
> +out:
> + logd("m_la %llu m_pa %llu m_llen %llu m_plen %llu m_flags 0%o",
> + map->m_la, map->m_pa,
> + map->m_llen, map->m_plen, map->m_flags);
> +
> + DBG_BUGON(err < 0 && err != -ENOMEM);
> + return err;
> +}
> diff --git a/include/erofs/defs.h b/include/erofs/defs.h
> index a9c769e..06d29a9 100644
> --- a/include/erofs/defs.h
> +++ b/include/erofs/defs.h
> @@ -173,5 +173,18 @@ typedef int64_t s64;
> #define __maybe_unused __attribute__((__unused__))
> #endif
>
> +struct __una_u32 { u32 x; } __packed;
> +
> +static inline u32 __get_unaligned_cpu32(const void *p)
> +{
> + const struct __una_u32 *ptr = (const struct __una_u32 *)p;
> + return ptr->x;
> +}
> +
> +static inline u32 get_unaligned_le32(const void *p)
> +{
> + return __get_unaligned_cpu32((const u8 *)p);
> +}
need to handle big-endian, anyway, minor for now.
Thanks,
Gao Xiang
[-- Attachment #2: Type: text/html, Size: 43219 bytes --]
next prev parent reply other threads:[~2020-10-17 6:24 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-15 13:39 [PATCH 1/5] erofs-utils: fix the conflict with the master branch Huang Jianan
2020-10-15 13:39 ` [PATCH 2/5] erofs-utils: fix the wrong name length of tail file in the directory Huang Jianan
2020-10-15 13:39 ` [PATCH 3/5] erofs-utils: fix the wrong address of inline dir content Huang Jianan
2020-10-16 15:36 ` Gao Xiang via Linux-erofs
2020-10-15 13:39 ` [PATCH 4/5] erofs-utils: support read special file Huang Jianan
2020-10-15 13:39 ` [PATCH 5/5] erofs-utils: support read compressed file Huang Jianan
2020-10-16 16:17 ` Gao Xiang via Linux-erofs
2020-10-17 6:24 ` jnhuang95 [this message]
2020-10-17 6:36 ` Gao Xiang via Linux-erofs
2020-10-16 15:41 ` [PATCH 1/5] erofs-utils: fix the conflict with the master branch Gao Xiang via Linux-erofs
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2020101714241160424912@gmail.com \
--to=jnhuang95@gmail.com \
--cc=guoweichao@oppo.com \
--cc=hsiangkao@aol.com \
--cc=linux-erofs@lists.ozlabs.org \
--cc=zhangshiming@oppo.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.