* [PATCH 1/2] erofs: introduce chunk-based file on-disk format
@ 2021-08-18 7:07 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-18 7:07 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi, Gao Xiang
Currently, uncompressed data except for tail-packing inline is
consecutive on disk.
In order to support chunk-based data deduplication, add a new
corresponding inode data layout.
In the future, the data source of chunks can be either (un)compressed.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
Documentation/filesystems/erofs.rst | 16 ++++++++++--
fs/erofs/erofs_fs.h | 40 +++++++++++++++++++++++++++--
2 files changed, 52 insertions(+), 4 deletions(-)
diff --git a/Documentation/filesystems/erofs.rst b/Documentation/filesystems/erofs.rst
index 868e3972227f..b46d0fc46eb6 100644
--- a/Documentation/filesystems/erofs.rst
+++ b/Documentation/filesystems/erofs.rst
@@ -156,13 +156,14 @@ may not. All metadatas can be now observed in two different spaces (views):
Xattrs, extents, data inline are followed by the corresponding inode with
proper alignment, and they could be optional for different data mappings.
- _currently_ total 4 valid data mappings are supported:
+ _currently_ total 5 data layouts are supported:
== ====================================================================
0 flat file data without data inline (no extent);
1 fixed-sized output data compression (with non-compacted indexes);
2 flat file data with tail packing data inline (no extent);
- 3 fixed-sized output data compression (with compacted indexes, v5.3+).
+ 3 fixed-sized output data compression (with compacted indexes, v5.3+);
+ 4 chunk-based file (v5.15+).
== ====================================================================
The size of the optional xattrs is indicated by i_xattr_count in inode
@@ -213,6 +214,17 @@ Note that apart from the offset of the first filename, nameoff0 also indicates
the total number of directory entries in this block since it is no need to
introduce another on-disk field at all.
+Chunk-based file
+----------------
+In order to support chunk-based file deduplication, a new inode data layout has
+been supported since Linux v5.15: Files are split in equal-sized data chunks
+with ``extents`` area of the inode metadata indicating how to get the chunk
+data: these can be simply as a 4-byte block address array or in the 8-byte
+chunk index form (see struct erofs_inode_chunk_index in erofs_fs.h for more
+details.)
+
+By the way, chunk-based files are all uncompressed for now.
+
Data compression
----------------
EROFS implements LZ4 fixed-sized output compression which generates fixed-sized
diff --git a/fs/erofs/erofs_fs.h b/fs/erofs/erofs_fs.h
index 0f8da74570b4..6210fe434930 100644
--- a/fs/erofs/erofs_fs.h
+++ b/fs/erofs/erofs_fs.h
@@ -4,6 +4,7 @@
*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_FS_H
#define __EROFS_FS_H
@@ -19,10 +20,12 @@
#define EROFS_FEATURE_INCOMPAT_LZ4_0PADDING 0x00000001
#define EROFS_FEATURE_INCOMPAT_COMPR_CFGS 0x00000002
#define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x00000002
+#define EROFS_FEATURE_INCOMPAT_CHUNKED_FILE 0x00000004
#define EROFS_ALL_FEATURE_INCOMPAT \
(EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \
EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
- EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)
+ EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER | \
+ EROFS_FEATURE_INCOMPAT_CHUNKED_FILE)
#define EROFS_SB_EXTSLOT_SIZE 16
@@ -64,13 +67,16 @@ struct erofs_super_block {
* inode, [xattrs], last_inline_data, ... | ... | no-holed data
* 3 - inode compression D:
* inode, [xattrs], map_header, extents ... | ...
- * 4~7 - reserved
+ * 4 - inode chunk-based E:
+ * inode, [xattrs], chunk indexes ... | ...
+ * 5~7 - reserved
*/
enum {
EROFS_INODE_FLAT_PLAIN = 0,
EROFS_INODE_FLAT_COMPRESSION_LEGACY = 1,
EROFS_INODE_FLAT_INLINE = 2,
EROFS_INODE_FLAT_COMPRESSION = 3,
+ EROFS_INODE_CHUNK_BASED = 4,
EROFS_INODE_DATALAYOUT_MAX
};
@@ -90,6 +96,19 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
#define EROFS_I_ALL \
((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
+/* indicate chunk blkbits, thus `chunksize = blocksize << chunk blkbits' */
+#define EROFS_CHUNK_FORMAT_BLKBITS_MASK 0x001F
+/* with chunk indexes or just a 4-byte blkaddr array */
+#define EROFS_CHUNK_FORMAT_INDEXES 0x0020
+
+#define EROFS_CHUNK_FORMAT_ALL \
+ (EROFS_CHUNK_FORMAT_BLKBITS_MASK | EROFS_CHUNK_FORMAT_INDEXES)
+
+struct erofs_inode_chunk_info {
+ __le16 format; /* chunk blkbits */
+ __le16 reserved;
+};
+
/* 32-byte reduced form of an ondisk inode */
struct erofs_inode_compact {
__le16 i_format; /* inode format hints */
@@ -107,6 +126,9 @@ struct erofs_inode_compact {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
__le32 i_ino; /* only used for 32-bit stat compatibility */
__le16 i_uid;
@@ -135,6 +157,9 @@ struct erofs_inode_extended {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
/* only used for 32-bit stat compatibility */
@@ -204,6 +229,15 @@ static inline unsigned int erofs_xattr_entry_size(struct erofs_xattr_entry *e)
e->e_name_len + le16_to_cpu(e->e_value_size));
}
+/* represent a zeroed chunk (hole) */
+#define EROFS_NULL_ADDR -1
+
+struct erofs_inode_chunk_index {
+ __le32 blkaddr;
+ __le16 device_id; /* back-end storage id, always 0 for now */
+ __le16 reserved; /* reserved, don't care */
+};
+
/* maximum supported size of a physical compression cluster */
#define Z_EROFS_PCLUSTER_MAX_SIZE (1024 * 1024)
@@ -338,6 +372,8 @@ static inline void erofs_check_ondisk_layout_definitions(void)
BUILD_BUG_ON(sizeof(struct erofs_inode_extended) != 64);
BUILD_BUG_ON(sizeof(struct erofs_xattr_ibody_header) != 12);
BUILD_BUG_ON(sizeof(struct erofs_xattr_entry) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_info) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_map_header) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_vle_decompressed_index) != 8);
BUILD_BUG_ON(sizeof(struct erofs_dirent) != 12);
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH 1/2] erofs: introduce chunk-based file on-disk format
@ 2021-08-18 7:07 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-18 7:07 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Joseph Qi, Eryu Guan, Liu Jiang
Currently, uncompressed data except for tail-packing inline is
consecutive on disk.
In order to support chunk-based data deduplication, add a new
corresponding inode data layout.
In the future, the data source of chunks can be either (un)compressed.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
Documentation/filesystems/erofs.rst | 16 ++++++++++--
fs/erofs/erofs_fs.h | 40 +++++++++++++++++++++++++++--
2 files changed, 52 insertions(+), 4 deletions(-)
diff --git a/Documentation/filesystems/erofs.rst b/Documentation/filesystems/erofs.rst
index 868e3972227f..b46d0fc46eb6 100644
--- a/Documentation/filesystems/erofs.rst
+++ b/Documentation/filesystems/erofs.rst
@@ -156,13 +156,14 @@ may not. All metadatas can be now observed in two different spaces (views):
Xattrs, extents, data inline are followed by the corresponding inode with
proper alignment, and they could be optional for different data mappings.
- _currently_ total 4 valid data mappings are supported:
+ _currently_ total 5 data layouts are supported:
== ====================================================================
0 flat file data without data inline (no extent);
1 fixed-sized output data compression (with non-compacted indexes);
2 flat file data with tail packing data inline (no extent);
- 3 fixed-sized output data compression (with compacted indexes, v5.3+).
+ 3 fixed-sized output data compression (with compacted indexes, v5.3+);
+ 4 chunk-based file (v5.15+).
== ====================================================================
The size of the optional xattrs is indicated by i_xattr_count in inode
@@ -213,6 +214,17 @@ Note that apart from the offset of the first filename, nameoff0 also indicates
the total number of directory entries in this block since it is no need to
introduce another on-disk field at all.
+Chunk-based file
+----------------
+In order to support chunk-based file deduplication, a new inode data layout has
+been supported since Linux v5.15: Files are split in equal-sized data chunks
+with ``extents`` area of the inode metadata indicating how to get the chunk
+data: these can be simply as a 4-byte block address array or in the 8-byte
+chunk index form (see struct erofs_inode_chunk_index in erofs_fs.h for more
+details.)
+
+By the way, chunk-based files are all uncompressed for now.
+
Data compression
----------------
EROFS implements LZ4 fixed-sized output compression which generates fixed-sized
diff --git a/fs/erofs/erofs_fs.h b/fs/erofs/erofs_fs.h
index 0f8da74570b4..6210fe434930 100644
--- a/fs/erofs/erofs_fs.h
+++ b/fs/erofs/erofs_fs.h
@@ -4,6 +4,7 @@
*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_FS_H
#define __EROFS_FS_H
@@ -19,10 +20,12 @@
#define EROFS_FEATURE_INCOMPAT_LZ4_0PADDING 0x00000001
#define EROFS_FEATURE_INCOMPAT_COMPR_CFGS 0x00000002
#define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x00000002
+#define EROFS_FEATURE_INCOMPAT_CHUNKED_FILE 0x00000004
#define EROFS_ALL_FEATURE_INCOMPAT \
(EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \
EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
- EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)
+ EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER | \
+ EROFS_FEATURE_INCOMPAT_CHUNKED_FILE)
#define EROFS_SB_EXTSLOT_SIZE 16
@@ -64,13 +67,16 @@ struct erofs_super_block {
* inode, [xattrs], last_inline_data, ... | ... | no-holed data
* 3 - inode compression D:
* inode, [xattrs], map_header, extents ... | ...
- * 4~7 - reserved
+ * 4 - inode chunk-based E:
+ * inode, [xattrs], chunk indexes ... | ...
+ * 5~7 - reserved
*/
enum {
EROFS_INODE_FLAT_PLAIN = 0,
EROFS_INODE_FLAT_COMPRESSION_LEGACY = 1,
EROFS_INODE_FLAT_INLINE = 2,
EROFS_INODE_FLAT_COMPRESSION = 3,
+ EROFS_INODE_CHUNK_BASED = 4,
EROFS_INODE_DATALAYOUT_MAX
};
@@ -90,6 +96,19 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
#define EROFS_I_ALL \
((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
+/* indicate chunk blkbits, thus `chunksize = blocksize << chunk blkbits' */
+#define EROFS_CHUNK_FORMAT_BLKBITS_MASK 0x001F
+/* with chunk indexes or just a 4-byte blkaddr array */
+#define EROFS_CHUNK_FORMAT_INDEXES 0x0020
+
+#define EROFS_CHUNK_FORMAT_ALL \
+ (EROFS_CHUNK_FORMAT_BLKBITS_MASK | EROFS_CHUNK_FORMAT_INDEXES)
+
+struct erofs_inode_chunk_info {
+ __le16 format; /* chunk blkbits */
+ __le16 reserved;
+};
+
/* 32-byte reduced form of an ondisk inode */
struct erofs_inode_compact {
__le16 i_format; /* inode format hints */
@@ -107,6 +126,9 @@ struct erofs_inode_compact {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
__le32 i_ino; /* only used for 32-bit stat compatibility */
__le16 i_uid;
@@ -135,6 +157,9 @@ struct erofs_inode_extended {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
/* only used for 32-bit stat compatibility */
@@ -204,6 +229,15 @@ static inline unsigned int erofs_xattr_entry_size(struct erofs_xattr_entry *e)
e->e_name_len + le16_to_cpu(e->e_value_size));
}
+/* represent a zeroed chunk (hole) */
+#define EROFS_NULL_ADDR -1
+
+struct erofs_inode_chunk_index {
+ __le32 blkaddr;
+ __le16 device_id; /* back-end storage id, always 0 for now */
+ __le16 reserved; /* reserved, don't care */
+};
+
/* maximum supported size of a physical compression cluster */
#define Z_EROFS_PCLUSTER_MAX_SIZE (1024 * 1024)
@@ -338,6 +372,8 @@ static inline void erofs_check_ondisk_layout_definitions(void)
BUILD_BUG_ON(sizeof(struct erofs_inode_extended) != 64);
BUILD_BUG_ON(sizeof(struct erofs_xattr_ibody_header) != 12);
BUILD_BUG_ON(sizeof(struct erofs_xattr_entry) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_info) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_map_header) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_vle_decompressed_index) != 8);
BUILD_BUG_ON(sizeof(struct erofs_dirent) != 12);
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH 2/2] erofs: support reading chunk-based uncompressed files
2021-08-18 7:07 ` Gao Xiang
@ 2021-08-18 7:07 ` Gao Xiang
-1 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-18 7:07 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi, Gao Xiang
Add runtime support for chunk-based uncompressed files
described in the previous patch.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
fs/erofs/inode.c | 18 ++++++++-
fs/erofs/internal.h | 5 +++
3 files changed, 102 insertions(+), 11 deletions(-)
diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index b2a22aabc9bc..78d625709481 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "internal.h"
#include <linux/prefetch.h>
@@ -37,13 +38,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
lastblk = nblocks - tailendpacking;
- if (offset >= inode->i_size) {
- /* leave out-of-bound access unmapped */
- map->m_flags = 0;
- map->m_plen = 0;
- goto out;
- }
-
/* there is no hole in flatmode */
map->m_flags = EROFS_MAP_MAPPED;
@@ -78,14 +72,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
goto err_out;
}
-out:
map->m_llen = map->m_plen;
-
err_out:
trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
return err;
}
+static int erofs_map_blocks(struct inode *inode,
+ struct erofs_map_blocks *map, int flags)
+{
+ struct super_block *sb = inode->i_sb;
+ struct erofs_inode *vi = EROFS_I(inode);
+ struct erofs_inode_chunk_index *idx;
+ struct page *page;
+ u64 chunknr;
+ unsigned int unit;
+ erofs_off_t pos;
+ int err = 0;
+
+ if (map->m_la >= inode->i_size) {
+ /* leave out-of-bound access unmapped */
+ map->m_flags = 0;
+ map->m_plen = 0;
+ goto out;
+ }
+
+ if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
+ return erofs_map_blocks_flatmode(inode, map, flags);
+
+ if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
+ unit = sizeof(*idx); /* chunk index */
+ else
+ unit = 4; /* block map */
+
+ chunknr = map->m_la >> vi->chunkbits;
+ pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
+ vi->xattr_isize, unit) + unit * chunknr;
+
+ page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
+ if (IS_ERR(page))
+ return PTR_ERR(page);
+
+ map->m_la = chunknr << vi->chunkbits;
+ map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
+ roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
+
+ /* handle block map */
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
+ __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
+
+ if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
+ map->m_flags = 0;
+ } else {
+ map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ }
+ goto out_unlock;
+ }
+ /* parse chunk indexes */
+ idx = page_address(page) + erofs_blkoff(pos);
+ switch (le32_to_cpu(idx->blkaddr)) {
+ case EROFS_NULL_ADDR:
+ map->m_flags = 0;
+ break;
+ default:
+ /* only one device is supported for now */
+ if (idx->device_id) {
+ erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
+ le32_to_cpu(idx->device_id),
+ chunknr, vi->nid);
+ err = -EFSCORRUPTED;
+ goto out_unlock;
+ }
+ map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ break;
+ }
+out_unlock:
+ unlock_page(page);
+ put_page(page);
+out:
+ map->m_llen = map->m_plen;
+ return err;
+}
+
static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
{
@@ -95,7 +165,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
map.m_la = offset;
map.m_llen = length;
- ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
+ ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
if (ret < 0)
return ret;
diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
index 92728da1d206..036a6cc97d10 100644
--- a/fs/erofs/inode.c
+++ b/fs/erofs/inode.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "xattr.h"
@@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
/* total blocks for compressed files */
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(die->i_u.compressed_blocks);
-
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ /* fill chunked inode summary info */
+ vi->chunkformat = __le16_to_cpu(die->i_u.c.format);
kfree(copied);
break;
case EROFS_INODE_LAYOUT_COMPACT:
@@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
inode->i_size = le32_to_cpu(dic->i_size);
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(dic->i_u.compressed_blocks);
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ vi->chunkformat = __le16_to_cpu(dic->i_u.c.format);
break;
default:
erofs_err(inode->i_sb,
@@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
goto err_out;
}
+ if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
+ erofs_err(inode->i_sb,
+ "unsupported chunk format %x of nid %llu",
+ vi->chunkformat, vi->nid);
+ err = -EOPNOTSUPP;
+ goto err_out;
+ }
+ vi->chunkbits = LOG_BLOCK_SIZE +
+ (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
+ }
inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
index 25b094085ca6..0a46e149aadd 100644
--- a/fs/erofs/internal.h
+++ b/fs/erofs/internal.h
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_INTERNAL_H
#define __EROFS_INTERNAL_H
@@ -260,6 +261,10 @@ struct erofs_inode {
union {
erofs_blk_t raw_blkaddr;
+ struct {
+ unsigned short chunkformat;
+ unsigned char chunkbits;
+ };
#ifdef CONFIG_EROFS_FS_ZIP
struct {
unsigned short z_advise;
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-18 7:07 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-18 7:07 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Joseph Qi, Eryu Guan, Liu Jiang
Add runtime support for chunk-based uncompressed files
described in the previous patch.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
fs/erofs/inode.c | 18 ++++++++-
fs/erofs/internal.h | 5 +++
3 files changed, 102 insertions(+), 11 deletions(-)
diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index b2a22aabc9bc..78d625709481 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "internal.h"
#include <linux/prefetch.h>
@@ -37,13 +38,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
lastblk = nblocks - tailendpacking;
- if (offset >= inode->i_size) {
- /* leave out-of-bound access unmapped */
- map->m_flags = 0;
- map->m_plen = 0;
- goto out;
- }
-
/* there is no hole in flatmode */
map->m_flags = EROFS_MAP_MAPPED;
@@ -78,14 +72,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
goto err_out;
}
-out:
map->m_llen = map->m_plen;
-
err_out:
trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
return err;
}
+static int erofs_map_blocks(struct inode *inode,
+ struct erofs_map_blocks *map, int flags)
+{
+ struct super_block *sb = inode->i_sb;
+ struct erofs_inode *vi = EROFS_I(inode);
+ struct erofs_inode_chunk_index *idx;
+ struct page *page;
+ u64 chunknr;
+ unsigned int unit;
+ erofs_off_t pos;
+ int err = 0;
+
+ if (map->m_la >= inode->i_size) {
+ /* leave out-of-bound access unmapped */
+ map->m_flags = 0;
+ map->m_plen = 0;
+ goto out;
+ }
+
+ if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
+ return erofs_map_blocks_flatmode(inode, map, flags);
+
+ if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
+ unit = sizeof(*idx); /* chunk index */
+ else
+ unit = 4; /* block map */
+
+ chunknr = map->m_la >> vi->chunkbits;
+ pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
+ vi->xattr_isize, unit) + unit * chunknr;
+
+ page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
+ if (IS_ERR(page))
+ return PTR_ERR(page);
+
+ map->m_la = chunknr << vi->chunkbits;
+ map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
+ roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
+
+ /* handle block map */
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
+ __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
+
+ if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
+ map->m_flags = 0;
+ } else {
+ map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ }
+ goto out_unlock;
+ }
+ /* parse chunk indexes */
+ idx = page_address(page) + erofs_blkoff(pos);
+ switch (le32_to_cpu(idx->blkaddr)) {
+ case EROFS_NULL_ADDR:
+ map->m_flags = 0;
+ break;
+ default:
+ /* only one device is supported for now */
+ if (idx->device_id) {
+ erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
+ le32_to_cpu(idx->device_id),
+ chunknr, vi->nid);
+ err = -EFSCORRUPTED;
+ goto out_unlock;
+ }
+ map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ break;
+ }
+out_unlock:
+ unlock_page(page);
+ put_page(page);
+out:
+ map->m_llen = map->m_plen;
+ return err;
+}
+
static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
{
@@ -95,7 +165,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
map.m_la = offset;
map.m_llen = length;
- ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
+ ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
if (ret < 0)
return ret;
diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
index 92728da1d206..036a6cc97d10 100644
--- a/fs/erofs/inode.c
+++ b/fs/erofs/inode.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "xattr.h"
@@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
/* total blocks for compressed files */
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(die->i_u.compressed_blocks);
-
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ /* fill chunked inode summary info */
+ vi->chunkformat = __le16_to_cpu(die->i_u.c.format);
kfree(copied);
break;
case EROFS_INODE_LAYOUT_COMPACT:
@@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
inode->i_size = le32_to_cpu(dic->i_size);
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(dic->i_u.compressed_blocks);
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ vi->chunkformat = __le16_to_cpu(dic->i_u.c.format);
break;
default:
erofs_err(inode->i_sb,
@@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
goto err_out;
}
+ if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
+ erofs_err(inode->i_sb,
+ "unsupported chunk format %x of nid %llu",
+ vi->chunkformat, vi->nid);
+ err = -EOPNOTSUPP;
+ goto err_out;
+ }
+ vi->chunkbits = LOG_BLOCK_SIZE +
+ (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
+ }
inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
index 25b094085ca6..0a46e149aadd 100644
--- a/fs/erofs/internal.h
+++ b/fs/erofs/internal.h
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_INTERNAL_H
#define __EROFS_INTERNAL_H
@@ -260,6 +261,10 @@ struct erofs_inode {
union {
erofs_blk_t raw_blkaddr;
+ struct {
+ unsigned short chunkformat;
+ unsigned char chunkbits;
+ };
#ifdef CONFIG_EROFS_FS_ZIP
struct {
unsigned short z_advise;
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: [PATCH 1/2] erofs: introduce chunk-based file on-disk format
2021-08-18 7:07 ` Gao Xiang
@ 2021-08-18 22:28 ` Liu Bo
-1 siblings, 0 replies; 37+ messages in thread
From: Liu Bo @ 2021-08-18 22:28 UTC (permalink / raw)
To: Gao Xiang
Cc: linux-erofs, Chao Yu, LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi
On Wed, Aug 18, 2021 at 03:07:12PM +0800, Gao Xiang wrote:
> Currently, uncompressed data except for tail-packing inline is
> consecutive on disk.
>
> In order to support chunk-based data deduplication, add a new
> corresponding inode data layout.
>
> In the future, the data source of chunks can be either (un)compressed.
>
> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
> ---
> Documentation/filesystems/erofs.rst | 16 ++++++++++--
> fs/erofs/erofs_fs.h | 40 +++++++++++++++++++++++++++--
> 2 files changed, 52 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/filesystems/erofs.rst b/Documentation/filesystems/erofs.rst
> index 868e3972227f..b46d0fc46eb6 100644
> --- a/Documentation/filesystems/erofs.rst
> +++ b/Documentation/filesystems/erofs.rst
> @@ -156,13 +156,14 @@ may not. All metadatas can be now observed in two different spaces (views):
>
> Xattrs, extents, data inline are followed by the corresponding inode with
> proper alignment, and they could be optional for different data mappings.
> - _currently_ total 4 valid data mappings are supported:
> + _currently_ total 5 data layouts are supported:
>
> == ====================================================================
> 0 flat file data without data inline (no extent);
> 1 fixed-sized output data compression (with non-compacted indexes);
> 2 flat file data with tail packing data inline (no extent);
> - 3 fixed-sized output data compression (with compacted indexes, v5.3+).
> + 3 fixed-sized output data compression (with compacted indexes, v5.3+);
> + 4 chunk-based file (v5.15+).
> == ====================================================================
>
> The size of the optional xattrs is indicated by i_xattr_count in inode
> @@ -213,6 +214,17 @@ Note that apart from the offset of the first filename, nameoff0 also indicates
> the total number of directory entries in this block since it is no need to
> introduce another on-disk field at all.
>
> +Chunk-based file
> +----------------
> +In order to support chunk-based file deduplication, a new inode data layout has
> +been supported since Linux v5.15: Files are split in equal-sized data chunks
> +with ``extents`` area of the inode metadata indicating how to get the chunk
> +data: these can be simply as a 4-byte block address array or in the 8-byte
> +chunk index form (see struct erofs_inode_chunk_index in erofs_fs.h for more
> +details.)
> +
> +By the way, chunk-based files are all uncompressed for now.
> +
> Data compression
> ----------------
> EROFS implements LZ4 fixed-sized output compression which generates fixed-sized
> diff --git a/fs/erofs/erofs_fs.h b/fs/erofs/erofs_fs.h
> index 0f8da74570b4..6210fe434930 100644
> --- a/fs/erofs/erofs_fs.h
> +++ b/fs/erofs/erofs_fs.h
> @@ -4,6 +4,7 @@
> *
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #ifndef __EROFS_FS_H
> #define __EROFS_FS_H
> @@ -19,10 +20,12 @@
> #define EROFS_FEATURE_INCOMPAT_LZ4_0PADDING 0x00000001
> #define EROFS_FEATURE_INCOMPAT_COMPR_CFGS 0x00000002
> #define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x00000002
> +#define EROFS_FEATURE_INCOMPAT_CHUNKED_FILE 0x00000004
> #define EROFS_ALL_FEATURE_INCOMPAT \
> (EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \
> EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
> - EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)
> + EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER | \
> + EROFS_FEATURE_INCOMPAT_CHUNKED_FILE)
>
> #define EROFS_SB_EXTSLOT_SIZE 16
>
> @@ -64,13 +67,16 @@ struct erofs_super_block {
> * inode, [xattrs], last_inline_data, ... | ... | no-holed data
> * 3 - inode compression D:
> * inode, [xattrs], map_header, extents ... | ...
> - * 4~7 - reserved
> + * 4 - inode chunk-based E:
> + * inode, [xattrs], chunk indexes ... | ...
> + * 5~7 - reserved
> */
> enum {
> EROFS_INODE_FLAT_PLAIN = 0,
> EROFS_INODE_FLAT_COMPRESSION_LEGACY = 1,
> EROFS_INODE_FLAT_INLINE = 2,
> EROFS_INODE_FLAT_COMPRESSION = 3,
> + EROFS_INODE_CHUNK_BASED = 4,
> EROFS_INODE_DATALAYOUT_MAX
> };
>
> @@ -90,6 +96,19 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
> #define EROFS_I_ALL \
> ((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
>
> +/* indicate chunk blkbits, thus `chunksize = blocksize << chunk blkbits' */
A typo in the quotation marks. (`chunksize = ) should be ('chunksize =)
Otherwise it looks good.
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
thanks,
liubo
> +#define EROFS_CHUNK_FORMAT_BLKBITS_MASK 0x001F
> +/* with chunk indexes or just a 4-byte blkaddr array */
> +#define EROFS_CHUNK_FORMAT_INDEXES 0x0020
> +
> +#define EROFS_CHUNK_FORMAT_ALL \
> + (EROFS_CHUNK_FORMAT_BLKBITS_MASK | EROFS_CHUNK_FORMAT_INDEXES)
> +
> +struct erofs_inode_chunk_info {
> + __le16 format; /* chunk blkbits */
> + __le16 reserved;
> +};
> +
> /* 32-byte reduced form of an ondisk inode */
> struct erofs_inode_compact {
> __le16 i_format; /* inode format hints */
> @@ -107,6 +126,9 @@ struct erofs_inode_compact {
>
> /* for device files, used to indicate old/new device # */
> __le32 rdev;
> +
> + /* for chunk-based files, it contains the summary info */
> + struct erofs_inode_chunk_info c;
> } i_u;
> __le32 i_ino; /* only used for 32-bit stat compatibility */
> __le16 i_uid;
> @@ -135,6 +157,9 @@ struct erofs_inode_extended {
>
> /* for device files, used to indicate old/new device # */
> __le32 rdev;
> +
> + /* for chunk-based files, it contains the summary info */
> + struct erofs_inode_chunk_info c;
> } i_u;
>
> /* only used for 32-bit stat compatibility */
> @@ -204,6 +229,15 @@ static inline unsigned int erofs_xattr_entry_size(struct erofs_xattr_entry *e)
> e->e_name_len + le16_to_cpu(e->e_value_size));
> }
>
> +/* represent a zeroed chunk (hole) */
> +#define EROFS_NULL_ADDR -1
> +
> +struct erofs_inode_chunk_index {
> + __le32 blkaddr;
> + __le16 device_id; /* back-end storage id, always 0 for now */
> + __le16 reserved; /* reserved, don't care */
> +};
> +
> /* maximum supported size of a physical compression cluster */
> #define Z_EROFS_PCLUSTER_MAX_SIZE (1024 * 1024)
>
> @@ -338,6 +372,8 @@ static inline void erofs_check_ondisk_layout_definitions(void)
> BUILD_BUG_ON(sizeof(struct erofs_inode_extended) != 64);
> BUILD_BUG_ON(sizeof(struct erofs_xattr_ibody_header) != 12);
> BUILD_BUG_ON(sizeof(struct erofs_xattr_entry) != 4);
> + BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_info) != 4);
> + BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) != 8);
> BUILD_BUG_ON(sizeof(struct z_erofs_map_header) != 8);
> BUILD_BUG_ON(sizeof(struct z_erofs_vle_decompressed_index) != 8);
> BUILD_BUG_ON(sizeof(struct erofs_dirent) != 12);
> --
> 2.24.4
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 1/2] erofs: introduce chunk-based file on-disk format
@ 2021-08-18 22:28 ` Liu Bo
0 siblings, 0 replies; 37+ messages in thread
From: Liu Bo @ 2021-08-18 22:28 UTC (permalink / raw)
To: Gao Xiang; +Cc: LKML, Peng Tao, Joseph Qi, Eryu Guan, Liu Jiang, linux-erofs
On Wed, Aug 18, 2021 at 03:07:12PM +0800, Gao Xiang wrote:
> Currently, uncompressed data except for tail-packing inline is
> consecutive on disk.
>
> In order to support chunk-based data deduplication, add a new
> corresponding inode data layout.
>
> In the future, the data source of chunks can be either (un)compressed.
>
> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
> ---
> Documentation/filesystems/erofs.rst | 16 ++++++++++--
> fs/erofs/erofs_fs.h | 40 +++++++++++++++++++++++++++--
> 2 files changed, 52 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/filesystems/erofs.rst b/Documentation/filesystems/erofs.rst
> index 868e3972227f..b46d0fc46eb6 100644
> --- a/Documentation/filesystems/erofs.rst
> +++ b/Documentation/filesystems/erofs.rst
> @@ -156,13 +156,14 @@ may not. All metadatas can be now observed in two different spaces (views):
>
> Xattrs, extents, data inline are followed by the corresponding inode with
> proper alignment, and they could be optional for different data mappings.
> - _currently_ total 4 valid data mappings are supported:
> + _currently_ total 5 data layouts are supported:
>
> == ====================================================================
> 0 flat file data without data inline (no extent);
> 1 fixed-sized output data compression (with non-compacted indexes);
> 2 flat file data with tail packing data inline (no extent);
> - 3 fixed-sized output data compression (with compacted indexes, v5.3+).
> + 3 fixed-sized output data compression (with compacted indexes, v5.3+);
> + 4 chunk-based file (v5.15+).
> == ====================================================================
>
> The size of the optional xattrs is indicated by i_xattr_count in inode
> @@ -213,6 +214,17 @@ Note that apart from the offset of the first filename, nameoff0 also indicates
> the total number of directory entries in this block since it is no need to
> introduce another on-disk field at all.
>
> +Chunk-based file
> +----------------
> +In order to support chunk-based file deduplication, a new inode data layout has
> +been supported since Linux v5.15: Files are split in equal-sized data chunks
> +with ``extents`` area of the inode metadata indicating how to get the chunk
> +data: these can be simply as a 4-byte block address array or in the 8-byte
> +chunk index form (see struct erofs_inode_chunk_index in erofs_fs.h for more
> +details.)
> +
> +By the way, chunk-based files are all uncompressed for now.
> +
> Data compression
> ----------------
> EROFS implements LZ4 fixed-sized output compression which generates fixed-sized
> diff --git a/fs/erofs/erofs_fs.h b/fs/erofs/erofs_fs.h
> index 0f8da74570b4..6210fe434930 100644
> --- a/fs/erofs/erofs_fs.h
> +++ b/fs/erofs/erofs_fs.h
> @@ -4,6 +4,7 @@
> *
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #ifndef __EROFS_FS_H
> #define __EROFS_FS_H
> @@ -19,10 +20,12 @@
> #define EROFS_FEATURE_INCOMPAT_LZ4_0PADDING 0x00000001
> #define EROFS_FEATURE_INCOMPAT_COMPR_CFGS 0x00000002
> #define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x00000002
> +#define EROFS_FEATURE_INCOMPAT_CHUNKED_FILE 0x00000004
> #define EROFS_ALL_FEATURE_INCOMPAT \
> (EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \
> EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
> - EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)
> + EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER | \
> + EROFS_FEATURE_INCOMPAT_CHUNKED_FILE)
>
> #define EROFS_SB_EXTSLOT_SIZE 16
>
> @@ -64,13 +67,16 @@ struct erofs_super_block {
> * inode, [xattrs], last_inline_data, ... | ... | no-holed data
> * 3 - inode compression D:
> * inode, [xattrs], map_header, extents ... | ...
> - * 4~7 - reserved
> + * 4 - inode chunk-based E:
> + * inode, [xattrs], chunk indexes ... | ...
> + * 5~7 - reserved
> */
> enum {
> EROFS_INODE_FLAT_PLAIN = 0,
> EROFS_INODE_FLAT_COMPRESSION_LEGACY = 1,
> EROFS_INODE_FLAT_INLINE = 2,
> EROFS_INODE_FLAT_COMPRESSION = 3,
> + EROFS_INODE_CHUNK_BASED = 4,
> EROFS_INODE_DATALAYOUT_MAX
> };
>
> @@ -90,6 +96,19 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
> #define EROFS_I_ALL \
> ((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
>
> +/* indicate chunk blkbits, thus `chunksize = blocksize << chunk blkbits' */
A typo in the quotation marks. (`chunksize = ) should be ('chunksize =)
Otherwise it looks good.
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
thanks,
liubo
> +#define EROFS_CHUNK_FORMAT_BLKBITS_MASK 0x001F
> +/* with chunk indexes or just a 4-byte blkaddr array */
> +#define EROFS_CHUNK_FORMAT_INDEXES 0x0020
> +
> +#define EROFS_CHUNK_FORMAT_ALL \
> + (EROFS_CHUNK_FORMAT_BLKBITS_MASK | EROFS_CHUNK_FORMAT_INDEXES)
> +
> +struct erofs_inode_chunk_info {
> + __le16 format; /* chunk blkbits */
> + __le16 reserved;
> +};
> +
> /* 32-byte reduced form of an ondisk inode */
> struct erofs_inode_compact {
> __le16 i_format; /* inode format hints */
> @@ -107,6 +126,9 @@ struct erofs_inode_compact {
>
> /* for device files, used to indicate old/new device # */
> __le32 rdev;
> +
> + /* for chunk-based files, it contains the summary info */
> + struct erofs_inode_chunk_info c;
> } i_u;
> __le32 i_ino; /* only used for 32-bit stat compatibility */
> __le16 i_uid;
> @@ -135,6 +157,9 @@ struct erofs_inode_extended {
>
> /* for device files, used to indicate old/new device # */
> __le32 rdev;
> +
> + /* for chunk-based files, it contains the summary info */
> + struct erofs_inode_chunk_info c;
> } i_u;
>
> /* only used for 32-bit stat compatibility */
> @@ -204,6 +229,15 @@ static inline unsigned int erofs_xattr_entry_size(struct erofs_xattr_entry *e)
> e->e_name_len + le16_to_cpu(e->e_value_size));
> }
>
> +/* represent a zeroed chunk (hole) */
> +#define EROFS_NULL_ADDR -1
> +
> +struct erofs_inode_chunk_index {
> + __le32 blkaddr;
> + __le16 device_id; /* back-end storage id, always 0 for now */
> + __le16 reserved; /* reserved, don't care */
> +};
> +
> /* maximum supported size of a physical compression cluster */
> #define Z_EROFS_PCLUSTER_MAX_SIZE (1024 * 1024)
>
> @@ -338,6 +372,8 @@ static inline void erofs_check_ondisk_layout_definitions(void)
> BUILD_BUG_ON(sizeof(struct erofs_inode_extended) != 64);
> BUILD_BUG_ON(sizeof(struct erofs_xattr_ibody_header) != 12);
> BUILD_BUG_ON(sizeof(struct erofs_xattr_entry) != 4);
> + BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_info) != 4);
> + BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) != 8);
> BUILD_BUG_ON(sizeof(struct z_erofs_map_header) != 8);
> BUILD_BUG_ON(sizeof(struct z_erofs_vle_decompressed_index) != 8);
> BUILD_BUG_ON(sizeof(struct erofs_dirent) != 12);
> --
> 2.24.4
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 2/2] erofs: support reading chunk-based uncompressed files
2021-08-18 7:07 ` Gao Xiang
@ 2021-08-18 22:29 ` Liu Bo
-1 siblings, 0 replies; 37+ messages in thread
From: Liu Bo @ 2021-08-18 22:29 UTC (permalink / raw)
To: Gao Xiang
Cc: linux-erofs, Chao Yu, LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi
On Wed, Aug 18, 2021 at 03:07:13PM +0800, Gao Xiang wrote:
> Add runtime support for chunk-based uncompressed files
> described in the previous patch.
>
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
thanks,
liubo
> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
> ---
> fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
> fs/erofs/inode.c | 18 ++++++++-
> fs/erofs/internal.h | 5 +++
> 3 files changed, 102 insertions(+), 11 deletions(-)
>
> diff --git a/fs/erofs/data.c b/fs/erofs/data.c
> index b2a22aabc9bc..78d625709481 100644
> --- a/fs/erofs/data.c
> +++ b/fs/erofs/data.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "internal.h"
> #include <linux/prefetch.h>
> @@ -37,13 +38,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
> lastblk = nblocks - tailendpacking;
>
> - if (offset >= inode->i_size) {
> - /* leave out-of-bound access unmapped */
> - map->m_flags = 0;
> - map->m_plen = 0;
> - goto out;
> - }
> -
> /* there is no hole in flatmode */
> map->m_flags = EROFS_MAP_MAPPED;
>
> @@ -78,14 +72,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> goto err_out;
> }
>
> -out:
> map->m_llen = map->m_plen;
> -
> err_out:
> trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
> return err;
> }
>
> +static int erofs_map_blocks(struct inode *inode,
> + struct erofs_map_blocks *map, int flags)
> +{
> + struct super_block *sb = inode->i_sb;
> + struct erofs_inode *vi = EROFS_I(inode);
> + struct erofs_inode_chunk_index *idx;
> + struct page *page;
> + u64 chunknr;
> + unsigned int unit;
> + erofs_off_t pos;
> + int err = 0;
> +
> + if (map->m_la >= inode->i_size) {
> + /* leave out-of-bound access unmapped */
> + map->m_flags = 0;
> + map->m_plen = 0;
> + goto out;
> + }
> +
> + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
> + return erofs_map_blocks_flatmode(inode, map, flags);
> +
> + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
> + unit = sizeof(*idx); /* chunk index */
> + else
> + unit = 4; /* block map */
> +
> + chunknr = map->m_la >> vi->chunkbits;
> + pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
> + vi->xattr_isize, unit) + unit * chunknr;
> +
> + page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
> + if (IS_ERR(page))
> + return PTR_ERR(page);
> +
> + map->m_la = chunknr << vi->chunkbits;
> + map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
> + roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
> +
> + /* handle block map */
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
> + __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
> +
> + if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
> + map->m_flags = 0;
> + } else {
> + map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + }
> + goto out_unlock;
> + }
> + /* parse chunk indexes */
> + idx = page_address(page) + erofs_blkoff(pos);
> + switch (le32_to_cpu(idx->blkaddr)) {
> + case EROFS_NULL_ADDR:
> + map->m_flags = 0;
> + break;
> + default:
> + /* only one device is supported for now */
> + if (idx->device_id) {
> + erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
> + le32_to_cpu(idx->device_id),
> + chunknr, vi->nid);
> + err = -EFSCORRUPTED;
> + goto out_unlock;
> + }
> + map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + break;
> + }
> +out_unlock:
> + unlock_page(page);
> + put_page(page);
> +out:
> + map->m_llen = map->m_plen;
> + return err;
> +}
> +
> static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
> {
> @@ -95,7 +165,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> map.m_la = offset;
> map.m_llen = length;
>
> - ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
> + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
> if (ret < 0)
> return ret;
>
> diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
> index 92728da1d206..036a6cc97d10 100644
> --- a/fs/erofs/inode.c
> +++ b/fs/erofs/inode.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "xattr.h"
>
> @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
> /* total blocks for compressed files */
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(die->i_u.compressed_blocks);
> -
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + /* fill chunked inode summary info */
> + vi->chunkformat = __le16_to_cpu(die->i_u.c.format);
> kfree(copied);
> break;
> case EROFS_INODE_LAYOUT_COMPACT:
> @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> inode->i_size = le32_to_cpu(dic->i_size);
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + vi->chunkformat = __le16_to_cpu(dic->i_u.c.format);
> break;
> default:
> erofs_err(inode->i_sb,
> @@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
> goto err_out;
> }
>
> + if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
> + erofs_err(inode->i_sb,
> + "unsupported chunk format %x of nid %llu",
> + vi->chunkformat, vi->nid);
> + err = -EOPNOTSUPP;
> + goto err_out;
> + }
> + vi->chunkbits = LOG_BLOCK_SIZE +
> + (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
> + }
> inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
> index 25b094085ca6..0a46e149aadd 100644
> --- a/fs/erofs/internal.h
> +++ b/fs/erofs/internal.h
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #ifndef __EROFS_INTERNAL_H
> #define __EROFS_INTERNAL_H
> @@ -260,6 +261,10 @@ struct erofs_inode {
>
> union {
> erofs_blk_t raw_blkaddr;
> + struct {
> + unsigned short chunkformat;
> + unsigned char chunkbits;
> + };
> #ifdef CONFIG_EROFS_FS_ZIP
> struct {
> unsigned short z_advise;
> --
> 2.24.4
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-18 22:29 ` Liu Bo
0 siblings, 0 replies; 37+ messages in thread
From: Liu Bo @ 2021-08-18 22:29 UTC (permalink / raw)
To: Gao Xiang; +Cc: LKML, Peng Tao, Joseph Qi, Eryu Guan, Liu Jiang, linux-erofs
On Wed, Aug 18, 2021 at 03:07:13PM +0800, Gao Xiang wrote:
> Add runtime support for chunk-based uncompressed files
> described in the previous patch.
>
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
thanks,
liubo
> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
> ---
> fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
> fs/erofs/inode.c | 18 ++++++++-
> fs/erofs/internal.h | 5 +++
> 3 files changed, 102 insertions(+), 11 deletions(-)
>
> diff --git a/fs/erofs/data.c b/fs/erofs/data.c
> index b2a22aabc9bc..78d625709481 100644
> --- a/fs/erofs/data.c
> +++ b/fs/erofs/data.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "internal.h"
> #include <linux/prefetch.h>
> @@ -37,13 +38,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
> lastblk = nblocks - tailendpacking;
>
> - if (offset >= inode->i_size) {
> - /* leave out-of-bound access unmapped */
> - map->m_flags = 0;
> - map->m_plen = 0;
> - goto out;
> - }
> -
> /* there is no hole in flatmode */
> map->m_flags = EROFS_MAP_MAPPED;
>
> @@ -78,14 +72,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> goto err_out;
> }
>
> -out:
> map->m_llen = map->m_plen;
> -
> err_out:
> trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
> return err;
> }
>
> +static int erofs_map_blocks(struct inode *inode,
> + struct erofs_map_blocks *map, int flags)
> +{
> + struct super_block *sb = inode->i_sb;
> + struct erofs_inode *vi = EROFS_I(inode);
> + struct erofs_inode_chunk_index *idx;
> + struct page *page;
> + u64 chunknr;
> + unsigned int unit;
> + erofs_off_t pos;
> + int err = 0;
> +
> + if (map->m_la >= inode->i_size) {
> + /* leave out-of-bound access unmapped */
> + map->m_flags = 0;
> + map->m_plen = 0;
> + goto out;
> + }
> +
> + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
> + return erofs_map_blocks_flatmode(inode, map, flags);
> +
> + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
> + unit = sizeof(*idx); /* chunk index */
> + else
> + unit = 4; /* block map */
> +
> + chunknr = map->m_la >> vi->chunkbits;
> + pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
> + vi->xattr_isize, unit) + unit * chunknr;
> +
> + page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
> + if (IS_ERR(page))
> + return PTR_ERR(page);
> +
> + map->m_la = chunknr << vi->chunkbits;
> + map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
> + roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
> +
> + /* handle block map */
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
> + __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
> +
> + if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
> + map->m_flags = 0;
> + } else {
> + map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + }
> + goto out_unlock;
> + }
> + /* parse chunk indexes */
> + idx = page_address(page) + erofs_blkoff(pos);
> + switch (le32_to_cpu(idx->blkaddr)) {
> + case EROFS_NULL_ADDR:
> + map->m_flags = 0;
> + break;
> + default:
> + /* only one device is supported for now */
> + if (idx->device_id) {
> + erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
> + le32_to_cpu(idx->device_id),
> + chunknr, vi->nid);
> + err = -EFSCORRUPTED;
> + goto out_unlock;
> + }
> + map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + break;
> + }
> +out_unlock:
> + unlock_page(page);
> + put_page(page);
> +out:
> + map->m_llen = map->m_plen;
> + return err;
> +}
> +
> static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
> {
> @@ -95,7 +165,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> map.m_la = offset;
> map.m_llen = length;
>
> - ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
> + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
> if (ret < 0)
> return ret;
>
> diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
> index 92728da1d206..036a6cc97d10 100644
> --- a/fs/erofs/inode.c
> +++ b/fs/erofs/inode.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "xattr.h"
>
> @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
> /* total blocks for compressed files */
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(die->i_u.compressed_blocks);
> -
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + /* fill chunked inode summary info */
> + vi->chunkformat = __le16_to_cpu(die->i_u.c.format);
> kfree(copied);
> break;
> case EROFS_INODE_LAYOUT_COMPACT:
> @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> inode->i_size = le32_to_cpu(dic->i_size);
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + vi->chunkformat = __le16_to_cpu(dic->i_u.c.format);
> break;
> default:
> erofs_err(inode->i_sb,
> @@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
> goto err_out;
> }
>
> + if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
> + erofs_err(inode->i_sb,
> + "unsupported chunk format %x of nid %llu",
> + vi->chunkformat, vi->nid);
> + err = -EOPNOTSUPP;
> + goto err_out;
> + }
> + vi->chunkbits = LOG_BLOCK_SIZE +
> + (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
> + }
> inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
> index 25b094085ca6..0a46e149aadd 100644
> --- a/fs/erofs/internal.h
> +++ b/fs/erofs/internal.h
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #ifndef __EROFS_INTERNAL_H
> #define __EROFS_INTERNAL_H
> @@ -260,6 +261,10 @@ struct erofs_inode {
>
> union {
> erofs_blk_t raw_blkaddr;
> + struct {
> + unsigned short chunkformat;
> + unsigned char chunkbits;
> + };
> #ifdef CONFIG_EROFS_FS_ZIP
> struct {
> unsigned short z_advise;
> --
> 2.24.4
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 1/2] erofs: introduce chunk-based file on-disk format
2021-08-18 22:28 ` Liu Bo
@ 2021-08-19 0:59 ` Gao Xiang
-1 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-19 0:59 UTC (permalink / raw)
To: Liu Bo
Cc: linux-erofs, Chao Yu, LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi
Hi Bo,
On Thu, Aug 19, 2021 at 06:28:04AM +0800, Liu Bo wrote:
> On Wed, Aug 18, 2021 at 03:07:12PM +0800, Gao Xiang wrote:
...
> > + EROFS_INODE_CHUNK_BASED = 4,
> > EROFS_INODE_DATALAYOUT_MAX
> > };
> >
> > @@ -90,6 +96,19 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
> > #define EROFS_I_ALL \
> > ((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
> >
> > +/* indicate chunk blkbits, thus `chunksize = blocksize << chunk blkbits' */
>
> A typo in the quotation marks. (`chunksize = ) should be ('chunksize =)
Such usage is like below:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0e389028ad75412ff624b304913bba14f8d46ec4
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=78128fabd022240852859c0b253972147593690b
I'm fine in either way. I'll update it in the next version or when
submitting.
>
> Otherwise it looks good.
>
> Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Thanks for the review!
Thanks,
Gao Xiang
>
> thanks,
> liubo
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 1/2] erofs: introduce chunk-based file on-disk format
@ 2021-08-19 0:59 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-19 0:59 UTC (permalink / raw)
To: Liu Bo; +Cc: LKML, Peng Tao, Joseph Qi, Eryu Guan, Liu Jiang, linux-erofs
Hi Bo,
On Thu, Aug 19, 2021 at 06:28:04AM +0800, Liu Bo wrote:
> On Wed, Aug 18, 2021 at 03:07:12PM +0800, Gao Xiang wrote:
...
> > + EROFS_INODE_CHUNK_BASED = 4,
> > EROFS_INODE_DATALAYOUT_MAX
> > };
> >
> > @@ -90,6 +96,19 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
> > #define EROFS_I_ALL \
> > ((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
> >
> > +/* indicate chunk blkbits, thus `chunksize = blocksize << chunk blkbits' */
>
> A typo in the quotation marks. (`chunksize = ) should be ('chunksize =)
Such usage is like below:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0e389028ad75412ff624b304913bba14f8d46ec4
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=78128fabd022240852859c0b253972147593690b
I'm fine in either way. I'll update it in the next version or when
submitting.
>
> Otherwise it looks good.
>
> Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Thanks for the review!
Thanks,
Gao Xiang
>
> thanks,
> liubo
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 1/2] erofs: introduce chunk-based file on-disk format
2021-08-18 7:07 ` Gao Xiang
@ 2021-08-19 3:32 ` Chao Yu
-1 siblings, 0 replies; 37+ messages in thread
From: Chao Yu @ 2021-08-19 3:32 UTC (permalink / raw)
To: Gao Xiang, linux-erofs, Liu Bo
Cc: LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi
On 2021/8/18 15:07, Gao Xiang wrote:
> Currently, uncompressed data except for tail-packing inline is
> consecutive on disk.
>
> In order to support chunk-based data deduplication, add a new
> corresponding inode data layout.
>
> In the future, the data source of chunks can be either (un)compressed.
>
> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Reviewed-by: Chao Yu <chao@kernel.org>
Thanks,
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 1/2] erofs: introduce chunk-based file on-disk format
@ 2021-08-19 3:32 ` Chao Yu
0 siblings, 0 replies; 37+ messages in thread
From: Chao Yu @ 2021-08-19 3:32 UTC (permalink / raw)
To: Gao Xiang, linux-erofs, Liu Bo
Cc: Joseph Qi, Eryu Guan, Liu Jiang, LKML, Peng Tao
On 2021/8/18 15:07, Gao Xiang wrote:
> Currently, uncompressed data except for tail-packing inline is
> consecutive on disk.
>
> In order to support chunk-based data deduplication, add a new
> corresponding inode data layout.
>
> In the future, the data source of chunks can be either (un)compressed.
>
> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Reviewed-by: Chao Yu <chao@kernel.org>
Thanks,
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 2/2] erofs: support reading chunk-based uncompressed files
2021-08-18 7:07 ` Gao Xiang
@ 2021-08-19 3:46 ` Chao Yu
-1 siblings, 0 replies; 37+ messages in thread
From: Chao Yu @ 2021-08-19 3:46 UTC (permalink / raw)
To: Gao Xiang, linux-erofs, Liu Bo
Cc: LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi
On 2021/8/18 15:07, Gao Xiang wrote:
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + /* fill chunked inode summary info */
> + vi->chunkformat = __le16_to_cpu(die->i_u.c.format);
le16_to_cpu(),
> kfree(copied);
> break;
> case EROFS_INODE_LAYOUT_COMPACT:
> @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> inode->i_size = le32_to_cpu(dic->i_size);
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + vi->chunkformat = __le16_to_cpu(dic->i_u.c.format);
Ditto.
Thanks,
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-19 3:46 ` Chao Yu
0 siblings, 0 replies; 37+ messages in thread
From: Chao Yu @ 2021-08-19 3:46 UTC (permalink / raw)
To: Gao Xiang, linux-erofs, Liu Bo
Cc: Joseph Qi, Eryu Guan, Liu Jiang, LKML, Peng Tao
On 2021/8/18 15:07, Gao Xiang wrote:
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + /* fill chunked inode summary info */
> + vi->chunkformat = __le16_to_cpu(die->i_u.c.format);
le16_to_cpu(),
> kfree(copied);
> break;
> case EROFS_INODE_LAYOUT_COMPACT:
> @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> inode->i_size = le32_to_cpu(dic->i_size);
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + vi->chunkformat = __le16_to_cpu(dic->i_u.c.format);
Ditto.
Thanks,
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 2/2] erofs: support reading chunk-based uncompressed files
2021-08-19 3:46 ` Chao Yu
@ 2021-08-19 5:14 ` Gao Xiang
-1 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-19 5:14 UTC (permalink / raw)
To: Chao Yu
Cc: linux-erofs, Liu Bo, LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi
Hi Chao,
On Thu, Aug 19, 2021 at 11:46:11AM +0800, Chao Yu wrote:
> On 2021/8/18 15:07, Gao Xiang wrote:
> > + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> > + /* fill chunked inode summary info */
> > + vi->chunkformat = __le16_to_cpu(die->i_u.c.format);
>
> le16_to_cpu(),
Thanks for the review and catching this, I didn't meant to use that.
Will send out the next version soon.
Thanks,
Gao Xiang
>
> > kfree(copied);
> > break;
> > case EROFS_INODE_LAYOUT_COMPACT:
> > @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> > inode->i_size = le32_to_cpu(dic->i_size);
> > if (erofs_inode_is_data_compressed(vi->datalayout))
> > nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> > + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> > + vi->chunkformat = __le16_to_cpu(dic->i_u.c.format);
>
> Ditto.
>
> Thanks,
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-19 5:14 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-19 5:14 UTC (permalink / raw)
To: Chao Yu
Cc: LKML, Peng Tao, Joseph Qi, Liu Bo, Eryu Guan, Liu Jiang, linux-erofs
Hi Chao,
On Thu, Aug 19, 2021 at 11:46:11AM +0800, Chao Yu wrote:
> On 2021/8/18 15:07, Gao Xiang wrote:
> > + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> > + /* fill chunked inode summary info */
> > + vi->chunkformat = __le16_to_cpu(die->i_u.c.format);
>
> le16_to_cpu(),
Thanks for the review and catching this, I didn't meant to use that.
Will send out the next version soon.
Thanks,
Gao Xiang
>
> > kfree(copied);
> > break;
> > case EROFS_INODE_LAYOUT_COMPACT:
> > @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> > inode->i_size = le32_to_cpu(dic->i_size);
> > if (erofs_inode_is_data_compressed(vi->datalayout))
> > nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> > + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> > + vi->chunkformat = __le16_to_cpu(dic->i_u.c.format);
>
> Ditto.
>
> Thanks,
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v2 1/2] erofs: introduce chunk-based file on-disk format
2021-08-18 7:07 ` Gao Xiang
@ 2021-08-19 6:33 ` Gao Xiang
-1 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-19 6:33 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi, Tao Ma, Gao Xiang
Currently, uncompressed data except for tail-packing inline is
consecutive on disk.
In order to support chunk-based data deduplication, add a new
corresponding inode data layout.
In the future, the data source of chunks can be either (un)compressed.
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Reviewed-by: Chao Yu <chao@kernel.org>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
changes since v1:
- update a quotation mark pair suggested by Bo;
- adjust the members of struct erofs_inode_chunk_index to keep in
sync with struct z_erofs_vle_decompressed_index for better
extendibility later.
Documentation/filesystems/erofs.rst | 16 +++++++++--
fs/erofs/erofs_fs.h | 43 +++++++++++++++++++++++++++--
2 files changed, 55 insertions(+), 4 deletions(-)
diff --git a/Documentation/filesystems/erofs.rst b/Documentation/filesystems/erofs.rst
index 868e3972227f..d484408a90c1 100644
--- a/Documentation/filesystems/erofs.rst
+++ b/Documentation/filesystems/erofs.rst
@@ -156,13 +156,14 @@ may not. All metadatas can be now observed in two different spaces (views):
Xattrs, extents, data inline are followed by the corresponding inode with
proper alignment, and they could be optional for different data mappings.
- _currently_ total 4 valid data mappings are supported:
+ _currently_ total 5 data layouts are supported:
== ====================================================================
0 flat file data without data inline (no extent);
1 fixed-sized output data compression (with non-compacted indexes);
2 flat file data with tail packing data inline (no extent);
- 3 fixed-sized output data compression (with compacted indexes, v5.3+).
+ 3 fixed-sized output data compression (with compacted indexes, v5.3+);
+ 4 chunk-based file (v5.15+).
== ====================================================================
The size of the optional xattrs is indicated by i_xattr_count in inode
@@ -213,6 +214,17 @@ Note that apart from the offset of the first filename, nameoff0 also indicates
the total number of directory entries in this block since it is no need to
introduce another on-disk field at all.
+Chunk-based file
+----------------
+In order to support chunk-based file deduplication, a new inode data layout has
+been supported since Linux v5.15: Files are split in equal-sized data chunks
+with ``extents`` area of the inode metadata indicating how to get the chunk
+data: these can be simply as a 4-byte block address array or in the 8-byte
+chunk index form (see struct erofs_inode_chunk_index in erofs_fs.h for more
+details.)
+
+By the way, chunk-based files are all uncompressed for now.
+
Data compression
----------------
EROFS implements LZ4 fixed-sized output compression which generates fixed-sized
diff --git a/fs/erofs/erofs_fs.h b/fs/erofs/erofs_fs.h
index 0f8da74570b4..f4c249996341 100644
--- a/fs/erofs/erofs_fs.h
+++ b/fs/erofs/erofs_fs.h
@@ -4,6 +4,7 @@
*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_FS_H
#define __EROFS_FS_H
@@ -19,10 +20,12 @@
#define EROFS_FEATURE_INCOMPAT_LZ4_0PADDING 0x00000001
#define EROFS_FEATURE_INCOMPAT_COMPR_CFGS 0x00000002
#define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x00000002
+#define EROFS_FEATURE_INCOMPAT_CHUNKED_FILE 0x00000004
#define EROFS_ALL_FEATURE_INCOMPAT \
(EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \
EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
- EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)
+ EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER | \
+ EROFS_FEATURE_INCOMPAT_CHUNKED_FILE)
#define EROFS_SB_EXTSLOT_SIZE 16
@@ -64,13 +67,16 @@ struct erofs_super_block {
* inode, [xattrs], last_inline_data, ... | ... | no-holed data
* 3 - inode compression D:
* inode, [xattrs], map_header, extents ... | ...
- * 4~7 - reserved
+ * 4 - inode chunk-based E:
+ * inode, [xattrs], chunk indexes ... | ...
+ * 5~7 - reserved
*/
enum {
EROFS_INODE_FLAT_PLAIN = 0,
EROFS_INODE_FLAT_COMPRESSION_LEGACY = 1,
EROFS_INODE_FLAT_INLINE = 2,
EROFS_INODE_FLAT_COMPRESSION = 3,
+ EROFS_INODE_CHUNK_BASED = 4,
EROFS_INODE_DATALAYOUT_MAX
};
@@ -90,6 +96,19 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
#define EROFS_I_ALL \
((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
+/* indicate chunk blkbits, thus 'chunksize = blocksize << chunk blkbits' */
+#define EROFS_CHUNK_FORMAT_BLKBITS_MASK 0x001F
+/* with chunk indexes or just a 4-byte blkaddr array */
+#define EROFS_CHUNK_FORMAT_INDEXES 0x0020
+
+#define EROFS_CHUNK_FORMAT_ALL \
+ (EROFS_CHUNK_FORMAT_BLKBITS_MASK | EROFS_CHUNK_FORMAT_INDEXES)
+
+struct erofs_inode_chunk_info {
+ __le16 format; /* chunk blkbits, etc. */
+ __le16 reserved;
+};
+
/* 32-byte reduced form of an ondisk inode */
struct erofs_inode_compact {
__le16 i_format; /* inode format hints */
@@ -107,6 +126,9 @@ struct erofs_inode_compact {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
__le32 i_ino; /* only used for 32-bit stat compatibility */
__le16 i_uid;
@@ -135,6 +157,9 @@ struct erofs_inode_extended {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
/* only used for 32-bit stat compatibility */
@@ -204,6 +229,15 @@ static inline unsigned int erofs_xattr_entry_size(struct erofs_xattr_entry *e)
e->e_name_len + le16_to_cpu(e->e_value_size));
}
+/* represent a zeroed chunk (hole) */
+#define EROFS_NULL_ADDR -1
+
+struct erofs_inode_chunk_index {
+ __le16 advise; /* always 0, don't care for now */
+ __le16 device_id; /* back-end storage id, always 0 for now */
+ __le32 blkaddr; /* start block address of this inode chunk */
+};
+
/* maximum supported size of a physical compression cluster */
#define Z_EROFS_PCLUSTER_MAX_SIZE (1024 * 1024)
@@ -338,9 +372,14 @@ static inline void erofs_check_ondisk_layout_definitions(void)
BUILD_BUG_ON(sizeof(struct erofs_inode_extended) != 64);
BUILD_BUG_ON(sizeof(struct erofs_xattr_ibody_header) != 12);
BUILD_BUG_ON(sizeof(struct erofs_xattr_entry) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_info) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_map_header) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_vle_decompressed_index) != 8);
BUILD_BUG_ON(sizeof(struct erofs_dirent) != 12);
+ /* keep in sync between 2 index structures for better extendibility */
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) !=
+ sizeof(struct z_erofs_vle_decompressed_index));
BUILD_BUG_ON(BIT(Z_EROFS_VLE_DI_CLUSTER_TYPE_BITS) <
Z_EROFS_VLE_CLUSTER_TYPE_MAX - 1);
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH v2 1/2] erofs: introduce chunk-based file on-disk format
@ 2021-08-19 6:33 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-19 6:33 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: Tao Ma, LKML, Peng Tao, Joseph Qi, Eryu Guan, Liu Jiang
Currently, uncompressed data except for tail-packing inline is
consecutive on disk.
In order to support chunk-based data deduplication, add a new
corresponding inode data layout.
In the future, the data source of chunks can be either (un)compressed.
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Reviewed-by: Chao Yu <chao@kernel.org>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
changes since v1:
- update a quotation mark pair suggested by Bo;
- adjust the members of struct erofs_inode_chunk_index to keep in
sync with struct z_erofs_vle_decompressed_index for better
extendibility later.
Documentation/filesystems/erofs.rst | 16 +++++++++--
fs/erofs/erofs_fs.h | 43 +++++++++++++++++++++++++++--
2 files changed, 55 insertions(+), 4 deletions(-)
diff --git a/Documentation/filesystems/erofs.rst b/Documentation/filesystems/erofs.rst
index 868e3972227f..d484408a90c1 100644
--- a/Documentation/filesystems/erofs.rst
+++ b/Documentation/filesystems/erofs.rst
@@ -156,13 +156,14 @@ may not. All metadatas can be now observed in two different spaces (views):
Xattrs, extents, data inline are followed by the corresponding inode with
proper alignment, and they could be optional for different data mappings.
- _currently_ total 4 valid data mappings are supported:
+ _currently_ total 5 data layouts are supported:
== ====================================================================
0 flat file data without data inline (no extent);
1 fixed-sized output data compression (with non-compacted indexes);
2 flat file data with tail packing data inline (no extent);
- 3 fixed-sized output data compression (with compacted indexes, v5.3+).
+ 3 fixed-sized output data compression (with compacted indexes, v5.3+);
+ 4 chunk-based file (v5.15+).
== ====================================================================
The size of the optional xattrs is indicated by i_xattr_count in inode
@@ -213,6 +214,17 @@ Note that apart from the offset of the first filename, nameoff0 also indicates
the total number of directory entries in this block since it is no need to
introduce another on-disk field at all.
+Chunk-based file
+----------------
+In order to support chunk-based file deduplication, a new inode data layout has
+been supported since Linux v5.15: Files are split in equal-sized data chunks
+with ``extents`` area of the inode metadata indicating how to get the chunk
+data: these can be simply as a 4-byte block address array or in the 8-byte
+chunk index form (see struct erofs_inode_chunk_index in erofs_fs.h for more
+details.)
+
+By the way, chunk-based files are all uncompressed for now.
+
Data compression
----------------
EROFS implements LZ4 fixed-sized output compression which generates fixed-sized
diff --git a/fs/erofs/erofs_fs.h b/fs/erofs/erofs_fs.h
index 0f8da74570b4..f4c249996341 100644
--- a/fs/erofs/erofs_fs.h
+++ b/fs/erofs/erofs_fs.h
@@ -4,6 +4,7 @@
*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_FS_H
#define __EROFS_FS_H
@@ -19,10 +20,12 @@
#define EROFS_FEATURE_INCOMPAT_LZ4_0PADDING 0x00000001
#define EROFS_FEATURE_INCOMPAT_COMPR_CFGS 0x00000002
#define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x00000002
+#define EROFS_FEATURE_INCOMPAT_CHUNKED_FILE 0x00000004
#define EROFS_ALL_FEATURE_INCOMPAT \
(EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \
EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
- EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)
+ EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER | \
+ EROFS_FEATURE_INCOMPAT_CHUNKED_FILE)
#define EROFS_SB_EXTSLOT_SIZE 16
@@ -64,13 +67,16 @@ struct erofs_super_block {
* inode, [xattrs], last_inline_data, ... | ... | no-holed data
* 3 - inode compression D:
* inode, [xattrs], map_header, extents ... | ...
- * 4~7 - reserved
+ * 4 - inode chunk-based E:
+ * inode, [xattrs], chunk indexes ... | ...
+ * 5~7 - reserved
*/
enum {
EROFS_INODE_FLAT_PLAIN = 0,
EROFS_INODE_FLAT_COMPRESSION_LEGACY = 1,
EROFS_INODE_FLAT_INLINE = 2,
EROFS_INODE_FLAT_COMPRESSION = 3,
+ EROFS_INODE_CHUNK_BASED = 4,
EROFS_INODE_DATALAYOUT_MAX
};
@@ -90,6 +96,19 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
#define EROFS_I_ALL \
((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
+/* indicate chunk blkbits, thus 'chunksize = blocksize << chunk blkbits' */
+#define EROFS_CHUNK_FORMAT_BLKBITS_MASK 0x001F
+/* with chunk indexes or just a 4-byte blkaddr array */
+#define EROFS_CHUNK_FORMAT_INDEXES 0x0020
+
+#define EROFS_CHUNK_FORMAT_ALL \
+ (EROFS_CHUNK_FORMAT_BLKBITS_MASK | EROFS_CHUNK_FORMAT_INDEXES)
+
+struct erofs_inode_chunk_info {
+ __le16 format; /* chunk blkbits, etc. */
+ __le16 reserved;
+};
+
/* 32-byte reduced form of an ondisk inode */
struct erofs_inode_compact {
__le16 i_format; /* inode format hints */
@@ -107,6 +126,9 @@ struct erofs_inode_compact {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
__le32 i_ino; /* only used for 32-bit stat compatibility */
__le16 i_uid;
@@ -135,6 +157,9 @@ struct erofs_inode_extended {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
/* only used for 32-bit stat compatibility */
@@ -204,6 +229,15 @@ static inline unsigned int erofs_xattr_entry_size(struct erofs_xattr_entry *e)
e->e_name_len + le16_to_cpu(e->e_value_size));
}
+/* represent a zeroed chunk (hole) */
+#define EROFS_NULL_ADDR -1
+
+struct erofs_inode_chunk_index {
+ __le16 advise; /* always 0, don't care for now */
+ __le16 device_id; /* back-end storage id, always 0 for now */
+ __le32 blkaddr; /* start block address of this inode chunk */
+};
+
/* maximum supported size of a physical compression cluster */
#define Z_EROFS_PCLUSTER_MAX_SIZE (1024 * 1024)
@@ -338,9 +372,14 @@ static inline void erofs_check_ondisk_layout_definitions(void)
BUILD_BUG_ON(sizeof(struct erofs_inode_extended) != 64);
BUILD_BUG_ON(sizeof(struct erofs_xattr_ibody_header) != 12);
BUILD_BUG_ON(sizeof(struct erofs_xattr_entry) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_info) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_map_header) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_vle_decompressed_index) != 8);
BUILD_BUG_ON(sizeof(struct erofs_dirent) != 12);
+ /* keep in sync between 2 index structures for better extendibility */
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) !=
+ sizeof(struct z_erofs_vle_decompressed_index));
BUILD_BUG_ON(BIT(Z_EROFS_VLE_DI_CLUSTER_TYPE_BITS) <
Z_EROFS_VLE_CLUSTER_TYPE_MAX - 1);
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
2021-08-19 6:33 ` Gao Xiang
@ 2021-08-19 6:33 ` Gao Xiang
-1 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-19 6:33 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi, Tao Ma, Gao Xiang
Add runtime support for chunk-based uncompressed files
described in the previous patch.
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
changes since v1:
- use le16_to_cpu instead of __le16_to_cpu pointed out by Chao.
fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
fs/erofs/inode.c | 18 ++++++++-
fs/erofs/internal.h | 5 +++
3 files changed, 102 insertions(+), 11 deletions(-)
diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index 09c46fbdb9b2..ee9a33485313 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "internal.h"
#include <linux/prefetch.h>
@@ -36,13 +37,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
lastblk = nblocks - tailendpacking;
- if (offset >= inode->i_size) {
- /* leave out-of-bound access unmapped */
- map->m_flags = 0;
- map->m_plen = 0;
- goto out;
- }
-
/* there is no hole in flatmode */
map->m_flags = EROFS_MAP_MAPPED;
@@ -77,14 +71,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
goto err_out;
}
-out:
map->m_llen = map->m_plen;
-
err_out:
trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
return err;
}
+static int erofs_map_blocks(struct inode *inode,
+ struct erofs_map_blocks *map, int flags)
+{
+ struct super_block *sb = inode->i_sb;
+ struct erofs_inode *vi = EROFS_I(inode);
+ struct erofs_inode_chunk_index *idx;
+ struct page *page;
+ u64 chunknr;
+ unsigned int unit;
+ erofs_off_t pos;
+ int err = 0;
+
+ if (map->m_la >= inode->i_size) {
+ /* leave out-of-bound access unmapped */
+ map->m_flags = 0;
+ map->m_plen = 0;
+ goto out;
+ }
+
+ if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
+ return erofs_map_blocks_flatmode(inode, map, flags);
+
+ if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
+ unit = sizeof(*idx); /* chunk index */
+ else
+ unit = 4; /* block map */
+
+ chunknr = map->m_la >> vi->chunkbits;
+ pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
+ vi->xattr_isize, unit) + unit * chunknr;
+
+ page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
+ if (IS_ERR(page))
+ return PTR_ERR(page);
+
+ map->m_la = chunknr << vi->chunkbits;
+ map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
+ roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
+
+ /* handle block map */
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
+ __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
+
+ if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
+ map->m_flags = 0;
+ } else {
+ map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ }
+ goto out_unlock;
+ }
+ /* parse chunk indexes */
+ idx = page_address(page) + erofs_blkoff(pos);
+ switch (le32_to_cpu(idx->blkaddr)) {
+ case EROFS_NULL_ADDR:
+ map->m_flags = 0;
+ break;
+ default:
+ /* only one device is supported for now */
+ if (idx->device_id) {
+ erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
+ le32_to_cpu(idx->device_id),
+ chunknr, vi->nid);
+ err = -EFSCORRUPTED;
+ goto out_unlock;
+ }
+ map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ break;
+ }
+out_unlock:
+ unlock_page(page);
+ put_page(page);
+out:
+ map->m_llen = map->m_plen;
+ return err;
+}
+
static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
{
@@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
map.m_la = offset;
map.m_llen = length;
- ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
+ ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
if (ret < 0)
return ret;
diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
index d13e0709599c..4408929bd6f5 100644
--- a/fs/erofs/inode.c
+++ b/fs/erofs/inode.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "xattr.h"
@@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
/* total blocks for compressed files */
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(die->i_u.compressed_blocks);
-
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ /* fill chunked inode summary info */
+ vi->chunkformat = le16_to_cpu(die->i_u.c.format);
kfree(copied);
break;
case EROFS_INODE_LAYOUT_COMPACT:
@@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
inode->i_size = le32_to_cpu(dic->i_size);
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(dic->i_u.compressed_blocks);
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
break;
default:
erofs_err(inode->i_sb,
@@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
goto err_out;
}
+ if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
+ erofs_err(inode->i_sb,
+ "unsupported chunk format %x of nid %llu",
+ vi->chunkformat, vi->nid);
+ err = -EOPNOTSUPP;
+ goto err_out;
+ }
+ vi->chunkbits = LOG_BLOCK_SIZE +
+ (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
+ }
inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
index 91089ab8a816..9524e155b38f 100644
--- a/fs/erofs/internal.h
+++ b/fs/erofs/internal.h
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_INTERNAL_H
#define __EROFS_INTERNAL_H
@@ -261,6 +262,10 @@ struct erofs_inode {
union {
erofs_blk_t raw_blkaddr;
+ struct {
+ unsigned short chunkformat;
+ unsigned char chunkbits;
+ };
#ifdef CONFIG_EROFS_FS_ZIP
struct {
unsigned short z_advise;
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-19 6:33 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-19 6:33 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: Tao Ma, LKML, Peng Tao, Joseph Qi, Eryu Guan, Liu Jiang
Add runtime support for chunk-based uncompressed files
described in the previous patch.
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
changes since v1:
- use le16_to_cpu instead of __le16_to_cpu pointed out by Chao.
fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
fs/erofs/inode.c | 18 ++++++++-
fs/erofs/internal.h | 5 +++
3 files changed, 102 insertions(+), 11 deletions(-)
diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index 09c46fbdb9b2..ee9a33485313 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "internal.h"
#include <linux/prefetch.h>
@@ -36,13 +37,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
lastblk = nblocks - tailendpacking;
- if (offset >= inode->i_size) {
- /* leave out-of-bound access unmapped */
- map->m_flags = 0;
- map->m_plen = 0;
- goto out;
- }
-
/* there is no hole in flatmode */
map->m_flags = EROFS_MAP_MAPPED;
@@ -77,14 +71,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
goto err_out;
}
-out:
map->m_llen = map->m_plen;
-
err_out:
trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
return err;
}
+static int erofs_map_blocks(struct inode *inode,
+ struct erofs_map_blocks *map, int flags)
+{
+ struct super_block *sb = inode->i_sb;
+ struct erofs_inode *vi = EROFS_I(inode);
+ struct erofs_inode_chunk_index *idx;
+ struct page *page;
+ u64 chunknr;
+ unsigned int unit;
+ erofs_off_t pos;
+ int err = 0;
+
+ if (map->m_la >= inode->i_size) {
+ /* leave out-of-bound access unmapped */
+ map->m_flags = 0;
+ map->m_plen = 0;
+ goto out;
+ }
+
+ if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
+ return erofs_map_blocks_flatmode(inode, map, flags);
+
+ if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
+ unit = sizeof(*idx); /* chunk index */
+ else
+ unit = 4; /* block map */
+
+ chunknr = map->m_la >> vi->chunkbits;
+ pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
+ vi->xattr_isize, unit) + unit * chunknr;
+
+ page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
+ if (IS_ERR(page))
+ return PTR_ERR(page);
+
+ map->m_la = chunknr << vi->chunkbits;
+ map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
+ roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
+
+ /* handle block map */
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
+ __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
+
+ if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
+ map->m_flags = 0;
+ } else {
+ map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ }
+ goto out_unlock;
+ }
+ /* parse chunk indexes */
+ idx = page_address(page) + erofs_blkoff(pos);
+ switch (le32_to_cpu(idx->blkaddr)) {
+ case EROFS_NULL_ADDR:
+ map->m_flags = 0;
+ break;
+ default:
+ /* only one device is supported for now */
+ if (idx->device_id) {
+ erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
+ le32_to_cpu(idx->device_id),
+ chunknr, vi->nid);
+ err = -EFSCORRUPTED;
+ goto out_unlock;
+ }
+ map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ break;
+ }
+out_unlock:
+ unlock_page(page);
+ put_page(page);
+out:
+ map->m_llen = map->m_plen;
+ return err;
+}
+
static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
{
@@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
map.m_la = offset;
map.m_llen = length;
- ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
+ ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
if (ret < 0)
return ret;
diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
index d13e0709599c..4408929bd6f5 100644
--- a/fs/erofs/inode.c
+++ b/fs/erofs/inode.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "xattr.h"
@@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
/* total blocks for compressed files */
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(die->i_u.compressed_blocks);
-
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ /* fill chunked inode summary info */
+ vi->chunkformat = le16_to_cpu(die->i_u.c.format);
kfree(copied);
break;
case EROFS_INODE_LAYOUT_COMPACT:
@@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
inode->i_size = le32_to_cpu(dic->i_size);
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(dic->i_u.compressed_blocks);
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
break;
default:
erofs_err(inode->i_sb,
@@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
goto err_out;
}
+ if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
+ erofs_err(inode->i_sb,
+ "unsupported chunk format %x of nid %llu",
+ vi->chunkformat, vi->nid);
+ err = -EOPNOTSUPP;
+ goto err_out;
+ }
+ vi->chunkbits = LOG_BLOCK_SIZE +
+ (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
+ }
inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
index 91089ab8a816..9524e155b38f 100644
--- a/fs/erofs/internal.h
+++ b/fs/erofs/internal.h
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_INTERNAL_H
#define __EROFS_INTERNAL_H
@@ -261,6 +262,10 @@ struct erofs_inode {
union {
erofs_blk_t raw_blkaddr;
+ struct {
+ unsigned short chunkformat;
+ unsigned char chunkbits;
+ };
#ifdef CONFIG_EROFS_FS_ZIP
struct {
unsigned short z_advise;
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
2021-08-19 6:33 ` Gao Xiang
@ 2021-08-19 6:37 ` Joseph Qi
-1 siblings, 0 replies; 37+ messages in thread
From: Joseph Qi @ 2021-08-19 6:37 UTC (permalink / raw)
To: Gao Xiang, linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Eryu Guan, Liu Jiang, Tao Ma
On 8/19/21 2:33 PM, Gao Xiang wrote:
> Add runtime support for chunk-based uncompressed files
> described in the previous patch.
>
> Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
> ---
> changes since v1:
> - use le16_to_cpu instead of __le16_to_cpu pointed out by Chao.
>
> fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
> fs/erofs/inode.c | 18 ++++++++-
> fs/erofs/internal.h | 5 +++
> 3 files changed, 102 insertions(+), 11 deletions(-)
>
> diff --git a/fs/erofs/data.c b/fs/erofs/data.c
> index 09c46fbdb9b2..ee9a33485313 100644
> --- a/fs/erofs/data.c
> +++ b/fs/erofs/data.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "internal.h"
> #include <linux/prefetch.h>
> @@ -36,13 +37,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
> lastblk = nblocks - tailendpacking;
>
> - if (offset >= inode->i_size) {
> - /* leave out-of-bound access unmapped */
> - map->m_flags = 0;
> - map->m_plen = 0;
> - goto out;
> - }
> -
> /* there is no hole in flatmode */
> map->m_flags = EROFS_MAP_MAPPED;
>
> @@ -77,14 +71,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> goto err_out;
> }
>
> -out:
> map->m_llen = map->m_plen;
> -
> err_out:
> trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
> return err;
> }
>
> +static int erofs_map_blocks(struct inode *inode,
> + struct erofs_map_blocks *map, int flags)
> +{
> + struct super_block *sb = inode->i_sb;
> + struct erofs_inode *vi = EROFS_I(inode);
> + struct erofs_inode_chunk_index *idx;
> + struct page *page;
> + u64 chunknr;
> + unsigned int unit;
> + erofs_off_t pos;
> + int err = 0;
> +
> + if (map->m_la >= inode->i_size) {
> + /* leave out-of-bound access unmapped */
> + map->m_flags = 0;
> + map->m_plen = 0;
> + goto out;
> + }
> +
> + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
> + return erofs_map_blocks_flatmode(inode, map, flags);
> +
> + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
> + unit = sizeof(*idx); /* chunk index */
> + else
> + unit = 4; /* block map */
> +
> + chunknr = map->m_la >> vi->chunkbits;
> + pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
> + vi->xattr_isize, unit) + unit * chunknr;
> +
> + page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
> + if (IS_ERR(page))
> + return PTR_ERR(page);
> +
> + map->m_la = chunknr << vi->chunkbits;
> + map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
> + roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
> +
> + /* handle block map */
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
> + __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
> +
> + if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
> + map->m_flags = 0;
> + } else {
> + map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + }
> + goto out_unlock;
> + }
> + /* parse chunk indexes */
> + idx = page_address(page) + erofs_blkoff(pos);
> + switch (le32_to_cpu(idx->blkaddr)) {
> + case EROFS_NULL_ADDR:
> + map->m_flags = 0;
> + break;
> + default:
> + /* only one device is supported for now */
> + if (idx->device_id) {
> + erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
> + le32_to_cpu(idx->device_id),
> + chunknr, vi->nid);
> + err = -EFSCORRUPTED;
> + goto out_unlock;
> + }
> + map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + break;
> + }
> +out_unlock:
> + unlock_page(page);
> + put_page(page);
> +out:
> + map->m_llen = map->m_plen;
> + return err;
> +}
> +
> static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
> {
> @@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> map.m_la = offset;
> map.m_llen = length;
>
> - ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
> + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
> if (ret < 0)
> return ret;
>
> diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
> index d13e0709599c..4408929bd6f5 100644
> --- a/fs/erofs/inode.c
> +++ b/fs/erofs/inode.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "xattr.h"
>
> @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
> /* total blocks for compressed files */
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(die->i_u.compressed_blocks);
> -
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + /* fill chunked inode summary info */
> + vi->chunkformat = le16_to_cpu(die->i_u.c.format);
Better to add braces for if/else.
Thanks,
Joseph
> kfree(copied);
> break;
> case EROFS_INODE_LAYOUT_COMPACT:
> @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> inode->i_size = le32_to_cpu(dic->i_size);
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
> break;
> default:
> erofs_err(inode->i_sb,
> @@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
> goto err_out;
> }
>
> + if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
> + erofs_err(inode->i_sb,
> + "unsupported chunk format %x of nid %llu",
> + vi->chunkformat, vi->nid);
> + err = -EOPNOTSUPP;
> + goto err_out;
> + }
> + vi->chunkbits = LOG_BLOCK_SIZE +
> + (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
> + }
> inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
> index 91089ab8a816..9524e155b38f 100644
> --- a/fs/erofs/internal.h
> +++ b/fs/erofs/internal.h
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #ifndef __EROFS_INTERNAL_H
> #define __EROFS_INTERNAL_H
> @@ -261,6 +262,10 @@ struct erofs_inode {
>
> union {
> erofs_blk_t raw_blkaddr;
> + struct {
> + unsigned short chunkformat;
> + unsigned char chunkbits;
> + };
> #ifdef CONFIG_EROFS_FS_ZIP
> struct {
> unsigned short z_advise;
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-19 6:37 ` Joseph Qi
0 siblings, 0 replies; 37+ messages in thread
From: Joseph Qi @ 2021-08-19 6:37 UTC (permalink / raw)
To: Gao Xiang, linux-erofs, Chao Yu, Liu Bo
Cc: Tao Ma, Eryu Guan, Liu Jiang, LKML, Peng Tao
On 8/19/21 2:33 PM, Gao Xiang wrote:
> Add runtime support for chunk-based uncompressed files
> described in the previous patch.
>
> Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
> ---
> changes since v1:
> - use le16_to_cpu instead of __le16_to_cpu pointed out by Chao.
>
> fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
> fs/erofs/inode.c | 18 ++++++++-
> fs/erofs/internal.h | 5 +++
> 3 files changed, 102 insertions(+), 11 deletions(-)
>
> diff --git a/fs/erofs/data.c b/fs/erofs/data.c
> index 09c46fbdb9b2..ee9a33485313 100644
> --- a/fs/erofs/data.c
> +++ b/fs/erofs/data.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "internal.h"
> #include <linux/prefetch.h>
> @@ -36,13 +37,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
> lastblk = nblocks - tailendpacking;
>
> - if (offset >= inode->i_size) {
> - /* leave out-of-bound access unmapped */
> - map->m_flags = 0;
> - map->m_plen = 0;
> - goto out;
> - }
> -
> /* there is no hole in flatmode */
> map->m_flags = EROFS_MAP_MAPPED;
>
> @@ -77,14 +71,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> goto err_out;
> }
>
> -out:
> map->m_llen = map->m_plen;
> -
> err_out:
> trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
> return err;
> }
>
> +static int erofs_map_blocks(struct inode *inode,
> + struct erofs_map_blocks *map, int flags)
> +{
> + struct super_block *sb = inode->i_sb;
> + struct erofs_inode *vi = EROFS_I(inode);
> + struct erofs_inode_chunk_index *idx;
> + struct page *page;
> + u64 chunknr;
> + unsigned int unit;
> + erofs_off_t pos;
> + int err = 0;
> +
> + if (map->m_la >= inode->i_size) {
> + /* leave out-of-bound access unmapped */
> + map->m_flags = 0;
> + map->m_plen = 0;
> + goto out;
> + }
> +
> + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
> + return erofs_map_blocks_flatmode(inode, map, flags);
> +
> + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
> + unit = sizeof(*idx); /* chunk index */
> + else
> + unit = 4; /* block map */
> +
> + chunknr = map->m_la >> vi->chunkbits;
> + pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
> + vi->xattr_isize, unit) + unit * chunknr;
> +
> + page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
> + if (IS_ERR(page))
> + return PTR_ERR(page);
> +
> + map->m_la = chunknr << vi->chunkbits;
> + map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
> + roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
> +
> + /* handle block map */
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
> + __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
> +
> + if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
> + map->m_flags = 0;
> + } else {
> + map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + }
> + goto out_unlock;
> + }
> + /* parse chunk indexes */
> + idx = page_address(page) + erofs_blkoff(pos);
> + switch (le32_to_cpu(idx->blkaddr)) {
> + case EROFS_NULL_ADDR:
> + map->m_flags = 0;
> + break;
> + default:
> + /* only one device is supported for now */
> + if (idx->device_id) {
> + erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
> + le32_to_cpu(idx->device_id),
> + chunknr, vi->nid);
> + err = -EFSCORRUPTED;
> + goto out_unlock;
> + }
> + map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + break;
> + }
> +out_unlock:
> + unlock_page(page);
> + put_page(page);
> +out:
> + map->m_llen = map->m_plen;
> + return err;
> +}
> +
> static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
> {
> @@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> map.m_la = offset;
> map.m_llen = length;
>
> - ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
> + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
> if (ret < 0)
> return ret;
>
> diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
> index d13e0709599c..4408929bd6f5 100644
> --- a/fs/erofs/inode.c
> +++ b/fs/erofs/inode.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "xattr.h"
>
> @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
> /* total blocks for compressed files */
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(die->i_u.compressed_blocks);
> -
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + /* fill chunked inode summary info */
> + vi->chunkformat = le16_to_cpu(die->i_u.c.format);
Better to add braces for if/else.
Thanks,
Joseph
> kfree(copied);
> break;
> case EROFS_INODE_LAYOUT_COMPACT:
> @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> inode->i_size = le32_to_cpu(dic->i_size);
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
> break;
> default:
> erofs_err(inode->i_sb,
> @@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
> goto err_out;
> }
>
> + if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
> + erofs_err(inode->i_sb,
> + "unsupported chunk format %x of nid %llu",
> + vi->chunkformat, vi->nid);
> + err = -EOPNOTSUPP;
> + goto err_out;
> + }
> + vi->chunkbits = LOG_BLOCK_SIZE +
> + (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
> + }
> inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
> index 91089ab8a816..9524e155b38f 100644
> --- a/fs/erofs/internal.h
> +++ b/fs/erofs/internal.h
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #ifndef __EROFS_INTERNAL_H
> #define __EROFS_INTERNAL_H
> @@ -261,6 +262,10 @@ struct erofs_inode {
>
> union {
> erofs_blk_t raw_blkaddr;
> + struct {
> + unsigned short chunkformat;
> + unsigned char chunkbits;
> + };
> #ifdef CONFIG_EROFS_FS_ZIP
> struct {
> unsigned short z_advise;
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
2021-08-19 6:37 ` Joseph Qi
@ 2021-08-19 7:12 ` Gao Xiang
-1 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-19 7:12 UTC (permalink / raw)
To: Joseph Qi
Cc: linux-erofs, Chao Yu, Liu Bo, LKML, Peng Tao, Eryu Guan,
Liu Jiang, Tao Ma
Hi Joseph,
On Thu, Aug 19, 2021 at 02:37:50PM +0800, Joseph Qi wrote:
>
>
> On 8/19/21 2:33 PM, Gao Xiang wrote:
...
> > diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
> > index d13e0709599c..4408929bd6f5 100644
> > --- a/fs/erofs/inode.c
> > +++ b/fs/erofs/inode.c
> > @@ -2,6 +2,7 @@
> > /*
> > * Copyright (C) 2017-2018 HUAWEI, Inc.
> > * https://www.huawei.com/
> > + * Copyright (C) 2021, Alibaba Cloud
> > */
> > #include "xattr.h"
> >
> > @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
> > /* total blocks for compressed files */
> > if (erofs_inode_is_data_compressed(vi->datalayout))
> > nblks = le32_to_cpu(die->i_u.compressed_blocks);
> > -
> > + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> > + /* fill chunked inode summary info */
> > + vi->chunkformat = le16_to_cpu(die->i_u.c.format);
>
> Better to add braces for if/else.
Thanks for the kind suggestion. Here is single statement, I've checked
coding-style in Documentation. It's no necessary to use brace for this.
And checkpatch didn't report anything.
Also, I found some reference at
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/vmscan.c?h=v5.13#n3066
But anyway, I could update it when applying, either looks good to me.
Thanks,
Gao Xiang
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-19 7:12 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-19 7:12 UTC (permalink / raw)
To: Joseph Qi
Cc: Tao Ma, LKML, Peng Tao, Liu Bo, Eryu Guan, Liu Jiang, linux-erofs
Hi Joseph,
On Thu, Aug 19, 2021 at 02:37:50PM +0800, Joseph Qi wrote:
>
>
> On 8/19/21 2:33 PM, Gao Xiang wrote:
...
> > diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
> > index d13e0709599c..4408929bd6f5 100644
> > --- a/fs/erofs/inode.c
> > +++ b/fs/erofs/inode.c
> > @@ -2,6 +2,7 @@
> > /*
> > * Copyright (C) 2017-2018 HUAWEI, Inc.
> > * https://www.huawei.com/
> > + * Copyright (C) 2021, Alibaba Cloud
> > */
> > #include "xattr.h"
> >
> > @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
> > /* total blocks for compressed files */
> > if (erofs_inode_is_data_compressed(vi->datalayout))
> > nblks = le32_to_cpu(die->i_u.compressed_blocks);
> > -
> > + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> > + /* fill chunked inode summary info */
> > + vi->chunkformat = le16_to_cpu(die->i_u.c.format);
>
> Better to add braces for if/else.
Thanks for the kind suggestion. Here is single statement, I've checked
coding-style in Documentation. It's no necessary to use brace for this.
And checkpatch didn't report anything.
Also, I found some reference at
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/vmscan.c?h=v5.13#n3066
But anyway, I could update it when applying, either looks good to me.
Thanks,
Gao Xiang
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
2021-08-19 6:33 ` Gao Xiang
@ 2021-08-20 9:04 ` Chao Yu
-1 siblings, 0 replies; 37+ messages in thread
From: Chao Yu @ 2021-08-20 9:04 UTC (permalink / raw)
To: Gao Xiang, linux-erofs, Liu Bo
Cc: LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi, Tao Ma
On 2021/8/19 14:33, Gao Xiang wrote:
> Add runtime support for chunk-based uncompressed files
> described in the previous patch.
>
> Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
> ---
> changes since v1:
> - use le16_to_cpu instead of __le16_to_cpu pointed out by Chao.
>
> fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
> fs/erofs/inode.c | 18 ++++++++-
> fs/erofs/internal.h | 5 +++
> 3 files changed, 102 insertions(+), 11 deletions(-)
>
> diff --git a/fs/erofs/data.c b/fs/erofs/data.c
> index 09c46fbdb9b2..ee9a33485313 100644
> --- a/fs/erofs/data.c
> +++ b/fs/erofs/data.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "internal.h"
> #include <linux/prefetch.h>
> @@ -36,13 +37,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
> lastblk = nblocks - tailendpacking;
>
> - if (offset >= inode->i_size) {
> - /* leave out-of-bound access unmapped */
> - map->m_flags = 0;
> - map->m_plen = 0;
> - goto out;
> - }
> -
> /* there is no hole in flatmode */
> map->m_flags = EROFS_MAP_MAPPED;
>
> @@ -77,14 +71,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> goto err_out;
> }
>
> -out:
> map->m_llen = map->m_plen;
> -
> err_out:
> trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
> return err;
> }
>
> +static int erofs_map_blocks(struct inode *inode,
> + struct erofs_map_blocks *map, int flags)
> +{
> + struct super_block *sb = inode->i_sb;
> + struct erofs_inode *vi = EROFS_I(inode);
> + struct erofs_inode_chunk_index *idx;
> + struct page *page;
> + u64 chunknr;
> + unsigned int unit;
> + erofs_off_t pos;
> + int err = 0;
> +
> + if (map->m_la >= inode->i_size) {
> + /* leave out-of-bound access unmapped */
> + map->m_flags = 0;
> + map->m_plen = 0;
> + goto out;
> + }
> +
> + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
> + return erofs_map_blocks_flatmode(inode, map, flags);
> +
> + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
> + unit = sizeof(*idx); /* chunk index */
> + else
> + unit = 4; /* block map */
You mean sizeof(__le32)?
Otherwise it looks good to me.
Reviewed-by: Chao Yu <chao@kernel.org>
Thanks,
> +
> + chunknr = map->m_la >> vi->chunkbits;
> + pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
> + vi->xattr_isize, unit) + unit * chunknr;
> +
> + page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
> + if (IS_ERR(page))
> + return PTR_ERR(page);
> +
> + map->m_la = chunknr << vi->chunkbits;
> + map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
> + roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
> +
> + /* handle block map */
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
> + __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
> +
> + if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
> + map->m_flags = 0;
> + } else {
> + map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + }
> + goto out_unlock;
> + }
> + /* parse chunk indexes */
> + idx = page_address(page) + erofs_blkoff(pos);
> + switch (le32_to_cpu(idx->blkaddr)) {
> + case EROFS_NULL_ADDR:
> + map->m_flags = 0;
> + break;
> + default:
> + /* only one device is supported for now */
> + if (idx->device_id) {
> + erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
> + le32_to_cpu(idx->device_id),
> + chunknr, vi->nid);
> + err = -EFSCORRUPTED;
> + goto out_unlock;
> + }
> + map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + break;
> + }
> +out_unlock:
> + unlock_page(page);
> + put_page(page);
> +out:
> + map->m_llen = map->m_plen;
> + return err;
> +}
> +
> static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
> {
> @@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> map.m_la = offset;
> map.m_llen = length;
>
> - ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
> + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
> if (ret < 0)
> return ret;
>
> diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
> index d13e0709599c..4408929bd6f5 100644
> --- a/fs/erofs/inode.c
> +++ b/fs/erofs/inode.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "xattr.h"
>
> @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
> /* total blocks for compressed files */
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(die->i_u.compressed_blocks);
> -
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + /* fill chunked inode summary info */
> + vi->chunkformat = le16_to_cpu(die->i_u.c.format);
> kfree(copied);
> break;
> case EROFS_INODE_LAYOUT_COMPACT:
> @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> inode->i_size = le32_to_cpu(dic->i_size);
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
> break;
> default:
> erofs_err(inode->i_sb,
> @@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
> goto err_out;
> }
>
> + if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
> + erofs_err(inode->i_sb,
> + "unsupported chunk format %x of nid %llu",
> + vi->chunkformat, vi->nid);
> + err = -EOPNOTSUPP;
> + goto err_out;
> + }
> + vi->chunkbits = LOG_BLOCK_SIZE +
> + (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
> + }
> inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
> index 91089ab8a816..9524e155b38f 100644
> --- a/fs/erofs/internal.h
> +++ b/fs/erofs/internal.h
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #ifndef __EROFS_INTERNAL_H
> #define __EROFS_INTERNAL_H
> @@ -261,6 +262,10 @@ struct erofs_inode {
>
> union {
> erofs_blk_t raw_blkaddr;
> + struct {
> + unsigned short chunkformat;
> + unsigned char chunkbits;
> + };
> #ifdef CONFIG_EROFS_FS_ZIP
> struct {
> unsigned short z_advise;
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-20 9:04 ` Chao Yu
0 siblings, 0 replies; 37+ messages in thread
From: Chao Yu @ 2021-08-20 9:04 UTC (permalink / raw)
To: Gao Xiang, linux-erofs, Liu Bo
Cc: Tao Ma, LKML, Peng Tao, Joseph Qi, Eryu Guan, Liu Jiang
On 2021/8/19 14:33, Gao Xiang wrote:
> Add runtime support for chunk-based uncompressed files
> described in the previous patch.
>
> Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
> Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
> ---
> changes since v1:
> - use le16_to_cpu instead of __le16_to_cpu pointed out by Chao.
>
> fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
> fs/erofs/inode.c | 18 ++++++++-
> fs/erofs/internal.h | 5 +++
> 3 files changed, 102 insertions(+), 11 deletions(-)
>
> diff --git a/fs/erofs/data.c b/fs/erofs/data.c
> index 09c46fbdb9b2..ee9a33485313 100644
> --- a/fs/erofs/data.c
> +++ b/fs/erofs/data.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "internal.h"
> #include <linux/prefetch.h>
> @@ -36,13 +37,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
> lastblk = nblocks - tailendpacking;
>
> - if (offset >= inode->i_size) {
> - /* leave out-of-bound access unmapped */
> - map->m_flags = 0;
> - map->m_plen = 0;
> - goto out;
> - }
> -
> /* there is no hole in flatmode */
> map->m_flags = EROFS_MAP_MAPPED;
>
> @@ -77,14 +71,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
> goto err_out;
> }
>
> -out:
> map->m_llen = map->m_plen;
> -
> err_out:
> trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
> return err;
> }
>
> +static int erofs_map_blocks(struct inode *inode,
> + struct erofs_map_blocks *map, int flags)
> +{
> + struct super_block *sb = inode->i_sb;
> + struct erofs_inode *vi = EROFS_I(inode);
> + struct erofs_inode_chunk_index *idx;
> + struct page *page;
> + u64 chunknr;
> + unsigned int unit;
> + erofs_off_t pos;
> + int err = 0;
> +
> + if (map->m_la >= inode->i_size) {
> + /* leave out-of-bound access unmapped */
> + map->m_flags = 0;
> + map->m_plen = 0;
> + goto out;
> + }
> +
> + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
> + return erofs_map_blocks_flatmode(inode, map, flags);
> +
> + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
> + unit = sizeof(*idx); /* chunk index */
> + else
> + unit = 4; /* block map */
You mean sizeof(__le32)?
Otherwise it looks good to me.
Reviewed-by: Chao Yu <chao@kernel.org>
Thanks,
> +
> + chunknr = map->m_la >> vi->chunkbits;
> + pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
> + vi->xattr_isize, unit) + unit * chunknr;
> +
> + page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
> + if (IS_ERR(page))
> + return PTR_ERR(page);
> +
> + map->m_la = chunknr << vi->chunkbits;
> + map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
> + roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
> +
> + /* handle block map */
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
> + __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
> +
> + if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
> + map->m_flags = 0;
> + } else {
> + map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + }
> + goto out_unlock;
> + }
> + /* parse chunk indexes */
> + idx = page_address(page) + erofs_blkoff(pos);
> + switch (le32_to_cpu(idx->blkaddr)) {
> + case EROFS_NULL_ADDR:
> + map->m_flags = 0;
> + break;
> + default:
> + /* only one device is supported for now */
> + if (idx->device_id) {
> + erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
> + le32_to_cpu(idx->device_id),
> + chunknr, vi->nid);
> + err = -EFSCORRUPTED;
> + goto out_unlock;
> + }
> + map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
> + map->m_flags = EROFS_MAP_MAPPED;
> + break;
> + }
> +out_unlock:
> + unlock_page(page);
> + put_page(page);
> +out:
> + map->m_llen = map->m_plen;
> + return err;
> +}
> +
> static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
> {
> @@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> map.m_la = offset;
> map.m_llen = length;
>
> - ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
> + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
> if (ret < 0)
> return ret;
>
> diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
> index d13e0709599c..4408929bd6f5 100644
> --- a/fs/erofs/inode.c
> +++ b/fs/erofs/inode.c
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #include "xattr.h"
>
> @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
> /* total blocks for compressed files */
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(die->i_u.compressed_blocks);
> -
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + /* fill chunked inode summary info */
> + vi->chunkformat = le16_to_cpu(die->i_u.c.format);
> kfree(copied);
> break;
> case EROFS_INODE_LAYOUT_COMPACT:
> @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> inode->i_size = le32_to_cpu(dic->i_size);
> if (erofs_inode_is_data_compressed(vi->datalayout))
> nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> + vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
> break;
> default:
> erofs_err(inode->i_sb,
> @@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
> goto err_out;
> }
>
> + if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
> + erofs_err(inode->i_sb,
> + "unsupported chunk format %x of nid %llu",
> + vi->chunkformat, vi->nid);
> + err = -EOPNOTSUPP;
> + goto err_out;
> + }
> + vi->chunkbits = LOG_BLOCK_SIZE +
> + (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
> + }
> inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
> inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
> index 91089ab8a816..9524e155b38f 100644
> --- a/fs/erofs/internal.h
> +++ b/fs/erofs/internal.h
> @@ -2,6 +2,7 @@
> /*
> * Copyright (C) 2017-2018 HUAWEI, Inc.
> * https://www.huawei.com/
> + * Copyright (C) 2021, Alibaba Cloud
> */
> #ifndef __EROFS_INTERNAL_H
> #define __EROFS_INTERNAL_H
> @@ -261,6 +262,10 @@ struct erofs_inode {
>
> union {
> erofs_blk_t raw_blkaddr;
> + struct {
> + unsigned short chunkformat;
> + unsigned char chunkbits;
> + };
> #ifdef CONFIG_EROFS_FS_ZIP
> struct {
> unsigned short z_advise;
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
2021-08-20 9:04 ` Chao Yu
@ 2021-08-20 9:12 ` Gao Xiang
-1 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-20 9:12 UTC (permalink / raw)
To: Chao Yu
Cc: linux-erofs, Liu Bo, LKML, Peng Tao, Eryu Guan, Liu Jiang,
Joseph Qi, Tao Ma
Hi Chao,
On Fri, Aug 20, 2021 at 05:04:13PM +0800, Chao Yu wrote:
> On 2021/8/19 14:33, Gao Xiang wrote:
...
> > }
> > +static int erofs_map_blocks(struct inode *inode,
> > + struct erofs_map_blocks *map, int flags)
> > +{
> > + struct super_block *sb = inode->i_sb;
> > + struct erofs_inode *vi = EROFS_I(inode);
> > + struct erofs_inode_chunk_index *idx;
> > + struct page *page;
> > + u64 chunknr;
> > + unsigned int unit;
> > + erofs_off_t pos;
> > + int err = 0;
> > +
> > + if (map->m_la >= inode->i_size) {
> > + /* leave out-of-bound access unmapped */
> > + map->m_flags = 0;
> > + map->m_plen = 0;
> > + goto out;
> > + }
> > +
> > + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
> > + return erofs_map_blocks_flatmode(inode, map, flags);
> > +
> > + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
> > + unit = sizeof(*idx); /* chunk index */
> > + else
> > + unit = 4; /* block map */
>
> You mean sizeof(__le32)?
Yeah, sizeof(__le32) == 4, either way works for me.
If some tendency about this, I will update when applying.
>
> Otherwise it looks good to me.
>
> Reviewed-by: Chao Yu <chao@kernel.org>
>
Thanks for the review!
Thanks,
Gao Xiang
> Thanks,
>
> > +
> > + chunknr = map->m_la >> vi->chunkbits;
> > + pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
> > + vi->xattr_isize, unit) + unit * chunknr;
> > +
> > + page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
> > + if (IS_ERR(page))
> > + return PTR_ERR(page);
> > +
> > + map->m_la = chunknr << vi->chunkbits;
> > + map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
> > + roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
> > +
> > + /* handle block map */
> > + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
> > + __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
> > +
> > + if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
> > + map->m_flags = 0;
> > + } else {
> > + map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
> > + map->m_flags = EROFS_MAP_MAPPED;
> > + }
> > + goto out_unlock;
> > + }
> > + /* parse chunk indexes */
> > + idx = page_address(page) + erofs_blkoff(pos);
> > + switch (le32_to_cpu(idx->blkaddr)) {
> > + case EROFS_NULL_ADDR:
> > + map->m_flags = 0;
> > + break;
> > + default:
> > + /* only one device is supported for now */
> > + if (idx->device_id) {
> > + erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
> > + le32_to_cpu(idx->device_id),
> > + chunknr, vi->nid);
> > + err = -EFSCORRUPTED;
> > + goto out_unlock;
> > + }
> > + map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
> > + map->m_flags = EROFS_MAP_MAPPED;
> > + break;
> > + }
> > +out_unlock:
> > + unlock_page(page);
> > + put_page(page);
> > +out:
> > + map->m_llen = map->m_plen;
> > + return err;
> > +}
> > +
> > static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> > unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
> > {
> > @@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> > map.m_la = offset;
> > map.m_llen = length;
> > - ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
> > + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
> > if (ret < 0)
> > return ret;
> > diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
> > index d13e0709599c..4408929bd6f5 100644
> > --- a/fs/erofs/inode.c
> > +++ b/fs/erofs/inode.c
> > @@ -2,6 +2,7 @@
> > /*
> > * Copyright (C) 2017-2018 HUAWEI, Inc.
> > * https://www.huawei.com/
> > + * Copyright (C) 2021, Alibaba Cloud
> > */
> > #include "xattr.h"
> > @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
> > /* total blocks for compressed files */
> > if (erofs_inode_is_data_compressed(vi->datalayout))
> > nblks = le32_to_cpu(die->i_u.compressed_blocks);
> > -
> > + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> > + /* fill chunked inode summary info */
> > + vi->chunkformat = le16_to_cpu(die->i_u.c.format);
> > kfree(copied);
> > break;
> > case EROFS_INODE_LAYOUT_COMPACT:
> > @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> > inode->i_size = le32_to_cpu(dic->i_size);
> > if (erofs_inode_is_data_compressed(vi->datalayout))
> > nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> > + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> > + vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
> > break;
> > default:
> > erofs_err(inode->i_sb,
> > @@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
> > goto err_out;
> > }
> > + if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
> > + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
> > + erofs_err(inode->i_sb,
> > + "unsupported chunk format %x of nid %llu",
> > + vi->chunkformat, vi->nid);
> > + err = -EOPNOTSUPP;
> > + goto err_out;
> > + }
> > + vi->chunkbits = LOG_BLOCK_SIZE +
> > + (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
> > + }
> > inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
> > inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
> > inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
> > diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
> > index 91089ab8a816..9524e155b38f 100644
> > --- a/fs/erofs/internal.h
> > +++ b/fs/erofs/internal.h
> > @@ -2,6 +2,7 @@
> > /*
> > * Copyright (C) 2017-2018 HUAWEI, Inc.
> > * https://www.huawei.com/
> > + * Copyright (C) 2021, Alibaba Cloud
> > */
> > #ifndef __EROFS_INTERNAL_H
> > #define __EROFS_INTERNAL_H
> > @@ -261,6 +262,10 @@ struct erofs_inode {
> > union {
> > erofs_blk_t raw_blkaddr;
> > + struct {
> > + unsigned short chunkformat;
> > + unsigned char chunkbits;
> > + };
> > #ifdef CONFIG_EROFS_FS_ZIP
> > struct {
> > unsigned short z_advise;
> >
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-20 9:12 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-20 9:12 UTC (permalink / raw)
To: Chao Yu
Cc: Tao Ma, LKML, Peng Tao, Joseph Qi, Liu Bo, Eryu Guan, Liu Jiang,
linux-erofs
Hi Chao,
On Fri, Aug 20, 2021 at 05:04:13PM +0800, Chao Yu wrote:
> On 2021/8/19 14:33, Gao Xiang wrote:
...
> > }
> > +static int erofs_map_blocks(struct inode *inode,
> > + struct erofs_map_blocks *map, int flags)
> > +{
> > + struct super_block *sb = inode->i_sb;
> > + struct erofs_inode *vi = EROFS_I(inode);
> > + struct erofs_inode_chunk_index *idx;
> > + struct page *page;
> > + u64 chunknr;
> > + unsigned int unit;
> > + erofs_off_t pos;
> > + int err = 0;
> > +
> > + if (map->m_la >= inode->i_size) {
> > + /* leave out-of-bound access unmapped */
> > + map->m_flags = 0;
> > + map->m_plen = 0;
> > + goto out;
> > + }
> > +
> > + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
> > + return erofs_map_blocks_flatmode(inode, map, flags);
> > +
> > + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
> > + unit = sizeof(*idx); /* chunk index */
> > + else
> > + unit = 4; /* block map */
>
> You mean sizeof(__le32)?
Yeah, sizeof(__le32) == 4, either way works for me.
If some tendency about this, I will update when applying.
>
> Otherwise it looks good to me.
>
> Reviewed-by: Chao Yu <chao@kernel.org>
>
Thanks for the review!
Thanks,
Gao Xiang
> Thanks,
>
> > +
> > + chunknr = map->m_la >> vi->chunkbits;
> > + pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
> > + vi->xattr_isize, unit) + unit * chunknr;
> > +
> > + page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
> > + if (IS_ERR(page))
> > + return PTR_ERR(page);
> > +
> > + map->m_la = chunknr << vi->chunkbits;
> > + map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
> > + roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
> > +
> > + /* handle block map */
> > + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
> > + __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
> > +
> > + if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
> > + map->m_flags = 0;
> > + } else {
> > + map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
> > + map->m_flags = EROFS_MAP_MAPPED;
> > + }
> > + goto out_unlock;
> > + }
> > + /* parse chunk indexes */
> > + idx = page_address(page) + erofs_blkoff(pos);
> > + switch (le32_to_cpu(idx->blkaddr)) {
> > + case EROFS_NULL_ADDR:
> > + map->m_flags = 0;
> > + break;
> > + default:
> > + /* only one device is supported for now */
> > + if (idx->device_id) {
> > + erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
> > + le32_to_cpu(idx->device_id),
> > + chunknr, vi->nid);
> > + err = -EFSCORRUPTED;
> > + goto out_unlock;
> > + }
> > + map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
> > + map->m_flags = EROFS_MAP_MAPPED;
> > + break;
> > + }
> > +out_unlock:
> > + unlock_page(page);
> > + put_page(page);
> > +out:
> > + map->m_llen = map->m_plen;
> > + return err;
> > +}
> > +
> > static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> > unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
> > {
> > @@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
> > map.m_la = offset;
> > map.m_llen = length;
> > - ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
> > + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
> > if (ret < 0)
> > return ret;
> > diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
> > index d13e0709599c..4408929bd6f5 100644
> > --- a/fs/erofs/inode.c
> > +++ b/fs/erofs/inode.c
> > @@ -2,6 +2,7 @@
> > /*
> > * Copyright (C) 2017-2018 HUAWEI, Inc.
> > * https://www.huawei.com/
> > + * Copyright (C) 2021, Alibaba Cloud
> > */
> > #include "xattr.h"
> > @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
> > /* total blocks for compressed files */
> > if (erofs_inode_is_data_compressed(vi->datalayout))
> > nblks = le32_to_cpu(die->i_u.compressed_blocks);
> > -
> > + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> > + /* fill chunked inode summary info */
> > + vi->chunkformat = le16_to_cpu(die->i_u.c.format);
> > kfree(copied);
> > break;
> > case EROFS_INODE_LAYOUT_COMPACT:
> > @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
> > inode->i_size = le32_to_cpu(dic->i_size);
> > if (erofs_inode_is_data_compressed(vi->datalayout))
> > nblks = le32_to_cpu(dic->i_u.compressed_blocks);
> > + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
> > + vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
> > break;
> > default:
> > erofs_err(inode->i_sb,
> > @@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
> > goto err_out;
> > }
> > + if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
> > + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
> > + erofs_err(inode->i_sb,
> > + "unsupported chunk format %x of nid %llu",
> > + vi->chunkformat, vi->nid);
> > + err = -EOPNOTSUPP;
> > + goto err_out;
> > + }
> > + vi->chunkbits = LOG_BLOCK_SIZE +
> > + (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
> > + }
> > inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
> > inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
> > inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
> > diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
> > index 91089ab8a816..9524e155b38f 100644
> > --- a/fs/erofs/internal.h
> > +++ b/fs/erofs/internal.h
> > @@ -2,6 +2,7 @@
> > /*
> > * Copyright (C) 2017-2018 HUAWEI, Inc.
> > * https://www.huawei.com/
> > + * Copyright (C) 2021, Alibaba Cloud
> > */
> > #ifndef __EROFS_INTERNAL_H
> > #define __EROFS_INTERNAL_H
> > @@ -261,6 +262,10 @@ struct erofs_inode {
> > union {
> > erofs_blk_t raw_blkaddr;
> > + struct {
> > + unsigned short chunkformat;
> > + unsigned char chunkbits;
> > + };
> > #ifdef CONFIG_EROFS_FS_ZIP
> > struct {
> > unsigned short z_advise;
> >
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
2021-08-20 9:12 ` Gao Xiang
@ 2021-08-20 9:29 ` Chao Yu
-1 siblings, 0 replies; 37+ messages in thread
From: Chao Yu @ 2021-08-20 9:29 UTC (permalink / raw)
To: Gao Xiang
Cc: linux-erofs, Liu Bo, LKML, Peng Tao, Eryu Guan, Liu Jiang,
Joseph Qi, Tao Ma
On 2021/8/20 17:12, Gao Xiang wrote:
> Hi Chao,
>
> On Fri, Aug 20, 2021 at 05:04:13PM +0800, Chao Yu wrote:
>> On 2021/8/19 14:33, Gao Xiang wrote:
>
> ...
>
>>> }
>>> +static int erofs_map_blocks(struct inode *inode,
>>> + struct erofs_map_blocks *map, int flags)
>>> +{
>>> + struct super_block *sb = inode->i_sb;
>>> + struct erofs_inode *vi = EROFS_I(inode);
>>> + struct erofs_inode_chunk_index *idx;
>>> + struct page *page;
>>> + u64 chunknr;
>>> + unsigned int unit;
>>> + erofs_off_t pos;
>>> + int err = 0;
>>> +
>>> + if (map->m_la >= inode->i_size) {
>>> + /* leave out-of-bound access unmapped */
>>> + map->m_flags = 0;
>>> + map->m_plen = 0;
>>> + goto out;
>>> + }
>>> +
>>> + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
>>> + return erofs_map_blocks_flatmode(inode, map, flags);
>>> +
>>> + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
>>> + unit = sizeof(*idx); /* chunk index */
>>> + else
>>> + unit = 4; /* block map */
>>
>> You mean sizeof(__le32)?
>
> Yeah, sizeof(__le32) == 4, either way works for me.
>
> If some tendency about this, I will update when applying.
Xiang,
Yeah, I preper:
#define EROFS_BLOCK_MAP_ENTRY_SIZE sizeof(__le32)
unit = EROFS_BLOCK_MAP_ENTRY_SIZE;
to improve readablity, but unit = sizeof(__le32) is fine as well.
Thanks,
>
>>
>> Otherwise it looks good to me.
>>
>> Reviewed-by: Chao Yu <chao@kernel.org>
>>
>
> Thanks for the review!
>
> Thanks,
> Gao Xiang
>
>> Thanks,
>>
>>> +
>>> + chunknr = map->m_la >> vi->chunkbits;
>>> + pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
>>> + vi->xattr_isize, unit) + unit * chunknr;
>>> +
>>> + page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
>>> + if (IS_ERR(page))
>>> + return PTR_ERR(page);
>>> +
>>> + map->m_la = chunknr << vi->chunkbits;
>>> + map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
>>> + roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
>>> +
>>> + /* handle block map */
>>> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
>>> + __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
>>> +
>>> + if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
>>> + map->m_flags = 0;
>>> + } else {
>>> + map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
>>> + map->m_flags = EROFS_MAP_MAPPED;
>>> + }
>>> + goto out_unlock;
>>> + }
>>> + /* parse chunk indexes */
>>> + idx = page_address(page) + erofs_blkoff(pos);
>>> + switch (le32_to_cpu(idx->blkaddr)) {
>>> + case EROFS_NULL_ADDR:
>>> + map->m_flags = 0;
>>> + break;
>>> + default:
>>> + /* only one device is supported for now */
>>> + if (idx->device_id) {
>>> + erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
>>> + le32_to_cpu(idx->device_id),
>>> + chunknr, vi->nid);
>>> + err = -EFSCORRUPTED;
>>> + goto out_unlock;
>>> + }
>>> + map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
>>> + map->m_flags = EROFS_MAP_MAPPED;
>>> + break;
>>> + }
>>> +out_unlock:
>>> + unlock_page(page);
>>> + put_page(page);
>>> +out:
>>> + map->m_llen = map->m_plen;
>>> + return err;
>>> +}
>>> +
>>> static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
>>> unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
>>> {
>>> @@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
>>> map.m_la = offset;
>>> map.m_llen = length;
>>> - ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
>>> + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
>>> if (ret < 0)
>>> return ret;
>>> diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
>>> index d13e0709599c..4408929bd6f5 100644
>>> --- a/fs/erofs/inode.c
>>> +++ b/fs/erofs/inode.c
>>> @@ -2,6 +2,7 @@
>>> /*
>>> * Copyright (C) 2017-2018 HUAWEI, Inc.
>>> * https://www.huawei.com/
>>> + * Copyright (C) 2021, Alibaba Cloud
>>> */
>>> #include "xattr.h"
>>> @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
>>> /* total blocks for compressed files */
>>> if (erofs_inode_is_data_compressed(vi->datalayout))
>>> nblks = le32_to_cpu(die->i_u.compressed_blocks);
>>> -
>>> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
>>> + /* fill chunked inode summary info */
>>> + vi->chunkformat = le16_to_cpu(die->i_u.c.format);
>>> kfree(copied);
>>> break;
>>> case EROFS_INODE_LAYOUT_COMPACT:
>>> @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
>>> inode->i_size = le32_to_cpu(dic->i_size);
>>> if (erofs_inode_is_data_compressed(vi->datalayout))
>>> nblks = le32_to_cpu(dic->i_u.compressed_blocks);
>>> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
>>> + vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
>>> break;
>>> default:
>>> erofs_err(inode->i_sb,
>>> @@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
>>> goto err_out;
>>> }
>>> + if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
>>> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
>>> + erofs_err(inode->i_sb,
>>> + "unsupported chunk format %x of nid %llu",
>>> + vi->chunkformat, vi->nid);
>>> + err = -EOPNOTSUPP;
>>> + goto err_out;
>>> + }
>>> + vi->chunkbits = LOG_BLOCK_SIZE +
>>> + (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
>>> + }
>>> inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
>>> inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
>>> inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
>>> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
>>> index 91089ab8a816..9524e155b38f 100644
>>> --- a/fs/erofs/internal.h
>>> +++ b/fs/erofs/internal.h
>>> @@ -2,6 +2,7 @@
>>> /*
>>> * Copyright (C) 2017-2018 HUAWEI, Inc.
>>> * https://www.huawei.com/
>>> + * Copyright (C) 2021, Alibaba Cloud
>>> */
>>> #ifndef __EROFS_INTERNAL_H
>>> #define __EROFS_INTERNAL_H
>>> @@ -261,6 +262,10 @@ struct erofs_inode {
>>> union {
>>> erofs_blk_t raw_blkaddr;
>>> + struct {
>>> + unsigned short chunkformat;
>>> + unsigned char chunkbits;
>>> + };
>>> #ifdef CONFIG_EROFS_FS_ZIP
>>> struct {
>>> unsigned short z_advise;
>>>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-20 9:29 ` Chao Yu
0 siblings, 0 replies; 37+ messages in thread
From: Chao Yu @ 2021-08-20 9:29 UTC (permalink / raw)
To: Gao Xiang
Cc: Tao Ma, LKML, Peng Tao, Joseph Qi, Liu Bo, Eryu Guan, Liu Jiang,
linux-erofs
On 2021/8/20 17:12, Gao Xiang wrote:
> Hi Chao,
>
> On Fri, Aug 20, 2021 at 05:04:13PM +0800, Chao Yu wrote:
>> On 2021/8/19 14:33, Gao Xiang wrote:
>
> ...
>
>>> }
>>> +static int erofs_map_blocks(struct inode *inode,
>>> + struct erofs_map_blocks *map, int flags)
>>> +{
>>> + struct super_block *sb = inode->i_sb;
>>> + struct erofs_inode *vi = EROFS_I(inode);
>>> + struct erofs_inode_chunk_index *idx;
>>> + struct page *page;
>>> + u64 chunknr;
>>> + unsigned int unit;
>>> + erofs_off_t pos;
>>> + int err = 0;
>>> +
>>> + if (map->m_la >= inode->i_size) {
>>> + /* leave out-of-bound access unmapped */
>>> + map->m_flags = 0;
>>> + map->m_plen = 0;
>>> + goto out;
>>> + }
>>> +
>>> + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
>>> + return erofs_map_blocks_flatmode(inode, map, flags);
>>> +
>>> + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
>>> + unit = sizeof(*idx); /* chunk index */
>>> + else
>>> + unit = 4; /* block map */
>>
>> You mean sizeof(__le32)?
>
> Yeah, sizeof(__le32) == 4, either way works for me.
>
> If some tendency about this, I will update when applying.
Xiang,
Yeah, I preper:
#define EROFS_BLOCK_MAP_ENTRY_SIZE sizeof(__le32)
unit = EROFS_BLOCK_MAP_ENTRY_SIZE;
to improve readablity, but unit = sizeof(__le32) is fine as well.
Thanks,
>
>>
>> Otherwise it looks good to me.
>>
>> Reviewed-by: Chao Yu <chao@kernel.org>
>>
>
> Thanks for the review!
>
> Thanks,
> Gao Xiang
>
>> Thanks,
>>
>>> +
>>> + chunknr = map->m_la >> vi->chunkbits;
>>> + pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
>>> + vi->xattr_isize, unit) + unit * chunknr;
>>> +
>>> + page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
>>> + if (IS_ERR(page))
>>> + return PTR_ERR(page);
>>> +
>>> + map->m_la = chunknr << vi->chunkbits;
>>> + map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
>>> + roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
>>> +
>>> + /* handle block map */
>>> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
>>> + __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
>>> +
>>> + if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
>>> + map->m_flags = 0;
>>> + } else {
>>> + map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
>>> + map->m_flags = EROFS_MAP_MAPPED;
>>> + }
>>> + goto out_unlock;
>>> + }
>>> + /* parse chunk indexes */
>>> + idx = page_address(page) + erofs_blkoff(pos);
>>> + switch (le32_to_cpu(idx->blkaddr)) {
>>> + case EROFS_NULL_ADDR:
>>> + map->m_flags = 0;
>>> + break;
>>> + default:
>>> + /* only one device is supported for now */
>>> + if (idx->device_id) {
>>> + erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
>>> + le32_to_cpu(idx->device_id),
>>> + chunknr, vi->nid);
>>> + err = -EFSCORRUPTED;
>>> + goto out_unlock;
>>> + }
>>> + map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
>>> + map->m_flags = EROFS_MAP_MAPPED;
>>> + break;
>>> + }
>>> +out_unlock:
>>> + unlock_page(page);
>>> + put_page(page);
>>> +out:
>>> + map->m_llen = map->m_plen;
>>> + return err;
>>> +}
>>> +
>>> static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
>>> unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
>>> {
>>> @@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
>>> map.m_la = offset;
>>> map.m_llen = length;
>>> - ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
>>> + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
>>> if (ret < 0)
>>> return ret;
>>> diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
>>> index d13e0709599c..4408929bd6f5 100644
>>> --- a/fs/erofs/inode.c
>>> +++ b/fs/erofs/inode.c
>>> @@ -2,6 +2,7 @@
>>> /*
>>> * Copyright (C) 2017-2018 HUAWEI, Inc.
>>> * https://www.huawei.com/
>>> + * Copyright (C) 2021, Alibaba Cloud
>>> */
>>> #include "xattr.h"
>>> @@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
>>> /* total blocks for compressed files */
>>> if (erofs_inode_is_data_compressed(vi->datalayout))
>>> nblks = le32_to_cpu(die->i_u.compressed_blocks);
>>> -
>>> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
>>> + /* fill chunked inode summary info */
>>> + vi->chunkformat = le16_to_cpu(die->i_u.c.format);
>>> kfree(copied);
>>> break;
>>> case EROFS_INODE_LAYOUT_COMPACT:
>>> @@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
>>> inode->i_size = le32_to_cpu(dic->i_size);
>>> if (erofs_inode_is_data_compressed(vi->datalayout))
>>> nblks = le32_to_cpu(dic->i_u.compressed_blocks);
>>> + else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
>>> + vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
>>> break;
>>> default:
>>> erofs_err(inode->i_sb,
>>> @@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
>>> goto err_out;
>>> }
>>> + if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
>>> + if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
>>> + erofs_err(inode->i_sb,
>>> + "unsupported chunk format %x of nid %llu",
>>> + vi->chunkformat, vi->nid);
>>> + err = -EOPNOTSUPP;
>>> + goto err_out;
>>> + }
>>> + vi->chunkbits = LOG_BLOCK_SIZE +
>>> + (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
>>> + }
>>> inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
>>> inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
>>> inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
>>> diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
>>> index 91089ab8a816..9524e155b38f 100644
>>> --- a/fs/erofs/internal.h
>>> +++ b/fs/erofs/internal.h
>>> @@ -2,6 +2,7 @@
>>> /*
>>> * Copyright (C) 2017-2018 HUAWEI, Inc.
>>> * https://www.huawei.com/
>>> + * Copyright (C) 2021, Alibaba Cloud
>>> */
>>> #ifndef __EROFS_INTERNAL_H
>>> #define __EROFS_INTERNAL_H
>>> @@ -261,6 +262,10 @@ struct erofs_inode {
>>> union {
>>> erofs_blk_t raw_blkaddr;
>>> + struct {
>>> + unsigned short chunkformat;
>>> + unsigned char chunkbits;
>>> + };
>>> #ifdef CONFIG_EROFS_FS_ZIP
>>> struct {
>>> unsigned short z_advise;
>>>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
2021-08-20 9:29 ` Chao Yu
@ 2021-08-20 9:33 ` Gao Xiang
-1 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-20 9:33 UTC (permalink / raw)
To: Chao Yu
Cc: linux-erofs, Liu Bo, LKML, Peng Tao, Eryu Guan, Liu Jiang,
Joseph Qi, Tao Ma
On Fri, Aug 20, 2021 at 05:29:36PM +0800, Chao Yu wrote:
> On 2021/8/20 17:12, Gao Xiang wrote:
> > Hi Chao,
> >
> > On Fri, Aug 20, 2021 at 05:04:13PM +0800, Chao Yu wrote:
> > > On 2021/8/19 14:33, Gao Xiang wrote:
> >
> > ...
> >
> > > > }
> > > > +static int erofs_map_blocks(struct inode *inode,
> > > > + struct erofs_map_blocks *map, int flags)
> > > > +{
> > > > + struct super_block *sb = inode->i_sb;
> > > > + struct erofs_inode *vi = EROFS_I(inode);
> > > > + struct erofs_inode_chunk_index *idx;
> > > > + struct page *page;
> > > > + u64 chunknr;
> > > > + unsigned int unit;
> > > > + erofs_off_t pos;
> > > > + int err = 0;
> > > > +
> > > > + if (map->m_la >= inode->i_size) {
> > > > + /* leave out-of-bound access unmapped */
> > > > + map->m_flags = 0;
> > > > + map->m_plen = 0;
> > > > + goto out;
> > > > + }
> > > > +
> > > > + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
> > > > + return erofs_map_blocks_flatmode(inode, map, flags);
> > > > +
> > > > + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
> > > > + unit = sizeof(*idx); /* chunk index */
> > > > + else
> > > > + unit = 4; /* block map */
> > >
> > > You mean sizeof(__le32)?
> >
> > Yeah, sizeof(__le32) == 4, either way works for me.
> >
> > If some tendency about this, I will update when applying.
>
> Xiang,
>
> Yeah, I preper:
>
> #define EROFS_BLOCK_MAP_ENTRY_SIZE sizeof(__le32)
>
> unit = EROFS_BLOCK_MAP_ENTRY_SIZE;
>
> to improve readablity, but unit = sizeof(__le32) is fine as well.
Ok, looks much better, let me revise v3 here.
Thanks,
Gao Xiang
>
> Thanks,
>
> >
> > >
> > > Otherwise it looks good to me.
> > >
> > > Reviewed-by: Chao Yu <chao@kernel.org>
> > >
> >
> > Thanks for the review!
> >
> > Thanks,
> > Gao Xiang
> >
> > > Thanks,
> > >
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-20 9:33 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-20 9:33 UTC (permalink / raw)
To: Chao Yu
Cc: Tao Ma, LKML, Peng Tao, Joseph Qi, Liu Bo, Eryu Guan, Liu Jiang,
linux-erofs
On Fri, Aug 20, 2021 at 05:29:36PM +0800, Chao Yu wrote:
> On 2021/8/20 17:12, Gao Xiang wrote:
> > Hi Chao,
> >
> > On Fri, Aug 20, 2021 at 05:04:13PM +0800, Chao Yu wrote:
> > > On 2021/8/19 14:33, Gao Xiang wrote:
> >
> > ...
> >
> > > > }
> > > > +static int erofs_map_blocks(struct inode *inode,
> > > > + struct erofs_map_blocks *map, int flags)
> > > > +{
> > > > + struct super_block *sb = inode->i_sb;
> > > > + struct erofs_inode *vi = EROFS_I(inode);
> > > > + struct erofs_inode_chunk_index *idx;
> > > > + struct page *page;
> > > > + u64 chunknr;
> > > > + unsigned int unit;
> > > > + erofs_off_t pos;
> > > > + int err = 0;
> > > > +
> > > > + if (map->m_la >= inode->i_size) {
> > > > + /* leave out-of-bound access unmapped */
> > > > + map->m_flags = 0;
> > > > + map->m_plen = 0;
> > > > + goto out;
> > > > + }
> > > > +
> > > > + if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
> > > > + return erofs_map_blocks_flatmode(inode, map, flags);
> > > > +
> > > > + if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
> > > > + unit = sizeof(*idx); /* chunk index */
> > > > + else
> > > > + unit = 4; /* block map */
> > >
> > > You mean sizeof(__le32)?
> >
> > Yeah, sizeof(__le32) == 4, either way works for me.
> >
> > If some tendency about this, I will update when applying.
>
> Xiang,
>
> Yeah, I preper:
>
> #define EROFS_BLOCK_MAP_ENTRY_SIZE sizeof(__le32)
>
> unit = EROFS_BLOCK_MAP_ENTRY_SIZE;
>
> to improve readablity, but unit = sizeof(__le32) is fine as well.
Ok, looks much better, let me revise v3 here.
Thanks,
Gao Xiang
>
> Thanks,
>
> >
> > >
> > > Otherwise it looks good to me.
> > >
> > > Reviewed-by: Chao Yu <chao@kernel.org>
> > >
> >
> > Thanks for the review!
> >
> > Thanks,
> > Gao Xiang
> >
> > > Thanks,
> > >
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v3 1/2] erofs: introduce chunk-based file on-disk format
2021-08-19 6:33 ` Gao Xiang
@ 2021-08-20 10:00 ` Gao Xiang
-1 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-20 10:00 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi, Gao Xiang
Currently, uncompressed data except for tail-packing inline is
consecutive on disk.
In order to support chunk-based data deduplication, add a new
corresponding inode data layout.
In the future, the data source of chunks can be either (un)compressed.
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Reviewed-by: Chao Yu <chao@kernel.org>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
changes since v2:
- introduce EROFS_BLOCK_MAP_ENTRY_SIZE suggested by Chao.
Documentation/filesystems/erofs.rst | 16 ++++++++--
fs/erofs/erofs_fs.h | 47 +++++++++++++++++++++++++++--
2 files changed, 59 insertions(+), 4 deletions(-)
diff --git a/Documentation/filesystems/erofs.rst b/Documentation/filesystems/erofs.rst
index 868e3972227f..d484408a90c1 100644
--- a/Documentation/filesystems/erofs.rst
+++ b/Documentation/filesystems/erofs.rst
@@ -156,13 +156,14 @@ may not. All metadatas can be now observed in two different spaces (views):
Xattrs, extents, data inline are followed by the corresponding inode with
proper alignment, and they could be optional for different data mappings.
- _currently_ total 4 valid data mappings are supported:
+ _currently_ total 5 data layouts are supported:
== ====================================================================
0 flat file data without data inline (no extent);
1 fixed-sized output data compression (with non-compacted indexes);
2 flat file data with tail packing data inline (no extent);
- 3 fixed-sized output data compression (with compacted indexes, v5.3+).
+ 3 fixed-sized output data compression (with compacted indexes, v5.3+);
+ 4 chunk-based file (v5.15+).
== ====================================================================
The size of the optional xattrs is indicated by i_xattr_count in inode
@@ -213,6 +214,17 @@ Note that apart from the offset of the first filename, nameoff0 also indicates
the total number of directory entries in this block since it is no need to
introduce another on-disk field at all.
+Chunk-based file
+----------------
+In order to support chunk-based data deduplication, a new inode data layout has
+been supported since Linux v5.15: Files are split in equal-sized data chunks
+with ``extents`` area of the inode metadata indicating how to get the chunk
+data: these can be simply as a 4-byte block address array or in the 8-byte
+chunk index form (see struct erofs_inode_chunk_index in erofs_fs.h for more
+details.)
+
+By the way, chunk-based files are all uncompressed for now.
+
Data compression
----------------
EROFS implements LZ4 fixed-sized output compression which generates fixed-sized
diff --git a/fs/erofs/erofs_fs.h b/fs/erofs/erofs_fs.h
index 0f8da74570b4..b0b23f41abc3 100644
--- a/fs/erofs/erofs_fs.h
+++ b/fs/erofs/erofs_fs.h
@@ -4,6 +4,7 @@
*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_FS_H
#define __EROFS_FS_H
@@ -19,10 +20,12 @@
#define EROFS_FEATURE_INCOMPAT_LZ4_0PADDING 0x00000001
#define EROFS_FEATURE_INCOMPAT_COMPR_CFGS 0x00000002
#define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x00000002
+#define EROFS_FEATURE_INCOMPAT_CHUNKED_FILE 0x00000004
#define EROFS_ALL_FEATURE_INCOMPAT \
(EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \
EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
- EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)
+ EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER | \
+ EROFS_FEATURE_INCOMPAT_CHUNKED_FILE)
#define EROFS_SB_EXTSLOT_SIZE 16
@@ -64,13 +67,16 @@ struct erofs_super_block {
* inode, [xattrs], last_inline_data, ... | ... | no-holed data
* 3 - inode compression D:
* inode, [xattrs], map_header, extents ... | ...
- * 4~7 - reserved
+ * 4 - inode chunk-based E:
+ * inode, [xattrs], chunk indexes ... | ...
+ * 5~7 - reserved
*/
enum {
EROFS_INODE_FLAT_PLAIN = 0,
EROFS_INODE_FLAT_COMPRESSION_LEGACY = 1,
EROFS_INODE_FLAT_INLINE = 2,
EROFS_INODE_FLAT_COMPRESSION = 3,
+ EROFS_INODE_CHUNK_BASED = 4,
EROFS_INODE_DATALAYOUT_MAX
};
@@ -90,6 +96,19 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
#define EROFS_I_ALL \
((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
+/* indicate chunk blkbits, thus 'chunksize = blocksize << chunk blkbits' */
+#define EROFS_CHUNK_FORMAT_BLKBITS_MASK 0x001F
+/* with chunk indexes or just a 4-byte blkaddr array */
+#define EROFS_CHUNK_FORMAT_INDEXES 0x0020
+
+#define EROFS_CHUNK_FORMAT_ALL \
+ (EROFS_CHUNK_FORMAT_BLKBITS_MASK | EROFS_CHUNK_FORMAT_INDEXES)
+
+struct erofs_inode_chunk_info {
+ __le16 format; /* chunk blkbits, etc. */
+ __le16 reserved;
+};
+
/* 32-byte reduced form of an ondisk inode */
struct erofs_inode_compact {
__le16 i_format; /* inode format hints */
@@ -107,6 +126,9 @@ struct erofs_inode_compact {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
__le32 i_ino; /* only used for 32-bit stat compatibility */
__le16 i_uid;
@@ -135,6 +157,9 @@ struct erofs_inode_extended {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
/* only used for 32-bit stat compatibility */
@@ -204,6 +229,19 @@ static inline unsigned int erofs_xattr_entry_size(struct erofs_xattr_entry *e)
e->e_name_len + le16_to_cpu(e->e_value_size));
}
+/* represent a zeroed chunk (hole) */
+#define EROFS_NULL_ADDR -1
+
+/* 4-byte block address array */
+#define EROFS_BLOCK_MAP_ENTRY_SIZE sizeof(__le32)
+
+/* 8-byte inode chunk indexes */
+struct erofs_inode_chunk_index {
+ __le16 advise; /* always 0, don't care for now */
+ __le16 device_id; /* back-end storage id, always 0 for now */
+ __le32 blkaddr; /* start block address of this inode chunk */
+};
+
/* maximum supported size of a physical compression cluster */
#define Z_EROFS_PCLUSTER_MAX_SIZE (1024 * 1024)
@@ -338,9 +376,14 @@ static inline void erofs_check_ondisk_layout_definitions(void)
BUILD_BUG_ON(sizeof(struct erofs_inode_extended) != 64);
BUILD_BUG_ON(sizeof(struct erofs_xattr_ibody_header) != 12);
BUILD_BUG_ON(sizeof(struct erofs_xattr_entry) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_info) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_map_header) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_vle_decompressed_index) != 8);
BUILD_BUG_ON(sizeof(struct erofs_dirent) != 12);
+ /* keep in sync between 2 index structures for better extendibility */
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) !=
+ sizeof(struct z_erofs_vle_decompressed_index));
BUILD_BUG_ON(BIT(Z_EROFS_VLE_DI_CLUSTER_TYPE_BITS) <
Z_EROFS_VLE_CLUSTER_TYPE_MAX - 1);
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH v3 1/2] erofs: introduce chunk-based file on-disk format
@ 2021-08-20 10:00 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-20 10:00 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Joseph Qi, Eryu Guan, Liu Jiang
Currently, uncompressed data except for tail-packing inline is
consecutive on disk.
In order to support chunk-based data deduplication, add a new
corresponding inode data layout.
In the future, the data source of chunks can be either (un)compressed.
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Reviewed-by: Chao Yu <chao@kernel.org>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
changes since v2:
- introduce EROFS_BLOCK_MAP_ENTRY_SIZE suggested by Chao.
Documentation/filesystems/erofs.rst | 16 ++++++++--
fs/erofs/erofs_fs.h | 47 +++++++++++++++++++++++++++--
2 files changed, 59 insertions(+), 4 deletions(-)
diff --git a/Documentation/filesystems/erofs.rst b/Documentation/filesystems/erofs.rst
index 868e3972227f..d484408a90c1 100644
--- a/Documentation/filesystems/erofs.rst
+++ b/Documentation/filesystems/erofs.rst
@@ -156,13 +156,14 @@ may not. All metadatas can be now observed in two different spaces (views):
Xattrs, extents, data inline are followed by the corresponding inode with
proper alignment, and they could be optional for different data mappings.
- _currently_ total 4 valid data mappings are supported:
+ _currently_ total 5 data layouts are supported:
== ====================================================================
0 flat file data without data inline (no extent);
1 fixed-sized output data compression (with non-compacted indexes);
2 flat file data with tail packing data inline (no extent);
- 3 fixed-sized output data compression (with compacted indexes, v5.3+).
+ 3 fixed-sized output data compression (with compacted indexes, v5.3+);
+ 4 chunk-based file (v5.15+).
== ====================================================================
The size of the optional xattrs is indicated by i_xattr_count in inode
@@ -213,6 +214,17 @@ Note that apart from the offset of the first filename, nameoff0 also indicates
the total number of directory entries in this block since it is no need to
introduce another on-disk field at all.
+Chunk-based file
+----------------
+In order to support chunk-based data deduplication, a new inode data layout has
+been supported since Linux v5.15: Files are split in equal-sized data chunks
+with ``extents`` area of the inode metadata indicating how to get the chunk
+data: these can be simply as a 4-byte block address array or in the 8-byte
+chunk index form (see struct erofs_inode_chunk_index in erofs_fs.h for more
+details.)
+
+By the way, chunk-based files are all uncompressed for now.
+
Data compression
----------------
EROFS implements LZ4 fixed-sized output compression which generates fixed-sized
diff --git a/fs/erofs/erofs_fs.h b/fs/erofs/erofs_fs.h
index 0f8da74570b4..b0b23f41abc3 100644
--- a/fs/erofs/erofs_fs.h
+++ b/fs/erofs/erofs_fs.h
@@ -4,6 +4,7 @@
*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_FS_H
#define __EROFS_FS_H
@@ -19,10 +20,12 @@
#define EROFS_FEATURE_INCOMPAT_LZ4_0PADDING 0x00000001
#define EROFS_FEATURE_INCOMPAT_COMPR_CFGS 0x00000002
#define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x00000002
+#define EROFS_FEATURE_INCOMPAT_CHUNKED_FILE 0x00000004
#define EROFS_ALL_FEATURE_INCOMPAT \
(EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \
EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
- EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)
+ EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER | \
+ EROFS_FEATURE_INCOMPAT_CHUNKED_FILE)
#define EROFS_SB_EXTSLOT_SIZE 16
@@ -64,13 +67,16 @@ struct erofs_super_block {
* inode, [xattrs], last_inline_data, ... | ... | no-holed data
* 3 - inode compression D:
* inode, [xattrs], map_header, extents ... | ...
- * 4~7 - reserved
+ * 4 - inode chunk-based E:
+ * inode, [xattrs], chunk indexes ... | ...
+ * 5~7 - reserved
*/
enum {
EROFS_INODE_FLAT_PLAIN = 0,
EROFS_INODE_FLAT_COMPRESSION_LEGACY = 1,
EROFS_INODE_FLAT_INLINE = 2,
EROFS_INODE_FLAT_COMPRESSION = 3,
+ EROFS_INODE_CHUNK_BASED = 4,
EROFS_INODE_DATALAYOUT_MAX
};
@@ -90,6 +96,19 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
#define EROFS_I_ALL \
((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
+/* indicate chunk blkbits, thus 'chunksize = blocksize << chunk blkbits' */
+#define EROFS_CHUNK_FORMAT_BLKBITS_MASK 0x001F
+/* with chunk indexes or just a 4-byte blkaddr array */
+#define EROFS_CHUNK_FORMAT_INDEXES 0x0020
+
+#define EROFS_CHUNK_FORMAT_ALL \
+ (EROFS_CHUNK_FORMAT_BLKBITS_MASK | EROFS_CHUNK_FORMAT_INDEXES)
+
+struct erofs_inode_chunk_info {
+ __le16 format; /* chunk blkbits, etc. */
+ __le16 reserved;
+};
+
/* 32-byte reduced form of an ondisk inode */
struct erofs_inode_compact {
__le16 i_format; /* inode format hints */
@@ -107,6 +126,9 @@ struct erofs_inode_compact {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
__le32 i_ino; /* only used for 32-bit stat compatibility */
__le16 i_uid;
@@ -135,6 +157,9 @@ struct erofs_inode_extended {
/* for device files, used to indicate old/new device # */
__le32 rdev;
+
+ /* for chunk-based files, it contains the summary info */
+ struct erofs_inode_chunk_info c;
} i_u;
/* only used for 32-bit stat compatibility */
@@ -204,6 +229,19 @@ static inline unsigned int erofs_xattr_entry_size(struct erofs_xattr_entry *e)
e->e_name_len + le16_to_cpu(e->e_value_size));
}
+/* represent a zeroed chunk (hole) */
+#define EROFS_NULL_ADDR -1
+
+/* 4-byte block address array */
+#define EROFS_BLOCK_MAP_ENTRY_SIZE sizeof(__le32)
+
+/* 8-byte inode chunk indexes */
+struct erofs_inode_chunk_index {
+ __le16 advise; /* always 0, don't care for now */
+ __le16 device_id; /* back-end storage id, always 0 for now */
+ __le32 blkaddr; /* start block address of this inode chunk */
+};
+
/* maximum supported size of a physical compression cluster */
#define Z_EROFS_PCLUSTER_MAX_SIZE (1024 * 1024)
@@ -338,9 +376,14 @@ static inline void erofs_check_ondisk_layout_definitions(void)
BUILD_BUG_ON(sizeof(struct erofs_inode_extended) != 64);
BUILD_BUG_ON(sizeof(struct erofs_xattr_ibody_header) != 12);
BUILD_BUG_ON(sizeof(struct erofs_xattr_entry) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_info) != 4);
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_map_header) != 8);
BUILD_BUG_ON(sizeof(struct z_erofs_vle_decompressed_index) != 8);
BUILD_BUG_ON(sizeof(struct erofs_dirent) != 12);
+ /* keep in sync between 2 index structures for better extendibility */
+ BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) !=
+ sizeof(struct z_erofs_vle_decompressed_index));
BUILD_BUG_ON(BIT(Z_EROFS_VLE_DI_CLUSTER_TYPE_BITS) <
Z_EROFS_VLE_CLUSTER_TYPE_MAX - 1);
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH v3 2/2] erofs: support reading chunk-based uncompressed files
2021-08-20 10:00 ` Gao Xiang
@ 2021-08-20 10:00 ` Gao Xiang
-1 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-20 10:00 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Eryu Guan, Liu Jiang, Joseph Qi, Gao Xiang
Add runtime support for chunk-based uncompressed files
described in the previous patch.
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Reviewed-by: Chao Yu <chao@kernel.org>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
changes since v2:
- use EROFS_BLOCK_MAP_ENTRY_SIZE instead of 4 suggested by Chao.
fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
fs/erofs/inode.c | 18 ++++++++-
fs/erofs/internal.h | 5 +++
3 files changed, 102 insertions(+), 11 deletions(-)
diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index 09c46fbdb9b2..9db829715652 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "internal.h"
#include <linux/prefetch.h>
@@ -36,13 +37,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
lastblk = nblocks - tailendpacking;
- if (offset >= inode->i_size) {
- /* leave out-of-bound access unmapped */
- map->m_flags = 0;
- map->m_plen = 0;
- goto out;
- }
-
/* there is no hole in flatmode */
map->m_flags = EROFS_MAP_MAPPED;
@@ -77,14 +71,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
goto err_out;
}
-out:
map->m_llen = map->m_plen;
-
err_out:
trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
return err;
}
+static int erofs_map_blocks(struct inode *inode,
+ struct erofs_map_blocks *map, int flags)
+{
+ struct super_block *sb = inode->i_sb;
+ struct erofs_inode *vi = EROFS_I(inode);
+ struct erofs_inode_chunk_index *idx;
+ struct page *page;
+ u64 chunknr;
+ unsigned int unit;
+ erofs_off_t pos;
+ int err = 0;
+
+ if (map->m_la >= inode->i_size) {
+ /* leave out-of-bound access unmapped */
+ map->m_flags = 0;
+ map->m_plen = 0;
+ goto out;
+ }
+
+ if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
+ return erofs_map_blocks_flatmode(inode, map, flags);
+
+ if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
+ unit = sizeof(*idx); /* chunk index */
+ else
+ unit = EROFS_BLOCK_MAP_ENTRY_SIZE; /* block map */
+
+ chunknr = map->m_la >> vi->chunkbits;
+ pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
+ vi->xattr_isize, unit) + unit * chunknr;
+
+ page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
+ if (IS_ERR(page))
+ return PTR_ERR(page);
+
+ map->m_la = chunknr << vi->chunkbits;
+ map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
+ roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
+
+ /* handle block map */
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
+ __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
+
+ if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
+ map->m_flags = 0;
+ } else {
+ map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ }
+ goto out_unlock;
+ }
+ /* parse chunk indexes */
+ idx = page_address(page) + erofs_blkoff(pos);
+ switch (le32_to_cpu(idx->blkaddr)) {
+ case EROFS_NULL_ADDR:
+ map->m_flags = 0;
+ break;
+ default:
+ /* only one device is supported for now */
+ if (idx->device_id) {
+ erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
+ le16_to_cpu(idx->device_id),
+ chunknr, vi->nid);
+ err = -EFSCORRUPTED;
+ goto out_unlock;
+ }
+ map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ break;
+ }
+out_unlock:
+ unlock_page(page);
+ put_page(page);
+out:
+ map->m_llen = map->m_plen;
+ return err;
+}
+
static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
{
@@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
map.m_la = offset;
map.m_llen = length;
- ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
+ ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
if (ret < 0)
return ret;
diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
index d13e0709599c..4408929bd6f5 100644
--- a/fs/erofs/inode.c
+++ b/fs/erofs/inode.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "xattr.h"
@@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
/* total blocks for compressed files */
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(die->i_u.compressed_blocks);
-
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ /* fill chunked inode summary info */
+ vi->chunkformat = le16_to_cpu(die->i_u.c.format);
kfree(copied);
break;
case EROFS_INODE_LAYOUT_COMPACT:
@@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
inode->i_size = le32_to_cpu(dic->i_size);
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(dic->i_u.compressed_blocks);
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
break;
default:
erofs_err(inode->i_sb,
@@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
goto err_out;
}
+ if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
+ erofs_err(inode->i_sb,
+ "unsupported chunk format %x of nid %llu",
+ vi->chunkformat, vi->nid);
+ err = -EOPNOTSUPP;
+ goto err_out;
+ }
+ vi->chunkbits = LOG_BLOCK_SIZE +
+ (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
+ }
inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
index 91089ab8a816..9524e155b38f 100644
--- a/fs/erofs/internal.h
+++ b/fs/erofs/internal.h
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_INTERNAL_H
#define __EROFS_INTERNAL_H
@@ -261,6 +262,10 @@ struct erofs_inode {
union {
erofs_blk_t raw_blkaddr;
+ struct {
+ unsigned short chunkformat;
+ unsigned char chunkbits;
+ };
#ifdef CONFIG_EROFS_FS_ZIP
struct {
unsigned short z_advise;
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* [PATCH v3 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-20 10:00 ` Gao Xiang
0 siblings, 0 replies; 37+ messages in thread
From: Gao Xiang @ 2021-08-20 10:00 UTC (permalink / raw)
To: linux-erofs, Chao Yu, Liu Bo
Cc: LKML, Peng Tao, Joseph Qi, Eryu Guan, Liu Jiang
Add runtime support for chunk-based uncompressed files
described in the previous patch.
Reviewed-by: Liu Bo <bo.liu@linux.alibaba.com>
Reviewed-by: Chao Yu <chao@kernel.org>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
---
changes since v2:
- use EROFS_BLOCK_MAP_ENTRY_SIZE instead of 4 suggested by Chao.
fs/erofs/data.c | 90 ++++++++++++++++++++++++++++++++++++++++-----
fs/erofs/inode.c | 18 ++++++++-
fs/erofs/internal.h | 5 +++
3 files changed, 102 insertions(+), 11 deletions(-)
diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index 09c46fbdb9b2..9db829715652 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "internal.h"
#include <linux/prefetch.h>
@@ -36,13 +37,6 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
nblocks = DIV_ROUND_UP(inode->i_size, PAGE_SIZE);
lastblk = nblocks - tailendpacking;
- if (offset >= inode->i_size) {
- /* leave out-of-bound access unmapped */
- map->m_flags = 0;
- map->m_plen = 0;
- goto out;
- }
-
/* there is no hole in flatmode */
map->m_flags = EROFS_MAP_MAPPED;
@@ -77,14 +71,90 @@ static int erofs_map_blocks_flatmode(struct inode *inode,
goto err_out;
}
-out:
map->m_llen = map->m_plen;
-
err_out:
trace_erofs_map_blocks_flatmode_exit(inode, map, flags, 0);
return err;
}
+static int erofs_map_blocks(struct inode *inode,
+ struct erofs_map_blocks *map, int flags)
+{
+ struct super_block *sb = inode->i_sb;
+ struct erofs_inode *vi = EROFS_I(inode);
+ struct erofs_inode_chunk_index *idx;
+ struct page *page;
+ u64 chunknr;
+ unsigned int unit;
+ erofs_off_t pos;
+ int err = 0;
+
+ if (map->m_la >= inode->i_size) {
+ /* leave out-of-bound access unmapped */
+ map->m_flags = 0;
+ map->m_plen = 0;
+ goto out;
+ }
+
+ if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
+ return erofs_map_blocks_flatmode(inode, map, flags);
+
+ if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
+ unit = sizeof(*idx); /* chunk index */
+ else
+ unit = EROFS_BLOCK_MAP_ENTRY_SIZE; /* block map */
+
+ chunknr = map->m_la >> vi->chunkbits;
+ pos = ALIGN(iloc(EROFS_SB(sb), vi->nid) + vi->inode_isize +
+ vi->xattr_isize, unit) + unit * chunknr;
+
+ page = erofs_get_meta_page(inode->i_sb, erofs_blknr(pos));
+ if (IS_ERR(page))
+ return PTR_ERR(page);
+
+ map->m_la = chunknr << vi->chunkbits;
+ map->m_plen = min_t(erofs_off_t, 1UL << vi->chunkbits,
+ roundup(inode->i_size - map->m_la, EROFS_BLKSIZ));
+
+ /* handle block map */
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)) {
+ __le32 *blkaddr = page_address(page) + erofs_blkoff(pos);
+
+ if (le32_to_cpu(*blkaddr) == EROFS_NULL_ADDR) {
+ map->m_flags = 0;
+ } else {
+ map->m_pa = blknr_to_addr(le32_to_cpu(*blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ }
+ goto out_unlock;
+ }
+ /* parse chunk indexes */
+ idx = page_address(page) + erofs_blkoff(pos);
+ switch (le32_to_cpu(idx->blkaddr)) {
+ case EROFS_NULL_ADDR:
+ map->m_flags = 0;
+ break;
+ default:
+ /* only one device is supported for now */
+ if (idx->device_id) {
+ erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
+ le16_to_cpu(idx->device_id),
+ chunknr, vi->nid);
+ err = -EFSCORRUPTED;
+ goto out_unlock;
+ }
+ map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
+ map->m_flags = EROFS_MAP_MAPPED;
+ break;
+ }
+out_unlock:
+ unlock_page(page);
+ put_page(page);
+out:
+ map->m_llen = map->m_plen;
+ return err;
+}
+
static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
{
@@ -94,7 +164,7 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
map.m_la = offset;
map.m_llen = length;
- ret = erofs_map_blocks_flatmode(inode, &map, EROFS_GET_BLOCKS_RAW);
+ ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
if (ret < 0)
return ret;
diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
index d13e0709599c..4408929bd6f5 100644
--- a/fs/erofs/inode.c
+++ b/fs/erofs/inode.c
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#include "xattr.h"
@@ -122,7 +123,9 @@ static struct page *erofs_read_inode(struct inode *inode,
/* total blocks for compressed files */
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(die->i_u.compressed_blocks);
-
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ /* fill chunked inode summary info */
+ vi->chunkformat = le16_to_cpu(die->i_u.c.format);
kfree(copied);
break;
case EROFS_INODE_LAYOUT_COMPACT:
@@ -160,6 +163,8 @@ static struct page *erofs_read_inode(struct inode *inode,
inode->i_size = le32_to_cpu(dic->i_size);
if (erofs_inode_is_data_compressed(vi->datalayout))
nblks = le32_to_cpu(dic->i_u.compressed_blocks);
+ else if (vi->datalayout == EROFS_INODE_CHUNK_BASED)
+ vi->chunkformat = le16_to_cpu(dic->i_u.c.format);
break;
default:
erofs_err(inode->i_sb,
@@ -169,6 +174,17 @@ static struct page *erofs_read_inode(struct inode *inode,
goto err_out;
}
+ if (vi->datalayout == EROFS_INODE_CHUNK_BASED) {
+ if (!(vi->chunkformat & EROFS_CHUNK_FORMAT_ALL)) {
+ erofs_err(inode->i_sb,
+ "unsupported chunk format %x of nid %llu",
+ vi->chunkformat, vi->nid);
+ err = -EOPNOTSUPP;
+ goto err_out;
+ }
+ vi->chunkbits = LOG_BLOCK_SIZE +
+ (vi->chunkformat & EROFS_CHUNK_FORMAT_BLKBITS_MASK);
+ }
inode->i_mtime.tv_sec = inode->i_ctime.tv_sec;
inode->i_atime.tv_sec = inode->i_ctime.tv_sec;
inode->i_mtime.tv_nsec = inode->i_ctime.tv_nsec;
diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
index 91089ab8a816..9524e155b38f 100644
--- a/fs/erofs/internal.h
+++ b/fs/erofs/internal.h
@@ -2,6 +2,7 @@
/*
* Copyright (C) 2017-2018 HUAWEI, Inc.
* https://www.huawei.com/
+ * Copyright (C) 2021, Alibaba Cloud
*/
#ifndef __EROFS_INTERNAL_H
#define __EROFS_INTERNAL_H
@@ -261,6 +262,10 @@ struct erofs_inode {
union {
erofs_blk_t raw_blkaddr;
+ struct {
+ unsigned short chunkformat;
+ unsigned char chunkbits;
+ };
#ifdef CONFIG_EROFS_FS_ZIP
struct {
unsigned short z_advise;
--
2.24.4
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files
@ 2021-08-21 10:53 kernel test robot
0 siblings, 0 replies; 37+ messages in thread
From: kernel test robot @ 2021-08-21 10:53 UTC (permalink / raw)
To: kbuild
[-- Attachment #1: Type: text/plain, Size: 17112 bytes --]
CC: clang-built-linux(a)googlegroups.com
CC: kbuild-all(a)lists.01.org
In-Reply-To: <20210819063310.177035-2-hsiangkao@linux.alibaba.com>
References: <20210819063310.177035-2-hsiangkao@linux.alibaba.com>
TO: Gao Xiang <hsiangkao@linux.alibaba.com>
Hi Gao,
I love your patch! Perhaps something to improve:
[auto build test WARNING on xiang-erofs/dev-test]
[also build test WARNING on next-20210820]
[cannot apply to v5.14-rc6]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Gao-Xiang/erofs-introduce-chunk-based-file-on-disk-format/20210820-190035
base: https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs.git dev-test
:::::: branch date: 24 hours ago
:::::: commit date: 24 hours ago
config: i386-randconfig-c001-20210820 (attached as .config)
compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project d9c5613e856cf2addfbf892fc4c1ce9ef9feceaa)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/7fe675691d4b693aa4d499a0902ede8aa74b8d32
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Gao-Xiang/erofs-introduce-chunk-based-file-on-disk-format/20210820-190035
git checkout 7fe675691d4b693aa4d499a0902ede8aa74b8d32
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=i386 clang-analyzer
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
clang-analyzer warnings: (new ones prefixed by >>)
^
include/linux/percpu-defs.h:446:2: note: expanded from macro '__this_cpu_read'
raw_cpu_read(pcp); \
^
include/linux/percpu-defs.h:420:28: note: expanded from macro 'raw_cpu_read'
#define raw_cpu_read(pcp) __pcpu_size_call_return(raw_cpu_read_, pcp)
^
include/linux/percpu-defs.h:320:2: note: expanded from macro '__pcpu_size_call_return'
switch(sizeof(variable)) { \
^
arch/x86/mm/tlb.c:1170:12: note: Execution continues on line 1170
int cpu = get_cpu();
^
include/linux/smp.h:268:42: note: expanded from macro 'get_cpu'
#define get_cpu() ({ preempt_disable(); __smp_processor_id(); })
^
arch/x86/include/asm/smp.h:159:30: note: expanded from macro '__smp_processor_id'
#define __smp_processor_id() __this_cpu_read(cpu_number)
^
include/linux/percpu-defs.h:446:2: note: expanded from macro '__this_cpu_read'
raw_cpu_read(pcp); \
^
include/linux/percpu-defs.h:420:28: note: expanded from macro 'raw_cpu_read'
#define raw_cpu_read(pcp) __pcpu_size_call_return(raw_cpu_read_, pcp)
^
include/linux/percpu-defs.h:323:42: note: expanded from macro '__pcpu_size_call_return'
case 4: pscr_ret__ = stem##4(variable); break; \
^
arch/x86/mm/tlb.c:1178:6: note: Assuming the condition is true
if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) {
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/mm/tlb.c:1178:2: note: Taking true branch
if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) {
^
arch/x86/mm/tlb.c:1179:3: note: Calling 'flush_tlb_multi'
flush_tlb_multi(&batch->cpumask, info);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/mm/tlb.c:868:2: note: Calling '__flush_tlb_multi'
__flush_tlb_multi(cpumask, info);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/paravirt.h:87:2: note: Assigned value is garbage or undefined
PVOP_VCALL2(mmu.flush_tlb_multi, cpumask, info);
^
arch/x86/include/asm/paravirt_types.h:547:2: note: expanded from macro 'PVOP_VCALL2'
__PVOP_VCALL(op, PVOP_CALL_ARG1(arg1), PVOP_CALL_ARG2(arg2))
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/paravirt_types.h:491:8: note: expanded from macro '__PVOP_VCALL'
(void)____PVOP_CALL(, op, CLBR_ANY, PVOP_VCALL_CLOBBERS, \
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
arch/x86/include/asm/paravirt_types.h:446:3: note: expanded from macro '____PVOP_CALL'
PVOP_CALL_ARGS; \
^~~~~~~~~~~~~~
arch/x86/include/asm/paravirt_types.h:386:16: note: expanded from macro 'PVOP_CALL_ARGS'
unsigned long __eax = __eax, __edx = __edx, __ecx = __ecx;
^ ~~~~~
Suppressed 6 warnings (6 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
9 warnings generated.
Suppressed 9 warnings (9 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
13 warnings generated.
fs/erofs/inode.c:50:3: warning: Value stored to 'err' is never read [clang-analyzer-deadcode.DeadStores]
err = -EOPNOTSUPP;
^ ~~~~~~~~~~~
fs/erofs/inode.c:50:3: note: Value stored to 'err' is never read
err = -EOPNOTSUPP;
^ ~~~~~~~~~~~
fs/erofs/inode.c:58:3: warning: Value stored to 'err' is never read [clang-analyzer-deadcode.DeadStores]
err = -EOPNOTSUPP;
^ ~~~~~~~~~~~
fs/erofs/inode.c:58:3: note: Value stored to 'err' is never read
err = -EOPNOTSUPP;
^ ~~~~~~~~~~~
fs/erofs/inode.c:74:5: warning: Value stored to 'err' is never read [clang-analyzer-deadcode.DeadStores]
err = -ENOMEM;
^ ~~~~~~~
fs/erofs/inode.c:74:5: note: Value stored to 'err' is never read
err = -ENOMEM;
^ ~~~~~~~
fs/erofs/inode.c:173:3: warning: Value stored to 'err' is never read [clang-analyzer-deadcode.DeadStores]
err = -EOPNOTSUPP;
^ ~~~~~~~~~~~
fs/erofs/inode.c:173:3: note: Value stored to 'err' is never read
err = -EOPNOTSUPP;
^ ~~~~~~~~~~~
fs/erofs/inode.c:182:4: warning: Value stored to 'err' is never read [clang-analyzer-deadcode.DeadStores]
err = -EOPNOTSUPP;
^ ~~~~~~~~~~~
fs/erofs/inode.c:182:4: note: Value stored to 'err' is never read
err = -EOPNOTSUPP;
^ ~~~~~~~~~~~
fs/erofs/inode.c:207:2: warning: Value stored to 'err' is never read [clang-analyzer-deadcode.DeadStores]
err = -EFSCORRUPTED;
^ ~~~~~~~~~~~~~
fs/erofs/inode.c:207:2: note: Value stored to 'err' is never read
err = -EFSCORRUPTED;
^ ~~~~~~~~~~~~~
Suppressed 7 warnings (7 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
8 warnings generated.
>> fs/erofs/data.c:178:20: warning: The left operand of '&' is a garbage value [clang-analyzer-core.UndefinedBinaryOperatorResult]
if (!(map.m_flags & EROFS_MAP_MAPPED)) {
~~~~~~~~~~~ ^
fs/erofs/data.c:167:8: note: Calling 'erofs_map_blocks'
ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fs/erofs/data.c:84:27: note: Left side of '&&' is false
struct erofs_inode *vi = EROFS_I(inode);
^
fs/erofs/internal.h:282:2: note: expanded from macro 'EROFS_I'
container_of(ptr, struct erofs_inode, vfs_inode)
^
include/linux/kernel.h:495:61: note: expanded from macro 'container_of'
BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
^
fs/erofs/data.c:84:27: note: Taking false branch
struct erofs_inode *vi = EROFS_I(inode);
^
fs/erofs/internal.h:282:2: note: expanded from macro 'EROFS_I'
container_of(ptr, struct erofs_inode, vfs_inode)
^
include/linux/kernel.h:495:2: note: expanded from macro 'container_of'
BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
^
include/linux/build_bug.h:39:37: note: expanded from macro 'BUILD_BUG_ON_MSG'
#define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
^
include/linux/compiler_types.h:328:2: note: expanded from macro 'compiletime_assert'
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
^
include/linux/compiler_types.h:316:2: note: expanded from macro '_compiletime_assert'
__compiletime_assert(condition, msg, prefix, suffix)
^
include/linux/compiler_types.h:308:3: note: expanded from macro '__compiletime_assert'
if (!(condition)) \
^
fs/erofs/data.c:84:27: note: Loop condition is false. Exiting loop
struct erofs_inode *vi = EROFS_I(inode);
^
fs/erofs/internal.h:282:2: note: expanded from macro 'EROFS_I'
container_of(ptr, struct erofs_inode, vfs_inode)
^
include/linux/kernel.h:495:2: note: expanded from macro 'container_of'
BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
^
include/linux/build_bug.h:39:37: note: expanded from macro 'BUILD_BUG_ON_MSG'
#define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
^
include/linux/compiler_types.h:328:2: note: expanded from macro 'compiletime_assert'
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
^
include/linux/compiler_types.h:316:2: note: expanded from macro '_compiletime_assert'
__compiletime_assert(condition, msg, prefix, suffix)
^
include/linux/compiler_types.h:306:2: note: expanded from macro '__compiletime_assert'
do { \
^
fs/erofs/data.c:92:6: note: Assuming field 'm_la' is < field 'i_size'
if (map->m_la >= inode->i_size) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
fs/erofs/data.c:92:2: note: Taking false branch
if (map->m_la >= inode->i_size) {
^
fs/erofs/data.c:99:6: note: Assuming field 'datalayout' is equal to EROFS_INODE_CHUNK_BASED
if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fs/erofs/data.c:99:2: note: Taking false branch
if (vi->datalayout != EROFS_INODE_CHUNK_BASED)
^
fs/erofs/data.c:102:6: note: Assuming the condition is false
if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fs/erofs/data.c:102:2: note: Taking false branch
if (vi->chunkformat & EROFS_CHUNK_FORMAT_INDEXES)
^
fs/erofs/data.c:112:2: note: Taking true branch
if (IS_ERR(page))
^
fs/erofs/data.c:113:3: note: Returning without writing to 'map->m_flags'
return PTR_ERR(page);
^
fs/erofs/data.c:167:8: note: Returning from 'erofs_map_blocks'
ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fs/erofs/data.c:168:6: note: 'ret' is >= 0
if (ret < 0)
^~~
fs/erofs/data.c:168:2: note: Taking false branch
if (ret < 0)
^
fs/erofs/data.c:178:20: note: The left operand of '&' is a garbage value
if (!(map.m_flags & EROFS_MAP_MAPPED)) {
~~~~~~~~~~~ ^
Suppressed 7 warnings (7 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
10 warnings generated.
Suppressed 10 warnings (10 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
7 warnings generated.
fs/nls/nls_base.c:227:10: warning: Dereference of null pointer [clang-analyzer-core.NullDereference]
*op++ = (u8) u;
vim +178 fs/erofs/data.c
7fe675691d4b69 Gao Xiang 2021-08-19 157
a08e67a0280215 Huang Jianan 2021-08-05 158 static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
a08e67a0280215 Huang Jianan 2021-08-05 159 unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
a08e67a0280215 Huang Jianan 2021-08-05 160 {
a08e67a0280215 Huang Jianan 2021-08-05 161 int ret;
a08e67a0280215 Huang Jianan 2021-08-05 162 struct erofs_map_blocks map;
a08e67a0280215 Huang Jianan 2021-08-05 163
a08e67a0280215 Huang Jianan 2021-08-05 164 map.m_la = offset;
a08e67a0280215 Huang Jianan 2021-08-05 165 map.m_llen = length;
a08e67a0280215 Huang Jianan 2021-08-05 166
7fe675691d4b69 Gao Xiang 2021-08-19 167 ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW);
a08e67a0280215 Huang Jianan 2021-08-05 168 if (ret < 0)
a08e67a0280215 Huang Jianan 2021-08-05 169 return ret;
a08e67a0280215 Huang Jianan 2021-08-05 170
a08e67a0280215 Huang Jianan 2021-08-05 171 iomap->bdev = inode->i_sb->s_bdev;
06252e9ce05b94 Gao Xiang 2021-08-05 172 iomap->dax_dev = EROFS_I_SB(inode)->dax_dev;
a08e67a0280215 Huang Jianan 2021-08-05 173 iomap->offset = map.m_la;
a08e67a0280215 Huang Jianan 2021-08-05 174 iomap->length = map.m_llen;
a08e67a0280215 Huang Jianan 2021-08-05 175 iomap->flags = 0;
771c994ea51f57 Gao Xiang 2021-08-05 176 iomap->private = NULL;
a08e67a0280215 Huang Jianan 2021-08-05 177
a08e67a0280215 Huang Jianan 2021-08-05 @178 if (!(map.m_flags & EROFS_MAP_MAPPED)) {
a08e67a0280215 Huang Jianan 2021-08-05 179 iomap->type = IOMAP_HOLE;
a08e67a0280215 Huang Jianan 2021-08-05 180 iomap->addr = IOMAP_NULL_ADDR;
a08e67a0280215 Huang Jianan 2021-08-05 181 if (!iomap->length)
a08e67a0280215 Huang Jianan 2021-08-05 182 iomap->length = length;
a08e67a0280215 Huang Jianan 2021-08-05 183 return 0;
a08e67a0280215 Huang Jianan 2021-08-05 184 }
a08e67a0280215 Huang Jianan 2021-08-05 185
a08e67a0280215 Huang Jianan 2021-08-05 186 if (map.m_flags & EROFS_MAP_META) {
771c994ea51f57 Gao Xiang 2021-08-05 187 struct page *ipage;
771c994ea51f57 Gao Xiang 2021-08-05 188
771c994ea51f57 Gao Xiang 2021-08-05 189 iomap->type = IOMAP_INLINE;
771c994ea51f57 Gao Xiang 2021-08-05 190 ipage = erofs_get_meta_page(inode->i_sb,
771c994ea51f57 Gao Xiang 2021-08-05 191 erofs_blknr(map.m_pa));
771c994ea51f57 Gao Xiang 2021-08-05 192 if (IS_ERR(ipage))
771c994ea51f57 Gao Xiang 2021-08-05 193 return PTR_ERR(ipage);
771c994ea51f57 Gao Xiang 2021-08-05 194 iomap->inline_data = page_address(ipage) +
771c994ea51f57 Gao Xiang 2021-08-05 195 erofs_blkoff(map.m_pa);
771c994ea51f57 Gao Xiang 2021-08-05 196 iomap->private = ipage;
771c994ea51f57 Gao Xiang 2021-08-05 197 } else {
a08e67a0280215 Huang Jianan 2021-08-05 198 iomap->type = IOMAP_MAPPED;
a08e67a0280215 Huang Jianan 2021-08-05 199 iomap->addr = map.m_pa;
771c994ea51f57 Gao Xiang 2021-08-05 200 }
a08e67a0280215 Huang Jianan 2021-08-05 201 return 0;
a08e67a0280215 Huang Jianan 2021-08-05 202 }
a08e67a0280215 Huang Jianan 2021-08-05 203
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 34927 bytes --]
^ permalink raw reply [flat|nested] 37+ messages in thread
end of thread, other threads:[~2021-08-21 10:53 UTC | newest]
Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-18 7:07 [PATCH 1/2] erofs: introduce chunk-based file on-disk format Gao Xiang
2021-08-18 7:07 ` Gao Xiang
2021-08-18 7:07 ` [PATCH 2/2] erofs: support reading chunk-based uncompressed files Gao Xiang
2021-08-18 7:07 ` Gao Xiang
2021-08-18 22:29 ` Liu Bo
2021-08-18 22:29 ` Liu Bo
2021-08-19 3:46 ` Chao Yu
2021-08-19 3:46 ` Chao Yu
2021-08-19 5:14 ` Gao Xiang
2021-08-19 5:14 ` Gao Xiang
2021-08-18 22:28 ` [PATCH 1/2] erofs: introduce chunk-based file on-disk format Liu Bo
2021-08-18 22:28 ` Liu Bo
2021-08-19 0:59 ` Gao Xiang
2021-08-19 0:59 ` Gao Xiang
2021-08-19 3:32 ` Chao Yu
2021-08-19 3:32 ` Chao Yu
2021-08-19 6:33 ` [PATCH v2 " Gao Xiang
2021-08-19 6:33 ` Gao Xiang
2021-08-19 6:33 ` [PATCH v2 2/2] erofs: support reading chunk-based uncompressed files Gao Xiang
2021-08-19 6:33 ` Gao Xiang
2021-08-19 6:37 ` Joseph Qi
2021-08-19 6:37 ` Joseph Qi
2021-08-19 7:12 ` Gao Xiang
2021-08-19 7:12 ` Gao Xiang
2021-08-20 9:04 ` Chao Yu
2021-08-20 9:04 ` Chao Yu
2021-08-20 9:12 ` Gao Xiang
2021-08-20 9:12 ` Gao Xiang
2021-08-20 9:29 ` Chao Yu
2021-08-20 9:29 ` Chao Yu
2021-08-20 9:33 ` Gao Xiang
2021-08-20 9:33 ` Gao Xiang
2021-08-20 10:00 ` [PATCH v3 1/2] erofs: introduce chunk-based file on-disk format Gao Xiang
2021-08-20 10:00 ` Gao Xiang
2021-08-20 10:00 ` [PATCH v3 2/2] erofs: support reading chunk-based uncompressed files Gao Xiang
2021-08-20 10:00 ` Gao Xiang
2021-08-21 10:53 [PATCH v2 " kernel test robot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.