* [PATCH] erofs: update documentation
[not found] <20191207023726.5359-1-hsiangkao.ref@aol.com>
@ 2019-12-07 2:37 ` Gao Xiang
2019-12-07 2:55 ` [PATCH v2] " Gao Xiang
0 siblings, 1 reply; 4+ messages in thread
From: Gao Xiang @ 2019-12-07 2:37 UTC (permalink / raw)
To: Chao Yu, Jonathan Corbet; +Cc: linux-erofs, linux-doc, linux-kernel, Gao Xiang
From: Gao Xiang <gaoxiang25@huawei.com>
Some on-disk structures, fields have been renamed in v5.4,
the corresponding document should be updated as well.
Also fix misrespresentation of file time and words about
fixed-sized output compression, data inline, etc.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
---
Hi,
I plan to update documentation together with
https://lore.kernel.org/r/20191201084040.29275-1-hsiangkao@aol.com/
for this round.
Comments about this are greatly welcomed.
Thanks,
Gao Xiang
Documentation/filesystems/erofs.txt | 27 ++++++++++++++-------------
1 file changed, 14 insertions(+), 13 deletions(-)
diff --git a/Documentation/filesystems/erofs.txt b/Documentation/filesystems/erofs.txt
index b0c085326e2e..061eb8bd8387 100644
--- a/Documentation/filesystems/erofs.txt
+++ b/Documentation/filesystems/erofs.txt
@@ -24,11 +24,11 @@ Here is the main features of EROFS:
- Metadata & data could be mixed by design;
- 2 inode versions for different requirements:
- v1 v2
+ compact (v1) extended (v2)
Inode metadata size: 32 bytes 64 bytes
Max file size: 4 GB 16 EB (also limited by max. vol size)
Max uids/gids: 65536 4294967296
- File creation time: no yes (64 + 32-bit timestamp)
+ File change time: no yes (64 + 32-bit timestamp)
Max hardlinks: 65536 4294967296
Metadata reserved: 4 bytes 14 bytes
@@ -39,7 +39,7 @@ Here is the main features of EROFS:
- Support POSIX.1e ACLs by using xattrs;
- Support transparent file compression as an option:
- LZ4 algorithm with 4 KB fixed-output compression for high performance;
+ LZ4 algorithm with 4 KB fixed-sized output compression for high performance.
The following git tree provides the file system user-space tools under
development (ex, formatting tool mkfs.erofs):
@@ -85,7 +85,7 @@ All data areas should be aligned with the block size, but metadata areas
may not. All metadatas can be now observed in two different spaces (views):
1. Inode metadata space
Each valid inode should be aligned with an inode slot, which is a fixed
- value (32 bytes) and designed to be kept in line with v1 inode size.
+ value (32 bytes) and designed to be kept in line with compact inode size.
Each inode can be directly found with the following formula:
inode offset = meta_blkaddr * block_size + 32 * nid
@@ -117,10 +117,10 @@ may not. All metadatas can be now observed in two different spaces (views):
|-> aligned with 4B
Inode could be 32 or 64 bytes, which can be distinguished from a common
- field which all inode versions have -- i_advise:
+ field which all inode versions have -- i_format:
__________________ __________________
- | i_advise | | i_advise |
+ | i_format | | i_format |
|__________________| |__________________|
| ... | | ... |
| | | |
@@ -129,12 +129,13 @@ may not. All metadatas can be now observed in two different spaces (views):
|__________________| 64 bytes
Xattrs, extents, data inline are followed by the corresponding inode with
- proper alignes, and they could be optional for different data mappings,
- _currently_ there are totally 3 valid data mappings supported:
+ proper alignment, and they could be optional for different data mappings.
+ _currently_ total 4 valid data mappings are supported:
- 1) flat file data without data inline (no extent);
- 2) fixed-output size data compression (must have extents);
- 3) flat file data with tail-end data inline (no extent);
+ 0 flat file data without data inline (no extent);
+ 1 fixed-sized output data compression (with non-compacted indexes);
+ 2 flat file data with tail packing data inline (no extent);
+ 3 fixed-sized output data compression (with compacted indexes).
The size of the optional xattrs is indicated by i_xattr_count in inode
header. Large xattrs or xattrs shared by many different files can be
@@ -182,8 +183,8 @@ introduce another on-disk field at all.
Compression
-----------
-Currently, EROFS supports 4KB fixed-output clustersize transparent file
-compression, as illustrated below:
+Currently, EROFS supports 4KB fixed-sized output transparent file compression,
+as illustrated below:
|---- Variant-Length Extent ----|-------- VLE --------|----- VLE -----
clusterofs clusterofs clusterofs
--
2.20.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH v2] erofs: update documentation
2019-12-07 2:37 ` [PATCH] erofs: update documentation Gao Xiang
@ 2019-12-07 2:55 ` Gao Xiang
0 siblings, 0 replies; 4+ messages in thread
From: Gao Xiang @ 2019-12-07 2:55 UTC (permalink / raw)
To: Chao Yu, Jonathan Corbet; +Cc: linux-erofs, linux-doc, linux-kernel, Gao Xiang
From: Gao Xiang <gaoxiang25@huawei.com>
Some on-disk structures, fields have been renamed in v5.4,
the corresponding document should be updated as well.
Also fix misrespresentation of file time and words about
fixed-sized output compression, data inline, etc.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
---
changes since v1:
- mark that "fixed-sized output compression (with compacted indexes)"
was firstly introduced in v5.3 in order to minimumize metadata;
- reduce whitespace between on-disk values and words about data mapping
details;
Documentation/filesystems/erofs.txt | 27 ++++++++++++++-------------
1 file changed, 14 insertions(+), 13 deletions(-)
diff --git a/Documentation/filesystems/erofs.txt b/Documentation/filesystems/erofs.txt
index b0c085326e2e..db6d39c3ae71 100644
--- a/Documentation/filesystems/erofs.txt
+++ b/Documentation/filesystems/erofs.txt
@@ -24,11 +24,11 @@ Here is the main features of EROFS:
- Metadata & data could be mixed by design;
- 2 inode versions for different requirements:
- v1 v2
+ compact (v1) extended (v2)
Inode metadata size: 32 bytes 64 bytes
Max file size: 4 GB 16 EB (also limited by max. vol size)
Max uids/gids: 65536 4294967296
- File creation time: no yes (64 + 32-bit timestamp)
+ File change time: no yes (64 + 32-bit timestamp)
Max hardlinks: 65536 4294967296
Metadata reserved: 4 bytes 14 bytes
@@ -39,7 +39,7 @@ Here is the main features of EROFS:
- Support POSIX.1e ACLs by using xattrs;
- Support transparent file compression as an option:
- LZ4 algorithm with 4 KB fixed-output compression for high performance;
+ LZ4 algorithm with 4 KB fixed-sized output compression for high performance.
The following git tree provides the file system user-space tools under
development (ex, formatting tool mkfs.erofs):
@@ -85,7 +85,7 @@ All data areas should be aligned with the block size, but metadata areas
may not. All metadatas can be now observed in two different spaces (views):
1. Inode metadata space
Each valid inode should be aligned with an inode slot, which is a fixed
- value (32 bytes) and designed to be kept in line with v1 inode size.
+ value (32 bytes) and designed to be kept in line with compact inode size.
Each inode can be directly found with the following formula:
inode offset = meta_blkaddr * block_size + 32 * nid
@@ -117,10 +117,10 @@ may not. All metadatas can be now observed in two different spaces (views):
|-> aligned with 4B
Inode could be 32 or 64 bytes, which can be distinguished from a common
- field which all inode versions have -- i_advise:
+ field which all inode versions have -- i_format:
__________________ __________________
- | i_advise | | i_advise |
+ | i_format | | i_format |
|__________________| |__________________|
| ... | | ... |
| | | |
@@ -129,12 +129,13 @@ may not. All metadatas can be now observed in two different spaces (views):
|__________________| 64 bytes
Xattrs, extents, data inline are followed by the corresponding inode with
- proper alignes, and they could be optional for different data mappings,
- _currently_ there are totally 3 valid data mappings supported:
+ proper alignment, and they could be optional for different data mappings.
+ _currently_ total 4 valid data mappings are supported:
- 1) flat file data without data inline (no extent);
- 2) fixed-output size data compression (must have extents);
- 3) flat file data with tail-end data inline (no extent);
+ 0 flat file data without data inline (no extent);
+ 1 fixed-sized output data compression (with non-compacted indexes);
+ 2 flat file data with tail packing data inline (no extent);
+ 3 fixed-sized output data compression (with compacted indexes, v5.3+).
The size of the optional xattrs is indicated by i_xattr_count in inode
header. Large xattrs or xattrs shared by many different files can be
@@ -182,8 +183,8 @@ introduce another on-disk field at all.
Compression
-----------
-Currently, EROFS supports 4KB fixed-output clustersize transparent file
-compression, as illustrated below:
+Currently, EROFS supports 4KB fixed-sized output transparent file compression,
+as illustrated below:
|---- Variant-Length Extent ----|-------- VLE --------|----- VLE -----
clusterofs clusterofs clusterofs
--
2.20.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH v2] erofs: update documentation
2023-09-28 13:48 Jingbo Xu
@ 2023-09-28 14:36 ` Gao Xiang
0 siblings, 0 replies; 4+ messages in thread
From: Gao Xiang @ 2023-09-28 14:36 UTC (permalink / raw)
To: Jingbo Xu, xiang, chao, linux-erofs; +Cc: huyue2, linux-doc, linux-kernel
On 2023/9/28 21:48, Jingbo Xu wrote:
> - update new features like bloom filter and DEFLATE.
>
> - add documentation for the long xattr name prefixes, which was
> landed upstream since v6.4.
>
> Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Thanks,
Gao Xiang
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH v2] erofs: update documentation
@ 2023-09-28 13:48 Jingbo Xu
2023-09-28 14:36 ` Gao Xiang
0 siblings, 1 reply; 4+ messages in thread
From: Jingbo Xu @ 2023-09-28 13:48 UTC (permalink / raw)
To: xiang, chao, linux-erofs; +Cc: huyue2, linux-doc, linux-kernel
- update new features like bloom filter and DEFLATE.
- add documentation for the long xattr name prefixes, which was
landed upstream since v6.4.
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
---
changes: fix minor grammar issues
v1: https://lore.kernel.org/all/20230928131600.84701-1-jefflexu@linux.alibaba.com/
---
Documentation/filesystems/erofs.rst | 40 ++++++++++++++++++++++++++---
1 file changed, 37 insertions(+), 3 deletions(-)
diff --git a/Documentation/filesystems/erofs.rst b/Documentation/filesystems/erofs.rst
index 4654ee57c1d5..f200d7874495 100644
--- a/Documentation/filesystems/erofs.rst
+++ b/Documentation/filesystems/erofs.rst
@@ -58,12 +58,14 @@ Here are the main features of EROFS:
- Support extended attributes as an option;
+ - Support a bloom filter that speeds up negative extended attribute lookups;
+
- Support POSIX.1e ACLs by using extended attributes;
- Support transparent data compression as an option:
- LZ4 and MicroLZMA algorithms can be used on a per-file basis; In addition,
- inplace decompression is also supported to avoid bounce compressed buffers
- and page cache thrashing.
+ LZ4, MicroLZMA and DEFLATE algorithms can be used on a per-file basis; In
+ addition, inplace decompression is also supported to avoid bounce compressed
+ buffers and unnecessary page cache thrashing.
- Support chunk-based data deduplication and rolling-hash compressed data
deduplication;
@@ -268,6 +270,38 @@ details.)
By the way, chunk-based files are all uncompressed for now.
+Long extended attribute name prefixes
+-------------------------------------
+There are use cases where extended attributes with different values can have
+only a few common prefixes (such as overlayfs xattrs). The predefined prefixes
+work inefficiently in both image size and runtime performance in such cases.
+
+The long xattr name prefixes feature is introduced to address this issue. The
+overall idea is that, apart from the existing predefined prefixes, the xattr
+entry could also refer to user-specified long xattr name prefixes, e.g.
+"trusted.overlay.".
+
+When referring to a long xattr name prefix, the highest bit (bit 7) of
+erofs_xattr_entry.e_name_index is set, while the lower bits (bit 0-6) as a whole
+represent the index of the referred long name prefix among all long name
+prefixes. Therefore, only the trailing part of the name apart from the long
+xattr name prefix is stored in erofs_xattr_entry.e_name, which could be empty if
+the full xattr name matches exactly as its long xattr name prefix.
+
+All long xattr prefixes are stored one by one in the packed inode as long as
+the packed inode is valid, or in the meta inode otherwise. The
+xattr_prefix_count (of the on-disk superblock) indicates the total number of
+long xattr name prefixes, while (xattr_prefix_start * 4) indicates the start
+offset of long name prefixes in the packed/meta inode. Note that, long extended
+attribute name prefixes are disabled if xattr_prefix_count is 0.
+
+Each long name prefix is stored in the format: ALIGN({__le16 len, data}, 4),
+where len represents the total size of the data part. The data part is actually
+represented by 'struct erofs_xattr_long_prefix', where base_index represents the
+index of the predefined xattr name prefix, e.g. EROFS_XATTR_INDEX_TRUSTED for
+"trusted.overlay." long name prefix, while the infix string keeps the string
+after stripping the short prefix, e.g. "overlay." for the example above.
+
Data compression
----------------
EROFS implements fixed-sized output compression which generates fixed-sized
--
2.19.1.6.gb485710b
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2023-09-28 14:36 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20191207023726.5359-1-hsiangkao.ref@aol.com>
2019-12-07 2:37 ` [PATCH] erofs: update documentation Gao Xiang
2019-12-07 2:55 ` [PATCH v2] " Gao Xiang
2023-09-28 13:48 Jingbo Xu
2023-09-28 14:36 ` Gao Xiang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).