* [PATCH 0/3] f2fs-tools: sload.f2fs to support compression @ 2020-12-02 0:57 ` Robin Hsu 0 siblings, 0 replies; 8+ messages in thread From: Robin Hsu @ 2020-12-02 0:57 UTC (permalink / raw) To: linux-f2fs-devel, jaegeuk, chao; +Cc: linux-kernel, Robin Hsu From: Robin Hsu <robinhsu@google.com> * 3 patch set: #1: added some #ifdef for easier support #2: main code change #3: automake changes Robin Hsu (3): f2fs-tools: Added #ifdef WITH_func f2fs-tools:sload.f2fs compression support f2fs-tools:sload.f2fs compress: Fixed automake configure.ac | 32 +++++++ fsck/Makefile.am | 9 +- fsck/compress_wrapper.c | 102 ++++++++++++++++++++ fsck/compress_wrapper.h | 22 +++++ fsck/fsck.h | 15 +++ fsck/main.c | 154 +++++++++++++++++++++++++++++- fsck/segment.c | 202 +++++++++++++++++++++++++++++++++++++--- fsck/sload.c | 67 +++++++++++++ include/f2fs_fs.h | 76 ++++++++++++++- lib/libf2fs_io.c | 33 +++++++ 10 files changed, 695 insertions(+), 17 deletions(-) create mode 100644 fsck/compress_wrapper.c create mode 100644 fsck/compress_wrapper.h -- 2.29.2.454.gaff20da3a2-goog ^ permalink raw reply [flat|nested] 8+ messages in thread
* [f2fs-dev] [PATCH 0/3] f2fs-tools: sload.f2fs to support compression @ 2020-12-02 0:57 ` Robin Hsu 0 siblings, 0 replies; 8+ messages in thread From: Robin Hsu @ 2020-12-02 0:57 UTC (permalink / raw) To: linux-f2fs-devel, jaegeuk, chao; +Cc: linux-kernel From: Robin Hsu <robinhsu@google.com> * 3 patch set: #1: added some #ifdef for easier support #2: main code change #3: automake changes Robin Hsu (3): f2fs-tools: Added #ifdef WITH_func f2fs-tools:sload.f2fs compression support f2fs-tools:sload.f2fs compress: Fixed automake configure.ac | 32 +++++++ fsck/Makefile.am | 9 +- fsck/compress_wrapper.c | 102 ++++++++++++++++++++ fsck/compress_wrapper.h | 22 +++++ fsck/fsck.h | 15 +++ fsck/main.c | 154 +++++++++++++++++++++++++++++- fsck/segment.c | 202 +++++++++++++++++++++++++++++++++++++--- fsck/sload.c | 67 +++++++++++++ include/f2fs_fs.h | 76 ++++++++++++++- lib/libf2fs_io.c | 33 +++++++ 10 files changed, 695 insertions(+), 17 deletions(-) create mode 100644 fsck/compress_wrapper.c create mode 100644 fsck/compress_wrapper.h -- 2.29.2.454.gaff20da3a2-goog _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/3] f2fs-tools: Added #ifdef WITH_func 2020-12-02 0:57 ` [f2fs-dev] " Robin Hsu @ 2020-12-02 0:57 ` Robin Hsu -1 siblings, 0 replies; 8+ messages in thread From: Robin Hsu @ 2020-12-02 0:57 UTC (permalink / raw) To: linux-f2fs-devel, jaegeuk, chao; +Cc: linux-kernel, Robin Hsu From: Robin Hsu <robinhsu@google.com> Add proprocessor defines (options) 'WITH_func', where func = DUMP, DEFRAG, RESIZE, or SLOAD Signed-off-by: Robin Hsu <robinhsu@google.com> --- fsck/main.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/fsck/main.c b/fsck/main.c index e70048b..b20498f 100644 --- a/fsck/main.c +++ b/fsck/main.c @@ -345,6 +345,7 @@ void f2fs_parse_options(int argc, char *argv[]) break; } } else if (!strcmp("dump.f2fs", prog)) { +#ifdef WITH_DUMP const char *option_string = "d:i:n:s:Sa:b:V"; static struct dump_option dump_opt = { .nid = 0, /* default root ino */ @@ -426,7 +427,9 @@ void f2fs_parse_options(int argc, char *argv[]) } c.private = &dump_opt; +#endif } else if (!strcmp("defrag.f2fs", prog)) { +#ifdef WITH_DEFRAG const char *option_string = "d:s:Sl:t:iV"; c.func = DEFRAG; @@ -484,7 +487,9 @@ void f2fs_parse_options(int argc, char *argv[]) if (err != NOERROR) break; } +#endif } else if (!strcmp("resize.f2fs", prog)) { +#ifdef WITH_RESIZE const char *option_string = "d:st:iV"; c.func = RESIZE; @@ -526,7 +531,9 @@ void f2fs_parse_options(int argc, char *argv[]) if (err != NOERROR) break; } +#endif } else if (!strcmp("sload.f2fs", prog)) { +#ifdef WITH_SLOAD const char *option_string = "C:d:f:p:s:St:T:V"; #ifdef HAVE_LIBSELINUX int max_nr_opt = (int)sizeof(c.seopt_file) / @@ -595,6 +602,7 @@ void f2fs_parse_options(int argc, char *argv[]) if (err != NOERROR) break; } +#endif /* WITH_SLOAD */ } if (err == NOERROR) { @@ -707,6 +715,7 @@ static int do_fsck(struct f2fs_sb_info *sbi) return FSCK_ERRORS_LEFT_UNCORRECTED; } +#ifdef WITH_DUMP static void do_dump(struct f2fs_sb_info *sbi) { struct dump_option *opt = (struct dump_option *)c.private; @@ -733,7 +742,9 @@ static void do_dump(struct f2fs_sb_info *sbi) print_cp_state(flag); } +#endif +#ifdef WITH_DEFRAG static int do_defrag(struct f2fs_sb_info *sbi) { struct f2fs_super_block *sb = F2FS_RAW_SUPER(sbi); @@ -782,7 +793,9 @@ out_range: c.defrag_target); return -1; } +#endif +#ifdef WITH_RESIZE static int do_resize(struct f2fs_sb_info *sbi) { if (!c.target_sectors) @@ -796,7 +809,9 @@ static int do_resize(struct f2fs_sb_info *sbi) return f2fs_resize(sbi); } +#endif +#ifdef WITH_SLOAD static int do_sload(struct f2fs_sb_info *sbi) { if (!c.from_dir) { @@ -808,6 +823,7 @@ static int do_sload(struct f2fs_sb_info *sbi) return f2fs_sload(sbi); } +#endif #if defined(__APPLE__) static u64 get_boottime_ns() -- 2.29.2.454.gaff20da3a2-goog ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [f2fs-dev] [PATCH 1/3] f2fs-tools: Added #ifdef WITH_func @ 2020-12-02 0:57 ` Robin Hsu 0 siblings, 0 replies; 8+ messages in thread From: Robin Hsu @ 2020-12-02 0:57 UTC (permalink / raw) To: linux-f2fs-devel, jaegeuk, chao; +Cc: linux-kernel From: Robin Hsu <robinhsu@google.com> Add proprocessor defines (options) 'WITH_func', where func = DUMP, DEFRAG, RESIZE, or SLOAD Signed-off-by: Robin Hsu <robinhsu@google.com> --- fsck/main.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/fsck/main.c b/fsck/main.c index e70048b..b20498f 100644 --- a/fsck/main.c +++ b/fsck/main.c @@ -345,6 +345,7 @@ void f2fs_parse_options(int argc, char *argv[]) break; } } else if (!strcmp("dump.f2fs", prog)) { +#ifdef WITH_DUMP const char *option_string = "d:i:n:s:Sa:b:V"; static struct dump_option dump_opt = { .nid = 0, /* default root ino */ @@ -426,7 +427,9 @@ void f2fs_parse_options(int argc, char *argv[]) } c.private = &dump_opt; +#endif } else if (!strcmp("defrag.f2fs", prog)) { +#ifdef WITH_DEFRAG const char *option_string = "d:s:Sl:t:iV"; c.func = DEFRAG; @@ -484,7 +487,9 @@ void f2fs_parse_options(int argc, char *argv[]) if (err != NOERROR) break; } +#endif } else if (!strcmp("resize.f2fs", prog)) { +#ifdef WITH_RESIZE const char *option_string = "d:st:iV"; c.func = RESIZE; @@ -526,7 +531,9 @@ void f2fs_parse_options(int argc, char *argv[]) if (err != NOERROR) break; } +#endif } else if (!strcmp("sload.f2fs", prog)) { +#ifdef WITH_SLOAD const char *option_string = "C:d:f:p:s:St:T:V"; #ifdef HAVE_LIBSELINUX int max_nr_opt = (int)sizeof(c.seopt_file) / @@ -595,6 +602,7 @@ void f2fs_parse_options(int argc, char *argv[]) if (err != NOERROR) break; } +#endif /* WITH_SLOAD */ } if (err == NOERROR) { @@ -707,6 +715,7 @@ static int do_fsck(struct f2fs_sb_info *sbi) return FSCK_ERRORS_LEFT_UNCORRECTED; } +#ifdef WITH_DUMP static void do_dump(struct f2fs_sb_info *sbi) { struct dump_option *opt = (struct dump_option *)c.private; @@ -733,7 +742,9 @@ static void do_dump(struct f2fs_sb_info *sbi) print_cp_state(flag); } +#endif +#ifdef WITH_DEFRAG static int do_defrag(struct f2fs_sb_info *sbi) { struct f2fs_super_block *sb = F2FS_RAW_SUPER(sbi); @@ -782,7 +793,9 @@ out_range: c.defrag_target); return -1; } +#endif +#ifdef WITH_RESIZE static int do_resize(struct f2fs_sb_info *sbi) { if (!c.target_sectors) @@ -796,7 +809,9 @@ static int do_resize(struct f2fs_sb_info *sbi) return f2fs_resize(sbi); } +#endif +#ifdef WITH_SLOAD static int do_sload(struct f2fs_sb_info *sbi) { if (!c.from_dir) { @@ -808,6 +823,7 @@ static int do_sload(struct f2fs_sb_info *sbi) return f2fs_sload(sbi); } +#endif #if defined(__APPLE__) static u64 get_boottime_ns() -- 2.29.2.454.gaff20da3a2-goog _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 2/3] f2fs-tools:sload.f2fs compression support 2020-12-02 0:57 ` [f2fs-dev] " Robin Hsu @ 2020-12-02 0:57 ` Robin Hsu -1 siblings, 0 replies; 8+ messages in thread From: Robin Hsu @ 2020-12-02 0:57 UTC (permalink / raw) To: linux-f2fs-devel, jaegeuk, chao; +Cc: linux-kernel, Robin Hsu From: Robin Hsu <robinhsu@google.com> Add F2FS compression support for sload * Support file extension filter, either default-accept or default-deny policy * Support choice of compression algorithm, LZO (version 2) or LZ4 (default) * Support custom log of cluster size * Support minimum number of compressed blocks per cluster (default 1). A cluster will not be compressed if the number can not be met. * suuport -r (read-only) option Signed-off-by: Robin Hsu <robinhsu@google.com> --- fsck/compress_wrapper.c | 102 ++++++++++++++++++++ fsck/compress_wrapper.h | 22 +++++ fsck/fsck.h | 15 +++ fsck/main.c | 138 ++++++++++++++++++++++++++- fsck/segment.c | 202 +++++++++++++++++++++++++++++++++++++--- fsck/sload.c | 67 +++++++++++++ include/f2fs_fs.h | 76 ++++++++++++++- lib/libf2fs_io.c | 33 +++++++ 8 files changed, 641 insertions(+), 14 deletions(-) create mode 100644 fsck/compress_wrapper.c create mode 100644 fsck/compress_wrapper.h diff --git a/fsck/compress_wrapper.c b/fsck/compress_wrapper.c new file mode 100644 index 0000000..2cdc4fd --- /dev/null +++ b/fsck/compress_wrapper.c @@ -0,0 +1,102 @@ +/** + * compress_wrapper.c + * + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : initial created, for sload compression support + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ +#include "f2fs.h" /* for config.h for general environment (non-Android) */ + +#include "compress_wrapper.h" +#ifdef HAVE_LIBLZO2 +#include <lzo/lzo1x.h> /* for lzo1x_1_15_compress() */ +#endif +#ifdef HAVE_LIBLZ4 +#include <lz4.h> /* for LZ4_compress_fast_extState() */ +#endif + +/* + * macro/constants borrowed from kernel header (GPL-2.0): + * include/linux/lzo.h, and include/linux/lz4.h + */ +#ifdef HAVE_LIBLZO2 +#define lzo1x_worst_compress(x) ((x) + (x) / 16 + 64 + 3 + 2) +#define LZO_WORK_SIZE ALIGN_UP(LZO1X_1_15_MEM_COMPRESS, 8) +#endif +#ifdef HAVE_LIBLZ4 +#define LZ4_MEMORY_USAGE 14 +#define LZ4_MAX_INPUT_SIZE 0x7E000000 /* 2 113 929 216 bytes */ +#define LZ4_STREAMSIZE (LZ4_STREAMSIZE_U64 * sizeof(long long)) +#define LZ4_MEM_COMPRESS LZ4_STREAMSIZE +#define LZ4_ACCELERATION_DEFAULT 1 +#define LZ4_WORK_SIZE ALIGN_UP(LZ4_MEM_COMPRESS, 8) +#endif + +#ifdef HAVE_LIBLZO2 +static void lzo_compress_init(struct compress_ctx *cc) +{ + size_t size = cc->cluster_size * F2FS_BLKSIZE; + size_t alloc = size + lzo1x_worst_compress(size) + + COMPRESS_HEADER_SIZE + LZO_WORK_SIZE; + ASSERT((cc->private = qbuf_alloc(alloc)) != NULL); + cc->rbuf = (char *) cc->private + LZO_WORK_SIZE; + cc->cbuf = (struct compress_data *)((char *) cc->rbuf + size); +} + +static int lzo_compress(struct compress_ctx *cc) +{ + int ret = lzo1x_1_15_compress(cc->rbuf, cc->rlen, cc->cbuf->cdata, + (lzo_uintp)(&cc->clen), cc->private); + cc->cbuf->clen = cpu_to_le32(cc->clen); + return ret; +} +#endif + +#ifdef HAVE_LIBLZ4 +static void lz4_compress_init(struct compress_ctx *cc) +{ + size_t size = cc->cluster_size * F2FS_BLKSIZE; + size_t alloc = size + LZ4_COMPRESSBOUND(size) + + COMPRESS_HEADER_SIZE + LZ4_WORK_SIZE; + ASSERT((cc->private = qbuf_alloc(alloc)) != NULL); + cc->rbuf = (char *) cc->private + LZ4_WORK_SIZE; + cc->cbuf = (struct compress_data *)((char *) cc->rbuf + size); +} + +static int lz4_compress(struct compress_ctx *cc) +{ + cc->clen = LZ4_compress_fast_extState(cc->private, cc->rbuf, + (char *)cc->cbuf->cdata, cc->rlen, + cc->rlen - F2FS_BLKSIZE * c.sldc_min_cbpc, + LZ4_ACCELERATION_DEFAULT); + + if (!cc->clen) + return 1; + + cc->cbuf->clen = cpu_to_le32(cc->clen); + return 0; +} +#endif + +const char *ca_names[] = { + "LZO", + "LZ4", + "", /* end of the name list */ +}; + +compress_ops compr_ops[] = { +#ifdef HAVE_LIBLZO2 + {lzo_compress_init, lzo_compress}, +#else + {NULL, NULL}, +#endif +#ifdef HAVE_LIBLZ4 + {lz4_compress_init, lz4_compress}, +#else + {NULL, NULL}, +#endif +}; diff --git a/fsck/compress_wrapper.h b/fsck/compress_wrapper.h new file mode 100644 index 0000000..ec33d43 --- /dev/null +++ b/fsck/compress_wrapper.h @@ -0,0 +1,22 @@ +/** + * compress_wrapper.h + * + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : initial created, for sload compression support + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef COMPRESS_WRAPPER_H +#define COMPRESS_WRAPPER_H + +#include "f2fs_fs.h" +extern compress_ops compr_ops[]; /* [0]: LZO, [1]: LZ4, */ + +#endif /* COMPRESS_WRAPPER_H */ diff --git a/fsck/fsck.h b/fsck/fsck.h index c5e85fe..4e866ec 100644 --- a/fsck/fsck.h +++ b/fsck/fsck.h @@ -3,6 +3,9 @@ * * Copyright (c) 2013 Samsung Electronics Co., Ltd. * http://www.samsung.com/ + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -266,6 +269,9 @@ int f2fs_resize(struct f2fs_sb_info *); /* sload.c */ int f2fs_sload(struct f2fs_sb_info *); +void sldc_erase_bufs(struct compress_ctx *cc); +void sload_countblk(void); +extern struct ext_tbl_op ext_filter; /* segment.c */ int reserve_new_block(struct f2fs_sb_info *, block_t *, @@ -282,7 +288,16 @@ block_t new_node_block(struct f2fs_sb_info *, struct quota_file; u64 f2fs_quota_size(struct quota_file *); u64 f2fs_read(struct f2fs_sb_info *, nid_t, u8 *, u64, pgoff_t); +enum wr_addr_type { + WR_NORMAL = 1, + WR_COMPRESS_DATA = 2, + WR_NULL_ADDR = NULL_ADDR, /* 0 */ + WR_NEW_ADDR = NEW_ADDR, /* -1U */ + WR_COMPRESS_ADDR = COMPRESS_ADDR, /* -2U */ +}; u64 f2fs_write(struct f2fs_sb_info *, nid_t, u8 *, u64, pgoff_t); +u64 f2fs_write_compress_data(struct f2fs_sb_info *, nid_t, u8 *, u64, pgoff_t); +u64 f2fs_write_addrtag(struct f2fs_sb_info *, nid_t, pgoff_t, unsigned int); void f2fs_filesize_update(struct f2fs_sb_info *, nid_t, u64); int get_dnode_of_data(struct f2fs_sb_info *, struct dnode_of_data *, diff --git a/fsck/main.c b/fsck/main.c index b20498f..6b8fb95 100644 --- a/fsck/main.c +++ b/fsck/main.c @@ -13,6 +13,9 @@ * Copyright (c) 2019 Google Inc. * Robin Hsu <robinhsu@google.com> * : add cache layer + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -25,6 +28,7 @@ #include <getopt.h> #include <stdbool.h> #include "quotaio.h" +#include "compress_wrapper.h" struct f2fs_fsck gfsck; @@ -134,6 +138,17 @@ void sload_usage() MSG(0, " -S sparse_mode\n"); MSG(0, " -t mount point [prefix of target fs path, default:/]\n"); MSG(0, " -T timestamp\n"); + MSG(0, " -c enable compression (default allow policy)\n"); + MSG(0, " ------------ Compression sub-options -----------------\n"); + MSG(0, " -L <log-of-blocks-per-cluster>, default 2\n"); + MSG(0, " -a <algorithm> compression algorithm, default LZ4\n"); + MSG(0, " -x <ext> compress files except for these extensions.\n"); + MSG(0, " -i <ext> compress files with these extensions only.\n"); + MSG(0, " * -i or -x: use it many times for multiple extensions.\n"); + MSG(0, " * -i and -x cannot be used together..\n"); + MSG(0, " -m <num> min compressed blocks per cluster\n"); + MSG(0, " -r readonly (IMMUTABLE) for compressed files\n"); + MSG(0, " ------------------------------------------------------\n"); MSG(0, " -d debug level [default:0]\n"); MSG(0, " -V print the version number and exit\n"); exit(1); @@ -534,7 +549,7 @@ void f2fs_parse_options(int argc, char *argv[]) #endif } else if (!strcmp("sload.f2fs", prog)) { #ifdef WITH_SLOAD - const char *option_string = "C:d:f:p:s:St:T:V"; + const char *option_string = "cL:a:i:x:m:rC:d:f:p:s:St:T:V"; #ifdef HAVE_LIBSELINUX int max_nr_opt = (int)sizeof(c.seopt_file) / sizeof(c.seopt_file[0]); @@ -543,8 +558,82 @@ void f2fs_parse_options(int argc, char *argv[]) char *p; c.func = SLOAD; + c.sldc_cc.log_cluster_size = 2; + c.sldc_ca = CA_LZ4; + c.sldc_min_cbpc = 1; + c.sldc_ef = &ext_filter; while ((option = getopt(argc, argv, option_string)) != EOF) { + unsigned int i; + int val; + switch (option) { + case 'c': /* compression support */ + c.sldc_en = true; + break; + case 'L': /* compression: log of blocks-per-cluster */ + c.sldc_got_opt = true; + val = atoi(optarg); + if (val < MIN_COMPRESS_LOG_SIZE || + val > MAX_COMPRESS_LOG_SIZE) { + MSG(0, "\tError: log of blocks per" + " cluster must be in the range" + " of %d .. %d.\n", + MIN_COMPRESS_LOG_SIZE, + MAX_COMPRESS_LOG_SIZE); + error_out(prog); + } + c.sldc_cc.log_cluster_size = val; + break; + case 'a': /* compression: choose algorithm */ + c.sldc_got_opt = true; + c.sldc_ca = (u8)-1; + for (i = 0; ca_names[i][0] != 0; i++) { + if (!strcmp(ca_names[i], optarg)) { + c.sldc_ca = i; + break; + } + } + if (c.sldc_ca == (u8)-1) { + MSG(0, "\tError: Unknown compression" + " algorithm %s\n", optarg); + error_out(prog); + } + break; + case 'i': /* compress only these extensions */ + c.sldc_got_opt = true; + if (c.sldc_policy == FP_ALLOW) { + MSG(0, "\tError: could not mix option" + " -i and -x\n"); + error_out(prog); + } + c.sldc_policy = FP_DENY; + c.sldc_ef->add(optarg); + break; + case 'x': /* compress except for these extensions */ + c.sldc_got_opt = true; + if (c.sldc_policy == FP_DENY) { + MSG(0, "\tError: could not mix option" + " -i and -x\n"); + error_out(prog); + } + c.sldc_policy = FP_ALLOW; + c.sldc_ef->add(optarg); + break; + case 'm': /* minimum compressed blocks per cluster */ + c.sldc_got_opt = true; + val = atoi(optarg); + if (val <= 0) { + MSG(0, "\tError: minimum compressed" + " blocks per cluster must be" + " positive.\n"); + error_out(prog); + } + c.sldc_min_cbpc = val; + break; + case 'r': /* compress file to set IMMUTABLE */ + c.sldc_got_opt = true; + c.sldc_immutable = true; + break; case 'C': c.fs_config_file = absolute_path(optarg); break; @@ -602,6 +691,27 @@ void f2fs_parse_options(int argc, char *argv[]) if (err != NOERROR) break; } + if (c.sldc_got_opt && !c.sldc_en) { + MSG(0, "\tError: compression sub-options are used" + " without the compression enable (-c) option\n" + ); + error_out(prog); + } + if (err == NOERROR && c.sldc_en) { + c.sldc_cc.cluster_size = 1 + << c.sldc_cc.log_cluster_size; + if (c.sldc_policy == FP_UNASSIGNED) + c.sldc_policy = FP_ALLOW; + if (c.sldc_min_cbpc >= c.sldc_cc.cluster_size) { + MSG(0, "\tError: minimum reduced blocks by" + " compression per cluster must be at" + " most one less than blocks per" + " cluster, i.e. %d\n", + c.sldc_cc.cluster_size - 1); + error_out(prog); + } + qbuf_init(); + } #endif /* WITH_SLOAD */ } @@ -812,6 +922,26 @@ static int do_resize(struct f2fs_sb_info *sbi) #endif #ifdef WITH_SLOAD +void init_compr(struct f2fs_sb_info *sbi) +{ + if (!(sbi->raw_super->feature + & cpu_to_le32(F2FS_FEATURE_COMPRESSION))) { + printf("Error: Compression (-c) was requested " + "but the file system is not created " + "with such feature.\n"); + abort(); + } + c.sldc_compr = compr_ops + c.sldc_ca; + if (c.sldc_compr == NULL) { + printf("Error: The selected compression algorithm is not\n" + " supported\n"); + abort(); + } + c.sldc_compr->init(&c.sldc_cc); + sldc_erase_bufs(&c.sldc_cc); + c.sldc_cc.rlen = c.sldc_cc.cluster_size * F2FS_BLKSIZE; +} + static int do_sload(struct f2fs_sb_info *sbi) { if (!c.from_dir) { @@ -821,6 +951,9 @@ static int do_sload(struct f2fs_sb_info *sbi) if (!c.mount_point) c.mount_point = "/"; + if (c.sldc_en) + init_compr(sbi); + return f2fs_sload(sbi); } #endif @@ -971,6 +1104,9 @@ retry: return ret2; } + if (c.func == SLOAD) + c.sldc_ef->destroy(); + printf("\nDone: %lf secs\n", (get_boottime_ns() - start) / 1000000000.0); return ret; diff --git a/fsck/segment.c b/fsck/segment.c index 0487f41..e4c8cea 100644 --- a/fsck/segment.c +++ b/fsck/segment.c @@ -8,6 +8,9 @@ * Hou Pengyang <houpengyang@huawei.com> * Liu Shuoran <liushuoran@huawei.com> * Jaegeuk Kim <jaegeuk@kernel.org> + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -16,6 +19,7 @@ #include "fsck.h" #include "node.h" #include "quotaio.h" +#include "compress_wrapper.h" int reserve_new_block(struct f2fs_sb_info *sbi, block_t *to, struct f2fs_summary *sum, int type, bool is_inode) @@ -228,8 +232,14 @@ u64 f2fs_read(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, return read_count; } -u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, - u64 count, pgoff_t offset) +/* + * Do not call this function directly. Instead, call one of the following: + * u64 f2fs_write(); + * u64 f2fs_write_compress_data(); + * u64 f2fs_write_addrtag(); + */ +static u64 f2fs_write_ex(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, + u64 count, pgoff_t offset, enum wr_addr_type addr_type) { struct dnode_of_data dn; struct node_info ni; @@ -243,6 +253,19 @@ u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, void* index_node = NULL; int idirty = 0; int err; + bool has_data = (addr_type == WR_NORMAL + || addr_type == WR_COMPRESS_DATA); + + if (count == 0) + return 0; + + /* + * Enforce calling from f2fs_write(), f2fs_write_compress_data(), + * and f2fs_write_addrtag(). Beside, check if is properly called. + */ + ASSERT((!has_data && buffer == NULL) || (has_data && buffer != NULL)); + if (addr_type != WR_NORMAL) + ASSERT(offset % F2FS_BLKSIZE == 0); /* block boundary only */ /* Memory allocation for block buffer and inode. */ blk_buffer = calloc(BLOCK_SZ, 2); @@ -265,15 +288,26 @@ u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, if (err) break; idirty |= dn.idirty; - if (index_node) - free(index_node); + free(index_node); index_node = (dn.node_blk == dn.inode_blk) ? - NULL : dn.node_blk; + NULL : dn.node_blk; remained_blkentries = ADDRS_PER_PAGE(sbi, - dn.node_blk, dn.inode_blk); + dn.node_blk, dn.inode_blk) - + dn.ofs_in_node; } ASSERT(remained_blkentries > 0); + if (!has_data) { + dn.data_blkaddr = addr_type; + set_data_blkaddr(&dn); + idirty |= dn.idirty; + if (dn.ndirty) + ASSERT(dev_write_block(dn.node_blk, + dn.node_blkaddr) >= 0); + written_count = 0; + break; + } + blkaddr = datablock_addr(dn.node_blk, dn.ofs_in_node); if (blkaddr == NULL_ADDR || blkaddr == NEW_ADDR) { err = new_data_block(sbi, blk_buffer, @@ -281,6 +315,7 @@ u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, if (err) break; blkaddr = dn.data_blkaddr; + idirty |= dn.idirty; } off_in_blk = offset % BLOCK_SZ; @@ -305,9 +340,10 @@ u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, dn.ofs_in_node++; if ((--remained_blkentries == 0 || count == 0) && (dn.ndirty)) - ASSERT(dev_write_block(dn.node_blk, dn.node_blkaddr) >= 0); + ASSERT(dev_write_block(dn.node_blk, dn.node_blkaddr) + >= 0); } - if (offset > le64_to_cpu(inode->i.i_size)) { + if (addr_type == WR_NORMAL && offset > le64_to_cpu(inode->i.i_size)) { inode->i.i_size = cpu_to_le64(offset); idirty = 1; } @@ -315,13 +351,33 @@ u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, ASSERT(inode == dn.inode_blk); ASSERT(write_inode(inode, ni.blk_addr) >= 0); } - if (index_node) - free(index_node); + + free(index_node); free(blk_buffer); return written_count; } +u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, + u64 count, pgoff_t offset) +{ + return f2fs_write_ex(sbi, ino, buffer, count, offset, WR_NORMAL); +} + +u64 f2fs_write_compress_data(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, + u64 count, pgoff_t offset) +{ + return f2fs_write_ex(sbi, ino, buffer, count, offset, WR_COMPRESS_DATA); +} + +u64 f2fs_write_addrtag(struct f2fs_sb_info *sbi, nid_t ino, pgoff_t offset, + unsigned int addrtag) +{ + ASSERT(addrtag == COMPRESS_ADDR || addrtag == NEW_ADDR + || addrtag == NULL_ADDR); + return f2fs_write_ex(sbi, ino, NULL, F2FS_BLKSIZE, offset, addrtag); +} + /* This function updates only inode->i.i_size */ void f2fs_filesize_update(struct f2fs_sb_info *sbi, nid_t ino, u64 filesize) { @@ -342,11 +398,59 @@ void f2fs_filesize_update(struct f2fs_sb_info *sbi, nid_t ino, u64 filesize) free(inode); } +#define MAX_BULKR_RETRY 5 +int bulkread(int fd, void *rbuf, size_t rsize, bool *eof) +{ + int n = 0; + int retry = MAX_BULKR_RETRY; + int cur; + + if (!rsize) + return 0; + + if (eof != NULL) + *eof = false; + while (rsize && (cur = read(fd, rbuf, rsize)) != 0) { + if (cur == -1) { + if (errno == EINTR && retry--) + continue; + return -1; + } + retry = MAX_BULKR_RETRY; + + rsize -= cur; + n += cur; + } + if (eof != NULL) + *eof = (cur == 0); + return n; +} + +u64 f2fs_fix_mutable(struct f2fs_sb_info *sbi, nid_t ino, pgoff_t offset, + unsigned int compressed) +{ + unsigned int i; + u64 wlen; + + if (c.sldc_immutable) + return 0; + + for (i = 0; i < compressed - 1; i++) { + wlen = f2fs_write_addrtag(sbi, ino, + offset + (i << F2FS_BLKSIZE_BITS), NEW_ADDR); + if (wlen) + return wlen; + } + return 0; +} + int f2fs_build_file(struct f2fs_sb_info *sbi, struct dentry *de) { int fd, n; pgoff_t off = 0; u8 buffer[BLOCK_SZ]; + struct node_info ni; + struct f2fs_node *node_blk; if (de->ino == 0) return -1; @@ -359,8 +463,6 @@ int f2fs_build_file(struct f2fs_sb_info *sbi, struct dentry *de) /* inline_data support */ if (de->size <= DEF_MAX_INLINE_DATA) { - struct node_info ni; - struct f2fs_node *node_blk; int ret; get_node_info(sbi, de->ino, &ni); @@ -385,6 +487,82 @@ int f2fs_build_file(struct f2fs_sb_info *sbi, struct dentry *de) node_blk->i.i_size = cpu_to_le64(de->size); ASSERT(write_inode(node_blk, ni.blk_addr) >= 0); free(node_blk); +#ifdef WITH_SLOAD + } else if (c.func == SLOAD && c.sldc_en && + c.sldc_ef->filter(de->full_path)) { + bool eof = false; + u8 *rbuf = c.sldc_cc.rbuf; + unsigned int cblocks = 0; + + node_blk = calloc(BLOCK_SZ, 1); + ASSERT(node_blk); + + /* read inode */ + get_node_info(sbi, de->ino, &ni); + ASSERT(dev_read_block(node_blk, ni.blk_addr) >= 0); + /* update inode meta */ + node_blk->i.i_compress_algrithm = c.sldc_ca; + node_blk->i.i_log_cluster_size = + c.sldc_cc.log_cluster_size; + node_blk->i.i_flags = cpu_to_le32( + F2FS_COMPR_FL | + (c.sldc_immutable ? FS_IMMUTABLE_FL : 0)); + ASSERT(write_inode(node_blk, ni.blk_addr) >= 0); + + while (!eof && (n = bulkread(fd, rbuf, c.sldc_cc.rlen, + &eof)) > 0) { + int ret = c.sldc_compr->compress(&c.sldc_cc); + u64 wlen; + u32 csize = ALIGN_UP(c.sldc_cc.clen + + COMPRESS_HEADER_SIZE, BLOCK_SZ); + unsigned int cur_cblk; + + if (ret || n < (int)(csize + BLOCK_SZ * + c.sldc_min_cbpc)) { + wlen = f2fs_write(sbi, de->ino, rbuf, n, off); + ASSERT((int)wlen == n); + } else { + wlen = f2fs_write_addrtag(sbi, de->ino, off, + WR_COMPRESS_ADDR); + ASSERT(!wlen); + wlen = f2fs_write_compress_data(sbi, de->ino, + (u8 *)c.sldc_cc.cbuf, + csize, off + BLOCK_SZ); + ASSERT(wlen == csize); + sldc_erase_bufs(&c.sldc_cc); + cur_cblk = (c.sldc_cc.rlen - csize) / BLOCK_SZ; + cblocks += cur_cblk; + wlen = f2fs_fix_mutable(sbi, de->ino, + off + BLOCK_SZ + csize, + cur_cblk); + ASSERT(!wlen); + } + off += n; + } + if (n == -1) { + fprintf(stderr, "Load file '%s' failed: ", + de->full_path); + perror(NULL); + } + /* read inode */ + get_node_info(sbi, de->ino, &ni); + ASSERT(dev_read_block(node_blk, ni.blk_addr) >= 0); + /* update inode meta */ + node_blk->i.i_size = cpu_to_le64(off); + if (!c.sldc_immutable) + node_blk->i.i_compr_blocks = cpu_to_le64(cblocks); + ASSERT(write_inode(node_blk, ni.blk_addr) >= 0); + free(node_blk); + + if (!c.sldc_immutable) { + sbi->total_valid_block_count += cblocks; + if (sbi->total_valid_block_count >= + sbi->user_block_count) { + ERR_MSG("Not enough space\n"); + ASSERT(0); + } + } +#endif } else { while ((n = read(fd, buffer, BLOCK_SZ)) > 0) { f2fs_write(sbi, de->ino, buffer, n, off); diff --git a/fsck/sload.c b/fsck/sload.c index 14012fb..13e523a 100644 --- a/fsck/sload.c +++ b/fsck/sload.c @@ -6,6 +6,9 @@ * Hou Pengyang <houpengyang@huawei.com> * Liu Shuoran <liushuoran@huawei.com> * Jaegeuk Kim <jaegeuk@kernel.org> + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -317,6 +320,70 @@ skip: return 0; } +typedef struct _ext_tbl { + const char *ext; + struct _ext_tbl *next; /* linked list */ +} ext_tbl_t; +static ext_tbl_t *ext_tbl; + +static bool ext_found(const char *ext) +{ + ext_tbl_t *p = ext_tbl; + + while (p != NULL && strcmp(ext, p->ext)) + p = p->next; + return (p != NULL); +} + +static const char *get_ext(const char *path) +{ + char *p = strrchr(path, '.'); + return p == NULL ? path + strlen(path) : p + 1; +} + +static bool ext_do_filter(const char *path) +{ + return (ext_found(get_ext(path)) == true) ^ (c.sldc_policy == FP_ALLOW); +} + +static void ext_filter_add(const char *ext) +{ + ext_tbl_t *node; + + ASSERT(ext != NULL); + if (ext_found(ext)) + return; /* ext was already registered */ + node = malloc(sizeof(ext_tbl_t)); + ASSERT(node != NULL); + node->ext = ext; + node->next = ext_tbl; + ext_tbl = node; +} + +static void ext_filter_destroy(void) +{ + ext_tbl_t *p; + + while (ext_tbl != NULL) { + p = ext_tbl; + ext_tbl = p->next; + free(p); + } +} + +struct ext_tbl_op ext_filter = { + .add = ext_filter_add, + .destroy = ext_filter_destroy, + .filter = ext_do_filter, +}; + +void sldc_erase_bufs(struct compress_ctx *cc) +{ + memset(cc->rbuf, 0, cc->cluster_size * F2FS_BLKSIZE); + memset(cc->cbuf->cdata, 0, cc->cluster_size * F2FS_BLKSIZE + - F2FS_BLKSIZE); +} + int f2fs_sload(struct f2fs_sb_info *sbi) { int ret = 0; diff --git a/include/f2fs_fs.h b/include/f2fs_fs.h index 1348e39..2a2dc15 100644 --- a/include/f2fs_fs.h +++ b/include/f2fs_fs.h @@ -5,6 +5,9 @@ * http://www.samsung.com/ * Copyright (c) 2019 Google Inc. * http://www.google.com/ + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support * * Dual licensed under the GPL or LGPL version 2 licenses. * @@ -68,6 +71,10 @@ typedef uint16_t u_int16_t; typedef uint8_t u_int8_t; #endif +/* codes from kernel's f2fs.h, GPL-v2.0 */ +#define MIN_COMPRESS_LOG_SIZE 2 +#define MAX_COMPRESS_LOG_SIZE 8 + typedef u_int64_t u64; typedef u_int32_t u32; typedef u_int16_t u16; @@ -93,6 +100,31 @@ typedef u32 __be32; typedef u64 __be64; #endif +/* + * code borrowed from kernel f2fs dirver: f2fs.h, GPL-2.0 + * : definitions of COMPRESS_DATA_RESERVED_SIZE, + * struct compress_data, COMPRESS_HEADER_SIZE, + * and struct compress_ctx + */ +#define COMPRESS_DATA_RESERVED_SIZE 4 +struct compress_data { + __le32 clen; /* compressed data size */ + __le32 chksum; /* checksum of compressed data */ + __le32 reserved[COMPRESS_DATA_RESERVED_SIZE]; /* reserved */ + u8 cdata[]; /* compressed data */ +}; +#define COMPRESS_HEADER_SIZE (sizeof(struct compress_data)) +/* compress context */ +struct compress_ctx { + unsigned int cluster_size; /* page count in cluster */ + unsigned int log_cluster_size; /* log of cluster size */ + void *rbuf; /* compression input buffer */ + struct compress_data *cbuf; /* comprsssion output header + data */ + size_t rlen; /* valid data length in rbuf */ + size_t clen; /* valid data length in cbuf */ + void *private; /* work buf for compress algorithm */ +}; + #if HAVE_BYTESWAP_H #include <byteswap.h> #else @@ -345,6 +377,25 @@ typedef struct { bool dbg_en; } dev_cache_config_t; +/* f2fs_configration: sldc_ca, the sload compress algorithm */ +enum {CA_LZO, CA_LZ4}; +extern const char *ca_names[]; + +typedef struct { + void (*init)(struct compress_ctx *cc); + int (*compress)(struct compress_ctx *cc); +} compress_ops; + +#define ALIGN_UP(value, size) ((value) + ((value) % (size) > 0 ? \ + (size) - (value) % (size) : 0)) + +enum filter_policy {FP_UNASSIGNED = 0, FP_ALLOW, FP_DENY}; +struct ext_tbl_op { + void (*add)(const char *); + void (*destroy)(void); + bool (*filter)(const char *); +}; + struct f2fs_configuration { u_int32_t reserved_segments; u_int32_t new_reserved_segments; @@ -441,6 +492,24 @@ struct f2fs_configuration { /* cache parameters */ dev_cache_config_t cache_config; + + /* quick dynamic buffer */ + bool qbuf_initialized; + size_t qbufsize; + void *qbuf; + + /* sldc: sload compression support */ + bool sldc_en; + bool sldc_use_allow_list; /* default false to use the deny list */ + struct compress_ctx sldc_cc; + u8 sldc_ca; /* compress algorithm: 0 = LZO, 1 = LZ4 */ + compress_ops *sldc_compr; + enum filter_policy sldc_policy; + /* max_cppc can used to specify minimum compression rate */ + unsigned int sldc_min_cbpc; /* min compressed pages per cluster */ + bool sldc_got_opt; + bool sldc_immutable; + struct ext_tbl_op *sldc_ef; /* extension filter */ }; #ifdef CONFIG_64BIT @@ -1226,6 +1295,11 @@ extern void f2fs_release_sparse_resource(void); extern int f2fs_finalize_device(void); extern int f2fs_fsync_device(void); +/* quick (shared) buffer */ +extern void qbuf_free(void); +extern void *qbuf_alloc(size_t size); +extern void qbuf_init(void); + extern void dcache_init(void); extern void dcache_release(void); @@ -1377,7 +1451,7 @@ int f2fs_reset_zone(int, void *); extern int f2fs_reset_zones(int); extern uint32_t f2fs_get_usable_segments(struct f2fs_super_block *sb); -#define SIZE_ALIGN(val, size) ((val) + (size) - 1) / (size) +#define SIZE_ALIGN(val, size) (((val) + (size) - 1) / (size)) #define SEG_ALIGN(blks) SIZE_ALIGN(blks, c.blks_per_seg) #define ZONE_ALIGN(blks) SIZE_ALIGN(blks, c.blks_per_seg * \ c.segs_per_zone) diff --git a/lib/libf2fs_io.c b/lib/libf2fs_io.c index 138285d..0280896 100644 --- a/lib/libf2fs_io.c +++ b/lib/libf2fs_io.c @@ -5,6 +5,9 @@ * http://www.samsung.com/ * Copyright (c) 2019 Google Inc. * http://www.google.com/ + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add quick-buffer for sload compression support * * Dual licensed under the GPL or LGPL version 2 licenses. */ @@ -106,6 +109,36 @@ static long dcache_relocate_offset0[] = { }; static int dcache_relocate_offset[16]; +/* quick (shared) buffer */ +static bool qbuf_initialized; +static void *qbuf; +static size_t qbufsize; +void qbuf_free(void) +{ + ASSERT(qbuf_initialized); + if (qbuf != NULL) { + free(qbuf); + qbuf = NULL; + qbufsize = 0; + } +} +void *qbuf_alloc(size_t size) +{ + ASSERT(qbuf_initialized); + if (size > qbufsize) { + qbuf_free(); + qbuf = malloc(size); + } + return qbuf; +} +void qbuf_init(void) +{ + if (qbuf_initialized) + return; + atexit(qbuf_free); + qbuf_initialized = true; +} + static void dcache_print_statistics(void) { long i; -- 2.29.2.454.gaff20da3a2-goog ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [f2fs-dev] [PATCH 2/3] f2fs-tools:sload.f2fs compression support @ 2020-12-02 0:57 ` Robin Hsu 0 siblings, 0 replies; 8+ messages in thread From: Robin Hsu @ 2020-12-02 0:57 UTC (permalink / raw) To: linux-f2fs-devel, jaegeuk, chao; +Cc: linux-kernel From: Robin Hsu <robinhsu@google.com> Add F2FS compression support for sload * Support file extension filter, either default-accept or default-deny policy * Support choice of compression algorithm, LZO (version 2) or LZ4 (default) * Support custom log of cluster size * Support minimum number of compressed blocks per cluster (default 1). A cluster will not be compressed if the number can not be met. * suuport -r (read-only) option Signed-off-by: Robin Hsu <robinhsu@google.com> --- fsck/compress_wrapper.c | 102 ++++++++++++++++++++ fsck/compress_wrapper.h | 22 +++++ fsck/fsck.h | 15 +++ fsck/main.c | 138 ++++++++++++++++++++++++++- fsck/segment.c | 202 +++++++++++++++++++++++++++++++++++++--- fsck/sload.c | 67 +++++++++++++ include/f2fs_fs.h | 76 ++++++++++++++- lib/libf2fs_io.c | 33 +++++++ 8 files changed, 641 insertions(+), 14 deletions(-) create mode 100644 fsck/compress_wrapper.c create mode 100644 fsck/compress_wrapper.h diff --git a/fsck/compress_wrapper.c b/fsck/compress_wrapper.c new file mode 100644 index 0000000..2cdc4fd --- /dev/null +++ b/fsck/compress_wrapper.c @@ -0,0 +1,102 @@ +/** + * compress_wrapper.c + * + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : initial created, for sload compression support + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ +#include "f2fs.h" /* for config.h for general environment (non-Android) */ + +#include "compress_wrapper.h" +#ifdef HAVE_LIBLZO2 +#include <lzo/lzo1x.h> /* for lzo1x_1_15_compress() */ +#endif +#ifdef HAVE_LIBLZ4 +#include <lz4.h> /* for LZ4_compress_fast_extState() */ +#endif + +/* + * macro/constants borrowed from kernel header (GPL-2.0): + * include/linux/lzo.h, and include/linux/lz4.h + */ +#ifdef HAVE_LIBLZO2 +#define lzo1x_worst_compress(x) ((x) + (x) / 16 + 64 + 3 + 2) +#define LZO_WORK_SIZE ALIGN_UP(LZO1X_1_15_MEM_COMPRESS, 8) +#endif +#ifdef HAVE_LIBLZ4 +#define LZ4_MEMORY_USAGE 14 +#define LZ4_MAX_INPUT_SIZE 0x7E000000 /* 2 113 929 216 bytes */ +#define LZ4_STREAMSIZE (LZ4_STREAMSIZE_U64 * sizeof(long long)) +#define LZ4_MEM_COMPRESS LZ4_STREAMSIZE +#define LZ4_ACCELERATION_DEFAULT 1 +#define LZ4_WORK_SIZE ALIGN_UP(LZ4_MEM_COMPRESS, 8) +#endif + +#ifdef HAVE_LIBLZO2 +static void lzo_compress_init(struct compress_ctx *cc) +{ + size_t size = cc->cluster_size * F2FS_BLKSIZE; + size_t alloc = size + lzo1x_worst_compress(size) + + COMPRESS_HEADER_SIZE + LZO_WORK_SIZE; + ASSERT((cc->private = qbuf_alloc(alloc)) != NULL); + cc->rbuf = (char *) cc->private + LZO_WORK_SIZE; + cc->cbuf = (struct compress_data *)((char *) cc->rbuf + size); +} + +static int lzo_compress(struct compress_ctx *cc) +{ + int ret = lzo1x_1_15_compress(cc->rbuf, cc->rlen, cc->cbuf->cdata, + (lzo_uintp)(&cc->clen), cc->private); + cc->cbuf->clen = cpu_to_le32(cc->clen); + return ret; +} +#endif + +#ifdef HAVE_LIBLZ4 +static void lz4_compress_init(struct compress_ctx *cc) +{ + size_t size = cc->cluster_size * F2FS_BLKSIZE; + size_t alloc = size + LZ4_COMPRESSBOUND(size) + + COMPRESS_HEADER_SIZE + LZ4_WORK_SIZE; + ASSERT((cc->private = qbuf_alloc(alloc)) != NULL); + cc->rbuf = (char *) cc->private + LZ4_WORK_SIZE; + cc->cbuf = (struct compress_data *)((char *) cc->rbuf + size); +} + +static int lz4_compress(struct compress_ctx *cc) +{ + cc->clen = LZ4_compress_fast_extState(cc->private, cc->rbuf, + (char *)cc->cbuf->cdata, cc->rlen, + cc->rlen - F2FS_BLKSIZE * c.sldc_min_cbpc, + LZ4_ACCELERATION_DEFAULT); + + if (!cc->clen) + return 1; + + cc->cbuf->clen = cpu_to_le32(cc->clen); + return 0; +} +#endif + +const char *ca_names[] = { + "LZO", + "LZ4", + "", /* end of the name list */ +}; + +compress_ops compr_ops[] = { +#ifdef HAVE_LIBLZO2 + {lzo_compress_init, lzo_compress}, +#else + {NULL, NULL}, +#endif +#ifdef HAVE_LIBLZ4 + {lz4_compress_init, lz4_compress}, +#else + {NULL, NULL}, +#endif +}; diff --git a/fsck/compress_wrapper.h b/fsck/compress_wrapper.h new file mode 100644 index 0000000..ec33d43 --- /dev/null +++ b/fsck/compress_wrapper.h @@ -0,0 +1,22 @@ +/** + * compress_wrapper.h + * + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : initial created, for sload compression support + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#ifndef COMPRESS_WRAPPER_H +#define COMPRESS_WRAPPER_H + +#include "f2fs_fs.h" +extern compress_ops compr_ops[]; /* [0]: LZO, [1]: LZ4, */ + +#endif /* COMPRESS_WRAPPER_H */ diff --git a/fsck/fsck.h b/fsck/fsck.h index c5e85fe..4e866ec 100644 --- a/fsck/fsck.h +++ b/fsck/fsck.h @@ -3,6 +3,9 @@ * * Copyright (c) 2013 Samsung Electronics Co., Ltd. * http://www.samsung.com/ + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -266,6 +269,9 @@ int f2fs_resize(struct f2fs_sb_info *); /* sload.c */ int f2fs_sload(struct f2fs_sb_info *); +void sldc_erase_bufs(struct compress_ctx *cc); +void sload_countblk(void); +extern struct ext_tbl_op ext_filter; /* segment.c */ int reserve_new_block(struct f2fs_sb_info *, block_t *, @@ -282,7 +288,16 @@ block_t new_node_block(struct f2fs_sb_info *, struct quota_file; u64 f2fs_quota_size(struct quota_file *); u64 f2fs_read(struct f2fs_sb_info *, nid_t, u8 *, u64, pgoff_t); +enum wr_addr_type { + WR_NORMAL = 1, + WR_COMPRESS_DATA = 2, + WR_NULL_ADDR = NULL_ADDR, /* 0 */ + WR_NEW_ADDR = NEW_ADDR, /* -1U */ + WR_COMPRESS_ADDR = COMPRESS_ADDR, /* -2U */ +}; u64 f2fs_write(struct f2fs_sb_info *, nid_t, u8 *, u64, pgoff_t); +u64 f2fs_write_compress_data(struct f2fs_sb_info *, nid_t, u8 *, u64, pgoff_t); +u64 f2fs_write_addrtag(struct f2fs_sb_info *, nid_t, pgoff_t, unsigned int); void f2fs_filesize_update(struct f2fs_sb_info *, nid_t, u64); int get_dnode_of_data(struct f2fs_sb_info *, struct dnode_of_data *, diff --git a/fsck/main.c b/fsck/main.c index b20498f..6b8fb95 100644 --- a/fsck/main.c +++ b/fsck/main.c @@ -13,6 +13,9 @@ * Copyright (c) 2019 Google Inc. * Robin Hsu <robinhsu@google.com> * : add cache layer + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -25,6 +28,7 @@ #include <getopt.h> #include <stdbool.h> #include "quotaio.h" +#include "compress_wrapper.h" struct f2fs_fsck gfsck; @@ -134,6 +138,17 @@ void sload_usage() MSG(0, " -S sparse_mode\n"); MSG(0, " -t mount point [prefix of target fs path, default:/]\n"); MSG(0, " -T timestamp\n"); + MSG(0, " -c enable compression (default allow policy)\n"); + MSG(0, " ------------ Compression sub-options -----------------\n"); + MSG(0, " -L <log-of-blocks-per-cluster>, default 2\n"); + MSG(0, " -a <algorithm> compression algorithm, default LZ4\n"); + MSG(0, " -x <ext> compress files except for these extensions.\n"); + MSG(0, " -i <ext> compress files with these extensions only.\n"); + MSG(0, " * -i or -x: use it many times for multiple extensions.\n"); + MSG(0, " * -i and -x cannot be used together..\n"); + MSG(0, " -m <num> min compressed blocks per cluster\n"); + MSG(0, " -r readonly (IMMUTABLE) for compressed files\n"); + MSG(0, " ------------------------------------------------------\n"); MSG(0, " -d debug level [default:0]\n"); MSG(0, " -V print the version number and exit\n"); exit(1); @@ -534,7 +549,7 @@ void f2fs_parse_options(int argc, char *argv[]) #endif } else if (!strcmp("sload.f2fs", prog)) { #ifdef WITH_SLOAD - const char *option_string = "C:d:f:p:s:St:T:V"; + const char *option_string = "cL:a:i:x:m:rC:d:f:p:s:St:T:V"; #ifdef HAVE_LIBSELINUX int max_nr_opt = (int)sizeof(c.seopt_file) / sizeof(c.seopt_file[0]); @@ -543,8 +558,82 @@ void f2fs_parse_options(int argc, char *argv[]) char *p; c.func = SLOAD; + c.sldc_cc.log_cluster_size = 2; + c.sldc_ca = CA_LZ4; + c.sldc_min_cbpc = 1; + c.sldc_ef = &ext_filter; while ((option = getopt(argc, argv, option_string)) != EOF) { + unsigned int i; + int val; + switch (option) { + case 'c': /* compression support */ + c.sldc_en = true; + break; + case 'L': /* compression: log of blocks-per-cluster */ + c.sldc_got_opt = true; + val = atoi(optarg); + if (val < MIN_COMPRESS_LOG_SIZE || + val > MAX_COMPRESS_LOG_SIZE) { + MSG(0, "\tError: log of blocks per" + " cluster must be in the range" + " of %d .. %d.\n", + MIN_COMPRESS_LOG_SIZE, + MAX_COMPRESS_LOG_SIZE); + error_out(prog); + } + c.sldc_cc.log_cluster_size = val; + break; + case 'a': /* compression: choose algorithm */ + c.sldc_got_opt = true; + c.sldc_ca = (u8)-1; + for (i = 0; ca_names[i][0] != 0; i++) { + if (!strcmp(ca_names[i], optarg)) { + c.sldc_ca = i; + break; + } + } + if (c.sldc_ca == (u8)-1) { + MSG(0, "\tError: Unknown compression" + " algorithm %s\n", optarg); + error_out(prog); + } + break; + case 'i': /* compress only these extensions */ + c.sldc_got_opt = true; + if (c.sldc_policy == FP_ALLOW) { + MSG(0, "\tError: could not mix option" + " -i and -x\n"); + error_out(prog); + } + c.sldc_policy = FP_DENY; + c.sldc_ef->add(optarg); + break; + case 'x': /* compress except for these extensions */ + c.sldc_got_opt = true; + if (c.sldc_policy == FP_DENY) { + MSG(0, "\tError: could not mix option" + " -i and -x\n"); + error_out(prog); + } + c.sldc_policy = FP_ALLOW; + c.sldc_ef->add(optarg); + break; + case 'm': /* minimum compressed blocks per cluster */ + c.sldc_got_opt = true; + val = atoi(optarg); + if (val <= 0) { + MSG(0, "\tError: minimum compressed" + " blocks per cluster must be" + " positive.\n"); + error_out(prog); + } + c.sldc_min_cbpc = val; + break; + case 'r': /* compress file to set IMMUTABLE */ + c.sldc_got_opt = true; + c.sldc_immutable = true; + break; case 'C': c.fs_config_file = absolute_path(optarg); break; @@ -602,6 +691,27 @@ void f2fs_parse_options(int argc, char *argv[]) if (err != NOERROR) break; } + if (c.sldc_got_opt && !c.sldc_en) { + MSG(0, "\tError: compression sub-options are used" + " without the compression enable (-c) option\n" + ); + error_out(prog); + } + if (err == NOERROR && c.sldc_en) { + c.sldc_cc.cluster_size = 1 + << c.sldc_cc.log_cluster_size; + if (c.sldc_policy == FP_UNASSIGNED) + c.sldc_policy = FP_ALLOW; + if (c.sldc_min_cbpc >= c.sldc_cc.cluster_size) { + MSG(0, "\tError: minimum reduced blocks by" + " compression per cluster must be at" + " most one less than blocks per" + " cluster, i.e. %d\n", + c.sldc_cc.cluster_size - 1); + error_out(prog); + } + qbuf_init(); + } #endif /* WITH_SLOAD */ } @@ -812,6 +922,26 @@ static int do_resize(struct f2fs_sb_info *sbi) #endif #ifdef WITH_SLOAD +void init_compr(struct f2fs_sb_info *sbi) +{ + if (!(sbi->raw_super->feature + & cpu_to_le32(F2FS_FEATURE_COMPRESSION))) { + printf("Error: Compression (-c) was requested " + "but the file system is not created " + "with such feature.\n"); + abort(); + } + c.sldc_compr = compr_ops + c.sldc_ca; + if (c.sldc_compr == NULL) { + printf("Error: The selected compression algorithm is not\n" + " supported\n"); + abort(); + } + c.sldc_compr->init(&c.sldc_cc); + sldc_erase_bufs(&c.sldc_cc); + c.sldc_cc.rlen = c.sldc_cc.cluster_size * F2FS_BLKSIZE; +} + static int do_sload(struct f2fs_sb_info *sbi) { if (!c.from_dir) { @@ -821,6 +951,9 @@ static int do_sload(struct f2fs_sb_info *sbi) if (!c.mount_point) c.mount_point = "/"; + if (c.sldc_en) + init_compr(sbi); + return f2fs_sload(sbi); } #endif @@ -971,6 +1104,9 @@ retry: return ret2; } + if (c.func == SLOAD) + c.sldc_ef->destroy(); + printf("\nDone: %lf secs\n", (get_boottime_ns() - start) / 1000000000.0); return ret; diff --git a/fsck/segment.c b/fsck/segment.c index 0487f41..e4c8cea 100644 --- a/fsck/segment.c +++ b/fsck/segment.c @@ -8,6 +8,9 @@ * Hou Pengyang <houpengyang@huawei.com> * Liu Shuoran <liushuoran@huawei.com> * Jaegeuk Kim <jaegeuk@kernel.org> + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -16,6 +19,7 @@ #include "fsck.h" #include "node.h" #include "quotaio.h" +#include "compress_wrapper.h" int reserve_new_block(struct f2fs_sb_info *sbi, block_t *to, struct f2fs_summary *sum, int type, bool is_inode) @@ -228,8 +232,14 @@ u64 f2fs_read(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, return read_count; } -u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, - u64 count, pgoff_t offset) +/* + * Do not call this function directly. Instead, call one of the following: + * u64 f2fs_write(); + * u64 f2fs_write_compress_data(); + * u64 f2fs_write_addrtag(); + */ +static u64 f2fs_write_ex(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, + u64 count, pgoff_t offset, enum wr_addr_type addr_type) { struct dnode_of_data dn; struct node_info ni; @@ -243,6 +253,19 @@ u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, void* index_node = NULL; int idirty = 0; int err; + bool has_data = (addr_type == WR_NORMAL + || addr_type == WR_COMPRESS_DATA); + + if (count == 0) + return 0; + + /* + * Enforce calling from f2fs_write(), f2fs_write_compress_data(), + * and f2fs_write_addrtag(). Beside, check if is properly called. + */ + ASSERT((!has_data && buffer == NULL) || (has_data && buffer != NULL)); + if (addr_type != WR_NORMAL) + ASSERT(offset % F2FS_BLKSIZE == 0); /* block boundary only */ /* Memory allocation for block buffer and inode. */ blk_buffer = calloc(BLOCK_SZ, 2); @@ -265,15 +288,26 @@ u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, if (err) break; idirty |= dn.idirty; - if (index_node) - free(index_node); + free(index_node); index_node = (dn.node_blk == dn.inode_blk) ? - NULL : dn.node_blk; + NULL : dn.node_blk; remained_blkentries = ADDRS_PER_PAGE(sbi, - dn.node_blk, dn.inode_blk); + dn.node_blk, dn.inode_blk) - + dn.ofs_in_node; } ASSERT(remained_blkentries > 0); + if (!has_data) { + dn.data_blkaddr = addr_type; + set_data_blkaddr(&dn); + idirty |= dn.idirty; + if (dn.ndirty) + ASSERT(dev_write_block(dn.node_blk, + dn.node_blkaddr) >= 0); + written_count = 0; + break; + } + blkaddr = datablock_addr(dn.node_blk, dn.ofs_in_node); if (blkaddr == NULL_ADDR || blkaddr == NEW_ADDR) { err = new_data_block(sbi, blk_buffer, @@ -281,6 +315,7 @@ u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, if (err) break; blkaddr = dn.data_blkaddr; + idirty |= dn.idirty; } off_in_blk = offset % BLOCK_SZ; @@ -305,9 +340,10 @@ u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, dn.ofs_in_node++; if ((--remained_blkentries == 0 || count == 0) && (dn.ndirty)) - ASSERT(dev_write_block(dn.node_blk, dn.node_blkaddr) >= 0); + ASSERT(dev_write_block(dn.node_blk, dn.node_blkaddr) + >= 0); } - if (offset > le64_to_cpu(inode->i.i_size)) { + if (addr_type == WR_NORMAL && offset > le64_to_cpu(inode->i.i_size)) { inode->i.i_size = cpu_to_le64(offset); idirty = 1; } @@ -315,13 +351,33 @@ u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, ASSERT(inode == dn.inode_blk); ASSERT(write_inode(inode, ni.blk_addr) >= 0); } - if (index_node) - free(index_node); + + free(index_node); free(blk_buffer); return written_count; } +u64 f2fs_write(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, + u64 count, pgoff_t offset) +{ + return f2fs_write_ex(sbi, ino, buffer, count, offset, WR_NORMAL); +} + +u64 f2fs_write_compress_data(struct f2fs_sb_info *sbi, nid_t ino, u8 *buffer, + u64 count, pgoff_t offset) +{ + return f2fs_write_ex(sbi, ino, buffer, count, offset, WR_COMPRESS_DATA); +} + +u64 f2fs_write_addrtag(struct f2fs_sb_info *sbi, nid_t ino, pgoff_t offset, + unsigned int addrtag) +{ + ASSERT(addrtag == COMPRESS_ADDR || addrtag == NEW_ADDR + || addrtag == NULL_ADDR); + return f2fs_write_ex(sbi, ino, NULL, F2FS_BLKSIZE, offset, addrtag); +} + /* This function updates only inode->i.i_size */ void f2fs_filesize_update(struct f2fs_sb_info *sbi, nid_t ino, u64 filesize) { @@ -342,11 +398,59 @@ void f2fs_filesize_update(struct f2fs_sb_info *sbi, nid_t ino, u64 filesize) free(inode); } +#define MAX_BULKR_RETRY 5 +int bulkread(int fd, void *rbuf, size_t rsize, bool *eof) +{ + int n = 0; + int retry = MAX_BULKR_RETRY; + int cur; + + if (!rsize) + return 0; + + if (eof != NULL) + *eof = false; + while (rsize && (cur = read(fd, rbuf, rsize)) != 0) { + if (cur == -1) { + if (errno == EINTR && retry--) + continue; + return -1; + } + retry = MAX_BULKR_RETRY; + + rsize -= cur; + n += cur; + } + if (eof != NULL) + *eof = (cur == 0); + return n; +} + +u64 f2fs_fix_mutable(struct f2fs_sb_info *sbi, nid_t ino, pgoff_t offset, + unsigned int compressed) +{ + unsigned int i; + u64 wlen; + + if (c.sldc_immutable) + return 0; + + for (i = 0; i < compressed - 1; i++) { + wlen = f2fs_write_addrtag(sbi, ino, + offset + (i << F2FS_BLKSIZE_BITS), NEW_ADDR); + if (wlen) + return wlen; + } + return 0; +} + int f2fs_build_file(struct f2fs_sb_info *sbi, struct dentry *de) { int fd, n; pgoff_t off = 0; u8 buffer[BLOCK_SZ]; + struct node_info ni; + struct f2fs_node *node_blk; if (de->ino == 0) return -1; @@ -359,8 +463,6 @@ int f2fs_build_file(struct f2fs_sb_info *sbi, struct dentry *de) /* inline_data support */ if (de->size <= DEF_MAX_INLINE_DATA) { - struct node_info ni; - struct f2fs_node *node_blk; int ret; get_node_info(sbi, de->ino, &ni); @@ -385,6 +487,82 @@ int f2fs_build_file(struct f2fs_sb_info *sbi, struct dentry *de) node_blk->i.i_size = cpu_to_le64(de->size); ASSERT(write_inode(node_blk, ni.blk_addr) >= 0); free(node_blk); +#ifdef WITH_SLOAD + } else if (c.func == SLOAD && c.sldc_en && + c.sldc_ef->filter(de->full_path)) { + bool eof = false; + u8 *rbuf = c.sldc_cc.rbuf; + unsigned int cblocks = 0; + + node_blk = calloc(BLOCK_SZ, 1); + ASSERT(node_blk); + + /* read inode */ + get_node_info(sbi, de->ino, &ni); + ASSERT(dev_read_block(node_blk, ni.blk_addr) >= 0); + /* update inode meta */ + node_blk->i.i_compress_algrithm = c.sldc_ca; + node_blk->i.i_log_cluster_size = + c.sldc_cc.log_cluster_size; + node_blk->i.i_flags = cpu_to_le32( + F2FS_COMPR_FL | + (c.sldc_immutable ? FS_IMMUTABLE_FL : 0)); + ASSERT(write_inode(node_blk, ni.blk_addr) >= 0); + + while (!eof && (n = bulkread(fd, rbuf, c.sldc_cc.rlen, + &eof)) > 0) { + int ret = c.sldc_compr->compress(&c.sldc_cc); + u64 wlen; + u32 csize = ALIGN_UP(c.sldc_cc.clen + + COMPRESS_HEADER_SIZE, BLOCK_SZ); + unsigned int cur_cblk; + + if (ret || n < (int)(csize + BLOCK_SZ * + c.sldc_min_cbpc)) { + wlen = f2fs_write(sbi, de->ino, rbuf, n, off); + ASSERT((int)wlen == n); + } else { + wlen = f2fs_write_addrtag(sbi, de->ino, off, + WR_COMPRESS_ADDR); + ASSERT(!wlen); + wlen = f2fs_write_compress_data(sbi, de->ino, + (u8 *)c.sldc_cc.cbuf, + csize, off + BLOCK_SZ); + ASSERT(wlen == csize); + sldc_erase_bufs(&c.sldc_cc); + cur_cblk = (c.sldc_cc.rlen - csize) / BLOCK_SZ; + cblocks += cur_cblk; + wlen = f2fs_fix_mutable(sbi, de->ino, + off + BLOCK_SZ + csize, + cur_cblk); + ASSERT(!wlen); + } + off += n; + } + if (n == -1) { + fprintf(stderr, "Load file '%s' failed: ", + de->full_path); + perror(NULL); + } + /* read inode */ + get_node_info(sbi, de->ino, &ni); + ASSERT(dev_read_block(node_blk, ni.blk_addr) >= 0); + /* update inode meta */ + node_blk->i.i_size = cpu_to_le64(off); + if (!c.sldc_immutable) + node_blk->i.i_compr_blocks = cpu_to_le64(cblocks); + ASSERT(write_inode(node_blk, ni.blk_addr) >= 0); + free(node_blk); + + if (!c.sldc_immutable) { + sbi->total_valid_block_count += cblocks; + if (sbi->total_valid_block_count >= + sbi->user_block_count) { + ERR_MSG("Not enough space\n"); + ASSERT(0); + } + } +#endif } else { while ((n = read(fd, buffer, BLOCK_SZ)) > 0) { f2fs_write(sbi, de->ino, buffer, n, off); diff --git a/fsck/sload.c b/fsck/sload.c index 14012fb..13e523a 100644 --- a/fsck/sload.c +++ b/fsck/sload.c @@ -6,6 +6,9 @@ * Hou Pengyang <houpengyang@huawei.com> * Liu Shuoran <liushuoran@huawei.com> * Jaegeuk Kim <jaegeuk@kernel.org> + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -317,6 +320,70 @@ skip: return 0; } +typedef struct _ext_tbl { + const char *ext; + struct _ext_tbl *next; /* linked list */ +} ext_tbl_t; +static ext_tbl_t *ext_tbl; + +static bool ext_found(const char *ext) +{ + ext_tbl_t *p = ext_tbl; + + while (p != NULL && strcmp(ext, p->ext)) + p = p->next; + return (p != NULL); +} + +static const char *get_ext(const char *path) +{ + char *p = strrchr(path, '.'); + return p == NULL ? path + strlen(path) : p + 1; +} + +static bool ext_do_filter(const char *path) +{ + return (ext_found(get_ext(path)) == true) ^ (c.sldc_policy == FP_ALLOW); +} + +static void ext_filter_add(const char *ext) +{ + ext_tbl_t *node; + + ASSERT(ext != NULL); + if (ext_found(ext)) + return; /* ext was already registered */ + node = malloc(sizeof(ext_tbl_t)); + ASSERT(node != NULL); + node->ext = ext; + node->next = ext_tbl; + ext_tbl = node; +} + +static void ext_filter_destroy(void) +{ + ext_tbl_t *p; + + while (ext_tbl != NULL) { + p = ext_tbl; + ext_tbl = p->next; + free(p); + } +} + +struct ext_tbl_op ext_filter = { + .add = ext_filter_add, + .destroy = ext_filter_destroy, + .filter = ext_do_filter, +}; + +void sldc_erase_bufs(struct compress_ctx *cc) +{ + memset(cc->rbuf, 0, cc->cluster_size * F2FS_BLKSIZE); + memset(cc->cbuf->cdata, 0, cc->cluster_size * F2FS_BLKSIZE + - F2FS_BLKSIZE); +} + int f2fs_sload(struct f2fs_sb_info *sbi) { int ret = 0; diff --git a/include/f2fs_fs.h b/include/f2fs_fs.h index 1348e39..2a2dc15 100644 --- a/include/f2fs_fs.h +++ b/include/f2fs_fs.h @@ -5,6 +5,9 @@ * http://www.samsung.com/ * Copyright (c) 2019 Google Inc. * http://www.google.com/ + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add sload compression support * * Dual licensed under the GPL or LGPL version 2 licenses. * @@ -68,6 +71,10 @@ typedef uint16_t u_int16_t; typedef uint8_t u_int8_t; #endif +/* codes from kernel's f2fs.h, GPL-v2.0 */ +#define MIN_COMPRESS_LOG_SIZE 2 +#define MAX_COMPRESS_LOG_SIZE 8 + typedef u_int64_t u64; typedef u_int32_t u32; typedef u_int16_t u16; @@ -93,6 +100,31 @@ typedef u32 __be32; typedef u64 __be64; #endif +/* + * code borrowed from kernel f2fs dirver: f2fs.h, GPL-2.0 + * : definitions of COMPRESS_DATA_RESERVED_SIZE, + * struct compress_data, COMPRESS_HEADER_SIZE, + * and struct compress_ctx + */ +#define COMPRESS_DATA_RESERVED_SIZE 4 +struct compress_data { + __le32 clen; /* compressed data size */ + __le32 chksum; /* checksum of compressed data */ + __le32 reserved[COMPRESS_DATA_RESERVED_SIZE]; /* reserved */ + u8 cdata[]; /* compressed data */ +}; +#define COMPRESS_HEADER_SIZE (sizeof(struct compress_data)) +/* compress context */ +struct compress_ctx { + unsigned int cluster_size; /* page count in cluster */ + unsigned int log_cluster_size; /* log of cluster size */ + void *rbuf; /* compression input buffer */ + struct compress_data *cbuf; /* comprsssion output header + data */ + size_t rlen; /* valid data length in rbuf */ + size_t clen; /* valid data length in cbuf */ + void *private; /* work buf for compress algorithm */ +}; + #if HAVE_BYTESWAP_H #include <byteswap.h> #else @@ -345,6 +377,25 @@ typedef struct { bool dbg_en; } dev_cache_config_t; +/* f2fs_configration: sldc_ca, the sload compress algorithm */ +enum {CA_LZO, CA_LZ4}; +extern const char *ca_names[]; + +typedef struct { + void (*init)(struct compress_ctx *cc); + int (*compress)(struct compress_ctx *cc); +} compress_ops; + +#define ALIGN_UP(value, size) ((value) + ((value) % (size) > 0 ? \ + (size) - (value) % (size) : 0)) + +enum filter_policy {FP_UNASSIGNED = 0, FP_ALLOW, FP_DENY}; +struct ext_tbl_op { + void (*add)(const char *); + void (*destroy)(void); + bool (*filter)(const char *); +}; + struct f2fs_configuration { u_int32_t reserved_segments; u_int32_t new_reserved_segments; @@ -441,6 +492,24 @@ struct f2fs_configuration { /* cache parameters */ dev_cache_config_t cache_config; + + /* quick dynamic buffer */ + bool qbuf_initialized; + size_t qbufsize; + void *qbuf; + + /* sldc: sload compression support */ + bool sldc_en; + bool sldc_use_allow_list; /* default false to use the deny list */ + struct compress_ctx sldc_cc; + u8 sldc_ca; /* compress algorithm: 0 = LZO, 1 = LZ4 */ + compress_ops *sldc_compr; + enum filter_policy sldc_policy; + /* max_cppc can used to specify minimum compression rate */ + unsigned int sldc_min_cbpc; /* min compressed pages per cluster */ + bool sldc_got_opt; + bool sldc_immutable; + struct ext_tbl_op *sldc_ef; /* extension filter */ }; #ifdef CONFIG_64BIT @@ -1226,6 +1295,11 @@ extern void f2fs_release_sparse_resource(void); extern int f2fs_finalize_device(void); extern int f2fs_fsync_device(void); +/* quick (shared) buffer */ +extern void qbuf_free(void); +extern void *qbuf_alloc(size_t size); +extern void qbuf_init(void); + extern void dcache_init(void); extern void dcache_release(void); @@ -1377,7 +1451,7 @@ int f2fs_reset_zone(int, void *); extern int f2fs_reset_zones(int); extern uint32_t f2fs_get_usable_segments(struct f2fs_super_block *sb); -#define SIZE_ALIGN(val, size) ((val) + (size) - 1) / (size) +#define SIZE_ALIGN(val, size) (((val) + (size) - 1) / (size)) #define SEG_ALIGN(blks) SIZE_ALIGN(blks, c.blks_per_seg) #define ZONE_ALIGN(blks) SIZE_ALIGN(blks, c.blks_per_seg * \ c.segs_per_zone) diff --git a/lib/libf2fs_io.c b/lib/libf2fs_io.c index 138285d..0280896 100644 --- a/lib/libf2fs_io.c +++ b/lib/libf2fs_io.c @@ -5,6 +5,9 @@ * http://www.samsung.com/ * Copyright (c) 2019 Google Inc. * http://www.google.com/ + * Copyright (c) 2020 Google Inc. + * Robin Hsu <robinhsu@google.com> + * : add quick-buffer for sload compression support * * Dual licensed under the GPL or LGPL version 2 licenses. */ @@ -106,6 +109,36 @@ static long dcache_relocate_offset0[] = { }; static int dcache_relocate_offset[16]; +/* quick (shared) buffer */ +static bool qbuf_initialized; +static void *qbuf; +static size_t qbufsize; +void qbuf_free(void) +{ + ASSERT(qbuf_initialized); + if (qbuf != NULL) { + free(qbuf); + qbuf = NULL; + qbufsize = 0; + } +} +void *qbuf_alloc(size_t size) +{ + ASSERT(qbuf_initialized); + if (size > qbufsize) { + qbuf_free(); + qbuf = malloc(size); + } + return qbuf; +} +void qbuf_init(void) +{ + if (qbuf_initialized) + return; + atexit(qbuf_free); + qbuf_initialized = true; +} + static void dcache_print_statistics(void) { long i; -- 2.29.2.454.gaff20da3a2-goog _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 3/3] f2fs-tools:sload.f2fs compress: Fixed automake 2020-12-02 0:57 ` [f2fs-dev] " Robin Hsu @ 2020-12-02 0:57 ` Robin Hsu -1 siblings, 0 replies; 8+ messages in thread From: Robin Hsu @ 2020-12-02 0:57 UTC (permalink / raw) To: linux-f2fs-devel, jaegeuk, chao; +Cc: linux-kernel, Robin Hsu From: Robin Hsu <robinhsu@google.com> Fixed automake for sload.f2fs compression support ./configure now will by default depends on liblzo2 and liblz4. To compile without liblzo2 (and thus not support liblzo2 compression), run ./configure with '--without-lzo2' option. To compile without liblz4 (and thus not support liblz4 compression), run ./configure with '--without-lz4' option. Signed-off-by: Robin Hsu <robinhsu@google.com> --- configure.ac | 32 ++++++++++++++++++++++++++++++++ fsck/Makefile.am | 9 ++++++--- 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/configure.ac b/configure.ac index 1e5619d..3a357b2 100644 --- a/configure.ac +++ b/configure.ac @@ -44,6 +44,14 @@ AC_ARG_WITH([blkid], AS_HELP_STRING([--without-blkid], [Ignore presence of libblkid and disable blkid support])) +AC_ARG_WITH([lzo2], + AS_HELP_STRING([--without-lzo2], + [Ignore liblzo ver.2 for sload (not supporting lzo compression)])) + +AC_ARG_WITH([lz4], + AS_HELP_STRING([--without-lz4], + [Ignore liblz4 for sload (not supporting lz4 compression)])) + # Checks for programs. AC_PROG_CC AC_PROG_LIBTOOL @@ -52,6 +60,30 @@ AC_PATH_PROG([LDCONFIG], [ldconfig], [$PATH:/sbin]) # Checks for libraries. +AS_IF([test "x$with_lzo2" != xno], + [AC_CHECK_LIB([lzo2], [main], + [AC_SUBST([liblzo2_LIBS], ["-llzo2"]) + AC_DEFINE([HAVE_LIBLZO2], [1], + [Define if you have liblzo2]) + ], + [if test "x$with_lzo2" != xcheck; then + AC_MSG_FAILURE( + [--with-lzo2 was given (or by default), but test for liblzo2 failed]) + fi + ], [])]) + +AS_IF([test "x$with_lz4" != xno], + [AC_CHECK_LIB([lz4], [main], + [AC_SUBST([liblz4_LIBS], ["-llz4"]) + AC_DEFINE([HAVE_LIBLZ4], [1], + [Define if you have liblz4]) + ], + [if test "x$with_lz4" != xcheck; then + AC_MSG_FAILURE( + [--with-lz4 was given (or by default), but test for liblz4 failed]) + fi + ], [])]) + PKG_CHECK_MODULES([libuuid], [uuid]) AS_IF([test "x$with_selinux" != "xno"], diff --git a/fsck/Makefile.am b/fsck/Makefile.am index 1fc7310..74bc4b8 100644 --- a/fsck/Makefile.am +++ b/fsck/Makefile.am @@ -3,12 +3,15 @@ AM_CPPFLAGS = ${libuuid_CFLAGS} -I$(top_srcdir)/include AM_CFLAGS = -Wall sbin_PROGRAMS = fsck.f2fs -noinst_HEADERS = common.h dict.h dqblk_v2.h f2fs.h fsck.h node.h quotaio.h quotaio_tree.h quotaio_v2.h xattr.h +noinst_HEADERS = common.h dict.h dqblk_v2.h f2fs.h fsck.h node.h quotaio.h \ + quotaio_tree.h quotaio_v2.h xattr.h compress_wrapper.h include_HEADERS = $(top_srcdir)/include/quota.h fsck_f2fs_SOURCES = main.c fsck.c dump.c mount.c defrag.c resize.c \ - node.c segment.c dir.c sload.c xattr.c \ + node.c segment.c dir.c sload.c xattr.c compress_wrapper.c \ dict.c mkquota.c quotaio.c quotaio_tree.c quotaio_v2.c -fsck_f2fs_LDADD = ${libselinux_LIBS} ${libuuid_LIBS} $(top_builddir)/lib/libf2fs.la +fsck_f2fs_LDADD = ${libselinux_LIBS} ${libuuid_LIBS} \ + ${liblzo2_LIBS} ${liblz4_LIBS} \ + $(top_builddir)/lib/libf2fs.la install-data-hook: ln -sf fsck.f2fs $(DESTDIR)/$(sbindir)/dump.f2fs -- 2.29.2.454.gaff20da3a2-goog ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [f2fs-dev] [PATCH 3/3] f2fs-tools:sload.f2fs compress: Fixed automake @ 2020-12-02 0:57 ` Robin Hsu 0 siblings, 0 replies; 8+ messages in thread From: Robin Hsu @ 2020-12-02 0:57 UTC (permalink / raw) To: linux-f2fs-devel, jaegeuk, chao; +Cc: linux-kernel From: Robin Hsu <robinhsu@google.com> Fixed automake for sload.f2fs compression support ./configure now will by default depends on liblzo2 and liblz4. To compile without liblzo2 (and thus not support liblzo2 compression), run ./configure with '--without-lzo2' option. To compile without liblz4 (and thus not support liblz4 compression), run ./configure with '--without-lz4' option. Signed-off-by: Robin Hsu <robinhsu@google.com> --- configure.ac | 32 ++++++++++++++++++++++++++++++++ fsck/Makefile.am | 9 ++++++--- 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/configure.ac b/configure.ac index 1e5619d..3a357b2 100644 --- a/configure.ac +++ b/configure.ac @@ -44,6 +44,14 @@ AC_ARG_WITH([blkid], AS_HELP_STRING([--without-blkid], [Ignore presence of libblkid and disable blkid support])) +AC_ARG_WITH([lzo2], + AS_HELP_STRING([--without-lzo2], + [Ignore liblzo ver.2 for sload (not supporting lzo compression)])) + +AC_ARG_WITH([lz4], + AS_HELP_STRING([--without-lz4], + [Ignore liblz4 for sload (not supporting lz4 compression)])) + # Checks for programs. AC_PROG_CC AC_PROG_LIBTOOL @@ -52,6 +60,30 @@ AC_PATH_PROG([LDCONFIG], [ldconfig], [$PATH:/sbin]) # Checks for libraries. +AS_IF([test "x$with_lzo2" != xno], + [AC_CHECK_LIB([lzo2], [main], + [AC_SUBST([liblzo2_LIBS], ["-llzo2"]) + AC_DEFINE([HAVE_LIBLZO2], [1], + [Define if you have liblzo2]) + ], + [if test "x$with_lzo2" != xcheck; then + AC_MSG_FAILURE( + [--with-lzo2 was given (or by default), but test for liblzo2 failed]) + fi + ], [])]) + +AS_IF([test "x$with_lz4" != xno], + [AC_CHECK_LIB([lz4], [main], + [AC_SUBST([liblz4_LIBS], ["-llz4"]) + AC_DEFINE([HAVE_LIBLZ4], [1], + [Define if you have liblz4]) + ], + [if test "x$with_lz4" != xcheck; then + AC_MSG_FAILURE( + [--with-lz4 was given (or by default), but test for liblz4 failed]) + fi + ], [])]) + PKG_CHECK_MODULES([libuuid], [uuid]) AS_IF([test "x$with_selinux" != "xno"], diff --git a/fsck/Makefile.am b/fsck/Makefile.am index 1fc7310..74bc4b8 100644 --- a/fsck/Makefile.am +++ b/fsck/Makefile.am @@ -3,12 +3,15 @@ AM_CPPFLAGS = ${libuuid_CFLAGS} -I$(top_srcdir)/include AM_CFLAGS = -Wall sbin_PROGRAMS = fsck.f2fs -noinst_HEADERS = common.h dict.h dqblk_v2.h f2fs.h fsck.h node.h quotaio.h quotaio_tree.h quotaio_v2.h xattr.h +noinst_HEADERS = common.h dict.h dqblk_v2.h f2fs.h fsck.h node.h quotaio.h \ + quotaio_tree.h quotaio_v2.h xattr.h compress_wrapper.h include_HEADERS = $(top_srcdir)/include/quota.h fsck_f2fs_SOURCES = main.c fsck.c dump.c mount.c defrag.c resize.c \ - node.c segment.c dir.c sload.c xattr.c \ + node.c segment.c dir.c sload.c xattr.c compress_wrapper.c \ dict.c mkquota.c quotaio.c quotaio_tree.c quotaio_v2.c -fsck_f2fs_LDADD = ${libselinux_LIBS} ${libuuid_LIBS} $(top_builddir)/lib/libf2fs.la +fsck_f2fs_LDADD = ${libselinux_LIBS} ${libuuid_LIBS} \ + ${liblzo2_LIBS} ${liblz4_LIBS} \ + $(top_builddir)/lib/libf2fs.la install-data-hook: ln -sf fsck.f2fs $(DESTDIR)/$(sbindir)/dump.f2fs -- 2.29.2.454.gaff20da3a2-goog _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-12-02 0:59 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-12-02 0:57 [PATCH 0/3] f2fs-tools: sload.f2fs to support compression Robin Hsu 2020-12-02 0:57 ` [f2fs-dev] " Robin Hsu 2020-12-02 0:57 ` [PATCH 1/3] f2fs-tools: Added #ifdef WITH_func Robin Hsu 2020-12-02 0:57 ` [f2fs-dev] " Robin Hsu 2020-12-02 0:57 ` [PATCH 2/3] f2fs-tools:sload.f2fs compression support Robin Hsu 2020-12-02 0:57 ` [f2fs-dev] " Robin Hsu 2020-12-02 0:57 ` [PATCH 3/3] f2fs-tools:sload.f2fs compress: Fixed automake Robin Hsu 2020-12-02 0:57 ` [f2fs-dev] " Robin Hsu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.