linux-erofs.lists.ozlabs.org archive mirror
 help / color / mirror / Atom feed
From: Gao Xiang via Linux-erofs <linux-erofs@lists.ozlabs.org>
To: linux-erofs@lists.ozlabs.org
Subject: [WIP] [PATCH v0.0-20200229 11/11] ez: lzma: add test program
Date: Sat, 29 Feb 2020 12:50:17 +0800	[thread overview]
Message-ID: <20200229045017.12424-12-hsiangkao@aol.com> (raw)
In-Reply-To: <20200229045017.12424-1-hsiangkao@aol.com>

Usage:
$ ./run.sh
$ ./a.out output.bin.lzma infile

It will compress the beginning as much
as possible into 4k RAW LZMA block.

Signed-off-by: Gao Xiang <hsiangkao@aol.com>
---
 lzma/lzma_encoder.c | 115 ++++++++++++++++++++++++++++++++++++++++++++
 lzma/run.sh         |   1 +
 2 files changed, 116 insertions(+)
 create mode 100755 lzma/run.sh

diff --git a/lzma/lzma_encoder.c b/lzma/lzma_encoder.c
index 98cde22..7c4b51d 100644
--- a/lzma/lzma_encoder.c
+++ b/lzma/lzma_encoder.c
@@ -757,3 +757,118 @@ void lzma_default_properties(struct lzma_properties *p, int level)
 	p->mf.depth = (16 + (p->mf.nice_len >> 1)) >> 1;
 }
 
+#include <stdlib.h>
+#include <stdio.h>
+#include <fcntl.h>
+#include <unistd.h>
+
+#if 0
+const char text[] = "HABEABDABABABHHHEAAAAAAAA";
+#elif 0
+const char text[] = "abcde_bcdefgh_abcdefghxxxxxxx";
+#else
+const char text[] = "The only time we actually leave the path spinning is if we're truncating "
+"a small amount and don't actually free an extent, which is not a common "
+"occurrence.  We have to set the path blocking in order to add the "
+"delayed ref anyway, so the first extent we find we set the path to "
+"blocking and stay blocking for the duration of the operation.  With the "
+"upcoming file extent map stuff there will be another case that we have "
+"to have the path blocking, so just swap to blocking always.";
+#endif
+
+static const uint8_t lzma_header[] = {
+	0x5D,				/* LZMA model properties (lc, lp, pb) in encoded form */
+	0x00, 0x00, 0x80, 0x00,		/* Dictionary size (32-bit unsigned integer, little-endian) */
+	0xFF, 0xFF, 0xFF, 0xFF,
+	0xFF, 0xFF, 0xFF, 0xFF,		/* Uncompressed size (64-bit unsigned integer, little-endian) */
+};
+
+int main(int argc, char *argv[])
+{
+	char *outfile;
+	struct lzma_encoder lzmaenc = {0};
+	struct lzma_properties props = {
+		.mf.dictsize = 65536,
+	};
+	struct lzma_encoder_destsize dstsize;
+	uint8_t buf[4096];
+	int inf, outf;
+
+	int err;
+
+	lzmaenc.mf.buffer = malloc(65536) + 1;
+	lzmaenc.mf.buffer[-1] = 0;
+
+	if (argc >= 3) {
+		int len;
+
+		inf = open(argv[2], O_RDONLY);
+
+		len = lseek(inf, 0, SEEK_END);
+		if (len >= 65535)
+			len = 65535;
+
+		printf("len: %d\n", len);
+
+		lseek(inf, 0, SEEK_SET);
+		read(inf, lzmaenc.mf.buffer, len);
+		close(inf);
+		lzmaenc.mf.iend = lzmaenc.mf.buffer + len;
+	} else {
+		memcpy(lzmaenc.mf.buffer, text, sizeof(text));
+		lzmaenc.mf.iend = lzmaenc.mf.buffer + sizeof(text);
+	}
+	lzmaenc.op = buf;
+	lzmaenc.oend = buf + sizeof(buf);
+	lzmaenc.finish = true;
+
+	lzmaenc.need_eopm = true;
+	dstsize.capacity = 4096 - sizeof(lzma_header); //UINT32_MAX;
+	lzmaenc.dstsize = &dstsize;
+
+
+	lzma_default_properties(&props, 5);
+	lzma_encoder_reset(&lzmaenc, &props);
+
+	err = __lzma_encode(&lzmaenc);
+
+	printf("%d\n", err);
+
+	rc_encode(&lzmaenc.rc, &lzmaenc.op, lzmaenc.oend);
+
+	if (err != -ERANGE) {
+		memcpy(lzmaenc.op, dstsize.ending, dstsize.esz);
+		lzmaenc.op += dstsize.esz;
+	} else {
+		encode_eopm(&lzmaenc);
+		rc_flush(&lzmaenc.rc);
+
+		rc_encode(&lzmaenc.rc, &lzmaenc.op, lzmaenc.oend);
+	}
+
+	printf("encoded length: %lu + %lu\n", lzmaenc.op - buf,
+	       sizeof(lzma_header));
+
+	if (argc < 2)
+		outfile = "output.bin.lzma";
+	else
+		outfile = argv[1];
+
+	outf = open(outfile, O_WRONLY | O_CREAT | O_TRUNC, 0644);
+	write(outf, lzma_header, sizeof(lzma_header));
+	write(outf, buf, lzmaenc.op - buf);
+	close(outf);
+
+#if 0
+	nliterals = lzma_get_optimum_fast(&lzmaenc, &back_res, &len_res);
+	printf("nlits %d (%d %d)\n", nliterals, back_res, len_res);
+	encode_sequence(&lzmaenc, nliterals, back_res, len_res, &position);
+	nliterals = lzma_get_optimum_fast(&lzmaenc, &back_res, &len_res);
+	printf("nlits %d (%d %d)\n", nliterals, back_res, len_res);
+	nliterals = lzma_get_optimum_fast(&lzmaenc, &back_res, &len_res);
+	printf("nlits %d (%d %d)\n", nliterals, back_res, len_res);
+	nliterals = lzma_get_optimum_fast(&lzmaenc, &back_res, &len_res);
+	printf("nlits %d (%d %d)\n", nliterals, back_res, len_res);
+#endif
+}
+
diff --git a/lzma/run.sh b/lzma/run.sh
new file mode 100755
index 0000000..57adc12
--- /dev/null
+++ b/lzma/run.sh
@@ -0,0 +1 @@
+gcc -Wall -g -I ../include lzma_encoder.c mf.c
-- 
2.20.1


  parent reply	other threads:[~2020-02-29  4:52 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20200229045017.12424-1-hsiangkao.ref@aol.com>
2020-02-29  4:50 ` [WIP] [PATCH v0.0-20200229 00/11] ez: LZMA fixed-sized output compression Gao Xiang via Linux-erofs
2020-02-29  4:50   ` [WIP] [PATCH v0.0-20200229 01/11] ez: add basic source files Gao Xiang via Linux-erofs
2020-02-29  4:50   ` [WIP] [PATCH v0.0-20200229 02/11] ez: add helpers for unaligned accesses Gao Xiang via Linux-erofs
2020-02-29  4:50   ` [WIP] [PATCH v0.0-20200229 03/11] ez: introduce bitops header file Gao Xiang via Linux-erofs
2020-02-29  4:50   ` [WIP] [PATCH v0.0-20200229 04/11] ez: lzma: add range encoder Gao Xiang via Linux-erofs
2020-02-29  4:50   ` [WIP] [PATCH v0.0-20200229 05/11] ez: lzma: add common header file Gao Xiang via Linux-erofs
2020-02-29  4:50   ` [WIP] [PATCH v0.0-20200229 06/11] ez: lzma: add byte hashtable generated with CRC32 Gao Xiang via Linux-erofs
2020-02-29  4:50   ` [WIP] [PATCH v0.0-20200229 07/11] ez: lzma: add LZMA matchfinder Gao Xiang via Linux-erofs
2020-02-29  4:50   ` [WIP] [PATCH v0.0-20200229 08/11] ez: lzma: add LZMA encoder Gao Xiang via Linux-erofs
2020-02-29  4:50   ` [WIP] [PATCH v0.0-20200229 09/11] ez: lzma: checkpoint feature for range encoder Gao Xiang via Linux-erofs
2020-02-29  4:50   ` [WIP] [PATCH v0.0-20200229 10/11] ez: lzma: add fixed-sized output compression Gao Xiang via Linux-erofs
2020-02-29  4:50   ` Gao Xiang via Linux-erofs [this message]
2020-02-29  4:58   ` [WIP] [PATCH v0.0-20200229 00/11] ez: LZMA " Eric Biggers
2020-02-29  5:27     ` Gao Xiang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200229045017.12424-12-hsiangkao@aol.com \
    --to=linux-erofs@lists.ozlabs.org \
    --cc=hsiangkao@aol.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).