From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26755C48BE8 for ; Mon, 14 Jun 2021 07:52:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1230261357 for ; Mon, 14 Jun 2021 07:52:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232622AbhFNHyL (ORCPT ); Mon, 14 Jun 2021 03:54:11 -0400 Received: from mail-ej1-f42.google.com ([209.85.218.42]:40676 "EHLO mail-ej1-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232623AbhFNHyK (ORCPT ); Mon, 14 Jun 2021 03:54:10 -0400 Received: by mail-ej1-f42.google.com with SMTP id my49so15063856ejc.7 for ; Mon, 14 Jun 2021 00:51:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fK0+f39KReYwmIZ4HszCv/9BYALVUEKPLQVsLsNwq94=; b=OOv7N4OfuJhyAxhW0lNISF/K2fJdgp5VZ6aoNGj2JwlSeTgrklZ37sf1qic5VUvXyT i3eb2tlGlu999P/whSSiuRCw+ABPJjRKfxJyTlgyufTqau/Z/DggkIshUs5Mf4V55a0F AFOgmO77o2Y5MZmOtZ3tiIbh2pXol/YdKF5wGYr1NKPHAQa2LLS0M7LfONY0JCbF3KQC Tdtwrhy2FXQ5c/nFbXyur2CsaH94fjbUikqaUFtkG+Oopsz221ELQxsJ2ZChcSTTfy4f U/0ogWgz1C04vdF+N9d6jPln5k6JA5KLthHwpWDgnQBRIEMAI4Ygh+tsXoCTgmKvHAZN Z3Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fK0+f39KReYwmIZ4HszCv/9BYALVUEKPLQVsLsNwq94=; b=a3eqjm/HaOlO4Xima81X1ZlCQCgfjyBJNxSV3Gjk5G6J0lAhDSD14q5TXLenWp3w+r oMrqC+GPNwlwj9N2iczbvrAgulB/fQj+phQ1QOlIkAzr0mUwYmZTRZt/Ir2c/h1CGY9N Ya9/FRa/2+x0xhzdPfP3zTAgra/lJqiLJRy1Si/Bd1tat8pFosp4zyBQvfeKcu25iS2k QJDiUTmxLnop+oeeoPdpMentnwQypO4ZMwlJ9dx/cfkIbgsK+C9PtMMaZxBHij/cP52z Pjkt585Xd3uqrGtKogSRAFd8Umk7YTnxv6nGsNr9NFGQF7ksaXBFAGVYzSQ7LNm/DC9p w2ZA== X-Gm-Message-State: AOAM532KjLsr+AeZXzG+NZmxiydX21adYKXZapHxsmPikxHddnmp5+jE dEiWsCLq1jvD2UT0MFGmAA3vmltCTO0kRg== X-Google-Smtp-Source: ABdhPJyU1wEzfgG1GIS53C5EGqItGWlo05979nYIo9kyfDUFhaiV7OZ2bGhehtNPUeg0YHtMhQMtUg== X-Received: by 2002:a17:906:1912:: with SMTP id a18mr14152458eje.117.1623657054072; Mon, 14 Jun 2021 00:50:54 -0700 (PDT) Received: from oberon.zico.biz ([83.222.187.186]) by smtp.gmail.com with ESMTPSA id f6sm6536444eja.108.2021.06.14.00.50.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Jun 2021 00:50:53 -0700 (PDT) From: "Tzvetomir Stoyanov (VMware)" To: rostedt@goodmis.org Cc: linux-trace-devel@vger.kernel.org Subject: [PATCH v6 23/45] trace-cmd library: Add APIs for read and write compressed data in chunks Date: Mon, 14 Jun 2021 10:50:07 +0300 Message-Id: <20210614075029.598048-24-tz.stoyanov@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210614075029.598048-1-tz.stoyanov@gmail.com> References: <20210614075029.598048-1-tz.stoyanov@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-trace-devel@vger.kernel.org New compression APIs are added for writing and reading compressed data from / to files in chunks: tracecmd_compress_copy_from() tracecmd_uncompress_copy_to() Format of the compressed data, used by these APIs, is: - 4 bytes, chunks count - for each chunk: - 4 bytes, size of compressed data in this chunk - 4 bytes, uncompressed size of the data in this chunk - data, bytes of Signed-off-by: Tzvetomir Stoyanov (VMware) --- .../include/private/trace-cmd-private.h | 5 +- lib/trace-cmd/trace-compress.c | 200 ++++++++++++++++++ 2 files changed, 204 insertions(+), 1 deletion(-) diff --git a/lib/trace-cmd/include/private/trace-cmd-private.h b/lib/trace-cmd/include/private/trace-cmd-private.h index 10089389..067ba34d 100644 --- a/lib/trace-cmd/include/private/trace-cmd-private.h +++ b/lib/trace-cmd/include/private/trace-cmd-private.h @@ -494,7 +494,10 @@ int tracecmd_compress_proto_register(const char *name, const char *version, int char *out, unsigned int *out_bytes), unsigned int (*comress_size)(unsigned int bytes), bool (*is_supported)(const char *name, const char *version)); - +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size); +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size); /* --- Plugin handling --- */ extern struct tep_plugin_option trace_ftrace_options[]; diff --git a/lib/trace-cmd/trace-compress.c b/lib/trace-cmd/trace-compress.c index 039f60c5..dfed7b0e 100644 --- a/lib/trace-cmd/trace-compress.c +++ b/lib/trace-cmd/trace-compress.c @@ -595,3 +595,203 @@ error: free(v); return -1; } + +/** + * tracecmd_compress_copy_from - Copy and compress data from a file + * @handle: compression handler + * @fd: file descriptor to uncompressed data to copy from + * @chunk_size: size of one compression chunk + * @read_size: in - max bytes to read from @fd, 0 to read till the EOF + * out - size of the uncompressed data read from @fd + * @write_size: return, size of the compressed data written into @handle + * + * This function reads uncompressed data from given @fd, compresses the data using the @handle + * compression context and writes the compressed data into the fd associated with the @handle. + * The data is compressed on chunks with given @chunk_size size. + * The compressed data is written in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_compress_copy_from(struct tracecmd_compression *handle, int fd, int chunk_size, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int rchunk = 0; + unsigned int chunks = 0; + unsigned int wsize = 0; + unsigned int rsize = 0; + unsigned int rmax = 0; + unsigned int csize; + unsigned int size; + unsigned int r; + off64_t offset; + char *buf_from; + char *buf_to; + int endian4; + int ret; + + if (!handle || !handle->proto || + !handle->proto->compress_block || !handle->proto->compress_size) + return 0; + if (read_size) + rmax = *read_size; + csize = handle->proto->compress_size(chunk_size); + buf_from = malloc(chunk_size); + if (!buf_from) + return -1; + buf_to = malloc(csize); + if (!buf_to) + return -1; + /* save the initial offset and write 0 chunks */ + offset = lseek64(handle->fd, 0, SEEK_CUR); + write_fd(handle->fd, &chunks, 4); + + do { + if (rmax > 0 && (rmax - rsize) < chunk_size) + rchunk = (rmax - rsize); + else + rchunk = chunk_size; + + r = read(fd, buf_from, rchunk); + if (r < 0 || (rmax > 0 && rsize >= rmax)) + break; + rsize += r; + size = csize; + if (r > 0) { + ret = handle->proto->compress_block(buf_from, r, buf_to, &size); + if (ret < 0) { + if (errno == EINTR) + continue; + break; + } + /* Write compressed data size */ + endian4 = tep_read_number(handle->tep, &size, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write uncompressed data size */ + endian4 = tep_read_number(handle->tep, &r, 4); + ret = write_fd(handle->fd, &endian4, 4); + if (ret != 4) + break; + /* Write the compressed data */ + ret = write_fd(handle->fd, buf_to, size); + if (ret != size) + break; + /* data + compress header */ + wsize += (size + 8); + chunks++; + } + } while (r > 0); + free(buf_from); + free(buf_to); + if (r) + return -1; + if (lseek64(handle->fd, offset, SEEK_SET) == (off_t)-1) + return -1; + endian4 = tep_read_number(handle->tep, &chunks, 4); + /* write chunks count*/ + write_fd(handle->fd, &chunks, 4); + lseek64(handle->fd, offset, SEEK_SET); + if (lseek64(handle->fd, 0, SEEK_END) == (off_t)-1) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} + +/** + * tracecmd_uncompress_copy_to - Uncompress data and copy to a file + * @handle: compression handler + * @fd: file descriptor to uncompressed data to copy into + * @read_size: return, size of the compressed data read from @handle + * @write_size: return, size of the uncompressed data written into @fd + * + * This function reads compressed data from the fd, associated with @handle, uncompresses it + * using the @handle compression context and writes the uncompressed data into the fd. + * The compressed data must be in the format: + * - 4 bytes, chunks count + * - for each chunk: + * - 4 bytes, size of compressed data in this chunk + * - 4 bytes, uncompressed size of the data in this chunk + * - data, bytes of + * + * On success 0 is returned, @read_size and @write_size are updated with the size of + * read and written data. + */ +int tracecmd_uncompress_copy_to(struct tracecmd_compression *handle, int fd, + unsigned long long *read_size, unsigned long long *write_size) +{ + unsigned int s_uncompressed; + unsigned int s_compressed; + unsigned int rsize = 0; + unsigned int wsize = 0; + char *bytes_out = NULL; + char *bytes_in = NULL; + int size_out; + int size_in; + int chunks; + char buf[4]; + char *tmp; + int ret; + + if (!handle || !handle->proto || !handle->proto->uncompress_block) + return -1; + + if (read(handle->fd, buf, 4) != 4) + return -1; + chunks = tep_read_number(handle->tep, buf, 4); + rsize += 4; + while (chunks) { + if (read(handle->fd, buf, 4) != 4) + break; + s_compressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (read(handle->fd, buf, 4) != 4) + break; + s_uncompressed = tep_read_number(handle->tep, buf, 4); + rsize += 4; + if (!bytes_in || size_in < s_compressed) { + tmp = realloc(bytes_in, s_compressed); + if (!tmp) + break; + bytes_in = tmp; + size_in = s_compressed; + } + + if (!bytes_out || size_out < s_uncompressed) { + tmp = realloc(bytes_out, s_uncompressed); + if (!tmp) + break; + bytes_out = tmp; + size_out = s_uncompressed; + } + + if (read_fd(handle->fd, bytes_in, s_compressed) < 0) + break; + rsize += s_compressed; + ret = handle->proto->uncompress_block(bytes_in, s_compressed, + bytes_out, &s_uncompressed); + if (ret) + break; + write_fd(fd, bytes_out, s_uncompressed); + wsize += s_uncompressed; + chunks--; + } + free(bytes_in); + free(bytes_out); + if (chunks) + return -1; + if (read_size) + *read_size = rsize; + if (write_size) + *write_size = wsize; + return 0; +} -- 2.31.1