Re: [PATCH v3 12/17] chunk-format: create read chunk API

From: "SZEDER Gábor" <szeder.dev@gmail.com>
To: Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Cc: git@vger.kernel.org, me@ttaylorr.com, gitster@pobox.com,
	l.s.r@web.de, Chris Torek <chris.torek@gmail.com>,
	Derrick Stolee <stolee@gmail.com>,
	Derrick Stolee <derrickstolee@github.com>,
	Derrick Stolee <dstolee@microsoft.com>
Subject: Re: [PATCH v3 12/17] chunk-format: create read chunk API
Date: Sun, 7 Feb 2021 21:20:42 +0100	[thread overview]
Message-ID: <20210207202042.GC1015009@szeder.dev> (raw)
In-Reply-To: <366eb2afee839feccdfd2244b231d2ad718c76d4.1612535453.git.gitgitgadget@gmail.com>

On Fri, Feb 05, 2021 at 02:30:47PM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <dstolee@microsoft.com>
> 
> Add the capability to read the table of contents, then pair the chunks
> with necessary logic using read_chunk_fn pointers. Callers will be added
> in future changes, but the typical outline will be:
> 
>  1. initialize a 'struct chunkfile' with init_chunkfile(NULL).
>  2. call read_table_of_contents().

A reader should call read_table_of_contents(), noted.

>  3. for each chunk to parse,
>     a. call pair_chunk() to assign a pointer with the chunk position, or
>     b. call read_chunk() to run a callback on the chunk start and size.
>  4. call free_chunkfile() to clear the 'struct chunkfile' data.

How could a user of this API learn about all chunks present in the
chunkfile, including unrecognized chunks?

> We are re-using the anonymous 'struct chunkfile' data, as it is internal
> to the chunk-format API. This gives it essentially two modes: write and
> read. If the same struct instance was used for both reads and writes,
> then there would be failures.
> 
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>

> diff --git a/chunk-format.h b/chunk-format.h
> index 9a1d770accec..0edcc57db4e7 100644
> --- a/chunk-format.h
> +++ b/chunk-format.h
> @@ -6,6 +6,19 @@
>  struct hashfile;
>  struct chunkfile;
>  
> +/*
> + * Initialize a 'struct chunkfile' for writing _or_ reading a file
> + * with the chunk format.
> + *
> + * If writing a file, supply a non-NULL 'struct hashfile *' that will
> + * be used to write.
> + *
> + * If reading a file, then supply the memory-mapped data to the
> + * pair_chunk() or read_chunk() methods, as appropriate.

And call read_table_of_contents() in between.

> + *
> + * DO NOT MIX THESE MODES. Use different 'struct chunkfile' instances
> + * for reading and writing.
> + */
>  struct chunkfile *init_chunkfile(struct hashfile *f);
>  void free_chunkfile(struct chunkfile *cf);
>  int get_num_chunks(struct chunkfile *cf);
> @@ -16,4 +29,37 @@ void add_chunk(struct chunkfile *cf,
>  	       chunk_write_fn fn);
>  int write_chunkfile(struct chunkfile *cf, void *data);
>  
> +int read_table_of_contents(struct chunkfile *cf,
> +			   const unsigned char *mfile,
> +			   size_t mfile_size,
> +			   uint64_t toc_offset,
> +			   int toc_length);
> +
> +#define CHUNK_NOT_FOUND (-2)
> +
> +/*
> + * Find 'chunk_id' in the given chunkfile and assign the
> + * given pointer to the position in the mmap'd file where
> + * that chunk begins.
> + *
> + * Returns CHUNK_NOT_FOUND if the chunk does not exist.
> + */
> +int pair_chunk(struct chunkfile *cf,
> +	       uint32_t chunk_id,
> +	       const unsigned char **p);
> +
> +typedef int (*chunk_read_fn)(const unsigned char *chunk_start,
> +			     size_t chunk_size, void *data);
> +/*
> + * Find 'chunk_id' in the given chunkfile and call the
> + * given chunk_read_fn method with the information for
> + * that chunk.
> + *
> + * Returns CHUNK_NOT_FOUND if the chunk does not exist.
> + */
> +int read_chunk(struct chunkfile *cf,
> +	       uint32_t chunk_id,
> +	       chunk_read_fn fn,
> +	       void *data);
> +
>  #endif
> -- 
> gitgitgadget
> 