[PATCH v5 0/3] git config cache & special querying api utilizing the cache

[PATCH v5 0/3] git config cache & special querying api utilizing the cache

[Tanay Abhra]

Hi,

[PATCH V5]: `config_set` now uses a single hashmap. Corrected style nits raised in
			the thread[7]. Thanks to Junio and Matthieu for their suggestions.

[PATCH v4]: Introduced `config_set` construct which points to a ordered set of
	config-files cached as hashmaps. Added relevant API functions. For more
	details see the documentation. Rewrote the git_config_get* family to use
	`config_set` internally. Added tests for both config_set API and git_config_get
	family. Added type specific API functions which parses the found value and
	converts it into a specific type.
	Most of the changes implemented are the result of discussion in [6].
	Thanks to Eric, Ramsay, Junio, Matthieu & Karsten for their suggestions
	and review.

[PATCH v3]: Added flag for NULL values that were causing segfaults in some cases.
	Added test-config for usage examples.
	Minor changes and corrections. Refer to discussion thread[5] for more details.
	Thanks to Matthieu, Jeff and Junio for their valuable suggestions.

[PATCH v2]:Changed the string_list to a struct instead of pointer to a struct.
	Added string-list initilization functions.
	Minor mistakes corrected acoording to review comments[4]. Thanks to
	Eric and Matthieu for their review.

[PATCH V1]:Most of the invaluable suggestions by Eric Sunshine, Torsten Bogershausen and
	Jeff King has been implemented[1]. Complete rewrite of config_cache*() family
	using git_config() as hook as suggested by Jeff. Thanks for the review.

[RFC V2]: Improved according to the suggestions by Eric Sunshine and Torsten Bogershausen.
	Added cache invalidation when config file is changed.[2]
	I am using git_config_set_multivar_in_file() as an update hook.

This is patch is for this year's GSoC. My project is
"Git Config API improvements". The link of my proposal is appended below [3].

The aim of this patch series is to generate a cache for querying values from
the config files in a non-callback manner as the current method reads and
parses the config files every time a value is queried for.

The cache is generated from hooking the update_cache function to the current
parsing and callback mechanism in config.c. It is implemented as an hashmap
using the hashmap-api with variables and its corresponding values list as
its members. The values in the list are sorted in order of increasing priority.
The cache is initialised the first time when any of the new query functions is
called. It is invalidated by using git_config_set_multivar_in_file() as an
update hook.

We add two new functions to the config-api, git_config_get_string() and
git_config_get_string_multi() for querying in a non callback manner from
the cache.

[1] http://marc.info/?t=140172066200006&r=1&w=2
[2] http://git.661346.n2.nabble.com/RFC-PATCH-0-2-Git-config-cache-amp-special-querying-api-utilizing-the-cache-td7611691.html
[3] https://drive.google.com/file/d/0B4suZ-aHqDcnSUZJRXVTTnZUN1E/edit?usp=sharing
[4] http://thread.gmane.org/gmane.comp.version-control.git/251073/focus=251369
[5] http://thread.gmane.org/gmane.comp.version-control.git/251704/
[6] http://thread.gmane.org/gmane.comp.version-control.git/252329/
[7] http://marc.info/?t=140428115200001&r=1&w=2


Tanay Abhra (2):
  config-hash.c
  test-config

 .gitignore                             |   1 +
 Documentation/technical/api-config.txt | 134 +++++++++++++++
 Makefile                               |   2 +
 cache.h                                |  34 ++++
 config-hash.c                          | 297 +++++++++++++++++++++++++++++++++
 config.c                               |   3 +
 t/t1308-config-hash.sh                 | 187 +++++++++++++++++++++
 test-config.c                          | 127 ++++++++++++++
 8 files changed, 785 insertions(+)
 create mode 100644 config-hash.c
 create mode 100755 t/t1308-config-hash.sh
 create mode 100644 test-config.c

-- 
1.9.0.GIT

[PATCH v5 1/2] add `config_set` API for caching config-like files

[Tanay Abhra]

Currently `git_config()` uses a callback mechanism and file rereads for
config values. Due to this approach, it is not uncommon for the config
files to be parsed several times during the run of a git program, with
different callbacks picking out different variables useful to themselves.

Add a `config_set`, that can be used to construct an in-memory cache for
config-like files that the caller specifies (i.e., files like `.gitmodules`,
`~/.gitconfig` etc.). Add two external functions `git_configset_get_value`
and `git_configset_get_value_multi` for querying from the config sets.
`git_configset_get_value` follows `last one wins` semantic (i.e. if there
are multiple matches for the queried key in the files of the configset the
value returned will be the last entry in `value_list`).
`git_configset_get_value_multi` returns a list of values sorted in order of
increasing priority (i.e. last match will be at the end of the list). Add
type specific query functions like `git_configset_get_bool` and similar.

Add a default `config_set`, `the_config_set` to cache all key-value pairs
read from usual config files (repo specific .git/config, user wide
~/.gitconfig and the global /etc/gitconfig). `the_config_set` uses a
single hashmap populated using `git_config()`.

Add two external functions `git_config_get_value` and 
`git_config_get_value_multi` for querying in a non-callback manner from
`the_config_set`. Also, add type specific query functions that are
implemented as a thin wrapper around the `config_set` API.

Signed-off-by: Tanay Abhra <tanayabh@gmail.com>
---
 Documentation/technical/api-config.txt | 134 +++++++++++++++
 Makefile                               |   1 +
 cache.h                                |  34 ++++
 config-hash.c                          | 297 +++++++++++++++++++++++++++++++++
 config.c                               |   3 +
 5 files changed, 469 insertions(+)
 create mode 100644 config-hash.c

diff --git a/Documentation/technical/api-config.txt b/Documentation/technical/api-config.txt
index 230b3a0..bdf86d0 100644
--- a/Documentation/technical/api-config.txt
+++ b/Documentation/technical/api-config.txt
@@ -77,6 +77,81 @@ To read a specific file in git-config format, use
 `git_config_from_file`. This takes the same callback and data parameters
 as `git_config`.
 
+Querying For Specific Variables
+-------------------------------
+
+For programs wanting to query for specific variables in a non-callback
+manner, the config API provides two functions `git_config_get_value`
+and `git_config_get_value_multi`. They both read values from an internal
+cache generated previously from reading the config files.
+
+`int git_config_get_value(const char *key, const char **value)`::
+
+	Finds the highest-priority value for the configuration variable `key`,
+	stores the pointer to it in `value` and returns 0. When the
+	configuration variable `key` is not found, returns 1 without touching
+	`value`. The caller should not free or modify `value`, as it is owned
+	by the cache.
+
+`const struct string_list *git_config_get_value_multi(const char *key)`::
+
+	Finds and returns the value list, sorted in order of increasing priority
+	for the configuration variable `key`. When the configuration variable
+	`key` is not found, returns NULL. The caller should not free or modify
+	the returned pointer, as it is owned by the cache.
+
+`void git_config_clear(void)`::
+
+	Resets and invalidate the config cache. 
+
+The config API also provides type specific API functions which do conversion
+as well as retrieval for the queried variable, including:
+
+`int git_config_get_int(const char *key, int *dest)`::
+
+	Finds and parses the value to an integer for the configuration variable
+	`key`. Dies on error; otherwise, stores pointer to the parsed integer in
+	`dest` and returns 0. When the configuration variable `key` is not found,
+	returns 1 without touching `dest`.
+
+`int git_config_get_ulong(const char *key, unsigned long *dest)`::
+
+	Similar to `git_config_get_int` but for unsigned longs.
+
+`int git_config_get_int(const char *key, int *dest)`::
+
+	Finds and parses the value into a boolean value, for the configuration
+	variable `key`respecting keywords like "true" and "false". Integer
+	values are converted into true/false values (when they are non-zero or
+	zero, respectively). Other values cause a die(). If parsing is successful,
+	stores the pointer to the parsed result in `dest` and returns 0. When the
+	configuration variable `key` is not found, returns 1 without touching
+	`dest`.
+
+`int git_config_get_bool_or_int(const char *key, int *is_bool, int *dest)`::
+
+	Similar to `git_config_get_bool`, except that integers are copied as-is,
+	and `is_bool` flag is unset.
+
+`int git_config_get_maybe_bool(const char *key, int *dest)`::
+
+	Similar to `git_config_get_bool`, except that it returns -1 on error
+	rather than dying.
+
+`int git_config_get_string(const char *key, const char **dest)`::
+
+	Allocates and copies the retrieved string into the `dest` parameter for
+	the configuration variable `key`; if NULL string is given, prints an
+	error message and returns -1. When the configuration variable `key` is
+	not found, returns 1 without touching `dest`.
+
+`int git_config_get_pathname(const char *key, const char **dest)`::
+
+	Similar to `git_config_get_string`, but expands `~` or `~user` into
+	the user's home directory when found at the beginning of the path.
+
+See test-config.c for usage examples.
+
 Value Parsing Helpers
 ---------------------
 
@@ -134,6 +209,65 @@ int read_file_with_include(const char *file, config_fn_t fn, void *data)
 `git_config` respects includes automatically. The lower-level
 `git_config_from_file` does not.
 
+Custom Configsets
+-----------------
+
+A `config_set` can be used to construct an in-memory cache for config-like
+files that the caller specifies (i.e., files like `.gitmodules`,
+`~/.gitconfig` etc.). For example,
+
+---------------------------------------
+struct config_set gm_config;
+git_configset_init(&gm_config);
+int b;
+/* we add config files to the config_set */
+git_configset_add_file(&gm_config, ".gitmodules");
+git_configset_add_file(&gm_config, ".gitmodules_alt");
+
+if (!git_configset_get_bool(gm_config, "submodule.frotz.ignore", &b)) {
+	/* hack hack hack */
+}
+
+/* when we are done with the configset */
+git_configset_clear(&gm_config);
+----------------------------------------
+
+Configset API provides functions for the above mentioned work flow, including:
+
+`void git_configset_init(struct config_set *cs)`::
+
+	Initializes the member variables of config_set `cs`.
+
+`int git_configset_add_file(struct config_set *cs, const char *filename)`::
+
+	Parses the file and adds the variable-value pairs to the `config_set`,
+	dies if there is an error in parsing the file.
+
+`int git_configset_get_value(struct config_set *cs, const char *key, const char **value)`::
+
+	Finds the highest-priority value for the configuration variable `key`
+	and config set `cs`, stores the pointer to it in `value` and returns 0.
+	When the configuration variable `key` is not found, returns 1 without
+	touching `value`. The caller should not free or modify `value`, as it
+	is owned by the cache.
+
+`const struct string_list *git_configset_get_value_multi(struct config_set *cs, const char *key)`::
+
+	Finds and returns the value list, sorted in order of increasing priority
+	for the configuration variable `key` and config set `cs`. When the
+	configuration variable `key` is not found, returns NULL. The caller
+	should not free or modify the returned pointer, as it is owned by the cache.
+
+`void git_configset_clear(struct config_set *cs)`::
+
+	Clears `config_set` structure, removes all saved variable-value pairs.
+
+In addition to above functions, the `config_set` API provides type specific
+functions in the vein of `git_config_get_int` and family but with an extra
+parameter, pointer to struct `config_set`.
+They all behave similarly to the `git_config_get*()` family described in
+"Querying For Specific Variables" above.
+
 Writing Config Files
 --------------------
 
diff --git a/Makefile b/Makefile
index 07ea105..d503f78 100644
--- a/Makefile
+++ b/Makefile
@@ -777,6 +777,7 @@ LIB_OBJS += commit.o
 LIB_OBJS += compat/obstack.o
 LIB_OBJS += compat/terminal.o
 LIB_OBJS += config.o
+LIB_OBJS += config-hash.o
 LIB_OBJS += connect.o
 LIB_OBJS += connected.o
 LIB_OBJS += convert.o
diff --git a/cache.h b/cache.h
index df65231..35bd71e 100644
--- a/cache.h
+++ b/cache.h
@@ -1325,6 +1325,40 @@ extern int parse_config_key(const char *var,
 			    const char **subsection, int *subsection_len,
 			    const char **key);
 
+/* config-hash.c */
+
+struct config_set {
+	struct hashmap config_hash;
+	int hash_initialized;
+};
+
+extern void git_configset_init(struct config_set *cs);
+extern int git_configset_add_file(struct config_set *cs, const char *filename);
+extern int git_configset_get_value(struct config_set *cs, const char *key, const char **value);
+extern const struct string_list *git_configset_get_value_multi(struct config_set *cs, const char *key);
+extern void git_configset_clear(struct config_set *cs);
+extern void git_configset_iter(struct config_set *cs, config_fn_t fn, void *data);
+extern int git_configset_get_string(struct config_set *cs, const char *key, const char **dest);
+extern int git_configset_get_int(struct config_set *cs, const char *key, int *dest);
+extern int git_configset_get_ulong(struct config_set *cs, const char *key, unsigned long *dest);
+extern int git_configset_get_bool(struct config_set *cs, const char *key, int *dest);
+extern int git_configset_get_bool_or_int(struct config_set *cs, const char *key, int *is_bool, int *dest);
+extern int git_configset_get_maybe_bool(struct config_set *cs, const char *key, int *dest);
+extern int git_configset_get_pathname(struct config_set *cs, const char *key, const char **dest);
+
+extern int git_config_get_value(const char *key, const char **value);
+extern const struct string_list *git_config_get_value_multi(const char *key);
+extern void git_config_clear(void);
+extern void git_config_iter(config_fn_t fn, void *data);
+extern int git_config_get_string(const char *key, const char **dest);
+extern int git_config_get_int(const char *key, int *dest);
+extern int git_config_get_ulong(const char *key, unsigned long *dest);
+extern int git_config_get_bool(const char *key, int *dest);
+extern int git_config_get_bool_or_int(const char *key, int *is_bool, int *dest);
+extern int git_config_get_maybe_bool(const char *key, int *dest);
+extern int git_config_get_pathname(const char *key, const char **dest);
+
+
 extern int committer_ident_sufficiently_given(void);
 extern int author_ident_sufficiently_given(void);
 
diff --git a/config-hash.c b/config-hash.c
new file mode 100644
index 0000000..4c56bd9
--- /dev/null
+++ b/config-hash.c
@@ -0,0 +1,297 @@
+#include "cache.h"
+#include "hashmap.h"
+#include "string-list.h"
+
+
+/*
+ * Default config_set that contains key-value pairs from the usual set of config
+ * config files (i.e repo specific .git/config, user wide ~/.gitconfig and the
+ * global /etc/gitconfig)
+ */
+static struct config_set the_config_set;
+
+struct config_hash_entry {
+	struct hashmap_entry ent;
+	char *key;
+	struct string_list value_list;
+};
+
+static int config_hash_add_value(const char *, const char *, struct hashmap *);
+
+static int config_hash_entry_cmp(const struct config_hash_entry *e1,
+				 const struct config_hash_entry *e2, const void *unused)
+{
+	return strcmp(e1->key, e2->key);
+}
+
+static void config_hash_init(struct hashmap *config_hash)
+{
+	hashmap_init(config_hash, (hashmap_cmp_fn)config_hash_entry_cmp, 0);
+}
+
+static int config_hash_callback(const char *key, const char *value, void *cb)
+{
+	struct config_set *cs = cb;
+	config_hash_add_value(key, value, &cs->config_hash);
+	return 0;
+}
+
+static int add_configset_hash(const char *filename, struct config_set *cs)
+{
+	int ret = 0;
+	if (!cs->hash_initialized)
+		config_hash_init(&cs->config_hash);
+	cs->hash_initialized = 1;
+	ret = git_config_from_file(config_hash_callback, filename, cs);
+	if (!ret)
+		return 0;
+	else {
+		hashmap_free(&cs->config_hash, 1);
+		cs->hash_initialized = 0;
+		return -1;
+	}
+}
+
+static struct config_hash_entry *config_hash_find_entry(const char *key,
+							struct hashmap *config_hash)
+{
+	struct config_hash_entry k;
+	struct config_hash_entry *found_entry;
+	char *normalized_key;
+	int ret;
+	ret = git_config_parse_key(key, &normalized_key, NULL);
+
+	if (ret)
+		return NULL;
+
+	hashmap_entry_init(&k, strhash(normalized_key));
+	k.key = normalized_key;
+	found_entry = hashmap_get(config_hash, &k, NULL);
+	free(normalized_key);
+	return found_entry;
+}
+
+static struct string_list *configset_get_list(const char *key, struct config_set *cs)
+{
+	struct config_hash_entry *e = config_hash_find_entry(key, &cs->config_hash);
+	return e ? &e->value_list : NULL;
+}
+
+static int config_hash_add_value(const char *key, const char *value, struct hashmap *config_hash)
+{
+	struct config_hash_entry *e;
+	e = config_hash_find_entry(key, config_hash);
+
+	if (!e) {
+		e = xmalloc(sizeof(*e));
+		hashmap_entry_init(e, strhash(key));
+		e->key = xstrdup(key);
+		memset(&e->value_list, 0, sizeof(e->value_list));
+		e->value_list.strdup_strings = 1;
+		hashmap_add(config_hash, e);
+	}
+	/*
+	 * Since the values are being fed by git_config*() callback mechanism, they
+	 * are already normalized. So simply add them without any further munging.
+	 */
+	string_list_append_nodup(&e->value_list, value ? xstrdup(value) : NULL);
+
+	return 0;
+}
+
+void git_configset_init(struct config_set *cs)
+{
+	cs->hash_initialized = 0;
+}
+
+int git_configset_add_file(struct config_set *cs, const char *filename)
+{
+	return add_configset_hash(filename, cs);
+}
+
+int git_configset_get_value(struct config_set *cs, const char *key, const char **value)
+{
+	struct string_list *values = NULL;
+	/*
+	 * Follows "last one wins" semantic, i.e., if there are multiple matches for the
+	 * queried key in the files of the configset, the value returned will be the last
+	 * value in the value list for that key.
+	 */
+	values = configset_get_list(key, cs);
+
+	if (!values)
+		return 1;
+	assert(values->nr > 0);
+	*value = values->items[values->nr - 1].string;
+	return 0;
+}
+
+const struct string_list *git_configset_get_value_multi(struct config_set *cs, const char *key)
+{
+	return configset_get_list(key, cs);
+}
+
+void git_configset_clear(struct config_set *cs)
+{
+	struct config_hash_entry *entry;
+	struct hashmap_iter iter;
+	if (!cs->hash_initialized)
+		return;
+
+	hashmap_iter_init( &cs->config_hash, &iter);
+	while ((entry = hashmap_iter_next(&iter))) {
+		free(entry->key);
+		string_list_clear(&entry->value_list, 0);
+	}
+	hashmap_free(&cs->config_hash, 1);
+}
+
+int git_configset_get_string(struct config_set *cs, const char *key, const char **dest)
+{
+	const char *value;
+	if (!git_configset_get_value(cs, key, &value))
+		return git_config_string(dest, key, value);
+	else
+		return 1;
+}
+
+int git_configset_get_int(struct config_set *cs, const char *key, int *dest)
+{
+	const char *value;
+	if (!git_configset_get_value(cs, key, &value)) {
+		*dest = git_config_int(key, value);
+		return 0;
+	} else
+		return 1;
+}
+
+int git_configset_get_ulong(struct config_set *cs, const char *key, unsigned long *dest)
+{
+	const char *value;
+	if (!git_configset_get_value(cs, key, &value)) {
+		*dest = git_config_ulong(key, value);
+		return 0;
+	} else
+		return 1;
+}
+
+int git_configset_get_bool(struct config_set *cs, const char *key, int *dest)
+{
+	const char *value;
+	if (!git_configset_get_value(cs, key, &value)) {
+		*dest = git_config_bool(key, value);
+		return 0;
+	} else
+		return 1;
+}
+
+int git_configset_get_bool_or_int(struct config_set *cs, const char *key,
+				int *is_bool, int *dest)
+{
+	const char *value;
+	if (!git_configset_get_value(cs, key, &value)) {
+		*dest = git_config_bool_or_int(key, value, is_bool);
+		return 0;
+	} else
+		return 1;
+}
+
+int git_configset_get_maybe_bool(struct config_set *cs, const char *key, int *dest)
+{
+	const char *value;
+	if (!git_configset_get_value(cs, key, &value)) {
+		*dest = git_config_maybe_bool(key, value);
+		if (*dest == -1)
+			return -1;
+		return 0;
+	} else
+		return 1;
+}
+
+int git_configset_get_pathname(struct config_set *cs, const char *key, const char **dest)
+{
+	const char *value;
+	if (!git_configset_get_value(cs, key, &value))
+		return git_config_pathname(dest, key, value);
+	else
+		return 1;
+}
+
+static int git_config_check_init(void)
+{
+	int ret = 0;
+	if (the_config_set.hash_initialized)
+		return 0;
+	config_hash_init(&the_config_set.config_hash);
+	the_config_set.hash_initialized = 1;
+	ret = git_config(config_hash_callback, &the_config_set);
+	if (ret >= 0)
+		return 0;
+	else {
+		hashmap_free(&the_config_set.config_hash, 1);
+		the_config_set.hash_initialized = 0;
+		return -1;
+	}
+}
+
+int git_config_get_value(const char *key, const char **value)
+{
+	git_config_check_init();
+	return git_configset_get_value(&the_config_set, key, value);
+}
+
+const struct string_list *git_config_get_value_multi(const char *key)
+{
+	git_config_check_init();
+	return git_configset_get_value_multi(&the_config_set, key);
+}
+
+void git_config_clear(void)
+{
+	if (!the_config_set.hash_initialized)
+		return;
+	git_configset_clear(&the_config_set);
+	the_config_set.hash_initialized = 0;
+}
+
+int git_config_get_string(const char *key, const char **dest)
+{
+	git_config_check_init();
+	return git_configset_get_string(&the_config_set, key, dest);
+}
+
+int git_config_get_int(const char *key, int *dest)
+{
+	git_config_check_init();
+	return git_configset_get_int(&the_config_set, key, dest);
+}
+
+int git_config_get_ulong(const char *key, unsigned long *dest)
+{
+	git_config_check_init();
+	return git_configset_get_ulong(&the_config_set, key, dest);
+}
+
+int git_config_get_bool(const char *key, int *dest)
+{
+	git_config_check_init();
+	return git_configset_get_bool(&the_config_set, key, dest);
+}
+
+int git_config_get_bool_or_int(const char *key, int *is_bool, int *dest)
+{
+	git_config_check_init();
+	return git_configset_get_bool_or_int(&the_config_set, key, is_bool, dest);
+}
+
+int git_config_get_maybe_bool(const char *key, int *dest)
+{
+	git_config_check_init();
+	return git_configset_get_maybe_bool(&the_config_set, key, dest);
+}
+
+int git_config_get_pathname(const char *key, const char **dest)
+{
+	git_config_check_init();
+	return git_configset_get_pathname(&the_config_set, key, dest);
+}
diff --git a/config.c b/config.c
index a1aef1c..6cffec7 100644
--- a/config.c
+++ b/config.c
@@ -1708,6 +1708,9 @@ int git_config_set_multivar_in_file(const char *config_filename,
 	lock = NULL;
 	ret = 0;
 
+	/* Invalidate the config cache */
+	git_config_clear();
+
 out_free:
 	if (lock)
 		rollback_lock_file(lock);
-- 
1.9.0.GIT

[PATCH v5 2/2] test-config: Add tests for the config_set API

[Tanay Abhra]

Expose the `config_set` C API as a set of simple commands in order to
facilitate testing. Add tests for the `config_set` API as well as for
`git_config_get_*()` family for the usual config files.

Signed-off-by: Tanay Abhra <tanayabh@gmail.com>
---
 .gitignore             |   1 +
 Makefile               |   1 +
 t/t1308-config-hash.sh | 187 +++++++++++++++++++++++++++++++++++++++++++++++++
 test-config.c          | 127 +++++++++++++++++++++++++++++++++
 4 files changed, 316 insertions(+)
 create mode 100755 t/t1308-config-hash.sh
 create mode 100644 test-config.c

diff --git a/.gitignore b/.gitignore
index 42294e5..eeb66e2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -176,6 +176,7 @@
 /gitweb/static/gitweb.js
 /gitweb/static/gitweb.min.*
 /test-chmtime
+/test-config
 /test-ctype
 /test-date
 /test-delta
diff --git a/Makefile b/Makefile
index d503f78..e875070 100644
--- a/Makefile
+++ b/Makefile
@@ -548,6 +548,7 @@ X =
 PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS))
 
 TEST_PROGRAMS_NEED_X += test-chmtime
+TEST_PROGRAMS_NEED_X += test-config
 TEST_PROGRAMS_NEED_X += test-ctype
 TEST_PROGRAMS_NEED_X += test-date
 TEST_PROGRAMS_NEED_X += test-delta
diff --git a/t/t1308-config-hash.sh b/t/t1308-config-hash.sh
new file mode 100755
index 0000000..eba7102
--- /dev/null
+++ b/t/t1308-config-hash.sh
@@ -0,0 +1,187 @@
+#!/bin/sh
+
+test_description='Test git config-hash API in different settings'
+
+. ./test-lib.sh
+
+test_expect_success 'clear default config' '
+	rm -f .git/config
+'
+
+test_expect_success 'initialize default config' '
+	cat >.git/config << EOF
+	[core]
+		penguin = very blue
+		Movie = BadPhysics
+		UPPERCASE = true
+		MixedCase = true
+		my =
+		foo
+		baz = sam
+	[Cores]
+		WhatEver = Second
+		baz = bar
+	[cores]
+		baz = bat
+	[CORES]
+		baz = ball
+	[my "Foo bAr"]
+		hi = mixed-case
+	[my "FOO BAR"]
+		hi = upper-case
+	[my "foo bar"]
+		hi = lower-case
+	[core]
+		baz = bat
+		baz = hask
+	[lamb]
+		chop = 65
+		head = none
+	[goat]
+		legs = 4
+		head = true
+		skin = false
+		nose = 1
+		horns
+	EOF
+'
+
+test_expect_success 'get value for a simple key' '
+	echo "very blue" >expect &&
+	test-config get_value core.penguin >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'get value for a key with value as an empty string' '
+	echo "" >expect &&
+	test-config get_value core.my >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'get value for a key with value as NULL' '
+	echo "(NULL)" >expect &&
+	test-config get_value core.foo >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'upper case key' '
+	echo "true" >expect &&
+	test-config get_value core.UPPERCASE >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'mixed case key' '
+	echo "true" >expect &&
+	test-config get_value core.MixedCase >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'key and value with mixed case' '
+	echo "BadPhysics" >expect &&
+	test-config get_value core.Movie >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'key with case sensitive subsection' '
+	echo "mixed-case" >expect &&
+	echo "upper-case" >>expect &&
+	echo "lower-case" >>expect &&
+	test-config get_value "my.Foo bAr.hi" >actual &&
+	test-config get_value "my.FOO BAR.hi" >>actual &&
+	test-config get_value "my.foo bar.hi" >>actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'key with case insensitive section header' '
+	echo "ball" >expect &&
+	echo "ball" >>expect &&
+	echo "ball" >>expect &&
+	test-config get_value cores.baz >actual &&
+	test-config get_value Cores.baz >>actual &&
+	test-config get_value CORES.baz >>actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'find value with misspelled key' '
+	echo "Value not found for \"my.fOo Bar.hi\"" >expect &&
+	test_must_fail test-config get_value "my.fOo Bar.hi" >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'find value with the highest priority' '
+	echo hask >expect &&
+	test-config get_value "core.baz">actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'find integer value for a key' '
+	echo 65 >expect &&
+	test-config get_int lamb.chop >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'find integer if value is non parse-able' '
+	echo 65 >expect &&
+	test_must_fail test-config get_int lamb.head >actual &&
+	test_must_fail test_cmp expect actual
+'
+
+test_expect_success 'find bool value for the entered key' '
+	cat >expect <<-\EOF &&
+	1
+	0
+	1
+	1
+	1
+	EOF
+	test-config get_bool goat.head >actual &&
+	test-config get_bool goat.skin >>actual &&
+	test-config get_bool goat.nose >>actual &&
+	test-config get_bool goat.horns >>actual &&
+	test-config get_bool goat.legs >>actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'find multiple values' '
+	cat >expect <<-\EOF &&
+	sam
+	bat
+	hask
+	EOF
+	test-config get_value_multi "core.baz">actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'find value from a configset' '
+	cat >config2 <<-\EOF &&
+	[core]
+		baz = lama
+	[my]
+		new = silk
+	[core]
+		baz = ball
+	EOF
+	echo silk >expect &&
+	test-config configset_get_value my.new config2 .git/config >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'find value with highest priority from a configset' '
+	echo hask > expect &&
+	test-config configset_get_value core.baz config2 .git/config  >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'find value_list for a key from a configset' '
+	cat >except <<-\EOF &&
+	sam
+	bat
+	hask
+	lama
+	ball
+	EOF
+	test-config configset_get_value core.baz config2 .git/config  >actual &&
+	test_cmp expect actual
+'
+
+test_done
diff --git a/test-config.c b/test-config.c
new file mode 100644
index 0000000..45ccd0a
--- /dev/null
+++ b/test-config.c
@@ -0,0 +1,127 @@
+#include "cache.h"
+#include "hashmap.h"
+#include "string-list.h"
+
+/*
+ * This program exposes the C API of the configuration mechanism
+ * as a set of simple commands in order to facilitate testing.
+ *
+ * Reads stdin and prints result of command to stdout:
+ *
+ * get_value -> prints the value with highest priority for the entered key
+ *
+ * get_value_multi -> prints all values for the entered key in increasing order
+ *		     of priority
+ *
+ * get_int -> print integer value for the entered key or die
+ *
+ * get_bool -> print bool value for the entered key or die
+ *
+ * configset_get_value -> returns value with the highest priority for the entered key
+ * 			from a config_set constructed from files entered as arguments.
+ *
+ * configset_get_value_multi -> returns value_list for the entered key sorted in
+ * 				ascending order of priority from a config_set 
+ * 				constructed from files entered as arguments.
+ *
+ * Examples:
+ *
+ * To print the value with highest priority for key "foo.bAr Baz.rock":
+ * 	test-config get_value "foo.bAr Baz.rock"
+ *
+ */
+
+
+int main(int argc, char **argv)
+{
+	int i, no_of_files;
+	int ret = 0;
+	const char *v;
+	int val;
+	const struct string_list *strptr;
+	struct config_set cs;
+	git_configset_init(&cs);
+	if (argc == 3 && !strcmp(argv[1], "get_value")) {
+		if (!git_config_get_value(argv[2], &v)) {
+			if (!v)
+				printf("(NULL)\n");
+			else
+				printf("%s\n", v);
+			return 0;
+		} else {
+			printf("Value not found for \"%s\"\n", argv[2]);
+			return -1;
+		}
+	} else if (argc == 3 && !strcmp(argv[1], "get_value_multi")) {
+		strptr = git_config_get_value_multi(argv[2]);
+		if (strptr) {
+			for (i = 0; i < strptr->nr; i++) {
+				v = strptr->items[i].string;
+				if (!v)
+					printf("(NULL)\n");
+				else
+					printf("%s\n", v);
+			}
+			return 0;
+		} else {
+			printf("Value not found for \"%s\"\n", argv[2]);
+			return -1;
+		}
+	} else if (argc == 3 && !strcmp(argv[1], "get_int")) {
+		if (!git_config_get_int(argv[2], &val)) {
+			printf("%d\n", val);
+			return 0;
+		} else {
+			printf("Value not found for \"%s\"\n", argv[2]);
+			return -1;
+		}
+	} else if (argc == 3 && !strcmp(argv[1], "get_bool")) {
+		if (!git_config_get_bool(argv[2], &val)) {
+			printf("%d\n", val);
+			return 0;
+		} else {
+			printf("Value not found for \"%s\"\n", argv[2]);
+			return -1;
+		}
+	} else if (!strcmp(argv[1], "configset_get_value")) {
+		for (i = 3; i < argc; i++) {
+			ret = git_configset_add_file(&cs, argv[i]);
+			if (ret)
+				return -1;
+		}
+		if (!git_configset_get_value(&cs, argv[2], &v)) {
+			if (!v)
+				printf("(NULL)\n");
+			else
+				printf("%s\n", v);
+			return 0;
+		} else {
+			printf("Value not found for \"%s\"\n", argv[2]);
+			return -1;
+		}
+
+	} else if (!strcmp(argv[1], "configset_get_value_multi")) {
+		for (i = 3; i < argc; i++) {
+			ret = git_configset_add_file(&cs, argv[i]);
+			if (ret)
+				return -1;
+		}
+		strptr = git_configset_get_value_multi(&cs, argv[2]);
+		if (strptr) {
+			for (i = 0; i < strptr->nr; i++) {
+				v = strptr->items[i].string;
+				if (!v)
+					printf("(NULL)\n");
+				else
+					printf("%s\n", v);
+			}
+			return 0;
+		} else {
+			printf("Value not found for \"%s\"\n", argv[2]);
+			return -1;
+		}
+	}
+
+	fprintf(stderr, "%s: Please check the syntax and the function name\n", argv[0]);
+	return 1;
+}
-- 
1.9.0.GIT

Re: [PATCH v5 1/2] add `config_set` API for caching config-like files

[Ramsay Jones]

On 06/07/14 08:19, Tanay Abhra wrote:
> Currently `git_config()` uses a callback mechanism and file rereads for
> config values. Due to this approach, it is not uncommon for the config
> files to be parsed several times during the run of a git program, with
> different callbacks picking out different variables useful to themselves.
> 
> Add a `config_set`, that can be used to construct an in-memory cache for
> config-like files that the caller specifies (i.e., files like `.gitmodules`,
> `~/.gitconfig` etc.). Add two external functions `git_configset_get_value`
> and `git_configset_get_value_multi` for querying from the config sets.
> `git_configset_get_value` follows `last one wins` semantic (i.e. if there
> are multiple matches for the queried key in the files of the configset the
> value returned will be the last entry in `value_list`).
> `git_configset_get_value_multi` returns a list of values sorted in order of
> increasing priority (i.e. last match will be at the end of the list). Add
> type specific query functions like `git_configset_get_bool` and similar.
> 
> Add a default `config_set`, `the_config_set` to cache all key-value pairs
> read from usual config files (repo specific .git/config, user wide
> ~/.gitconfig and the global /etc/gitconfig). `the_config_set` uses a
> single hashmap populated using `git_config()`.
> 
> Add two external functions `git_config_get_value` and 
> `git_config_get_value_multi` for querying in a non-callback manner from
> `the_config_set`. Also, add type specific query functions that are
> implemented as a thin wrapper around the `config_set` API.
> 
> Signed-off-by: Tanay Abhra <tanayabh@gmail.com>
> ---
>  Documentation/technical/api-config.txt | 134 +++++++++++++++
>  Makefile                               |   1 +
>  cache.h                                |  34 ++++
>  config-hash.c                          | 297 +++++++++++++++++++++++++++++++++
>  config.c                               |   3 +
>  5 files changed, 469 insertions(+)
>  create mode 100644 config-hash.c
> 
> diff --git a/Documentation/technical/api-config.txt b/Documentation/technical/api-config.txt
> index 230b3a0..bdf86d0 100644
> --- a/Documentation/technical/api-config.txt
> +++ b/Documentation/technical/api-config.txt
> @@ -77,6 +77,81 @@ To read a specific file in git-config format, use
>  `git_config_from_file`. This takes the same callback and data parameters
>  as `git_config`.
>  
> +Querying For Specific Variables
> +-------------------------------
> +
> +For programs wanting to query for specific variables in a non-callback
> +manner, the config API provides two functions `git_config_get_value`
> +and `git_config_get_value_multi`. They both read values from an internal
> +cache generated previously from reading the config files.
> +
> +`int git_config_get_value(const char *key, const char **value)`::
> +
> +	Finds the highest-priority value for the configuration variable `key`,
> +	stores the pointer to it in `value` and returns 0. When the
> +	configuration variable `key` is not found, returns 1 without touching
> +	`value`. The caller should not free or modify `value`, as it is owned
> +	by the cache.
> +
> +`const struct string_list *git_config_get_value_multi(const char *key)`::
> +
> +	Finds and returns the value list, sorted in order of increasing priority
> +	for the configuration variable `key`. When the configuration variable
> +	`key` is not found, returns NULL. The caller should not free or modify
> +	the returned pointer, as it is owned by the cache.
> +
> +`void git_config_clear(void)`::
> +
> +	Resets and invalidate the config cache. 
> +
> +The config API also provides type specific API functions which do conversion
> +as well as retrieval for the queried variable, including:
> +
> +`int git_config_get_int(const char *key, int *dest)`::
> +
> +	Finds and parses the value to an integer for the configuration variable
> +	`key`. Dies on error; otherwise, stores pointer to the parsed integer in

... stores the value of the parsed integer in `dest` ...

> +	`dest` and returns 0. When the configuration variable `key` is not found,
> +	returns 1 without touching `dest`.
> +
> +`int git_config_get_ulong(const char *key, unsigned long *dest)`::
> +
> +	Similar to `git_config_get_int` but for unsigned longs.
> +
> +`int git_config_get_int(const char *key, int *dest)`::
-----------------------^^^

git_config_get_bool

> +
> +	Finds and parses the value into a boolean value, for the configuration
> +	variable `key`respecting keywords like "true" and "false". Integer
> +	values are converted into true/false values (when they are non-zero or
> +	zero, respectively). Other values cause a die(). If parsing is successful,
> +	stores the pointer to the parsed result in `dest` and returns 0. When the

... stores the value of the parsed result in `dest` ...

> +	configuration variable `key` is not found, returns 1 without touching
> +	`dest`.
> +
> +`int git_config_get_bool_or_int(const char *key, int *is_bool, int *dest)`::
> +
> +	Similar to `git_config_get_bool`, except that integers are copied as-is,
> +	and `is_bool` flag is unset.
> +
> +`int git_config_get_maybe_bool(const char *key, int *dest)`::
> +
> +	Similar to `git_config_get_bool`, except that it returns -1 on error
> +	rather than dying.
> +
> +`int git_config_get_string(const char *key, const char **dest)`::
> +
> +	Allocates and copies the retrieved string into the `dest` parameter for
> +	the configuration variable `key`; if NULL string is given, prints an
> +	error message and returns -1. When the configuration variable `key` is
> +	not found, returns 1 without touching `dest`.
> +
> +`int git_config_get_pathname(const char *key, const char **dest)`::
> +
> +	Similar to `git_config_get_string`, but expands `~` or `~user` into
> +	the user's home directory when found at the beginning of the path.
> +
> +See test-config.c for usage examples.
> +

Sorry, I didn't have time to do more than squint at the code, but at first
glance it looks good.

ATB,
Ramsay Jones

Re: [PATCH v5 1/2] add `config_set` API for caching config-like files

[Matthieu Moy]

Tanay Abhra <tanayabh@gmail.com> writes:

> Add a default `config_set`, `the_config_set` to cache all key-value pairs
> read from usual config files (repo specific .git/config, user wide
> ~/.gitconfig and the global /etc/gitconfig). `the_config_set` uses a
> single hashmap populated using `git_config()`.

"uses a single hashmap" is no longer relevant. Any config_set use a
single hashmap.

(The "populated using git_config()" part is still relevant OTOH)

> +`void git_configset_init(struct config_set *cs)`::
> +
> +	Initializes the member variables of config_set `cs`.

I'd say just "initializes the config_set `cs`".

> +/*
> + * Default config_set that contains key-value pairs from the usual set of config
> + * config files (i.e repo specific .git/config, user wide ~/.gitconfig and the
> + * global /etc/gitconfig)

To be technically correct, "user wide ~/.gitconfig, XDG config file and
the global ...".

> +static int add_configset_hash(const char *filename, struct config_set *cs)
> +{
> +	int ret = 0;
> +	if (!cs->hash_initialized)
> +		config_hash_init(&cs->config_hash);
> +	cs->hash_initialized = 1;

That would seem more natural to me to have a function config_set_init
instead of config_hash_init, that would set hash_initialized.

config_hash_init is currently a one-liner, so there's no risk of making
it too complex.

> +static int config_hash_add_value(const char *key, const char *value, struct hashmap *config_hash)
> +{
> +	struct config_hash_entry *e;
> +	e = config_hash_find_entry(key, config_hash);
> +
> +	if (!e) {
> +		e = xmalloc(sizeof(*e));
> +		hashmap_entry_init(e, strhash(key));

Nitpick: you're hashing the same string twice. Once for
config_hash_find_entry, and another here. It would be slightly faster to
allow config_hash_find_entry to return the hashcode (as a by-address
parameter).

But you may want to keep the code as it is for simplicity.

It took me some time to understand why you normalize in
config_hash_find_entry and not here. My understanding is that
config_hash_find_entry is used to query the hashmap from the user API
(so you need normalization), but this particular codepath receives
normalized keys (the comment right below states that for values).

It's probably worth a comment here and/or in config_hash_find_entry.

> +	}
> +	/*
> +	 * Since the values are being fed by git_config*() callback mechanism, they
> +	 * are already normalized. So simply add them without any further munging.
> +	 */
> +	string_list_append_nodup(&e->value_list, value ? xstrdup(value) : NULL);
> +
> +	return 0;
> +}

[...]

> +int git_configset_add_file(struct config_set *cs, const char *filename)
> +{
> +	return add_configset_hash(filename, cs);
> +}

Why have two functions doing the same, with arguments in different
orders?

I guess it's just historical, and you can keep only one.

> +void git_configset_clear(struct config_set *cs)
> +{
> +	struct config_hash_entry *entry;
> +	struct hashmap_iter iter;
> +	if (!cs->hash_initialized)
> +		return;
> +
> +	hashmap_iter_init( &cs->config_hash, &iter);

No space after (.

> +int git_config_get_string(const char *key, const char **dest)
> +{
> +	git_config_check_init();
> +	return git_configset_get_string(&the_config_set, key, dest);
> +}
> +
> +int git_config_get_int(const char *key, int *dest)
> +{
> +	git_config_check_init();
> +	return git_configset_get_int(&the_config_set, key, dest);
> +}

Reading this list, I initially thought it might be worth factoring this
in a macro. But the list isn't that long, and contains several
special-cases (3 parameters instead of 2). So, don't bother.

OK, we're getting close. v6 should be really good :-).

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH v5 2/2] test-config: Add tests for the config_set API

[Matthieu Moy]

Tanay Abhra <tanayabh@gmail.com> writes:

> +test_expect_success 'get value for a simple key' '
> +	echo "very blue" >expect &&
> +	test-config get_value core.penguin >actual &&
> +	test_cmp expect actual
> +'

All these tests would greatly benefit from a helper like

test_expect_config () {
	echo "1" >expect &&
	test-config get_value "$2" >actual &&
	test_cmp expect actual
}

Then, all the 3-liners below would become 1-liners.

Should not block inclusion, but may be worth considering.

> +test_expect_success 'get value for a key with value as an empty string' '
> +	echo "" >expect &&
> +	test-config get_value core.my >actual &&
> +	test_cmp expect actual
> +'
> +
> +test_expect_success 'get value for a key with value as NULL' '
> +	echo "(NULL)" >expect &&
> +	test-config get_value core.foo >actual &&
> +	test_cmp expect actual
> +'
[...]

> +test_expect_success 'key with case sensitive subsection' '
> +	echo "mixed-case" >expect &&
> +	echo "upper-case" >>expect &&
> +	echo "lower-case" >>expect &&
> +	test-config get_value "my.Foo bAr.hi" >actual &&
> +	test-config get_value "my.FOO BAR.hi" >>actual &&
> +	test-config get_value "my.foo bar.hi" >>actual &&
> +	test_cmp expect actual
> +'

This would become a 3-liner with my helper.

> +test_expect_success 'key with case insensitive section header' '
> +	echo "ball" >expect &&
> +	echo "ball" >>expect &&
> +	echo "ball" >>expect &&
> +	test-config get_value cores.baz >actual &&
> +	test-config get_value Cores.baz >>actual &&
> +	test-config get_value CORES.baz >>actual &&
> +	test_cmp expect actual
> +'

I think you miss a simple case: get_value with a case that doesn't exist
in the config file, like "get_value coreS.baz".

> +test_expect_success 'find value with the highest priority' '
> +	echo hask >expect &&
> +	test-config get_value "core.baz">actual &&

Space before >.

> diff --git a/test-config.c b/test-config.c

No time for a real review of this file, but from a quick look, it seems
OK.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH v5 2/2] test-config: Add tests for the config_set API

[Matthieu Moy]

Tanay Abhra <tanayabh@gmail.com> writes:

> diff --git a/test-config.c b/test-config.c
> new file mode 100644
> index 0000000..45ccd0a
> --- /dev/null
> +++ b/test-config.c
> @@ -0,0 +1,127 @@
> +#include "cache.h"
> +#include "hashmap.h"

Useless include, you're not using the hashmap directly.

> +int main(int argc, char **argv)
> +{
> +	int i, no_of_files;

Unused no_of_files.

With

CFLAGS += -Wdeclaration-after-statement -Wall -Werror

in config.mak, my git.git refuses to compile with your patch. You should
have similar options for hacking on git.git.

> +	if (argc == 3 && !strcmp(argv[1], "get_value")) {

You should do something like

if (argc < 2) {
	fprintf(stderr, "Please, provide a command name on the command-line\n");
	return 1;
}

before this. Otherwise, some argv[1] below are invalid on mis-use. No
need for thorough checks since it's just a test program, but better
avoid undefined behavior and segfaults anyway...

> +		if (!git_config_get_value(argv[2], &v)) {
> +			if (!v)
> +				printf("(NULL)\n");
> +			else
> +				printf("%s\n", v);
> +			return 0;
> +		} else {
> +			printf("Value not found for \"%s\"\n", argv[2]);
> +			return -1;

Avoid returning negative values from main. Your shell's $? won't see -1,
but most likely 255 or so, but I think it even depends on the OS.

You don't seem to use main's return for anything but error, so 0 =
everything OK; 1 = some error is the common convention.

> +		} else {
> +			printf("Value not found for \"%s\"\n", argv[2]);
> +			return -1;
> +		}
> +
> +	} else if (!strcmp(argv[1], "configset_get_value_multi")) {

Why a blank line before this "else if" and not before other "else if"s?

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH v5 2/2] test-config: Add tests for the config_set API

[Ramkumar Ramachandra]

A couple of quick nits.

Tanay Abhra wrote:
> +test_expect_success 'clear default config' '
> +       rm -f .git/config
> +'

Unnecessary; a fresh temporary directory is created for each test run.

> +test_expect_success 'initialize default config' '

You might want to mark this as "setup".

Re: [PATCH v5 2/2] test-config: Add tests for the config_set API

[Matthieu Moy]

Ramkumar Ramachandra <artagnon@gmail.com> writes:

> A couple of quick nits.
>
> Tanay Abhra wrote:
>> +test_expect_success 'clear default config' '
>> +       rm -f .git/config
>> +'
>
> Unnecessary; a fresh temporary directory is created for each test run.

Hmm, fresh, but not empty.

Anyway, the next test does a cat > on this file, so it will erase its
content, so the "rm -f" is actually not needed.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/