[PATCH RFC v2 0/4] Add an external testing library for unit tests

From: steadmon@google.com
To: git@vger.kernel.org
Cc: Josh Steadmon <steadmon@google.com>,
	calvinwan@gmail.com, szeder.dev@gmail.com,
	phillip.wood123@gmail.com, chooglen@google.com, avarab@gmail.com,
	gitster@pobox.com, sandals@crustytoothpaste.net,
	Calvin Wan <calvinwan@google.com>,
	Phillip Wood <phillip.wood@dunelm.org.uk>
Subject: [PATCH RFC v2 0/4] Add an external testing library for unit tests
Date: Wed, 17 May 2023 16:56:30 -0700	[thread overview]
Message-ID: <20230517-unit-tests-v2-v2-0-21b5b60f4b32@google.com> (raw)

In our current testing environment, we spend a significant amount of
effort crafting end-to-end tests for error conditions that could easily
be captured by unit tests (or we simply forgo some hard-to-setup and
rare error conditions). Unit tests additionally provide stability to the
codebase and can simplify debugging through isolation. Turning parts of
Git into libraries[1] gives us the ability to run unit tests on the
libraries and to write unit tests in C. Writing unit tests in pure C,
rather than with our current shell/test-tool helper setup, simplifies
test setup, simplifies passing data around (no shell-isms required), and
reduces testing runtime by not spawning a separate process for every
test invocation.

Unit testing in C requires a separate testing harness that we ideally
would like to be TAP-style and to come with a non-restrictive license.
Fortunately, there already exists a C TAP harness library[2] with an MIT
license (at least for the files included in this series). Phillip Wood
has also proposed an alternative implementation[3]. I have not had a
chance to review that patch in much detail, but I have hopefully made
the Makefile integration here somewhat pluggable so that we can easily
switch to his version if it proves superior. For now, I have continued
with the C TAP library.

The first patch is a small cleanup to allow linking common_exit()
separately from common-main.o. In the second patch, I've added a rough
draft project plan listing some goals. Patch 3 adds the C TAP libraries.
Patch 4 is a modified version of Calvin's previous implemenation with
better integration to our Makefiles.

[1] https://lore.kernel.org/git/CAJoAoZ=Cig_kLocxKGax31sU7Xe4==BGzC__Bg2_pr7krNq6MA@mail.gmail.com/
[2] https://github.com/rra/c-tap-harness/
[3]: https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/

Signed-off-by: Josh Steadmon <steadmon@google.com>
---
Calvin Wan (1):
      Add C TAP harness

Josh Steadmon (3):
      common-main: split common_exit() into a new file
      unit tests: Add a project plan document
      unit test: add basic example and build rules

 .gitignore                             |    2 +
 Documentation/Makefile                 |    1 +
 Documentation/technical/unit-tests.txt |   47 +
 Makefile                               |   25 +-
 common-exit.c                          |   26 +
 common-main.c                          |   24 -
 t/Makefile                             |   10 +
 t/runtests.c                           | 1789 ++++++++++++++++++++++++++++++++
 t/strbuf-test.c                        |   54 +
 t/tap/basic.c                          | 1029 ++++++++++++++++++
 t/tap/basic.h                          |  198 ++++
 t/tap/macros.h                         |  109 ++
 12 files changed, 3289 insertions(+), 25 deletions(-)
---
base-commit: 69c786637d7a7fe3b2b8f7d989af095f5f49c3a8
change-id: 20230517-unit-tests-v2-94f50a7ccd8a

Best regards,
-- 
Josh Steadmon <steadmon@google.com>