[RFD] Libification proposal: separate internal and external interfaces

[RFD] Libification proposal: separate internal and external interfaces

[Calvin Wan]

This proposal was originally written by Kyle Lippincott, but he’s
currently on vacation for the next two weeks so I’m helping start this
discussion for him (from here on out Kyle is the “I”).

TL;DR: I'm proposing that when creating a library for code from the Git
codebase, we have two interfaces to this library: the "internal" one
that the rest of the Git codebase uses, and the "external" one for use
by other projects. The external interface will have a different coding
style and platform support than the rest of the codebase.

When thinking about potential issues and complications with
libification, I encountered a few broad categories of issues, and I'd
like to list them briefly (edit: turns out I can't be brief to save my
life) and float a proposal that may help minimize them.

Definitions
-----------
- When I say "Git" or "the git executable/binary" or whatever, I'm
  referring to "the collection of binaries, tests, etc. that are part of
  the main git repo" unless I say otherwise.
- Similarly, when I say "internal" I mean "for use by <that collection
  of programs>". When I say "external" I mean for use by stuff that's
  not part of the Git repository.

Assumptions
-----------
- Libraries that we're providing can be either statically or dynamically
  linked. Git will link statically to its own Git libraries. External
  projects may use either.
- Git must continue to be compilable and usable on all platforms it's
  currently supported on. Libification can't take that away. However,
  since libification is producing new interfaces for new use cases,
  there is no requirement that we make these new interfaces usable on
  all platforms, especially at first.
- We'd like as little churn and "uglification" of the main codebase as
  possible.

Issues
------
- Symbol name collisions: Since C doesn't have namespacing or other
  official name mangling mechanisms, all of the symbols inside of the
  library that aren't static are going to be at risk of colliding with
  symbols in the external project. This is especially a problem for
  common symbols like "error()".

- Header files: This is actually several related problems:
  - Git codebase's header files assume that anything that's brought in
    via <git-compat-util.h> is available; this includes system header
    files, but also macro definitions, including ones that change how
    various headers behave. Example: _GNU_SOURCE and
    _FILE_OFFSET_BITS=64 cause headers like <unistd.h> to change
    behavior; _GNU_SOURCE makes it provide different/additional
    functionality, and _FILE_OFFSET_BITS=64 makes types like `off_t` be
    64-bit (on some platforms it might be 32-bit without this define).
  - <git-compat-util.h> is expected to be included as the first header
    file in the translation unit, so as to make _GNU_SOURCE and similar
    #defines have the desired effect. If a translation unit (in an
    external library consumer) has already included <unistd.h>, we can't
    rely on them having had _GNU_SOURCE defined ahead of time
  - We can't just `#include <git-compat-util.h>` at the top of our
    external interface headers,
  - Git's header files make regular use of inlining. We can't assume
    that external projects are going to use static linking, and we can't
    assume that external projects are going to use a C-compatible
    language (they might not use our header files at all), so inline
    functions seem risky at the interface layer.

- Compatibility: Using code from the git codebase as a library is a new
  use case, we do not have the backwards compatibility requirements that
  we do for Git itself. We should take full advantage of this, and
  explicitly state what compatibility guarantees we are providing (or
  not providing).

Proposal
--------
Let's have a distinction between the "internal" interface (used by Git),
and the "external" interface (used by everyone else). The "external"
interface has several differences from the rest of the git codebase:

- Minimal. Only include symbols and types that we explicitly want to be
  part of the interface
    - This is both for API evolution abilities and providing a "well-lit
      path" to usage. Internal header files may have a lot of similar
      but slightly different functions that can be very confusing, or
      are highly specialized.
    - Most languages will not be able to include our headers. Reducing
      the interface to the minimal necessary means it's easier to
      identify when the interfaces change and update the
      non-C-compatible-language bindings.
    - The external interface should have as little code/new
      functionality as possible. All actual functionality should be in
      the internal interface(s).

- No inline functions. This is similar to minimal. We should put as
  little as possible in the header files, especially since many use
  cases involve using the library from a language that can't even
  #include them at all.

- Self Contained. The header files must work if they are the first/only
  #include in the external project. They must include everything they
  need, and not assume it was already handled for them.

- Tolerant. The header files probably won't be the first/only #include
  in the external project's translation unit, and they should still
  work. This means not using types like `off_t` or `struct stat` in the
  interfaces provided, since their sizes are dependent on the execution
  environment (what's been included, #defines, CFLAGS, etc.)

- Non-interfering. Our header files must not change fundamental things
  about the execution environment. This means they must not do things
  like #define _GNU_SOURCE or #define _FILE_OFFSET_BITS=64

- Limited Platform Compatibility. The external interfaces are able to
  assume that <stdint.h> and other C99 (or maybe even C11+)
  functionality exists and use it immediately, without weather balloons
  or #ifdefs. If some platform requires special handling, that platform
  isn't supported, at least initially.

- Non-colliding. Symbol names in these external interface headers should
  have a standard, obvious prefix as a manual namespacing solution.
  Something like `gitlib_`. (Prior art: cURL uses a `curl_` prefix for
  externally-visible symbols, libgit2 uses `git_<module_name>_foo`
  names; for internal symbols, they use `Curl_` and `git_<module>__foo`
  (with a double underscore), respectively)

- Translating. The external interface provides "external" symbol names,
  and potentially more compatible function interfaces than the internal
  interface does, and exists to translate from one domain to another.
  Most functions in the external interface will be just a single call to
  the internal interface. Examples:
    - Internal interface is `void foo();`; external interface would be
      `void gitlib_foo() { foo(); }`
    - Internal interface is `void foo(off_t val);`; external interface
      could be `void gitlib_foo(int64_t val) { foo(val); }` -- here we
      accept int64_t instead of off_t due to the issues around the size
      of off_t
    - Internal interface is `void foo(strbuf *s);`; the external
      interface might be `void gitlib_foo(char *s, size_t s_len) {
      strbuf sb; strbuf_init(&sb, s_len + 1); strbuf_add(&sb, s, s_len);
      foo(&sb); } ` -- since strbufs own the memory they hold, strings
      that come via the external interface might need to be copied to be
      memory safe.
    - Internal interface was `void foo();` but gained a new parameter.
      We don't need to expose this parameter in the external interface,
      and instead can just use a sensible default. External interface
      can remain `void gitlib_foo() { foo(NULL); }`

Proof of Concept
----------------
I think we should continue with the git-std-lib work as a manual
separation of the .c files and associated header files that comprise the
very lowest level of functionality in the git codebase. This manual
separation would only produce a library with an "internal" interface. We
should also start to apply these ideas by defining an "external"
interface which has a subset of the functionality in git-std-lib.

Automatic symbol hiding
-----------------------
One of the main driving forces behind my proposal above is avoiding
significant churn in the git codebase, for example needing to rename
every function in the codebase that's not static. While many function
names are unlikely to collide, such as `parse_oid_hex`, others are
significantly more likely, like `error` or `hex_to_bytes`. Needing to
rename all "plausible" collisions to things that are unlikely to
collide, like `GIT_error` or `GIT_hex_to_bytes` is tedious, error prone,
and unpleasant.

I have possibly discovered a truly remarkable solution, but this
footnote is too small to contain it. Wait, no it's not. This isn't fully
tested yet, but has shown promise in my initial tests using clang on a
Linux machine.
- Compile the "internal" interface(s) and all supporting code with
  `-fvisibility=hidden` to produce .o files for each .c file
- Compile the "external" interface(s) without hiding the symbols
- Produce a .a file that contains that code, for use by git itself
- "Partially link" the everything, using `ld -r`, to produce a single .o
  file
- Use `objcopy --localize-hidden` to actually hide the internal symbols
  from the "partially linked" .o file

This should leave us with two static libraries: one that has the symbols
marked as "hidden" but still usable, for use in git itself, and one that
contains the external interface, but doesn't expose the hidden symbols.

There may be similar solutions possible on other platforms, or there may
not, and we may need to do the great renaming (either in the code
itself, or via something like a giant set of linker scripts). While my
proposal to have a separation between the internal and external
interfaces is a requirement for making this automatic symbol hiding
solution work, I don't think that a failure to make the automatic symbol
hiding solution work means that we shouldn't have the internal/external
split. It's only one contributing point in favor of having the
internal/external split.

Re: [RFD] Libification proposal: separate internal and external interfaces

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3884 bytes --]

On 2024-04-02 at 14:18:51, Calvin Wan wrote:
> Issues
> ------
> - Symbol name collisions: Since C doesn't have namespacing or other
>   official name mangling mechanisms, all of the symbols inside of the
>   library that aren't static are going to be at risk of colliding with
>   symbols in the external project. This is especially a problem for
>   common symbols like "error()".

Yes, I think this is important.  We'll want to avoid the mistake that
OpenSSL and Perl made by just exporting random symbols or using more
than one namespace.

We'll also need to consider that libgit2 is currently using `git_` and
thus we'll either need to use something different or avoid conflicts.
Perhaps `gitlib_` might be useful.

I might also suggest that we enable symbol versioning on platforms that
support it, which is a nice way to avoid conflicts.

> - Header files: This is actually several related problems:
>   - Git codebase's header files assume that anything that's brought in
>     via <git-compat-util.h> is available; this includes system header
>     files, but also macro definitions, including ones that change how
>     various headers behave. Example: _GNU_SOURCE and
>     _FILE_OFFSET_BITS=64 cause headers like <unistd.h> to change
>     behavior; _GNU_SOURCE makes it provide different/additional
>     functionality, and _FILE_OFFSET_BITS=64 makes types like `off_t` be
>     64-bit (on some platforms it might be 32-bit without this define).

I should point out that _FILE_OFFSET_BITS=64 is effectively standard
these days.  Nobody wants their files limited to 2 GiB.

> - Tolerant. The header files probably won't be the first/only #include
>   in the external project's translation unit, and they should still
>   work. This means not using types like `off_t` or `struct stat` in the
>   interfaces provided, since their sizes are dependent on the execution
>   environment (what's been included, #defines, CFLAGS, etc.)

Sure.  If we need a file size type, it should be something like
`int64_t` or `uint64_t` and not `off_t`.

> - Limited Platform Compatibility. The external interfaces are able to
>   assume that <stdint.h> and other C99 (or maybe even C11+)
>   functionality exists and use it immediately, without weather balloons
>   or #ifdefs. If some platform requires special handling, that platform
>   isn't supported, at least initially.

I think this is fine.  It's 2024.  We should assume that C11 (a
13-year-old spec) is available and so is POSIX 1003.1-2008 (except for
Windows).  We may want to have a nice `#ifdef __STDC__ < 200112L` (and a
similar check for POSIX) to just produce an `#error` if the system is
too old.

We should therefore also assume that threading is available because
POSIX 1003.1-2008 requires it as part of the base specification (and
Windows obviously also supports it).  Because this is a library, we'll
want to avoid interfaces that are clearly not thread safe and document
the requirements for thread safety (either that the object must be
externally synchronized or that it is internally synchronized).

> - Non-colliding. Symbol names in these external interface headers should
>   have a standard, obvious prefix as a manual namespacing solution.
>   Something like `gitlib_`. (Prior art: cURL uses a `curl_` prefix for
>   externally-visible symbols, libgit2 uses `git_<module_name>_foo`
>   names; for internal symbols, they use `Curl_` and `git_<module>__foo`
>   (with a double underscore), respectively)

Yeah, I think we came up with the same idea.  I do like the
`gitlib_module_foo` approach.

Overall, I read through this and I didn't see anything I obviously
disagreed with.  Everything seemed to be either reasonable or something
I didn't have a strong opinion on.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

RE: [RFD] Libification proposal: separate internal and external interfaces

[rsbecker]

On Sunday, April 7, 2024 5:34 PM, brian m. carlson wrote:
>On 2024-04-02 at 14:18:51, Calvin Wan wrote:
>> Issues
>> ------
>> - Symbol name collisions: Since C doesn't have namespacing or other
>>   official name mangling mechanisms, all of the symbols inside of the
>>   library that aren't static are going to be at risk of colliding with
>>   symbols in the external project. This is especially a problem for
>>   common symbols like "error()".
>
>Yes, I think this is important.  We'll want to avoid the mistake that OpenSSL and Perl
>made by just exporting random symbols or using more than one namespace.
>
>We'll also need to consider that libgit2 is currently using `git_` and thus we'll either
>need to use something different or avoid conflicts.
>Perhaps `gitlib_` might be useful.

Or `GITL_`

>
>I might also suggest that we enable symbol versioning on platforms that support it,
>which is a nice way to avoid conflicts.
>
>> - Header files: This is actually several related problems:
>>   - Git codebase's header files assume that anything that's brought in
>>     via <git-compat-util.h> is available; this includes system header
>>     files, but also macro definitions, including ones that change how
>>     various headers behave. Example: _GNU_SOURCE and
>>     _FILE_OFFSET_BITS=64 cause headers like <unistd.h> to change
>>     behavior; _GNU_SOURCE makes it provide different/additional
>>     functionality, and _FILE_OFFSET_BITS=64 makes types like `off_t` be
>>     64-bit (on some platforms it might be 32-bit without this define).
>
>I should point out that _FILE_OFFSET_BITS=64 is effectively standard these days.
>Nobody wants their files limited to 2 GiB.

This is not the default on some platforms. It is still required but can be put into a knob.

>> - Tolerant. The header files probably won't be the first/only #include
>>   in the external project's translation unit, and they should still
>>   work. This means not using types like `off_t` or `struct stat` in the
>>   interfaces provided, since their sizes are dependent on the execution
>>   environment (what's been included, #defines, CFLAGS, etc.)
>
>Sure.  If we need a file size type, it should be something like `int64_t` or `uint64_t`
>and not `off_t`. Not all platforms have explicit 64-bit APIs compiles. NonStop and S/390 do require explicit controls to make this work.

This will result in compile warnings with some platforms with APIs that do use off_t for arguments and results.

>
>> - Limited Platform Compatibility. The external interfaces are able to
>>   assume that <stdint.h> and other C99 (or maybe even C11+)
>>   functionality exists and use it immediately, without weather balloons
>>   or #ifdefs. If some platform requires special handling, that platform
>>   isn't supported, at least initially.
>
>I think this is fine.  It's 2024.  We should assume that C11 (a 13-year-old spec) is
>available and so is POSIX 1003.1-2008 (except for Windows).  We may want to
>have a nice `#ifdef __STDC__ < 200112L` (and a similar check for POSIX) to just
>produce an `#error` if the system is too old.

I wish this were the case. I have 2 years before C11 is guaranteed to be available on platforms that I maintain. Can we wait until mid-late 2025 before doing this? I do not want to lose access to git.

>We should therefore also assume that threading is available because POSIX 1003.1-
>2008 requires it as part of the base specification (and Windows obviously also
>supports it).  Because this is a library, we'll want to avoid interfaces that are clearly
>not thread safe and document the requirements for thread safety (either that the
>object must be externally synchronized or that it is internally synchronized).

This is not a particularly good assumption. Some architectures do not support kernel-level threading. PUT threading is possible on some platforms but attempts to port git to PUT require that all dependencies also support PUT, which is not a good assumption. Most of the dependencies that I have to work with actually do not support PUT or any other form of threading.

>> - Non-colliding. Symbol names in these external interface headers should
>>   have a standard, obvious prefix as a manual namespacing solution.
>>   Something like `gitlib_`. (Prior art: cURL uses a `curl_` prefix for
>>   externally-visible symbols, libgit2 uses `git_<module_name>_foo`
>>   names; for internal symbols, they use `Curl_` and `git_<module>__foo`
>>   (with a double underscore), respectively)
>
>Yeah, I think we came up with the same idea.  I do like the `gitlib_module_foo`
>approach.
>
>Overall, I read through this and I didn't see anything I obviously disagreed with.
>Everything seemed to be either reasonable or something I didn't have a strong
>opinion on.

Significantly changing git dependencies and requirement are not intended to be part of the libification project. There are version serious consequences to doing that. Losing access to git is definitely not desirable from my point of view.

--Randall

Re: [RFD] Libification proposal: separate internal and external interfaces

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 5450 bytes --]

On 2024-04-07 at 21:48:07, rsbecker@nexbridge.com wrote:
> On Sunday, April 7, 2024 5:34 PM, brian m. carlson wrote:
> >We'll also need to consider that libgit2 is currently using `git_` and thus we'll either
> >need to use something different or avoid conflicts.
> >Perhaps `gitlib_` might be useful.
> 
> Or `GITL_`

It's more common to use lowercase, so I think we should do that.

> >I should point out that _FILE_OFFSET_BITS=64 is effectively standard these days.
> >Nobody wants their files limited to 2 GiB.
> 
> This is not the default on some platforms. It is still required but can be put into a knob.

True, but everyone compiles with it.  If you're a software vendor and
your build only supports 2 GiB files, everyone's going to be mad.  We
should be able to assume that vendors are prudent and reasonable people
who will compile their binaries accordingly and not have to return an
error if a requested file size is larger than 32 bits.

> >Sure.  If we need a file size type, it should be something like `int64_t` or `uint64_t`
> >and not `off_t`. Not all platforms have explicit 64-bit APIs compiles. NonStop and S/390 do require explicit controls to make this work.
> 
> This will result in compile warnings with some platforms with APIs
> that do use off_t for arguments and results.

There will need to be a cast.  Our APIs should use a standard 64-bit
type and not expose the platform `off_t`.

> >
> >> - Limited Platform Compatibility. The external interfaces are able to
> >>   assume that <stdint.h> and other C99 (or maybe even C11+)
> >>   functionality exists and use it immediately, without weather balloons
> >>   or #ifdefs. If some platform requires special handling, that platform
> >>   isn't supported, at least initially.
> >
> >I think this is fine.  It's 2024.  We should assume that C11 (a 13-year-old spec) is
> >available and so is POSIX 1003.1-2008 (except for Windows).  We may want to
> >have a nice `#ifdef __STDC__ < 200112L` (and a similar check for POSIX) to just
> >produce an `#error` if the system is too old.
> 
> I wish this were the case. I have 2 years before C11 is guaranteed to
> be available on platforms that I maintain. Can we wait until mid-late
> 2025 before doing this? I do not want to lose access to git.

As mentioned in the original proposal, we don't have to support all
platforms in the libified code.  The main Git binaries will continue to
function and be supported, but the new libified code will rely on newer
features.  You will still be able to have all the Git binaries and
functionality, but if you want the new shared library to compile
against, you'll have to furnish a newer compiler.

As a side note, I don't think requiring porters to support a 13-year-old
spec is unreasonable for new, independent parts of the codebase.  As a
side note, if your platform supported GCC or LLVM, then this would be an
easy goal to achieve.  I know it doesn't right now, but it might be an
incentive to get it there.

> This is not a particularly good assumption. Some architectures do not
> support kernel-level threading. PUT threading is possible on some
> platforms but attempts to port git to PUT require that all
> dependencies also support PUT, which is not a good assumption. Most of
> the dependencies that I have to work with actually do not support PUT
> or any other form of threading.

I don't believe this is true.  Every architecture which supports Linux
or any other modern Unix has threading support.  You're compiling on
ia64 and x86, which definitely do support threading and have had it on
Linux and NetBSD for years (before they removed ia64 support).

Threading is a reasonable thing to have on a modern operating system,
and if we're adding new, independent functionality, that should be able
to safely work with threading.  You can continue to use the binaries
without threading, just not the new shared libraries.

I'll add on that I'm not opposed to continuing to support NonStop in
principle, but I _am_ opposed to having to continue to support really
ancient versions of common standards[0].  POSIX 1003.1-2008 is about 16
years old; it could drive a car in much of North America.  I think it's
the responsibility of porters to provide this functionality, and in
general most open source OSes do this with little or no paid staff, so
it should be reasonable to expect corporate-backed operating systems to
do so.

I would strongly recommend that NonStop start supporting modern compiler
and POSIX standards, as well at least one of GCC or LLVM, and that you
push very hard to make that happen.  It's not just Git that's pushing
forward here; there's a lot of open source software that simply will not
compile with the functionality you support, including the increasing
amount of software written in Go or Rust, and it's unreasonable to
expect people not to want to use standard functionality that's over a
decade old.  If NonStop doesn't move in that direction, I anticipate we
will eventually drop support for it sooner or later.

[0] This is not just NonStop; Microsoft's long-time refusal to implement
C99 or a newer version of C really ground my gears and I basically gave
up on supporting it for all personal projects as a result.  Fortunately,
they've now come up to a modern standard.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

RE: [RFD] Libification proposal: separate internal and external interfaces

[rsbecker]

On Sunday, April 7, 2024 9:10 PM, brian m. carlson wrote:
>On 2024-04-07 at 21:48:07, rsbecker@nexbridge.com wrote:
>> On Sunday, April 7, 2024 5:34 PM, brian m. carlson wrote:
>> >We'll also need to consider that libgit2 is currently using `git_`
>> >and thus we'll either need to use something different or avoid conflicts.
>> >Perhaps `gitlib_` might be useful.
>>
>> Or `GITL_`
>
>It's more common to use lowercase, so I think we should do that.
>
>> >I should point out that _FILE_OFFSET_BITS=64 is effectively standard these days.
>> >Nobody wants their files limited to 2 GiB.
>>
>> This is not the default on some platforms. It is still required but can be put into a
>knob.
>
>True, but everyone compiles with it.  If you're a software vendor and your build only
>supports 2 GiB files, everyone's going to be mad.  We should be able to assume that
>vendors are prudent and reasonable people who will compile their binaries
>accordingly and not have to return an error if a requested file size is larger than 32
>bits.

I am not disagreeing that it "should be" standard. But it is not a compiler default, so still needs to be supplied when git is built. Removing the define will drop the limit to 2Gb on some platforms. I'm not sure where this is going.

>> >Sure.  If we need a file size type, it should be something like
>> >`int64_t` or `uint64_t` and not `off_t`. Not all platforms have explicit 64-bit APIs
>compiles. NonStop and S/390 do require explicit controls to make this work.
>>
>> This will result in compile warnings with some platforms with APIs
>> that do use off_t for arguments and results.

Where _FILE_OFFSET_BITS=64, off_t should be 64 bits. If we are saying that git is going to force 64-bit types for offsets, then the types should be explicitly different, like git_off_t rather than hard-binding them to int64_t. That would allow git_compat_util.h or others to force the type broadly instead of making assumptions about the size.
>
>There will need to be a cast.  Our APIs should use a standard 64-bit type and not
>expose the platform `off_t`.
>
>> >
>> >> - Limited Platform Compatibility. The external interfaces are able to
>> >>   assume that <stdint.h> and other C99 (or maybe even C11+)
>> >>   functionality exists and use it immediately, without weather balloons
>> >>   or #ifdefs. If some platform requires special handling, that platform
>> >>   isn't supported, at least initially.
>> >
>> >I think this is fine.  It's 2024.  We should assume that C11 (a
>> >13-year-old spec) is available and so is POSIX 1003.1-2008 (except
>> >for Windows).  We may want to have a nice `#ifdef __STDC__ < 200112L`
>> >(and a similar check for POSIX) to just produce an `#error` if the system is too old.
>>
>> I wish this were the case. I have 2 years before C11 is guaranteed to
>> be available on platforms that I maintain. Can we wait until mid-late
>> 2025 before doing this? I do not want to lose access to git.
>
>As mentioned in the original proposal, we don't have to support all platforms in the
>libified code.  The main Git binaries will continue to function and be supported, but
>the new libified code will rely on newer features.  You will still be able to have all the
>Git binaries and functionality, but if you want the new shared library to compile
>against, you'll have to furnish a newer compiler.

I am not sure this original assumption still holds, if git is now going to depend on the libified code, which seems to be the direction after recent discussion.

>As a side note, I don't think requiring porters to support a 13-year-old spec is
>unreasonable for new, independent parts of the codebase.  As a side note, if your
>platform supported GCC or LLVM, then this would be an easy goal to achieve.  I
>know it doesn't right now, but it might be an incentive to get it there.
>
>> This is not a particularly good assumption. Some architectures do not
>> support kernel-level threading. PUT threading is possible on some
>> platforms but attempts to port git to PUT require that all
>> dependencies also support PUT, which is not a good assumption. Most of
>> the dependencies that I have to work with actually do not support PUT
>> or any other form of threading.
>
>I don't believe this is true.  Every architecture which supports Linux or any other
>modern Unix has threading support.  You're compiling on
>ia64 and x86, which definitely do support threading and have had it on Linux and
>NetBSD for years (before they removed ia64 support).

Just be cause the chipset supports threading, which it does, does not mean that every platform has kernel-level threading available to applications. For SMP, sure, you can make that assumption, but for MPP architectures (few as they are), PUT has been the standard - it is available on NonStop and works correctly, but porting git to PUT on NonStop has been tried at least twice since I have been involved, with bad results each time - the failures may have been in the test suite, but if one does not have a test suite that supports what is being tested, one should not release it. Where I have an issue with requiring PUT as a minimum, is that dependencies for git, and the list is not small, do not generally support PUT, even if git does; rather make a similar assumption that hardware threading is sufficient - It is not and there is ample evidence of that. Cooperative threading (PUT) is much more complex to code than a kernel thread implementation, and our attempts at porting git to PUT on NonStop failed because of problems in dependencies and the git test suites. This is a much longer discussion, but I do not have confidence that everything git requires is properly reentrant and can handle PUT. They all "should", but that does not appear to be the case.

With that said, I can try spinning up an effort again to try the PUT build and dive deeper into this, but I do not relish having to debug someone else's thread code, at least not without some explicit explanation or help of the thread design. My $team will avoid trying to debug threaded code that does not work in favour of a rewrite, unless the problems are glaringly obvious. My $DAYJOB is already unhappy with the time I have spent debugging other project's thread problems - ones that would impact git PUT builds.

I am not saying it is impossible to do, but it is a very big ask, to justify not only git but all git dependencies to implement PUT threading correctly. If kernel threads are ever externalized on NonStop, the topic becomes much easier to handle. Again, ia64 goes to the curl because PUT is all we will ever get on ia64. Whenever this requirement is put in place, it locks git on NonStop to that version. The upshot is that I do not think libification has taken into account threading issues, so this may be moot.

>Threading is a reasonable thing to have on a modern operating system, and if we're
>adding new, independent functionality, that should be able to safely work with
>threading.  You can continue to use the binaries without threading, just not the new
>shared libraries.
>
>I'll add on that I'm not opposed to continuing to support NonStop in principle, but I
>_am_ opposed to having to continue to support really ancient versions of common
>standards[0].  POSIX 1003.1-2008 is about 16 years old; it could drive a car in much
>of North America.  I think it's the responsibility of porters to provide this
>functionality, and in general most open source OSes do this with little or no paid
>staff, so it should be reasonable to expect corporate-backed operating systems to
>do so.

I agree with that, but I have very little influence on what the NonStop team does internally. I am not on team.

>I would strongly recommend that NonStop start supporting modern compiler and
>POSIX standards, as well at least one of GCC or LLVM, and that you push very hard
>to make that happen.  It's not just Git that's pushing forward here; there's a lot of
>open source software that simply will not compile with the functionality you
>support, including the increasing amount of software written in Go or Rust, and it's
>unreasonable to expect people not to want to use standard functionality that's over
>a decade old.  If NonStop doesn't move in that direction, I anticipate we will
>eventually drop support for it sooner or later.

This is a great recommendation that will not happen any time soon. It is something the community has requested repeatedly but there are inside reasons for not doing this. My team spent years trying to port GCC without success (I cannot go into why, but it is non-trivial and will not happen). At least on the more modern NonStop x86, we do have C11, just not on ia64. With the latter going off support relatively soon, we could move to C11 safely in about 2 years. With that notice, it would give us time to adapt, but would absolutely kick git to the curb on ia64.

This does bring up the question of how much notice is needed (or at worst considered "considerate") to allow platforms to cope with fundamental architectural dependency changes in git.

>[0] This is not just NonStop; Microsoft's long-time refusal to implement
>C99 or a newer version of C really ground my gears and I basically gave up on
>supporting it for all personal projects as a result.  Fortunately, they've now come up
>to a modern standard.
>--
>brian m. carlson (they/them or he/him)
>Toronto, Ontario, CA

I have to say that I this thread is making me very nervous about the future of a supportable git on anything but Linux.

Re: [RFD] Libification proposal: separate internal and external interfaces

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> As mentioned in the original proposal, we don't have to support all
> platforms in the libified code.  The main Git binaries will continue to
> function and be supported, but the new libified code will rely on newer
> features.  You will still be able to have all the Git binaries and
> functionality, but if you want the new shared library to compile
> against, you'll have to furnish a newer compiler.

I thought one of the yardstick to gauge the success of this
"libification" effort, if not the purpose of this effort, is to
allow Git to be its first client.

I am not sure how it would supposed to work.  Unless you are giving
parallel implementations of "main Git binaries", one with the native
code and the other replaced the native code with thin wrappers
around the library calls, that is.

Puzzled...

Re: [RFD] Libification proposal: separate internal and external interfaces

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

On 2024-04-08 at 21:29:27, Junio C Hamano wrote:
> I thought one of the yardstick to gauge the success of this
> "libification" effort, if not the purpose of this effort, is to
> allow Git to be its first client.
> 
> I am not sure how it would supposed to work.  Unless you are giving
> parallel implementations of "main Git binaries", one with the native
> code and the other replaced the native code with thin wrappers
> around the library calls, that is.

I think the plan as proposed in the original file was to have an
internal and external library and to have the binaries use the internal
library.  However, perhaps I misunderstood the proposal, in which case
clarification on the part of the proposers would be helpful.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [RFD] Libification proposal: separate internal and external interfaces

[Phillip Wood]

On 08/04/2024 22:29, Junio C Hamano wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
>> As mentioned in the original proposal, we don't have to support all
>> platforms in the libified code.  The main Git binaries will continue to
>> function and be supported, but the new libified code will rely on newer
>> features.  You will still be able to have all the Git binaries and
>> functionality, but if you want the new shared library to compile
>> against, you'll have to furnish a newer compiler.
> 
> I thought one of the yardstick to gauge the success of this
> "libification" effort, if not the purpose of this effort, is to
> allow Git to be its first client.

Indeed, the last set of patches allow git to be built with the same 
library that external programs can use which I thought was very welcome. 
This proposal seems to be backing away from that.

We could have a single version of the library with a set of external 
headers that export a limited set of functions in a gitlib_ namespace 
and are wrapped internally with definitions like

static inline int foo(int x)
{
	return libgit_foo(x);
}

but that still leaves the problem of symbol visibility for the symbols 
that are consumed internally but not exposed in the external headers.

Best Wishes

Phillip

Re: [RFD] Libification proposal: separate internal and external interfaces

[Calvin Wan]

On Mon, Apr 8, 2024 at 5:35 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> On 2024-04-08 at 21:29:27, Junio C Hamano wrote:
> > I thought one of the yardstick to gauge the success of this
> > "libification" effort, if not the purpose of this effort, is to
> > allow Git to be its first client.
> >
> > I am not sure how it would supposed to work.  Unless you are giving
> > parallel implementations of "main Git binaries", one with the native
> > code and the other replaced the native code with thin wrappers
> > around the library calls, that is.
>
> I think the plan as proposed in the original file was to have an
> internal and external library and to have the binaries use the internal
> library.  However, perhaps I misunderstood the proposal, in which case
> clarification on the part of the proposers would be helpful.

We are still working on the specifics of how we would accomplish this
(part of the reason this is marked RFD). Thin wrappers make sense for
functions that should be slightly altered for external users (e.g.
anything with a die() should have their error codes bubbled up). On
the other hand, there are functions that are already suitable for
external users out of the box, so we could just provide a separate
header for those. But those functions may still need to be renamed
anyways to avoid symbol collision, so thin wrappers may end up being
the default going forward.

Re: [RFD] Libification proposal: separate internal and external interfaces

[Calvin Wan]

On Tue, Apr 9, 2024 at 2:40 AM Phillip Wood <phillip.wood123@gmail.com> wrote:
>
> Indeed, the last set of patches allow git to be built with the same
> library that external programs can use which I thought was very welcome.
> This proposal seems to be backing away from that.

One of the questions I was grappling with was, do we want to expose
all functions to external users? While it is debatable whether we
would like that exposure for some of the files in libstdgit, functions
in wrapper.c and usage.c seem like clear candidates for functions that
have no value being exposed externally while also making it more
difficult to maintain such a library in the future.

Re: [RFD] Libification proposal: separate internal and external interfaces

[Calvin Wan]

Thanks everyone for your initial comments on this discussion. I wanted
to provide some examples of how an internal/external interface could
look in practice -- originally I had intended to use git-std-lib v6 as
that example, but found that it fell short due to feedback that only
being able to expose a smaller subset of functions in that library would
be insufficient for users since they should have the same tools that we
have for building Git. In this reply, I have two examples of paths
forward that such an interface could look like for future libraries
(both methods would require a non-trivial amount of code change so this
seemed like a better idea than completely refactoring git-std-lib twice).

Part of the reason for wanting to expose a smaller subset of library
functions initially was to avoid having to expose functions that do
things a library function shouldn't, mainly those with die() calls. I
chose `strbuf_grow()` as the example function to be libified with an
internal/external interface since it has a die() call and in a library,
we would want to pass that error up rather than die()ing. I have two
ideas for how such an interface could look. For reference, this is how
`strbuf_grow()` currently looks:

void strbuf_grow(struct strbuf *sb, size_t extra)
{
	int new_buf = !sb->alloc;
	if (unsigned_add_overflows(extra, 1) ||
	    unsigned_add_overflows(sb->len, extra + 1))
		die("you want to use way too much memory");
	if (new_buf)
		sb->buf = NULL;
	ALLOC_GROW(sb->buf, sb->len + extra + 1, sb->alloc);
	if (new_buf)
		sb->buf[0] = '\0';
}

The first idea involves turning `strbuf_grow()` into a wrapper function
that invokes its equivalent library function, eg.
`libgit_strbuf_grow()`:

int libgit_strbuf_grow(struct strbuf *sb, size_t extra)
{
	int new_buf = !sb->alloc;
	if (unsigned_add_overflows(extra, 1) ||
	    unsigned_add_overflows(sb->len, extra + 1))
		return -1;
	if (new_buf)
		sb->buf = NULL;
	ALLOC_GROW(sb->buf, sb->len + extra + 1, sb->alloc);
	if (new_buf)
		sb->buf[0] = '\0';
        return 0;
}

void strbuf_grow(struct strbuf *sb, size_t extra)
{
        if (libgit_strbuf_grow(sb, extra))
                die("you want to use way too much memory");
}

(Note a context object could also be added as a parameter to
`libgit_strbuf_grow()` for error messages and possibly global variables.)

In this scenario, we would be exposing `libgit_strbuf_grow()` to
external consumers of the library, while not having to refactor internal
uses of `strbuf_grow()`. This method would reduce initial churn within
the codebase, however, we would want to eventually get rid of
`strbuf_grow()` and use `libgit_strbuf_grow()` internally as well. I
envision that it would be easier to remove die()'s all at once, from top
down, once libification has progressed further since top level callers
do not have to worry about refactoring any callers to accomodate passing
up error messages/codes. 

The shortfall of this approach is that we'd be carrying two different
functions for every library function until we are able to remove all of
them. It would also create additional toil for Git contributors to
figure out which version of the function should be used.

The second idea removes the need for two different functions by removing
the wrapper function and instead refactoring all callers of
`strbuf_grow()` (and subsequently callers of other library functions).

int libgit_strbuf_grow(struct strbuf *sb, size_t extra)
{
	int new_buf = !sb->alloc;
	if (unsigned_add_overflows(extra, 1) ||
	    unsigned_add_overflows(sb->len, extra + 1))
		return -1;
	if (new_buf)
		sb->buf = NULL;
	ALLOC_GROW(sb->buf, sb->len + extra + 1, sb->alloc);
	if (new_buf)
		sb->buf[0] = '\0';
        return 0;
}

void strbuf_grow_caller() {
	strbuf *sb;
	size_t extra;

	// if only success/failure is passed up
	if (libgit_strbuf_grow(sb, extra))
                die("you want to use way too much memory");
	
	// if context object is used
	if (libgit_strbuf_grow(sb, extra, context_obj))
                die(context_obj->error_msg);
	
	// if there are multiple error codes that can be passed up
	if (libgit_strbuf_grow(sb, extra) == -1)
                die("you want to use way too much memory");
	else if (libgit_strbuf_grow(sb, extra) == -2)
		die("some other error");
}

One shortcoming of this approach is the need to refactor all callers of
library functions, but that can be handled better and the churn made
more visible with a coccinelle patch. Another shortcoming is the need
for lengthier code blocks whenever calling a library function, however,
it could also be seen as a benefit since the caller would understand the
function can die(). These error messages would also ideally be passed up
as well in the future rather than die()ing.

While I tried to find a solution that avoided the shortcomings of both
approaches, I think that answer simply does not exist so the ideas above
are what I believe to be the least disruptive options. I'm wondering
which interface would be more suitable, and also open to hearing if
there are any other ideas!

Re: [RFD] Libification proposal: separate internal and external interfaces

[Junio C Hamano]

Calvin Wan <calvinwan@google.com> writes:

Thanks for writing this down.

> The first idea involves turning `strbuf_grow()` into a wrapper function
> that invokes its equivalent library function, eg.
> `libgit_strbuf_grow()`:
>
> int libgit_strbuf_grow(struct strbuf *sb, size_t extra)
> {
> 	int new_buf = !sb->alloc;
> 	if (unsigned_add_overflows(extra, 1) ||
> 	    unsigned_add_overflows(sb->len, extra + 1))
> 		return -1;
> 	if (new_buf)
> 		sb->buf = NULL;
> 	ALLOC_GROW(sb->buf, sb->len + extra + 1, sb->alloc);
> 	if (new_buf)
> 		sb->buf[0] = '\0';
>         return 0;
> }
>
> void strbuf_grow(struct strbuf *sb, size_t extra)
> {
>         if (libgit_strbuf_grow(sb, extra))
>                 die("you want to use way too much memory");
> }
>
> (Note a context object could also be added as a parameter to
> `libgit_strbuf_grow()` for error messages and possibly global variables.)
>
> In this scenario, we would be exposing `libgit_strbuf_grow()` to
> external consumers of the library, while not having to refactor internal
> uses of `strbuf_grow()`.

Yes, this is how I envision the things will go.  And it is a good
place to STOP for "git" the command line program everybody knows for
the past 20 years.  It is good that you mentioned the "context"
here, too.  As to the naming, I am OK with "libgit_".  Just wanted
to say that "libgit2" does not seem to use it as their prefix (and
they do not use "libgit2_" as their prefix, either) so if somebody
wants to bikeshed, they do not have to worry about them.

> This method would reduce initial churn within
> the codebase, however, we would want to eventually get rid of
> `strbuf_grow()` and use `libgit_strbuf_grow()` internally as well. 

The "however" and everything that follows wants justification.

I think after we pass the "Traditional API git offered are thin
wrappers around git-std-lib" point, the returns (the "clean-up"
value) will quickly diminish.  Even with diminished returns, there
may still be "clean-up" value left, but it would be simpler to
consider that as outside the scope of "libification" discussion.
Once the "Traditional API are thin wrappers" state is achieved, the
"libification" is done.

> The second idea removes the need for two different functions by removing
> the wrapper function and instead refactoring all callers of
> `strbuf_grow()` (and subsequently callers of other library functions).
> ...
> One shortcoming of this approach is the need to refactor all callers of
> library functions, but that can be handled better and the churn made

There is one huge downside you did not mention.

The appraoch makes "git the command line program" to commit to the
"errors must percolate all the way up to the top", and the
addditional effort to take us there after we have achieved the
libification goals has dubious trade-off.

I do not see why it bothers you, the libification person when he
wears git-std-lib hat, that "git the command line program" that is a
mere one of customers of git-std-lib happens to have a function
whose name is strbuf_grow() and uses libgit_strbuf_grow().  It
shouldn't bother you more than "git the command line program" having
a program called cmd_show() that uses many functions from libgit_
suite.  After we reach the "our implementation of traditional API
are all wrappers around git-std-lib API functions" [*] state, we may
decide to refactor further and may replace those "wrappers" with
direct calls, but that can happen far in the future, when the
libified result is rock solid.

If you try to do everything from the top to bottom at once, you
cannot easily keep the system (the combination of git-std-lib still
work-in-progress plus the client code that is "git the command line
program") as solid as the first approach.

Re: [RFD] Libification proposal: separate internal and external interfaces

[phillip.wood123]

Hi Calvin

On 22/04/2024 17:26, Calvin Wan wrote:
> The first idea involves turning `strbuf_grow()` into a wrapper function
> that invokes its equivalent library function, eg.
> `libgit_strbuf_grow()`:
> 
> int libgit_strbuf_grow(struct strbuf *sb, size_t extra)
> {
> 	int new_buf = !sb->alloc;
> 	if (unsigned_add_overflows(extra, 1) ||
> 	    unsigned_add_overflows(sb->len, extra + 1))
> 		return -1;
> 	if (new_buf)
> 		sb->buf = NULL;
> 	ALLOC_GROW(sb->buf, sb->len + extra + 1, sb->alloc);
> 	if (new_buf)
> 		sb->buf[0] = '\0';
>          return 0;
> }
> 
> void strbuf_grow(struct strbuf *sb, size_t extra)
> {
>          if (libgit_strbuf_grow(sb, extra))
>                  die("you want to use way too much memory");
> }
> 
> (Note a context object could also be added as a parameter to
> `libgit_strbuf_grow()` for error messages and possibly global variables.)

I agree with Junio that this is a good route forward and that we should 
not consider removing the wrappers to be part of the libification 
project. ALLOC_GROW() can die so I think we'd need a way to propagate an 
error message up to the wrapper even for relatively simple functions 
like this. For the allocation functions we'd either need to use a static 
string for the message which is not a good fit for other functions that 
want to dynamically format their messages (for example to include a path 
name into the error message), or pre-allocate the error messages.

> The shortfall of this approach is that we'd be carrying two different
> functions for every library function until we are able to remove all of
> them. It would also create additional toil for Git contributors to
> figure out which version of the function should be used.

Hopefully it should be clear if a function is part of the library or not 
so I don't think this should be a big problem.

Best Wishes

Phillip