LF IT: Statement of Work for CTI: glibc project hosting

LF IT: Statement of Work for CTI: glibc project hosting

[Carlos O'Donell]

With permission from Konstantin (LF IT) I'm posting the final SOW
publicly to the list.

The cost breakdown is considered confidential information and redacted
from the public posting, but the summarized cost is provided for the
yearly ongoing costs.

Thank you everyone for contributing to the SOW!

--- 8< --- 8< ---

Statement of Work for CTI: glibc project hosting
========================================================================

1. Background
========================================================================

The Linux Foundation provides a comprehensive range of financial,
operational, promotional, technical and managerial services to open
source projects. With more than 20 years of experience driving the
growth of large-scale open source projects, The Linux Foundation is a
premier partner for collaborative projects.

The following statement of work (SOW) describes services The Linux
Foundation (LF) will provide to the Core Toolchain Infrastructure
Project (CTI) to host core project collaboration services for glibc.

2. Project Description
========================================================================

The Core Toolchain Infrastructure (CTI) Project’s mission is to support
the GNU Toolchain community with secure infrastructure and state of the
art services required to support the community’s development efforts to
be a trusted foundation in a secure supply chain.

The CTI project aspires to move forward the goal of creating a long-term
sustainable set of secure and state of the art services and
infrastructure for the GNU Toolchain and related packages.

Some of the project’s major goals include:
    • Secure and state of the art infrastructure.
    • Continuity planning for infrastructure, development, and governance.
    • Security policy planning.

Glibc is a GNU C standard library used near-universally across all Linux
distributions as the base library layer for the operating system.

3. Scope of Work
========================================================================

When choosing solutions and deploying services, the guiding principle is
to remain compliant with the GNU Ethical Repository Criteria, Grade B.

The CTI project requests that all service providers use FOSS to provide
the services that are used by the projects. The use of FOSS for the
services is a mandatory requirement.

1. Mailing lists for decentralized patch-based development

  a. LF will set up mailing list hosting under the
     lists.coretoolcain.dev domain, using the following technology
     stack:

       i. postfix
       ii. mlmmj
       iii. public-inbox

  b. For scalability and IP reputation reasons, LF will reuse its current
     mailing list platform already deployed for Linux kernel development
     (subspace.kernel.org).

  c. Individual glibc mailing lists will be created on request and do
     not require to be enumerated in this SOW.

  d. Mailing list archives will be available either via lore.kernel.org,
     or a dedicated coretoolchain.dev domain name, depending on project
     preferences and to be determined at a later date.

2. Bug tracking software (Bugzilla)

  a. LF will set up a bug tracking system using Bugzilla Harmony
     (https://github.com/bugzilla/harmony)

  b. This will be a new, standalone installation dedicated exclusively
     to glibc under the coretoolchain.dev domain.

  c. LF will work with the CTI leadership to properly configure the 
     software for use with the glibc project.

  d. LF will migrate the existing database using the database export 
     provided by the CTI team.

  e. Once installed and available, CTI members will manually remove any
     projects and components that are not related to glibc.

  f. LF will work with the glibc community to offer the tooling to
     integrate git repositories, mailing lists, and the bugtracker
     (e.g. using the bugspray project).

3. Git repository hosting (Gitolite)

  a. LF will set up the necessary glibc repositories reusing the
     existing gitolite.coretoolchain.dev service.

  b. LF will work with the CTI leadership to migrate existing
     repositories and grant the necessary permissions on a schedule to
     be agreed with the CTI project..

  c. LF will work with the glibc project community to analyze, port,
     and adapt the existing set of repository post-commit hooks to
     ensure that they are functioning after the migration, or are
     replaced with suitable alternatives if existing implementations
     cannot be used directly due to security or other considerations.

  d. LF will set up mirroring and replication using grokmirror to offer
     multiple redundant sites for git repository access.

4. Documentation repository and website (Sphinx)

  a. LF will work with the glibc community to help migrate the wiki site
     from MoinMoin Wiki to a set of restructured-text documents
     (e.g. using pandoc)

  b. LF will provide a publishing framework to automatically build and
     deploy documentation to the dedicated docs website (exact domain
     name to be established)

5. Static website (Sphinx)

  a. LF will provide hosting for a separate static website for glibc
     (exact domain name to be established)

6. Patch tracking services (Patchwork)

  a. LF will deploy the latest version of Patchwork patch tracking
     software under the coretoolchain.dev domain. It will be a
     standalone instance, dedicated to the glibc project.

  b. LF will migrate the existing database using the database export
     provided.

  c. Once patchwork is configured, CTI members will manually remove
     any projects not associated with glibc.

  d. LF will set up automation and integration services between mailing
     lists, patchwork, and git repositories (using git-patchwork-bot).

  e. LF will work with the glibc community to set up projects and
     access as appropriate.

7. Video conferencing (BigBlueButton)

  a. LF will provide video conferencing to the project using
     BigBlueButton (BBB), either with a 3rd-party provider or as a
     fully self-hosted service.

  b. LF will work with the glibc project to set up conference rooms and
     access as required.

3.1. Run Operations Support
========================================================================

Both during the deployment and the ongoing support stages, LF IT will
provide the following services:

1. Email-based help desk support.

2. Monitoring, alerting, and off-site backups. These services are
   included in the cost of ongoing support and maintenance.

3. OS and software updates for security and new features. Occasional
   downtime will be required in order to restart affected services –
   communicated and coordinated ahead of time, unless addressing a
   critical issue.
   
4. Mailing list anti-abuse services. Core Projects IT team employs
   multiple self-hosted and third-party services to identify and block
   spam and malicious content that may be posted to the mailing lists
   or public sites.

3.2 Platform Fault Tolerance and Redundancy
========================================================================

The infrastructure will be deployed with a cloud vendor with multiple
points of redundancy built-in:

    • Multiple hardware nodes with live automatic failover

    • Primary/Secondary database design with automatic replication and
      failover

    • Redundant virtual IPs with failover and multipath networking

    • Public git repositories mirrored to multiple worldwide frontends
      with geo-ip based routing for requests

    • All critical data is encrypted and backed up to off-site locations
      with high levels of data retention and redundancy

While this design should provide a lot of built-in redundancy, it will
be deployed within a single public cloud environment, so a
provider-level outage would still impact those services that are not
mirrored out to multiple public frontends.

3.3 Service Level Promise
========================================================================

The Linux Foundation IT offers a “promise of our best effort” for all IT
support:

    • LF IT provides 24/7 monitoring and engineering support to handle
      all service outages or high-risk security events. On-call staff
      will do initial evaluation of impact and escalate to other members
      of LF IT as necessary.

    • Members of the Core Projects IT team will respond to incoming
      project requests, aiming to respond within 1 standard North 
      America business day. Should the primary engineer be away, there
      will be a back-up staff member familiar with the project who can
      provide support in their absence.

4. Project Cost
========================================================================

LF IT Engineering and Support Costs:

Yearly running cost (incl. estimated infra costs):
	~$276,000 USD

4.1 Hosting Cost Estimates (pass-through)
========================================================================

The costs are high-watermark estimates for the glibc projects as billed
directly by the hosting provider. Depending on the final deployment,
they may be lower; justification will be provided by the LF IT team for
any costs exceeding the estimated high-watermark amount.

5. Terms and Conditions
========================================================================

The Linux Foundation and CTI shall make available to each other a 
designated representative with each authorization to make binding 
decisions regarding the obligations covered by this Agreement.

The Linux Foundation ensures that all services will be performed in a 
professional manner using qualified personnel. The Linux Foundation
reserves the right to reassign personnel to achieve the best allocation
of resources.

The Linux Foundation reserves the right to engage qualified independent
contractors as necessary to complete any work covered by this Agreement.

Nothing in this Agreement may be construed as creating an
employer/employee relationship between any of the Linux Foundation’s
personnel and. All work performed by the Linux Foundation personnel
under this agreement is done on behalf of the Linux Foundation and may
not be considered to be work made for hire.

6. Communication
========================================================================

Communication is essential to the success of every project. Meeting
attendance and regular correspondence by phone and email help keep a
project on track. The Linux Foundation teams welcome feedback at any time.

7. Agreement
========================================================================

This Statement of Work governs services provided by the Linux Foundation
to CTI. It lays out the mutual responsibilities and processes necessary
to complete the services described herein with the best result. 

This agreement covers the period through December 31, 2025 and is
renewable on an annual basis.

The undersigned agree to respect this Statement of Work, for the mutual
benefit of both parties.

--- 8< --- 8< ---

-- 
Cheers,
Carlos.