CY23 - Status of service enumeration.

CY23 - Status of service enumeration.

[Carlos O'Donell]

My current notes on the status of service enumeration are as follows:

- glibc is fairly complete here:
  https://lore.kernel.org/gti-tac/804cee43-a120-3acf-a302-b1640f2285a9@redhat.com/

- gdb is fairly complete here:
  https://lore.kernel.org/gti-tac/b755774c-3912-2dac-957c-91f659c95119@efficios.com/

- binutils is not yet analyzed, but likely very similar to gdb.

- gcc is not yet analyzed.

I would like to see service enumeration completed this month so we can follow
up and present all the details at our GTI TAC meeting.

Joseph,

Have you had a chance to enumerate the gcc services?

Joel, Simon,

Is binutils going to be any different from gdb?

With the exception of the website?


-- 
Cheers,
Carlos.

Re: CY23 - Status of service enumeration.

[Simon Marchi]

> Joel, Simon,
> 
> Is binutils going to be any different from gdb?
I don't know about the binutils process.  Perhaps we can ask Nick
Clifton to review our list for GDB and complement it with any
information specific for binutils?

Simon

Re: CY23 - Status of service enumeration.

[Carlos O'Donell]

On 2/6/23 14:11, Simon Marchi wrote:
>> Joel, Simon,
>>
>> Is binutils going to be any different from gdb?
> I don't know about the binutils process.  Perhaps we can ask Nick
> Clifton to review our list for GDB and complement it with any
> information specific for binutils?

Simon,

Good idea!

Nick,

Would it be possible for you to look over the gdb list of services here:

https://lore.kernel.org/gti-tac/Y6aOvWPYmFXgyva5@adacore.com/T/#m9b3f915a4ba6cc5aab388ed28c02e5ee9f18e65c

... and comment on if there is anything that binutils does differently?

-- 
Cheers,
Carlos.

Re: CY23 - Status of service enumeration.

[Joseph Myers]

On Mon, 6 Feb 2023, Carlos O'Donell wrote:

> Joseph,
> 
> Have you had a chance to enumerate the gcc services?

No.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: CY23 - Status of service enumeration.

[Nick Clifton]

Hi Carlos,

>>> Is binutils going to be any different from gdb?
>> I don't know about the binutils process.  Perhaps we can ask Nick
>> Clifton to review our list for GDB and complement it with any
>> information specific for binutils?

It is basically the same.  We use the same git repository as GDB,
the same bugzilla and buildbot systems and the same website hosting.

There are a few discrepancies however:

   * The website contents are not generated by scripts, instead
     they are hand made.  The documentation parts of the binutils
     web site are created by building the docs locally and then
     uploading to the site.

   * We are not using patchwork at the moment.  Given the relatively
     light patch review load for the binutils, it may never be used.

   * We do not currently have daily snapshots of the binutils sources
     although these used to happen in the (long distant) past.

   * Not quite as many mailing lists as GDB, but basically similar.

Do you want a full list similar to the one produced by Simon and Joel
or is this discrepancy information sufficient ?

Cheers
   Nick

Re: CY23 - Status of service enumeration.

[Carlos O'Donell]

On 2/7/23 07:55, Nick Clifton wrote:
> Hi Carlos,
> 
>>>> Is binutils going to be any different from gdb?
>>> I don't know about the binutils process.  Perhaps we can ask Nick
>>> Clifton to review our list for GDB and complement it with any
>>> information specific for binutils?
> 
> It is basically the same.  We use the same git repository as GDB,
> the same bugzilla and buildbot systems and the same website hosting.
> 
> There are a few discrepancies however:
> 
>   * The website contents are not generated by scripts, instead
>     they are hand made.  The documentation parts of the binutils
>     web site are created by building the docs locally and then
>     uploading to the site.
> 
>   * We are not using patchwork at the moment.  Given the relatively
>     light patch review load for the binutils, it may never be used.
> 
>   * We do not currently have daily snapshots of the binutils sources
>     although these used to happen in the (long distant) past.
> 
>   * Not quite as many mailing lists as GDB, but basically similar.
> 
> Do you want a full list similar to the one produced by Simon and Joel
> or is this discrepancy information sufficient ?

It would be great to have a full list similar to the gdb one :-)

-- 
Cheers,
Carlos.

Re: CY23 - Status of service enumeration.

[Nick Clifton]

Hi Carlos,

> It would be great to have a full list similar to the gdb one :-)

Right - here we go.
--------------------------------------------------------------------
                Services used by the GNU Binutils project

* Git Repository
   * Need push access
   * Ability for project maintainers to request push access for new
     contributors (or to do it themselves)
   * Ability for users to create user-branches (e.g. users/simark/foo),
     which will be ignored by the notification-sending script.

* Git hooks

   * git-hooks (https://github.com/AdaCore/git-hooks) installed in
     the binutils-gdb.git repository.

   * binutils-gdb.git's git-hooks configuration pointing to the following
     scripts locally installed on the machine hosting the repository,
     to which the binutils & GDB admins need access (currently done
     via SSH access)

       - binutils-gdb.git/hooks-bin/email_to.py
         Small python script which determine which mailing-list(s) to use
         when sending commit email notifications. No special requirements
         other than a Python3 interpreter.

       - binutils-gdb.git/hooks-bin/commit-extra-checker.py

         Verifies that we do not have this issue:

         # With commits created using "git am" of patches sent via the gdb@ or
         # gdb-patches@ mailing list, it's possible that the author information
         # gets changed into "xxx via xxx@sourceware.org". Catch and reject those,
         # so the person doing the push can fix this before the push is allowed.

         No special requirements other than a Python3 interpreter.

	Note - although the comment refers to gdb mailing lists, this hook also
         catches sent to the binutils mailing lists.

       - /git/binutils-gdb.git/hooks-bin/style_checker

         An empty script at the moment (set because the git-hooks require
         it to be set).

       - /sourceware/infra/bin/email-to-bugzilla

         A perl script whose maintainership is unclear. Comments at
         the beginning of the script mention David Hampton <hampton@cisco.com>
         as contributor and Greg A. Woods <woods@web.net> as having "greatly
         hacked" it. Beyond that, don't know.

           . The goal is that when a commit is pushed to the git
             repository, we check for any mention of a Bugzilla bugs, and
             post a message to those bugs
           . It accesses the Bugzilla database directly to check if a given bug
             number exists (could easily be changed to use some HTTP request)
           . It posts to bugzilla by sending an email to a local email account
             (Joel: I think sourceware-bugzilla@localhost)

         Joel's note: If I understood the script correctly, then there
         must be a handler in the local MTA to catch those emails and
         send them to bugzilla for further processing. Don't know how
         that works, though.

       - binutils-gdb.git/hooks-bin/post-receive

         Bash wrapper scripts which essentially calls the following
         script: /usr/local/bin/irkerhook.py. I suspect the origin
         of this script is this github repository:
         https://gitlab.com/esr/irker

         If not, ISTR that it was Tom Tromey who got it installed,
         so we could ask him.

   * The git-hooks themselves send emails about commits to the binutils-cvs
     mailing list (each repository configures the destination, and for
     binutils-gdb, this this a "dynamic" configuration via the email_to.py
     script mentioned above allowing the destination to vary depending
     on which files got modified, whether they are binutils fils or GDB
     file, or both).

* Web-based navigation of the Git repository
   * Need to be able to browse the git repository online
   * https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git
   * If it exists, a more featureful git web frontends would be nice, one
     that allows searching files by name for instance.
   * URLs to a given commit in the Web UI should be easily computable
     using their SHA1; e.g.
         "https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=%(rev)s"
     We use those in the commit emails being sent by the git-hooks.

* Mailing lists
   * Those I know for sure are used / useful:
     * binutils@
     * binutils-cvs@ (now misnamed, but can be used to follow commits to the repo)
   * There is also:
     * bug-binutils@gnu.org
         which is effectively the same as the binutils@sourceware.org list,
         but hosted by the FSF.

* Mailing list web interface
   * The original one, which I find not so useful now that we have
     public-inbox: https://sourceware.org/pipermail/binutils/
   * However, old links need to keep working
   * public-inbox or something equivalent is necessary:
     * have a way to browse easily message threads that span multiple
       months / years
     * have a way to download raw messages, to apply patches locally

* Website
   * Handwritten pages in a couple of CVS repositories:
     - The gnu.org copy is maintained on https://savannah.gnu.org/;
     - The sourceware.org copy is maintained on sourceware.org itself,
       in /cvs/gdb/htdocs
     - It used to be that both versions were kept in sync by duplicating
       all the commits in both repositories. But this was recently changed
       in favor of installing redirects from the gnu.org website to
       the sourceware.org. This was the first step towards a possible
       transition of the web-pages to Git.
   * The documentation part of the website is created by building the
       relevant files locally and then uploading them to the website.
   * SFTP access to the machine hosting the website is used in order to
       upload new or edited files and to change symbolic links.

* Wiki
   * Currently, anyone with write access can give someone else write access.
   * At the very least, we need a way for project maintainers to request
     write access for a new member, or do it themselves

* Bug tracker (bugzilla)
   * Sends notifications of new bugs and comments to binutils@
   * Sends notifications of comments on a bug to users who are watching
     that bug
   * Accepts replies by email, both from users and from the
     email-to-bugzilla script.

* Release hosting
   * Releases, the release manager must be able to upload there
     * https://sourceware.org/pub/binutils/releases/
   * Pre-release snapshots are available here:
     * https://sourceware.org/pub/binutils/snapshots/
------------------------------------------------------------------------

Re: CY23 - Status of service enumeration.

[Carlos O'Donell]

On 2/7/23 09:41, Nick Clifton wrote:
> Hi Carlos,
> 
>> It would be great to have a full list similar to the gdb one :-)
> 
> Right - here we go.
> --------------------------------------------------------------------
>                Services used by the GNU Binutils project

Thank you Nick!

The list looks great.

Are there any cron scripts that binutils might be using directly?

Just trying to think of anything we might have missed.

> 
> * Git Repository
>   * Need push access
>   * Ability for project maintainers to request push access for new
>     contributors (or to do it themselves)
>   * Ability for users to create user-branches (e.g. users/simark/foo),
>     which will be ignored by the notification-sending script.
> 
> * Git hooks
> 
>   * git-hooks (https://github.com/AdaCore/git-hooks) installed in
>     the binutils-gdb.git repository.
> 
>   * binutils-gdb.git's git-hooks configuration pointing to the following
>     scripts locally installed on the machine hosting the repository,
>     to which the binutils & GDB admins need access (currently done
>     via SSH access)
> 
>       - binutils-gdb.git/hooks-bin/email_to.py
>         Small python script which determine which mailing-list(s) to use
>         when sending commit email notifications. No special requirements
>         other than a Python3 interpreter.
> 
>       - binutils-gdb.git/hooks-bin/commit-extra-checker.py
> 
>         Verifies that we do not have this issue:
> 
>         # With commits created using "git am" of patches sent via the gdb@ or
>         # gdb-patches@ mailing list, it's possible that the author information
>         # gets changed into "xxx via xxx@sourceware.org". Catch and reject those,
>         # so the person doing the push can fix this before the push is allowed.
> 
>         No special requirements other than a Python3 interpreter.
> 
>     Note - although the comment refers to gdb mailing lists, this hook also
>         catches sent to the binutils mailing lists.
> 
>       - /git/binutils-gdb.git/hooks-bin/style_checker
> 
>         An empty script at the moment (set because the git-hooks require
>         it to be set).
> 
>       - /sourceware/infra/bin/email-to-bugzilla
> 
>         A perl script whose maintainership is unclear. Comments at
>         the beginning of the script mention David Hampton <hampton@cisco.com>
>         as contributor and Greg A. Woods <woods@web.net> as having "greatly
>         hacked" it. Beyond that, don't know.
> 
>           . The goal is that when a commit is pushed to the git
>             repository, we check for any mention of a Bugzilla bugs, and
>             post a message to those bugs
>           . It accesses the Bugzilla database directly to check if a given bug
>             number exists (could easily be changed to use some HTTP request)
>           . It posts to bugzilla by sending an email to a local email account
>             (Joel: I think sourceware-bugzilla@localhost)
> 
>         Joel's note: If I understood the script correctly, then there
>         must be a handler in the local MTA to catch those emails and
>         send them to bugzilla for further processing. Don't know how
>         that works, though.
> 
>       - binutils-gdb.git/hooks-bin/post-receive
> 
>         Bash wrapper scripts which essentially calls the following
>         script: /usr/local/bin/irkerhook.py. I suspect the origin
>         of this script is this github repository:
>         https://gitlab.com/esr/irker
> 
>         If not, ISTR that it was Tom Tromey who got it installed,
>         so we could ask him.
> 
>   * The git-hooks themselves send emails about commits to the binutils-cvs
>     mailing list (each repository configures the destination, and for
>     binutils-gdb, this this a "dynamic" configuration via the email_to.py
>     script mentioned above allowing the destination to vary depending
>     on which files got modified, whether they are binutils fils or GDB
>     file, or both).
> 
> * Web-based navigation of the Git repository
>   * Need to be able to browse the git repository online
>   * https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git
>   * If it exists, a more featureful git web frontends would be nice, one
>     that allows searching files by name for instance.
>   * URLs to a given commit in the Web UI should be easily computable
>     using their SHA1; e.g.
>         "https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=%(rev)s"
>     We use those in the commit emails being sent by the git-hooks.
> 
> * Mailing lists
>   * Those I know for sure are used / useful:
>     * binutils@
>     * binutils-cvs@ (now misnamed, but can be used to follow commits to the repo)
>   * There is also:
>     * bug-binutils@gnu.org
>         which is effectively the same as the binutils@sourceware.org list,
>         but hosted by the FSF.
> 
> * Mailing list web interface
>   * The original one, which I find not so useful now that we have
>     public-inbox: https://sourceware.org/pipermail/binutils/
>   * However, old links need to keep working
>   * public-inbox or something equivalent is necessary:
>     * have a way to browse easily message threads that span multiple
>       months / years
>     * have a way to download raw messages, to apply patches locally
> 
> * Website
>   * Handwritten pages in a couple of CVS repositories:
>     - The gnu.org copy is maintained on https://savannah.gnu.org/;
>     - The sourceware.org copy is maintained on sourceware.org itself,
>       in /cvs/gdb/htdocs
>     - It used to be that both versions were kept in sync by duplicating
>       all the commits in both repositories. But this was recently changed
>       in favor of installing redirects from the gnu.org website to
>       the sourceware.org. This was the first step towards a possible
>       transition of the web-pages to Git.
>   * The documentation part of the website is created by building the
>       relevant files locally and then uploading them to the website.
>   * SFTP access to the machine hosting the website is used in order to
>       upload new or edited files and to change symbolic links.
> 
> * Wiki
>   * Currently, anyone with write access can give someone else write access.
>   * At the very least, we need a way for project maintainers to request
>     write access for a new member, or do it themselves
> 
> * Bug tracker (bugzilla)
>   * Sends notifications of new bugs and comments to binutils@
>   * Sends notifications of comments on a bug to users who are watching
>     that bug
>   * Accepts replies by email, both from users and from the
>     email-to-bugzilla script.
> 
> * Release hosting
>   * Releases, the release manager must be able to upload there
>     * https://sourceware.org/pub/binutils/releases/
>   * Pre-release snapshots are available here:
>     * https://sourceware.org/pub/binutils/snapshots/
> ------------------------------------------------------------------------
> 

-- 
Cheers,
Carlos.

Re: CY23 - Status of service enumeration.

[Nick Clifton]

Hi Carlos,

> Are there any cron scripts that binutils might be using directly?

None.  At least none that I could find on sourceware.

Cheers
   Nick