Kernel ABI Monitoring and toolchain support

Kernel ABI Monitoring and toolchain support

[Nick Desaulniers]

Hi Matthias and Giuliano,
Jose and I are currently in the planning process to put together a
Kernel+Toolchain microconference track at Linux Plumbers Conference
this year (Sept 12-14) in Dublin, Ireland.

Would you all be interested in a leading session on Kernel ABI
Monitoring and perhaps what toolchain vendors can do to help, as part
of our MC (which itself is currently still in planning phase).
-- 
Thanks,
~Nick Desaulniers

Re: Kernel ABI Monitoring and toolchain support

[Jose E. Marchesi]

[Adding Dodji Seketeli, libabigail maintainer,
 and Guillermo Martinez, who works in the CTF support for libabigail
 including kernel support.]

> Hi Matthias and Giuliano,
> Jose and I are currently in the planning process to put together a
> Kernel+Toolchain microconference track at Linux Plumbers Conference
> this year (Sept 12-14) in Dublin, Ireland.
>
> Would you all be interested in a leading session on Kernel ABI
> Monitoring and perhaps what toolchain vendors can do to help, as part
> of our MC (which itself is currently still in planning phase).

Re: Kernel ABI Monitoring and toolchain support

[Guillermo E. Martinez]

Hello guys,

On our side we are working on:

  * Gives the support  to extract the ABI for Linux kernel image 
  and Linux  kernel modules using the CFT backed in with
  libabigail tools.
  * Add support to compare packages with CTF debug format by using
  abipkgdiff and other related libabigail tools.

   * https://sourceware.org/pipermail/libabigail/2022q2/004340.html
   * https://sourceware.org/pipermail/libabigail/2022q2/004345.html

If you think that it could be a point to talk as part of your MC,  I can for sure
provides more details about it.

Thanks for your comments,
Guillermo

On Wednesday, April 20, 2022 12:34:59 PM CDT Giuliano Procida wrote:
> Sorry for the late reply, I've been on leave and Mattias is still on leave.
> 
> That sounds quite interesting.
> 
> Matthias and I could certainly talk about what we've been busy with and
> what are the main pain points for Android (not just the kernel).
> 
> On what more would we like from the toolchain... it falls into at least two
> buckets:
> 
> 1. make the possible easier - stricter conformance to DWARF standards, full
> DWARF 5 support, getting things like ARM64 ABI extensions (e.g., for
> HWASAN) into things like elfutils at the same time as the compile-link
> toolchain, more consistency between Clang and GCC debug info for the same
> sources, the same for Clang and Clang with full LTO
> 2. make the impossible possible - extending ABI monitoring coverage beyond
> just architecture, symbols and types / dealing with header constants,
> macros and more
> 
> Giuliano.
> 
> 
> On Wed, 20 Apr 2022 at 17:02, Jose E. Marchesi <jose.marchesi@oracle.com>
> wrote:
> 
> >
> > [Adding Dodji Seketeli, libabigail maintainer,
> >  and Guillermo Martinez, who works in the CTF support for libabigail
> >  including kernel support.]
> >
> > > Hi Matthias and Giuliano,
> > > Jose and I are currently in the planning process to put together a
> > > Kernel+Toolchain microconference track at Linux Plumbers Conference
> > > this year (Sept 12-14) in Dublin, Ireland.
> > >
> > > Would you all be interested in a leading session on Kernel ABI
> > > Monitoring and perhaps what toolchain vendors can do to help, as part
> > > of our MC (which itself is currently still in planning phase).
> >
> 

Re: Kernel ABI Monitoring and toolchain support

[Matthias Männich]

Hi!

That is a very late response, sorry about that.

In the Android MC we will have a talk about the latest news monitoring
Android Kernel ABIs already, but I definitely see value in a more
generic session in the Kernel+Toolchain MC. Is that schedule already
finalized or is there still a slot available and interest from your
side?

In particular, I would be interested in discussing ways to standardize
ABI and type information in a way that it can be embedded into
binaries in a less ambiguous way. In other words, what can we do to
not rely entirely on intermediate formats like CTF or DWARF to make
sense of an ABI? Maybe CTF is already a good starting point, yet some
additions are needed (e.g. other language features like for C++)?

Cheers,
Matthias


On Sat, May 7, 2022 at 3:15 AM Guillermo E. Martinez
<guillermo.e.martinez@oracle.com> wrote:
>
> Hello guys,
>
> On our side we are working on:
>
>   * Gives the support  to extract the ABI for Linux kernel image
>   and Linux  kernel modules using the CFT backed in with
>   libabigail tools.
>   * Add support to compare packages with CTF debug format by using
>   abipkgdiff and other related libabigail tools.
>
>    * https://sourceware.org/pipermail/libabigail/2022q2/004340.html
>    * https://sourceware.org/pipermail/libabigail/2022q2/004345.html
>
> If you think that it could be a point to talk as part of your MC,  I can for sure
> provides more details about it.
>
> Thanks for your comments,
> Guillermo
>
> On Wednesday, April 20, 2022 12:34:59 PM CDT Giuliano Procida wrote:
> > Sorry for the late reply, I've been on leave and Mattias is still on leave.
> >
> > That sounds quite interesting.
> >
> > Matthias and I could certainly talk about what we've been busy with and
> > what are the main pain points for Android (not just the kernel).
> >
> > On what more would we like from the toolchain... it falls into at least two
> > buckets:
> >
> > 1. make the possible easier - stricter conformance to DWARF standards, full
> > DWARF 5 support, getting things like ARM64 ABI extensions (e.g., for
> > HWASAN) into things like elfutils at the same time as the compile-link
> > toolchain, more consistency between Clang and GCC debug info for the same
> > sources, the same for Clang and Clang with full LTO
> > 2. make the impossible possible - extending ABI monitoring coverage beyond
> > just architecture, symbols and types / dealing with header constants,
> > macros and more
> >
> > Giuliano.
> >
> >
> > On Wed, 20 Apr 2022 at 17:02, Jose E. Marchesi <jose.marchesi@oracle.com>
> > wrote:
> >
> > >
> > > [Adding Dodji Seketeli, libabigail maintainer,
> > >  and Guillermo Martinez, who works in the CTF support for libabigail
> > >  including kernel support.]
> > >
> > > > Hi Matthias and Giuliano,
> > > > Jose and I are currently in the planning process to put together a
> > > > Kernel+Toolchain microconference track at Linux Plumbers Conference
> > > > this year (Sept 12-14) in Dublin, Ireland.
> > > >
> > > > Would you all be interested in a leading session on Kernel ABI
> > > > Monitoring and perhaps what toolchain vendors can do to help, as part
> > > > of our MC (which itself is currently still in planning phase).
> > >
> >
>
>
>
>

Re: Kernel ABI Monitoring and toolchain support

[Nick Desaulniers]

On Mon, Jun 20, 2022 at 6:42 AM Matthias Männich <maennich@google.com> wrote:
>
> Hi!
>
> That is a very late response, sorry about that.
>
> In the Android MC we will have a talk about the latest news monitoring
> Android Kernel ABIs already, but I definitely see value in a more
> generic session in the Kernel+Toolchain MC. Is that schedule already
> finalized or is there still a slot available and interest from your
> side?

We haven't finalized the schedule yet.  Matthias, Guillermo, Giuliano
can one of you perhaps fill out a proposal then for the ideas
discussed in this thread on https://lpc.events/event/16/abstracts/
with the "Track" set to "Toolchains Track?"

>
> In particular, I would be interested in discussing ways to standardize
> ABI and type information in a way that it can be embedded into
> binaries in a less ambiguous way. In other words, what can we do to
> not rely entirely on intermediate formats like CTF or DWARF to make
> sense of an ABI? Maybe CTF is already a good starting point, yet some
> additions are needed (e.g. other language features like for C++)?
>
> Cheers,
> Matthias
>
>
> On Sat, May 7, 2022 at 3:15 AM Guillermo E. Martinez
> <guillermo.e.martinez@oracle.com> wrote:
> >
> > Hello guys,
> >
> > On our side we are working on:
> >
> >   * Gives the support  to extract the ABI for Linux kernel image
> >   and Linux  kernel modules using the CFT backed in with
> >   libabigail tools.
> >   * Add support to compare packages with CTF debug format by using
> >   abipkgdiff and other related libabigail tools.
> >
> >    * https://sourceware.org/pipermail/libabigail/2022q2/004340.html
> >    * https://sourceware.org/pipermail/libabigail/2022q2/004345.html
> >
> > If you think that it could be a point to talk as part of your MC,  I can for sure
> > provides more details about it.
> >
> > Thanks for your comments,
> > Guillermo
> >
> > On Wednesday, April 20, 2022 12:34:59 PM CDT Giuliano Procida wrote:
> > > Sorry for the late reply, I've been on leave and Mattias is still on leave.
> > >
> > > That sounds quite interesting.
> > >
> > > Matthias and I could certainly talk about what we've been busy with and
> > > what are the main pain points for Android (not just the kernel).
> > >
> > > On what more would we like from the toolchain... it falls into at least two
> > > buckets:
> > >
> > > 1. make the possible easier - stricter conformance to DWARF standards, full
> > > DWARF 5 support, getting things like ARM64 ABI extensions (e.g., for
> > > HWASAN) into things like elfutils at the same time as the compile-link
> > > toolchain, more consistency between Clang and GCC debug info for the same
> > > sources, the same for Clang and Clang with full LTO
> > > 2. make the impossible possible - extending ABI monitoring coverage beyond
> > > just architecture, symbols and types / dealing with header constants,
> > > macros and more
> > >
> > > Giuliano.
> > >
> > >
> > > On Wed, 20 Apr 2022 at 17:02, Jose E. Marchesi <jose.marchesi@oracle.com>
> > > wrote:
> > >
> > > >
> > > > [Adding Dodji Seketeli, libabigail maintainer,
> > > >  and Guillermo Martinez, who works in the CTF support for libabigail
> > > >  including kernel support.]
> > > >
> > > > > Hi Matthias and Giuliano,
> > > > > Jose and I are currently in the planning process to put together a
> > > > > Kernel+Toolchain microconference track at Linux Plumbers Conference
> > > > > this year (Sept 12-14) in Dublin, Ireland.
> > > > >
> > > > > Would you all be interested in a leading session on Kernel ABI
> > > > > Monitoring and perhaps what toolchain vendors can do to help, as part
> > > > > of our MC (which itself is currently still in planning phase).
> > > >
> > >
> >
> >
> >
> >



-- 
Thanks,
~Nick Desaulniers

Re: Kernel ABI Monitoring and toolchain support

[Elena Zannoni]

On 6/21/22 15:19, Nick Desaulniers wrote:
> On Mon, Jun 20, 2022 at 6:42 AM Matthias Männich <maennich@google.com> wrote:
>> Hi!
>>
>> That is a very late response, sorry about that.
>>
>> In the Android MC we will have a talk about the latest news monitoring
>> Android Kernel ABIs already, but I definitely see value in a more
>> generic session in the Kernel+Toolchain MC. Is that schedule already
>> finalized or is there still a slot available and interest from your
>> side?
> We haven't finalized the schedule yet.  Matthias, Guillermo, Giuliano
> can one of you perhaps fill out a proposal then for the ideas
> discussed in this thread on https://lpc.events/event/16/abstracts/
> with the "Track" set to "Toolchains Track?"
>

Hi,

actually Guillermo was already working on that, will send something by the end of the day

elena

>> In particular, I would be interested in discussing ways to standardize
>> ABI and type information in a way that it can be embedded into
>> binaries in a less ambiguous way. In other words, what can we do to
>> not rely entirely on intermediate formats like CTF or DWARF to make
>> sense of an ABI? Maybe CTF is already a good starting point, yet some
>> additions are needed (e.g. other language features like for C++)?
>>
>> Cheers,
>> Matthias
>>
>>
>> On Sat, May 7, 2022 at 3:15 AM Guillermo E. Martinez
>> <guillermo.e.martinez@oracle.com> wrote:
>>> Hello guys,
>>>
>>> On our side we are working on:
>>>
>>>   * Gives the support  to extract the ABI for Linux kernel image
>>>   and Linux  kernel modules using the CFT backed in with
>>>   libabigail tools.
>>>   * Add support to compare packages with CTF debug format by using
>>>   abipkgdiff and other related libabigail tools.
>>>
>>>    * https://sourceware.org/pipermail/libabigail/2022q2/004340.html
>>>    * https://sourceware.org/pipermail/libabigail/2022q2/004345.html
>>>
>>> If you think that it could be a point to talk as part of your MC,  I can for sure
>>> provides more details about it.
>>>
>>> Thanks for your comments,
>>> Guillermo
>>>
>>> On Wednesday, April 20, 2022 12:34:59 PM CDT Giuliano Procida wrote:
>>>> Sorry for the late reply, I've been on leave and Mattias is still on leave.
>>>>
>>>> That sounds quite interesting.
>>>>
>>>> Matthias and I could certainly talk about what we've been busy with and
>>>> what are the main pain points for Android (not just the kernel).
>>>>
>>>> On what more would we like from the toolchain... it falls into at least two
>>>> buckets:
>>>>
>>>> 1. make the possible easier - stricter conformance to DWARF standards, full
>>>> DWARF 5 support, getting things like ARM64 ABI extensions (e.g., for
>>>> HWASAN) into things like elfutils at the same time as the compile-link
>>>> toolchain, more consistency between Clang and GCC debug info for the same
>>>> sources, the same for Clang and Clang with full LTO
>>>> 2. make the impossible possible - extending ABI monitoring coverage beyond
>>>> just architecture, symbols and types / dealing with header constants,
>>>> macros and more
>>>>
>>>> Giuliano.
>>>>
>>>>
>>>> On Wed, 20 Apr 2022 at 17:02, Jose E. Marchesi <jose.marchesi@oracle.com>
>>>> wrote:
>>>>
>>>>> [Adding Dodji Seketeli, libabigail maintainer,
>>>>>  and Guillermo Martinez, who works in the CTF support for libabigail
>>>>>  including kernel support.]
>>>>>
>>>>>> Hi Matthias and Giuliano,
>>>>>> Jose and I are currently in the planning process to put together a
>>>>>> Kernel+Toolchain microconference track at Linux Plumbers Conference
>>>>>> this year (Sept 12-14) in Dublin, Ireland.
>>>>>>
>>>>>> Would you all be interested in a leading session on Kernel ABI
>>>>>> Monitoring and perhaps what toolchain vendors can do to help, as part
>>>>>> of our MC (which itself is currently still in planning phase).
>>>
>>>
>>>
>
>

Re: Kernel ABI Monitoring and toolchain support

[Guillermo E. Martinez]

On Tuesday, June 21, 2022 4:27:18 PM CDT Elena Zannoni wrote:

Hello guys, 

The paragraphs bellow summarize the proposal  for Kernel ABI analysis
talk in LPC, these ideas comes from the thread mail bellow.

Please let me know your comments to submit it
to https://lpc.events/event/16/abstracts/

guillermo

---
Kernel ABI Monitoring and toolchain support

The new CTF(Compact C Type Format) supported  in  libabigail  is  able
to extract a  corpus  representation  for  the  debug  information  in
Kernel binary and its modules, i.e, entire Kernel  release  (kernel  +
modules).  Using CTF reader improvements the time to extract and build
the corpus compared with DWARF reader,  for  example,  extracting  ABI
information  from  the  Linux  kernel  takes  up  to  ~4.5times   less
time, this was done using a Kernel  compiled  by  GCC,  nowadays  LLVM
doesn't support binaries generation with CTF debug info, would be nice
to have this.

But what about of the modules inserted  (loaded)  at  runtime  in  the
Kernel image?.  To make the comparison it uses kABI  scripts  this  is
useful among other things to load modules with compatible  kABI,  this
mechanism allows modules to be used with a  different  kernel  version
that of the kernel for which it was built.  So what of using a  single
notion  of   ABI   (libabigail)   also   for   the   modules   loader?

Since we add support for  CTF  in  libabigail,  is  needed  the  patch
for building the Kernel  with  CTF  enabled  in  the  Kernel  upstream
configuration.  Also some GCC  attributes  that  affect  the  ABI  and
are used by kernel hackers like noreturn,  interrupt,  etc.   are  not
represented in DWARF/CTF debug  format  and  therefore  they  are  not
present in the corpus.

A stricter conformance to DWARF standards would be nice, full DWARF  5
support, getting things like ARM64 ABI extensions (e.g.,  for  HWASAN)
into things like  elfutils  at  the  same  time  as  the  compile-link
toolchain, more consistency between Clang and GCC debug info  for  the
same sources, the same for Clang and Clang  with  full  LTO.   And  an
extending ABI monitoring coverage beyond  just  architecture,  symbols
and  types  /  dealing  with  header  constants,   macros   and   more

The  interest  in  discussing  ways  to  standardize  ABI   and   type
information in a way that it can be embedded into binaries in  a  less
ambiguous way.  In other words, what can we do to not rely entirely on
intermediate formats like CTF or DWARF to make sense of an ABI?  Maybe
CTF is already a good starting point, yet some  additions  are  needed
(e.g. other language features like for C++)?

> On 6/21/22 15:19, Nick Desaulniers wrote:
> > On Mon, Jun 20, 2022 at 6:42 AM Matthias Männich <maennich@google.com> wrote:
> >> Hi!
> >>
> >> That is a very late response, sorry about that.
> >>
> >> In the Android MC we will have a talk about the latest news monitoring
> >> Android Kernel ABIs already, but I definitely see value in a more
> >> generic session in the Kernel+Toolchain MC. Is that schedule already
> >> finalized or is there still a slot available and interest from your
> >> side?
> > We haven't finalized the schedule yet.  Matthias, Guillermo, Giuliano
> > can one of you perhaps fill out a proposal then for the ideas
> > discussed in this thread on https://lpc.events/event/16/abstracts/
> > with the "Track" set to "Toolchains Track?"
> >
> 
> Hi,
> 
> actually Guillermo was already working on that, will send something by the end of the day
> 
> elena
> 
> >> In particular, I would be interested in discussing ways to standardize
> >> ABI and type information in a way that it can be embedded into
> >> binaries in a less ambiguous way. In other words, what can we do to
> >> not rely entirely on intermediate formats like CTF or DWARF to make
> >> sense of an ABI? Maybe CTF is already a good starting point, yet some
> >> additions are needed (e.g. other language features like for C++)?
> >>
> >> Cheers,
> >> Matthias
> >>
> >>
> >> On Sat, May 7, 2022 at 3:15 AM Guillermo E. Martinez
> >> <guillermo.e.martinez@oracle.com> wrote:
> >>> Hello guys,
> >>>
> >>> On our side we are working on:
> >>>
> >>>   * Gives the support  to extract the ABI for Linux kernel image
> >>>   and Linux  kernel modules using the CFT backed in with
> >>>   libabigail tools.
> >>>   * Add support to compare packages with CTF debug format by using
> >>>   abipkgdiff and other related libabigail tools.
> >>>
> >>>    * https://sourceware.org/pipermail/libabigail/2022q2/004340.html
> >>>    * https://sourceware.org/pipermail/libabigail/2022q2/004345.html
> >>>
> >>> If you think that it could be a point to talk as part of your MC,  I can for sure
> >>> provides more details about it.
> >>>
> >>> Thanks for your comments,
> >>> Guillermo
> >>>
> >>> On Wednesday, April 20, 2022 12:34:59 PM CDT Giuliano Procida wrote:
> >>>> Sorry for the late reply, I've been on leave and Mattias is still on leave.
> >>>>
> >>>> That sounds quite interesting.
> >>>>
> >>>> Matthias and I could certainly talk about what we've been busy with and
> >>>> what are the main pain points for Android (not just the kernel).
> >>>>
> >>>> On what more would we like from the toolchain... it falls into at least two
> >>>> buckets:
> >>>>
> >>>> 1. make the possible easier - stricter conformance to DWARF standards, full
> >>>> DWARF 5 support, getting things like ARM64 ABI extensions (e.g., for
> >>>> HWASAN) into things like elfutils at the same time as the compile-link
> >>>> toolchain, more consistency between Clang and GCC debug info for the same
> >>>> sources, the same for Clang and Clang with full LTO
> >>>> 2. make the impossible possible - extending ABI monitoring coverage beyond
> >>>> just architecture, symbols and types / dealing with header constants,
> >>>> macros and more
> >>>>
> >>>> Giuliano.
> >>>>
> >>>>
> >>>> On Wed, 20 Apr 2022 at 17:02, Jose E. Marchesi <jose.marchesi@oracle.com>
> >>>> wrote:
> >>>>
> >>>>> [Adding Dodji Seketeli, libabigail maintainer,
> >>>>>  and Guillermo Martinez, who works in the CTF support for libabigail
> >>>>>  including kernel support.]
> >>>>>
> >>>>>> Hi Matthias and Giuliano,
> >>>>>> Jose and I are currently in the planning process to put together a
> >>>>>> Kernel+Toolchain microconference track at Linux Plumbers Conference
> >>>>>> this year (Sept 12-14) in Dublin, Ireland.
> >>>>>>
> >>>>>> Would you all be interested in a leading session on Kernel ABI
> >>>>>> Monitoring and perhaps what toolchain vendors can do to help, as part
> >>>>>> of our MC (which itself is currently still in planning phase).
> >>>
> >>>
> >>>
> >
> >
> 
> 

Re: Kernel ABI Monitoring and toolchain support

[Guillermo E. Martinez]

Hello,

Any comment?

Thanks,
guillermo

> Hello guys, 
> 
> The paragraphs bellow summarize the proposal  for Kernel ABI analysis
> talk in LPC, these ideas comes from the thread mail bellow.
> 
> Please let me know your comments to submit it
> to https://lpc.events/event/16/abstracts/
> 
> guillermo
> 
> ---
> Kernel ABI Monitoring and toolchain support
> 
> The new CTF(Compact C Type Format) supported  in  libabigail  is  able
> to extract a  corpus  representation  for  the  debug  information  in
> Kernel binary and its modules, i.e, entire Kernel  release  (kernel  +
> modules).  Using CTF reader improvements the time to extract and build
> the corpus compared with DWARF reader,  for  example,  extracting  ABI
> information  from  the  Linux  kernel  takes  up  to  ~4.5times   less
> time, this was done using a Kernel  compiled  by  GCC,  nowadays  LLVM
> doesn't support binaries generation with CTF debug info, would be nice
> to have this.
> 
> But what about of the modules inserted  (loaded)  at  runtime  in  the
> Kernel image?.  To make the comparison it uses kABI  scripts  this  is
> useful among other things to load modules with compatible  kABI,  this
> mechanism allows modules to be used with a  different  kernel  version
> that of the kernel for which it was built.  So what of using a  single
> notion  of   ABI   (libabigail)   also   for   the   modules   loader?
> 
> Since we add support for  CTF  in  libabigail,  is  needed  the  patch
> for building the Kernel  with  CTF  enabled  in  the  Kernel  upstream
> configuration.  Also some GCC  attributes  that  affect  the  ABI  and
> are used by kernel hackers like noreturn,  interrupt,  etc.   are  not
> represented in DWARF/CTF debug  format  and  therefore  they  are  not
> present in the corpus.
> 
> A stricter conformance to DWARF standards would be nice, full DWARF  5
> support, getting things like ARM64 ABI extensions (e.g.,  for  HWASAN)
> into things like  elfutils  at  the  same  time  as  the  compile-link
> toolchain, more consistency between Clang and GCC debug info  for  the
> same sources, the same for Clang and Clang  with  full  LTO.   And  an
> extending ABI monitoring coverage beyond  just  architecture,  symbols
> and  types  /  dealing  with  header  constants,   macros   and   more
> 
> The  interest  in  discussing  ways  to  standardize  ABI   and   type
> information in a way that it can be embedded into binaries in  a  less
> ambiguous way.  In other words, what can we do to not rely entirely on
> intermediate formats like CTF or DWARF to make sense of an ABI?  Maybe
> CTF is already a good starting point, yet some  additions  are  needed
> (e.g. other language features like for C++)?
> 
> > On 6/21/22 15:19, Nick Desaulniers wrote:
> > > On Mon, Jun 20, 2022 at 6:42 AM Matthias Männich <maennich@google.com> wrote:
> > >> Hi!
> > >>
> > >> That is a very late response, sorry about that.
> > >>
> > >> In the Android MC we will have a talk about the latest news monitoring
> > >> Android Kernel ABIs already, but I definitely see value in a more
> > >> generic session in the Kernel+Toolchain MC. Is that schedule already
> > >> finalized or is there still a slot available and interest from your
> > >> side?
> > > We haven't finalized the schedule yet.  Matthias, Guillermo, Giuliano
> > > can one of you perhaps fill out a proposal then for the ideas
> > > discussed in this thread on https://lpc.events/event/16/abstracts/
> > > with the "Track" set to "Toolchains Track?"
> > >
> > 
> > Hi,
> > 
> > actually Guillermo was already working on that, will send something by the end of the day
> > 
> > elena
> > 
> > >> In particular, I would be interested in discussing ways to standardize
> > >> ABI and type information in a way that it can be embedded into
> > >> binaries in a less ambiguous way. In other words, what can we do to
> > >> not rely entirely on intermediate formats like CTF or DWARF to make
> > >> sense of an ABI? Maybe CTF is already a good starting point, yet some
> > >> additions are needed (e.g. other language features like for C++)?
> > >>
> > >> Cheers,
> > >> Matthias
> > >>
> > >>
> > >> On Sat, May 7, 2022 at 3:15 AM Guillermo E. Martinez
> > >> <guillermo.e.martinez@oracle.com> wrote:
> > >>> Hello guys,
> > >>>
> > >>> On our side we are working on:
> > >>>
> > >>>   * Gives the support  to extract the ABI for Linux kernel image
> > >>>   and Linux  kernel modules using the CFT backed in with
> > >>>   libabigail tools.
> > >>>   * Add support to compare packages with CTF debug format by using
> > >>>   abipkgdiff and other related libabigail tools.
> > >>>
> > >>>    * https://sourceware.org/pipermail/libabigail/2022q2/004340.html
> > >>>    * https://sourceware.org/pipermail/libabigail/2022q2/004345.html
> > >>>
> > >>> If you think that it could be a point to talk as part of your MC,  I can for sure
> > >>> provides more details about it.
> > >>>
> > >>> Thanks for your comments,
> > >>> Guillermo
> > >>>
> > >>> On Wednesday, April 20, 2022 12:34:59 PM CDT Giuliano Procida wrote:
> > >>>> Sorry for the late reply, I've been on leave and Mattias is still on leave.
> > >>>>
> > >>>> That sounds quite interesting.
> > >>>>
> > >>>> Matthias and I could certainly talk about what we've been busy with and
> > >>>> what are the main pain points for Android (not just the kernel).
> > >>>>
> > >>>> On what more would we like from the toolchain... it falls into at least two
> > >>>> buckets:
> > >>>>
> > >>>> 1. make the possible easier - stricter conformance to DWARF standards, full
> > >>>> DWARF 5 support, getting things like ARM64 ABI extensions (e.g., for
> > >>>> HWASAN) into things like elfutils at the same time as the compile-link
> > >>>> toolchain, more consistency between Clang and GCC debug info for the same
> > >>>> sources, the same for Clang and Clang with full LTO
> > >>>> 2. make the impossible possible - extending ABI monitoring coverage beyond
> > >>>> just architecture, symbols and types / dealing with header constants,
> > >>>> macros and more
> > >>>>
> > >>>> Giuliano.
> > >>>>
> > >>>>
> > >>>> On Wed, 20 Apr 2022 at 17:02, Jose E. Marchesi <jose.marchesi@oracle.com>
> > >>>> wrote:
> > >>>>
> > >>>>> [Adding Dodji Seketeli, libabigail maintainer,
> > >>>>>  and Guillermo Martinez, who works in the CTF support for libabigail
> > >>>>>  including kernel support.]
> > >>>>>
> > >>>>>> Hi Matthias and Giuliano,
> > >>>>>> Jose and I are currently in the planning process to put together a
> > >>>>>> Kernel+Toolchain microconference track at Linux Plumbers Conference
> > >>>>>> this year (Sept 12-14) in Dublin, Ireland.
> > >>>>>>
> > >>>>>> Would you all be interested in a leading session on Kernel ABI
> > >>>>>> Monitoring and perhaps what toolchain vendors can do to help, as part
> > >>>>>> of our MC (which itself is currently still in planning phase).
> > >>>
> > >>>
> > >>>
> > >
> > >
> > 
> > 
> 
> 
> 
> 
> 

Re: Kernel ABI Monitoring and toolchain support

[Nick Desaulniers]

On Wed, Jun 22, 2022 at 8:27 AM Guillermo E. Martinez
<guillermo.e.martinez@oracle.com> wrote:
>
> Hello,
>
> Any comment?

LGTM; please file a formal submission at
https://lpc.events/event/16/abstracts/
with the "Track" set to "Toolchains Track."

>
> Thanks,
> guillermo
>
> > Hello guys,
> >
> > The paragraphs bellow summarize the proposal  for Kernel ABI analysis
> > talk in LPC, these ideas comes from the thread mail bellow.
> >
> > Please let me know your comments to submit it
> > to https://lpc.events/event/16/abstracts/
> >
> > guillermo
> >
> > ---
> > Kernel ABI Monitoring and toolchain support
> >
> > The new CTF(Compact C Type Format) supported  in  libabigail  is  able
> > to extract a  corpus  representation  for  the  debug  information  in
> > Kernel binary and its modules, i.e, entire Kernel  release  (kernel  +
> > modules).  Using CTF reader improvements the time to extract and build
> > the corpus compared with DWARF reader,  for  example,  extracting  ABI
> > information  from  the  Linux  kernel  takes  up  to  ~4.5times   less
> > time, this was done using a Kernel  compiled  by  GCC,  nowadays  LLVM
> > doesn't support binaries generation with CTF debug info, would be nice
> > to have this.
> >
> > But what about of the modules inserted  (loaded)  at  runtime  in  the
> > Kernel image?.  To make the comparison it uses kABI  scripts  this  is
> > useful among other things to load modules with compatible  kABI,  this
> > mechanism allows modules to be used with a  different  kernel  version
> > that of the kernel for which it was built.  So what of using a  single
> > notion  of   ABI   (libabigail)   also   for   the   modules   loader?
> >
> > Since we add support for  CTF  in  libabigail,  is  needed  the  patch
> > for building the Kernel  with  CTF  enabled  in  the  Kernel  upstream
> > configuration.  Also some GCC  attributes  that  affect  the  ABI  and
> > are used by kernel hackers like noreturn,  interrupt,  etc.   are  not
> > represented in DWARF/CTF debug  format  and  therefore  they  are  not
> > present in the corpus.
> >
> > A stricter conformance to DWARF standards would be nice, full DWARF  5
> > support, getting things like ARM64 ABI extensions (e.g.,  for  HWASAN)
> > into things like  elfutils  at  the  same  time  as  the  compile-link
> > toolchain, more consistency between Clang and GCC debug info  for  the
> > same sources, the same for Clang and Clang  with  full  LTO.   And  an
> > extending ABI monitoring coverage beyond  just  architecture,  symbols
> > and  types  /  dealing  with  header  constants,   macros   and   more
> >
> > The  interest  in  discussing  ways  to  standardize  ABI   and   type
> > information in a way that it can be embedded into binaries in  a  less
> > ambiguous way.  In other words, what can we do to not rely entirely on
> > intermediate formats like CTF or DWARF to make sense of an ABI?  Maybe
> > CTF is already a good starting point, yet some  additions  are  needed
> > (e.g. other language features like for C++)?
> >
> > > On 6/21/22 15:19, Nick Desaulniers wrote:
> > > > On Mon, Jun 20, 2022 at 6:42 AM Matthias Männich <maennich@google.com> wrote:
> > > >> Hi!
> > > >>
> > > >> That is a very late response, sorry about that.
> > > >>
> > > >> In the Android MC we will have a talk about the latest news monitoring
> > > >> Android Kernel ABIs already, but I definitely see value in a more
> > > >> generic session in the Kernel+Toolchain MC. Is that schedule already
> > > >> finalized or is there still a slot available and interest from your
> > > >> side?
> > > > We haven't finalized the schedule yet.  Matthias, Guillermo, Giuliano
> > > > can one of you perhaps fill out a proposal then for the ideas
> > > > discussed in this thread on https://lpc.events/event/16/abstracts/
> > > > with the "Track" set to "Toolchains Track?"
> > > >
> > >
> > > Hi,
> > >
> > > actually Guillermo was already working on that, will send something by the end of the day
> > >
> > > elena
> > >
> > > >> In particular, I would be interested in discussing ways to standardize
> > > >> ABI and type information in a way that it can be embedded into
> > > >> binaries in a less ambiguous way. In other words, what can we do to
> > > >> not rely entirely on intermediate formats like CTF or DWARF to make
> > > >> sense of an ABI? Maybe CTF is already a good starting point, yet some
> > > >> additions are needed (e.g. other language features like for C++)?
> > > >>
> > > >> Cheers,
> > > >> Matthias
> > > >>
> > > >>
> > > >> On Sat, May 7, 2022 at 3:15 AM Guillermo E. Martinez
> > > >> <guillermo.e.martinez@oracle.com> wrote:
> > > >>> Hello guys,
> > > >>>
> > > >>> On our side we are working on:
> > > >>>
> > > >>>   * Gives the support  to extract the ABI for Linux kernel image
> > > >>>   and Linux  kernel modules using the CFT backed in with
> > > >>>   libabigail tools.
> > > >>>   * Add support to compare packages with CTF debug format by using
> > > >>>   abipkgdiff and other related libabigail tools.
> > > >>>
> > > >>>    * https://sourceware.org/pipermail/libabigail/2022q2/004340.html
> > > >>>    * https://sourceware.org/pipermail/libabigail/2022q2/004345.html
> > > >>>
> > > >>> If you think that it could be a point to talk as part of your MC,  I can for sure
> > > >>> provides more details about it.
> > > >>>
> > > >>> Thanks for your comments,
> > > >>> Guillermo
> > > >>>
> > > >>> On Wednesday, April 20, 2022 12:34:59 PM CDT Giuliano Procida wrote:
> > > >>>> Sorry for the late reply, I've been on leave and Mattias is still on leave.
> > > >>>>
> > > >>>> That sounds quite interesting.
> > > >>>>
> > > >>>> Matthias and I could certainly talk about what we've been busy with and
> > > >>>> what are the main pain points for Android (not just the kernel).
> > > >>>>
> > > >>>> On what more would we like from the toolchain... it falls into at least two
> > > >>>> buckets:
> > > >>>>
> > > >>>> 1. make the possible easier - stricter conformance to DWARF standards, full
> > > >>>> DWARF 5 support, getting things like ARM64 ABI extensions (e.g., for
> > > >>>> HWASAN) into things like elfutils at the same time as the compile-link
> > > >>>> toolchain, more consistency between Clang and GCC debug info for the same
> > > >>>> sources, the same for Clang and Clang with full LTO
> > > >>>> 2. make the impossible possible - extending ABI monitoring coverage beyond
> > > >>>> just architecture, symbols and types / dealing with header constants,
> > > >>>> macros and more
> > > >>>>
> > > >>>> Giuliano.
> > > >>>>
> > > >>>>
> > > >>>> On Wed, 20 Apr 2022 at 17:02, Jose E. Marchesi <jose.marchesi@oracle.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> [Adding Dodji Seketeli, libabigail maintainer,
> > > >>>>>  and Guillermo Martinez, who works in the CTF support for libabigail
> > > >>>>>  including kernel support.]
> > > >>>>>
> > > >>>>>> Hi Matthias and Giuliano,
> > > >>>>>> Jose and I are currently in the planning process to put together a
> > > >>>>>> Kernel+Toolchain microconference track at Linux Plumbers Conference
> > > >>>>>> this year (Sept 12-14) in Dublin, Ireland.
> > > >>>>>>
> > > >>>>>> Would you all be interested in a leading session on Kernel ABI
> > > >>>>>> Monitoring and perhaps what toolchain vendors can do to help, as part
> > > >>>>>> of our MC (which itself is currently still in planning phase).
> > > >>>
> > > >>>
> > > >>>
> > > >
> > > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>


-- 
Thanks,
~Nick Desaulniers

Re: Kernel ABI Monitoring and toolchain support

[Dodji Seketeli]

Matthias Männich <maennich@google.com> a écrit:

> Hi!
>
> That is a very late response, sorry about that.
>
> In the Android MC we will have a talk about the latest news monitoring
> Android Kernel ABIs already, but I definitely see value in a more
> generic session in the Kernel+Toolchain MC. Is that schedule already
> finalized or is there still a slot available and interest from your
> side?
>
> In particular, I would be interested in discussing ways to standardize
> ABI and type information in a way that it can be embedded into
> binaries in a less ambiguous way. In other words, what can we do to
> not rely entirely on intermediate formats like CTF or DWARF to make
> sense of an ABI? Maybe CTF is already a good starting point, yet some
> additions are needed (e.g. other language features like for C++)?

I'd be super interested in this topic, for what it's worth.  I won't be
able to attend the conference physically, though, only remotely.

Would that work?

Thanks.

-- 
		Dodji

Re: Kernel ABI Monitoring and toolchain support

[Dodji Seketeli]

Hey,

Matthias Männich <maennich@google.com> a écrit:

> That is a very late response, sorry about that.

No problem Matthias :-)

I totally forgot about this thread to begin with, so thank you for
reviving it.

> In the Android MC we will have a talk about the latest news monitoring
> Android Kernel ABIs already, but I definitely see value in a more
> generic session in the Kernel+Toolchain MC. Is that schedule already
> finalized or is there still a slot available and interest from your
> side?
>
> In particular, I would be interested in discussing ways to standardize
> ABI and type information in a way that it can be embedded into
> binaries in a less ambiguous way. In other words, what can we do to
> not rely entirely on intermediate formats like CTF or DWARF to make
> sense of an ABI? Maybe CTF is already a good starting point, yet some
> additions are needed (e.g. other language features like for C++)?

Do you have something in particular in mind, Matthias?  I am very
curious about this.  I know the idea is to discuss this at the
conference, but a little bit of a teaser would be awesome, I guess ;-)

Cheers,

>
> Cheers,
> Matthias
>
>
> On Sat, May 7, 2022 at 3:15 AM Guillermo E. Martinez
> <guillermo.e.martinez@oracle.com> wrote:
>>
>> Hello guys,
>>
>> On our side we are working on:
>>
>>   * Gives the support  to extract the ABI for Linux kernel image
>>   and Linux  kernel modules using the CFT backed in with
>>   libabigail tools.
>>   * Add support to compare packages with CTF debug format by using
>>   abipkgdiff and other related libabigail tools.
>>
>>    * https://sourceware.org/pipermail/libabigail/2022q2/004340.html
>>    * https://sourceware.org/pipermail/libabigail/2022q2/004345.html
>>
>> If you think that it could be a point to talk as part of your MC,  I can for sure
>> provides more details about it.
>>
>> Thanks for your comments,
>> Guillermo
>>
>> On Wednesday, April 20, 2022 12:34:59 PM CDT Giuliano Procida wrote:
>> > Sorry for the late reply, I've been on leave and Mattias is still on leave.
>> >
>> > That sounds quite interesting.
>> >
>> > Matthias and I could certainly talk about what we've been busy with and
>> > what are the main pain points for Android (not just the kernel).
>> >
>> > On what more would we like from the toolchain... it falls into at least two
>> > buckets:
>> >
>> > 1. make the possible easier - stricter conformance to DWARF standards, full
>> > DWARF 5 support, getting things like ARM64 ABI extensions (e.g., for
>> > HWASAN) into things like elfutils at the same time as the compile-link
>> > toolchain, more consistency between Clang and GCC debug info for the same
>> > sources, the same for Clang and Clang with full LTO
>> > 2. make the impossible possible - extending ABI monitoring coverage beyond
>> > just architecture, symbols and types / dealing with header constants,
>> > macros and more
>> >
>> > Giuliano.
>> >
>> >
>> > On Wed, 20 Apr 2022 at 17:02, Jose E. Marchesi <jose.marchesi@oracle.com>
>> > wrote:
>> >
>> > >
>> > > [Adding Dodji Seketeli, libabigail maintainer,
>> > >  and Guillermo Martinez, who works in the CTF support for libabigail
>> > >  including kernel support.]
>> > >
>> > > > Hi Matthias and Giuliano,
>> > > > Jose and I are currently in the planning process to put together a
>> > > > Kernel+Toolchain microconference track at Linux Plumbers Conference
>> > > > this year (Sept 12-14) in Dublin, Ireland.
>> > > >
>> > > > Would you all be interested in a leading session on Kernel ABI
>> > > > Monitoring and perhaps what toolchain vendors can do to help, as part
>> > > > of our MC (which itself is currently still in planning phase).
>> > >
>> >
>>
>>
>>
>>

-- 
		Dodji