Release of LKST 1.3

Release of LKST 1.3

[Yumiko Sugita]

Dear all,

    I'd like to announce publication of Linux Kernel State Tracer (LKST) 1.3,
  which is a tracer for Linux kernel.

    LKST's main purpose is debugging, fault analysis and performance analysis
of enterprise systems.
    For the purpose, LKST has these features,

    (1)It is possible to change dynamically which events are recorded.
        Users can obtain information about the events which they concern
       only interesting events.
        And it reduces the overhead of components which is not related
       with a fault.

    (2)It is possible to change each function invoked by each events.
        A default function invoked by events is just recording occuring of the events.
       But, if it is necessary, this function can be changed to another
       function.
        And LKST supports installing the function by using a kernel module
        LKST also supports a maskset, which controls what kind of events should be
       recorded, can be changed dynamically. For example, LKST usually traces
       a few events for good performance, and when the kernel be in a particular
       status, LKST can change a maskset to get more detail information.

    (3)It is possible to create new buffers and change to one of them.
         By changing to other buffer, Users can leave the information
       which they want.


    LKST binaries, source code and documents are available in the following site,
         https://sourceforge.net/projects/lkst/
         http://sourceforge.jp/projects/lkst/
         http://oss.hitachi.co.jp/sdl/english/lkst.html (now updating)

    We prepared a mailing list written below in order to let users know 
update of LKST.

  lkst-users@lists.sourceforge.net
  lkst-users@lists.sourceforge.jp

  To subscribe, please refer following URL,

  http://lists.sourceforge.net/lists/listinfo/lkst-users
  http://lists.sourceforge.jp/mailman/listinfo/lkst-users

    And if you have any comments, please send to the above list, or to 
another mailing
  list written below.

  lkst-develop@lists.sourceforge.net
  lkst-develop@lists.sourceforge.jp


  With kindest regards,
  All of the LKST developers
----------------
  Yumiko Sugita
  Hitachi,Ltd., Systems Development Laboratory
  

Re: Release of LKST 1.3

[Robert Schwebel]

On Wed, Sep 18, 2002 at 09:20:55PM +0900, Yumiko Sugita wrote:
> I'd like to announce publication of Linux Kernel State Tracer (LKST)
> 1.3, which is a tracer for Linux kernel.

Can you tell us what's the difference between this and the Linux Trace
Toolkit (LTT)? 

Robert
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Braunschweiger Str. 79,  31134 Hildesheim, Germany
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
    Phone: +49-5121-28619-0 |  Fax: +49-5121-28619-4

Re: Release of LKST 1.3

[Vamsi Krishna S.]

Hi,

>From the howto-1.3.txt

> Known Problems:
>
> - GCC 2.96, which is a kernel compiler in RedHat, with -O2 option generates
> wrong code around Kernel Hooks that caused the kernel panic. To avoid this
> problem, you use the GCC 2.95, which is a standard kernel compiler.

I use 2.96 here all the time and haven't seen it miscompile hooks. Can you 
please elaborate what the problems are and give a me small sample code
to demonstrate the problem.

Thanks,
Vamsi.

--
Vamsi Krishna S.
Linux Technology Center,
IBM Software Lab, Bangalore.
Ph: +91 80 5044959
Internet: vamsi@in.ibm.com

Re: [Lkst-develop] Re: Release of LKST 1.3

[Yumiko Sugita]

Hello, Robert

   Thank you for your inquiry. I apologize for the delay in replying.

   Let me first explain the background of our development work.
   We began development of the Linux Kernel State Tracer (LKST)
in response to a domestic need to improve Reliability, Availability,
and Serviceability (RAS) with respect to enterprise systems.
The following requirements were applied to LKST:

   * Capable of handling a variety of information about system errors.
   * Little trace overhead (or To control trace overhead)
   * Short development time

   As we had to achieve a short development time, we elected to
develop LKST using our own methodology (based on know-how of
tracer development that we carried out for other OS's) different
from other known tools such as LTT.
# This is not to say that we developed all functions on our own.
#LKST at present connects with Kernel Hooks (GKHI) and LKCD.

   Consequently, LKST, which is oriented to enterprise systems,
has the following features different from those of LTT.
# These LKST features are also being enhanced at this time.

(1) Little overhead and good scalability when tracing on a large-scale
    SMP system
   * To make lock mechanism overhead as little as possible, we
      designed that the buffers are not shared among CPUs.

(2) Easy to extend/expand the function (User-based extendibility)
   * Without recompiling kernel, user can change/add/modify the kind
     of events and information to be recorded at anytime.
      For example, LKST usually traces very few events for the purpose
    of good performance.  Once the kernel get into the particular status
    that user specified, LKST will trace and record more detail information.

(3) Preservation of trace information
   * Recovery of trace information collected at the time of a system crash
     in connection with LKCD.
   * Saving of specific event information during tracing.
      For example, switching to another buffer after the occurrence of
     a specific event enables the information on that event to be left
     in the previous buffer.

(4) Collection of even more kernel event information
   * Information on more than 50 kernel events can be collected for
     kernel debugging.

  The demand for RAS functions in Linux should grow in the years to come.
It is our hope that LKST becomes one means of implementing such functions.


Best regards,
----------------
  Yumiko Sugita
  Hitachi,Ltd., Systems Development Laboratory


At 07:58 02/09/19 +0200, Robert Schwebel wrote:
>On Wed, Sep 18, 2002 at 09:20:55PM +0900, Yumiko Sugita wrote:
> > I'd like to announce publication of Linux Kernel State Tracer (LKST)
> > 1.3, which is a tracer for Linux kernel.
>
>Can you tell us what's the difference between this and the Linux Trace
>Toolkit (LTT)?
>
>Robert
>--
>  Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
>  Pengutronix - Linux Solutions for Science and Industry
>    Braunschweiger Str. 79,  31134 Hildesheim, Germany
>    Handelsregister:  Amtsgericht Hildesheim, HRA 2686
>     Phone: +49-5121-28619-0 |  Fax: +49-5121-28619-4
>_______________________________________________
>Lkst-develop mailing list
>Lkst-develop@lists.sourceforge.jp
>http://lists.sourceforge.jp/cgi-bin/mailman/listinfo/lkst-develop

Re: [Lkst-develop] Re: Release of LKST 1.3

[Karim Yaghmour]

Just a couple of general observations here.

Yumiko Sugita wrote:
>    Consequently, LKST, which is oriented to enterprise systems,
> has the following features different from those of LTT.
> # These LKST features are also being enhanced at this time.
> 
> (1) Little overhead and good scalability when tracing on a large-scale
>     SMP system
>    * To make lock mechanism overhead as little as possible, we
>       designed that the buffers are not shared among CPUs.

I was wondering whether you followed the recent discussion about LTT on the
LKML? Clearly this is not a problem for LTT since we don't use any form
of locking whatsoever anymore. IBM's work on the lockless scheme has
solved this problem and their current work on the per-CPU buffering solves
the rest of the issue.

> (2) Easy to extend/expand the function (User-based extendibility)
>    * Without recompiling kernel, user can change/add/modify the kind
>      of events and information to be recorded at anytime.

Ditto with LTT.

>       For example, LKST usually traces very few events for the purpose
>     of good performance.  Once the kernel get into the particular status
>     that user specified, LKST will trace and record more detail information.

This implies callbacks, which do exist in LTT and which Ingo Molnar explicitly
asked us to remove.

> (3) Preservation of trace information
>    * Recovery of trace information collected at the time of a system crash
>      in connection with LKCD.

Connection with LKCD is really not a problem, but this points to the main
purpose of the tool, which in the case of LKCD is kernel debugging. LTT isn't
aimed as a kernel debugger, so although LKCD is on our to-do list, it's
certainly not our priority.

As for handling multiple output streams (which LKCD can be one of them), we
already have very detailed plans on how LTT is going to integrate this (as I've
mentioned a number of times before on this list). However, before we go down
this road we need to make sure that the core tracing functionality is
lightweight and fits the general requirements set for kernel code. Once this
core lighweight functionality is there, we can build a rich and solid feature
set around it.

>    * Saving of specific event information during tracing.
>       For example, switching to another buffer after the occurrence of
>      a specific event enables the information on that event to be left
>      in the previous buffer.

Again, callbacks and triggers. A while back, I had written a state machine
engine for LTT. Basically, you could provide it with an event-driven state
machine and it would callback your functions depending on the sequence of
events. All of this obviously implies callbacks, which, as I said earlier,
we've been explicitly asked to remove.

> (4) Collection of even more kernel event information
>    * Information on more than 50 kernel events can be collected for
>      kernel debugging.

Well, I think this is where LTT and LKST cannot be compared. If LKST is
a kernel debugging tool, as it has always been advertised, then any comparison
of LKST should be made with the other tracing tools which are used for
kernel debugging, such as the ones mentioned by Ingo and Andi earlier on
this list.

LTT was built from the ground up to help users understand the dynamic
behavior of the system. As such, it cannot be compared to any kernel
debugging tool since it isn't one.

>   The demand for RAS functions in Linux should grow in the years to come.
> It is our hope that LKST becomes one means of implementing such functions.

There was a RAS BoF at the OLS this year where tracing was intensively discussed.
All the attendees agreed to unify their efforts around LTT. At this meeting,
Richard Moore of IBM presented a tracing to-do list
(http://opersys.com/LTT/ltt-to-do-list.txt) which we are using a basic
check list for our ongoing work. Instead of implementing yet another tracing
system, I think that the LKST team would benefit much from contributing to
LTT, which has already a proven track record and has been adopted by the
community as much as the industry.

Karim

===================================================
                 Karim Yaghmour
               karim@opersys.com
      Embedded and Real-Time Linux Expert
===================================================

Re: [Lkst-develop] Re: Release of LKST 1.3

[Yumiko Sugita]

Hello, Mr. Karim

  I'm Yumiko Sugita, a member of LKST developers.
  I read your mail. Thank you for your advices.

  The features for SMP are one of our important aims,
so we, LKST developpers, developed them. We welcome
IBM's work on them.

  We think callback feature is useful for kernel developers.
Are there any problems?　Are you going to remove callbacks
from LTT?  Is the main reason security?  If you have some
cases of security problems about callbacks, please teach
them and give some advice to us.

  After future, we'll join community actively. We'll use LTT
and want to concern LTT, so we'll join the discussion of you
and other LTT developers about Linux RAS.
  We hope to co-operate you and other developers about
Linux RAS.

Best regards,

*〜*〜*〜*〜*〜
Yumiko Sugita
Hitachi, Ltd., Systems Development Laboratory
Email : sugita@sdl.hitachi.co.jp
                              　　　　　  〜*〜*〜*〜*〜*

At 11:48 02/09/27 -0400, Karim Yaghmour wrote:

>Just a couple of general observations here.
>
>Yumiko Sugita wrote:
> >    Consequently, LKST, which is oriented to enterprise systems,
> > has the following features different from those of LTT.
> > # These LKST features are also being enhanced at this time.
> >
> > (1) Little overhead and good scalability when tracing on a large-scale
> >     SMP system
> >    * To make lock mechanism overhead as little as possible, we
> >       designed that the buffers are not shared among CPUs.
>
>I was wondering whether you followed the recent discussion about LTT on the
>LKML? Clearly this is not a problem for LTT since we don't use any form
>of locking whatsoever anymore. IBM's work on the lockless scheme has
>solved this problem and their current work on the per-CPU buffering solves
>the rest of the issue.
>
> > (2) Easy to extend/expand the function (User-based extendibility)
> >    * Without recompiling kernel, user can change/add/modify the kind
> >      of events and information to be recorded at anytime.
>
>Ditto with LTT.
>
> >       For example, LKST usually traces very few events for the purpose
> >     of good performance.  Once the kernel get into the particular status
> >     that user specified, LKST will trace and record more detail information.
>
>This implies callbacks, which do exist in LTT and which Ingo Molnar explicitly
>asked us to remove.
>
> > (3) Preservation of trace information
> >    * Recovery of trace information collected at the time of a system crash
> >      in connection with LKCD.
>
>Connection with LKCD is really not a problem, but this points to the main
>purpose of the tool, which in the case of LKCD is kernel debugging. LTT isn't
>aimed as a kernel debugger, so although LKCD is on our to-do list, it's
>certainly not our priority.
>
>As for handling multiple output streams (which LKCD can be one of them), we
>already have very detailed plans on how LTT is going to integrate this (as I've
>mentioned a number of times before on this list). However, before we go down
>this road we need to make sure that the core tracing functionality is
>lightweight and fits the general requirements set for kernel code. Once this
>core lighweight functionality is there, we can build a rich and solid feature
>set around it.
>
> >    * Saving of specific event information during tracing.
> >       For example, switching to another buffer after the occurrence of
> >      a specific event enables the information on that event to be left
> >      in the previous buffer.
>
>Again, callbacks and triggers. A while back, I had written a state machine
>engine for LTT. Basically, you could provide it with an event-driven state
>machine and it would callback your functions depending on the sequence of
>events. All of this obviously implies callbacks, which, as I said earlier,
>we've been explicitly asked to remove.
>
> > (4) Collection of even more kernel event information
> >    * Information on more than 50 kernel events can be collected for
> >      kernel debugging.
>
>Well, I think this is where LTT and LKST cannot be compared. If LKST is
>a kernel debugging tool, as it has always been advertised, then any comparison
>of LKST should be made with the other tracing tools which are used for
>kernel debugging, such as the ones mentioned by Ingo and Andi earlier on
>this list.
>
>LTT was built from the ground up to help users understand the dynamic
>behavior of the system. As such, it cannot be compared to any kernel
>debugging tool since it isn't one.
>
> >   The demand for RAS functions in Linux should grow in the years to come.
> > It is our hope that LKST becomes one means of implementing such functions.
>
>There was a RAS BoF at the OLS this year where tracing was intensively 
>discussed.
>All the attendees agreed to unify their efforts around LTT. At this meeting,
>Richard Moore of IBM presented a tracing to-do list
>(http://opersys.com/LTT/ltt-to-do-list.txt) which we are using a basic
>check list for our ongoing work. Instead of implementing yet another tracing
>system, I think that the LKST team would benefit much from contributing to
>LTT, which has already a proven track record and has been adopted by the
>community as much as the industry.
>
>Karim
>
>===================================================
>                  Karim Yaghmour
>                karim@opersys.com
>       Embedded and Real-Time Linux Expert
>===================================================
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Re: [Lkst-develop] Re: Release of LKST 1.3

[Karim Yaghmour]

... sorry for the delay, I'm very busy lately ...

Yumiko Sugita wrote:
>   We think callback feature is useful for kernel developers.
> Are there any problems?^[$B!!^[(BAre you going to remove callbacks
> from LTT?  Is the main reason security?  If you have some
> cases of security problems about callbacks, please teach
> them and give some advice to us.

The issue of callbacks was covered by one of Ingo's comments about
LTT. Here's the excerpt from his mail:
> okay. The thing is that generic callbacks and data hooks in the task
> structure are an invitation for various types of abuses - security and GPL
> type abuses. People do get very nervous when seeing such stuff - eg. read
> back Christoph Hellwig's comment from a few weeks ago. It's a red flag for
> many people. Provide a clean and concentrated set of APIs, no callbacks,
> no unnecessery hooks. I can see the technical reasons why you have added
> it - it's in theory an extensible interface, but generally we tend to add
> such stuff when it's needed - if it's needed at all.

(You can get the complete copy from:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103276662708853&w=2)

If you would like to provide callbacks for _kernel developers_ then
these callbacks should live as an outside patch, as with any other
facility that is useful to kernel development only.

If there is a legitimate need for such hooks later on, then we can
add them when needed, as Ingo suggested. None of it is really
complicated. These callbacks would also have to be exported as
GPL-only, in order to avoid any sort of abuse. The main issue we are
concentrating on at this time, however, is to make sure that the core
infrastructure is lightweight and solid. Any additional features will
be added on top of this core infrastructure.

>   After future, we'll join community actively. We'll use LTT
> and want to concern LTT, so we'll join the discussion of you
> and other LTT developers about Linux RAS.
>   We hope to co-operate you and other developers about
> Linux RAS.

We certainly welcome any contribution and will be happy to help
you integrate your features into a common tracing infrastructure.
Feel free to join in the discussion around the LTT development
mailing list (See the project's web site for details on how to
subscribe).

Best regards,

Karim

===================================================
                 Karim Yaghmour
               karim@opersys.com
      Embedded and Real-Time Linux Expert
===================================================