Re: Enabling and disabling events

Re: Enabling and disabling events

[Ikaheimonen, JP]

Hi Daniel,
as you can see, things easily get too complicated.

Currently in LTTng you can only disable events that are enabled in the first place. In other words, an event is born disabled, as if 'lttng disable-event "*"' had been issued whenever a channel is created. We already have a mechanism to un-disable events - this is the enable-event command. I do not see any reason to add another command to do exactly what enable-event already does. If we want to 'disable all events except 'a'' we can call lttng enable-event 'a'. Symmetrically, I would like to have 'enable all events except 'a'', but currently this is impossible.

I still propose keeping the disable-event and enable-event commands asymmetrical. Based on your thoughts below, I also suggest keeping two lists for the enabled events: one for the enablements, and one for the exceptions to the enablements. The items in the latter list would point to specific items in the former list, so that the exceptions would be tied to certain enablements.

So for instance,

$ lttng enable-event -u 'x'
$ lttng list session
Events:
  x [enabled]
$ lttng enable-event -u "x*" --except "xa*"
$ lttng list session
Events:
  x [enabled]
  x* [enabled, exception xa*]
$ lttng enable-event -u "x*" --except "xb*"
$ lttng list session
  x [enabled]
  x* [enabled, exception xa*]
  x* [enabled, exception xb*]

Again, all the enablements would be checked, and the event would be enabled if any of the enablements match.

If an event enablement has an exception, it must be used in the disable-event command also:
$ lttng disable-event -u "x*" --except "xb*"
$ lttng list session
  x [enabled]
  x* [enabled, exception xa*]
  x* [disabled, exception xb*]

Comments are very welcome.

JP Ikaheimonen | SW Development Engineer http://go.mentor.com/sourceryanalyzer

Mentor Embedded(tm) | Nucleus(r) | Linux(r) | Android(tm) | Services | UI | Multi-OS

Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions.
Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

Re: Enabling and disabling events

[Thibault, Daniel]

> De : Ikaheimonen, JP [mailto:jp_ikaheimonen@mentor.com] 
> Envoyé : 10 septembre 2013 07:41
>
> As you can see, things easily get too complicated.
>
> Currently in LTTng you can only disable events that are enabled in the first place. In other words, an event is born disabled, as if 'lttng disable-event "*"' had been issued whenever a > channel is created. We already have a mechanism to un-disable events - this is the enable-event command. I do not see any reason to add another command to do exactly what
> enable-event already does. If we want to 'disable all events except 'a'' we can call lttng enable-event 'a'. Symmetrically, I would like to have 'enable all events except 'a'', but
> currently this is impossible.

   I disagree with the perception that "an event is born disabled".  I perceive quite the opposite, in fact.  The problem, as I've mentioned elsewhere before, is that enable-events is a double-duty command, which serves to both *create* (i.e define) events and to toggle their enablement.  I would much rather have a separate define-event command.  Right now you can define, enable, disable until the trace is started, and once that has happened you can only enable and disable.  With separate define-event and enable-event commands the code would be simpler (and thus less bug-prone), and so would the user experience.

   Getting back to event birth, events are initially non-existent, then they transition to defined (and enabled), and can later toggle between enabled and disabled (but cannot be undefined).  It is in that sense that they are born enabled.

> I still propose keeping the disable-event and enable-event commands asymmetrical. Based on your thoughts below, I also suggest keeping two lists for the enabled events: one for > the enablements, and one for the exceptions to the enablements. The items in the latter list would point to specific items in the former list, so that the exceptions would be tied to > certain enablements.
>
> So for instance,
>
> $ lttng enable-event -u 'x'
> $ lttng list session
> Events:
>   x [enabled]
> $ lttng enable-event -u "x*" --except "xa*"
> $ lttng list session
> Events:
>   x [enabled]
>   x* [enabled, exception xa*]

   I think the exception should be in the event_name string (recall that enable-event accepts a *list* of event_names), something like:
$ lttng enable-event -u "x*!xa*"
$ lttng enable-event -u "x*!xa*","y*!yy*"
   (Presumably, if an '!' appears in an actual event name, it would need to be escaped in the event_name argument of enable-event)

> $ lttng enable-event -u "x*" --except "xb*"
> $ lttng list session
>   x [enabled]
>   x* [enabled, exception xa*]
>   x* [enabled, exception xb*]
>
> Again, all the enablements would be checked, and the event would be enabled if any of the enablements match.

   What if the user types lttng enable-event -u "xa*!x*"?  Clearly the argument collapses to an empty string (it does not enable anything), so the interface should leave the current list of enablements and disablements alone, and issue a warning to the user.

> If an event enablement has an exception, it must be used in the disable-event command also:
> $ lttng disable-event -u "x*" --except "xb*"
> $ lttng list session
>   x [enabled]
>   x* [enabled, exception xa*]
>   x* [disabled, exception xb*]

   I don't like that restriction at all.  You should be able to use any argument for either enable-event or disable-event.

> Comments are very welcome.
>
> JP Ikaheimonen | SW Development Engineer http://go.mentor.com/sourceryanalyzer

   I still say the lttng daemon should process the resulting enablement/disablement sets to simplify them whenever possible.  Thus:

$ lttng enable-event -u 'x'
$ lttng list session
Events:
   x [enabled]
$ lttng enable-event -u "x*!xa*"
$ lttng list session
Events:
   x* [enabled, exception xa*]

   The 'x [enabled]' rule is removed because it is subsumed by the 'x* [enabled, exception xa*]' rule.

$ lttng enable-event -u "x*!xb*"
$ lttng list session
  x* [enabled]

   In this case the union of "x*!xa*" and "x*!xb*" results in "x*".

   If the user wants to do "all events beginning with x except for those beginning with xa or xb", we would have to accept the '[]' regular expression operator (e.g. enable-event "x*!x[ab]*) (but note that the '*' is a wildcard here, not the regular expression repetition operator).  It may be easier to go to full-blown regular expressions.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

Re: Enabling and disabling events

[Ikaheimonen, JP]

Hi Daniel,

I can see your point about how the events are created. I do not completely agree with your views, but it is not essential that I do.

How about the following (this is based on the ideas you have brought forth):

We keep the enable-event command as it is. However, we enhance the event name specification syntax somewhat. We could of course do a full regex matching, but I fear that would get too slow.  Instead we'd have something like

$ lttng enable-event -u "*!mine*!alsomine"

This would enable any event ('*'), except those who match with mine* or alsomine.

The matching logic would separate the name specification into parts, using the ! character as a separator. If the new event name matches the first part but does not match any of the latter parts, then the event gets enabled.

I don't like collating the event enablers into unified specifications. I'd like to have each enable-event command create its own entries into the enabler list. I feel a bit uneasy if the event enabler code gets too sophisticated.

I'm still unsure how the disable-event command should work. I'm playing around with the idea that the event specification in the disable-event command should exactly match the event specification given in the enable-event command. It does not feel good, but any solution around it seems to bring about the ideas about disable-lists which I would like to avoid.

Cheers,
JP Ikaheimonen | SW Development Engineer http://go.mentor.com/sourceryanalyzer

Mentor Embedded(tm) | Nucleus(r) | Linux(r) | Android(tm) | Services | UI | Multi-OS

Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions.
Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

Re: Enabling and disabling events

[Ikaheimonen, JP]

>   I think processing the list of rules is a must because it shows the user precisely and concisely what is going to happen during his session.  

I will need to think about the best way to collate (unify) the enablement specifications. It seems to be an interesting set-theoretic exercise. A solvable one, it seems. And, I must agree, it is necessary, if only for the disable-event processing.

Thank you, Daniel, for your comments.

JP

Re: Enabling and disabling events

[Thibault, Daniel]

> We keep the enable-event command as it is. However, we enhance the event name
> specification syntax somewhat. We could of course do a full regex matching, but I
> fear that would get too slow.  Instead we'd have something like
> 
> $ lttng enable-event -u "*!mine*!alsomine"
>
> This would enable any event ('*'), except those who match with mine* or alsomine.
>
> The matching logic would separate the name specification into parts, using the
> ! character as a separator. If the new event name matches the first part but does
> not match any of the latter parts, then the event gets enabled.

   Simple enough.

> I don't like collating the event enablers into unified specifications. I'd like to have
> each enable-event command create its own entries into the enabler list. I feel a
> bit uneasy if the event enabler code gets too sophisticated.
>
> I'm still unsure how the disable-event command should work. I'm playing around
> with the idea that the event specification in the disable-event command should
> exactly match the event specification given in the enable-event command. It
> does not feel good, but any solution around it seems to bring about the ideas
> about disable-lists which I would like to avoid.
>
> JP Ikaheimonen | SW Development Engineer http://go.mentor.com/sourceryanalyzer

   We could do without a separate disabling list if you stick to the ! notation.  But some unification processing would need to be done.  For example (showing desirable user feedback):

$ lttng enable-event -u --all
All UST events are enabled (channel channel0, session session)
$ lttng list session
    Enabled Events:
      * (type: tracepoint)
$ lttng disable-event -u "mine*"
UST event mine* disabled (channel channel0, session session)
$ lttng list session
    Enabled Events:
      *!mine* (type: tracepoint)
$ lttng enable-event -u "min*"
UST event min* enabled (channel channel0, session session)
Exception removed from UST event * (channel channel0, session session)
$ lttng list session
    Enabled Events:
      * (type: tracepoint)
$ lttng disable-event -u "mine*"
UST event mine* disabled (channel channel0, session session)
Exception added to UST events (channel channel0, session session)
$ lttng list session
    Enabled Events:
      *!mine* (type: tracepoint)
$ lttng enable-event -u "miner*"
UST event miner* enabled (channel channel0, session session)
$ lttng list session
    Enabled Events:
      *!mine* (type: tracepoint)
      miner* (type: tracepoint)

   Note that the display uses "Enabled Events:" and drops the per-line "[enabled]" since "[disabled]" can never happen in this scheme.  (So-called Prussian approach: everything not expressly permitted is forbidden)

   The feedback for '$ lttng disable-event -u "mine*"' reads 'Exception added to UST events ...' because multiple exceptions could be added and we don't want the feedback to get too cluttered.

   In the last case ($ lttng enable-event -u "miner*"), assuming identifiers can use a..z, A..Z, 0..9 and _, we could have unified into something like:

$ lttng list session
    Enabled Events:
      *!mina*!minb*!minc*!mind*!minf*!ming*!minh*... (type: tracepoint) [enabled]

   But that would generate a huge enabling specification line (1+6*(25+26+10+1) = 373 characters) which is not very user-friendly.  Keeping two rules (ORed together, remember) is cleaner (and would be processed faster).

   Further use-case:

$ lttng list session
    Enabled Events:
      (none)
$ lttng disable-event -u "mine*"
Error: Event mine*: UST event not enabled (channel channel0, session session), nothing to disable
Warning: Some command(s) went wrong
$ lttng list session
    Enabled Events:
      (none)

   And using ! with disable-event:

$ lttng enable-event -u --all
All UST events are enabled (channel channel0, session session)
$ lttng list session
    Enabled Events:
      * (type: tracepoint)
$ lttng disable-event -u "mine*!miner*"
UST event mine* disabled (channel channel0, session session)
UST event miner* enabled (channel channel0, session session)
$ lttng list session
    Enabled Events:
      *!mine* (type: tracepoint)
      miner* (type: tracepoint)

$ lttng enable-event -u "miners*"
UST event miners* is enabled (channel channel0, session session)
$ lttng list session
    Enabled Events:
      miners* (type: tracepoint)
$ lttng disable-event -u "mine*!miner*"
Error: Event mine*!miner*: UST event to disable matches exception, nothing to disable (channel channel0, session session)
Warning: Some command(s) went wrong
$ lttng list session
    Enabled Events:
      miners* (type: tracepoint)

   Note that exceptions to the disable-event argument serve only to maintain existing enablements: disable-event a!b is not the same as disable-event a followed by enable-event b.  It should be read as "disable event a but do not disable event b" (not disabling an event is not the same as enabling it).

$ lttng list session
    Enabled Events:
      miners* (type: tracepoint)
$ lttng disable-event -u "mine*!min*"
Error: Event mine*!min*: Exception matches condition, nothing to disable (channel channel0, session session)
Warning: Some command(s) went wrong

   With filters, the filter field would remain attached to the rule and nothing else would change.  (Note that currently 'lttng list' cannot display the filters that have been set so far)  Filters are also Prussian: of their three logical states (true, false, unknown), only the true state allows the event to pass (unknown occurs if any part of the filter fails to bind to the event's fields and context).

$ lttng enable-event -u provider:event --filter "intfield > 500"
Filter 'intfield > 500' successfully set for UST event provider:event (channel channel0, session session)
$ lttng list session
    Enabled Events:
      provider:event (type: tracepoint) [with filter 'intfield > 500']
$ lttng enable-event -u --all
All UST events are enabled (channel channel0, session session)
$ lttng list session
    Enabled Events:
      * (type: tracepoint)

   I think processing the list of rules is a must because it shows the user precisely and concisely what is going to happen during his session.  Right now, you can get a messy listing like this:

$ lttng list session
    Events:
      sample:event (type: tracepoint) [disabled]
      sample:event (type: tracepoint) [disabled] [with filter]
      sample:ev* (type: tracepoint) [enabled] [with filter]
      * (type: tracepoint) [enabled]

   The user can't figure out what is enabled and what is disabled unless he remembers the rules are all ORed together.  He'll mentally boil the set of rules down to just "* [enabled]", but a nasty suspicion will remain regarding all those remaining disregarded rules.  Processing the list of rules to its simplest expression at the time enable-event/disable-event is issued also saves processing later -whenever the daemon needs to figure out whether a newly registered event should be traced or not- because it'll have fewer rules to try out.

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

Enabling and disabling events

[Thibault, Daniel]

   Suppose we do something like this (edited for brevity):

$ lttng enable-event -u sample:message --loglevel notice
$ lttng list session
Events:
  sample:message (loglevel: TRACE_NOTICE (5)) [enabled]
$ lttng disable-event -u sample:message
$ lttng list session
Events:
  sample:message (loglevel: TRACE_NOTICE (5)) [disabled]
$ lttng enable-event -u sample:message
$ lttng list session
Events:
  sample:message [enabled]
  sample:message (loglevel: TRACE_NOTICE (5)) [disabled]
$ lttng enable-event -u sample:message --loglevel err
$ lttng list session
Events:
  sample:message [enabled]
  sample:message (loglevel: TRACE_NOTICE (5)) [disabled]
  sample:message (loglevel: TRACE_ERR (3)) [enabled]

This could be understood to mean that sample:message will be logged unless its loglevel lies in the TRACE_WARNING (4) to TRACE_NOTICE (5) range.  But that is not what lttng is doing: it looks up all the rules which match the candidate event, then ORs the results together.  (Two rules are different if their string key or loglevel setting are different; currently lttng treats a loglevel-only setting as it it matched the corresponding loglevel setting)  So in this case there is no way that sample:message could not be logged, since the first "rule" (i.e. sample:message [enabled]) trumps the others.

Wouldn't it be much clearer if the enable-event command did some clean up in the list of rules?  The previous use case would then become:

$ lttng enable-event -u sample:message --loglevel notice
$ lttng list session
Events:
  sample:message (loglevel: TRACE_NOTICE (5)) [enabled]
$ lttng disable-event -u sample:message
$ lttng list session
Events:
  sample:message (loglevel: TRACE_NOTICE (5)) [disabled]
$ lttng enable-event -u sample:message
$ lttng list session
Events:
  sample:message [enabled]
$ lttng enable-event -u sample:message --loglevel err
$ lttng list session
Events:
  sample:message [enabled]

In other words, we have a partial ordering on the loglevel key that goes:
LTTNG_EVENT_LOGLEVEL_ALL > LTTNG_EVENT_LOGLEVEL_RANGE(m),
LTTNG_EVENT_LOGLEVEL_ALL > LTTNG_EVENT_LOGLEVEL_SINGLE(n),
LTTNG_EVENT_LOGLEVEL_RANGE(m) > LTTNG_EVENT_LOGLEVEL_SINGLE(n) if m>=n

It is tempting to extend the partial ordering to the text key, but I think we then need to change the interpretation of the rules from a simple "OR":

$ lttng enable-event -u "sample:mes*"
$ lttng list session
Events:
  sample:mes* [enabled]
$ lttng enable-event -u sample:message
$ lttng list session
Events:
  sample:mes* [enabled]
$ lttng disable-event -u sample:message
$ lttng list session
Events:
  sample:mes* [enabled]
  sample:message [disabled]

The user's wish of "enabling sample:mes* except for sample:message" would be achieved if the daemon treated the disabling rules as having priority over the enabling ones.  Instead of having a single list with each list entry having an enabled/disabled attribute, the daemon would need to maintain two lists, one for enabling rules, the other for disabling rules (and the list entries of both lists would be attribute-less).

This approach would dovetail nicely with the proposed exclusion special character (!).  A command like:

$ lttng enable-event -u "sample:mes*" "!sample:message"

Would readily be resolved as an entry in each list.

But then how could we deal with:

$ lttng disable-event -u "sample:mes*" "!sample:message"

(In other words, "disabling sample:mes* except for sample:message")  Maybe we'd need two sets of disabling rules: a weak one (enabling rules have priority) and a strong one (enabling rules don't have priority).  Although that could still be reasonably understandable when displayed with 'lttng list session' if the rules were presented in three ordered blocks (strong disabling rules, enabling rules, weak disabling rules), I suspect the scheme is getting too complex.

This needs more thought.   :-)

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>