From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mathieu Desnoyers Subject: Re: Double free or corruption error (fasttop) Date: Wed, 21 Mar 2018 13:27:17 -0400 (EDT) Message-ID: <268788496.560.1521653237589.JavaMail.zimbra__31013.7883214795$1521653132$gmane$org@efficios.com> References: <173668501.337.1521487906796.JavaMail.zimbra@efficios.com> <1883907448.467.1521489713355.JavaMail.zimbra@efficios.com> <568213680.975.1521562064453.JavaMail.zimbra@efficios.com> <446537616.1725.1521579509437.JavaMail.zimbra@efficios.com> <1870456531.1798.1521582159880.JavaMail.zimbra@efficios.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_556_408660655.1521653237585" Return-path: Received: from mail.efficios.com (mail.efficios.com [167.114.142.138]) by lists.lttng.org (Postfix) with ESMTPS id 405xZQ2PdWzxVB for ; Wed, 21 Mar 2018 13:27:25 -0400 (EDT) In-Reply-To: <1870456531.1798.1521582159880.JavaMail.zimbra@efficios.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" To: Shehab Elsayed Cc: lttng-dev List-Id: lttng-dev@lists.lttng.org ------=_Part_556_408660655.1521653237585 Content-Type: multipart/alternative; boundary="=_421cce1e-7b82-4ae2-ba5c-12e34a0f1321" --=_421cce1e-7b82-4ae2-ba5c-12e34a0f1321 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit ----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers wrote: > ----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers > wrote: >> ----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers >> wrote: >>> ----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed wrote: >>>> I did "echo "-1" > /proc/sys/kernel/perf_event_paranoid " and made sure the >>>> value was actually written by "cat /proc/sys/kernel/perf_event_paranoid" >>>> It executed normally 2 times but then got the same error. >>> Can you provide the output when reproducing the issue with the >>> LTTNG_UST_DEBUG=1 environment variable set when starting >>> your test program ? >> I just noticed something that might explain what goes wrong here: >> lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock(). Pthread >> mutex >> in your case is instrumented with the pthread wrapper. This "add_thread_field" >> is invoked >> the first time the perf counter is hit by each given thread. When this happens, >> the >> instrumented pthread mutex will try to call into the instrumentation tracepoint >> again, >> which will call add_thread_field() (again), and so on until we reach the >> libringbuffer >> "lib_ring_buffer_nesting" threshold of 4 calls deep. >> I suspect this situation where we recursively call add_thread_field is >> unexpected, >> which may trigger your double-free here. >> Will investigate more. > Can you try with the attached patch applied ? Here is an updated v2 of the patch which tests the notrace tls counter sooner (before evaluating filter). Please let me know how it goes. Thanks, Mathieu > Thanks, > Mathieu >> Thanks, >> Mathieu >>> Thanks, >>> Mathieu >>>> Shehab Y. Elsayed, MSc. >>>> PhD Student >>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>>> University of Toronto >>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | >>>> shehabyomn@gmail.com ] >>>> On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers < [ >>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] > >>>> wrote: >>>>> ----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed < [ >>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote: >>>>>> cat /proc/sys/kernel/perf_event_paranoid ---> returns 1 >>>>>> I run the program as a normal user >>>>>> The glibc version I get by running "ldd --version" is 2.17 >>>>> Can you reproduce the issue after you do this as root ? >>>>> echo "-1" > /proc/sys/kernel/perf_event_paranoid >>>>> Based on this documentation of the Linux kernel: >>>>> Documentation/sysctl/kernel.txt: >>>>> perf_event_paranoid: >>>>> Controls use of the performance events system by unprivileged >>>>> users (without CAP_SYS_ADMIN). The default value is 2. >>>>> -1: Allow use of (almost) all events by all users >>>>> Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK >>>>> >=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN >>>>> Disallow raw tracepoint access by users without CAP_SYS_ADMIN >>>>> >=1: Disallow CPU event access by users without CAP_SYS_ADMIN >>>>> >=2: Disallow kernel profiling by users without CAP_SYS_ADMIN >>>>> Thanks, >>>>> Mathieu >>>>>> Shehab Y. Elsayed, MSc. >>>>>> PhD Student >>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>>>>> University of Toronto >>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | >>>>>> shehabyomn@gmail.com ] >>>>>> On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers < [ >>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] > >>>>>> wrote: >>>>>>> ---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers < [ >>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] > >>>>>>> wrote: >>>>>>>> ----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed < [ >>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote: >>>>>>>>> Yes, I tried with only one of those contexts and I still ran into the same >>>>>>>>> problem. >>>>>>>> What is the setting returned by >>>>>>>> cat /proc/sys/kernel/perf_event_paranoid >>>>>>>> on your system ? And do you run your test program as root or normal user ? >>>>>>>> Please CC the mailing list on your reply. >>>>>>> I will also need to know what glibc version you have on your system. >>>>>>> Thanks, >>>>>>> Mathieu >>>>>>>> Thanks, >>>>>>>> Mathieu >>>>>>>>> Shehab Y. Elsayed, MSc. >>>>>>>>> PhD Student >>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>>>>>>>> University of Toronto >>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | >>>>>>>>> shehabyomn@gmail.com ] >>>>>>>>> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [ >>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] > >>>>>>>>> wrote: >>>>>>>>>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [ >>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote: >>>>>>>>>>> Hi Mathieu, >>>>>>>>>>> Thank you very much for your reply. >>>>>>>>>>> I manually built lttng-ust from source (commit #: >>>>>>>>>>> 8a208943e21700211beee3ea64180a5a534c7d2a). >>>>>>>>>>> This is how I set up the tracing session: >>>>>>>>>>> 1- lttng create lu_ncb_8_native -o {path} >>>>>>>>>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req >>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq >>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock >>>>>>>>>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock >>>>>>>>>>> 3- lttng add-context -u -t procname >>>>>>>>>>> lttng add-context -u -t vpid >>>>>>>>>>> lttng add-context -u -t pthread_id >>>>>>>>>>> lttng add-context -u -t vtid >>>>>>>>>>> lttng add-context -u -t ip >>>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles >>>>>>>>>>> lttng add-context -u -t perf:thread:cycles >>>>>>>>>>> lttng add-context -u -t perf:thread:instructions >>>>>>>>>>> 4- lttng start >>>>>>>>>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096 >>>>>>>>>>> -b32 >>>>>>>>>>> 6- lttng stop >>>>>>>>>>> 7- lttng destroy >>>>>>>>>> Can you reproduce if you remove the following contexts ? >>>>>>>>>> lttng add-context -u -t perf:thread:cpu-cycles >>>>>>>>>> lttng add-context -u -t perf:thread:cycles >>>>>>>>>> lttng add-context -u -t perf:thread:instructions >>>>>>>>>> And if you only keep a single of those contexts ? >>>>>>>>>> Thanks, >>>>>>>>>> Mathieu >>>>>>>>>>> Shehab Y. Elsayed, MSc. >>>>>>>>>>> PhD Student >>>>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>>>>>>>>>> University of Toronto >>>>>>>>>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# | >>>>>>>>>>> shehabyomn@gmail.com ] >>>>>>>>>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [ >>>>>>>>>>> mailto:mathieu.desnoyers@efficios.com | mathieu.desnoyers@efficios.com ] > >>>>>>>>>>> wrote: >>>>>>>>>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [ >>>>>>>>>>>> mailto:shehabyomn@gmail.com | shehabyomn@gmail.com ] > wrote: >>>>>>>>>>>>> Hello All, >>>>>>>>>>>>> I am trying to instrument a pthread application using the provided pthread >>>>>>>>>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop >>>>>>>>>>>>> )" error. >>>>>>>>>>>> Please provide more information about the version of lttng-ust you are using, >>>>>>>>>>>> and how you setup >>>>>>>>>>>> your tracing session. >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Mathieu >>>>>>>>>>>>> Here is a description of what I have tried and noticed: >>>>>>>>>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as >>>>>>>>>>>>> expected. >>>>>>>>>>>>> 2- From my experiments, the problem happens (more frequently at least) when >>>>>>>>>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and >>>>>>>>>>>>> instructions). >>>>>>>>>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _ >>>>>>>>>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents. >>>>>>>>>>>>> 4- Here is the backtrace printed after exiting: >>>>>>>>>>>>> ======= Backtrace : ========= >>>>>>>>>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited] >>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng >>>>>>>>>>>>> _destroy_context+0x35)[0x7ffff7471575] >>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng >>>>>>>>>>>>> _session_destroy+0x21c)[0x7ffff747363c] >>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906] >>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>>>>>>>>> +0x9f)[0x7ffff746dccf] >>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>>>>>>>>> +0x9f)[0x7ffff746dccf] >>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref >>>>>>>>>>>>> +0x9f)[0x7ffff746dccf] >>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi >>>>>>>>>>>>> _exit+0x68)[0x7ffff746ead8] >>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3] >>>>>>>>>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57] >>>>>>>>>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a] >>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49] >>>>>>>>>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95] >>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51] >>>>>>>>>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35] >>>>>>>>>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44] >>>>>>>>>>>>> 5- Also, this is a backtrace I obtained from gdb : >>>>>>>>>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6 >>>>>>>>>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6 >>>>>>>>>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6 >>>>>>>>>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6 >>>>>>>>>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field ( >>>>>>>>>>>>> field=) at lttng -context- perf -counters.c:418 >>>>>>>>>>>>> #5 0x00007ffff767a575 in lttng _destroy_context ( >>>>>>>>>>>>> ctx =0x7ffff0011090) at lttng -context.c:278 >>>>>>>>>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap ( >>>>>>>>>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172 >>>>>>>>>>>>> #7 lttng _session_destroy (session=0x7ffff0000900) >>>>>>>>>>>>> at lttng -events.c:247 >>>>>>>>>>>>> #8 0x00007ffff7676906 in lttng _release_session ( >>>>>>>>>>>>> objd =) at lttng - ust - abi .c:601 >>>>>>>>>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1, >>>>>>>>>>>>> is_owner=) at lttng - ust - abi .c:216 >>>>>>>>>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2, >>>>>>>>>>>>> is_owner=) at lttng - ust - abi .c:216 >>>>>>>>>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id@entry=18, >>>>>>>>>>>>> is_owner=is_owner@entry=1) at lttng - ust - abi .c:216 >>>>>>>>>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy () >>>>>>>>>>>>> at lttng - ust - abi .c:235 >>>>>>>>>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002 >>>>>>>>>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1) >>>>>>>>>>>>> at lttng - ust -comm.c:1807 >>>>>>>>>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit () >>>>>>>>>>>>> at lttng - ust -comm.c:1874 >>>>>>>>>>>>> #16 0x00007ffff7dec85a in _ dl _ fini () >>>>>>>>>>>>> from /lib64/ ld - linux -x86-64.so.2 >>>>>>>>>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers () >>>>>>>>>>>>> from /lib64/ libc .so.6 >>>>>>>>>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6 >>>>>>>>>>>>> #19 0x0000000000401b51 in main ( argc =, >>>>>>>>>>>>> argv =) at lu .c:368 >>>>>>>>>>>>> Any ideas, why this happens and how to fix it? >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Shehab >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> lttng-dev mailing list >>>>>>>>>>>>> [ mailto:lttng-dev@lists.lttng.org | lttng-dev@lists.lttng.org ] >>>>>>>>>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | >>>>>>>>>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] >>>>>>>>>>>> -- >>>>>>>>>>>> Mathieu Desnoyers >>>>>>>>>>>> EfficiOS Inc. >>>>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>>>>>>>>> -- >>>>>>>>>> Mathieu Desnoyers >>>>>>>>>> EfficiOS Inc. >>>>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>>>>>>> -- >>>>>>>> Mathieu Desnoyers >>>>>>>> EfficiOS Inc. >>>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>>>>>> -- >>>>>>> Mathieu Desnoyers >>>>>>> EfficiOS Inc. >>>>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>>>> -- >>>>> Mathieu Desnoyers >>>>> EfficiOS Inc. >>>>> [ http://www.efficios.com/ | http://www.efficios.com ] >>> -- >>> Mathieu Desnoyers >>> EfficiOS Inc. >>> http://www.efficios.com >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> http://www.efficios.com > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com --=_421cce1e-7b82-4ae2-ba5c-12e34a0f1321 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers <mathieu.desnoy= ers@efficios.com> wrote:
----- On Mar 20, 2018, at 4:58 PM, Mathieu Des= noyers <mathieu.desnoyers@efficios.com> wrote:
<= blockquote style=3D"border-left:2px solid #1010FF;margin-left:5px;padding-l= eft:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:non= e;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">
----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers= <mathieu.desnoyers@efficios.com> wrote:

----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <s= hehabyomn@gmail.com> wrote:
I did "echo "-1" = > /proc/sys/kernel/p= erf_event_paranoid" and made sure the value was actually written by = "cat /proc/sys/kernel/perf_event_paranoid"

It executed no= rmally 2 times but then got the same error.

Can you provide the output when reproducing the issue with the
<= div>LTTNG_UST_DEBUG=3D1 environment variable set when starting
your test program ?
I just noticed som= ething that might explain what goes wrong here:

lttng-con= text-perf-counters.c: add_thread_field() grabs the ust_lock(). Pthread mute= x
in your case is instrumented with the pthread wrapper. This= "add_thread_field" is invoked
the first time the perf counte= r is hit by each given thread. When this happens, the
instrum= ented pthread mutex will try to call into the instrumentation tracepoint ag= ain,
which will call add_thread_field() (again), and so on un= til we reach the libringbuffer
"lib_ring_buffer_nesting" thre= shold of 4 calls deep.

I suspect this situation where we = recursively call add_thread_field is unexpected,
which may tr= igger your double-free here.

Will investigate more.
=
Can you try with the attached patch applied ?=
Here is an updated v2 of the patch whic= h tests the notrace tls counter sooner (before evaluating
fil= ter). Please let me know how it goes.
Thanks,

Mathieu





Thanks,

Mathieu





Thanks,

Mathieu




Thanks,

Mathieu




Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer E= ngineering
University of Toronto =
E-mail: shehabyomn@gmail.com

On Mon, Mar 19, 2018 at 4:01 P= M, Mathieu Desnoyers <mathieu.desnoyers@efficios.com><= /span> wrote:


----- On Mar 19, 2018, a= t 3:53 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote:
cat /p= roc/sys/kernel/perf_event_paranoid ---> returns 1

I = run the program as a normal user

The glibc version I get by ru= nning "ldd --version" is 2.17

Can you rep= roduce the issue after you do this as root ?

echo "-1" &g= t; /proc/sys/kernel/perf_even= t_paranoid

Based on this documentation of the Linu= x kernel:

Documentation/sysctl/kernel.txt:

<= div>perf_event_paranoid:

Controls use of the performance events syst= em by unprivileged
users (without CAP_SYS_ADMIN).  The default valu= e is 2.

 -1: Allow use of (almost) all events by all users
&= nbsp;    Ignore mlock limit after perf_event_mlock_kb withou= t CAP_IPC_LOCK
>=3D0: Disallow ftrace function tracepoint by users wi= thout CAP_SYS_ADMIN
     Disallow raw tracepoint acc= ess by users without CAP_SYS_ADMIN
>=3D1: Disallow CPU event access b= y users without CAP_SYS_ADMIN
>=3D2: Disallow kernel profiling by use= rs without CAP_SYS_ADMIN

Thanks,

Mathieu


=
=
Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer E= ngineering
University of Toronto E-mail: shehabyomn@gmail.com

On Mon, Mar 19, 2018 at 3:31 P= M, Mathieu Desnoyers <mathieu.desnoyers@efficios.com><= /span> wrote:
---- On Mar 19, 2= 018, at 3:26 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrot= e:
----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <shehabyomn@gmail.co= m> wrote:
Yes, I tried with only one of those c= ontexts and I still ran into the same problem.
What = is the setting returned by

cat /proc/sys/kernel/perf_even= t_paranoid

on your system ? And do you run your test prog= ram as root or normal user ?

Please CC the mailing list o= n your reply.

I will also nee= d to know what glibc version you have on your system.

Tha= nks,

Mathieu


=


Thanks,

Mathieu


Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer E= ngineering
University of Toronto E-mail: shehabyomn@gmail.com

On Mon, Mar 19, 2018 at 2:24 P= M, Mathieu Desnoyers <mathieu.desnoyers@efficios.com><= /span> wrote:
----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed <shehabyomn@gmail.com> wro= te:
Hi= Mathieu,

Thank you very much for your reply.

I manually buil= t lttng-ust from source (commit #: 8a208943e21700211beee3ea64180a5a534c7d2a= ).

This is how I set up the tracing session:
1- lttng= create lu_ncb_8_native -o {path}
2- lttng enable-event --userspac= e lttng_ust_pthread:pthread_mutex_lock_req
    lttng enab= le-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
 &nbs= p;  lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_loc= k_trylock
    lttng enable-event --userspace lttng_ust_pt= hread:pthread_mutex_lock_unlock
3- lttng add-context -u -t procnam= e
    lttng add-context -u -t vpid
    = lttng add-context -u -t pthread_id
    lttng add-contex= t -u -t vtid
    lttng add-context -u -t ip
 &nb= sp;  lttng add-context -u -t perf:thread:cpu-cycles
  &n= bsp; lttng add-context -u -t perf:thread:cycles
    ltt= ng add-context -u -t perf:thread:instructions
4- lttng start
5-= LD_PRELOAD=3D/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -= n8096 -b32
6- lttng stop
7- lttng destroy

Can you reproduce if you remove the following contexts ?=

    lttng add-context -u -t perf:th= read:cpu-cycles
    lttng add-context -u -t perf:thread:c= ycles
    lttng add-context -u -t perf:thread:instruction= s

And if you only keep a single of those contexts ?

Thanks,

Mathieu






Shehab Y. Elsayed, MSc.
PhD Student
The Edwards S. Rogers Sr. Dept. of Electrical and Computer E= ngineering
University of Toronto E-mail: shehabyomn@gmail.com

On Mon, Mar 19, 2018 at 12:21 = PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>= wrote:

----- On Mar 16, 2018, at 5:37 PM, Sheh= ab Elsayed <sh= ehabyomn@gmail.com> wrote:
Hello All,

I am try= ing to instrument a pthread applicati= on using the provided pthread wrapper= , but I sometimes run into a "Double free or corruption error (fasttop)" error.

Please provide mor= e information about the version of lttng-ust you are using, and how you set= up
your tracing session.

Thanks,
<= br>
Mathieu



Here is a description of what I have tried and not= iced:
1- The problem isn't consistent. It sometimes happen and som= etimes works as expected.
2- From my experiments, the problem happ= ens (more frequently at least) when adding performance counter contexts (I = tried cycles, cpu_cycles and instruct= ions).
3- I am testing using lu_ncb from splash3 benchmark suite = after setting LD_PRELOAD to use the pthread= wrapper as described in the LTTng documents.
4- Here is the ba= cktrace printed after exiting:
=3D=3D=3D=3D=3D=3D=3D Backtrace: =3D=3D=3D=3D=3D=3D=3D=3D=3D
/lib64= /libc.so.6([Thread 0x7ffff5611700 (<= span id=3D"m_5949702530543711213m_4191731141621744608m_-2379611272366386122= m_-3928334406880738982:3qh.14">LWP 97229) exited]
/usr/local/lib/libl= ttng-ust.so.0(lttng_destroy_context+0x35)[0x7ffff7471575]
/usr/local/lib/liblttng-ust.so.0(lttng_session_destroy+0x21c)[0x7ffff7473= 63c]
/usr/local/lib/liblttng-ust.so.0(+0x1e906)[0x7ffff746d906]
/usr/local/lib/liblttng-ust.so.0(lttng<= /span>_ust_objd_unref+0x9f)[0x7ff= ff746dccf]
/usr/local/lib/liblttng-us= t.so.0(lttng_ust_objd_unref+0x9f)[0x7ffff746dccf]
/usr/local/lib/ust.so.0(lttng_ust_objd_unref+0x9f)[0x7ffff746dccf]
/liblttng-ust.so.0(lttng_ust_abi_exit+0x68)[0x7ffff746ead8]
/usr/local/lib/liblttng<= /span>-ust.so.0(+0x191d3)[0x7ffff746= 81d3]
/usr/local/lib/liblttng-ust.so.0(lttng_ust_exit+0x67)[0x7ffff745ed57]
/lib64/ld-linux-x86-= 64.so.2(+0xf85a)[0x7ffff7dec85a]
/lib64/= libc.so.6(+0x38a49)[0x7ffff6ca6a49]
/lib64/libc.so.6(+0x38a95)[0x7ffff6ca6a95]
/aenao-99/elsayed9/LTTn= g/data/scripts/tmp/lu_ncb[0x401b51]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff6c8fb35]<= br>/aenao-99/elsayed9/LTTng/data/scripts/tmp/lu_ncb[0x401c44]
5- Also, this is a backtrace I obtained from gdb:
#0  0x00007ffff6eac1d7 in raise () from /lib64/<= span id=3D"m_5949702530543711213m_4191731141621744608m_-2379611272366386122= m_-3928334406880738982:3qh.79">libc.so.6
#1  0x00007ffff6ead= 8c8 in abort () from /lib64/libc.so.= 6
#2  0x00007ffff6eebf07 in __libc<= /span>_message () from /lib64/libc.s= o.6
#3  0x00007ffff6ef3503 in _int_free () from /lib64/libc.so.6
#4  0x00007ffff768ad25 in lttng_destroy_perf_counter_field (
    field=3D<optimiz= ed out>) at lttng-context-perf-counters.c:418
#5  0x00007ffff= 767a575 in lttng_destroy_context (ctx=3D0x7ffff0011090) at lttng-context.c:278
#6  0x00007ffff7= 67c63c in _lttng_channel_unmap (
lttn= g_chan=3D0x7ffff0010f40) at <= span id=3D"m_5949702530543711213m_4191731141621744608m_-2379611272366386122= m_-3928334406880738982:3qh.95">lttng-events.c:172
#7  lttng_session_destroy (session=3D0x7ffff00= 00900)
    at lttng-events.c:247
#8  0x00007ffff7676906 in lttng_release_session (
obj= d=3D<optimized out>) at lttng= -ust-abi.c:601
#9  0x00007ffff7676ccf in lttng_ust= _objd_unref (id=3D1,
    is_owner=3D<optimized o= ut>) at lttng-ust-abi.c:216=
#10 0x00007ffff7676ccf in lttng= _ust_objd_unref (id=3D2,
&= nbsp;   is_owner=3D<optimized out>) at lttng-ust-abi.c:216
#11 0x00007ffff7676ccf in <= span id=3D"m_5949702530543711213m_4191731141621744608m_-2379611272366386122= m_-3928334406880738982:3qh.117">lttng_ust_objd_unref (id=3Did@entry=3D18,
    i= s_owner=3Dis_owner@entry=3D1) at lttng-ust-abi.c:216
#12 0x00007ffff7677ad8 in objd_table_destroy ()
    at lttng-ust-abi.c:235
#13 lttng_ust= _abi_exit () at lttng-ust-abi.c:1002
#14 0x00007ffff76711d3 in= lttng_ust_cleanup (exiting=3D1)
    at lttng-ust-comm.c:1807
#15 0x00007ffff7667d57 in ust_exit ()
 = ;   at lttng-ust-comm.c:1874
#16 0x00007ffff7dec85a in _<= span id=3D"m_5949702530543711213m_4191731141621744608m_-2379611272366386122= m_-3928334406880738982:3qh.142">dl_= fini ()
   from /lib64/ld-linux-x86-64.so.2
#17= 0x00007ffff6eafa49 in __run_exit_handlers ()
   from /lib64/<= span id=3D"m_5949702530543711213m_4191731141621744608m_-2379611272366386122= m_-3928334406880738982:3qh.146">libc.so.6
#18 0x00007ffff6eafa95 = in exit () from /lib64/libc.so.6#19 0x0000000000401b51 in main (argc=3D<optimized out>,
argv=3D<optimized out>) at lu.c:= 368

Any ideas, why this happens and how to fix it?

Thanks,
Shehab
<= div>



_______= ________________________________________
lttng-dev mailing list
lttng-dev@lists.l= ttng.org
https://lists.lttng.org/cgi-bin/mailman/listi= nfo/lttng-dev

-- =
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

<= /div>

<= div>--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.=
http://www.effici= os.com


--
<= div>Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
<= /div>

--
Mathieu DesnoyersEfficiOS Inc.
http://www.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.
http://ww= w.efficios.com


--
Mathieu Desnoyers
EfficiOS Inc.=
http://www.efficios.com
--=_421cce1e-7b82-4ae2-ba5c-12e34a0f1321-- ------=_Part_556_408660655.1521653237585 Content-Type: text/x-patch; name=0001-Fix-perf-event-mutex-with-pthread-wrapper.patch Content-Disposition: attachment; filename=0001-Fix-perf-event-mutex-with-pthread-wrapper.patch Content-Transfer-Encoding: base64 RnJvbSA4NjcxYzZhZTgzOTYxNWMwNzc5MDQxYWJhZGE0NzAzNjgwZmJjMDAzIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBNYXRoaWV1IERlc25veWVycyA8bWF0aGlldS5kZXNub3llcnNA ZWZmaWNpb3MuY29tPgpEYXRlOiBUdWUsIDIwIE1hciAyMDE4IDE3OjMyOjM2IC0wNDAwClN1Ympl Y3Q6IFtSRkMgUEFUQ0ggdjJdIEZpeDogcGVyZiBldmVudCBtdXRleCB3aXRoIHB0aHJlYWQgd3Jh cHBlcgoKV2UgZG8gbm90IHdhbnQgdG8gcmVjdXJzZSBpbiB0aGUgcHRocmVhZCBtdXRleCBpbnN0 cnVtZW50YXRpb24gd2hlbgpzZXR0aW5nIHVwIHRoZSBwZXJmIGNvdW50ZXJzIGZvciBhIGdpdmVu IHRocmVhZC4KCkludHJvZHVjZSBhICJub3RyYWNlIiBwZXItdGhyZWFkIGNvdW50ZXIgdG8gaW5o aWJpdCB0cmFjaW5nIGZvciB0aGUKY3VycmVudCB0aHJlYWQuCgpTaWduZWQtb2ZmLWJ5OiBNYXRo aWV1IERlc25veWVycyA8bWF0aGlldS5kZXNub3llcnNAZWZmaWNpb3MuY29tPgotLS0KIGluY2x1 ZGUvbHR0bmcvdXN0LWV2ZW50cy5oICAgICAgICAgICAgICAgICB8ICA2ICsrKysrKwogaW5jbHVk ZS9sdHRuZy91c3QtdHJhY2Vwb2ludC1ldmVudC5oICAgICAgIHwgIDcgKystLS0tLQogbGlibHR0 bmctdXN0L2x0dG5nLWNvbnRleHQtcGVyZi1jb3VudGVycy5jIHwgIDIgKysKIGxpYmx0dG5nLXVz dC9sdHRuZy1ldmVudHMuYyAgICAgICAgICAgICAgICB8IDE0ICsrKysrKysrKysrKysrCiBsaWJs dHRuZy11c3QvbHR0bmctdHJhY2VyLWNvcmUuaCAgICAgICAgICAgfCAgMyArKysKIGxpYmx0dG5n LXVzdC9sdHRuZy11c3QtY29tbS5jICAgICAgICAgICAgICB8IDIzICsrKysrKysrKysrKysrKysr KysrKysrCiA2IGZpbGVzIGNoYW5nZWQsIDUwIGluc2VydGlvbnMoKyksIDUgZGVsZXRpb25zKC0p CgpkaWZmIC0tZ2l0IGEvaW5jbHVkZS9sdHRuZy91c3QtZXZlbnRzLmggYi9pbmNsdWRlL2x0dG5n L3VzdC1ldmVudHMuaAppbmRleCA4NjczMzUwMy4uZjhjMTMwZjkgMTAwNjQ0Ci0tLSBhL2luY2x1 ZGUvbHR0bmcvdXN0LWV2ZW50cy5oCisrKyBiL2luY2x1ZGUvbHR0bmcvdXN0LWV2ZW50cy5oCkBA IC03MzgsNiArNzM4LDEyIEBAIHN0cnVjdCBsdHRuZ19lbnVtICpsdHRuZ191c3RfZW51bV9nZXRf ZnJvbV9kZXNjKHN0cnVjdCBsdHRuZ19zZXNzaW9uICpzZXNzaW9uLAogdm9pZCBsdHRuZ191c3Rf ZGxfdXBkYXRlKHZvaWQgKmlwKTsKIHZvaWQgbHR0bmdfdXN0X2ZpeHVwX2ZkX3RyYWNrZXJfdGxz KHZvaWQpOwogCit2b2lkIGx0dG5nX3VzdF9iZWdpbl9ub3RyYWNlKHZvaWQpOwordm9pZCBsdHRu Z191c3RfZW5kX25vdHJhY2Uodm9pZCk7CisKK2ludCBsdHRuZ191c3RfZG9fdHJhY2Uoc3RydWN0 IGx0dG5nX3Nlc3Npb24gKnNlc3Npb24sCisJCXN0cnVjdCBsdHRuZ19jaGFubmVsICpjaGFuLCBz dHJ1Y3QgbHR0bmdfZXZlbnQgKmV2ZW50KTsKKwogLyogRm9yIGJhY2t3YXJkIGNvbXBhdGliaWxp dHkuIExlYXZlIHRob3NlIGV4cG9ydGVkIHN5bWJvbHMgaW4gcGxhY2UuICovCiBleHRlcm4gc3Ry dWN0IGx0dG5nX2N0eCAqbHR0bmdfc3RhdGljX2N0eDsKIHZvaWQgbHR0bmdfY29udGV4dF9pbml0 KHZvaWQpOwpkaWZmIC0tZ2l0IGEvaW5jbHVkZS9sdHRuZy91c3QtdHJhY2Vwb2ludC1ldmVudC5o IGIvaW5jbHVkZS9sdHRuZy91c3QtdHJhY2Vwb2ludC1ldmVudC5oCmluZGV4IGVjMjkyZDI0Li4w YmMwMDJmNyAxMDA2NDQKLS0tIGEvaW5jbHVkZS9sdHRuZy91c3QtdHJhY2Vwb2ludC1ldmVudC5o CisrKyBiL2luY2x1ZGUvbHR0bmcvdXN0LXRyYWNlcG9pbnQtZXZlbnQuaApAQCAtNzY2LDExICs3 NjYsOCBAQCB2b2lkIF9fZXZlbnRfcHJvYmVfXyMjX3Byb3ZpZGVyIyNfX18jI19uYW1lKF9UUF9B UkdTX0RBVEFfUFJPVE8oX2FyZ3MpKQkgICAgICBcCiAJCSh2b2lkKSBfX2R5bmFtaWNfbGVuX2lk eDsJLyogZG9uJ3Qgd2FybiBpZiB1bnVzZWQgKi8gICAgXAogCWlmICghX1RQX1NFU1NJT05fQ0hF Q0soc2Vzc2lvbiwgX19jaGFuLT5zZXNzaW9uKSkJCSAgICAgIFwKIAkJcmV0dXJuOwkJCQkJCQkg ICAgICBcCi0JaWYgKGNhYV91bmxpa2VseSghQ01NX0FDQ0VTU19PTkNFKF9fY2hhbi0+c2Vzc2lv bi0+YWN0aXZlKSkpCSAgICAgIFwKLQkJcmV0dXJuOwkJCQkJCQkgICAgICBcCi0JaWYgKGNhYV91 bmxpa2VseSghQ01NX0FDQ0VTU19PTkNFKF9fY2hhbi0+ZW5hYmxlZCkpKQkJICAgICAgXAotCQly ZXR1cm47CQkJCQkJCSAgICAgIFwKLQlpZiAoY2FhX3VubGlrZWx5KCFDTU1fQUNDRVNTX09OQ0Uo X19ldmVudC0+ZW5hYmxlZCkpKQkJICAgICAgXAorCWlmIChjYWFfdW5saWtlbHkoIWx0dG5nX3Vz dF9kb190cmFjZShfX2NoYW4tPnNlc3Npb24sIF9fY2hhbiwJICAgICAgXAorCQkJCQkgICAgIF9f ZXZlbnQpKSkJCQkgICAgICBcCiAJCXJldHVybjsJCQkJCQkJICAgICAgXAogCWlmIChjYWFfdW5s aWtlbHkoIVRQX1JDVV9MSU5LX1RFU1QoKSkpCQkJCSAgICAgIFwKIAkJcmV0dXJuOwkJCQkJCQkg ICAgICBcCmRpZmYgLS1naXQgYS9saWJsdHRuZy11c3QvbHR0bmctY29udGV4dC1wZXJmLWNvdW50 ZXJzLmMgYi9saWJsdHRuZy11c3QvbHR0bmctY29udGV4dC1wZXJmLWNvdW50ZXJzLmMKaW5kZXgg YTE1NDE3Y2MuLjEzMWFhYTU0IDEwMDY0NAotLS0gYS9saWJsdHRuZy11c3QvbHR0bmctY29udGV4 dC1wZXJmLWNvdW50ZXJzLmMKKysrIGIvbGlibHR0bmctdXN0L2x0dG5nLWNvbnRleHQtcGVyZi1j b3VudGVycy5jCkBAIC0yOTEsNiArMjkxLDcgQEAgc3RydWN0IGx0dG5nX3BlcmZfY291bnRlcl90 aHJlYWRfZmllbGQgKgogCXJldCA9IHB0aHJlYWRfc2lnbWFzayhTSUdfQkxPQ0ssICZuZXdtYXNr LCAmb2xkbWFzayk7CiAJaWYgKHJldCkKIAkJYWJvcnQoKTsKKwlsdHRuZ191c3RfYmVnaW5fbm90 cmFjZSgpOwogCS8qIENoZWNrIGFnYWluIHdpdGggc2lnbmFscyBkaXNhYmxlZCAqLwogCWNkc19s aXN0X2Zvcl9lYWNoX2VudHJ5X3JjdSh0aHJlYWRfZmllbGQsICZwZXJmX3RocmVhZC0+cmN1X2Zp ZWxkX2xpc3QsCiAJCQlyY3VfZmllbGRfbm9kZSkgewpAQCAtMzE1LDYgKzMxNiw3IEBAIHN0cnVj dCBsdHRuZ19wZXJmX2NvdW50ZXJfdGhyZWFkX2ZpZWxkICoKIAkJCSZwZXJmX2ZpZWxkLT50aHJl YWRfZmllbGRfbGlzdCk7CiAJdXN0X3VubG9jaygpOwogc2tpcDoKKwlsdHRuZ191c3RfZW5kX25v dHJhY2UoKTsKIAlyZXQgPSBwdGhyZWFkX3NpZ21hc2soU0lHX1NFVE1BU0ssICZvbGRtYXNrLCBO VUxMKTsKIAlpZiAocmV0KQogCQlhYm9ydCgpOwpkaWZmIC0tZ2l0IGEvbGlibHR0bmctdXN0L2x0 dG5nLWV2ZW50cy5jIGIvbGlibHR0bmctdXN0L2x0dG5nLWV2ZW50cy5jCmluZGV4IDI1NWM0Yjk1 Li5kOGU0ZmQ3OSAxMDA2NDQKLS0tIGEvbGlibHR0bmctdXN0L2x0dG5nLWV2ZW50cy5jCisrKyBi L2xpYmx0dG5nLXVzdC9sdHRuZy1ldmVudHMuYwpAQCAtMTI4NCwzICsxMjg0LDE3IEBAIHZvaWQg bHR0bmdfdXN0X2NvbnRleHRfc2V0X3Nlc3Npb25fcHJvdmlkZXIoY29uc3QgY2hhciAqbmFtZSwK IAkJfQogCX0KIH0KKworaW50IGx0dG5nX3VzdF9kb190cmFjZShzdHJ1Y3QgbHR0bmdfc2Vzc2lv biAqc2Vzc2lvbiwKKwkJc3RydWN0IGx0dG5nX2NoYW5uZWwgKmNoYW4sIHN0cnVjdCBsdHRuZ19l dmVudCAqZXZlbnQpCit7CisJaWYgKGNhYV91bmxpa2VseSghQ01NX0FDQ0VTU19PTkNFKHNlc3Np b24tPmFjdGl2ZSkpKQorCQlyZXR1cm4gMDsKKwlpZiAoY2FhX3VubGlrZWx5KCFDTU1fQUNDRVNT X09OQ0UoY2hhbi0+ZW5hYmxlZCkpKQorCQlyZXR1cm4gMDsKKwlpZiAoY2FhX3VubGlrZWx5KCFD TU1fQUNDRVNTX09OQ0UoZXZlbnQtPmVuYWJsZWQpKSkKKwkJcmV0dXJuIDA7CisJaWYgKGNhYV91 bmxpa2VseShVUkNVX1RMUyhsdHRuZ191c3Rfbm90cmFjZV90aHJlYWQpKSA+IDApCisJCXJldHVy biAwOworCXJldHVybiAxOworfQpkaWZmIC0tZ2l0IGEvbGlibHR0bmctdXN0L2x0dG5nLXRyYWNl ci1jb3JlLmggYi9saWJsdHRuZy11c3QvbHR0bmctdHJhY2VyLWNvcmUuaAppbmRleCBiYTIzMmYz Mi4uOThkNzliZTggMTAwNjQ0Ci0tLSBhL2xpYmx0dG5nLXVzdC9sdHRuZy10cmFjZXItY29yZS5o CisrKyBiL2xpYmx0dG5nLXVzdC9sdHRuZy10cmFjZXItY29yZS5oCkBAIC0yNSw2ICsyNSw3IEBA CiAjaW5jbHVkZSA8c3RkZGVmLmg+CiAjaW5jbHVkZSA8dXJjdS9hcmNoLmg+CiAjaW5jbHVkZSA8 dXJjdS9saXN0Lmg+CisjaW5jbHVkZSA8dXJjdS90bHMtY29tcGF0Lmg+CiAjaW5jbHVkZSA8bHR0 bmcvdXN0LXRyYWNlci5oPgogI2luY2x1ZGUgPGx0dG5nL2J1Zy5oPgogI2luY2x1ZGUgPGx0dG5n L3JpbmdidWZmZXItY29uZmlnLmg+CkBAIC0zNyw2ICszOCw4IEBAIHN0cnVjdCBsdHRuZ19jdHhf ZmllbGQ7CiBzdHJ1Y3QgbHR0bmdfdXN0X2xpYl9yaW5nX2J1ZmZlcl9jdHg7CiBzdHJ1Y3QgbHR0 bmdfY3R4X3ZhbHVlOwogCitleHRlcm4gREVDTEFSRV9VUkNVX1RMUyhpbnQsIGx0dG5nX3VzdF9u b3RyYWNlX3RocmVhZCk7CisKIGludCB1c3RfbG9jayh2b2lkKSBfX2F0dHJpYnV0ZV9fICgod2Fy bl91bnVzZWRfcmVzdWx0KSk7CiB2b2lkIHVzdF9sb2NrX25vY2hlY2sodm9pZCk7CiB2b2lkIHVz dF91bmxvY2sodm9pZCk7CmRpZmYgLS1naXQgYS9saWJsdHRuZy11c3QvbHR0bmctdXN0LWNvbW0u YyBiL2xpYmx0dG5nLXVzdC9sdHRuZy11c3QtY29tbS5jCmluZGV4IGQ0YWRkMWMwLi45NTIwZjI0 NiAxMDA2NDQKLS0tIGEvbGlibHR0bmctdXN0L2x0dG5nLXVzdC1jb21tLmMKKysrIGIvbGlibHR0 bmctdXN0L2x0dG5nLXVzdC1jb21tLmMKQEAgLTkyLDYgKzkyLDkgQEAgc3RhdGljIHB0aHJlYWRf bXV0ZXhfdCB1c3RfbXV0ZXggPSBQVEhSRUFEX01VVEVYX0lOSVRJQUxJWkVSOwogLyogQWxsb3cg bmVzdGluZyB0aGUgdXN0X211dGV4IHdpdGhpbiB0aGUgc2FtZSB0aHJlYWQuICovCiBzdGF0aWMg REVGSU5FX1VSQ1VfVExTKGludCwgdXN0X211dGV4X25lc3QpOwogCisvKiBEbyBub3QgdHJhY2Ug ZXZlbnRzIGZvciB0aGUgY3VycmVudCB0aHJlYWQuICovCitERUZJTkVfVVJDVV9UTFMoaW50LCBs dHRuZ191c3Rfbm90cmFjZV90aHJlYWQpIF9fYXR0cmlidXRlX18oKHZpc2liaWxpdHkoImhpZGRl biIpKSk7CisKIC8qCiAgKiB1c3RfZXhpdF9tdXRleCBwcm90ZWN0cyB0aHJlYWRfYWN0aXZlIHZh cmlhYmxlIHdydCB0aHJlYWQgZXhpdC4gSXQKICAqIGNhbm5vdCBiZSBkb25lIGJ5IHVzdF9tdXRl eCBiZWNhdXNlIHB0aHJlYWRfY2FuY2VsKCksIHdoaWNoIHRha2VzIGFuCkBAIC0xMjEsNiArMTI0 LDE5IEBAIHN0YXRpYyBpbnQgbHR0bmdfdXN0X2NvbW1fc2hvdWxkX3F1aXQ7CiBpbnQgbHR0bmdf dXN0X2xvYWRlZCBfX2F0dHJpYnV0ZV9fKCh3ZWFrKSk7CiAKIC8qCisgKiBJbmhpYml0IGx0dG5n LXVzdCB0cmFjaW5nIGZvciB0aGlzIHRocmVhZC4KKyAqLwordm9pZCBsdHRuZ191c3RfYmVnaW5f bm90cmFjZSh2b2lkKQoreworCVVSQ1VfVExTKGx0dG5nX3VzdF9ub3RyYWNlX3RocmVhZCkrKzsK K30KKwordm9pZCBsdHRuZ191c3RfZW5kX25vdHJhY2Uodm9pZCkKK3sKKwktLVVSQ1VfVExTKGx0 dG5nX3VzdF9ub3RyYWNlX3RocmVhZCk7Cit9CisKKy8qCiAgKiBSZXR1cm4gMCBvbiBzdWNjZXNz LCAtMSBpZiBzaG91bGQgcXVpdC4KICAqIFRoZSBsb2NrIGlzIHRha2VuIGluIGJvdGggY2FzZXMu CiAgKiBTaWduYWwtc2FmZS4KQEAgLTM5Miw2ICs0MDgsMTIgQEAgdm9pZCBsdHRuZ19maXh1cF91 c3RfbXV0ZXhfbmVzdF90bHModm9pZCkKIAlhc20gdm9sYXRpbGUgKCIiIDogOiAibSIgKFVSQ1Vf VExTKHVzdF9tdXRleF9uZXN0KSkpOwogfQogCitzdGF0aWMKK3ZvaWQgbHR0bmdfZml4dXBfbHR0 bmdfdXN0X25vdHJhY2VfdGxzKHZvaWQpCit7CisJYXNtIHZvbGF0aWxlICgiIiA6IDogIm0iIChV UkNVX1RMUyhsdHRuZ191c3Rfbm90cmFjZV90aHJlYWQpKSk7Cit9CisKIC8qCiAgKiBGaXh1cCB1 cmN1IGJwIFRMUy4KICAqLwpAQCAtNDEwLDYgKzQzMiw3IEBAIHZvaWQgbHR0bmdfdXN0X2ZpeHVw X3Rscyh2b2lkKQogCWx0dG5nX2ZpeHVwX25lc3RfY291bnRfdGxzKCk7CiAJbHR0bmdfZml4dXBf cHJvY25hbWVfdGxzKCk7CiAJbHR0bmdfZml4dXBfdXN0X211dGV4X25lc3RfdGxzKCk7CisJbHR0 bmdfZml4dXBfbHR0bmdfdXN0X25vdHJhY2VfdGxzKCk7CiAJbHR0bmdfdXN0X2ZpeHVwX2ZkX3Ry YWNrZXJfdGxzKCk7CiB9CiAKLS0gCjIuMTEuMAoK ------=_Part_556_408660655.1521653237585 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbHR0bmctZGV2 IG1haWxpbmcgbGlzdApsdHRuZy1kZXZAbGlzdHMubHR0bmcub3JnCmh0dHBzOi8vbGlzdHMubHR0 bmcub3JnL2NnaS1iaW4vbWFpbG1hbi9saXN0aW5mby9sdHRuZy1kZXYK ------=_Part_556_408660655.1521653237585--