Hi, I installed KeKriek version because it is the one recommanded by the apt-get of Debian Buster 10.9 (LTTng 2.10). Debian -- Détails du paquet lttng-tools dans buster I tried multiple times to install LTTng 2.12 without success, so I stayed in LTTng 2.10. Find below the output of : lttng-sessiond -vvv --verbose-consumer https://paste.ubuntu.com/p/NrtyCqhHZc/ (I have got the same with LTTNG_CONSUMERD32_BIN=/usr/local/lib32/lttng/libexec/lttng-consumerd LTTNG_CONSUMERD32_LIBDIR=/usr/local/lib32 lttng-sessiond -vvv --verbose-consumer) Yes, I had the issue below few days ago with LTTng 2.12 "For 2.12 there is some problems when compiling only lttng-consumerd (fd-tracker lib and libcommon problem)." Thank you ! The following modification is working : lttng-sessiond --consumerd32-libdir=/usr/local/lib32 --consumerd32-path=/usr/local/lib32/lttng/libexec/lttng-consumerd --daemonize Regards, Julien ________________________________ De : Jonathan Rajotte-Julien Envoyé : lundi 7 juin 2021 17:24 À : MONTET Julien Cc : lttng-dev Objet : Re: [lttng-dev] LTTng 32 bits on 64 bits systems - kernel events... but no user-event Hi, > I am currently trying to add lttng 32bits on a 64bits system with a custom > kernel 4.19.177+ x86_64 (Debian 10.9) > My former attempt only with only a 100% arm32 bits was successful (raspberry). > I am facing now a strange issue : I can record kernel event but not userspace > event... but I can see these user-event (see below)! Ok. > (tuto : [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2F%23doc-instrumenting-32-bit-app-on-64-bit-system&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7C9b85f601067749845a2408d929c85983%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637586762735252731%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5NB2nNrI17uwSE1Xpn1Uf6sbrRXbEePidPNXUh4ps7k%3D&reserved=0 > | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2F%23doc-instrumenting-32-bit-app-on-64-bit-system&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7C9b85f601067749845a2408d929c85983%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637586762735252731%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5NB2nNrI17uwSE1Xpn1Uf6sbrRXbEePidPNXUh4ps7k%3D&reserved=0 ] ) > I followed two times the tutorial : on lttng 2.12 and lttng 2.10. > I am on lttng (LTTng Trace Control) 2.10.11 - KeKriek now Any particular reasons? > All installation seemed to install without any error. > The only thing I added is at the very end where I did a : ln -s > /usr/local/bin/lttng /usr/bin/lttng (to directly call commands like "lttng > start") > Here some interesting outputs, I am using this projet as an example : [ > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2F%23doc-tracing-your-own-user-application&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7C9b85f601067749845a2408d929c85983%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637586762735262680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6RTEjlZmpq2CZoIUD%2Fs5ySx60Zqd4JMByVejF8nQ7Vk%3D&reserved=0 | > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2F%23doc-tracing-your-own-user-application&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7C9b85f601067749845a2408d929c85983%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637586762735262680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6RTEjlZmpq2CZoIUD%2Fs5ySx60Zqd4JMByVejF8nQ7Vk%3D&reserved=0 ] > lttng list -u : OK [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpaste.ubuntu.com%2Fp%2FhZptnNzySw%2F&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7C9b85f601067749845a2408d929c85983%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637586762735262680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=o71InJDxz1UgYSztgj5ENRr48MmLhsvQvPCO4GeOJ40%3D&reserved=0 | Ubuntu Pastebin ] > LTTNG_UST_DEBUG=1 ./hello : seems OK : [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpaste.ubuntu.com%2Fp%2Fw6xHrJsWJ9%2F&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7C9b85f601067749845a2408d929c85983%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637586762735262680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FWcFiJDvDaGP0BxZaHVsk7MH91zwvcgmw7vOnDkkHGg%3D&reserved=0 > | Ubuntu Pastebin ] > kernel-event output : [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpaste.ubuntu.com%2Fp%2F5KfyS8wVdb%2F&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7C9b85f601067749845a2408d929c85983%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637586762735262680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uHI4%2FeLn3gupvZO0DKksc7msLSacG%2Bkp2daShuKr7Wg%3D&reserved=0 | Ubuntu Pastebin > ] > user-event ouput : none - the output folder stated with lttng create > my-user-session --output=/tmp/my-user-session doesn't even appear > apt-file search lttng : [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpaste.ubuntu.com%2Fp%2FxMcCc7bqwk%2F&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7C9b85f601067749845a2408d929c85983%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637586762735262680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BXXEGKYhAexDJnfDDb%2Fc2QjscpT%2FH49pkT0yI%2FlLh8Q%3D&reserved=0 | Ubuntu > Pastebin ] > lsmod | grep lttng : [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpaste.ubuntu.com%2Fp%2F7YNkNsh9Fr%2F&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7C9b85f601067749845a2408d929c85983%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637586762735262680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OYHATk5CBw3pIRB9Z3Cw56BiHJCknRzh09s1F%2B4wR%2FY%3D&reserved=0 | Ubuntu Pastebin > ] Can you provide the config.log from the configure steps We are missing the most important logs: lttng-sessiond. $ lttng-sessiond -vvv --verbose-consumer > Note : file /usr/local/bin/lttng (the exe I am using) gives : > /usr/local/bin/lttng: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), > dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux > 3.2.0, BuildID[sha1]=542a108c40e386c17027c2c8f8fcb30e278b7748, with debug_info, > not stripped This is normal. Concretely to support tracing of 32 bit applications only a 32 bits lttng-consumerd executable is needed which is launched by the lttng-sessiond executable. > I am wondering if the installation didn't pick a 64bits RCU and analyses > programms only on this. Hmmm, what indicate this? I seriously doubt that this can happen. > However, I have currently absolutely no idea how to debug this. I decided to do the process on a clean ubuntu 20.04 machine. For 2.12 there is some problems when compiling only lttng-consumerd (fd-tracker lib and libcommon problem). I plan on providing a patch shortly. But the major problem is that the path passed at configure time for the lttng-consumerd32 bit daemon are simply not used at runtime. See this bug I just opened [1]. This is valid for 2.10 all the way to master AFAIK. [1] https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.lttng.org%2Fissues%2F1318&data=04%7C01%7Cjulien.montet%40reseau.eseo.fr%7C9b85f601067749845a2408d929c85983%7C4d7ad1591265437ab9f62946247d5bf9%7C0%7C0%7C637586762735262680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EV2j354mEqvEeW46AAnyn7pbutOdn0HtWOjI%2BMbUf3Y%3D&reserved=0 In the meantime you will have to pass the lttng-consumerd 32 bit path as argument or env variables to lttng-sessiond as such: $ lttng-sessiond -vvv --verbose-consumer --consumerd32-libdir=/usr/local/lib32 --consumerd32-path=/usr/local/lib32/lttng/libexec/lttng-consumerd or $ LTTNG_CONSUMERD32_BIN=/usr/local/lib32/lttng/libexec/lttng-consumerd LTTNG_CONSUMERD32_LIBDIR=/usr/local/lib32 lttng-sessiond -vvv --verbose-consumer From there the 32bits application is traced correctly and data is gathered without any problem (at least on 2.12). ``` [15:22:24.500655502] (+?.?????????) ubuntu2004.localdomain lttng_ust_statedump:start: { cpu_id = 3 }, { } [15:22:24.500661538] (+0.000006036) ubuntu2004.localdomain lttng_ust_statedump:procname: { cpu_id = 3 }, { procname = "hello" } [15:22:24.502611224] (+0.001949686) ubuntu2004.localdomain lttng_ust_statedump:bin_info: { cpu_id = 3 }, { baddr = 0xF7EC9000, memsz = 20596, path = "/usr/lib32/libdl-2.31.so", is_pic = 1, has_build_id = 1, has_debug_link = 1 } [15:22:24.502614501] (+0.000003277) ubuntu2004.localdomain lttng_ust_statedump:build_id: { cpu_id = 3 }, { baddr = 0xF7EC9000, _build_id_length = 20, build_id = [ [0] = 0x75, [1] = 0x1E, [2] = 0xDD, [3] = 0x47, [4] = 0x72, [5] = 0x3, [6] ``` Cheers