RE: Linux coresight arm64 : Incorrect data in cstrace.bin

["Pandey, Radhey Shyam" <radhey.shyam.pandey@amd.com>]

> -----Original Message-----
> From: Pandey, Radhey Shyam
> Sent: Thursday, July 6, 2023 5:40 PM
> To: James Clark <james.clark@arm.com>
> Cc: coresight@lists.linaro.org; linux-arm-kernel@lists.infradead.org; Suzuki K
> Poulose <suzuki.poulose@arm.com>; mathieu.poirier@linaro.org;
> mike.leach@linaro.org; leo.yan@linaro.org;
> alexander.shishkin@linux.intel.com; Sarangi, Anirudha
> <anirudha.sarangi@amd.com>
> Subject: RE: Linux coresight arm64 : Incorrect data in cstrace.bin
> 
> > -----Original Message-----
> > From: James Clark <james.clark@arm.com>
> > Sent: Wednesday, July 5, 2023 5:44 PM
> > To: Pandey, Radhey Shyam <radhey.shyam.pandey@amd.com>
> > Cc: coresight@lists.linaro.org; linux-arm-kernel@lists.infradead.org;
> > Suzuki K Poulose <suzuki.poulose@arm.com>; mathieu.poirier@linaro.org;
> > mike.leach@linaro.org; leo.yan@linaro.org;
> > alexander.shishkin@linux.intel.com; Sarangi, Anirudha
> > <anirudha.sarangi@amd.com>
> > Subject: Re: Linux coresight arm64 : Incorrect data in cstrace.bin
> >
> >
> >
> > On 05/07/2023 12:48, Pandey, Radhey Shyam wrote:
> > >> -----Original Message-----
> > >> From: James Clark <james.clark@arm.com>
> > >> Sent: Wednesday, July 5, 2023 4:29 PM
> > >> To: Pandey, Radhey Shyam <radhey.shyam.pandey@amd.com>
> > >> Cc: coresight@lists.linaro.org;
> > >> linux-arm-kernel@lists.infradead.org;
> > >> Suzuki K Poulose <suzuki.poulose@arm.com>;
> > >> mathieu.poirier@linaro.org; mike.leach@linaro.org;
> > >> leo.yan@linaro.org; alexander.shishkin@linux.intel.com
> > >> Subject: Re: Linux coresight arm64 : Incorrect data in cstrace.bin
> > >>
> > >>
> > >>
> > >> On 05/07/2023 10:56, Pandey, Radhey Shyam wrote:
> > >>>> -----Original Message-----
> > >>>> From: Suzuki K Poulose <suzuki.poulose@arm.com>
> > >>>> Sent: Wednesday, July 5, 2023 1:57 PM
> > >>>> To: Pandey, Radhey Shyam <radhey.shyam.pandey@amd.com>;
> > >>>> mathieu.poirier@linaro.org; mike.leach@linaro.org;
> > >>>> leo.yan@linaro.org; alexander.shishkin@linux.intel.com
> > >>>> Cc: coresight@lists.linaro.org;
> > >>>> linux-arm-kernel@lists.infradead.org
> > >>>> Subject: Re: Linux coresight arm64 : Incorrect data in
> > >>>> cstrace.bin
> > >>>>
> > >>>> Hi Radhe Shyam
> > >>>>
> > >>>> On 05/07/2023 05:38, Pandey, Radhey Shyam wrote:
> > >>>>> Hi,
> > >>>>>
> > >>>>> I am using linux 6.1 kernel coresight framework to capture ETM
> trace.
> > >>>>> Enabled coresight driver and added coresight component device
> > >>>>> node in DTS. With that, we could probe all coresight components.
> > >>>>>
> > >>>>> But when we do capture using sysfs and then read the trace.bin
> > >>>>> using ptm2human we see invalid trace data.  Trace content
> > >>>>> changes on each capture. Any pointers to help narrow down the
> issue?
> > >>>>>
> > >>>>> Tried: sysfs capture and perf (with OpenCSD enabled)
> > >>>>>
> > >>>>> Development board: VCK190 :
> > >>>>>   https://docs.xilinx.com/r/en-US/ug1366-vck190-eval-bd
> > >>>>> https://www.xilinx.com/support/documents/architecture-
> > >>>> manuals/am011-ve
> > >>>>> rsal-acap-trm.pdf
> > >>>>>
> > >>>>> xilinx-vck190-20231:/home/petalinux# dmesg | grep -i amba
> > >>>>> [    0.301054] Serial: AMBA PL011 UART driver
> > >>>>> [    0.306226] amba f0c20000.funnel: Fixing up cyclic dependency
> with
> > >>>> f0d70000.etm
> > >>>>> [    0.313624] amba f0c20000.funnel: Fixing up cyclic dependency
> with
> > >>>> f0d30000.etm
> > >>>>> [    0.321110] amba f0c30000.etf: Fixing up cyclic dependency with
> > >>>> f0c20000.funnel
> > >>>>> [    1.470247] Serial: AMBA driver
> > >>>>> xilinx-vck190-20231:/home/petalinux# dmesg | grep -i coresight
> > >>>>> [    1.865006]  cs_system_cfg: CoreSight Configuration manager
> > initialised
> > >>>>> [    1.876858] coresight etm0: CPU0: etm v4.0 initialized
> > >>>>> [    1.885666] coresight etm1: CPU1: etm v4.0 initialized
> > >>>>> [    1.894435] coresight-cpu-debug f0d00000.debug1: Coresight
> debug-
> > >> CPU0
> > >>>> initialized
> > >>>>> [    1.902072] coresight-cpu-debug f0d40000.debug1: Coresight
> debug-
> > >> CPU1
> > >>>> initialized
> > >>>>>
> > >>>>> xilinx-vck190-20231:/home/petalinux# ls
> /sys/bus/coresight/devices/
> > >>>>> etm0      etm1      funnel0   tmc_etf0
> > >>>>>
> > >>>>>   cd /sys/bus/coresight/devices/ echo 1 > tmc_etf0/enable_sink
> > >>>>> echo 1 > etm0/enable_source echo 0 > etm0/enable_source echo 0 >
> > >>>>> tmc_etf0/enable_sink cd /root/ dd if=/dev/tmc_etf0
> > >>>>> of=cstrace_28Jun.bin
> > >>>>>
> > >>>>> ./ptm2human/ptm2human -e -i cstrace.bin -d
> > >>>>
> > >>>> Please note that ptm2human is for PTM trace decoding and
> > >>>> ETMv4 uses a different format and thus is not compatible.
> > >>>>
> > >>>>>
> > >>>>> I also tried OpenCSD integration with PERF.
> > >>>>
> > >>>> Have you made sure the perf is "linked" to the opencsd ?
> > >>>
> > >>> Earlier I statically linked but after adding CORESIGHT=1.
> > >>>
> > >>>
> > >>> linux-xlnx$ make ARCH=arm64 NO_LIBELF=1 NO_JVMTI=1 VF=1
> > >> CORESIGHT=1 -C
> > >>> tools/perf/
> > >>> make: Entering directory 'linux-xlnx/tools/perf'
> > >>>   BUILD:   Doing 'make -j24' parallel build
> > >>>
> > >>> Makefile.config:520: *** Error: No libopencsd library found or the
> > >>> version
> > >> is not up-to-date.
> > >>> Please install recent libopencsd to build with CORESIGHT=1.  Stop.
> > >>>
> > >>>
> > >>> echo $CSINCLUDES
> > >>> /scratch/development/coresight/my-opencsd/decoder/include
> > >>> radheys@xhdradheys41:/scratch/development/linux-xlnx$ ls
> > >>> /scratch/development/coresight/my-opencsd/decoder/include
> > >>> common  i_dec  interfaces  mem_acc  opencsd  opencsd.h
> > >>> pkt_printers
> > >>>
> > >>> echo $CSLIBS
> > >>> /scratch/development/coresight/my-opencsd/decoder/lib/builddir
> > >>> radheys@xhdradheys41:/scratch/development/linux-xlnx$ ls
> > >> /scratch/development/coresight/my-opencsd/decoder/lib/builddir
> > >>> libopencsd.a        libopencsd_c_api.so    libopencsd_c_api.so.1.4.0
> > >> libopencsd.so.1
> > >>> libopencsd_c_api.a  libopencsd_c_api.so.1  libopencsd.so
> > >> libopencsd.so.1.4.0
> > >>>
> > >>> Anything I am missing to fix this opencsd lib not found?
> > >>
> > >> There is a minimum version of 1.1.1 but it looks like you are using
> > >> higher than that so that part should be fine.
> > >>
> > >> Personally I "make install" OpenCSD to the system path. I had a
> > >> play around with $CSLIBS it and maybe there is also some stickyness
> > >> to the feature detection so a make clean might help to make sure
> > >> the errors you are seeing are real.
> > >>
> > >> Lastly you can print the output of why the feature test compilation
> > >> failed which might help. I see some linking issues in there if I
> > >> delete my system installed version of OpenCSD, so there might be a
> > >> bug with just using CSINCLUDES and CSLIBS:
> > >>
> > >>  $ cat  tools/build/feature/test-libopencsd.make.output
> > >>
> > >>   /usr/bin/ld: warning: libopencsd.so.1, needed by opencsd-
> > >> local/decoder/lib/builddir//libopencsd_c_api.so, not found (try
> > >> using -rpath or -rpath-link)
> > >>   /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so:
> > >> undefined reference to `vtable for StmTrcPacket'
> > >>   /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so:
> > >> undefined reference to `DecodeTree::getDecoderStats(unsigned char,
> > >> _ocsd_decode_stats**)'
> > >>   /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so:
> > >> undefined reference to
> > >> `DecodeTree::addRawFramePrinter(RawFramePrinter**, unsigned int)'
> > >>
> > >> Can you try the make install and see if that works? If that works
> > >> but the other way doesn't I can try looking into why its not and make a
> fix.
> > >
> > > I tried make install but still it errors out. Looking at make.output
> > > I am seeing multiple undefined references.
> > >
> > > NOTE I have to still set CSINCLUDES and CSLIBS as we are
> > > cross-compiling Perf on a x86 host machine.
> >
> > This should still work if you make install them to the arch specific
> > folder for example /usr/lib/aarch64-linux-gnu/. If do 'apt install
> libopencsd-dev:arm64'
> > I would get the arm ones in there.
> >
> > Then if I cross build like this I get openCSD linked:
> >
> >   make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C tools/perf
> > CORESIGHT=1
> >
> > So for now you should be able to work around the issue in this way.
> 
> I tried copying opencsd libraries to /usr/lib/aarch64-linux-gnu but still I see
> undefined reference. Is it because I am using cross-compiler from server
> (toolchain path set to $PATH variable)?
> 
> /proj/<snip>/Vitis/2023.2/gnu/aarch64/lin/aarch64-linux/bin/aarch64-linux-
> gnu-gcc
> I assume CSLIBS and CSINCLUDES should be sufficient to cross compile perf
> with coresight support?

FYI, I tried with static linking and could proceed to perf compilation.

make LDFLAGS=-static NO_LIBELF=1 NO_JVMTI=1 VF=1 CORESIGHT=1 -C tools/perf/
<snip>
...                        libbfd-liberty-z: [ OFF ]
...                              libopencsd: [ on  ]
...                           libunwind-x86: [ OFF ]

But it ended with compilation failure. Seems like OpenCSD version mismatch.
I will try to downgrade opencsd library and see if it fixes.

util/cs-etm-decoder/cs-etm-decoder.c: In function 'cs_etm_decoder__gen_trace_elem_printer':
util/cs-etm-decoder/cs-etm-decoder.c:567:9: error: enumeration value 
'OCSD_GEN_TRC_ELEM_INSTRUMENTATION' not handled in switch [-Werror=switch-enum]
  567 |         switch (elem->elem_type) {

> 
> >
> > >
> > > cat tools/build/feature/test-libopencsd.make.output <snip library
> > > paths>
> > >
> > > aarch64-xilinx-linux-ld.real: warning: libopencsd.so.1, needed by
> > > libopencsd_c_api.so, not found (try using -rpath or -rpath-link)
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `vtable for StmTrcPacket'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `DecodeTree::getDecoderStats(unsigned char, _ocsd_decode_stats**)'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `DecodeTree::addRawFramePrinter(RawFramePrinter**, unsigned int)'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `DecodeTree::logMappedRanges()'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `OcsdLibDcdRegister::deregisterAllDecoders()'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `PtmTrcPacket::toString(std::__cxx11::basic_string<char,
> > std::char_traits<char>, std::allocator<char> >&) const'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `DecodeTree::s_error_logger'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `DecodeTree::addBinFileRegionMemAcc(_ocsd_file_mem_region const*,
> int,
> > _ocsd_mem_space_acc_t, std::__cxx11::basic_string<char,
> > std::char_traits<char>, std::allocator<char> > const&)'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `EtmV4ITrcPacket::~EtmV4ITrcPacket()'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `OcsdLibDcdRegister::getDecoderRegister()'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `typeinfo for TraceComponent'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `EtmV3TrcPacket::~EtmV3TrcPacket()'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `OcsdTraceElement::toString(std::__cxx11::basic_string<char,
> > std::char_traits<char>, std::allocator<char> >&) const'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `DecodeTree::addGenElemPrinter(TrcGenericElementPrinter**)'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `DecodeTree::DestroyDecodeTree(DecodeTree*)'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `ocsdError::getErrorString[abi:cxx11](ocsdError const&)'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `DecodeTree::setGenTraceElemOutI(ITrcGenElemIn*)'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `ocsdError::ocsdError(ocsdError const*)'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `ocsdError::ocsdError(_ocsd_err_severity_t, _ocsd_err_t, unsigned int,
> > unsigned char, std::__cxx11::basic_string<char,
> > std::char_traits<char>, std::allocator<char> > const&)'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `EtmV4ITrcPacket::toString(std::__cxx11::basic_string<char,
> > std::char_traits<char>, std::allocator<char> >&) const'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `DecodeTree::createDecoder(std::__cxx11::basic_string<char,
> > std::char_traits<char>, std::allocator<char> > const&, int, CSConfig const*)'
> > > aarch64-xilinx-linux-ld.real:  libopencsd_c_api.so: undefined
> > > reference to
> > `StmTrcPacket::StmTrcPacket()'
> > >
> >
> > Yes this looks like what I see as well. There is probably some issue
> > with CSLIBS not being propagated to all parts of the build. I would
> > recommend using the system library path for now and I will look into a fix
> next week.
> > Unless you are happy to look into it and send a fix that would be great too.
> >
> > James
> >
> > > Thanks,
> > > Radhey
> > >
> > >>
> > >> Thanks
> > >> James
> > >
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel