perf record --kcore does not work when /proc/modules changed during copy

perf record --kcore does not work when /proc/modules changed during copy

[Daniel Dao]

Hi Perf tools maintainers,

`perf record --kcore` frequently did not work on a somewhat busy system.
For example:

  sudo perf record --kcore -- sleep 1
  ERROR: Failed to copy kcore

Using strace to look at the invocation, the failure looks like:

  ...
  openat(AT_FDCWD, "/proc/modules", O_RDONLY) = 56
  openat(AT_FDCWD, "perf.data/kcore_dir/modules", O_RDONLY) = 57
  read(56, "mpls_gso 16384 0 - Live 0xffffff"..., 4096) = 4070
  read(57, "mpls_gso 16384 0 - Live 0xffffff"..., 4070) = 4070
  read(56, "xt_conntrack 24576 22 - Live 0xf"..., 4096) = 3738
  read(57, "xt_conntrack 24576 22 - Live 0xf"..., 3738) = 3738
  close(57)                               = 0
  close(56)                               = 0
  close(55)                               = 0
  unlink("perf.data/kcore_dir/kcore")     = 0
  close(54)                               = 0
  unlink("perf.data/kcore_dir/modules")   = 0
  unlink("perf.data/kcore_dir/kallsyms")  = 0
  write(2, "ERROR: Failed to copy kcore\n", 28ERROR: Failed to copy kcore
  ...

We can see that the verification of proc/modules failed because /proc/modules
output changed after we copied kcore. When i looked at it, they are caused by
changes of module refcount which seems expected on busy systems, such as

  < tcp_bbr 40960 12644 - Live 0x0000000000000000
  ---
  > tcp_bbr 40960 12678 - Live 0x0000000000000000

Any suggestions on how to make this work is much appreciated.

Cheers,
Daniel.

Re: perf record --kcore does not work when /proc/modules changed during copy

[Adrian Hunter]

On 2/09/22 18:46, Daniel Dao wrote:
> Hi Perf tools maintainers,
> 
> `perf record --kcore` frequently did not work on a somewhat busy system.
> For example:
> 
>   sudo perf record --kcore -- sleep 1
>   ERROR: Failed to copy kcore
> 
> Using strace to look at the invocation, the failure looks like:
> 
>   ...
>   openat(AT_FDCWD, "/proc/modules", O_RDONLY) = 56
>   openat(AT_FDCWD, "perf.data/kcore_dir/modules", O_RDONLY) = 57
>   read(56, "mpls_gso 16384 0 - Live 0xffffff"..., 4096) = 4070
>   read(57, "mpls_gso 16384 0 - Live 0xffffff"..., 4070) = 4070
>   read(56, "xt_conntrack 24576 22 - Live 0xf"..., 4096) = 3738
>   read(57, "xt_conntrack 24576 22 - Live 0xf"..., 3738) = 3738
>   close(57)                               = 0
>   close(56)                               = 0
>   close(55)                               = 0
>   unlink("perf.data/kcore_dir/kcore")     = 0
>   close(54)                               = 0
>   unlink("perf.data/kcore_dir/modules")   = 0
>   unlink("perf.data/kcore_dir/kallsyms")  = 0
>   write(2, "ERROR: Failed to copy kcore\n", 28ERROR: Failed to copy kcore
>   ...
> 
> We can see that the verification of proc/modules failed because /proc/modules
> output changed after we copied kcore. When i looked at it, they are caused by
> changes of module refcount which seems expected on busy systems, such as
> 
>   < tcp_bbr 40960 12644 - Live 0x0000000000000000
>   ---
>   > tcp_bbr 40960 12678 - Live 0x0000000000000000
> 
> Any suggestions on how to make this work is much appreciated.

At the moment I can't think why the proc/modules file needs to be
compared at all since we also compare proc/kallsyms which should be
enough.