[PATCH v2] perf test record.sh: Raise limit of open file descriptors

[PATCH v2] perf test record.sh: Raise limit of open file descriptors

[vmolnaro]

From: Veronika Molnarova <vmolnaro@redhat.com>

Subtest for system-wide record with '--threads=cpu' option fails due
to a limit of open file descriptors on systems with 128 or more CPUs
as the default limit is set to 1024.

The number of open file descriptors should be slightly above
nmb_events*nmb_cpus + nmb_cpus(for perf.data.n) + 4*nmb_cpus(for pipes),
which equals 8*nmb_cpus. Therefore, temporarily raise the limit to
16*nmb_cpus for the test.

Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>
---
 tools/perf/tests/shell/record.sh | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
index 4fbc74805d52..8acd55d09a1c 100755
--- a/tools/perf/tests/shell/record.sh
+++ b/tools/perf/tests/shell/record.sh
@@ -11,6 +11,14 @@ err=0
 perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
 testprog="perf test -w thloop"
 testsym="test_loop"
+default_fd_limit=$(ulimit -n)
+# With option --threads=cpu the number of open file descriptors should be
+# equal to sum of:    nmb_cpus * nmb_events (2+dummy),
+#                     nmb_threads for perf.data.n (equal to nmb_cpus) and
+#                     2*nmb_cpus of pipes = 4*nmb_cpus (each pipe has 2 ends)
+# All together it needs 8*nmb_cpus file descriptors plus some are also used 
+# outside of testing, thus raising the limit to 16*nmb_cpus
+min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16))
 
 cleanup() {
   rm -rf "${perfdata}"
@@ -154,10 +162,18 @@ test_workload() {
   echo "Basic target workload test [Success]"
 }
 
+# raise the limit of file descriptors to minimum
+if [[ $default_fd_limit -lt $min_fd_limit ]]; then
+	ulimit -n $min_fd_limit
+fi
+
 test_per_thread
 test_register_capture
 test_system_wide
 test_workload
 
+# restore the default value
+ulimit -n $default_fd_limit
+
 cleanup
 exit $err
-- 
2.41.0

Re: [PATCH v2] perf test record.sh: Raise limit of open file descriptors

[Michael Petlan]

Hello Arnaldo,
kind reminder, any chance this being merged soon?
Thanks,
Michael

On Mon, 11 Mar 2024, vmolnaro@redhat.com wrote:
> From: Veronika Molnarova <vmolnaro@redhat.com>
> 
> Subtest for system-wide record with '--threads=cpu' option fails due
> to a limit of open file descriptors on systems with 128 or more CPUs
> as the default limit is set to 1024.
> 
> The number of open file descriptors should be slightly above
> nmb_events*nmb_cpus + nmb_cpus(for perf.data.n) + 4*nmb_cpus(for pipes),
> which equals 8*nmb_cpus. Therefore, temporarily raise the limit to
> 16*nmb_cpus for the test.
>
> Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>

Acked-by: Michael Petlan <mpetlan@redhat.com>

> ---
>  tools/perf/tests/shell/record.sh | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
> index 4fbc74805d52..8acd55d09a1c 100755
> --- a/tools/perf/tests/shell/record.sh
> +++ b/tools/perf/tests/shell/record.sh
> @@ -11,6 +11,14 @@ err=0
>  perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
>  testprog="perf test -w thloop"
>  testsym="test_loop"
> +default_fd_limit=$(ulimit -n)
> +# With option --threads=cpu the number of open file descriptors should be
> +# equal to sum of:    nmb_cpus * nmb_events (2+dummy),
> +#                     nmb_threads for perf.data.n (equal to nmb_cpus) and
> +#                     2*nmb_cpus of pipes = 4*nmb_cpus (each pipe has 2 ends)
> +# All together it needs 8*nmb_cpus file descriptors plus some are also used 
> +# outside of testing, thus raising the limit to 16*nmb_cpus
> +min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16))
>  
>  cleanup() {
>    rm -rf "${perfdata}"
> @@ -154,10 +162,18 @@ test_workload() {
>    echo "Basic target workload test [Success]"
>  }
>  
> +# raise the limit of file descriptors to minimum
> +if [[ $default_fd_limit -lt $min_fd_limit ]]; then
> +	ulimit -n $min_fd_limit
> +fi
> +
>  test_per_thread
>  test_register_capture
>  test_system_wide
>  test_workload
>  
> +# restore the default value
> +ulimit -n $default_fd_limit
> +
>  cleanup
>  exit $err
> -- 
> 2.41.0
> 
> 

Re: [PATCH v2] perf test record.sh: Raise limit of open file descriptors

[Arnaldo Carvalho de Melo]

On Wed, Mar 20, 2024 at 01:14:32PM +0100, Michael Petlan wrote:
> Hello Arnaldo,
> kind reminder, any chance this being merged soon?
> Thanks,
> Michael
> 
> On Mon, 11 Mar 2024, vmolnaro@redhat.com wrote:
> > From: Veronika Molnarova <vmolnaro@redhat.com>
> > 
> > Subtest for system-wide record with '--threads=cpu' option fails due
> > to a limit of open file descriptors on systems with 128 or more CPUs
> > as the default limit is set to 1024.
> > 
> > The number of open file descriptors should be slightly above
> > nmb_events*nmb_cpus + nmb_cpus(for perf.data.n) + 4*nmb_cpus(for pipes),
> > which equals 8*nmb_cpus. Therefore, temporarily raise the limit to
> > 16*nmb_cpus for the test.
> >
> > Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>
> 
> Acked-by: Michael Petlan <mpetlan@redhat.com>

Its not applying, can you please check?

Please use the
git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git
repo, tmp.perf-tools-next branch.

- Arnaldo

⬢[acme@toolbox perf-tools-next]$ b4 am -ctsl --cc-trailers alpine.LRH.2.20.2403201310220.4040@Diego
Grabbing thread from lore.kernel.org/all/alpine.LRH.2.20.2403201310220.4040@Diego/t.mbox.gz
Checking for newer revisions
Grabbing search results from lore.kernel.org
Analyzing 2 messages in the thread
Checking attestation on all messages, may take a moment...
---
  [PATCH v2] perf test record.sh: Raise limit of open file descriptors
    + Acked-by: Michael Petlan <mpetlan@redhat.com>
    + Link: https://lore.kernel.org/r/20240311081127.7652-1-vmolnaro@redhat.com
    + Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
  ---
  NOTE: install dkimpy for DKIM signature verification
---
Total patches: 1
---
 Link: https://lore.kernel.org/r/20240311081127.7652-1-vmolnaro@redhat.com
 Base: not specified
       git am ./v2_20240311_vmolnaro_perf_test_record_sh_raise_limit_of_open_file_descriptors.mbx
⬢[acme@toolbox perf-tools-next]$ vim ./v2_20240311_vmolnaro_perf_test_record_sh_raise_limit_of_open_file_descriptors.mbx
⬢[acme@toolbox perf-tools-next]$ patch -p1 < ./v2_20240311_vmolnaro_perf_test_record_sh_raise_limit_of_open_file_descriptors.mbx
patching file tools/perf/tests/shell/record.sh
Hunk #1 succeeded at 12 with fuzz 2 (offset 1 line).
Hunk #2 FAILED at 162.
1 out of 2 hunks FAILED -- saving rejects to file tools/perf/tests/shell/record.sh.rej
⬢[acme@toolbox perf-tools-next]$ 

[PATCH] perf test record.sh: Raise limit of open file descriptors

[vmolnaro]

From: Veronika Molnarova <vmolnaro@redhat.com>

Subtest for system-wide record with '--threads=cpu' option fails due
to a limit of open file descriptors on systems with 128 or more CPUs
as the default limit is set to 1024.

The number of open file descriptors should be slightly above
nmb_events*nmb_cpus + nmb_cpus(for perf.data.n) + 4*nmb_cpus(for pipes),
which equals 8*nmb_cpus. Therefore, temporarily raise the limit to
16*nmb_cpus for the test.

Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>
---
Reworked the patch as the testfile has already changed since the patch
was sent causing the applying to fail. Should be good now.
 tools/perf/tests/shell/record.sh | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
index 4fbc74805d52..8dc90fa08682 100755
--- a/tools/perf/tests/shell/record.sh
+++ b/tools/perf/tests/shell/record.sh
@@ -11,6 +11,14 @@ err=0
 perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
 testprog="perf test -w thloop"
 testsym="test_loop"
+default_fd_limit=$(ulimit -n)
+# With option --threads=cpu the number of open file descriptors should be
+# equal to sum of:    nmb_cpus * nmb_events (2+dummy),
+#                     nmb_threads for perf.data.n (equal to nmb_cpus) and
+#                     2*nmb_cpus of pipes = 4*nmb_cpus (each pipe has 2 ends)
+# All together it needs 8*nmb_cpus file descriptors plus some are also used
+# outside of testing, thus raising the limit to 16*nmb_cpus
+min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16))
 
 cleanup() {
   rm -rf "${perfdata}"
@@ -154,10 +162,18 @@ test_workload() {
   echo "Basic target workload test [Success]"
 }
 
+# raise the limit of file descriptors to minimum
+if [[ $default_fd_limit -lt $min_fd_limit ]]; then
+	ulimit -n $min_fd_limit
+fi
+
 test_per_thread
 test_register_capture
 test_system_wide
 test_workload
 
+# restore the default value
+ulimit -n $default_fd_limit
+
 cleanup
 exit $err
-- 
2.43.0

Re: [PATCH v2] perf test record.sh: Raise limit of open file descriptors

[Arnaldo Carvalho de Melo]

On Wed, Mar 20, 2024 at 11:19:30AM -0300, Arnaldo Carvalho de Melo wrote:
> On Wed, Mar 20, 2024 at 01:14:32PM +0100, Michael Petlan wrote:
> > Hello Arnaldo,
> > kind reminder, any chance this being merged soon?
> > Thanks,
> > Michael
> > 
> > On Mon, 11 Mar 2024, vmolnaro@redhat.com wrote:
> > > From: Veronika Molnarova <vmolnaro@redhat.com>
> > > 
> > > Subtest for system-wide record with '--threads=cpu' option fails due
> > > to a limit of open file descriptors on systems with 128 or more CPUs
> > > as the default limit is set to 1024.
> > > 
> > > The number of open file descriptors should be slightly above
> > > nmb_events*nmb_cpus + nmb_cpus(for perf.data.n) + 4*nmb_cpus(for pipes),
> > > which equals 8*nmb_cpus. Therefore, temporarily raise the limit to
> > > 16*nmb_cpus for the test.
> > >
> > > Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>
> > 
> > Acked-by: Michael Petlan <mpetlan@redhat.com>
> 
> Its not applying, can you please check?
> 
> Please use the
> git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git
> repo, tmp.perf-tools-next branch.

ping?

- Arnaldo
 
> - Arnaldo
> 
> ⬢[acme@toolbox perf-tools-next]$ b4 am -ctsl --cc-trailers alpine.LRH.2.20.2403201310220.4040@Diego
> Grabbing thread from lore.kernel.org/all/alpine.LRH.2.20.2403201310220.4040@Diego/t.mbox.gz
> Checking for newer revisions
> Grabbing search results from lore.kernel.org
> Analyzing 2 messages in the thread
> Checking attestation on all messages, may take a moment...
> ---
>   [PATCH v2] perf test record.sh: Raise limit of open file descriptors
>     + Acked-by: Michael Petlan <mpetlan@redhat.com>
>     + Link: https://lore.kernel.org/r/20240311081127.7652-1-vmolnaro@redhat.com
>     + Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
>   ---
>   NOTE: install dkimpy for DKIM signature verification
> ---
> Total patches: 1
> ---
>  Link: https://lore.kernel.org/r/20240311081127.7652-1-vmolnaro@redhat.com
>  Base: not specified
>        git am ./v2_20240311_vmolnaro_perf_test_record_sh_raise_limit_of_open_file_descriptors.mbx
> ⬢[acme@toolbox perf-tools-next]$ vim ./v2_20240311_vmolnaro_perf_test_record_sh_raise_limit_of_open_file_descriptors.mbx
> ⬢[acme@toolbox perf-tools-next]$ patch -p1 < ./v2_20240311_vmolnaro_perf_test_record_sh_raise_limit_of_open_file_descriptors.mbx
> patching file tools/perf/tests/shell/record.sh
> Hunk #1 succeeded at 12 with fuzz 2 (offset 1 line).
> Hunk #2 FAILED at 162.
> 1 out of 2 hunks FAILED -- saving rejects to file tools/perf/tests/shell/record.sh.rej
> ⬢[acme@toolbox perf-tools-next]$ 

[PATCH v2] perf test record.sh: Raise limit of open file descriptors

[vmolnaro]

From: Veronika Molnarova <vmolnaro@redhat.com>

Subtest for system-wide record with '--threads=cpu' option fails due
to a limit of open file descriptors on systems with 128 or more CPUs
as the default limit is set to 1024.

The number of open file descriptors should be slightly above
nmb_events*nmb_cpus + nmb_cpus(for perf.data.n) + 4*nmb_cpus(for pipes),
which equals 8*nmb_cpus. Therefore, temporarily raise the limit to
16*nmb_cpus for the test.

Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>
---
Reworked the patch as the testfile has already changed since the patch
was sent causing the applying to fail, now applying upstream without
problems.
 tools/perf/tests/shell/record.sh | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
index 3d1a7759a7b2..344c0e40ebe8 100755
--- a/tools/perf/tests/shell/record.sh
+++ b/tools/perf/tests/shell/record.sh
@@ -21,6 +21,15 @@ testprog="perf test -w thloop"
 cpu_pmu_dir="/sys/bus/event_source/devices/cpu*"
 br_cntr_file="/caps/branch_counter_nr"
 br_cntr_output="branch stack counters"
+default_fd_limit=$(ulimit -n)
+# With option --threads=cpu the number of open file descriptors should be
+# equal to sum of:    nmb_cpus * nmb_events (2+dummy),
+#                     nmb_threads for perf.data.n (equal to nmb_cpus) and
+#                     2*nmb_cpus of pipes = 4*nmb_cpus (each pipe has 2 ends)
+# All together it needs 8*nmb_cpus file descriptors plus some are also used
+# outside of testing, thus raising the limit to 16*nmb_cpus
+min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16))
+
 
 cleanup() {
   rm -rf "${perfdata}"
@@ -190,11 +199,19 @@ test_branch_counter() {
   echo "Basic branch counter test [Success]"
 }
 
+# raise the limit of file descriptors to minimum
+if [[ $default_fd_limit -lt $min_fd_limit ]]; then
+       ulimit -n $min_fd_limit
+fi
+
 test_per_thread
 test_register_capture
 test_system_wide
 test_workload
 test_branch_counter
 
+# restore the default value
+ulimit -n $default_fd_limit
+
 cleanup
 exit $err
-- 
2.43.0

Re: [PATCH v2] perf test record.sh: Raise limit of open file descriptors

[Michael Petlan]

Hello Arnaldo,
what do you think about the v2?
Thanks.
Michael

On Thu, 30 Nov 2023, vmolnaro@redhat.com wrote:
> From: Veronika Molnarova <vmolnaro@redhat.com>
> 
> Subtest for system-wide record with '--threads=cpu' option fails due
> to a limit of open file descriptors on systems with 128 or more CPUs
> as the default limit is set to 1024.
> 
> The number of open file descriptors should be slightly above
> nmb_events*nmb_cpus + nmb_cpus(for perf.data.n) + 4*nmb_cpus(for pipes),
> which equals 8*nmb_cpus. Therefore, temporarily raise the limit to
> 16*nmb_cpus for the test.
> 
> Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>
> ---
>  tools/perf/tests/shell/record.sh | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
> index 4fbc74805d52..8acd55d09a1c 100755
> --- a/tools/perf/tests/shell/record.sh
> +++ b/tools/perf/tests/shell/record.sh
> @@ -11,6 +11,14 @@ err=0
>  perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
>  testprog="perf test -w thloop"
>  testsym="test_loop"
> +default_fd_limit=$(ulimit -n)
> +# With option --threads=cpu the number of open file descriptors should be
> +# equal to sum of:    nmb_cpus * nmb_events (2+dummy),
> +#                     nmb_threads for perf.data.n (equal to nmb_cpus) and
> +#                     2*nmb_cpus of pipes = 4*nmb_cpus (each pipe has 2 ends)
> +# All together it needs 8*nmb_cpus file descriptors plus some are also used 
> +# outside of testing, thus raising the limit to 16*nmb_cpus
> +min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16))
>  
>  cleanup() {
>    rm -rf "${perfdata}"
> @@ -154,10 +162,18 @@ test_workload() {
>    echo "Basic target workload test [Success]"
>  }
>  
> +# raise the limit of file descriptors to minimum
> +if [[ $default_fd_limit -lt $min_fd_limit ]]; then
> +	ulimit -n $min_fd_limit
> +fi
> +
>  test_per_thread
>  test_register_capture
>  test_system_wide
>  test_workload
>  
> +# restore the default value
> +ulimit -n $default_fd_limit
> +
>  cleanup
>  exit $err
> -- 
> 2.41.0
> 
> 
> 

[PATCH v2] perf test record.sh: Raise limit of open file descriptors

[vmolnaro]

From: Veronika Molnarova <vmolnaro@redhat.com>

Subtest for system-wide record with '--threads=cpu' option fails due
to a limit of open file descriptors on systems with 128 or more CPUs
as the default limit is set to 1024.

The number of open file descriptors should be slightly above
nmb_events*nmb_cpus + nmb_cpus(for perf.data.n) + 4*nmb_cpus(for pipes),
which equals 8*nmb_cpus. Therefore, temporarily raise the limit to
16*nmb_cpus for the test.

Signed-off-by: Veronika Molnarova <vmolnaro@redhat.com>
---
 tools/perf/tests/shell/record.sh | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
index 4fbc74805d52..8acd55d09a1c 100755
--- a/tools/perf/tests/shell/record.sh
+++ b/tools/perf/tests/shell/record.sh
@@ -11,6 +11,14 @@ err=0
 perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
 testprog="perf test -w thloop"
 testsym="test_loop"
+default_fd_limit=$(ulimit -n)
+# With option --threads=cpu the number of open file descriptors should be
+# equal to sum of:    nmb_cpus * nmb_events (2+dummy),
+#                     nmb_threads for perf.data.n (equal to nmb_cpus) and
+#                     2*nmb_cpus of pipes = 4*nmb_cpus (each pipe has 2 ends)
+# All together it needs 8*nmb_cpus file descriptors plus some are also used 
+# outside of testing, thus raising the limit to 16*nmb_cpus
+min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16))
 
 cleanup() {
   rm -rf "${perfdata}"
@@ -154,10 +162,18 @@ test_workload() {
   echo "Basic target workload test [Success]"
 }
 
+# raise the limit of file descriptors to minimum
+if [[ $default_fd_limit -lt $min_fd_limit ]]; then
+	ulimit -n $min_fd_limit
+fi
+
 test_per_thread
 test_register_capture
 test_system_wide
 test_workload
 
+# restore the default value
+ulimit -n $default_fd_limit
+
 cleanup
 exit $err
-- 
2.41.0