[RFT PATCH -perfbook 0/2] Retouch plots in QQA 15.31

[RFT PATCH -perfbook 0/2] Retouch plots in QQA 15.31

[Akira Yokosawa]

Hi Paul,

I'm not sure if you are already working on those figures in .png,
but I'm sending updates in scripts.  I'm dropping changes and
additions of .eps and .png files generated by the updated scripts.

Here is a list of commands you'd need to reflect the changes in
perfbook.pdf.

    pushd CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/
    sh plots.sh

    evince coe-nvalue.eps &
    evince coe.eps &
    evince fre.eps &
    evince rfe.eps &
    display fre.png &

If those plots look reasonable to you, please copy them to the parent
directory and do the "make":

    cp *.eps fre.png ../
    popd
    make

If those figures in the resulting perfbook.pdf looks good to you,
I'd like you to go forward and commit the changes and additions:

    git add CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/*.eps
    git add CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/fre.png
    git add CodeSamples/cpu/data/*.eps
    git add CodeSamples/cpu/data/fre.png

    git commit -a # with a suitable tag such as Requested-by: me

Furthermore, if you think those .png figures, now obsoleted by the
new .eps figures, are not needed anymore, please "git rm" them as well.
    
If there is anything I've missed, please let me know.

        Thanks, Akira
--
Akira Yokosawa (2):
  CodeSamples/cpu/data: plots.sh: Fix typo in xlabel for fre plot
  CodeSamples/cpu/data: Use "smooth frequency" for plotting histogram

 .../expand.pl                                 | 42 +++++++++
 .../plots.sh                                  | 85 ++++++++++++++++++-
 2 files changed, 126 insertions(+), 1 deletion(-)
 create mode 100755 CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/expand.pl


base-commit: 9caac64bfabf85439c480ce928dddef969354369
-- 
2.34.1

[RFT PATCH -perfbook 1/2] CodeSamples/cpu/data/.../plots.sh: Fix typo in xlabel for fre plot

[Akira Yokosawa]

Fix trivial typo in xlabel.

Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
---
 .../cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/plots.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/plots.sh b/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/plots.sh
index ab2f318f..5b589858 100644
--- a/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/plots.sh
+++ b/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/plots.sh
@@ -37,7 +37,7 @@ plot "coe.dat" w points pt 6 pointsize 0.2
 gnuplot << ---EOF---
 set term png
 set output "fre.png"
-set xlabel "Store-to-Load Latency (Timestamp Periods)"
+set xlabel "Load-to-Store Latency (Timestamp Periods)"
 set ylabel "Number of Samples"
 #set logscale y
 set xrange [-250:50]
-- 
2.34.1

[RFT PATCH -perfbook 2/2] CodeSamples/cpu/data: Use "smooth frequency" for plotting histogram

[Akira Yokosawa]

Add gnuplot scripts for generating EPS plots.

Use "smooth frequency" feature of gnuplot for plotting distribution
histograms.

Distribution data in coe.dat, fre.dat, and rfe.dat are converted
on-the-fly into single-column data by using expand.pl as a filter.

Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
---
 .../expand.pl                                 | 42 ++++++++++
 .../plots.sh                                  | 83 +++++++++++++++++++
 2 files changed, 125 insertions(+)
 create mode 100755 CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/expand.pl

diff --git a/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/expand.pl b/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/expand.pl
new file mode 100755
index 00000000..30719692
--- /dev/null
+++ b/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/expand.pl
@@ -0,0 +1,42 @@
+#!/usr/bin/env perl
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Expand frequency data of the form:
+#
+#   1000 4
+#   1004 2
+#   1050 1
+#
+# into:
+#
+#   1000
+#   1000
+#   1000
+#   1000
+#   1004
+#   1004
+#   1050
+#
+# Copyright (C) Akira Yokosawa, 2024
+#
+# Authors: Akira Yokosawa <akiyks@gmail.com>
+
+use strict;
+use warnings;
+
+my $line;
+my $freq;
+my $count;
+
+open(my $fn, '<:encoding(UTF-8)', $ARGV[0])
+    or die "Could not open file '$ARGV[0]' $!";
+
+while($line = <$fn>) {
+    if ($line =~ /(\-?[0-9]+)\s+([0-9]+)/ ) {
+	$freq = $1;
+	$count = $2;
+	for (my $i = 0; $i < $count; $i++) {
+	    print "$freq\n";
+	}
+    }
+}
diff --git a/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/plots.sh b/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/plots.sh
index 5b589858..ca083bca 100644
--- a/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/plots.sh
+++ b/CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/plots.sh
@@ -67,3 +67,86 @@ set nokey
 # set label 5 "refcnt" at 0.15,2.8 left
 plot "rfe.dat" w points pt 6 pointsize 0.5
 ---EOF---
+
+# For eps output
+font=../../../../fonts/uhvr8a.pfb
+fontsize=8
+przsize="nosquare 0.55,0.25"
+
+gnuplot << ---EOF---
+set term postscript portrait color ${fontsize} enhanced fontfile "${font}" "NimbusSanL-Regu" 
+set size $przsize
+set output "coe-nvals.eps"
+
+set xlabel "Time (Timestamp Periods)"
+set ylabel "Number of Simultaneous Values"
+#set logscale y
+set xrange [:10000]
+#set yrange [100:10000]
+set nokey
+plot "coe-nvals.dat" w l
+---EOF---
+
+gnuplot << ---EOF---
+set term postscript portrait color ${fontsize} enhanced fontfile "${font}" "NimbusSanL-Regu"
+set size $przsize
+set output "coe.eps"
+stats "< perl expand.pl coe.dat" nooutput
+N = STATS_records
+
+set xlabel "Store-to-Store Latency (Timestamp Periods)"
+set ylabel "Frequency (arb. unit)"
+#set logscale y
+set xrange [-1000:3000]
+set yrange [0:]
+set nokey
+#set border 3
+set yzeroaxis
+set boxwidth 40
+#set style fill solid 1.0 noborder
+bin_width = 40
+bin(x) = bin_width * floor(x/bin_width)
+plot "< perl expand.pl coe.dat" using (bin(column(1))):(20./N) smooth frequency with boxes
+---EOF---
+
+gnuplot << ---EOF---
+set term postscript portrait color ${fontsize} enhanced fontfile "${font}" "NimbusSanL-Regu"
+set size $przsize
+set output "fre.eps"
+stats "< perl expand.pl fre.dat" nooutput
+N = STATS_records
+
+set xlabel "Load-to-Store Latency (Timestamp Periods)"
+set ylabel "Frequency (arb. unit)"
+#set logscale y
+set xrange [-250:50]
+set yrange [0:]
+set nokey
+set yzeroaxis
+set boxwidth 2
+#set style fill solid 1.0 noborder
+bin_width = 2
+bin(x) = bin_width * floor(x/bin_width)
+plot "< perl expand.pl fre.dat" using (bin(column(1))):(20./N) smooth frequency with boxes
+---EOF---
+
+gnuplot << ---EOF---
+set term postscript portrait color ${fontsize} enhanced fontfile "${font}" "NimbusSanL-Regu"
+set size $przsize
+set output "rfe.eps"
+stats "< perl expand.pl rfe.dat" nooutput
+N = STATS_records
+
+set xlabel "Store-to-Load Latency (Timestamp Periods)"
+set ylabel "Frequency (arb. unit)"
+#set logscale y
+set xrange [0:350]
+set yrange [0:]
+set nokey
+set yzeroaxis
+set boxwidth 2
+#set style fill solid 1.0 noborder
+bin_width = 2
+bin(x) = bin_width * floor(x/bin_width)
+plot "< perl expand.pl rfe.dat" using (bin(column(1))):(20./N) smooth frequency with boxes
+---EOF---
-- 
2.34.1

Re: [RFT PATCH -perfbook 0/2] Retouch plots in QQA 15.31

[Paul E. McKenney]

On Wed, Jan 31, 2024 at 12:31:06AM +0900, Akira Yokosawa wrote:
> Hi Paul,
> 
> I'm not sure if you are already working on those figures in .png,
> but I'm sending updates in scripts.  I'm dropping changes and
> additions of .eps and .png files generated by the updated scripts.
> 
> Here is a list of commands you'd need to reflect the changes in
> perfbook.pdf.
> 
>     pushd CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/

I removed this directory due to my having tweaked the data to paper over
bugs in early versions of my scripts.  So I focused instead on
CodeSamples/cpu/data/kerneltest054.05.atn6.2024.01.22-12.28.01.

Ah, and I never did push out the commit where I removed the
CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02
directory.  Apologies, and it is out there now.

>     sh plots.sh

This failed until I installed the groff package to get pfbtops.
I of course also had to move expand.pl to this directory.

>     evince coe-nvalue.eps &
>     evince coe.eps &
>     evince fre.eps &
>     evince rfe.eps &
>     display fre.png &

I do like the vertical dashed line at x=0.  I see artifacts on my
screen due to pixel/plot misalignment, but increasing size makes
things look good.  It will be fine on paper.  And the fill is
good compared to the funny datapoint labels.

I adjusted coe.eps's xrange to include the entire curve in the plot.

And it does look a lot more professional than my original, so thank
you for that!

> If those plots look reasonable to you, please copy them to the parent
> directory and do the "make":
> 
>     cp *.eps fre.png ../

I don't understand why fre.png is needed.  I am leaving it out, so
perhaps I will learn the hard way.  ;-)

>     popd
>     make
> 
> If those figures in the resulting perfbook.pdf looks good to you,
> I'd like you to go forward and commit the changes and additions:
> 
>     git add CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/*.eps
>     git add CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/fre.png
>     git add CodeSamples/cpu/data/*.eps
>     git add CodeSamples/cpu/data/fre.png
> 
>     git commit -a # with a suitable tag such as Requested-by: me
> 
> Furthermore, if you think those .png figures, now obsoleted by the
> new .eps figures, are not needed anymore, please "git rm" them as well.

Done.

I kept the old .png files where they are generated because I did not
remove that part of the plots.sh file.  Showing both ways of doing
this is not necessarily a bad thing.  ;-)

> If there is anything I've missed, please let me know.

Queued and pushed, thank you!

							Thanx, Paul

>         Thanks, Akira
> --
> Akira Yokosawa (2):
>   CodeSamples/cpu/data: plots.sh: Fix typo in xlabel for fre plot
>   CodeSamples/cpu/data: Use "smooth frequency" for plotting histogram
> 
>  .../expand.pl                                 | 42 +++++++++
>  .../plots.sh                                  | 85 ++++++++++++++++++-
>  2 files changed, 126 insertions(+), 1 deletion(-)
>  create mode 100755 CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/expand.pl
> 
> 
> base-commit: 9caac64bfabf85439c480ce928dddef969354369
> -- 
> 2.34.1
> 
> 

Re: [RFT PATCH -perfbook 0/2] Retouch plots in QQA 15.31

[Paul E. McKenney]

On Tue, Jan 30, 2024 at 09:57:06AM -0800, Paul E. McKenney wrote:
> On Wed, Jan 31, 2024 at 12:31:06AM +0900, Akira Yokosawa wrote:
> > Hi Paul,
> > 
> > I'm not sure if you are already working on those figures in .png,
> > but I'm sending updates in scripts.  I'm dropping changes and
> > additions of .eps and .png files generated by the updated scripts.
> > 
> > Here is a list of commands you'd need to reflect the changes in
> > perfbook.pdf.
> > 
> >     pushd CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/
> 
> I removed this directory due to my having tweaked the data to paper over
> bugs in early versions of my scripts.  So I focused instead on
> CodeSamples/cpu/data/kerneltest054.05.atn6.2024.01.22-12.28.01.
> 
> Ah, and I never did push out the commit where I removed the
> CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02
> directory.  Apologies, and it is out there now.
> 
> >     sh plots.sh
> 
> This failed until I installed the groff package to get pfbtops.
> I of course also had to move expand.pl to this directory.
> 
> >     evince coe-nvalue.eps &
> >     evince coe.eps &
> >     evince fre.eps &
> >     evince rfe.eps &
> >     display fre.png &
> 
> I do like the vertical dashed line at x=0.  I see artifacts on my
> screen due to pixel/plot misalignment, but increasing size makes
> things look good.  It will be fine on paper.  And the fill is
> good compared to the funny datapoint labels.
> 
> I adjusted coe.eps's xrange to include the entire curve in the plot.
> 
> And it does look a lot more professional than my original, so thank
> you for that!
> 
> > If those plots look reasonable to you, please copy them to the parent
> > directory and do the "make":
> > 
> >     cp *.eps fre.png ../
> 
> I don't understand why fre.png is needed.  I am leaving it out, so
> perhaps I will learn the hard way.  ;-)
> 
> >     popd
> >     make
> > 
> > If those figures in the resulting perfbook.pdf looks good to you,
> > I'd like you to go forward and commit the changes and additions:
> > 
> >     git add CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/*.eps
> >     git add CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/fre.png
> >     git add CodeSamples/cpu/data/*.eps
> >     git add CodeSamples/cpu/data/fre.png
> > 
> >     git commit -a # with a suitable tag such as Requested-by: me
> > 
> > Furthermore, if you think those .png figures, now obsoleted by the
> > new .eps figures, are not needed anymore, please "git rm" them as well.
> 
> Done.
> 
> I kept the old .png files where they are generated because I did not
> remove that part of the plots.sh file.  Showing both ways of doing
> this is not necessarily a bad thing.  ;-)
> 
> > If there is anything I've missed, please let me know.
> 
> Queued and pushed, thank you!

Except that github choked on the 366MB coe.out file.  The gzip command
compresses it to 72MB, and "tar -cJf" gets it down to 33MB, so maybe I
rebase the compressed version into the commit that created that file.

And change the script that collects the data to do the compression.  ;-)

Now kernel.org had no problem with the full file, but it might also
be good to avoid imposing too much on their storage-space largesse...

Other thoughts?

							Thanx, Paul

> >         Thanks, Akira
> > --
> > Akira Yokosawa (2):
> >   CodeSamples/cpu/data: plots.sh: Fix typo in xlabel for fre plot
> >   CodeSamples/cpu/data: Use "smooth frequency" for plotting histogram
> > 
> >  .../expand.pl                                 | 42 +++++++++
> >  .../plots.sh                                  | 85 ++++++++++++++++++-
> >  2 files changed, 126 insertions(+), 1 deletion(-)
> >  create mode 100755 CodeSamples/cpu/data/kerneltest009.05.atn6.2024.01.17-08.25.02/expand.pl
> > 
> > 
> > base-commit: 9caac64bfabf85439c480ce928dddef969354369
> > -- 
> > 2.34.1
> > 
> > 

Re: [RFT PATCH -perfbook 0/2] Retouch plots in QQA 15.31

[Akira Yokosawa]

[-- Attachment #1: Type: text/plain, Size: 1984 bytes --]

On 2024/01/31 5:20, Paul E. McKenney wrote:
[...]
> 
> Except that github choked on the 366MB coe.out file.  The gzip command
> compresses it to 72MB, and "tar -cJf" gets it down to 33MB, so maybe I
> rebase the compressed version into the commit that created that file.

Quote from github's file size limit at:
    https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github#file-size-limits

    GitHub limits the size of files allowed in repositories. If you
    attempt to add or update a file that is larger than 50 MiB, you
    will receive a warning from Git. The changes will still
    successfully push to your repository, but you can consider removing
    the commit to minimize performance impact. For more information,
    see "Removing files from a repository's history."

    ...

    GitHub blocks files larger than 100 MiB.

You hit the github's limit.

> 
> And change the script that collects the data to do the compression.  ;-)
> 
> Now kernel.org had no problem with the full file, but it might also
> be good to avoid imposing too much on their storage-space largesse...
> 
> Other thoughts?

Well, no.  Using xz compression looks reasonable to me.

By the way, you added the following in the answer to QQA 15.31:

    Alert readers may have noticed that the distribution shown in
    Figure E.10 is monomodal, which those in Figure E.12 and
    Figure E.14 are trimodal.

In coe.png, whose bin_width is 2 I think, I see a spike around
time period = [38, 40].  Count of time period 40 is 140 and the
largest count in coe.dat.

Attached coe.png is the histogram version of the same plot.
The spike is more evident there.

This spike causes slight but significant enough bump in the
right shoulder of Figure E.10 whose bin_width is 40.
You can still say the distribution is monomodal, but I just
wanted to make sure.

Or I can replace the EPS plot with the one with bin_width=2,
if you prefer.

        Thanks, Akira

[-- Attachment #2: coe-hist.png --]
[-- Type: image/png, Size: 6958 bytes --]

Re: [RFT PATCH -perfbook 0/2] Retouch plots in QQA 15.31

[Paul E. McKenney]

On Sat, Feb 03, 2024 at 11:12:39AM +0900, Akira Yokosawa wrote:
> On 2024/01/31 5:20, Paul E. McKenney wrote:
> [...]
> > 
> > Except that github choked on the 366MB coe.out file.  The gzip command
> > compresses it to 72MB, and "tar -cJf" gets it down to 33MB, so maybe I
> > rebase the compressed version into the commit that created that file.
> 
> Quote from github's file size limit at:
>     https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github#file-size-limits
> 
>     GitHub limits the size of files allowed in repositories. If you
>     attempt to add or update a file that is larger than 50 MiB, you
>     will receive a warning from Git. The changes will still
>     successfully push to your repository, but you can consider removing
>     the commit to minimize performance impact. For more information,
>     see "Removing files from a repository's history."
> 
>     ...
> 
>     GitHub blocks files larger than 100 MiB.
> 
> You hit the github's limit.
> 
> > 
> > And change the script that collects the data to do the compression.  ;-)
> > 
> > Now kernel.org had no problem with the full file, but it might also
> > be good to avoid imposing too much on their storage-space largesse...
> > 
> > Other thoughts?
> 
> Well, no.  Using xz compression looks reasonable to me.

I will need to figure something else out if I ever get access to a system
with (say) 500 CPUs.  If nothing else, split it into one compressed file
per CPU.  I am sure that github will love that.  ;-)

> By the way, you added the following in the answer to QQA 15.31:
> 
>     Alert readers may have noticed that the distribution shown in
>     Figure E.10 is monomodal, which those in Figure E.12 and
>     Figure E.14 are trimodal.
> 
> In coe.png, whose bin_width is 2 I think, I see a spike around
> time period = [38, 40].  Count of time period 40 is 140 and the
> largest count in coe.dat.
> 
> Attached coe.png is the histogram version of the same plot.
> The spike is more evident there.
> 
> This spike causes slight but significant enough bump in the
> right shoulder of Figure E.10 whose bin_width is 40.
> You can still say the distribution is monomodal, but I just
> wanted to make sure.
> 
> Or I can replace the EPS plot with the one with bin_width=2,
> if you prefer.

Thank you for checking, and please do replace that plot.  I will
then reword that text to say something like "almost monomodal" and
"emphatically trimodal".

							Thanx, Paul