Re: [meta-oe][PATCH] add a storage I/O benchmark test suite

[Paolo Valente <paolo.valente@linaro.org>]

[RESENDING: FIRST ATTEMPT REJECTED BECAUSE I WAS NOT SUBSCRIBED TO THE LIST]

> Il giorno 23 ott 2018, alle ore 15:33, Anders Roxell <anders.roxell@linaro.org> ha scritto:
> 
> CCing in the package maintainer.
> 

Thanks Anders, hi everybody,

> On Sun, 21 Oct 2018 at 17:10, Khem Raj <raj.khem@gmail.com> wrote:
>> 
>> On Sun, Oct 21, 2018 at 12:29 AM Randy MacLeod
>> <randy.macleod@windriver.com> wrote:
>>> 
>>> On 10/19/18 10:54 AM, Anders Roxell wrote:
>>>> Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
>>>> ---
>>>> .../recipes-benchmark/s-suite/s-suite_git.bb  | 30 +++++++++++++++++++
>>>> 1 file changed, 30 insertions(+)
>>>> create mode 100644 meta-oe/recipes-benchmark/s-suite/s-suite_git.bb
>>>> 
>>>> diff --git a/meta-oe/recipes-benchmark/s-suite/s-suite_git.bb b/meta-oe/recipes-benchmark/s-suite/s-suite_git.bb
>>>> new file mode 100644
>>>> index 000000000000..77d172814065
>>>> --- /dev/null
>>>> +++ b/meta-oe/recipes-benchmark/s-suite/s-suite_git.bb
>>>> @@ -0,0 +1,30 @@
>>>> +SUMMARY = "Small collection of benchmarks for storage I/O"
>>>> +LICENSE = "GPLv2"
>>>> +LIC_FILES_CHKSUM = "file://COPYING;md5=b529aaa6a0c50f15d29f89609b5c22f3"
>>>> +
>>>> +SRCREV = "79698f645bfb28d0d966484ddad3a1efb562246d"
>>>> +PV = "0.0+git${SRCPV}"
>>>> +SRC_URI = "git://github.com/Algodev-github/S.git;protocol=https;branch=master"
>>>> +
>>>> +S = "${WORKDIR}/git"
>>>> +
>>>> +# installing in /opt/S-suite since the package has
>>>> +# dependencies to the directory structure.
>>>> +do_install() {
>>>> +    install -d ${D}/opt/S-suite
>>>> +    for i in $(find ${S}/* -type d); do
>>>> +        install -d ${D}/opt/S-suite/$(basename $i)
>>>> +        install -m0755 -p ${S}/$(basename $i)/* ${D}/opt/S-suite/$(basename $i)
>>>> +    done
>>>> +
>>>> +    install -m0755 ${S}/config_params.sh ${D}/opt/S-suite
>>>> +    install -m0755 ${S}/def_config_params.sh ${D}/opt/S-suite
>>>> +}
>>>> +
>>>> +RDEPENDS_${PN} = "bash bc coreutils gawk g++ gcc fio libaio libaio-dev sysstat"
>>>> +
>>>> +FILES_${PN} = "/opt/S-suite/"
>>>> +
>>>> +# added to INSANE_SKIP since s-suite have an runtime
>>>> +# dependency (RDEPENDS) on libaio-dev.
>>>> +INSANE_SKIP_${PN} += "dev-deps"
>>>> 
>>> 
>>> Thanks for the submission but from:
>>> 
>>>   https://github.com/Algodev-github/S
>>> 
>>>   "These benchmarks have been written just for internal use,
>>>    and are more a yard than a well-finished work."
>>> 
>>> 
>>> That doesn't lead to a lot of confidence that these suite will
>>> be maintained for years.

Gosh.  I put that disclaimer in the README when I first wrote it more
than 10 years ago.  After then, thousands of changes have been made by
me and many other contributors.  Unfortunately, I put the suite on
GitHub only recently.  But as you can see from the log, there have
been 448 commits in (less than) the last five months.

In addition, the S suite is now used in a new Phoronix benchmark and
in a new mmtests benchmark.

So, believe me, the S suite ATM is definitely in good shape, and seems
to be here to stay.

I'll remove that line, right after finishing a non-trivial new commit
I'm working on.

>>> What's the value to meta-oe users to add
>>> this recipe compared to something that has been around for years
>>> if not decades such as Bonnie++:
>>>   https://layers.openembedded.org/layerindex/recipe/864/
>>> 
>> 

The added value is that bonnie++ does not perform *any* of the benchmarks in the S suite, apart from that on throughput. I have made the S suite, many years ago, because I needed to run benchmarks that no suite provided. Here are some examples:
- start-up times of real applications under real background workloads
- playback quality (drop rate) of video and audio under real background workloads
- speed of code-development tasks (make, git checkout, git merge, git
 grep) under real background workloads, plus measurement of system
 responsiveness while one of these dev tasks is executed
- minimum per-client bandwidth guaranteed to a set of clients doing
 any possible type of I/O
- maximum per-client latency guaranteed to a set of clients doing any
 possible type of I/O

For any doubt, here I am.

Thanks,
Paolo

>> how I read this is that they have not made proper installation
>> targets and its dependencies or letting user know of the dependencies
>> which actually is not a problem with OE since both
>> of these would be taken care of in the recipe with
>> do_install and DEPENDS
> 
> Do you mean DEPENDS or RDEPENDS?
> 
> Cheers,
> Anders