[Fuego] Fuego criteria file documentation

[Fuego] Fuego criteria file documentation

[Bird, Timothy]

Hello all (but especially Daniel),

I've been writing the documentation for new features in the Fuego 1.2 release,
and I've put together a page about the new criteria file.

The page is here:
http://fuegotest.org/wiki/criteria.json

Can you please take a look and see the information there is correct?

Thanks,
 -- Tim

Re: [Fuego] Fuego criteria file documentation

[Daniel Sangorrin]

Hi Tim,

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> Sent: Thursday, November 02, 2017 9:53 AM
> To: fuego@lists.linuxfoundation.org
> Cc: Daniel Sangorrin
> Subject: Fuego criteria file documentation
> 
> Hello all (but especially Daniel),
> 
> I've been writing the documentation for new features in the Fuego 1.2 release,
> and I've put together a page about the new criteria file.
> 
> The page is here:
> http://fuegotest.org/wiki/criteria.json
> 
> Can you please take a look and see the information there is correct?

Thanks for putting all this important documentation together.
I just reviewed it and I think that it reflects accurately the current status of criteria.json.

I have some comments about the section "Customizing the criteria.json file for a board":

-  I found a small errata: FUEGO_CRITERA_JSON_PATH -> FUEGO_CRITERIA_JSON_PATH

- FUEGO_CRITERIA_JSON_PATH: it seems to me that this functionality is not completely
implemented yet. For example, there are no checks on ftc add-job/run-test and the 
variable is not stored together with other test flags (such as REBUILD or REBOOT).
Personally, I would add a "--criteria-json-path" parameter to ftc add-job/run-test.
What do you think? I can work on this.

- I think we should provide a flag "--store-criteria" (by default active) to indicate whether criteria.json
should be stored on the log folder for sharing or future reference.

- The same we are doing for criteria.json is probably required for specs.json and chart_config.json.

Thanks,
Daniel

Re: [Fuego] Fuego criteria file documentation

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Sunday, November 05, 2017 9:03 PM
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> >
> > Hello all (but especially Daniel),
> >
> > I've been writing the documentation for new features in the Fuego 1.2
> release,
> > and I've put together a page about the new criteria file.
> >
> > The page is here:
> > http://fuegotest.org/wiki/criteria.json
> >
> > Can you please take a look and see the information there is correct?
> 
> Thanks for putting all this important documentation together.
> I just reviewed it and I think that it reflects accurately the current status of
> criteria.json.
> 
> I have some comments about the section "Customizing the criteria.json file
> for a board":
> 
> -  I found a small errata: FUEGO_CRITERA_JSON_PATH ->
> FUEGO_CRITERIA_JSON_PATH
Fixed.  Thanks.

> 
> - FUEGO_CRITERIA_JSON_PATH: it seems to me that this functionality is not
> completely
> implemented yet. For example, there are no checks on ftc add-job/run-test
> and the
> variable is not stored together with other test flags (such as REBUILD or
> REBOOT).
> Personally, I would add a "--criteria-json-path" parameter to ftc add-job/run-
> test.
> What do you think? I can work on this.

Sounds good.  I think it would be better as just "--criteria-path".  I don't
want to encode the file format in the command line argument.
Using the same rationale, maybe we should remove "JSON_" from
FUEGO_CRITERIA_JSON_PATH as well (so it's just FUEGO_CRITERIA_PATH).

> 
> - I think we should provide a flag "--store-criteria" (by default active) to
> indicate whether criteria.json
> should be stored on the log folder for sharing or future reference.
I would lean towards always storing it, and not having a flag.

> 
> - The same we are doing for criteria.json is probably required for specs.json
> and chart_config.json.
I would agree with saving the spec that was used in the test.  Since someone
might change the spec after running the test, it should get recorded for an
individual run.  I'm not sure we need the whole file, and specs are short enough
that it might just be easier to save the variable definitions from the spec, in
the run.json file.  Let me know what you think.

With regard to chart_config.json - that's a presentation detail that isn't tied
to specific run.  It's tied more to a test than to a run.
 -- Tim

Re: [Fuego] Fuego criteria file documentation

[Daniel Sangorrin]

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > -----Original Message-----
> > From: Daniel Sangorrin on Sunday, November 05, 2017 9:03 PM
> > > -----Original Message-----
> > > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > >
> > > Hello all (but especially Daniel),
> > >
> > > I've been writing the documentation for new features in the Fuego 1.2
> > release,
> > > and I've put together a page about the new criteria file.
> > >
> > > The page is here:
> > > http://fuegotest.org/wiki/criteria.json
> > >
> > > Can you please take a look and see the information there is correct?
> >
> > Thanks for putting all this important documentation together.
> > I just reviewed it and I think that it reflects accurately the current status of
> > criteria.json.
> >
> > I have some comments about the section "Customizing the criteria.json file
> > for a board":
> >
> > -  I found a small errata: FUEGO_CRITERA_JSON_PATH ->
> > FUEGO_CRITERIA_JSON_PATH
> Fixed.  Thanks.
> 
> >
> > - FUEGO_CRITERIA_JSON_PATH: it seems to me that this functionality is not
> > completely
> > implemented yet. For example, there are no checks on ftc add-job/run-test
> > and the
> > variable is not stored together with other test flags (such as REBUILD or
> > REBOOT).
> > Personally, I would add a "--criteria-json-path" parameter to ftc add-job/run-
> > test.
> > What do you think? I can work on this.
> 
> Sounds good.  I think it would be better as just "--criteria-path".  I don't
> want to encode the file format in the command line argument.
> Using the same rationale, maybe we should remove "JSON_" from
> FUEGO_CRITERIA_JSON_PATH as well (so it's just FUEGO_CRITERIA_PATH).

OK, I will work on this.

> > - I think we should provide a flag "--store-criteria" (by default active) to
> > indicate whether criteria.json
> > should be stored on the log folder for sharing or future reference.
> I would lean towards always storing it, and not having a flag.

OK. I will work on this as well.
Should we store criteria.json files that have not been modified as well?
Probably we could infer it from the "fuego_core_version" variable but it would
be a bit tedious (git checkout etc..)

> > - The same we are doing for criteria.json is probably required for specs.json
> > and chart_config.json.
> I would agree with saving the spec that was used in the test.  Since someone
> might change the spec after running the test, it should get recorded for an
> individual run.  I'm not sure we need the whole file, and specs are short enough
> that it might just be easier to save the variable definitions from the spec, in
> the run.json file.  Let me know what you think.
> 
> With regard to chart_config.json - that's a presentation detail that isn't tied
> to specific run.  It's tied more to a test than to a run.

For chart_config.json I meant that we should also provide a parameter (--chart-config-path?)
for the user to customize what gets displayed. For example, depending on the board, spec or
even the user's taste/goal the test results to display may differ. Suppose I am
working a specific part of the kernel and I want LTP to only display a table with results 
of those testcases that I consider relevant. 

When sharing the results to a centralized server, having the chart_config.json 
could be beneficial if the user wants to specify what results should be displayed
and which not. It would also easy backwards-compatibility if the test names
change.

Thanks,
Daniel

Re: [Fuego] Fuego criteria file documentation

[Liu, Wenlong]

[-- Attachment #1: Type: text/plain, Size: 2065 bytes --]


> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org
> [mailto:fuego-bounces@lists.linuxfoundation.org] On Behalf Of Bird,
> Timothy
> Sent: Thursday, November 02, 2017 8:53 AM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] Fuego criteria file documentation
> 
> Hello all (but especially Daniel),
> 
> I've been writing the documentation for new features in the Fuego 1.2 release,
> and I've put together a page about the new criteria file.
> 
> The page is here:
> http://fuegotest.org/wiki/criteria.json
> 
> Can you please take a look and see the information there is correct?
I have seen the current Fuego criteria documentation.
One question as below.

As shown in the example of this documentation,
{
    "schema_version":"1.0",
    "criteria": [
    {
        "tguid": "syscall",
        "min_pass": 1000,
        "max_fail": 5
    },
    {
        "tguid": "timers",
        "fail_ok_list": ["leapsec_timer"]
    },
    {
        "tguid": "pty",
        "must_pass_list": ["hangup01"]
    }
    ]
}
Yes, the example can works well.
However, you know, for LTP test, there are more than 70 sub-testsuites.

For sub-testsuite named "net.nfs", the criteria json file cannot works as I expected even though I added "net.nfs" as above.
When function "split_test_id(test_case_id)" was executed, the testset name was changed to "default" and we cannot find the correct criteria json file. 
The reason might be that "net.nfs" has a character ".".

So, any advice for this situation or it should be fixed?

PS: 
* Attached is my criteria json file for AGL, although it cannot works as I expected(reference only).
  (LTP Version:20170516; Target: R-car M3)
* I didn't list all network related tests in this criteria json file because of the issue above.

Best regards
Liu

> Thanks,
>  -- Tim
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
> 




[-- Attachment #2: m3ulcb-Functional.LTP-criteria.json --]
[-- Type: application/octet-stream, Size: 1596 bytes --]

{
    "schema_version":"1.0",
    "criteria": [
    {
        "tguid": "syscall",
        "min_pass": 1000,
        "max_fail": 5
    },
    {
        "tguid": "admin_tools",
        "fail_ok_list": ["su01", "cron_allow01", "cron_dirs_checks01", "acl_test01"]
    },
    {
        "tguid": "commands",
        "fail_ok_list": ["cron", "lsmod01"]
    },
    {
        "tguid": "containers",
        "fail_ok_list": ["netns_sysfs", "netns_breakns_ns_exec_ipv6_netlink", "netns_breakns_ns_exec_ipv6_ioctl", "netns_breakns_ip_ipv6_netlink", "netns_comm_ns_exec_ipv6_ioctl", "netns_comm_ip_ipv6_netlink", "netns_comm_ns_exec_ipv6_netlink", "netns_breakns_ip_ipv6_ioctl", "userns06", "netns_comm_ip_ipv6_ioctl"]
    },
    {
        "tguid": "controllers",
        "max_fail": 5
    },
    {
        "tguid": "cpuhotplug",
        "fail_ok_list": ["cpuhotplug03", "cpuhotplug04", "cpuhotplug05"]
    },
    {
        "tguid": "fs",
        "fail_ok_list": ["isofs", "quota_remount_test01"]
    },
    {
        "tguid": "hugetlb",
        "fail_ok_list": ["hugeshmat04", "hugeshmat05", "hugeshmat06"]
    },
    {
        "tguid": "ima",
        "fail_ok_list": ["ima01", "ima02", "ima03", "ima04"]
    },
    {
        "tguid": "smack",
        "min_pass": 6
    },
    {
        "tguid": "stress.part1",
        "fail_ok_list": ["nfs01", "nfs02", "nfs03", "nfs04", "nfs05", "nfsx-linux"]
    },
    {
        "tguid": "net.tirpc_tests",
        "fail_ok_list": ["tirpc_authdes_seccreate", "tirpc_authdes_create"]
    },
    {
        "tguid": "net.rpc",
        "fail_ok_list": ["rup"]
    }
    ]
}

Re: [Fuego] Fuego criteria file documentation

[Bird, Timothy]

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Liu, Wenlong
> Sent: Monday, November 13, 2017 2:38 AM
> To: fuego@lists.linuxfoundation.org
> Subject: Re: [Fuego] Fuego criteria file documentation
> 
> 
> > -----Original Message-----
> > From: fuego-bounces@lists.linuxfoundation.org
> > [mailto:fuego-bounces@lists.linuxfoundation.org] On Behalf Of Bird,
> > Timothy
> > Sent: Thursday, November 02, 2017 8:53 AM
> > To: fuego@lists.linuxfoundation.org
> > Subject: [Fuego] Fuego criteria file documentation
> >
> > Hello all (but especially Daniel),
> >
> > I've been writing the documentation for new features in the Fuego 1.2
> release,
> > and I've put together a page about the new criteria file.
> >
> > The page is here:
> > http://fuegotest.org/wiki/criteria.json
> >
> > Can you please take a look and see the information there is correct?
> I have seen the current Fuego criteria documentation.
> One question as below.
> 
> As shown in the example of this documentation,
> {
>     "schema_version":"1.0",
>     "criteria": [
>     {
>         "tguid": "syscall",
>         "min_pass": 1000,
>         "max_fail": 5
>     },
>     {
>         "tguid": "timers",
>         "fail_ok_list": ["leapsec_timer"]
>     },
>     {
>         "tguid": "pty",
>         "must_pass_list": ["hangup01"]
>     }
>     ]
> }
> Yes, the example can works well.
> However, you know, for LTP test, there are more than 70 sub-testsuites.
> 
> For sub-testsuite named "net.nfs", the criteria json file cannot works as I
> expected even though I added "net.nfs" as above.
> When function "split_test_id(test_case_id)" was executed, the testset name
> was changed to "default" and we cannot find the correct criteria json file.
> The reason might be that "net.nfs" has a character ".".

That was indeed the cause.  The full test_case_id included multiple
periods, which split_test_id did not handle correctly.

> 
> So, any advice for this situation or it should be fixed?
I corrected this routine, with commit: f50cff2 in the master branch.

However, I didn't try to use a criteria file with with net.nfs.  Can you
check out the latest code and try again?

> 
> PS:
> * Attached is my criteria json file for AGL, although it cannot works as I
> expected(reference only).
>   (LTP Version:20170516; Target: R-car M3)
> * I didn't list all network related tests in this criteria json file because of the
> issue above.

Thanks for the bug report.
 -- Tim

Re: [Fuego] Fuego criteria file documentation

[Liu, Wenlong]

[-- Attachment #1: Type: text/plain, Size: 4438 bytes --]


> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> Sent: Tuesday, November 14, 2017 9:53 AM
> To: Liu, Wenlong/刘 文龙 <liuwl.fnst@cn.fujitsu.com>;
> fuego@lists.linuxfoundation.org
> Subject: RE: Fuego criteria file documentation
> 
> 
> 
> > -----Original Message-----
> > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > bounces@lists.linuxfoundation.org] On Behalf Of Liu, Wenlong
> > Sent: Monday, November 13, 2017 2:38 AM
> > To: fuego@lists.linuxfoundation.org
> > Subject: Re: [Fuego] Fuego criteria file documentation
> >
> >
> > > -----Original Message-----
> > > From: fuego-bounces@lists.linuxfoundation.org
> > > [mailto:fuego-bounces@lists.linuxfoundation.org] On Behalf Of Bird,
> > > Timothy
> > > Sent: Thursday, November 02, 2017 8:53 AM
> > > To: fuego@lists.linuxfoundation.org
> > > Subject: [Fuego] Fuego criteria file documentation
> > >
> > > Hello all (but especially Daniel),
> > >
> > > I've been writing the documentation for new features in the Fuego
> > > 1.2
> > release,
> > > and I've put together a page about the new criteria file.
> > >
> > > The page is here:
> > > http://fuegotest.org/wiki/criteria.json
> > >
> > > Can you please take a look and see the information there is correct?
> > I have seen the current Fuego criteria documentation.
> > One question as below.
> >
> > As shown in the example of this documentation, {
> >     "schema_version":"1.0",
> >     "criteria": [
> >     {
> >         "tguid": "syscall",
> >         "min_pass": 1000,
> >         "max_fail": 5
> >     },
> >     {
> >         "tguid": "timers",
> >         "fail_ok_list": ["leapsec_timer"]
> >     },
> >     {
> >         "tguid": "pty",
> >         "must_pass_list": ["hangup01"]
> >     }
> >     ]
> > }
> > Yes, the example can works well.
> > However, you know, for LTP test, there are more than 70 sub-testsuites.
> >
> > For sub-testsuite named "net.nfs", the criteria json file cannot works
> > as I expected even though I added "net.nfs" as above.
> > When function "split_test_id(test_case_id)" was executed, the testset
> > name was changed to "default" and we cannot find the correct criteria
> json file.
> > The reason might be that "net.nfs" has a character ".".
> 
> That was indeed the cause.  The full test_case_id included multiple periods,
> which split_test_id did not handle correctly.
> 
> >
> > So, any advice for this situation or it should be fixed?
> I corrected this routine, with commit: f50cff2 in the master branch.
> 
> However, I didn't try to use a criteria file with with net.nfs.  Can you
> check out the latest code and try again?

I have tested this and find one problem.
It'll give the error log as below,

 ... ...
 Writing merged results to  /fuego-rw/logs/Functional.LTP/results.json
 Writing flat results to /fuego-rw/logs/Functional.LTP/flat_plot_data.txt
 Traceback (most recent call last):
   File "/fuego-core/engine/tests/Functional.LTP/parser.py", line 206, in <module>
     sys.exit(plib.process(test_results))
   File "/fuego-core/engine/scripts/parser/common.py", line 577, in process
     make_chart_data(test_logdir, TESTDIR, CHART_CONFIG_JSON, data_lines)
   File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 703, in make_chart_data
     chart_list = make_testset_summary_table(test_name, chart_config, entries)
   File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 622, in make_testset_summary_table
     ssb_list.sort(ssb_cmp)
   File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 543, in ssb_cmp
     spec,ts,bn = ssb.split(".")
 ValueError: too many values to unpack
 ERROR: results did not satisfy the threshold
 ... ...

My test job name is ubuntu.stress.part3.Functional.LTP.
When Fuego try to make the summary table, function "ssb_cmp" cannot handle this situation.

In order to continue my test work, I temporarily used the solution in the attachment.
But I strongly doubt if it'll be a good solution.

> >
> > PS:
> > * Attached is my criteria json file for AGL, although it cannot works
> > as I expected(reference only).
> >   (LTP Version:20170516; Target: R-car M3)
> > * I didn't list all network related tests in this criteria json file
> > because of the issue above.
> 
> Thanks for the bug report.
>  -- Tim
> 
> 




[-- Attachment #2: 0001-chart-handle-test_set-or-test_spec-names-with-embedd.patch --]
[-- Type: application/octet-stream, Size: 2601 bytes --]

From bdd6e148af8036bb262fceee45444f91edc82a9a Mon Sep 17 00:00:00 2001
From: Liu Wenlong <liuwl.fnst@cn.fujitsu.com>
Date: Fri, 17 Nov 2017 01:27:57 -0500
Subject: [PATCH 1/4] chart: handle test_set or test_spec names with embedded
 periods

Function "make_testset_summary_table" will give a exception if
test_set or test_spec names with embedded periods.

Test_spec names should use underscores, but Fuego naming rules allow
a test set name to have a period.
Change delimiter from "." to "|" will be easy to handle cases where
the test_case_id or test_spec comes in with more than one period for
function make_testset_summary_table.

Signed-off-by: Liu Wenlong <liuwl.fnst@cn.fujitsu.com>
---
 engine/scripts/parser/prepare_chart_data.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/engine/scripts/parser/prepare_chart_data.py b/engine/scripts/parser/prepare_chart_data.py
index f86906e..681b3fa 100644
--- a/engine/scripts/parser/prepare_chart_data.py
+++ b/engine/scripts/parser/prepare_chart_data.py
@@ -540,8 +540,8 @@ def make_testcase_table(test_name, chart_config, entries):
 #     print d["Points"],d["Name"]
 
 def ssb_cmp(ssb, ssb2):
-    spec,ts,bn = ssb.split(".")
-    spec2,ts2,bn2 = ssb2.split(".")
+    spec,ts,bn = ssb.split("|")
+    spec2,ts2,bn2 = ssb2.split("|")
     if spec == spec2:
         if bn == bn2:
             return cmp(ts,ts2)
@@ -582,7 +582,7 @@ def make_testset_summary_table(test_name, chart_config, entries):
             if entry.kernel not in kernel_list:
                 kernel_list.append(entry.kernel)
 
-            ssb_key = entry.spec + "." + entry.test_set + "." + str(entry.build_number)
+            ssb_key = entry.spec + "|" + entry.test_set + "|" + str(entry.build_number)
             if not ssb_map.has_key(ssb_key):
                 ssb_map[ssb_key] = [0,0,0,0]
 
@@ -624,7 +624,7 @@ def make_testset_summary_table(test_name, chart_config, entries):
         spec_rows = {}
         sbn_rows = {}
         for ssb in ssb_list:
-            spec,ts,bn = ssb.split(".")
+            spec,ts,bn = ssb.split("|")
             if not spec in spec_rows:
                 spec_rows[spec] = 1
             else:
@@ -639,7 +639,7 @@ def make_testset_summary_table(test_name, chart_config, entries):
         last_bn = ""
         last_spec = ""
         for ssb in ssb_list:
-            spec, test_set, build_number = ssb.split(".")
+            spec, test_set, build_number = ssb.split("|")
             if spec != last_spec:
                 last_spec = spec
                 build_number = build_number
-- 
2.7.4

Re: [Fuego] Fuego criteria file documentation

[Bird, Timothy]

See comments inline below

> -----Original Message-----
> From: Liu, Wenlong on Sunday, November 19, 2017 10:54 PM
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > > -----Original Message-----
> > > From Liu, Wenlong on Monday, November 13, 2017 2:38 AM
> > > > -----Original Message-----
> > > > From: fuego-bounces@lists.linuxfoundation.org
> > > > [mailto:fuego-bounces@lists.linuxfoundation.org] On Behalf Of Bird,
> > > > Timothy
> > > > Sent: Thursday, November 02, 2017 8:53 AM
> > > > To: fuego@lists.linuxfoundation.org
> > > > Subject: [Fuego] Fuego criteria file documentation
> > > >
> > > > Hello all (but especially Daniel),
> > > >
> > > > I've been writing the documentation for new features in the Fuego
> > > > 1.2
> > > release,
> > > > and I've put together a page about the new criteria file.
> > > >
> > > > The page is here:
> > > > http://fuegotest.org/wiki/criteria.json
> > > >
> > > > Can you please take a look and see the information there is correct?
> > > I have seen the current Fuego criteria documentation.
> > > One question as below.
> > >
> > > As shown in the example of this documentation, {
> > >     "schema_version":"1.0",
> > >     "criteria": [
> > >     {
> > >         "tguid": "syscall",
> > >         "min_pass": 1000,
> > >         "max_fail": 5
> > >     },
> > >     {
> > >         "tguid": "timers",
> > >         "fail_ok_list": ["leapsec_timer"]
> > >     },
> > >     {
> > >         "tguid": "pty",
> > >         "must_pass_list": ["hangup01"]
> > >     }
> > >     ]
> > > }
> > > Yes, the example can works well.
> > > However, you know, for LTP test, there are more than 70 sub-testsuites.
> > >
> > > For sub-testsuite named "net.nfs", the criteria json file cannot works
> > > as I expected even though I added "net.nfs" as above.
> > > When function "split_test_id(test_case_id)" was executed, the testset
> > > name was changed to "default" and we cannot find the correct criteria
> > json file.
> > > The reason might be that "net.nfs" has a character ".".
> >
> > That was indeed the cause.  The full test_case_id included multiple periods,
> > which split_test_id did not handle correctly.
> >
> > >
> > > So, any advice for this situation or it should be fixed?
> > I corrected this routine, with commit: f50cff2 in the master branch.
> >
> > However, I didn't try to use a criteria file with with net.nfs.  Can you
> > check out the latest code and try again?
> 
> I have tested this and find one problem.
> It'll give the error log as below,
> 
>  ... ...
>  Writing merged results to  /fuego-rw/logs/Functional.LTP/results.json
>  Writing flat results to /fuego-rw/logs/Functional.LTP/flat_plot_data.txt
>  Traceback (most recent call last):
>    File "/fuego-core/engine/tests/Functional.LTP/parser.py", line 206, in
> <module>
>      sys.exit(plib.process(test_results))
>    File "/fuego-core/engine/scripts/parser/common.py", line 577, in process
>      make_chart_data(test_logdir, TESTDIR, CHART_CONFIG_JSON, data_lines)
>    File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 703,
> in make_chart_data
>      chart_list = make_testset_summary_table(test_name, chart_config,
> entries)
>    File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 622,
> in make_testset_summary_table
>      ssb_list.sort(ssb_cmp)
>    File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 543,
> in ssb_cmp
>      spec,ts,bn = ssb.split(".")
>  ValueError: too many values to unpack
>  ERROR: results did not satisfy the threshold
>  ... ...
> 
> My test job name is ubuntu.stress.part3.Functional.LTP.
> When Fuego try to make the summary table, function "ssb_cmp" cannot
> handle this situation.
> 
> In order to continue my test work, I temporarily used the solution in the
> attachment.
> But I strongly doubt if it'll be a good solution.
Since the delimiter is only used internally, I think the solution is OK.
I changed the delimiter from "|" to "," purely out of personal
preference, but otherwise kept your patch, and applied it.

Please try it out to make sure I didn't mess something up.

Thanks!
 -- Tim