From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by yocto-www.yoctoproject.org (Postfix, from userid 118) id 0D279E00AA5; Fri, 20 Jan 2017 02:17:15 -0800 (PST) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on yocto-www.yoctoproject.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-HAM-Report: * -5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/, high * trust * [192.35.17.28 listed in list.dnswl.org] * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] Received: from goliath.siemens.de (goliath.siemens.de [192.35.17.28]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id D3E00E00A73 for ; Fri, 20 Jan 2017 02:17:12 -0800 (PST) Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id v0KAH4c9000334 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 20 Jan 2017 11:17:05 +0100 Received: from [139.16.79.32] (MD1FZU9C.ad001.siemens.net [139.16.79.32]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTPS id v0KAH3qw020114 (version=TLSv1.2 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 20 Jan 2017 11:17:04 +0100 To: "Esquivel, Benjamin" , =?UTF-8?B?QW7DrWJhbCBMaW3Ds24=?= , "yocto@yoctoproject.org" , Christian Schuler References: <1484832823-10892-1-git-send-email-pascal.bach@siemens.com> <5880FA7A.5070506@linux.intel.com> <1372815145309948926EAD7C6C26148D49E09CD1@FMSMSX103.amr.corp.intel.com> From: Pascal Bach Message-ID: <3f9a4351-8a4c-1ff9-a1a9-4fbf5d4eee20@siemens.com> Date: Fri, 20 Jan 2017 11:17:03 +0100 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.5.0 MIME-Version: 1.0 In-Reply-To: <1372815145309948926EAD7C6C26148D49E09CD1@FMSMSX103.amr.corp.intel.com> Subject: Re: [ptest-runner][PATCH] feat: generate xml-file X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jan 2017 10:17:15 -0000 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Hi Benjamin and Anibal See inline repsonse. On 19.01.2017 18:53, Esquivel, Benjamin wrote: > Sure Anibal, see my response inline. > >> -----Original Message----- >> From: Aníbal Limón [mailto:anibal.limon@linux.intel.com] >> Sent: Thursday, January 19, 2017 11:42 AM >> To: Pascal Bach ; yocto@yoctoproject.org; >> Christian Schuler ; Esquivel, Benjamin >> >> Subject: Re: [yocto] [ptest-runner][PATCH] feat: generate xml-file >> >> Hi Pascal and Christian, >> >> Your patch looks good at first look, only a few comments about the code >> inline. >> >> We are trying to standardize the results output format across our testing >> components in order to build up-on a Test reporting tool, so let us review the >> XML format. >> >> Benjamin do you have a wiki page link with test result format information? > https://wiki.yoctoproject.org/wiki/QA/xUnit_XML_Template Thanks, the current XML is just a minimal set but it should comply to the xUnit spec. But enhancements are of course welcome. >> Cheers, >> alimon >> >> On 01/19/2017 07:33 AM, Pascal Bach wrote: >>> From: Christian Schuler >>> >>> add a feature that allows the user to generate an XUnit xml-file. >>> This file shows if the ptests run correctly or not. e.g. bash, zlib. >>> It doesn't list the underlaying tests of the package. >>> The feature is activated by console input -x . >>> Unit-tests of the feature are included. >>> >>> Signed-off-by: Christian Schuler >>> Signed-off-by: Pascal Bach >>> --- >>> README.md | 1 + >>> main.c | 15 +++++++++++---- >>> tests/utils.c | 46 +++++++++++++++++++++++++++++++++++++++++++-- >>> utils.c | 60 >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- >>> utils.h | 7 ++++++- >>> 5 files changed, 121 insertions(+), 8 deletions(-) >>> >>> diff --git a/README.md b/README.md >>> index ed7a589..fedab04 100644 >>> --- a/README.md >>> +++ b/README.md >>> @@ -14,6 +14,7 @@ Now the ptest-runner2 support the next features: >>> - List available ptests. >>> - Specify the timeout for avoid blocking indefinetly. >>> - Only run certain ptests. >>> +- XML-ouput >>> >>> Proposed features: >>> >>> diff --git a/main.c b/main.c >>> index 31bf3b5..8dfdad6 100644 >>> --- a/main.c >>> +++ b/main.c >>> @@ -41,7 +41,7 @@ >>> static inline void >>> print_usage(FILE *stream, char *progname) { >>> - fprintf(stream, "Usage: %s [-d directory] [-l list] [-t timeout] " >>> + fprintf(stream, "Usage: %s [-d directory] [-l list] [-t timeout] [-x xml- >> filename]" >>> "[-h] [ptest1 ptest2 ...]\n", progname); } >>> >>> @@ -50,6 +50,7 @@ static struct { >>> int list; >>> int timeout; >>> char **ptests; >>> + char * xml_filename; >>> } opts; >>> >>> int >>> @@ -70,8 +71,9 @@ main(int argc, char *argv[]) >>> opts.list = 0; >>> opts.timeout = DEFAULT_TIMEOUT; >>> opts.ptests = NULL; >>> + opts.xml_filename = NULL; >>> >>> - while ((opt = getopt(argc, argv, "d:lt:h")) != -1) { >>> + while ((opt = getopt(argc, argv, "d:ltx:h")) != -1) { >>> switch (opt) { >>> case 'd': >>> free(opts.directory); >>> @@ -88,6 +90,11 @@ main(int argc, char *argv[]) >>> print_usage(stdout, argv[0]); >>> exit(0); >>> break; >>> + case 'x': >>> + free(opts.xml_filename); >>> + opts.xml_filename = strdup(optarg); >>> + CHECK_ALLOCATION(opts.xml_filename, 1, >> 1); >>> + break; >>> default: >>> print_usage(stdout, argv[0]); >>> exit(1); >>> @@ -131,9 +138,9 @@ main(int argc, char *argv[]) >>> run = filter_ptests(head, opts.ptests, ptest_num); >>> CHECK_ALLOCATION(run, ptest_num, 1); >>> ptest_list_free_all(head); >>> - } >>> + } >>> >>> - rc = run_ptests(run, opts.timeout, argv[0], stdout, stderr); >>> + rc = run_ptests(run, opts.timeout, argv[0], stdout, stderr, >>> +opts.xml_filename); >> Will be a good idea modify the run_ptest signature to pass the whole options >> to it. >> >> I mean: >> >> run_ptests(run, opts, argv[0], stdout, stderr) >> >> If you could do a first patch with this change and then add your functionality >> realted to xml_filename will be good. >> >>> ptest_list_free_all(run); >>> >>> diff --git a/tests/utils.c b/tests/utils.c index 1332798..dd869e9 >>> 100644 >>> --- a/tests/utils.c >>> +++ b/tests/utils.c >>> @@ -173,7 +173,7 @@ START_TEST(test_run_ptests) >>> ptest_list_remove(head, "hang", 1); >>> ptest_list_remove(head, "fail", 1); >>> >>> - rc = run_ptests(head, timeout, "test_run_ptests", fp_stdout, >> fp_stderr); >>> + rc = run_ptests(head, timeout, "test_run_ptests", fp_stdout, >>> +fp_stderr, NULL); >>> ck_assert(rc == 0); >>> ptest_list_free_all(head); >>> >>> @@ -234,6 +234,46 @@ START_TEST(test_run_fail_ptest) >>> ptest_list_free_all(head); >>> END_TEST >>> >>> +START_TEST(test_xml_pass) >>> + char ref[] = "\n> name='ptests' tests='2'>\n" >>> + "\n> result='/'>\n" >>> + "\n" >>> + "\n" >>> + "\n" >>> + "\n" >>> + "\n" >>> + "\n" >>> + "\n" >>> + "\n" >>> + "A Timeout has occured or more tests than expected >> failed.\n" >>> + "\n" >>> + "\n" >>> + "\n" >>> + "\n" >>> + ""; >>> + char buffer[strlen(ref)]; >>> + >>> + FILE *fp; >>> + fp = xml_create(2, "./test_xml"); >>> + xml_add_case(fp, 0,"."); >>> + xml_add_case(fp, 1,"."); >>> + xml_finish(fp); >>> + >>> + fp = fopen("./test_xml", "a+"); >>> + fseek(fp, 0, SEEK_SET); >>> + >>> + fread(buffer, strlen(ref), 1, fp); >>> + ck_assert(strncmp(ref, buffer, strlen(ref)) == 0); >>> + fclose(fp); >>> + unlink("./test_xml"); >>> + >>> +END_TEST >>> + >>> +START_TEST(test_xml_fail) >>> + ck_assert(xml_create(2, "./") == NULL); >>> + >>> +END_TEST >>> + >>> Suite * >>> utils_suite() >>> { >>> @@ -249,6 +289,8 @@ utils_suite() >>> tcase_add_test(tc_core, test_run_ptests); >>> tcase_add_test(tc_core, test_run_timeout_ptest); >>> tcase_add_test(tc_core, test_run_fail_ptest); >>> + tcase_add_test(tc_core, test_xml_pass); >>> + tcase_add_test(tc_core, test_xml_fail); >>> >>> suite_add_tcase(s, tc_core); >>> >>> @@ -276,7 +318,7 @@ test_ptest_expected_failure(struct ptest_list >> *head, const int timeout, char *pr >>> ck_assert(ptest_list_length(filtered) == 1); >>> >>> h_analizer( >>> - run_ptests(filtered, timeout, progname, fp_stdout, >> fp_stderr), >>> + run_ptests(filtered, timeout, progname, fp_stdout, >> fp_stderr, >>> +NULL), >>> fp_stdout, fp_stderr >>> ); >>> >>> diff --git a/utils.c b/utils.c >>> index 77427e0..430fb7a 100644 >>> --- a/utils.c >>> +++ b/utils.c >>> @@ -303,9 +303,10 @@ wait_child(const char *ptest_dir, const char >>> *run_ptest, pid_t pid, >>> >>> int >>> run_ptests(struct ptest_list *head, int timeout, const char *progname, >>> - FILE *fp, FILE *fp_stderr) >>> + FILE *fp, FILE *fp_stderr, char *xml_filename) >>> { >>> int rc = 0; >>> + XML *xh; >>> >>> struct ptest_list *p; >>> char stime[GET_STIME_BUF_SIZE]; >>> @@ -314,6 +315,12 @@ run_ptests(struct ptest_list *head, int timeout, >> const char *progname, >>> int pipefd_stdout[2]; >>> int pipefd_stderr[2]; >>> >>> + if(xml_filename) { >>> + xh = xml_create(ptest_list_length(head), xml_filename); >>> + if(!xh) >>> + exit(EXIT_FAILURE); >>> + } >>> + >>> do >>> { >>> if ((rc = pipe2(pipefd_stdout, O_NONBLOCK)) == -1) @@ - >> 355,6 +362,9 >>> @@ run_ptests(struct ptest_list *head, int timeout, const char >> *progname, >>> fprintf(fp, "ERROR: Exit status is >> %d\n", status); >>> rc += 1; >>> } >>> + if (xml_filename) >>> + xml_add_case(xh, status, ptest_dir); >>> + >>> fprintf(fp, "END: %s\n", ptest_dir); >>> fprintf(fp, "%s\n", get_stime(stime, >> GET_STIME_BUF_SIZE)); >>> } >>> @@ -368,5 +378,53 @@ run_ptests(struct ptest_list *head, int timeout, >> const char *progname, >>> if (rc == -1) >>> fprintf(fp_stderr, "run_ptests fails: %s", strerror(errno)); >>> >>> + if (xml_filename) >>> + xml_finish(xh); >>> + >>> return rc; >>> } >>> + >>> +FILE * >>> +xml_create(int test_count, char *xml_filename) { >>> + FILE *xh; >>> + >>> + if ((xh = fopen(xml_filename, "w"))) { >>> + fprintf(xh, "\n"); >>> + fprintf(xh, "\n", >> test_count); >>> + } >>> + else { >>> + fprintf(stderr, "File could not be created. Make sure the path >> is correct and you have the right permissions."); >>> + return NULL; >>> + } >>> + >>> + return xh; >>> +} >>> + >>> +void >>> +xml_add_case(FILE *xh, int status, const char *ptest_dir) { >>> + fprintf(xh, "\n", ptest_dir); >>> + fprintf(xh, "\n"); >>> + >>> + if(status == 0){ >>> + fprintf(xh, "\n"); >>> + fprintf(xh, "\n"); >>> + } >>> + else{ >>> + fprintf(xh, "\n", status); >>> + fprintf(xh, "\n"); >>> + fprintf(xh, "A Timeout has occured or more tests than >> expected failed.\n"); >>> + fprintf(xh, "\n"); >>> + fprintf(xh, "\n"); >>> + } >>> + fprintf(xh, "\n"); >>> + fprintf(xh, "\n"); >>> +} >>> + >>> +void >>> +xml_finish(FILE *xh) >>> +{ >>> + fprintf(xh, ""); >>> + fclose(xh); >>> +} >>> diff --git a/utils.h b/utils.h >>> index 4dc23ef..99f6100 100644 >>> --- a/utils.h >>> +++ b/utils.h >>> @@ -30,10 +30,15 @@ >>> #define CHECK_ALLOCATION(p, size, exit_on_null) \ >>> check_allocation1(p, size, __FILE__, __LINE__, exit_on_null) >>> >>> +typedef FILE XML; >>> + >>> extern void check_allocation1(void *, size_t, char *, int, int); >>> extern struct ptest_list *get_available_ptests(const char *); extern >>> int print_ptests(struct ptest_list *, FILE *); extern struct >>> ptest_list *filter_ptests(struct ptest_list *, char **, int); -extern >>> int run_ptests(struct ptest_list *, int, const char *progname, FILE *, >>> FILE *); >>> +extern int run_ptests(struct ptest_list *, int, const char *progname, >>> +FILE *, FILE *, char *xml_filename); extern XML * xml_create(int >>> +test_count, char *filename); extern void xml_add_case(XML *, int >>> +status, const char *ptest_dir); extern void xml_finish(XML *); >> I'm trying to follow BSD code-style so please remove variable names from >> prototypes. Also i make a mistake with progname that was before on the >> run_ptests prototype. We will do a rework of the patch that addresses the style issue. The XML we leave untouched for the moment. Pascal >> Also i don't wrote about code style into README, i will put a section into it, >> sorry for that. >> >> Best regards, >> alimon >> >>> #endif >>>