From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Sony.onmicrosoft.com; s=selector1-Sony-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XFx96kpgOcXnaW+Sf7Yo5M2LXl5LMshitD3hG9CKd3Q=; b=Rpxgxcwcpc0MrJXBBhyYnw2+GRHD1i8aLq/AZlhawi9lvXcv44WN7ujZG5PLKr8hbUxdDnVH29ayJeGIblHe3UouORG0abNakMjDqErjRCeA74cTntD9IwbpF5tmtL09v33UB0i2xQYREVpSucyQZgfyR+WzXYRt8XbfKUiAVZI= From: Date: Fri, 13 Jul 2018 17:05:34 +0000 Message-ID: References: <1530708469-14477-1-git-send-email-lixm.fnst@cn.fujitsu.com> <62079D2F712F7747B0BDCC6821B8DF0D013CF2F1@G08CNEXMBPEKD02.g08.fujitsu.local> In-Reply-To: <62079D2F712F7747B0BDCC6821B8DF0D013CF2F1@G08CNEXMBPEKD02.g08.fujitsu.local> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Fuego] [PATCH] LTP_Smack: add a new job to test "smack" module List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: lixm.fnst@cn.fujitsu.com, fuego@lists.linuxfoundation.org Li, Thank you for this test. I will make some comments about the test in this e-mail (see below), and then address Daniel's comments in a separa= te e-mail.=20 > -----Original Message----- > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego- > bounces@lists.linuxfoundation.org] On Behalf Of Li, Xiaoming > Sent: Thursday, July 05, 2018 10:50 PM > To: fuego@lists.linuxfoundation.org > Subject: [Fuego] [PATCH] LTP_Smack: add a new job to test "smack" module >=20 >=20 > Signed-off-by: Li Xiaoming > --- > engine/tests/Functional.LTP_Smack/fuego_test.sh | 77 +++++++ > engine/tests/Functional.LTP_Smack/parser.py | 270 > ++++++++++++++++++++++++ > engine/tests/Functional.LTP_Smack/spec.json | 7 + > engine/tests/Functional.LTP_Smack/test_mount.sh | 28 +++ > 4 files changed, 382 insertions(+) > create mode 100755 engine/tests/Functional.LTP_Smack/fuego_test.sh > create mode 100755 engine/tests/Functional.LTP_Smack/parser.py > create mode 100644 engine/tests/Functional.LTP_Smack/spec.json > create mode 100644 engine/tests/Functional.LTP_Smack/test_mount.sh >=20 > diff --git a/engine/tests/Functional.LTP_Smack/fuego_test.sh > b/engine/tests/Functional.LTP_Smack/fuego_test.sh > new file mode 100755 > index 0000000..2fc2fe5 > --- /dev/null > +++ b/engine/tests/Functional.LTP_Smack/fuego_test.sh > @@ -0,0 +1,77 @@ > +# Don't allow jobs to share build directories # the > +"test_successfully_built" flag for one spec function test_build { > + # check for LTP build directory > + LTP_BUILD_DIR=3D"${WORKSPACE}/$(echo $JOB_BUILD_DIR | sed > s/LTP_one_test/LTP/ | sed s/$TESTSPEC/default/)" This needs to be: LTP_BUILD_DIR=3D"${WORKSPACE}/$(echo $JOB_BUILD_DIR | sed s/LTP_Smack/LTP/ | sed s/$TESTSPEC/default/)" This construct is used to convert this test's build directory into the LTP build directory. But this test is not LTP_one_test, so the sed substitutio= n from that test won't work. I'm curious - did this work for you? It looks like you have the right subs= titution below, so maybe this would only affect the build. I think that in the wors= t case, this will never detect a pre-built LTP, and might not cause a build w= hen one was needed (or vice-versa - trigger a build even when one was not needed). > + echo "LTP_BUILD_DIR=3D${LTP_BUILD_DIR}" > + > + # if not already built, build LTP > + if [ ! -e ${LTP_BUILD_DIR}/fuego_test_successfully_built ] ; then > + echo "Building parent LTP test..." > + ftc run-test -b $NODE_NAME -t Functional.LTP -p pcb > + # NOTE: vars used in ftc run-test should not leak into this envi= ronment > + # that is, none of our test vars should have changed. > + fi > +} > + > +function test_deploy { > + # set LTP_BUILD_DIR (possibly again), in case test_build was skipped > + LTP_BUILD_DIR=3D"${WORKSPACE}/$(echo $JOB_BUILD_DIR | sed > s/LTP_Smack/LTP/ | sed s/$TESTSPEC/default/)" OK - this sets the correct LTP_BUILD_DIR =20 > + echo "LTP_BUILD_DIR=3D${LTP_BUILD_DIR}" > + > + local bdir=3D"$BOARD_TESTDIR/fuego.$TESTDIR" > + echo "bdir=3D${bdir}" > + > + # copy helper files, runltp, ltp-pan and the > + # test program to the board > + cmd "mkdir -p $bdir/bin $bdir/runtest $bdir/testcases/bin " > + put ${LTP_BUILD_DIR}/target_bin/IDcheck.sh $bdir/ > + put ${LTP_BUILD_DIR}/target_bin/ver_linux $bdir/ > + put ${LTP_BUILD_DIR}/target_bin/Version $bdir/ > + put ${LTP_BUILD_DIR}/target_bin/runltp $bdir/ > + put ${LTP_BUILD_DIR}/target_bin/bin/ltp-pan $bdir/bin/ > + > + put ${LTP_BUILD_DIR}/target_bin/runtest/smack $bdir/runtest > + > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_set_direct.sh > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_set_current.sh > $bdir/testcases/bin > + put > ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_set_socket_labels > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_set_onlycap.sh > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_set_cipso.sh > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_set_doi.sh > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_file_access.sh > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_notroot > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_set_ambient.sh > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_set_netlabel.sh > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_common.sh > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/smack_set_load.sh > $bdir/testcases/bin Using a wildcard here will reduce the lines needed above: put ${LTP_BUILD_DIR}/target_bin/testcase/bin/smack_* $bdir/testcase/bin > + > + # smack test cases need them > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/test.sh > $bdir/testcases/bin > + put ${LTP_BUILD_DIR}/target_bin/testcases/bin/tst_ansi_color.sh > $bdir/testcases/bin > + > + # test_mount.sh set the smack env > + put $TEST_HOME/test_mount.sh $bdir/ } > + > +function test_run { > + > + local bdir=3D"$BOARD_TESTDIR/fuego.$TESTDIR" > + local odir=3D"$BOARD_TESTDIR/fuego.$TESTDIR/result/default" > + echo "test_run__bdir:" $bdir > + > + report "cd $bdir; chmod +x test_mount.sh; ./test_mount.sh start" > + report "cd $bdir; mkdir -p $odir; ./runltp -f smack -l $odir/result.= log -o > $odir/output.log" > + report "cd $bdir; ./test_mount.sh end" > +} > + > +function test_fetch_results { > + echo "Fetching LTP Smack results" > + rm -rf result/ > + get $BOARD_TESTDIR/fuego.$TESTDIR/result $LOGDIR } > + > +function test_processing { > + return > +} > + > diff --git a/engine/tests/Functional.LTP_Smack/parser.py > b/engine/tests/Functional.LTP_Smack/parser.py Is this a direct copy of Functional.LTP/parser.py? If so, is there a way t= o=20 share that one (maybe a symlink)? If we do keep this parser.py separate, we don't need the posix parsing at all, so a lot of this code could be eliminated. > new file mode 100755 > index 0000000..2dc44a8 > --- /dev/null > +++ b/engine/tests/Functional.LTP_Smack/parser.py > @@ -0,0 +1,270 @@ > +#!/usr/bin/python > +# -*- coding: UTF-8 -*- > +import os, os.path, re, sys > +sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser') > +import common as plib > + > +SAVEDIR=3Dos.getcwd() > +LOGDIR=3Dos.environ["LOGDIR"] > + > +def abort(msg): > + print msg > + os.chdir(SAVEDIR) > + sys.exit(1) > + > +def split_output_per_testcase (test_category): > + ''' > + For each test category/group (e.g. syscalls) there is an output.= log > + file that contains the output log of each testcase. This functio= n > + splits output.log into per-testcase files > + ''' > + > + # open input > + try: > + output_all =3D open("%s/output.log" % test_category) > + except IOError: > + abort('"%s/result.log" cannot be opened.' % test_category) > + > + # prepare for outputs > + out_dir =3D test_category + "/outputs" > + try: > + os.mkdir(out_dir) > + except OSError: > + pass > + > + lines =3D output_all.readlines() > + output_all.close() > + for line in lines: > + m =3D re.compile("^<<>>").match(line) > + if m is not None: > + loop_end =3D 0 > + in_loop =3D 1 > + try: > + output_each =3D open(out_dir+"/tmp.log", "w") > + except IOError: > + abort('"%s/tmp.log" cannot be opened.' % out_dir) > + > + m =3D re.compile("^tag=3D([^ ]*)").match(line) > + if m is not None: > + test_case =3D m.group(1) > + > + m =3D re.compile("^<<>>").match(line) > + if m is not None: > + loop_end =3D 1 > + > + if in_loop: > + output_each.write("%s" % line) > + > + if in_loop & loop_end: > + output_each.close() > + os.rename(out_dir+"/tmp.log", out_dir+"/%s.log" % test_case) > + in_loop =3D 0 > + > +def read_output (test_category, test_case): > + ''' > + Reads one of the files splitted by split_output_per_testcase > + ''' > + case_log =3D "%s/outputs/%s.log" % (test_category, test_case) > + try: > + output_each =3D open(case_log) > + except IOError: > + abort('"%s"" cannot be opened.' % (case_log)) > + > + output =3D output_each.read() > + output_each.close() > + > + m =3D re.compile("<<>>\n(.*)\n<<>>", > re.M | re.S).search(output) > + if m is not None: > + result =3D m.group(1) > + else: > + result =3D "" > + > + return result > + > + > +# Check for results dir, and cd there > +try: > + os.chdir(LOGDIR+"/result") > +except: > + print "WARNING: no result directory (probably a build only test)." > + sys.exit(3) > + > +# there are three types of results - regular, posix and realtime # > +parse the regular results, first, into test_results > + > +# Loop that proceses each test folder > +tests =3D os.listdir('.') > +tests.sort() > +test_results =3D {} > +for test_category in tests: > + if not os.path.isdir(test_category): > + continue > + > + split_output_per_testcase(test_category) > + > + ## Check result.log > + try: > + f =3D open("%s/result.log" % test_category) > + except IOError: > + print '"%s/result.log" cannot be opened.' % test_category > + continue > + > + lines =3D f.readlines() > + f.close() > + regc =3D re.compile("^tag=3D([^ ]*) stime=3D([^ ]*) dur=3D([^ ]*) ex= it=3D([^ ]*) > stat=3D([^ ]*) core=3D([^ ]*) cu=3D([^ ]*) cs=3D([^ ]*)") > + for line in lines: > + m =3D regc.match(line) > + if m is not None: > + test_case =3D m.group(1) > + result =3D m.group(5) > + > + errtype =3D [] > + decision =3D 0 # 0: PASS, 1: FAIL > + > + if int(result) =3D=3D 0: > + errtype.append("PASS") > + > + if int(result) & 32 !=3D 0: > + errtype.append("CONF") > + decision =3D 0 > + > + if int(result) & 16 !=3D 0: > + errtype.append("INFO") > + decision =3D 1 > + > + if int(result) & 4 !=3D 0: > + errtype.append("WARN") > + decision =3D 1 > + > + if int(result) & 2 !=3D 0: > + errtype.append("BROK") > + decision =3D 1 > + > + if int(result) & 1 !=3D 0: > + errtype.append("FAIL") > + decision =3D 1 > + > + if int(result) & 0x100 !=3D 0: > + decision =3D 1 > + errtype.append("ERRNO") > + > + if int(result) & 0x200 !=3D 0: > + decision =3D 1 > + errtype.append("TERRNO") > + > + if int(result) & 0x300 !=3D 0: > + decision =3D 1 > + errtype.append("RERRNO") > + > + if decision =3D=3D 0: > + print "%s:%s passed" % (test_category, test_case) > + status =3D "PASS" > + else: > + print "%s:%s failed" % (test_category, test_case) > + status =3D "FAIL" > + > + # FIXTHIS: show errtype > + # FIXTHIS: add sub-test data > + test_results[test_category + '.' + test_case] =3D status > + > + # put test output to console log > + output =3D read_output(test_category, test_case) > + print output > + > +# now process posix results - from pts.log file posix_results =3D {} > +pts_logfile=3DLOGDIR+"/result/pts.log" > +lines =3D [] > +if os.path.exists(pts_logfile): > + try: > + f =3D open(pts_logfile) > + lines =3D f.readlines() > + f.close() > + except IOError: > + print '"%s" cannot be opened.' % pts_logfile > + > +regc =3D re.compile(r"^conformance/([^/]*)/([^/]*)/([^/]*): execution: > +(.*)") for line in lines: > + m =3D regc.match(line) > + if m: > + test_set =3D m.group(2) > + test_case =3D m.group(3) > + result =3D m.group(4) > + > + test_id =3D test_set+"."+test_case > + status =3D "ERROR" > + if result.startswith("PASS"): > + status =3D "PASS" > + elif result.startswith("FAIL"): > + status =3D "FAIL" > + elif result.startswith("UNTESTED"): > + status =3D "SKIP" > + posix_results[test_id] =3D status > + > +# hope no posix tests have the same test_ids as regular tests > +test_results.update(posix_results) > + > +if os.path.exists('rt.log'): > + rt_results =3D {} > + with open('rt.log') as f: > + rt_testcase_regex =3D "^--- Running testcase (.*) ---$" > + rt_results_regex =3D "^\s*Result:\s*(.*)$" > + for line in f: > + m =3D re.match(rt_testcase_regex, line.rstrip()) > + if m: > + test_case =3D m.group(1) > + m =3D re.match(rt_results_regex, line.rstrip()) > + if m: > + test_result =3D m.group(1) > + test_id =3D "rt." + test_case > + rt_results[test_id] =3D test_result > + test_results.update(rt_results) > + > +os.chdir(SAVEDIR) > +sys.exit(plib.process(test_results)) > + > +# Posix Test Suite processing > +#last_was_conformance =3D False > +#set_pts_format =3D False > +#fills =3D {'UNRESOLVED':brok_fill, 'FAILED':fail_fill, 'PASS':pass_fill= , > +'UNTESTED':conf_fill, 'UNSUPPORTED':info_fill} > + > +#def pts_set_style(ws): > + #for r in range(1, ws.get_highest_row()): > + #ws.cell(row=3Dr, column=3D1).style.fill =3D fills[str(ws.cell(r= ow=3Dr, > column=3D1).value)] > + ## adjust column widths > + #dims =3D{} > + #for row in ws.rows: > + #for cell in row: > + #if cell.value: > + #dims[cell.column] =3D max((dims.get(cell.column, 0), le= n(cell.value) > + 2)) > + #for col, value in dims.items(): > + #ws.column_dimensions[col].width =3D value > + > +#if os.path.exists('pts.log'): > + ## create one sheet per test group and fill the cells with the resul= ts > + #with open('pts.log') as f: > + #for line in f: > + #line =3D line.rstrip() > + #if not line: > + #continue > + #splitted =3D line.split(':') > + #if splitted[0] in ['AIO', 'MEM', 'MSG', 'SEM', 'SIG', 'THR'= , 'TMR', 'TPS']: > + #if set_pts_format: > + #pts_set_style(ws) > + #ws =3D book.create_sheet(title=3Dsplitted[0]) > + #ws.append(["Test", "Result", "Log"]) > + #last_was_conformance =3D False > + #set_pts_format =3D True > + #elif splitted[0].startswith('conformance'): > + #last_was_conformance =3D True > + #ws.append([os.path.basename(splitted[0]), splitted[2].l= strip()]) > + #else: > + #if last_was_conformance: > + #cell =3D ws.cell(row=3Dws.get_highest_row() - 1, co= lumn=3D2) > + #if cell.value: > + #cell.value =3D str(cell.value) + '\n' + line > + #else: > + #cell.value =3D line > + > + > + > diff --git a/engine/tests/Functional.LTP_Smack/spec.json > b/engine/tests/Functional.LTP_Smack/spec.json > new file mode 100644 > index 0000000..5d03076 > --- /dev/null > +++ b/engine/tests/Functional.LTP_Smack/spec.json > @@ -0,0 +1,7 @@ > +{ > + "testName": "Functional.LTP_Smack", > + "specs": { > + "default": { > + } > + } > +} > diff --git a/engine/tests/Functional.LTP_Smack/test_mount.sh > b/engine/tests/Functional.LTP_Smack/test_mount.sh > new file mode 100644 > index 0000000..9d78154 > --- /dev/null > +++ b/engine/tests/Functional.LTP_Smack/test_mount.sh > @@ -0,0 +1,28 @@ > +#!/bin/sh > + > +if [ "$1" =3D=3D "start" ]; then > + touch test_mount.log > + mount | grep -v /sys/fs/smackfs | grep /smack > /dev/null > + if [ $? -eq 0 ]; then > + exit 0 > + fi > + > + if [ ! -d /smack ]; then > + mkdir /smack > /dev/null > + echo "NEW_DIR" >> test_mount.log > + fi > + > + mount -t smackfs smackfs /smack > + echo "NEW_MOUNT" >> test_mount.log > +fi > + > + > +if [ "$1" =3D=3D "end" ]; then > + if grep "NEW_MOUNT" test_mount.log > /dev/null; then > + umount /smack > + fi > + > + if grep "NEW_DIR" test_mount.log > /dev/null; then > + rmdir /smack > + fi > +fi Do we need to remove test_mount.log, when 'end' is called? Or does the default test cleanup get rid of it? Just in case the user specifies no Target_PreCleanup=3Dfalse and Target_PostCleanup=3Dfalse, I think it would be good to explicitly remove it when we're done with it. As stated above, more comments on a separate thread. -- Tim