[XTF PATCH 0/3] Support getting output from either console or log file

[XTF PATCH 0/3] Support getting output from either console or log file

[Wei Liu]

Wei Liu (3):
  xtf-runner: introduce get_xen_version
  xtf-runner: options for different modes to get output
  xtf-runner: support two modes for getting output

 xtf-runner | 144 ++++++++++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 123 insertions(+), 21 deletions(-)

-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[XTF PATCH 1/3] xtf-runner: introduce get_xen_version

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xtf-runner | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/xtf-runner b/xtf-runner
index c063699..7b69c45 100755
--- a/xtf-runner
+++ b/xtf-runner
@@ -151,6 +151,19 @@ def __repr__(self):
         return "TestInfo(%s)" % (self.name, )
 
 
+def get_xen_version():
+    """Get the version string of Xen"""
+
+    for line in check_output(['xl', 'info']).splitlines():
+        if not line.startswith("xen_version"):
+            continue
+
+        version_str = line.split()[2:][0]
+        break
+
+    return version_str
+
+
 def parse_test_instance_string(arg):
     """Parse a test instance string.
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[XTF PATCH 2/3] xtf-runner: options for different modes to get output

[Wei Liu]

Only options and help text. Implementation will come later.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xtf-runner | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/xtf-runner b/xtf-runner
index 7b69c45..73a4e0d 100755
--- a/xtf-runner
+++ b/xtf-runner
@@ -532,6 +532,20 @@ def main():
                   "  all tests in the selection, printing a summary of their\n"
                   "  results at the end.\n"
                   "\n"
+                  "  To determine how runner should get output from Xen, use\n"
+                  "  --mode option. The default value is \"auto\", which\n"
+                  "  will choose the mode based on Xen version. Other\n"
+                  "  supported values are \"console\" and \"logfile\", which\n"
+                  "  respectively means to get log from xenconsole and\n"
+                  "  to get log from xencosonsoled output.\n"
+                  "\n"
+                  "  The \"logfile\" mode requires users to configure\n"
+                  "  xenconsoled to log guest console output. This mode\n"
+                  "  is useful for Xen version < 4.8. Note that runner will\n"
+                  "  append a custom string to guest log file before\n"
+                  "  running each test. Also see --logfile-dir\n"
+                  "  and --logfile-pattern options.\n"
+                  "\n"
                   "Selections:\n"
                   "  A selection is zero or more of any of the following\n"
                   "  parameters: Categories, Environments and Tests.\n"
@@ -609,6 +623,19 @@ def main():
                       dest = "host", help = "Restrict selection to applicable"
                       " tests for the current host",
                       )
+    parser.add_option("-m", "--mode", action = "store",
+                      dest = "mode", default = "auto", type = "string",
+                      help = "Instruct how runner gets its output (see below)")
+    parser.add_option("--logfile-dir", action = "store",
+                      dest = "logfile_dir", default = "/var/log/xen/console/",
+                      type = "string",
+                      help = 'Specify the directory to look for console logs, \
+defaults to "/var/log/xen/console/"')
+    parser.add_option("--logfile-pattern", action = "store",
+                      dest = "logfile_pattern", default = "guest-%s.log",
+                      type = "string",
+                      help = 'Specify the log file name pattern, \
+defaults to "guest-%s"')
 
     opts, args = parser.parse_args()
     opts.args = args
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[XTF PATCH 3/3] xtf-runner: support two modes for getting output

[Wei Liu]

We need two modes for getting output:

1. Use console directly with newer (>=4.8) Xen
2. Use log files for older Xen

This patch implements both. The default behaviour is to choose mode
automatically based on Xen version. User can also explicitly specify
which mode to use.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xtf-runner | 104 ++++++++++++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 83 insertions(+), 21 deletions(-)

diff --git a/xtf-runner b/xtf-runner
index 73a4e0d..2d4d6db 100755
--- a/xtf-runner
+++ b/xtf-runner
@@ -11,6 +11,8 @@
 
 from optparse import OptionParser
 from subprocess import Popen, PIPE, call as subproc_call, check_output
+from distutils.version import LooseVersion
+import time
 
 try:
     import json
@@ -438,30 +440,19 @@ def list_tests(opts):
     for sel in opts.selection:
         print sel
 
+def run_test_console(opts, test):
+    """ Run a specific test via xenconsole"""
 
-def run_test(test):
-    """ Run a specific test """
-
-    cmd = ['xl', 'create', '-p', test.cfg_path()]
-    print "Executing '%s'" % (" ".join(cmd), )
-    rc = subproc_call(cmd)
-    if rc:
-        raise RunnerError("Failed to create VM")
-
-    cmd = ['xl', 'console', test.vm_name()]
+    cmd = ['xl', 'create', '-Fc', test.cfg_path()]
     print "Executing '%s'" % (" ".join(cmd), )
-    console = Popen(cmd, stdout = PIPE)
+    guest = Popen(cmd, stdout = PIPE, stderr = PIPE)
 
-    cmd = ['xl', 'unpause', test.vm_name()]
-    print "Executing '%s'" % (" ".join(cmd), )
-    rc = subproc_call(cmd)
-    if rc:
-        raise RunnerError("Failed to unpause VM")
+    # stdout is console output, stderr is xl output
+    stdout, stderr = guest.communicate()
 
-    stdout, _ = console.communicate()
-
-    if console.returncode:
-        raise RunnerError("Failed to obtain VM console")
+    if guest.returncode:
+        print stderr
+        raise RunnerError("Failed to communicate with guest")
 
     lines = stdout.splitlines()
 
@@ -483,6 +474,64 @@ def run_test(test):
     return "ERROR"
 
 
+def run_test_logfile(opts, test):
+    """ Run a specific test via grepping log file"""
+
+    fn = opts.logfile_dir + (opts.logfile_pattern % test)
+    local_time = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime())
+
+    # Use time to generate unique stamps
+    start_stamp = "===== XTF TEST START %s =====" % local_time
+    end_stamp = "===== XTF TEST END %s =====" % local_time
+
+    print "Using %s" % fn
+
+    f = open(fn, "ab")
+    f.write(start_stamp + "\n")
+    f.close()
+
+    cmd = ['xl', 'create', '-F', test.cfg_path()]
+    print "Executing '%s'" % (" ".join(cmd), )
+    rc = subproc_call(cmd)
+    if rc:
+        raise RunnerError("Failed to run test")
+
+    f = open(fn, "ab")
+    f.write(end_stamp + "\n")
+    f.close()
+
+    f = open(fn, "rb")
+    output = f.readlines()
+    f.close()
+    lines = []
+    found = False
+    for line in output:
+        if end_stamp in line:
+            break
+        if start_stamp in line:
+            found = True
+            continue
+        if not found:
+            continue
+        lines.append(line)
+
+    print "".join(lines)
+
+    if len(lines) == 0:
+        raise RunnerError("Log file output empty")
+
+    test_result = lines[-1]
+    if not "Test result:" in test_result:
+        return "ERROR"
+
+    for res in all_results:
+
+        if res in test_result:
+            return res
+
+    return "ERROR"
+
+
 def run_tests(opts):
     """ Run tests """
 
@@ -490,12 +539,25 @@ def run_tests(opts):
     if not len(tests):
         raise RunnerError("No tests to run")
 
+    if opts.mode == "auto":
+        xen_version = LooseVersion(get_xen_version())
+        if xen_version < LooseVersion("4.8"):
+            run_test = run_test_logfile
+        else:
+            run_test = run_test_console
+    elif opts.mode == "console":
+        run_test = run_test_console
+    elif opts.mode == "logfile":
+        run_test = run_test_logfile
+    else:
+        raise RunnerError("Unrecognised mode")
+
     rc = all_results.index('SUCCESS')
     results = []
 
     for test in tests:
 
-        res = run_test(test)
+        res = run_test(opts, test)
         res_idx = all_results.index(res)
         if res_idx > rc:
             rc = res_idx
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output

[Wei Liu]

On Wed, Aug 10, 2016 at 04:07:30PM +0100, Wei Liu wrote:
[...]
>  
> +def run_test_logfile(opts, test):
> +    """ Run a specific test via grepping log file"""
> +
> +    fn = opts.logfile_dir + (opts.logfile_pattern % test)
> +    local_time = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime())
> +
> +    # Use time to generate unique stamps
> +    start_stamp = "===== XTF TEST START %s =====" % local_time
> +    end_stamp = "===== XTF TEST END %s =====" % local_time
> +
> +    print "Using %s" % fn
> +
> +    f = open(fn, "ab")
> +    f.write(start_stamp + "\n")
> +    f.close()
> +

I think it would make more sense for the micro VM itself to write
stamps?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output

[Wei Liu]

On Thu, Aug 11, 2016 at 09:33:57AM +0100, Wei Liu wrote:
> On Wed, Aug 10, 2016 at 04:07:30PM +0100, Wei Liu wrote:
> [...]
> >  
> > +def run_test_logfile(opts, test):
> > +    """ Run a specific test via grepping log file"""
> > +
> > +    fn = opts.logfile_dir + (opts.logfile_pattern % test)
> > +    local_time = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime())
> > +
> > +    # Use time to generate unique stamps
> > +    start_stamp = "===== XTF TEST START %s =====" % local_time
> > +    end_stamp = "===== XTF TEST END %s =====" % local_time
> > +
> > +    print "Using %s" % fn
> > +
> > +    f = open(fn, "ab")
> > +    f.write(start_stamp + "\n")
> > +    f.close()
> > +
> 
> I think it would make more sense for the micro VM itself to write
> stamps?

I want to pass a stamp generated by the runner to micro VMs, otherwise
runner wouldn't be able to tell which stamps are the right one.

For PV guests it works because there is start_info->cmd_line.
Unfortunately I can't seem to find a place for putting in a command line
for hvm guest in the ABI. Newer version of Xen will have a boot ABI that
supports command line. This mode is mainly for old versions of Xen so I
don't see how it is possibly at this stage to uniformly support both old
and new versions of Xen.

Maybe we need to live with running adding the stamp? Let me know if I
miss anything.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output

[Andrew Cooper]

On 11/08/16 10:44, Wei Liu wrote:
> On Thu, Aug 11, 2016 at 09:33:57AM +0100, Wei Liu wrote:
>> On Wed, Aug 10, 2016 at 04:07:30PM +0100, Wei Liu wrote:
>> [...]
>>>  
>>> +def run_test_logfile(opts, test):
>>> +    """ Run a specific test via grepping log file"""
>>> +
>>> +    fn = opts.logfile_dir + (opts.logfile_pattern % test)
>>> +    local_time = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime())
>>> +
>>> +    # Use time to generate unique stamps
>>> +    start_stamp = "===== XTF TEST START %s =====" % local_time
>>> +    end_stamp = "===== XTF TEST END %s =====" % local_time
>>> +
>>> +    print "Using %s" % fn
>>> +
>>> +    f = open(fn, "ab")
>>> +    f.write(start_stamp + "\n")
>>> +    f.close()
>>> +
>> I think it would make more sense for the micro VM itself to write
>> stamps?
> I want to pass a stamp generated by the runner to micro VMs, otherwise
> runner wouldn't be able to tell which stamps are the right one.
>
> For PV guests it works because there is start_info->cmd_line.
> Unfortunately I can't seem to find a place for putting in a command line
> for hvm guest in the ABI. Newer version of Xen will have a boot ABI that
> supports command line. This mode is mainly for old versions of Xen so I
> don't see how it is possibly at this stage to uniformly support both old
> and new versions of Xen.
>
> Maybe we need to live with running adding the stamp? Let me know if I
> miss anything.

I haven't managed to come up with a reasonable way to get a command line
into an HVM guest yet.  The best I managed was a xenstore key, but that
gets in the way of doing xenstore ring testing in XTF, and still
requires going behind the back of the toolstack.

Can't you just open the log file as read, seek to the end, run the test
and read again from the same FD?  It would be rather more simple than
marking the logs.

Sadly, whatever method we use here is going to have to be clever enough
to cope with the log files being rotated, and I can't think of a clever
way of doing that ATM.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output

[Wei Liu]

On Thu, Aug 11, 2016 at 10:56:23AM +0100, Andrew Cooper wrote:
> On 11/08/16 10:44, Wei Liu wrote:
> > On Thu, Aug 11, 2016 at 09:33:57AM +0100, Wei Liu wrote:
> >> On Wed, Aug 10, 2016 at 04:07:30PM +0100, Wei Liu wrote:
> >> [...]
> >>>  
> >>> +def run_test_logfile(opts, test):
> >>> +    """ Run a specific test via grepping log file"""
> >>> +
> >>> +    fn = opts.logfile_dir + (opts.logfile_pattern % test)
> >>> +    local_time = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime())
> >>> +
> >>> +    # Use time to generate unique stamps
> >>> +    start_stamp = "===== XTF TEST START %s =====" % local_time
> >>> +    end_stamp = "===== XTF TEST END %s =====" % local_time
> >>> +
> >>> +    print "Using %s" % fn
> >>> +
> >>> +    f = open(fn, "ab")
> >>> +    f.write(start_stamp + "\n")
> >>> +    f.close()
> >>> +
> >> I think it would make more sense for the micro VM itself to write
> >> stamps?
> > I want to pass a stamp generated by the runner to micro VMs, otherwise
> > runner wouldn't be able to tell which stamps are the right one.
> >
> > For PV guests it works because there is start_info->cmd_line.
> > Unfortunately I can't seem to find a place for putting in a command line
> > for hvm guest in the ABI. Newer version of Xen will have a boot ABI that
> > supports command line. This mode is mainly for old versions of Xen so I
> > don't see how it is possibly at this stage to uniformly support both old
> > and new versions of Xen.
> >
> > Maybe we need to live with running adding the stamp? Let me know if I
> > miss anything.
> 
> I haven't managed to come up with a reasonable way to get a command line
> into an HVM guest yet.  The best I managed was a xenstore key, but that
> gets in the way of doing xenstore ring testing in XTF, and still
> requires going behind the back of the toolstack.
> 
> Can't you just open the log file as read, seek to the end, run the test
> and read again from the same FD?  It would be rather more simple than
> marking the logs.
> 

The reason I want to put stamps (I think Ian's, too) is to make sure we
can still get the right bits out if same test case is run consequently,
multiple times. The only race-free way of doing it is to have the micro
vm itself prints out stamps. Using the runner to print stamps won't
solve it -- I only realised that after posting this series.

Given the current restrictions, I can live with the method you suggest,
too.

> Sadly, whatever method we use here is going to have to be clever enough
> to cope with the log files being rotated, and I can't think of a clever
> way of doing that ATM.
> 

Require logrorate to be disabled. :-)

Wei.

> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 1/3] xtf-runner: introduce get_xen_version

[Andrew Cooper]

On 10/08/16 16:07, Wei Liu wrote:
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  xtf-runner | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> diff --git a/xtf-runner b/xtf-runner
> index c063699..7b69c45 100755
> --- a/xtf-runner
> +++ b/xtf-runner
> @@ -151,6 +151,19 @@ def __repr__(self):
>          return "TestInfo(%s)" % (self.name, )
>  
>  
> +def get_xen_version():
> +    """Get the version string of Xen"""
> +
> +    for line in check_output(['xl', 'info']).splitlines():
> +        if not line.startswith("xen_version"):
> +            continue
> +
> +        version_str = line.split()[2:][0]
> +        break
> +
> +    return version_str

This will hit a name error if xen_version isn't found in the output.

A better option would be to "return line.split()[2:][0]" directly and
raise a RunnerError() at this point.

However, xen_version was introduced by me a while ago, but not so very
long ago, and won't work on older versions of Xen.

I think at this point, it would just be easier to use the python libxc
bindings, so

from xen.lowlevel.xc import xc
libxc = xc()

info = libxc.xeninfo()
return "%s.%s" % (info["xen_major"], info["xen_minor"])

Make sure the import statement is inside the function call to avoid
breaking the usecase of `xtf-runner --list` offhost.

~Andrew

> +
> +
>  def parse_test_instance_string(arg):
>      """Parse a test instance string.
>  


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 1/3] xtf-runner: introduce get_xen_version

[Wei Liu]

On Thu, Aug 11, 2016 at 11:10:41AM +0100, Andrew Cooper wrote:
> On 10/08/16 16:07, Wei Liu wrote:
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> >  xtf-runner | 13 +++++++++++++
> >  1 file changed, 13 insertions(+)
> >
> > diff --git a/xtf-runner b/xtf-runner
> > index c063699..7b69c45 100755
> > --- a/xtf-runner
> > +++ b/xtf-runner
> > @@ -151,6 +151,19 @@ def __repr__(self):
> >          return "TestInfo(%s)" % (self.name, )
> >  
> >  
> > +def get_xen_version():
> > +    """Get the version string of Xen"""
> > +
> > +    for line in check_output(['xl', 'info']).splitlines():
> > +        if not line.startswith("xen_version"):
> > +            continue
> > +
> > +        version_str = line.split()[2:][0]
> > +        break
> > +
> > +    return version_str
> 
> This will hit a name error if xen_version isn't found in the output.
> 
> A better option would be to "return line.split()[2:][0]" directly and
> raise a RunnerError() at this point.
> 
> However, xen_version was introduced by me a while ago, but not so very
> long ago, and won't work on older versions of Xen.
> 

Oh, didn't notice that. I thought it was available to all versions of
Xen.

> I think at this point, it would just be easier to use the python libxc
> bindings, so
> 
> from xen.lowlevel.xc import xc
> libxc = xc()
> 
> info = libxc.xeninfo()
> return "%s.%s" % (info["xen_major"], info["xen_minor"])
> 
> Make sure the import statement is inside the function call to avoid
> breaking the usecase of `xtf-runner --list` offhost.
> 

NP.

Wei.

> ~Andrew
> 
> > +
> > +
> >  def parse_test_instance_string(arg):
> >      """Parse a test instance string.
> >  
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 1/3] xtf-runner: introduce get_xen_version

[Andrew Cooper]

On 11/08/16 11:12, Wei Liu wrote:
> On Thu, Aug 11, 2016 at 11:10:41AM +0100, Andrew Cooper wrote:
>> On 10/08/16 16:07, Wei Liu wrote:
>>> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
>>> ---
>>>  xtf-runner | 13 +++++++++++++
>>>  1 file changed, 13 insertions(+)
>>>
>>> diff --git a/xtf-runner b/xtf-runner
>>> index c063699..7b69c45 100755
>>> --- a/xtf-runner
>>> +++ b/xtf-runner
>>> @@ -151,6 +151,19 @@ def __repr__(self):
>>>          return "TestInfo(%s)" % (self.name, )
>>>  
>>>  
>>> +def get_xen_version():
>>> +    """Get the version string of Xen"""
>>> +
>>> +    for line in check_output(['xl', 'info']).splitlines():
>>> +        if not line.startswith("xen_version"):
>>> +            continue
>>> +
>>> +        version_str = line.split()[2:][0]
>>> +        break
>>> +
>>> +    return version_str
>> This will hit a name error if xen_version isn't found in the output.
>>
>> A better option would be to "return line.split()[2:][0]" directly and
>> raise a RunnerError() at this point.
>>
>> However, xen_version was introduced by me a while ago, but not so very
>> long ago, and won't work on older versions of Xen.
>>
> Oh, didn't notice that. I thought it was available to all versions of
> Xen.
>
>> I think at this point, it would just be easier to use the python libxc
>> bindings, so
>>
>> from xen.lowlevel.xc import xc
>> libxc = xc()
>>
>> info = libxc.xeninfo()
>> return "%s.%s" % (info["xen_major"], info["xen_minor"])

Erm - these should be %d not %s.

Sorry for the misinformation.

~Andrew

>>
>> Make sure the import statement is inside the function call to avoid
>> breaking the usecase of `xtf-runner --list` offhost.
>>
> NP.
>
> Wei.
>
>> ~Andrew
>>
>>> +
>>> +
>>>  def parse_test_instance_string(arg):
>>>      """Parse a test instance string.
>>>  


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output

[Ian Jackson]

Wei Liu writes ("[XTF PATCH 3/3] xtf-runner: support two modes for getting output"):
> We need two modes for getting output:
...
> +    # Use time to generate unique stamps
> +    start_stamp = "===== XTF TEST START %s =====" % local_time
> +    end_stamp = "===== XTF TEST END %s =====" % local_time

This will go wrong if someone runs the same test very rapidly in a
loop.  It needs to be augmented, at least.

Ideally with the domid, but AFAICT that's not available here.  If you
can't think of anything else, use, in addition to the timestamp, the
pid of the xtf-runner process, plus a counter.  (The counter is
necessary in case the same xtf-runner process is used to run the same
test multiple times.)

> +    f = open(fn, "rb")
> +    output = f.readlines()
> +    f.close()
> +    lines = []
> +    found = False
> +    for line in output:
> +        if end_stamp in line:
> +            break
> +        if start_stamp in line:
> +            found = True
> +            continue
> +        if not found:
> +            continue
> +        lines.append(line)

This is accidentally quadratic in the number of test executions.
Do we care ?

If we do care, we should read the file backwards.  AFAICT "tac" can do
this in a manner that's not quadratic in the size of the logfile:

mariner:iwj> time tac v | head | sha256sum 
55748d8a2243c7a8978eccf24bb4f603091ecf4be836294e7009695426971485  -

real    0m0.019s
user    0m0.000s
sys     0m0.000s
mariner:iwj> time cat v | tail | tac | sha256sum 
55748d8a2243c7a8978eccf24bb4f603091ecf4be836294e7009695426971485  -

real    0m11.286s
user    0m0.296s
sys     0m1.096s
mariner:iwj> ll --hu v
-rw-rw-r-- 1 iwj iwj 638M Aug 11 11:42 v
mariner:iwj>

Or you could write your own implementation.  The algorithm is to read
increasingly large chunks of the tail of the file, into memory, until
you find all of the part you're looking for.

HTH.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output

[Wei Liu]

On Thu, Aug 11, 2016 at 11:49:29AM +0100, Ian Jackson wrote:
> Wei Liu writes ("[XTF PATCH 3/3] xtf-runner: support two modes for getting output"):
> > We need two modes for getting output:
> ...
> > +    # Use time to generate unique stamps
> > +    start_stamp = "===== XTF TEST START %s =====" % local_time
> > +    end_stamp = "===== XTF TEST END %s =====" % local_time
> 
> This will go wrong if someone runs the same test very rapidly in a
> loop.  It needs to be augmented, at least.
> 
> Ideally with the domid, but AFAICT that's not available here.  If you
> can't think of anything else, use, in addition to the timestamp, the
> pid of the xtf-runner process, plus a counter.  (The counter is
> necessary in case the same xtf-runner process is used to run the same
> test multiple times.)
> 
> > +    f = open(fn, "rb")
> > +    output = f.readlines()
> > +    f.close()
> > +    lines = []
> > +    found = False
> > +    for line in output:
> > +        if end_stamp in line:
> > +            break
> > +        if start_stamp in line:
> > +            found = True
> > +            continue
> > +        if not found:
> > +            continue
> > +        lines.append(line)
> 
> This is accidentally quadratic in the number of test executions.
> Do we care ?
> 

I don't think so. That's why I used this simple algorithm. I don't
expect the log files to even exceed a few MBs. We can optimise later if
there is a test that would generate a huge amount of output.

Anyway, I think the stamps and the algorithm issues will become moot if
we seek to the end of the file and read it after the test is finished as
Andrew suggested. What do you think of that?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output

[Ian Jackson]

Andrew Cooper writes ("Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output"):
> Can't you just open the log file as read, seek to the end, run the test
> and read again from the same FD?  It would be rather more simple than
> marking the logs.

This is a good suggestion, I think.  Much better than mine in my other
email.

> Sadly, whatever method we use here is going to have to be clever enough
> to cope with the log files being rotated, and I can't think of a clever
> way of doing that ATM.

In an automatic test system the logfiles won't get big enough to be
rotated.  In a manual setup this is just a "random hazard" which you
seem to be prepared to accept...

But: it is easy for the runner to see if the logfile was rotated.  It
can open the logfile a second time and check if it has the same inum.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output

[Wei Liu]

On Thu, Aug 11, 2016 at 12:08:03PM +0100, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output"):
> > Can't you just open the log file as read, seek to the end, run the test
> > and read again from the same FD?  It would be rather more simple than
> > marking the logs.
> 
> This is a good suggestion, I think.  Much better than mine in my other
> email.
> 
> > Sadly, whatever method we use here is going to have to be clever enough
> > to cope with the log files being rotated, and I can't think of a clever
> > way of doing that ATM.
> 
> In an automatic test system the logfiles won't get big enough to be
> rotated.  In a manual setup this is just a "random hazard" which you
> seem to be prepared to accept...
> 

That's what I thought as well.

> But: it is easy for the runner to see if the logfile was rotated.  It
> can open the logfile a second time and check if it has the same inum.
> 

For now let's keep things simple. Doesn't seem to buy us much even if we
implement this.

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output

[Andrew Cooper]

On 11/08/16 12:19, Wei Liu wrote:
> On Thu, Aug 11, 2016 at 12:08:03PM +0100, Ian Jackson wrote:
>> Andrew Cooper writes ("Re: [XTF PATCH 3/3] xtf-runner: support two modes for getting output"):
>>> Can't you just open the log file as read, seek to the end, run the test
>>> and read again from the same FD?  It would be rather more simple than
>>> marking the logs.
>> This is a good suggestion, I think.  Much better than mine in my other
>> email.
>>
>>> Sadly, whatever method we use here is going to have to be clever enough
>>> to cope with the log files being rotated, and I can't think of a clever
>>> way of doing that ATM.
>> In an automatic test system the logfiles won't get big enough to be
>> rotated.  In a manual setup this is just a "random hazard" which you
>> seem to be prepared to accept...
>>
> That's what I thought as well.
>
>> But: it is easy for the runner to see if the logfile was rotated.  It
>> can open the logfile a second time and check if it has the same inum.
>>
> For now let's keep things simple. Doesn't seem to buy us much even if we
> implement this.

I expect anyone developing tests by hand to be using upstream, and
therefore the non-logfile mode.

It is worth a warning in the code about logfile rotation, but lets not
waste time now for a feature people will hopefully never need.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel