[CRIU] [PATCH] zdtm.py: add support of TAP reports

[Pavel Emelyanov]

On 12/29/2015 03:32 PM, Sergey Bronnikov wrote:
> Usually we run CRIU tests automatically using Jenkins and it reports status as
> PASS/FAIL for overall testsuite on used environment. You should dig in log files
> to figure out how many tests were failed and skipped.
> Jenkins has a plugin with TAP support, it allows to show extended status for
> testrun with passed/failed/skip tests on the graph without digging in log files.
> 
> TAP is a simple protocol for reporting test results (https://testanything.org/)
> Jenkins TAP Plugin: https://wiki.jenkins-ci.org/display/JENKINS/TAP+Plugin
> 
> Signed-off-by: Sergey Bronnikov <sergeyb at openvz.org>
> ---
>  Makefile     |  1 +
>  test/zdtm.py | 22 ++++++++++++++++++++++
>  2 files changed, 23 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 8a0a230..948f1cd 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -287,6 +287,7 @@ clean: clean-built
>  	$(Q) $(RM) ./*.pyc
>  	$(Q) $(RM) -r build
>  	$(Q) $(RM) -r usr
> +	$(Q) $(RM) ./test/*.tap
>  
>  distclean: clean
>  	$(E) "  DISTCLEAN"
> diff --git a/test/zdtm.py b/test/zdtm.py
> index eb59290..e0df4d6 100755
> --- a/test/zdtm.py
> +++ b/test/zdtm.py
> @@ -18,9 +18,13 @@ import imp
>  import socket
>  import fcntl
>  import errno
> +import datetime
>  
>  os.chdir(os.path.dirname(os.path.abspath(__file__)))
>  
> +now = datetime.datetime.now()
> +report = "testreport-" + now.strftime("%Y-%m-%d-%H-%M-%S") + ".tap"

Can we put tap report into --report dir?

> +
>  prev_line = None
>  def traceit(f, e, a):
>  	if e == "line":
> @@ -952,6 +956,7 @@ class launcher:
>  		self.__max = int(opts['parallel'] or 1)
>  		self.__subs = {}
>  		self.__fail = False
> +		init_tap(nr_tests)
>  
>  	def __show_progress(self):
>  		perc = self.__nr * 16 / self.__total
> @@ -960,6 +965,7 @@ class launcher:
>  	def skip(self, name, reason):
>  		print "Skipping %s (%s)" % (name, reason)
>  		self.__nr += 1
> +		write_tap("not ok " + str(self.__nr) + " - " + str(name.split('/')[-1:][0]) + " # SKIP " + reason)
>  
>  	def run_test(self, name, desc, flavor):
>  
> @@ -989,6 +995,11 @@ class launcher:
>  				stdout = log, stderr = subprocess.STDOUT)
>  		self.__subs[sub.pid] = { 'sub': sub, 'log': logf }
>  
> +		if sub:
> +			write_tap("ok " + str(self.__nr) + " - " + str(name.split('/')[-1:][0]))
> +		else:
> +			write_tap("not ok " + str(self.__nr) + " - " + str(name.split('/')[-1:][0]))
> +
>  		if test_flag(desc, 'excl'):
>  			self.wait()

The launcher would abort tests execution on the very first failure and report
will contain incomplete data.

>  
> @@ -1074,6 +1085,17 @@ def print_fname(fname, typ):
>  def print_sep(title, sep = "=", width = 80):
>  	print (" " + title + " ").center(width, sep)
>  
> +def init_tap(total):
> +	f = open(report, 'a')
> +	print >> f, "TAP version 13"
> +	print >> f, "1.." + str(total)
> +	f.close()
> +
> +def write_tap(tcase):
> +	f = open(report, 'a')
> +	print >> f, tcase
> +	f.close()

Why not open report file once in launcher and feed data into it eventually instead
of constant re-opening one?

> +
>  def grep_errors(fname):
>  	first = True
>  	for l in open(fname):
> 

-- Pavel