[CRIU] [PATCH][RFC] zdtm: Move towards the new generation of criu testing

Andrew Vagin avagin at odin.com
Mon Oct 5 01:56:30 PDT 2015


On Thu, Oct 01, 2015 at 03:11:32PM +0300, Pavel Emelyanov wrote:
> Hi.
> 
> After recent set with fault-injection it seemed to me that the
> existing 1.2KLOC shell script would be quite hard to extend and
> maintain in the future. I propose to bring our test system onto
> the next level and write it on a more suitable language :)
> 
> So, things to get fixed in the first place
> 
> 1. Introduce per-test extendable description. Right now this
>    description is scatered over test itself, .opts file, scripts
>    and the zdtm.sh itself (list forming and random hacks).
> 
>    In new approach all the tests' metadata is in the zdtm.list
>    file in yaml format.
> 
> 2. Introduce the environments in which test can be run. Now we
>    have 3 -- on host, in namespaces and in user-namespaces --
>    then we'll have the 4th -- with faults injected.
> 
>    In this approach there's a thing call 'flavor' that creates
>    and environment for a test to be run in.
> 
> So here's the first version -- the minimal wannabe replacement
> for zdtm.sh written in python. It can
> 
> 1. List tests :)
> 
> 2. Run tests in 3 flavors (\union supported by the particular
>    test) -- host, namespace, user-namespace. Default steps are
>    start, dump, restore, stop and check.
> 
> 3. Run non-standard paths, in particular
> 
>    a. Don't CR at all checking that the test itself is OK
> 
>    b. Don't restore after dump and check test survived
> 
>    c. Do several pre-dumps before dump
> 
>    d. Run the CR sequence again on restored test to check that
>       restored process is in dumpable state
> 
> 4. Check opened descriptors and maps before to coincide
>    with those after C/R (has issue, will fix)
> 
> Plans for the nearest changes
> 
> 1. Make it functional as zdtm.sh is. This includes
> 
>    a. Add page-server
> 
>    b. Exclude tests from run list
> 
>    c. Ability to only start the test
> 
>    d. Run tests in parallel
> 
> 2. Ability to add more than just zdtm/ tests into run. Right now
>    all non-zdtm tests are scatered over the test/ directory and
>    each has its own script to launch :)
> 
>    In the future I plan to add other tests in the zdtm.list and
>    rename one (and zdtm.py too).
> 
> 3. Ability to check not only CLI, but RPC as well
> 
> 4. Add fault-injection facility

Good job!

Thank you for this work. It looks good.

> 
> Signed-off-by: Pavel Emelyanov <xemul at parallels.com>
> ---
>  test/zdtm.list | 328 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  test/zdtm.py   | 348 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 676 insertions(+)
>  create mode 100644 test/zdtm.list
>  create mode 100755 test/zdtm.py
> 
> diff --git a/test/zdtm.list b/test/zdtm.list
> new file mode 100644
> index 0000000..d1cd184
> --- /dev/null
> +++ b/test/zdtm.list
> @@ -0,0 +1,328 @@
> +static/pipe00:
> +static/pipe01:
> +static/pipe02:
> +static/busyloop00:
> +static/cwd00:
> +static/cwd01:
> +static/cwd02:
> +static/env00:
> +static/maps00:
> +static/maps01:
> +  flags: suid
> +  flavor: h ns
> +static/maps02:
> +static/maps04:
> +static/maps05:
> +static/mlock_setuid:
> +  flags: suid
> +  flavor: h ns
> +static/maps_file_prot:
> +static/mprotect00:
> +static/mtime_mmap:
> +static/sleeping00:
> +static/write_read00:
> +static/write_read01:
> +static/write_read02:
> +static/write_read10:
> +static/wait00:
> +static/vdso00:
> +static/sched_prio00:
> +  flags: suid
> +  flavor: h ns
> +static/sched_policy00:
> +  flags: suid
> +  flavor: h ns
> +static/file_shared:
> +static/file_append:
> +static/timers:
> +static/posix_timers:
> +static/futex:
> +static/futex-rl:
> +static/xids00:
> +static/groups:
> +  flags: suid
> +static/pthread00:
> +static/pthread01:
> +static/umask00:
> +streaming/pipe_loop00:
> +streaming/pipe_shared00:
> +transition/file_read:
> +static/sockets00:
> +  flags: suid
> +static/sockets01:
> +static/sockets02:
> +static/sock_opts00:
> +  flags: suid
> +static/sock_opts01:
> +  flags: suid
> +static/sockets_spair:
> +static/sockets_dgram:
> +static/socket_dgram_data:
> +static/socket_queues:
> +static/deleted_unix_sock:
> +static/sk-unix-unconn:
> +static/sk-unix-rel:
> +static/pid00:
> +  flags: suid
> +static/pstree:
> +static/caps00:
> +  flags: suid
> +static/cmdlinenv00:
> +  flags: suid
> +static/socket_listen:
> +static/socket_listen6:
> +static/packet_sock:
> +  flags: suid
> +static/packet_sock_mmap:
> +  flags: suid
> +static/socket_udp:
> +static/sock_filter:
> +static/socket6_udp:
> +static/socket_udplite:
> +static/selfexe00:
> +static/link10:
> +static/unlink_fstat00:
> +static/unlink_fstat01:
> +static/unlink_fstat02:
> +static/unlink_fstat03:
> +  opts: --link-remap
> +static/unlink_mmap00:
> +static/unlink_mmap01:
> +static/unlink_mmap02:
> +static/rmdir_open:
> +static/eventfs00:
> +static/signalfd00:
> +static/inotify00:
> +  opts: --link-remap
> +static/inotify_irmap:
> +  flags: suid
> +static/fanotify00:
> +  flags: suid
> +  flavor: h ns
> +static/unbound_sock:
> +static/fifo-rowo-pair:
> +static/fifo-ghost:
> +static/fifo:
> +static/fifo_wronly:
> +static/fifo_ro:
> +static/unlink_fifo:
> +static/unlink_fifo_wronly:
> +static/zombie00:
> +static/rlimits00:
> +transition/fork:
> +transition/fork2:
> +transition/thread-bomb:
> +static/pty00:
> +static/pty01:
> +static/pty04:
> +static/tty02:
> +static/tty03:
> +static/console:
> +  flags: suid
> +  flavor: h ns
> +static/vt:
> +  flags: suid
> +  flavor: h ns
> +static/child_opened_proc:
> +static/cow01:
> +  flags: suid
> +  flavor: h ns
> +static/pdeath_sig:
> +static/fdt_shared:
> +static/file_locks00:
> +  flags: excl
> +  opts: --file-locks
> +static/file_locks01:
> +  flags: excl
> +  opts: --file-locks
> +static/file_locks02:
> +  flags: excl
> +  opts: --file-locks
> +static/file_locks03:
> +  flags: excl
> +  opts: --file-locks
> +static/file_locks04:
> +  flags: excl
> +  opts: --file-locks
> +static/file_locks05:
> +  flags: excl
> +  opts: --file-locks
> +static/sigpending:
> +static/sigaltstack:
> +static/sk-netlink:
> +  flags: suid
> +static/proc-self:
> +static/grow_map:
> +static/grow_map02:
> +static/grow_map03:
> +static/stopped:
> +static/chroot:
> +  flags: suid
> +static/chroot-file:
> +  flags: suid
> +static/rtc:
> +  flags: nouns suid crlib
> +transition/maps007:
> +  flags: suid
> +static/dumpable01:
> +static/dumpable02:
> +  flavor: h ns
> +static/deleted_dev:
> +  flags: suid
> +  flavor: h ns
> +static/fpu00:
> +  arch: x86_64
> +static/fpu01:
> +  arch: x86_64
> +static/mmx00:
> +  arch: x86_64
> +static/sse00:
> +  arch: x86_64
> +static/sse20:
> +  arch: x86_64
> +static/vdso01:
> +  arch: x86_64
> +static/vsx:
> +  arch: ppc64le
> +static/file_fown:
> +  flavor: h
> +static/socket-ext:
> +  flavor: h
> +  opts: --ext-unix-sk
> +static/socket-tcp:
> +  flavor: h
> +  opts: --tcp-established
> +static/socket-tcp6:
> +  flavor: h
> +  opts: --tcp-established
> +streaming/socket-tcp:
> +  flavor: h
> +  opts: --tcp-established
> +streaming/socket-tcp6:
> +  flavor: h
> +  opts: --tcp-established
> +static/socket-tcpbuf:
> +  flavor: h
> +  opts: --tcp-established
> +static/socket-tcpbuf-local:
> +  flavor: h
> +  opts: --tcp-established
> +static/socket-tcpbuf6:
> +  flavor: h
> +  opts: --tcp-established
> +static/pty03:
> +  flavor: h
> +static/mountpoints:
> +  flags: suid
> +  flavor: h
> +static/utsname:
> +  flavor: h
> +static/ipc_namespace:
> +  flavor: h
> +static/shm:
> +  flavor: h
> +static/msgque:
> +  flavor: h
> +static/sem:
> +  flavor: h
> +transition/ipc:
> +  flavor: h
> +static/netns-nf:
> +  flavor: h
> +static/netns:
> +  flavor: h
> +static/cgroup00:
> +  flags: suid
> +  flavor: h
> +  opts: --manage-cgroups
> +static/cgroup01:
> +  flags: suid
> +  flavor: h
> +  opts: --manage-cgroups
> +static/cgroup02:
> +  flags: suid
> +  flavor: h
> +  opts: --manage-cgroups --cgroup-root /newroot --cgroup-root name=zdtmtst:/zdtmtstroot
> +static/remap_dead_pid:
> +  flavor: h
> +static/poll:
> +  flavor: h
> +static/apparmor:
> +  flags: suid
> +static/different_creds:
> +  flags: suid crfail
> +  flavor: h
> +static/aio00:
> +  feature: aio
> +static/timerfd:
> +  feature: timerfd
> +static/session00:
> +  flavor: ns uns
> +static/session01:
> +  flavor: ns uns
> +static/tempfs:
> +  flags: suid
> +  flavor: ns uns
> +static/tempfs_ro:
> +  flags: suid
> +  flavor: ns
> +static/mnt_ro_bind:
> +  flags: suid
> +  flavor: ns uns
> +static/mount_paths:
> +  flags: suid
> +  flavor: ns uns
> +static/bind-mount:
> +  flags: suid
> +  flavor: ns uns
> +static/netns-dev:
> +  flags: suid
> +  flavor: ns uns
> +static/mnt_ext_auto:
> +  flavor: ns uns
> +  feature: mntid
> +  opts: --ext-mount-map auto --enable-external-sharing
> +static/mnt_ext_master:
> +  flavor: ns uns
> +  feature: mntid
> +  opts: --ext-mount-map auto --enable-external-masters
> +static/mntns_open:
> +  flags: suid
> +  flavor: ns uns
> +  feature: mntid
> +static/mntns_link_remap:
> +  flags: suid
> +  flavor: ns
> +  feature: mntid
> +  opts: --link-remap
> +static/mntns_link_ghost:
> +  flags: suid
> +  flavor: ns
> +  feature: mntid
> +static/mntns_shared_bind:
> +  flags: suid
> +  flavor: ns uns
> +  feature: mntid
> +static/mntns_shared_bind02:
> +  flags: suid
> +  flavor: ns uns
> +  feature: mntid
> +static/mntns_root_bind:
> +  flags: suid
> +  flavor: ns uns
> +  feature: mntid
> +static/mntns_deleted:
> +  flags: suid
> +  flavor: ns uns
> +  feature: mntid
> +static/tun:
> +  flags: suid
> +  flavor: ns uns
> +  feature: tun
> +static/seccomp_strict:
> +  flags: suid
> +  flavor: h
> +  feature: seccomp_suspend
> +static/clean_mntns:
> +  flags: suid
> +  flavor: ns
> diff --git a/test/zdtm.py b/test/zdtm.py
> new file mode 100755
> index 0000000..53be6f2
> --- /dev/null
> +++ b/test/zdtm.py
> @@ -0,0 +1,348 @@
> +#!/bin/env python
> +import argparse
> +import yaml
> +import os
> +import subprocess
> +import time
> +import tempfile
> +import shutil
> +import re
> +import stat
> +import signal
> +import atexit
> +
> +default_test={ }
> +zdtm_root = None
> +arch = os.uname()[4]
> +
> +def clean_zdtm_root():
> +	global zdtm_root
> +	if zdtm_root:
> +		os.rmdir(zdtm_root)
> +
> +def make_zdtm_root():
> +	global zdtm_root
> +	if not zdtm_root:
> +		zdtm_root = tempfile.mkdtemp("", "criu-root-", "/tmp")
> +		atexit.register(clean_zdtm_root)
> +	return zdtm_root
> +
> +class host_flavor:
> +	def __init__(self, opts):
> +		self.name = "host"
> +		self.ns = False
> +		self.root = None
> +
> +	def init(self, test_bin):
> +		pass
> +
> +	def fini(self):
> +		pass
> +
> +class ns_flavor:
> +	def __init__(self, opts):
> +		self.name = "ns"
> +		self.ns = True
> +		self.uns = False
> +		self.root = make_zdtm_root()
> +
> +	def init(self, test_bin):
> +		print "Construct root for %s" % test_bin
> +		subprocess.check_call(["mount", "--make-private", "--bind", ".", self.root])
> +
> +		if not os.access(self.root + "/.constructed", os.F_OK):
> +			for dir in ["/bin", "/etc", "/lib", "/lib64", "/dev", "/tmp"]:
> +				os.mkdir(self.root + dir)
> +
> +			os.mknod(self.root + "/dev/tty", stat.S_IFCHR | 0666, os.makedev(5, 0))
> +			os.mknod(self.root + "/.constructed", stat.S_IFREG | 0600)
> +
> +		ldd = subprocess.Popen(["ldd", test_bin], stdout = subprocess.PIPE)
> +		xl = re.compile('^(linux-gate.so|linux-vdso(64)?.so|not a dynamic)')
> +		libs = map(lambda x: x[1] == '=>' and x[2] or x[0],		\
> +				map(lambda x: x.split(),			\
> +					filter(lambda x: not xl.match(x),	\
> +						map(lambda x: x.strip(),	\
> +							filter(lambda x: x.startswith('\t'), ldd.stdout.readlines())))))
> +		ldd.wait()
> +
> +		for lib in libs:
> +			tlib = self.root + lib
> +			if not os.access(tlib, os.F_OK):
> +				dst = tempfile.mktemp(".tso", "", self.root + os.path.dirname(lib))
> +				shutil.copy2(lib, dst)
> +				os.rename(dst, tlib)
> +
> +	def fini(self):
> +		subprocess.check_call(["mount", "--make-private", self.root])
> +		subprocess.check_call(["umount", "-l", self.root])
> +
> +class userns_flavor(ns_flavor):
> +	def __init__(self, opts):
> +		ns_flavor.__init__(self, opts)
> +		self.name = "userns"
> +		self.uns = True
> +
> +flavors = { 'h': host_flavor, 'ns': ns_flavor, 'uns': userns_flavor }
> +
> +def tail(path):
> +	p = subprocess.Popen(['tail', '-n1', path],
> +			stdout = subprocess.PIPE)
> +	return p.stdout.readline()
> +
> +def test_flag(tdesc, flag):
> +	return flag in tdesc.get('flags', '').split()
> +
> +class test_fail_exc:
> +	def __init__(self, step):
> +		self.step = step
> +
> +class zdtm_test:
> +	def __init__(self, name, desc, flavor):
> +		self.__name = name
> +		self.__desc = desc
> +		self.__make_action('cleanout')
> +		self.__pid = 0
> +		self.__flavor = flavor
> +
> +	def __getpath(self, typ = ''):
> +		return os.path.join("zdtm/live/", self.__name + typ)
> +
> +	def __make_action(self, act, env = None, root = None):
> +		tpath = self.__getpath('.' + act)
> +		s_args = ['make', '--no-print-directory', \
> +			 	'-C', os.path.dirname(tpath), \
> +				      os.path.basename(tpath)]
> +
> +		if env:
> +			env = dict(os.environ, **env)
> +
> +		s = subprocess.Popen(s_args, env = env, cwd = root)
> +		s.wait()
> +
> +	def __pidfile(self):
> +		if self.__flavor.ns:
> +			return self.__getpath('.init.pid')
> +		else:
> +			return self.__getpath('.pid')
> +
> +	def __wait_task_die(self):
> +		tmo = 0.1
> +		while tmo < 3:
> +			try:
> +				os.kill(int(self.__pid), 0)
> +			except: # Died
> +				break
> +
> +			print "Wait for %s to die for %f" % (self.__name, tmo)
> +			time.sleep(tmo)
> +			tmo *= 2
> +		else:
> +			raise test_fail_exc("test die")
> +
> +	def start(self):
> +		env = {}
> +		self.__flavor.init(self.__getpath())
> +
> +		print "Start test"
> +		if not test_flag(self.__desc, 'suid'):
> +			env['ZDTM_UID'] = "18943"
> +			env['ZDTM_GID'] = "58467"
> +			env['ZDTM_GROUPS'] = "27495 48244"
> +		else:
> +			print "Test is SUID"
> +
> +		if self.__flavor.ns:
> +			env['ZDTM_NEWNS'] = "1"
> +			env['ZDTM_PIDFILE'] = os.path.realpath(self.__getpath('.init.pid'))
> +			env['ZDTM_ROOT'] = self.__flavor.root
> +
> +			if self.__flavor.uns:
> +				env['ZDTM_USERNS'] = "1"
> +
> +		self.__make_action('pid', env, self.__flavor.root)
> +
> +		try:
> +			os.kill(int(self.getpid()), 0)
> +		except:
> +			raise test_fail_exc("start")
> +
> +	def kill(self, sig = signal.SIGKILL):
> +		if self.__pid:
> +			os.kill(int(self.__pid), sig)
> +			self.gone(sig == signal.SIGKILL)
> +
> +		self.__flavor.fini()
> +
> +	def stop(self):
> +		print "Stop test"
> +		self.kill(signal.SIGTERM)
> +
> +		res = tail(self.__getpath('.out'))
> +		if not 'PASS' in res.split():
> +			raise test_fail_exc("result check")
> +
> +	def getpid(self):
> +		if self.__pid == 0:
> +			self.__pid = open(self.__pidfile()).readline().strip()
> +
> +		return self.__pid
> +
> +	def getname(self):
> +		return self.__name
> +
> +	def getcropts(self):
> +		opts = self.__desc.get('opts', []) + ["--pidfile", os.path.realpath(self.__pidfile())]
> +		if self.__flavor.ns:
> +			opts += ["--root", self.__flavor.root]
> +		return opts
> +
> +	def gone(self, force = True):
> +		self.__wait_task_die()
> +		self.__pid = 0
> +		if force or self.__flavor.ns:
> +			os.unlink(self.__pidfile())
> +
> +class criu_cli:
> +	def __init__(self, test):
> +		self.__test = test
> +		self.__dump_path = "dump/" + test.getname() + "/" + test.getpid()
> +		self.__iter = 0
> +		os.makedirs(self.__dump_path)
> +
> +	def __ddir(self):
> +		return os.path.join(self.__dump_path, "%d" % self.__iter)
> +
> +	def __criu(self, action, opts = [], log = None):
> +		if not log:
> +			log = action + ".log"
> +
> +		s_args = ["../criu", action, "-o", log, "-D", self.__ddir(), "-v4"]
> +		s_args += opts
> +		s_args += self.__test.getcropts()
> +
> +		print "Run CRIU: [" + " ".join(s_args) + "]"
> +		cr = subprocess.Popen(s_args)
> +		ret = cr.wait()
> +		if ret != 0:
> +			raise test_fail_exc("CRIU %s" % action)
> +
> +	def dump(self, action, opts = []):
> +		self.__iter += 1
> +		os.mkdir(self.__ddir())
> +
> +		a_opts = ["-t", self.__test.getpid()]
> +		if self.__iter > 1:
> +			a_opts += ["--prev-images-dir", "../%d" % (self.__iter - 1), "--track-mem"]
> +
> +		self.__criu(action, opts = a_opts + opts)
> +
> +	def restore(self):
> +		self.__criu("restore", opts = ["--restore-detached"])
> +
> +def cr(test, opts):
> +	if opts['nocr']:
> +		return
> +
> +	cr_api = criu_cli(test)
> +
> +	for i in xrange(0, int(opts['iters'] or 1)):
> +		for p in xrange(0, int(opts['pre'] or 0)):
> +			cr_api.dump("pre-dump")
> +
> +		if opts['norst']:
> +			cr_api.dump("dump", opts = ["--leave-running"])
> +		else:
> +			cr_api.dump("dump")
> +			test.gone()
> +			cr_api.restore()
> +
> +def get_visible_state(test):
> +	fds = os.listdir("/proc/%s/fdinfo" % test.getpid())
> +	maps = map(lambda x: x.split()[0], open("/proc/%s/maps" % test.getpid()).readlines())
> +	return (set(fds), set(maps))
> +
> +def check_visible_state(test, state):
> +	new = get_visible_state(test)
> +	if new[0] != state[0]:
> +		raise test_fail_exc("fds compare")
> +	if new[1] != state[1]:
> +		raise test_fail_exc("maps compare")
> +
> +def run_test_flav(tname, tdesc, flavor, opts):
> +	print "Run %s in %s" % (tname, flavor)
> +
> +	f = flavors[flavor](opts)
> +	t = zdtm_test(tname, tdesc, f)
> +
> +	try:
> +		t.start()
> +		s = get_visible_state(t)
> +		cr(t, opts)
> +		check_visible_state(t, s)
> +		t.stop()
> +	except test_fail_exc as e:
> +		t.kill()
> +		print "Test %s FAIL at %s" % (tname, e.step)
> +	else:
> +		print "Test %s PASS\n" % tname
> +
> +def run_test(tname, opts, tlist):
> +	global arch
> +
> +	tdesc = tlist.get(tname, default_test) or default_test
> +	if tdesc.get('arch', arch) != arch:
> +		print "Test not available for %s" % arch
> +		return
> +
> +	tflav = tdesc.get('flavor', 'h ns uns').split()
> +	rflav = (opts['flavor'] or 'h,ns,uns').split(',')
> +	flav = set(tflav) & set(rflav)
> +
> +	if flav:
> +		for f in flav:
> +			run_test_flav(tname, tdesc, f, opts)
> +	else:
> +		print "Selected flavor(s) not supported for test"
> +
> +def run_tests(opts, tlist):
> +	if opts['all']:
> +		torun = tlist
> +	elif opts['test']:
> +		torun = opts['test']
> +	else:
> +		print "Specify test with -t <name> or -a"
> +		return
> +
> +	for t in torun:
> +		run_test(t, opts, tlist)
> +
> +def list_tests(opts, tlist):
> +	for t in tlist:
> +		print t
> +
> +#
> +# main() starts here
> +#
> +
> +p = argparse.ArgumentParser("ZDTM test suite")
> +sp = p.add_subparsers(help = "Use --help for list of actions")
> +
> +rp = sp.add_parser("run", help = "Run test(s)")
> +rp.set_defaults(action = run_tests)
> +rp.add_argument("-a", "--all", action = 'store_true')
> +rp.add_argument("-t", "--test", help = "Test name", action = 'append')
> +rp.add_argument("-f", "--flavor", help = "Flavor to run")
> +
> +rp.add_argument("--pre", help = "Do some pre-dumps before dump")
> +rp.add_argument("--nocr", help = "Do not CR anything, just check test works", action = 'store_true')
> +rp.add_argument("--norst", help = "Don't restore tasks, leave them running after dump", action = 'store_true')
> +rp.add_argument("--iters", help = "Do CR cycle several times before check")
> +
> +lp = sp.add_parser("list", help = "List tests")
> +lp.set_defaults(action = list_tests)
> +
> +opts = vars(p.parse_args())
> +tlist = yaml.load(open("zdtm.list"))
> +
> +opts['action'](opts, tlist)
> -- 
> 1.9.3
> 


More information about the CRIU mailing list