[CRIU] [PATCH][RFC] zdtm: Move towards the new generation of criu testing
Pavel Emelyanov
xemul at parallels.com
Thu Oct 1 05:11:32 PDT 2015
Hi.
After recent set with fault-injection it seemed to me that the
existing 1.2KLOC shell script would be quite hard to extend and
maintain in the future. I propose to bring our test system onto
the next level and write it on a more suitable language :)
So, things to get fixed in the first place
1. Introduce per-test extendable description. Right now this
description is scatered over test itself, .opts file, scripts
and the zdtm.sh itself (list forming and random hacks).
In new approach all the tests' metadata is in the zdtm.list
file in yaml format.
2. Introduce the environments in which test can be run. Now we
have 3 -- on host, in namespaces and in user-namespaces --
then we'll have the 4th -- with faults injected.
In this approach there's a thing call 'flavor' that creates
and environment for a test to be run in.
So here's the first version -- the minimal wannabe replacement
for zdtm.sh written in python. It can
1. List tests :)
2. Run tests in 3 flavors (\union supported by the particular
test) -- host, namespace, user-namespace. Default steps are
start, dump, restore, stop and check.
3. Run non-standard paths, in particular
a. Don't CR at all checking that the test itself is OK
b. Don't restore after dump and check test survived
c. Do several pre-dumps before dump
d. Run the CR sequence again on restored test to check that
restored process is in dumpable state
4. Check opened descriptors and maps before to coincide
with those after C/R (has issue, will fix)
Plans for the nearest changes
1. Make it functional as zdtm.sh is. This includes
a. Add page-server
b. Exclude tests from run list
c. Ability to only start the test
d. Run tests in parallel
2. Ability to add more than just zdtm/ tests into run. Right now
all non-zdtm tests are scatered over the test/ directory and
each has its own script to launch :)
In the future I plan to add other tests in the zdtm.list and
rename one (and zdtm.py too).
3. Ability to check not only CLI, but RPC as well
4. Add fault-injection facility
Signed-off-by: Pavel Emelyanov <xemul at parallels.com>
---
test/zdtm.list | 328 +++++++++++++++++++++++++++++++++++++++++++++++++++++
test/zdtm.py | 348 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 676 insertions(+)
create mode 100644 test/zdtm.list
create mode 100755 test/zdtm.py
diff --git a/test/zdtm.list b/test/zdtm.list
new file mode 100644
index 0000000..d1cd184
--- /dev/null
+++ b/test/zdtm.list
@@ -0,0 +1,328 @@
+static/pipe00:
+static/pipe01:
+static/pipe02:
+static/busyloop00:
+static/cwd00:
+static/cwd01:
+static/cwd02:
+static/env00:
+static/maps00:
+static/maps01:
+ flags: suid
+ flavor: h ns
+static/maps02:
+static/maps04:
+static/maps05:
+static/mlock_setuid:
+ flags: suid
+ flavor: h ns
+static/maps_file_prot:
+static/mprotect00:
+static/mtime_mmap:
+static/sleeping00:
+static/write_read00:
+static/write_read01:
+static/write_read02:
+static/write_read10:
+static/wait00:
+static/vdso00:
+static/sched_prio00:
+ flags: suid
+ flavor: h ns
+static/sched_policy00:
+ flags: suid
+ flavor: h ns
+static/file_shared:
+static/file_append:
+static/timers:
+static/posix_timers:
+static/futex:
+static/futex-rl:
+static/xids00:
+static/groups:
+ flags: suid
+static/pthread00:
+static/pthread01:
+static/umask00:
+streaming/pipe_loop00:
+streaming/pipe_shared00:
+transition/file_read:
+static/sockets00:
+ flags: suid
+static/sockets01:
+static/sockets02:
+static/sock_opts00:
+ flags: suid
+static/sock_opts01:
+ flags: suid
+static/sockets_spair:
+static/sockets_dgram:
+static/socket_dgram_data:
+static/socket_queues:
+static/deleted_unix_sock:
+static/sk-unix-unconn:
+static/sk-unix-rel:
+static/pid00:
+ flags: suid
+static/pstree:
+static/caps00:
+ flags: suid
+static/cmdlinenv00:
+ flags: suid
+static/socket_listen:
+static/socket_listen6:
+static/packet_sock:
+ flags: suid
+static/packet_sock_mmap:
+ flags: suid
+static/socket_udp:
+static/sock_filter:
+static/socket6_udp:
+static/socket_udplite:
+static/selfexe00:
+static/link10:
+static/unlink_fstat00:
+static/unlink_fstat01:
+static/unlink_fstat02:
+static/unlink_fstat03:
+ opts: --link-remap
+static/unlink_mmap00:
+static/unlink_mmap01:
+static/unlink_mmap02:
+static/rmdir_open:
+static/eventfs00:
+static/signalfd00:
+static/inotify00:
+ opts: --link-remap
+static/inotify_irmap:
+ flags: suid
+static/fanotify00:
+ flags: suid
+ flavor: h ns
+static/unbound_sock:
+static/fifo-rowo-pair:
+static/fifo-ghost:
+static/fifo:
+static/fifo_wronly:
+static/fifo_ro:
+static/unlink_fifo:
+static/unlink_fifo_wronly:
+static/zombie00:
+static/rlimits00:
+transition/fork:
+transition/fork2:
+transition/thread-bomb:
+static/pty00:
+static/pty01:
+static/pty04:
+static/tty02:
+static/tty03:
+static/console:
+ flags: suid
+ flavor: h ns
+static/vt:
+ flags: suid
+ flavor: h ns
+static/child_opened_proc:
+static/cow01:
+ flags: suid
+ flavor: h ns
+static/pdeath_sig:
+static/fdt_shared:
+static/file_locks00:
+ flags: excl
+ opts: --file-locks
+static/file_locks01:
+ flags: excl
+ opts: --file-locks
+static/file_locks02:
+ flags: excl
+ opts: --file-locks
+static/file_locks03:
+ flags: excl
+ opts: --file-locks
+static/file_locks04:
+ flags: excl
+ opts: --file-locks
+static/file_locks05:
+ flags: excl
+ opts: --file-locks
+static/sigpending:
+static/sigaltstack:
+static/sk-netlink:
+ flags: suid
+static/proc-self:
+static/grow_map:
+static/grow_map02:
+static/grow_map03:
+static/stopped:
+static/chroot:
+ flags: suid
+static/chroot-file:
+ flags: suid
+static/rtc:
+ flags: nouns suid crlib
+transition/maps007:
+ flags: suid
+static/dumpable01:
+static/dumpable02:
+ flavor: h ns
+static/deleted_dev:
+ flags: suid
+ flavor: h ns
+static/fpu00:
+ arch: x86_64
+static/fpu01:
+ arch: x86_64
+static/mmx00:
+ arch: x86_64
+static/sse00:
+ arch: x86_64
+static/sse20:
+ arch: x86_64
+static/vdso01:
+ arch: x86_64
+static/vsx:
+ arch: ppc64le
+static/file_fown:
+ flavor: h
+static/socket-ext:
+ flavor: h
+ opts: --ext-unix-sk
+static/socket-tcp:
+ flavor: h
+ opts: --tcp-established
+static/socket-tcp6:
+ flavor: h
+ opts: --tcp-established
+streaming/socket-tcp:
+ flavor: h
+ opts: --tcp-established
+streaming/socket-tcp6:
+ flavor: h
+ opts: --tcp-established
+static/socket-tcpbuf:
+ flavor: h
+ opts: --tcp-established
+static/socket-tcpbuf-local:
+ flavor: h
+ opts: --tcp-established
+static/socket-tcpbuf6:
+ flavor: h
+ opts: --tcp-established
+static/pty03:
+ flavor: h
+static/mountpoints:
+ flags: suid
+ flavor: h
+static/utsname:
+ flavor: h
+static/ipc_namespace:
+ flavor: h
+static/shm:
+ flavor: h
+static/msgque:
+ flavor: h
+static/sem:
+ flavor: h
+transition/ipc:
+ flavor: h
+static/netns-nf:
+ flavor: h
+static/netns:
+ flavor: h
+static/cgroup00:
+ flags: suid
+ flavor: h
+ opts: --manage-cgroups
+static/cgroup01:
+ flags: suid
+ flavor: h
+ opts: --manage-cgroups
+static/cgroup02:
+ flags: suid
+ flavor: h
+ opts: --manage-cgroups --cgroup-root /newroot --cgroup-root name=zdtmtst:/zdtmtstroot
+static/remap_dead_pid:
+ flavor: h
+static/poll:
+ flavor: h
+static/apparmor:
+ flags: suid
+static/different_creds:
+ flags: suid crfail
+ flavor: h
+static/aio00:
+ feature: aio
+static/timerfd:
+ feature: timerfd
+static/session00:
+ flavor: ns uns
+static/session01:
+ flavor: ns uns
+static/tempfs:
+ flags: suid
+ flavor: ns uns
+static/tempfs_ro:
+ flags: suid
+ flavor: ns
+static/mnt_ro_bind:
+ flags: suid
+ flavor: ns uns
+static/mount_paths:
+ flags: suid
+ flavor: ns uns
+static/bind-mount:
+ flags: suid
+ flavor: ns uns
+static/netns-dev:
+ flags: suid
+ flavor: ns uns
+static/mnt_ext_auto:
+ flavor: ns uns
+ feature: mntid
+ opts: --ext-mount-map auto --enable-external-sharing
+static/mnt_ext_master:
+ flavor: ns uns
+ feature: mntid
+ opts: --ext-mount-map auto --enable-external-masters
+static/mntns_open:
+ flags: suid
+ flavor: ns uns
+ feature: mntid
+static/mntns_link_remap:
+ flags: suid
+ flavor: ns
+ feature: mntid
+ opts: --link-remap
+static/mntns_link_ghost:
+ flags: suid
+ flavor: ns
+ feature: mntid
+static/mntns_shared_bind:
+ flags: suid
+ flavor: ns uns
+ feature: mntid
+static/mntns_shared_bind02:
+ flags: suid
+ flavor: ns uns
+ feature: mntid
+static/mntns_root_bind:
+ flags: suid
+ flavor: ns uns
+ feature: mntid
+static/mntns_deleted:
+ flags: suid
+ flavor: ns uns
+ feature: mntid
+static/tun:
+ flags: suid
+ flavor: ns uns
+ feature: tun
+static/seccomp_strict:
+ flags: suid
+ flavor: h
+ feature: seccomp_suspend
+static/clean_mntns:
+ flags: suid
+ flavor: ns
diff --git a/test/zdtm.py b/test/zdtm.py
new file mode 100755
index 0000000..53be6f2
--- /dev/null
+++ b/test/zdtm.py
@@ -0,0 +1,348 @@
+#!/bin/env python
+import argparse
+import yaml
+import os
+import subprocess
+import time
+import tempfile
+import shutil
+import re
+import stat
+import signal
+import atexit
+
+default_test={ }
+zdtm_root = None
+arch = os.uname()[4]
+
+def clean_zdtm_root():
+ global zdtm_root
+ if zdtm_root:
+ os.rmdir(zdtm_root)
+
+def make_zdtm_root():
+ global zdtm_root
+ if not zdtm_root:
+ zdtm_root = tempfile.mkdtemp("", "criu-root-", "/tmp")
+ atexit.register(clean_zdtm_root)
+ return zdtm_root
+
+class host_flavor:
+ def __init__(self, opts):
+ self.name = "host"
+ self.ns = False
+ self.root = None
+
+ def init(self, test_bin):
+ pass
+
+ def fini(self):
+ pass
+
+class ns_flavor:
+ def __init__(self, opts):
+ self.name = "ns"
+ self.ns = True
+ self.uns = False
+ self.root = make_zdtm_root()
+
+ def init(self, test_bin):
+ print "Construct root for %s" % test_bin
+ subprocess.check_call(["mount", "--make-private", "--bind", ".", self.root])
+
+ if not os.access(self.root + "/.constructed", os.F_OK):
+ for dir in ["/bin", "/etc", "/lib", "/lib64", "/dev", "/tmp"]:
+ os.mkdir(self.root + dir)
+
+ os.mknod(self.root + "/dev/tty", stat.S_IFCHR | 0666, os.makedev(5, 0))
+ os.mknod(self.root + "/.constructed", stat.S_IFREG | 0600)
+
+ ldd = subprocess.Popen(["ldd", test_bin], stdout = subprocess.PIPE)
+ xl = re.compile('^(linux-gate.so|linux-vdso(64)?.so|not a dynamic)')
+ libs = map(lambda x: x[1] == '=>' and x[2] or x[0], \
+ map(lambda x: x.split(), \
+ filter(lambda x: not xl.match(x), \
+ map(lambda x: x.strip(), \
+ filter(lambda x: x.startswith('\t'), ldd.stdout.readlines())))))
+ ldd.wait()
+
+ for lib in libs:
+ tlib = self.root + lib
+ if not os.access(tlib, os.F_OK):
+ dst = tempfile.mktemp(".tso", "", self.root + os.path.dirname(lib))
+ shutil.copy2(lib, dst)
+ os.rename(dst, tlib)
+
+ def fini(self):
+ subprocess.check_call(["mount", "--make-private", self.root])
+ subprocess.check_call(["umount", "-l", self.root])
+
+class userns_flavor(ns_flavor):
+ def __init__(self, opts):
+ ns_flavor.__init__(self, opts)
+ self.name = "userns"
+ self.uns = True
+
+flavors = { 'h': host_flavor, 'ns': ns_flavor, 'uns': userns_flavor }
+
+def tail(path):
+ p = subprocess.Popen(['tail', '-n1', path],
+ stdout = subprocess.PIPE)
+ return p.stdout.readline()
+
+def test_flag(tdesc, flag):
+ return flag in tdesc.get('flags', '').split()
+
+class test_fail_exc:
+ def __init__(self, step):
+ self.step = step
+
+class zdtm_test:
+ def __init__(self, name, desc, flavor):
+ self.__name = name
+ self.__desc = desc
+ self.__make_action('cleanout')
+ self.__pid = 0
+ self.__flavor = flavor
+
+ def __getpath(self, typ = ''):
+ return os.path.join("zdtm/live/", self.__name + typ)
+
+ def __make_action(self, act, env = None, root = None):
+ tpath = self.__getpath('.' + act)
+ s_args = ['make', '--no-print-directory', \
+ '-C', os.path.dirname(tpath), \
+ os.path.basename(tpath)]
+
+ if env:
+ env = dict(os.environ, **env)
+
+ s = subprocess.Popen(s_args, env = env, cwd = root)
+ s.wait()
+
+ def __pidfile(self):
+ if self.__flavor.ns:
+ return self.__getpath('.init.pid')
+ else:
+ return self.__getpath('.pid')
+
+ def __wait_task_die(self):
+ tmo = 0.1
+ while tmo < 3:
+ try:
+ os.kill(int(self.__pid), 0)
+ except: # Died
+ break
+
+ print "Wait for %s to die for %f" % (self.__name, tmo)
+ time.sleep(tmo)
+ tmo *= 2
+ else:
+ raise test_fail_exc("test die")
+
+ def start(self):
+ env = {}
+ self.__flavor.init(self.__getpath())
+
+ print "Start test"
+ if not test_flag(self.__desc, 'suid'):
+ env['ZDTM_UID'] = "18943"
+ env['ZDTM_GID'] = "58467"
+ env['ZDTM_GROUPS'] = "27495 48244"
+ else:
+ print "Test is SUID"
+
+ if self.__flavor.ns:
+ env['ZDTM_NEWNS'] = "1"
+ env['ZDTM_PIDFILE'] = os.path.realpath(self.__getpath('.init.pid'))
+ env['ZDTM_ROOT'] = self.__flavor.root
+
+ if self.__flavor.uns:
+ env['ZDTM_USERNS'] = "1"
+
+ self.__make_action('pid', env, self.__flavor.root)
+
+ try:
+ os.kill(int(self.getpid()), 0)
+ except:
+ raise test_fail_exc("start")
+
+ def kill(self, sig = signal.SIGKILL):
+ if self.__pid:
+ os.kill(int(self.__pid), sig)
+ self.gone(sig == signal.SIGKILL)
+
+ self.__flavor.fini()
+
+ def stop(self):
+ print "Stop test"
+ self.kill(signal.SIGTERM)
+
+ res = tail(self.__getpath('.out'))
+ if not 'PASS' in res.split():
+ raise test_fail_exc("result check")
+
+ def getpid(self):
+ if self.__pid == 0:
+ self.__pid = open(self.__pidfile()).readline().strip()
+
+ return self.__pid
+
+ def getname(self):
+ return self.__name
+
+ def getcropts(self):
+ opts = self.__desc.get('opts', []) + ["--pidfile", os.path.realpath(self.__pidfile())]
+ if self.__flavor.ns:
+ opts += ["--root", self.__flavor.root]
+ return opts
+
+ def gone(self, force = True):
+ self.__wait_task_die()
+ self.__pid = 0
+ if force or self.__flavor.ns:
+ os.unlink(self.__pidfile())
+
+class criu_cli:
+ def __init__(self, test):
+ self.__test = test
+ self.__dump_path = "dump/" + test.getname() + "/" + test.getpid()
+ self.__iter = 0
+ os.makedirs(self.__dump_path)
+
+ def __ddir(self):
+ return os.path.join(self.__dump_path, "%d" % self.__iter)
+
+ def __criu(self, action, opts = [], log = None):
+ if not log:
+ log = action + ".log"
+
+ s_args = ["../criu", action, "-o", log, "-D", self.__ddir(), "-v4"]
+ s_args += opts
+ s_args += self.__test.getcropts()
+
+ print "Run CRIU: [" + " ".join(s_args) + "]"
+ cr = subprocess.Popen(s_args)
+ ret = cr.wait()
+ if ret != 0:
+ raise test_fail_exc("CRIU %s" % action)
+
+ def dump(self, action, opts = []):
+ self.__iter += 1
+ os.mkdir(self.__ddir())
+
+ a_opts = ["-t", self.__test.getpid()]
+ if self.__iter > 1:
+ a_opts += ["--prev-images-dir", "../%d" % (self.__iter - 1), "--track-mem"]
+
+ self.__criu(action, opts = a_opts + opts)
+
+ def restore(self):
+ self.__criu("restore", opts = ["--restore-detached"])
+
+def cr(test, opts):
+ if opts['nocr']:
+ return
+
+ cr_api = criu_cli(test)
+
+ for i in xrange(0, int(opts['iters'] or 1)):
+ for p in xrange(0, int(opts['pre'] or 0)):
+ cr_api.dump("pre-dump")
+
+ if opts['norst']:
+ cr_api.dump("dump", opts = ["--leave-running"])
+ else:
+ cr_api.dump("dump")
+ test.gone()
+ cr_api.restore()
+
+def get_visible_state(test):
+ fds = os.listdir("/proc/%s/fdinfo" % test.getpid())
+ maps = map(lambda x: x.split()[0], open("/proc/%s/maps" % test.getpid()).readlines())
+ return (set(fds), set(maps))
+
+def check_visible_state(test, state):
+ new = get_visible_state(test)
+ if new[0] != state[0]:
+ raise test_fail_exc("fds compare")
+ if new[1] != state[1]:
+ raise test_fail_exc("maps compare")
+
+def run_test_flav(tname, tdesc, flavor, opts):
+ print "Run %s in %s" % (tname, flavor)
+
+ f = flavors[flavor](opts)
+ t = zdtm_test(tname, tdesc, f)
+
+ try:
+ t.start()
+ s = get_visible_state(t)
+ cr(t, opts)
+ check_visible_state(t, s)
+ t.stop()
+ except test_fail_exc as e:
+ t.kill()
+ print "Test %s FAIL at %s" % (tname, e.step)
+ else:
+ print "Test %s PASS\n" % tname
+
+def run_test(tname, opts, tlist):
+ global arch
+
+ tdesc = tlist.get(tname, default_test) or default_test
+ if tdesc.get('arch', arch) != arch:
+ print "Test not available for %s" % arch
+ return
+
+ tflav = tdesc.get('flavor', 'h ns uns').split()
+ rflav = (opts['flavor'] or 'h,ns,uns').split(',')
+ flav = set(tflav) & set(rflav)
+
+ if flav:
+ for f in flav:
+ run_test_flav(tname, tdesc, f, opts)
+ else:
+ print "Selected flavor(s) not supported for test"
+
+def run_tests(opts, tlist):
+ if opts['all']:
+ torun = tlist
+ elif opts['test']:
+ torun = opts['test']
+ else:
+ print "Specify test with -t <name> or -a"
+ return
+
+ for t in torun:
+ run_test(t, opts, tlist)
+
+def list_tests(opts, tlist):
+ for t in tlist:
+ print t
+
+#
+# main() starts here
+#
+
+p = argparse.ArgumentParser("ZDTM test suite")
+sp = p.add_subparsers(help = "Use --help for list of actions")
+
+rp = sp.add_parser("run", help = "Run test(s)")
+rp.set_defaults(action = run_tests)
+rp.add_argument("-a", "--all", action = 'store_true')
+rp.add_argument("-t", "--test", help = "Test name", action = 'append')
+rp.add_argument("-f", "--flavor", help = "Flavor to run")
+
+rp.add_argument("--pre", help = "Do some pre-dumps before dump")
+rp.add_argument("--nocr", help = "Do not CR anything, just check test works", action = 'store_true')
+rp.add_argument("--norst", help = "Don't restore tasks, leave them running after dump", action = 'store_true')
+rp.add_argument("--iters", help = "Do CR cycle several times before check")
+
+lp = sp.add_parser("list", help = "List tests")
+lp.set_defaults(action = list_tests)
+
+opts = vars(p.parse_args())
+tlist = yaml.load(open("zdtm.list"))
+
+opts['action'](opts, tlist)
--
1.9.3
More information about the CRIU
mailing list