[CRIU] skip() does not work in zdtm tests

[Andrew Vagin]

On Tue, Jun 24, 2014 at 08:29:57PM -0700, Filipe Brandenburger wrote:
> Hi Andrew,
> 
> I had in mind something like using exit codes to signal whether a test
> should be skipped or when there was an error executing some of the
> commands of the test...
> 
> I recalled there was some framework that used some error codes for it,
> I googled it and saw I was thinking of GNU autotest which uses exit
> code 77 for skip and 99 for hard failure:
> 
> https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Writing-Testsuites.html#index-AT_005fCHECK-2296
> 
> I was thinking of the current macros in zdtm (pass(), fail(), err(),
> warn()) and I think they have some shortcomings:
> 1) they always print the errno value, but in some cases it doesn't
> make sense to print it;
> 2) there's a name clash, for instance there's a BSD function err()
> with a different prototype
> (http://www.freebsd.org/cgi/man.cgi?query=err)
> 3) they are not really fatal, while some of them should.
> 
> I was thinking of replacing them with a new set of functions and/or
> macros that address the three points above. Using a prefix such as
> zdtm_* to avoid the name clash. Defining two sets of functions:
> zdtm_err() and zdtm_warn() vs. zdtm_perr() and zdtm_pwarn() where the
> "p" variation prints the errno information (think perror() for the
> "p") and make some of them exit when appropriate, e.g. zdtm_err()
> would exit(99), zdtm_skip() would exit(77) and zdtm_fail() would
> exit(1).
> 
> What do you think of that idea?

I don't have objection yet.

To be honest, I don't like to complicate zdtm. My experience shows that
developers write tests only if a framework is enough simple and if it
doesn't require to spend much time for learning.

I don't say that what you want to do will complicate zdtm too much,
it's just my fears.

> 
> I will try to write a few patches to try this concept and send them to
> you later this week.

Ok

> 
> Cheers,
> Filipe
> 
> 
> On Mon, Jun 23, 2014 at 11:54 PM, Andrew Vagin <avagin at parallels.com> wrote:
> > On Mon, Jun 23, 2014 at 11:18:04PM -0700, Filipe Brandenburger wrote:
> >> Hi Andrew,
> >>
> >> I've been looking at conditionally skipping a zdtm test. I noticed
> >> there's a skip() macro that logs a SKIP: ... message but I tried that
> >> with an exit 0 and that doesn't really seem to work...
> >
> > we have SKIP o_O. I have heard first time about it:).
> >
> > zdtm was started for the OpenVZ project, then it was adapted for CRIU,
> > so we don't use all its features.
> >
> > Pls, look at the attached path. I think we need smth like it.
> >
> > Thanks.
> >
> >>
> >> I wrote a short proof of concept (attached) and tried it. Without
> >> skipping the test, everything works as expected:
> >>
> >> $ sudo test/zdtm.sh ns/static/skiptest
> >> Execute zdtm/live/static/skiptest
> >> ./skiptest --pidfile=skiptest.pid --outfile=skiptest.out
> >>   PID TTY          TIME CMD
> >> 13953 ?        00:00:00 skiptest
> >> Dump 13953
> >> Restore
> >> Check results 13979
> >> Waiting...
> >> 05:21:08.607:     4: PASS
> >> Test: zdtm/live/static/skiptest, Result: PASS
> >> ZDTM tests PASS.
> >>
> >>
> >> But when I set the env var that makes it try to skip, it fails, with
> >> an ugly error message:
> >>
> >> $ sudo env SKIP_TEST=1 test/zdtm.sh ns/static/skiptest
> >> Execute zdtm/live/static/skiptest
> >> ./skiptest --pidfile=skiptest.pid --outfile=skiptest.out
> >> The test failed (0, 1)
> >> make: *** [skiptest.pid] Error 1
> >> ERROR: fail to start zdtm/live/static/skiptest
> >> cat: /home/filbranden/devel/criu/test/zdtm/live/static/skiptest.init.pid:
> >> No such file or directory
> >> error: list of process IDs must follow -p
> >> Test: zdtm/live/static/skiptest, Result: FAIL
> >> ==================================== ERROR
> >> Test: zdtm/live/static/skiptest, Namespace: 1
> >> Output file: /home/filbranden/devel/criu/test/zdtm/live/static/skiptest.out
> >> ------------------------------------------------------------------------------
> >> 05:20:53.014:     4: SKIP: skiptest.c:12: I would like to skip this test.
> >> 05:20:53.014:     3: ERR: test.c:176: Test exited with unexpectedly with code 0
> >>  (errno = 11 (No such file or directory))
> >>   PID   SID COMMAND
> >>     1     1 skiptest
> >>     5     1 ps
> >> waitpid() failed: No child processes
> >> ------------------------------------- END -------------------------------------
> >> ================================= ERROR OVER
> >>
> >>
> >> It seems the problem is the way in which test_init() sets up
> >> parent/child processes for the test runner and isn't really ready for
> >> a test that doesn't go into test_daemon() and test_waitsig()...
> >>
> >> I was wondering what was the initial plan for skip, whether it is
> >> actually supposed to work and maybe I've been using it wrong (even
> >> though the other examples I saw seem to be using the same pattern.) Do
> >> you have a suggestion on how to fix this behavior?
> >>
> >> Thanks!
> >> Filipe
> >
> >