[CRIU] skip() does not work in zdtm tests

[Filipe Brandenburger]

Hi Andrew,

I had in mind something like using exit codes to signal whether a test
should be skipped or when there was an error executing some of the
commands of the test...

I recalled there was some framework that used some error codes for it,
I googled it and saw I was thinking of GNU autotest which uses exit
code 77 for skip and 99 for hard failure:

https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Writing-Testsuites.html#index-AT_005fCHECK-2296

I was thinking of the current macros in zdtm (pass(), fail(), err(),
warn()) and I think they have some shortcomings:
1) they always print the errno value, but in some cases it doesn't
make sense to print it;
2) there's a name clash, for instance there's a BSD function err()
with a different prototype
(http://www.freebsd.org/cgi/man.cgi?query=err)
3) they are not really fatal, while some of them should.

I was thinking of replacing them with a new set of functions and/or
macros that address the three points above. Using a prefix such as
zdtm_* to avoid the name clash. Defining two sets of functions:
zdtm_err() and zdtm_warn() vs. zdtm_perr() and zdtm_pwarn() where the
"p" variation prints the errno information (think perror() for the
"p") and make some of them exit when appropriate, e.g. zdtm_err()
would exit(99), zdtm_skip() would exit(77) and zdtm_fail() would
exit(1).

What do you think of that idea?

I will try to write a few patches to try this concept and send them to
you later this week.

Cheers,
Filipe


On Mon, Jun 23, 2014 at 11:54 PM, Andrew Vagin <avagin at parallels.com> wrote:
> On Mon, Jun 23, 2014 at 11:18:04PM -0700, Filipe Brandenburger wrote:
>> Hi Andrew,
>>
>> I've been looking at conditionally skipping a zdtm test. I noticed
>> there's a skip() macro that logs a SKIP: ... message but I tried that
>> with an exit 0 and that doesn't really seem to work...
>
> we have SKIP o_O. I have heard first time about it:).
>
> zdtm was started for the OpenVZ project, then it was adapted for CRIU,
> so we don't use all its features.
>
> Pls, look at the attached path. I think we need smth like it.
>
> Thanks.
>
>>
>> I wrote a short proof of concept (attached) and tried it. Without
>> skipping the test, everything works as expected:
>>
>> $ sudo test/zdtm.sh ns/static/skiptest
>> Execute zdtm/live/static/skiptest
>> ./skiptest --pidfile=skiptest.pid --outfile=skiptest.out
>>   PID TTY          TIME CMD
>> 13953 ?        00:00:00 skiptest
>> Dump 13953
>> Restore
>> Check results 13979
>> Waiting...
>> 05:21:08.607:     4: PASS
>> Test: zdtm/live/static/skiptest, Result: PASS
>> ZDTM tests PASS.
>>
>>
>> But when I set the env var that makes it try to skip, it fails, with
>> an ugly error message:
>>
>> $ sudo env SKIP_TEST=1 test/zdtm.sh ns/static/skiptest
>> Execute zdtm/live/static/skiptest
>> ./skiptest --pidfile=skiptest.pid --outfile=skiptest.out
>> The test failed (0, 1)
>> make: *** [skiptest.pid] Error 1
>> ERROR: fail to start zdtm/live/static/skiptest
>> cat: /home/filbranden/devel/criu/test/zdtm/live/static/skiptest.init.pid:
>> No such file or directory
>> error: list of process IDs must follow -p
>> Test: zdtm/live/static/skiptest, Result: FAIL
>> ==================================== ERROR
>> Test: zdtm/live/static/skiptest, Namespace: 1
>> Output file: /home/filbranden/devel/criu/test/zdtm/live/static/skiptest.out
>> ------------------------------------------------------------------------------
>> 05:20:53.014:     4: SKIP: skiptest.c:12: I would like to skip this test.
>> 05:20:53.014:     3: ERR: test.c:176: Test exited with unexpectedly with code 0
>>  (errno = 11 (No such file or directory))
>>   PID   SID COMMAND
>>     1     1 skiptest
>>     5     1 ps
>> waitpid() failed: No child processes
>> ------------------------------------- END -------------------------------------
>> ================================= ERROR OVER
>>
>>
>> It seems the problem is the way in which test_init() sets up
>> parent/child processes for the test runner and isn't really ready for
>> a test that doesn't go into test_daemon() and test_waitsig()...
>>
>> I was wondering what was the initial plan for skip, whether it is
>> actually supposed to work and maybe I've been using it wrong (even
>> though the other examples I saw seem to be using the same pattern.) Do
>> you have a suggestion on how to fix this behavior?
>>
>> Thanks!
>> Filipe
>
>