[CRIU] [PATCH][RFC] zdtm: Move towards the new generation of criu testing

[Christopher Covington]

On 10/01/2015 09:46 AM, Pavel Emelyanov wrote:
> On 10/01/2015 03:59 PM, Christopher Covington wrote:
>> Hi Pavel,
>>
>> On 10/01/2015 08:11 AM, Pavel Emelyanov wrote:
>>> Hi.
>>>
>>> After recent set with fault-injection it seemed to me that the
>>> existing 1.2KLOC shell script would be quite hard to extend and
>>> maintain in the future. I propose to bring our test system onto
>>> the next level and write it on a more suitable language :)
>>
>> Great idea.
> 
> BTW, won't you have issues on your ARM boxes? IIRC you had some
> strange ones even with shell-based zdtm, will python be OK for 
> your setups?

I've mostly given up on the minimal, embedded style root filesytem. But if I
did have the time to support it, I'd maybe want to run zdtm.py on an x86 host
and either have it generate shell scripts that could be run on a minimal guest
or have zdtm.py remotely control a minimal guest (over tty or ssh).

>>> So, things to get fixed in the first place
>>>
>>> 1. Introduce per-test extendable description. Right now this
>>>    description is scatered over test itself, .opts file, scripts
>>>    and the zdtm.sh itself (list forming and random hacks).
>>>
>>>    In new approach all the tests' metadata is in the zdtm.list
>>>    file in yaml format.
>>
>> So you've reduced the scattering from test, .opts, scripts, and zdtm.sh (4) to
>> test and zdtm.list (2). Do you think it's useful to have the information in
>> two places (the redundancy could help catch errors for example)? Why not have
>> all the information in the test?
> 
> Yes, yes, suggestions are welcome :) I'm not 100% happy with zdtm.list I 
> currently have.
> 
>> For example:
>>
>> A) Build the list by recursively finding executables. 
> 
> Tried that :) First of all, we don't run everything that executes :) E.g.
> the zdtm/ dir contains some tests that we know don't work, but they're
> still compile.

Perhaps such tests should be self-skipping if they aren't already (run, check
required features, print skip message, and exit).

> Second, some tests out there can be scripts.

chmod +x script

>> This can take care of
>> the arch flag as there is no point building x86-specific tests for ARM and
>> Power (and some of them won't build even if you wanted to).
>>
>> B) Determine the flags, flavor, opts, etc. by parsing the output of `$test
>> --help`.
> 
> And this is what I was thinking. Executing every binary twice heavily slows 
> things down :(

C hello world is slower than string parsing in Python? Good for Python, I guess.

Just brainstorming, but another option might be to automatically generate
zdtm.list at build time or first run based on information in the C source or ELF.

Thanks,
Christopher Covington

-- 
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project