[CRIU] rewriting images via criu (or crit?)

[Ruslan Kuprieiev]

On 27.10.2014 19:28, Ruslan Kuprieiev wrote:
> On 27.10.2014 19:10, Tycho Andersen wrote:
>> On Mon, Oct 27, 2014 at 07:06:33PM +0200, Ruslan Kuprieiev wrote:
>>> On 27.10.2014 18:42, Tycho Andersen wrote:
>>>> On Mon, Oct 27, 2014 at 11:32:18AM -0500, Tycho Andersen wrote:
>>>>> On Mon, Oct 27, 2014 at 06:04:39PM +0200, Ruslan Kuprieiev wrote:
>>>>>> On 27.10.2014 17:32, Tycho Andersen wrote:
>>>>>>> On Mon, Oct 27, 2014 at 05:10:33PM +0200, Ruslan Kuprieiev wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I was recently thinking about converting/modifying images and i 
>>>>>>>> want to
>>>>>>>> share some of my thoughts.
>>>>>>>> For now I see 4 main ways:
>>>>>>>>
>>>>>>>> 1) Use existing patch set for crit written in python.
>>>>>>>> Pros:
>>>>>>>>      1. Already works.
>>>>>>>>      2. Python project is easy to support.
>>>>>>>>      3. Adding xpath-like language for modifying images should 
>>>>>>>> be easy in
>>>>>>>> Python.
>>>>>>>>      4. All needed dependencies are available in (any?) repo.
>>>>>>>> Cons:
>>>>>>>>      1. Performance(maybe we could use Cython to make it better?).
>>>>>>>>
>>>>>>>> 2) Rewrite crit in C
>>>>>>>> Pros:
>>>>>>>>      1. Performance.
>>>>>>>> Cons:
>>>>>>>>      1. C project is hard to support.
>>>>>>>>      3. Adding xpath-like language for modifying images is not 
>>>>>>>> as easy as in
>>>>>>>> Python.
>>>>>>>>      4. Requires protobuf-c-text, which is quite new and isn't 
>>>>>>>> available in
>>>>>>>> repos.
>>>>>>>>
>>>>>>>> 3) Add crit functionality to criu ("criu convert", "criu modify").
>>>>>>>> Pros:
>>>>>>>>      1. Performance
>>>>>>>>      2. Allows us to reuse a bunch of existing code.
>>>>>>>>      3. Having such tool shipped along with "criu show" looks 
>>>>>>>> like a nice
>>>>>>>> feature.
>>>>>>>>      4. We can easily teach criu how to use images in text 
>>>>>>>> format, which
>>>>>>>> seems
>>>>>>>>          quite handy.
>>>>>>>> Cons:
>>>>>>>>      1. criu is already complex enough and adding more stuff 
>>>>>>>> might be too
>>>>>>>> much,
>>>>>>>>          especially xpath-like language engine.
>>>>>>>>      2. Hard to support.
>>>>>>>>      3. Requires protobuf-c-text, which is quite new and isn't 
>>>>>>>> available in
>>>>>>>> repos.
>>>>>>>>      4. Requires heavy patching to existing code.
>>>>>>>>
>>>>>>>> 4) Use 1), but for _high-speed_ image modification use criu 
>>>>>>>> plugins and
>>>>>>>> hooks
>>>>>>>>      to modify images on restore.
>>>>>>>> Pros:
>>>>>>>>      1. Performance.
>>>>>>>>      2. No heavy patching.
>>>>>>>> Cons:
>>>>>>>>      Can't see any.
>>>>>>>>
>>>>>>>> I prefer 4) because it seems like a good compromise.
>>>>>>>> What do you guys think?
>>>>>>> Does that mean we'd need to expose all the protobuf headers? Will
>>>>>>> users then need to link against libprotobuf?
>>>>>> We would need to expose proto files with crit anyway(just the way 
>>>>>> we are
>>>>>> shipping rpc.proto), so exposing protobuf headers shouldn't be a 
>>>>>> problem.
>>>>> Why do we need to expose the proto files if we have the language to
>>>>> modify the images?
>>> Well, for example, for developer needs. So one can recompile proto 
>>> files
>>> using protoc.
>>>
>>>>>> Plugin indeed should be linked against libprotobuf-c. Is that a 
>>>>>> problem?
>>>>> It would take some amount of non-trivial work for lxc to link against
>>>>> libprotobuf-c (lxc is in the main ubuntu archive, so all of its
>>>>> dependencies must be too; the main archive has extra reviews and
>>>>> restrictions). Not to say it couldn't be done, but it would be 
>>>>> nice to
>>>>> avoid it.
>>> Why lxc even should link to libprotobuf-c? If you are able to call 
>>> tool that
>>> is
>>> linked to libprotobuf-c (criu, for example), then you should be able 
>>> to just
>>> pass
>>> plugin to criu.
>>>
>>>>> Another major benefit is that the proto files are essentially an
>>>>> internal API for criu. When we make changes to the proto files, we 
>>>>> can
>>>>> do that as long as we write some code that handles the old style
>>>>> images. If we make these "public", then we have to have some way to
>>>>> version them.
>>>>>
>>>>>> I mean, yes, plugin isn't very flexible tool, but it should be 
>>>>>> faster than
>>>>>> calling
>>>>>> a separate tool to modify image. For flexibility one could use 
>>>>>> crit written
>>>>>> in python.
>>>>> Isn't it mostly just saving you an extra fork()? Assuming the
>>>>> performance critical bits of crit were written in C, shouldn't it be
>>>>> essentially equivalent to doing it in criu?
>>>> Ah, I guess not. You only have to read the images once in this case.
>>>> That may be a significant enough difference to warrant the change, I'm
>>>> not sure.
>>> For plugin:
>>> 1) call criu with plugin
>>>      1. open img
>>>      2. modify img in RAM
>>>      3. restore
>>>
>>> For tool:
>>> 1) call tool to modify image
>>>      1. open img
>>>      2. parse xpath-like cmd
>>>      2. modfiy img
>>>      3. write img
>>> 2) call criu
>>>      1. open img
>>>      2. restore
>>>
>>> Not sure how slow dlopen is, but it looks like using plugin should be
>>> significantly faster.
>> The slowness comes not from dlopen, but from reading the images twice,
>> which could be very slow if the images are very large.
>>
>> What about putting this rewriting stuff into p.haul? p.haul has to
>> read the images anyway in order to move them across hosts, it may as
>> well do the editing of them in the process.
>
> Yes, that sounds great. All main crit features are stored inside pycriu
> module, so after we add xpath-like language to modify images, it could
> be used by p.haul.
> We  might consider adding some-kind of plugin to p.haul too, so one can
> write python script with commands to execute in p.haul. I mean, how cool
> is that, being able to write plugin is python =). Sounds great to me.
> And it will be faster than parsing xpath-like command, yet not so 
> flexible.
> Supporting both xpath-like and python plugin sounds good.
> What do you think about it?
>

Btw, in p.haul we would need to read image from tmpfs anyway.
Using criu plugin should be faster. We might come back to it
if p.haul will be not fast enough for you.

>>> I'm assuming you need to modify images of migrating process, right?
>>> So you just need to fix some values in image as fast as possible, 
>>> without
>>> saving the image somewhere on disk, right?
>> Right, although optionally being able to save things to disk would be
>> handy too.
>
> But after restore, right? =)
> We can achieve that with p.haul.
>
>>> And other people who use criu, for example, for testing, could use
>>> "crit modify" to modify image, save it somewhere and restore a lot
>>> of times.
>>>
>>>> Tycho
>