[CRIU] rewriting images via criu (or crit?)

[Ruslan Kuprieiev]

On 27.10.2014 19:10, Tycho Andersen wrote:
> On Mon, Oct 27, 2014 at 07:06:33PM +0200, Ruslan Kuprieiev wrote:
>> On 27.10.2014 18:42, Tycho Andersen wrote:
>>> On Mon, Oct 27, 2014 at 11:32:18AM -0500, Tycho Andersen wrote:
>>>> On Mon, Oct 27, 2014 at 06:04:39PM +0200, Ruslan Kuprieiev wrote:
>>>>> On 27.10.2014 17:32, Tycho Andersen wrote:
>>>>>> On Mon, Oct 27, 2014 at 05:10:33PM +0200, Ruslan Kuprieiev wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I was recently thinking about converting/modifying images and i want to
>>>>>>> share some of my thoughts.
>>>>>>> For now I see 4 main ways:
>>>>>>>
>>>>>>> 1) Use existing patch set for crit written in python.
>>>>>>> Pros:
>>>>>>>      1. Already works.
>>>>>>>      2. Python project is easy to support.
>>>>>>>      3. Adding xpath-like language for modifying images should be easy in
>>>>>>> Python.
>>>>>>>      4. All needed dependencies are available in (any?) repo.
>>>>>>> Cons:
>>>>>>>      1. Performance(maybe we could use Cython to make it better?).
>>>>>>>
>>>>>>> 2) Rewrite crit in C
>>>>>>> Pros:
>>>>>>>      1. Performance.
>>>>>>> Cons:
>>>>>>>      1. C project is hard to support.
>>>>>>>      3. Adding xpath-like language for modifying images is not as easy as in
>>>>>>> Python.
>>>>>>>      4. Requires protobuf-c-text, which is quite new and isn't available in
>>>>>>> repos.
>>>>>>>
>>>>>>> 3) Add crit functionality to criu ("criu convert", "criu modify").
>>>>>>> Pros:
>>>>>>>      1. Performance
>>>>>>>      2. Allows us to reuse a bunch of existing code.
>>>>>>>      3. Having such tool shipped along with "criu show" looks like a nice
>>>>>>> feature.
>>>>>>>      4. We can easily teach criu how to use images in text format, which
>>>>>>> seems
>>>>>>>          quite handy.
>>>>>>> Cons:
>>>>>>>      1. criu is already complex enough and adding more stuff might be too
>>>>>>> much,
>>>>>>>          especially xpath-like language engine.
>>>>>>>      2. Hard to support.
>>>>>>>      3. Requires protobuf-c-text, which is quite new and isn't available in
>>>>>>> repos.
>>>>>>>      4. Requires heavy patching to existing code.
>>>>>>>
>>>>>>> 4) Use 1), but for _high-speed_ image modification use criu plugins and
>>>>>>> hooks
>>>>>>>      to modify images on restore.
>>>>>>> Pros:
>>>>>>>      1. Performance.
>>>>>>>      2. No heavy patching.
>>>>>>> Cons:
>>>>>>>      Can't see any.
>>>>>>>
>>>>>>> I prefer 4) because it seems like a good compromise.
>>>>>>> What do you guys think?
>>>>>> Does that mean we'd need to expose all the protobuf headers? Will
>>>>>> users then need to link against libprotobuf?
>>>>> We would need to expose proto files with crit anyway(just the way we are
>>>>> shipping rpc.proto), so exposing protobuf headers shouldn't be a problem.
>>>> Why do we need to expose the proto files if we have the language to
>>>> modify the images?
>> Well, for example, for developer needs. So one can recompile proto files
>> using protoc.
>>
>>>>> Plugin indeed should be linked against libprotobuf-c. Is that a problem?
>>>> It would take some amount of non-trivial work for lxc to link against
>>>> libprotobuf-c (lxc is in the main ubuntu archive, so all of its
>>>> dependencies must be too; the main archive has extra reviews and
>>>> restrictions). Not to say it couldn't be done, but it would be nice to
>>>> avoid it.
>> Why lxc even should link to libprotobuf-c? If you are able to call tool that
>> is
>> linked to libprotobuf-c (criu, for example), then you should be able to just
>> pass
>> plugin to criu.
>>
>>>> Another major benefit is that the proto files are essentially an
>>>> internal API for criu. When we make changes to the proto files, we can
>>>> do that as long as we write some code that handles the old style
>>>> images. If we make these "public", then we have to have some way to
>>>> version them.
>>>>
>>>>> I mean, yes, plugin isn't very flexible tool, but it should be faster than
>>>>> calling
>>>>> a separate tool to modify image. For flexibility one could use crit written
>>>>> in python.
>>>> Isn't it mostly just saving you an extra fork()? Assuming the
>>>> performance critical bits of crit were written in C, shouldn't it be
>>>> essentially equivalent to doing it in criu?
>>> Ah, I guess not. You only have to read the images once in this case.
>>> That may be a significant enough difference to warrant the change, I'm
>>> not sure.
>> For plugin:
>> 1) call criu with plugin
>>      1. open img
>>      2. modify img in RAM
>>      3. restore
>>
>> For tool:
>> 1) call tool to modify image
>>      1. open img
>>      2. parse xpath-like cmd
>>      2. modfiy img
>>      3. write img
>> 2) call criu
>>      1. open img
>>      2. restore
>>
>> Not sure how slow dlopen is, but it looks like using plugin should be
>> significantly faster.
> The slowness comes not from dlopen, but from reading the images twice,
> which could be very slow if the images are very large.
>
> What about putting this rewriting stuff into p.haul? p.haul has to
> read the images anyway in order to move them across hosts, it may as
> well do the editing of them in the process.

Yes, that sounds great. All main crit features are stored inside pycriu
module, so after we add xpath-like language to modify images, it could
be used by p.haul.
We  might consider adding some-kind of plugin to p.haul too, so one can
write python script with commands to execute in p.haul. I mean, how cool
is that, being able to write plugin is python =). Sounds great to me.
And it will be faster than parsing xpath-like command, yet not so flexible.
Supporting both xpath-like and python plugin sounds good.
What do you think about it?

>> I'm assuming you need to modify images of migrating process, right?
>> So you just need to fix some values in image as fast as possible, without
>> saving the image somewhere on disk, right?
> Right, although optionally being able to save things to disk would be
> handy too.

But after restore, right? =)
We can achieve that with p.haul.

>> And other people who use criu, for example, for testing, could use
>> "crit modify" to modify image, save it somewhere and restore a lot
>> of times.
>>
>>> Tycho