[Devel] Re: [RFC v3][PATCH 1/9] Create syscalls: sys_checkpoint, sys_restart

[Oren Laadan]

Serge E. Hallyn wrote:
> Quoting Oren Laadan (orenl at cs.columbia.edu):
>>
>> Serge E. Hallyn wrote:
>>> Quoting Oren Laadan (orenl at cs.columbia.edu):
>>>> Create trivial sys_checkpoint and sys_restore system calls. They will
>>>> enable to checkpoint and restart an entire container, to and from a
>>>> checkpoint image file descriptor.
>>>>
>>>> The syscalls take a file descriptor (for the image file) and flags as
>>>> arguments. For sys_checkpoint the first argument identifies the target
>>>> container; for sys_restart it will identify the checkpoint image.
>>>>
>>>> Signed-off-by: Oren Laadan <orenl at cs.columbia.edu>
>>>> ---
>> [...]
>>
>>>> +/**
>>>> + * sys_checkpoint - checkpoint a container
>>>> + * @pid: pid of the container init(1) process
>>>> + * @fd: file to which dump the checkpoint image
>>>> + * @flags: checkpoint operation flags
>>>> + */
>>>> +asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags)
>>>> +{
>>>> +	pr_debug("sys_checkpoint not implemented yet\n");
>>>> +	return -ENOSYS;
>>>> +}
>>>> +/**
>>>> + * sys_restart - restart a container
>>>> + * @crid: checkpoint image identifier
>>> So can we compare your api to Andrey's?
>>>
>>> You've explained before that crid is used to tie together multiple
>>> calls to checkpoint, but why do you have to specify it for restart?
>>> Can't it just come from the fd?  Or, the fd will be passed in
>>> seek()d to the right position for the data for this task, so the crid
>>> won't be available there?
>> I added the 'crid' inside to support a mode of operation in which we
>> would like the checkpoint data to remain in memory across multiple
>> system calls. Here are example scenarios:
>>
>> 1) We will want to reduce down time by first buffering the checkpoint
>> image in memory, then resuming the container, and only then writing
>> the data back to a (the) file descriptor.
>> So instead of:
>>   freeze -> checkpoint and write back -> unfreeze
>> We want:
>>   freeze -> checkpoint to buffer -> unfreeze -> write back
>> I envision each of these steps to be a separate invocation of a syscall.
>> to the 'crid' returned by the sys_checkpoint() at the 2nd step, will be
>> used to identify that data in the 4th step. (Note, that between the
>> unfreeze and the write-back, another checkpoint may be already taken).
>>
>> 2) A task may want to take a checkpoint (e.g. of itself, or a whole
>> container) and keep that checkpoint in memory; at a later time it may
>> want to revert to that checkpoint. Moreover, it may keep multiple such
>> checkpoints (to where it may want to return). 'crid' tells sys_restart
>> which one to use.
>>
>> Note that this 'crid' will in fact be tied to resources that are kept
>> by the kernel - e.g. references to COW pages (when we add that).
>> Louis suggested to use a specialized FD instead of a numeric 'crid'
>> (that is: create a anonymous inode and a struct file that represent
>> that checkpoint in the kernel, and return an FD to it). This approach
>> has pros and cons of 'crid' (see the archives of the containers
>> mailing list). For now I kept 'crid', but I'm definitely open to change
>> it to a FD.
>>
>> Oren.
> 
> Oh, so the crid identifies one checkpoint inside the file - the single
> file can store multiple checkpoints?

Not quite. Let me rephrase the motivation first:

There are occasions when we would like to keep the checkpoint data in the
kernel for some (relatively long) time, between syscalls. By "checkpoint
data" I mean references to memory contents (pages) and all the other data.

The two scenarios above are two examples: between the syscall to checkpoint
and the syscall to unfreeze and then write-back the data to a file (first
example), and for some time until a task may want to "go back in time"
(second example, useful for ultra fast "undo" for a task).

Note that in both cases when I say "keep in kernel" I mean before it is
written to a file, or to the network. Simply in memory, in some efficient
manner.

Subsequent syscalls will need to refer to a specific checkpoint data that
is kept in memory - e.g. to write-back to a file-descriptor, or to clean
up, or to restart from it. (At any single time a specific container may
have multiple checkpoints associated with it - eg. because they have not
yet been written back to storage but already taken).

Once the data is written back to a file descriptor, the in-kernel data can
be discarded and cleaned-up.

The main reason why I want to keep the data in the kernel and not instead
copy to user space, is efficiency: most of the checkpoint data is the memory
footprint; by keeping the data in the kernel, one can merely keep a COW
reference instead of a whole copy of everything (save space and copy time).

So, if we have keep data in kernel between syscalls, then we must have a
way to refer to it. The current implementation uses a very simple 'crid'
value to do that - although, clearly, at the moment it isn't used.

I hope this explains better.

Oren.

> 
>>> Andrey, how will the 'ctid' in your patchset be used?  It sounds
>>> like it's actually going to set some integer id on the created
>>> container?  We actually don't have container ids (or even
>>> containers) right now, so we probably don't want that in our api,
>>> right?
> 
> -serge
_______________________________________________
Containers mailing list
Containers at lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/containers