<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">чт, 18 июн. 2020 г. в 19:46, Felix Kuehling <<a href="mailto:felix.kuehling@gmail.com">felix.kuehling@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><br>
</p>
<div>Am 2020-06-16 um 4:58 a.m. schrieb
Pavel Emelyanov:<br>
</div>
<blockquote type="cite">
<div dir="auto">
<div>Hi, Felix
<div dir="auto"><br>
</div>
<div dir="auto">Please, see my comments inline.</div>
</div>
</div>
</blockquote>
<p>Thank you for your reply. I'm responding inline ...</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="auto">
<div><br>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">вт, 16 июн. 2020 г., 4:04
Felix Kuehling <<a href="mailto:felix.kuehling@gmail.com" target="_blank">felix.kuehling@gmail.com</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="x-unicode">
<p>Hi all,</p>
<p>I'm investigating the possibility of making CRIU
work with ROCm, the AMD Radeon Open Compute
Platform. I need some advice, but I'll give you some
background first.</p>
<p>ROCm uses the /dev/kfd device as well as
/dev/dri/renderD*. I'm planning to do most of the
state saving using /dev/kfd with a
cr_plugin_dump_file callback in a plugin. I've spent
some time reading documentation on <a href="http://criu.org" rel="noreferrer" target="_blank">criu.org</a>
and also CRIU source code. At this point I believe I
have a fairly good understanding of the low level
details of saving kernel mode state associated with
ROCm processes.</p>
<p>I have more trouble with restoring the state. The
main issue is the way KFD maps system memory for
device access using HMM (or get_user pages and MMU
notifiers with DKMS on older kernels). This requires
the VMAs to be at the expected virtual addresses
before we try to mirror them into the GPU page
table.</p>
</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">What if the system memory of this area is shared
between several processes? And mapped in all of them in
different virtual address. Presumably the requirement is just
to have them mapped at correct virtual address of the caller
process?</div>
</div>
</blockquote>
<p>Yes. The GPUs support per-process virtual address spaces and each
process mirrors its own virtual memory mappings to its own GPU
virtual address space. Whether or not the mappings share the same
physical pages should not matter.</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="x-unicode">
<p> Resuming execution on the GPU also needs to be
delayed until after the GPU memory mappings have
been restored.<br>
</p>
<p>At the time of the cr_plugin_restore_file callback,
the VMAs are not at the right place in the restored
process, so this is too early to restore the GPU
memory mappings.</p>
</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto">True. At this point some vmas are in so called
premapped area and some don't yet exist.</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="x-unicode">
<p>I can send the mappings and their properties to KFD
but KFD needs to wait for some later trigger event
before it activates the mappings and their MMU
notifiers.</p>
<p>So this is my question: What would be a good
trigger event to indicate that VMAs have been moved
to their proper location by the restorer parasite
code?</p>
</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto">We have "restore stages" that are used to
synchronize all the processes at specific points. The last 3
refer to places where all the memory is in needed virtual
address. If you wait for a stage to complete and don't start
the next one, then you can safely execute whatever is needed
with all tasks' memory being mapped at final virtual address.</div>
</div>
</blockquote>
<p>I see. I'll need some more time to understand all the stages and
the details of the synchronization mechanism. But I've already
seen that there is synchronization in the restorer PIE code. So
that's very encouraging.</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="auto">
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="x-unicode">
<p> I have considered two possibilities that will not
work. I'm hoping you can give me some better ideas:</p>
<ul>
<li>cr_plugin_fini</li>
<ul>
<li>Doesn't get called in all the child processes,
not sure if there is synchronization with the
child processes' restore completion<br>
</li>
</ul>
</ul>
</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto">Yes, it's called in the master process after all
stages completed. It's a cleanup hook.</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="x-unicode">
<ul>
<ul>
<li> <br>
</li>
</ul>
<li>An MMU notifier on the munmap of the restorer
parasite blob itself</li>
<ul>
<li>In cr_plugin_restore_file this address is not
known yet</li>
</ul>
</ul>
</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto">Can the restoring code run in criu main process
instead of one of the child ones? If yes, this could make
things simpler. You can add yet another plugin invocations
near apply_memfd_seal, this is the place where all child
processes are stopped with their vmas properly restored. But
again, this place runs in the context of criu master process ,
not children.</div>
</div>
</blockquote>
Yes, I think I can make that work. I'll need to identify the target
process to resume in an ioctl call. To establish authentication to
allow such operations, I'd use ptrace attached status, so that only
a ptrace-attached parent process is allowed to save/restore state of
a child process. Is the master process ptrace-attached to the child
processes at the apply_memfd_seal stage?<br></div></blockquote><div><br></div><div>No, but that time no. On restore the ptrace takes part only close to the very-very end. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>Am I understanding you correctly, you're suggesting a change to
CRIU itself to add a new plugin callback to enable my use case?</p></div></blockquote><div>If it's properly justified and sanely coded, then yes.<br></div><div><br></div><div>-- Pavel</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>Thank you,<br>
Felix</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="auto">
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="x-unicode">
<ul>
<ul>
</ul>
</ul>
<p>I noticed that the child processes are resumed
through sigreturn. I'm not familiar with this
mechanism. Does this mean there is some signal I may
be able to intercept just before execution of the
child process resumes?</p>
</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto">No, sigreturn is just the mechanics we use to
restore tasks registers :)</div>
<div dir="auto"><br>
</div>
<div dir="auto">-- Pavel</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div lang="x-unicode">
<p>Thank you in advance for your insights.</p>
<p>Best regards,<br>
Felix</p>
<p><br>
</p>
</div>
</div>
_______________________________________________<br>
CRIU mailing list<br>
<a href="mailto:CRIU@openvz.org" rel="noreferrer" target="_blank">CRIU@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/criu" rel="noreferrer noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/criu</a><br>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div></div>