<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Hi all,</p>
<p>I'm investigating the possibility of making CRIU work with ROCm,
the AMD Radeon Open Compute Platform. I need some advice, but I'll
give you some background first.</p>
<p>ROCm uses the /dev/kfd device as well as /dev/dri/renderD*. I'm
planning to do most of the state saving using /dev/kfd with a
cr_plugin_dump_file callback in a plugin. I've spent some time
reading documentation on criu.org and also CRIU source code. At
this point I believe I have a fairly good understanding of the low
level details of saving kernel mode state associated with ROCm
processes.</p>
<p>I have more trouble with restoring the state. The main issue is
the way KFD maps system memory for device access using HMM (or
get_user pages and MMU notifiers with DKMS on older kernels). This
requires the VMAs to be at the expected virtual addresses before
we try to mirror them into the GPU page table. Resuming execution
on the GPU also needs to be delayed until after the GPU memory
mappings have been restored.<br>
</p>
<p>At the time of the cr_plugin_restore_file callback, the VMAs are
not at the right place in the restored process, so this is too
early to restore the GPU memory mappings. I can send the mappings
and their properties to KFD but KFD needs to wait for some later
trigger event before it activates the mappings and their MMU
notifiers.</p>
<p>So this is my question: What would be a good trigger event to
indicate that VMAs have been moved to their proper location by the
restorer parasite code? I have considered two possibilities that
will not work. I'm hoping you can give me some better ideas:</p>
<ul>
<li>cr_plugin_fini</li>
<ul>
<li>Doesn't get called in all the child processes, not sure if
there is synchronization with the child processes' restore
completion<br>
</li>
</ul>
<li>An MMU notifier on the munmap of the restorer parasite blob
itself</li>
<ul>
<li>In cr_plugin_restore_file this address is not known yet</li>
</ul>
</ul>
<p>I noticed that the child processes are resumed through sigreturn.
I'm not familiar with this mechanism. Does this mean there is some
signal I may be able to intercept just before execution of the
child process resumes?</p>
<p>Thank you in advance for your insights.</p>
<p>Best regards,<br>
Felix</p>
<p><br>
</p>
</body>
</html>