<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">It looks like the issue might be with the destination server, CRIU works fine on the source server but I get this error when trying to suspend and restore on the destination server:</div><div dir="ltr"><br></div><div dir="ltr"><div dir="ltr">(00.134699) Error (criu/namespaces.c:481): Can't dump nested user namespace for 5051</div><div dir="ltr">(00.134708) Error (criu/namespaces.c:836): Can't make userns id</div><div dir="ltr">(00.140699) Error (criu/util.c:684): exited, status=1</div><div dir="ltr">(00.142811) Error (criu/util.c:684): exited, status=1</div><div dir="ltr">(00.144311) Error (criu/cr-dump.c:2275): Dumping FAILED.</div><div dir="ltr">Failed to checkpoint the Container</div><div dir="ltr">All dump files and logs were saved to /vz/private/10597/dump/Dump.fail</div><div dir="ltr">Checkpointing failed</div><div dir="ltr">Container is already running</div><div dir="ltr"><br></div><div>Also confirmed that this command works on both servers: "python /usr/lib/python2.7/site-packages/phaul/shell/phaul_client.py --help"</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 22, 2021 at 6:40 AM Pavel Vokhmyanin <<a href="mailto:pvokhmyanin@virtuozzo.com">pvokhmyanin@virtuozzo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="RU">
<div class="gmail-m_4984974556797412031WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I see, versions are correct, no problems there.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">It might be a bit awkward to debug this over the mail further, but lets try couple basic tests.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Can you tell me, does “vzctl suspend %CTID%; vzctl resume %CTID%” succeed? This should test whether CRIU has all dependencies
and is operational.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Besides, is there an exception if you run “python /usr/lib/python2.7/site-packages/phaul/shell/phaul_client.py --help”
?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Right now we know that phaul-client doesn’t start on your source server, but can’t tell exactly what the problem is.
Hopefully these tests will give us more clues.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)">Pavel Vokhmyanin</span></b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(89,89,89)">Software Developer, Virtuozzo R&D</span><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)"> </span><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)">Otradnaya street 2b/9, “Otradnoe” Techno Park | Moscow | Russia</span><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)">Phone: +7 (495) 139 80 17, ext 77449 |
<a href="mailto:pvokhmyanin@virtuozzo.com" target="_blank"><span style="color:rgb(5,99,193)">pvokhmyanin@virtuozzo.com</span></a></span><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)">Skype: pvokhmyanin</span><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)"><a href="https://virtuozzo.com/" target="_blank"><span style="color:rgb(5,99,193)">Virtuozzo.com</span></a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"> <a href="mailto:users-bounces@openvz.org" target="_blank">users-bounces@openvz.org</a> [mailto:<a href="mailto:users-bounces@openvz.org" target="_blank">users-bounces@openvz.org</a>]
<b>On Behalf Of </b>Joe Dougherty<br>
<b>Sent:</b> Monday, March 22, 2021 12:56 PM<br>
<b>To:</b> OpenVZ users<br>
<b>Subject:</b> Re: [Users] Unable to perform migration between two fairly new OpenVZ 7 nodes<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Source server:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"># rpm -q vzmigrate phaul<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">vzmigrate-7.0.138-1.vz7.x86_64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">phaul-0.1.76-1.vz7.noarch<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Destination server:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"># rpm -q vzmigrate phaul<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">vzmigrate-7.0.138-1.vz7.x86_64<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">phaul-0.1.76-1.vz7.noarch<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Mon, Mar 22, 2021 at 5:50 AM Pavel Vokhmyanin <<a href="mailto:pvokhmyanin@virtuozzo.com" target="_blank">pvokhmyanin@virtuozzo.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hello Joe,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">These symptoms indicate that phaul failed to start.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Recently there were changes in arguments for phaul. If you have an old vzmigrate and
new phaul or vice-versa, you could get this behavior.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">You should have either phaul <=0.1.78 + vzmigrate <=7.0.140 OR phaul 0.1.79 + vzmgirate
>=7.0.142.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Can you elaborate what package versions you’re using on the source server?</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"># rpm –q vzmigrate phaul</span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)"> </span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)">Pavel Vokhmyanin</span></b><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(89,89,89)">Software Developer, Virtuozzo R&D</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)">Otradnaya street 2b/9, “Otradnoe” Techno Park | Moscow | Russia</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)">Phone: +7 (495) 139 80 17, ext 77449 |
<a href="mailto:pvokhmyanin@virtuozzo.com" target="_blank"><span style="color:rgb(5,99,193)">pvokhmyanin@virtuozzo.com</span></a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)">Skype: pvokhmyanin</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9pt;font-family:Calibri,sans-serif;color:rgb(64,64,64)"><a href="https://virtuozzo.com/" target="_blank"><span style="color:rgb(5,99,193)">Virtuozzo.com</span></a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">
<a href="mailto:users-bounces@openvz.org" target="_blank">users-bounces@openvz.org</a> [mailto:<a href="mailto:users-bounces@openvz.org" target="_blank">users-bounces@openvz.org</a>]
<b>On Behalf Of </b>Joe Dougherty<br>
<b>Sent:</b> Monday, March 22, 2021 2:26 AM<br>
<b>To:</b> OpenVZ users<br>
<b>Subject:</b> [Users] Unable to perform migration between two fairly new OpenVZ 7 nodes</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">I'm attempting to perform migrations between two nodes, one build about a month ago and the other build this morning. I can migrate the containers if I stop them first, but attempts
to migrate them while powered on (both using --live and without) fail due to phaul-service.<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The command I'm running:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">vzmigrate -vvv --keep-dst --ssh="-p 2200 -i /root/.ssh/key" server2 10597<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Here's the tail end of the output where it fails:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">2021-03-21 19:13:08.979: Warm migration stage started<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:08.979: Compression is enabled<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.063: Io multiplexer peer aborted<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.063: 2021-03-21 16:13:10.778: Phaul service failed to live migrate CT<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.064: 2021-03-21 16:13:10.778: cmd 'runphaulmigr' error [-73] : Phaul service failed to live migrate CT<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.064: Phaul service failed to live migrate CT<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.064: Phaul failed to live migrate CT (/var/log/phaul.log)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.064: 2021-03-21 16:13:10.779: cleaning : rename : /vz/private/10597 -> /vz/private/10597.migrated<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.064: 2021-03-21 16:13:10.779: cleaning : destroy CT 10597<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.073: 2021-03-21 16:13:10.788: cleaning : 'rmdir' dir : /vz/root/10597<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.074: 2021-03-21 16:13:10.788: can not find entry for delete : [/vz/root/10597]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.074: 2021-03-21 16:13:10.788: cleaning : rename : /vz/private/10597 -> /vz/private/10597.migrated<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.074: 2021-03-21 16:13:10.788: can not move '/vz/private/10597' -> '/vz/private/10597.migrated' : No such file or directory<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.074: 2021-03-21 16:13:10.788: Can't do correct cleaning: can not move '/vz/private/10597' -> '/vz/private/10597.migrated' : No such file or directory<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.074: 2021-03-21 16:13:10.788: unlocking 10597<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.075: Can't move/copy CT 10597 -> CT 10597, [], [] : Phaul failed to live migrate CT (/var/log/phaul.log)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.075: cleaning : 'rm' file : /vz/dump/10597-criu_err.log<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:09.075: can not find entry for delete : [/vz/dump/10597-criu_err.log]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:10.075: unlocking 10597<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2021-03-21 19:13:10.075: close channel<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">There is no log file.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"># cat /var/log/phaul.log<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">cat: /var/log/phaul.log: No such file or directory<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Any ideas on how I can fix this so I can begin performing migrations without having to power off the containers first?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thank you.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-Joe<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal"><b><i><span style="font-family:Verdana,sans-serif">-Joe Dougherty</span></i></b><u></u><u></u></p>
<div>
<p class="MsoNormal"><b><i><span style="font-family:Verdana,sans-serif">Chief Operating Officer</span></i></b><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><b><i><span style="font-family:Verdana,sans-serif">Secure Dragon LLC </span></i></b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><b><i><a href="http://www.SecureDragon.net" target="_blank"><span style="font-family:Verdana,sans-serif">www.SecureDragon.net</span></a></i></b><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><b><i><font face="verdana, sans-serif">-Joe<br></font></i></b><div></div></div></div></div></div>