<div dir="ltr">Thank you for the follow-up. It seems the requirements for live migration have been tightened considerably. <br><br>We used to live migrate between different architectures, but now both sides of a live migration must be exactly identical. Almost makes one want to go back to the openvz 6 days.<br><br>I'll work on finding an exact clone of my primary OVZ server and try again.<br><br>Thanks for the prompt response.<div><br></div><div>Jake</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jun 16, 2019 at 4:38 AM Konstantin Khorenko <<a href="mailto:khorenko@virtuozzo.com">khorenko@virtuozzo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 06/15/2019 08:21 PM, jjs - mainphrame wrote:<br>
> Greetings -<br>
><br>
> Live migration, which worked beautifully with openvz 10 years ago, has stopped working in the current openvz 7 environment.<br>
><br>
> When I first built ovz 7 servers a few years ago, live migration worked as it should. Within the past few months it stopped working. Thinking it might be a problem with tighter<br>
> requirements on CPU matching, I repurposed the amd based ovz host and replaced it, so that both ovz hosts would be running on intel hardware.<br>
><br>
> Unfortunately that did not change the issue. Note below - both hosts running up to date ovz-7 non-factory.<br>
><br>
> The command -<br>
> [root@annie ~]# vzmigrate --online --nodeps hachi 1987<br>
> Connecttion to destination node (hachi) is successfully established<br>
> Moving/copying CT 1987 -> CT 1987, [], [] ...<br>
> locking 1987<br>
> Checking bindmounts<br>
> Check cluster ID<br>
> Checking keep dir for private area copy<br>
> Checking technologies<br>
> Checking IP addresses on destination node<br>
> Checking RATE parameters in config<br>
> Checking ploop format 2<br>
> copy CT private /vz/private/1987<br>
> Live migration stage started<br>
> Compression is enabled<br>
> Phaul service failed to live migrate CT<br>
> Phaul failed to live migrate CT (/var/log/phaul.log)<br>
> Can't move/copy CT 1987 -> CT 1987, [], [] : Phaul failed to live migrate CT (/var/log/phaul.log)<br>
> unlocking 1987<br>
> [root@annie ~]#<br>
><br>
> Contents of phaul.log -<br>
> --<br>
> 10:01:16.569: 17149:<br>
> 10:01:16.569: 17149:<br>
> 10:01:16.570: 17149:<br>
> 10:01:16.570: 17149: Starting p.haul<br>
> 10:01:16.570: 17149: Use existing connections, fdrpc=11 fdmem=13 fdfs=root.hdd/root.hds:15<br>
> 10:01:16.589: 17149: Setting up local<br>
> 10:01:16.589: 17149: Loading config file from /etc/vz/conf/<br>
> 10:01:16.590: 17149: Initialize ploop hauler<br>
> 10:01:16.590: 17149: `- /vz/private/1987/root.hdd/root.hds<br>
> 10:01:16.616: 17149: Passing (ctl:12, data:10) pair to CRIU<br>
> 10:01:16.616: 17149: Set maximum number of open file descriptors to 1048576<br>
> 10:01:16.618: 17149: Setting up remote<br>
> 10:01:16.704: 17149: Start migration in live mode<br>
> 10:01:16.704: 17149: Checking criu version<br>
> 10:01:16.757: 17149: Checking for Dirty Tracking<br>
> 10:01:16.758: 17149: `- Explicitly enabled<br>
> 10:01:16.758: 17149: Preliminary FS migration<br>
> 10:01:34.968: 17149: Fs driver transfer 1327497216 bytes (~1266Mb)<br>
> 10:01:34.968: 17149: * Iteration 0<br>
> 10:01:35.074: 17149: Making directory /vz/dump/1987/dmp-VTSUxn-19.06.15-10.01/img/1<br>
> 10:01:35.075: 17149: Issuing pre-dump command to service<br>
> 10:01:36.120: 17149: Dumped 28561 pages, 0 skipped<br>
> 10:01:36.120: 17149: Fs driver transfer 0 bytes<br>
> 10:01:36.120: 17149: Checking iteration progress:<br>
> 10:01:36.120: 17149: > Proceed to next iteration<br>
> 10:01:36.120: 17149: * Iteration 1<br>
> 10:01:36.122: 17149: Making directory /vz/dump/1987/dmp-VTSUxn-19.06.15-10.01/img/2<br>
> 10:01:36.122: 17149: Issuing pre-dump command to service<br>
> 10:01:37.751: 17149: Dumped 360 pages, 28201 skipped<br>
> 10:01:37.751: 17149: Fs driver transfer 0 bytes<br>
> 10:01:37.751: 17149: Checking iteration progress:<br>
> 10:01:37.751: 17149: > Proceed to next iteration<br>
> 10:01:37.751: 17149: * Iteration 2<br>
> 10:01:37.754: 17149: Making directory /vz/dump/1987/dmp-VTSUxn-19.06.15-10.01/img/3<br>
> 10:01:37.754: 17149: Issuing pre-dump command to service<br>
> 10:01:38.485: 17149: Dumped 361 pages, 28200 skipped<br>
> 10:01:38.485: 17149: Fs driver transfer 0 bytes<br>
> 10:01:38.485: 17149: Checking iteration progress:<br>
> 10:01:38.485: 17149: > Too many iterations<br>
> 10:01:38.485: 17149: Final dump and restore<br>
> 10:01:38.487: 17149: Making directory /vz/dump/1987/dmp-VTSUxn-19.06.15-10.01/img/4<br>
> 10:01:38.545: 17149: Issuing dump command to service<br>
> 10:01:38.547: 17149: Notify (pre-dump)<br>
> 10:01:38.555: 17149: Notify (network-lock)<br>
> 10:01:38.579: 17149: Action script /usr/libexec/criu/scripts/nfs-ports-allow.sh finished with exit code 0<br>
> 10:01:38.580: 17149: Notify (post-network-lock)<br>
> 10:01:41.047: 17149: Final FS and images sync<br>
> 10:01:41.441: 17149: Sending images to target<br>
> 10:01:41.442: 17149: Pack<br>
> 10:01:41.493: 17149: Add htype images<br>
> 10:01:41.722: 17149: Asking target host to restore<br>
> 10:01:42.635: 17149: Remote exception<br>
> 10:01:42.636: 17149: I/O operation on closed file<br>
> Traceback (most recent call last):<br>
> File "/usr/libexec/phaul/p.haul", line 9, in <module><br>
> load_entry_point('phaul==0.1', 'console_scripts', 'p.haul')()<br>
> File "/usr/lib/python2.7/site-packages/phaul/shell/phaul_client.py", line 49, in main<br>
> worker.start_migration()<br>
> File "/usr/lib/python2.7/site-packages/phaul/iters.py", line 161, in start_migration<br>
> self.__start_live_migration()<br>
> File "/usr/lib/python2.7/site-packages/phaul/iters.py", line 232, in __start_live_migration<br>
> self.target_host.restore_from_images()<br>
> File "/usr/lib/python2.7/site-packages/phaul/xem_rpc_client.py", line 26, in __call__<br>
> raise Exception(resp[1])<br>
> Exception: I/O operation on closed file<br>
> --<br>
><br>
> Dump directory contents are also available -<br>
><br>
> Shall I open a bug report?<br>
<br>
Yes, file a bug please.<br>
<br>
--<br>
Best regards,<br>
<br>
Konstantin Khorenko,<br>
Virtuozzo Linux Kernel Team<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.openvz.org/mailman/listinfo/users</a><br>
</blockquote></div>