<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hello</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">If it can help, what I did so far to
try to re-enable dead CTs <br>
</div>
<div class="moz-cite-prefix"><br>
</div>
# prlctl stop ldap2<br>
<div class="moz-cite-prefix">Stopping the CT...<br>
Failed to stop the CT: PRL_ERR_VZCTL_OPERATION_FAILED (Details:
Cannot lock the Container<br>
)<br>
# cat /vz/lock/144dc737-b4e3-4c03-852c-25a6df06cee4.lck <br>
6227<br>
resuming<br>
# ps auwx | grep 6227<br>
root 6227 0.0 0.0 92140 6984 ? S 15:10 0:00
/usr/sbin/vzctl resume 144dc737-b4e3-4c03-852c-25a6df06cee4<br>
# kill -9 6227<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">still cannot stop the CT (Cannot lock
the Container...) <br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"># df |grep
144dc737-b4e3-4c03-852c-25a6df06cee4<br>
/dev/ploop11432p1 10188052 2546636 7100848 27%
/vz/root/144dc737-b4e3-4c03-852c-25a6df06cee4<br>
none 1048576 0 1048576 0%
/vz/private/144dc737-b4e3-4c03-852c-25a6df06cee4/dump/Dump/.criu.cgyard.56I2ls<br>
# umount /dev/ploop11432p1<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"># ploop check -F
/vz/private/144dc737-b4e3-4c03-852c-25a6df06cee4/root.hdd/root.hds
<br>
Reopen rw
/vz/private/144dc737-b4e3-4c03-852c-25a6df06cee4/root.hdd/root.hds<br>
Error in ploop_check (check.c:663): Dirty flag is set<br>
<br>
</div>
<div class="moz-cite-prefix"># ploop mount
/vz/private/144dc737-b4e3-4c03-852c-25a6df06cee4/root.hdd/DiskDescriptor.xml<br>
Error in ploop_mount_image (ploop.c:2495): Image
/vz/private/144dc737-b4e3-4c03-852c-25a6df06cee4/root.hdd/root.hds
already used by device /dev/ploop11432</div>
<div class="moz-cite-prefix"># df -H | grep ploop11432<br>
=> nothing <br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">I am lost , any help appreciated .</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Thanks . <br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Le 06/07/2020 à 15:37, Jehan Procaccia
IMT a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:954ec91b-fba0-ef55-d876-1838715ff5f7@imtbs-tsp.eu">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Hello,</p>
<p>I am back to the initial pb related to that post , since I
updated to <em>OpenVZ release 7.0.14 (136) | </em><em><i>Virtuozzo
Linux release 7.8.0 (609)</i></em> , I am also facing CT
corrupted status . <br>
</p>
<p>I don't see the exact same error as mentioned by Kevin Drysdale
below (ploop/fsck) , but I am not able to enter certain CT
neither can I stop them <br>
</p>
<p><i>[root@olb~]# prlctl stop trans8</i><i><br>
</i><i>Stopping the CT...</i><i><br>
</i><i>Failed to stop the CT: PRL_ERR_VZCTL_OPERATION_FAILED
(Details: Cannot lock the Container</i><i><br>
</i><i>)</i><i><br>
</i></p>
<p><i>[root@olb ~]# prlctl enter trans8</i><i><br>
</i><i>Unable to get init pid</i><i><br>
</i><i>enter into CT failed</i><i><br>
</i><i><br>
</i><i>exited from CT 02faecdd-ddb6-42eb-8103-202508f18256</i><br>
</p>
<p>For those CTs that fail to enter or stop, I noticed that there
is a 2nd device mounted with name ending in <i>dump/Dump/.criu.cgyard.4EJB8c</i><i><br>
</i></p>
<p><i>[root@olb ~]# df -H |grep
02faecdd-ddb6-42eb-8103-202508f18256</i><i><br>
</i><i>/dev/ploop53152p1 11G 2,2G 7,7G 23%
/vz/root/02faecdd-ddb6-42eb-8103-202508f18256</i><i><br>
</i><i>none 537M 0 537M 0%
/vz/private/02faecdd-ddb6-42eb-8103-202508f18256/dump/Dump/.criu.cgyard.4EJB8c</i></p>
<p><br>
<i><i>[root@olb ~]# prlctl list | grep
02faecdd-ddb6-42eb-8103-202508f18256</i><i><br>
</i><i>{02faecdd-ddb6-42eb-8103-202508f18256} running
157.159.196.17 CT isptrans8</i><i><br>
</i></i></p>
<p>I rebooted the whole hardware node, and since reboot here is
the related vzctl.log<br>
</p>
<p><i>2020-07-06T15:10:38+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : Removing the stale lock
file /vz/lock/02faecdd-ddb6-42eb-8103-202508f18256.lck</i><i><br>
</i><i>2020-07-06T15:10:38+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : Restoring the Container
...</i><i><br>
</i><i>2020-07-06T15:10:38+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : Mount image:
/vz/private/02faecdd-ddb6-42eb-8103-202508f18256/root.hdd </i><i><br>
</i><i>2020-07-06T15:10:38+0200 : Opening delta
/vz/private/02faecdd-ddb6-42eb-8103-202508f18256/root.hdd/root.hds</i><i><br>
</i><i>2020-07-06T15:10:38+0200 : Opening delta
/vz/private/02faecdd-ddb6-42eb-8103-202508f18256/root.hdd/root.hds</i><i><br>
</i><i>2020-07-06T15:10:38+0200 : Opening delta
/vz/private/02faecdd-ddb6-42eb-8103-202508f18256/root.hdd/root.hds</i><i><br>
</i><i>2020-07-06T15:10:38+0200 : Adding delta
dev=/dev/ploop53152
img=/vz/private/02faecdd-ddb6-42eb-8103-202508f18256/root.hdd/root.hds
(rw)</i><i><br>
</i><i>2020-07-06T15:10:39+0200 : Mounted /dev/ploop53152p1 at
/vz/root/02faecdd-ddb6-42eb-8103-202508f18256 fstype=ext4
data=',balloon_ino=12' </i><i><br>
</i><i>2020-07-06T15:10:39+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : Container is mounted</i><i><br>
</i><i>2020-07-06T15:10:40+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : Setting permissions for
image=/vz/private/02faecdd-ddb6-42eb-8103-202508f18256/root.hdd</i><i><br>
</i><i>2020-07-06T15:10:40+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : Configure memguarantee:
0%</i><i><br>
</i><i>2020-07-06T15:18:12+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : Unable to get init pid</i><i><br>
</i><i>2020-07-06T15:18:12+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : enter into CT failed</i><i><br>
</i><i>2020-07-06T15:19:49+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : Cannot lock the
Container</i><i><br>
</i><i>2020-07-06T15:25:33+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : Unable to get init pid</i><i><br>
</i><i>2020-07-06T15:25:33+0200 vzctl : CT
02faecdd-ddb6-42eb-8103-202508f18256 : enter into CT failed</i><br>
</p>
<p>on another CT failing to enter / stop same kind of logs + <i>Error
(criu </i>: <br>
</p>
<p><i>2020-07-06T15:10:38+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Restoring the Container
...</i><i><br>
</i><i>2020-07-06T15:10:38+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Mount image:
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd </i><i><br>
</i><i>2020-07-06T15:10:38+0200 : Opening delta
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd/root.hds</i><i><br>
</i><i>2020-07-06T15:10:39+0200 : Opening delta
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd/root.hds</i><i><br>
</i><i>2020-07-06T15:10:39+0200 : Opening delta
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd/root.hds</i><i><br>
</i><i>2020-07-06T15:10:39+0200 : Adding delta
dev=/dev/ploop36049
img=/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd/root.hds
(rw)</i><i><br>
</i><i>2020-07-06T15:10:41+0200 : Mounted /dev/ploop36049p1 at
/vz/root/4ae48335-5b63-475d-8629-c8d742cb0ba0 fstype=ext4
data=',balloon_ino=12' </i><i><br>
</i><i>2020-07-06T15:10:41+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Container is mounted</i><i><br>
</i><i>2020-07-06T15:10:41+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Setting permissions for
image=/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd</i><i><br>
</i><i>2020-07-06T15:10:41+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Configure memguarantee:
0%</i><i><br>
</i><i>2020-07-06T15:10:57+0200 vzeventd : Run:
/etc/vz/vzevent.d/ve-stop
id=4ae48335-5b63-475d-8629-c8d742cb0ba0</i><i><br>
</i><i>2020-07-06T15:10:57+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : (03.038774) Error
(criu/util.c:666): exited, status=4</i><i><br>
</i><i>2020-07-06T15:10:57+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : (14.446513) 1:
Error (criu/files.c:230): Empty list on file desc id 0x1f(5)</i><i><br>
</i><i>2020-07-06T15:10:57+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : (14.446518) 1:
Error (criu/files.c:231): BUG at criu/files.c:231</i><i><br>
</i><i>2020-07-06T15:10:57+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : (15.589529) Error
(criu/cr-restore.c:1612): 7130 killed by signal 11:
Segmentation fault</i><i><br>
</i><i>2020-07-06T15:10:57+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : (15.604550) Error
(criu/cr-restore.c:2614): Restoring FAILED.</i><i><br>
</i><i>2020-07-06T15:10:57+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : The restore log was
saved in
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/dump/Dump/restore.log</i><i><br>
</i><i>2020-07-06T15:10:57+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : criu exited with rc=17</i><i><br>
</i><i>2020-07-06T15:10:57+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Unmount image:
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd
(190)</i><i><br>
</i><i>2020-07-06T15:10:57+0200 : Unmounting file system at
/vz/root/4ae48335-5b63-475d-8629-c8d742cb0ba0</i><i><br>
</i><i>2020-07-06T15:11:31+0200 : Opening delta
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd/root.hds</i><i><br>
</i><i>2020-07-06T15:11:31+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Container is unmounted</i><i><br>
</i><i>2020-07-06T15:11:31+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Failed to restore the
Container</i><i><br>
</i><i>2020-07-06T15:11:31+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Restoring the Container
...</i><i><br>
</i><i>2020-07-06T15:11:31+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Mount image:
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd </i><i><br>
</i><i>2020-07-06T15:11:31+0200 : Opening delta
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd/root.hds</i><i><br>
</i><i>2020-07-06T15:11:31+0200 : Opening delta
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd/root.hds</i><i><br>
</i><i>2020-07-06T15:11:31+0200 : Opening delta
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd/root.hds</i><i><br>
</i><i>2020-07-06T15:11:31+0200 : Adding delta
dev=/dev/ploop36049
img=/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd/root.hds
(rw)</i><i><br>
</i><i>2020-07-06T15:11:31+0200 : Mounted /dev/ploop36049p1 at
/vz/root/4ae48335-5b63-475d-8629-c8d742cb0ba0 fstype=ext4
data=',balloon_ino=12' </i><i><br>
</i><i>2020-07-06T15:11:31+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Container is mounted</i><i><br>
</i><i>2020-07-06T15:11:31+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Setting permissions for
image=/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd</i><i><br>
</i><i>2020-07-06T15:11:31+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Configure memguarantee:
0%</i><i><br>
</i><i>2020-07-06T15:14:18+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : Unable to get init pid</i><i><br>
</i><i>2020-07-06T15:14:18+0200 vzctl : CT
4ae48335-5b63-475d-8629-c8d742cb0ba0 : enter into CT failed</i><i><br>
</i></p>
<p>in prl-disp.log<br>
</p>
<p><i>07-06 15:10:30.797 F /virtuozzo:4836:4836/ register CT:
4ae48335-5b63-475d-8629-c8d742cb0ba0</i><i><br>
</i><i>07-06 15:10:38.717 F /disp:4836:6163/ Processing command
'DspCmdVmStartEx' 1036 for CT
uuid='{4ae48335-5b63-475d-8629-c8d742cb0ba0}' </i><i><br>
</i><i>07-06 15:10:38.738 I /virtuozzo:4836:6234/
/usr/sbin/vzctl resume 4ae48335-5b63-475d-8629-c8d742cb0ba0</i><i><br>
</i><i>07-06 15:10:48.542 I /disp:4836:5196/ vzevent: state=6,
envid=4ae48335-5b63-475d-8629-c8d742cb0ba0</i><i><br>
</i><i>07-06 15:10:57.364 I /disp:4836:5196/ vzevent: state=8,
envid=4ae48335-5b63-475d-8629-c8d742cb0ba0</i><i><br>
</i><i>07-06 15:10:57.475 I /disp:4836:5196/ vzevent: state=12,
envid=4ae48335-5b63-475d-8629-c8d742cb0ba0</i><i><br>
</i><i>07-06 15:11:31.161 F /virtuozzo:4836:6234/
/usr/sbin/vzctl utility failed: /usr/sbin/vzctl resume
4ae48335-5b63-475d-8629-c8d742cb0ba0 [6]</i><i><br>
</i><i>Mount image:
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd </i><i><br>
</i><i>Setting permissions for
image=/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd</i><i><br>
</i><i>Unmount image:
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/root.hdd
(190)</i><i><br>
</i><i>The restore log was saved in
/vz/private/4ae48335-5b63-475d-8629-c8d742cb0ba0/dump/Dump/restore.log</i><i><br>
</i><i>07-06 15:11:31.162 I /virtuozzo:4836:6234/
/usr/sbin/vzctl start 4ae48335-5b63-475d-8629-c8d742cb0ba0</i><br>
<br>
</p>
<p>Is this related to the update ? how can I renable those CT ? <br>
</p>
<p>Thanks . <br>
</p>
<p><br>
<em></em></p>
<blockquote type="cite"
cite="mid:daa01f5d-4a44-34a0-7e99-1592c9ce5dcf@imtbs-tsp.eu">
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Le 29/06/2020 à 12:30, Kevin
Drysdale a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:alpine.DEB.2.22.394.2006291128590.619775@drysdalk.iomartplc.com">Hello,
<br>
<br>
After updating one of our OpenVZ VPS hosting nodes at the end
of last week, we've started to have issues with corruption
apparently occurring inside containers. Issues of this nature
have never affected the node previously, and there do not
appear to be any hardware issues that could explain this. <br>
<br>
Specifically, a few hours after updating, we began to see
containers experiencing errors such as this in the logs: <br>
<br>
[90471.678994] EXT4-fs (ploop35454p1): error count since last
fsck: 25 <br>
[90471.679022] EXT4-fs (ploop35454p1): initial error at time
1593205255: ext4_ext_find_extent:904: inode 136399 <br>
[90471.679030] EXT4-fs (ploop35454p1): last error at time
1593232922: ext4_ext_find_extent:904: inode 136399 <br>
[95189.954569] EXT4-fs (ploop42983p1): error count since last
fsck: 67 <br>
[95189.954582] EXT4-fs (ploop42983p1): initial error at time
1593210174: htree_dirblock_to_tree:918: inode 926441: block
3683060 <br>
[95189.954589] EXT4-fs (ploop42983p1): last error at time
1593276902: ext4_iget:4435: inode 1849777 <br>
[95714.207432] EXT4-fs (ploop60706p1): error count since last
fsck: 42 <br>
[95714.207447] EXT4-fs (ploop60706p1): initial error at time
1593210489: ext4_ext_find_extent:904: inode 136272 <br>
[95714.207452] EXT4-fs (ploop60706p1): last error at time
1593231063: ext4_ext_find_extent:904: inode 136272 <br>
<br>
Shutting the containers down and manually mounting and
e2fsck'ing their filesystems did clear these errors, but each
of the containers (which were mostly used for running Plesk)
had widespread issues with corrupt or missing files after the
fsck's completed, necessitating their being restored from
backup. <br>
<br>
Concurrently, we also began to see messages like this
appearing in /var/log/vzctl.log, which again have never
appeared at any point prior to this update being installed: <br>
<br>
/var/log/vzctl.log:2020-06-26T21:05:19+0100 : Error in
fill_hole (check.c:240): Warning: ploop image
'/vz/private/8288448/root.hdd/root.hds' is sparse <br>
/var/log/vzctl.log:2020-06-26T21:09:41+0100 : Error in
fill_hole (check.c:240): Warning: ploop image
'/vz/private/8288450/root.hdd/root.hds' is sparse <br>
/var/log/vzctl.log:2020-06-26T21:16:22+0100 : Error in
fill_hole (check.c:240): Warning: ploop image
'/vz/private/8288451/root.hdd/root.hds' is sparse <br>
/var/log/vzctl.log:2020-06-26T21:19:57+0100 : Error in
fill_hole (check.c:240): Warning: ploop image
'/vz/private/8288452/root.hdd/root.hds' is sparse <br>
<br>
The basic procedure we follow when updating our nodes is as
follows: <br>
<br>
1, Update the standby node we keep spare for this process <br>
2. vzmigrate all containers from the live node being updated
to the standby node <br>
3. Update the live node <br>
4. Reboot the live node <br>
5. vzmigrate the containers from the standby node back to the
live node they originally came from <br>
<br>
So the only tool which has been used to affect these
containers is 'vzmigrate' itself, so I'm at something of a
loss as to how to explain the root.hdd images for these
containers containing sparse gaps. This is something we have
never done, as we have always been aware that OpenVZ does not
support their use inside a container's hard drive image. And
the fact that these images have suddenly become sparse at the
same time they have started to exhibit filesystem corruption
is somewhat concerning. <br>
<br>
We can restore all affected containers from backups, but I
wanted to get in touch with the list to see if anyone else at
any other site has experienced these or similar issues after
applying the 7.0.14 (136) update. <br>
<br>
Thank you, <br>
Kevin Drysdale. <br>
<br>
<br>
<br>
<br>
_______________________________________________ <br>
Users mailing list <br>
<a class="moz-txt-link-abbreviated"
href="mailto:Users@openvz.org" moz-do-not-send="true">Users@openvz.org</a>
<br>
<a class="moz-txt-link-freetext"
href="https://lists.openvz.org/mailman/listinfo/users"
moz-do-not-send="true">https://lists.openvz.org/mailman/listinfo/users</a>
<br>
</blockquote>
<p><br>
</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@openvz.org" moz-do-not-send="true">Users@openvz.org</a>
<a class="moz-txt-link-freetext" href="https://lists.openvz.org/mailman/listinfo/users" moz-do-not-send="true">https://lists.openvz.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<p><br>
</p>
</blockquote>
<p><br>
</p>
</body>
</html>