<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.E-MailFormatvorlage17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE-AT" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">I am trying to dump/restore a process running third-party software that is now using POSIX named semaphores after a recent update.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">After initial dump/restores failed due to invisible, seemingly random /dev/shm/* files, I took a deep dive into the intricacies of named<o:p></o:p></p>
<p class="MsoNormal">POSIX semaphores. They create random-named files (mktemp) first which are later unlinked once the desired sem.X file has been created (linked).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I wrote a simple producer / consumer process pair incrementing/decrementing a System V shared-memory counter protected by a POSIX named semaphore to<o:p></o:p></p>
<p class="MsoNormal">simulate the productive software and tried to dump/restore it using the latest build (criu-devel).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">After starting the producer process, the named semaphore is available in /dev/shm as expected:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">raini@rd-devel:~/Development/semtest$ sudo setsid unshare -i ./producer<o:p></o:p></p>
<p class="MsoNormal">raini@rd-devel:~/Development/semtest$ ls -lhai /dev/shm/*<o:p></o:p></p>
<p class="MsoNormal">5 -rw-r--r-- 1 root root 32 Dez 13 17:32 /dev/shm/sem.pcsync<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The problematic links can already be seen in map_files:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">raini@rd-devel:~/Development/semtest$ sudo ls -l /proc/$(pidof producer)/map_files/<o:p></o:p></p>
<p class="MsoNormal">...<o:p></o:p></p>
<p class="MsoNormal">lrw------- 1 root root 64 Dez 13 17:33 7f599057e000-7f599057f000 -> '/SYSV00000539 (deleted)'<o:p></o:p></p>
<p class="MsoNormal">lrw------- 1 root root 64 Dez 13 17:33 7f599057f000-7f5990580000 -> '/dev/shm/sem.4y8Ymq (deleted)'<o:p></o:p></p>
<p class="MsoNormal">...<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Dumping the process fails due to the invisible file created in sem_open:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">raini@rd-devel:~/Development/semtest/dump$ sudo criu dump -t $(pidof producer) --shell-job -D . -vvvv<o:p></o:p></p>
<p class="MsoNormal">...<o:p></o:p></p>
<p class="MsoNormal">(00.003612) Dumping path for -3 fd via self 12 [/dev/shm/sem.4y8Ymq (deleted)]<o:p></o:p></p>
<p class="MsoNormal">(00.003618) Strip ' (deleted)' tag from './dev/shm/sem.4y8Ymq (deleted)'<o:p></o:p></p>
<p class="MsoNormal">(00.003624) Error (criu/files-reg.c:991): Can't create link remap for /dev/shm/sem.4y8Ymq. Use link-remap option.<o:p></o:p></p>
<p class="MsoNormal">(00.003629) Error (criu/cr-dump.c:1269): Collect mappings (pid: 1895) failed with -1<o:p></o:p></p>
<p class="MsoNormal">(00.003672) Unlock network<o:p></o:p></p>
<p class="MsoNormal">(00.003677) Unfreezing tasks into 1<o:p></o:p></p>
<p class="MsoNormal">(00.003679) Unseizing 1895 into 1<o:p></o:p></p>
<p class="MsoNormal">(00.003700) Error (criu/cr-dump.c:1788): Dumping FAILED.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Using the link-remap option as instructed, the dump succeeds and files.img contains both the random named semaphore file<o:p></o:p></p>
<p class="MsoNormal">and a link remap file, but not the named semaphore file "sem.pcsync" as created by the producer process:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">raini@rd-devel:~/Development/semtest/dump$ sudo crit decode --pretty -i files.img | grep shm<o:p></o:p></p>
<p class="MsoNormal"> "name": "/dev/shm/link_remap.4"<o:p></o:p></p>
<p class="MsoNormal"> "name": "/dev/shm/sem.4y8Ymq",<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Since the semaphore file is not removed (sem_unlink) it still persists, together with the link_remap file that's not cleaned up:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">raini@rd-devel:~/Development/semtest/dump$ ls -lhai /dev/shm/<o:p></o:p></p>
<p class="MsoNormal">total 8,0K<o:p></o:p></p>
<p class="MsoNormal">1 drwxrwxrwt 2 root root 80 Dez 13 17:42 .<o:p></o:p></p>
<p class="MsoNormal">1 drwxr-xr-x 20 root root 4,2K Dez 13 17:13 ..<o:p></o:p></p>
<p class="MsoNormal">5 -rw-r--r-- 2 root root 32 Dez 13 17:32 link_remap.4<o:p></o:p></p>
<p class="MsoNormal">5 -rw-r--r-- 2 root root 32 Dez 13 17:32 sem.pcsync<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If I try to restore locally directly after dumping, the restore succeeds and semaphore operations (sem_wait/sem_post) still work as expected:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">raini@rd-devel:~/Development/semtest/dump$ sudo criu restore -d -D . --shell-job --link-remap -vvvv<o:p></o:p></p>
<p class="MsoNormal">...<o:p></o:p></p>
<p class="MsoNormal">(00.003590) Found regular file mapping, OK<o:p></o:p></p>
<p class="MsoNormal">(00.003592) path: /SYSV00000539 (deleted)<o:p></o:p></p>
<p class="MsoNormal">(00.003615) Found regular file mapping, OK<o:p></o:p></p>
<p class="MsoNormal">(00.003630) Dumping path for -3 fd via self 12 [/dev/shm/sem.W5abgh (deleted)]<o:p></o:p></p>
<p class="MsoNormal">(00.003635) Strip ' (deleted)' tag from './dev/shm/sem.W5abgh (deleted)'<o:p></o:p></p>
<p class="MsoNormal">(00.003659) Only file size could be stored for validation for file /dev/shm/sem.W5abgh<o:p></o:p></p>
<p class="MsoNormal">(00.003672) Found regular file mapping, OK<o:p></o:p></p>
<p class="MsoNormal">(00.003689) Dumping path for -3 fd via self 12 [/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2]<o:p></o:p></p>
<p class="MsoNormal">...<o:p></o:p></p>
<p class="MsoNormal">(00.011397) Namespaces dump complete<o:p></o:p></p>
<p class="MsoNormal">(00.011446) cg: All tasks in criu's cgroups. Nothing to dump.<o:p></o:p></p>
<p class="MsoNormal">(00.011449) unix: Dumping external sockets<o:p></o:p></p>
<p class="MsoNormal">(00.011469) tty: Unpaired slave 0<o:p></o:p></p>
<p class="MsoNormal">(00.011474) Writing image inventory (version 1)<o:p></o:p></p>
<p class="MsoNormal">(00.011536) Running post-dump scripts<o:p></o:p></p>
<p class="MsoNormal">(00.011541) Unfreezing tasks into 2<o:p></o:p></p>
<p class="MsoNormal">(00.011543) Unseizing 3037 into 2<o:p></o:p></p>
<p class="MsoNormal">(00.011717) Writing stats<o:p></o:p></p>
<p class="MsoNormal">(00.011748) Dumping finished successfully<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">However, if I repeat the same procedure and restore either on another machine or on the same machine after reboot, the restore fails:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">raini@rd-devel:~/Development/semtest/dump$ sudo criu restore -d -D . --shell-job --link-remap -vvvv<o:p></o:p></p>
<p class="MsoNormal">...<o:p></o:p></p>
<p class="MsoNormal">(00.066370) 3037: Opening 0x007f7659a78000-0x007f7659a79000 0000000000000000 (81) vma<o:p></o:p></p>
<p class="MsoNormal">(00.066387) 3037: Warn (criu/files-reg.c:1748): Can't link dev/shm/link_remap.4 -> dev/shm/sem.W5abgh<o:p></o:p></p>
<p class="MsoNormal">(00.066415) 3037: Error (criu/files-reg.c:2125): Can't link dev/shm/link_remap.4 -> dev/shm/sem.W5abgh: No such file or directory<o:p></o:p></p>
<p class="MsoNormal">(00.066432) 3037: Error (criu/mem.c:1349): `- Can't open vma<o:p></o:p></p>
<p class="MsoNormal">(00.066502) Error (criu/cr-restore.c:2470): Restoring FAILED.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I would have assumed that the link_remap.4 file would be restored by criu itself, however restore fails if it does not exist, which it wouldn't in a productive scenario.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Furthermore I noticed that even if the link_remap.4 file created by the original dump persists but the /dev/shm.pcsync semaphore file is deleted, it is not restored during a successful restore.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">How can I get criu to consistently dump/restore POSIX named semaphores?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If it is not supported - what's the problem compared to SysV semaphores/shared-memory and how could I tackle adding named semaphore support within criu?<o:p></o:p></p>
</div>
<br>
<br>
<br>
<br>
<br>
<div style="font-size:10pt; font-family: 'Calibri',sans-serif;">FN 181686 k. HG Wien, UID-Nr. ATU 47056901, zertifiziert nach ISO 9001:2015 Nr. 04036/0<br>
Der Inhalt dieser E-Mail ist vertraulich und ausschließlich für den bezeichneten Adressaten bestimmt. Wenn Sie nicht der vorgesehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, so beachten Sie bitte, dass jede Form der Kenntnisnahme, Veröffentlichung,
Vervielfältigung oder Weitergabe des Inhaltes dieser E-Mail unzulässig ist. Wir bitten Sie, sich in diesem Fall mit dem Absender der E-Mail in Verbindung zu setzen und die E-Mail zu vernichten. Für Übermittlungsfehler oder sonstige Irrtümer bei der Übermittlung
besteht keine Haftung.<br>
This e-mail is intended solely for the person to whom it is addressed and may contain confidential or legally privileged information. Access to this e-mail by anyone else is unauthorized. If an addressing or transmission error has misdirected this e-mail, please
notify the author by replying to this e-mail and destroy this e-mail and any attachments. E-mail may be susceptible to data corruption, interception, unauthorized amendment, viruses and delays or the consequences thereof. If you are not the intended recipient,
be advised that you have received this e-mail in error and that any use, dissemination, forwarding, printing or copying of this e-mail is strictly prohibited.
</div>
</body>
</html>