<div dir="ltr">Thank you very much!<div>I have found the bad guy which leads to restore failed in container! </div><div><br></div><div>Here it is:</div><div>root@dslab:/home/dslab/tools/criu# ./crit show /home/checkpoint/bad_cr/ids-68.img </div><div>{</div><div> "magic": "IDS", </div><div> "entries": [</div><div> {</div><div> "vm_id": 4, </div><div> "files_id": 4, </div><div> "fs_id": 4, </div><div> "sighand_id": 4, </div><div> "pid_ns_id": 7, </div><div> "net_ns_id": 8, </div><div> "ipc_ns_id": 9, </div><div> "uts_ns_id": 10, </div><div> "mnt_ns_id": 11, </div><div> "user_ns_id": 6</div><div> }</div><div> ]</div><div>} </div><div><br></div><div><div>root@dslab:/home/dslab/tools/criu# ./crit show /home/checkpoint/bad_cr/core-68.img </div><div>{</div><div> "magic": "CORE", </div><div> "entries": [</div><div> {</div><div> "mtype": "X86_64", </div><div>...</div><div> "tc": {<br></div><div> "task_state": 1, </div><div> "exit_code": 0, </div><div> "personality": 0, </div><div> "flags": 1077944384, </div><div> "blk_sigset": "0x0", </div><div> "comm": "dhclient", </div><div> "timers": {</div><div> "real": {</div><div> "isec": 0, </div><div> "iusec": 0, </div><div> "vsec": 0, </div><div> "vusec": 0</div><div> }, </div><div> "virt": {</div><div> "isec": 0, </div><div> "iusec": 0, </div><div> "vsec": 0, </div><div> "vusec": 0</div><div> }, </div><div> "prof": {</div><div> "isec": 0, </div><div> "iusec": 0, </div><div> "vsec": 0, </div><div> "vusec": 0</div><div> }</div><div> }, </div><div> "rlimits": {</div><div> "rlimits": [</div><div> {</div><div> "cur": 18446744073709551615, </div><div> "max": 18446744073709551615</div><div> }, </div><div> {</div><div> "cur": 18446744073709551615, </div><div> "max": 18446744073709551615</div><div> }, </div><div> {</div><div> "cur": 18446744073709551615, </div><div> "max": 18446744073709551615</div><div> }, </div><div> {</div><div> "cur": 8388608, </div><div> "max": 18446744073709551615</div><div> }, </div><div> {</div><div> "cur": 0, </div><div> "max": 18446744073709551615</div><div> }, </div><div> {</div><div> "cur": 18446744073709551615, </div><div> "max": 18446744073709551615</div><div> }, </div><div> {</div><div> "cur": 31333, </div><div> "max": 31333</div><div> }, </div><div> {</div><div> "cur": 65536, </div><div> "max": 65536</div><div> }, </div><div> {</div><div> "cur": 65536, </div><div> "max": 65536</div><div> }, </div><div> {</div><div> "cur": 18446744073709551615, </div><div> "max": 18446744073709551615</div><div> }, </div><div> {</div><div> "cur": 18446744073709551615, </div><div> "max": 18446744073709551615</div><div> }, </div><div> {</div><div> "cur": 31333, </div><div> "max": 31333</div><div> }, </div><div> {</div><div> "cur": 819200, </div><div> "max": 819200</div><div> }, </div><div> {</div><div> "cur": 0, </div><div> "max": 0</div><div> }, </div><div> {</div><div> "cur": 0, </div><div> "max": 0</div><div> }, </div><div> {</div><div> "cur": 18446744073709551615, </div><div> "max": 18446744073709551615</div><div> }</div><div> ]</div><div> }, </div><div> "cg_set": 4, </div><div> "signals_s": {}</div><div> }, </div><div> "thread_core": {</div><div> "futex_rla": 0, </div><div> "futex_rla_len": 24, </div><div> "sched_nice": 0, </div><div> "sched_policy": 0, </div><div> "sas": {</div><div> "ss_sp": 0, </div><div> "ss_size": 0, </div><div> "ss_flags": 2</div><div> }, </div><div> "signals_p": {}</div><div> }</div><div> }</div><div> ]</div><div>}</div></div><div><br></div><div>dhclient maybe use <span style="font-size:14px">SOCK_PACKET to acquire IP address. When I kill this process, CRIU works well !</span><br></div><div><span style="font-size:14px"><br></span></div><div><br></div><div>Best regards</div><div>From Jason Lee </div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-10-19 16:51 GMT+08:00 Pavel Emelyanov <span dir="ltr"><<a href="mailto:xemul@parallels.com" target="_blank">xemul@parallels.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 10/16/2015 06:05 AM, Jason Lee wrote:<br>
> In actually,I have done as you said but there are no pid info in fdinfo-img,just id,flag,type and fd.<br>
<br>
</span>Yes, fdinfo is a table, without pids. Pid is ... implicit here ;) E.g. fdinfo-4.img<br>
means that this is table number 4. Now you need to look at ids-*.img and check which<br>
one of them has files_id being 4. The respective image file name would contain the<br>
pid of the task owning one.<br>
<div><div class="h5"><br>
> root@dslab:/home/dslab/tools/criu# ./crit show /home/checkpoint/c2/fdinfo-4.img<br>
> {<br>
> "magic": "FDINFO",<br>
> "entries": [<br>
> {<br>
> "id": 56,<br>
> "flags": 0,<br>
> "type": "REG",<br>
> "fd": 0<br>
> },<br>
> {<br>
> "id": 57,<br>
> "flags": 0,<br>
> "type": "REG",<br>
> "fd": 1<br>
> },<br>
> {<br>
> "id": 58,<br>
> "flags": 0,<br>
> "type": "REG",<br>
> "fd": 2<br>
> },<br>
> {<br>
> "id": 59,<br>
> "flags": 1,<br>
> "type": "UNIXSK",<br>
> "fd": 3<br>
> },<br>
> {<br>
> "id": 60,<br>
> "flags": 0,<br>
> "type": "REG",<br>
> "fd": 4<br>
> },<br>
> {<br>
> "id": 61,<br>
> "flags": 1,<br>
> "type": "PACKETSK",<br>
> "fd": 5<br>
> },<br>
> {<br>
> "id": 62,<br>
> "flags": 1,<br>
> "type": "INETSK",<br>
> "fd": 6<br>
> },<br>
> {<br>
> "id": 63,<br>
> "flags": 0,<br>
> "type": "INETSK",<br>
> "fd": 20<br>
> },<br>
> {<br>
> "id": 64,<br>
> "flags": 0,<br>
> "type": "INETSK",<br>
> "fd": 21<br>
> }<br>
> ]<br>
> }<br>
><br>
> In packetsk.img, there are the pid under "fown",It's the pid which I look for ?<br>
><br>
> root@dslab:/home/dslab/tools/criu# ./crit show /home/checkpoint/c2/packetsk.img<br>
> {<br>
> "magic": "PACKETSK",<br>
> "entries": [<br>
> {<br>
> "id": 61,<br>
> "type": 10,<br>
> "protocol": 768,<br>
> "flags": "0x80002",<br>
> "ifindex": 73,<br>
> "fown": {<br>
> "uid": 0,<br>
> "euid": 0,<br>
> "signum": 0,<br>
> "pid_type": 0,<br>
> "pid": 0<br>
> },<br>
> ...<br>
> }<br>
><br>
> In the checkpoint dir,there are several core-$pid.img:<br>
> root@dslab:/home/checkpoint# ls c2/core-*<br>
> c2/core-1.img c2/core-20.img c2/core-68.img c2/core-89.img c2/core-90.img c2/core-92.img<br>
><br>
> So I grep each core img<br>
> root@dslab:/home/dslab/tools/criu# ./crit show /home/checkpoint/c2/core-1.img | grep comm<br>
> "comm": "systemd",<br>
> root@dslab:/home/dslab/tools/criu# ./crit show /home/checkpoint/c2/core-20.img | grep comm<br>
> "comm": "systemd-journal",<br>
> root@dslab:/home/dslab/tools/criu# ./crit show /home/checkpoint/c2/core-68.img | grep comm<br>
> "comm": "dhclient",<br>
> root@dslab:/home/dslab/tools/criu# ./crit show /home/checkpoint/c2/core-89.img | grep comm<br>
> "comm": "sshd",<br>
> root@dslab:/home/dslab/tools/criu# ./crit show /home/checkpoint/c2/core-90.img | grep comm<br>
> "comm": "rc.local",<br>
> root@dslab:/home/dslab/tools/criu# ./crit show /home/checkpoint/c2/core-92.img | grep comm<br>
> "comm": "a.out",<br>
><br>
> a.out is just the test app without network usage.Maybe something wrong exist in other processes?<br>
><br>
</div></div>> 2015-10-16 2:10 GMT+08:00 Tycho Andersen <<a href="mailto:tycho.andersen@canonical.com">tycho.andersen@canonical.com</a> <mailto:<a href="mailto:tycho.andersen@canonical.com">tycho.andersen@canonical.com</a>>>:<br>
<span class="">><br>
> On Thu, Oct 15, 2015 at 08:37:02PM +0800, Jason Lee wrote:<br>
> > In this server,I only run the latest version LXC and CRIU. There are no any<br>
> > my own applications<br>
> > using socket in LXC(maybe sshd or dhclient?).My distribution linux is<br>
> > debian 8<br>
> > so I don't know which applications use SOCK_PACKET ...<br>
><br>
> If you look to see which pid (crit show fdinfo-$pid.img) has an fd<br>
> info with the id 61 (your SOCK_PACKET socket), you can get the command<br>
> name from core.img (crit show core-$pid.img | grep comm). I'd be<br>
> curious to know what application this is, because it if is something<br>
> like systemd, it will come downstream to ubuntu and we'll need to fix<br>
> this soon :)<br>
><br>
> Tycho<br>
><br>
> > To avoid this situation, I think using two network card is better than<br>
> > net-bridge,isn't it ?<br>
> ><br>
> > I wish criu could solve this issue in next version!<br>
> ><br>
> ><br>
> > - Jason<br>
> ><br>
</span>> > 2015-10-15 20:08 GMT+08:00 Jason Lee <<a href="mailto:ldm5235@gmail.com">ldm5235@gmail.com</a> <mailto:<a href="mailto:ldm5235@gmail.com">ldm5235@gmail.com</a>>>:<br>
<span class="im HOEnZb">> ><br>
> > ><br>
> > > ---------- Forwarded message ----------<br>
> > > From: Pavel Emelyanov <<a href="mailto:xemul@parallels.com">xemul@parallels.com</a> <mailto:<a href="mailto:xemul@parallels.com">xemul@parallels.com</a>>><br>
> > > Date: 2015-10-15 20:06 GMT+08:00<br>
> > > Subject: Re: [CRIU] lxc-checkpoint restore failed<br>
</span><div class="HOEnZb"><div class="h5">> > > To: Jason Lee <<a href="mailto:ldm5235@gmail.com">ldm5235@gmail.com</a> <mailto:<a href="mailto:ldm5235@gmail.com">ldm5235@gmail.com</a>>><br>
> > > Cc: Tycho Andersen <<a href="mailto:tycho.andersen@canonical.com">tycho.andersen@canonical.com</a> <mailto:<a href="mailto:tycho.andersen@canonical.com">tycho.andersen@canonical.com</a>>>, <a href="mailto:criu@openvz.org">criu@openvz.org</a> <mailto:<a href="mailto:criu@openvz.org">criu@openvz.org</a>><br>
> > ><br>
> > ><br>
> > > On 10/15/2015 03:04 PM, Pavel Emelyanov wrote:<br>
> > > > On 10/15/2015 02:58 PM, Jason Lee wrote:<br>
> > > >> OK!<br>
> > > >> Here it is:<br>
> > > >><br>
> > > >> root@dslab:/home/dslab/tools/criu# ./crit show<br>
> > > /home/checkpoint/c2/packetsk.img<br>
> > > >> {<br>
> > > >> "magic": "PACKETSK",<br>
> > > >> "entries": [<br>
> > > >> {<br>
> > > >> "id": 61,<br>
> > > >> "type": 10,<br>
> > > ><br>
> > > > Here it is. This is SOCK_PACKET which we didn't support (and didn't put<br>
> > > check<br>
> > > > for it on dump). Which software uses this thing? AF_PACKET sockets are<br>
> > > typically<br>
> > > > SOCK_RAW or SOCK_DGRAM, SOCK_PACKET is, frankly speaking, new to me :)<br>
> > ><br>
> > > Just FIY, I've created an issue for this feature:<br>
> > > <a href="https://github.com/xemul/criu/issues/73" rel="noreferrer" target="_blank">https://github.com/xemul/criu/issues/73</a><br>
> > ><br>
> > > -- Pavel<br>
> > ><br>
> > ><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div>