[CRIU] Using p.haul migration failure
Sowmini Varadhan
sowmini.varadhan at oracle.com
Sat Nov 1 19:08:22 PDT 2014
On (10/28/14 15:57), Pavel Emelyanov wrote:
>
> Please, note, that your dump stage will fail anyway, as there are some
> mount points in your program, that CRIU doesn't support for now. Let's
> check what it is.
>
> Thanks,
> Pavel
yes, I must be hitting this point.
having hacked the python to work around the veth.pair issue,
I'm now able to get further, but now I hit this error
Final dump and restore
Making directory /var/local/p.haul-fs/dmp-sYVUup-14.11.01-21.43/img/3
Issuing dump command to service
Downing veth6GI2X1
Notify (network-lock)
Upping veth6GI2X1
Notify (network-unlock)
Traceback (most recent call last):
File "/home/sowmini/src/p.haul/p.haul", line 39, in <module>
worker.start_migration()
File "/home/sowmini/src/p.haul/p_haul_iters.py", line 182, in start_migration
raise Exception("Dump failed")
Exception: Dump failed
Examining the log file, seems like the issue is this
(00.054526) Error (files-reg.c:792): Unable to look up the 18 mount
(00.054542) BUF 0x7f3998ccc000 >
(00.054549) ----------------------------------------
(00.054552) Error (cr-dump.c:1596): Dump files (pid: 2681) failed with -1
full log file (criu_dump.3.log) is attached.
Where is this now getting stuck? Any workarounds?
--Sowmini
-------------- next part --------------
(00.000031) ========================================
(00.000054) Dumping processes (pid: 2681)
(00.000058) ========================================
(00.000120) Found anon-shmem device at 4
(00.000128) Reset 20940's dirty tracking
(00.000163) ... done
(00.000188) Dirty track supported on kernel
(00.000222) irmap: Searching irmap cache in work dir
(00.000230) BUF++
(00.000235) BUF 0x7f3998cda000 <
(00.000244) irmap: ... done
(00.000246) irmap: Loading irmap cache
(00.000249) BUF 0x7f3998cda000 >
(00.000263) cpu: fpu:1 fxsr:1 xsave:0
(00.000324) vdso: Parsing at 7fffdd5fe000 7fffdd600000
(00.000327) vdso: PT_LOAD p_vaddr: 0
(00.000329) vdso: DT_HASH: 0x120
(00.000331) vdso: DT_STRTAB: 0x268
(00.000333) vdso: DT_SYMTAB: 0x160
(00.000334) vdso: DT_STRSZ: 94
(00.000336) vdso: DT_SYMENT: 24
(00.000338) vdso: nbucket 3 nchain 11 bucket 0x7fffdd5fe128 chain 0x7fffdd5fe134
(00.000344) vdso: rt [vdso] 7fffdd5fe000-7fffdd600000 [vvar] 7fffdd5fc000-7fffdd5fe000
(00.000395) Writing image inventory (version 1)
(00.000409) BUF 0x7f3998cda000 <
(00.000437) Collected 1.pid namespace
(00.000444) Collected 2.net namespace
(00.000449) Collected 3.ipc namespace
(00.000453) Collected 4.uts namespace
(00.000458) Collected 5.mnt namespace
(00.000460) cg: Dumping cgroups for 20940
(00.000499) cg: `- New css ID 1
(00.000502) cg: `- [blkio] -> [/user/1000.user/3.session]
(00.000505) cg: `- [cpu] -> [/user/1000.user/3.session]
(00.000506) cg: `- [cpuacct] -> [/user/1000.user/3.session]
(00.000508) cg: `- [cpuset] -> [/user/1000.user/3.session]
(00.000510) cg: `- [devices] -> [/user/1000.user/3.session]
(00.000512) cg: `- [freezer] -> [/user/1000.user/3.session]
(00.000514) cg: `- [hugetlb] -> [/user/1000.user/3.session]
(00.000516) cg: `- [memory] -> [/user/1000.user/3.session]
(00.000517) cg: `- [name=systemd] -> [/user/1000.user/3.session]
(00.000519) cg: `- [net_cls] -> [/user/1000.user/3.session]
(00.000521) cg: `- [perf_event] -> [/user/1000.user/3.session]
(00.000523) cg: Set 1 is criu one
(00.000533) BUF 0x7f3998cda000 >
(00.000537) Re-using ps socket 4
(00.000584) Seized task 2681, state 1
(00.000615) Seized task 2688, state 0
(00.000657) Seized task 2692, state 0
(00.000741) Collected 2692 in 1 state
(00.000810) Collected 2688 in 1 state
(00.000817) Seized task 2690, state 0
(00.000877) Collected 2690 in 1 state
(00.000895) Collected 2681 in 1 state
(00.000905) Will take pid namespace in the image
(00.000907) Collected 6.pid namespace
(00.000912) Will take net namespace in the image
(00.000914) Collected 7.net namespace
(00.000917) Will take ipc namespace in the image
(00.000919) Collected 8.ipc namespace
(00.000923) Will take uts namespace in the image
(00.000925) Collected 9.uts namespace
(00.000929) Will take mnt namespace in the image
(00.000930) Collected 10.mnt namespace
(00.000982) Lock network
(00.000984) Running network-lock scripts
(00.000995) RPC
(00.021518) lockinfo: 1:2 1 1126 00:13:11451 0 EOF
(00.021536) lockinfo: 2:2 1 1054 00:10:11349 0 EOF
(00.021619) type unsupported source /dev/disk/by-uuid/3e92507e-2d93-48a4-b229-4ba7cdf7c5df mnt_id 0x4d s_dev 0xfd00001 /var/lib/lxc/iperfs/rootfs @ ./ flags 0x200000 options errors=remount-ro,data=ordered
(00.021633) type proc source proc mnt_id 0x4e s_dev 0x25 / @ ./proc flags 0x20000e options
(00.021639) type proc source proc mnt_id 0x4f s_dev 0x25 /sys @ ./proc/sys flags 0x200001 options
(00.021644) type proc source proc mnt_id 0x50 s_dev 0x25 /sysrq-trigger @ ./proc/sysrq-trigger flags 0x200001 options
(00.021649) type sysfs source sysfs mnt_id 0x51 s_dev 0x26 / @ ./sys flags 0x200001 options
(00.021655) type unsupported source /dev/disk/by-uuid/3e92507e-2d93-48a4-b229-4ba7cdf7c5df mnt_id 0x52 s_dev 0xfd00001 /lib @ ./lib flags 0x200001 options errors=remount-ro,data=ordered
(00.021660) type unsupported source /dev/disk/by-uuid/3e92507e-2d93-48a4-b229-4ba7cdf7c5df mnt_id 0x53 s_dev 0xfd00001 /usr/lib @ ./usr/lib flags 0x200001 options errors=remount-ro,data=ordered
(00.021665) type unsupported source /dev/disk/by-uuid/3e92507e-2d93-48a4-b229-4ba7cdf7c5df mnt_id 0x54 s_dev 0xfd00001 /lib64 @ ./lib64 flags 0x200001 options errors=remount-ro,data=ordered
(00.021678) type securityfs source none mnt_id 0x55 s_dev 0xa / @ ./sys/kernel/security flags 0x200001 options
(00.021684) type devpts source devpts mnt_id 0x3b s_dev 0x27 / @ ./dev/pts flags 0x200000 options gid=5,mode=620,ptmxmode=666
(00.021697) type tmpfs source shm mnt_id 0x3c s_dev 0x28 / @ ./dev/shm flags 0x200000 options
(00.021711) Building mountpoints tree
(00.021715) Building plain mount tree
(00.021717) Working on 60->77
(00.021720) Working on 59->77
(00.021722) Working on 85->81
(00.021724) Working on 84->77
(00.021725) Working on 83->77
(00.021727) Working on 82->77
(00.021729) Working on 81->77
(00.021731) Working on 80->78
(00.021733) Working on 79->78
(00.021735) Working on 78->77
(00.021737) Working on 77->58
(00.021739) Resorting siblings on 77
(00.021741) Resorting siblings on 60
(00.021743) Resorting siblings on 59
(00.021745) Resorting siblings on 84
(00.021747) Resorting siblings on 83
(00.021749) Resorting siblings on 82
(00.021751) Resorting siblings on 81
(00.021753) Resorting siblings on 85
(00.021755) Resorting siblings on 78
(00.021757) Resorting siblings on 80
(00.021758) Resorting siblings on 79
(00.021760) Done:
(00.021762) [./](77->58)
(00.021764) [./lib64](84->77)
(00.021766) <--
(00.021768) [./proc](78->77)
(00.021770) [./proc/sysrq-trigger](80->78)
(00.021772) <--
(00.021774) [./proc/sys](79->78)
(00.021776) <--
(00.021778) <--
(00.021779) [./sys](81->77)
(00.021781) [./sys/kernel/security](85->81)
(00.021783) <--
(00.021785) <--
(00.021787) [./lib](82->77)
(00.021789) <--
(00.021791) [./usr/lib](83->77)
(00.021792) <--
(00.021794) [./dev/shm](60->77)
(00.021796) <--
(00.021798) [./dev/pts](59->77)
(00.021800) <--
(00.021802) <--
(00.021809) Error (mount.c:453): 84:./lib64 doesn't have a proper root mount
(00.021812) Collecting netns 7/2681
(00.021814) Switching to 2681's net for collecting sockets
(00.021900) Collected: ino 0x3647 peer_ino 0 family 1 type 2 state 7 name /dev/log
(00.021907) Collected: ino 0x1b607 peer_ino 0 family 1 type 5 state 7 name (null)
(00.021986) Collect packet sock 14552 2048
(00.021989) Got 0 mreqs
(00.022006) Collect netlink sock 0x361e
(00.022008) Collect netlink sock 0x3625
(00.022010) Collect netlink sock 0x1b606
(00.022012) Collect netlink sock 0x3624
(00.022014) Collect netlink sock 0x3621
(00.022016) Collect netlink sock 0x361f
(00.022018) Collect netlink sock 0x3620
(00.052867) BUF 0x7f3998cda000 <
(00.052882) BUF 0x7f3998cdb000 <
(00.052894) BUF 0x7f3998cdc000 <
(00.052903) BUF 0x7f3998cdd000 <
(00.052911) BUF 0x7f3998cde000 <
(00.052920) BUF 0x7f3998cdf000 <
(00.052929) BUF 0x7f3998ce0000 <
(00.052938) BUF 0x7f3998ce1000 <
(00.053199) BUF 0x7f3998ce2000 <
(00.053222) BUF 0x7f3998ce3000 <
(00.053233) BUF 0x7f3998ce4000 <
(00.053244) BUF 0x7f3998ce5000 <
(00.053252) BUF 0x7f3998ce6000 <
(00.053261) BUF 0x7f3998ce7000 <
(00.053270) BUF 0x7f3998ce8000 <
(00.053279) BUF 0x7f3998ce9000 <
(00.053288) BUF++
(00.053295) BUF 0x7f3998cc1000 <
(00.053304) BUF 0x7f3998cc2000 <
(00.053314) BUF 0x7f3998cc3000 <
(00.053325) BUF 0x7f3998cc4000 <
(00.053334) BUF 0x7f3998cc5000 <
(00.053339) ========================================
(00.053341) Dumping task (pid: 2681)
(00.053344) ========================================
(00.053345) Obtaining task stat ... (00.053418)
(00.053422) Collecting mappings (pid: 2681)
(00.053424) ----------------------------------------
(00.053430) BUF 0x7f3998cc6000 <
(00.053513) BUF 0x7f3998cc7000 <
(00.053522) BUF 0x7f3998cc7000 >
(00.053531) vma 7d1000 borrows vfi from previous 400000
(00.053557) BUF 0x7f3998cc6000 >
(00.053561) Collected, longest area occupies 466 pages
(00.053564) 0x400000-0x5d2000 (1864K) prot 0x5 flags 0x2 off 0 reg fp shmid: 0
(00.053567) 0x7d1000-0x7d4000 (12K) prot 0x3 flags 0x2 off 0x1d1000 reg fp shmid: 0
(00.053569) 0x7d4000-0x7da000 (24K) prot 0x3 flags 0x22 off 0 reg ap shmid: 0
(00.053571) 0x1e19000-0x1e3c000 (140K) prot 0x3 flags 0x22 off 0 reg heap ap shmid: 0
(00.053577) 0x7fffd2aae000-0x7fffd2ad0000 (136K) prot 0x3 flags 0x122 off 0 reg ap shmid: 0
(00.053580) 0x7fffd2bfc000-0x7fffd2bfe000 (8K) prot 0x1 flags 0x22 off 0 reg vvar ap shmid: 0
(00.053582) 0x7fffd2bfe000-0x7fffd2c00000 (8K) prot 0x5 flags 0x22 off 0 reg vdso ap shmid: 0
(00.053584) 0xffffffffff600000-0xffffffffff601000 (4K) prot 0x5 flags 0x22 off 0 vsys ap shmid: 0
(00.053587) ----------------------------------------
(00.053591)
(00.053593) Collecting fds (pid: 2681)
(00.053595) ----------------------------------------
(00.053607) Found 3 file descriptors
(00.053609) ----------------------------------------
(00.053614) BUF 0x7f3998cc6000 <
(00.053617) BUF 0x7f3998cc6000 >
(00.053621) Dump private signals of 2681
(00.053627) Dump shared signals of 2681
(00.053635) Parasite syscall_ip at 0x400000
(00.053733) Putting parasite blob into 0x7f3998cb3000->0x7f66a33c1000
(00.053753) Dumping GP/FPU registers for 2681
(00.053760) xsave runtime structure
(00.053762) -----------------------
(00.053763) cwd:37f swd:0 twd:0 fop:0 mxcsr:1f80 mxcsr_mask:ffff
(00.053766) magic1:0 extended_size:0 xstate_bv:0 xstate_size:0
(00.053768) xstate_bv: 0
(00.053770) -----------------------
(00.053772) Putting tsock into pid 2681
pie: Running daemon thread leader
pie: __sent ack msg: 2 2 0
pie: Daemon wais for command
(00.053873) Wait for parasite being daemonized...
(00.053876) Wait for ack 2 on daemon socket
(00.053881) Fetched ack: 2 2 0
(00.053883) Parasite 2681 has been switched to daemon mode
pie: __fetched msg: 15 0 0
pie: __sent ack msg: 15 15 0
pie: Daemon wais for command
(00.053909) Sent msg to daemon 15 0 0
(00.053914) Wait for ack 15 on daemon socket
(00.053918) Fetched ack: 15 15 0
pie: __fetched msg: 17 0 0
pie: __sent ack msg: 17 17 0
pie: Daemon wais for command
(00.053980) Sent msg to daemon 17 0 0
(00.053990) Wait for ack 17 on daemon socket
(00.053995) Fetched ack: 17 17 0
pie: __fetched msg: 12 0 0
pie: __sent ack msg: 12 12 0
pie: Daemon wais for command
(00.054091) Sent msg to daemon 12 0 0
(00.054096) Wait for ack 12 on daemon socket
(00.054100) Fetched ack: 12 12 0
(00.054103) sid=1 pgid=1 pid=1
(00.054123) BUF 0x7f3998cc6000 <
(00.054136) BUF 0x7f3998cc7000 <
(00.054176) BUF 0x7f3998cc8000 <
(00.054199) BUF 0x7f3998cc9000 <
(00.054213) BUF 0x7f3998cca000 <
(00.054226) BUF 0x7f3998ccb000 <
(00.054235)
(00.054238) Dumping opened files (pid: 2681)
(00.054240) ----------------------------------------
pie: __fetched msg: 14 0 0
pie: __sent ack msg: 14 14 0
pie: Daemon wais for command
(00.054274) Sent msg to daemon 14 0 0
(00.054282) Wait for ack 14 on daemon socket
(00.054286) Fetched ack: 14 14 0
(00.054300) BUF 0x7f3998ccc000 <
(00.054414) BUF 0x7f3998ccd000 <
(00.054460) BUF 0x7f3998ccd000 >
(00.054489) 2681 fdinfo 0: pos: 0x 0 flags: 100002/0
(00.054517) tty: Dumping tty 40 with id 0x1
(00.054526) Error (files-reg.c:792): Unable to look up the 18 mount
(00.054542) BUF 0x7f3998ccc000 >
(00.054549) ----------------------------------------
(00.054552) Error (cr-dump.c:1596): Dump files (pid: 2681) failed with -1
(00.054562) BUF 0x7f3998cc6000 >
(00.054571) BUF 0x7f3998cc7000 >
(00.054580) BUF 0x7f3998cc8000 >
(00.054590) BUF 0x7f3998cc9000 >
(00.054599) BUF 0x7f3998cca000 >
(00.054609) BUF 0x7f3998ccb000 >
(00.054627) Waiting for 2681 to trap
(00.054633) Daemon 2681 exited trapping
(00.054640) Sent msg to daemon 6 0 0
pie: __fetched msg: 6 0 0
pie: 1: new_sp=0x7f66a33ca008 ip 0x49e94e
(00.054701) 2681 was trapped
(00.054780) `- Expecting exit
(00.054844) 2681 was trapped
(00.054854) 2681 is going to execute the syscall f
(00.054874) 2681 was stopped
(00.054893) 2681 was trapped
(00.054897) 2681 is going to execute the syscall 1
(00.054904) 2681 was trapped
(00.054906) `- Expecting exit
(00.054913) 2681 was trapped
(00.054916) 2681 is going to execute the syscall b
(00.054934) 2681 was stopped
(00.054962) Disconnect from the page server (null):0
(00.055592) BUF 0x7f3998cda000 >
(00.055610) BUF 0x7f3998cdb000 >
(00.055617) BUF 0x7f3998cdc000 >
(00.055627) BUF 0x7f3998cdd000 >
(00.055633) BUF 0x7f3998cde000 >
(00.055639) BUF 0x7f3998cdf000 >
(00.055646) BUF 0x7f3998ce0000 >
(00.055653) BUF 0x7f3998ce1000 >
(00.055661) BUF 0x7f3998ce2000 >
(00.055668) BUF 0x7f3998ce3000 >
(00.055676) BUF 0x7f3998ce4000 >
(00.055682) BUF 0x7f3998ce5000 >
(00.055688) BUF 0x7f3998ce6000 >
(00.055696) BUF 0x7f3998ce7000 >
(00.055703) BUF 0x7f3998ce8000 >
(00.055709) BUF 0x7f3998ce9000 >
(00.055716) BUF 0x7f3998cc1000 >
(00.055723) BUF 0x7f3998cc2000 >
(00.055729) BUF 0x7f3998cc3000 >
(00.055736) BUF 0x7f3998cc4000 >
(00.055742) BUF 0x7f3998cc5000 >
(00.055746) Unlock network
(00.055749) Running network-unlock scripts
(00.055757) RPC
(00.078125) Unfreezing tasks into 1
(00.078152) Unseizing 2681 into 1
(00.078194) Unseizing 2688 into 1
(00.078201) Unseizing 2692 into 1
(00.078229) Unseizing 2690 into 1
(00.078296) Error (cr-dump.c:1952): Dumping FAILED.
More information about the CRIU
mailing list