[Users] can't resume Oracle CT

Tue Nov 20 01:00:27 EST 2012

Hello,

1. Looks like new OpenVZ suspend/resume feature doesn't work for a CT that
contains running Oracle DB.

Resume always trigger kill-everything cycle and then CT dies.
Although I had no any issues with container before suspend (all user
beancounters are zeroes and
nothing in logs).

HN and CT are both 6.3 with latest yum updates.
Kernel: 2.6.32-042stab063.2.
Oracle 11.2.0.3.

Here's what I have in HN /var/log/messages after vzctl resume 908:

> Nov 19 22:35:31 kernel: [24961.760236] CT: 908: started
> Nov 19 22:39:19 kernel: [25189.757112] >>> 908 oom generation 0 starts
> Nov 19 22:39:19 kernel: [25189.757120] __ratelimit: 4 callbacks suppressed
> Nov 19 22:39:19 kernel: [25189.757125] 41252 (crond) invoked loc
> oom-killer: gfp 0x200d2 order 0 oomkilladj=0
> Nov 19 22:39:19 kernel: [25189.757135] UB-908-Mem-Info:
> Nov 19 22:39:19 kernel: [25189.757139] Node 0 DMA scan:0 a_anon:0 i_anon:0
> a_file:0 i_file:0 unevictable:0
> Nov 19 22:39:19 kernel: [25189.757145] Node 0 DMA32 scan:5804378
> a_anon:233525 i_anon:77815 a_file:0 i_file:0 unevictable:0
> Nov 19 22:39:19 kernel: [25189.757152] Total 311339 anon:311339 file:0
> a_anon:233514 i_anon:77825 a_file:0 i_file:0 unevictable:0
> Nov 19 22:39:19 kernel: [25189.757157] RAM: 315097 / 307200 [1] SWAP:
> 21285 / 230400 [0] KMEM: 13336576 / inf [0] DCSZ: 680503 / inf [0] OOMG: 0
> / inf [0] Dirty 0 Wback 0 Dche 31 Prnd 0
> Nov 19 22:39:19 kernel: [25189.757272] Out of memory in UB: Kill process
> 41261 (oracle) score 3 or sacrifice child
> Nov 19 22:39:19 kernel: [25189.757305] OOM killed process 41261 (oracle)
> vm:950828kB, rss:0kB, swap:8364kB
> Nov 19 22:39:20 kernel: [25189.818729] OOM killed process oracle
> (pid=41261, ve=908) exited, free=129959.
> Nov 19 22:39:20 kernel: [25189.818736] UB-908-Mem-Info:
> Nov 19 22:39:20 kernel: [25189.818741] Node 0 DMA scan:0 a_anon:0 i_anon:0
> a_file:0 i_file:0 unevictable:0
> Nov 19 22:39:20 kernel: [25189.818747] Node 0 DMA32 scan:6025211
> a_anon:233532 i_anon:77815 a_file:0 i_file:0 unevictable:0
> Nov 19 22:39:20 kernel: [25189.818754] Total 311334 anon:311334 file:0
> a_anon:233514 i_anon:77820 a_file:0 i_file:0 unevictable:0
> Nov 19 22:39:20 kernel: [25189.818759] RAM: 315062 / 307200 [6] SWAP:
> 19350 / 230400 [0] KMEM: 13193216 / inf [0] DCSZ: 680503 / inf [0] OOMG: 0
> / inf
>
(full version is attached).

/etc/vz/conf/908.conf :

> ONBOOT="yes"
> BOOTORDER="350"
> CPUUNITS="250"
> IOPRIO="5"
>
> PHYSPAGES="0:1300M"
> SWAPPAGES="0:1000M"
>
> DISK_QUOTA="yes"
> DISKSPACE="22G:25G"
> DISKINODES="900000:1100000"
> QUOTATIME="0"
>
> VE_ROOT="/vz/root/$VEID"
> VE_PRIVATE="/vz/private/$VEID"
> OSTEMPLATE="centos-6-x86_64-CT5xx"
> ORIGIN_SAMPLE="vswap-1g"
> IP_ADDRESS="172.9.x.x"
> HOSTNAME="oraxx"
>

and then a bunch of parameters taken from
http://wiki.openvz.org/Oracle_10g_VE
all equal to "unlimited". All except PHYSPAGES that is shown above.

I did this a few times - decreased amount of memory that Oracle consumes,
it is a consistent bug,
or obviously I don't understand something. Also wanted to notice that every
time I suspended CT,
the dump file was around 1.3Gb even after I decreased amount of memory
assigned to Oracle by 300M.
Does OpenVZ dump what PHYSPAGES says? I was surprised by this - my
expectation was it'll dump
only memory pages that are in use/allocated.

2. Is any way to disable automatic suspend/resume - I read that HN reboot
causes all CTs to suspend,
   and when it boots up they got resumed. I don't want this behavior for
Oracle CT now.

Best regards,

Ruslan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvz.org/pipermail/users/attachments/20121119/d563cda8/attachment-0001.html>
-------------- next part --------------
Nov 19 22:35:31 kernel: [24961.760236] CT: 908: started
Nov 19 22:39:19 kernel: [25189.757112] >>> 908 oom generation 0 starts
Nov 19 22:39:19 kernel: [25189.757120] __ratelimit: 4 callbacks suppressed
Nov 19 22:39:19 kernel: [25189.757125] 41252 (crond) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:19 kernel: [25189.757135] UB-908-Mem-Info:
Nov 19 22:39:19 kernel: [25189.757139] Node 0 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
Nov 19 22:39:19 kernel: [25189.757145] Node 0 DMA32 scan:5804378 a_anon:233525 i_anon:77815 a_file:0 i_file:0 unevictable:0
Nov 19 22:39:19 kernel: [25189.757152] Total 311339 anon:311339 file:0 a_anon:233514 i_anon:77825 a_file:0 i_file:0 unevictable:0
Nov 19 22:39:19 kernel: [25189.757157] RAM: 315097 / 307200 [1] SWAP: 21285 / 230400 [0] KMEM: 13336576 / inf [0] DCSZ: 680503 / inf [0] OOMG: 0 / inf [0] Dirty 0 Wback 0 Dche 31 Prnd 0
Nov 19 22:39:19 kernel: [25189.757272] Out of memory in UB: Kill process 41261 (oracle) score 3 or sacrifice child
Nov 19 22:39:19 kernel: [25189.757305] OOM killed process 41261 (oracle) vm:950828kB, rss:0kB, swap:8364kB
Nov 19 22:39:20 kernel: [25189.818729] OOM killed process oracle (pid=41261, ve=908) exited, free=129959.
Nov 19 22:39:20 kernel: [25189.818736] UB-908-Mem-Info:
Nov 19 22:39:20 kernel: [25189.818741] Node 0 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
Nov 19 22:39:20 kernel: [25189.818747] Node 0 DMA32 scan:6025211 a_anon:233532 i_anon:77815 a_file:0 i_file:0 unevictable:0
Nov 19 22:39:20 kernel: [25189.818754] Total 311334 anon:311334 file:0 a_anon:233514 i_anon:77820 a_file:0 i_file:0 unevictable:0
Nov 19 22:39:20 kernel: [25189.818759] RAM: 315062 / 307200 [6] SWAP: 19350 / 230400 [0] KMEM: 13193216 / inf [0] DCSZ: 680503 / inf [0] OOMG: 0 / inf [1] Dirty 0 Wback 0 Dche 31 Prnd 0
Nov 19 22:39:20 kernel: [25189.818773] <<< 908 oom generation 0 ends (task died)
Nov 19 22:39:21 kernel: [25191.537162] >>> 908 oom generation 1 starts
Nov 19 22:39:21 kernel: [25191.537173] 41263 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:21 kernel: [25191.537178] UB-908-Mem-Info:
Nov 19 22:39:21 kernel: [25191.537183] Node 0 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
Nov 19 22:39:21 kernel: [25191.537188] Node 0 DMA32 scan:12382939 a_anon:233450 i_anon:77815 a_file:0 i_file:0 unevictable:0
Nov 19 22:39:21 kernel: [25191.537195] Total 311265 anon:311265 file:0 a_anon:233450 i_anon:77815 a_file:0 i_file:0 unevictable:0
Nov 19 22:39:21 kernel: [25191.537201] RAM: 315046 / 307200 [22] SWAP: 19350 / 230400 [0] KMEM: 13127680 / inf [0] DCSZ: 680503 / inf [0] OOMG: 0 / inf [1] Dirty 0 Wback 0 Dche 34 Prnd 0
Nov 19 22:39:21 kernel: [25191.537329] Out of memory in UB: Kill process 41266 (oracle) score 2 or sacrifice child
Nov 19 22:39:21 kernel: [25191.537358] OOM killed process 41266 (oracle) vm:950284kB, rss:0kB, swap:5376kB
Nov 19 22:39:21 kernel: [25191.575838] OOM killed process oracle (pid=41266, ve=908) exited, free=131261.
Nov 19 22:39:21 kernel: [25191.575845] UB-908-Mem-Info:
Nov 19 22:39:21 kernel: [25191.575850] Node 0 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
Nov 19 22:39:21 kernel: [25191.575855] Node 0 DMA32 scan:12533467 a_anon:233482 i_anon:77815 a_file:0 i_file:1 unevictable:0
Nov 19 22:39:21 kernel: [25191.575862] Total 311302 anon:311301 file:1 a_anon:233486 i_anon:77815 a_file:0 i_file:1 unevictable:0
Nov 19 22:39:21 kernel: [25191.575867] RAM: 315005 / 307200 [23] SWAP: 18060 / 230400 [0] KMEM: 12955648 / inf [0] DCSZ: 680503 / inf [0] OOMG: 0 / inf [2] Dirty 0 Wback 0 Dche 34 Prnd 0
Nov 19 22:39:21 kernel: [25191.575881] <<< 908 oom generation 1 ends (task died)
Nov 19 22:39:23 kernel: [25193.307978] >>> 908 oom generation 2 starts
Nov 19 22:39:23 kernel: [25193.307987] 41049 (init) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:23 kernel: [25193.308059] Out of memory in UB: Kill process 41273 (oracle) score 2 or sacrifice child
Nov 19 22:39:23 kernel: [25193.308059] OOM killed process 41273 (oracle) vm:948964kB, rss:0kB, swap:5700kB
Nov 19 22:39:23 kernel: [25193.354600] OOM killed process oracle (pid=41273, ve=908) exited, free=132819.
Nov 19 22:39:23 kernel: [25193.354608] <<< 908 oom generation 2 ends (task died)
Nov 19 22:39:25 kernel: [25194.914864] >>> 908 oom generation 3 starts
Nov 19 22:39:25 kernel: [25194.914875] 41274 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:25 kernel: [25194.914971] Out of memory in UB: Kill process 41278 (oracle) score 2 or sacrifice child
Nov 19 22:39:25 kernel: [25194.915005] OOM killed process 41278 (oracle) vm:950136kB, rss:0kB, swap:5880kB
Nov 19 22:39:25 kernel: [25194.918696] OOM killed process oracle (pid=41278, ve=908) exited, free=134338.
Nov 19 22:39:25 kernel: [25194.918702] <<< 908 oom generation 3 ends (task died)
Nov 19 22:39:26 kernel: [25196.588809] >>> 908 oom generation 4 starts
Nov 19 22:39:26 kernel: [25196.588818] 41254 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:26 kernel: [25196.588923] Out of memory in UB: Kill process 41249 (tnslsnr) score 1 or sacrifice child
Nov 19 22:39:26 kernel: [25196.588966] OOM killed process 41249 (tnslsnr) vm:214448kB, rss:0kB, swap:3572kB
Nov 19 22:39:26 kernel: [25196.592308] OOM killed process tnslsnr (pid=41250, ve=908) exited, free=135332.
Nov 19 22:39:26 kernel: [25196.592316] <<< 908 oom generation 4 ends (task died)
Nov 19 22:39:28 kernel: [25198.259227] >>> 908 oom generation 5 starts
Nov 19 22:39:28 kernel: [25198.259237] 41268 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:28 kernel: [25198.259334] Out of memory in UB: Kill process 41253 (oracle) score 1 or sacrifice child
Nov 19 22:39:28 kernel: [25198.259365] OOM killed process 41253 (oracle) vm:946592kB, rss:0kB, swap:2144kB
Nov 19 22:39:28 kernel: [25198.266014] OOM killed process oracle (pid=41253, ve=908) exited, free=135894.
Nov 19 22:39:28 kernel: [25198.266021] <<< 908 oom generation 5 ends (task died)
Nov 19 22:39:30 kernel: [25199.824964] >>> 908 oom generation 6 starts
Nov 19 22:39:30 kernel: [25199.824974] 41267 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:30 kernel: [25199.825088] Out of memory in UB: Kill process 41258 (oracle) score 1 or sacrifice child
Nov 19 22:39:30 kernel: [25199.825122] OOM killed process 41258 (oracle) vm:944840kB, rss:0kB, swap:2308kB
Nov 19 22:39:30 kernel: [25199.848294] OOM killed process oracle (pid=41258, ve=908) exited, free=136576.
Nov 19 22:39:30 kernel: [25199.848304] <<< 908 oom generation 6 ends (task died)
Nov 19 22:39:31 kernel: [25201.334676] >>> 908 oom generation 7 starts
Nov 19 22:39:31 kernel: [25201.334685] 41254 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:31 kernel: [25201.334778] Out of memory in UB: Kill process 41259 (oracle) score 1 or sacrifice child
Nov 19 22:39:31 kernel: [25201.334807] OOM killed process 41259 (oracle) vm:944816kB, rss:0kB, swap:2544kB
Nov 19 22:39:31 kernel: [25201.374441] OOM killed process oracle (pid=41259, ve=908) exited, free=137258.
Nov 19 22:39:31 kernel: [25201.374450] <<< 908 oom generation 7 ends (task died)
Nov 19 22:39:33 kernel: [25202.890073] >>> 908 oom generation 8 starts
Nov 19 22:39:33 kernel: [25202.890082] 41272 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:33 kernel: [25202.890178] Out of memory in UB: Kill process 41262 (oracle) score 1 or sacrifice child
Nov 19 22:39:33 kernel: [25202.890207] OOM killed process 41262 (oracle) vm:959856kB, rss:0kB, swap:2332kB
Nov 19 22:39:33 kernel: [25202.916123] OOM killed process oracle (pid=41262, ve=908) exited, free=137878.
Nov 19 22:39:33 kernel: [25202.916132] <<< 908 oom generation 8 ends (task died)
Nov 19 22:39:34 kernel: [25204.331244] >>> 908 oom generation 9 starts
Nov 19 22:39:34 kernel: [25204.331254] 41260 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:34 kernel: [25204.331355] Out of memory in UB: Kill process 41263 (oracle) score 1 or sacrifice child
Nov 19 22:39:34 kernel: [25204.331389] OOM killed process 41263 (oracle) vm:944304kB, rss:0kB, swap:2132kB
Nov 19 22:39:34 kernel: [25204.348255] OOM killed process oracle (pid=41263, ve=908) exited, free=138529.
Nov 19 22:39:34 kernel: [25204.348263] <<< 908 oom generation 9 ends (task died)
Nov 19 22:39:35 kernel: [25205.688469] >>> 908 oom generation 10 starts
Nov 19 22:39:35 kernel: [25205.688478] 41255 (oracle) invoked loc oom-killer: gfp 0x8000d0 order 0 oomkilladj=0
Nov 19 22:39:35 kernel: [25205.688570] Out of memory in UB: Kill process 41264 (oracle) score 1 or sacrifice child
Nov 19 22:39:35 kernel: [25205.688605] OOM killed process 41264 (oracle) vm:944868kB, rss:0kB, swap:2576kB
Nov 19 22:39:35 kernel: [25205.721417] OOM killed process oracle (pid=41264, ve=908) exited, free=139242.
Nov 19 22:39:35 kernel: [25205.721424] <<< 908 oom generation 10 ends (task died)
Nov 19 22:39:37 kernel: [25206.882250] >>> 908 oom generation 11 starts
Nov 19 22:39:37 kernel: [25206.882258] 41270 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:37 kernel: [25206.882366] Out of memory in UB: Kill process 41265 (oracle) score 1 or sacrifice child
Nov 19 22:39:37 kernel: [25206.882394] OOM killed process 41265 (oracle) vm:944828kB, rss:0kB, swap:2348kB
Nov 19 22:39:37 kernel: [25206.900401] OOM killed process oracle (pid=41265, ve=908) exited, free=139893.
Nov 19 22:39:37 kernel: [25206.900408] <<< 908 oom generation 11 ends (task died)
Nov 19 22:39:38 kernel: [25207.973016] >>> 908 oom generation 12 starts
Nov 19 22:39:38 kernel: [25207.973026] 41257 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:38 kernel: [25207.973113] Out of memory in UB: Kill process 41267 (oracle) score 1 or sacrifice child
Nov 19 22:39:38 kernel: [25207.973143] OOM killed process 41267 (oracle) vm:945540kB, rss:0kB, swap:3132kB
Nov 19 22:39:38 kernel: [25207.976752] OOM killed process oracle (pid=41267, ve=908) exited, free=140414.
Nov 19 22:39:38 kernel: [25207.976759] <<< 908 oom generation 12 ends (task died)
Nov 19 22:39:39 kernel: [25209.059461] >>> 908 oom generation 13 starts
Nov 19 22:39:39 kernel: [25209.059471] 41255 (oracle) invoked loc oom-killer: gfp 0x8000d0 order 0 oomkilladj=0
Nov 19 22:39:39 kernel: [25209.059554] Out of memory in UB: Kill process 41268 (oracle) score 1 or sacrifice child
Nov 19 22:39:39 kernel: [25209.059583] OOM killed process 41268 (oracle) vm:946340kB, rss:0kB, swap:2668kB
Nov 19 22:39:39 kernel: [25209.071057] OOM killed process oracle (pid=41268, ve=908) exited, free=141127.
Nov 19 22:39:39 kernel: [25209.071057] <<< 908 oom generation 13 ends (task died)
Nov 19 22:39:40 kernel: [25210.008570] >>> 908 oom generation 14 starts
Nov 19 22:39:40 kernel: [25210.008579] 41269 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:40 kernel: [25210.008664] Out of memory in UB: Kill process 41270 (oracle) score 1 or sacrifice child
Nov 19 22:39:40 kernel: [25210.008698] OOM killed process 41270 (oracle) vm:968048kB, rss:0kB, swap:2344kB
Nov 19 22:39:40 kernel: [25210.031335] OOM killed process oracle (pid=41270, ve=908) exited, free=141623.
Nov 19 22:39:40 kernel: [25210.031342] <<< 908 oom generation 14 ends (task died)
Nov 19 22:39:41 kernel: [25210.858544] >>> 908 oom generation 15 starts
Nov 19 22:39:41 kernel: [25210.858553] 41255 (oracle) invoked loc oom-killer: gfp 0x8000d0 order 0 oomkilladj=0
Nov 19 22:39:41 kernel: [25210.858635] Out of memory in UB: Kill process 41271 (oracle) score 1 or sacrifice child
Nov 19 22:39:41 kernel: [25210.858664] OOM killed process 41271 (oracle) vm:968048kB, rss:0kB, swap:2340kB
Nov 19 22:39:41 kernel: [25210.861841] OOM killed process oracle (pid=41271, ve=908) exited, free=142274.
Nov 19 22:39:41 kernel: [25210.861847] <<< 908 oom generation 15 ends (task died)
Nov 19 22:39:41 kernel: [25211.695788] >>> 908 oom generation 16 starts
Nov 19 22:39:41 kernel: [25211.695797] 41257 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:41 kernel: [25211.695880] Out of memory in UB: Kill process 41275 (oracle) score 1 or sacrifice child
Nov 19 22:39:41 kernel: [25211.696072] OOM killed process 41275 (oracle) vm:945892kB, rss:0kB, swap:2932kB
Nov 19 22:39:41 kernel: [25211.712591] OOM killed process oracle (pid=41275, ve=908) exited, free=143082.
Nov 19 22:39:41 kernel: [25211.712599] <<< 908 oom generation 16 ends (task died)
Nov 19 22:39:42 kernel: [25212.432003] >>> 908 oom generation 17 starts
Nov 19 22:39:42 kernel: [25212.432011] 41256 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:42 kernel: [25212.432094] Out of memory in UB: Kill process 41276 (oracle) score 1 or sacrifice child
Nov 19 22:39:42 kernel: [25212.432122] OOM killed process 41276 (oracle) vm:944336kB, rss:0kB, swap:2332kB
Nov 19 22:39:42 kernel: [25212.440045] OOM killed process oracle (pid=41276, ve=908) exited, free=143733.
Nov 19 22:39:42 kernel: [25212.440052] <<< 908 oom generation 17 ends (task died)
Nov 19 22:39:43 kernel: [25213.254015] >>> 908 oom generation 18 starts
Nov 19 22:39:43 kernel: [25213.254024] 41252 (crond) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:43 kernel: [25213.254118] Out of memory in UB: Kill process 41277 (oracle) score 1 or sacrifice child
Nov 19 22:39:43 kernel: [25213.254163] OOM killed process 41277 (oracle) vm:946544kB, rss:0kB, swap:3336kB
Nov 19 22:39:43 kernel: [25213.256786] OOM killed process oracle (pid=41277, ve=908) exited, free=144601.
Nov 19 22:39:43 kernel: [25213.256794] <<< 908 oom generation 18 ends (task died)
Nov 19 22:39:44 kernel: [25214.154887] >>> 908 oom generation 19 starts
Nov 19 22:39:44 kernel: [25214.154897] 41269 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:44 kernel: [25214.154980] Out of memory in UB: Kill process 41279 (oracle) score 1 or sacrifice child
Nov 19 22:39:44 kernel: [25214.155018] OOM killed process 41279 (oracle) vm:946524kB, rss:0kB, swap:3200kB
Nov 19 22:39:44 kernel: [25214.157593] OOM killed process oracle (pid=41279, ve=908) exited, free=145469.
Nov 19 22:39:44 kernel: [25214.157599] <<< 908 oom generation 19 ends (task died)
Nov 19 22:39:45 kernel: [25214.869670] >>> 908 oom generation 20 starts
Nov 19 22:39:45 kernel: [25214.869680] 41254 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:45 kernel: [25214.869757] Out of memory in UB: Kill process 41254 (oracle) score 0 or sacrifice child
Nov 19 22:39:45 kernel: [25214.869797] OOM killed process 41254 (oracle) vm:944304kB, rss:0kB, swap:1984kB
Nov 19 22:39:45 kernel: [25214.884558] OOM killed process oracle (pid=41254, ve=908) exited, free=146027.
Nov 19 22:39:45 kernel: [25214.884565] <<< 908 oom generation 20 ends (task died)
Nov 19 22:39:45 kernel: [25215.453540] >>> 908 oom generation 21 starts
Nov 19 22:39:45 kernel: [25215.453548] 41274 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:45 kernel: [25215.453630] Out of memory in UB: Kill process 41255 (oracle) score 0 or sacrifice child
Nov 19 22:39:45 kernel: [25215.453662] OOM killed process 41255 (oracle) vm:944304kB, rss:0kB, swap:1992kB
Nov 19 22:39:45 kernel: [25215.459161] OOM killed process oracle (pid=41255, ve=908) exited, free=146523.
Nov 19 22:39:45 kernel: [25215.459169] <<< 908 oom generation 21 ends (task died)
Nov 19 22:39:46 kernel: [25216.041000] >>> 908 oom generation 22 starts
Nov 19 22:39:46 kernel: [25216.041010] 41260 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:46 kernel: [25216.041088] Out of memory in UB: Kill process 41256 (oracle) score 0 or sacrifice child
Nov 19 22:39:46 kernel: [25216.041119] OOM killed process 41256 (oracle) vm:944304kB, rss:0kB, swap:1988kB
Nov 19 22:39:46 kernel: [25216.053146] OOM killed process oracle (pid=41256, ve=908) exited, free=147083.
Nov 19 22:39:46 kernel: [25216.053154] <<< 908 oom generation 22 ends (task died)
Nov 19 22:39:46 kernel: [25216.551532] >>> 908 oom generation 23 starts
Nov 19 22:39:46 kernel: [25216.551541] 41269 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:46 kernel: [25216.551619] Out of memory in UB: Kill process 41257 (oracle) score 0 or sacrifice child
Nov 19 22:39:46 kernel: [25216.551650] OOM kill task 41274 (oracle) in ub 908
Nov 19 22:39:46 kernel: [25216.551653] OOM killer in rage, 1 tasks killed in ub 908
Nov 19 22:39:46 kernel: [25216.551669] OOM killed process 41257 (oracle) vm:944304kB, rss:0kB, swap:1976kB
Nov 19 22:39:46 kernel: [25216.555316] OOM killed process oracle (pid=41257, ve=908) exited, free=148137.
Nov 19 22:39:46 kernel: [25216.555323] <<< 908 oom generation 23 ends (task died)
Nov 19 22:39:47 kernel: [25216.913712] >>> 908 oom generation 24 starts
Nov 19 22:39:47 kernel: [25216.913721] 41272 (oracle) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:47 kernel: [25216.913802] Out of memory in UB: Kill process 41260 (oracle) score 0 or sacrifice child
Nov 19 22:39:47 kernel: [25216.913833] OOM kill task 41272 (oracle) in ub 908
Nov 19 22:39:47 kernel: [25216.913843] OOM kill task 41269 (oracle) in ub 908
Nov 19 22:39:47 kernel: [25216.913847] OOM killer in rage, 2 tasks killed in ub 908
Nov 19 22:39:47 kernel: [25216.913864] OOM killed process 41260 (oracle) vm:944304kB, rss:0kB, swap:1984kB
Nov 19 22:39:47 kernel: [25216.921592] OOM killed process oracle (pid=41260, ve=908) exited, free=149749.
Nov 19 22:39:47 kernel: [25216.921600] <<< 908 oom generation 24 ends (task died)
Nov 19 22:39:47 kernel: [25217.129940] >>> 908 oom generation 25 starts
Nov 19 22:39:47 kernel: [25217.129949] 41049 (init) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:47 kernel: [25217.130020] Out of memory in UB: Kill process 41252 (crond) score -700 or sacrifice child
Nov 19 22:39:47 kernel: [25217.130099] OOM kill task 41252 (crond) in ub 908
Nov 19 22:39:47 kernel: [25217.130114] OOM killer in rage, 1 tasks killed in ub 908
Nov 19 22:39:47 kernel: [25217.130128] OOM killed process 41252 (crond) vm:117208kB, rss:0kB, swap:600kB
Nov 19 22:39:47 kernel: [25217.130903] OOM killed process crond (pid=41252, ve=908) exited, free=149966.
Nov 19 22:39:47 kernel: [25217.130910] <<< 908 oom generation 25 ends (task died)
Nov 19 22:39:47 kernel: [25217.281487] >>> 908 oom generation 26 starts
Nov 19 22:39:47 kernel: [25217.281496] 41049 (init) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:47 kernel: [25217.281564] Out of memory in UB: Kill process 41248 (udevd) score -890 or sacrifice child
Nov 19 22:39:47 kernel: [25217.281603] OOM kill task 41248 (udevd) in ub 908
Nov 19 22:39:47 kernel: [25217.281616] OOM killer in rage, 1 tasks killed in ub 908
Nov 19 22:39:47 kernel: [25217.281630] OOM killed process 41248 (udevd) vm:10660kB, rss:0kB, swap:272kB
Nov 19 22:39:47 kernel: [25217.282046] OOM killed process udevd (pid=41248, ve=908) exited, free=150059.
Nov 19 22:39:47 kernel: [25217.282083] <<< 908 oom generation 26 ends (task died)
Nov 19 22:39:47 kernel: [25217.432122] >>> 908 oom generation 27 starts
Nov 19 22:39:47 kernel: [25217.432131] 41049 (init) invoked loc oom-killer: gfp 0x200d2 order 0 oomkilladj=0
Nov 19 22:39:47 kernel: [25217.432199] Out of memory in UB: Kill process 41049 (init) score -900 or sacrifice child
Nov 19 22:39:47 kernel: [25217.432235] OOM kill task 41049 (init) in ub 908
Nov 19 22:39:47 kernel: [25217.432247] OOM killer in rage, 1 tasks killed in ub 908
Nov 19 22:39:47 kernel: [25217.432261] OOM killed process 41049 (init) vm:19232kB, rss:0kB, swap:236kB
Nov 19 22:39:47 kernel: [25217.432734] OOM killed process init (pid=41049, ve=908) exited, free=150121.
Nov 19 22:39:47 kernel: [25217.432738] <<< 908 oom generation 27 ends (task died)
Nov 19 22:39:48 kernel: [25217.886247] CT: 908: stopped