<div dir="ltr"><div>Thanks for your answers, I asked some followup regarding resource usage in containers in a different thread.</div><div><br></div><div>btw. as I mentioned, I am trying to benchmark/observe the behavior of live migration for "realistic" production systems, basically looking at J2EE enterprise or service-oriented applications. Can anyone tell me if such benchmarking has been done? </div><div>Or any suggestions that you may have on how to check if live migration is "practical" for real scenarios? </div><div>Also how would one define "practical"? </div><div><br></div><div>Currently I am looking into the Dacapo Benchmark DayTrader Application (<a href="http://www.dacapobench.org">http://www.dacapobench.org</a>)</div><div>I got a migration suspend time of 12-15 seconds.</div><div><br></div><div>Thanks</div><div>Nipun</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 28, 2014 at 4:49 AM, Kir Kolyshkin <span dir="ltr"><<a href="mailto:kir@openvz.org" target="_blank">kir@openvz.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
On 11/28/2014 12:47 AM, Pavel Odintsov wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Nice suggestion! Old fashioned UBC is real nightmare and was deprecated.<br>
</blockquote>
<br></span>
In fact they are not deprecated, but rather optional. The beauty of it<br>
is you can still limit say network buffers or number of processes (or<br>
have out of memory killer guarantee) -- but you don't<br>
HAVE to do it, as the default setup (only setting ram/swap) should be<br>
secure enough.<span class="HOEnZb"><font color="#888888"><br>
<br>
Kir.</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On Fri, Nov 28, 2014 at 10:35 AM, CoolCold <<a href="mailto:coolthecold@gmail.com" target="_blank">coolthecold@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello!<br>
I'd recommend set only ram/swap limits (via --ram/--swap) , letting other<br>
settings be mostly unlimited (while ram/swap limits are not overflowed of<br>
course) - <a href="http://wiki.openvz.org/VSwap" target="_blank">http://wiki.openvz.org/VSwap</a><br>
<br>
On Fri, Nov 28, 2014 at 3:31 AM, Nipun Arora <<a href="mailto:nipunarora2512@gmail.com" target="_blank">nipunarora2512@gmail.com</a>><br>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Nevermind, I figured it out by changing the fail counters in<br>
/proc/user_beans<br>
<br>
Thanks<br>
Nipun<br>
<br>
On Thu, Nov 27, 2014 at 7:14 PM, Nipun Arora <<a href="mailto:nipunarora2512@gmail.com" target="_blank">nipunarora2512@gmail.com</a>><br>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks, the speed is improved by an order of magnitude :)<br>
<br>
btw. is there any benchmark, that you all have looked into for testing<br>
how good/practical live migration is for real-world systems?<br>
Additionally, I'm trying to run a java application(dacapo benchmark), but<br>
keep having trouble in getting java to run..<br>
<br>
java -version<br>
<br>
Error occurred during initialization of VM<br>
<br>
Could not reserve enough space for object heap<br>
<br>
Could not create the Java virtual machine.<br>
<br>
<br>
I've put my vz conf file below, can anyone suggest what could be the<br>
problem?<br>
<br>
Thanks<br>
Nipun<br>
<br>
# UBC parameters (in form of barrier:limit)<br>
<br>
KMEMSIZE="14372700:14790164"<br>
<br>
LOCKEDPAGES="2048:2048"<br>
<br>
PRIVVMPAGES="65536:69632"<br>
<br>
SHMPAGES="21504:21504"<br>
<br>
NUMPROC="240:240"<br>
<br>
PHYSPAGES="0:131072"<br>
<br>
VMGUARPAGES="33792:unlimited"<br>
<br>
OOMGUARPAGES="26112:unlimited"<br>
<br>
NUMTCPSOCK="360:360"<br>
<br>
NUMFLOCK="188:206"<br>
<br>
NUMPTY="16:16"<br>
<br>
NUMSIGINFO="256:256"<br>
<br>
TCPSNDBUF="1720320:2703360"<br>
<br>
TCPRCVBUF="1720320:2703360"<br>
<br>
OTHERSOCKBUF="1126080:2097152"<br>
<br>
DGRAMRCVBUF="262144:262144"<br>
<br>
NUMOTHERSOCK="1200"<br>
<br>
DCACHESIZE="3409920:3624960"<br>
<br>
NUMFILE="9312:9312"<br>
<br>
AVNUMPROC="180:180"<br>
<br>
NUMIPTENT="128:128"<br>
<br>
<br>
# Disk quota parameters (in form of softlimit:hardlimit)<br>
<br>
DISKSPACE="3145728:3145728"<br>
<br>
DISKINODES="131072:144179"<br>
<br>
QUOTATIME="0"<br>
<br>
<br>
# CPU fair scheduler parameter<br>
<br>
CPUUNITS="1000"<br>
<br>
<br>
NETFILTER="stateless"<br>
<br>
VE_ROOT="/vz/root/101"<br>
<br>
VE_PRIVATE="/vz/private/101"<br>
<br>
OSTEMPLATE="centos-6-x86_64"<br>
<br>
ORIGIN_SAMPLE="basic"<br>
<br>
HOSTNAME="test"<br>
<br>
IP_ADDRESS="192.168.1.101"<br>
<br>
NAMESERVER="8.8.8.8 8.8.4.4"<br>
<br>
CPULIMIT="25"<br>
<br>
SWAPPAGES="0:262144"<br>
<br>
<br>
<br>
On Mon, Nov 24, 2014 at 12:16 PM, Kir Kolyshkin <<a href="mailto:kir@openvz.org" target="_blank">kir@openvz.org</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On 11/23/2014 07:13 PM, Nipun Arora wrote:<br>
<br>
Thanks, I will try your suggestions, and get back to you.<br>
btw... any idea what could be used to share the base image on both<br>
containers?<br>
Like hardlink it in what way? Once both containers start, won't they<br>
have to write to different locations?<br>
<br>
<br>
ploop is composed as a set of stacked images, with all of them but the<br>
top one being read-only.<br>
<br>
<br>
I understand that some file systems have a copy on write mechanism,<br>
where after a snapshot all future writes are written to a additional linked<br>
disks.<br>
Does ploop operate in a similar way?<br>
<br>
<br>
yes<br>
<br>
<br>
<a href="http://wiki.qemu.org/Features/Snapshots" target="_blank">http://wiki.qemu.org/Features/<u></u>Snapshots</a><br>
<br>
<br>
<a href="http://openvz.livejournal.com/44508.html" target="_blank">http://openvz.livejournal.com/<u></u>44508.html</a><br>
<br>
<br>
<br>
The cloning with a modified vzmigrate script helps.<br>
<br>
- Nipun<br>
<br>
On Sun, Nov 23, 2014 at 5:29 PM, Kir Kolyshkin <<a href="mailto:kir@openvz.org" target="_blank">kir@openvz.org</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On 11/23/2014 04:59 AM, Nipun Arora wrote:<br>
<br>
Hi Kir,<br>
<br>
Thanks for the response, I'll update it, and tell you about the<br>
results.<br>
<br>
1. A follow up question... I found that the write I/O speed of<br>
500-1Mbps increased the suspend time to several minutes.(mostly pcopy<br>
stage)<br>
This seems extremely high for a relatively low I/O workload, which is<br>
why I was wondering if there are any special things I need to take care of.<br>
(I ran fio (flexible i/o writer) with fixed throughput while doing live<br>
migration)<br>
<br>
<br>
Please retry with vzctl 4.8 and ploop 1.12.1 (make sure they are on<br>
both sides).<br>
There was a 5 second wait for the remote side to finish syncing<br>
copied ploop data. It helped a case with not much I/O activity in<br>
container, but<br>
ruined the case you are talking about.<br>
<br>
Newer ploop and vzctl implement a feedback channel for ploop copy that<br>
eliminates<br>
that wait time.<br>
<br>
<a href="http://git.openvz.org/?p=ploop;a=commit;h=20d754c91079165b" target="_blank">http://git.openvz.org/?p=<u></u>ploop;a=commit;h=<u></u>20d754c91079165b</a><br>
<a href="http://git.openvz.org/?p=vzctl;a=commit;h=374b759dec45255d4" target="_blank">http://git.openvz.org/?p=<u></u>vzctl;a=commit;h=<u></u>374b759dec45255d4</a><br>
<br>
There are some other major improvements as well, such as async send for<br>
ploop.<br>
<br>
<a href="http://git.openvz.org/?p=ploop;a=commit;h=a55e26e9606e0b" target="_blank">http://git.openvz.org/?p=<u></u>ploop;a=commit;h=<u></u>a55e26e9606e0b</a><br>
<br>
<br>
2. For my purposes, I have modified the live migration script to allow<br>
me to do cloning... i.e. I start both the containers instead of deleting the<br>
original. I need to do this "cloning" from time to time for the same target<br>
container...<br>
<br>
a. Which means that lets say we cloned container C1 to container<br>
C2, and let both execute at time t0, this works with no apparent loss of<br>
service.<br>
<br>
b. Now at time t1 I would like to again clone C1 to C2, and<br>
would like to optimize the rsync process as most of the ploop file for C1<br>
and C2 should still be the same (i.e. less time to sync). Can anyone suggest<br>
what would be the best way to realize the second point?<br>
<br>
<br>
You can create a ploop snapshot and use shared base image for both<br>
containers<br>
(instead of copying the base delta, hardlink it). This is not supported<br>
by tools<br>
(for example, since base delta is now shared you can't merge down to<br>
it, but the<br>
tools are not aware) so you need to figure it out by yourself and be<br>
accurate<br>
but it should work.<br>
<br>
<br>
<br>
<br>
Thanks<br>
Nipun<br>
<br>
On Sun, Nov 23, 2014 at 12:56 AM, Kir Kolyshkin <<a href="mailto:kir@openvz.org" target="_blank">kir@openvz.org</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On 11/22/2014 09:09 AM, Nipun Arora wrote:<br>
<br>
Hi All,<br>
<br>
I was wondering if anyone can suggest what is the most optimal way to<br>
do the following<br>
<br>
1. Can anyone clarify if ploop is the best layout for minimum suspend<br>
time during live migration?<br>
<br>
<br>
Yes (due to ploop copy which only copies the modified blocks).<br>
<br>
<br>
2. I tried migrating a ploop device where I increased the --diskspace<br>
to 5G,<br>
and found that the suspend time taken by live migration increased to<br>
57 seconds<br>
(mainly undump and restore increased)...<br>
whereas a 2G diskspace was taking 2-3 seconds suspend time... Is this<br>
expected?<br>
<br>
<br>
No. Undump and restore times depends mostly on amount of RAM used by a<br>
container.<br>
<br>
Having said that, live migration stages influence each other, although<br>
it's less so<br>
in the latest vzctl release (I won't go into details here if you allow<br>
me -- just make sure<br>
you test with vzctl 4.8).<br>
<br>
<br>
3. I tried running a write intensive workload, and found that beyond<br>
100-150Kbps,<br>
the suspend time during live migration rapidly increased? Is this an<br>
expected trend?<br>
<br>
<br>
Sure. With increased writing speed, the amount of data that needs to<br>
be copied after CT<br>
is suspended increases.<br>
<br>
<br>
I am using vzctl 4.7, and ploop 1.11 in centos 6.5<br>
<br>
<br>
You need to update vzctl and ploop and rerun your tests, there should<br>
be<br>
some improvement (in particular with respect to issue #3).<br>
<br>
<br>
Thanks<br>
Nipun<br>
<br>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/<u></u>mailman/listinfo/users</a><br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/<u></u>mailman/listinfo/users</a><br>
<br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/<u></u>mailman/listinfo/users</a><br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/<u></u>mailman/listinfo/users</a><br>
<br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/<u></u>mailman/listinfo/users</a><br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/<u></u>mailman/listinfo/users</a><br>
<br>
</blockquote></blockquote>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/<u></u>mailman/listinfo/users</a><br>
<br>
</blockquote>
<br>
<br>
--<br>
Best regards,<br>
[COOLCOLD-RIPN]<br>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/<u></u>mailman/listinfo/users</a><br>
<br>
</blockquote>
<br>
<br>
</blockquote>
<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@openvz.org" target="_blank">Users@openvz.org</a><br>
<a href="https://lists.openvz.org/mailman/listinfo/users" target="_blank">https://lists.openvz.org/<u></u>mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div>