[Users] Fwd: Fwd: Container networking broken after upgrade to 2.6.18-308.8.2.el5.028stab101.1, stange kernel errors. CPT ERR

Tue Feb 5 06:50:11 EST 2013

Hello.
I've created a bug in Bugzilla:
https://bugzilla.openvz.org/show_bug.cgi?id=2504

Could you, please, register in OpenVz Bugzilla and answer a few questions in this bug?
And next time create a bug report (probably even instead of sending a letter to users@ list).

04.02.2013 15:14, Konstantin Khorenko пишет:
> Stas,
>
> please take a look at the error reported.
> Is it clear what happens?
>
> Known issue or a new one?
>
> If new, please file a bug for it as well.
>
> Thank you.
>
>
> -------- Original Message --------
> Subject:     Fwd: [Users] Container networking broken after upgrade to 2.6.18-308.8.2.el5.028stab101.1, stange kernel errors. CPT ERR
> Date:     Fri, 1 Feb 2013 19:34:36 +0000
> From:     Kirill Korotaev <dev at parallels.com>
> To:     Kirill Kolyshkin <kir at parallels.com>
> CC:     Alexey Kobets <kobets at parallels.com>, Konstantin Khorenko <khorenko at parallels.com>
>
>
>
> Guys, can it be related to vz service trying to suspend CTs now by default?
> Kir, why you do not follow users@ mailing list at all? :(
>
>
> Begin forwarded message:
>
>> *From: *P J <pauljflists at gmail.com <mailto:pauljflists at gmail.com>>
>> *Subject: **[Users] Container networking broken after upgrade to 2.6.18-308.8.2.el5.028stab101.1, stange kernel errors. CPT ERR*
>> *Date: *January 30, 2013 12:55:26 PM MST
>> *To: *<users at openvz.org <mailto:users at openvz.org>>
>> *Reply-To: *OpenVZ users <users at openvz.org <mailto:users at openvz.org>>
>>
>> Hey Guys,
>>
>> I've spent days and countless hours trying to figure out what is going on here and I'm totally out of ideas, I've tried posting on the openvz forum but for
>> some reason the post is not approved, so now I'm reaching out to you all.
>>
>> I've even emailed OpenVZ for paid commercial support, I'm happy to pay for support for this issue as we have been using OpenVZ for many years problem free on
>> many servers.
>>
>> Here is the issue:
>>
>> One of oru CentOS 5.9 x86_64 servers, last weekend I upgraded the kernel from 2.6.18-194.8.1.el5.028stab070.4 to the latest
>> 2.6.18-308.8.2.el5.028stab101.1. (we use ksplice so I don't have to reboot the HN too often)
>>
>> Installed VZ packages:
>>
>> vzctl-core-4.1.2-1
>> ovzkernel-2.6.18-308.8.2.el5.028stab101.1
>> vzctl-4.1.2-1
>> ovzkernel-2.6.18-194.8.1.el5.028stab070.4
>> ovzkernel-devel-2.6.18-308.8.2.el5.028stab101.1
>> vzquota-3.1-1
>> ovzkernel-2.6.18-308.el5.028stab099.3
>> ovzkernel-2.6.18-238.9.1.el5.028stab089.1
>>
>> Upon booting the new 2.6.18-308.8.2.el5.028stab101.1 kernel, I started seeing strange kernel errors when starting the containers:
>>
>>
>> "Jan 26 20:46:06 vz02 kernel: CT: 103: started
>> Jan 26 20:46:08 vz02 kernel: CPT ERR: ffff81031e46a000,103 :NLMERR: -22
>> Jan 26 20:46:08 vz02 last message repeated 8 times
>> Jan 26 20:46:12 vz02 kernel: CT: 104: started
>> Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104 :open_listening_socket: sock_create_kern: -97
>> Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104 :rst_sockets: open_listening_socket: -97
>> Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104 :rst_sockets: -97
>> Jan 26 20:46:14 vz02 kernel: CT: 104: stopped
>> Jan 26 20:46:15 vz02 kernel: CT: 104: started
>> Jan 26 20:46:17 vz02 kernel: CT: 105: started
>> Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :NLMERR: -22
>> Jan 26 20:46:19 vz02 last message repeated 8 times
>> Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :open_listening_socket: sock_create_kern: -97
>> Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :rst_sockets: open_listening_socket: -97
>> Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :rst_sockets: -97
>> "
>>
>> --snip-- this goes on and on...
>>
>> The containers did start up, but networking was not working. I could not ping them, they could not ping out.
>> Their interfaces were up, most of them use venet.
>>
>> So I did a vzctl restart on each container, they threw out the same error messages, but networking started working again.
>>
>> What makes even less sense is, we have other identical servers, same hardware, same version of CentOS, same VZ kernel - no issues. What am I missing here?
>>
>> Here is a copy of our vz.conf, nothing fancy:
>>
>> "
>> VIRTUOZZO=yes
>> LOCKDIR=/vz/lock
>> DUMPDIR=/vz/dump
>> VE0CPUUNITS=1000
>>
>> ## Logging parameters
>> LOGGING=yes
>> LOGFILE=/var/log/vzctl.log
>> LOG_LEVEL=0
>> VERBOSE=0
>>
>> ## Disk quota parameters
>> DISK_QUOTA=yes
>> VZFASTBOOT=no
>>
>> # Disable module loading. If set, vz initscript do not load any modules.
>> #MODULES_DISABLED=yes
>>
>> # The name of the device whose IP address will be used as source IP for CT.
>> # By default automatically assigned.
>> #VE_ROUTE_SRC_DEV="eth0"
>>
>> # Controls which interfaces to send ARP requests and modify APR tables on.
>> NEIGHBOUR_DEVS=all
>> ERROR_ON_ARPFAIL="no"
>>
>>
>> ## Template parameters
>> TEMPLATE=/vz/template
>>
>> ## Defaults for containers
>> VE_ROOT=/vz/root/$VEID
>> VE_PRIVATE=/vz/private/$VEID
>> CONFIGFILE="vps.basic"
>> DEF_OSTEMPLATE="fedora-core-4"
>>
>> ## Load vzwdog module
>> VZWDOG="no"
>>
>> ## IPv4 iptables kernel modules
>> #IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ipt_state iptable_nat "
>>
>> IPTABLES="iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack
>> ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_MASQUERADE"
>>
>> ## Enable IPv6
>> IPV6="no"
>>
>> ## IPv6 ip6tables kernel modules
>> IP6TABLES="ip6_tables ip6table_filter ip6table_mangle ip6t_REJECT"
>> "
>>
>> Here is a example container config.
>>
>> "
>> ONBOOT="yes"
>>
>> NUMPROC="5102:5102"
>> AVNUMPROC="2551:2551"
>> NUMTCPSOCK="5102:5102"
>> NUMOTHERSOCK="5102:5102"
>> VMGUARPAGES="262144:9223372036854775807"
>>
>> # Secondary parameters
>> KMEMSIZE="209012940:229914234"
>> TCPSNDBUF="48773188:69670980"
>> TCPRCVBUF="48773188:69670980"
>> OTHERSOCKBUF="24386594:45284386"
>> DGRAMRCVBUF="24386594:24386594"
>> OOMGUARPAGES="151485:9223372036854775807"
>> PRIVVMPAGES="256000:262140"
>>
>> # Auxiliary parameters
>> LOCKEDPAGES="10205:10205"
>> SHMPAGES="90891:90891"
>> PHYSPAGES="0:9223372036854775807"
>> NUMFILE="81632:81632"
>> NUMFLOCK="1000:1100"
>> NUMPTY="510:510"
>> NUMSIGINFO="1024:1024"
>> DCACHESIZE="45650516:47020032"
>>
>> NUMIPTENT="3072:3072"
>> # Disk Resource Limits
>> DISKINODES="4560000:4800000"
>> DISKSPACE="39845888:41943040"
>>
>> # Quota Resource Limits
>> QUOTATIME="0"
>> QUOTAUGIDLIMIT="3000"
>>
>> # CPU Resource Limits
>> CPUUNITS="1000"
>> #RATE="eth0:1:6000"
>>
>> # IPTables config
>> IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ip_conntrack ip_conntrack_ftp
>> ipt_LOG ipt_conntrack ipt_helper ipt_state iptable_nat ip_nat_ftp ipt_TOS ipt_REDIRECT"
>>
>> # Default Devices
>> #DEVICES="c:10:229:rw c:10:200:rw "
>>
>> IP_ADDRESS="1.2.3.4"
>> HOSTNAME="nscache01.xxx.com <http://nscache01.xxx.com/>"
>> VE_ROOT="/vz/root/$VEID"
>> VE_PRIVATE="/vz/private/$VEID"
>> OSTEMPLATE="centos-5-x86_64"
>> ORIGIN_SAMPLE="512mb"
>> NAMESERVER="1.2.3.4"
>> NAME="nscache01-xxx"
>> "
>>
>> Any ideas? Configuration issue? Kernel bug?
>>
>> I also noticed a container that used to run OpenVPN no longer works, so even though networking is now "working" something is still going on...
>>
>> The one thing I did do before upgrading the kernel is I had to remove ploop as there were some dependency issues.
>> We don't use any ploop based containers so I don't believe this should affect us? We did the same on another server and had no issue...
>>
>> If anyone wants to respond directly to me that provides paid support, OpenVZ developers or anyone else I'm of course happy to pay, I don't expect someone to
>> spend hours troubleshooting something for free.
>>
>> Or, of course if any other list readers have any ideas please let me know! :)
>>
>> Thanks in advance for your help.
>>
>> -PJF
>> _______________________________________________
>> Users mailing list
>> Users at openvz.org <mailto:Users at openvz.org>
>> https://lists.openvz.org/mailman/listinfo/users
>
>
>

-- 
Best regards,
Stanislav Kinsbursky