[Users] Container networking broken after upgrade to 2.6.18-308.8.2.el5.028stab101.1, stange kernel errors. CPT ERR

[P J]

On Wed, Jan 30, 2013 at 11:55 AM, P J <pauljflists at gmail.com> wrote:

> Hey Guys,
>
> I've spent days and countless hours trying to figure out what is going on
> here and I'm totally out of ideas, I've tried posting on the openvz forum
> but for some reason the post is not approved, so now I'm reaching out to
> you all.
>
> I've even emailed OpenVZ for paid commercial support, I'm happy to pay for
> support for this issue as we have been using OpenVZ for many years problem
> free on many servers.
>
> Here is the issue:
>
> One of oru CentOS 5.9 x86_64 servers, last weekend I upgraded the kernel
> from 2.6.18-194.8.1.el5.028stab070.4 to the latest
> 2.6.18-308.8.2.el5.028stab101.1. (we use ksplice so I don't have to reboot
> the HN too often)
>
> Installed VZ packages:
>
> vzctl-core-4.1.2-1
> ovzkernel-2.6.18-308.8.2.el5.028stab101.1
> vzctl-4.1.2-1
> ovzkernel-2.6.18-194.8.1.el5.028stab070.4
> ovzkernel-devel-2.6.18-308.8.2.el5.028stab101.1
> vzquota-3.1-1
> ovzkernel-2.6.18-308.el5.028stab099.3
> ovzkernel-2.6.18-238.9.1.el5.028stab089.1
>
> Upon booting the new 2.6.18-308.8.2.el5.028stab101.1 kernel, I started
> seeing strange kernel errors when starting the containers:
>
>
> "Jan 26 20:46:06 vz02 kernel: CT: 103: started
> Jan 26 20:46:08 vz02 kernel: CPT ERR: ffff81031e46a000,103 :NLMERR: -22
> Jan 26 20:46:08 vz02 last message repeated 8 times
> Jan 26 20:46:12 vz02 kernel: CT: 104: started
> Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104
> :open_listening_socket: sock_create_kern: -97
> Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104 :rst_sockets:
> open_listening_socket: -97
> Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104 :rst_sockets:
> -97
> Jan 26 20:46:14 vz02 kernel: CT: 104: stopped
> Jan 26 20:46:15 vz02 kernel: CT: 104: started
> Jan 26 20:46:17 vz02 kernel: CT: 105: started
> Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :NLMERR: -22
> Jan 26 20:46:19 vz02 last message repeated 8 times
> Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105
> :open_listening_socket: sock_create_kern: -97
> Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :rst_sockets:
> open_listening_socket: -97
> Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :rst_sockets:
> -97
> "
>
> --snip-- this goes on and on...
>
> The containers did start up, but networking was not working. I could not
> ping them, they could not ping out.
> Their interfaces were up, most of them use venet.
>
> So I did a vzctl restart on each container, they threw out the same error
> messages, but networking started working again.
>
> What makes even less sense is, we have other identical servers, same
> hardware, same version of CentOS, same VZ kernel - no issues. What am I
> missing here?
>
> Here is a copy of our vz.conf, nothing fancy:
>
> "
> VIRTUOZZO=yes
> LOCKDIR=/vz/lock
> DUMPDIR=/vz/dump
> VE0CPUUNITS=1000
>
> ## Logging parameters
> LOGGING=yes
> LOGFILE=/var/log/vzctl.log
> LOG_LEVEL=0
> VERBOSE=0
>
> ## Disk quota parameters
> DISK_QUOTA=yes
> VZFASTBOOT=no
>
> # Disable module loading. If set, vz initscript do not load any modules.
> #MODULES_DISABLED=yes
>
> # The name of the device whose IP address will be used as source IP for CT.
> # By default automatically assigned.
> #VE_ROUTE_SRC_DEV="eth0"
>
> # Controls which interfaces to send ARP requests and modify APR tables on.
> NEIGHBOUR_DEVS=all
> ERROR_ON_ARPFAIL="no"
>
>
> ## Template parameters
> TEMPLATE=/vz/template
>
> ## Defaults for containers
> VE_ROOT=/vz/root/$VEID
> VE_PRIVATE=/vz/private/$VEID
> CONFIGFILE="vps.basic"
> DEF_OSTEMPLATE="fedora-core-4"
>
> ## Load vzwdog module
> VZWDOG="no"
>
> ## IPv4 iptables kernel modules
> #IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter
> iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ipt_state
> iptable_nat "
>
> IPTABLES="iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos
> ipt_TOS ipt_REJECT ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_LOG ipt_length
> ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state
> ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_MASQUERADE"
>
> ## Enable IPv6
> IPV6="no"
>
> ## IPv6 ip6tables kernel modules
> IP6TABLES="ip6_tables ip6table_filter ip6table_mangle ip6t_REJECT"
> "
>
> Here is a example container config.
>
> "
> ONBOOT="yes"
>
> NUMPROC="5102:5102"
> AVNUMPROC="2551:2551"
> NUMTCPSOCK="5102:5102"
> NUMOTHERSOCK="5102:5102"
> VMGUARPAGES="262144:9223372036854775807"
>
> # Secondary parameters
> KMEMSIZE="209012940:229914234"
> TCPSNDBUF="48773188:69670980"
> TCPRCVBUF="48773188:69670980"
> OTHERSOCKBUF="24386594:45284386"
> DGRAMRCVBUF="24386594:24386594"
> OOMGUARPAGES="151485:9223372036854775807"
> PRIVVMPAGES="256000:262140"
>
> # Auxiliary parameters
> LOCKEDPAGES="10205:10205"
> SHMPAGES="90891:90891"
> PHYSPAGES="0:9223372036854775807"
> NUMFILE="81632:81632"
> NUMFLOCK="1000:1100"
> NUMPTY="510:510"
> NUMSIGINFO="1024:1024"
> DCACHESIZE="45650516:47020032"
>
> NUMIPTENT="3072:3072"
> # Disk Resource Limits
> DISKINODES="4560000:4800000"
> DISKSPACE="39845888:41943040"
>
> # Quota Resource Limits
> QUOTATIME="0"
> QUOTAUGIDLIMIT="3000"
>
> # CPU Resource Limits
> CPUUNITS="1000"
> #RATE="eth0:1:6000"
>
> # IPTables config
> IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter
> iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ip_conntrack
> ip_conntrack_ftp ipt_LOG ipt_conntrack ipt_helper ipt_state iptable_nat
> ip_nat_ftp ipt_TOS ipt_REDIRECT"
>
> # Default Devices
> #DEVICES="c:10:229:rw c:10:200:rw "
>
> IP_ADDRESS="1.2.3.4"
> HOSTNAME="nscache01.xxx.com"
> VE_ROOT="/vz/root/$VEID"
> VE_PRIVATE="/vz/private/$VEID"
> OSTEMPLATE="centos-5-x86_64"
> ORIGIN_SAMPLE="512mb"
> NAMESERVER="1.2.3.4"
> NAME="nscache01-xxx"
> "
>
> Any ideas? Configuration issue? Kernel bug?
>
> I also noticed a container that used to run OpenVPN no longer works, so
> even though networking is now "working" something is still going on...
>
> The one thing I did do before upgrading the kernel is I had to remove
> ploop as there were some dependency issues.
> We don't use any ploop based containers so I don't believe this should
> affect us? We did the same on another server and had no issue...
>
> If anyone wants to respond directly to me that provides paid support,
> OpenVZ developers or anyone else I'm of course happy to pay, I don't expect
> someone to spend hours troubleshooting something for free.
>
> Or, of course if any other list readers have any ideas please let me know!
> :)
>
> Thanks in advance for your help.
>
> -PJF
>

I posted on OpenVZ's bugzilla last week, was hoping for a response from
someone on the list on maybe where I should be looking.

Nobody has any ideas?

Most importantly what the:

Jan 26 20:46:08 vz02 kernel: CPT ERR: ffff81031e46a000,103 :NLMERR: -22
Jan 26 20:46:08 vz02 last message repeated 8 times
Jan 26 20:46:12 vz02 kernel: CT: 104: started
Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104
:open_listening_socket: sock_create_kern: -97
Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104
:rst_sockets: open_listening_socket: -97
Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104 :rst_sockets: -97

Errors are?

Thanks in advance.

-PJF
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openvz.org/pipermail/users/attachments/20130204/20f73e9c/attachment.html>