[CRIU] Failing second checkpoint with iptables-restore v1.8.0 (nf_tables)

[Adrian Reber]

On Thu, Sep 06, 2018 at 06:04:13PM +0200, Adrian Reber wrote:
> On Thu, Sep 06, 2018 at 10:38:01AM +0200, Adrian Reber wrote:
> > On Wed, Sep 05, 2018 at 04:14:04PM +0200, Adrian Reber wrote:
> > > On Wed, Sep 05, 2018 at 11:14:14AM +0200, Adrian Reber wrote:
> > > > I got a report of a checkpoint failure that is happening when dumping a
> > > > already restored runc container.
> > > > 
> > > > On the second 'runc checkpoint' the log says:
> > > > 
> > > > ip6tables-restore v1.8.0 (nf_tables): 
> > > > line 2: CHAIN_USER_FLUSH failed (Device or resource busy): chain CRIU
> > > > line 2: CHAIN_USER_ADD failed (File exists): chain CRIU
> > > > (00.132118) Error (criu/util.c:811): exited, status=4
> > > > (00.132144) Unfreezing tasks into 1
> > > > (00.132154) 	Unseizing 15729 into 1
> > > > (00.132175) Error (criu/cr-dump.c:1720): Dumping FAILED.
> > > > 
> > > > This is criu 3.10 on a 4.18 kernel. This is the first time I am seeing a
> > > > system with 'iptables-restore v1.8.0 (nf_tables)'. Not sure if that is
> > > > related.
> > > > 
> > > > I cannot reproduce it with runc currently. So right now I just wanted to
> > > > reach it out if this is something anybody has already seen.
> > > 
> > > I was able to reproduce this error and it is related to iptables 1.8.0
> > > which has a multi call binary:
> > > 
> > > /usr/sbin/iptables-restore -> xtables-nft-multi
> > > 
> > > and the command line options of this iptables-restore program are
> > > different (and with less features) than the iptables-restore from 1.6.*.
> > > 
> > > I started to talk with one of the iptables maintainers.
> > 
> > Upstream provided a fix:
> > 
> > https://marc.info/?l=netfilter-devel&m=153616769327109&w=2
> > 
> > But it still fails. I tried to simulate what CRIU does and I think it is
> > something like this:
> > 
> > echo -e "*filter\n:CRIU - [0:0]\n-I INPUT -j CRIU\n-A CRIU -m mark --mark 42 -j ACCEPT\nCOMMIT\n" | iptables-restore -w 1 --noflush
> > 
> > With iptables 1.4.x and 1.6.x I can run the command multiple times and
> > it just works. It keeps on adding the same rules to iptables.
> > 
> > With iptables 1.8.0 this fails when running the second time with:
> > 
> > iptables-restore v1.8.0 (nf_tables): 
> > line 2: CHAIN_USER_FLUSH failed (Device or resource busy): chain CRIU
> > line 2: CHAIN_USER_ADD failed (File exists): chain CRIU
> > 
> > But CRIU does not seem to add the rules twice, so it is unclear why I
> > get the same error as adding the rules twice, although CRIU adds the
> > rules only once.
> > 
> > Is this related to some iptables locking? Maybe someone who still
> > remembers the code can help out.
> 
> So I really love answering my own emails ;)
> 
> Another upstream iptables patch which brings us closer to a working
> iptables-restore again:
> 
> https://marc.info/?l=netfilter-devel&m=153624401918001&w=2
> 
> Still not fixed completely as I still get errors from iptables, but we
> are getting closer.

There is a V2 of the patch above:

https://marc.info/?l=netfilter-devel&m=153625521421967&w=2

And now 'runc checkpoint' works again.

		Adrian