[CRIU] [PATCHv2 1/2] net: add nftables c/r

[Alexander Mikhalitsyn]

On Thu, 14 Nov 2019 20:01:09 +0300
Kirill Gorkunov <gorcunov at virtuozzo.com> wrote:

> On Thu, Nov 14, 2019 at 11:32:05AM +0300, Cyrill Gorcunov wrote:
> >
> > Still if Alexander would be able to estimate what we need to fetch by hands
> > this would be great!
> 
> Been talking to Andrew f2f so we've got a consensus -- for vanilla criu
> we definitely should use netlink protocol and fetch rulesets directly,
> which should not be that hard but maybe a little time consuming task.
> For vz7 instance we should figure out how urgent the support for nft
> rules is needed and in case of real need we could make a faster fix
> via execve and then once vanilla variant is ready -- switch to netlink.
Dear friends,

I've made some investigation and tried to write initial implementation of nftables dump/restore using libnftnl. I would like to tell you what I did and what conclusions I came to:

I) It's *not* simple. Why?

In nft we operate the concept of "ruleset" (table, chains, rules, sets). We need to dump ruleset i.e. all tables, all chains (that assigned to some table), all rules (that assigned to some chain), all sets. To do that we need to form sequence of NETLINK messages: NFT_MSG_GETTABLE, NFT_MSG_GETCHAIN, NFT_MSG_GETRULE, NFT_MSG_GETSET (and for each set element NFT_MSG_GETSETELEM). To each of this requests kernel respond with special NETLINK messages that describes table, rule, etc. But. We *can't* simply save this binary netlink response messages and send that when needed to recreate for example table of rule! Why? Because this message contains more data, that we need to restore structure (message type ID the same when we get table or when we create table - that's ok).

Consider example (dump/creation of the same table)

Message that we can construct from netlink response to GETTABLE request:
(I've only changed flags to R (request) and A (ack))
----------------	------------------
|  0000000068  |	| message length |
| 02560 | R-A- |	|  type | flags  |
|  1574068571  |	| sequence number|
|  0000004410  |	|     port ID    |
----------------	------------------
|00002|-B|01024|	|len |flags| type|
|00017|--|00001|	|len |flags| type|
| 74 65 73 74  |	|      data      |	 t e s t
| 31 32 33 33  |	|      data      |	 1 2 3 3
| 34 34 73 64  |	|      data      |	 4 4 s d
| 00 00 00 00  |	|      data      |	        
|00008|--|00002|	|len |flags| type|
| 00 00 00 00  |	|      data      |	        
|00008|--|00003|	|len |flags| type|
| 00 00 00 00  |	|      data      |	        
|00012|--|00004|	|len |flags| type|
| 00 00 00 00  |	|      data      |	        
| 00 00 00 24  |	|      data      |	       $
----------------	------------------

This is the *same* message that I construct by using libnftnl
----------------	------------------
|  0000000040  |	| message length |
| 02560 | R-A- |	|  type | flags  |
|  1574068472  |	| sequence number|
|  0000004217  |	|     port ID    |
----------------	------------------
|00017|--|00001|	|len |flags| type|
| 74 65 73 74  |	|      data      |	 t e s t
| 31 32 33 33  |	|      data      |	 1 2 3 3
| 34 34 73 64  |	|      data      |	 4 4 s d
| 00 00 00 00  |	|      data      |	        
----------------	------------------

As you see in first message we have some additional data - it's for example "kernel handle" of table, and other attributes. Idealy, after GETTABLE request I need to do several things with response:
1. change flags to R A (request, ack).
2. Obtain and remove extra attributes (NFTA_TABLE_USE attribute for table and so on)

II) Approach, described in previous point looks ugly for me. Why? Because I need to make some changes in raw netlink messages and it may provoke bugs and so on.

But...

III) I think that we have two possible solutions to this problem:
1. We could try some hack -> in libnftnl we have several helper functions that could convert raw netlink messages to libnftnl-defined structures (struct nftnl_table, struct nftnl_chain, etc). We can parse response netlink messages to netlink structures, then vice versa construct payload to create table (chain, rule, set)! :) Looks a bit uggly. But it's better than make some changes in binary NETLINK messages by hands. A? When we use this way all extra attributes will be stripped! Look on nftnl_table_nlmsg_build_payload() function from libnftnl (file table.c) and compare it to nftnl_table_parse_attr_cb() for instance.
2. Another option is to make new feature in libnftnl library to dump ruleset to binary file (or json/xml) and then reconstruct ruleset from file. It's hard. But in this case we make some good feature in libnftnl and guys from netlink will support this code.

I would like to hear your opinion and advices. :)

Thank you
Regards, Alex