[Users] Some details of vSwap implementation in Virtuozzo 7

[Konstantin Khorenko]

On 07/21/2020 06:55 PM, Daniel Pearson wrote:
> Thanks for posting this, I have some doubts on your explanation along
> with some information we received in tickets as well. For clarification
> we primarily run web-server VM's.
>
> We do not use swappages, this is set 0:0 within the containers. However,
> over a long period of time (100~ days) regardless of the kernel we use,
> we get very odd processes that drop into the node level swap regardless
> of free memory.

Hi Daniel,

OK, and this is completely unrelated to vSwap.
You don't have vSwap configured on your nodes (don't use swappages for Containers),

(you can check if vSwap is used on a node at all:
     # grep Tswap /proc/meminfo
     Tswap:                 0 kB)

thus your question relates to a generic memory managing algorithm of a Hardware Node.

And yes, it's completely fine if mm decides to move some anonymous memory into swap
(physical!, no vSwap here), and use RAM for more caches instead.

> So during these 100 days available memory has never gone to zero, nor
> has it gone to the point where this many processes should swap. However,
> we get 20GB worth of swap, the majority of which end up being very
> active applications such as MySQL, httpd processes,

And here comes your real question - why active applications might get into the swap?
(leaving apart the question if those applications are really _always_ active,
because if the are not active _for some time_, then no surprise)

One of very common reason is some process - does not matter on the host on in a Container,
which generates _a lot of_ pagecache.
Most often it's some kind of homebrew backup processes - just because it's their job to read
files from disk while performing a backup - and thus they generate a lot of pagecache.

Once again - it's possible such processes are run on the host (so hoster performs backups)
or in a Container (CT owner performs backups on his own).

And mm does not know if this pagecache is "useful" and will be used by someone a bit later or not,
and by default considers it as useful. And if your disks are very fast, new pagecache is generated
so fast that even those application you consider as "active" are not active enough from algorithm's
point of view and thus pages from these processes might go into swap.

What can be done with such "backup" processes?

1) if those processes are run on the host:
    a) rewrite backups not to generate pagecache.
       For example:
       # dd if=/vz/private/123/root.hdd/root.hds iflag=direct of=- | gzip
       (note, i'm not talking now about the consistency, that you must backup snapshots, not live image, etc.
        It's a different theme. Use commercial Virtuozzo backups at last :). )

    b) if you cannot rewrite backup software, run backup in a separate memory cgroup
       and limit pagecache for this cgroup.
       This is a Virtuozzo specific memcg limit, mainstream does not have it => LXC/Proxmox too.
       We've implemented this feature especially to fight with such processes which generate a lot of pagecache by their nature.

       memcg::memory.cache.limit_in_bytes

2) if those processes are run by a Container owner:
    a) you can limit the io/iops for the CT (thus pagecache is generated slower and active processes
       are considered as active enough not to get into swap)
    b) you can disable tcache for this Container memcg::memory.disable_cleancache
       (raise your hand if you wish me to explain what tcache is)
    c) you can limit the max amount of memory which can be used for pagecache for this Container
       memcg::memory.cache.limit_in_bytes


These are my guesses surely, but this is the most often reason.

> <skipped>
>
> "b) if there is free memory on the host, the Container's memory is saved
> in a special swap cache in host's RAM and no real write to host's
> physical swap occurs"
>
> But how can this be shown or proven? I do not believe this bit functions
> at all on the nodes we run.

Yes, i think vSwap does not work on your nodes if no swap is configured for Containers.
How to check the number of tswap pages at the moment on the Hardware Node overall
(tswap - is a backend for vSwap, so these are exactly pages which were going to be put into physical swap from Containers,
but were put into "tswap" - a special cache in Hardware Node RAM)

# grep Tswap /proc/meminfo

If you wish to check if vSwap works at all (i mean - if it avoids physical swap io if there is free RAM on the node),
* you can take an idle node for the experiment with, say 3X RAM,
* create a Container with, say, X RAM + X vSwap,
* run a memory eater process for 2X RAM in the Container,
* check on the node the io in block device with swap (say, in /proc/diskstats)

Hope that helps.

--
Best regards,

Konstantin Khorenko,
Virtuozzo Linux Kernel Team