[Users] Ploop Incremental backups strategy

Tue Sep 15 18:25:24 MSK 2020

Hello Steffan.

In order to download backups, you must configure OVZDB as follow:

SOURCE => SSH keys 1 => MASTER (where backups will be downloaded) => SSH 
keys 2 => REMOTE (where the backups are saved)

For instance:

MASTER_SSH_PATH="root at localhost:/home/backup/openvz-diff-backups" 
(backups will downloaded to "localhost")

REMOTE_SSH_PATH="root at 192.168.1.99:/home/backup/openvz-diff-backups" 
(the machine where backups are currently stored)

When you will download backups, OVZDB will first connects to "localhost" 
(to access MASTER "storage") and then will connect to 192.168.1.99 (to 
access REMOTE "storage").

I known this is counter intuitive but,

- when you make backups, OVZDB will store them on MASTER (ex: 192.168.1.99)

- when you download backups, the "previous" MASTER (192.168.1.99) must 
be configured as the REMOTE

Tonight I will try to post more infos.

Have a great day!

Le 15/09/2020 à 15:43, mailinglist at tikklik.nl a écrit :
> Hello tranxene 50,
>
> What i find out abouth the script is a strange (i think strange) loop
>
>  From my office i do a:
> openvz-diff-backups download 1 2020-09-15_03-42-51
>
> I see a SSH connection to the backupserver on the NOC
> but in the ssh log of the backupserver i see;
> refused connect from 127.0.0.1 (127.0.0.1)
>
> So the script is trying to make a new connection back to the server at my office, hat only will work if i give the server access to my office
> But why is it setting up another connection, from the server back tot he office
>
>
>
> -----Oorspronkelijk bericht-----
> Van: users-bounces at openvz.org <users-bounces at openvz.org> Namens mailinglist at tikklik.nl
> Verzonden: dinsdag 15 september 2020 11:20
> Aan: 'OpenVZ users' <users at openvz.org>
> Onderwerp: Re: [Users] Ploop Incremental backups strategy
>
> So the backup function tot he master backupserver works great.
> But looks like there is a bug somewhere.
>
> If from the live server i do
> openvz-diff-backups restore 1 list
> im getting a list of backups
> 1 - *Latest* backup - mode: live_plop - date: 2020-09-15_03-42-51 - task: backup - exit: success - keep: until it is deleted - host:
>
> But if i do:
> openvz-diff-backups download 1 2020-09-15_03-42-51
>
> im getting
> [2020-09-15 11:19:11] [     1] *Error*    - SSH error: connection failure (host: "192.168.1.99" -> "127.0.0.1" - error: "255") - please check SSH private/public keys
> [2020-09-15 11:19:11] [     1] *Notice*   - SSH error: unable to close remote SSH mux (host: "192.168.1.99" -> "127.0.0.1" - error: "255") - ignoring (please, do not panic)
>
>
> -----Oorspronkelijk bericht-----
> Van: users-bounces at openvz.org <users-bounces at openvz.org> Namens tranxene50
> Verzonden: zaterdag 12 september 2020 03:35
> Aan: users at openvz.org
> Onderwerp: Re: [Users] Ploop Incremental backups strategy
>
> Hello Steffan.
>
> To quickly answer your questions:
>
> 1) every successful backup is considered as a full backup, even if it is an "incremental/differential/full" backup under the hood (I know, this is counter intuitive)
>
> In short, you do not have to care about choosing which backup you want to restore because there is no computation to "rebuild" a backup.
>
> 2) "openvz-diff-backups --help" will show you all options
>
> You are right, the doc is missing: I have spent 99% of my time working on the code - and testing it - but only 1% to "build" a simple single website page ...
>
> 3) there is no intrinsic performance differences between push or pull
> modes: it only depends of the hardware that your are using
>
> But your question was about "external" servers having access to your backup server => use "pull" mode.
>
> This means that every client should save all backups on their own servers and, once it is done, you can download them on your backup server.
>
> If this is not possible, create a container to accept their backups
> (push) and another container to "pull" these backups.
>
> *WARNING AGAIN*: never, ever, store OVZDB backups in a container using Ploop layout: please, use bindfs to mount a directory from the host into the container
>
> Have a good night!
>
> Le 11/09/2020 à 16:05, tranxene50 a écrit :
>> Hello Steffan.
>>
>> I am writing a "survival" guide in English: please let me some time to
>> finish it. ;-)
>>
>> Have a nice day!
>>
>> Le 11/09/2020 à 15:53, mailinglist at tikklik.nl a écrit :
>>> Thanx, im looking in to it, it looks very nice Im missing some
>>> documentation so i have some quastions...
>>>
>>> 1)
>>> So if you take live backups
>>> Then the seccond backup is incremental any you can in case of
>>> emergancy take the last incremental backup to restore the whole CT?
>>>
>>> 2) in your cron you use this, " -l 6 -q" can you explain what that does?
>>>
>>> 3)Push and pull
>>> I miss documentation how to setup.
>>> Have you any experiance in performance on push vs pull On my own
>>> nodes i can install the script on the live node But i also have
>>> clients with nodes i backup, but i dont want them to give access
>>> directly tot he backup server.
>>>
>>> 4) at night i want to backup, from tha main backup server to a remote
>>> sitelocation
>>>       In case of a big desaster 😊
>>>      what do i need tor sync?
>>>      Everything with a date file/map name?
>>>     and include the data dir?
>>>
>>> Mayby for the future the option to pull the last update to a remote
>>> server 😊
>>>
>>> Thanxs for the time.
>>>
>>> Steffan
>>>
>>> -----Oorspronkelijk bericht-----
>>> Van: users-bounces at openvz.org <users-bounces at openvz.org> Namens
>>> tranxene50
>>> Verzonden: woensdag 9 september 2020 01:02
>>> Aan: users at openvz.org
>>> Onderwerp: Re: [Users] Ploop Incremental backups strategy
>>>
>>> Hello!
>>>
>>> Please forgive my bad English, I live in France.
>>>
>>> A few years ago, as a hobby in the beginning, I created a file based
>>> "incremental" backup tool (that heavily relies on rsync) :
>>> https://www.openvz-diff-backups.fr
>>>
>>> It works flawlessly with OpenVZ 6 but only have been rapidly tested
>>> with OpenVZ 7.
>>>
>>> The main difference between OpenVZ 6 and 7 is that memory dump
>>> (checkpoint) is now done using CRIU instead of OpenVZ "Legacy" kernel.
>>>
>>> But,  globally, the process is still the same: create a ploop
>>> snapshot, mount it, sync it (with rsync) and then create a "diff"
>>> backup (using rsync again with --link-dest).
>>>
>>> As far as I know, this is one of the rarest GPL tools (any hint
>>> appreciated!) able to backup/restore CT files and, most importantly,
>>> full memory state.
>>>
>>> Restoring a "live" backup is like resuming an OS (Windows, Linux,
>>> MacOS,
>>> etc) after it had been put to sleep: the container will resume and
>>> works again, just like nothing had happen.
>>>
>>> So you can cheat and pretend 100% uptime even if a container was down
>>> most of the time... (evidently this a joke and, please, do not play
>>> at this game: /var/log/* will betray you)
>>>
>>> Note: to migrate CT between OpenVZ 6 and 7, you must use "cold"
>>> backups because memory dumps are incompatible (OpenVZ Kernel vs CRIU).
>>>
>>> To answer your questions:
>>>
>>> 1) you can restore any previous backup because each one is considered
>>> as a full backup (no diff/incremental computing: it is just
>>> files/directories/other - and hard links)
>>>
>>> 2) because backups are just a bunch of files (and mostly hard links),
>>> you can easily browse any backup of a CT and copy any
>>> files/directories needed
>>>
>>> At the moment, development of openvz-diff-backups is on pause -
>>> because it fulfills all my needs with OpenVZ 6 - but I am in the
>>> process of moving to OpenVZ 7 in a few months.
>>>
>>> So, if you encounter a bug or an issue, please leave me a message:
>>> the tool has a very conservative approach and is designed to cleanly
>>> stop if anything unexpected/unknown/abnormal happens.
>>>
>>> Have a nice day!
>>>
>>> Le 08/09/2020 à 14:24, mailinglist at tikklik.nl a écrit :
>>>> Hello,
>>>>
>>>> Using openvz7
>>>> Im looking ad my backup strategy
>>>> I allways used rsync on openvz6
>>>> And looks like on openvz7 this can also be done on /vz/root/VEID But
>>>> i dont know if i can do a full restore from that...
>>>>
>>>>
>>>> Now im looking at snapshots
>>>> https://github.com/TamCore/vzpbackup
>>>> it can make full backups and incremental backups
>>>>
>>>> Is someone using this script?
>>>> i have some question hope someone can help
>>>>
>>>> the incremental backups are nice to save space and time on a remote
>>>> backupserver But how does a restore works.
>>>> For what i know you can only restore a full snapshot, the
>>>> incremental backups have only the changed files
>>>>
>>>> Question 2.
>>>> Is it possible to extract a file from the backup for a single file
>>>> restore?
>>>> And if so can someone tell me how?
>>>>
>>>>
>>>> Thanxs
>>>> Steffan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at openvz.org
>>>> https://lists.openvz.org/mailman/listinfo/users
>>> --
>>> tranxene50
>>> tranxene50 at openvz-diff-backups.fr
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at openvz.org
>>> https://lists.openvz.org/mailman/listinfo/users
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at openvz.org
>>> https://lists.openvz.org/mailman/listinfo/users
> --
> tranxene50
> tranxene50 at openvz-diff-backups.fr
>
> _______________________________________________
> Users mailing list
> Users at openvz.org
> https://lists.openvz.org/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> Users at openvz.org
> https://lists.openvz.org/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> Users at openvz.org
> https://lists.openvz.org/mailman/listinfo/users

-- 
tranxene50
tranxene50 at openvz-diff-backups.fr