[Users] ploop snapshots are dangerous
Gena Makhomed
gmm at csdoc.com
Mon Feb 1 19:05:44 MSK 2021
Hello All,
ploop snapshots are dangerous - they can lead to data loss
if partition there ploop file located has no free space.
Detailed information:
I use file-based backups technology for ploop backups.
https://wiki.openvz.org/Ploop/Backup#File-based_backup
Backup process:
# Take a snapshot without suspending a CT and saving its config
vzctl snapshot $CTID --id $ID --skip-suspend --skip-config
# Mount the snapshot taken
vzctl snapshot-mount $CTID --id $ID --target $MNTDIR
# Perform a backup using your favorite backup tool
(using rsync)
# Unmount the snapshot
vzctl snapshot-umount $CTID --id $ID
# Delete (merge) the snapshot
vzctl snapshot-delete $CTID --id $ID
for $CTID container ploop file has size 2.7 TiB.
After the moment of starting backup process -
snapshot file grow in size to 500+ GiB.
Backup process of this container take more then 24 hours.
And then I run command
# Delete (merge) the snapshot
vzctl snapshot-delete $CTID --id $ID
free size on the partition very quickly changed to 0.
and as result - ploop file leaved in inconsistent state.
and as result all data from this ploop and snapshot was lost.
and I can't rescue this container data in such situation.
ploop snapshots are dangerous - they can lead to data loss.
===========================================================
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/lv-vz 3.6T 3.6T 0 100% /vz
kernel: [6621861.851163] ploop64503: host disk is almost full
(3985838080 < 4294971392); CT sees -ENOSPC !
kernel: [6622046.948480] dio_submit_alloc() 683 ploop51810 set error -28
kernel: [6622046.948649] Pid: 4963, comm: ploop51810 veid: 0 Not tainted
2.6.32-042stab136.1 #1
kernel: [6622046.948999] Call Trace:
kernel: [6622046.949175] [<ffffffffa064ab62>] ?
dio_submit_alloc+0x502/0x830 [pio_direct]
kernel: [6622046.949497] [<ffffffff8100980f>] ? __switch_to+0xcf/0x470
kernel: [6622046.949664] [<ffffffff813be610>] ?
scsi_requeue_run_queue+0x0/0x20
kernel: [6622046.949833] [<ffffffff813be610>] ?
scsi_requeue_run_queue+0x0/0x20
kernel: [6622046.950001] [<ffffffffa063f036>] ?
ploop1_allocate+0x46/0x8c [pfmt_ploop1]
kernel: [6622046.950196] [<ffffffffa0619167>] ?
ploop_req_state_process+0x407/0xa60 [ploop]
kernel: [6622046.950516] [<ffffffffa061dbda>] ?
ploop_thread+0x1ca/0x7c0 [ploop]
kernel: [6622046.950685] [<ffffffff81563a7b>] ? schedule+0x5ab/0xe60
kernel: [6622046.950853] [<ffffffff810b4d80>] ?
autoremove_wake_function+0x0/0x40
kernel: [6622046.951023] [<ffffffffa061da10>] ? ploop_thread+0x0/0x7c0
[ploop]
kernel: [6622046.951209] [<ffffffff810b4980>] ? kthread+0xa0/0xc0
kernel: [6622046.951377] [<ffffffff815708d0>] ? child_rip+0x20/0x30
kernel: [6622046.951542] [<ffffffff810b48e0>] ? kthread+0x0/0xc0
kernel: [6622046.951707] [<ffffffff815708b0>] ? child_rip+0x0/0x30
kernel: [6622046.951871] No space left on device! Either free some space
on disk or abort ploop51810 manually.
kernel: [6622056.943265] dio_submit_alloc() 683 ploop51810 set error -28
...................
kernel: [6637722.768763] EXT4-fs error (device ploop51810p1):
__ext4_get_inode_loc: unable to read inode block - inode=8659230,
block=34603569
kernel: [6637722.768826] EXT4-fs error (device ploop51810p1):
ext4_lookup: deleted inode referenced: 8784049
kernel: [6637722.784161] EXT4-fs error (device ploop51810p1):
ext4_lookup: deleted inode referenced: 8784023
kernel: [6637722.784172] EXT4-fs error (device ploop51810p1):
ext4_lookup: deleted inode referenced: 8783937
kernel: [6637722.791399] EXT4-fs error (device ploop51810p1):
ext4_dx_find_entry: bad entry in directory #37879916: rec_len % 4 != 0 -
block=151527766offset=0(81920), inode=17236994, rec_len=1026, name_len=7
kernel: [6637722.792023] EXT4-fs error (device ploop51810p1):
ext4_dx_find_entry: bad entry in directory #37879916: rec_len % 4 != 0 -
block=151529532offset=0(323584), inode=2156920843, rec_len=1922, name_len=10
kernel: [6637722.792602] EXT4-fs error (device ploop51810p1):
ext4_dx_find_entry: bad entry in directory #37879916: rec_len % 4 != 0 -
block=151527778offset=0(131072), inode=18678797, rec_len=1037, name_len=29
kernel: [6637722.795204] EXT4-fs error (device ploop51810p1):
ext4_lookup: deleted inode referenced: 12062682
kernel: [6637722.795284] EXT4-fs error (device ploop51810p1):
ext4_lookup: deleted inode referenced: 8783957
kernel: [6637722.795316] EXT4-fs error (device ploop51810p1):
ext4_lookup: deleted inode referenced: 8783959
kernel: [6637722.797955]
kernel: [6637722.798173] EXT4-fs error (device ploop51810p1):
ext4_dx_find_entry: bad entry in directory #37879916: rec_len % 4 != 0 -
block=151527573offset=0(40960), inode=914454813, rec_len=29569, name_len=129
kernel: [6637722.798728] EXT4-fs error (device ploop51810p1):
ext4_dx_find_entry: bad entry in directory #37879916: rec_len % 4 != 0 -
block=151527569offset=0(24576), inode=2170257725, rec_len=31523,
name_len=129
kernel: [6637722.799308] EXT4-fs error (device ploop51810p1):
ext4_dx_find_entry: bad entry in directory #37879916: rec_len % 4 != 0 -
block=151527779offset=1028(136196), inode=17761286, rec_len=1030,
name_len=15
kernel: [6637722.799937] EXT4-fs error (device ploop51810p1):
ext4_dx_find_entry: bad entry in directory #37879916: rec_len % 4 != 0 -
block=151527806offset=3116(220204), inode=18940943, rec_len=1039,
name_len=33
kernel: [6637722.800512] EXT4-fs error (device ploop51810p1):
ext4_dx_find_entry: bad entry in directory #37879916: directory entry
across blocks - block=151527765offset=0(77824), inode=234954500,
rec_len=7940, name_len=1
kernel: [6637722.801094] EXT4-fs error (device ploop51810p1):
ext4_dx_find_entry: bad entry in directory #37879916: rec_len % 4 != 0 -
block=151527565offset=0(8192), inode=2172331393, rec_len=33086, name_len=84
kernel: [6637722.801732] EXT4-fs error (device ploop51810p1):
ext4_dx_find_entry: bad entry in directory #37879916: rec_len % 4 != 0 -
block=151527565offset=0(8192), inode=2172331393, rec_len=33086, name_len=84
......................................
kernel: [6637723.689638] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 3410050, running e2fsck is recommended.
kernel: [6637723.689971] EXT4-fs warning (device ploop51810p1):
dx_probe: dx entry: limit != root limit
kernel: [6637723.690296] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 3410050, running e2fsck is recommended.
kernel: [6637723.690631] EXT4-fs warning (device ploop51810p1):
dx_probe: dx entry: limit != root limit
kernel: [6637723.690954] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 3410050, running e2fsck is recommended.
kernel: [6637723.691293] EXT4-fs warning (device ploop51810p1):
dx_probe: dx entry: limit != root limit
kernel: [6637723.691616] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 3410050, running e2fsck is recommended.
kernel: [6637723.691950] EXT4-fs warning (device ploop51810p1):
dx_probe: dx entry: limit != root limit
kernel: [6637723.692275] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 3410050, running e2fsck is recommended.
kernel: [6637723.692610] EXT4-fs warning (device ploop51810p1):
dx_probe: dx entry: limit != root limit
kernel: [6637723.692932] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 3410050, running e2fsck is recommended.
kernel: [6637723.693265] EXT4-fs warning (device ploop51810p1):
dx_probe: dx entry: limit != root limit
kernel: [6637723.693588] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 3410050, running e2fsck is recommended.
kernel: [6637723.693922] EXT4-fs warning (device ploop51810p1):
dx_probe: dx entry: limit != root limit
kernel: [6637723.694244] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 3410050, running e2fsck is recommended.
....................
kernel: [6637725.783053] EXT4-fs error (device ploop51810p1):
ext4_readdir: bad entry in directory #21758770: rec_len is smaller than
minimal - block=87040442offset=0(0), inode=0, rec_len=0, name_len=0
kernel: [6637725.783667] EXT4-fs warning (device ploop51810p1):
dx_probe: dx entry: limit != root limit
kernel: [6637725.783989] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 21496939, running e2fsck is recommended.
kernel: [6637725.788231] EXT4-fs error (device ploop51810p1):
ext4_readdir: bad entry in directory #21496939: rec_len is smaller than
minimal - block=85992025offset=0(0), inode=0, rec_len=0, name_len=0
kernel: [6637725.788841] EXT4-fs warning (device ploop51810p1):
dx_probe: dx entry: limit != root limit
kernel: [6637725.789163] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 3541157, running e2fsck is recommended.
kernel: [6637725.789494] EXT4-fs error (device ploop51810p1):
ext4_readdir: bad entry in directory #3541157: rec_len is smaller than
minimal - block=14165118offset=0(0), inode=0, rec_len=0, name_len=0
kernel: [6637725.790009] EXT4-fs warning (device ploop51810p1):
dx_probe: dx entry: limit != root limit
kernel: [6637725.790331] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 11798733, running e2fsck is recommended.
....................
kernel: [6637738.990054] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
kernel: [6637738.990393]
kernel: [6637743.986582] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
kernel: [6637743.986904]
kernel: [6637748.983077] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
kernel: [6637748.983400]
kernel: [6637752.339337] EXT4-fs (ploop51810p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 37880203; err -30
kernel: [6637752.339659]
kernel: [6637752.339814] EXT4-fs (ploop51810p1): ext4_da_writepages:
jbd2_start: 8192 pages, ino 37881592; err -30
kernel: [6637752.340135]
kernel: [6637752.340290] EXT4-fs (ploop51810p1): ext4_da_writepages:
jbd2_start: 8192 pages, ino 37881500; err -30
kernel: [6637752.340615]
kernel: [6637753.979592] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
kernel: [6637753.979915]
kernel: [6637758.976072] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
..................................................
kernel: [6643633.255145] EXT4-fs warning (device ploop51810p1):
dx_probe: Unrecognised inode hash code 6
kernel: [6643633.255178] EXT4-fs warning (device ploop51810p1):
dx_probe: Corrupt dir inode 27930449, running e2fsck is recommended.
kernel: [6643634.114901] CT: 171: stopped
kernel: [6643636.239696] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
kernel: [6643636.239722]
kernel: [6643636.490317] __ratelimit: 79 callbacks suppressed
kernel: [6643636.490330] Buffer I/O error on device ploop51810p1,
logical block 0
kernel: [6643636.490346] lost page write due to I/O error on ploop51810p1
kernel: [6643636.490397] EXT4-fs error (device ploop51810p1):
ext4_put_super: Couldn't clean up the journal
kernel: [6643641.235203] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
kernel: [6643641.235229]
kernel: [6643646.230710] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
kernel: [6643646.230737]
kernel: [6643651.226214] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
kernel: [6643651.226239]
kernel: [6643656.221719] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
kernel: [6643656.222049]
kernel: [6643661.217225] EXT4-fs (ploop34148p1): ext4_da_writepages:
jbd2_start: 1024 pages, ino 525116; err -30
kernel: [6643661.217548]
.................................................
--
Best regards,
Gena
More information about the Users
mailing list