[Users] flashcache

[Pavel Snajdr]

On 07/09/2014 06:58 PM, Kir Kolyshkin wrote:
> On 07/08/2014 11:54 PM, Pavel Snajdr wrote:
>> On 07/08/2014 07:52 PM, Scott Dowdle wrote:
>>> Greetings,
>>>
>>> ----- Original Message -----
>>>> (offtopic) We can not use ZFS. Unfortunately, NAS with something like
>>>> Nexenta is to expensive for us.
>>> From what I've gathered from a few presentations, ZFS on Linux (http://zfsonlinux.org/) is as stable but more performant than it is on the OpenSolaris forks... so you can build your own if you can spare the people to learn the best practices.
>>>
>>> I don't have a use for ZFS myself so I'm not really advocating it.
>>>
>>> TYL,
>>>
>> Hi all,
>>
>> we run tens of OpenVZ nodes (bigger boxes, 256G RAM, 12cores+, 90 CTs at
>> least). We've used to run ext4+flashcache, but ext4 has proven to be a
>> bottleneck. That was the primary motivation behind ploop as far as I know.
>>
>> We've switched to ZFS on Linux around the time Ploop was announced and I
>> didn't have second thoughts since. ZFS really *is* in my experience the
>> best filesystem there is at the moment for this kind of deployment  -
>> especially if you use dedicated SSDs for ZIL and L2ARC, although the
>> latter is less important. You will know what I'm talking about when you
>> try this on boxes with lots of CTs doing LAMP load - databases and their
>> synchronous writes are the real problem, which ZFS with dedicated ZIL
>> device solves.
>>
>> Also there is the ARC caching, which is smarter then linux VFS cache -
>> we're able to achieve about 99% of hitrate at about 99% of the time,
>> even under high loads.
>>
>> Having said all that, I recommend everyone to give ZFS a chance, but I'm
>> aware this is yet another out-of-mainline code and that doesn't suit
>> everyone that well.
>>
> 
> Are you using per-container ZVOL or something else?

That would mean I'd need to do another filesystem on top of ZFS, which
would in turn mean I'd add another unnecessary layer of indirection. ZFS
is a pooled storage like BTRFS is, we're giving one dataset to each
container.

vzctl tries to move the VE_PRIVATE folder around, so we had to add one
more directory to put the VE_PRIVATE data into (see the first ls).

Example from production:

[root at node2.prg.vpsfree.cz]
 ~ # zpool status vz
  pool: vz
 state: ONLINE
  scan: scrub repaired 0 in 1h24m with 0 errors on Tue Jul  8 16:22:17 2014
config:

	NAME        STATE     READ WRITE CKSUM
	vz          ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    sda     ONLINE       0     0     0
	    sdb     ONLINE       0     0     0
	  mirror-1  ONLINE       0     0     0
	    sde     ONLINE       0     0     0
	    sdf     ONLINE       0     0     0
	  mirror-2  ONLINE       0     0     0
	    sdg     ONLINE       0     0     0
	    sdh     ONLINE       0     0     0
	logs
	  mirror-3  ONLINE       0     0     0
	    sdc3    ONLINE       0     0     0
	    sdd3    ONLINE       0     0     0
	cache
	  sdc5      ONLINE       0     0     0
	  sdd5      ONLINE       0     0     0

errors: No known data errors

[root at node2.prg.vpsfree.cz]
 ~ # zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
vz                432G  2.25T    36K  /vz
vz/private        427G  2.25T   111K  /vz/private
vz/private/101   17.7G  42.3G  17.7G  /vz/private/101
<snip>
vz/root           104K  2.25T   104K  /vz/root
vz/template      5.38G  2.25T  5.38G  /vz/template

[root at node2.prg.vpsfree.cz]
 ~ # zfs get compressratio vz/private/101
NAME            PROPERTY       VALUE  SOURCE
vz/private/101  compressratio  1.38x  -

[root at node2.prg.vpsfree.cz]
 ~ # ls /vz/private/101
private

[root at node2.prg.vpsfree.cz]
 ~ # ls /vz/private/101/private/
aquota.group  aquota.user  b  bin  boot  dev  etc  git  home  lib
<snip>

[root at node2.prg.vpsfree.cz]
 ~ # cat /etc/vz/conf/101.conf | grep -P "PRIVATE|ROOT"
VE_ROOT="/vz/root/101"
VE_PRIVATE="/vz/private/101/private"


> _______________________________________________
> Users mailing list
> Users at openvz.org
> https://lists.openvz.org/mailman/listinfo/users
>