[Devel] [PATCH 3/6] memcg, slab: cleanup barrier usage when accessing memcg_caches

[Michal Hocko]

On Thu 19-12-13 13:29:59, Vladimir Davydov wrote:
> On 12/19/2013 01:21 PM, Michal Hocko wrote:
> > On Thu 19-12-13 13:16:01, Vladimir Davydov wrote:
> >> On 12/19/2013 01:10 PM, Michal Hocko wrote:
> >>> On Thu 19-12-13 10:37:27, Vladimir Davydov wrote:
> >>>> On 12/18/2013 09:14 PM, Michal Hocko wrote:
> >>>>> On Wed 18-12-13 17:16:54, Vladimir Davydov wrote:
> >>>>>> First, in memcg_create_kmem_cache() we should issue the write barrier
> >>>>>> after the kmem_cache is initialized, but before storing the pointer to
> >>>>>> it in its parent's memcg_params.
> >>>>>>
> >>>>>> Second, we should always issue the read barrier after
> >>>>>> cache_from_memcg_idx() to conform with the write barrier.
> >>>>>>
> >>>>>> Third, its better to use smp_* versions of barriers, because we don't
> >>>>>> need them on UP systems.
> >>>>> Please be (much) more verbose on Why. Barriers are tricky and should be
> >>>>> documented accordingly. So if you say that we should issue a barrier
> >>>>> always be specific why we should do it.
> >>>> In short, we have kmem_cache::memcg_params::memcg_caches is an array of
> >>>> pointers to per-memcg caches. We access it lock-free so we should use
> >>>> memory barriers during initialization. Obviously we should place a write
> >>>> barrier just before we set the pointer in order to make sure nobody will
> >>>> see a partially initialized structure. Besides there must be a read
> >>>> barrier between reading the pointer and accessing the structure, to
> >>>> conform with the write barrier. It's all that similar to rcu_assign and
> >>>> rcu_deref. Currently the barrier usage looks rather strange:
> >>>>
> >>>> memcg_create_kmem_cache:
> >>>>     initialize kmem
> >>>>     set the pointer in memcg_caches
> >>>>     wmb() // ???
> >>>>
> >>>> __memcg_kmem_get_cache:
> >>>>     <...>
> >>>>     read_barrier_depends() // ???
> >>>>     cachep = root_cache->memcg_params->memcg_caches[memcg_id]
> >>>>     <...>
> >>> Why do we need explicit memory barriers when we can use RCU?
> >>> __memcg_kmem_get_cache already dereferences within rcu_read_lock.
> >> Because it's not RCU, IMO. RCU implies freeing the old version after a
> >> grace period, while kmem_caches are freed immediately. We simply want to
> >> be sure the kmem_cache is fully initialized. And we do not require
> >> calling this in an RCU critical section.
> > And you can use rcu_dereference and rcu_assign for that as well.
> 
> rcu_dereference() will complain if called outside an RCU critical
> section, while cache_from_memcg_idx() is called w/o RCU protection from
> some places.

Does anything prevents us from using RCU from those callers as well?

> > It hides all the juicy details about memory barriers.
> 
> IMO, a memory barrier with a good comment looks better than an
> rcu_dereference() without RCU protection :-)

OK, let's wait for a good comment then ;)

> > Besides that nothing prevents us from freeing from rcu callback. Or?
> 
> It's an overhead we can live without there. The point is that we can
> access a cache only if it is active. I mean no allocation can go from a
> cache that has already been destroyed. It would be a bug. So there is no
> point in introducing RCU-protection for kmem_caches there. It would only
> confuse, IMO.

My point was that the current state is a disaster. Implicit assumptions
on different locking with memory barriers to make it even more juicy.
This should be cleaned up really. Replacing explicit memory barriers by
RCU sounds like a straightforward and much easier to follow for many
people (unlike memory barriers).

I do not insist on RCU but please make this code comprehensible. My head
is spinning anytime I look down there and try to find out which locks
are actually held and whether that is safe.

> 
> Thanks.

-- 
Michal Hocko
SUSE Labs