[Devel] [PATCH 3/6] memcg, slab: cleanup barrier usage when accessing memcg_caches

[Michal Hocko]

On Thu 19-12-13 13:16:01, Vladimir Davydov wrote:
> On 12/19/2013 01:10 PM, Michal Hocko wrote:
> > On Thu 19-12-13 10:37:27, Vladimir Davydov wrote:
> >> On 12/18/2013 09:14 PM, Michal Hocko wrote:
> >>> On Wed 18-12-13 17:16:54, Vladimir Davydov wrote:
> >>>> First, in memcg_create_kmem_cache() we should issue the write barrier
> >>>> after the kmem_cache is initialized, but before storing the pointer to
> >>>> it in its parent's memcg_params.
> >>>>
> >>>> Second, we should always issue the read barrier after
> >>>> cache_from_memcg_idx() to conform with the write barrier.
> >>>>
> >>>> Third, its better to use smp_* versions of barriers, because we don't
> >>>> need them on UP systems.
> >>> Please be (much) more verbose on Why. Barriers are tricky and should be
> >>> documented accordingly. So if you say that we should issue a barrier
> >>> always be specific why we should do it.
> >> In short, we have kmem_cache::memcg_params::memcg_caches is an array of
> >> pointers to per-memcg caches. We access it lock-free so we should use
> >> memory barriers during initialization. Obviously we should place a write
> >> barrier just before we set the pointer in order to make sure nobody will
> >> see a partially initialized structure. Besides there must be a read
> >> barrier between reading the pointer and accessing the structure, to
> >> conform with the write barrier. It's all that similar to rcu_assign and
> >> rcu_deref. Currently the barrier usage looks rather strange:
> >>
> >> memcg_create_kmem_cache:
> >>     initialize kmem
> >>     set the pointer in memcg_caches
> >>     wmb() // ???
> >>
> >> __memcg_kmem_get_cache:
> >>     <...>
> >>     read_barrier_depends() // ???
> >>     cachep = root_cache->memcg_params->memcg_caches[memcg_id]
> >>     <...>
> > Why do we need explicit memory barriers when we can use RCU?
> > __memcg_kmem_get_cache already dereferences within rcu_read_lock.
> 
> Because it's not RCU, IMO. RCU implies freeing the old version after a
> grace period, while kmem_caches are freed immediately. We simply want to
> be sure the kmem_cache is fully initialized. And we do not require
> calling this in an RCU critical section.

And you can use rcu_dereference and rcu_assign for that as well. It
hides all the juicy details about memory barriers. Besides that nothing
prevents us from freeing from rcu callback. Or?
 
> > Btw. cache_from_memcg_idx is desperately asking for a comment about
> > required locking.
> 
> Actually, I placed a reference to the comment there ;-) but no problem,
> I move it to cache_from_memcg_idx().
> 
> Thanks.

-- 
Michal Hocko
SUSE Labs