<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><span>> From: Kirill Tkhai <ktkhai@virtuozzo.com><br>
</span>
<div>> Sent: Tuesday, July 21, 2020 4:52 PM<br>
</div>
<div>> To: Valeriy Vdovin <Valeriy.Vdovin@virtuozzo.com>; devel@openvz.org > <devel@openvz.org><br>
</div>
<div>> Cc: Valeriy Vdovin <valeriy.vdovin@virtuozz.com><br>
</div>
<div>> Subject: Re: [PATCH RHEL7 v20 06/14] ve/cgroup: unmark ve-root cgroups at container stop<br>
</div>
<div>><br>
</div>
<div>> On 25.06.2020 17:29, Valeriy Vdovin wrote:<br>
</div>
<div>> > Signed-off-by: Valeriy Vdovin <valeriy.vdovin@virtuozzo.com><br>
</div>
<div>> > Reviewed-by: Kirill Tkhai <ktkhai@virtuozzo.com><br>
</div>
<div>> > ---<br>
</div>
<div>> > include/linux/cgroup.h | 1 +<br>
</div>
<div>> > kernel/cgroup.c | 38 ++++++++++++++++++++++++++++++++++++++<br>
</div>
<div>> > kernel/ve/ve.c | 2 ++<br>
</div>
<div>> > 3 files changed, 41 insertions(+)<br>
</div>
<div>> ><br>
</div>
<div>> > diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h<br>
</div>
<div>> > index ac60aaed..6e2c206 100644<br>
</div>
<div>> > --- a/include/linux/cgroup.h<br>
</div>
<div>> > +++ b/include/linux/cgroup.h<br>
</div>
<div>> > @@ -671,6 +671,7 @@ int cgroup_task_count(const struct cgroup *cgrp);<br>
</div>
<div>> ><br>
</div>
<span>> > #ifdef CONFIG_VE</span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>> > void cgroup_mark_ve_roots(struct ve_struct *ve);<br>
</span>
<div>> > +void cgroup_unmark_ve_roots(struct ve_struct *ve);<br>
</div>
<div>> > #endif<br>
</div>
<div>> ><br>
</div>
<div>> > /*<br>
</div>
<div>> > diff --git a/kernel/cgroup.c b/kernel/cgroup.c<br>
</div>
<div>> > index ce576c5..6e3871a 100644<br>
</div>
<div>> > --- a/kernel/cgroup.c<br>
</div>
<div>> > +++ b/kernel/cgroup.c<br>
</div>
<div>> > @@ -637,6 +637,31 @@ static struct css_set *find_css_set(<br>
</div>
<div>> > }<br>
</div>
<div>> ><br>
</div>
<div>> > /*<br>
</div>
<div>> > + * Walk each cgroup link of a given css_set and find a cgroup that<br>
</div>
<div>> > + * is the child of cgroupfs_root in argument.<br>
</div>
<div>> > + */<br>
</div>
<div>> > +static struct cgroup *css_cgroup_from_root(struct css_set *css_set,<br>
</div>
<div>> > + struct cgroupfs_root *root)<br>
</div>
<div>> > +{<br>
</div>
<div>> > + struct cgroup *res = NULL;<br>
</div>
<div>> > + struct cg_cgroup_link *link;<br>
</div>
<div>> > +<br>
</div>
<div>> > + BUG_ON(!mutex_is_locked(&cgroup_mutex));<br>
</div>
<div>> > + read_lock(&css_set_lock);<br>
</div>
<div>> > +<br>
</div>
<div>> > + list_for_each_entry(link, &css_set->cg_links, cg_link_list) {<br>
</div>
<div>> > + struct cgroup *c = link->cgrp;<br>
</div>
<div>> > + if (c->root == root) {<br>
</div>
<div>> > + res = c;<br>
</div>
<div>> > + break;<br>
</div>
<div>> > + }<br>
</div>
<div>> > + }<br>
</div>
<div>> > + read_unlock(&css_set_lock);<br>
</div>
<div>> > + BUG_ON(!res);<br>
</div>
<div>> > + return res;<br>
</div>
<div>> > +}<br>
</div>
<div>> > +<br>
</div>
<div>> > +/*<br>
</div>
<div>> > * Return the cgroup for "task" from the given hierarchy. Must be<br>
</div>
<div>> > * called with cgroup_mutex held.<br>
</div>
<div>> > */<br>
</div>
<div>> > @@ -4329,6 +4354,19 @@ void cgroup_mark_ve_roots(struct ve_struct *ve)<br>
</div>
<div>> > mutex_unlock(&cgroup_mutex);<br>
</div>
<div>> > }<br>
</div>
<span>> ></span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>> > +void cgroup_unmark_ve_roots(struct ve_struct *ve)<br>
</span>
<div>> > +{<br>
</div>
<div>> > + struct cgroup *cgrp;<br>
</div>
<div>> > + struct cgroupfs_root *root;<br>
</div>
<div>> > + <br>
</div>
<div>> > + mutex_lock(&cgroup_mutex);<br>
</div>
<div>> > + for_each_active_root(root) {<br>
</div>
<div>> > + cgrp = css_cgroup_from_root(ve->root_css_set, root);<br>
</div>
<div>> > + clear_bit(CGRP_VE_ROOT, &cgrp->flags);<br>
</div>
<div>> > + } <br>
</div>
<div>> > + mutex_unlock(&cgroup_mutex);<br>
</div>
<div>> > +}<br>
</div>
<div>> > + <br>
</div>
<div>> > struct cgroup *cgroup_get_ve_root(struct cgroup *cgrp)<br>
</div>
<div>> > {<br>
</div>
<div>> > struct cgroup *ve_root = NULL;<br>
</div>
<div>> > diff --git a/kernel/ve/ve.c b/kernel/ve/ve.c<br>
</div>
<div>> > index 73cfee6..711050c 100644<br>
</div>
<div>> > --- a/kernel/ve/ve.c<br>
</div>
<div>> > +++ b/kernel/ve/ve.c<br>
</div>
<div>> > @@ -623,6 +623,8 @@ void ve_exit_ns(struct pid_namespace *pid_ns)<br>
</div>
<div>> > if (!ve->ve_ns || ve->ve_ns->pid_ns != pid_ns)<br>
</div>
<div>> > return;<br>
</div>
<div>> > <br>
</div>
<div>> > + cgroup_unmark_ve_roots(ve);<br>
</div>
<div>> <br>
</div>
<div>> Is there a problem that ve workqueue works will run after we unmark roots?<br>
</div>
<div>> Maybe we should call this cgroup_unmark_ve_roots() after ve_workqueue_stop()?<br>
</div>
<div><br>
</div>
<div>When a cgroup gets empty it's decided to which workqueue it should be put to await for
<br>
</div>
<div>release. Thus when we unmark ve root, we prevent any new empty cgroups from entering this<br>
</div>
<div>workqueue. After that we are safe to stop the workqueue by waiting for all the current jobs to<br>
</div>
<div>complete.<br>
</div>
<div><br>
</div>
<div>> > + <br>
</div>
<div>> > ve_workqueue_stop(ve);<br>
</div>
<div>> > <br>
</div>
<div>> > /* <br>
</div>
<div>> ><br>
</div>
<span></span><br>
</div>
</body>
</html>