* CGroups: idr_remove called for id=65536 which is not allocated.
@ 2016-07-13 13:31 John Garcia
2016-07-13 15:22 ` Tejun Heo
0 siblings, 1 reply; 6+ messages in thread
From: John Garcia @ 2016-07-13 13:31 UTC (permalink / raw)
To: gregkh; +Cc: tj, cgroups, linux-kernel
Hi esteemed kernel devs! Please be sure to CC answers to me personally
so they're flagged for my attention.
I've filed https://bugzilla.kernel.org/show_bug.cgi?id=124641 in
Bugzilla, so I'll try not to repeat too many key details here; GKH has
asked that I bring the conversation to email, LKML, and the cgroups
list.
Long story short, we see the behavior described in this LKML thread
http://lkml.iu.edu/hypermail/linux/kernel/1606.2/00736.html and (we
think) addressed by the patch therein - memory cgroups don't seem to
be automatically garbage collected during runtime and must be manually
released to allow space for new cgroups after the 65336 limit is
reached. At this time, the only way we've found to release them is to
send echo 1 > /proc/sys/vm/drop_caches
to the kernel. The title of this message is the dmesg line we
typically see in machines that have breached the limit, and are trying
in vain to remove a cgroup beyond the 65536 ceiling.
We've asked CoreOS to cut a build with the patch from LKML and we'll
test it shortly to see if it alleviates the problem. We'll update this
list with results of testing against that patch and also test against
a 4.6.x kernel as soon as possible. Let us know if you need the full
stack trace or other diagnostic info.
Cheers!
--
John Garcia
Technical Support Engineer
Mesosphere.io
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CGroups: idr_remove called for id=65536 which is not allocated.
2016-07-13 13:31 CGroups: idr_remove called for id=65536 which is not allocated John Garcia
@ 2016-07-13 15:22 ` Tejun Heo
2016-07-13 20:10 ` John Garcia
0 siblings, 1 reply; 6+ messages in thread
From: Tejun Heo @ 2016-07-13 15:22 UTC (permalink / raw)
To: John Garcia; +Cc: gregkh, cgroups, linux-kernel, Johannes Weiner
Hello,
On Wed, Jul 13, 2016 at 06:31:18AM -0700, John Garcia wrote:
> Hi esteemed kernel devs! Please be sure to CC answers to me personally
> so they're flagged for my attention.
>
> I've filed https://bugzilla.kernel.org/show_bug.cgi?id=124641 in
> Bugzilla, so I'll try not to repeat too many key details here; GKH has
> asked that I bring the conversation to email, LKML, and the cgroups
> list.
>
> Long story short, we see the behavior described in this LKML thread
> http://lkml.iu.edu/hypermail/linux/kernel/1606.2/00736.html and (we
> think) addressed by the patch therein - memory cgroups don't seem to
> be automatically garbage collected during runtime and must be manually
> released to allow space for new cgroups after the 65336 limit is
> reached. At this time, the only way we've found to release them is to
It's not that memcg doesn't gc the dead csses but that the memory
lying around keeps pinning the memcg struct down. There's nothing
wrong with it. As soon as there's memory pressure, the memory will
get reclaimed and the memcg structs will be freed. The problem is
caused by the memcg struct keeping pinning memcg id which is a pretty
limited resource. The above patch fixes the issue by the lifetime of
decoupling memcg id from that of memcg struct.
> send echo 1 > /proc/sys/vm/drop_caches
> to the kernel. The title of this message is the dmesg line we
> typically see in machines that have breached the limit, and are trying
> in vain to remove a cgroup beyond the 65536 ceiling.
The idr_remove message is from a bug in creation failure path. It is
harmless and removed by b00c52dae6d9 ("cgroup: remove redundant
cleanup in css_create").
> We've asked CoreOS to cut a build with the patch from LKML and we'll
> test it shortly to see if it alleviates the problem. We'll update this
> list with results of testing against that patch and also test against
> a 4.6.x kernel as soon as possible. Let us know if you need the full
> stack trace or other diagnostic info.
With the patches backported, there shouldn't be any more issues. If
you encounter any more issues with it, please let us know.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CGroups: idr_remove called for id=65536 which is not allocated.
2016-07-13 15:22 ` Tejun Heo
@ 2016-07-13 20:10 ` John Garcia
2016-07-13 20:14 ` Tejun Heo
0 siblings, 1 reply; 6+ messages in thread
From: John Garcia @ 2016-07-13 20:10 UTC (permalink / raw)
To: Tejun Heo; +Cc: gregkh, cgroups, linux-kernel, Johannes Weiner
Hi Tejun, thanks for the response!
It sounds like everything is good then, is it likely this will get
backported to the next LTS 4.4 kernel?
Thanks!
John
On Wed, Jul 13, 2016 at 8:22 AM, Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> On Wed, Jul 13, 2016 at 06:31:18AM -0700, John Garcia wrote:
>> Hi esteemed kernel devs! Please be sure to CC answers to me personally
>> so they're flagged for my attention.
>>
>> I've filed https://bugzilla.kernel.org/show_bug.cgi?id=124641 in
>> Bugzilla, so I'll try not to repeat too many key details here; GKH has
>> asked that I bring the conversation to email, LKML, and the cgroups
>> list.
>>
>> Long story short, we see the behavior described in this LKML thread
>> http://lkml.iu.edu/hypermail/linux/kernel/1606.2/00736.html and (we
>> think) addressed by the patch therein - memory cgroups don't seem to
>> be automatically garbage collected during runtime and must be manually
>> released to allow space for new cgroups after the 65336 limit is
>> reached. At this time, the only way we've found to release them is to
>
> It's not that memcg doesn't gc the dead csses but that the memory
> lying around keeps pinning the memcg struct down. There's nothing
> wrong with it. As soon as there's memory pressure, the memory will
> get reclaimed and the memcg structs will be freed. The problem is
> caused by the memcg struct keeping pinning memcg id which is a pretty
> limited resource. The above patch fixes the issue by the lifetime of
> decoupling memcg id from that of memcg struct.
>
>> send echo 1 > /proc/sys/vm/drop_caches
>> to the kernel. The title of this message is the dmesg line we
>> typically see in machines that have breached the limit, and are trying
>> in vain to remove a cgroup beyond the 65536 ceiling.
>
> The idr_remove message is from a bug in creation failure path. It is
> harmless and removed by b00c52dae6d9 ("cgroup: remove redundant
> cleanup in css_create").
>
>> We've asked CoreOS to cut a build with the patch from LKML and we'll
>> test it shortly to see if it alleviates the problem. We'll update this
>> list with results of testing against that patch and also test against
>> a 4.6.x kernel as soon as possible. Let us know if you need the full
>> stack trace or other diagnostic info.
>
> With the patches backported, there shouldn't be any more issues. If
> you encounter any more issues with it, please let us know.
>
> Thanks.
>
> --
> tejun
--
John Garcia
Technical Support Engineer
Mesosphere.io
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CGroups: idr_remove called for id=65536 which is not allocated.
2016-07-13 20:10 ` John Garcia
@ 2016-07-13 20:14 ` Tejun Heo
2016-07-13 23:02 ` Greg KH
0 siblings, 1 reply; 6+ messages in thread
From: Tejun Heo @ 2016-07-13 20:14 UTC (permalink / raw)
To: John Garcia; +Cc: gregkh, cgroups, linux-kernel, Johannes Weiner
On Wed, Jul 13, 2016 at 01:10:05PM -0700, John Garcia wrote:
> Hi Tejun, thanks for the response!
>
> It sounds like everything is good then, is it likely this will get
> backported to the next LTS 4.4 kernel?
Yeah, it should.
Thanks!
--
tejun
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CGroups: idr_remove called for id=65536 which is not allocated.
2016-07-13 20:14 ` Tejun Heo
@ 2016-07-13 23:02 ` Greg KH
2016-07-14 15:30 ` Tejun Heo
0 siblings, 1 reply; 6+ messages in thread
From: Greg KH @ 2016-07-13 23:02 UTC (permalink / raw)
To: Tejun Heo; +Cc: John Garcia, cgroups, linux-kernel, Johannes Weiner
On Wed, Jul 13, 2016 at 04:14:17PM -0400, Tejun Heo wrote:
> On Wed, Jul 13, 2016 at 01:10:05PM -0700, John Garcia wrote:
> > Hi Tejun, thanks for the response!
> >
> > It sounds like everything is good then, is it likely this will get
> > backported to the next LTS 4.4 kernel?
>
> Yeah, it should.
Which commits exactly should be backported? I'll be glad to do that...
thanks,
greg k-h
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: CGroups: idr_remove called for id=65536 which is not allocated.
2016-07-13 23:02 ` Greg KH
@ 2016-07-14 15:30 ` Tejun Heo
0 siblings, 0 replies; 6+ messages in thread
From: Tejun Heo @ 2016-07-14 15:30 UTC (permalink / raw)
To: Greg KH; +Cc: John Garcia, cgroups, linux-kernel, Johannes Weiner
Hello, Greg.
On Thu, Jul 14, 2016 at 08:02:26AM +0900, Greg KH wrote:
> On Wed, Jul 13, 2016 at 04:14:17PM -0400, Tejun Heo wrote:
> > On Wed, Jul 13, 2016 at 01:10:05PM -0700, John Garcia wrote:
> > > Hi Tejun, thanks for the response!
> > >
> > > It sounds like everything is good then, is it likely this will get
> > > backported to the next LTS 4.4 kernel?
> >
> > Yeah, it should.
>
> Which commits exactly should be backported? I'll be glad to do that...
Hmmm... actually, the patches aren't tagged for -stable. The patches
are queued in mm and the relevant patches are
mm: memcontrol: fix cgroup creation failure after many small jobs
mm: memcontrol: fix cgroup creation failure after many small jobs fix
Johannes will ping Andrew so that they can out for v4.8 with -stable
cc'd.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-07-14 15:30 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-07-13 13:31 CGroups: idr_remove called for id=65536 which is not allocated John Garcia
2016-07-13 15:22 ` Tejun Heo
2016-07-13 20:10 ` John Garcia
2016-07-13 20:14 ` Tejun Heo
2016-07-13 23:02 ` Greg KH
2016-07-14 15:30 ` Tejun Heo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).