From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECD0FC48BCF for ; Wed, 9 Jun 2021 11:17:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7596761364 for ; Wed, 9 Jun 2021 11:17:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7596761364 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B37416B0036; Wed, 9 Jun 2021 07:17:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE72E6B006E; Wed, 9 Jun 2021 07:17:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 987B96B0070; Wed, 9 Jun 2021 07:17:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0145.hostedemail.com [216.40.44.145]) by kanga.kvack.org (Postfix) with ESMTP id 6B4796B0036 for ; Wed, 9 Jun 2021 07:17:46 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0F309180AD817 for ; Wed, 9 Jun 2021 11:17:46 +0000 (UTC) X-FDA: 78233935332.40.A0F7D98 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 5141C8019357 for ; Wed, 9 Jun 2021 11:17:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623237465; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8E1p+wWGhwc5LDKImO4GlXYNuAUCpHRsvl3yalaOpY4=; b=h9uURE+dLkuBUgNCYUpeR9ACFQUPIKpOxi2aSsNNknGYPaE40lhExslSXN2xgDLL7tGcaP RUedk5faE4PMOqWD9RxODpB4VDkF7Lt69u0xhkAGQqWAcL0nh5vNOOol9cDJMzDNubeE9W k9Jo5Dz015/xENngEDIlX9H/zbmgi1c= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-165--SiTPfg_NJCwwaRd0NwlpA-1; Wed, 09 Jun 2021 07:17:44 -0400 X-MC-Unique: -SiTPfg_NJCwwaRd0NwlpA-1 Received: by mail-wr1-f70.google.com with SMTP id n4-20020a5d42040000b0290119fef97609so788220wrq.18 for ; Wed, 09 Jun 2021 04:17:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:cc:references:from:organization:subject :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=8E1p+wWGhwc5LDKImO4GlXYNuAUCpHRsvl3yalaOpY4=; b=TFUDez0FmYn2tJ1tPg9kzuWUrUIwZ1F0t/yqWUuUZGkRzpmY8y26cXvAvftry+J72H td+oRYZ4jKUIH3fOTlhXW0UR5kn1xSUkMubTQN2dBNiIjkYI//she0DB3YIYDKcXVsf6 UfjVdwlcXUxpk7f2oVdjXYmw55kfd4AkTMbQ2Xs0cjV2xTAENL6gKQ5KG6GE2mFRmbgp nj/dNHilwuXxJYXh/Ut+y+X+MVjFti4qsLt2Bd7DYufdzs/xDPryi8WQjTZwZWwqWuiD K0ntqJ9Lmd5JeLw6sKaPUFd5wKSA9HIZqD9Qv8wf1psfM+LzrTZ6d11A6VtUE+wPBP4F +LIw== X-Gm-Message-State: AOAM531kTAkULNDd4xiu4pO5fCzUeNXrfEOd+78gkcuMMQa5lS8mho8Y w5Do0oYmd7gd/xs9pu2DQNLindYdrwWj/iRp9wNxy7Jf5Oq5+07ryHjELvGQD6PidM946IHmWpL kgP799ZLHK0M= X-Received: by 2002:adf:e507:: with SMTP id j7mr26921886wrm.178.1623237462893; Wed, 09 Jun 2021 04:17:42 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxuR4C5IxItTaoDqkr5XVupzAy1SkPFt9xFcRl0AJq+emXXfaUYYL532d4tAQ75Anb4NZ6wHQ== X-Received: by 2002:adf:e507:: with SMTP id j7mr26921861wrm.178.1623237462598; Wed, 09 Jun 2021 04:17:42 -0700 (PDT) Received: from [192.168.3.132] (p5b0c611d.dip0.t-ipconnect.de. [91.12.97.29]) by smtp.gmail.com with ESMTPSA id 89sm25483595wrq.14.2021.06.09.04.17.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 09 Jun 2021 04:17:42 -0700 (PDT) To: Paul Menzel , Andrew Morton Cc: linux-mm@kvack.org, it+linux-mm@molgen.mpg.de References: From: David Hildenbrand Organization: Red Hat Subject: Re: Cannot allocate memory despite buff/cache non-zero Message-ID: Date: Wed, 9 Jun 2021 13:17:41 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5141C8019357 Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h9uURE+d; spf=none (imf27.hostedemail.com: domain of david@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: 71b3wbisot4aiqfrnp65ds17pd46jgzp X-HE-Tag: 1623237461-510461 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 04.06.21 13:36, Paul Menzel wrote: > Dear Linux folks, >=20 >=20 > On a 1 TB RAM compute server with Linux 5.10.24 and memory > overcommitting disabled, we ran into a situation where processes like > SSH couldn=E2=80=99t allocate memory anymore. >=20 > $ more /proc/cmdline > BOOT_IMAGE=3D/boot/bzImage-5.10.24.mx64.375 root=3DLABEL=3Droot r= o > crashkernel=3D256M console=3DttyS0,115200n8 console=3Dtty0 init=3D/bin/= systemd > audit=3D0 random.trust_cpu=3Don >=20 > 2021-06-03T22:00:28+02:00 godsavethequeen sshd[89163]: > pam_systemd(sshd:session): Failed to create session: Unit > session-25654.scope not found. > 2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]: error: > do_exec_no_pty: fork: Cannot allocate memory > 2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]: > pam_unix(sshd:session): session closed for user root > 2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: fork= : > Cannot allocate memory > 2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: > ssh_msg_send: write: Broken pipe > 2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: > send_rexec_state: ssh_msg_send failed >=20 > $ free -h > total used free shared buff/cac= he > available > Mem: 1.0T 606G 2.6G 2.2M 39= 5G > 391G > Swap: 0B 0B 0B >=20 > Looking at this, I would have expected, that the pages(?) in buff/cache > would be moved/deleted to make memory available. >=20 > Looking at `/proc/meminfo` (attached): >=20 > MemTotal: 1052411824 kB > MemFree: 2709976 kB > MemAvailable: 410847908 kB > [=E2=80=A6] > CommitLimit: 1052411824 kB > Committed_AS: 1052455260 kB > [=E2=80=A6] >=20 With memory overcommit disabled, each accountable mapping=20 (mm/mmap.c:accountable_mapping()) will count towards Committed_AS. So=20 you might still have plenty of free memory in the system reserved for=20 these mappings, yet Linux won't allow for more accountable mappings.=20 That's why you see the "Cannot allocate memory" messages. mmap() failed. > Committed_AS is greater than the commit limit (total memory). >=20 > Is such behavior expected? We're talking about 43436 kB that exceed the CommitLimit. The CommitLimit might change (grow/shrink) when a) The number of hugetlb pages changes b) Swap space is resized If CommitLimit did not change, Committed_AS should actually not exceed=20 it. IIUC, it can only happen temporarily while trying creation of a new=20 mapping. We increase Committed_AS unconditionally and decrease it again=20 if we reject it. --=20 Thanks, David / dhildenb