From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81869C433B4 for ; Sat, 10 Apr 2021 03:07:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ECE0D6101B for ; Sat, 10 Apr 2021 03:07:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ECE0D6101B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 289EE6B006C; Fri, 9 Apr 2021 23:07:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 23A436B006E; Fri, 9 Apr 2021 23:07:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DB6B6B0070; Fri, 9 Apr 2021 23:07:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0043.hostedemail.com [216.40.44.43]) by kanga.kvack.org (Postfix) with ESMTP id E4D106B006C for ; Fri, 9 Apr 2021 23:07:20 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 913B96D71 for ; Sat, 10 Apr 2021 03:07:20 +0000 (UTC) X-FDA: 78014971440.30.0854513 Received: from mail-io1-f52.google.com (mail-io1-f52.google.com [209.85.166.52]) by imf05.hostedemail.com (Postfix) with ESMTP id B023FE00010B for ; Sat, 10 Apr 2021 03:07:19 +0000 (UTC) Received: by mail-io1-f52.google.com with SMTP id j26so7886707iog.13 for ; Fri, 09 Apr 2021 20:07:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=HVNC0d59MT6xpFEDMgjLN8K/lOHIofFkyNv+jPDCMzY=; b=V9jKrooI8Haf1uI3I2fEntbb+NYU3UsCPjwssnKN96Xu8dLwsGjfZ2WNbH+AMKa3Fk y0RdsN7Y9Mp5ddgZiWLriuMUFvlEoJBt00+qw7eAeWHeepZsfLDhgcAFC7FxIQhPgFjt wI8wZXfd0pIa3sfNlmOK32loMbbA8zpWu/TZIA1cwpkAUiqMma/F04XSnp1R8mhen8GV jUe1BTc/HywJQIBifiapO0Cb91d5IwO3yIhaf6IxIF/pffhXYS/BU9cVvImdYY/OTgmh focPNd35E+/ELmZQ8pQJdwzDnwKss2AER02PBwOZTPKILLLrMafQsHXrvssCgmjS5Acl emRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=HVNC0d59MT6xpFEDMgjLN8K/lOHIofFkyNv+jPDCMzY=; b=M33+xrHTeZ8wccFqxhtm9Rs6AMt+zphMEBB+bxg/oTxFvMvY8zGUlUzgQiBVh/DdWh fL8ATGqnZkDx6ftf/z8NmemQa/3L3wFHJItRfudTMLErUzG9Oe3asp/iLEpHMq7GQYNz S/qFycY+iJX1Q6hEA9nup4QagbfTqHeHxHxlkv9c1BDJo4nSWAW+TN2IoIWqnZD7A9Zh 2PdvIcASihVZIB6WAKG9O0Aw6yJoRm628x/ajm7vr5b1n7ivhtupMJxGPzCAlHE+z0ji rVZ4Fj+1M4EyztcRrIZl1ti5nPR5TrvTXi5lJR3zhRhZu8nBJPGodambBfqAMMkxU2Yu /1Lw== X-Gm-Message-State: AOAM5311YVuOVa2QmoG1gEMcIH3QT4SztBKMPSr2jCUB4tyYEzRj5tdi cFAuamfHnRPZegL6HF0XkdbN5U8cVY2Llil26Nfvpg== X-Google-Smtp-Source: ABdhPJy1JsEpa0RjSGy/7SGGuduw2fX6ZNYpbqt4u0QfFvAtjUX0rEOfdWgfHjjynFGUZEkr2BTVdUrIYql0OyJhtKc= X-Received: by 2002:a5e:8c16:: with SMTP id n22mr13978920ioj.156.1618024039488; Fri, 09 Apr 2021 20:07:19 -0700 (PDT) MIME-Version: 1.0 References: <20210401183216.443C4443@viggo.jf.intel.com> <20210401183219.DC1928FA@viggo.jf.intel.com> In-Reply-To: <20210401183219.DC1928FA@viggo.jf.intel.com> From: Wei Xu Date: Fri, 9 Apr 2021 20:07:08 -0700 Message-ID: Subject: Re: [PATCH 02/10] mm/numa: automatically generate node migration order To: Dave Hansen Cc: Linux MM , Linux Kernel Mailing List , Yang Shi , David Rientjes , Huang Ying , Dan Williams , David Hildenbrand , Oscar Salvador Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: ugtqm5p1bp9ysah11au9dzbop7a5aahi X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B023FE00010B Received-SPF: none (google.com>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=mail-io1-f52.google.com; client-ip=209.85.166.52 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618024039-439507 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Apr 1, 2021 at 11:35 AM Dave Hansen wrote: > +/* > + * node_demotion[] example: > + * > + * Consider a system with two sockets. Each socket has > + * three classes of memory attached: fast, medium and slow. > + * Each memory class is placed in its own NUMA node. The > + * CPUs are placed in the node with the "fast" memory. The > + * 6 NUMA nodes (0-5) might be split among the sockets like > + * this: > + * > + * Socket A: 0, 1, 2 > + * Socket B: 3, 4, 5 > + * > + * When Node 0 fills up, its memory should be migrated to > + * Node 1. When Node 1 fills up, it should be migrated to > + * Node 2. The migration path start on the nodes with the > + * processors (since allocations default to this node) and > + * fast memory, progress through medium and end with the > + * slow memory: > + * > + * 0 -> 1 -> 2 -> stop > + * 3 -> 4 -> 5 -> stop > + * > + * This is represented in the node_demotion[] like this: > + * > + * { 1, // Node 0 migrates to 1 > + * 2, // Node 1 migrates to 2 > + * -1, // Node 2 does not migrate > + * 4, // Node 3 migrates to 4 > + * 5, // Node 4 migrates to 5 > + * -1} // Node 5 does not migrate > + */ In this example, if we want to support multiple nodes as the demotion target of a source node, we can group these nodes into three tiers (classes): fast class: 0 -> {1, 4} // 1 is the preferred 3 -> {4, 1} // 4 is the preferred medium class: 1 -> {2, 5} // 2 is the preferred 4 -> {5, 2} // 5 is the preferred slow class: 2 -> stop 5 -> stop This can guarantee there are no cycles, either. Does it sound sensible? > +again: > + this_pass = next_pass; > + next_pass = NODE_MASK_NONE; > + /* > + * To avoid cycles in the migration "graph", ensure > + * that migration sources are not future targets by > + * setting them in 'used_targets'. Do this only > + * once per pass so that multiple source nodes can > + * share a target node. > + * > + * 'used_targets' will become unavailable in future > + * passes. This limits some opportunities for > + * multiple source nodes to share a destination. > + */ > + nodes_or(used_targets, used_targets, this_pass); > + for_each_node_mask(node, this_pass) { > + int target_node = establish_migrate_target(node, &used_targets); > + > + if (target_node == NUMA_NO_NODE) > + continue; > + > + /* Visit targets from this pass in the next pass: */ > + node_set(target_node, next_pass); > + } > + /* Is another pass necessary? */ > + if (!nodes_empty(next_pass)) > + goto again; This goto seems like exactly a "do {} while" loop. Any particular reason not to use "do {} while" here?