From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F610C433DF for ; Mon, 15 Jun 2020 20:21:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 42FD5207D3 for ; Mon, 15 Jun 2020 20:21:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Z1naqXXF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731466AbgFOUVr (ORCPT ); Mon, 15 Jun 2020 16:21:47 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:56645 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1730054AbgFOUVp (ORCPT ); Mon, 15 Jun 2020 16:21:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1592252504; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc; bh=LHOwL6jMIuTyNQ2/9f6CpOOFyVxyxBr/Q//T3vuo+z4=; b=Z1naqXXFx9GCjqCrmMt5y51MLloki08r7nFcQzh6rWT+t2zgaoKX0pUjtQSj/n6bq7okiv XtP/TAfoHNj7Itv/Q9Je9o1PZSMzzRyLiF0PiewDR3NvNtyh5iKngDqGdFgsyGWFtRwDaa QOYAZ2cNy54PCL+xrxeVt75xhWn+l40= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-52-dF99QhNIPdijJG7skzE-nw-1; Mon, 15 Jun 2020 16:21:35 -0400 X-MC-Unique: dF99QhNIPdijJG7skzE-nw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2781B873402; Mon, 15 Jun 2020 20:21:34 +0000 (UTC) Received: from virtlab500.virt.lab.eng.bos.redhat.com (virtlab500.virt.lab.eng.bos.redhat.com [10.19.152.160]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4F0806ED96; Mon, 15 Jun 2020 20:21:33 +0000 (UTC) From: Nitesh Narayan Lal To: linux-kernel@vger.kernel.org, frederic@kernel.org, mtosatti@redhat.com, sassmann@redhat.com, jeffrey.t.kirsher@intel.com, jacob.e.keller@intel.com, jlelli@redhat.com Subject: [PATCH v1 0/1] limit the i40e msix vectors based on housekeeping CPUs Date: Mon, 15 Jun 2020 16:21:24 -0400 Message-Id: <20200615202125.27831-1-nitesh@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Issue ===== With the current implementation at the time of i40e_init_msix(), i40e creates vectors only based on the number of online CPUs. This would be problematic for RT setup that includes a large number of isolated but very few housekeeping CPUs. This is because in those setups an attempt to move all IRQs from isolated to housekeeping CPUs may easily fail due to per CPU vector limit. Setup For The Issue =================== I have triggered this issue on a setup that had a total of 72 cores among which 68 were isolated and only 4 were left for housekeeping tasks. I was using tuned's realtime-virtual-host profile to configure the system. However, Tuned reported the error message 'Failed to set SMP affinity of IRQ xxx to '00000040,00000010,00000005': [Errno 28] No space left on the device' for several IRQs in tuned.log. Note: There were other IRQs as well pinned to the housekeeping CPUs that were generated by other drivers. Fix === - In this proposed fix I have replaced num_online_cpus in i40e_init_msix() with the number of housekeeping CPUs. - The reason why I chose to include both HK_FLAG_DOMAIN & HK_FLAG_WQ is because we would also need IRQ isolation with something like systemd's CPU affinity. Testing ======= To test this change I had added a tracepoint in i40e_init_msix() to find the number of CPUs derived for vector creation with and without tuned's realtime-virtual-host profile. As per expectation with the profile applied I was only getting the number of housekeeping CPUs and all available CPUs without it. Nitesh Narayan Lal (1): i40e: limit the msix vectors based on housekeeping CPUs drivers/net/ethernet/intel/i40e/i40e_main.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) --