From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6870BC433E1 for ; Thu, 6 Aug 2020 17:12:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C16F223135 for ; Thu, 6 Aug 2020 17:12:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="Uu02hQeX"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="1ench/3Y" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729897AbgHFRMJ (ORCPT ); Thu, 6 Aug 2020 13:12:09 -0400 Received: from Galois.linutronix.de ([193.142.43.55]:58938 "EHLO galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729728AbgHFRKI (ORCPT ); Thu, 6 Aug 2020 13:10:08 -0400 From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1596733806; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=yKVD0ICRirwJxf/czTIoL9qaNHOqC5llD5fZwtykD6w=; b=Uu02hQeXnceW1clbDFU34w01Na01ihe8gw5vdOZssFKm0ZWXV+ig/hiWrMPj3DyEDWsWmp d5q6kG0wmvOmqMIX3dWRl6kqRLWtU/fisAqjfMYs/MNmgVZG/RmheVA60Im0TeiUhxBCZi 9hqt3C2EPyjAChYbG+bZD0bMlT5GYo2qRQV4t2yqJImq4xW2cQe2ctC176iuoCQtFZxNNG MI4kIWnaPSegaZ8F6XUsUtUkLhfST8e4RlThO9ng/MYX/egAzO5ulxnw6h/KaTv4bLrWqA /2u4RF35uMj/YCiBZQtAEM2JK91u6lf5m4pEzA64GZc0kJ2gbDtWTsZWc4f9MQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1596733806; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=yKVD0ICRirwJxf/czTIoL9qaNHOqC5llD5fZwtykD6w=; b=1ench/3Yz4ELlGwrmwd8zDNRPuZX7xmwBO6TcJKMNpbwZC7PP+JR9mKgtVvQyp3pUUu5qC WOvTlcXJ7KEJIRBQ== To: "Dey\, Megha" , Jason Gunthorpe Cc: Marc Zyngier , "Jiang\, Dave" , "vkoul\@kernel.org" , "bhelgaas\@google.com" , "rafael\@kernel.org" , "gregkh\@linuxfoundation.org" , "hpa\@zytor.com" , "alex.williamson\@redhat.com" , "Pan\, Jacob jun" , "Raj\, Ashok" , "Liu\, Yi L" , "Lu\, Baolu" , "Tian\, Kevin" , "Kumar\, Sanjay K" , "Luck\, Tony" , "Lin\, Jing" , "Williams\, Dan J" , "kwankhede\@nvidia.com" , "eric.auger\@redhat.com" , "parav\@mellanox.com" , "Hansen\, Dave" , "netanelg\@mellanox.com" , "shahafs\@mellanox.com" , "yan.y.zhao\@linux.intel.com" , "pbonzini\@redhat.com" , "Ortiz\, Samuel" , "Hossain\, Mona" , "dmaengine\@vger.kernel.org" , "linux-kernel\@vger.kernel.org" , "x86\@kernel.org" , "linux-pci\@vger.kernel.org" , "kvm\@vger.kernel.org" Subject: RE: [PATCH RFC v2 02/18] irq/dev-msi: Add support for a new DEV_MSI irq domain In-Reply-To: References: <159534667974.28840.2045034360240786644.stgit@djiang5-desk3.ch.intel.com> <159534734833.28840.10067945890695808535.stgit@djiang5-desk3.ch.intel.com> <878sfbxtzi.wl-maz@kernel.org> <20200722195928.GN2021248@mellanox.com> <96a1eb5ccc724790b5404a642583919d@intel.com> <20200805221548.GK19097@mellanox.com> <70465fd3a7ae428a82e19f98daa779e8@intel.com> <20200805225330.GL19097@mellanox.com> <630e6a4dc17b49aba32675377f5a50e0@intel.com> <20200806001927.GM19097@mellanox.com> Date: Thu, 06 Aug 2020 19:10:05 +0200 Message-ID: <87tuxfhf9u.fsf@nanos.tec.linutronix.de> MIME-Version: 1.0 Content-Type: text/plain Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org Megha, "Dey, Megha" writes: >> -----Original Message----- >> From: Jason Gunthorpe >> Subject: Re: [PATCH RFC v2 02/18] irq/dev-msi: Add support for a new DEV_MSI >> irq domain can you please fix your mail client not to copy the whole header of the mail you are replying to into the mail body? >> > > Well, I had suggested to pass in the parent struct device, but it >> Oops, I was thinking of platform_msi_domain_alloc_irqs() not >> create_device_domain() >> >> ie call it in the device driver that wishes to consume the extra MSIs. >> >> Is there a harm if each device driver creates a new irq_domain for its use? > > Well, the only harm is if we want to reuse the irq domain. You cannot reuse the irq domain if you create a domain per driver. The way how hierarchical domains work is: vector --- DMAR-MSI | |-- .... | |-- IR-0 --- IO/APIC-0 | | | |-- IO/APIC-1 | | | |-- PCI/MSI-0 | | | |-- HPET/MSI-0 | |-- IR-1 --- PCI/MSI-1 | | The outermost domain is what the actual device driver uses. I.e. for PCI-MSI it's the msi domain which is associated to the bus the device is connected to. Each domain has its own interrupt chip instance and its own data set. Domains of the same type share the code, but neither the data nor the interrupt chip instance. Also there is a strict parent child relationship in terms of resources. Let's look at PCI. PCI/MSI-0 depends on IR-0 which depends on the vector domain. That's reflecting both the flow of the interrupt and the steps required for various tasks, e.g. allocation/deallocation and also interrupt chip operations. In order to allocate a PCI/MSI interrupt in domain PCI/MSI-0 a slot in the remapping unit and a vector needs to be allocated. If you disable interrupt remapping all the outermost domains in the scheme above become childs of the vector domain. So if we look at DEV/MSI as a infrastructure domain then the scheme looks like this: vector --- DMAR-MSI | |-- .... | |-- IR-0 --- IO/APIC-0 | | | |-- IO/APIC-1 | | | |-- PCI/MSI-0 | | | |-- HPET/MSI-0 | | | |-- DEV/MSI-0 | |-- IR-1 --- PCI/MSI-1 | | | |-- DEV/MSI-1 But if you make it per device then you have multiple DEV/MSI domains per IR unit. What's the right thing to do? If the DEV/MSI domain has it's own per IR unit resource management, then you need one per IR unit. If the resource management is solely per device then having a domain per device is the right choice. Thanks, tglx