From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8768B205E0A for ; Mon, 22 Apr 2024 05:22:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713763358; cv=none; b=Z9r0792OWNBI7kd1hHNjgYBk5tT+kZyBVLV4CC05g996prR6QkqOhmTWhONVRSWvENMe21fBEIpnqRAPS9COT0YFTUdaz9D7B6XdRrMjRgPgEgfZZ6bmwtoIFOvDOw5FjWnJ/7dhMHVFwV0uV6EGoc4x2oFpGmFTPNrSpbVjx/s= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713763358; c=relaxed/simple; bh=ZKqatucqlCT9kWZz8IX/UslCh9aEuPFTXZOmstfs/CU=; h=Message-ID:Date:MIME-Version:Cc:Subject:To:References:From: In-Reply-To:Content-Type; b=MQGcraHufh5PqR+Kdqv6HatDqNUqXDvRjf7IK1tYd9/PJQS6AuN6vCfbOf5S+/cqS7XJUdOTJNSQ9lisM+mCLNhCPok/LiMuuzjsbk6d47MaR2z1jYETEuvQ6ymC/lM/02V0BJw/cjP1nWfIpQ+JEAwY2cnN4nWPdT972qNTJzg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=lE/YPNbh; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="lE/YPNbh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1713763356; x=1745299356; h=message-id:date:mime-version:cc:subject:to:references: from:in-reply-to:content-transfer-encoding; bh=ZKqatucqlCT9kWZz8IX/UslCh9aEuPFTXZOmstfs/CU=; b=lE/YPNbhnJwgEwITJyFieqdud+Hsr95PUSs0sHRDw1sktoQSPoPHH9nt nEXUatZQrQc3E+lnRBMoDu2Rc4mSlxE2puInTb0io7HpGx/klZaFpYVKd VR9RmTFzkBhNG6nk5264xZFTUvK402s7bCYucU4vBEE/k7ACk0GRjzjfQ f7vbvhN4OYbgSq1JCn/RbXLyXNxfbi0BD6LPfmq86ZKBOPcXEzy8e0OMU K44ywQe4HBguzo6t3z+1Fxd7Chu1r8AIT4BnGiUW9aFxRrxD3u/zvgxBe UKAYai4kx0YejnhLv2rlnewuYEJ0I+iMNm++cbWOpkNEunUeNSOWYavnz g==; X-CSE-ConnectionGUID: QpeR37FCRRWfRhUsBHG3kw== X-CSE-MsgGUID: nnLUs64gReWNPI5kwnug5Q== X-IronPort-AV: E=McAfee;i="6600,9927,11051"; a="34683065" X-IronPort-AV: E=Sophos;i="6.07,219,1708416000"; d="scan'208";a="34683065" Received: from fmviesa006.fm.intel.com ([10.60.135.146]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Apr 2024 22:22:31 -0700 X-CSE-ConnectionGUID: 3AUe3Y+NQx2mxryv5T9TlQ== X-CSE-MsgGUID: N7gOqtsFQ8W23/Oj5az/fg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,219,1708416000"; d="scan'208";a="23952539" Received: from unknown (HELO [10.239.159.127]) ([10.239.159.127]) by fmviesa006.fm.intel.com with ESMTP; 21 Apr 2024 22:22:26 -0700 Message-ID: <2f2750cd-a5bf-4486-8f50-c93d246f8b0c@linux.intel.com> Date: Mon, 22 Apr 2024 13:21:05 +0800 Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Cc: baolu.lu@linux.intel.com, Palmer Dabbelt , Albert Ou , Anup Patel , Sunil V L , Nick Kossifidis , Sebastien Boeuf , Rob Herring , Krzysztof Kozlowski , Conor Dooley , devicetree@vger.kernel.org, iommu@lists.linux.dev, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux@rivosinc.com Subject: Re: [PATCH v2 7/7] iommu/riscv: Paging domain support To: Tomasz Jeznach , Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley References: <301244bc3ff5da484b46d3fecc931cdad7d2806f.1713456598.git.tjeznach@rivosinc.com> Content-Language: en-US From: Baolu Lu In-Reply-To: <301244bc3ff5da484b46d3fecc931cdad7d2806f.1713456598.git.tjeznach@rivosinc.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 4/19/24 12:32 AM, Tomasz Jeznach wrote: > Introduce first-stage address translation support. > > Page table configured by the IOMMU driver will use the same format > as the CPU’s MMU, and will fallback to identity translation if the > page table format configured for the MMU is not supported by the > IOMMU hardware. > > This change introduces IOTINVAL.VMA command, required to invalidate > any cached IOATC entries after mapping is updated and/or removed from > the paging domain. Invalidations for the non-leaf page entries will > be added to the driver code in separate patch series, following spec > update to clarify non-leaf cache invalidation command. With this patch, > allowing only 4K mappings and keeping non-leaf page entries in memory > this should be a reasonable simplification. > > Signed-off-by: Tomasz Jeznach > --- > drivers/iommu/riscv/Kconfig | 1 + > drivers/iommu/riscv/iommu.c | 467 +++++++++++++++++++++++++++++++++++- > 2 files changed, 466 insertions(+), 2 deletions(-) > [...] > + > static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, > struct device *dev, > struct iommu_domain *iommu_domain) > { > struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); > + struct riscv_iommu_domain *domain; > struct riscv_iommu_dc *dc; > + struct riscv_iommu_bond *bond = NULL, *b; > + struct riscv_iommu_command cmd; > u64 fsc, ta, tc; > int i; > > @@ -769,6 +838,20 @@ static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, > ta = 0; > tc = RISCV_IOMMU_DC_TC_V; > fsc = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_MODE_BARE); > + } else if (iommu_domain->type & __IOMMU_DOMAIN_PAGING) { > + domain = iommu_domain_to_riscv(iommu_domain); > + > + ta = FIELD_PREP(RISCV_IOMMU_PC_TA_PSCID, domain->pscid); > + tc = RISCV_IOMMU_DC_TC_V; > + if (domain->amo_enabled) > + tc |= RISCV_IOMMU_DC_TC_SADE; > + fsc = FIELD_PREP(RISCV_IOMMU_PC_FSC_MODE, domain->pgd_mode) | > + FIELD_PREP(RISCV_IOMMU_PC_FSC_PPN, virt_to_pfn(domain->pgd_root)); > + > + bond = kzalloc(sizeof(*bond), GFP_KERNEL); > + if (!bond) > + return -ENOMEM; > + bond->dev = dev; > } else { > /* This should never happen. */ > return -ENODEV; > @@ -787,12 +870,390 @@ static int riscv_iommu_attach_domain(struct riscv_iommu_device *iommu, > xchg64(&dc->ta, ta); > xchg64(&dc->tc, tc); > > - /* Device context invalidation will be required. Ignoring for now. */ > + if (!(tc & RISCV_IOMMU_DC_TC_V)) > + continue; > + > + /* Invalidate device context cache */ > + riscv_iommu_cmd_iodir_inval_ddt(&cmd); > + riscv_iommu_cmd_iodir_set_did(&cmd, fwspec->ids[i]); > + riscv_iommu_cmd_send(iommu, &cmd, 0); > + > + if (FIELD_GET(RISCV_IOMMU_PC_FSC_MODE, fsc) == RISCV_IOMMU_DC_FSC_MODE_BARE) > + continue; > + > + /* Invalidate last valid PSCID */ > + riscv_iommu_cmd_inval_vma(&cmd); > + riscv_iommu_cmd_inval_set_pscid(&cmd, FIELD_GET(RISCV_IOMMU_DC_TA_PSCID, ta)); > + riscv_iommu_cmd_send(iommu, &cmd, 0); > + } > + > + /* Synchronize directory update */ > + riscv_iommu_cmd_iofence(&cmd); > + riscv_iommu_cmd_send(iommu, &cmd, RISCV_IOMMU_IOTINVAL_TIMEOUT); > + > + /* Track domain to devices mapping. */ > + if (bond) > + list_add_rcu(&bond->list, &domain->bonds); > + > + /* Remove tracking from previous domain, if needed. */ > + iommu_domain = iommu_get_domain_for_dev(dev); Calling iommu_get_domain_for_dev() in the domain attaching path is very fragile because it heavily depends on the order of calling the attach callback and setting the domain pointer in the core. Perhaps the driver can use dev_iommu_priv_set/get() to keep the active domain in the per-device private data? > + if (iommu_domain && !!(iommu_domain->type & __IOMMU_DOMAIN_PAGING)) { > + domain = iommu_domain_to_riscv(iommu_domain); > + bond = NULL; > + rcu_read_lock(); > + list_for_each_entry_rcu(b, &domain->bonds, list) { > + if (b->dev == dev) { > + bond = b; > + break; > + } > + } > + rcu_read_unlock(); > + > + if (bond) { > + list_del_rcu(&bond->list); > + kfree_rcu(bond, rcu); > + } > + } > + > + return 0; > +} Best regards, baolu