From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752407AbdBGB6N (ORCPT ); Mon, 6 Feb 2017 20:58:13 -0500 Received: from mail-sn1nam01on0076.outbound.protection.outlook.com ([104.47.32.76]:44242 "EHLO NAM01-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751805AbdBGB6L (ORCPT ); Mon, 6 Feb 2017 20:58:11 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Suravee.Suthikulpanit@amd.com; Subject: Re: [PATCH v8 9/9] perf/amd/iommu: Enable support for multiple IOMMUs To: Peter Zijlstra References: <1484551416-5440-1-git-send-email-Suravee.Suthikulpanit@amd.com> <1484551416-5440-10-git-send-email-Suravee.Suthikulpanit@amd.com> <20170125094653.GO6515@twins.programming.kicks-ass.net> CC: , , , , From: Suravee Suthikulpanit Message-ID: Date: Tue, 7 Feb 2017 08:57:52 +0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <20170125094653.GO6515@twins.programming.kicks-ass.net> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [171.96.222.170] X-ClientProxiedBy: SG2PR01CA0086.apcprd01.prod.exchangelabs.com (10.170.138.140) To BN6PR12MB1443.namprd12.prod.outlook.com (10.172.24.20) X-MS-Office365-Filtering-Correlation-Id: 6d5b60b5-8950-475d-b12f-08d44efcc3b0 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(48565401081);SRVR:BN6PR12MB1443; X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1443;3:w/qZf42BrGIbZyS/jcxhhmWH4ANMwiCzCROEZWJXkhW9jFHCOthpE5cppVhVKHCa5v9OVUI5HsZ0rV2QfW+0HU04RXFaJzZMcjk0pHthvPuo/Cjc1k+/5CeJZWC4sBqvW4Idkf6p/eaR9v6l18qTWC58+4gPruHm12ctl5iq6WBWfp96ddEQu1FkbM6zphdxhDQdXvrzJUHee+Mv05F+BaR/ogmOSE3pw1h5Hso+IZh3IcX7t8YlQXSzGi24yWf76sGfCyqbeTJm2kLSz9jHfJhciBKETvBu85Sjp3vhAfY=;25:IC5yiqhzH0mKVe1h91419Who052/rWMNcL1dm1bun2P6zjbij3KBMN69BoEFPVbatGTMcEHFU/hNFwC5YZ7q89Lr5LeSgG+RSenxsigY8DJFe+mxaSPqpRIJQ0YxxOUxv721kcxJgrMDK3EhgLsLteSDL2Uw7o2RddJ016b+SEQI+Rqg2FVq0KP5+9vusvDGYtuzQf3Fx2LXuEmOO5toaO7E6FiI79OA8NCkMHDjIId+LIsmuA1/UEGK8J06SxaEhZ9S7EIld/xz4LsF7pB6OGCl4I45CmtmzTCFgrfPgRrZP9wYFZsFQhWEbPdH9KbsllM7shVQYT9GFtcUrvRqTWtUp57LrdUqxemR+lCUJgFjoh1Qx+m9kmWRt4CqzA3oHLRmswtcb1RO38tQ2JmTPUh5vqzC77dh5Kdh3UgQ/f5PNdiRRAaOLlAoKtOQvBx4EhTXYBFSBeZ+AFAPmPiGHw== X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1443;31:XKm4ucUGzXiPpF18V4IFqdvpZSelfSKZ5oqegGsgLqnPy0R4fX0Ab/O9v1VmppqACVwJ10R3sBwlp0/aMGqjP91C+8xSvrqRYJBJxFjbBQKsPSEpiWTu91GNZe2CfMJZkJGLdaJMdsZASzDrgKN0PnmQRbAz5idsuSURUh45JRts+54qR134zNlWnY9WhIqBWnK1RXH7kYA/6u33DvK77ADwWsYPCxCIi3SVYfLR811ZCMt7Tgnusr4niYkkAXhI5DyRMQAW3tUBkf7QVl46aA==;20:uIbVE5IrmuxfP/U7oop5zXOi21RIdDfYmVUZ2bpCPZj8yh7mIesktvFMAAIpEAThNDfQOGOM+yfqx32Dsk9JFcwnn+RS4QY5J7TII/z4+kHrrdaqoQhgkttKD49HW8pD8Jm4MMB3w7j3pwwqswkgeosc1h5xJ0iAg0yEyBQ4X19A1Iroy8CUZWelnTUaJjhVBs2ofKGb+o/8u/bapCDr/6ty9nsmqVfhwVbtABoEZUTd6ipOdgAnD7SH7o9KyD7Xx1hWpd3Y0SixPiCul/Srf6RkAfmKXdmjctGq3IWX8NGx+Ly5uxJYt7vwA6Qs3xBWTs8/KtenDA5tCCbNw2a56xwbe5iJagfkrp/xj4f8SK/AsqP74U4NYRSBy9mqoTuzWLgFdmnqbUf/WJ6OuFNH/zamiOJTUId7tRpJiBdYc9wf8w0gN+o/UxSCm31LkmlbKqQkKS+8ToeXR3W0OTrvm9324mYvMPZqXvgyNCz+0oQkpfYZiR7yiZ//2Bp11dfE X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(20170203043)(10201501046)(3002001)(6055026)(6041248)(20161123555025)(20161123564025)(20161123560025)(20161123558025)(20161123562025)(6072148);SRVR:BN6PR12MB1443;BCL:0;PCL:0;RULEID:;SRVR:BN6PR12MB1443; X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1443;4:ukoeKelk+o+dH1R3ph+l6b+821XO25Ixm+Xc4PKSuD75VvmCHe192yVvQeeYSqKTM3op0XW4k6wSRUG8bjvSLSUPZC3nmsAD1un4hquASL/RB/JU/xnW+LJ5XcWQGALhA5xhVAFMajl3qo7ucurX8wU9VgG7h4y8ete2uNJMjdo29Nm23lpHmzXu7YxmsfwTHEWRICULq0zAdAvq6cK5gelxZ9YQrJdeZ2B/jdXU0D3HXTvut7FOEeXPRlv86svXyZrzcsHXfN+8sGqOWfQqI4JOICU3SZdR5KtAhzD45A8qJcFSgn7wfeL46za1hn8ivzMypHEqFJ1adqfhaJ+5SZdFe0gOskJOQKejflK3k4oQ9tmQ6Ph7B6/MYbyfEO9jqPgl4R3Nli/Fsf2xeoi7JAtVCdnlAVGXYUwiRK4CZy4iME8BJf+g7/MBfxI5ps1ezI2/J5BMnIim4rTlhR5fS+T+KFHwThHF3ocwOZ4nKjWPcYXCs6WUzAymI27KVzXWxfcJg3w4Px81Uf/pwSqXSCz0vsVSLkk8Aq2pPpseWuhAeT0bd/Lyh5pWDle0XYovABNf9YeKci4K+s5oUKFPLwm96BvpB4d4qhm4hU9oLvmKkVNOfjLssgVeslbNfdkp25MmsNaGnjMfmjljShwXIw== X-Forefront-PRVS: 0211965D06 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(7916002)(39850400002)(39410400002)(39840400002)(39860400002)(39450400003)(24454002)(189002)(199003)(38730400002)(3846002)(6116002)(2950100002)(230700001)(6916009)(33646002)(6666003)(54906002)(54356999)(86362001)(31696002)(6506006)(23746002)(4326007)(81156014)(25786008)(8676002)(83506001)(6512007)(229853002)(68736007)(50986999)(6486002)(81166006)(76176999)(36756003)(2906002)(106356001)(66066001)(65956001)(105586002)(47776003)(65806001)(6246003)(101416001)(53546003)(42186005)(50466002)(64126003)(7736002)(65826007)(5660300001)(31686004)(305945005)(4001350100001)(97736004)(189998001)(53936002)(110136004)(92566002);DIR:OUT;SFP:1101;SCL:1;SRVR:BN6PR12MB1443;H:Suravees-MacBook-Pro.local;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;BN6PR12MB1443;23:fo1F1TT6mD/pmXGGR4GBIHxmzaSk69X51VBBu?= =?Windows-1252?Q?ss74FKYn6Q4cRVQPDCILd1SY39+OQpIZdRt6PdZG8LQ/JKMuZHoxY2bL?= =?Windows-1252?Q?/GD9o8ykFERNOnLvus7qD9UVi1ODj20kfGddkDPFp3mqF06MCvcudCK6?= =?Windows-1252?Q?mb2l/MUWae615kBk2w/chjf04pV9UixKQokYmizlXJT6IYDIzyfZ6R/f?= =?Windows-1252?Q?3djyFu+s9mSzzXggDWGL8guwha60j21ktS2fK6bc6aivRgaOnCUBUkMu?= =?Windows-1252?Q?S4048xflbxmJ6DUd0HR/EV/BesHmP7KkjofiuZyq1YjEliyb3cf/HHHv?= =?Windows-1252?Q?4V/QCcm8TaDyGqEN7hIkxwDP7Wo/OiO5GKLTDA4FFgIN2rYtrWz18H/K?= =?Windows-1252?Q?bskgfWX9E9pCY7PM9dqDANWrKDpQT+VPzlB4uhvALVyqvGU3uYDRizCT?= =?Windows-1252?Q?9pF3GOLnSRXBMHKpx6vjt4MXg9W11H0WIpGEMeHmvbrI84LJAMUcjRnQ?= =?Windows-1252?Q?v6ipHXTYR5KcU5ZG+fD9bZhbM58T8inCWsf3L4HxK8aeJq9Q5MLO+FXY?= =?Windows-1252?Q?7G0Llm5cB3ZvDKVjq2OYNrNnUsbNrEoD2D8LDTSn36lfwg7xkev9p0I1?= =?Windows-1252?Q?r8ATP3iytrICt7gNojmjJSTnCdVRbXWOpTavOQxPON/eQAfGbk/oKGdR?= =?Windows-1252?Q?7dF3ke9aAvmuwPdOUrR2U/NuAQGlCMPkhOOgFt+FZfHVnGFJvuAUNtPy?= =?Windows-1252?Q?MJchVkvrRoaMZbVARi4eAtf3GZyhFtlPy9jm4cQDYwdZmJGFncw/CUNp?= =?Windows-1252?Q?zXn0g0fQnf7zUWW6Vx3gCBreqfp4uQTxWI+nD5Nm6Xa86OnMNZvNFrPn?= =?Windows-1252?Q?JfPG5KW2shZE2+lr8bQAqSg9lZCOdiw9uA27aqaVqRwevDf3xIlJlgRy?= =?Windows-1252?Q?cISesfEtdu9Pxx4GtM+pJTWdrmAaXZ/se+ZrEhgALW1m5vswr6vWycvG?= =?Windows-1252?Q?T+DwJdHpe14iExAG5udqSA75L2ycpONE6DmA7YnIq/nVaNCl32HAHY2v?= =?Windows-1252?Q?s3k5+5FOvoF0Z47FDrvqmQCxVy5eB71Ap4ri6nF8USslTZC0xjaubO8l?= =?Windows-1252?Q?nRTHDYwNCqacxpQ/LS+oZ96wzTw8u0YGVakBUZlN79916ozvh1e+gx4F?= =?Windows-1252?Q?qnb3BDPgobzXhYwDx+8qZ1W0Bxz7V0T2oKh/RwR0Cx5xPf34o4GqdSQU?= =?Windows-1252?Q?hPqCV2lLANAGUHTUB8l44t0UPfXN8Gk7QdyI/GBAzZvjRiQxc1iLaWu1?= =?Windows-1252?Q?4prqknnhWwBtxRcAcj5FD5ezZeQZupnM7Ji+dlRLsMg7VFWQAs5VMhAr?= =?Windows-1252?Q?XQ3pdItJoPm8yncza/Z60yWC2Mjf6LvBimPTpD+UiZmO5M3d9D9a5y2l?= =?Windows-1252?Q?mf+jDJaFcJPucdZXpBM3I/LxMtDOqDGVyUxJd7VV9HbDUdy5SURBFckG?= =?Windows-1252?Q?xpXuT4QA1sgSamf49C6AY5YHY3IUN5KTGxMUjJtUxQ4PVOvM/chN6i/7?= =?Windows-1252?Q?9naw1ZWUHt1rIBQ+kEZGVx3W+0UVF4jChH7pIE7RP7+AwjgOP0P2ndXP?= =?Windows-1252?B?QT09?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1443;6:jtUgYr+o+0Ip7WSgJe3B0BZX82uOYSs5N6NyLq16u9Q18FqW0ze2x7WzL3EEwu9Q3He7JAYsrkThGKRPMYpdsvv51TJvRhjfy+lIJ1ephhjWdtGG2mgsp7KhoNojYjQ8YOiXerlwlgxpQG9W79TzvlWipiC3CmHvFi9N/drUbh1PRe4WgrX2pWf69LHFYJcldmDESDZw4Nrvlq2pyTS05qioZmcCMQa7PiUGDOZyPmKAeahufxxpGkEVFeCelWJyL+ZRqyLJLpVycDq863KL15c12TXd1xozvAyi6C7tiTfw/2Kry2FHPscxwbbkCg20VtnwzABLGxZ9Fm4d7t33Gc5KR57+b7WTSZaay0Faq8fiij/J8x3vuwAMwAsGmJJ5E10oLwMQbLnL2w6ClRPK4w0wSRye95PRdWt2wSYd88Q=;5:hmAA9Z0eVtKMG+ul80bMBx/hN890IA8ye6wa/oSUpKB2CFcT9cqz9aN8blsJNivNSzUHQGVvwnfpk8DvOiHnImfEvVBoMYPC0eJb92Jcnv6onWHtoNah1Ux60z3xvXYTFseXj8aTBWDil12xJE9puNahPxtaifMkZaZSkMndUAg=;24:M79aZjpZ6JnvwXOsVBwigNyX+SYxugNl0ewnMUzr+/4Q/WxzRNDAI6vjJzgTGV5x8JGq41rufbrv7drjxMO3H/MVNAvq1xS6Hm2c6tfbZmA= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1443;7:X7ZQSYpcPjAkanTOMtjZfdfVNkzAkJPIWSwfYf1+EMUYesu1smnMT42kS60zv8NkYNE1FVwr2Zx5UK/qB+AewZfpAnW+i50DmqRKjH1293TmhPQvVVHzYFWT7/Ya8i3378QpIjozIrXYbjQ98MtYqh1Sp1HyZAdPJibV6ixcEfPV4arLqOjss6p17M37HI9a6914yJwgJiTgKgZ0Uon9vRdDckBPDnRpMwQ/JmLtos5/pN/f6pqRxPY1ORDhrqXHa3neYDaRkCq5aod8j2ucZipUzElO1NaCr+0bRZIMHPc1N7mvRv3UtVm+iHXDC6BZr6rm2/ao7GfHPQy21omFHbvk6XHEvfBtWTGm4Od9KyFASr8meAaQG32nyynbkFLzS0oYKh+64t2u7LlxDLoAu9nF+R+H1YYJkDb50cZVSErvHfSeKGhCtU+ItL3lJ2HryYOZyYRoa/VAFaDfvPU5XLitm1McufJwke7t8Kr6U4CNlgqr7x/IllbNnyjlk4139ErJXrB8++cYdZkge9WHOw==;20:qGY4QAyqtsHQG3r74xRzbKDd1QDkDQZkMEGXQkjcjgnnmnm4nUSQ14NChniFHU81Qgdx5B2iBxaq1QSwzCkqprRjOea6yQlupRxVYpsZtkKIg7vxt2w0ZPJjgaHlxvUfmqKf0vhzEpsiuoOUkG2r/OT+H+spGcJqo9pYhPXKMq0TR6AzhRcFFCrkB7/AmB+8dfJzALHIGo0qvFzYRixwmbVkyKWNET/43uragm5OOOlqxOzF+iyAIav565QooFCD X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2017 01:58:06.7377 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR12MB1443 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Peter, On 1/25/17 16:46, Peter Zijlstra wrote: > On Mon, Jan 16, 2017 at 01:23:36AM -0600, Suravee Suthikulpanit wrote: > >> + pi = container_of(event->pmu, struct perf_amd_iommu, pmu); >> + hwc->idx = pi->idx; >> + hwc->config = event->attr.config; >> + hwc->extra_reg.config = event->attr.config1; > >> static void perf_iommu_enable_event(struct perf_event *ev) >> { >> + struct hw_perf_event *hwc = &ev->hw; >> u8 csource = _GET_CSOURCE(ev); >> u16 devid = _GET_DEVID(ev); >> u8 bank = _GET_BANK(ev); >> @@ -253,30 +248,34 @@ static void perf_iommu_enable_event(struct perf_event *ev) >> u64 reg = 0ULL; >> >> reg = csource; >> - amd_iommu_pc_set_reg(0, bank, cntr, >> + amd_iommu_pc_set_reg(hwc->idx, bank, cntr, >> IOMMU_PC_COUNTER_SRC_REG, ®); > > Please explain about this IOMMU crud, this looks like fail. > hwc->idx should be the counter, not a random pmu index. Ok, I thought this was not used by the code, so I used it for holding PMU index. This might not be a good idea. I will change this part per Boris comment in the reply of this thread. > But instead it looks like you get the counter form: > > #define _GET_CNTR(ev) ((u8)(ev->hw.extra_reg.reg)) > > Which is absolutely insane. > So, the IOMMU counters are grouped into bank, and there could be many banks. I use the extra_reg.reg to hold the bank and counter indices. This will be used to program onto the counter configuration register. This is handled in get_next_avail_iommu_bnk_cntr() and clear_avail_iommu_bnk_cntr(). Thanks, Suravee