From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CB92C4708A for ; Wed, 26 May 2021 17:54:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 56DD4613D2 for ; Wed, 26 May 2021 17:54:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235159AbhEZR4P (ORCPT ); Wed, 26 May 2021 13:56:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234508AbhEZR4O (ORCPT ); Wed, 26 May 2021 13:56:14 -0400 Received: from mail-il1-x12a.google.com (mail-il1-x12a.google.com [IPv6:2607:f8b0:4864:20::12a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56C8DC061760 for ; Wed, 26 May 2021 10:54:42 -0700 (PDT) Received: by mail-il1-x12a.google.com with SMTP id k4so1691112ili.4 for ; Wed, 26 May 2021 10:54:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=subject:from:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=ACtspkCT/fGXAK2XAfeoVeEU1qeByhAQDFj8vp7z1Kw=; b=d1FGpRxRysaEV+riouzevu+1R3rw1n0T3AJ+3iiM+0TJAZT5/eoQscxY6vIbmrGOC4 yLD7Z7pbhiFgNKcwlv3Ecl27TrPcZndmuXiXU89xc17fy8oHwN6CykHAbCz8WTV2ouzk +MVA6GZKqWdz3RyZeZe0dJ/KzYkUvzjLmztlXZNJeAjZSpm8XppaK1uW8H25bmP+D6Vy 0uFso1DZPuM28/bJ7TX1jt5XotMoU8OiruIojax6KqUCNrCLICfRxsvK4b+vCe72ba+v KN/1BSKivxhQAFapgQmpO5oBF5AzkEoUO3dggcgvWs8MiXdMZdSwLjgPUS70taZoaVDi s5Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=ACtspkCT/fGXAK2XAfeoVeEU1qeByhAQDFj8vp7z1Kw=; b=d9pRvGbkoop+pzaxz2zhpLrhuP3PYwAvdd7sQrTQ10xqii4ykJAohcAlMkHwol0SgQ KdNiX9eYcrnC7DgcJeYNnoTRACkdNPmQV6GG9fHIrIr/MLTDr7AI0VMDtBjQzBBpYQky SG+4B/PK3b4jpXAtH7xthUCXVi45ZL7zD9Px0cuMppWyx4bxCUj7s0sox9ZFzkf8jO4b dwkJ8JPb51TPeSDbJevHKDfp4aW8X8gu2ktmoHw+tUDzzkz/FWYv3oSmAa/iLceIYbsR 7+/pzZoM2iIVX6y8hZexBpX3BAuuxvHdmvAwmCP7XqtUOBf4PCZCcxk/EfQzhX9Dnhst Lh+g== X-Gm-Message-State: AOAM533VWd24jUmotkEBvY3hnpSk4px4YCOSSbzCxfs/dBKRAMrNeZ1u VNxYwdgFqVn8l6DbEezka5yn2w== X-Google-Smtp-Source: ABdhPJybzwpznpNhH9BPdawI9/2rQI1ZannI33T0sQ0em8q+W0UoAy8iDp9ZoISdJbaKp0tEPuizQg== X-Received: by 2002:a05:6e02:54d:: with SMTP id i13mr27905858ils.26.1622051681431; Wed, 26 May 2021 10:54:41 -0700 (PDT) Received: from [192.168.1.30] ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id b10sm12182ioz.35.2021.05.26.10.54.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 26 May 2021 10:54:41 -0700 (PDT) Subject: Re: [RFC PATCH 2/9] audit,io_uring,io-wq: add some basic audit support to io_uring From: Jens Axboe To: Paul Moore Cc: Pavel Begunkov , linux-security-module@vger.kernel.org, selinux@vger.kernel.org, linux-audit@redhat.com, io-uring@vger.kernel.org, linux-fsdevel@vger.kernel.org, Kumar Kartikeya Dwivedi , Alexander Viro References: <162163367115.8379.8459012634106035341.stgit@sifl> <162163379461.8379.9691291608621179559.stgit@sifl> <162219f9-7844-0c78-388f-9b5c06557d06@gmail.com> <8943629d-3c69-3529-ca79-d7f8e2c60c16@kernel.dk> <9e69e4b6-2b87-a688-d604-c7f70be894f5@kernel.dk> <3bef7c8a-ee70-d91d-74db-367ad0137d00@kernel.dk> Message-ID: Date: Wed, 26 May 2021 11:54:40 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <3bef7c8a-ee70-d91d-74db-367ad0137d00@kernel.dk> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On 5/26/21 11:31 AM, Jens Axboe wrote: > On 5/26/21 11:15 AM, Jens Axboe wrote: >> On 5/25/21 8:04 PM, Paul Moore wrote: >>> On Tue, May 25, 2021 at 9:11 PM Jens Axboe wrote: >>>> On 5/24/21 1:59 PM, Paul Moore wrote: >>>>> That said, audit is not for everyone, and we have build time and >>>>> runtime options to help make life easier. Beyond simply disabling >>>>> audit at compile time a number of Linux distributions effectively >>>>> shortcut audit at runtime by adding a "never" rule to the audit >>>>> filter, for example: >>>>> >>>>> % auditctl -a task,never >>>> >>>> As has been brought up, the issue we're facing is that distros have >>>> CONFIG_AUDIT=y and hence the above is the best real world case outside >>>> of people doing custom kernels. My question would then be how much >>>> overhead the above will add, considering it's an entry/exit call per op. >>>> If auditctl is turned off, what is the expectation in turns of overhead? >>> >>> I commented on that case in my last email to Pavel, but I'll try to go >>> over it again in a little more detail. >>> >>> As we discussed earlier in this thread, we can skip the req->opcode >>> check before both the _entry and _exit calls, so we are left with just >>> the bare audit calls in the io_uring code. As the _entry and _exit >>> functions are small, I've copied them and their supporting functions >>> below and I'll try to explain what would happen in CONFIG_AUDIT=y, >>> "task,never" case. >>> >>> + static inline struct audit_context *audit_context(void) >>> + { >>> + return current->audit_context; >>> + } >>> >>> + static inline bool audit_dummy_context(void) >>> + { >>> + void *p = audit_context(); >>> + return !p || *(int *)p; >>> + } >>> >>> + static inline void audit_uring_entry(u8 op) >>> + { >>> + if (unlikely(audit_enabled && audit_context())) >>> + __audit_uring_entry(op); >>> + } >>> >>> We have one if statement where the conditional checks on two >>> individual conditions. The first (audit_enabled) is simply a check to >>> see if anyone has "turned on" auditing at runtime; historically this >>> worked rather well, and still does in a number of places, but ever >>> since systemd has taken to forcing audit on regardless of the admin's >>> audit configuration it is less useful. The second (audit_context()) >>> is a check to see if an audit_context has been allocated for the >>> current task. In the case of "task,never" current->audit_context will >>> be NULL (see audit_alloc()) and the __audit_uring_entry() slowpath >>> will never be called. >>> >>> Worst case here is checking the value of audit_enabled and >>> current->audit_context. Depending on which you think is more likely >>> we can change the order of the check so that the >>> current->audit_context check is first if you feel that is more likely >>> to be NULL than audit_enabled is to be false (it may be that way now). >>> >>> + static inline void audit_uring_exit(int success, long code) >>> + { >>> + if (unlikely(!audit_dummy_context())) >>> + __audit_uring_exit(success, code); >>> + } >>> >>> The exit call is very similar to the entry call, but in the >>> "task,never" case it is very simple as the first check to be performed >>> is the current->audit_context check which we know to be NULL. The >>> __audit_uring_exit() slowpath will never be called. >> >> I actually ran some numbers this morning. The test base is 5.13+, and >> CONFIG_AUDIT=y and CONFIG_AUDITSYSCALL=y is set for both the baseline >> test and the test with this series applied. I used your git branch as of >> this morning. >> >> The test case is my usual peak perf test, which is random reads at >> QD=128 and using polled IO. It's a single core test, not threaded. I ran >> two different tests - one was having a thread just do the IO, the other >> is using SQPOLL to do the IO for us. The device is capable than more >> IOPS than a single core can deliver, so we're CPU limited in this test. >> Hence it's a good test case as it does actual work, and shows software >> overhead quite nicely. Runs are very stable (less than 0.5% difference >> between runs on the same base), yet I did average 4 runs. >> >> Kernel SQPOLL IOPS Perf diff >> --------------------------------------------------------- >> 5.13 0 3029872 0.0% >> 5.13 1 3031056 0.0% >> 5.13 + audit 0 2894160 -4.5% >> 5.13 + audit 1 2886168 -4.8% >> >> That's an immediate drop in perf of almost 5%. Looking at a quick >> profile of it (nothing fancy, just checking for 'audit' in the profile) >> shows this: >> >> + 2.17% io_uring [kernel.vmlinux] [k] __audit_uring_entry >> + 0.71% io_uring [kernel.vmlinux] [k] __audit_uring_exit >> 0.07% io_uring [kernel.vmlinux] [k] __audit_syscall_entry >> 0.02% io_uring [kernel.vmlinux] [k] __audit_syscall_exit >> >> Note that this is with _no_ rules! > > io_uring also supports a NOP command, which basically just measures > reqs/sec through the interface. Ran that as well: > > Kernel SQPOLL IOPS Perf diff > --------------------------------------------------------- > 5.13 0 31.05M 0.0% > 5.13 + audit 0 25.31M -18.5% > > and profile for the latter includes: > > + 5.19% io_uring [kernel.vmlinux] [k] __audit_uring_entry > + 4.31% io_uring [kernel.vmlinux] [k] __audit_uring_exit > 0.26% io_uring [kernel.vmlinux] [k] __audit_syscall_entry > 0.08% io_uring [kernel.vmlinux] [k] __audit_syscall_exit As Pavel correctly pointed it, looks like auditing is enabled. And indeed it was! Hence the above numbers is without having turned off auditing. Running the NOPs after having turned off audit, we get 30.6M IOPS, which is down about 1.5% from the baseline. The results for the polled random read test above did _not_ change from this, they are still down the same amount. Note, and I should have included this in the first email, this is not any kind of argument for or against audit logging. It's purely meant to be a set of numbers that show how the current series impacts performance. -- Jens Axboe From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39FC9C47082 for ; Wed, 26 May 2021 18:11:21 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 921CB613D7 for ; Wed, 26 May 2021 18:11:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 921CB613D7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=linux-audit-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-229-Joaea1x6M-ObVhtqvttFiQ-1; Wed, 26 May 2021 14:11:17 -0400 X-MC-Unique: Joaea1x6M-ObVhtqvttFiQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E26051007477; Wed, 26 May 2021 18:11:13 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1BB5E5D6D3; Wed, 26 May 2021 18:11:13 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id D385B180B463; Wed, 26 May 2021 18:11:11 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 14QHspQ4023948 for ; Wed, 26 May 2021 13:54:51 -0400 Received: by smtp.corp.redhat.com (Postfix) id 60DF410E60FC; Wed, 26 May 2021 17:54:51 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast05.extmail.prod.ext.rdu2.redhat.com [10.11.55.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 59BF210E60F6 for ; Wed, 26 May 2021 17:54:45 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E48C983395C for ; Wed, 26 May 2021 17:54:44 +0000 (UTC) Received: from mail-il1-f178.google.com (mail-il1-f178.google.com [209.85.166.178]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-55--N3Za5X8P1CNO4AaFejUYQ-1; Wed, 26 May 2021 13:54:42 -0400 X-MC-Unique: -N3Za5X8P1CNO4AaFejUYQ-1 Received: by mail-il1-f178.google.com with SMTP id w2so1676547ilo.7 for ; Wed, 26 May 2021 10:54:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=ACtspkCT/fGXAK2XAfeoVeEU1qeByhAQDFj8vp7z1Kw=; b=MV1kyLhG0Fm1PklcxOq65S3XNEAOFxLqyl1w3iO37wyHTglarM1U08W3cpLKagbIkB 7Jb3yknk30Rg3FF+ZhR78cpKc4UOInjADLmVW4u/E1q9uSkcbu09tFbSyiKAQTRnDdZd XdYBI9/WBvXu4OZy8qzEM99p7EvlFU+HET3lZQOajtFKwO3UkFXLc+eVwKxnR/A5RC9R mv0w8fTK6BuxYDSipEqXDYtxa0ALASTxbtclZgh+YCwFgXPfoxMVSL0bymBBJ8XVZz3A +iVwhH1Kv61FRwmOOKmxVuLWfpwzAXXy+YH7XsBgEIv8RZckqlDckTxsPLbhW+xekZLS AlWQ== X-Gm-Message-State: AOAM530v18tjS8VSwBQEI1lfpG9UfIVjGXjoB2vW+rAjogFJPhSnZLuY 1osRJvhqsQQpEM67evHm7wWXvQ== X-Google-Smtp-Source: ABdhPJybzwpznpNhH9BPdawI9/2rQI1ZannI33T0sQ0em8q+W0UoAy8iDp9ZoISdJbaKp0tEPuizQg== X-Received: by 2002:a05:6e02:54d:: with SMTP id i13mr27905858ils.26.1622051681431; Wed, 26 May 2021 10:54:41 -0700 (PDT) Received: from [192.168.1.30] ([65.144.74.34]) by smtp.gmail.com with ESMTPSA id b10sm12182ioz.35.2021.05.26.10.54.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 26 May 2021 10:54:41 -0700 (PDT) Subject: Re: [RFC PATCH 2/9] audit,io_uring,io-wq: add some basic audit support to io_uring From: Jens Axboe To: Paul Moore References: <162163367115.8379.8459012634106035341.stgit@sifl> <162163379461.8379.9691291608621179559.stgit@sifl> <162219f9-7844-0c78-388f-9b5c06557d06@gmail.com> <8943629d-3c69-3529-ca79-d7f8e2c60c16@kernel.dk> <9e69e4b6-2b87-a688-d604-c7f70be894f5@kernel.dk> <3bef7c8a-ee70-d91d-74db-367ad0137d00@kernel.dk> Message-ID: Date: Wed, 26 May 2021 11:54:40 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <3bef7c8a-ee70-d91d-74db-367ad0137d00@kernel.dk> X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-loop: linux-audit@redhat.com X-Mailman-Approved-At: Wed, 26 May 2021 14:09:40 -0400 Cc: selinux@vger.kernel.org, io-uring@vger.kernel.org, linux-security-module@vger.kernel.org, linux-audit@redhat.com, Kumar Kartikeya Dwivedi , linux-fsdevel@vger.kernel.org, Pavel Begunkov , Alexander Viro X-BeenThere: linux-audit@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Linux Audit Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=linux-audit-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On 5/26/21 11:31 AM, Jens Axboe wrote: > On 5/26/21 11:15 AM, Jens Axboe wrote: >> On 5/25/21 8:04 PM, Paul Moore wrote: >>> On Tue, May 25, 2021 at 9:11 PM Jens Axboe wrote: >>>> On 5/24/21 1:59 PM, Paul Moore wrote: >>>>> That said, audit is not for everyone, and we have build time and >>>>> runtime options to help make life easier. Beyond simply disabling >>>>> audit at compile time a number of Linux distributions effectively >>>>> shortcut audit at runtime by adding a "never" rule to the audit >>>>> filter, for example: >>>>> >>>>> % auditctl -a task,never >>>> >>>> As has been brought up, the issue we're facing is that distros have >>>> CONFIG_AUDIT=y and hence the above is the best real world case outside >>>> of people doing custom kernels. My question would then be how much >>>> overhead the above will add, considering it's an entry/exit call per op. >>>> If auditctl is turned off, what is the expectation in turns of overhead? >>> >>> I commented on that case in my last email to Pavel, but I'll try to go >>> over it again in a little more detail. >>> >>> As we discussed earlier in this thread, we can skip the req->opcode >>> check before both the _entry and _exit calls, so we are left with just >>> the bare audit calls in the io_uring code. As the _entry and _exit >>> functions are small, I've copied them and their supporting functions >>> below and I'll try to explain what would happen in CONFIG_AUDIT=y, >>> "task,never" case. >>> >>> + static inline struct audit_context *audit_context(void) >>> + { >>> + return current->audit_context; >>> + } >>> >>> + static inline bool audit_dummy_context(void) >>> + { >>> + void *p = audit_context(); >>> + return !p || *(int *)p; >>> + } >>> >>> + static inline void audit_uring_entry(u8 op) >>> + { >>> + if (unlikely(audit_enabled && audit_context())) >>> + __audit_uring_entry(op); >>> + } >>> >>> We have one if statement where the conditional checks on two >>> individual conditions. The first (audit_enabled) is simply a check to >>> see if anyone has "turned on" auditing at runtime; historically this >>> worked rather well, and still does in a number of places, but ever >>> since systemd has taken to forcing audit on regardless of the admin's >>> audit configuration it is less useful. The second (audit_context()) >>> is a check to see if an audit_context has been allocated for the >>> current task. In the case of "task,never" current->audit_context will >>> be NULL (see audit_alloc()) and the __audit_uring_entry() slowpath >>> will never be called. >>> >>> Worst case here is checking the value of audit_enabled and >>> current->audit_context. Depending on which you think is more likely >>> we can change the order of the check so that the >>> current->audit_context check is first if you feel that is more likely >>> to be NULL than audit_enabled is to be false (it may be that way now). >>> >>> + static inline void audit_uring_exit(int success, long code) >>> + { >>> + if (unlikely(!audit_dummy_context())) >>> + __audit_uring_exit(success, code); >>> + } >>> >>> The exit call is very similar to the entry call, but in the >>> "task,never" case it is very simple as the first check to be performed >>> is the current->audit_context check which we know to be NULL. The >>> __audit_uring_exit() slowpath will never be called. >> >> I actually ran some numbers this morning. The test base is 5.13+, and >> CONFIG_AUDIT=y and CONFIG_AUDITSYSCALL=y is set for both the baseline >> test and the test with this series applied. I used your git branch as of >> this morning. >> >> The test case is my usual peak perf test, which is random reads at >> QD=128 and using polled IO. It's a single core test, not threaded. I ran >> two different tests - one was having a thread just do the IO, the other >> is using SQPOLL to do the IO for us. The device is capable than more >> IOPS than a single core can deliver, so we're CPU limited in this test. >> Hence it's a good test case as it does actual work, and shows software >> overhead quite nicely. Runs are very stable (less than 0.5% difference >> between runs on the same base), yet I did average 4 runs. >> >> Kernel SQPOLL IOPS Perf diff >> --------------------------------------------------------- >> 5.13 0 3029872 0.0% >> 5.13 1 3031056 0.0% >> 5.13 + audit 0 2894160 -4.5% >> 5.13 + audit 1 2886168 -4.8% >> >> That's an immediate drop in perf of almost 5%. Looking at a quick >> profile of it (nothing fancy, just checking for 'audit' in the profile) >> shows this: >> >> + 2.17% io_uring [kernel.vmlinux] [k] __audit_uring_entry >> + 0.71% io_uring [kernel.vmlinux] [k] __audit_uring_exit >> 0.07% io_uring [kernel.vmlinux] [k] __audit_syscall_entry >> 0.02% io_uring [kernel.vmlinux] [k] __audit_syscall_exit >> >> Note that this is with _no_ rules! > > io_uring also supports a NOP command, which basically just measures > reqs/sec through the interface. Ran that as well: > > Kernel SQPOLL IOPS Perf diff > --------------------------------------------------------- > 5.13 0 31.05M 0.0% > 5.13 + audit 0 25.31M -18.5% > > and profile for the latter includes: > > + 5.19% io_uring [kernel.vmlinux] [k] __audit_uring_entry > + 4.31% io_uring [kernel.vmlinux] [k] __audit_uring_exit > 0.26% io_uring [kernel.vmlinux] [k] __audit_syscall_entry > 0.08% io_uring [kernel.vmlinux] [k] __audit_syscall_exit As Pavel correctly pointed it, looks like auditing is enabled. And indeed it was! Hence the above numbers is without having turned off auditing. Running the NOPs after having turned off audit, we get 30.6M IOPS, which is down about 1.5% from the baseline. The results for the polled random read test above did _not_ change from this, they are still down the same amount. Note, and I should have included this in the first email, this is not any kind of argument for or against audit logging. It's purely meant to be a set of numbers that show how the current series impacts performance. -- Jens Axboe -- Linux-audit mailing list Linux-audit@redhat.com https://listman.redhat.com/mailman/listinfo/linux-audit