From mboxrd@z Thu Jan 1 00:00:00 1970 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754710AbeANBG6 (ORCPT + 1 other); Sat, 13 Jan 2018 20:06:58 -0500 Received: from mail.kernel.org ([198.145.29.99]:33028 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752945AbeANBG4 (ORCPT ); Sat, 13 Jan 2018 20:06:56 -0500 DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E85DC21775 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=mhiramat@kernel.org Date: Sun, 14 Jan 2018 10:06:51 +0900 From: Masami Hiramatsu To: Akinobu Mita Cc: mhiramat@kernel.org, Alexei Starovoitov , Josef Bacik , Steven Rostedt , Ingo Molnar , "David S. Miller" , netdev , LKML , ast@kernel.org, kernel-team@fb.com, daniel@iogearbox.net, linux-btrfs@vger.kernel.org, darrick.wong@oracle.com, Josef Bacik Subject: Re: [PATCH bpf-next v5 5/5] error-injection: Support fault injection framework Message-Id: <20180114100651.d0d22c2b5a839b399fd08414@kernel.org> In-Reply-To: References: <151577961239.17713.16735014086815016675.stgit@devbox> <151577976308.17713.15328322734236027874.stgit@devbox> X-Mailer: Sylpheed 3.5.1 (GTK+ 2.24.31; x86_64-redhat-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Return-Path: On Sat, 13 Jan 2018 22:28:29 +0900 Akinobu Mita wrote: > 2018-01-13 2:56 GMT+09:00 Masami Hiramatsu : > > Support in-kernel fault-injection framework via debugfs. > > This allows you to inject a conditional error to specified > > function using debugfs interfaces. > > > > Here is the result of test script described in > > Documentation/fault-injection/fault-injection.txt > > > > =========== > > # ./test_fail_function.sh > > 1+0 records in > > 1+0 records out > > 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227404 s, 46.1 MB/s > > btrfs-progs v4.4 > > See http://btrfs.wiki.kernel.org for more information. > > > > Label: (null) > > UUID: bfa96010-12e9-4360-aed0-42eec7af5798 > > Node size: 16384 > > Sector size: 4096 > > Filesystem size: 1001.00MiB > > Block group profiles: > > Data: single 8.00MiB > > Metadata: DUP 58.00MiB > > System: DUP 12.00MiB > > SSD detected: no > > Incompat features: extref, skinny-metadata > > Number of devices: 1 > > Devices: > > ID SIZE PATH > > 1 1001.00MiB /dev/loop2 > > > > mount: mount /dev/loop2 on /opt/tmpmnt failed: Cannot allocate memory > > SUCCESS! > > =========== > > > > > > Signed-off-by: Masami Hiramatsu > > Reviewed-by: Josef Bacik > > --- > > Changes in v3: > > - Check and adjust error value for each target function > > - Clear kporbe flag for reuse > > - Add more documents and example > > Changes in v5: > > - Support multi-function error injection > > --- > > Documentation/fault-injection/fault-injection.txt | 68 ++++ > > kernel/Makefile | 1 > > kernel/fail_function.c | 349 +++++++++++++++++++++ > > lib/Kconfig.debug | 10 + > > 4 files changed, 428 insertions(+) > > create mode 100644 kernel/fail_function.c > > > > diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt > > index 918972babcd8..f4a32463ca48 100644 > > --- a/Documentation/fault-injection/fault-injection.txt > > +++ b/Documentation/fault-injection/fault-injection.txt > > @@ -30,6 +30,12 @@ o fail_mmc_request > > injects MMC data errors on devices permitted by setting > > debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request > > > > +o fail_function > > + > > + injects error return on specific functions, which are marked by > > + ALLOW_ERROR_INJECTION() macro, by setting debugfs entries > > + under /sys/kernel/debug/fail_function. No boot option supported. > > + > > Configure fault-injection capabilities behavior > > ----------------------------------------------- > > > > @@ -123,6 +129,29 @@ configuration of fault-injection capabilities. > > default is 'N', setting it to 'Y' will disable failure injections > > when dealing with private (address space) futexes. > > > > +- /sys/kernel/debug/fail_function/inject: > > + > > + Format: { 'function-name' | '!function-name' | '' } > > + specifies the target function of error injection by name. > > + If the function name leads '!' prefix, given function is > > + removed from injection list. If nothing specified ('') > > + injection list is cleared. > > + > > +- /sys/kernel/debug/fail_function/injectable: > > + > > + (read only) shows error injectable functions and what type of > > + error values can be specified. The error type will be one of > > + below; > > + - NULL: retval must be 0. > > + - ERRNO: retval must be -1 to -MAX_ERRNO (-4096). > > + - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096). > > + > > +- /sys/kernel/debug/fail_function//retval: > > + > > + specifies the "error" return value to inject to the given > > + function for given function. This will be created when > > + user specifies new injection entry. > > + > > o Boot option > > > > In order to inject faults while debugfs is not available (early boot time), > > @@ -268,6 +297,45 @@ trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT > > echo "Injecting errors into the module $module... (interrupt to stop)" > > sleep 1000000 > > > > +------------------------------------------------------------------------------ > > + > > +o Inject open_ctree error while btrfs mount > > + > > +#!/bin/bash > > + > > +rm -f testfile.img > > +dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1 > > +DEVICE=$(losetup --show -f testfile.img) > > +mkfs.btrfs -f $DEVICE > > +mkdir -p tmpmnt > > + > > +FAILTYPE=fail_function > > +FAILFUNC=open_ctree > > +echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject > > +echo -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval > > +echo N > /sys/kernel/debug/$FAILTYPE/task-filter > > +echo 100 > /sys/kernel/debug/$FAILTYPE/probability > > +echo 0 > /sys/kernel/debug/$FAILTYPE/interval > > +echo -1 > /sys/kernel/debug/$FAILTYPE/times > > +echo 0 > /sys/kernel/debug/$FAILTYPE/space > > +echo 1 > /sys/kernel/debug/$FAILTYPE/verbose > > I expected that the fault_attr is created for each target function. > (i.e. /sys/kernel/debug/fail_function// directory contains not > only 'retval', but also 'probability', 'interval', ...) > > Or is there any good reason to share a single fault_attr for all > error injected functions? Good question. I actually considered that. But since fault-attr only considers no relationship between 2 fault points (in time too), this means each time should_fail() will be done as independent trial. So, in most case user will just copy the setting for those 2 points. (another reason is not to change the document so much :), all interface are still under /sys/kernel/debug/fail_*/ ...) Thank you, -- Masami Hiramatsu