Re: [PATCH bpf-next v5 5/5] error-injection: Support fault injection framework

[Masami Hiramatsu <mhiramat@kernel.org>]

On Sat, 13 Jan 2018 22:28:29 +0900
Akinobu Mita <akinobu.mita@gmail.com> wrote:

> 2018-01-13 2:56 GMT+09:00 Masami Hiramatsu <mhiramat@kernel.org>:
> > Support in-kernel fault-injection framework via debugfs.
> > This allows you to inject a conditional error to specified
> > function using debugfs interfaces.
> >
> > Here is the result of test script described in
> > Documentation/fault-injection/fault-injection.txt
> >
> >   ===========
> >   # ./test_fail_function.sh
> >   1+0 records in
> >   1+0 records out
> >   1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227404 s, 46.1 MB/s
> >   btrfs-progs v4.4
> >   See http://btrfs.wiki.kernel.org for more information.
> >
> >   Label:              (null)
> >   UUID:               bfa96010-12e9-4360-aed0-42eec7af5798
> >   Node size:          16384
> >   Sector size:        4096
> >   Filesystem size:    1001.00MiB
> >   Block group profiles:
> >     Data:             single            8.00MiB
> >     Metadata:         DUP              58.00MiB
> >     System:           DUP              12.00MiB
> >   SSD detected:       no
> >   Incompat features:  extref, skinny-metadata
> >   Number of devices:  1
> >   Devices:
> >      ID        SIZE  PATH
> >       1  1001.00MiB  /dev/loop2
> >
> >   mount: mount /dev/loop2 on /opt/tmpmnt failed: Cannot allocate memory
> >   SUCCESS!
> >   ===========
> >
> >
> > Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> > Reviewed-by: Josef Bacik <jbacik@fb.com>
> > ---
> >   Changes in v3:
> >    - Check and adjust error value for each target function
> >    - Clear kporbe flag for reuse
> >    - Add more documents and example
> >   Changes in v5:
> >    - Support multi-function error injection
> > ---
> >  Documentation/fault-injection/fault-injection.txt |   68 ++++
> >  kernel/Makefile                                   |    1
> >  kernel/fail_function.c                            |  349 +++++++++++++++++++++
> >  lib/Kconfig.debug                                 |   10 +
> >  4 files changed, 428 insertions(+)
> >  create mode 100644 kernel/fail_function.c
> >
> > diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt
> > index 918972babcd8..f4a32463ca48 100644
> > --- a/Documentation/fault-injection/fault-injection.txt
> > +++ b/Documentation/fault-injection/fault-injection.txt
> > @@ -30,6 +30,12 @@ o fail_mmc_request
> >    injects MMC data errors on devices permitted by setting
> >    debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
> >
> > +o fail_function
> > +
> > +  injects error return on specific functions, which are marked by
> > +  ALLOW_ERROR_INJECTION() macro, by setting debugfs entries
> > +  under /sys/kernel/debug/fail_function. No boot option supported.
> > +
> >  Configure fault-injection capabilities behavior
> >  -----------------------------------------------
> >
> > @@ -123,6 +129,29 @@ configuration of fault-injection capabilities.
> >         default is 'N', setting it to 'Y' will disable failure injections
> >         when dealing with private (address space) futexes.
> >
> > +- /sys/kernel/debug/fail_function/inject:
> > +
> > +       Format: { 'function-name' | '!function-name' | '' }
> > +       specifies the target function of error injection by name.
> > +       If the function name leads '!' prefix, given function is
> > +       removed from injection list. If nothing specified ('')
> > +       injection list is cleared.
> > +
> > +- /sys/kernel/debug/fail_function/injectable:
> > +
> > +       (read only) shows error injectable functions and what type of
> > +       error values can be specified. The error type will be one of
> > +       below;
> > +       - NULL: retval must be 0.
> > +       - ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
> > +       - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
> > +
> > +- /sys/kernel/debug/fail_function/<functiuon-name>/retval:
> > +
> > +       specifies the "error" return value to inject to the given
> > +       function for given function. This will be created when
> > +       user specifies new injection entry.
> > +
> >  o Boot option
> >
> >  In order to inject faults while debugfs is not available (early boot time),
> > @@ -268,6 +297,45 @@ trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
> >  echo "Injecting errors into the module $module... (interrupt to stop)"
> >  sleep 1000000
> >
> > +------------------------------------------------------------------------------
> > +
> > +o Inject open_ctree error while btrfs mount
> > +
> > +#!/bin/bash
> > +
> > +rm -f testfile.img
> > +dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1
> > +DEVICE=$(losetup --show -f testfile.img)
> > +mkfs.btrfs -f $DEVICE
> > +mkdir -p tmpmnt
> > +
> > +FAILTYPE=fail_function
> > +FAILFUNC=open_ctree
> > +echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
> > +echo -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
> > +echo N > /sys/kernel/debug/$FAILTYPE/task-filter
> > +echo 100 > /sys/kernel/debug/$FAILTYPE/probability
> > +echo 0 > /sys/kernel/debug/$FAILTYPE/interval
> > +echo -1 > /sys/kernel/debug/$FAILTYPE/times
> > +echo 0 > /sys/kernel/debug/$FAILTYPE/space
> > +echo 1 > /sys/kernel/debug/$FAILTYPE/verbose
> 
> I expected that the fault_attr is created for each target function.
> (i.e. /sys/kernel/debug/fail_function/<func>/ directory contains not
> only 'retval', but also 'probability', 'interval', ...)
> 
> Or is there any good reason to share a single fault_attr for all
> error injected functions?

Good question. I actually considered that. But since fault-attr only
considers no relationship between 2 fault points (in time too),
this means each time should_fail() will be done as independent trial.
So, in most case user will just copy the setting for those 2 points.
(another reason is not to change the document so much :), all interface
are still under /sys/kernel/debug/fail_*/ ...)

Thank you,

-- 
Masami Hiramatsu <mhiramat@kernel.org>