[patch 0/6] RFC: fault-injection capabilities (v2)

[patch 0/6] RFC: fault-injection capabilities (v2)

[Akinobu Mita]

Changes from last version

- use lightweight random simulator instead of get_random_int()
- added per queue filter for disk IO failures
  (/sys/blocks/sda/sda1/make-it-fail, /sys/blocks/sda/make-it-fail)
- added process filter
  (/debug/{failslab,fail_page_alloc,fail_make_request}/process-filter,
   /proc/<pid>/make-it-fail)

---
This patch set provides some fault-injection capabilities.

- kmalloc failures

- alloc_pages() failures

- disk IO errors

We can see what really happens if those failures happen.

In order to enable these fault-injection capabilities:

1. Enable relevant config options (CONFIG_FAILSLAB, CONFIG_PAGE_ALLOC,
   CONFIG_MAKE_REQUEST) and runtime configuration kernel module
   (CONFIG_SHOULD_FAIL_KNOBS)

2. build and boot with this kernel

3. modprobe should_fail_knob

4. configure fault-injection capabilities behavior by debugfs

For example about kmalloc failures:

/debug/failslab/probability

	specifies how often it should fail in percent.

/debug/failslab/interval

	specifies the interval of failures.

/debug/failslab/times

	specifies how many times failures may happen at most.

/debug/failslab/space

	specifies the size of free space where memory can be allocated
	safely in bytes.

/debug/failslab/process-filter

	enable process filter.

5. see what really happens.

[patch 1/6] fault-injection capabilities infrastructure

[Akinobu Mita]

[-- Attachment #1: should-fail.patch --]
[-- Type: text/plain, Size: 4597 bytes --]

This patch provides some functions for implement fault-injection
capabilities.

Lightweight random simulator is taken from crasher module for SUSE kernel.
The function should_fail() is taken from failmalloc-1.0
(http://www.nongnu.org/failmalloc/)

Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 include/linux/should_fail.h |   45 ++++++++++++++++++++++++
 lib/Kconfig.debug           |    4 ++
 lib/Makefile                |    1 
 lib/should_fail.c           |   82 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 132 insertions(+)

Index: work-shouldfail/include/linux/should_fail.h
===================================================================
--- /dev/null
+++ work-shouldfail/include/linux/should_fail.h
@@ -0,0 +1,45 @@
+#ifndef _LINUX_SHOULD_FAIL_H
+#define _LINUX_SHOULD_FAIL_H
+
+#ifdef CONFIG_SHOULD_FAIL
+
+#include <linux/types.h>
+#include <asm/atomic.h>
+
+struct should_fail_data {
+
+	/* how often it should fail in percent. */
+	unsigned long probability;
+
+	/* the interval of failures. */
+	unsigned long interval;
+
+	/*
+	 * how many times failures may happen at most.
+	 * A value of '-1' means infinity.
+	 */
+	atomic_t times;
+
+	/*
+	 * the size of free space where memory can be allocated safely.
+	 * A value of '0' means infinity.
+	 */
+	atomic_t space;
+
+	unsigned long count;
+};
+
+#define DEFINE_SHOULD_FAIL(name) \
+	struct should_fail_data name = { .times = ATOMIC_INIT(-1), }
+
+int setup_should_fail(struct should_fail_data *data, char *str);
+void should_fail_srandom(unsigned long entropy);
+int should_fail(struct should_fail_data *data, ssize_t size);
+
+#else
+
+#define should_fail(data, size)	(0)
+
+#endif /* CONFIG_SHOULD_FAIL */
+
+#endif /* _LINUX_SHOULD_FAIL_H */
Index: work-shouldfail/lib/should_fail.c
===================================================================
--- /dev/null
+++ work-shouldfail/lib/should_fail.c
@@ -0,0 +1,82 @@
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/random.h>
+#include <linux/stat.h>
+#include <linux/types.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/should_fail.h>
+
+int setup_should_fail(struct should_fail_data *data, char *str)
+{
+	unsigned long probability;
+	unsigned long interval;
+	int times;
+	int space;
+
+	/* "<probability>,<interval>,<times>,<space>" */
+	if (sscanf(str, "%lu,%lu,%d,%d", &probability, &interval, &times,
+		   &space) < 4)
+		return 0;
+
+	data->probability = probability;
+	data->interval = interval;
+	atomic_set(&data->times, times);
+	atomic_set(&data->space, space);
+
+	return 1;
+}
+
+#define failure_probability(data)	(data)->probability
+#define failure_interval(data)		(data)->interval
+#define max_failures(data)		(data)->times
+#define current_space(data)		(data)->space
+#define atomic_dec_not_zero(v)		atomic_add_unless((v), -1, 0)
+
+static unsigned long rand_seed = 152L;
+
+static unsigned long should_fail_random(void)
+{
+	rand_seed = rand_seed * 690690L+1;
+	return rand_seed ^ jiffies;
+}
+
+void should_fail_srandom(unsigned long entropy)
+{
+	rand_seed ^= entropy;
+	should_fail_random();
+}
+
+/*
+ * This code is stolen from failmalloc-1.0
+ * http://www.nongnu.org/failmalloc/
+ */
+
+int should_fail(struct should_fail_data *data, ssize_t size)
+{
+	if (atomic_read(&max_failures(data)) == 0)
+		return 0;
+
+	if (atomic_read(&current_space(data)) > size) {
+		atomic_sub(size, &current_space(data));
+		return 0;
+	}
+
+	if (failure_interval(data) > 1) {
+		data->count++;
+		if (data->count % failure_interval(data))
+			return 0;
+	}
+
+	if (failure_probability(data) > should_fail_random() % 100)
+		goto fail;
+
+	return 0;
+
+fail:
+
+	if (atomic_read(&max_failures(data)) != -1)
+		atomic_dec_not_zero(&max_failures(data));
+
+	return 1;
+}
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -368,3 +368,7 @@ config RCU_TORTURE_TEST
 	  at boot time (you probably don't).
 	  Say M if you want the RCU torture tests to build as a module.
 	  Say N if you are unsure.
+
+config SHOULD_FAIL
+	bool
+
Index: work-shouldfail/lib/Makefile
===================================================================
--- work-shouldfail.orig/lib/Makefile
+++ work-shouldfail/lib/Makefile
@@ -51,6 +51,7 @@ obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o
 obj-$(CONFIG_SMP) += percpu_counter.o
 
 obj-$(CONFIG_SWIOTLB) += swiotlb.o
+obj-$(CONFIG_SHOULD_FAIL) += should_fail.o
 
 hostprogs-y	:= gen_crc32table
 clean-files	:= crc32table.h

--

[patch 2/6] fault-injection capability for kmalloc.

[Akinobu Mita]

[-- Attachment #1: failslab.patch --]
[-- Type: text/plain, Size: 2972 bytes --]

This patch provides fault-injection capability for kmalloc.

Boot option:

	failslab=<probability>,<interval>,<times>,<space>

	<probability>

		specifies how often it should fail in percent.

	<interval>

		specifies the interval of failures.

	<times>

		specifies how many times failures may happen at most.

	<space>

		specifies the size of free space where memory can be allocated
		safely in bytes.

Example:

	failslab=100,10,-1,0

slab allocation (kmalloc, kmem_cache_alloc,..) fails once per 10 times.

Cc: Pekka Enberg <penberg@cs.helsinki.fi>
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 include/linux/should_fail.h |    4 ++++
 lib/Kconfig.debug           |    7 +++++++
 mm/slab.c                   |   21 +++++++++++++++++++++
 3 files changed, 32 insertions(+)

Index: work-shouldfail/mm/slab.c
===================================================================
--- work-shouldfail.orig/mm/slab.c
+++ work-shouldfail/mm/slab.c
@@ -108,6 +108,7 @@
 #include	<linux/mempolicy.h>
 #include	<linux/mutex.h>
 #include	<linux/rtmutex.h>
+#include	<linux/should_fail.h>
 
 #include	<asm/uaccess.h>
 #include	<asm/cacheflush.h>
@@ -2963,11 +2964,31 @@ static void *cache_alloc_debugcheck_afte
 #define cache_alloc_debugcheck_after(a,b,objp,d) (objp)
 #endif
 
+#ifdef CONFIG_FAILSLAB
+
+static DEFINE_SHOULD_FAIL(failslab_data);
+
+static int __init setup_failslab(char *str)
+{
+	should_fail_srandom(jiffies);
+	return setup_should_fail(&failslab_data, str);
+}
+__setup("failslab=", setup_failslab);
+
+struct should_fail_data *failslab = &failslab_data;
+EXPORT_SYMBOL_GPL(failslab);
+
+#endif
+
 static inline void *____cache_alloc(struct kmem_cache *cachep, gfp_t flags)
 {
 	void *objp;
 	struct array_cache *ac;
 
+	if (!(flags & __GFP_NOFAIL) && cachep != &cache_cache &&
+	    should_fail(failslab, obj_size(cachep)))
+		return NULL;
+
 #ifdef CONFIG_NUMA
 	if (unlikely(current->flags & (PF_SPREAD_SLAB | PF_MEMPOLICY))) {
 		objp = alternate_node_alloc(cachep, flags);
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -372,3 +372,10 @@ config RCU_TORTURE_TEST
 config SHOULD_FAIL
 	bool
 
+config FAILSLAB
+	bool "fault-injection capabilitiy for kmalloc"
+	depends on DEBUG_KERNEL
+	select SHOULD_FAIL
+	help
+	  This option provides fault-injection capabilitiy for kmalloc.
+
Index: work-shouldfail/include/linux/should_fail.h
===================================================================
--- work-shouldfail.orig/include/linux/should_fail.h
+++ work-shouldfail/include/linux/should_fail.h
@@ -36,6 +36,10 @@ int setup_should_fail(struct should_fail
 void should_fail_srandom(unsigned long entropy);
 int should_fail(struct should_fail_data *data, ssize_t size);
 
+#ifdef CONFIG_FAILSLAB
+extern struct should_fail_data *failslab;
+#endif
+
 #else
 
 #define should_fail(data, size)	(0)

--

[patch 3/6] fault-injection capability for alloc_pages()

[Akinobu Mita]

[-- Attachment #1: fail_alloc_pages.patch --]
[-- Type: text/plain, Size: 2915 bytes --]

This patch provides fault-injection capability for alloc_pages()

boot option:

	fail_page_alloc=<probability>,<interval>,<times>,<space>

	<probability>

		specifies how often it should fail in percent.

	<interval>

		specifies the interval of failures.

	<times>

		specifies how many times failures may happen at most.

	<space>

		specifies the size of free space where memory can be allocated
		safely in pages.

Example:

	fail_page_alloc=100,10,-1,0

page allocation fails once per 10 times.

Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 include/linux/should_fail.h |    4 ++++
 lib/Kconfig.debug           |    7 +++++++
 mm/page_alloc.c             |   21 +++++++++++++++++++++
 3 files changed, 32 insertions(+)

Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -379,3 +379,10 @@ config FAILSLAB
 	help
 	  This option provides fault-injection capabilitiy for kmalloc.
 
+config FAIL_PAGE_ALLOC
+	bool "fault-injection capabilitiy for alloc_pages()"
+	depends on DEBUG_KERNEL
+	select SHOULD_FAIL
+	help
+	  This option provides fault-injection capabilitiy for alloc_pages().
+
Index: work-shouldfail/mm/page_alloc.c
===================================================================
--- work-shouldfail.orig/mm/page_alloc.c
+++ work-shouldfail/mm/page_alloc.c
@@ -37,6 +37,7 @@
 #include <linux/vmalloc.h>
 #include <linux/mempolicy.h>
 #include <linux/stop_machine.h>
+#include <linux/should_fail.h>
 
 #include <asm/tlbflush.h>
 #include <asm/div64.h>
@@ -903,6 +904,22 @@ get_page_from_freelist(gfp_t gfp_mask, u
 	return page;
 }
 
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+
+static DEFINE_SHOULD_FAIL(fail_page_alloc_data);
+
+static int __init setup_fail_page_alloc(char *str)
+{
+	should_fail_srandom(jiffies);
+	return setup_should_fail(&fail_page_alloc_data, str);
+}
+__setup("fail_page_alloc=", setup_fail_page_alloc);
+
+struct should_fail_data *fail_page_alloc = &fail_page_alloc_data;
+EXPORT_SYMBOL_GPL(fail_page_alloc);
+
+#endif
+
 /*
  * This is the 'heart' of the zoned buddy allocator.
  */
@@ -921,6 +938,10 @@ __alloc_pages(gfp_t gfp_mask, unsigned i
 
 	might_sleep_if(wait);
 
+	if (!(gfp_mask & __GFP_NOFAIL) &&
+	    should_fail(fail_page_alloc, 1 << order))
+		return NULL;
+
 restart:
 	z = zonelist->zones;  /* the list of zones suitable for gfp_mask */
 
Index: work-shouldfail/include/linux/should_fail.h
===================================================================
--- work-shouldfail.orig/include/linux/should_fail.h
+++ work-shouldfail/include/linux/should_fail.h
@@ -40,6 +40,10 @@ int should_fail(struct should_fail_data 
 extern struct should_fail_data *failslab;
 #endif
 
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+extern struct should_fail_data *fail_page_alloc;
+#endif
+
 #else
 
 #define should_fail(data, size)	(0)

--

[patch 4/6] fault-injection capability for disk IO

[Akinobu Mita]

[-- Attachment #1: fail_make_request.patch --]
[-- Type: text/plain, Size: 6647 bytes --]

This patch provides fault-injection capability for disk IO.

Boot option:

	fail_make_request=<probability>,<interval>,<times>,<space>

	<probability>

		specifies how often it should fail in percent.

	<interval>

		specifies the interval of failures.

	<times>

		specifies how many times failures may happen at most.

	<space>

		specifies the size of free space where disk IO can be issued
		safely in bytes.

Example:

	fail_make_request=100,10,-1,0
	echo 1 > /sys/blocks/hda/hda1/make-it-fail

generic_make_request() on /dev/hda1 fails once per 10 times.

Cc: Jens Axboe <axboe@suse.de>
Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 block/genhd.c               |   31 +++++++++++++++++++++++++++++++
 block/ll_rw_blk.c           |   32 ++++++++++++++++++++++++++++++++
 fs/partitions/check.c       |   27 +++++++++++++++++++++++++++
 include/linux/genhd.h       |    4 ++++
 include/linux/should_fail.h |    4 ++++
 lib/Kconfig.debug           |    7 +++++++
 6 files changed, 105 insertions(+)

Index: work-shouldfail/block/ll_rw_blk.c
===================================================================
--- work-shouldfail.orig/block/ll_rw_blk.c
+++ work-shouldfail/block/ll_rw_blk.c
@@ -28,6 +28,7 @@
 #include <linux/interrupt.h>
 #include <linux/cpu.h>
 #include <linux/blktrace_api.h>
+#include <linux/should_fail.h>
 
 /*
  * for max sense size
@@ -2993,6 +2994,34 @@ static void handle_bad_sector(struct bio
 	set_bit(BIO_EOF, &bio->bi_flags);
 }
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static DEFINE_SHOULD_FAIL(fail_make_request_data);
+
+static int __init setup_fail_make_request(char *str)
+{
+	should_fail_srandom(jiffies);
+	return setup_should_fail(&fail_make_request_data, str);
+}
+__setup("fail_make_request=", setup_fail_make_request);
+
+struct should_fail_data *fail_make_request = &fail_make_request_data;
+EXPORT_SYMBOL_GPL(fail_make_request);
+
+static int should_fail_request(struct should_fail_data *data, struct bio *bio)
+{
+	if ((bio->bi_bdev->bd_disk->flags & GENHD_FL_FAIL) ||
+	    (bio->bi_bdev->bd_part && bio->bi_bdev->bd_part->make_it_fail))
+		return should_fail(data, bio->bi_size);
+
+	return 0;
+}
+#else
+
+#define should_fail_request(data, bio)	(0)
+
+#endif
+
 /**
  * generic_make_request: hand a buffer to its device driver for I/O
  * @bio:  The bio describing the location in memory and on the device.
@@ -3077,6 +3106,9 @@ end_io:
 		if (unlikely(test_bit(QUEUE_FLAG_DEAD, &q->queue_flags)))
 			goto end_io;
 
+		if (should_fail_request(fail_make_request, bio))
+			goto end_io;
+
 		/*
 		 * If this device has partitions, remap block n
 		 * of partition p to block n+start(p) of the disk.
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -386,3 +386,10 @@ config FAIL_PAGE_ALLOC
 	help
 	  This option provides fault-injection capabilitiy for alloc_pages().
 
+config FAIL_MAKE_REQUEST
+	bool "fault-injection capabilitiy for disk IO"
+	depends on DEBUG_KERNEL
+	select SHOULD_FAIL
+	help
+	  This option provides fault-injection capabilitiy to disk IO.
+
Index: work-shouldfail/include/linux/should_fail.h
===================================================================
--- work-shouldfail.orig/include/linux/should_fail.h
+++ work-shouldfail/include/linux/should_fail.h
@@ -44,6 +44,10 @@ extern struct should_fail_data *failslab
 extern struct should_fail_data *fail_page_alloc;
 #endif
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+extern struct should_fail_data *fail_make_request;
+#endif
+
 #else
 
 #define should_fail(data, size)	(0)
Index: work-shouldfail/block/genhd.c
===================================================================
--- work-shouldfail.orig/block/genhd.c
+++ work-shouldfail/block/genhd.c
@@ -412,6 +412,34 @@ static struct disk_attribute disk_attr_s
 	.show	= disk_stats_read
 };
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static ssize_t disk_fail_store(struct gendisk * disk,
+			       const char *buf, size_t count)
+{
+	int i;
+
+	if (count > 0 && sscanf(buf, "%d", &i) > 0) {
+		if (i == 0)
+			disk->flags &= ~GENHD_FL_FAIL;
+		else
+			disk->flags |= GENHD_FL_FAIL;
+	}
+
+	return count;
+}
+static ssize_t disk_fail_read(struct gendisk * disk, char *page)
+{
+	return sprintf(page, "%d\n", disk->flags & GENHD_FL_FAIL ? 1 : 0);
+}
+static struct disk_attribute disk_attr_fail = {
+	.attr = {.name = "make-it-fail", .mode = S_IRUGO | S_IWUSR },
+	.store	= disk_fail_store,
+	.show	= disk_fail_read
+};
+
+#endif
+
 static struct attribute * default_attrs[] = {
 	&disk_attr_uevent.attr,
 	&disk_attr_dev.attr,
@@ -419,6 +447,9 @@ static struct attribute * default_attrs[
 	&disk_attr_removable.attr,
 	&disk_attr_size.attr,
 	&disk_attr_stat.attr,
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	&disk_attr_fail.attr,
+#endif
 	NULL,
 };
 
Index: work-shouldfail/include/linux/genhd.h
===================================================================
--- work-shouldfail.orig/include/linux/genhd.h
+++ work-shouldfail/include/linux/genhd.h
@@ -81,6 +81,9 @@ struct hd_struct {
 	struct kobject *holder_dir;
 	unsigned ios[2], sectors[2];	/* READs and WRITEs */
 	int policy, partno;
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	int make_it_fail;
+#endif
 };
 
 #define GENHD_FL_REMOVABLE			1
@@ -88,6 +91,7 @@ struct hd_struct {
 #define GENHD_FL_CD				8
 #define GENHD_FL_UP				16
 #define GENHD_FL_SUPPRESS_PARTITION_INFO	32
+#define GENHD_FL_FAIL				64
 
 struct disk_stats {
 	unsigned long sectors[2];	/* READs and WRITEs */
Index: work-shouldfail/fs/partitions/check.c
===================================================================
--- work-shouldfail.orig/fs/partitions/check.c
+++ work-shouldfail/fs/partitions/check.c
@@ -265,12 +265,39 @@ static struct part_attribute part_attr_s
 	.show	= part_stat_read
 };
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static ssize_t part_fail_store(struct hd_struct * p,
+			       const char *buf, size_t count)
+{
+	int i;
+
+	if (count > 0 && sscanf(buf, "%d", &i) > 0)
+		p->make_it_fail = (i == 0) ? 0 : 1;
+
+	return count;
+}
+static ssize_t part_fail_read(struct hd_struct * p, char *page)
+{
+	return sprintf(page, "%d\n", p->make_it_fail);
+}
+static struct part_attribute part_attr_fail = {
+	.attr = {.name = "make-it-fail", .mode = S_IRUGO | S_IWUSR },
+	.store	= part_fail_store,
+	.show	= part_fail_read
+};
+
+#endif
+
 static struct attribute * default_attrs[] = {
 	&part_attr_uevent.attr,
 	&part_attr_dev.attr,
 	&part_attr_start.attr,
 	&part_attr_size.attr,
 	&part_attr_stat.attr,
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	&part_attr_fail.attr,
+#endif
 	NULL,
 };
 

--

[patch 5/6] debugfs entries for configuration

[Akinobu Mita]

[-- Attachment #1: knobs.patch --]
[-- Type: text/plain, Size: 5778 bytes --]

This kernel module provides debugfs entries for fault-injection
capabilities configuation.

/debug/
|-- fail_make_request
|   |-- interval
|   |-- probability
|   |-- space
|   `-- times
|-- fail_page_alloc
|   |-- interval
|   |-- probability
|   |-- space
|   `-- times
`-- failslab
    |-- interval
    |-- probability
    |-- space
    `-- times

Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 lib/Kconfig.debug       |    8 ++
 lib/Makefile            |    1 
 lib/should_fail_knobs.c |  168 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 177 insertions(+)

Index: work-shouldfail/lib/should_fail_knobs.c
===================================================================
--- /dev/null
+++ work-shouldfail/lib/should_fail_knobs.c
@@ -0,0 +1,168 @@
+#include <linux/module.h>
+#include <linux/should_fail.h>
+#include <linux/debugfs.h>
+
+struct should_fail_knobs {
+	struct dentry *dir;
+	struct dentry *probability_file;
+	struct dentry *interval_file;
+	struct dentry *times_file;
+	struct dentry *space_file;
+};
+
+static void debugfs_ul_set(void *data, u64 val)
+{
+	*(unsigned long *)data = val;
+}
+
+static u64 debugfs_ul_get(void *data)
+{
+	return *(unsigned long *)data;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_ul, debugfs_ul_get, debugfs_ul_set, "%llu\n");
+
+static struct dentry *debugfs_create_ul(const char *name, mode_t mode,
+				struct dentry *parent, unsigned long *value)
+{
+	return debugfs_create_file(name, mode, parent, value, &fops_ul);
+}
+
+static void debugfs_atomic_t_set(void *data, u64 val)
+{
+	atomic_set((atomic_t *)data, val);
+}
+
+static u64 debugfs_atomic_t_get(void *data)
+{
+	return atomic_read((atomic_t *)data);
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_atomic_t, debugfs_atomic_t_get,
+			debugfs_atomic_t_set, "%lld\n");
+
+static struct dentry *debugfs_create_atomic_t(const char *name, mode_t mode,
+				struct dentry *parent, atomic_t *value)
+{
+	return debugfs_create_file(name, mode, parent, value, &fops_atomic_t);
+}
+
+static void cleanup_should_fail_knobs(struct should_fail_knobs *knobs)
+{
+	if (knobs->dir) {
+		if (knobs->probability_file) {
+			debugfs_remove(knobs->probability_file);
+			knobs->probability_file = NULL;
+		}
+		if (knobs->interval_file) {
+			debugfs_remove(knobs->interval_file);
+			knobs->interval_file = NULL;
+		}
+		if (knobs->times_file) {
+			debugfs_remove(knobs->times_file);
+			knobs->times_file = NULL;
+		}
+		if (knobs->space_file) {
+			debugfs_remove(knobs->space_file);
+			knobs->space_file = NULL;
+		}
+		debugfs_remove(knobs->dir);
+		knobs->dir = NULL;
+	}
+}
+
+static int init_should_fail_knobs(struct should_fail_knobs *knobs,
+			   struct should_fail_data *data, const char *name)
+{
+	mode_t mode = S_IFREG | S_IRUSR | S_IWUSR;
+	struct dentry *dir;
+	struct dentry *file;
+
+	memset(knobs, 0, sizeof(*knobs));
+
+	dir = debugfs_create_dir(name, NULL);
+	if (!dir)
+		goto fail;
+	knobs->dir = dir;
+
+	file = debugfs_create_ul("probability", mode, dir, &data->probability);
+	if (!file)
+		goto fail;
+	knobs->probability_file = file;
+
+	file = debugfs_create_ul("interval", mode, dir, &data->interval);
+	if (!file)
+		goto fail;
+	knobs->interval_file = file;
+
+	file = debugfs_create_atomic_t("times", mode, dir, &data->times);
+	if (!file)
+		goto fail;
+	knobs->times_file = file;
+
+	file = debugfs_create_atomic_t("space", mode, dir, &data->space);
+	if (!file)
+		goto fail;
+	knobs->space_file = file;
+
+	return 0;
+fail:
+	cleanup_should_fail_knobs(knobs);
+	return -ENOMEM;
+}
+
+#ifdef CONFIG_FAILSLAB
+static struct should_fail_knobs failslab_knobs;
+#endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+static struct should_fail_knobs fail_page_alloc_knobs;
+#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+static struct should_fail_knobs fail_make_request_knobs;
+#endif
+
+static void cleanup_knobs(void)
+{
+#ifdef CONFIG_FAILSLAB
+	cleanup_should_fail_knobs(&fail_make_request_knobs);
+#endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+	cleanup_should_fail_knobs(&fail_page_alloc_knobs);
+#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	cleanup_should_fail_knobs(&failslab_knobs);
+#endif
+}
+
+static int init_knobs(void)
+{
+	int err;
+
+#ifdef CONFIG_FAILSLAB
+	err = init_should_fail_knobs(&failslab_knobs, failslab, "failslab");
+	if (err)
+		goto fail;
+#endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+	err = init_should_fail_knobs(&fail_page_alloc_knobs, fail_page_alloc,
+				     "fail_page_alloc");
+	if (err)
+		goto fail;
+#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	err = init_should_fail_knobs(&fail_make_request_knobs,
+				     fail_make_request, "fail_make_request");
+	if (err)
+		goto fail;
+#endif
+
+	return 0;
+fail:
+	cleanup_knobs();
+
+	return err;
+}
+
+module_init(init_knobs);
+module_exit(cleanup_knobs);
+MODULE_LICENSE("GPL");
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
--- work-shouldfail.orig/lib/Kconfig.debug
+++ work-shouldfail/lib/Kconfig.debug
@@ -393,3 +393,11 @@ config FAIL_MAKE_REQUEST
 	help
 	  This option provides fault-injection capabilitiy to disk IO.
 
+config SHOULD_FAIL_KNOBS
+	tristate "runtime configuration for fault-injection capabilities"
+	depends on DEBUG_KERNEL && SYSFS && SHOULD_FAIL
+	select DEBUG_FS
+	help
+	  This option provides kernel module that provides runtime
+	  configuration interface by debugfs.
+
Index: work-shouldfail/lib/Makefile
===================================================================
--- work-shouldfail.orig/lib/Makefile
+++ work-shouldfail/lib/Makefile
@@ -52,6 +52,7 @@ obj-$(CONFIG_SMP) += percpu_counter.o
 
 obj-$(CONFIG_SWIOTLB) += swiotlb.o
 obj-$(CONFIG_SHOULD_FAIL) += should_fail.o
+obj-$(CONFIG_SHOULD_FAIL_KNOBS) += should_fail_knobs.o
 
 hostprogs-y	:= gen_crc32table
 clean-files	:= crc32table.h

--

[patch 6/6] process filtering for fault-injection capabilities

[Akinobu Mita]

[-- Attachment #1: process-filter.patch --]
[-- Type: text/plain, Size: 7012 bytes --]

This patch provides process filtering feature.

Example:

# modprobe should_fail_knobs
# echo 1 > /debug/failslab/process-filter
# echo 30 > /debug/failslab/probability
# find /lib/modules/`uname -r` -name '*.ko' -exec basename {} .ko \; > list
# for i in `cat list`;do failmodprobe $i;done

failmodprobe:
-------------
#!/bin/bash
echo 1 > /proc/self/make-it-fail
exec modprobe $*

Signed-off-by: Akinobu Mita <mita@miraclelinux.com>

 fs/proc/base.c              |   77 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/sched.h       |    3 +
 include/linux/should_fail.h |    3 +
 lib/should_fail.c           |   11 ++++--
 lib/should_fail_knobs.c     |   10 +++++
 5 files changed, 101 insertions(+), 3 deletions(-)

Index: work-shouldfail/fs/proc/base.c
===================================================================
--- work-shouldfail.orig/fs/proc/base.c
+++ work-shouldfail/fs/proc/base.c
@@ -138,6 +138,9 @@ enum pid_directory_inos {
 #endif
 	PROC_TGID_OOM_SCORE,
 	PROC_TGID_OOM_ADJUST,
+#ifdef CONFIG_SHOULD_FAIL
+	PROC_TGID_SHOULD_FAIL,
+#endif
 	PROC_TID_INO,
 	PROC_TID_STATUS,
 	PROC_TID_MEM,
@@ -181,6 +184,9 @@ enum pid_directory_inos {
 #endif
 	PROC_TID_OOM_SCORE,
 	PROC_TID_OOM_ADJUST,
+#ifdef CONFIG_SHOULD_FAIL
+	PROC_TID_SHOULD_FAIL,
+#endif
 
 	/* Add new entries before this */
 	PROC_TID_FD_DIR = 0x8000,	/* 0x8000-0xffff */
@@ -240,6 +246,9 @@ static struct pid_entry tgid_base_stuff[
 #ifdef CONFIG_AUDITSYSCALL
 	E(PROC_TGID_LOGINUID, "loginuid", S_IFREG|S_IWUSR|S_IRUGO),
 #endif
+#ifdef CONFIG_SHOULD_FAIL
+	E(PROC_TGID_SHOULD_FAIL, "make-it-fail", S_IFREG|S_IWUSR|S_IRUGO),
+#endif
 	{0,0,NULL,0}
 };
 static struct pid_entry tid_base_stuff[] = {
@@ -282,6 +291,9 @@ static struct pid_entry tid_base_stuff[]
 #ifdef CONFIG_AUDITSYSCALL
 	E(PROC_TID_LOGINUID, "loginuid", S_IFREG|S_IWUSR|S_IRUGO),
 #endif
+#ifdef CONFIG_SHOULD_FAIL
+	E(PROC_TID_SHOULD_FAIL, "make-it-fail", S_IFREG|S_IWUSR|S_IRUGO),
+#endif
 	{0,0,NULL,0}
 };
 
@@ -992,6 +1004,65 @@ static struct file_operations proc_login
 };
 #endif
 
+#ifdef CONFIG_SHOULD_FAIL
+static ssize_t proc_should_fail_read(struct file * file, char __user * buf,
+				     size_t count, loff_t *ppos)
+{
+	struct task_struct *task = get_proc_task(file->f_dentry->d_inode);
+	char buffer[PROC_NUMBUF];
+	size_t len;
+	int make_it_fail;
+	loff_t __ppos = *ppos;
+
+	if (!task)
+		return -ESRCH;
+	make_it_fail = task->make_it_fail;
+	put_task_struct(task);
+
+	len = snprintf(buffer, sizeof(buffer), "%i\n", make_it_fail);
+	if (__ppos >= len)
+		return 0;
+	if (count > len-__ppos)
+		count = len-__ppos;
+	if (copy_to_user(buf, buffer + __ppos, count))
+		return -EFAULT;
+	*ppos = __ppos + count;
+	return count;
+}
+
+static ssize_t proc_should_fail_write(struct file * file,
+			const char __user * buf, size_t count, loff_t *ppos)
+{
+	struct task_struct *task;
+	char buffer[PROC_NUMBUF], *end;
+	int make_it_fail;
+
+	if (!capable(CAP_SYS_RESOURCE))
+		return -EPERM;
+	memset(buffer, 0, sizeof(buffer));
+	if (count > sizeof(buffer) - 1)
+		count = sizeof(buffer) - 1;
+	if (copy_from_user(buffer, buf, count))
+		return -EFAULT;
+	make_it_fail = simple_strtol(buffer, &end, 0);
+	if (*end == '\n')
+		end++;
+	task = get_proc_task(file->f_dentry->d_inode);
+	if (!task)
+		return -ESRCH;
+	task->make_it_fail = make_it_fail;
+	put_task_struct(task);
+	if (end - buffer == 0)
+		return -EIO;
+	return end - buffer;
+}
+
+static struct file_operations proc_should_fail_operations = {
+	.read		= proc_should_fail_read,
+	.write		= proc_should_fail_write,
+};
+#endif
+
 #ifdef CONFIG_SECCOMP
 static ssize_t seccomp_read(struct file *file, char __user *buf,
 			    size_t count, loff_t *ppos)
@@ -1834,6 +1905,12 @@ static struct dentry *proc_pident_lookup
 			inode->i_fop = &proc_loginuid_operations;
 			break;
 #endif
+#ifdef CONFIG_SHOULD_FAIL
+		case PROC_TID_SHOULD_FAIL:
+		case PROC_TGID_SHOULD_FAIL:
+			inode->i_fop = &proc_should_fail_operations;
+			break;
+#endif
 		default:
 			printk("procfs: impossible type (%d)",p->type);
 			iput(inode);
Index: work-shouldfail/include/linux/sched.h
===================================================================
--- work-shouldfail.orig/include/linux/sched.h
+++ work-shouldfail/include/linux/sched.h
@@ -997,6 +997,9 @@ struct task_struct {
 	spinlock_t delays_lock;
 	struct task_delay_info *delays;
 #endif
+#ifdef CONFIG_SHOULD_FAIL
+	int make_it_fail;
+#endif
 };
 
 static inline pid_t process_group(struct task_struct *tsk)
Index: work-shouldfail/include/linux/should_fail.h
===================================================================
--- work-shouldfail.orig/include/linux/should_fail.h
+++ work-shouldfail/include/linux/should_fail.h
@@ -27,6 +27,9 @@ struct should_fail_data {
 	atomic_t space;
 
 	unsigned long count;
+
+	/* process filter */
+	unsigned long process_filter;
 };
 
 #define DEFINE_SHOULD_FAIL(name) \
Index: work-shouldfail/lib/should_fail.c
===================================================================
--- work-shouldfail.orig/lib/should_fail.c
+++ work-shouldfail/lib/should_fail.c
@@ -13,16 +13,18 @@ int setup_should_fail(struct should_fail
 	unsigned long interval;
 	int times;
 	int space;
+	unsigned long filter;
 
-	/* "<probability>,<interval>,<times>,<space>" */
-	if (sscanf(str, "%lu,%lu,%d,%d", &probability, &interval, &times,
-		   &space) < 4)
+	/* "<probability>,<interval>,<times>,<space>,<process-filter>" */
+	if (sscanf(str, "%lu,%lu,%d,%d,%d", &probability, &interval, &times,
+		   &space, &filter) < 5)
 		return 0;
 
 	data->probability = probability;
 	data->interval = interval;
 	atomic_set(&data->times, times);
 	atomic_set(&data->space, space);
+	data->process_filter = filter;
 
 	return 1;
 }
@@ -54,6 +56,9 @@ void should_fail_srandom(unsigned long e
 
 int should_fail(struct should_fail_data *data, ssize_t size)
 {
+	if (data->process_filter && !current->make_it_fail)
+		return 0;
+
 	if (atomic_read(&max_failures(data)) == 0)
 		return 0;
 
Index: work-shouldfail/lib/should_fail_knobs.c
===================================================================
--- work-shouldfail.orig/lib/should_fail_knobs.c
+++ work-shouldfail/lib/should_fail_knobs.c
@@ -8,6 +8,7 @@ struct should_fail_knobs {
 	struct dentry *interval_file;
 	struct dentry *times_file;
 	struct dentry *space_file;
+	struct dentry *filter_file;
 };
 
 static void debugfs_ul_set(void *data, u64 val)
@@ -66,6 +67,10 @@ static void cleanup_should_fail_knobs(st
 			debugfs_remove(knobs->space_file);
 			knobs->space_file = NULL;
 		}
+		if (knobs->filter_file) {
+			debugfs_remove(knobs->filter_file);
+			knobs->filter_file = NULL;
+		}
 		debugfs_remove(knobs->dir);
 		knobs->dir = NULL;
 	}
@@ -105,6 +110,11 @@ static int init_should_fail_knobs(struct
 		goto fail;
 	knobs->space_file = file;
 
+	file = debugfs_create_ul("process-filter", mode, dir, &data->process_filter);
+	if (!file)
+		goto fail;
+	knobs->filter_file = file;
+
 	return 0;
 fail:
 	cleanup_should_fail_knobs(knobs);

--

Re: [patch 1/6] fault-injection capabilities infrastructure

[Andi Kleen]

On Thursday 31 August 2006 12:07, Akinobu Mita wrote:
>
> Index: work-shouldfail/lib/should_fail.c
> ===================================================================
> --- /dev/null
> +++ work-shouldfail/lib/should_fail.c
> @@ -0,0 +1,82 @@

You really need some more comments in this file, describing
what it does in a high level way. Or ideally something
in Documentation/* describing what the various modules
do and how to use them? 


-Andi

Re: [patch 3/6] fault-injection capability for alloc_pages()

[Andi Kleen]

On Thursday 31 August 2006 12:07, Akinobu Mita wrote:
> This patch provides fault-injection capability for alloc_pages()
> 
> boot option:
> 
> 	fail_page_alloc=<probability>,<interval>,<times>,<space>
> 
> 	<probability>
> 
> 		specifies how often it should fail in percent.
> 
> 	<interval>
> 
> 		specifies the interval of failures.
> 
> 	<times>
> 
> 		specifies how many times failures may happen at most.
> 
> 	<space>
> 
> 		specifies the size of free space where memory can be allocated
> 		safely in pages.
> 
> Example:
> 
> 	fail_page_alloc=100,10,-1,0
> 
> page allocation fails once per 10 times.

I still think this will need some better filters to be useful. At least
a optional uid filter perhaps (make sure to handle the interrupt case
correctly, interrupts don't belong to the uid) , and perhaps an option to only 
fail GFP_ATOMIC.

With arbitary failing the system will just be unusable, right? Or would
you run some system you use this way? @)

Another possibility would be to look up __builtin_return_address(0) in 
the module table and allow failing only for a specific module.


-andi

Re: [patch 3/6] fault-injection capability for alloc_pages()

[Akinobu Mita]

On Thu, Aug 31, 2006 at 12:25:02PM +0200, Andi Kleen wrote:

> I still think this will need some better filters to be useful. At least
> a optional uid filter perhaps (make sure to handle the interrupt case
> correctly, interrupts don't belong to the uid) , and perhaps an option to only 
> fail GFP_ATOMIC.

I wrote process filter. Please patch 6/6. But I forgot to ignore
in_interrupt() case.

> With arbitary failing the system will just be unusable, right? Or would
> you run some system you use this way? @)
> 
> Another possibility would be to look up __builtin_return_address(0) in 
> the module table and allow failing only for a specific module.

That will be useful. Thanks.

Re: [patch 3/6] fault-injection capability for alloc_pages()

[Andi Kleen]

On Thursday 31 August 2006 12:35, Akinobu Mita wrote:
> On Thu, Aug 31, 2006 at 12:25:02PM +0200, Andi Kleen wrote:
> 
> > I still think this will need some better filters to be useful. At least
> > a optional uid filter perhaps (make sure to handle the interrupt case
> > correctly, interrupts don't belong to the uid) , and perhaps an option to only 
> > fail GFP_ATOMIC.
> 
> I wrote process filter.

Oops sorry. I overlooked that.

> Please patch 6/6. But I forgot to ignore 
> in_interrupt() case.

Ok fine then.

> 
> > With arbitary failing the system will just be unusable, right? Or would
> > you run some system you use this way? @)
> > 
> > Another possibility would be to look up __builtin_return_address(0) in 
> > the module table and allow failing only for a specific module.
> 
> That will be useful. Thanks.

It might unfortunately need architecture specific code. But I guess a i386
only implementation as start would be useful enough.

-Andi

Re: [patch 0/6] RFC: fault-injection capabilities (v2)

[Josh Triplett]

On Thu, 2006-08-31 at 19:07 +0900, Akinobu Mita wrote:
> This patch set provides some fault-injection capabilities.
> 
> - kmalloc failures
> 
> - alloc_pages() failures
> 
> - disk IO errors
> 
> We can see what really happens if those failures happen.

Looks very useful for testing error paths; nice work.

Should this perhaps taint the kernel when used?

- Josh Triplett

Re: [patch 0/6] RFC: fault-injection capabilities (v2)

[Kyle Moffett]

On Aug 31, 2006, at 13:29:20, Josh Triplett wrote:
> On Thu, 2006-08-31 at 19:07 +0900, Akinobu Mita wrote:
>> This patch set provides some fault-injection capabilities.
>>
>> - kmalloc failures
>>
>> - alloc_pages() failures
>>
>> - disk IO errors
>>
>> We can see what really happens if those failures happen.
>
> Looks very useful for testing error paths; nice work.
>
> Should this perhaps taint the kernel when used?

It shouldn't; these are all failures that could quite possibly happen  
during normal operation even without this enabled, they're just a few  
orders of magnitude less likely (in most situations).

Cheers,
Kyle moffett