* [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices
@ 2019-08-01 20:00 Branden Bonaby
2019-08-01 20:00 ` [PATCH 1/3] drivers: hv: vmbus: Introduce latency testing Branden Bonaby
` (3 more replies)
0 siblings, 4 replies; 10+ messages in thread
From: Branden Bonaby @ 2019-08-01 20:00 UTC (permalink / raw)
To: kys, haiyangz, sthemmin, sashal
Cc: Branden Bonaby, linux-hyperv, linux-kernel
This patchset introduces a testing framework for Hyper-V drivers.
This framework allows us to introduce delays in the packet receive
path on a per-device basis. While the current code only supports
introducing arbitrary delays in the host/guest communication path,
we intend to expand this to support error injection in the future.
Branden Bonaby (3):
drivers: hv: vmbus: Introduce latency testing
drivers: hv: vmbus: add fuzz test attributes to sysfs
tools: hv: add vmbus testing tool
Documentation/ABI/stable/sysfs-bus-vmbus | 22 ++
drivers/hv/connection.c | 5 +
drivers/hv/ring_buffer.c | 10 +
drivers/hv/vmbus_drv.c | 97 ++++++-
include/linux/hyperv.h | 14 +
tools/hv/vmbus_testing | 326 +++++++++++++++++++++++
6 files changed, 473 insertions(+), 1 deletion(-)
create mode 100644 tools/hv/vmbus_testing
--
2.17.1
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 1/3] drivers: hv: vmbus: Introduce latency testing 2019-08-01 20:00 [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices Branden Bonaby @ 2019-08-01 20:00 ` Branden Bonaby 2019-08-02 7:32 ` Vitaly Kuznetsov 2019-08-01 20:00 ` [PATCH 2/3] drivers: hv: vmbus: add fuzz test attributes to sysfs Branden Bonaby ` (2 subsequent siblings) 3 siblings, 1 reply; 10+ messages in thread From: Branden Bonaby @ 2019-08-01 20:00 UTC (permalink / raw) To: kys, haiyangz, sthemmin, sashal Cc: Branden Bonaby, linux-hyperv, linux-kernel Introduce user specified latency in the packet reception path. Signed-off-by: Branden Bonaby <brandonbonaby94@gmail.com> --- drivers/hv/connection.c | 5 +++++ drivers/hv/ring_buffer.c | 10 ++++++++++ include/linux/hyperv.h | 14 ++++++++++++++ 3 files changed, 29 insertions(+) diff --git a/drivers/hv/connection.c b/drivers/hv/connection.c index 09829e15d4a0..2a2c22f5570e 100644 --- a/drivers/hv/connection.c +++ b/drivers/hv/connection.c @@ -354,9 +354,14 @@ void vmbus_on_event(unsigned long data) { struct vmbus_channel *channel = (void *) data; unsigned long time_limit = jiffies + 2; + struct vmbus_channel *test_channel = !channel->primary_channel ? + channel : + channel->primary_channel; trace_vmbus_on_event(channel); + if (unlikely(test_channel->fuzz_testing_buffer_delay > 0)) + udelay(test_channel->fuzz_testing_buffer_delay); do { void (*callback_fn)(void *); diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c index 9a03b163cbbd..d7627c9023d6 100644 --- a/drivers/hv/ring_buffer.c +++ b/drivers/hv/ring_buffer.c @@ -395,7 +395,12 @@ struct vmpacket_descriptor *hv_pkt_iter_first(struct vmbus_channel *channel) { struct hv_ring_buffer_info *rbi = &channel->inbound; struct vmpacket_descriptor *desc; + struct vmbus_channel *test_channel = !channel->primary_channel ? + channel : + channel->primary_channel; + if (unlikely(test_channel->fuzz_testing_message_delay > 0)) + udelay(test_channel->fuzz_testing_message_delay); if (hv_pkt_iter_avail(rbi) < sizeof(struct vmpacket_descriptor)) return NULL; @@ -420,7 +425,12 @@ __hv_pkt_iter_next(struct vmbus_channel *channel, struct hv_ring_buffer_info *rbi = &channel->inbound; u32 packetlen = desc->len8 << 3; u32 dsize = rbi->ring_datasize; + struct vmbus_channel *test_channel = !channel->primary_channel ? + channel : + channel->primary_channel; + if (unlikely(test_channel->fuzz_testing_message_delay > 0)) + udelay(test_channel->fuzz_testing_message_delay); /* bump offset to next potential packet */ rbi->priv_read_index += packetlen + VMBUS_PKT_TRAILER; if (rbi->priv_read_index >= dsize) diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h index 6256cc34c4a6..8d068956dd67 100644 --- a/include/linux/hyperv.h +++ b/include/linux/hyperv.h @@ -23,6 +23,7 @@ #include <linux/mod_devicetable.h> #include <linux/interrupt.h> #include <linux/reciprocal_div.h> +#include <linux/delay.h> #define MAX_PAGE_BUFFER_COUNT 32 #define MAX_MULTIPAGE_BUFFER_COUNT 32 /* 128K */ @@ -926,6 +927,19 @@ struct vmbus_channel { * full outbound ring buffer. */ u64 out_full_first; + + /* enabling/disabling fuzz testing on the channel (default is false)*/ + bool fuzz_testing_state; + + /* Buffer delay will delay the guest from emptying the ring buffer + * for a specific amount of time. The delay is in microseconds and will + * be between 1 to a maximum of 1000, its default is 0 (no delay). + * The Message delay will delay guest reading on a per message basis + * in microseconds between 1 to 1000 with the default being 0 + * (no delay). + */ + u32 fuzz_testing_buffer_delay; + u32 fuzz_testing_message_delay; }; static inline bool is_hvsock_channel(const struct vmbus_channel *c) -- 2.17.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 1/3] drivers: hv: vmbus: Introduce latency testing 2019-08-01 20:00 ` [PATCH 1/3] drivers: hv: vmbus: Introduce latency testing Branden Bonaby @ 2019-08-02 7:32 ` Vitaly Kuznetsov 2019-08-02 19:02 ` Branden Bonaby 0 siblings, 1 reply; 10+ messages in thread From: Vitaly Kuznetsov @ 2019-08-02 7:32 UTC (permalink / raw) To: Branden Bonaby, kys, haiyangz, sthemmin, sashal Cc: Branden Bonaby, linux-hyperv, linux-kernel Branden Bonaby <brandonbonaby94@gmail.com> writes: > Introduce user specified latency in the packet reception path. > > Signed-off-by: Branden Bonaby <brandonbonaby94@gmail.com> > --- > drivers/hv/connection.c | 5 +++++ > drivers/hv/ring_buffer.c | 10 ++++++++++ > include/linux/hyperv.h | 14 ++++++++++++++ > 3 files changed, 29 insertions(+) > > diff --git a/drivers/hv/connection.c b/drivers/hv/connection.c > index 09829e15d4a0..2a2c22f5570e 100644 > --- a/drivers/hv/connection.c > +++ b/drivers/hv/connection.c > @@ -354,9 +354,14 @@ void vmbus_on_event(unsigned long data) > { > struct vmbus_channel *channel = (void *) data; > unsigned long time_limit = jiffies + 2; > + struct vmbus_channel *test_channel = !channel->primary_channel ? > + channel : > + channel->primary_channel; > > trace_vmbus_on_event(channel); > > + if (unlikely(test_channel->fuzz_testing_buffer_delay > 0)) > + udelay(test_channel->fuzz_testing_buffer_delay); > do { > void (*callback_fn)(void *); > > diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c > index 9a03b163cbbd..d7627c9023d6 100644 > --- a/drivers/hv/ring_buffer.c > +++ b/drivers/hv/ring_buffer.c > @@ -395,7 +395,12 @@ struct vmpacket_descriptor *hv_pkt_iter_first(struct vmbus_channel *channel) > { > struct hv_ring_buffer_info *rbi = &channel->inbound; > struct vmpacket_descriptor *desc; > + struct vmbus_channel *test_channel = !channel->primary_channel ? > + channel : > + channel->primary_channel; > > + if (unlikely(test_channel->fuzz_testing_message_delay > 0)) > + udelay(test_channel->fuzz_testing_message_delay); > if (hv_pkt_iter_avail(rbi) < sizeof(struct vmpacket_descriptor)) > return NULL; > > @@ -420,7 +425,12 @@ __hv_pkt_iter_next(struct vmbus_channel *channel, > struct hv_ring_buffer_info *rbi = &channel->inbound; > u32 packetlen = desc->len8 << 3; > u32 dsize = rbi->ring_datasize; > + struct vmbus_channel *test_channel = !channel->primary_channel ? > + channel : > + channel->primary_channel; This pattern is repeated 3 times so a define is justified. I would also reversed the logic: test_channel = channel->primary_channel ? channel->primary_channel : channel; > > + if (unlikely(test_channel->fuzz_testing_message_delay > 0)) > + udelay(test_channel->fuzz_testing_message_delay); unlikely() is good but if it was under #ifdef it would've been even better. > /* bump offset to next potential packet */ > rbi->priv_read_index += packetlen + VMBUS_PKT_TRAILER; > if (rbi->priv_read_index >= dsize) > diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h > index 6256cc34c4a6..8d068956dd67 100644 > --- a/include/linux/hyperv.h > +++ b/include/linux/hyperv.h > @@ -23,6 +23,7 @@ > #include <linux/mod_devicetable.h> > #include <linux/interrupt.h> > #include <linux/reciprocal_div.h> > +#include <linux/delay.h> > > #define MAX_PAGE_BUFFER_COUNT 32 > #define MAX_MULTIPAGE_BUFFER_COUNT 32 /* 128K */ > @@ -926,6 +927,19 @@ struct vmbus_channel { > * full outbound ring buffer. > */ > u64 out_full_first; > + > + /* enabling/disabling fuzz testing on the channel (default is false)*/ > + bool fuzz_testing_state; > + > + /* Buffer delay will delay the guest from emptying the ring buffer > + * for a specific amount of time. The delay is in microseconds and will > + * be between 1 to a maximum of 1000, its default is 0 (no delay). > + * The Message delay will delay guest reading on a per message basis > + * in microseconds between 1 to 1000 with the default being 0 > + * (no delay). > + */ > + u32 fuzz_testing_buffer_delay; > + u32 fuzz_testing_message_delay; > }; > > static inline bool is_hvsock_channel(const struct vmbus_channel *c) -- Vitaly ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 1/3] drivers: hv: vmbus: Introduce latency testing 2019-08-02 7:32 ` Vitaly Kuznetsov @ 2019-08-02 19:02 ` Branden Bonaby 0 siblings, 0 replies; 10+ messages in thread From: Branden Bonaby @ 2019-08-02 19:02 UTC (permalink / raw) To: Vitaly Kuznetsov Cc: kys, haiyangz, sthemmin, sashal, linux-hyperv, linux-kernel On Fri, Aug 02, 2019 at 09:32:59AM +0200, Vitaly Kuznetsov wrote: > Branden Bonaby <brandonbonaby94@gmail.com> writes: > > > Introduce user specified latency in the packet reception path. > > > > Signed-off-by: Branden Bonaby <brandonbonaby94@gmail.com> > > --- > > drivers/hv/connection.c | 5 +++++ > > drivers/hv/ring_buffer.c | 10 ++++++++++ > > include/linux/hyperv.h | 14 ++++++++++++++ > > 3 files changed, 29 insertions(+) > > > > diff --git a/drivers/hv/connection.c b/drivers/hv/connection.c > > index 09829e15d4a0..2a2c22f5570e 100644 > > --- a/drivers/hv/connection.c > > +++ b/drivers/hv/connection.c > > @@ -354,9 +354,14 @@ void vmbus_on_event(unsigned long data) > > { > > struct vmbus_channel *channel = (void *) data; > > unsigned long time_limit = jiffies + 2; > > + struct vmbus_channel *test_channel = !channel->primary_channel ? > > + channel : > > + channel->primary_channel; > > > > trace_vmbus_on_event(channel); > > > > + if (unlikely(test_channel->fuzz_testing_buffer_delay > 0)) > > + udelay(test_channel->fuzz_testing_buffer_delay); > > do { > > void (*callback_fn)(void *); > > > > diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c > > index 9a03b163cbbd..d7627c9023d6 100644 > > --- a/drivers/hv/ring_buffer.c > > +++ b/drivers/hv/ring_buffer.c > > @@ -395,7 +395,12 @@ struct vmpacket_descriptor *hv_pkt_iter_first(struct vmbus_channel *channel) > > { > > struct hv_ring_buffer_info *rbi = &channel->inbound; > > struct vmpacket_descriptor *desc; > > + struct vmbus_channel *test_channel = !channel->primary_channel ? > > + channel : > > + channel->primary_channel; > > > > + if (unlikely(test_channel->fuzz_testing_message_delay > 0)) > > + udelay(test_channel->fuzz_testing_message_delay); > > if (hv_pkt_iter_avail(rbi) < sizeof(struct vmpacket_descriptor)) > > return NULL; > > > > @@ -420,7 +425,12 @@ __hv_pkt_iter_next(struct vmbus_channel *channel, > > struct hv_ring_buffer_info *rbi = &channel->inbound; > > u32 packetlen = desc->len8 << 3; > > u32 dsize = rbi->ring_datasize; > > + struct vmbus_channel *test_channel = !channel->primary_channel ? > > + channel : > > + channel->primary_channel; > > This pattern is repeated 3 times so a define is justified. I would also > reversed the logic: > > test_channel = channel->primary_channel ? channel->primary_channel : channel; > > > > > + if (unlikely(test_channel->fuzz_testing_message_delay > 0)) > > + udelay(test_channel->fuzz_testing_message_delay); > > unlikely() is good but if it was under #ifdef it would've been even better. > > > /* bump offset to next potential packet */ > -- > Vitaly Makes sense, I'll address the repeated code and will change the way I handled that if statement. Using an ifdef CONFIG_HYPERV_TESTING seems like a good thing to add in here like you suggested. ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 2/3] drivers: hv: vmbus: add fuzz test attributes to sysfs 2019-08-01 20:00 [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices Branden Bonaby 2019-08-01 20:00 ` [PATCH 1/3] drivers: hv: vmbus: Introduce latency testing Branden Bonaby @ 2019-08-01 20:00 ` Branden Bonaby 2019-08-02 7:34 ` Vitaly Kuznetsov 2019-08-01 20:00 ` [PATCH 3/3] tools: hv: add vmbus testing tool Branden Bonaby 2019-08-02 7:30 ` [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices Vitaly Kuznetsov 3 siblings, 1 reply; 10+ messages in thread From: Branden Bonaby @ 2019-08-01 20:00 UTC (permalink / raw) To: kys, haiyangz, sthemmin, sashal Cc: Branden Bonaby, linux-hyperv, linux-kernel Expose the test parameters as part of the sysfs channel attributes. We will control the testing state via these attributes. Signed-off-by: Branden Bonaby <brandonbonaby94@gmail.com> --- Documentation/ABI/stable/sysfs-bus-vmbus | 22 ++++++ drivers/hv/vmbus_drv.c | 97 +++++++++++++++++++++++- 2 files changed, 118 insertions(+), 1 deletion(-) diff --git a/Documentation/ABI/stable/sysfs-bus-vmbus b/Documentation/ABI/stable/sysfs-bus-vmbus index 8e8d167eca31..239fcb6fdc75 100644 --- a/Documentation/ABI/stable/sysfs-bus-vmbus +++ b/Documentation/ABI/stable/sysfs-bus-vmbus @@ -185,3 +185,25 @@ Contact: Michael Kelley <mikelley@microsoft.com> Description: Total number of write operations that encountered an outbound ring buffer full condition Users: Debugging tools + +What: /sys/bus/vmbus/devices/<UUID>/fuzz_test_state +Date: July 2019 +KernelVersion: 5.2 +Contact: Branden Bonaby <brandonbonaby94@gmail.com> +Description: Fuzz testing status of a vmbus device, whether its in an ON + state or a OFF state +Users: Debugging tools + +What: /sys/bus/vmbus/devices/<UUID>/fuzz_test_buffer_delay +Date: July 2019 +KernelVersion: 5.2 +Contact: Branden Bonaby <brandonbonaby94@gmail.com> +Description: Fuzz testing buffer delay value between 0 - 1000 +Users: Debugging tools + +What: /sys/bus/vmbus/devices/<UUID>/fuzz_test_message_delay +Date: July 2019 +KernelVersion: 5.2 +Contact: Branden Bonaby <brandonbonaby94@gmail.com> +Description: Fuzz testing message delay value between 0 - 1000 +Users: Debugging tools diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c index 92b1874b3eb3..0c71fd66ef81 100644 --- a/drivers/hv/vmbus_drv.c +++ b/drivers/hv/vmbus_drv.c @@ -22,7 +22,7 @@ #include <linux/clockchips.h> #include <linux/cpu.h> #include <linux/sched/task_stack.h> - +#include <linux/kernel.h> #include <asm/mshyperv.h> #include <linux/notifier.h> #include <linux/ptrace.h> @@ -584,6 +584,98 @@ static ssize_t driver_override_show(struct device *dev, } static DEVICE_ATTR_RW(driver_override); +static ssize_t fuzz_test_state_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct hv_device *hv_dev = device_to_hv_device(dev); + struct vmbus_channel *channel = hv_dev->channel; + int state; + int delay = kstrtoint(buf, 0, &state); + + if (delay) + return count; + if (state) + channel->fuzz_testing_state = 1; + else + channel->fuzz_testing_state = 0; + return count; +} + +static ssize_t fuzz_test_state_show(struct device *dev, + struct device_attribute *dev_attr, + char *buf) +{ + struct hv_device *hv_dev = device_to_hv_device(dev); + struct vmbus_channel *channel = hv_dev->channel; + + return sprintf(buf, "%u\n", channel->fuzz_testing_state); +} +static DEVICE_ATTR_RW(fuzz_test_state); + +static ssize_t fuzz_test_buffer_delay_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct hv_device *hv_dev = device_to_hv_device(dev); + struct vmbus_channel *channel = hv_dev->channel; + int val; + int delay = kstrtoint(buf, 0, &val); + + if (delay) + return count; + if (val >= 1 && val <= 1000) + channel->fuzz_testing_buffer_delay = val; + /*Best to not use else statement here since we want + *the buffer delay to remain the same if val > 1000 + */ + else if (val <= 0) + channel->fuzz_testing_buffer_delay = 0; + return count; +} + +static ssize_t fuzz_test_buffer_delay_show(struct device *dev, + struct device_attribute *dev_attr, + char *buf) +{ + struct hv_device *hv_dev = device_to_hv_device(dev); + struct vmbus_channel *channel = hv_dev->channel; + + return sprintf(buf, "%u\n", channel->fuzz_testing_buffer_delay); +} +static DEVICE_ATTR_RW(fuzz_test_buffer_delay); + +static ssize_t fuzz_test_message_delay_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct hv_device *hv_dev = device_to_hv_device(dev); + struct vmbus_channel *channel = hv_dev->channel; + int val; + int delay = kstrtoint(buf, 0, &val); + + if (delay) + return count; + if (val >= 1 && val <= 1000) + channel->fuzz_testing_message_delay = val; + /*Best to not use else statement here since we want + *the message delay to remain the same if val > 1000 + */ + else if (val <= 0) + channel->fuzz_testing_message_delay = 0; + return count; +} + +static ssize_t fuzz_test_message_delay_show(struct device *dev, + struct device_attribute *dev_attr, + char *buf) +{ + struct hv_device *hv_dev = device_to_hv_device(dev); + struct vmbus_channel *channel = hv_dev->channel; + + return sprintf(buf, "%u\n", channel->fuzz_testing_message_delay); +} +static DEVICE_ATTR_RW(fuzz_test_message_delay); /* Set up per device attributes in /sys/bus/vmbus/devices/<bus device> */ static struct attribute *vmbus_dev_attrs[] = { &dev_attr_id.attr, @@ -615,6 +707,9 @@ static struct attribute *vmbus_dev_attrs[] = { &dev_attr_vendor.attr, &dev_attr_device.attr, &dev_attr_driver_override.attr, + &dev_attr_fuzz_test_state.attr, + &dev_attr_fuzz_test_buffer_delay.attr, + &dev_attr_fuzz_test_message_delay.attr, NULL, }; -- 2.17.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drivers: hv: vmbus: add fuzz test attributes to sysfs 2019-08-01 20:00 ` [PATCH 2/3] drivers: hv: vmbus: add fuzz test attributes to sysfs Branden Bonaby @ 2019-08-02 7:34 ` Vitaly Kuznetsov 2019-08-02 19:02 ` Branden Bonaby 0 siblings, 1 reply; 10+ messages in thread From: Vitaly Kuznetsov @ 2019-08-02 7:34 UTC (permalink / raw) To: Branden Bonaby Cc: linux-hyperv, linux-kernel, kys, haiyangz, sthemmin, sashal Branden Bonaby <brandonbonaby94@gmail.com> writes: > Expose the test parameters as part of the sysfs channel attributes. > We will control the testing state via these attributes. > > Signed-off-by: Branden Bonaby <brandonbonaby94@gmail.com> > --- > Documentation/ABI/stable/sysfs-bus-vmbus | 22 ++++++ > drivers/hv/vmbus_drv.c | 97 +++++++++++++++++++++++- > 2 files changed, 118 insertions(+), 1 deletion(-) > > diff --git a/Documentation/ABI/stable/sysfs-bus-vmbus b/Documentation/ABI/stable/sysfs-bus-vmbus > index 8e8d167eca31..239fcb6fdc75 100644 > --- a/Documentation/ABI/stable/sysfs-bus-vmbus > +++ b/Documentation/ABI/stable/sysfs-bus-vmbus > @@ -185,3 +185,25 @@ Contact: Michael Kelley <mikelley@microsoft.com> > Description: Total number of write operations that encountered an outbound > ring buffer full condition > Users: Debugging tools > + > +What: /sys/bus/vmbus/devices/<UUID>/fuzz_test_state I would prefer this to go under /sys/kernel/debug/ as this is clearly a debug/test feature. > +Date: July 2019 > +KernelVersion: 5.2 > +Contact: Branden Bonaby <brandonbonaby94@gmail.com> > +Description: Fuzz testing status of a vmbus device, whether its in an ON > + state or a OFF state > +Users: Debugging tools > + > +What: /sys/bus/vmbus/devices/<UUID>/fuzz_test_buffer_delay > +Date: July 2019 > +KernelVersion: 5.2 > +Contact: Branden Bonaby <brandonbonaby94@gmail.com> > +Description: Fuzz testing buffer delay value between 0 - 1000 > +Users: Debugging tools > + > +What: /sys/bus/vmbus/devices/<UUID>/fuzz_test_message_delay > +Date: July 2019 > +KernelVersion: 5.2 > +Contact: Branden Bonaby <brandonbonaby94@gmail.com> > +Description: Fuzz testing message delay value between 0 - 1000 > +Users: Debugging tools > diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c > index 92b1874b3eb3..0c71fd66ef81 100644 > --- a/drivers/hv/vmbus_drv.c > +++ b/drivers/hv/vmbus_drv.c > @@ -22,7 +22,7 @@ > #include <linux/clockchips.h> > #include <linux/cpu.h> > #include <linux/sched/task_stack.h> > - > +#include <linux/kernel.h> > #include <asm/mshyperv.h> > #include <linux/notifier.h> > #include <linux/ptrace.h> > @@ -584,6 +584,98 @@ static ssize_t driver_override_show(struct device *dev, > } > static DEVICE_ATTR_RW(driver_override); > > +static ssize_t fuzz_test_state_store(struct device *dev, > + struct device_attribute *attr, > + const char *buf, size_t count) > +{ > + struct hv_device *hv_dev = device_to_hv_device(dev); > + struct vmbus_channel *channel = hv_dev->channel; > + int state; > + int delay = kstrtoint(buf, 0, &state); > + > + if (delay) > + return count; > + if (state) > + channel->fuzz_testing_state = 1; > + else > + channel->fuzz_testing_state = 0; > + return count; > +} > + > +static ssize_t fuzz_test_state_show(struct device *dev, > + struct device_attribute *dev_attr, > + char *buf) > +{ > + struct hv_device *hv_dev = device_to_hv_device(dev); > + struct vmbus_channel *channel = hv_dev->channel; > + > + return sprintf(buf, "%u\n", channel->fuzz_testing_state); > +} > +static DEVICE_ATTR_RW(fuzz_test_state); > + > +static ssize_t fuzz_test_buffer_delay_store(struct device *dev, > + struct device_attribute *attr, > + const char *buf, size_t count) > +{ > + struct hv_device *hv_dev = device_to_hv_device(dev); > + struct vmbus_channel *channel = hv_dev->channel; > + int val; > + int delay = kstrtoint(buf, 0, &val); > + > + if (delay) > + return count; > + if (val >= 1 && val <= 1000) > + channel->fuzz_testing_buffer_delay = val; > + /*Best to not use else statement here since we want > + *the buffer delay to remain the same if val > 1000 > + */ > + else if (val <= 0) > + channel->fuzz_testing_buffer_delay = 0; > + return count; > +} > + > +static ssize_t fuzz_test_buffer_delay_show(struct device *dev, > + struct device_attribute *dev_attr, > + char *buf) > +{ > + struct hv_device *hv_dev = device_to_hv_device(dev); > + struct vmbus_channel *channel = hv_dev->channel; > + > + return sprintf(buf, "%u\n", channel->fuzz_testing_buffer_delay); > +} > +static DEVICE_ATTR_RW(fuzz_test_buffer_delay); > + > +static ssize_t fuzz_test_message_delay_store(struct device *dev, > + struct device_attribute *attr, > + const char *buf, size_t count) > +{ > + struct hv_device *hv_dev = device_to_hv_device(dev); > + struct vmbus_channel *channel = hv_dev->channel; > + int val; > + int delay = kstrtoint(buf, 0, &val); > + > + if (delay) > + return count; > + if (val >= 1 && val <= 1000) > + channel->fuzz_testing_message_delay = val; > + /*Best to not use else statement here since we want > + *the message delay to remain the same if val > 1000 > + */ > + else if (val <= 0) > + channel->fuzz_testing_message_delay = 0; > + return count; > +} > + > +static ssize_t fuzz_test_message_delay_show(struct device *dev, > + struct device_attribute *dev_attr, > + char *buf) > +{ > + struct hv_device *hv_dev = device_to_hv_device(dev); > + struct vmbus_channel *channel = hv_dev->channel; > + > + return sprintf(buf, "%u\n", channel->fuzz_testing_message_delay); > +} > +static DEVICE_ATTR_RW(fuzz_test_message_delay); > /* Set up per device attributes in /sys/bus/vmbus/devices/<bus device> */ > static struct attribute *vmbus_dev_attrs[] = { > &dev_attr_id.attr, > @@ -615,6 +707,9 @@ static struct attribute *vmbus_dev_attrs[] = { > &dev_attr_vendor.attr, > &dev_attr_device.attr, > &dev_attr_driver_override.attr, > + &dev_attr_fuzz_test_state.attr, > + &dev_attr_fuzz_test_buffer_delay.attr, > + &dev_attr_fuzz_test_message_delay.attr, > NULL, > }; -- Vitaly ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drivers: hv: vmbus: add fuzz test attributes to sysfs 2019-08-02 7:34 ` Vitaly Kuznetsov @ 2019-08-02 19:02 ` Branden Bonaby 0 siblings, 0 replies; 10+ messages in thread From: Branden Bonaby @ 2019-08-02 19:02 UTC (permalink / raw) To: Vitaly Kuznetsov Cc: linux-hyperv, linux-kernel, kys, haiyangz, sthemmin, sashal On Fri, Aug 02, 2019 at 09:34:40AM +0200, Vitaly Kuznetsov wrote: > Branden Bonaby <brandonbonaby94@gmail.com> writes: > > > Expose the test parameters as part of the sysfs channel attributes. > > We will control the testing state via these attributes. > > > > Signed-off-by: Branden Bonaby <brandonbonaby94@gmail.com> > > --- > > Documentation/ABI/stable/sysfs-bus-vmbus | 22 ++++++ > > drivers/hv/vmbus_drv.c | 97 +++++++++++++++++++++++- > > 2 files changed, 118 insertions(+), 1 deletion(-) > > > > diff --git a/Documentation/ABI/stable/sysfs-bus-vmbus b/Documentation/ABI/stable/sysfs-bus-vmbus > > index 8e8d167eca31..239fcb6fdc75 100644 > > --- a/Documentation/ABI/stable/sysfs-bus-vmbus > > +++ b/Documentation/ABI/stable/sysfs-bus-vmbus > > @@ -185,3 +185,25 @@ Contact: Michael Kelley <mikelley@microsoft.com> > > Description: Total number of write operations that encountered an outbound > > ring buffer full condition > > Users: Debugging tools > > + > > +What: /sys/bus/vmbus/devices/<UUID>/fuzz_test_state > > I would prefer this to go under /sys/kernel/debug/ as this is clearly a > debug/test feature. > -- > Vitaly Alright, it is testing so I see what you mean and why the code in this patch should go in debugfs. Will fix that and resend. Branden Bonaby ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 3/3] tools: hv: add vmbus testing tool 2019-08-01 20:00 [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices Branden Bonaby 2019-08-01 20:00 ` [PATCH 1/3] drivers: hv: vmbus: Introduce latency testing Branden Bonaby 2019-08-01 20:00 ` [PATCH 2/3] drivers: hv: vmbus: add fuzz test attributes to sysfs Branden Bonaby @ 2019-08-01 20:00 ` Branden Bonaby 2019-08-02 7:30 ` [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices Vitaly Kuznetsov 3 siblings, 0 replies; 10+ messages in thread From: Branden Bonaby @ 2019-08-01 20:00 UTC (permalink / raw) To: kys, haiyangz, sthemmin, sashal Cc: Branden Bonaby, linux-hyperv, linux-kernel This is a userspace tool to drive the testing. Currently it supports introducing user specified delay in the host to guest communication path on a per-channel basis. Signed-off-by: Branden Bonaby <brandonbonaby94@gmail.com> --- tools/hv/vmbus_testing | 326 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 326 insertions(+) create mode 100644 tools/hv/vmbus_testing diff --git a/tools/hv/vmbus_testing b/tools/hv/vmbus_testing new file mode 100644 index 000000000000..8f5cd997952b --- /dev/null +++ b/tools/hv/vmbus_testing @@ -0,0 +1,326 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# +#Program to allow users to fuzz test Hyper-V drivers +#by interfacing with Hyper-V sysfs directories +#author: Branden Bonaby <brandonbonaby94@gmail.com> + +import os +import cmd +import argparse +from collections import defaultdict +from argparse import RawDescriptionHelpFormatter + +#sysfs paths for vmbus must exist (same as in lsvmbus) +vmbus_sys_path = '/sys/bus/vmbus/devices' +if not os.path.isdir(vmbus_sys_path): + print("{} doesn't exist: exiting...".format(vmbus_sys_path)) + exit(-1) + +#Do not change unless, you change the sysfs attributes +#in "/sys/bus/vmbus/devices/<UUID>/". All fuzz testing +#attributes will start with "fuzz_test". +pathlen = len(vmbus_sys_path) +fuzz_state_location = "fuzz_test_state" +fuzz_states = {0 : "Disable", 1 : "Enable"} +fuzz_methods ={1 : "Delay_testing"} +fuzz_delay_types = {1 : "fuzz_test_buffer_delay", 2 :"fuzz_test_message_delay"} + +def parse_args(): + parser = argparse.ArgumentParser(description = "vmbus_testing "\ + "[-s] [0|1] [-q] [-p] <sysfs-path>\n""vmbus_testing [-s]"\ + " [0|1] [-q] [-p] <sysfs-path> delay [-d] [val] [val] [-E|-D]\n" + "vmbus_testing [-q] disable-all\n" + "vmbus_testing [-q] view [-v|-V]\n"\ + "vmbus_testing --version", + epilog = "Current testing options {}".format(fuzz_methods), + prog = 'vmbus_testing', + formatter_class = RawDescriptionHelpFormatter) + subparsers = parser.add_subparsers(dest="action") + parser.add_argument('--version', action='version',\ + version = '%(prog)s 1.0') + parser.add_argument("-q","--quiet",action = "store_true",\ + help = "silence none important test messages") + parser.add_argument("-s","--state",metavar = "", type = int,\ + choices = range(0,2),\ + help = "Turn testing ON or OFF for a single device."\ + " The value (1) will turn testing ON. The value"\ + " of (0) will turn testing OFF with the default set"\ + " to (0).") + parser.add_argument("-p","--path", metavar = "",\ + help = "Refers to the sysfs path to a vmbus device." + " If the path is not a valid path to a sysfs device,"\ + " the program will exit. The path must be the"\ + " absolute path; use the lsvmbus command to find"\ + " the path.") + parser_delay = subparsers.add_parser("delay",\ + help = "Delay buffer/message reads in microseconds.", + description = "vmbus_testing -s [0|1] [-q] -p "\ + "<sysfs-path> delay -d "\ + "[buffer-delay-value] [message-delay-value]\n" + "vmbus_testing [-q] delay [buffer-delay-value] "\ + "[message-delay-value] -E\n" + "vmbus_testing [-q] delay [buffer-delay-value] "\ + "[message-delay-value] -D", + formatter_class = RawDescriptionHelpFormatter) + delay_group = parser_delay.add_mutually_exclusive_group() + delay_group.add_argument("-E","--en-all-delay", action = "store_true",\ + help = "Enable Buffer/Message Delay testing on ALL"\ + " devices. Use -d option with this to set the values"\ + " for both the buffer delay and the message delay. No"\ + " value can be (0) or less than (-1). If testing is"\ + " disabled on a device prior to running this command,"\ + " testing will be enabled on the device as a result"\ + " of this command.") + delay_group.add_argument("-D","--dis-all-delay", action="store_true",\ + help = "Disable Buffer/Message delay testing on ALL"\ + " devices. A value equal to (-1) will keep the"\ + " current delay value, and a value equal to (0) will"\ + " remove delay testing for the specfied delay column."\ + " only values (-1) and (0) will be accepted but at"\ + " least one value must be a (0) or a (-1).") + parser_delay.add_argument("-d","--delay-time", metavar="", nargs=2,\ + type = check_range, default =[0,0], required = (True),\ + help = "Buffer/message delay time. A value of (0) will"\ + "disable delay testing on the specified delay column,"\ + " while a value of (-1) will ignore the specfied"\ + " delay column. The default values are [0] & [0]."\ + " The first column represents the buffer delay value"\ + " and the second represents the message delay value."\ + " Value constraints: -1 <= value <= 1000.") + parser_dis_all = subparsers.add_parser("disable-all",\ + help = "Disable ALL testing on all vmbus devices.", + description = "vmbus_testing disable-all", + formatter_class = RawDescriptionHelpFormatter) + parser_view = subparsers.add_parser("view",\ + help = "View testing on vmbus devices.", + description = "vmbus_testing view -V\n" + "vmbus_testing -p <sysfs-path> view -v", + formatter_class = RawDescriptionHelpFormatter) + view_group = parser_view.add_mutually_exclusive_group() + view_group.add_argument("-V","--view-all-states",action = "store_true",\ + help = "View the test status for all vmbus devices.") + view_group.add_argument("-v","--view-single-device",\ + action = "store_true",help = "View test values for a"\ + " single vmbus device.") + + return parser.parse_args() + +#value checking for range checking input in parser +def check_range(arg1): + try: + val = int(arg1) + except ValueError as err: + raise argparse.ArgumentTypeError(str(err)) + if val < -1 or val > 1000: + message = ("\n\nError, Expected -1 <= value <= 1000, got value"\ + " {}\n").format(val) + raise argparse.ArgumentTypeError(message) + return val + +def main(): + try: + dev_list = [] + for dir in os.listdir(vmbus_sys_path):\ + dev_list.append(os.path.join(vmbus_sys_path,dir)) + #key value, pairs + #key = sysfs device path + #value = list of fuzz testing attributes. + device_and_files = defaultdict(list) + for dev in dev_list: + for f in os.listdir(dev): + if (f.startswith("fuzz_test")): + device_and_files[dev].append(f) + + device_and_files.default_factory = None + args = parse_args() + path = args.path + state = args.state + quiet = args.quiet + if (not quiet): + print("*** Use lsvmbus to get vmbus device"\ + " information.*** ") + if (state is not None and validate_args_path(path,dev_list)): + if (state is not get_test_state(path)): + change_test_state(path,quiet) + state = get_test_state(path) + if (state is 0 and path is not None): + disable_testing_single_device(path,0,quiet) + return + #Use subparsers as the key for different fuzz testing methods + if (args.action == "delay"): + delay = args.delay_time + if (validate_delay_values(args,delay)): + delay_test_all_devices(dev_list,delay,quiet) + elif (validate_args_path(path,dev_list)): + if(get_test_state(path) is 1): + delay_test_store(path,delay,quiet) + return + print("device testing OFF, use -s 1 to turn ON") + elif (args.action == "disable-all"): + disable_all_testing(dev_list,quiet) + elif (args.action == "view"): + if (args.view_all_states): + all_devices_test_status(dev_list) + elif (args.view_single_device): + if (validate_args_path(path,dev_list)): + device_test_values(device_and_files,\ + path) + return + print("Error,(check path) usage: -p [path]"\ + " view -v") + except AttributeError: + print("check usage, 1 or more elements not provided") + exit(-1) + +#Validate delay values to make sure they are acceptable to +#to either enable all delays on a device or disable all +#delays on a device +def validate_delay_values(args,delay): + if (args.en_all_delay): + for i in delay: + if (i < -1 or i == 0): + print("\nError, Values must be"\ + " equal to -1 or be > 0, use"\ + " -d option") + exit(-1) + return True + elif (args.dis_all_delay): + for i in delay: + if (i < -1 or i > 0): + print("\nError, at least 1 value" + " is not a (0) or a (-1)") + exit(-1) + return True + else: + return False + + +#Validate argument path +def validate_args_path(path,dev_list): + if (path in dev_list): + return True + else: + return False + +#display Testing status of single device +def device_test_values(device_and_files,path): + for test in device_and_files.get(path): + print("{}".format(test), end = '') + print((" value = {}")\ + .format(read_test_files(os.path.join(path,test)))) + +#display Testing state of devices +def all_devices_test_status(dev_list): + for device in dev_list: + if (get_test_state(device) is 1): + print("Testing = ON for: {}".format(device.split("/")[5])) + else: + print("Testing = OFF for: {}".format(device.split("/")[5])) + +#read the vmbus device files, path must be absolute path before calling +def read_test_files(path): + try: + with open(path,"r") as f: + state = f.readline().replace("\n","") + return int(state) + + except IOError as e: + errno, strerror = e.args + print("I/O error({0}): {1} on file {2}"\ + .format(errno,strerror,path)) + exit(-1) + except ValueError: + print ("Element to int conversion error in: \n{}".format(path)) + exit(-1) + +#writing to vmbus device files, path must be absolute path before calling +def write_test_files(path,value): + try: + with open(path,"w") as f: + f.write("{}".format(value)) + except IOError as e: + errno, strerror = e.args + print("I/O error({0}): {1} on file {2}"\ + .format(errno,strerror,path)) + exit(-1) + +#change testing state of device +def change_test_state(device,quiet): + state_path = os.path.join(device,fuzz_state_location) + if (get_test_state(device) is 0): + write_test_files(state_path,1) + if (not quiet): + print("Testing = ON for device: {}"\ + .format(state_path.split("/")[5])) + else: + write_test_files(state_path,0) + if (not quiet): + print("Testing = OFF for device: {}"\ + .format(state_path.split("/")[5])) + +#get testing state of device +def get_test_state(device): + #state == 1 - test = ON + #state == 0 - test = OFF + return read_test_files(os.path.join(device,fuzz_state_location)) + +#Enter 1 - 1000 microseconds, into a single device using the +#fuzz_test_buffer_delay and fuzz_test_message_delay +#sysfs attributes +def delay_test_store(device,delay_length,quiet): + + try: + # delay[0]- buffer delay, delay[1]- message delay + buff_test = os.path.join(device,fuzz_delay_types.get(1)) + mess_test = os.path.join(device,fuzz_delay_types.get(2)) + + if (delay_length[0] >= 0): + write_test_files(buff_test,delay_length[0]) + if (delay_length[1] >= 0): + write_test_files(mess_test,delay_length[1]) + if (not quiet): + print("Buffer delay testing = {} for: {}"\ + .format(read_test_files(buff_test),\ + buff_test.split("/")[5])) + print("Message delay testing = {} for: {}"\ + .format(read_test_files(mess_test),\ + mess_test.split("/")[5])) + except IOError as e: + errno, strerror = e.args + print("I/O error({0}): {1} on files {2}{3}"\ + .format(errno,strerror,buff_test,mess_test)) + exit(-1) + +#enabling/disabling delay testing on all devices +def delay_test_all_devices(dev_list,delay,quiet): + + for device in (dev_list): + if (get_test_state(device) is 0): + change_test_state(device,quiet) + delay_test_store(device,delay,quiet) + +#disabling testing on single device +def disable_testing_single_device(device,test_type,quiet): + + #test_type represents corresponding key + #delay method in delay_methods dict. + #special type 0 , used to disable all + #testing on SINGLE device. + + if (test_type is 1 or test_type is 0): + #disable list [buffer,message] + disable_delay = [0,0] + if (get_test_state(device) is 1): + change_test_state(device,quiet) + delay_test_store(device,disable_delay,quiet) + +#disabling testing on ALL devices +def disable_all_testing(dev_list,quiet): + + #delay disable list [buffer,message] + for device in dev_list: + disable_testing_single_device(device,0,quiet) + +if __name__ == "__main__": + main() -- 2.17.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices 2019-08-01 20:00 [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices Branden Bonaby ` (2 preceding siblings ...) 2019-08-01 20:00 ` [PATCH 3/3] tools: hv: add vmbus testing tool Branden Bonaby @ 2019-08-02 7:30 ` Vitaly Kuznetsov 2019-08-02 19:02 ` Branden Bonaby 3 siblings, 1 reply; 10+ messages in thread From: Vitaly Kuznetsov @ 2019-08-02 7:30 UTC (permalink / raw) To: Branden Bonaby Cc: linux-hyperv, linux-kernel, kys, haiyangz, sthemmin, sashal Branden Bonaby <brandonbonaby94@gmail.com> writes: > This patchset introduces a testing framework for Hyper-V drivers. > This framework allows us to introduce delays in the packet receive > path on a per-device basis. While the current code only supports > introducing arbitrary delays in the host/guest communication path, > we intend to expand this to support error injection in the future. > > Branden Bonaby (3): > drivers: hv: vmbus: Introduce latency testing > drivers: hv: vmbus: add fuzz test attributes to sysfs > tools: hv: add vmbus testing tool > > Documentation/ABI/stable/sysfs-bus-vmbus | 22 ++ > drivers/hv/connection.c | 5 + > drivers/hv/ring_buffer.c | 10 + > drivers/hv/vmbus_drv.c | 97 ++++++- Can we have something like CONFIG_HYPERV_TESTING and put this new code under #ifdef? > include/linux/hyperv.h | 14 + > tools/hv/vmbus_testing | 326 +++++++++++++++++++++++ > 6 files changed, 473 insertions(+), 1 deletion(-) > create mode 100644 tools/hv/vmbus_testing -- Vitaly ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices 2019-08-02 7:30 ` [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices Vitaly Kuznetsov @ 2019-08-02 19:02 ` Branden Bonaby 0 siblings, 0 replies; 10+ messages in thread From: Branden Bonaby @ 2019-08-02 19:02 UTC (permalink / raw) To: Vitaly Kuznetsov Cc: linux-hyperv, linux-kernel, kys, haiyangz, sthemmin, sashal On Fri, Aug 02, 2019 at 09:30:18AM +0200, Vitaly Kuznetsov wrote: > Branden Bonaby <brandonbonaby94@gmail.com> writes: > > > This patchset introduces a testing framework for Hyper-V drivers. > > This framework allows us to introduce delays in the packet receive > > path on a per-device basis. While the current code only supports > > introducing arbitrary delays in the host/guest communication path, > > we intend to expand this to support error injection in the future. > > > > Branden Bonaby (3): > > drivers: hv: vmbus: Introduce latency testing > > drivers: hv: vmbus: add fuzz test attributes to sysfs > > tools: hv: add vmbus testing tool > > > > Documentation/ABI/stable/sysfs-bus-vmbus | 22 ++ > > drivers/hv/connection.c | 5 + > > drivers/hv/ring_buffer.c | 10 + > > drivers/hv/vmbus_drv.c | 97 ++++++- > > Can we have something like CONFIG_HYPERV_TESTING and put this new > code under #ifdef? > > > include/linux/hyperv.h | 14 + > > tools/hv/vmbus_testing | 326 +++++++++++++++++++++++ > > 6 files changed, 473 insertions(+), 1 deletion(-) > > create mode 100644 tools/hv/vmbus_testing > > -- > Vitaly You're right, it would be better to do it that way with ifdef's. Will edit my patches and resend. Branden Bonaby ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2019-08-02 19:02 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-08-01 20:00 [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices Branden Bonaby 2019-08-01 20:00 ` [PATCH 1/3] drivers: hv: vmbus: Introduce latency testing Branden Bonaby 2019-08-02 7:32 ` Vitaly Kuznetsov 2019-08-02 19:02 ` Branden Bonaby 2019-08-01 20:00 ` [PATCH 2/3] drivers: hv: vmbus: add fuzz test attributes to sysfs Branden Bonaby 2019-08-02 7:34 ` Vitaly Kuznetsov 2019-08-02 19:02 ` Branden Bonaby 2019-08-01 20:00 ` [PATCH 3/3] tools: hv: add vmbus testing tool Branden Bonaby 2019-08-02 7:30 ` [PATCH 0/3] hv: vmbus: add fuzz testing to hv devices Vitaly Kuznetsov 2019-08-02 19:02 ` Branden Bonaby
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).